Safenetwork Git Portal Discussion

Apps
1

This topic began on 18th October 2020, a few days after I first started thinking about how to build a decentralised alternative for github on the Safe Network.

It has progressed rapily, from some outline ideas on implementation, and discussions to help envisage a “Safe-git” UI. Within two months the essential concept was proven and now has the beginnings of a demonstration product that looks the part.

We’re now moving from the early vision preserved below, to planning to build a usable product. Everyone is welcome to contribute ideas, feedback suggestions and messages of encouragement.

Please read on if you would like to get more involved, contribute general points here or open an issue on the github repo for specific issues or more detailed discussions: happyborg/p2p-git-portal-poc

Safe-git UI Vision

past-and-future

Git Portal like github/gitlab/gitea

A no-server no-backend (ideally in Rust) with web frontend, e.g. using Svelte.

Git Portal Features

  • operate directly on repositories in peer-to-peer storage
  • clone, upload, fetch/push between p2p storage, local device and remotes (some by CLI)
  • extensions (Issues, PRs etc.) supported without interfering with repository commits

Git Portal UI

Windows, MacOS and Linux

  • Static HTML Web UI and “git extra” CLI for viewing, creating, editing etc of Issues
  • Cross platform Desktop GUI (e.g. using Tauri)

Progress Update 24th December 2020

What an amazing few weeks. In this update we move from concept pretty much proven to having a brand new look, courtesy of new contributor @Miguelo.

Status Summary

We have proven the essentials of a statically hosted (no-server no-backend) peer-to-peer alternative to github. A Git Portal including features like github issues for bug tracking, discussion etc.

This has been achieved surprisingly quickly - I began discussion on this on 18th October 2020, so its taken slightly over two months to get here. To achieve this we used as much existing code as we could find, notably the excellent Go libraries git-bug and go-git.

Some of this will be replaced, but I envisage continuing to use the git-bug CLI and file format alongside Git Portal as a p2p web UI. We have a good relationship with the author of git-bug and it would be great to continue and build on that, supporting his work to improve and extend its features, with Pull Requests for example.

Now for details of the latest update.

:christmas_tree: Git Portal Christmas Update :christmas_tree:

Git Portal got a new look!

Here it is, a working proof-of-concept for a decentralised alternative to GitHub running from static storage, and later Safe Network (or any static hosting or p2p storage such as Solid or IPFS etc). It isn’t feature complete yet so you can’t create and edit issues in the UI, but you can see one’s which have been created automatically for “New” repositories on the Issues tab.

gitportal-poc-screenshot
gitportal-poc-screenshot1344×1112 137 KB

@Miguelo is busy with Christmas so has asked me to do the announcement, but the credit for this fantastic UI is his. In a very short time he’s transformed the look and operation of the UI into something that really looks the part as well as demonstrating the key features needed. To get this look he used Tailwind CSS building on the Svelte framework with Go/Wasm backend, plus his own customisations.

Can you see which repository he’s cloned for display, and a who gets mentioned in the Commits tab! Nice touch :smiley:

Thank you @Nigel for your vocal support and for providing the Safe Git logo which appears at the top left. :pray:

Try Git Portal

Anyone can try the demo out by visiting http://gitch.happybeing.com/. Be patient for the first time loading takes a little while (you may need to refresh if you’ve visited this link before). Once the red message goes away you are set.

  • Create Repository: To start, enter a repository name under “Create repository” and click “Create” to make the new repository. This is instant, and let’s you browse the files in the .git directory, or view the commits (none yet) or Issues (a few are created automatically).

  • Clone Repository: To clone from github you need to use a proxy as described in the README. If you just want to do a quick test, click on the gear :gear: icon and enter the following proxy URL: https://cors.isomorphic-git.org/. That’s handy for testing but not for regular use. Then click “Clone” and wait while it fetches the data. For large repositories this can take quite a while, so one has been entered for you to try, which should complete in under a minute.

  • Upload Repository: You can choose a repository directory to upload, but be aware this will fail for large repositories at the moment. It works for repositories up to about 2,000 files, but fails if they have 10,000 (including all the commits etc in .git). Again, uploading can take a while so be patient!

Issues

If you think something went wrong or just want to see what’s going on under the hood, open the browser console (Ctrl-Shift-I in Firefox or Chrome/Chromium). Any issues, please report here or on happyborg/github.

Help!

18th March 2021: I’ve paused my work on Git Portal and am considering whether I want to continue alone on such a large project, or on code at all.

With perfect timing MaidSafe have announced substantial funding is available to third party developers. So if you have relevant skills and might want to contribute or even take the lead over time, let’s talk.

See Announcing the BambooGarden Fund and my note here.

This a chance contribute to something you want to happen by either using what you know or learning something new. We’ll help you get going or with learning as you go.

It’s Not Over Yet

There are many features not implemented, but I’ve shown that they should be possible, though there are limitations to the approach.

The main difficulty is the size of the bundle with a 12MB Go runtime, the current demo is 16MB which is a lot for a web app. I’m looking into different ways of around this for the web, and it will be possible to bundle this into a desktop app using Tauri.

Next: add git-bug

  • get compiling to web assembly :white_check_mark:
  • make into a package :white_check_mark:
  • modify to use go-billy filesystem for git and git-bug operations :white_check_mark:
  • integrate into the git portal :white_check_mark:
  • add a simple file viewer :white_check_mark:
  • get help with UI styles and UX / API design :white_check_mark:
  • NEW: provide UI for creating and editing issues
  • NEW: refine UX closer to the github experience
  • NEW: evaluate product implementation options

For more, see:

31 Likes
2

Proof of Concept: gitex

gx extract [--type="issues"] [--overwrite] <user>/<repo>
Extract ‘github extras’ from github repository to .gitex/

gx clone <user>/<repo> [path-of-clone]
Clone github repo and populate .gitex subdirectory.

gx fork <user>/<repo> <path-to-fork> [path-of-clone]
Maybe? Clones, creates a new repo, sets origin and upstream remotes in the clone,
and pushes clone to the new repo (origin).

gx issues [--json]
List all issues

gx issues new

gx issues print [--json] <issue-number>
gx issues close <issue-number>

gx comment add <issue-number> [<text>]
gx comment print [--json] <issue-number> <comment-number>
gx comment replace <issue-number> <comment-number> [<text>]

gx sync
gx issues sync
Get latest communal extras for current repo, drop any local duplicates.

Note: the first letter of issues, comment and pull-request can be used instead of the full word. For example: gx i new is equivalent to gx issues new.

Proof of Concept: git UI

Location: (drop-down of recent URIs / filesystem paths)

Github URI

User - github-repos list - clone to path
Repo - clone to path

Local filesystem path

Local directory: list repo subdirectories
Local repo directory: Tabs: README / Issues / PRs

Local repo directory Issues tab: github style issues list - visit / open
Local repo directory Issue page: github style issue page - comment / edit / delete

Local repo directory PRs tab: TODO placeholder

11 Likes
3

Maybe in a .gitex directory at the root of the repo?

eg

.gitex/issues
.gitex/wiki
.gitex/releases
...

This could contain pretty-printed json files, one file per issue / wiki page / release note etc? If gitex specifies a json-schema then it would be simple to ensure any additional data has a certain form and it gives the UI some confidence of what data to base their views on.

I feel like keeping it in the repo is useful, since it allows this idea to work independent of Safe Network. Don’t get me wrong it’s good business to build a moat and get people invested in Safe Network, but for interoperability it seems neat to have it in the repo. Although maybe a .gitex direction is not interoperable with, say, github which could create confusion.

One observation I had on github (which might impact the .gitex subdirectory structure) is issue ids and pull request ids are linked. Eg if you have /issue/44 you can’t have pull/44 - the underlying id comes from the same source. Pull Requests are just issues-with-attached-code. So that’s a small leak of the underlying abstractions used by github. No need to have an issues schema and a pulls schema, they’re the same underlying structure.

7 Likes
4

Thanks @mav, useful thoughts and great tip on the issue/PR structure. Those are the first things I’d implement so it’s helpful to know.

I was planning to start with a .gitex/ subdirectory and JSON files for issues (starting with what the github APIs deliver) but not have its contents under git control as I thought it would complicate things to have commits of things like issues. Any thoughts on that?

I haven’t thought any of this through, I drew the diagram today and then wrote down some ideas. So it’s very early stage.

I wanted something useful to do which I can progress while awaiting the testnets and APIs. I’ll start with gitex and then make a simple Svelte based Tauri app to do stuff with the .gitex/ data, and try to create a core library that can be used to create a Safe-git terminal UI.

I saw and experimental ‘modern’ git CLI replacement called bit earlier which might give inspiration for this, but for now I’ll keep things simple.

9 Likes
5

Mmmm. Yeah. This would pollute the git history. Good point.

I wonder if something like how git tags works is a useful concept to consider? Tags are a metadata in the repo that are related to the commit history but is not adding new history. I wonder if tags might have conceptual relation to gitex type features?

Maybe a hair-brained idea but what if gitex is a branch based off the initial commit? gitex command could be an alias for roughly git checkout gitex; do gitex stuff and autocommit; git checkout $PREV_BRANCH … ? Seems a bit hacky.

Overall the gitex concept is great and I can see it being very popular if it’s well implemented. Having a standardized non-vendor-lock-in way to do the ‘social’ aspects of code that made github so successful would be pretty amazing.

6 Likes
6

This is incredible @happybeing … thank you for all the hard work and thought you put into the network. I know that, like the Maid team, you will be rewarded greatly down the track for your efforts.

13 Likes
7

Thanks @TylerAbeoJordan, I do this for fun but part of that is being appreciated. It wouldn’t be much fun if nobody thought what I was doing was worthwhile!

@mav your input it’s invaluable as usual. I don’t know much about guy under the hood, so not sure if I’ll come up with something implemented that will, at least from the start. First step is proof of concept, though so no pressure, but if you want to keep thinking up interesting ideas like this I would appreciate that.

On the tricky parts of social. Firstly, there’s an element of collaboration with issues, so I was thinking to handle this there would be the remote repo view, and your local repo view, which includes your most recent pull of the remote (.gitex/issues) plus anything you’ve done locally, such as create an issue. So you will see a new issue you made or a comment you added, immediately but only visible to you.

At the point you made the issue, perhaps this is appended to an issues data structure (eg Sequence?) for the given remote, which anyone can write to. But when you or anyone ‘pulls’ you all get the same view (whatever is in the Sequence), and locally gitex can drop any duplicates you created locally. So we have a separate mechanism from git push/pull that works in parallel, managed by gitex.

On the other social aspects, maybe a similar approach will work for that, but I’ve not really thought much about it yet. Maybe we’re need a more general social API/library and to tap into that. It’s crucial as you say, a big area so I’ll try to think about it in the background. Again appreciate any thoughts on this.

A Question for Everyone

Are there any things which github does badly, or doesn’t do but could?

Initially I’m going to try replicating familiar github features, but at some point we want to break free and if there are some nice innovations that would be very cool. Even just in concept, we can document things for later.

7 Likes
8

/brainstorming mode on

Some form of documentation links perhaps. I See 2 forms here (possibly 3).

  1. Code documentation (will differ per language though)
  2. Research docs to explain what/why the library/app is implementing (so background to the design)

If 2 could be linked from the code in an easy way then it might be helpful (i.e. this method implements “this part of this doc”.

The other thing could be a real wiki, but using git/PR’s etc. to update it. So updates to the wiki need agreement in the team. This might be 2 above actually. Github tries to do a wiki, but tbh it’s not that great. I think there is a cleaner way and one that will help code reviewers / potential users to clearly understand the reason for the library as well as how it’s implemented.

Another thing is linked crates, so like a dependency graph (rust does this well, so may be language constrained). But not so much actual dependencies, more like, this crate has these deps, it is meant for this project which has other purposes (vault / client for us). Maybe that is a project crate though?

7 Likes
9

Warning: to create maximum confusion I’ve renamed the new CLI to gx (from gitex) mainly to save characters, but left the extras subdirectory as .gitex for now, though I’m inclined to shorten that too.

I did some searching on gitex and gx and there are a couple of git related things called gitex and ‘git extensions’ (plus a well SEO’d annual GITEX technology conference). Whereas I found only one thing called gx which is a package manager written in go that was quite popular, but describes itself as alpha quality and has had no gh activity for 2 years, (and the author is busy with Filecoin Lotus!)

3 Likes
10

Open source is awesome.

I stumbled on git-dit via StackExchange and I shall look into using this instead of creating gx at least for the time being.

It’s written in Rust which is always a good sign, and @mav it sounds suspiciously like your ‘hacky’ suggestion which may well be the way to go. It at least gives me a quick way of trying out the general idea without having to build a CLI to start with.

@mav If you have time to read this short discussion of how it works I’d be grateful for your comments. I don’t know what ‘patch sets’ are for example, or what he means by an ‘empty commit’, so I don’t understand the effects of the implementation choices he’s made.

EDIT: browsing the issues I note:

  • they don’t commit files (so I don’t yet know how issue data is stored or shared)
  • they use an email style model (but I’m taking this to mean for their threading model rather than actual emails at this point)
  • they use MIME encoding e.g. to handle file embedding
4 Likes
11

Another from the same StackExchange is sit, a serverless modular system for sharing and querying information with modules for different workflows including a sit issue tracker. Also Rust, also dormant for a couple of years but worth looking into.

3 Likes
12

FYI -

3 Likes
13

A slightly different approach is being taken by radicle using git to provide efficient propagation of information, by using git packfiles to avoid the overhead they found using a storage system (IPFS).

radicle is a protocol for discovering peers, using git and it’s packfiles to scale the propagation of changes along with gossip.

I’m not sure if there’s anything for me or more generally Safe, to learn from their approach, moving from replicated storage (IPFS) to git packfiles but it’s an interesting comment they make. I don’t have time to look in depth at all these systems though, so if others do it would be interesting to hear back.

2 Likes
14

Search is not that great for code. I find grep a lot more useful and am regularly frustrated by github code search. Github will only show the top N results in each file, so if you’re looking for some specific use of a variable maybe it’s ‘below the fold’ which you can never know from the search results.

Another thing they don’t do well is data export / import for metadata such as issues etc (hmmm sounds like exactly what this topic is addressing what a coincidence). The reason I say it explicitly here is if gx is designed specifically to enable easy import/export to/from github gitlab etc then it has a much stronger chance of catching on. Being aware of the way github does things a) saves a lot of work/mistakes by using prior learning from github and b) maximizes possible traction by giving a relatively simple path for adoption. The examples you link seem to want to operate in a vacuum and I feel like that’s a mistake because they miss out on all the potential network effects already in play from the big web git providers.

It sounds like git-dit have a fairly atomic approach where each comment / action is a unit (like sequential emails are individual), where I was thinking more like each issue (containing many actions) would be a unit with git-style updates to that unit for each action taken. Not sure about the difference in practice, maybe the flexibility of everything separate is better, or maybe it introduces unnecessary complexity? Perhaps a little too far down in the weeds for the current stage of thinking.

Really cool to see so much prior work in this space.

Since it was mentioned but not linked, here’s bit (worth looking at their inspiration too):

3 Likes
15

gitx

Naming wise, the shorter ‘gitx’ pronounced the same as ‘git-ex’ would also be nice.

3 Likes
16

Several good points @mav, can you say more about the or support for export/import. I only looked briefly but found export of issues very easy and it looked comprehensive. Using curl you can quickly get a list of issues, and the issues themselves as JSON files, which seemed good.

I agree with your take that extending github is a better first step. Kindof using Microsoft’s own death grip against them (embrace, extend, extinguish) :face_with_symbols_over_mouth:.

So anyway, yes we need interoperability. Good point and thanks for looking.

Thanks @jlpell, ‘gitx’ is nice.

I’ve built git-dit so tomorrow I’m going to play with it and maybe start work on a UI while continuing research.

Please keep the feedback flowing, it’s encouraging and helpful.

My sense from mastodon is that there are plenty of people wanting this, which is good. Maybe some will join in and help build it.

EDIT: A co-author of git-dit contacted me on mastodon and offered to answer questions on git-git.

I’m getting quite a bit of interest on mastodon over my git toots, and I don’t yet have much social reach, so I find that very encouraging. So if anyone is tempted to join mastodon I think it could be a good way of expanding awareness and bringing in new like minded folks, users and developers.

4 Likes
17

:flushed: my bad! :slight_smile: looks like import/export is actually very well covered, eg github to gitlab lists a lot of aspects available for export/import.

Looks like the migrating projects to gitlab will give a solid overview of which features are available on each platform.

3 Likes
18

Good find, thanks @mav and I’m shocked to see you miss something!

4 Likes
19

Screenshot that :slight_smile: Just as you should any long late at night early morning post I make with no typo’s :slight_smile:

5 Likes
20

He was definitely right about GitHub search sucking though. You can’t even look for a “quoted phrase”. Imagine you could do something like ‘go to definition’ or ‘find usages’ like in an IDE :open_mouth:. That’d be amazing.

3 Likes