Safenetwork Git Portal Discussion

This is a topic inviting ideas and participation in defining and building a Safe-git UI.

All are welcome, I have outlined an approach which like everything will be open to debate and change. I’d like to start with something simple and extend that incrementally, so have proposed something based on spawning git commands, along with a new CLI (to be written) called gx to add support for github style functionality (e.g. github issues), but without a server.

You can contribute general points here, or for more specific discussions, open an issue on the github repo: safenetwork-gitportal

Safe-git UI Vision

past-and-future

github/gitlab/gitea like GUI

Possibly a Tauri based Rust backend with web frontend, e.g. using Svelte:

  • operates on a local repo
  • spawns git commands
  • a new gx CLI to create issues, comments etc (e.g. stored in .gitex)
  • data for extas held in the repo (e.g. issues.json and issues/issue-001.json, or perhaps in RDF/turtle)
  • git push/pull remotes on Safe (e.g. via FUSE mounted Safe filesyste)
  • cross platform GUI: Windows, MacOS and Linux (e.g. using Tauri)
  • cross platform terminal UI: Windows, MacOS and Linux (e.g. using tui-rs)
  • web UI for mobile (e.g. static HTML hosted on Safe Network)

Discussions to Have

Existing git UIs

There are many graphical, web and terminal UIs for git, so these may be adapted or learned from.

Which should we look at and why? If you know any well your input would be very helpful.

gx requirements

gx is a new CLI we need to create that adds features not handled by git, such as github issues.

  • gx based features must work in a way which allows a GUI to support them by spawning git and gx commands. Doing so enables anyone to operate using the CLI, or to build their own Safe-git UI with the tools of their choice.

  • where gx adds something based on a github feature, it should also provide a way to import this from a github repo using the github APIs.

  • what features do we want, and what difficulties do they present?

  • How to store and publish gx extras?

Note that Safe has a built in perpetual history for public data which negates the need for .gitex to be part of the repo’s git history. But then we need a way for third parties to create issues in a way that makes them accessible to everyone interested in a git repository published on Safe.

Development Tools

I’m keen to try out Tauri and Svelte at least for proof of concept, but if we define the underlying data storage and structures for gx, anyone will be able to build a GUI with the tools of their choice (e.g. spawning git and gx commands).

So you can build your own Safe-git UI tools while collaborating on a platform we can create and extend together.

Update - Current Status

I’ve been looking at an existing tool, git-bug which adds support for issues tracking to a git repo without polluting your commit history. It’s very nice!

By running git-bug in the browser, it may be possible to host a github like portal as a Safe web app, which along with the git-bug and git command lines gives us a complete alternative to github which I’m sure others will want to use and extend.

I had a good chat with author of git-bug to understand his project and see how we might assist each other, and have already had two offers of help from the Mastodon community. There’s a lot of interest in a github alternative there, along with decentralisation and open source, so all this is very encouraging.

git-bug is written in Golang, so currently I’m investigating use of Golang -> wasm in the browser, and how to shoehorn git-bug into a safe:// URI.

For more see this post and the posts which follow it:

Safenetwork Git Portal Discussion

19 Likes

Proof of Concept: gitex

gx extract [--type="issues"] [--overwrite] <user>/<repo>
Extract ‘github extras’ from github repository to .gitex/

gx clone <user>/<repo> [path-of-clone]
Clone github repo and populate .gitex subdirectory.

gx fork <user>/<repo> <path-to-fork> [path-of-clone]
Maybe? Clones, creates a new repo, sets origin and upstream remotes in the clone,
and pushes clone to the new repo (origin).

gx issues [--json]
List all issues

gx issues new

gx issues print [--json] <issue-number>
gx issues close <issue-number>

gx comment add <issue-number> [<text>]
gx comment print [--json] <issue-number> <comment-number>
gx comment replace <issue-number> <comment-number> [<text>]

gx sync
gx issues sync
Get latest communal extras for current repo, drop any local duplicates.

Note: the first letter of issues, comment and pull-request can be used instead of the full word. For example: gx i new is equivalent to gx issues new.

Proof of Concept: git UI

Location: (drop-down of recent URIs / filesystem paths)

Github URI

User - github-repos list - clone to path
Repo - clone to path

Local filesystem path

Local directory: list repo subdirectories
Local repo directory: Tabs: README / Issues / PRs

Local repo directory Issues tab: github style issues list - visit / open
Local repo directory Issue page: github style issue page - comment / edit / delete

Local repo directory PRs tab: TODO placeholder

9 Likes

Maybe in a .gitex directory at the root of the repo?

eg

.gitex/issues
.gitex/wiki
.gitex/releases
...

This could contain pretty-printed json files, one file per issue / wiki page / release note etc? If gitex specifies a json-schema then it would be simple to ensure any additional data has a certain form and it gives the UI some confidence of what data to base their views on.

I feel like keeping it in the repo is useful, since it allows this idea to work independent of Safe Network. Don’t get me wrong it’s good business to build a moat and get people invested in Safe Network, but for interoperability it seems neat to have it in the repo. Although maybe a .gitex direction is not interoperable with, say, github which could create confusion.

One observation I had on github (which might impact the .gitex subdirectory structure) is issue ids and pull request ids are linked. Eg if you have /issue/44 you can’t have pull/44 - the underlying id comes from the same source. Pull Requests are just issues-with-attached-code. So that’s a small leak of the underlying abstractions used by github. No need to have an issues schema and a pulls schema, they’re the same underlying structure.

6 Likes

Thanks @mav, useful thoughts and great tip on the issue/PR structure. Those are the first things I’d implement so it’s helpful to know.

I was planning to start with a .gitex/ subdirectory and JSON files for issues (starting with what the github APIs deliver) but not have its contents under git control as I thought it would complicate things to have commits of things like issues. Any thoughts on that?

I haven’t thought any of this through, I drew the diagram today and then wrote down some ideas. So it’s very early stage.

I wanted something useful to do which I can progress while awaiting the testnets and APIs. I’ll start with gitex and then make a simple Svelte based Tauri app to do stuff with the .gitex/ data, and try to create a core library that can be used to create a Safe-git terminal UI.

I saw and experimental ‘modern’ git CLI replacement called bit earlier which might give inspiration for this, but for now I’ll keep things simple.

9 Likes

Mmmm. Yeah. This would pollute the git history. Good point.

I wonder if something like how git tags works is a useful concept to consider? Tags are a metadata in the repo that are related to the commit history but is not adding new history. I wonder if tags might have conceptual relation to gitex type features?

Maybe a hair-brained idea but what if gitex is a branch based off the initial commit? gitex command could be an alias for roughly git checkout gitex; do gitex stuff and autocommit; git checkout $PREV_BRANCH … ? Seems a bit hacky.

Overall the gitex concept is great and I can see it being very popular if it’s well implemented. Having a standardized non-vendor-lock-in way to do the ‘social’ aspects of code that made github so successful would be pretty amazing.

6 Likes

This is incredible @happybeing … thank you for all the hard work and thought you put into the network. I know that, like the Maid team, you will be rewarded greatly down the track for your efforts.

12 Likes

Thanks @TylerAbeoJordan, I do this for fun but part of that is being appreciated. It wouldn’t be much fun if nobody thought what I was doing was worthwhile!

@mav your input it’s invaluable as usual. I don’t know much about guy under the hood, so not sure if I’ll come up with something implemented that will, at least from the start. First step is proof of concept, though so no pressure, but if you want to keep thinking up interesting ideas like this I would appreciate that.

On the tricky parts of social. Firstly, there’s an element of collaboration with issues, so I was thinking to handle this there would be the remote repo view, and your local repo view, which includes your most recent pull of the remote (.gitex/issues) plus anything you’ve done locally, such as create an issue. So you will see a new issue you made or a comment you added, immediately but only visible to you.

At the point you made the issue, perhaps this is appended to an issues data structure (eg Sequence?) for the given remote, which anyone can write to. But when you or anyone ‘pulls’ you all get the same view (whatever is in the Sequence), and locally gitex can drop any duplicates you created locally. So we have a separate mechanism from git push/pull that works in parallel, managed by gitex.

On the other social aspects, maybe a similar approach will work for that, but I’ve not really thought much about it yet. Maybe we’re need a more general social API/library and to tap into that. It’s crucial as you say, a big area so I’ll try to think about it in the background. Again appreciate any thoughts on this.

A Question for Everyone

Are there any things which github does badly, or doesn’t do but could?

Initially I’m going to try replicating familiar github features, but at some point we want to break free and if there are some nice innovations that would be very cool. Even just in concept, we can document things for later.

6 Likes

/brainstorming mode on

Some form of documentation links perhaps. I See 2 forms here (possibly 3).

  1. Code documentation (will differ per language though)
  2. Research docs to explain what/why the library/app is implementing (so background to the design)

If 2 could be linked from the code in an easy way then it might be helpful (i.e. this method implements “this part of this doc”.

The other thing could be a real wiki, but using git/PR’s etc. to update it. So updates to the wiki need agreement in the team. This might be 2 above actually. Github tries to do a wiki, but tbh it’s not that great. I think there is a cleaner way and one that will help code reviewers / potential users to clearly understand the reason for the library as well as how it’s implemented.

Another thing is linked crates, so like a dependency graph (rust does this well, so may be language constrained). But not so much actual dependencies, more like, this crate has these deps, it is meant for this project which has other purposes (vault / client for us). Maybe that is a project crate though?

7 Likes

Warning: to create maximum confusion I’ve renamed the new CLI to gx (from gitex) mainly to save characters, but left the extras subdirectory as .gitex for now, though I’m inclined to shorten that too.

I did some searching on gitex and gx and there are a couple of git related things called gitex and ‘git extensions’ (plus a well SEO’d annual GITEX technology conference). Whereas I found only one thing called gx which is a package manager written in go that was quite popular, but describes itself as alpha quality and has had no gh activity for 2 years, (and the author is busy with Filecoin Lotus!)

3 Likes

Open source is awesome.

I stumbled on git-dit via StackExchange and I shall look into using this instead of creating gx at least for the time being.

It’s written in Rust which is always a good sign, and @mav it sounds suspiciously like your ‘hacky’ suggestion which may well be the way to go. It at least gives me a quick way of trying out the general idea without having to build a CLI to start with.

@mav If you have time to read this short discussion of how it works I’d be grateful for your comments. I don’t know what ‘patch sets’ are for example, or what he means by an ‘empty commit’, so I don’t understand the effects of the implementation choices he’s made.

EDIT: browsing the issues I note:

  • they don’t commit files (so I don’t yet know how issue data is stored or shared)
  • they use an email style model (but I’m taking this to mean for their threading model rather than actual emails at this point)
  • they use MIME encoding e.g. to handle file embedding
4 Likes

Another from the same StackExchange is sit, a serverless modular system for sharing and querying information with modules for different workflows including a sit issue tracker. Also Rust, also dormant for a couple of years but worth looking into.

3 Likes

FYI -

3 Likes

A slightly different approach is being taken by radicle using git to provide efficient propagation of information, by using git packfiles to avoid the overhead they found using a storage system (IPFS).

radicle is a protocol for discovering peers, using git and it’s packfiles to scale the propagation of changes along with gossip.

I’m not sure if there’s anything for me or more generally Safe, to learn from their approach, moving from replicated storage (IPFS) to git packfiles but it’s an interesting comment they make. I don’t have time to look in depth at all these systems though, so if others do it would be interesting to hear back.

2 Likes

Search is not that great for code. I find grep a lot more useful and am regularly frustrated by github code search. Github will only show the top N results in each file, so if you’re looking for some specific use of a variable maybe it’s ‘below the fold’ which you can never know from the search results.

Another thing they don’t do well is data export / import for metadata such as issues etc (hmmm sounds like exactly what this topic is addressing what a coincidence). The reason I say it explicitly here is if gx is designed specifically to enable easy import/export to/from github gitlab etc then it has a much stronger chance of catching on. Being aware of the way github does things a) saves a lot of work/mistakes by using prior learning from github and b) maximizes possible traction by giving a relatively simple path for adoption. The examples you link seem to want to operate in a vacuum and I feel like that’s a mistake because they miss out on all the potential network effects already in play from the big web git providers.

It sounds like git-dit have a fairly atomic approach where each comment / action is a unit (like sequential emails are individual), where I was thinking more like each issue (containing many actions) would be a unit with git-style updates to that unit for each action taken. Not sure about the difference in practice, maybe the flexibility of everything separate is better, or maybe it introduces unnecessary complexity? Perhaps a little too far down in the weeds for the current stage of thinking.

Really cool to see so much prior work in this space.

Since it was mentioned but not linked, here’s bit (worth looking at their inspiration too):

3 Likes

gitx

Naming wise, the shorter ‘gitx’ pronounced the same as ‘git-ex’ would also be nice.

3 Likes

Several good points @mav, can you say more about the or support for export/import. I only looked briefly but found export of issues very easy and it looked comprehensive. Using curl you can quickly get a list of issues, and the issues themselves as JSON files, which seemed good.

I agree with your take that extending github is a better first step. Kindof using Microsoft’s own death grip against them (embrace, extend, extinguish) :face_with_symbols_over_mouth:.

So anyway, yes we need interoperability. Good point and thanks for looking.

Thanks @jlpell, ‘gitx’ is nice.

I’ve built git-dit so tomorrow I’m going to play with it and maybe start work on a UI while continuing research.

Please keep the feedback flowing, it’s encouraging and helpful.

My sense from mastodon is that there are plenty of people wanting this, which is good. Maybe some will join in and help build it.

EDIT: A co-author of git-dit contacted me on mastodon and offered to answer questions on git-git.

I’m getting quite a bit of interest on mastodon over my git toots, and I don’t yet have much social reach, so I find that very encouraging. So if anyone is tempted to join mastodon I think it could be a good way of expanding awareness and bringing in new like minded folks, users and developers.

4 Likes

:flushed: my bad! :slight_smile: looks like import/export is actually very well covered, eg github to gitlab lists a lot of aspects available for export/import.

Looks like the migrating projects to gitlab will give a solid overview of which features are available on each platform.

3 Likes

Good find, thanks @mav and I’m shocked to see you miss something!

4 Likes

Screenshot that :slight_smile: Just as you should any long late at night early morning post I make with no typo’s :slight_smile:

5 Likes

He was definitely right about GitHub search sucking though. You can’t even look for a “quoted phrase”. Imagine you could do something like ‘go to definition’ or ‘find usages’ like in an IDE :open_mouth:. That’d be amazing.

3 Likes