Safenetwork Git Portal Discussion

So are the main benefits speed of loading a page? I’ve been kind of passively following along not fully realizing the potential benefit. Does this mean if you have and existing JS/ReactJS/Typescript app that you could use this bindgen to compile it to WASM for a better/faster web experience on the Safe Network where loading times and latency might be an issue??

Big thank you to @Southside, @JPL and anyone else who had a go for trying this out. It’s incredibly helpful, picked out one bug and helped me fix errors in the documentation so more people should find this useful.

I’ve since made a few tweaks and will announce the thing on social media this week as I start building the Git Portal Prototype using this and the work from the Proof-of-Concept.

Those interested can follow the next phase at Git Portal Prototype. Ideally also the extension of the proof of concept to allow creation and editing issues and comments, and adding this to the UI, but I’m not sure who will do that as I need to press on.

Opportunities to Contribute

There’s plenty to do if anyone would like to jump in with front-end (I can help you learn Svelte), applying Rust git libraries, JavaScript git modules (for stuff we don’t yet have in Rust), and writing a git-bug compatible library in Rust to support github like ‘issues’.

I’d also welcome help setting down coding guidelines and implementing support for tests, documentation and error handling. @StephenC do MaidSafe have anything written down on those areas than you can share?

I’ll keep plugging away and eventually may get something useful, but it’s a lot for one person.

And of course trying things out helps a lot as noted.

Thank you!

10 Likes

The benefits include small code size so yes the app will load fast, but that’s not such a big deal for you as your bundler probably does a good enough job with JavaScript.

The main reason to use WASM in a Web app will be to make use of a different language such as Rust, for speed, code quality and security. Again, not such a big issue for you I think. I’d suggest switching to Svelte before considering WASM for a conventional app.

I think the benefits I listed are more important for the Git Portal, but it would have been possible to build this in Svelte plus JS. I went straight to Wasm so that I could use existing code written in Go (git-bug primarily, but I also benefited from go-git and go-billy) which saved me months of work getting to a proof of concept. Go adds 12MB to the WASM though (in Rust my WASM file is <200k) so I wasn’t willing to build on that foundation.

Having got there it makes sense to switch from Go to Rust target than reimplement git-bug in JavaScript.

For most web apps you either won’t need this or could use WASM to reimplement any parts that are running too slowly. If you do, don’t follow my route (WASI/WasmerJS) unless you find you really need WASI. You can just use WASM, and that’s well catered for by wasm-pack.

I’m using WASI for very particular reasons that are unlikely to apply elsewhere. WASI isn’t really intended for use in the browser, but is a way to create apps that run on multiple platforms (like NodeJS) so most of the WASI tooling, documentation and examples are about that use case.

2 Likes

I’d like to help out and also have a go at Svelte. I should warn you I’m not the quickest though so put something padded on your desk to avoid concussion. Do you have any suggestions for starting materials?

4 Likes

I’ll wear a helmet.

That would depend on your learning style. Take a look at the tutorial, docs and examples (REPLs) at https://svelte.dev. There’s a lot there but if you are familiar with React not much to unlearn - most things I think are simpler and easy to pick up when you need (e.g. by searching the examples there).

It’s probably worth trying something starting from the official svelte template or directly in the REPLs by modifying things you find there (and if you sign up you can save REPLs and come back to them). I think a lot of people begin ideas as a REPL and then copy the files (manually) into a project. If you want to we could start a topic which others can follow, or if you prefer have a PM thread.

Does that help?

2 Likes

Thanks, I’ll have a look at those and take it from there.

2 Likes

Our contributing to the Safe Network doc maybe touches on a couple of bits that could be of interest. Many of our repos also have issue templates set up, which for us are pretty generic.

Git hooks can be useful as long as they’re not too invasive. We use in sn_api here.

For testing we don’t have formal documentation any more. The switch to first Jenkins, then GitHub Actions, now CD via GitHub Actions, plus our repos morphing so much over the years, and a small team, make it a relatively new process that we’re always refining, and relatively straight forward to manage. I can definitely recommend GHA once you have tests set up - it’s a no-brainer IMO.
Been using GHA for a while now & managed to build up some knowledge, albeit from a more or less purely Rust perspective, so I’m happy to help out wherever I can in setting something up there - DM me if I can help.

Documentation & error handling - these are areas that I think we have a way to go to improve on, we just haven’t found that right formula that we are really happy with. You will have no doubt read the updates where we’re often mentioning improving the docs, and recently there has been a focus on error handling. I’d say we’re still experimenting here. The refactoring work undertaken over the past year has helped simplify things a lot for the current team so hopefully we’ll be in a better place with these two soon. We don’t have anything written down that I can pass over, but if you have any specific questions then fire them over and I’ll let you know our experience on it.

9 Likes

Thanks @StephenC I hadn’t thought of looking at this so will do. I’ll look out for things and share anything useful that I find or set up.

4 Likes

Mini Git Portal update - commits pushed today:

  • Setup test framework for API using uvu and puppeteer
  • Add documentation generation using JSDoc with docdash template

See: GitHub - happybeing/p2p-git-portal-wasi: Peer-to-peer Git Portal in Svelte, Rust/WASI on WasmerJS

10 Likes

Found this in my recent searching and thought you might be interested to see it. -

Had to add this quote " Finally I made the step to read the rust book and boy was it worth it. I mean, Rust is the Dark Soul of modern programming languages, not because it is intrinsically difficult, but because how unforgiving the compiler is."

3 Likes

Mini Git Portal Update (4th Feb 2021)

Three really nice things have been added or enhanced:

  • automated tests
  • continuous integration on github
  • improve generation of documentation (for internal APIs)
  • all the above updated in the README

Development and Production Testing

I mentioned previously that I had set up a test framework, but now I’ve supercharged it. What I had was the basics, but now I have it firing up two servers so the tests can run fully automatically.

One server is for the app, which can either be run on a new build (for production testing) or using webpack-dev-middleware for hot reloading during development. :tada:

The second server is a trivial githttp server stolen from wasm-git (which I worked on recently - and had a PR accepted! - while evaluating options for Git Portal). That means I can create more complex internal tests without using a proxy if I want, though the latter will also be an option during development.

It alls sounds a bit geeky to non-developers, but its a big plus for the project and I’m chuffed because I’ve never done any of those things before. All I have to do now is write the tests! And it will make it much easier for others to jump in and help out.

Continuous Integration on GitHub

Encouraged by @StephenC (thanks for offering to help but I managed with a little trial and error) the production tests mentioned above are now run automatically by GitHub whenever code is pushed to main or a PR is submitted. And all tests currently pass!

I’m chuffed as it’s the first time I’ve set up continuous integration on one of my projects and a useful feature that makes it much easier to collaborate with others…so more PRs please!

Improved Documentation Generation (for internal APIs)

To help folks contribute and make the project easier to maintain I wanted the APIs used by the front-end to communicate with the application-logic to be documented as well as tested, and had a basic set-up in place (from the previous update).

But to make it useable I had to do some more tweaking and ended up writing my first JSDoc plugin. It’s simple, but also a new skill that means I can do much more with this when needed.

The main change this time is to shorten the symbol names. Previously JSDoc was prefixing everything with “export.” which was making the menus unusable, and generally polluting the documentation with useless information.

I also learned a bit more about how to use JSDoc and will at some point be writing some guidelines to make it easier for others to jump in and help so…

Help Welcome!

I’m only one person, and a slow one at that :grinning:, trying to create a GitHub alternative and even though there are lots of economies doing this on Safe Network (see below), I won’t be able to complete this all on my own.

So if you want to contribute in any way, don’t be shy. Testing always helps, following the README and finding things that don’t work is invaluable and of course there is code to write in Rust or JavaScript, Svelte stuff to do, front-end UI to envision and build, new post-GitHub features to imagine and much more. All are welcome and I’ll help you skill up wherever you fancy trying something new.

And it is all built on exciting new technology such as [cough] Safe, Rust/WASM, WasmerJS/WASI, my old favourite Svelte and the fancy automated testing and CI I described above.

This is fun, and an exciting valuable project to work on.

17 Likes

Why Safe is a Massive Deal for Developers

I mentioned above that I’m a bit slow, here’s why I mentioned that (re-posted from something I wrote today on another topic).

Something that I’ve only just realised is the absolutely staggering leverage developers will gain by developing on Safe Network.

I had already appreciated some big gains, such as the ability to scale a website or application to any degree with zero infrastructure costs. That alone has massive implications, including turning the financial models for business upside down, and of course decentralising power and blowing the doors off innovation.

But it took me starting to build Git Portal to appreciate that this network is going to be much more than a way to scale with economics built in that people have been dreaming of for decades but has not yet materialised.

So what’s this big new realisation? Well look, here I am just one moderately skilled coder and I’m attempting to build an alternative to GitHub, had a proof of concept running in less than a month and all but complete in two. How is this possible?

It’s possible because Safe provides me with not just a ready made scalable platform, but the option of avoiding all the complicated bits that have put me off attempting this for years.

A few years ago I looked inside SSB-Git and recoiled in horror at the prospect of figuring out how git works under the hood and integrating that into a decentralised protocol as they had done.

Now we see Radicle building a decentralised git system. They have a large team because they’re doing that and more - building a decentralised protocol that fits git like a glove, as well as expanding git’s capabilities in the way I’m doing for Git Portal (adding issues, PRs, social etc). It may well be effective but we know how hard it is to build those protocols! And there’s a large development cost which has to be paid for somehow.

I never stuck with SSB (Scuttlebutt) whenever I tried it because it just doesn’t work for me, taking forever to update and using enormous bandwidth and storage for a single social app.

So what’s different about Git Portal? The big difference is that it is not using the protocols, nor the data types, just the peer-to-peer storage that sits on top of them. This has reduced the work needed enormously. No protocols to design, write, or even to interface with. And I don’t even need to interface with the storage much, because a consequence of having a standard storage API is that I can put Git Portal together from existing git libraries that already sit on a storage API.

There’s still a lot of code to write, and I have had to figure out a way to glue it all together, but it’s probably between one and five percent of the work needed to build something identical in functionality that uses its own protocol.

The same will be true for many apps now that we don’t just have the data types, but local first CRDT types, as well as the option of just using a filesystem (device or in-browser). Others can use the work I’ve done and have even less work to do.

This is going to be a massive incentive for anyone building things that can take advantage of those development economies.

19 Likes

Git Portal Micro-Update

Lots of activity but the best part is that Git Portal had two new contributors! :tada:

By @dredd :pray:

  • Windows support added (for development)
  • build and test commands tidied up (renamed and rationalised)

By @happybeing :pray:

  • added CONTRIBUTING.md for anyone wishing to contribute. Contains guidelines for non-coders, as well as for front-end and Rust subsystem covering documentation, testing, error handling etc.
  • added Windows build to github actions (CI) so everything is built, and tests run on Linux and Windows when code is pushed or merged
  • added Rust checks to CI and Rust source code is checked with clippy (lint) and rustfmt (layout)
  • added Semgrep to CI to scan code and dependencies for security issues (untested, so :crossed_fingers:). Thank you to @StephenC who made this trivial by doing the hard work and leaving me with a copy/paste job.

By @janriemer :pray:

  • proof read and updated CONTRIBUTING.md with useful links plus fixes to spelling and grammar

Help Welcome!

Ad hoc contributions are welcome so don’t think you need to make a big commitment to help out. If you are interested in any area, coding (Svelte, JavaScript or Rust) or related to documentation, testing or feedback get in touch. For specifics, check out CONTRIBUTING.md.

19 Likes

I’ve paused my work on Git Portal and am considering whether I want to continue alone on such a large project, or on code at all.

With almost perfect timing, MaidSafe have announced substantial funding is available to third party developers. So if you have relevant skills and might want to contribute or even take the lead over time, let’s talk.

See Announcing the BambooGarden Fund and my note here.

13 Likes

@happybeing … if you have the time, could you make a blog post of that? It’s really good and it’d be awesome to be able to share it around to people who might be interested in developing on Safe.

1 Like

I’ve already shared it as is, linking directly to the reply and I think that works well, bringing people to the community as well as the content itself. I do this whenever I write or see a suitable reply, David often writes something that works for, or has a good quote etc.

If anyone wishes to make a blog of it by all means used what you like.

2 Likes

Keep up the excellent work, inspiration and your wisdom please @happybeing

5 Likes

It’s a shame you are pausing on this. It looked like you were enjoying breaking new ground?

I’ve no idea what your day job is, but it seems you have a passion for this. It would be a shame to see it go to waste.

3 Likes

It’s not wasted even if this code isn’t used, and I’ve been working away flat out knowing it’s too much for me to complete on my own. Perhaps now we have a fund others will be tempted to join in, take it over, or come up with their own solutions.

I think I’m done for the time being, though still considering. It’s been a long haul, not just Git Portal, but learning to code again, so many new tools on several Safe projects over the different APIs, and lots of my time on social media working to help the project.

It’s been great but a few weeks ago I felt the need to pause. The pause is continuing so I can see where I want to go from here. There’s still lots to do, but it feels like the project has reached a point where it will soon be unstoppable.

8 Likes

It’s always hard to get a balance. I’ve abandoned trying to work on projects in my spare time; my family need my time more.

It would be great to combine safe network with my day job though. That would far more fulfilling than what I often end up working on. Having worked mostly in fintech over the last few years, it feels like my soul needs to be taken off the market.

It would would great if other senior community members could focus more on safe network. Now it is coming to a head, it feels like there are a few mutually beneficial angles.

Anyway, no pressure. I just feel that sometimes things feel right and we shouldn’t fight that. Sometimes it leads to a happier place all round.

2 Likes