MaidSafe Dev Update - December 14, 2017

Peruse (the custom browser that will eventually replace SAFE Browser) has been updated, you can download the latest version here. It adds localhost support, as well as the ability to open http links in your clearnet browser of choice automatically.

Keep in mind that this is a proof of concept and has a few known limitations:

  • Placeholder authenticator messages (the prompt doesn’t show the app’s info and/or permissions requested as the current authenticator does)
  • Potentially slow startup (network connection occurs on startup as opposed to first fetch)
  • No reconnect notifications (if you start without the invite IP updated, you’ll need to restart, for example)
  • UI has not been a focus yet
  • No page loading indicator right now

Community Engagement Program (CEP)

Followers of the project and regular visitors to this forum will have seen a lot of discussion on the forum about marketing, and as mentioned in previous updates this is something that is getting much more focus and resources with @SarahPentland joining the team a couple of months ago and @dugcampbell (Dug Campbell) joining this week. There are a couple of areas that we will be focussing on, but one is much more community involvement. The SAFE Network must have the strongest community in the space and even after 3+ years we continue to be astonished by the energy, intelligence, commitment and creativity of the people on this forum and it would be almost criminal for us not to use these skills to all our advantage. That is why next week we are bringing back the Community Engagement Program (CEP) as the first step in this process.

The purpose of the CEP is to match MaidSafe requirements with skills in the community, taking the form of a community funded ‘kickstarter’ like process to encourage community members, individuals and teams to bid for projects. Before this was more focused on technical requirements and longer standing forum members will recall that the CEP gave birth to the SAFE Browser, currently part of Alpha 2, and also led us to hire @joshuef. Our intention with this relaunch is to not only focus on development tasks but to also incorporate non-technical requirements, such as design, animations, videos, content management…etc…Sarah and Dug are currently working on an explainer video and some other resources to make the process clearer and to educate newer members about its existence and we look forward to launching this next week.

SAFE Authenticator & API

The DOM API playground tool was updated to the latest changes in the DOM API and it’s now fully compatible with the SAFE Browser v0.8.0. This update included a refactor where the code snippets were changed to make use of async/await as we hope it should make it simpler/easier for developers trying to understand them and to then use them in their own apps. Although intended for mock development, it has also been published at safe://codeplay.discover for initial discovery. Please keep in mind that it can quickly consume mutation requests.

The scripts to automate the SAFE Browser packaging process to generate versions for mock and non-mock routing have been merged and we are currently testing them to have them ready for our next release of the browser.

During the last few days, we were also going through the list of bugs reported for the browser and plugins, not only those reported on GitHub but also some others discovered internally, and we started fixing some of them to be able to release a patch for the browser soon.

Peruse has been updated, you can download the latest version 0.2.0 here. It adds localhost support, as well as the ability to open http links in your clearnet browser of choice automatically.

We’ve dropped the custom localhost protocol (as found in the SAFE Browser), as this was causing a number of issues in Electron. Now http://localhost:<port> and http://127.0.0.1:<port> are whitelisted HTTP URLs for use in Peruse. This sidesteps these Electron issues and enables more robust local development (most notably hot-reloading of your applications).

Otherwise, this week has seen a lot of refactoring of the Peruse store to enable syncing across browser windows and increasing test coverage, which puts us in a good position to implement history and favourites to round out a bit more of the basic browser functionality.

As a first step in integrating the JNI bindings, we are implementing a few minimal APIs, setting up the test suite and testing the desktop JAR files from a quick example app. Once these steps are covered, we will be starting to expand the APIs.

The Java project is structured such that the api project will be a dependency for safe_app (desktop) and safe_app_android. Only the platform-specific variations will get coded into the desktop or Android project. For example, the loading of the native library is different in desktop and Android. Hence this part will be handled in the platform-specific project while the majority of the API code will reside in the api project.

The progress with Java bindings has been good. We are well placed to cover these three steps and get it merged with the upstream master by this week.

A pull request adding ReSharper code inspection to the AppVeyor CI script and removing identified linter issues has been merged.

SAFE Client Libs

As we keep supporting more and more platforms and processor architectures, maintaining their releases has become tedious. To streamline the process, we came up with a plan to automate it as much as possible. The first step of this plan was to write a script which automatically builds and packages all necessary libraries and other support files for each supported platform and architecture. This script is now ready and is undergoing review and testing. After that is done, the next step will be to include it in our continuous integration system.

Today we are releasing a first community preview of the auto-generated Java bindings. It consists of two parts: the Java files, defining the API, and libsafe_app, which glues JNI functions to the Rust implementation of SAFE App. For now, the released version is available for the x86_64 Linux platform only, but as soon as we’re done with the build automation scripts, it should be available for all the platforms that we support.

We’ve also received some valuable feedback from the front-end team and started implementing it already, so there will be more news and updates in the coming days. The feedback concerns the minor stuff, so the Java bindings should be wrapped up pretty soon.

Routing & Crust

As we’ve been drilling down a bit further into the details of the proposed implementation of Data Chains, we’ve come across some problematic examples which have resulted in further team discussions and refining/updating the protocols. As expected, the chief culprit of the problems is the section merge procedure. That’s what happens when a section of peers becomes so small that it needs to join up with one or more neighbouring sections.

This has proved troublesome in the past, and we’ve learned a lot from previous work about how this can go wrong. That experience is invaluable, it’s made us all very aware of how easy it is to overlook edge cases which can hurt the network. This is just a symptom of decentralisation: if all the peers agreed simultaneously about how the network was changing around them, it would be relatively trivial to handle. However, at a given moment, since half the peers in a section can “see” one thing and the other half see something slightly different, it makes achieving eventual consistency in this case particularly tricky.

As a result of this, we’ve not progressed the code as we’d have hoped - instead, we’ve been largely focussed on resolving the remaining issues in the merge protocol.

We are also delighted to have our work closely scrutinised by forum members. Special shout out to @mav for his interest and contribution. Seeing so much interest from the community is awesome and gives us even more motivation to work on the project :slight_smile:

@mav’s interest in the project also led to him discovering a serious bug in our simulation - it was caused by relocated nodes not being actually removed from the source section, which resulted in their duplication and artificial increase in the number of older nodes. The results of the simulation changed dramatically after this bug has been fixed. This exposed a flaw in our current design which made the network accept barely any new peers. Fixing this flaw will be an important task for the near future. Summarising, @mav’s efforts are quite a significant contribution to the project - much appreciated! :slight_smile:

uTP support has finally landed in Crust’s dev branch. uTP is a UDP-based transport protocol which has several advantages over TCP. Its congestion control algorithm is designed to use all the spare bandwidth on an internet connection while yielding to anything else which tries to use the same connection. This makes it a perfect protocol for peer-to-peer downloads from the SAFE Network. UDP hole-punching is also much more reliable than TCP hole-punching, meaning that we will be able to make direct peer-to-peer connections more often than we were able to with just TCP.

Some parts of uTP support still haven’t made it into Crust. Specifically, the rendezvous server implementation used for UDP hole-punching still needs to be integrated from our p2p crate. But we expect this to be completed within days.

The next stage for uTP support in Crust is, of course, testing. We plan to do some more internal testing first to make sure there are no glaring bugs, then release binaries to the community so that we can get some testing done in the real world.

83 Likes

Nice update! Thanks.

8 Likes

Very very cool! @mav glad to see people being this committed and challenging the way stuff is done!

28 Likes

Great work @mav! Another excellent update :smile: I may be wrong but it looks like data chains and crust are heading down their respective tracks to come together in relatively short order for some much anticipated community testnets! Good luck @maidsafe nipping that section merge issue, you are some very talented folk and we’re all cheering for you.

Can’t wait to see what’s in store with CEP :thinking: glad to see that resurrected :smiley:

16 Likes

No unicycle? Disappointed.

8 Likes

Big thanks to everybody, and this time especially @mav.

10 Likes

Woot - great news!!!

13 Likes

@mav :clap::clap::clap::clap::+1::heart_eyes::ok_hand::ok_hand::ok_hand::sunglasses::sunglasses::sunglasses: :heart::heart::heart::heart::heart:

12 Likes

Thank you @mav and the Maidsafe team!

6 Likes

oh - and just to mention it I really like this plan a lot! :hugs:

11 Likes

It is already shared in the largest crypto-forum in Spanish language (6 K total users 2.2 K active):

Now to wait for the news of the Community Engagement Program (CEP).

It would be good that yourselves pass by there since the thread has fallen a lot lately and they will see a bit of movement. There are really many people interested but they are asleep.

@digipl you have us orphans friend …

13 Likes

How does that Phil Collins song go " I can feel it in the air tonight - Hold on". You guys are motoring. @mav great job

2 Likes

I don’t pretend to be an expert on data chains or any of this but I have tried to follow how this all works. So perhaps a link to exactly this problem is being resolved or discussed might be good. More details would be appreciated. :smile:

Also while it’s on my mind. How far along the road map are we exactly?

2 Likes

@maidsafe thank you so much for Peruse all working smoothly in my mac, localhost now works with gulp, I don’t have any issue anymore (so far :smile: ) and web hosting manager also can use, awesome works guys!!!

11 Likes

Good on ya @mav

Thanks as always for the update and we do have a great community here :+1:

13 Likes

Thanks Maidsafe devs for coding and these updates they give super powers.

Now this is real contribution to the SAFE Network, I salute you @mav

I love it, how you super ants keep testing and testing and testing…

Salutation also to all the super ants on this forum, you know who you are :stuck_out_tongue:

Oooh yeah about this @maidsafe

I’m not trying to pump these guys their video, but they got really good explainer video’s + over 1M views per vid, I would put my money up for a SAFE Network explainer vid done by Kurzgesagt.
https://www.youtube.com/results?search_query=kurzgesagt

12 Likes

Awesome!

Boy oh boy. I opted for this little over a month ago after making three inspirational video’s which I shared in that same topic. @nicklambert replied they were discussing community involvement in other areas besides technical development, but I didn’t expect it to come to fruition that quick. Great news!

Work has been sucking the life out of me these last few weeks, but I’ll be happy to contribute some of my creativity once this CEP goes live.

@SarahPentland, @dugcampbell :
count-me-in-5fo1a2

23 Likes

Wow ! this is really cool :smiley:

Thank you for the rewrite of the doc on http://docs.maidsafe.net/beaker-plugin-safe-app/ , too !

Excellent news, I’m eager to see what fruits the community engagement program will bring !!

9 Likes

Thursday is starting to become one of my favourite days of the week. Go team!

15 Likes

Hi

I think we need an online course as part of the marketing plan. I am not sure how much resource it would take up but I think it would really help break down the network into manageable pieces. We could have videos explaining things, a bit like Khan academy.

Obviously its not a priority for the team but maybe it could be something for the CEP. @SarahPentland @dugcampbell

10 Likes