MaidSafe Dev Update - May 31, 2018

Here are some of the main things to highlight this week:

  • In case you’ve missed it, please do go ahead and listen to @fergish’s interview with @pierrechevalier83, one of the authors of PARSEC here at MaidSafe.
  • As mentioned last week, Pierre has recorded a video that explains the entire algorithm while attempting to remain accessible to everyone without the need for a mathematical background. The idea is that this video would be a good introduction to reading the whitepaper as it explains the reasoning behind the algorithm in an intuitive fashion. We’re finishing off the post-production on the video this evening and will share this on the forum and across social media channels tomorrow.
  • @Sotros25 has stepped up and is running a new SAFE Network meetup in Chicago. The date of the first get-together isn’t fixed yet. You can sign up to the group here.
  • The PR for integrating the new C# bindings has been merged with the upstream repository for safe_app_csharp. safe_app_csharp is also packaged and published on NuGet :tada:. The C# mobile apps are also updated to use the published bindings and a PR is waiting to be merged with the upstream repository.
  • With our intent to replace our C library dependencies with pure Rust implementations for more security and portability, we have merged the first PR to replace the Brotli compression library in Self Encryption and confirmed that it works with the existing data on the Alpha 2 network.
  • We updated the PARSEC RFC with an illustration that demonstrates an end-to-end application of PARSEC on a gossip graph. Hopefully, that helps in understanding the protocol.

Marketing

New Website

During the past week, we’ve been working closely with the UX/front-end team on the development of the new website. We’ve had a number of highly productive sessions debating not only design but also creating content.

PARSEC Marketing

We’d like to thank everyone who has worked online to share the PARSEC message since it was released last week. A huge number of you have clapped, retweeted, shared, upvoted, commented and generally engaged with the wider community to ensure that PARSEC receives the publicity that it deserves. As of tomorrow, we’ll also now have Pierre’s video so hopefully this should make engagement even easier. As soon as this is released, we’ll embed this within the Medium article to make it easy to find/share.

Interviews & Podcasts

Speaking of which, in case you’ve missed it, please do go ahead and listen to @fergish’s interview with @pierrechevalier83, one of the authors of PARSEC here at MaidSafe. This week, we’ve also carried out a few interviews and @dugcampbell recorded a podcast with the FutureTechPodcast. We’ll share these on Twitter and the usual social platforms as soon as they go live.

SAFE Network: Chicago

@Sotros25 has stepped up and is running a new SAFE Network meetup in Chicago. The date of the first get-together isn’t fixed yet. You can sign up to the group here. If you live nearby (or even if you don’t but just want to support the effort!) please do sign up so that we can support @Sotros25 as she gets this new group up and running. Moving forwards, she will also be helping to bring meetup organisers together to help with coordinating events etc so if you feel you could make an impact, we’d encourage you to go to the meetup thread and get involved.

User Experience & Website Design

After collecting creative ideas with the help of the Mood Board, @JimCollinson and @shona together came up with two distinct layout patterns (i.e., one using a broken grid layout and another using a normal grid layout). Both layouts were presented to the internal team and inputs have been collected. In the coming week, we are planning to complete the design for a couple more pages. We are also planning to start implementing the website in parallel.

SAFE Browser, Authenticator & API

Several changes and improvements to our safe_app_nodejs package have been reviewed and tested during the last couple of weeks covering different aspects, adding more features, cleaner code and consistency in the API, as well as trying to make sure it’s aligned with what is being exposed by the underlying native safe_app lib:

  • The PR to support multirange Range requests was merged in safe-app-nodejs as well as its counterpart in Peruse browser to support such a feature for content streaming.
  • A couple of changes to normalise the API names and align them to safe_app lib names.
  • Support for forcing the use of mock by providing forceUseMock boolean to the initialisation options of each safeApp instance, i.e. each safeApp instance can either use mock vault or the actual network independently now, assuming both versions of the native libs were installed/downloaded.
  • Minor refactor in the webFetch function to split it into several pieces to make it easier to understand and maintain.
  • The network state callback is now also invoked when going from INIT to CONNECTED state.
  • Allow to set the metadata when creating a file with NFS emulation, which is stored in the NFS container of the file. The metadata is opaque for the API and the network and it’s up to the application what format and serialisation to use for it.

A tremendous amount of work has been going into Peruse as well, towards being able to share with the community a new unofficial package soon as part of our testing process on all the new functionalities and enhancements it brings.

One of the latest enhancements that is being worked on in Peruse is the ability to have a development version of it which by default runs against the Mock routing/vault without the need to set any environment variable or parameter when launching it. We are aware that having to set an environment variable to enable mock routing has been causing problems or inconvenience to many people and we fully agree this should be enhanced by allowing people to launch it simply from the files manager in their preferred OS.

As you know, the way Peruse will expose the Web API is one of the main enhancements compared to our Beaker browser, since the new Web API will be simply the API exposed by safe_app_nodejs with only a couple of exceptions like disabling the openUri function from it, removing completely the need for managing application handles rather than JS objects. Due to this, we have been also working on updating our Web API Playground tool to be able to use it with the new Peruse package.

We’ve also dedicated some effort to research more about the possibilities around the integration of Solid applications and the full support for semantic web applications. We believe the SAFE Network is a perfect fit for the semantic web and we are trying to understand how we can facilitate and encourage developers to create these types of applications on the SAFE Network. Thanks to @happybeing for all the initial efforts he took to showcase the integration.

The PR for integrating the new C# bindings has been merged with the upstream repository for safe_app_csharp. We were using cargo-script to download the native dependencies. Now, we have moved to Cake script. It helps us to clean up the CI scripts from Rust dependencies and make it faster. safe_app_csharp is also packaged and published on NuGet :tada:. The C# mobile apps are also updated to use the published bindings and a PR is waiting to be merged with the upstream repository.

SAFE Client Libs

With our intent to replace our C library dependencies with pure Rust implementations for more security and portability, we have merged the first PR to replace the Brotli compression library in Self Encryption and confirmed that it works with the existing data on the Alpha 2 network. We’ve started research on the next big step to replace one of our largest dependencies: rust_sodium, the Rust wrapper for the common cryptographic library libsodium. This means we’ll potentially change our code to use a new set of cryptographic primitives - however, this should not be taken as a “writing our own encryption” intent, which is immensely discouraged by security experts and the crypto community. Instead, we’re investigating the possibility of using a set of cryptographic functions as robust and proven as the functions provided by libsodium, only written in Rust (which also provides an extra layer of security - so we won’t see the goto fail kind of errors here). The libraries we’re looking forward to using are Dalek and Miscreant. If this approach is proven working, we’ll gradually move all of our code from rust_sodium, replacing it with an analogous set of functions, implemented in Rust. The JIRA task to track the progress can be found here.

This week we investigated an issue that was reported by the front-end team regarding deserialisation errors when fetching immutable data. It turns out that the ImmutableData API can only be used to fetch immutable data that were created and encrypted through this API. We are considering adding functionality to the API to allow fetching arbitrary, possibly unencrypted immutable data from the network.

We also started work on a Rust upgrade of all our Rust crates to stable 1.26.1, nightly 2018-05-29, and rustfmt-nightly 0.8.2. This will give us access to the long-awaited impl Trait! This upgrade will also allow us to resume MAID-2650, which is blocked due to a bug in dependency resolution on our current nightly. We hope that upgrading nightly will allow that task to finish smoothly. After this upgrade, we intend to stay at this Rust version for longer, as the upgrades take a considerable amount of time to get done. Instead of upgrading every six weeks with each new Rust compiler release, we’ll look into the new compiler features individually and upgrade only in cases where there’s something really valuable to us (like the impl Trait feature, for example).

Routing

There have been a few parallel tasks which have been tackled in Routing this week. @pierrechevalier83 took a couple of days off while he was meeting members of the Rust community at RustFest in Paris. He spent the rest of the week spreading the word about PARSEC. He was a guest on @fergish’s podcast (SAFE Crossroads) to explain the high-level motivations behind PARSEC, and how it fits in the wider crypto-ecosystem and in the SAFE Network. He also prepared a video that explains the entire algorithm while attempting to remain accessible to everyone without the need for a mathematical background. The idea is that this video would be a good introduction to reading the whitepaper as it explains the reasoning behind the algorithm in an intuitive fashion. The podcast is available here. The video and the slides will be released shortly.

@bart has been refining the PARSEC whitepaper; looking for ways in which it can be improved or clarified. This led to a more precise definition of the algorithm’s synchrony assumption and an improvement to the proof of properties of the concrete coin. Some of his efforts were also directed towards creating a detailed description of the application of PARSEC to a sharded, permissionless network.

We updated the PARSEC RFC with an illustration that demonstrates an end-to-end application of PARSEC on a gossip graph. Hopefully, that helps in understanding the protocol.

Work has also begun on specifying the implementation details required for PARSEC, with the expectation that tasks for these should appear in Jira very soon. That will allow us to get a decent handle on timescales for implementing PARSEC (including tests and example code). Some of the heavy-lifting has already been done here with regards to the implementation since we created a script which produces graph visualisations in order to better understand the protocol as we were developing it. All that aside, the implementation will require much less effort than the design phase did.

Crust

A while ago we worked on integrating the Crust dev branch into Routing. That revealed some bugs in other crates like tokio-utp. Since then we’ve been mostly working on those issues and netsim, which helps us to test some edge cases. But since PARSEC’s appearance, it feels like Alpha 3 is just around the corner. If it was an ocean, we could probably smell it… :smiley: So to prepare for the swim we started to give more attention to the integration with Routing again. We started redesigning the encryption interface. Soon Crust will use traits for traffic encryption, so users will be able to provide specific implementations. Another advantage of this is that you won’t need to pass your private encryption keys to Crust. Also, encryption itself will be optional, since we’ll be able to implement null encryption primitives that simply serialize/deserialize data without actually encrypting it. This should make testing and debugging a lot easier in some cases.

Besides that, we fixed a bug in the tokio-utp congestion control implementation. The whole purpose of congestion control is to make sure we’re not oversaturating network traffic with outgoing packets. Now that’s pretty important. We use the LEDBAT algorithm which uses packet delays as the main congestion measure. It turns out that in some cases the delay calculation would overflow and cause programs to panic (crash). Fortunately, we found the root cause, fixed it and ended up with a bunch of more tests :slight_smile: But the work with tokio-utp is not done yet: the work on graceful connection shutdown is still in progress.

Before starting working on encryption, we also spent some time on netsim. Its API was slightly changed again, it is now possible to introduce latency on Ethernet connections and soon it won’t be dependant on libclang. netsim highly relies on Linux APIs to create network interfaces, virtual network devices, etc. On Linux, libc is used to interact with the system. That’s a C library and there’s a Rust crate to simplify the interoperation: libc. Unfortunately, it was lacking some functionality, but Andrew was kind enough to provide the fixes: 1004, 1003, 1002, 1001, 1000, 999, 998, 995. Long live OSS, right? :slight_smile:

74 Likes

Second :slight_smile: first attempt at this, just got lucky.

2 Likes

Third! @lubinew real triggerfinger

3 Likes

Whoooo (20 char)…

Lets read now.
Well after the huge update of last week this update is still giving goosebumps.Specially the video regarding PARSEC would be very useful.

Good decision :+1:

:heart_eyes:

Waiting eagerly

Done. Quite big update after a week gap. The team is sounding more optimistic in each following update. Well smelling the ocean is not enough lets make the voyage. :star_struck:

10 Likes

Excellent news!

How long was the design phase? :stuck_out_tongue:

4 Likes

Ok guys… Amsterdam reporting in.
PARSEC update and general progress has caused wild fire adoption in my company. We all say hello… Wish the decorum team.were here in arena… :slight_smile:

IMG_20180531_214251

Ace update as usual.

42 Likes

Someone is engaging with Vlad Zamfir Ethereum dev about parsec on Twitter. He’s asking some questions that someone more knowledgeable like @pierrechevalier83 @bart @qi_ma or @ustulation could prolly answer. https://twitter.com/anonfreely/status/1002262842064752640?s=21

UPDATE: This is a retweet of the convo between Vlad, Bart, and David on PARSEC from anon. Has a lot of good hashtags let’s retweet it! https://twitter.com/anonfreely/status/1002305421909135361?s=21

As always amazing dev update! These things are getting so big it’s hard to comment on everything!!! That’s a good thing by the way! :smiley:
Keep up the great work @maidsafe!

28 Likes

Yes, trying to answer his questions right now :smiley:

27 Likes

Looks like we are both here :smiley: I just attempted to answer :wink:

20 Likes

:slight_smile: can’t wait for the new website to pop up and amaze the world =)

ha! glad to hear that this will be handled automatically now =D [i was one of those who didn’t recognize there is an enviroment-variable necessary and switched for mock to the ‘old’ browser xD …]

4 Likes

I had replied but not pressed enter :frowning: sry guys was caught in hangouts there for a while. Cheers for the replies, all helps.

13 Likes

Any other feedback about PARSEC from cryptocurrency devs or academics? Has there been any response from the DAG-based currencies that MaidSafe has reached out to?

5 Likes

Okay I’ll admit it. I did not understand anything you just said there. I think clarification and more information is required.

Also please define what a “semantic web” is and what applications for this would be. Am I just missing some articles or something?

1 Like

Moving right along! Great work once again team, looking forward to the new video.

One project so far, but only an inquiry at this stage, not naming any names on that one as it was not public. Nothing concrete as of yet, but it is a lot to take in. So we will see.

8 Likes

Was about to reply but I see a few more knowledgeable bods are lining up beneath me so I’ll wait …

It would be cool if it was SOLID and worked on both SAFENetwork and clear net! :sunglasses:

4 Likes

It’s actually great you ask if it’s not clear @Blindsite2k. Currently the DOM API exposed in Beaker browser is a thin layer coded in the safe-app plugin that it’s shipped with it. Due to some technicalities that may not be important now (ask if you wanna know though), these functions return handles for the objects you create and interact with thru this DOM API, e.g. you get a handle for the safeApp instance you create to connect to the network, for a MutableData object that you would then use to commit to the network you also get a handle, etc.

In Peruse we were working on a couple of approaches to be able to not only remove these handles, but also in trying to have the DOM API to be much more similar and cohesive with the API exposed by safe_app_nodejs. The latter could be specially beneficial for developers trying to have apps which can be released as desktop apps but also deployed as web apps, so their internal code to interact with the SAFE Network will be almost the same and therefore very simple to port. This also makes it straight forward to be able to use the Web API once you learned how to use the safe_app_nodejs API, and vice-versa. I hope it helps in clarifying.

18 Likes

The semantic web is an idea of Tim Berners Lee to create a WWW of data rather than documents. Documents are made to be read by humans whereas the semantic web would be made to be read by machines and humans. I’ve found it hard to find a good high level description that doesn’t get too complex too quick but this works, for me at least

“The Semantic Web, Web 3.0, the Linked Data Web, the Web of Data…whatever you call it, the Semantic Web represents the next major evolution in connecting information. It enables data to be linked from a source to any other source and to be understood by computers so that they can perform increasingly sophisticated tasks on our behalf.”

@happybeing has been working on integrating SAFE with SOLID, a project built on semantic web protocols.

10 Likes

Great update as always!

Really great, will have a look tomorrow after a refreshing sleep and some coffee :+1:. Probably will study the video by @pierrechevalier83 first.

9 Likes