SAFE Network Dev Update - September 27, 2018


#1

Summary

Here are some of the main things to highlight this week:

  • We have been working on delivering and running a test of Crust that should help us to see how the NAT traversal works for different people using all kinds of operating systems in a variety of cities, countries, and continents. So far we have been running this test internally at the company, with some encouraging results: 75% of connections are successful. Further to some tweaks and bug fixing we will be looking for your help to run a public test in the very near future!
  • Here’s a sneak peak at what we’re building to support the tests. Have a poke about, and kick the tyres of the evolving frontend UX prototype of the Crust Testnet analytics dashboard.
  • @goindeep has posted a really good article titled “The Problem with Money…Online” to the SAFE Network publication on Medium. Have a read… give it some claps!
  • We had an internal review of the progress that has been made on a PoC for XOR URLs we started working on a few weeks ago. We recorded this meeting and we are happy to share it with you all.
  • For those who’ve been digging into the APIs (or maybe those who’ve been put off by some complexity), there’s a preRFC on API improvements/additions over on the Dev Forum, where eyeballs and ideas would be greatly appreciated.
  • We got the Malice in PARSEC RFC agreed and added to the GitHub repo.

Marketing

We’ve been busy bees again this week with a few things to update you on…

Big news is the internal (meaning internal to the company and including all remote members of the team) Crust test we have been running. Broadly speaking we saw around 75% of nodes successfully connect via hole punching. While this very early result is really encouraging the sample size is very small and further to some tweaks and bug fixing we will be looking for your help to run a public test in the very near future! For the remaining 25% it will still be possible to port forward to connect. Great to see the hard work of @ustulation and @povilasb pay off, so we can almost forgive Povilas for daring to take a holiday :smile: (seriously though great job guys!)

This represents the start of several releases that we’ll be making as we roll out Alpha 3 and we will keep you updated each step of the way via a series of posts and articles. Each will focus on a different feature(s), and at times will accompany new pieces of software for you to get your hands dirty with.

As you may be aware, Crust is a critical part not only of Alpha 3, but also to our vision of the decentralised web as it enables users, in a very non-technical plug and play way, to join the network with commodity PCs from behind home routers. @dirvine recently highlighted how important this layer is with one of his customary and very colourful explanations when asked how fundamental Crust was within the network “…you wouldn’t build a skyscraper on sand, would you”!

Our approach to a decentralised network is clearly distinct from all other projects for whom a decentralised web means running servers from home and/or going through the complexities and risks of configuring a home Router. Their approach is not wrong, it is just a different approach which can sometimes become blurred in the marketing noise of ICOs and YouTube interviews. But when one of your objectives is to build a new fully inclusive and secure web, then we believe our technology is better suited.

It is important to note that the entire MaidSafe team have been contributing towards Alpha 3 which is something that has not always happened given the diversity of roles here at MaidSafe. Great to see even the HR, Finance and Admin peeps all involved in Crust testing!

We got the Malice in PARSEC RFC agreed and added to the GitHub repo. While we’re discussing RFCs it’s probably best to highlight that currently we’re looking to focus on critical-path RFC items where possible so the team aren’t distracted away from their work on Alpha 3 features.

We will be seeing a few more RFCs being published soon to kick the tyres of some new planned features. @Joshuef is already working on a DNS RFC which is soon to drop, and a Web ID RFC after that, along with others from the back end team.

We would like to take this opportunity to shout out to @mav, @neo, @happybeing, @jlpell, @draw and others who are extremely generous with their time and really engage in the RFC process. The community is surely much stronger for having you guys involved.

@pierrechevalier83 attended the second London meet-up on Wednesday at the Flying Horse pub. Thanks to Oscar (@opacey) for taking command here. It looks to have went really well with community superstar @JPL even doing an impromptu SAFE Network demo!
Read Oscar’s summary of what was covered here.

Last weekend @dugcampbell gave a talk to the Modern Heroes Conference in Italy entitled ‘The Internet is Broken: Welcome to the SAFE Network’. The conference was a good opportunity to get exposure to many different influencers so we were glad that Dug could represent the SAFE community.

Very pleased to see one of our forum regulars (@goindeep) has posted a really good article titled “The Problem with Money…Online” to the SAFE Network publication on Medium. Have a read… give it some claps!

Recruitment

Since the last recruitment update there has been a lot going on. At the end of August we had offered a candidate the role of Dev Ops Engineer, however, the applicant decided to accept a role closer to home taking us back to square one in the recruitment process. We have been busy trying to find a replacement for this role which has not been an easy task. We are however excited to announce that we found another candidate who really impressed @StephenC and @ustulation. The candidate has been offered the role and accepted (watch this space for more information). @StephenC is delighted to have an additional member to his team and hasn’t stopped smiling since he heard the news the candidate accepted :). Now that we have successfully filled the Dev Ops Engineer role, Victoria will be switching her attention back to the other role for Stephen’s team which is the Software Test Engineer position.

@nicklambert and @victoria have been kept busy interviewing potential applicants for the Marketing Strategist position this week. We have met some really interesting candidates so far, each bringing something unique to the table. These interviews will continue this week and we hope to progress to the next stage shortly - tune in next week for more information.

The other role we are recruiting for at present and probably the most important is that of a Network Engineer to join the wonderful @povilasb in the Crust team. Adding an additional body to the team to work alongside Povilas will enable us to ensure the library is moving towards its strategic aims of being fully asynchronous, with less dependencies and providing greater platform coverage.

Recruitment will be making a regular appearance on the dev updates going forward to keep you up to date with all the goings on.

User Experience

What a fun week. Full speed, hands dirty, pixel smithing. It’s all about the Crust Testnet that you’ll soon be able to get up close and personal with, helping us punch a few holes from all corners of the globe.

But how do we make the results and data produced by this test as useful and meaningful as possible? Sure everyone loves digesting raw log files before bed, but to make things really fly we need a dashboard to slice, dice and analyse.

So here’s a sneak peek at what we’re building to support the tests. Have a poke about the evolving frontend UX prototype of the Crust Testnet analytics dashboard.

Questions and critique welcome. You know what to do.

SAFE API & Apps

Now that the major issues with safe_app_java have been resolved, we have raised a pull request to get these fixes upstream. With the CI passing and the code under review, we are putting together a sample Android application that demonstrates the safe_app APIs on mobile. In parallel, we are also working on the getting started guide for developing Java/Android apps for SAFE Network.

We started working on having all the experimental APIs be exposed only when a specific flag/argument is set, and hide them by default. We are planning to implement this on both the safe-app-nodejs package and the SAFE Browser. The idea is that after having such a feature in the safe-app-nodejs package, the SAFE Browser should be able to expose a switch in the UI to enable the experimental APIs, and/or even experimental browser features.

We continued working on fixing some issues in the SAFE Browser towards having a new release in the next few weeks. Apart from what’s mentioned above about the flag for experimental APIs/features, we are focusing on further stabilising it now, as we see the community has been actively using it and a few bugs were reported, which we really appreciate.

In addition to this, as many of you may remember, we shared a post a few weeks ago about a proposal related to how to share data through URLs without the need to use public names, i.e. by simply sharing a URL which contains the information needed to fetch the data from any location on the network (which we started calling XOR URLs). This week we had an internal review of the progress that has been made on a PoC we started working on since then, that we recorded and that we are happy to share with you all.

The recording is just a meeting in which we review some scenarios where the XOR URLs would be used, and how they open up possibilities for other things we believe will make a difference, and allow users to browse native linked data. We hope this shows what the vision is and how we are proposing to move forward with these specific aspects of the network. Please note we had an issue and @hunterlester’s voice was not recorded, but we are still sharing the recording as is, since the answers to his questions should let you know what the questions were. The following are screenshots of some of the things we reviewed during that meeting:

Finally, for those who’ve been digging into the APIs (or maybe those who’ve been put off by some complexity), there’s a preRFC on API improvements/additions over on the Dev Forum, where eyeballs and ideas would be greatly appreciated.

SAFE Client Libs

This week we finished integrating SAFE Crypto into SAFE Core (MAID-2725) and we have been working to replace Rust Sodium in SAFE Authenticator and SAFE App with SAFE Crypto (MAID-2726). The work is actually finished, but a single test has started to fail and required investigation. In addition, we finished adding new required functionality to SAFE Crypto, having the PRs reviewed on GitHub, and responding to review comments.

Routing

As expected, our focus has continued to be testing of dynamic membership in PARSEC this week. We finished implementing the smaller functional tests for this part, and fixed the few minor issues in the production code that we detected last week.

As we have been writing more functional and integration tests, we found opportunities to improve the testing framework.
That work has now been completed too and creating future functional and integration tests will become somewhat simpler and produce higher quality tests.

Work has also gone into larger integration tests for peer addition and removal, and these have begun to bear fruit in that we have identified a couple new issues in the production code. We’ve already addressed these issues. The integration tests have been written and are being reviewed. We expect to reach feature completion on the dynamic membership part of milestone 2 soon. This will be followed by a longer period of soak testing where we hope to catch any outliers before being able to call these features “production ready”.

As anticipated last week, we’ve managed to make some parallel progress on the next main batch of work; namely handling malice within PARSEC. A couple of tasks involving the detection and handling of incorrect Genesis blocks have been tackled.

We also had discussions on some key aspects of the upcoming tasks. These discussions lead us to notice the need to adjust the way we handle meta-voting in preparation for detecting gossip from invalid creators. Work has begun on that task also.

Next week, we expect to make progress on malice detection. There is a lot of work to be done there, as you can see by reading the RFC so this should keep us busy for a little while. Onward and upward! :smile:

Crust

Povilas is also finally back from vacation and working full speed. This week, we have been working on delivering and running a test of Crust that should help us to see how the NAT traversal works for different people using all kinds of operating systems in a variety of cities, countries and continents. Nikita volunteered to develop the Crust client and proxy apps, both Rust CLIs and in collaboration with the rest of the team, has done a great job with a quick delivery of the same. The test works as follows: we have a proxy that runs on one of our droplets and clients have it as a hardcoded contact. Once clients are connected with the proxy, they send their connection information which is subsequently forwarded to other clients. With that information, the clients can attempt to connect with each other: once complete, they send the statistics to the proxy that are then aggregated and from which we can infer the percentage of successful connections. So far we have been running this test internally at the company, with some encouraging results: 75% of connections are successful. David Geddes managed to (not-so) secretly click Spandan, Stephen and Nikita, with Povilas on a Hangout, going through some test results:

We’ve also been extending the Crust and p2p crates to expose some connection related information to cater for the test we’ve been running internally. We’ve updated the p2p crate to return detected NAT type during hole punching (PR) and public address (PR). Then we implemented a couple of helper methods in Crust: Service::probe_nat() and Service::connect_all(), which will allow to use Crust to build some diagnostics tools that collect useful information about misc connections: direct, hole punched, uTP, TCP, etc.


Pre-Dev-Update Thread! Yay! :D
#2

This week I’m first (like Remco Evenepoel). Going to read now!

Edit: it seems you’ve been busy indeed!

“…you wouldn’t build a skyscraper on sand, would you”!

This reminded me of the foundation works of the Jeddah Tower that should become 1km high. Maybe not on sand, but not the best of surfaces to build on, result: a lot of foundation piles up to 105m deep…
Anyway looking forward to test Crust and thanks for the shout out.


#4

Second by just a thousand of seconds. This update I like.


#5

This is amazing news! Great work again. Can’t wait to try it out! Cheers


#6

One to watch at Flanders!


#7

A nice fat update, thank you :smile:

Just posting to highlight Hyperswarm in case it’s relevant to the p2p connection testing featured today (I believe Hyperswarm includes hole punching). It’s by the same guy who created the fuse-bindings npm package I’m using for SAFE Drive. I guess you’ve seen it and I haven’t looked into it myself, but in case it is relevant…


#8

Quick question! Is the XOR URL sharing exposed in the experimental dev branch of the WebID Peruse browser?? Curious if these changes are available yet or how quickly things will change.

A lot to be excited for here and so many people on the team, all doing such amazing and exciting things! What’s even cooler to me is how much of an actual team @maidsafe is. Seeing screenshots and hangouts where you’re all working together is such an inspiring thing. All I can say is WOW!!! Great job everyone!

Edit: I kind of get an answer to my question at the end of the video regarding when API’s could change. These are pre-RFC PoC’s but it’d still be cool to mess with if it’s available in an experimental branch!?


#9

Juicy update. Great to hear how the testing and alpha 3 is coming along.

Looks like the meet up last night was pretty good too. Hope to make one of them in the coming months.

I’ll be looking out to hopefully get involved in the testing.

Keep up the amazing work Maidsafe!


#10

@goindeep Great Article…Keep them coming!


#11

@bochaco Super screencast demo Gabriel, it’s long but really nicely walks through some nice features that will I think be an important piece of the SAFE jigsaw.

I know there’s an RFC coming, but so I don’t forget I want to jot down the main thought it prompted in me…

Which is that if all the resources linked in website are addressed directly (as in the demo) using CID encoded xor addresses, whenever a page/resource in a website is changed, all the other pages that reference it (with a safe: link) will also need to be changed.

I see two issues there:

  1. one it could force a lot of extra updates
  2. what happens when two pages reference each other (or there’s any circular references)? A tree would be fine, but most websites are graphs. I don’t think that works because when you change the content of one, its address changes, so you have to change the other, and now you are in an infinite loop.

So I think a website might still need a container where the URLs of individual files can be looked up. I was thinking it could be an immutable data, but then you still get into 2. so I think websites will always need to go via a mutable data (or something that has a fixed URI regardless of its content) in order to allow mutual links to work.

I can’t immediately see a way around that. Maybe something to chew over in the RFC.

None of that takes away from this work though, or the very neat solution that CID refs to SAFE data creates for sharing, linking and so on. And I love the built in rendering of data types in the browser. It’s all very juicy!

One sneaky thing near the beginning, words along the lines of “we’re probably going to replace NFS containers with RDF” lol :laughing: . I’m saying nothing!


#12

It’s a bit voyeuristic, but I often follow the github activity of team members, so you might find interesting stuff here https://github.com/bochaco


#13

Must be referring to @foreverjoyful and the unsurpassed talent he brings to the table :laughing: .


#14

Yes, correct @Nigel, this is very early stage, so it’s not even in the dev branch, this is the branch where I have the code for that demo: https://github.com/bochaco/safe_app_nodejs/tree/issue-261-feature
If you wanna play around with it you’d need to build Peruse yourself making it dependent on that branch of safe-app-nodejs.

If the link is to an immutable resource, then yes, it’s a case by case thing.
It also depends if the link references a version or not, if you look at the XOR URL discussion you will see we are envisioning that the version of the resource could also be part of the link, so you could have a link to the latest version of a resource (no version info in the link), or to an specific version of it which is part of the URL/link.


#15

That is what your demo was showing, if I understand it, so you can’t create a website that way unless it is a tree rather than a graph.

If you are using mutable data versions, then if all versions of an immutable data are always available, then I can see how you can get around this. I think!

Is it the case that all versions of an immutable data are always available?

(I can see that might be how it’s implemented, but I haven’t heard that stated, and it’s a big deal, so why not! :slight_smile: ).

If so, it makes David’s vision complete, but it’s such a mind expander that I am not sure it’s true?


#16

That was just one of the scenarios we were looking at, we were not saying that all websites need to be that way, it’s a case by case, and as you say, if it’s a tree you have that option assuming you want to make it immutable, which again it will depend on what you are trying to achieve.

This is already implied by the fact that it’s being immutable, unless we think immutable doesn’t imply undeletable :slight_smile: , the question is if there will be an immutable data type which can be referenced using a version, if there is, then you certainly will want to be able to reference versions from links as well.


#17

Sorry, I said immutable but meant mutable. What I meant to say was…


#18

Just to give an example of something I personally would love to be able to do, once I start using a webapp, and I trust its behaviour, I can always use that version of the webapp, regardless if the author updates/upgrades it, so I can always use an immutable version of a webapp.


#19

Agreed, but it would be tricky to do this without involving any mutable data. Such apps would not be able to reference ‘back’ up the tree! Or is this another of my ‘it’s late’ moments? :slight_smile:


#20

Yes, it seems that’s correct, you’ll need something that can be amended/mutated with a new version but keep its address, which at the moment we don’t have.


#21

I see it could work for a lot of apps, single page apps might work quite well. A problem will be knowing whether a Web app is immutable, or if it has a link to something that is mutable.