SAFE Network Dev Update - November 8, 2018


#1

Summary

Here are some of the main things to highlight this week:

  • We concluded v2 of our Crust Test yesterday.
  • As of today, we’ve launched an email subscriber list. Sign up (using the form on the footer of the safenetwork.tech website) to receive the latest news on all SAFE Network developments, updates and releases.
  • We published a new post on Medium titled ‘Why The Internet Needs Evolution (Again)’.
  • We have an Executive Assistant joining the team on Monday to take over Nadia’s old role, we have a Software Test Engineer joining Stephen’s team on December 6 and we have a Marketing Strategist joining the marketing team on December 10.
  • The RFC for the Public Name System has been proposed. We’ve set up a discussion topic on the Dev Forum for those interested in digging into proposed RDF examples and the meat of content resolution on the network (and using RDF data to provide clarity/structure and flexibility).
  • The next SAFE Client Libs release (0.9.0) is on a good track and is scheduled to be released next week.
  • The Routing team has been working on preparing Parsec for a clean Routing integration while also progressing on malice detection tasks.

v2 of the Crust Test

As you will have seen yesterday we concluded v2 of our Crust Test. Thank you to everybody who took part once again - we hope you found it as enjoyable as we did to test-drive the power of Crust by connecting to other community members around the world.

Early next week, we’ll be sharing a summary of the results together with a few of our thoughts. But for the purposes of today’s update, here’s a quick recap:

  • We saw a total of 11,584 unique connection attempts.
  • Of those attempts, we had a 75% success rate in connecting using one of TCP hole punching, UDP hole punching or Direct Connections.
  • We had computers taking part from 37 countries:
    • Argentina, Australia, Austria, Belgium, Brazil, Bulgaria, Canada, China, Croatia, Czechia, Finland, France, Germany, Hong Kong, India, Indonesia, Ireland, Malta, Mexico, Netherlands, New Zealand, Norway, Poland, Republic of Korea, Republic of Lithuania, Romania, Russia, Slovakia, South Africa, Spain, Sweden, Switzerland, Taiwan, Turkey, Ukraine, United Kingdom, United States.
  • Connections into and out of China were very strong, with an 88% success rate overall.

Many of you will now be fascinated to hear whether these results are good/bad and in line with our expectations. We plan to carry out a little bit more analysis on the results received to date and we’ll be summarising a few thoughts in detail in a post next week. Keep an eye on Medium to find out more - or even better, sign up for email updates to receive them straight into your inbox. But without spoiling the surprise - there’s a little work to do but we’re very happy :smile:

Marketing

As of today, we’ve launched an email subscriber list. You’ll find it on the footer of the safenetwork.tech website. Why is this big news, you might wonder? Well, there have been many discussions over recent months internally as we’ve tried to balance the benefits of pushing notifications to interested community members outside the Forum whilst resisting creating lists of supporters. But there’s a clear benefit in this case: only those who wish to sign up to the list should do so. We don’t mind what name or email address you use. We’d just like to have the opportunity to ensure that you all receive all SAFE Network news when it’s hot off the press. So if that sounds good to you please do go and sign up now.

You’ve hopefully seen the ‘Why The Internet Needs Evolution (Again)’ post on Medium. We’ve gone down the route of outsourcing an infographic for this post so we’d appreciate you sharing this far and wide if you agree with the sentiments. And it wasn’t just marketing that have been working on new content, @Pierrechevalier83 also treated us to his own perspective on how SAFE and SOLID complement each other well in his Medium post, having been at Mozfest a couple of weeks back.

Finally, with a couple of new starts in the team recently, we’ve updated the MaidSafe site on the SAFE Network - you can get online with the Alpha 2 Network today and check it out at safe://maidsafe.safenet.

Recruitment

This week has been an exciting week for the recruitment team as all of the roles (bar one) have been filled :tada: :tada: :tada: :tada:

We have received confirmation that our Marketing Strategist will be joining the team on Monday, December 10.

@Victoria conducted final stage interviews for the Executive Assistant role and was really impressed with the quality of candidates. The successful candidate has been offered the role and she will be joining the team on Monday, November 12.

@StephenC, @Chriso and @victoria conducted final stage interviews for the Software Test Engineer role last week. We have now offered the successful candidate the role. He has accepted and will be joining the team on December 6.

The Network Engineer role is still ongoing, however, we have a number of candidates who are in the early stages of our recruitment process at the moment. Hopefully, we can start to make progress on this role soon :crossed_fingers:

QA

It’s been a busy and productive week for the QA team at MaidSafe. As mentioned in the recruitment section above, we have offered the Software Test Engineer role, which will bring our QA team up to full strength. Bugs, here we come! :muscle:

@Chriso has settled into the company well and has already contributed to automating our build and deploy process for the v2 Crust Test. This proved to be a great learning experience for him getting to know Travis CI, AppVeyor, the Crust inner workings, as well as how we operate as a team in projects such as this.
With the closure of the v2 Crust Test, Chris will now be focussing on getting to know our other repositories and their CI processes. This week he is meeting with @Krishna and @ustulation to get an overview of the front and back end. He will be looking to identify areas he can streamline by using his DevOps automation experience.

SAFE API & Apps

We have found room for improvements in the safe-getting-started-android application. They have been implemented and the pull request has been updated. The getting started guide has been updated to match the latest code and a pull request has been raised to the DevHub repo.

Now that we’re done with the dashboard for the Crust Test, our (@AshwinKumar and @ravinderjangra) focus is back to mobile. We are currently working on updating the packages for safe-authenticator-mobile and safe-email-app-csharp. Other minor enhancements planned are using Xamarin.Essentials instead of Xamarin.Auth for SecureStorage, using packageReference instead of package.config and adding a revoke permission feature and respective tests.

The RFC for the Public Name System has been proposed. We’ve set up a discussion topic on the Dev Forum for those interested in digging into proposed RDF examples and the meat of content resolution on the network (and using RDF data to provide clarity/structure and flexibility). As some of you know, understanding and making RDF data can be tricky. We’d like as many people as possible to help us in reviewing and making sure we have the correct representation for this type of data. In this case, it can be summarised as trying to come up with a valid and simple RDF compliant structure for our Resolvable Map and Files Map.

This week we reviewed the current state of the SAFE Browser and safe_app_nodejs repositories, and worked on a plan for a new release of them. Since it has been confirmed that a new release of safe_client_libs library will be available next week, which brings in several enhancements and bug fixes we are interested in, we plan to upgrade the safe_app library in safe_app_nodejs & the Web Hosting Manager application, as well as upgrading the safe_authenticator library in the SAFE Browser. Therefore, we started working towards this new release this week, which will also incorporate ~30 solved issues / function enhancements made on the browser.

SAFE Client Libs

The next SAFE Client Libs release (0.9.0) is on a good track and is scheduled to be released next week. It is feature complete but we want to be sure we fix all known outstanding problems before the public release. The front-end team is helping us to understand what’s missing and how to reproduce some of the bugs that they found. Some of the bugs that we fixed include the MockVault corruption, error when trying to authorise a mock app in the real network, and a multitude of problems related to Java/JNI bindings. However, the mobile team has reported that the Java bindings still have some deficiencies left so we’ll be working on fixing them quickly.

We’ve also started to have more in-depth discussions with the front-end team about implementing new requirements and features in Client Libs. One of the targets that we’re considering is the XOR addressing scheme that was proposed in August. We’re discussing the option of making it a part of SAFE Client Libs so that all supported platforms and languages can use it, and with our proven bindgen tool we’d be able to achieve this with less effort. We think this addition could also lay the foundation for further expansion of features linked to RDF and Solid, and likely there will be more news on this in the coming weeks.

Routing

The Routing team had a very interesting week.

We started test-driving the Parsec algorithm in the context of the Routing code. This work is still private at the moment as we took a few shortcuts such as bypassing some code paths manually. For instance, we don’t want to include malice detection and handling yet as we don’t want to be aiming at a moving target while this aspect keeps being worked on. In this experiment, we simply commented out the code paths that we wanted to avoid running. We will publish this code once these kinds of workarounds are replaced with cleaner alternatives such as feature gates.

This experiment gave us encouraging results and important information:
We were happy to confirm that the Parsec API (Application Programming Interface) conforms to our needs in Routing and it should be really easy to integrate both crates. This is great news as the API goes from our best estimate of our needs to a proven correct abstraction.
With this first trial, we could also start getting insight into how Parsec is doing. It is working as expected so far as far as behaviour goes, but we will need to prioritize performance before we can run more complex and life-like scenarios.
The fact that performance is currently less than ideal didn’t come as a surprise as we have left this topic aside so far with plans to tackle it later in the milestone.
However, we now decided to jump some of this work up the priority list: we will improve performance with dynamic membership and without malice detection in parallel with the continuing malice detection work.

From the Parsec side, some work was put in to prepare for a clean Routing integration:
We merged three pull requests related to improving the feature flags that can be passed to Routing to enable or disable specific functionality. This means Routing will now cleanly be able to run Parsec with fewer dependencies and with the malice detection disabled.
We also completed a small modification to the API as we allowed for some arbitrary data to be passed alongside an Observation::Add or Observation::Remove. We also started work on consolidating the customisable interesting_event definition that we started a while ago. We wrote a new test for it and are now improving the implementation to make sure that only opaque payloads use the custom logic.

While all of this work was ongoing, we still progressed our malice detection tasks with another kind of malformed event being detected: one where the self_parent pretends to have a different creator.

We also identified a bug in the invalid gossip creator detection we had coded last week and fixed it. This bug escaped initial testing which shows that the tests we ran could have been better. We are working on making our tests a bit more aggressive to avoid these kinds of issues in the future.

The long-running task of updating the generated graphs to match recent changes to our data representations finally came to an end this week as we merged the last outstanding pull request for it.

All in all, it has been a productive week for Routing and we can’t wait to be able to demonstrate Parsec’s integration in Routing once the initial hurdles are overcome.

Crust

Since we had pretty good results with Crust testing, we will be focusing on alpha-3 integration. Our tests showed that the development branch of Crust is not that stable yet, hence we’ll stick to the Crust master branch for the alpha-3 network, which is based on the Mio networking framework. Although this version of Crust seems to be stable, there is some extra work to be done: encryption, bootstrap cache and p2p integration support are not there yet, but work is in progress. We created a new library (socket-collection) which abstracts some common socket behavior and will make it easier for Crust to use different networking protocols. You can check the current library plan and progress on GitHub.


Crust Test v2 (concluded)
#2

Oh I love every update

Now that is a pretty impressive number :flushed: nice!

Oookay - that makes alpha 3 sound closer than I was afraid it would be :face_with_raised_eyebrow: :tada:


#4

Once again Thursday is the best day in the week! Thank you Team @maidsafe for the perfect job!


#5

Great news, another amazing update! Glad to hear that your happy about the CRUST test!


#6

This is awesome to see!

image


#7

Whoa - sounds like a busy week! I cant wait to see crust, parsec and routing come together!


#8

That’s a lot of countries!!! Truly a global and decentralized connections test.

Sounds like the team is full -1, full steam ahead :smiley:

Really impressed by the progress in PARSEC/Routing.

The XOR URL addressing scheme along with changes to the API storing RDF and then the Public Name System will really help bake in SOLID to the network. I think it will make a huge impact in the long term. WebID manager was already such a great teaser for what can be done.

Glad Josh got his Public Name System proposal out! I’m really curious what kind of an impact it could potentially have. By the way everyone, let’s spare the PNS jokes, okay? I mean I know it’s gonna be big but let’s all be mature. Unlike that Nigel imposter over on the dev forum. :roll_eyes: :sweat_smile:

Great work @maidsafe! :love_you_gesture:


#9

Great update Maidsafe devs,

:+1: :+1: :+1:

Super impressive super ants worldwide, still 158 countries to go :checkered_flag:

Last but not least really LOVED the v2 Crust Test dashboard, keep rocking :ant:s

:stuck_out_tongue:


#10

Great to hear about so much steady progress again!

Really enjoyed both Medium posts, super exciting to see these high level discussions happening about how SAFE (and Solid) will fit into the world.
I wonder if, like mass user generated content on web 2.0, the major development of web 3.0 could be the mass economic involvement of users, particularly as producers/vendors/providers.
Not an easy thing to achieve, but from all angles, it seems absolutely key to me that it becomes normal for everyone to run a vault, whether big or small. True power to the people!


#11

Sounds like another week of extremely good progress and well designed pieces seeming to come together. Thanks for all your hard work. This project is a thing the world needs.


#12

Wow no one sniped to say FIRST!

This poor sap’s missing a shoulder:

edit: wow they all are! Maybe it’s some…style.

You can tell I read the update slowly this time… should be my norm, now.
Not only to catch minuscule things like that, but this felt like one of the stronger updates.
Possibly solely from reading it adequately.

Oh maybe I’m thick but I didn’t see the e-mail subscription part of the website at the footer.


#13

The new race is to get a comment in before somebody can say first :joy:

It’s like silent disco, only silent win!


#14

Another fine update!

we have a Marketing Strategist

So this tweet from today about that job is out of date, I assume?
Edit: if you click for more info in the tweet, you get eventually: ‘This job is no longer available.’


#15

The “Evolution of the Internet” image is wicked


#16

Thanks as always for this update! One question: to what extent will the content in the emailer overlap with content on Medium and/or in these weekly updates?

I keep up-to-date with Medium posts and these weekly updates, and have been trying to prune the amount of emails I receive, but if content in the emailers won’t overlap significantly with other content, I’d definitely sign up.

I imagine it’s also more timely than content pushed out in other channels?


#17

Hi @dlux, the content for the new mailing list will be a mix of exclusive content and a curation of existing posts from other platforms. We will be continually updating the format in the first few emails so any feedback will be greatly appreciated. I would say sign up and if you find that there is no much overlap you can always unsubscribe.


#18

How is the malice detection for Crust doing? I remember there being anti-spam features so one node couldn’t join a group spamming the hell out of them. Is that part still active? Will users be able to join Alpha 3 without being part of the forum?

Great update btw. More and more souls joining the MaidSafe caravan. Warm welcome to all of them.


#19

Amazing progress and loving all the articles being put out. Great work!

Sounds like the crust test went well too.

I’d love to see some of these updates, progress reports, etc being delivered in video format sometimes. I think they can capture a wider audience if done well.

Keep up the awesome work every one!


#20

Great update …

Also - to everyone reading the medium article, don’t forget to hold down the mouse button on the clapper to give max claps - it will raise the ranking of the article and hence raise the awareness of the project!


#21

Anybody else think this is a bit cumbersome being at the very bottom? Maybe a link at the top that will take you directly the signup or maybe even have it right at the top?