SAFE Network Dev Update - December 6, 2018

Summary

Here are some of the main things to highlight this week:

  • We have 7 new team members joining us this week (of which 3 are interns).
  • We are happy to announce that snapshot releases of the Android library are available as safe-app-android and safe-app-android-dev on the oss-snapshot-local maven repository. The supporting API documentation is also available for reference.
  • The API documentation for safe_app_csharp is now available for reference.
  • We made available a new patch release of the safe_app_nodejs package (v0.10.1), which includes a fix for two minor issues. For more details about the changes, please refer to the changelog.
  • We published a new branch in the Routing repository called fleming because we feel like the changes we’ve been working on behind closed doors for the last little while are definitely a step in the right direction. Please note that it is still very early days as aspects such as performance still need significant work before the code can make it into a testnet.
  • As of this week, the Routing team has split into two sub-teams.

Marketing

Marketing has been focused this week on creating material to help explain dynamic membership within the context of PARSEC. You’ll see the outcome of this work during the next week, when we’ll also be sending out a new subscriber newsletter (reminder: if you haven’t already, please sign up today in the footer at https://safenetwork.tech/). We’ve also finally taken down the SAFE Network Wiki. Whilst it has contained some great information over the years, much of the content is now dated and instead will be covered by a combination of the existing websites and other future work. This will enable us to keep the information up-to-date moving forwards. Finally, next week, @dugcampbell will be out speaking about SAFE at a meetup at Strathclyde University (https://www.meetup.com/TheCryptograph-com-Cryptocurrency-Meetup/). We’re also looking forward to welcoming a new start to the team from Monday - but more of that to follow next week!

Recruitment

After not having a recruitment update last week, we have a number of new starts who have joined the team this week :tada:

We would like to welcome Calum (@karamu) who has joined the QA team today as Software Test Engineer. He will be working alongside @StephenC, @chriso and @srini.

We would also like to welcome some new faces to our Chennai office who joined at the start of the week. We have 6 new starters of which 3 are interns.

@vigneshwara and @murali (intern) have joined the team to work alongside @AshwinKumar, @lionel.faber and @ravinderjangra.
@siddique and @Ruthra (intern) have joined the UI/UX team and will be working alongside @Shankar and @JimCollinson.
@Manav (intern) has joined the JS team and will be working with @bochaco, @joshuef and @hunterlester.
@Yogesh will be working in the SAFE Client Libs teams with @nbaksalyar and @marcin.

We also have another new team member joining us next week but we will need to keep you in suspense until the next dev update :smile:

SAFE API & Apps

We have started planning the final test and release phase of the safe_app_java and safe_app_csharp APIs.

As part of this, we are happy to announce that snapshot releases of the Android library are available as safe-app-android and safe-app-android-dev on the oss-snapshot-local maven repository. To include these snapshot libraries in an Android application, follow the instructions available in the safe_app_java repository. The supporting API documentation is also available for reference. Until the release of the SafeAuthenticator mobile application, developers can try out the API using the safe-app-android-dev package which is built for the mock network.

@ravinder_jangra and @ashwinkumar have completed the tasks planned for safe_app_csharp v0.2.0. They added the Azure DevOps CI to build and run tests on .NET Core MacOS, Android x86_64 and iOS simulator. The API documentation for safe_app_csharp is now available for reference. The documentation is autogenerated and published from CI and in sync with the master branch.

The internal testing of the SAFE Browser has been going very well, with some issues found which were also being fixed in the last few days. The browser is now ready for a last iteration of tests and verification before we can release it. We also discovered that one of the issues spotted in the browser was due to how errors were being thrown from the safe_app_nodejs package when the experimental web interface was being accessed and the experimental APIs were not enabled. We therefore worked on an enhancement for this in the safe_app_nodejs package and made available a patch release v0.10.1, which also includes a fix for another minor issue in the fetch experimental API. As always, for more details about the changes, please refer to the changelog.

In parallel to the last iteration of internal testing for the SAFE browser, we’ve been working on our example applications to make them fully compatible with the new browser and API. We were making the necessary changes to Patter, WebID Manager app, Web API Playground tool, and the Web Hosting Manager desktop application, and we will be publishing them along with the SAFE browser.

We are now also starting to analyse what the next steps are for the browser and the front-end applications, so apart from some technical meetings we are planning to have, we already started to work on enhancing the documentation in our safe_app_nodejs package, not only content-wise but also trying to use a different doc generator to improve the generated document style and readability.

SAFE Client Libs

This week we have been looking into making the Redland RDF libraries, written in C, available for use by Rust code. We successfully used bindgen to generate unsafe Rust bindings for the rasqal and raptor libraries, and we are currently working on testing these bindings, both for parsing serialised RDF files (using raptor) and for querying them with SPARQL (using rasqal). We also looked into using immunant’s c2rust tool to translate the Redland RDF storage code into Rust automatically. The results were promising: although the generated Rust code required some manual adjustments, we were able to get an in-memory RDF storage implemented and are looking to expand the code (in Rust, of course!) to support MutableData storage of RDF.

In parallel, with the input from the tests, we are starting to draft the first RFC about making RDF a part of the SAFE Network stack. As this feature involves a number of changes in some of our core libraries, with this RFC we plan to define the scope of the future changes more precisely and answer the question of how it should be implemented.

Routing

The focus this week has largely been on Parsec again; further forms of malice-detection and malice-handling are now covered and we have become more focused on improving Parsec’s performance.

As we continue to pick off the malice-detection tasks, we’re continually re-evaluating their correctness. Indeed, in one particular case, a situation that we originally thought would indicate malice turned out to be possible with only honest nodes. Our more complex test cases helped us identify this, so we simply stopped considering it as malicious behaviour. We live to learn :smiley:

As well as these tasks, we made some inroads into improving the documentation in the Parsec source code; this being one of the primary means by which other developers will come to understand how to use Parsec.

Today, we are also releasing our initial jab at moving the Routing codebase towards where it needs to be for the Fleming milestone: Routing nodes from home. It is still very early days as aspects such as performance still need significant work before the code can make it into a testnet, but we reached a point where we feel like the changes to the Routing crate on which we’ve been working behind closed doors for the last little while are definitely a step in the right direction, and hence worth sharing with you, the community. We published a new branch in the Routing repository called Fleming. Changes include integrating Parsec for getting a rigorous ordered consensus where we used to only rely on eventual consistency through the data-chain. This is a version of Parsec that completely ignores malicious activity for now as we are still bringing that to completion on the Parsec side. Scenarios that used to be nightmarishly complicated with eventual consistency, such as complex edge cases involving multiple sections merging have become much more manageable. We got a complex test scenario called aggressive_churn to work with the new code. It is not the end of the road though as Parsec is currently much slower than we want despite recent improvements, which indicates that we must pay a lot of attention to this aspect before we can start thinking of any testnet.

As of this week, the Routing team has split into two sub-teams.

One sub-team will be looking to build on our work in the Routing codebase and separate as much as possible distinct abstractions within the codebase. We will attempt to formulate these abstractions into independent layers with minimal coupling between different layers. This will be a difficult job; Routing is necessarily a complex block of code, but it has accrued some technical debt since its inception which makes it particularly difficult to rationalise about the implementation and impact of high-level requirements of Routing. Splitting Routing into independent parts will lower that barrier as well as making the code more testable and robust.

The second sub-team will remain behind in Parsec, but with the focus shifting very much onto performance improvements until the Routing code can function to a satisfying degree. Once performance is acceptable, this subteam will continue to tackle all other “Routing milestone 2” items.

The two sub-teams will remain closely connected, with briefings and inter-team code reviews planned, to ensure that neither team becomes too specialised and that all members continue to stay up-to-date on both Parsec and Routing.

Crust

We finally finished the tasks planned for socket-collection v0.4.0 and published a new version on crates.io. This version includes encryption, enhanced documentation, more tests and a protocol agnostic socket that wraps the TCP and UDP protocols and provides a uniform interface. With that in place, we were able to start working on the Crust library itself. The next release is planned out here. We already did some refactoring: updated deprecated Mio code, removed pseudo-hole punching code, the proper NAT Traversal will be covered by the p2p crate, fixed service discovery listener and started working on socket-collection integration.

95 Likes

Taking this one home, cheers!

By the way “great update as always” I dare saying before reading it. Now that’s trust!

26 Likes

Thanks so much for working so hard. Remember to take a nice healthy break at Christmas.

Edit: Welcome to the new team members!

17 Likes

Looked at routing Fleming on github and there seems to be alot that have been added, nicely done!

Is there any percentage goal for efficiency improvement related to Parsec and routing, which you guys think will be needed to achieve reasonable performance in Fleming alpha 3?

14 Likes

All these new recruits.
Hope you don’t pay them at Swiss salaries, cause I will be worried for the budget.

Can we hope for a Christmas present on any deliverable ?

15 Likes

I approve of this update

9 Likes

Is it try or a touchdown? A six or a home run?
Either way the crowd is cheering!
Great update start to finish.

13 Likes

Amazing work guys. Shifting into performance for PARSEC is fantastic. Great progress on the browser, client libs, sockets as well. Do remember to take solid time off for the holidays.

12 Likes

Thank you for the hard work Team MaidSafe!

I really want to know the names of everyone to thank, so I made a small list, but I know there are people missing from it… Can I ask to tell me who is missing?

|1|Adam
|2|Andreas|
|3|Ashwin|
|4|Bartlomiej|
|5|David|
|6|David Geddes|
|7|Dug|
|8|Chris|
|9|Francis|
|10|Fraser|
|11|Gabriel|
|12|Hunter|
|13|Jim|
|14|Jon|
|15|Josh|
|16|JP|
|17|Kayley|
|18|Krishna|
|19|Lionel|
|20|Lindsey|
|21|Marcin|
|22|Nadia|
|23|Nick|
|24|Nikita|
|25|Pierre|
|26|Povilas|
|27|Qi|
|28|Ravi|
|29|Sarah|
|30|Shankar|
|31|Sharon|
|32|Shriram|
|33|Spandan|
|34|Srini|
|35|Stephen|
|36|Victoria|
|37|Viv|
|38|Vinitha|

And welcome to all new team members!

30 Likes

Thank you guys for the great update.
:clap::clap::pray::pray:

10 Likes

It really feels like it’s full steam ahead! Another amazing update team!

9 Likes

Thanks team for hard work but It looks like alpha 3 is still far away

4 Likes

Without a lot of hard work and powering into the unknown, alpha3 will always be far away.
Alpha3 is closer now than it was last week. Next week it will be another little bit closer.
If this was easy, any bugger could (and would) have done it.

Well done guys, Keep it up. Have you noticed we’re impatient, but hopefully confident and realistic?

17 Likes

These updates are getting so good they are hard to parse! I feel like commenting my major points of support is itself a mini sub-update! :smile: this is of course a great thing.

Cheers to the PARSEC and routing team(s)! This RDF stuff is exciting as well. Am I right that the direction that is being taken is to allow for easily searchable data network wide through SPARQL? Something I’d love a brief on but will be looking into more to get some sort of grasp.

17 Likes

That’s quite a team you have there now! Great to see Fleming branch starting to take shape too!

12 Likes

Great news.

I’ve updated safe-network-explained to replace wiki links with links that will work.

Here’s the changes

If anyone has suggestions for replacements it would be appreciated:

  • Glossary (currently changed to How It Works)
  • Attacks On The Safe Network (currently changed to the forum homepage)
  • How Will MaidSafeCoin Be Converted To Safecoin (currently changed to What Is Maidsafecoin)
18 Likes

Yes, that’s part of the beauty of Linked Data. SPARQL as a language supports federated queries, which are basically queries that agregate data from multiple sources. You can think about these multiple sources as just multiple Mutable Data objects on the SAFE Network, so it can be one of the ways of using it.

20 Likes

That’s brilliant! Thanks for the confirmation Nikita. I’ll do some reading tonight :slightly_smiling_face:

8 Likes

Thank you team. Not able to see when A3 will be complete. Team is definitely growing. Just before the xmas break. And developments sound very positive. Other than that i can only pray every night for completion and no glitches :relaxed:

8 Likes

Welcome to the new starters, especially the interns. How good will it be to have on their resumes they were assisting when Alpha 3 on Safenet happened.

11 Likes