MaidSafe Dev Update - January 25, 2018

Here are some of the main things to highlight this week:

  • The SAFE Network Primer is now available from our website. See this forum post for more info. Please read it and share it widely!
  • Our dedicated new Indian office is now open. We are currently looking to expand our team there - Javascript UI developer (x2), UI Designer (x1), Android Developer (x2) and .Net Developers (x2).
  • Nadia Burborough is joining us in the Ayr office as a Technical Executive Assistant, starting this Friday.
  • Support for streaming files (using the HTTP Range header) is being added to the Peruse browser. A new version of Peruse containing this feature will likely be released very soon.
  • Today, we’re publishing new versions of SAFE Client Libs: safe_app 0.6.0, safe_authenticator 0.6.0, safe_core 0.29.0, and ffi_utils 0.5.0.
  • The Routing milestones for Alpha-3 have been published here. In the coming days, the Routing team will detail the first of these and commence with the tasks which will also be added to Jira.
  • uTP is now working reliably in Crust. Also, Crust will soon have fully encrypted communications.

Marketing

Release of SAFE Network Primer

Today we’re releasing the SAFE Network Primer. This is a document written by community members @jpl and @polpolrene. It’s a fantastic piece of work by both and one that we’d love you all to read. You can download the Primer from this post. Please share this document widely - it’s a great introduction to the SAFE Network and one that will be supported by the content of the upcoming SAFE Academy.

CEP Donations Reminder

A quick reminder that the donation addresses for the successful Safecoin CEP are available here. If you’d like to support Joeri and Hypercube’s work, you have until 23:59 GMT on Friday 26th January to donate (using any combination of MAID, Bitcoin, Bitcoin Cash or Ether).

First Website Meeting

We held our initial ideation session this week with the website development agency that we have selected to design and build the new maidsafe.net site. We’ve identified a number of broad themes that we want to highlight in the upcoming website and are anticipating launch in early April. Having discussed high-level site architecture, the next step is for wireframes to be produced that help us understand and improve user journeys through the site. Once those are agreed, the design phase begins and in tandem with that the MaidSafe team will be working on site content (further to having carried out search term analysis). We’ll then progress to development, then testing.

Community Articles

We’ve continued to pull together our Medium presence. @dirvine now has copied across posts from his personal blog to his own Medium publication and @SarahPentland has had an article (‘Don’t all jump on the blockchain bandwagon’) doing well after being picked up by The Startup. From the community itself, we’ve also shared @goindeep’s recent articles (such as this one) whilst @suriana has had a recent article on the SAFE Network picked up by Hacker Noon. We know that we have some compelling messages and Medium is showing great potential for getting some of this content in front of a new audience - so, as ever, please support the message by clapping the articles as you see fit!

Opening a dedicated office space for the MaidSafe front-end team

Last but certainly not least, this week (Monday 22nd of Jan) saw the opening of a dedicated office in Chennai, India. The team there will be leading the front-end R&D for the SAFE Network. It’s an exciting expansion for us and we’re now currently recruiting for a range of roles there in order to grow the team there - Javascript UI developer (x2), UI Designer (x1), Android Developer (x2) and .Net Developers (x2). For more details about any of the roles, please email careers@maidsafe.net. India was viewed as an ideal location for a number of reasons, not least the fact that many within the existing team were already individually based there and having a central office will only enhance their collaborative efforts. In addition, the economy there continues to grow while there is huge demand for Internet-related technologies amongst the country’s vast population. Here are some photos of the new office:

Recruitment

Recruitment continues to get a lot of focus and is something that @Victoria is prioritising above all else. We are pleased to announce that we have hired Nadia Burborough as a Technical Executive Assistant, a role which will primarily help support @Viv keep updated with the activity across the expanding teams. This will be a challenging role as it will require Nadia to rapidly understand the network from a technical perspective while also requiring excellent organisational skills. With Nadia’s extensive project management experience within IT, she is well suited to the task ahead and we look forward to her starting in the Ayr office this Friday!

Some of the other roles are also progressing at pace, two candidates have made it through to the final interview for the UI/UX job, while several engineers have been taking the now infamous MaidSafe coding challenge. We do put a premium on quality here so hiring ratios tend to be very low but we know this is the way to go. We’ll continue to keep you guys updated.

SAFE Authenticator & API

We continue with our work on evolving and stabilising the Peruse browser. We are fixing bugs, making sure all features supported by our Beaker browser fork are also supported by Peruse, and also adding new features which weren’t implemented in Beaker.

As an example of these new features, the CONSTANTS object already available in safe_app_nodejs is now being exposed in the DOM API of Peruse at window.safeApp.CONSTANTS.

Some issues affecting the authorisation requests for containers permissions and shared MutableData permissions were also solved in Peruse.

The ability to store the browser’s state, i.e. browsing history and bookmarks, is now also implemented in Peruse and we are in the middle of internal testing/fixing cycles.

One of the features that was missing in the Beaker browser, and required by some applications being developed by the community, was the support for streaming with the use of the HTTP Range header. The first draft code is already in place and we are doing some minor changes to the way we enable this with the webFetch API function of safe_app_nodejs. Initial testing of this new feature with Peruse was successful so we are envisioning being able to release this very soon to the community. We also took the opportunity to make some minor enhancements and additions to our tests in safe_app_nodejs to keep increasing our test coverage.

React is being upgraded in Peruse to v16, which brings some new features & enhancements, and a reduction of its footprint, and we will be also upgrading Electron to overcome a remote code execution vulnerability which has been discovered affecting Electron apps (on Windows only) that use custom protocol handlers like in the case of our browser app.

We always try to keep up-to-date with tools and libraries as we move forward in the development of our front-end applications, and we try to experiment with them as time allows. This new Parcel release caught our attention as it supports compiling Rust to WebAssembly allowing to import a .rs file just like you can import a .js file, which is something we are aiming at being able to do in the medium term to import our safe_client_libs packages in the browser and/or apps.

In safe_app_csharp, the integration of the Authenticator functions is in progress along with the scope of the tests being expanded. A few minor nitpicks are left and after that we will start integrating the new C# API with the Xamarin apps. The C# API library will be published to NuGet only after the integration testing is completed.

SAFE Client Libs

Today, we’re publishing new versions of SAFE Client Libs: safe_app 0.6.0, safe_authenticator 0.6.0, safe_core 0.29.0, and ffi_utils 0.5.0. Changes in these new versions include: the catch_unwind_error_code function has been removed, the encode_auth_resp and related functions now return FFI_RESULT_OK instead of an error when authentication is not granted, test app creation APIs have been improved, and some callback parameters have been renamed to conform to naming conventions. We’ve also updated the Rust compiler to the latest available stable version (1.21.1), which is now uniform with the rest of our crates.

As we are moving the Client Libs team to Routing, the development pace will slow down a bit in the following weeks, so expect this section to shrink in size in future updates. @marcin will be supervising the further development and maintenance of SAFE Client Libs and he’ll be communicating and working with the front-end team.

Routing & Crust

In Routing, due to the simulation results, we have better insights on some of the Network variables and now we feel more confident in coming up with the milestones for Alpha-3. In the coming days, we will detail the first one into Jira tasks and start working on it. Coming back to the simulation, we wanted to try various triggers for peer ageing so that we can have purely random target sections for relocations as they are a step in the right direction security-wise. We wrote down the rules roughly here of what we wanted to see in the simulation. The main point to note in this one is the trigger for relocation - we hash the Live blocks to find a suitable candidate to be relocated. The simulator has a parameter that can be tweaked to express how many times to hash the given Live block to calculate for relocation trigger. By changing this number we can find a sweet spot that gives us enough churn for peers to age without causing the Network to stall. Also since relocation is random, it adds an extra layer of security. This complements the existing simulation which approaches relocation in a different way. Also, the ability to choose a number for recursive hashing of a Live block allows for a much finer grained tuning. @adam coded those rules and we ran the simulator with good results. Also, we would like to take the chance to thank @tfa who provided some valuable tweaks and evaluation results, which helped us comparing the two simulation outputs.

One important thing that comes up while tweaking the parameters is the average and peak section sizes. While the average is pretty good, the peak sizes sometimes make us wonder if we could be more efficient given those many nodes in a section. Currently, the Elders are all connected to every peer in their section and the swarm strategy is O(n^2) for votes and blocks. We will also be deeply looking into secure gossip and finding out if we can design a system which addresses both: not forcing connection among everyone (thus frugal use of socket descriptors) and also making swarm ideally logarithmic.

This week we finally got uTP working reliably in Crust. There was one last issue with the p2p crate: the UDP hole punching implementation wouldn’t send some final packets, causing the other peer to timeout. However, there is still one known unsolved bug in the tokio-utp crate which is quite obscure and nondeterministic. But no worries, we’re on it and we’ll hopefully fix it before someone will actually notice it :slight_smile: Also, one of the goodies is that Crust won’t try to do hole punching anymore if one of the peers has direct access, hence more reliable connections. Until now, all messages in Crust’s development branch would be transferred unencrypted. Fortunately, the work has been started and some of the messages are already encrypted. Hopefully, by the end of next week Crust will have completely secure communications. Another exciting thing worth mentioning is that we also started implementing some of our 2018 goals which were a lot about testing. Andrew started working on a network simulation Rust library. It will allow us to create Linux virtual network devices from Rust. Since Linux gives us the ability to programmatically control those interfaces, we’re able to manipulate raw Ethernet or IP packets, hence we can introduce latencies, packet drops, etc. into our tests. This is all super exciting so stay tuned!

97 Likes

Tjohoo! Nice update.

11 Likes

I’m getting this strong feeling 2018 is gonna be great! big start into the new year, keep it coming! :wink:

18 Likes

You guys genuinely spoil us with these updates, no other Team comes close :+1:

Welcome Nadia :slight_smile:

Ignore all the noise and keep grinding away, @maidsafe we’re all proud of what you do :wink:

28 Likes

Are the folks in the India office photos from a modeling agency? they look very chic to me :sunglasses:!
Congratulations @Krishna_Kumar, @Shankar and @srini, I’m sure you’ll enjoy it!

34 Likes

Streaming and uTP! This project is above all the rest!!! Encrypted crust comms! This is such big news I can hardly contain myself, sorry my fan boy is showing :blush: huge thanks to the @maidsafe team you guys don’t even know how important to us you are. :heart_eyes:

Ps the Indian office looks sweet!

31 Likes

Wonderful update. :lollipop::lollipop::lollipop:

11 Likes

Huge thanks to @polpolrene and @JPL for the Primer, this doc is going to inform and please so many people of potential interest! Great work.

22 Likes

Sharing that primer… such great work.

13 Likes

great detailed update, sounds like so much is happening!

10 Likes

Thanks! Quite some weeks ago I poked some folks here on the forum and I would like to thank them all for feedback/text/help etc. Huge thanks to John (@JPL) who is a real professional when it comes to this stuff. I wrote quite some blobs of text :grin: and he added way more and made it very nice and structured. To finish it off he also knew the right person for design and graphs and stuff like that. So great team/community effort here :+1:.

Can’t wait for the dev updates the moment these places are filled. It’s gonna be so much text.

Welcome Nadia!

Overall great update, so much stuff going on. Keep up the good work.

39 Likes

Awesome momentum here. Well done @maidsafe

15 Likes

Moving right along, this is awsome! Great work!

10 Likes

So basically… the many months of design is over and work to implement the long awaited datachains is about to begin.

I think thats what it says in a few more words :wink:

18 Likes

The SAFE Network Primer is a great resource and should be made available in a wide range of file formats (and deserves it’s own page on maidsafes site). PDF is great for reading, but to go viral… formats that encourage re-use/ mashups could be made available.

Kudos to @jpl and @polpolrene for bringing together this unifying resource. It should serve as ‘the script’ for any CEP based storytelling and ‘the brochure’ to be presented at meetups.

I did find the colored text sections a little hard to read on-screen…due to the color contrast being quite light.

16 Likes

Thanks Maidsafe devs, this update is really a trip

That office looks like StarTrek enterprise some how

It’s like we’ve reached the event horizon :crazy_face:

Welcome Nadia

12 Likes

Wow, what a update! Love it all, very exciting. Big thanks to @jpl and @polpolrene for the contribution to the Network Primer, for me, I will use it together with new SAFE videos to spread the word. Encrypted messages, uTP, launching of Indian office, peruse streaming, routing and crust progress, wow wow, need to read this update many times and reply again later. So much good things, Wow! Thanks David and the Maidsafe team for the exciting progress in this early beginning of 2018.

11 Likes

Well done everyone. So good to see the milestones for Datachains. Those asking for improvements to the Roadmap should heed notice. It feels like the accelerator is just starting to be pushed down, even though the devs have been flat chat for quite some time. BTW if I was in Chennai, one of those 2 seats at the bar would be my workstation

10 Likes

@polpolrene is being far too modest here:

He has a much better understanding of a lot of the technical stuff and the background to the project than I do and actually wrote the lion’s share, even more impressive given that English is not his first language. Thanks to @bochaco and @fergish for sense-checking and to @frabrunelle for eagle-eyed typo spotting. Other folks helped out too including from the MaidSafe core team, but this is turning into an Oscar’s speech now, suffice to say it was a team effort and you all know who you are :grin:

Great update by the way. Have only skimmed it so far but the Chennai office looks amazing.

29 Likes

I agree. The contrast is too light to easily read on my monitor. The original authors may have really great monitors, but more contrast would make it much easier for me to read.

Mutable Data and Immutable Data size is shown as 1 Mb. I take a small b to mean bits. One (1) megabit = 1/8 megabyte, so 1Mb is about 125 KBytes. I think that’s not correct, and so 1 Mb should become 1 MB.

4 Likes