SAFE Network Dev Update - February 14, 2019

#1

Summary

Here are some of the main things to highlight this week:

  • We published an interview with Devolution creator @Glen_Simister.
  • As we mentioned last week, we now have a good idea of how to replace our concrete coin with a common coin to make PARSEC fully asynchronous. However, we realised that the impact on the Routing crate would add another substantial amount of work. So while we have worked out what should be done and have a clear picture of how to get there, we’ve decided to maintain our strict focus on delivering SAFE Fleming as soon as possible and will therefore document our learnings about the common coin but keep the implementation as a lower priority item for now.

Marketing

Following recent additions to the Marketing Team, we’ve made the decision that we need to dive much deeper into the tech behind Fleming and, ultimately, Maxwell. This is a vital stage as we build up our knowledge, particularly of areas that have evolved in recent months, and refine our messaging in order to ensure that both our content and research is on point. Translated, this means that we’re spending a great deal of our time focused on capturing and recording that knowledge, with the intention of repurposing that information in many ways that will be of benefit to different groups as we move forwards. We’ll have more to say on this over the next few weeks but as it’s time-consuming but vitally important work, we wanted to let you all know. One other thing to point out in the meantime: yesterday we published an interview with Devolution creator @Glen_Simister this week that’s essential reading for anyone who’s looking into building SAFE DApps.

SAFE API & Apps

The planning and scoping for the next safe_app_java milestone has been completed. The code sprint for the same has begun, which includes improving the test suite, generating the coverage report, integrating with coveralls.io and fixing some existing issues. The progress of this milestone can be tracked on the GitHub project board.

The SafeAuthenticator mobile application is undergoing its second development sprint for the UI/UX enhancement milestone. Issues reported by QA in the first sprint are also being addressed. The progress of the milestone can be tracked on the project board.

Regarding the safe_browser, we are continuing to work on some internal improvements in the codebase structure, mainly towards our goal of being able to upgrade to Electron 4, but also as a way to reduce the CI times. Late last week we merged several of these improvements, e.g. we now run all end-to-end tests against the packaged browser app only, avoiding having many CI instances running and install steps in them, both dev and prod packages are built on the same CI instance. We also upgraded several of the dependencies, among other enhancements which reduce the CI times for the safe_browser, making our dev life a bit easier. In addition, we continue tackling down high priority issues to be able to release a new version of the browser soon, based on the original plan we will be creating a first release candidate in the next few days so we can take it to a first internal testing phase.

On some other threads, we are working on documentation for some of our projects, like the browser, safe_app_nodejs and some of the example applications. We understand that current documentation is not good enough for new developers who would like to contribute, therefore we are producing better documentation showing how the codebase is structured, making it easier for devs to locate answers when attempting to implement a new feature or even fix a bug.

SAFE Client Libs

This week we continued working on the documentation internally, creating Confluence pages with Rust development info, contribution instructions, useful info about RDF, Solid, and the Redland libraries, and technical details about the Client trait.

We’re starting to consider how we want to go about making the docs we have public. We’re currently leaning towards migrating the docs to HackMD which has a number of advantages, including a Book mode to structure the pages, the ability to keep pages private but set them to public-readable once they’re ready, and a smoother editing experience than Confluence. HackMD also uses Markdown, which makes it easier to edit than Confluence and also means that we can copy/paste the docs into a Readme file, or even into documentation comments in the Rust code itself. There are other options of course, like hosting the docs on GitHub – it has some of the same advantages, except that there is no way to keep some documents in a repository private while others are public.

Our priority is still the RDF storage proof of concept, which we are working hard on. @yogesh spent some time this week getting up to speed on the Redland libraries while @marcin helped him resolve some issues with building them.

Routing

This week most of the team came back refreshed from their holiday.

On the Fleming front, we continued to refine our Sybil attacks models. We ran more comparisons between large sections composed mostly of elders and large sections with a smaller number of elders. This confirmed what we observed last week: it appears that large sections with a smaller number of elders can be quite resilient to Sybil attacks. We also started modeling network growth and the split of sections that occur as they become too big. The preliminary results suggest only a moderate impact compared to our previous models. This is positive as it means that we can easily extrapolate from the results we already obtained in constant size networks.

As we mentioned last week, we now have a good idea of how to replace our concrete coin with a common coin to make PARSEC fully asynchronous. We started implementing the changes and made good progress. This allowed us to get a good feel for the amount of implementation effort required. The changes to PARSEC are quite manageable: we could mostly implement all that’s needed in a couple of weeks (note that this is before making all the tests pass so another week or two would be expected); however, we realised that the impact on the Routing crate would add another substantial amount of work.

So while we have worked out what should be done and have a clear picture of how to get there, we’ve decided to maintain our strict focus on delivering SAFE Fleming as soon as possible and will therefore document our learnings about the common coin but keep the implementation as a lower priority item for now. This will enable us to come back to the implementation at a time when it is our highest priority item and start with a very good idea of what’s involved. You should expect to see an RFC detailing our lessons learnt and an update to the PARSEC whitepaper in the near future.

We also continued working on performance. We created more relevant benchmarks to support further improvements. In particular, we enabled replaying PARSEC scenarios showcasing exactly how the Routing layer uses PARSEC. This will be crucial to confirm we optimize for real use cases.

A different kind of performance improvement happened this week: the QA team generously helped us improve the CI speed from ~40 mins to ~10 mins. This makes the development workflow that little bit more streamlined, which is much appreciated :smile:

We also finally merged a long-standing performance improvement pull request that reduced the amount of re-processing for a 10-40% speed improvement depending on the section size in our benchmarks.

Crust

We finally finished our work on the bootstrap cache. Externally reachable nodes (ones that have a public IP address) are now stored in an LRU cache. We reused our existing lru_time_cache library. Although, it required a few changes to make it work for Crust and fortunately it will be more flexible as well:

  1. LruCache::iter() order was changed to produce the most recent items first.
  2. New methods prefixed with notify_ were introduced that return expired entries that were evicted from the cache.

Now we want to answer the questions of how Crust can help network upgrades, restarting nodes rejoin the network, maximize bootstrap success rate, scale up to thousands of connections, etc. All of that will require Crust API changes which hopefully will make the library simpler too.

91 Likes
Sections replacing Groups
#3

beautiful!!! gogo maidsafe!

13 Likes
#4

thanks for the update, no idea when we can expect any testnets?

9 Likes
#5

Does this mean documentation that is only to be seen by MaidSafe? GitHub is great to allow contributions and with branches one can work on documentation without it being on the website.

6 Likes
#6

yes, lets have Fleming first please :slight_smile:

17 Likes
#7

I’m looking forward to the extra SAFE Client Lib documentation.

Indeed.Wallooning isn’t that important :wink:

10 Likes
#8

It would be good to get the async changes cemented into the white paper, so good call. Also won’t complain about getting SAFE-Fleming sooner :slightly_smiling_face: I’m assuming the change to the common coin, performance wise would be similar but if for some reason it was not it would be a pain to deal with any unknown unknowns or extensive testing in the face of a public release that is highly anticipated.

Come on Fleming!!!

14 Likes
#9

Is there still alot to be done in the performance area, or are we nearly there?

1 Like
#10

I wait the safe network as a child waiting for the Christmas presents, what you guys and gals are doing is awesome.
I will try to visit you this year in Ayr (if it is allowed) I might be able to put my grain of sand into this lovely project.

Go Ants! Go!

16 Likes
#11

This all makes sense to me. Thanks for the transparency and reasoning behind the decision. All in good time … :slight_smile:

10 Likes
#12

This update feels different. The focus on marketing, and getting documentation ready, and delaying a 2 week PARSEC implementation, in favour of working for the Fleming release, makes it feel like we are getting close! Super job team!

16 Likes
#13

Excited to see the focus on improving documentation for building apps, and the marketing focus on tech, which seem sort of related to me.

It’s always seemed to me that this project has the potential to attract people such as myself who wouldn’t usually be drawn towards tech, and I feel like this could be one of its ‘killer apps’ - getting people involved on a level beyond being just consumers.

In using SAFE to try to get to grips with some basic coding myself I’ve found documentation for a lot of other technologies very fragmented and badly presented as well, so don’t beat yourselves up too much! If I were to be cynical it almost seems like some people may have profited over the years from keeping things esoteric!

I’m lucky to have friends and family who have been able to help me at points, but I’ve realised from them that even with a lot of experience in the field both the high level concepts and little details of SAFE can be a bit baffling (in some ways more so than for someone totally new to it all,) as well as the problems of just trying to move into a field of computing that one is not used to.

I’m still not quite at a point where I feel I’ve got anything to contribute, but along the way I’ve been forming ideas about how best to present help for developing on SAFE at a basic level, so I’ll certainly try and chip in at some point.

Apart from sound technology, a wide knowledge base among a diverse range of people seems in the long term fairly essential for building something sustainable.

15 Likes
#14

Every time i read something like this it gets me all excited then nothing happens…stay calm, stay calm, stay calm, stay calm…

2 Likes
#15

I would hope anything not necessary for Fleming would go on a to do list for the future

2 Likes
#16

This is a hugely important viewpoint I’ve held for a long time and am glad to hear others having it as well, like in this interview.

Especially since having SAFE accessible to as many users and builders as possible is the overall goal, for SAFE to achieve it’s fullest potential.

#WebDesignersLivesMatter

9 Likes
#17

100% agree… (20chars)

2 Likes
#18

A bit off topic, but did MaidSafe develop a scanner?

“MaidSafe provides a Panic Button capability for those situations where a housekeeper may encounter an unsafe situation“ :joy:

4 Likes
#19

I knew @maidsafe would be disruptive across many industries but I didn’t think they’d go this far! :joy: only jk

12 Likes
#20

Great update Maidsafe devs & marketing team.

This reads like an opportunity to someone if they can code to implement this :wink:

:gem: found

another :gem: found

:bug:

Hang in there super :ant:s

:stuck_out_tongue:

2 Likes
#21

Is Fleming the testnet? First Alpha?