MaidSafe Dev Update :safe: 1st December 2015

This week’s Dev Update has very much been a collaborative affair, with contributions from the entire team; I can take very little credit, especially the Routing section with @Fraser leading the update charge here. So without further ado let’s get into it…

##Routing

As @dirvine stated yesterday in a Dev Update leak :wink: Routing: the patient is now stable and off the critical list, this is now very much the case with brilliant progress made this week, which has been borne out in real testing. We have run 65 (one more than group size) nodes with a close group size of 8 on droplets, all nodes could populate close group number of nodes in their routing tables. :slight_smile: This is a significant breakthrough and a testament to the hard work of the team. Does it mean we are fully out of the bug infested woods? Nearly, but not quite. Over to @Fraser for the detail.

This week has again seen significant effort put into cleaning up and fixing Routing with most of the normal working hours consumed in a multi-dev hangout with one dev (normally Spandan) coding and drafting documentation in a screen-share, so that all the others can view and discuss the progress in real-time. There have been several night-time code pushes too where simpler tasks have been identified late afternoon, implemented overnight, and reviewed by all five devs the following morning. Despite the heavy focus on discussion and documentation, there have been around 45 non-merge commits made to the ‘cleanup’ branch in Routing and we’re starting to see positive results.

From my own perspective, a significant step forward is that we now have five core devs with fairly intimate knowledge of the codebase in Routing rather than one, along with the knowledge and (draft) documentation of the process flows and design rationale which has gone into the library. This puts us in a very strong technical position moving forward.

On a more tangible basis, we now have much less code in Routing, we’ve significantly reduced the complexity, and we’ve ironed out quite a few issues and to-dos, some of which have been fairly long-standing. We’re not finished yet, although I hope we’ll soon be at a stage where the remaining tasks can be documented and implemented by just one or two of us rather than consuming the time of five developers.

The main features not covered at the moment are churn (handling nodes connecting and disconnecting from the network), swarm (handling message distribution amongst a close group of nodes) and caching (handling data being cached and retrieved on an opportunistic basis). When I say “not covered”, I mean that these haven’t been reviewed and documented in the same way as most of the other Routing processes during this current effort. Much, if not all of this code is implemented already - we just need to check its validity, make fixes where required and document the results.

I sincerely expect that next week’s update will contain news that we have implemented a working test network of Routing nodes!

For the more technical-minded, a brief summary of the week’s actual code changes are:

  • reduced complexity of process for joining the network (including process for relocating nodes)
  • reduced complexity of connection establishment
  • incorporated helpers provided by the new maidsafe_utilities crate, the main one being using its event sender to avoid a suboptimal event loop containing a micro-sleep
  • incorporated changes to Crust (we now explicitly tell Crust when to start listening and we handle failed connect attempts)
  • fixed bug in RoutingTable
  • reduced number of memory allocations
  • removed several unused functions and unwanted lint exceptions
  • removed all unwrap() calls from the non-test code and replaced those in the test code with more useful macros
  • refactored some excessively large functions into smaller, more legible ones
  • refactored or removed existing tests
  • improved debug output

##Crust

From a Crust perspective work continues on the Crust speedup RFC, with the guys hoping to implement enough of it so that we can run benchmarks and compare results against the updated Crust this week. Crust has also seen a few updates to better suit Routing and its requirements (e.g. removing default acceptors). As mentioned in last week’s update the team continued to investigate the RUST-UTP issues and work to ensure that we have a rock solid UTP connect function and are making good headway.

##Client

The RFC for exposing SAFE Network Data types was raised last week, which details the IPC APIs which would be exposed from the Launcher. The APIs would allow developers to create and manage custom structured data types and also store raw data / immutable data to the network. Another RFC to expand the DNS APIs has been raised too.

This week we are aiming for a couple more RFCs: one to provide authenticated access from Launcher and another to make the safe_nfs API non-blocking. The aim of the authenticated access RFC is to present developers with a simple and common approach to authorise applications (e.g. standalone apps, add-on, etc).

A comparative analysis on developing browser add-ons / extensions for browsers (Chrome, Safari) is being explored, with the aim being to increase understanding of the restrictions which other platforms impose, for example, Chrome does not support custom scheme (safe:) handling. So common standards have to be defined for SAFE Network add-ons on how and what would be exposed from the add-on.

##MaidSafe Website

With the completion of all the implementation tasks, we now have an initial draft of the end user site. This will allow us to more easily review the site and make changes to the design and implementation where appropriate. A considerable amount of effort has been put in by Scott overhauling all the visuals and ensuring that the new site works well across all platforms.

While we very much hope that you like what has been produced, the new site should be viewed as a starting point, rather than a finished project. With the roll out of the network commencing soon, information regarding getting started, installer downloads and refinement of the FAQs are likely to be among the first revisions. We look forward to your comments once the site goes live.

##Documentation

As mentioned last week we have significantly increased our focus on Documentation. So this week we have had a couple of planning meetings and been evaluating readme.io to see if it fits the bill as a good tool to create and importantly maintain good SAFE Network documentation. As a proof of concept we have created a custom theme and are working on some initial “Getting started” guides for SAFE Launcher CLI and the SAFE_DNS and SAFE_NFS examples. So far it looks like this could be a good platform to use to quickly and create easy to follow documents, which also allows for community contributions; a good start.

All in all things feel like they are coming together after a couple of weeks of focus in specific areas. The feeling of confidence that the team now have about Routing is palpable and with Crust also getting a lot of focus we hope to have solid, working and efficient transport layers which we can build upon. It also feels good to start producing documentation that will enable developers to easily integrate with the network and our code. While what we are producing is very much a starting point, the importance of engaging with the community and third party developers cannot be over-emphasised and we will be looking to increase momentum as the rollout of the network develops.

48 Likes

Thanks team. I love the diligence of these updates, and the tone of this one seems to me to have a little swagger in it too :thumbsup:

Hey @Ross, nice headings man :stuck_out_tongue:

I don’t know if many people can appreciate this without having worked in development, but it takes real guts (and experience) to work this diligently, to ignore the pressure, and not cut quickly to delivery. The results will be directly related to this though, I’m convinced of that. Thanks again, everyone at MaidSafe.

PS can we have the transcript too please!

12 Likes

Did I miss the installer release(s) or did those get moved to a back burner due to the bug hunting?

1 Like

I am really looking forward to a public test network. I think many are waiting to see if it can actually work as designed.

No doubts here, ofc - just excitement! :slight_smile:

9 Likes

Ah, you won’t miss it when it’s there. We’ll have a party here on the forum ;-). Some more bugs and than we could see the network go live. That means installers for all to test and run the network.

9 Likes

When the installers and some API examples hit, I will be one happy SAFE-er for sure!!
Lol!

6 Likes

@happybeing we changed our meetings up a little a few weeks ago and the transcript was the output of that meeting that is why I have left it out of the last few updates, I should have explained that :slight_smile: 10 points for spotting though ;-). @justine_mclevy still maintains it, so I am sure we can post a link tomorrow.

5 Likes

Nope you didn’t miss that - once we have a public network to connect to the installers will follow - we will shout it from the rooftops :smiley:

8 Likes

For the sake of clarity, this is not the public network right?

3 Likes

Here’s the transcript @happybeing!

9 Likes

Great news, and full marks for persistence

I have one question, I thought Group Size was 32, but the update said

So was group_size set to 64 for this test or is that what the network will use now?

Yes 64 is optimal routing table size. so close group (32) is first half after that we fill the routing table using kademlia rules. So we need tests of over 64 nodes to ensure we are actually routing and not actually connected. Nice catch

8 Likes

Thank you for the update.

I want to share thoughts from an end user point of view with regard to readme.io consideration. I have experimented (purely for fun, I have never made money from the 20+ websites I have built) with various web development tech’s since early 2000’s. I can tell you the one I still use the most is wordpress. The reason why I use it the most today and why it is so successful is because I have always been able to find a tutorial or a simple step-by-step instruction to manipulate it the way I want it. The few times I have hit a dead end (Or realized I don’t want to spend 10 hours figuring it out) I have been able to quickly find several people (most recently on freelancer.com) to do that very specific task (which wordpress made easy to know what I needed) for a reasonable price I have been happy to pay (even though I do not make money).

To have rock solid documentation would secure the SAFE network in a place where few open source services go, like wordpress. Definitely, light years beyond what all the current alternatives to SAFE that are out there…I have experimented with all of them. There is something to say about little old me setting up the first working cross continental crust node back in June or when ever that was, and that was me just following the tutorial steps I found on the forum. Imagine what would happen if there were paid “INDIVIDUALS” who built the readme docs and then were available to be hired by someone like me to help me solve my SAFE development?

What I’m saying is documentation is almost as important, if not more important, than the code (There have always been better/cheaper alternatives to wordpress available to me…used dream weaver before that). My hesitancy with readme.io or any online company that has a terms of service agreement and privacy policy as the largest section of information on their website, is that the way I want to see the most valuable resource the SAFE network hosted?..NO. I understand time is something that is very precious here, and I can’t imagine what is at stake, but why can this not be done with the same ideals and philosophy that keeps me so passionately obsessed with all things SAFE? Why can I not know the names/individual-effort (to give credit where credit is due) of the designers, maintainers, curators of arguably the most valuable jewel that SAFE has…the written language I understand (documentation)? How is it not possible for MAIDSAFE to hire a group of individuals to build the same exact thing that it needs; without the risk of tripping at the finish line when the ABC agency sees this as a very low hanging fruit? Is it because of …well, x company will give us a good deal or everyone uses them… Hell, why don’t I just use gmail or MSN browser then, that is a good deal?

I am looking at this from my given bias (There are a lot of graphic designers/artists in my life), and admittedly I am from the 10,000 miles up vantage point. This is just my first reaction after reading through readme.io’s website and spending time pondering this topic. This question of “content” and how important is artists (all forms of artistry) keeps coming up a lot. I know MAIDSAFE/SAFE can not be all things to all people, but I think there are specific moments in time that can have decades of ramifications (Look at what has happened to the control/benefit of the artists of the world with the advent of the internet?). Is there not a way to avoid throwing support towards an opposing force like readme.io or any like service.

I am already 100% committed to this project, wherever it goes and I will support whatever direction it goes as long as I can keep understanding the why in my head. Thank you again for the world changing efforts you all have already provided.

15 Likes

Love these updates guys. Solid work and thanks for the dedication.

4 Likes

@MrAnderson firstly thank you for your input, much appreciated and you make some very good points. Certainly given me / us plenty to think about - I will look through their TOS and Privacy Policy and be sure to discuss it at our next meeting. You are correct often the easiest route is not the best route.

4 Likes

When the test network goes live will safecoin be going into implimentation or will we be getting free puts or what? What exactly does a live test network mean from a practical standpoint? What will we actually be able to do and have at our disposal? I’m as excited as the next guy but then again the last time I downloaded installers not much was functional so I’m a big pragmatic about the whole thing? Can we upload files? If so how? Generate safecoin? Again if so how? Will there be apps? If so which ones? Etc etc.

No Safecoin yet.

This is a public network with storage and browsing only. So we should be able to upload files (including html = websites), create shares, register a public domain, and download anything we have permission to access - including browsing any public websites.

It should just work though, so I’m hopeful - bugs excepted - that everyone will be able to participate in this, albeit with help from the community.

I don’t think it’s any more than that, but that’s a giant leap for Troon kind… Hey, I may be a bit slow, but did anyone else notice that Troon is Tron with an extra ‘o’? Hmm suspicious!

7 Likes

Okay, awesome! We can upload and view files. Can we KEEP those files or is this a temporary thing? As in is the network going to be wiped by a giant reset button of doom at some point?

Oh and HOW do we upload all these files? Do we have a lifestuff app yet or something equivalant to it yet?

Also while I think of it: After the test network is implimented how long do you think it’ll take for the full network with safecoin and everything to be launched?

3 Likes

It’s temporary at this point, and will be until there’s some kind of community / MaidSafe agreement that it’s stable.

We may well not know in advance when we start using the “final” network, so don’t expect a “OK this one will be it” warning in advance - it may well be more like “OK, well we haven’t needed to reset it for a while, so let’s call this it”. I don’t know.

I don’t know any more details, and I’m not going to speculate because predictions of what and when are not worthwhile.

1 Like

So…, is there any update on how the documentation will be done?