MaidSafe Dev Update :safe: 10th November 2015

Hiya,

Last week was another busy week, after the sprint we have been working on different action points. Starting from the handover of @benjaminbollen, fixing issues in crust and routing, new RFCs etc.

As mentioned in the last week’s update from @Fraser, the handover process from @benjaminbollen went as planned. Andrew has quickly got his head around the Routing implementation and with Andrew’s expertise, we can only expect the Routing library to get even better in the near future.

Andrew has also raised a WIP RFC to detail an approach to refactor the Crust code base, so that we can use stabler version of Rust rather than depending on the nightly version of Rust and also make the library efficient and easy to use as a generic transport lib. While Andrew is busy with the RFC and the handover process, @Peter_Jankuliak and @vinipsmaker have been resolving the bugs in the Crust library.

Unexpectedly @qi_ma was unwell and we were short of him for couple of days last week. Also, @brian_s would be away for the next two weeks on annual leave. Since both the maintainers of routing are not available this week (Andrew and Brian), the Client and Vault guys (@ustulation and @fraser) are stepping in to Routing to review the code and make improvements, assisted by David. When @qi_ma resumes work, he would also be working with Fraser and Spandan in the routing library.

While others are busy working on the core libraries, I get some space to do feasibility analysis for getting the add-on compatible with the launcher workflow. I completed a proof of concept to expose the network API from the add-on, so that the application developers would be able to interact with the network using the API. Once the analysis is complete I will detail more on how the API would be exposed for developers.

We would be continuing improving the libraries until the issues in Routing and Crust are addressed. This interim period will help us improve the code base without adding any feature as such.

This week we have the updates from the maintainers for certain modules that are being actively worked on:

Launcher (@ustulation and @krishna_kumar)

@ustulation had raised an RFC for unregistered client access. The primary objective of this is to allow the applications to access the network to fetch public (unencrypted) data. For example, in the proposed RFC, the Firefox addon will be able to read public data, such as websites, from the network through the launcher without authentication. It would only need an unregistered client to fetch the data from the network.

In addition, the RFC also details about the discovery mechanism, where the application would ping launcher for access rather than the launcher starting the application. At present the application is to be added/configured in the launcher and then the application should be started from the launcher. So an alternative approach is detailed in the RFC, where a discovery mechanism would be in place, through which the application will connect to the launcher. Once it connects the launcher will ask the user for his consent to allow the application to connect with the SAFE Network. We would definitely like to get input from the community on this proposal, so please feel free to comment on this RFC within GitHub.

The reason for the alternate approach of authorising the application is that there can be restricted environments in which the applications are being developed, such as with a Firefox add-on or Chrome extensions. They do not allow the extensions to intercept the command line arguments being passed and this may become an issue as we expand to other platforms of application development. Thus we are forced to search for an alternate approach for scalability.

Scott has been working on the mockups for the Launcher UI. The mockups may be updated based on the discovery mechanism once the design and approach is confirmed.

Crust (@Peter_Jankuliak and @vinipsmaker)

So from Crust’s perspective, last week was still about bug fixing. The first bug was quite an elusive one, it only showed up on Windows CI machines when running tests. On top of that, it wasn’t any single test that made this bug manifest itself, it was only when all the tests were run that we saw it. Luckily, @andrew found out that if we run the tests on a virtual machine we can reproduce it, from there, it was just a matter of hours for us to have a fix ready.

Later that day @fraser went ahead and tried running the crust_peer example on multiple real network PCs (which we call droplets). He has done this before but this time he was testing Crust over the uTP protocol. Unfortunately, the results were not positive, only a few machines connected and some of them froze. After some investigation, the issue was traced to the rust-utp library. In particular, on few occasions, the library failed to send some packets. We managed to fix this problem (in the same PR) and we can now reliably connect two peers and exchange data using uTP, but yet again, our bug squashing quest is not over as we noticed that when using many (>10) uTP connections, they start to consume CPU heavily to the point where nothing gets sent over the channel. Peter is going to look at this this week, his suspicion is that there is a problem in rust-utp’s congestion control algorithm.

Routing (@Fraser and @ustulation)

With Ben’s departure at the end of last week and Brian being on leave for the next fortnight, Spandan and Fraser have taken on the task of looking at the ongoing Routing bug. Spandan has seen Routing from Client’s perspective while Fraser has seen it from the Vault one, but neither has had to dive into the inner working of Routing yet. So this is a good opportunity for consumers of the Routing library to see what goes on in there.

They’ll be starting by doing some housekeeping work (e.g. updating API functions to return errors where appropriate, and adding further log statements to help the debugging efforts) as well as creating a new common utility library to avoid duplicating helper functions across several libraries.

The hope is that over the next couple of days of housekeeping, they’ll become more familiar with the core Routing modules and will increase the “debuggability?!” of the library which should help nail the remaining bug(s). As mentioned above, during this time, Andrew will be finalising his RFC, after which he should be well placed to join Spandan and Fraser in the Routing bughunt.

Website (@Scott)
I’d first like to say Hi! :slight_smile: This is my first update here and is probably the first time many of you have heard from me. My name’s Scott and I’m the UI Designer here at MaidSafe.

I don’t know how much you guys know about the new websites, but the plan for MaidSafe’s web presence is to have two connected websites; one for consumers and the other for developers. The consumer site is the one we’ve been working to get out first. The objectives of this site is to capture new visitors, get them up to speed on the project and allow them to easily join the network when launched. A lot of the design work for this site is almost complete and I’ve been working with Shankar to get it implemented. Over the next few weeks we’ll have the site ready and will launch it soon after!

Overall we obviously have another very busy week ahead of us. We are hopeful that we can quickly find and fix the bugs in the networking layers that will allow us to launch the network on droplets and release the deliverables of the last sprint. While we have a reduced team working this week, we do have MaidSafe’s most experienced heads giving this 100% of their time, so we are optimistic about resolving the issues as quickly as possible.

While we are very focussed on fixing the issues that are stopping us from getting the network up and running, the amount achieved by the core team in the past few weeks and months should not be overlooked. Once we have a strong and reliable network in place we will be able to move forward confident that we are building all the exciting new features on a very solid platform. The community support we continue to receive is nothing short of amazing and is a real motivation for the entire team, we are now getting very close and the future looks very bright for the SAFE Network and it’s amazing community.

Here is the weekly dev update :slight_smile:

50 Likes

Time to introduce the “dislike” button.

2 Likes

“Time to realize this is what happens with software development” - Grasshopper.

9 Likes

I’m going to like the OP. I’ll also like the first reply, because everyone needs a little love. (Imagine if there was a dislike button; I might be persuaded to not give any love. Hmm.)

1 Like

Short sighted much? (Extra characters as required to post)

1 Like

As a software developer myself I know how unexpected bugs are and the happiness of finding clues that lead to the solution. Keep up the good work, team!

17 Likes

no, i see the price is climbing now, cheers

2 Likes

hyena doesn’t need any love though.

I don’t care if it takes til December to fix one bug. Great job everyone; these sprints are fantastically done.

12 Likes

Ah another minor bump on the road. Good job for continue to work on amazing project!

OT: About the mpid, anybody here know what is the limit size to send a message data? I know this isn’t on the works until the connection is established and ready to be used.

4 Likes

You make plans and then life happens :smile:

It just means the builders of The First SAFE Web have a few more days to get their act together (3 of 19 are ready so far) and much less excuse for not being ready to upload as soon as the network is live.

We could make it a race. Can we make all 19 websites ready before @Fraser @ustulation and @dirvine can squash those bugs?!

If not, I don’t think we have any right to complain because our task is far far easier, and if you aren’t planning to join and help by uploading some web content to support the network, you have even less business passing judgement or making negative comments.

The team need and deserve the support of this community. They’ve been through a lot this last 12 months, and achieved amazing things including a complete rewrite into a language none of them had used before. Right now they are on the brink of delivering the first live public version of SAFEnetwork. That’s amazing :smile:

Let’s be ready to cheer that moment, and to make the most of it by having some websites up there to show to others, and to prove to ourselves, as well as to test it out and reveal any remaining kinks and bugs.

Best wishes to @Ross & @qi_ma and get well soon!

16 Likes

If they didn’t find (hard to fix) bugs then I would be out of here in a shot

If they didn’t ever tell us they had hard to fix bugs then I would not trust their code.

If you program decent sized programs then expect bugs. It is like a parent who expects their child to grow up perfect without having to correct the child. Either the child is like “Jesus” or the parents are deluded.

Congrats to the progress of the dev team & others. Many thanks for keeping us updated

7 Likes

Someone said it was a rust language library causing the routing issue possibly. Anybody post on their forum?

1 Like

reasonable, but how to solve it? change language to re-write? :joy:

3 Likes

Will the Firefox addon support Windows when we start the first global network after the crust & routing issues are resolved? Right now the readme says Windows is not supported. I think we’d miss a lot of regular joe testers if we hit the media with the first global network without Windows browsing support.

2 Likes

Last time we had Waterfox support…

So the readme is outdated?

No there was a rust lib issue with libc, but on nightly. Crust still forces us on nightly (bleeding edge) but this will be getting squashed via some changes in crust that are long overdue. The issues in routing have been that crust had some bugs (as per the update) making routing very hard to test properly. Routing also had some issues in it’s error handling (it was ignoring lots of errors) and that is being sorted right now. You have to add a lot of extra code to bypass rust safety as far as errors are concerned and unfortunately some of this happened in routing, so fixing that is essential for an error free lib.

So this one was our fault and we paid the price, which we are now putting right. Just a symptom of going too fast with not enough ability to test effectively during the sprint. Should be fine this week though I would hope.

14 Likes

Wow, nice!
That means, that the critical bugs are solved (mostly) and we can connect to the network in the next weeks?

2 Likes

Still Waterfox is supported.

No, the README isn’t outdated. Since the add-on is specific for Firefox, the current version of the addon is not supported on Firefox windows. Waterfox was just an alternative to allow the windows users to experience the same as in other platforms.

Once the add-on is made compatible with the launcher, windows should be supported.

4 Likes

The freedom of information to flow freely is such a wonderful thing!

4 Likes