MaidSafe Dev Update :safe: 15th December 2015

Thanks to the entire team (again) for their contributions to the update this week. Pssst! the Routing example is functioning as expected… that statement seems a little underwhelming in itself, but trust me: serious time and effort has gone into getting to the stage of being able to say that again, I shall let the Devs in Routing bring you up to speed with the good news.

##Routing

As mentioned last week, after implementing the basic functionality documented in the new flowcharts, we took a further pass through the flow of messages with an eye on security and further reducing vagueness / complexity. Thanks to daily hangout sessions running from 06:30 this has now been completed with additional security validations included per message hop.

Running multiple instances of the key_value_store example confirms the expected behaviour with each node’s routing table getting populated and updated as they should. Running a further instance as a client shows that putting and getting is working as expected, albeit we’ve only run the small testnet on a single machine so far.

Having node discovery working i.e., routing table updates, removes the most problematic bug that was present from before the latest refactor. Full churn handling while not switched on in the implementation at the moment is not predicted to be a major cause for concern as churn and refresh have already been tested and proven prior to the work we just carried out. We anticipate that reinstating the logic will be relatively pain free. Brian is working on this now as the rest of the team have moved to confirm vaults are OK with the revised API. .

We’ve also slightly refactored the API (as mentioned earlier) to make life easier for Vaults and Core which interface with Routing. The API functions used to return local errors and network errors as asynchronous events, whereas only network errors really suited this paradigm. Take a Put request, for example. The client would call Routing::put and this function would return nothing. If a local error happened before the actual request could be sent to the network (e.g. the node isn’t connected to the network) this would be reported to the client in a separate error event. This is fine for a network error (e.g. once sending the data, the client’s managers reject the request because the account doesn’t have enough safecoin) since there can be a significant delay between making the request and finding out that it failed. But for local errors, we find out more or less immediately. So the API has been changed to have the Routing::put function (and all other functions) return any local errors right there - which saves the client from figuring out which particular request a subsequent error event related to.

Current work is now focused on documenting the updated source code, re-enabling churn and refresh and updating/adding tests. uTP integration remains to be done, tying in the recent changes to Crust. Also Core and Vault updates are necessary due to the changes introduced in the Routing API mentioned above. Overall the codebase is smaller and cleaner now and should simplify the process of feature inclusion. We hope to move to using the utp / NAT traversal API from Crust very soon all going well in Vaults (if we can confirm Vaults tomorrow). These are ferociously tight timescales we have internally. We hope this level of commitment will satisfy those who wish timescales to be better put on the forum etc. This current work is not project work in terms of the sprints, which are pretty much specced and allocated time as we move on. This is a rapid attack on some code that went sideways and therefore cannot be succinctly evaluated in terms of completion. None of us like this speed and pressure so the sooner it’s complete and we are back on track the better for us all. We percieve this will be very soon now and we will know even more in next update, if we are not finished by then (we can always hope, but it’s very doubtful for next week).

##Crust

Last week @qi_ma was diagnosing the service::test::network_test hanging in Windows. After some debugging into the cause for this issue, we realised this was coming because of issues in the rust-utp multiplexer thread from UtpWrapper. With the rust-utp module from upstream not supporting read and write concurrently, we’ve had to introduce this wrapper object to expose the same functionality to Crust. A quick fix to resolve the hanging issue was submitted via Crust PR 442, which breaks the infinite loop (causing the hanging) after a mandatory time out period has passed. However, reviewers spotted that the PR modifies expected behavior (you can catch the full discussion in the linked PR). To maintain the expected design, a suitable approach needs to get implemented here with proper notifier / correspondent messages. Also required is to keep sending a keep-alive or ping-pong style msg to detect connection status. With @vinipsmaker looking into the rust-utp issues, He’s going to be taking this PR and updating it to the approach best suited for Crust.

@vinipsmaker investigated the issues around uTP in Crust and rust-utp. we think we may have found the issue in rust-utp and it’s related to timeouts that are not reached (explained portions of the issue we’ve had with windows hanging tests too) and we are not resending packets because of that. Currently he’s working on implementing a suitable fix for the same which should let us know if this was the issue and hopefully have it addressed accordingly.

##Client

This week we continued with the work to standardise the way web applications interact with the SAFE Network. The initial restrictions imposed by the browsers have been identified. Out of the four browsers we are investigating (Firefox, Chrome, Safari and Internet Explorer); Safari has the most restricted addon environment, it does not provide any option to communicate with the Launcher through an add-on.

So alternate methods to overcome these restrictions have been investigated and documented and these alternative approaches and the add-on strategy taken by other similar applications are currently being analysed. One more challenge here is to try ensure that our agreed approach will work with all platforms from mobile to desktop, ensuring that the app devs don’t have to cater for each platform separately.

Launcher on mobile platforms has not been specified yet, further work is required to establish whether a similar approach would also be a good fit. All of these are ideal behaviours to have, but as we explore more we discover that limitations across varying platforms never fail to surprise at every stage of the process. It is essential now to complete the exploration, investigation and standardisation and lessen the chances of hitting surprises further down the line that then hinder the progress at that later stage.

##MaidSafe Website

After the completion of design and implementation of the end user site, it is now being reviewed internally, with the help from a couple of community members. I would anticipate the site going live toward the end of this week, at which point we will optimise the site for search engines as well as setup and utilise some of the tools that exist with Google analytics, to allow us to improve the user’s experience on the site. Once live, we look forward to making iterative improvements and if there is anything you would like to see on the site, please let us know via the forum.

##Documentation

Work has been completed documenting the SAFE Launcher, specifically using the CLI of Launcher and SAFE NFS and SAFE DNS examples and this is currently under internal review. The API documentation is also being worked on with code examples still to be completed here, this should be finished and shared with the community very soon.


Overall this past week has been challenging, but very worthwhile and we are pleased to have some very positive outcomes to share. With Routing now working against the example, we will look to enable churn before turning our attention from tomorrow to Vaults, which will need to be updated to correspond to the changes made in Routing. This week we also plan to get the droplet testing going with Client and in the Crust library we shall continue to focus on integrating uTP; with all of these in place and working as intended we will be able to present you with the Rust 5 deliverables. As David suggested on the forum, this is weeks away rather than months and as soon as we have reviewed Vaults and have a handle on the
uTP integration we will be able to tell you more. Thanks again for all the continued support; Rome wasn’t built in a day, and when you are making something that has the potential to be truly world changing, it does take a little time.


Link to the Team Update

53 Likes

Thanks so much for the update! :slight_smile:

6 Likes

It really does seem that you guys are optimistic about those “weeks” remaining till a major breakthrough from the tone of this update :smile:

Looking forward to a happy new year launch then!

4 Likes

Getting place, mitic thread! Thanks for all!

7 Likes

Keep up the great work :smile:

6 Likes

step by step
keep slayin+

4 Likes

Oh mama! Perfectionism at its finest! I love to watch the manifestation of clean high quality products. Send the bar into orbit. With the leadership of SAFE, so many other projects will scramble to achieve a similar level of refinement. These are interesting times indeed.:smile:

7 Likes

Great work! Congrats for the working routing example!

5 Likes

Thank you! I’m cheering for you all.

5 Likes

Patience is a virtue guys.

2 Likes

I’d just like to know who these “guys” are that have been questioning the team’s commitment and putting pressure on them to speed up. I think all the opposite has been repeatedly stated by those calling for rough timescales. Most of those asking have already expressed that they were “satisfied” early in the previous thread with answers received from team.
Patience may be a virtue, but constructing straw men isn’t. There was a communication issue, not an impatient, hurrying up, pressurizing issue as far as I can see…not virtuous at all :smiley:

4 Likes

You will gain patience when you see the vision.

The dev team will gain inspiration and fresh resolve when they see the yum-yums.

YOU can make this happen…

4 Likes

I want to highlight this so it is not misinterpreted (e.g as “launch”). This is an indication that the Rust-5 deliverables are currently weeks away - that’s a first public test network. I think this means something like what we played with earlier but with everyone participating in one public network rather than in isolated local networks.

@al_kafir I think you are overlooking the reasoning I gave about pressure - that calling for timescales can itself create pressure, particularly when it becomes a chorus, and in some cases more than just asking. Also, that providing timescales certainly does create pressure, especially when, as currently, it is very difficult to estimate reliably.

I hope you and everyone can see from this update that some of the team do currently feel under great pressure because of this overrun. We’ve already seen how this can be unhealthy for the project because it seems to have been a factor in what caused this delay, and obviously can be unhealthy for the team.

To All At MaidSafe

I’d like to say to the team that I know you are likely to work extra hard in this situation, but my experience that is not in your or the project’s interests if it goes on too long. So be willing to take a pause when needed, and carry on doing the great work on this amazing project. And thank you all for all your work and good humor.

6 Likes

No, I saw it… :smiley: The thing is, lots of things can create pressure for devs …every step of the way has been pretty pressured. Your posts on the other thread were also well after things had been resolved to mine and others satisfaction I think.
I’d say your “reasoning” seems not so much reasoning, as opinion really…Do you mean the devs have conveyed to you that they feel this is an extra needless pressure some members of the community are unreasonably putting on them - or is it just opinion, I mean?
If just opinion, then I could just as easily “reason” that directly requesting that the lead dev comment on various other projects etc is a distraction from the project and pressures devs to answer.(I don’t think that) - I don’t see any difference…it is just opinion. I’d choose not to make an issue of my opinion though…especially as I have no clue in all reality how pressured this makes devs feel. I would trust that the devs would PM or make their discomfort known themselves if this were the case.[quote=“happybeing, post:14, topic:6360”]
Also, that providing timescales certainly does create pressure
[/quote]
It would appear the implication/inference/basic premise is that this pressure is either created for no reason or for specious reasons. Some aspects of creating a new internet thingy, create pressure, some un-avoidable and necessary - timescales is one of them I think.
Your reasoning appears flawed to me in any case:… :smiley:
To continue not addressing the issue would have caused the issue to fester and for FUD to proliferate - I believe we were close to that point - trolling had started. The people who raised the issue, brought it into the light to be thrashed out and the issue was pretty well resolved quite early on in the thread. Where would we be now, by not asking questions?..I shudder to think…lol… :smiley:
Edit:

The fundamental issue to me mainly boils down to the Natives getting restless due to feeling uninformed. As I mentioned previously, the communication issue appears to be much less of a problem for “coder” Natives. I don’t think this is very surprising as coders have a good grasp af what is involved and understand progress being made.
I say I don’t find it surprising, because if I were to invent a similar “Decorating tech thingy”, I would be surrounded by decorators all talking about and understanding decorating; the tendency would be to inadvertently neglect to clearly explain things to the other non-decorating users
It may be all about devs presently, but if Safe Network becomes widely used, then the proportion of devs to non-techie users will be miniscule. It’s going to be all about the non-techie users eventually. At this point, the non-techie investors need to feel informed too – by simpler means that following Github etc. Hopefully, the new roadmap will be clear and include layman’s terms and rough timescales. :smile:

1 Like

@al_kafir I’m going to respond to you by PM because I don’t think we need to continue this debate here and doubt anyone else is interested in continuing it. I’d rather leave it too, but I want to explain to you why I disagree with what you’ve posted here. If anyone is interested, we can always invite them to the PM.

EDIT: In response to @iJRN’s request below I’ve resonded in public here: Developer Update Timescales Discussion

3 Likes

I am interested, please keep the conversation public. @happybeing @Al_Kafir

3 Likes

I don’t debate by PM anyway. :smiley:

4 Likes

4 posts were merged into an existing topic: Developer Update Timescales Discussion

chorus

Agreed. I’ll also add, it seems to me that @Al_Kafir is talking about those (new?) users whom I suspect have freshly invested in some Maidsafecoin. Then this hypothetical speculator creates an account here to ask “When is this going live?”, mostly from a speculation/forecast launch perspective for profit, less so from a buy-in to the whole project/vision perspective, that is until one of us inevitably replies with a patient even toned comment, letting them know that revolutionary software takes time to build, test and launch successfully.

3 Likes