MaidSafe Dev Update :safe: 21st September 2015

Hey guys,

We have finished the stabiliser sprint. This essentially was a small and feature light sprint which basically means that no additional features in any crate would be added. The main purpose of this sprint was to address all the TODO points regarding workarounds and designs that may have been done in haste in prior sprints. As a result the crates have much better error handling logic and test cases written for them.

@BenMS and @brian_s have made sure that churn handling was properly integrated into routing. This will allow the vault network to not lose data when other vaults join or leave - the very incidents which trigger churn. We ran 40 stable nodes and one additional node which kept refreshing every 5 seconds to simulate churn, for just over 43 hours. After tens of thousands of iterations, we saw the routing table still populated correctly and the network continuing to be stable. So a great result!

@Peter_Jankuliak and @vinipsmaker have worked and stabilised one important aspect in Crust - it can now connect to peers more than once. Another important thing to note is that Crust is now no longer blocking as async workaround for the blocking Rust-networking library has been put cleverly by the duo. It involves making a false connection initiated by a local socket to a blocking TCP acceptor/listener from a different thread so that it can be terminated gracefully. Hence it is a better programming paradigm and the user of Crust can expect a deterministic and graceful exit on demand.

@Fraser and @qi_ma made sure that churn handling, enabling the network to manage users turning their computers off and on, was fixed and properly tested in safe_vault. Also completed was the propagation of PUT error on low-balance to the Clients. So if the safecoin balance is not sufficient then the vaults would not allow the user to mutate the network (i.e., Put something on it). Instead an error would be returned so that the user has a chance to purchase more safecoins and reattempt the PUT.

Finally, the Client modules. Observer pattern for error handling and other miscellaneous async events was coded and stabilised. This allows the user of safe_client to tap into any one of the various categories of events and act upon it if additional actions are wanted over and above what the client module requires. It is much like boost::signals2 where the clean up of uninterested observers is done lazily during the next capture of an event. Also, previously safe_ffi restricted the retrieval of files to just the home directory for the service. That restriction was lifted and the users can now retrieve files from any node in the hierarchy of folders and directories starting from the service home directory.

Please note that the name of safe_client crate is planned to be changed. The new library name will be decided shortly and we will let you guys know when that is complete. We also are planning on making a stable branch for safe_ffi while the master would be the current working branch during development cycles.

To make sure that this sprint hasn’t introduced any regression, we tested the self-authentication example and it works well across 18 nodes, so lots of reason for optimism as we move into the next stage.

With that we now embark on our journey for the next sprint, Rust - 5, which will be a major feature-sprint for a lot of crates. We begin planning this week and will see RFC’s getting raised and discussed. Some of the high level objectives are as follows:
<1> uTP Hole Punching in crust - This will be our crux concept for peer-to-peer networking. Once fully in place it will allow the nodes (clients and vaults) to talk to each other beyond NAT boundaries and thus achieve decentralisation in a true sense in the public network. Community run nodes being part of the droplet network seems right around the corner !!
<2> Connection Management Implementation in Routing
<3> Messaging Infrastructure in safe_vault - This should be quite a fun deliverable, but is also a big one in itself. The messaging APIs for inter-node communication would be put in place. We might see the use of MPID (MaidSafe Public Identity) in this iteration. It would allow chat engines and clients to work on SAFE-Network.
<4> Messaging Infrastructure and Launcher implementation in the Client Modules - Again quite a lot of work might go into this. Launcher is expected to be the only application the users gives their credentials (PIN, Keyword and Password) to. Launcher would then authorise Apps on behalf of the user giving each one of them a separate environment (a directory) to work with. This not only prevents Apps from knowing the user credentials, but also removes the ability for them to tinker with folders/data of other Apps or user’s files and folders. Sharing of data between the Apps will be done in a controlled and monitored manner if so desired.

Please note that we have not yet concluded if all of these items will make it into the sprint, so this is not a final list by any means. We should be in a position to confirm the final list during next week’s update.

Also, it is planned to open GitHub issues and solutions to the community when rust-5 is underway. So anyone in the community interested can solve GitHub issues. If the solutions are satisfactory, the successful contributor will be awarded a bounty, similar to as explained here. We are working out a few details in the planning phase about the github issue task estimation values and indicating the same in issues accordingly. More info about this will be published once we have the answers in place.

Here’s the staff meeting transcript.

36 Likes

hahaaaaa - just wanted to be first to post :smiley: :heart_eyes: I love updates!

4 Likes

hahahaha That was even faster than initial increase in views for a Stackoverflow post :smile:

4 Likes

The start of a million before next summer :wink: Great work, thanks for the update.

6 Likes

how cool would a chatroom inside the pre-alpha-safe-network be?!?!
for starters a terminal-based chat would be enough and absolutely awesome!

4 Likes

Rust-5 is a biggin!! Can’t wait to see all these get ticked :smile: way to go maidsafe

2 Likes

Now the real fun is about to start, hopefully after this sprint we have the first global and truly public version of SAFE up and running!

I also really hope the sprint can be finished in the allotted time. It’s a big one again, and those tend to require extensions so far. I still have hope for test_safecoin to be up and running at the start of the new year!

5 Likes

Could anyone give a non-tech update?

  1. What is the status of the Safe App Launcher?
  2. When do we see Apps?
  3. How many sprints are required until Safe App Launcher?
2 Likes

Sure :smile:

It is currently in it’s planning phase. Launcher RFC is currently a work in progress. so questions still exist in it’s implementation design which needs answered and reviewed before it’s accepted for implementation (we hope as part of the rust-5 sprint. not confirmed yet though)

Sorry gonna be a long answer for this one. It’s quite a generic question. “soon” is probably the best answer for this. depends on what we refer to as apps. If we are talking about apps that you can use in the desktop environment showcasing a feature, well you have dns examples and sorts already to try. If we refer to third party apps interfacing with the network with the help of launcher for data storage, that should be possible once the launcher is in place with the nfs REST API. If it’s apps related to features not exposed yet(messaging, safecoin, POSIX friendly data storage…) gonna be when these features are implemented in the network layers itself. Mobile apps - when we establish the port layers for the corresponding platforms.

This should be answered when we conclude with the Launcher RFC work in this planning phase for rust-5 Sprint ( before rust-5 starts it’s implementation work). Once the RFC work is complete, the tasks required and the effort needed will be clear for us to draw the exact estimates of when these should make their way to production code.

Thats exactly the reason we havent fixed the objectives as “all of them” for rust-5. We’re hoping to have the work required laid out with RFCs in place during planning and based on the results from it, have an accurate estimation of the work we can achieve in meaningful fragments as deliverables.

10 Likes

How long will the planning for the next sprint take? I know I read how the sprints are going to be short going forward and was wondering if the next one will be 8 days like the previous?

2 Likes

Happy Sprint everyone! Rust-5 looks to be huge.

3 Likes

I´d also be interested in how long the next Sprint ist going to take.

Another thing: I feel bit stupid but can anyone explain to me what one can extract from the transcript. i suppose it´s a list with what people are up to currently, but the bulletpoints do not seem self-explanatory to me. Any help/advice?

Very exciting. The thing I see as most important to me is the hole punching with ability to connect on a global scale with what is already available on a local network.

So I can move further on hardware and mesh.

4 Likes

Currently we are estimating this planning phase for rust-5 to conclude next week(02/10/15). Unless we hit any blockers with any of the RFCs, this should be comfortable enough to finalise by and then list the final objectives for rust-5 and it’s duration for implementation.

We can almost certainly say implementation for rust-5 is going to be longer than 8 days. rust-4 was a specialisation sprint which didn’t introduce new features into the system. Ideally we’d hope to have the implementation within 3 weeks(maybe 4 as max).

5 Likes

This project needs to push forward the code bounty program very aggressively. Don´t get me wrong, I think all of you are doing a very good job, but the breadth of this project is too much huge.
Just my opinion. :thumbsup:

3 Likes

The bounty program is growing, but it can also be counter-productive if it diverts MaidSafe staff from activity they are best placed to do, so I think it needs to be balanced growth rather than pushed aggressively. This is for the core, which is indeed a big and broad project, but bounties are not needed for people to develop apps and services on top of SAFE Network, since they can be done speculatively for rewards through use, or as part of the sharing economy of open source.

4 Likes

It has been kept up because that was what we started with, so it is not so useful now we have the more explanatory Dev Updates. I still read it though, because I find it helps me know who’s who and who’s doing what which is not usually apparent from the Dev Update.

For me it still provides helpful insight into MaidSafe as a whole, new recruits, non-technical folk and so on, all of which I find useful even if the bullet technical points aren’t always understandable.

3 Likes

Thanks for the info.

1 Like

Ok, then we arrive to the point. Too small core dev team for a too wide huge task.
I think the solution for this problem is not burn the current core team. The solution is hire more people. Yes is well known that Maidsafe is looking for devs and is a hard task because very specific background is needed to fill the place. Ok, then use the bounty program to be familiar with other devs, maybe you find some good people unexpectedly. With the current system in bounty program this is not gonna happen.

That’s your opinion, not a fact. The bounty programme has been going a few weeks and is being expanded as we see in the OP. The dev team have been making great progress which and we can see things really happening, so I really don’t see the reason why your position is so negative. More bounties is good, so long as it helps, and it is already happening at an accelerating rate.

4 Likes