MaidSafe Dev Update - August 10, 2017


#1

Last week, we released Test 18. The focus during this week has been on getting all the issues that need to get completed for Alpha 2 (including those reported by the community) added to JIRA and making sure there are no loose ends.

With Test 17, we had the rate limiter quite relaxed and a lot of traffic was allowed. In Test 18, it is kind of the opposite since there are many more reattempts and retries that need to be happening from the clients. We are now working on striking a good balance between these two extremes and we’ll be updating the rate limiter to improve its performance and give a better user experience in terms of data traffic. With these changes, a lot more future requests should go through in the first attempt.

Advisory and Endorsement Requests

Members of the team, especially David, are getting quite a lot of direct approaches, either through lengthy DMs and LinkedIn messages, from people looking for advice on other projects, or to endorse ICOs. While we would like to be in a position to help, our focus on network development does not allow us time to read these messages in many cases, let alone respond. Please do not be offended by this, it doesn’t mean that we don’t see an importance in what you’re doing, we are simply very busy and your time is likely to be better spent engaging with other potential advisors.

SAFE Authenticator & API

Issues that were raised in Test 18 have been gathered and organised in JIRA. The new IPC for sharing MutableData is being integrated this week. Integrating the IPC functions in safe_app_nodejs and authenticator is almost completed. This feature has been the priority for us this week and @bochaco and @shankar are getting this integrated and tested end-to-end with the applications.

We have also fixed a few issues this week. The native modules were not being loaded by safe_browser on a few Windows machines which did not have the 2015 Visual C++ redistributable. When the user tries to log in, the authenticator kept spinning and it was misleading as if it was not connecting to the network. We are now handling this case and the proper error message is presented. safe_browser issue #99 is also resolved.

When the browser is trying to load a page even if the requested page/resource is not available, the loader in the browser keeps on spinning. This issue is now resolved in the master branch of beaker-plugin-safe-app.

Based on feedback from the dev community, the getHomeContainer API is being renamed to getOwnContainer. This API is used to fetch the application’s own/root container. The DOM APIs have also been updated with this change. Please do keep a note of this change in case you are trying to build the browser or your apps with the latest code in the master branch of safe_app_nodejs and beaker-plugin-safe-app. Also, @hunterlester will be resolving safe_browser issue #100 soon.

The Web Hosting Manager app updates the progress bar only after the file is uploaded to the network. If a big file is uploaded, the progress bar takes a long time to update, leaving the user guessing whether the application is hung or still working. This will need to use the NFS API to write as a stream instead of passing the entire content at once. safe_app_nodejs already exposes APIs for the same. The Web Hosting Manager app must be refactored to use the API to write as streams and update the progress bar as chunks of data are being uploaded. @hunterlester should be able to resolve this issue this week.

@joshuef has raised a pull request for storing the history and bookmarks in the network. The feature looks stable after a couple of testing iterations and we’re expecting it to be merged soon. @joshuef is looking into the DOM APIs to enable returning the error code along with the error message. Right now, only the error message is returned as beaker browser does not allow the APIs to return complex objects. @joshuef is trying to implement this feature for returning errors.

SAFE Client Libs

The back-end team has been working hard on fixing bugs reported by the front-end team and the community, and improving the features that were released last week. We’ve started with refactoring the account packet structure and improving safe_core, the essential parts of the SAFE Client Libs. The account packet is the data structure that holds the user account details, keys, and some extra information, such as pointers to the root directories - and that is the part that we wanted to improve because it remained almost intact since the SAFE Launcher days, while now we have a much more complicated design, with apps and the Authenticator. So now we’ve got rid of unnecessary indirections (the pointer to the access container is now stored as a part of the account packet) and inefficiencies (the root directory has been removed and the containers info is now stored directly in the access container). Along the way, we’ve reduced the number of network requests and parallelised some operations (which previously introduced useless synchronisation delays), so overall these changes should make the UI more snappy. They’re mostly completed and likely to be merged before the end of this week. For more details, you can see this JIRA task.

In parallel, @canndrew and @adam have been working on improving the new API for sharing Mutable Data. The basic implementation has been completed and merged and now we’re working on implementing useful comments and suggestions from @bochaco and the front-end team. First, we’ve decided to clearly define the key for the metadata entry of shared Mutable Data objects because allowing an app to provide its own metadata key could lead to security issues. For example, if a password manager app sets the metadata entry to say something like “this data must not be shared with other apps”, another malicious app requesting shared access could point to some other entry to be looked up for metadata, effectively bypassing the caution message and making a user confused about the intent of the sharing request. Second, the permissions part of the request has been simplified: now we assume that an app wants to request only Allow permissions, delegating more fine-grained permission settings (involving blacklists and Deny) to apps having the ManagePermissions privilege. All these changes go in this pull request and we’re currently reviewing them and perfecting the details.

Next, we’ll be focusing on optimising apps revocation. We’ve started with this pull request, which fixes some flaws and edge cases, such as attempts to authorise an application during an ongoing app revocation. This is not allowed now, as authorisation might interfere with re-encryption of revoked containers. Another issue could happen if an app tried to update or insert new data into a container which failed to be re-encrypted - in some circumstances it used an incorrect set of keys and mangled the data. Tomorrow we’ll continue addressing similar minor (but crucial) problems.

On top of that, there was a lot of smaller improvements. @marcin has been doing important work and added more automatic tests, confirming and fixing bugs found by the front-end team, such as a case when subsequent app re-authorisation didn’t result in the creation of app’s own container (if it wasn’t created on the first request). @marcin also added more NFS tests and verified that a user can still log into his account and browse the network even if he’s exhausted his account balance. He’s also removed a legacy module, public_id, which remained from the SAFE Launcher and wasn’t actually used by the front-end. @canndrew has updated the Routing dependency to the latest version, removing the necessity of retrying requests if a request rate limit has been exceeded - now that this is the Routing’s responsibility, we’ve considerably simplified this part of the code in safe_core. Finally, @adam has implemented an environment variable switch that disables the on-disk vault storage for tests. This change should speed up both front-end and back-end tests a bit.

Routing & Vault

As stated in the introduction, the rate-limiter for Test 18 is stricter compared to the previous test network. Message resend logic has also not been too efficient as Routing breaks the user messages from upper libraries into user message parts, each of which can be a maximum of 20 KiB. So if there was a user message which was 200 KiB, Routing would have broken it into 10 chunks of 20 KiB each. If a few of them were rejected, we’d currently be sending all parts again from the Client libs. The user library would then have to resend the entire message again after a pre-defined waiting period. To optimise this, we’ve modified Routing to handle the message resend at that layer directly. With the new changes, Routing will now selectively re-transmit the user message parts for which the error occurred. This encapsulates the message resend logic away from the upper libs. Routing also optimises route traffic by allowing the upper libs to indicate a maximum wait period after which message relay via further routes will not be attempted. This not only applies to rate limiter errors but also other parts such as the Ack-Manager.

There have been a few patches that have been brought in to master to address certain bootstrap bugs from Test 18 and the Rate limiter has also been refined this last week to allow clients to scale their data traffic capacity based on concurrent client connections active at the proxy at a given time. The PR addressing the same is currently waiting to be reviewed and merged. Rate limiter throughput calculations have also been updated to reflect the actual data throughput relayed by the proxy nodes to the clients.


#3

Alpha 2 is so close I can taste it :stuck_out_tongue:


#4

Alpha 2 is very close for like last 4 months :thinking:. Why so much delay :joy:. I am not tasting it for next 4 months :grimacing:.


#5

You kids are so quick @nigel & @Shan_Awan
I will read now :stuck_out_tongue:

Thanks Maidsafe devs for the long hours you gals/guys make to get this baby running.

This should be forbidden, people know damn well that they should not bother the core devs working incredibly hard to get this network up and running. Create your apps and promote it yourself, hey I found this incase somebody wants to contact David:

:face_with_raised_eyebrow:


#6

So great, this is what SAFE brings us people, a online universal database where your stuff is secured until you unlock it.

I have some sort of idea, but where will app devs use this?

Thanks for the update, as always very detailed. Keep the coffee flowing in Troon :coffee:


#7

I am sure I am not the only one who checked to see if that was real.:grinning:


#8

Sorry for the silly joke, but I wished it was true, people should stand behind their own project and not dare to bother the core devs. Crybabies pay up, 1 pound for every Maidsafecoin out there.


#9

I understand people just want answers from those who can answer it best but it’s just not the best timing atm. I think after the core components are there then there will probably be more willingness to engage with lost devs. It’d be much better if the documentation and SDK’s were enough at that point though. Which I’m betting will be :slightly_smiling_face:


#10

Cool. These update brings so much to digest in one day. Really some hard work. Improving API …WOW!!


#11

Yeah, got to sift through the masses of people contacting you to just get paid and leave,

And find the 0.00001% who are here to actually change the world no matter what. I’ve been dealing with that a bit too. MaidSafe is one of the few that’s not about just trying to scrape off of everyone. There’s something that’s actually world changing here. Futuristic tech will improve everyone’s lives so much more than dead dollars yet few can see that.

Sorry to see that it’s gotten so poisonous and loud that the team even had to write a paragraph on the update itself…

< /rant>


#12

Agree, but they need a couple of PR people now to deal with incoming and outgoing and also to facilitate development of collaborative work.


#13

we are very very very very very close i guess… :disappointed_relieved:


#14

I wishhhhhhhhhhhhh :joy:


#15

Great update - brilliant to see more issues being sorted with the tests & features maturing.

@Wallet111 @Shan_Awan

Stop your whinging & do something useful with test 18, which is a functional precursor to Alpha 2.

Nobody has ever achieved what MaidSafe are doing here, so as has been said so many times it’s not going to be a predictable path where everything gets implemented or fixed in one hit or on a set timeline.

If it takes another 3 tests & 6 weeks or more to make sure Alpha 2 is fully baked, great… just hope trolls don’t whinge on every dev update until then :smiley:


#16

As trolling goes, it’s extremely mild. I feel similarly about “soon” but hold it back.

Good work (better edit that in!) Safers (both Troon and external).

Feels like diminishing returns now (in a good way), and we will see alpha 2 in the coming weeks though. However it’s been so iterative that I feel like I’ve been there, got the t shirt and I am champing ah the bit to hear about data chains each week. Bring on the next chapter.

May I ask if the “Alpha 2” announcement , whatever week it comes in, will be on a Thursday as they usually are, or is there a non zero chance of it being made on a different day of the week? This is purely a selfish and financially motivated question. I order my trading with an eye on Thursdays as we close in on milestones, and it would be a nasty shock to say the least if Alpha 2 was announced on a Monday afternoon ! (Assuming it will move the price at all)


#17

Apps can set Metadata to the MutableData. This Metadata will have name and description.
For example, if the web hosting app is creating a service container for a public id krishna, it will set its metadata as

{
  name: 'Service Container for Public ID Krishna',
  description: 'This is the root service container where all the services for the Publid ID krishna will be stored'
} 

When email app is requesting access for this service container, it will send the XOR name and tag type in the request.
If we present just the XOR name in the UI, it wont be helpful for the user to decide on what the MutableData is related to.

Authenticator will try to retrieve the metadata if it is available and present it in the UI. It is not mandatory to set the Metadata for requesting access, it is only a good to have as a standard practice to help the user decide before they grant the permission.

Once the IPC is integrated with the example apps and tested, we can walk through the code flow in the dev forum using a simple snippet.


MaidSafe Dev Update - August 17, 2017
#18

Not quite sure what day the alpha 2 announcement will be on @danski. We are looking closely at how and where we do this.

I know what you mean, but the alternate approach is to try and make bigger jumps and release software that has significantly more bugs in it. So we have chosen the more iterative and steady approach and less (hopefully) bugs.


#19

Teaser. Pure marketing. My guess is Alpha2 is not too far off but within site, No comment would signal not within sight. Historically this means 6 months. Jan Feb 2018


#20

no comment

Only because I like logical paradox’s :wink:

it’s a short number of weeks though, but not next week. Lots of jira issues being killed off right now and a few more to go after Thursday, even by best most optimistic estimate. :smiley: So not too far but not this week or next I feel. Thing is MaidSafe staff do not know either and we don’t guess but we can see the current workload and resources we have. There is a few holidays this last few weeks and next which all adds to time, unfortunately.


#21

Bingo! 4-5 weeks. Thanks. :wink: