MaidSafe Dev Update - July 6, 2017

Earlier this week, we invited a first batch of forum users to participate in the internal testing for Test 17. Since then, we’ve invited two more batches (of 7 users each). Thank you to everyone who gave us feedback in this topic and tried out the SAFE Browser and the example apps, this certainly worked out great and the feedback has helped improve different parts of the Authenticator to make things clearer. We’re hoping to have some new binaries for you to test with very soon.

Along with the rate limiter, one feature we’re working on this week is to have a form of client whitelist. Without features such as bootstrap cache and scalable proxy nodes, this is intended to help spread the load of clients amongst proxies and also limit the number of clients that are allowed to interact with the same proxies.

Just like invitation-based accounts, this is a measure that we are adding at least until Alpha 3 or maybe later.

The same system that gives out invitation tokens will also allow you to register your endpoint (IP address) with the network. Once you acquire a token (which can start out purely for forum users and then scale to people without forum accounts), you will also get an option to add yourself to the whitelist. This will assign four proxy nodes from the network and it will flag you as a whitelisted client on those 4 machines. Now when you run client applications, you will be limited to connecting to those 4 machines. Rate Limiter will also restrict a single client from an endpoint per node, this will thereby mean a maximum of four parallel client instances per token would be permitted by the network at any time. For example, if you use the browser and the email app, they will be forced to use 3 different proxy nodes (unregistered client + authenticator from browser and email app). This means you can use up to 2 client apps at a time excluding the browser. Also, for anyone with dynamic IP addresses (most of us), the invitation token system should also allow anyone to update their registered IP with the network.

We’re hoping to put this to the test very soon with the forum users that are on the internal test network, and once confirmed as functioning ok, it should allow us to scale the invites to the entire forum quickly.

New tutorial videos

@hunterlester has created new YouTube videos demonstrating many features of the API. This was done off his own back and he has done a great job, however, see for yourself:

If you would like to start playing with the tools, here are a couple of repo links:

Thanks Hunter :+1:

New developer options for Vault networks

For people wanting to run local networks and test networks, we are going to introduce a bunch of dev options as part of the config files for Crust, Routing and Vaults. This will let people selectively enable and disable features (e.g. being able to enable/disable the external reachability checks, the resource proof requirement, the rate limiting and the ability to run multiple vaults on the same LAN). That way, people don’t have to fork the project, build it themselves and keep track of what the changes are to enable and disable features. See this JIRA task for more info.

This is a feature we’ve seen quite a few people in the community interested in and we’ve been wanting to add something like this for a while. We want to let you know that we intend to support these dev options in the long-term :slightly_smiling_face:

Once implemented, we are also thinking that when we put out the client-only networks (like the upcoming testnets and Alpha 2), we might release the actual vault binaries that we are using and people can use them to run local networks and droplet networks (they will be able to choose which options they want enabled or disabled by editing the config files). This will also help people who want to run tests and profile the network.

SAFE Authenticator & API

The participation of the forum users in the internal testing has been very positive, especially for the front-end team since a good amount of high-quality feedback has been provided in relation to the SAFE Browser and sample apps UI & UX, which is extremely valuable as creating a good UX is one of the most challenging tasks and having many different perspectives is definitely the way to achieve it.

Based on this feedback, the authorisation pop-up in the Authenticator is being enhanced to make it easier for the end user to visualise and understand the permissions that an application is requesting access to. Some screenshots of these changes were shared in this post.

All the issues being reported are tracked and prioritised as we receive them, and the bugs and enhancements related to the SAFE Browser are being treated with higher priority as it’s an official and very important component of the SAFE Network, whilst those related to the email & web manager sample applications generally have a lower priority as they are mainly intended to showcase the use of the SAFE Network APIs, with the exception of bugs that can prevent the execution of the sunny day scenarios.

One of the important aspects of having people participating in the internal testing is that the applications are being tested in different platforms/OSs. However, for the moment, we are supporting only the following set of platforms as they are the ones the team works with and that our CI setup currently supports:

  • Windows 8/10
  • Ubuntu 16.04 (LTS)
  • Debian 8
  • Fedora 25
  • and OS X Sierra

While many other platform variants “may” work, they might not work completely due to the various window manager combinations and other factors that come into play (especially in Linux), and we do not expect to be able to chase down bugs in these variants quickly as we’d likely be working on other features/bugs on the supported versions too. We’ll certainly welcome anyone who wishes to submit PRs for these other variants. As long as they are small issues/changes, we will try to handle them where possible. We’re also hoping to support some containerised binaries via Docker/VM that should hopefully be able to provide the app required environment on other platform variants too, but this isn’t active just yet.

We are also working on enhancing the invitation token system which was used previously for managing the invites that were distributed for creating an account. The plan here is to expand the scope in such a way it will map the IP address of the user with the invite to add to the whitelisted clients in the network. If the user wants to connect from a different IP, the server will allow the user to update the IP to the current one.

Additionally, we internally keep testing the APIs, and @hunterlester keeps working on the development of a playground tool to exercise the DOM API, as well as on the recording of other tutorial videos.

SAFE Client Libs & Vault

We’ve been working on improving SAFE Client Libs and fixing issues in the NFS subsystem in a large batch. The previous implementation and design of NFS (Network File System) has been battle-tested now, and it showed its flaws. It was implemented as a high-level API without providing enough flexibility for end users and it didn’t properly use the NFS functions provided by safe_core. We decided to reimplement it in a better and a less opinionated way, that is more resembling the POSIX-style file APIs in operating systems. Despite the extensiveness of these changes, they should not affect the apps’ code a lot, as they’re mostly confined to the application libraries (safe_app_nodejs and others). These changes are implemented, the pull request is being reviewed now and it will be merged soon.

There’s another improvement in the Mock Vault: now it’s possible to access it concurrently. As the local Vault is stored in a single shared file, there could be a problem in accessing or modifying it from several processes at once. It could result in data races when a process could not see changes made by another one. This was not consistent with the behaviour of the real network and complicated local testing of apps, as described in this topic. We’ve also improved the test coverage for the rate limiter error response, covering it from the apps and authenticator point of view.

On the vaults side, during the internal testing, we found a bug in the code that detects mutation requests conflicting with already pending mutations. A conflicting mutation wasn’t rejected, which resulted in reaching a code branch that should have been unreachable, so this bug could result in unwanted panics and in vaults shutting down. @qi_ma has investigated the case, simulated this behaviour in mock-crust tests, and evaluated whether such a scenario might cause any other potential issues. This bug is fixed now, and along the way we’ve also improved the test coverage.

Routing & Crust

Crust now gets a new feature added into it - a whitelist for nodes and a whitelist for clients. These are provided as optional fields in the config file which if not present would indicate that everyone is allowed. If populated however then only the clients flagged as whitelisted would be allowed to bootstrap off that proxy node; all others would be rejected. Similarly for nodes, only those marked whitelisted will be allowed to either bootstrap or connect to the node the config file belongs to. Further, this is dynamically refreshed, which means that if there was someone whitelisted who is currently connected to us but is no longer so, then the next refresh for the config file will purge such a peer. Also true the other way around - if someone was not whitelisted but is currently marked as whitelisted, then the next refresh of the config will allow such a peer to connect to us. We have made the bootstrap handler check for any updates to the config so that each time someone attempts to bootstrap off us, we will check if they should be allowed to. The WIP PR is here.

This eases some burden off Routing, as now if it gets a successful connection to the peer, it does not have to check for the peer being whitelisted or not. This will allow us to selectively (and dynamically) whitelist clients in the upcoming testnet as they consume their invitation token, thus reducing chances of spam. If a spammer is detected then the config file can be made to block such a peer from ever establishing a persistent connection to us.

The Rate-limiter PR in Routing has been merged and should be feature complete. With that, all the throttling and banning (of malicious clients) logic should be in place. Of course, we will need to thoroughly test it to tweak values and see the returns and are thus looking forward to the upcoming testnets to gain some practical insights on the topic. There is a slight tweak to the approach in this commit that is being tested before merge to see if it improves our tackling of the situation.

54 Likes

First ! hey my friend @JPL where are you now ?
I have to say, waiting a full week for the test 17 invite has inflicted more damage to my psychic health than watching the Juventus lose the champion’s league final to the Real.

All the pieces of the puzzle seem to be coming together nicely. Congrats @maidsafe
@hunterlester those videos are amazing. keep them coming, please, it is so helpful !!!

21 Likes

Damn bankers, damn damn. :slight_smile:

12 Likes

Another good update, looking forward to test the updated binaries once they are released.

If any specific tests are needed on specific platforms please let us know. Right now we are kind of freewheeling and different people are looking at different (parts of) applications.

7 Likes

I know I’ve been busy with a move, etc, but today I’m free for…

#Live Q&A was here!! // ended

#VIEW TEST17 VIDEO #1 HERE
Me simply going through all the software released in TEST 17 (which works great!) so everyone can see.

#VIEW TEST 17 VIDEO #2 HERE
Great little video where I showcase all the websites that people have uploaded to TEST 17

#VIEW TEST 17 VIDEO #3 HERE
Great overview of @Joseph_Meagher 's JS API demo site with explanations from him

16 Likes

:point_up_2:
I’ll say it again… rockstar! along with everyone else at Maidsafe of course :wink:
Exciting times :tada:

9 Likes

Great work here @maidsafe congrats

4 Likes

Someone is trying to join our current live video / audio chatroom and it’s saying this:

I would suggest you don’t try logging in from VPN

1 Like

@maidsafe is killing it! Happy to see the team at a point to direct attention to community wants/needs and hoping all the UI/UX feedback helps out with the upcoming testnet and alpha 2 :slight_smile:

10 Likes

@SwissPrivateBanker Oh, you know, I was kicking back with a nice long drink. It’s far too hot to be worried about being first and whatnot. Next week may well be cooler though and I’ll have had a week off. Keep practicing :wink:

7 Likes

If it’s not this hot or hotter… it’s cold.:rofl:

5 Likes

@hunterlester Thanks for the videos. Along with @whiteoutmashup 's intros it’s just what the project needs. Could you suggest a logial order for watching them? The first one on the list starts “In the last video…” . It would also be helpful to list any pre-requisites, eg node. I’m not a developer so perhaps it’s all very obvious, but I’m just trying to put myself in the shoes of a dev who’s new to SAFE. Cheers

9 Likes

Yeah but you don’t live in Britain. We’re not good with anything above pleasantly mild.

4 Likes

@hunterlester I’ll second that … very interesting to see those videos.
My only comment is the font size doesn’t make the detail easy to see. :thumbsup:

6 Likes

That’s good to know. Is the font too small even if you enlarge the video to full screen?

Let me know, I’ll play with adjusting the font for videos.

@JPL Thank you. I’ve updated that video to reference the other video that I mention.

10 Likes

Finished uploading my first TEST 17 video, 2 more uploading as we speak…

Sorry the appear.in got full; lots of people tried to see my demo live but couldn’t. Now you can!

Test 17 Video #2 here

TEST 17 DEMO VIDEO #3 HERE!

10 Likes

However, for the moment, we are supporting only the following set of platforms as they are the ones the team works with and that our CI setup currently supports

Windows 8 support seems a bizarre choice given market share.

I would suggest the biggest share of corporate desktops is win 7 and the majority using windows for music creation are also using win 7 (XP does not have WASAPI Exclusive Mode support)

Win 8 in my opinion is a waste of developer time… a dog from the outset and most touch based machines would/should migrate to win 10.

2016 market share

13 Likes

Shine on, crazy diamonds!

4 Likes

Thank you for sharing this information. I work on Windows 10. I want to read about what it would take to support Windows 7.

6 Likes

Thanks @maidsafe team for all your hard work,

Super fun to see the pieces of this puzzle coming together and super ants running tests.

@hunterlester many thanks for those vids, these are really needed especially now.

@whiteoutmashups good looking bro, vids just say so much more about the SAFE Network :stuck_out_tongue:

4 Likes