MaidSafe Dev Update - August 24, 2017

Today, we started testing Test 19 internally. As mentioned last week, we are continuing to work through JIRA tasks and fixing issues that need to be completed for Alpha 2. The network itself is looking good in internal tests and we don’t expect to make major changes to Routing & Vault before Alpha 2. However, there are still a few front-end issues we need to fix before releasing Test 19. Here are some of the issues we intend to address in the next few days:

  • MAID-2288: Uploading a directory with many files fails in Web Hosting Manager
  • MAID-2289: Permissions list in Authenticator always shows a full list of permissions
  • MAID-2270: Update Web API playground tool with changes in containers permissions API
  • MAID-2290: Show empty log page / log size
  • As well as other minor issues and UI changes

We initially hoped to get this testnet out tonight, but we feel these pending issues might affect general UX quite a bit. We’re hoping to have these resolved quite soon and get the binaries out to you to test these new features. It is fantastic mind you, to not be waiting on backend, vaults and routing etc. as these bugs are annoying but not keeping us awake at night. We do feel the pain of delays, probably more than most and we are desperate for the energy test network releases give us all. While backend libs are stable and Alpha 3 work continuing from the guys there, Alpha 2 and these pending front-end tasks remain our highest priority to complete as soon as possible to get the new test network going to keep progressing with the roadmap.

Recruitment Continues

As discussed in previous updates we are growing the (local) team across areas including engineering and marketing. We are also now looking to recruit a Help Desk Manager and a Release Cycle Manager to aid with the ongoing rollout process.

The Help Desk Manager will plan, set up and run a help desk (using JIRA) as the network advances through the up and coming test iterations toward beta and beyond. They will also record relevant information from test networks and compile update reports, as well as identify trends in support requests and perform root cause analysis.

The Release Cycle Manager will be multifaceted and cover quite a lot of areas including a combination of: thoroughly testing all new releases, ensuring that all risks, impacts and dependencies are all clearly understood and communicated. They will also write, maintain and manage all the technical documentation, something that we all appreciate needs worked on. We also envision that the successful candidate would get involved in helping to showcase and demonstrate new feature releases to all stakeholders.

What we are looking for with both roles is a very proactive approach to helping all users of SAFE software and this is another important step in positioning the company to be more outward facing and commercially minded, and to garner ever more feedback from users with a view to improving our future releases.

SAFE Authenticator & API

The main focus and effort during the last week on the front-end side was finalising the integration with some changes required in the safe_client_libs and safe_app_nodejs APIs to enable the sharing of MutableData which impacted several artifacts including the UIs and UX of the applications.

As explained in the previous dev update, both the Web Hosting Manager and email app now support sharing the same public ID for creating/adding email and web services, regardless of which of them created the public ID. This implies an authorisation request sent for the user to allow/deny sharing the MutableData behind the services’ container that is associated with that public ID. Internal integration tests of this new flow in Test 19 are going as expected and no issues have been discovered so far. The authorisation pop-up in this scenario displays the MutableData’s metadata to the user, but since this information is not enforced to be set at creation time, it can be empty. Therefore, in order to help the user make a more conscious decision at the moment of authorising a MutableData to be shared, the Authenticator has been enhanced to not only display the MutableData’s metadata information, but also to show the list of applications that are already authorised to access it, which includes the application which originally created it. The tests so far are showing that this functionality is also working as expected.

In parallel to the above, safe_app_nodejs and authenticator were adapted to accommodate to enhancements made in the way that the containers and MutableData permissions are retrieved through the APIs, making it more homogenous. One of this enhancements is to be able to retrieve not only the list of containers that the application has access to, but also to get the set of permissions the application was granted for each of the containers. This forced us to rename one of the APIs exposed by safe_app_nodejs and the DOM API, from getContainersNames to getContainersPermissions, to make it coherent with its semantic.

One of the other goals for the last few days was to try to work on some minor issues reported by the community either on GitHub or on the forums, and several of them were fixed or verified to be already solved, while others are being actively reviewed at the moment, while we also try to provide as much of a help as possible to the developers asking questions.

As mentioned in the introduction above, the main focus for the next few days is to solve a few issues that we prefer to close before sharing the new binaries with the community, as well as perform more tests to make sure that all the pieces are working fine together; in the front-end there are several integration points and covering all flows is not a trivial task.

SAFE Client Libs

This week, we’ve been focused on fixing known bugs and reviewing existing functionality in the SAFE Client Libs. We started with issues that were mentioned in some of the previous updates: we suspected that requesting app revocation from several Authenticators simultaneously (e.g. from several machines at once) wouldn’t work, and this was a known limitation. Testing this feature is hard, as we need to simulate this behaviour from several processes or several threads, which are not deterministic - that is, we need to run the tests multiple times to verify that they’re working as intended and there’s no undefined behaviour. @ustulation outlined the test scenarios in the JIRA task MAID-2235, and it’s been implemented by @adam in this pull request. The test results have shown that parallel revocation was not working due to some minor errors in the flow, and these bugs are fixed now by this PR (which is currently being reviewed and will be merged soon).

We also discovered another obscure bug that could result in the mock-vault data corruption (it doesn’t pertain to the real network). In some cases, after a user deleted an entry from a Mutable Data object on the network, authorised apps could not load anything from the mock-vault, so it was deemed corrupted, as described in this bug report. But as it turned out, the case was a bit more involved: we didn’t truncate the file contents before writing to it, so there could be a case when some junk data remained at the end of the mock-vault file. This is now fixed by this pull request.

In the mean time, @marcin worked on adding more tests to our suite, verifying that existing functionality works as designed. For instance, we have confirmed that the complete removal of revoked apps is working fine and that sharing access to arbitrary Mutable Data objects on the network cater for edge cases. Apart from that, @marcin has implemented a new API function to fetch a list of apps that have access to a certain Mutable Data object on the network. Other minor improvements include bug fixing in the SAFE App library (the file_close function didn’t return an expected value) and refactoring of access container functions (which are now made more flexible and generic) - both changes are landed in a single pull request.

To conclude: we published a new version of the system_uri library (the changes optimise apps IPC communication on Windows and Linux) and released our core libraries, safe_app and safe_authenticator, as version 0.3.0, and safe_core as version 0.26.0.

Routing, Vault & Crust

Routing had version 0.33.0 published, which contains the major features of improved rate_limiter and couple of bug fixes, as described in the previous dev updates. We are also investigating a new swarming mechanism, and aiming to make it not only secure but also avoiding the complexity of O(N^2). As an additional benefit, it is also being evaluated whether it is possible to reduce the number of connections a node has to maintain. This would allow larger sections, which improves security.

The Vault PR to integrate with the latest Routing is finally merged. This utilises the new Routing rate_limiter features: soft capacity and resending parts. The previous re-sends on exceeding limit carried out by the test_client is no longer required. And the new version of Vault, 0.17.0, also got published.

Crust has been getting some love again this week. One noticeable improvement is that it now uses file-system notifications to update its configuration whenever its config file is edited. An effect of this is that if a node has a configured whitelist of peers, changes to that list will take effect immediately, dropping connections to de-listed peers and allowing connections from newly-whitelisted peers. There has also been a small amount of general refactoring in Crust, which has fixed a race condition which would occasionally cause our continuous integration tests to fail. Additionally, there’s been some small bug-fixes to Crust’s test suite.

85 Likes

Among the first. Reading now

3 Likes

Awesome cool update .

7 Likes

Nice update, keep up the good work!

15 Likes

@marcin has gotten up to speed quickly it seems! I like the perfectionist mentality of this team personally. With how long Alpha 2 should be running its good to get all this out of the way now. Keep up the great work!

18 Likes

Great update.

It’s good to see some more focus in recruitment shifting towards user facing tasks like documentation and issue tracking as launch keeps edging closer!

The focus on user experience will make the safe network stand out in the crypto world which is generally very much by techies for techies.

Looking forward to test 19 and Alpha 2, and even more to seeing the beginnings of public data chains testing following Alpha 2 launch :slight_smile:

Well done all, and keep up the fantastic work!

18 Likes

Live Q&A here if anyone wants!

I know I’m a few hours late, been traveling, but I’m here if anyone wants to! Thanks

5 Likes

I’m really impressed by the amount of progress every week.

Where can I read about the existing and / or new swarming mechanism?

16 Likes

F Yeah! More engineers and marketers.

If there is anything we can do let us know.

3 Likes

The frequent development updates are great. Gives more transparency and progress of the network. SafeNet is an unreal project.

7 Likes

lol I just found this link, where you can see who are the most active users on this forums.

safe to say I’m the 3rd most active on this forum? after Mr Irvine and Mark :slight_smile:

Also I give the most :heart:'s / love :smiley:

18 Likes

That link is for the people who have given the most likes and you are the top

Most replies is
https://safenetforum.org/u?order=post_count&period=all
and @happybeing is the highest there

And from either of those links you can select other sortings to see who has viewed the most posts for instance

5 Likes

yeah I just liked to think I’m #1 :smiley: haha

but yes there are many criteria. I never knew about that link before. Was a cool find on the Discourse Meta forums. And @neo you are definitely wayyy up there too!

4 Likes

I just mentioned the others in case people wanted to see other sortings

2 Likes

Yeah I always really appreciate it when you like my posts. I think you contribute a lot to this forum and you put in a lot of work to support the project without always getting a lot in return.

11 Likes

hey it’s been an awesome ride with everyone, and i’m sure that once SAFE works as promised, that will be more than enough to make up for all the work all of us have put in. There are so many good souls on here every day, helping, answering basic questions, while receiving nothing in return, and we all just want to see the better world that SAFE will allow :smiley: that’s truly worth more than anything

14 Likes

So what does it mean, there are 2 different teams? One for the backend and one for the frontend?
And the backend team is working on Alpha 3 tasks already? So the team on Github where dirvine is in, is only for fronted development? Sorry, guys I’am a little bit confused :slight_smile: hope someone can explain it to me :wink:

1 Like

Yes there are different teams. Each major component has a “team” and some components are classed as backend and some are front end (UI) and soon to be help desk, test coordinator too which eventually would have teams under them.

7 Likes

This is very cool btw. Missed this yesterday.

8 Likes

Anyone know where I can get a life? :wink:

Whoops, there I go replying again :scream:

23 Likes