SAFE Network Dev Update - January 30, 2020

Summary

Here are some of the main things to highlight this week:

Vaults Phase 2

Project plan

We have made some progress with executing client tests against a real vault network, but this time locally on a single machine. With a network of vaults, all routing nodes being Elders, the client can bootstrap to them and all tests pass. This was a machine with sufficient memory to handle a load of 7 Vaults. The good news is that the components are working together nicely. But, the memory requirement is still higher than expected. With these observations, we can test the network functionalities and carry out network debugging more conveniently.

SAFE API

Project plan

New safe-cli v0.8.1 and safe-authd v0.0.4 releases have been published which incorporate all the fixes and features we’ve been working on since last week’s release. As before, you can update your CLI with $ safe update, and if you already updated to (or installed) authd v0.0.3 last week, you can update it to v0.0.4 with $ safe auth update, or otherwise install it with $ safe auth install. In any case, the full instructions can be found in the CLI User Guide.

This new version of the CLI adds two new commands as we’ve been commenting in our previous dev updates. A xorurl decode command is now available which allows extracting all the information that is encoded in a XOR URL:

$ safe xorurl decode safe://hnyynyzonskbrgd57kt8c1pnb14qg8oh8wjo7xiku4mh4tc67wjax3c54sbnc
Information decoded from XOR-URL: safe://hnyynyzonskbrgd57kt8c1pnb14qg8oh8wjo7xiku4mh4tc67wjax3c54sbnc
Xorname: e02b282430f7d544ec93441969c63c387a261d7d553d2f9a8b3dda270fcb37ab
Type tag: 1100
Native data type: PublishedSeqAppendOnlyData
Path: none
Sub names: []
Content version: latest

As suggested by the community, this new version also adds a files ls command which outputs the content of any FilesContainer in an analogous way to the traditional ls Linux command, also allowing us to specify subfolders in the path of the safe:// URL to browse the hierarchy of the target FilesContainer, e.g.:

$ safe files ls safe://hnyynyi6tgumo67yoauewe3ee3ojh37sbyr7rnh3nd6kkqhbo9decpjk64bnc
Files of FilesContainer (version 4) at "safe://hnyynyi6tgumo67yoauewe3ee3ojh37sbyr7rnh3nd6kkqhbo9decpjk64bnc":
Total: 6
SIZE  CREATED               MODIFIED              NAME
11    2020-01-28T20:26:05Z  2020-01-28T20:29:04Z  another.md
38    2020-01-28T20:35:43Z  2020-01-28T20:35:43Z  files-added/
30    2020-01-28T20:31:01Z  2020-01-28T20:31:01Z  new-files/
10    2020-01-28T20:29:04Z  2020-01-28T20:29:04Z  new.md
23    2020-01-28T20:26:05Z  2020-01-28T20:26:05Z  subfolder/

Some enhancements have also been made to the authd install procedure for Windows: it now checks if there is an instance of authd already registered as a Windows service, and requests the user to first uninstall it before being able to install a new authd as a service. This is to make sure the authd service is registered with the correct and up to date safe-authd.exe installation path.

A minor change, but very useful when it comes to troubleshooting issues, is the addition of the authd binary version to the authd status report. You can now check which authd version is running and responding to CLI requests by checking the version reported in the $ safe auth status response:

$ safe auth status
Sending request to authd to obtain an status report...
+------------------------------------------+-------+
| SAFE Authenticator status                |       |
+------------------------------------------+-------+
| Authenticator daemon version             | 0.0.4 |
+------------------------------------------+-------+
| Logged in to a SAFE account?             | Yes   |
+------------------------------------------+-------+
| Number of pending authorisation requests | 0     |
+------------------------------------------+-------+
| Number of notifications subscribers      | 0     |
+------------------------------------------+-------+

Note that once you have updated to the new safe-authd v0.0.4 (safe auth update) you will need to restart authd to launch the new version (safe auth restart).

There was also a bug fixed which was affecting accounts created with safe-authd. Due to the way it was parsing JSON-RPC messages, the account credentials were being passed to the network with enclosing quotes ("..."). This means that with this new version of authd, accounts previously created won’t be found and login will fail. Thus for those actively using the CLI and authd, once you have updated to the latest version you would need to either create new accounts, or as a workaround if you really want to use any previously created account, just make sure you enter the credentials with enclosing quotes ("...") and that will allow you to login with them. For example, if my passphrase was mypassphrase and my password was mypassword then I can still access that account after upgrading by entering my details as "mypassphrase" and "mypassword".

Labelled Data, Indexing and Token Authorisation

RFC, Project Plan

This week, @yogesh has joined forces with @joshuef for token implementation, and they’ve started working on feedback from the initial POC implementation, as well as starting deeper planning of the next steps for tokens. This involves a new, separate RFC (coming soon) just for tokens, to give clarity there.

With this, we’ll hopefully start to make some good progress here over the coming weeks!

Data Types Refinement

RFC, Project plan

Clarification of the Private | Public scope implications for Map has been added on request by @tfa. Additionally, details of possible extensions to PrivateMap deletion options have been added.
The Sequence PR is close to being merged, as most required changes have been implemented. A few minor things have been deferred to follow-up PRs.

Data Hierarchy Refinement

RFC

Discussions in the RFC have been ongoing during the week, with several suggestions from the community. Among these are using an additional encrypting layer of nodes as a way to deal with obfuscation of vaults (@jlpell), and a suggestion for natively supporting RDF through the introduction of a Triplet data type, i.e. giving a native triplestore (@JoeSmithJr).
Also, discussions around management of Shell metadata growth has gone into some previous work done in this area, specifically building trees of the native types. @bochaco added a few diagrams with examples he’s been working on previously.

SAFE Network App (desktop)

With new changes and APIs being added to the CLI and safe-authd, we decided to take advantage of this and simplify how authd was included in the SAFE Network App. We’ll now be using, managing and requiring the install of the CLI and authd from the SAFE Network App UI. Users will be informed if either the CLI or safe-authd aren’t installed and prompted to install them. Only then will Login options show up.

As well as using the same codebase as the CLI and reducing some Windows checks in the app, we also streamlined admin prompts on Windows (wait for prompt to be accepted before attempting another). So now you should get one prompt at install, and one at startup (as opposed to possibly getting a barrage of prompts). Which is nice.

Finally, we added another handy feature: the SAFE Network App will only attempt to stop the auth daemon if it wasn’t already started. So this should not interfere with any authd instances started from the CLI for example.

SAFE Browser (desktop)

We’ve released a couple of alpha browser releases this week. Mostly maintenance, but also fixing a couple of issues @latch has been hitting as he’s working towards a WASM-based web app. On macOS, there was an issue where opening devtools could crash the whole browser. An annoying one to debug as devmode didn’t have the problem (which was in itself, intermittent). In the end, it was a simple case of needing to URI decode some devtools requests, which were (falsely) being treated as safe requests.

These issues have also been raising some good questions about our API and how far we should be including HTTP type responses (or not). More on that on the Dev Forum.

SAFE Browser / SAFE Authenticator (mobile)

We spent this week testing the authenticator and mobile browser apps on Android and iOS and in the process we made lots of improvements and bug fixes.

The testing in the authenticator app led to the discovery of an interesting bug in safe-authd. We noticed in the mobile authenticator that when we created an account on a vault, for example the shared vault, we could not log in through the CLI or through the SAFE Network App using the new account details. This was also an issue in reverse, so no account created through the CLI or the SAFE Network App would work on the mobile authenticator. After some debugging from @ravinder he was able to narrow down the cause, then work with @bochaco to pinpoint the cause of the bug in safe-authd :mag: You can read a bit more about this bug and what the implications of it being fixed are in the SAFE API section of this update. It was an important one to catch as the longer it was in place the more accounts were created incorrectly via the CLI and the SAFE Network App.

In the authenticator app, we were using some custom controls which on reflection we decided were outdated and occasionally showed some buggy behaviour. So this week we decided to remove all of those controls and replace them with the material design controls. We feel that the result is that the app now looks much better and feels smooth, plus this enabled us to remove around 1k lines of code.

With several bug fixes implemented in both apps, we are making good progress, but there are still some outstanding bugs and tests to get through before we are satisfied that these apps are ready for you to get your hands on. For example, tests to confirm correct behaviour of the safecoin related changes in the authenticator app are still pending. Some further browser app related bugs were reported by the internal team yesterday and will be debugged and fixed in the coming days.

Things are taking shape nicely here as we progress towards our new app releases.

Safecoin / Farming

RFC

This week we divided research into two areas: CRDTs and DBCs.

CRDTs offer eventual consistency between replicas (Elders in our case) without the performance hit that comes with requiring that all nodes see data in the same order, e.g. via PARSEC consensus. However, most CRDTs do not enable us to enforce a rule such as balance >= 0. To this end, we’ve started looking at the Bounded Counter, a little known type that does enable enforcing such rules, at the cost that each replica must serialise operations internally, e.g. by keeping a hash-chain of updates. Research is ongoing, but the approach seems to have promise.

DBCs, as discussed last week, offer a mechanism for offline payments, such as gift cards. However, in combination with a recipient’s public key, they could also be utilised as a full-blown payment system. In such a system, the user’s wallet would hold individual certificates of specific amounts, equivalent to cash bills. Before making a payment, the payer would typically split/combine the certs as necessary to obtain the proper amount for payment, and then encrypt the certs to the recipient’s public key and sends them to their inbox. New DBCs would be issued via farming. At this time, we have an informal brainstorming document, where team members have been batting around ideas and questions.

One inspiration for such a system comes from the November 2019 Scrit Whitepaper, which describes a multi-sig based DBC payment system, albeit with epochs and other complexities that we do not believe would be needed for the SAFE Network.

Node Ageing

Project plan

Continuing with Routing code clean up, we made good progress toward having a unique message type for both Direct messages from one node to another, and Routing messages that are relayed through nodes before reaching their destination. Initial clean up work is already merged, and we are finalising the remaining work.

Additionally, we finished the work we started when allowing Adults to verify updates to section Elders. We re-used the same simpler mechanism for verifying a relocation request. We no longer need to have signatures accumulated through Parsec and so can re-use the same process shared with messages between sections.

Useful Links


Feel free to send us translations of this Dev Update and we’ll list them here:

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the SAFE Network together :tada:

75 Likes

wownhsdnbbd

Hey does this mean that I’m first?

Now read :laughing:

Update:

Thanks for another great update Maidsafe devs

Can’t wait :clap: :clap: :clap:

28 Likes

first …

nah… :unamused:

12 Likes

Can I get you a drink of water or something?

8 Likes

i cheated I quick typed “wownhsdnbbd” I just had a beer
Thanks cheers :stuck_out_tongue:

You actually win, because you’ve been playing a lot with the CLI (I wish I could) :sob:

13 Likes

Oh I know you cheated…

Yeah - doing something useful with it would be better

safe://pies doesn’t really count

11 Likes

I win cause I actually read the updated then commented! :rofl:

Nice that this has being given a little thought, lets hope it does not distract though, perhaps the distraction is worth it?

21 Likes

Sorry but I dont understand the updates any more,The routing part that the team focused on this week is not mentioned in any of the Git routing projects tasks to be completed for example.
Is the remaining work on vaults (vaults 2b,3,4) on hold until the memory issue is solved?This is also not clear to me.

3 Likes

That rule was superseded some time ago in favour of the Instant Gratification Rule

14 Likes

we should make an event, like a mini survey, of some questions, so we can know who, surelly, readit & post first e.e

2 Likes

Great update! Glad to see the issues with Vaults are clearing up. Also, find the two research areas on Safecoin quite interesting. Can’t wait to find out more :smile:

21 Likes

Thank you for your hard work, everybody.

Please try not to feel like everybody is just tearing at you demanding the impossible, David. We, the baby ants, are on your side too.

21 Likes

Thanks so much to the Maidsafe team for working so hard. We all appreciate it!

And Congratulations to @19eddyjohn75 for being first! What happened to “Macho Man” Randy Savage on your profile photo? And who is this new guy you have?

18 Likes

I thought all researches done long time ago, and they only code right now?

2 Likes

Later at Vaults Phase 99, it would be great to see zombie vaults able to come back from cold and still be useful! Cold boot of the network would be a strange capability - something that you can kill and it still comes back, is beyond even something that will not die!! :zombie:

Midway between then and now, I wonder that a vault that loses connectivity for a short time, while perhaps losing kudos relative to another that does not, should still be able to recover its state in the network.

I’ve only recently noticed that both home routers here reboot on occasion… I don’t know what that is - auto-updates are off - but if it’s common that home routers do reboot every week or so, then there would be a minutes loss of connection on a fairly frequent basis across many nodes. Having nodes that can adapt to that might be useful? :thinking:

12 Likes

@davidpbrown

I’ve thought about that issue as well. Not only power and telco outages but also people moving residences. Depending upon precisely how performance is ranked and status lost, if there is not some tolerance for nodes going temporarily offline, nodes could eventually become ‘centralized’ in datacenters.

11 Likes

Depends how recoverable a node is - that is, how useful it can remain. I wasn’t expect a move of location would be possible but its an interesting thought for longer term. Also, brownouts get talked of on occasion as a feature of the reality of some electric grids… I don’t know enough about whether internet services go down and it’s just internal to homes that is affected; if they are deliberately short periods, then UPS can cover that off for a while.

4 Likes

DBCs will be epic! Node ageing progress also epic! Great job again team, full steam ahead, exciting times! I’m shouting a lot.

19 Likes

At the risk of troll-feeding, research will never end b/c new challenges and tasks will always arise. What concluded was abstract research further afield from actual development and implementation.

20 Likes

Good to see that I am not the only one here that doesn’t understand the updates. Cheering madly as MVP is another 7 days closer

8 Likes