SAFE Network Dev Update - July 18, 2019

Marketing

We started off this week with a bang at a fantastic meet up in Glasgow on Monday evening. A huge thank you to all that came and a bigger thanks to @DavidMc0 who set up the entire event. We started off the night with @Jean-Philippe giving an overview of the projects he’s working on, followed by a Q&A then plenty of lively discussions on how SAFE will change our relationship with data. We can’t wait for another one already, so keep your eyes peeled for the next date!

Also this week, we’re delighted to announce that our very own UX mastermind @jimcollinson will be speaking at the SAFE Brighton meet up :tada:. It’s been incredible to see the community so engaged with his work so we would suggest you take advantage of this fantastic opportunity to hear him in person and of course, it’s the perfect occasion to ask all and any questions. Register and sign up details are here.

Another week so of course…another Tweetstorm. Please give it some love and RT and of course send any suggestions of stories our way!

AND…we’re not done. As development speeds up, one of the aspects we’re keen to support is community contributions as our libraries receive more interest. So, we’ve been investigating the GitHub Sponsors program, which lets us sponsor developers and those of you who help with documentation and other non-technical parts of the Network on a monthly basis. Everyone who contributes to an open-source project is eligible to become a sponsored developer, you just need to sign up to the waitlist to be part of the next beta phase.

Interested? Good! Head over here to find out more and pop your name down on the waitlist.

One more thing worth mentioning. We’ve been squirreling away in the background, looking at ways we can become more succinct visually when presenting ourselves to the world. Why is this important? There’s a whole field around brand identity and the use of visual cues that are part of creating strong brands. It’s often an undervalued part of marketing. So by starting to really hone in on our brand and paying attention to how we visually, and tonally, communicate with the world, we’re setting ourselves up in good stead for launch. Working in collaboration with UX, we’re in the final stretch of getting these agreed so you’ll start seeing this roll out soon. It’s not a rebrand or a 180-degree shift. Instead, it’s small, yet fundamental, nuances that will start to bring some consistency to SAFE marketing materials. So things like YouTube bookends and Medium headers, as well as social images have been considered. Whilst we’re not going to retrospectively apply all these, you’ll start to see these changes in some of the newer content we’ll be pushing out.

SAFE CLI

Project plan

This last week we made really good progress in finalising the integration of the CLI with the safe_client_libs API. Even though that may sound trivial, this put the CLI project in an excellent position as this shows the features we’ve been developing so far can effectively work with the SAFE Network and the new data types as originally envisioned. Today we merged the last PR we had for the integration effort. The integration of CLI with SCL also means we can now use all the available commands we have in CLI with the mock-network feature, and using our safe_auth CLI for authorising the CLI application by using the $ safe auth command.

We have already started and also made good progress on the first PR (#160) in relation to the NRS features, the other main goal of the current milestone, which is focused on allowing users to create public names and link websites to them using the CLI. We are now full steam ahead with finalising this PR as well as the rest of the planned NRS features, like not only being able to create public names and NRS containers, but also add, remove and update the linked websites/files from there, or fetching them and rendering their content using NRS-URLs with paths.

Lastly, we had to make some minor changes to the CLI UX making some args optional as they are not really required based on how the real SCL API work with Keys and coin balances. For example, when querying the balance of a Key we only need the secret key and therefore providing the Key’s XOR-URL is now totally optional. We will be also working on making the necessary updates to the CLI User Guide.

SAFE Browser

In the last weeks we’ve been reducing time spent on the desktop SAFE Browser for now, with most staff resources being assigned to other projects like the “SAFE Network App” and CLI development.

We’ll continue to offer patch releases as make sense, and so, just as a wee reminder for any who may have missed it, last week we made available our latest Browser patch release v0.14.1.

Mock interface

Project plan

It’s been yet another intense week for the SAFE Client Libs team. The experimental work on the Safecoin API was merged into the stable branch and handed over to the SAFE CLI team for integration. With this, the client API is mostly stabilized and all remaining changes are mostly internal to Client Libs.

Speaking of internals, we’ve updates the MDataInfo struct to use the new Mutable Data types. This enabled the team to migrate internal components such as access containers, the root config directory etc. to the new data types. Additionally, the deduction of PUTs has been replaced with Safecoin debits for mutation operations. With this, we will soon be ready to completely deprecate the legacy data types and their API for a smooth transition to Vault integration.

Along with the usual improvements and bug-fixes, an additional change has been made where apps cannot send requests for client-specific operations such as manipulating authenticated keys or deleting unpublished data. Previously, this was prevented only at the vaults. Now, the API doesn’t give users the feasibility to send such requests and even if they do (by changing the always open-sourced code) vaults will just drop the requests.

The team also spent some time planning the integration of quic-p2p in SAFE Client Libs. We have an initial plan in place and implementation will be starting soon once the team wraps up a few tasks in the currently active milestone.

Vaults

Project plan

Following last week’s completion of Immutable Data handling, we’ve turned our attention to Mutable and Append-Only Data. Both are now done, or nearly done, with the final PRs in the pipeline. At this point, it’s mostly about making sure we catch all corner cases in the tests. During the implementation of these data types, some improvements flowed back tosafe-nd to make sure the API is ergonomic to use.

Finally, handling of authorisation keys for apps is also implemented and with this we are very close to completing phase 1 :tada:, and soon we can test the implementation against SAFE Client Libs.

Secure Message Delivery

Project plan

A few initial issues in the SMD project are now addressed in an open PR, which will get merged soon. This opens the road to more concurrent work on this project.

The PR also contains a small “BLS emulation” layer. Our initial plan for SMD depended on BLS being in place, but it’s not complete yet and we don’t want to rush it. This approach enables us to stick to the plan as much as possible, and when we actually integrate BLS, we will be able to replace the BLS emulation with the actual threshold_crypto crate.

We also completed some additional cleanups. The main one simplifying our routing logic to keep the message signing group the same: always all the elders.

Integration of quic-p2p into Routing

Project plan

Quic-p2p has caught up with the mock-quic-p2p changes we made to accommodate Routing’s needs. This lets us update the dependency in Routing and restore CI builds of production code that were temporarily suspended. With this, all the issues in the project are closed and the integration of quic-p2p into Routing is complete :tada:

BLS cryptography

Project plan

Work continues on implementing our DKG (Distributed Key Generation, the way our peers generate their secret and public keys).

We agreed on the plan for the feature mentioned last week: allowing not-yet-full members of a section to participate in the DKG. We updated the project plan, and started with the implementation. The first part is in a PR raised and awaiting a final review. Once the related work is complete, we will have what is needed to start integration of the DKG process into the membership changes flows in Routing.

76 Likes

First! Teeheheeeeeee

23 Likes

Second I reckon! Yep :grin: and here’s my own mini update from yesterday:

Dev Update Comments

Lots of good news, so much progress. :smile:

Is this still a separate CLI mock, or is it now using the same /tmp/MockVault as the browser? If not, is that the intention at some point?

One of the great things about the CLI is going to be setting things up which can then be tested in the Browser. I was doing this today using SAFE Drive, but obviously it doesn’t support new data types and other features, so I look forward to using the CLI for this.

31 Likes

First! Twenty Charachters.
Edit: Third.

14 Likes

That’s one big step for a week, towards one giant leap for mankind.

Edit: Fourth? I need to move across the street of 6 Forbes Drive, Ayr with fiber optic Internet too to beat the other guys, lol!

19 Likes

Really excited to hear this!! Sounds like a great way to get a broader, more distributed set of devs involved with SAFE. It’ll pay off massive dividends in growing the ecosystem and facilitating mass adoption. :smile:

19 Likes

Very soon it will use a single real vault, no mock (needed, but we will still have one) . The vault will be a phase I vault, so not connect to others, but allow many clients to connect to it. It’s coming

So you see. We have SCL which apps use. That speaks to a single vault (at launch many vaults (it’s client handlers or closest Elders if you prefer). So we connect SCL to the vault phase I and that is a network. SCL has a ton of tests, so we use them to test all the vault functions (apart from farming). So you have mini networks you can use for local testing, even hosting for others. It is like a client(s) → server model.

Then we add in routing, so decentralise the vault. That allows farming etc. That’s Fleming.

Then we add some more (anti spam etc. not much though) security and upgrade capability, that’s Maxwell and we are in Beta.

The long road got a lot shorter very fast and with this process testing is much cleaner as we don’t introduce the complexity of routing up front, we test SCL and vaults first then test routing decentralised the vault.

26 Likes

Gonna be super important to decentralise the core team/devs.

20 Likes

It’s now the same mock vault as the browser :slight_smile: With just one change, the mock vault file is now called /tmp/SCL-Mock. This is because we wanted developers to try out the CLI and the new data types without affecting any of your brilliant on-going projects using the alpha-2 code :smile:

12 Likes

Thanks David and Lionel. I’m looking forward to being able to use the CLI and Browser with the same vault.

Will this switch effectively disable large parts of the old API when using the new data types, or will old code still run, but creating old data types?

I’m wondering about the migration process for code I have working. Is it effectively ‘ditch and start again’ or will much of the API be backward compatible, allowing existing apps to run without migration to the new APIs?

If old apps will still run, will the data they create be accessible to both APIs?

7 Likes

Great news! Great update. Now Beta feels closer than ever, as all the pieces fall into place. That’s what consistent years of integrity in planning, inventing and development help to accomplish. Cheers!

14 Likes

This is the plan.

The guys are working hard to allow old and new, I am much more in favour of migration to new only though. I will let them answer, but I hope not to scrap and start again but adapt to new API where the calls will be slightly different (there is safecoin). I will let the guys chime in here, but I would like to get everyone to the new API as soon as we can, although we are also looking (you will see in the CLI) at having a high level API that is even easier to use. In whatever case though we will make migrations as easy as possible as we don’t want to lose apps.

14 Likes

I’m not too affected either way, and all my code is built on SafenetworkJS, so in theory I’ll only have to migrate that.

I think there’s a benefit in taking old apps across to Fleming as is, but if that isn’t supported I’m OK with that.

11 Likes

It is a goal for sure. Even if it may mean we help migrate on some occasions. We will try to make it painless though. I am excited to see what folk (hint hint) do with RDF now :slight_smile:

13 Likes

In Client Libs the new API is very similar to the older API. Even though the older API will be fully deprecated, migrating to the new API will be very straight forward.

The MutableData type still remains as unpublished sequenced MutableData and ImmutableData is now Published ImmutableData.

I see the new data types as additions rather than replacements :slight_smile:
So what’s new is,

  • Unpublished ImmutableData
  • Unsequenced unpublished Mutable Data
  • (Published / Unpublished) (Sequenced / Unsequenced) AppendOnly Data

@happybeing I believe that your code will be using the safe_app_nodejs package. Right now, I’m not sure what the plan with those API, but, as David mentioned we are planning a uniform high-level API in Rust so all languages can have similar public API.

12 Likes

Thanks again Lionel. What is sequenced and unsequenced?

I made a table which I hope summarises the above:

New%20Data%20Types%20table%2020-47-26

5 Likes

Had to look it up …

6 Likes

Thanks @lionel.faber very helpful

Sequenced means updates are indexed (sequenced).

10 Likes

And is published == public, or is it more subtle than that?

Can you define what unpublished/published means?

I can try to look it up, but i think it’s worth bringing into this thread for completeness.

6 Likes

I absolutely hope that you focus 100% on the new and don’t waste time on keeping it compatible for anyone having worked with old API. How many are there? Me, Nigel and Mark? The rest are dormant. It’s not worth it. Just make your life’s easy and do what works best for you to get the new stuff as good as it can be. We will adapt. And we are just a few, there are thousands of people, who don’t yet know it, who waits to start using the new stuff.


The update:

So much is happening that my head is spinning. I want to take part of it all, but I can’t! We are getting to MVP, beta and release with big steps though. :heart_eyes:

33 Likes