Update 09 November, 2023

For the next testnet we hope to put to bed the memory issues that sank the last one, so that’s been our focus this week.

The pay-one-node branch is looking good, and with a bit more testing will be ready to merge. It should reduce memory usage compared to the previous approach of paying all nodes that store a chunk. As we mentioned last week, ultimately we should be able to offer upload speed and assurance trade-off options in the client.

Paying just one node also simplifies the royalty payments. Compensating five nodes means five separate royalty messages being broadcast for the same chunk, which then need to be deduplicated by the receiving nodes. This is complicated by the fact that these CashNoteRedemptions are encrypted. Using one node simplifies this a lot. In addition, these gossiped messages no longer need to be encrypted (with no loss of security) so we’re putting that in now.

The exact impact on royalty payments memory use needs to be tested further on a larger testnet, but it looks like some other simple optimisations are possible too. For example, setting each node to subscribe only to relevant messages, not all messages, has seen a big reduction in mem, and we’re looking to stop unnecessary rebroadcasting too.

On GossipSub itself, we’ve made verification stricter for testing. Rather than just expecting a minimum number of messages now we check for the exact amount.

@southside’s PR to improve logging outputs is now in the codebase. And thanks also to @loziniak for a PR to clone ClientRegister. :palms_up_together:

Now we’re going to hand over to @anselme to walk us through how payments work.

Payments on the Safe Network

This is a summary of how we pay for data on the Safe Network as well as an introduction to money transfers in general.

Transfers

Before we get into data payments, let us first understand how basic transfers work. Everyone on the Safe Network, both clients (wallet apps, Safe CLI, etc) and nodes (data storage) have a public/private keypair. The public key (aka MainPublicKey) is used to receive tokens. It is equivalent to bitcoin or ethereum addresses. The private key (aka MainSecretKey) is used to sign transactions and prove ownership of money.

This is what a MainPublicKey looks like:

93d01b3c6c0d41c4e50c855c753f906ba478f97a838415e6d74615a4b037a7101e724f935727bbf23d17293ab74a3027

You can obtain your own with the command: safe wallet address

Money in a wallet is stored as CashNotes. These are like a cheque for a given value. They contain all the necessary information for the owner to spend the value in them, but don’t have value on their own as they are useless without the private key.

When someone wants to send money, they create a Transaction where they spend their own CashNotes in exchange for new CashNotes that are owned by the recipient. As those CashNotes contain secrets that might affect the privacy of the sender and the recipient (the MainPublicKey), they are never sent or stored on the Network.

Instead, the sender creates very small CashNoteRedemptions that contain the minimum necessary information that the recipient needs to verify the transaction and re-create their own CashNote on their side.

Those CashNoteRedemptions are encrypted and packed into a Transfer which is sent directly to the recipient. The recipient of the Transfer can then decrypt it using their MainSecretKey, verify and rebuild the CashNotes, before adding them to their wallet. At that point, the transfer is complete!

You can send a Transfer with: safe wallet send <amount> <to>

Example: safe wallet send 42 93d01b3c6c0d41c4e50c855c753f906ba478f97a838415e6d74615a4b037a7101e724f935727bbf23d17293ab74a3027

This will print out the encrypted Transfer that can be published or sent directly to the recipient.

They can receive it with: safe wallet receive <transfer>

Data Payments

Data on the Safe Network is paid for using those same Transfers.

Before data is stored on the Network, it is prepared locally by the user: cut into chunks and encrypted. Each of those chunks have a unique hash value from which we derive their NetworkAddress.

Nodes on the Network also have a NetworkAddress. Data is stored on the nodes which have the closest NetworkAddress to the data itself. This is why we say that data is content-addressable.

By querying the Network, a client can get the closest nodes to a NetworkAddress. Once we know them, we first perform a StoreCost query to ask the price for holding data at that location. The price depends on the storage capacity of the nodes, which ultimately depends on demand.

Once we know the StoreCost we can safely send the data along with the Transfer paying for it. Upon receiving the data, nodes verify the Transfer, take the payment, verify the data itself and store it.

Along with that Transfer, another small Transfer is sent: NetworkRoyalties, These are sent to a pool and help keep the Network alive. When receiving data payments, nodes make sure that those NetworkRoyalties are also valid before storing the data.

All of this is done automatically when you upload data using the CLI:

safe files upload some_file

General progress

@chriso has finished work on a promising WinSW service manager extension to help with node management for Windows and submitted a PR for it.

@roland Submitted a PR to resume CLI uploads across multiple runs. This stores chunks locally and moves/removes them once paid/verified. He also refactored the codebase to release RPC client as a binary to fix some testnet deployment errors.

Several team members have been looking into the mechanism of paying royalties to Foundation nodes, as described above, including @bochaco, @bzee and @qi_ma.

@qi_ma also created a PR to avoid duplicated hash work, making the most of the latest changes to libp2p.

@jimcollinson has been looking at what’s needed for an MVP launch, with a focus on on-ramps and off-ramps for a soft launch.

And @joshuef has been drawing up a priority list of the remaining technical work before we enter beta.


Useful Links

Feel free to reply below with links to translations of this dev update and moderators will add them here:

:russia: Russian ; :germany: German ; :spain: Spanish ; :france: French; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!

56 Likes

First now to read the peak of the week

20 Likes

:star_struck: :star_struck:

Great update team as always. While all the efforts this week are fantastic and appreciated, I loved to see this:

I expect we won’t get there this year with time running out, but looking like beta early next year! :tada:

Thanks to all for the continuing hard efforts to get it over the finish line.

Cheers :beers:

24 Likes

I really enjoyed reading the summary of how the payments on the safe network work (current workflow).

It was well written and easy to comprehend!

Thank you team! Looking forward to the next testnet, :clap: .

23 Likes

You mean easy ways for folk to buy/sell SNT?

23 Likes

Thanks so much to the entire Maidsafe team for all of your hard work! :horse_racing:

And to all of the great people helping with the testnets! :horse_racing:

19 Likes

Yes basically that’s it

27 Likes

What happened to high network usage of previous version?
Was it bug? Was it fixed?

2 Likes

Its in the first line and we will see if it got fixed by the performance of the next testnet(s)

Thanks to all for a great update. The explanation of the payments and money transfers was especially welcome - its good to have the basics spelt out in simple language. I will test my understanding of it by trying to create a diagram to visualise the process. THen I will likely cheat and ask ChatGPT to do it for me.
@chriso’s work on running nodes as a service will hopefully simplify much for both Windows and Linux users and should help to widen the pool of testers - especially for Windows users.

Again thanks to everyone involved.

PS PRs need not be large pieces of work, the one I submitted was utterly trivial, hardly worth a mention and could have been reduced to a change in a single line of code, Of course they are not all so simple but if you see something that you think is not right - even just a typo, raise an issue, and if you think you can sort it, fire in a PR.
I can assure you that it will be welcomed and you will get plenty of help and encouragement to navigate the Github process, get it completed and into the codebase.
Don’t hold back, dive in and let your contribution (no matter how trivial) be part of history.

Remember the little bits we as a community do means the big brain devs can get on with the complex stuff. Having said that, hat-tip to @lozniak who looks like adding a very non-trivial piece of work soon.

19 Likes

Thanks for a super update and explanation of the payments process.

I’m wondering why the intermediate CNR rather than sending the CN encrypted?

And is any data lost to the recipient by only sending the CNR?

Also, I’m curious about what the network does during this process and what data related to wallets etc is stored in the network. Is there anywhere on the network that knows the balance of a wallet for example?

13 Likes

One for @Anselme (wrong time zone for now though :wink: )

7 Likes

Thx 4 the update Maidsafe devs

Can’t wait for the next testnet

Good to see the gents that can code making PR
@southside’s @loziniak :clap: :clap: :clap:

That’s why I liked this Github feature

Would be fun to see Copilot help along with Maidsafe’s codebase.

Nice to see payments explained, testing payments feels so second nature, because it’s made so easy.

Keep hacking super ants

6 Likes

What prevents multiple spends of a CashNote?

5 Likes

I was going to ask similar.

I know we’ve had previous methodology to send SNT and prevent double spend.

But the sentence that says the CN is never stored or sent on the network raised the immediate question, how is network/nodes to know this is not SNT that has also been sent to 1000 other nodes

I do like the idea though that CN are never actually sent.

@Anselme @joshuef (sorry if I tagged wrong people)

10 Likes

All those are excellent questions and I was hoping someone would ask!

A CashNote (CN) is quite big (kilobytes) and its size is variable. Depending on the size of the transaction it was created in, can become HUGE (lots of kilobytes, the sky is the limit). On the other hand a CashNoteRedemption (CNR) is 64 bytes. This is the main reason why we send CNR instead of CN.

There is no data lost to the recipient by sending the CNR as all the data that they need can be found by it on the Network in the SignedSpends referred to by the CNR.

The only data stored on the Network for transactions are those SignedSpends. Those are what prevent a user from double spending CashNotes. A SignedSpend is a piece of data addressed at the UniquePubkey of the spent CashNote. It is signed by its owner and verified by both nodes AND recipients of successors of this CashNote (any CN that is an output of the Tx where this CashNote was spent as an input).

When a wallet receives a CNR, it checks on the Network that:

  • all the inputs of the Transaction (that created our CN) where spent (there is a SignedSpend entry for them)
  • all the inputs where spent in the Transaction that created our CN and not another one!
  • that there in ONLY ONE SignedSpend entry for each (else it’s a double spend)
  • that the SignedSpend’s signatures are correct

Currently we don’t store anything wallet related on the Network. There is no way to know the balance of a wallet as the information that links a CN to its owner is secret. On the other hand, by following along all SignedSpends from Genesis, one can verify the supply, know the UniquePubkey and value of every single CashNote that existed. But they cannot link those UniquePubkeys to any wallet’s MainPublicKey.

17 Likes

The owner information about CashNotes is stored on the network? or can be restored, by the owner, based on SignedSpend?

1 Like

If by owner you mean MainPublicKey then no it is not stored on the Network.

It cannot be restored even by the owner from the SignedSpend only. On the other hand, if you have the original spent CashNote, then you can link it to its SignedSpend on the Network and identify its owner with the information stored in the CashNote. This is why we avoid sending CashNotes as they could reveal the owners if leaked!

9 Likes

Is there a good reason for keeping CN instead of only storing CNR locally? What’s the benefit?

2 Likes

Ah so while the CN is not stored there is a subset of information stored on the network.

I see what you did there. You worded it the way you did in the update just to get people to ask. Sneaky LOL

So a follow up question if I may, and if a negative answer then can it be implemented sometime in the future.

That is if I was too eager and generated a CN/CNR for an item that I was going to buy, I was sure I would buy it and generated the CN/CNR right away, then plans changed (out of stock, change of mind) can I reverse the CN/CNR since I never sent it to the seller. If I cannot then is it possible to make it so I can???

Yes i realise there has to be a point where I cannot since bad actors would reverse spends defrauding people.

Maybe it will have to be built into the wallet app such that the CN/CNR is only generated when sending the SNT

2 Likes