At this point the biggest problem is that there are just too few vaults and I bet they are geographically localised in just a few areas of the world.
In future the closest vault in XOR space would be closer than we are seeing with only a hundred or so vaults.
I am also wondering about the method of downloading/copying that is being done on your system?
Maybe we need to streamline the copy/download process where “prefetch” is done so that the chunks are already on the way to you while the earlier ones are being got.
Streaming in SAFE is more parallel because most video/audio streamers do prefetch already.
Could you try playing a video clip with VLC that is >20Mb and see how well it plays after it starts.
This is a question on what performance of safe network would be, I wonder if decentralization of data of 1 mega bytes chunks and having only a few copies would slow the speeds of downloads.
I would think 10-50 megabyte chunks with the growing capacity with having up to 10 copies the network.
I would like to know adjusting theses levels what would be the effect on the network?
I would think it could be an option as 1 MB chunk for security safe data and 50MB for public data.
maidsafe could create a high speed coin within the network as a cloud service where revenue could be shared with the participates. one could exchange 100 safe coins for 0ne 100 MB coin which is stored on a 100 vaults. or something that makes mathematical sense where vaults are uploading a file at full capacity.
Sorry bad wording I did actually mean distance as in traversing all the hops. So you are right.
Also I was not referring to caching at all. I think that is still turned off.
The point I was making is that with video streaming the player requests the first x blocks of data at once and buffers, then as one block finishes playing it requests the next block to keep the buffer full. So it night request block 1-100 then when all there it starts playing, when block 1 finishes it requests block 101. (blocks != chunks)
So video streaming may currently work better than file copying.
Why?
When you request multiple chunks at once all of them are being retrieved from different vaults (with different paths) so you are not bottlenecked with all going up the one uplink. This is why a player like VLC will play a vid better then a simple copy of the vid to your HD because the copy is requesting one chunk at a time and the resultant lag for each chunk is sequential. Whereas the buffered stream the lag is only for the first chunk then the others overlap.
So when copying a file from SAFE to your computer we may need one that requests multiple chunks at a time to cause the lag to be overlapped.
No VLC simply requests data ahead of time and buffers. So then lags are hidden and it requests multiple blocks (video blocks) at once so fill the buffer quicker. How many video blocks per chunk I would not know, that is why I would be interested to know how well it works, or not work.
I just downloaded some files (up to 25mb.) as well and my download went up to 200 KB/sec. and next it went down to like 50 and next it went up again. so yes, it took some time but it did work. I’m not concerned though, some other downloads/uploads went quite fast. Going over 2 or 4 hops shouldn’t make a big difference. When the chunks are on their way they should arrive over several relay_nodes. We have not reached any limits on what the network can handle, these are just issues that need to be solved. They probably didn’t have any of these problems over the droplet servers. This is a real P2P-network. Will be improved, I have no worries.
One of the things I’m working on and hope to have ready soon is profiling of the vault, to better understand where the performance bottlenecks are. I agree with you, performance is below what I would expect, and I am really keen to dig further and find out where the bottlenecks are.
I think they deserve some real credit. Look at the hole punching thing. It took a lot of time and energy of them to get that fixed. But I see no user here claiming they can’t connect, so they fixed that one quite well. Same for the Vault running parallel with MIO. CPU usage is extremely low for all users now. Next step was us connecting with some big limitations like 100 PUTS etc. And now we’re at the next step, and that’s of course a place where we see some trouble. The devs are testing this testnet like mad themselves, so they see all the things you pointed out as well. But like I said, they’ve shown over and over again that they fix the problems that show up.
At the moment the network messages are very inefficient and not optimised at all. This is both in size of messages and also number that are sent. Another issue is all nodes are treated equally, so a single bad node in a group will slow the whole group to almost that nodes speed.
These are all related to security and data persistence though. The next steps for vaults and routing is security and data resilience (Secure data republishing and validating per group). This is an interesting part as the security looks like it will actually facilitate less messages, or at least more efficient messages. We will know very soon though. We are certainly looking at an ability to handle the slower nodes and allowing the network to work at the speeds of the fastest nodes per group rather than the situation now where many times it’s the slowest node. These are all connected and underway at the moment. Much of the work is coded and being actively audited and debated right now.
After securing the data integrity and persistence the big push in routing etc. will be efficiency. That will be finalising the wire format. This part will show us what the actual speed will be and where it needs to be very fast. For me upload is OK to take a while, not from the client though, in the network. So the client can upload fast and the network can take a little longer to situate and securely store the data (eventual consistency). Get requests need to be very fast and almost by default Post will be to (as you are personally proving ownership).
Hopefully as the routing team are doing that the additional features will happen in parallel (nfs API, messaging safecoin etc.).
That’s a short precis of the current work that is happening. Again increasing the team is essential to aid the speed of delivery and amount of testing etc. All the feedback certainly helps us progress faster as well though as we can focus on what’s important right now.
The GROUP_SIZE, currently used by the tests, is it 8? Will this be eventually 24? What is the estimated impact on speed of increasing this group size? This is probably a problem for later, I’m just curious.
This is an aspect we are looking at as part of the overall security. group size of routing group nodes may not relate so much to copies of data as is now. So separation of concerns between routing groups and data groups (Again). This separation may allow larger groups, but this depends on the security requirements of those. Bare groups consensus without any additional security steps shows that approx size of 19 (IIRC) gives 50% resistance and 29 this figure goes us to 80% resistance (to getting a close group, hops require a similar security and that is covered by the current work in routing table where groups are again based on common leading bits in a more tree like single group per data bucket), but at a cost. Atm that cost would be too great as data is even across the group, when this changes the cost may be much simpler to pay.
Thank you for the response. I obviously meant to say 32 instead of 24, what has been mentioned in the past as the eventual closed group size.
I guess I’ve to look at it more thorougly.
I’m probably understanding it wrong, but am I a correct that the closed groups described in Introduction & Technical Overview of SAFE Consensus | by MaidSafe | safenetwork | Medium are all ‘routing groups’? Concerned about e.g. guarding a unique datachunk.
And that ‘data groups’ are/will be more about how much copies of those unique datachunks there will be?
There are no assurances. You’re not taking into account how small the amount of current users are. It’s likely under a hundred for the test networks. Even if it’s closer to the amount of people on the forum, thats still only around 2000. That’s not really enough for the network, it will need even more. Most consumer internet upload speeds are not that great.
However, don’t forget that there will probably be people who are willing to provide high speed large size vaults to the network. People rent seedboxes, I’m sure they’d rent SAFE vaults, or just host their own.
Once the network goes from being simply a test network to the real deal, you’re gonna see a lot better performance, imo.
I was satisfied with 20kbs for torrents for a while, now it is standard to get 15-20Mbps with bittorent. Even though that network is totally different, I see things evolving very quickly for SAFE.
I think we have to remember that there are very good use cases even if safe net remains at the current speed. Compare to blockchains, this thing is greased lightning!
