[Status: offline] SAFE Network Testnet - Vaults from home with IGD - Iteration 2

[Southside]

Are we up or down?..
ubuntu@ip-nnn-nnn-nnn-nnn:~$ safe auth create-acc --test-coins
[2020-07-17T15:44:11Z DEBUG safe] Starting SAFE CLI…
[2020-07-17T15:44:11Z DEBUG safe::cli] Processing command: CmdArgs { cmd: Some(Auth { cmd: Some(Create { config_file_str: None, sk: None, test_coins: true }) }), output_fmt: None, output_json: false, dry: false, xorurl_base: None, endpoint: None }
[2020-07-17T15:44:11Z DEBUG safe_api::api::authd_client::authd_client_api] Creating new authd client for endpoint https://localhost:33000
Passphrase:
Password:
Creating a SafeKey with test-coins…
[2020-07-17T15:44:26Z INFO safe_api::api::app::safe_client] Creating test SafeKey with 1000.110000000 test coins
[2020-07-17T15:44:26Z TRACE safe_core::client] Create test balance of 1000.110000000 for FullId { keypair: Keypair::Bls(…), public_id: Client(PublicKey::Bls(a7c14f…)) }
[2020-07-17T15:44:26Z TRACE safe_core::config_handler] Not available: /home/ubuntu/.config/safe_core/safe_core.config
[2020-07-17T15:44:26Z TRACE safe_core::config_handler] Reading: /home/ubuntu/.config/safe_vault/vault_connection_info.config
[2020-07-17T15:44:26Z TRACE safe_core::connection_manager] Trying to bootstrap with group Client(PublicKey::Bls(a7c14f…))
[2020-07-17T15:44:56Z TRACE safe_core::client] Error connecting to network! Retrying… (1)
[2020-07-17T15:44:56Z TRACE safe_core::connection_manager] Dropped ConnectionManager - terminating gracefully
[2020-07-17T15:44:56Z TRACE safe_core::connection_manager::connection_group] Gracefully terminating quic-p2p event loop
[2020-07-17T15:44:56Z TRACE safe_core::connection_manager] Trying to bootstrap with group Client(PublicKey::Bls(a7c14f…))
[2020-07-17T15:45:26Z TRACE safe_core::client] Error connecting to network! Retrying… (2)
[2020-07-17T15:45:26Z TRACE safe_core::connection_manager] Dropped ConnectionManager - terminating gracefully
[2020-07-17T15:45:26Z TRACE safe_core::connection_manager::connection_group] Gracefully terminating quic-p2p event loop
[2020-07-17T15:45:26Z TRACE safe_core::connection_manager] Trying to bootstrap with group Client(PublicKey::Bls(a7c14f…))
[2020-07-17T15:45:56Z TRACE safe_core::connection_manager] Dropped ConnectionManager - terminating gracefully
[2020-07-17T15:45:56Z TRACE safe_core::connection_manager::connection_group] Gracefully terminating quic-p2p event loop
[2020-07-17T15:45:56Z TRACE safe_api::api::authd_client::authd_client_api] SafeAuthdClient instance being dropped…
[2020-07-17T15:45:56Z ERROR safe] safe-cli error: [Error] NetDataError - Failed to allocate test coins: Unexpected: Connection timed out when bootstrapping to the network - CoreError::Unexpected::{“Connection timed out when bootstrapping to the network”}

[Josh]

Seemingly down.

[status: offline] SAFE Network Testnet - Vaults from home with IGD - Iteration 2.2

Thanks for all the help guys! And great job I must say. We have been doing quite the bashing and reproduced the OOM issue on some of the nodes. We should have what we need to continue investigating the issue. I’ll leave the remaining nodes running, however with a reduced number of elders there is no certainity that all requests will go through. The logs after nodes fail, will help plan ahead for the next steps. Thanks once again for all your help. Have a good weekend.

[Southside]

Thank you.

[Southside]

Do we have any figures for the no of vaults connected, total uploaded/downloaded etc?

[MaxSan]

isnt that the entire point of a self encrypted chunked file… Am i missing something?

[Josh]

I also noticed that some chunks were fully readable. Which at first did my head in. Then I remembered that public data is not encrypted. (I believe that’s the case)
So it seems the file types of chunks that are not encrypted are somehow still recognized.

That is my take on it, if I am wrong I’d gladly be corrected.

[Nigel]

I thought everything was encrypted but public data everyone can access the data map. I’d prefer that so even if something nasty that was public was on a hard drive no one would be the wiser. Kind of becomes a slippery slope otherwise imo.

[Josh]

@Nigel

I’d prefer that so even if something nasty that was public was on a hard drive no one would be the wiser.

Agreed, I’m totally out of my depth here so I am just guessing. I could see bits of Javascript/HTML that were legible, parts of files… chunks.
Not parts of images or anything like that.
I vaguely remember this discussion in the past so don’t quote me but may have to do with performance.

I’ll post a screenshot in a minute.

[KafkaLee]

I think chunk obfuscation is important foundation of maidsafe. why it is not implemented yet? Its published back in 2010 by @dirvine it may overkill? https://www.google.com/url?sa=t&source=web&rct=j&url=http://docs.maidsafe.net/Whitepapers/pdf/SelfEncryptingData.pdf&ved=2ahUKEwigw4i64tXqAhUNa94KHWNaAV0QFjABegQIBBAB&usg=AOvVaw3WWO3kP0PcIA39SusH6WJx

[Nigel]

I believe data obfuscation is on the roadmap at safenetwork.tech in future work AFAIK.

[digipl]

Not sure if CRDT change something but the idea is that all data greater than 3 KB. will be chunks where self-encryption could be applied regardless which data type they belong to.

[RFC] Data Hierarchy Refinement

Data Hierarchy Refinement Note: Bare in mind, that this is very much a work in progress, and there are inconsistencies as well as unfinished parts of this proposal. We have chosen to share it with the public in an earlier stage of iterations, for earlier exchange of ideas. If the proposal turns out to be desired and accepted, the earliest time of implementation would be post-Fleming. Summary This is a proposal describing a system where all data types are built of chunks, and together with de…

The earliest time of implementation would be post-Fleming

[Sascha]

There is something fishy about the “router problem”. I always use the same phone for tethering. Sometimes it works - sometimes I get this:

sascha@Knut-Mint:~$ tail -f ~/.safe/vault/local-vault/safe_vault.log
INFO 2020-07-18T11:17:20.667798070+03:00 [src/bin/safe_vault.rs:114] 

Running safe-vault v0.25.1
==========================
INFO 2020-07-18T11:17:30.892900120+03:00 [src/vault.rs:135] Initializing new Vault as Infant
ERROR 2020-07-18T11:18:00.893737680+03:00 [src/bin/safe_vault.rs:169] Cannot start vault due to error: Routing(Network(IgdNotSupported))
ERROR 2020-07-18T11:18:00.893765515+03:00 [src/bin/safe_vault.rs:170] Automatic Port forwarding Failed. Check if UPnP is enabled in your router's settings and try again. Note that not all routers are supported in this testnet. Visit https://forum.autonomi.community for more information.

[KafkaLee]

thanks digipl it explained a lot.

[dirvine]

@Sascha

There is something fishy about the “router problem”.

It may be the igd crate has a short timeout. We will check that out @ravinderjangra and @lionel.faber are looking for bugs like this to kill off.

[Traktion]

@Nigel

I thought everything was encrypted but public data everyone can access the data map. I’d prefer that so even if something nasty that was public was on a hard drive no one would be the wiser. Kind of becomes a slippery slope otherwise imo.

IIRC, it was when the content was public and small enough to fit in a single file with the data map, so it doesn’t get broken up and self encrypted. My memory may be deceiving me though.

It seems a bit of a weakness though.

[JPL]

Yes, I think if it’s under 3KB it doesn’t currently get encrypted. I remember adding a bunch of zeros to an HTML file and experimenting with that, way back when.

What can anyone know about a file?

Anything above 3KB and less than 3MB in length is split into 3 chunks using self encryption. Greater than 3MB then it is split into 1MB chunks using self encryption. This means that to have a single chunk for a file then it has to be less than 3KB Now it has been discussed that somebody could self encrypt a file independently of the network and use the chunks generated to know the hash (XOR) address of the chunks and then see if they are there. This would mean in theory someone could find i…

[dirvine]

The idea being a tiny file would be in a dir, the dir is encrypted as a file and that puts it over the 3Kb so it is unwrapped recursively. i.e. you should never have small file datamaps out somewhere on their own, if part of a filesystem it’s cool. The root dir will always be over 3Kb IIRC.

If there is no choice then the data map itself should be encrypted, but I don’t know where we would need to do this as files are for now part of filesystems. It would be a simple thing to encrypt them though with the users keys if necessary.

[neo]

@dirvine

The idea being a tiny file would be in a dir,

That brings up an interesting situation.

I have come across a number of situations where a directory full of small files (like 1KB to 5KB, mostly 1KB to 2KB) and its painfully enough that windows keeps getting slower with each version to just display the file list or work with it. Sometimes thousands of files.

Now if these files are then in the directory file, then that makes for a one large directory file. And what about working with the directory.

Just a thought

[dirvine]

@neo

Now if these files are then in the directory file, then that makes for a one large directory file.

In these cases the DM should be recursively encrypted to produce a small initial DM.

[joshuef]

For those interested in running local testnets, there’s a new browser alpha release available: Release v0.18.0-alpha.4 · maidsafe/sn_browser · GitHub

Remember, it’s still alpha and may be buggy, so if you’re not comfortable with investigating/reporting bugs this may not be for you! (But if you are, in !)