[Status: offline] SAFE Network Testnet - Vaults from home with IGD - Iteration 2

Development Releases
83

This did the trick, Maidsafe just spoiled us with Snapp, I use to log into the CLI without even thinking about it. :sweat_smile:

5 Likes
84

Gives us a target to confirm now and that is any growable arrays etc. in code that must not grow indefinitely. As we get parsec out this should be much simpler to confirm, but we can start on vaults right now. Nice one folks

13 Likes
85

Thatā€™s strange. I am feeling that I have already experienced this situation. This happens to me from time to time.

5 Likes
86

Is this parsec problem?

1 Like
87

No I donā€™t think so but parsec is a mem leak for sure. We have it contained but still it is a lump of mem we could use. It is much more likely a small issue we have in parts of the code that cache or hold data for a period or list of values. Both are wrong but you can make them more right by using duration and list length, but we should not have unbounded mem containers at all where possible. There is a design pattern that stops this and we recently and currently are looking to cement that. Basically nodes should use very little ram and data to be held has to be on disk where possible. So a little housekeeping for us, but the great news is we got pretty much all functions and features working here. So all in all this is great as the functional design seems right on track, just some practical data handling tidy up and thatā€™s not that hard at all and generally will mean even less code and much less us humans imagining we know how long or how many things we can keep in memory, the key is as close to zero as possible works, anything further than that fails. So much more message passing rather then do everything nodes is the key.

Itā€™s just decentralised networking and getting us away from code that works on servers with ulimited ram and cpu (clusters). Nodes must be small, resource light and with extreme efficiency. All in all a great place to be in for us all.

15 Likes
88

DĆ©jĆ  vu for sure! :smile:

5 Likes
89

Can anyone confirm @Southsideā€™s whereabouts at the time.

17 Likes
90

:rofl::rofl::rofl::rofl::rofl::rofl:

2 Likes
91

UPDATE:

We have been looking into the issue and trying to identify the cause. From our observations, it doesnā€™t look like there was a sudden spike in memory that caused the OOM error. So it is most likely that certain components of the testnet is using more memory than it should. To help us identify this we will be doing memory profiling on the Vault processes and we would like you all to join network once again. But the goal this time is to take it down. :smile:

Instructions to join are exactly the same (see OP). Do reach out if you have any questions.

Thanks in advance!

13 Likes
92

Maybe start a new thread to make sure to get the attention?

(Because I would like to continue joking about @Southside and not drown this important message with that :wink: )

6 Likes
93

Thatā€™s great. We get to wreck another network :blush:.
Maybe take the [status: offline] away so ppl are aware

9 Likes
94

Iā€™m sure he has that planned.

2 Likes
95

My wife. Bed

However the wee batch file I ran around 3AM that sent 1000 10kb files to the network should have finished long before 6.

And anyway, if it got broken, far better we know about it now.

7 Likes
96

Testing autonomous networks is like testing race cars.

Thrash the absolute bejasus out of it and see what breaks.
Change ONE thing
Repeat until it lasts the whole race.

Donā€™t forget to make it stronger and add lightness.

12 Likes
97

Is there anything you would like me definitely NOT to do?

Cos otherwise Iā€™ll just regard this as a challenge :slight_smile:

How about I repeat the 10kb x 1000 and see if it was that that broke it? Which would be pretty poor for someting we want to put into production ā€¦ However I am conscious that there is finite capacity on the droplets - and that they cost moneyā€¦

3 Likes
98

That shouldnā€™t be a problem @Southside
The droplets we created are of fixed size. And I highly doubt that disk size is the cause of the issue. Out of memory issues occour when the RAM utilization is very high for a process.

Nevertheless, the droplets we have deployed are of fixed size and so no dynamic pricing. So no worries there as well. Take it away!

8 Likes
99

Just a confirmation of the issue I have had all along with the Fleming testnets. This is on Ubuntu 19.04 connecting to the internet via a mobile hotspot.

The vault is working fine and has been accepted by the section, but the SAFE CLI canā€™t create an account:

safe auth create-acc --test-coins
[2020-07-17T14:11:56Z DEBUG safe] Starting SAFE CLI...
[2020-07-17T14:11:56Z DEBUG safe::cli] Processing command: CmdArgs { cmd: Some(Auth { cmd: Some(Create { config_file_str: None, sk: None, test_coins: true }) }), output_fmt: None, output_json: false, dry: false, xorurl_base: None, endpoint: None }
[2020-07-17T14:11:56Z DEBUG safe_api::api::authd_client::authd_client_api] Creating new authd client for endpoint https://localhost:33000
Passphrase: 
Password: 
Creating a SafeKey with test-coins...
[2020-07-17T14:11:58Z INFO  safe_api::api::app::safe_client] Creating test SafeKey with 1000.110000000 test coins
[2020-07-17T14:11:58Z TRACE safe_core::client] Create test balance of 1000.110000000 for FullId { keypair: Keypair::Bls(..), public_id: Client(PublicKey::Bls(997ba6..)) }
[2020-07-17T14:11:58Z TRACE safe_core::config_handler] Not available: /home/mrh/.config/safe_core/safe_core.config
[2020-07-17T14:11:58Z TRACE safe_core::config_handler] Reading: /home/mrh/.config/safe_vault/vault_connection_info.config
[2020-07-17T14:11:58Z TRACE safe_core::connection_manager] Trying to bootstrap with group Client(PublicKey::Bls(997ba6..))
[2020-07-17T14:11:58Z TRACE safe_core::connection_manager::connection_group] Bootstrapped; SocketAddr: V4(157.245.43.31:12000)
[2020-07-17T14:11:58Z TRACE safe_core::connection_manager::connection_group] 0: Sent user message
[2020-07-17T14:11:58Z TRACE safe_core::connection_manager::connection_group] HandshakeResponse::Join, transitioning to Joining state ([(4ffdd7.., V4(64.227.45.157:12000)), (52bfaa.., V4(157.245.43.31:12000)), (5da285.., V4(165.22.115.111:12000)), (613b3f.., V4(206.189.27.255:12000)), (85fca6.., V4(165.22.118.218:12000)), (ad28a9.., V4(161.35.44.247:12000)), (b65dd2.., V4(178.62.118.12:12000))])
[2020-07-17T14:11:58Z TRACE safe_core::connection_manager::connection_group] 0: Recvd connection failure for 157.245.43.31:12000, Connection cancelled
[2020-07-17T14:11:59Z TRACE safe_core::connection_manager::connection_group] 0: Sent user message
[2020-07-17T14:11:59Z TRACE safe_core::connection_manager::connection_group] 0: Sent user message
[2020-07-17T14:11:59Z TRACE safe_core::connection_manager::connection_group] 0: Sent user message
[2020-07-17T14:11:59Z TRACE safe_core::connection_manager::connection_group] 0: Sent user message
[2020-07-17T14:11:59Z TRACE safe_core::connection_manager::connection_group] 0: Sent user message
[2020-07-17T14:11:59Z TRACE safe_core::connection_manager::connection_group] 0: Sent user message
[2020-07-17T14:11:59Z TRACE safe_core::connection_manager::connection_group] Got the challenge from V4(206.189.27.255:12000)
[2020-07-17T14:11:59Z TRACE safe_core::connection_manager::connection_group] 0: Sent user message
[2020-07-17T14:11:59Z TRACE safe_core::connection_manager::connection_group] Got the challenge from V4(161.35.44.247:12000)
[2020-07-17T14:11:59Z TRACE safe_core::connection_manager::connection_group] 0: Sent user message
[2020-07-17T14:11:59Z TRACE safe_core::connection_manager::connection_group] 0: Sent user message
[2020-07-17T14:11:59Z TRACE safe_core::connection_manager::connection_group] Got the challenge from V4(64.227.45.157:12000)
[2020-07-17T14:11:59Z TRACE safe_core::connection_manager::connection_group] 0: Sent user message
[2020-07-17T14:12:28Z TRACE safe_core::client] Error connecting to network! Retrying... (1)
[2020-07-17T14:12:28Z TRACE safe_core::connection_manager] Dropped ConnectionManager - terminating gracefully
[2020-07-17T14:12:28Z TRACE safe_core::connection_manager::connection_group] Gracefully terminating quic-p2p event loop
[2020-07-17T14:12:28Z TRACE safe_core::connection_manager] Trying to bootstrap with group Client(PublicKey::Bls(997ba6..))
[2020-07-17T14:12:28Z TRACE safe_core::connection_manager::connection_group] Bootstrapped; SocketAddr: V4(157.245.43.31:12000)
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] 1: Sent user message
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] HandshakeResponse::Join, transitioning to Joining state ([(4ffdd7.., V4(64.227.45.157:12000)), (52bfaa.., V4(157.245.43.31:12000)), (5da285.., V4(165.22.115.111:12000)), (613b3f.., V4(206.189.27.255:12000)), (85fca6.., V4(165.22.118.218:12000)), (ad28a9.., V4(161.35.44.247:12000)), (b65dd2.., V4(178.62.118.12:12000))])
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] 1: Recvd connection failure for 157.245.43.31:12000, Connection cancelled
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] 1: Sent user message
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] 1: Sent user message
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] 1: Sent user message
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] 1: Sent user message
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] 1: Sent user message
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] 1: Sent user message
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] Got the challenge from V4(161.35.44.247:12000)
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] Got the challenge from V4(206.189.27.255:12000)
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] Got the challenge from V4(165.22.115.111:12000)
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] 1: Sent user message
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] 1: Sent user message
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] 1: Sent user message
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] 1: Sent user message
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] Got the challenge from V4(178.62.118.12:12000)
[2020-07-17T14:12:29Z TRACE safe_core::connection_manager::connection_group] 1: Sent user message
[2020-07-17T14:12:58Z TRACE safe_core::client] Error connecting to network! Retrying... (2)
[2020-07-17T14:12:58Z TRACE safe_core::connection_manager] Dropped ConnectionManager - terminating gracefully
[2020-07-17T14:12:58Z TRACE safe_core::connection_manager::connection_group] Gracefully terminating quic-p2p event loop
[2020-07-17T14:12:58Z TRACE safe_core::connection_manager] Trying to bootstrap with group Client(PublicKey::Bls(997ba6..))
[2020-07-17T14:12:58Z TRACE safe_core::connection_manager::connection_group] Bootstrapped; SocketAddr: V4(157.245.43.31:12000)
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] 2: Sent user message
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] HandshakeResponse::Join, transitioning to Joining state ([(4ffdd7.., V4(64.227.45.157:12000)), (52bfaa.., V4(157.245.43.31:12000)), (5da285.., V4(165.22.115.111:12000)), (613b3f.., V4(206.189.27.255:12000)), (85fca6.., V4(165.22.118.218:12000)), (ad28a9.., V4(161.35.44.247:12000)), (b65dd2.., V4(178.62.118.12:12000))])
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] 2: Recvd connection failure for 157.245.43.31:12000, Connection cancelled
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] 2: Sent user message
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] 2: Sent user message
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] Got the challenge from V4(206.189.27.255:12000)
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] 2: Sent user message
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] 2: Sent user message
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] 2: Sent user message
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] Got the challenge from V4(161.35.44.247:12000)
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] 2: Sent user message
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] Got the challenge from V4(178.62.118.12:12000)
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] Got the challenge from V4(165.22.115.111:12000)
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] 2: Sent user message
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] 2: Sent user message
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] 2: Sent user message
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] 2: Sent user message
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] 2: Sent user message
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] Got the challenge from V4(157.245.43.31:12000)
[2020-07-17T14:12:59Z TRACE safe_core::connection_manager::connection_group] 2: Sent user message
[2020-07-17T14:13:28Z TRACE safe_core::connection_manager] Dropped ConnectionManager - terminating gracefully
[2020-07-17T14:13:28Z TRACE safe_core::connection_manager::connection_group] Gracefully terminating quic-p2p event loop
[2020-07-17T14:13:28Z TRACE safe_api::api::authd_client::authd_client_api] SafeAuthdClient instance being dropped...
[2020-07-17T14:13:28Z ERROR safe] safe-cli error: [Error] NetDataError - Failed to allocate test coins: Unexpected: Connection timed out when bootstrapping to the network - CoreError::Unexpected::{"Connection timed out when bootstrapping to the network"}
4 Likes
100

Probably obvious to others but I did not do this when starting my new vault.
too verbose, you donā€™t lie! :woozy_face:

2 Likes
101

Bummer I went so quick through the setup that I was totally blind for that one :flushed:

2 Likes
102

:thinking:

Network gave up on the back of Zokiā€™s scriptā€¦

5 Likes