Update 20 April, 2023

Development Updates
55

Very exciting to see all these new changes at play and being able to upload files again!

I forced nodes to go offline systematically by altering the duration in between a forceful safenode pid termination starting from 15 seconds → 30 seconds → 60 seconds → 120 seconds → 240 seconds, and observe what impact would be there on the safe files upload command (early termination?), and at what safenode total pid count levels would be at when an error was encountered.

Successfully stored file to ChunkAddress(d1b43c(11010001)..)
Storing file "ac66da72-4046-4933-8432-ee52908f8d5f.txt" of 10485760 bytes..
Successfully stored file to ChunkAddress(721174(01110010)..)
Storing file "2fdf7ca4-a0b4-46a7-bac1-0a91f534f4f2.txt" of 10485760 bytes..
Successfully stored file to ChunkAddress(32e7be(00110010)..)
Storing file "82a9793b-e2e3-429c-ad14-7b4d10b4ade3.txt" of 10485760 bytes..
...

04/22/2023 04:28:24 - Initial # of Safe Nodes: 35 - 15 Seconds:
04/22/2023 04:28:39 - Total Safe Node PIDs: 35;   SAFE PID Killed: 16339 : SAFE CLI RUNNING
04/22/2023 04:28:54 - Total Safe Node PIDs: 34;   SAFE PID Killed: 16346 : SAFE CLI RUNNING
04/22/2023 04:29:09 - Total Safe Node PIDs: 33;   SAFE PID Killed: 16353 : SAFE CLI RUNNING
04/22/2023 04:29:24 - Total Safe Node PIDs: 32;   SAFE PID Killed: 16360 : SAFE CLI TERMINATED

04/22/2023 04:23:39 - Initial # of Safe Nodes: 35 - 30 Seconds:
04/22/2023 04:24:09 - Total Safe Node PIDs: 35;   SAFE PID Killed: 16033 : SAFE CLI RUNNING
04/22/2023 04:24:39 - Total Safe Node PIDs: 34;   SAFE PID Killed: 16040 : SAFE CLI RUNNING
04/22/2023 04:25:09 - Total Safe Node PIDs: 33;   SAFE PID Killed: 16050 : SAFE CLI RUNNING
04/22/2023 04:25:39 - Total Safe Node PIDs: 32;   SAFE PID Killed: 16057 : SAFE CLI TERMINATED

04/22/2023 04:03:07 - Initial # of Safe Nodes: 35 - 60 Seconds:
04/22/2023 04:04:07 - Total Safe Node PIDs: 35;   SAFE PID Killed: 15361 : SAFE CLI RUNNING
04/22/2023 04:05:07 - Total Safe Node PIDs: 34;   SAFE PID Killed: 15368 : SAFE CLI RUNNING
04/22/2023 04:06:07 - Total Safe Node PIDs: 33;   SAFE PID Killed: 15375 : SAFE CLI RUNNING
04/22/2023 04:07:07 - Total Safe Node PIDs: 32;   SAFE PID Killed: 15382 : SAFE CLI RUNNING
04/22/2023 04:08:07 - Total Safe Node PIDs: 31;   SAFE PID Killed: 15390 : SAFE CLI TERMINATED

04/22/2023 04:09:53 - Initial # of Safe Nodes: 35 - 120 Seconds:
04/22/2023 04:11:53 - Total Safe Node PIDs: 35;   SAFE PID Killed: 15712 : SAFE CLI RUNNING
04/22/2023 04:13:53 - Total Safe Node PIDs: 34;   SAFE PID Killed: 15719 : SAFE CLI RUNNING
04/22/2023 04:15:53 - Total Safe Node PIDs: 33;   SAFE PID Killed: 15726 : SAFE CLI RUNNING
04/22/2023 04:17:53 - Total Safe Node PIDs: 32;   SAFE PID Killed: 15733 : SAFE CLI RUNNING
04/22/2023 04:19:53 - Total Safe Node PIDs: 31;   SAFE PID Killed: 15740 : SAFE CLI RUNNING
04/22/2023 04:21:53 - Total Safe Node PIDs: 30;   SAFE PID Killed: 15748 : SAFE CLI TERMINATED

04/22/2023 04:47:21 - Initial # of Safe Nodes: 35 - 240 Seconds:
04/22/2023 04:51:21 - Total Safe Node PIDs: 35;   SAFE PID Killed: 16955 : SAFE CLI RUNNING
04/22/2023 04:55:21 - Total Safe Node PIDs: 34;   SAFE PID Killed: 16962 : SAFE CLI RUNNING
04/22/2023 04:59:21 - Total Safe Node PIDs: 33;   SAFE PID Killed: 16969 : SAFE CLI RUNNING
04/22/2023 05:03:21 - Total Safe Node PIDs: 32;   SAFE PID Killed: 16976 : SAFE CLI RUNNING
04/22/2023 05:07:22 - Total Safe Node PIDs: 31;   SAFE PID Killed: 16983 : SAFE CLI TERMINATED

safe files upload terminated with the following in all the runs above (same stack trace outside of the guid.txt):

thread 'main' panicked at 'Did not store file "f588525d-d6a6-46b6-adcb-d75456ca289d.txt" to all nodes in the close group! Network Error Outbound Error.', safenode/src/bin/cli/files.rs:66:21
stack backtrace:
   0: rust_begin_unwind
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:579:5
   1: core::panicking::panic_fmt
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/panicking.rs:64:14
   2: safe::main::{{closure}}
   3: tokio::runtime::park::CachedParkThread::block_on
   4: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
   5: tokio::runtime::runtime::Runtime::block_on
   6: safe::main

I didn’t carry out the tests dozens off times repeatedly and maybe the testing itself was flawed given all the changes being discussed lately with the cleaner and simpler network design, but overall the CPU, memory, and disk space were not maxed out on my local VM.

I am not sure what conclusions to draw from above, I was simply curious to see how the network would behave if random nodes were dropped either fast or slowly from a live local network… prior to the JoshNet testnet. I was hoping to see the safe files upload continuing to chug along as long as it can given the minimum network size and/or overall health criteria of the network was met. It seems if the duration between random nodes being killed was increased, the safe files upload continued to operate for a longer period, where as if the nodes were killed off faster, the safe files upload error’ed out earlier.

This all might be expected at this stage of development with a 35 node local testnet.

Overall, great weekly update team! Awesome to read all these latest changes, :smile: .

17 Likes