InstallNet [2023-06-19 Testnet] [Offline]

Southside · June 20, 2023, 2:00pm

Apologies I should have been clearer

dependant solely on the first few bytes of the actual hashed chunk content matching the relevant node addresses.

storage_guy · June 20, 2023, 2:02pm

Could be normal behaviour or a hangover from testing when running a local network or a bit of both. Check to see if the file is available locally before going out to the network. Maybe basic functionality is or was tested on something that stores data locally.

digipl · June 20, 2023, 2:06pm

Imac High Sierra 10.13.6

I have this error message:

Successfully stored file “termux-app_v0.118.0+github-debug_arm64-v8a.apk” to 0b6d3d46935b2b1526b186046d0e7d1c378487b972df663e13fa1996cc995be6
Writing 94 bytes to “/Users/imac27/.safe/client/uploaded_files/file_names_2023-06-20_14-03-29”
> Error: Too many open files (os error 24)
**> **
> Location:
> sn_cli/src/subcommands/files.rs:145:5

storage_guy · June 20, 2023, 2:08pm

This is on a AWS Instance (t4g.medium) with 2vCPU and 4GB RAM. So I didn’t do any port forwarding or specify a port with ‘–port’. So the nodes all have a port that was decided on by safenode.

Southside · June 20, 2023, 2:20pm

Sometimes I need to step back, take a breath and restate the fundamentals to myself.
It is often a good idea to check that what I think are the fundamentals are still valid.

==============================================
Meanwhile will somebody please say YES or NO?

SAFE urls are ALWAYS EXACTLY 65 hexadecimals long - no ifs, no buts

storage_guy · June 20, 2023, 2:25pm

I’m wondering if each chunk is spread across a number of nodes according to each chunk’s hash or whether the chunks for a file are spread across a number of nodes according to the hash of the file?

happybeing · June 20, 2023, 2:37pm

It’s hash of chunk.

Shu · June 20, 2023, 2:39pm

Hip hip hooray! Finally got a chunk written!

PUT request for Record key: Key(b"2\x11\xe9\xcf\x9b-\xd5\xdc\xb7\xf1v\x84\xfa\xb9\x0fd\xb8\x18Q\xc1\xaaC[O(H\xe7/\x1b\x8a\x14\x01")
Wrote record to disk! filename: 3211e9cf9b2dd5dcb7f17684fab90f64b81851c1aa435b4f2848e72f1b8a1401

@joshuef - I don’t see any Cmds, Request IDs, or Chunk Addresses in the logs?

Neither on the dashboard nor in the raw log files. I found those metrics useful in assessing the total duration of the workflow on an individual Request Id basis, and the the distribution of those metrics over the lifetime of the safenode pid.

I do see INFO, TRACE, DEBUG log levels etc since the safenode pid was started with SN_LOG=all. Hmm, .

Did something change in the code, or the issue is on my end? Either way, I will circle back on digging into this a bit later in the day today.

Southside · June 20, 2023, 2:40pm

AIUI each node is responsible for storing the chunks whose hash is closest in XOR space to the node address.

I dont know the exact mechanism by which this is repeated 4 or 8 ($redundancy_factor) times.

But I bet someone will be along soon to enlighten me

chriso · June 20, 2023, 2:40pm

Thanks for the feedback. Does this happen on any file upload?

digipl · June 20, 2023, 2:56pm

Only a couple of times.

But on another uploads, also a couple of times, I got this other error even though it just ended with a message of successfully stored file:

thread ‘’ panicked at ‘assertion failed: old_path.exists()’, /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/file-rotate-0.7.4/src/lib.rs:564:9
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

And, sometimes, I get an error when trying to download files:

Downloading file “pruebaur” with address 783a66dbd82835c26aa1fb926b04264d83c427b7bc2e5971988a01b931910e04
Did not get file “pruebaur” from the network! Chunks error Not all chunks were retrieved, expected 3, retrieved 2, missing [2b2908(00101011)…]…
Error: Too many open files (os error 24)

Location:
sn_cli/src/subcommands/files.rs:160:47

happybeing · June 20, 2023, 2:56pm

You might find it useful to look at the code. You can search for the ‘missing’ strings (they may be conditional on build features) or for trace, info, debug to understand what might be useful.

VSCode makes this very convenient.

Shu · June 20, 2023, 2:59pm

Thanks, I may compile a debug version and run it with SN_LOG=all, and see what happens.

I thought SN_LOG=all would be a wildcard * for all logging, but that might be only be a limited subset when compiled in release mode vs debug. Makes sense.

happybeing · June 20, 2023, 5:53pm

I was thinking of the ‘features’ compiler flag. This, for example is used to control the metrics that were added recently, but there may be other things that are built selectively, independently of the ‘debug’ flag.

Shu · June 20, 2023, 6:11pm

I will look into it more in a few hours from now. I don’t remember now if I used a debug or release bin on the prior testnet. My build script didn’t have any --features flag when compiling earlier (assuming I used a debug version).

A quick peek, if I am looking at the right place (hopefully), seems to have the following with metrics coming in as default already:

[features]
default=["metrics"]
local-discovery=["sn_networking/local-discovery"]
otlp = ["sn_logging/otlp"]
metrics = ["sn_logging/process-metrics"]

And, yes, you are right, I will probably have to start searching for those strings via VSCode, . Thanks.

chriso · June 20, 2023, 6:15pm

I’m not sure exactly what it is you’re trying to do here, but regarding the all thing, it’s just an alias to enable logging for all the crates:

github.com

maidsafe/safe_network/blob/ef92331b27616837b9d8b20897b19cc06a229679/sn_logging/src/lib.rs#L226


      
                      .unwrap_or_else(|_| println!("Error initializing logger"));
              });
          }
          
          
fn get_logging_targets(logging_env_value: &str) -> Result<Vec<(String, Level)>> {
              let mut targets = Vec::new();
              let crates = logging_env_value.split(',');
              for c in crates {
                  // TODO: are there other default short-circuits wanted?
                  // Could we have a default set if NOT on a release commit?
                  if c == ALL_SN_LOGS {
                      // short-circuit to get all logs
                      return Ok(vec![
                          ("safenode".to_string(), Level::TRACE),
                          ("safe".to_string(), Level::TRACE),
                          ("sn_build_info".to_string(), Level::TRACE),
                          ("sn_cli".to_string(), Level::TRACE),
                          ("sn_client".to_string(), Level::TRACE),
                          ("sn_logging".to_string(), Level::TRACE),
                          ("sn_networking".to_string(), Level::TRACE),
                          ("sn_node".to_string(), Level::TRACE),

Shu · June 20, 2023, 6:31pm

I was expecting to see messages such as these from earlier testnet:

[2023-06-07T21:56:11.611796Z TRACE sn_networking::msg] Received request with id: RequestId(78778), req: Cmd(StoreChunk { chunk: Chunk { address: ChunkAddress(5250f1(01010010)..) }, payment: None })
[2023-06-07T21:56:11.614907Z TRACE sn_networking::msg] ResponseSent for request_id: RequestId(78778) and peer: PeerId("12D3KooWC6wddzjx5YVWVJqUzkkFCcDSdTJywsXXittUCL7Z9EwR")

Seems its at this location currently:

github.com

maidsafe/safe_network/blob/main/sn_networking/src/msg/mod.rs#L33


      
              event: request_response::Event<Request, Response>,
          ) -> Result<(), Error> {
              match event {
                  request_response::Event::Message { message, .. } => match message {
                      Message::Request {
                          request,
                          channel,
                          request_id,
                          ..
                      } => {
                          trace!("Received request with id: {request_id:?}, req: {request:?}");
                          if let Err(error) = self
                              .event_sender
                              .send(NetworkEvent::RequestReceived {
                                  req: request,
                                  channel: MsgResponder::FromPeer(channel),
                              })
                              .await
                          {
                              error!("Failed in notify {request_id:?} NetworkEvent: {error:?}");
                          }

github.com

maidsafe/safe_network/blob/main/sn_networking/src/msg/mod.rs#L103


      
                          }
                      }
                      request_response::Event::InboundFailure {
                          peer,
                          request_id,
                          error,
                      } => {
                          warn!("RequestResponse: InboundFailure for request_id: {request_id:?} and peer: {peer:?}, with error: {error:?}");
                      }
                      request_response::Event::ResponseSent { peer, request_id } => {
                          trace!("ResponseSent for request_id: {request_id:?} and peer: {peer:?}");
                      }
                  }
                  Ok(())
              }
          }

When my node earlier today stored its first chunk, I didn’t find any RequestId, StoreChunk, or ChunkAddress in the logs, so I will circle back in a few hrs from now to figure out what happened here (might even have to start learning some Rust ).

Thanks all for the extra input/assistance.

Vort · June 20, 2023, 7:26pm

I saw records without chunks in previous testnet.
My guess is that some of records holds service information (Kad?).
However I doubt it because it would mean than some records are free to store - no one pays for them - which is strange.

chriso · June 20, 2023, 7:28pm

@Shu Was this the format of the output from before?

[2023-06-20T19:19:05.069297Z WARN safenode] {"message":"No peers given. As `local-discovery` feature is disabled, we will not be able to connect to the network."}
[2023-06-20T19:19:05.069475Z INFO safenode] {"message":"\nRunning safenode v0.83.40\n========================="}
[2023-06-20T19:19:05.071368Z INFO safenode] {"message":"Starting node ..."}
[2023-06-20T19:19:05.071926Z INFO sn_networking] {"message":"Generating a new keypair"}
[2023-06-20T19:19:05.072532Z INFO sn_networking] {"message":"Node (PID: 209270) with PeerId: 12D3KooWLaW9rTzBGMyv3MnVYapv4Ur9hvrHf9ZXifuQnUegffrM"}
[2023-06-20T19:19:05.072692Z INFO sn_networking] {"message":"PeerId: 12D3KooWLaW9rTzBGMyv3MnVYapv4Ur9hvrHf9ZXifuQnUegffrM has replication interval of 385.368521509s"}
[2023-06-20T19:19:05.078959Z INFO sn_networking::event] {"message":"Local node is listening on \"/ip4/127.0.0.1/tcp/41021/p2p/12D3KooWLaW9rTzBGMyv3MnVYapv4Ur9hvrHf9ZXifuQnUegffrM\""}
[2023-06-20T19:19:05.079258Z INFO sn_networking::event] {"message":"Local node is listening on \"/ip4/192.168.1.86/tcp/41021/p2p/12D3KooWLaW9rTzBGMyv3MnVYapv4Ur9hvrHf9ZXifuQnUegffrM\""}
[2023-06-20T19:19:05.079419Z INFO sn_networking::event] {"message":"Local node is listening on \"/ip4/172.18.0.1/tcp/41021/p2p/12D3KooWLaW9rTzBGMyv3MnVYapv4Ur9hvrHf9ZXifuQnUegffrM\""}
[2023-06-20T19:19:05.079557Z INFO sn_networking::event] {"message":"Local node is listening on \"/ip4/172.17.0.1/tcp/41021/p2p/12D3KooWLaW9rTzBGMyv3MnVYapv4Ur9hvrHf9ZXifuQnUegffrM\""}

Shu · June 20, 2023, 7:39pm

No, the format below for PID: 36112 (which started 19hrs ago or more based on my first post on this topic) was different:

Running safenode v0.83.35
=========================
[2023-06-20T00:12:23.689350Z INFO safenode] Starting node ...
[2023-06-20T00:12:23.689652Z INFO sn_networking] Node (PID: 36112) with PeerId: XYZ
[2023-06-20T00:12:23.689679Z INFO sn_networking] PeerId: XYZ has replication interval of 232.06966236s
[2023-06-20T00:12:23.694285Z INFO sn_networking::event] Local node is listening on "/ip4/127.0.0.1/tcp/12000/p2p/XYZ"
[2023-06-20T00:12:23.694453Z INFO sn_networking::event] Local node is listening on "/ip4/192.168.X.X/tcp/12000/p2p/XYZ"

Seems your sample messages are wrapped in a message variable.

Granted, I haven’t kept up with the latest releases after this initial topic was created, my original concern was I couldn’t find any RequestId or StoreChunks or ChunkAddress from sn_networking class in the logs for the version of safenode I used with this testnet from my end.

Maybe the issue is on my end, but I will be able to dig in much deeper in a few hours. Thanks for the assistance so far.

Topic		Replies	Views
Update June 22, 2023 Updates	17	1917	June 25, 2023
Update 11th April, 2024 Updates	23	1203	April 18, 2024
SAFE Network Dev Update - July 30, 2020 Updates	34	2635	August 6, 2020
Testnet - How to get started Features	11	967	April 15, 2021
Safe Network Dev Update - December 17, 2020 Updates	194	8129	January 14, 2021

InstallNet [2023-06-19 Testnet] [Offline]

Related topics