Yes, for now you’ll need different users to avoid the clash.
The node manager will easily support running different versions of nodes connected to different networks, but there isn’t really any such solution when you need different client binary versions, unless you either use a different user, or name them differently or something like that.
We used to have a networks
command in our old safe
that allowed the use of different networks, but that doesn’t really help when you need a different binary version.
All good. I’ll just make users for alpha and beta tests. Make my life easier anyway than using different command sets.
A new alpha is now running. The original post has been updated with new versions and faucet address.
/var/log/safenode/safenode9/safenode.log:[2024-04-04T18:56:14.225973Z INFO sn_networking::record_store] Cost is now 10 for quoting_metrics QuotingMetrics { close_records_stored: 0, max_records: 2048, received_payment_count: 0, live_time: 140 }
nodes up and running faucet working as expected - uploading fast
vdash 0.17.2 should work fine with that - let me know if you try and it shows records. My nodes aren’t and I don’t have that message in the log yet.
I’m running:
safenode --version
safenode cli 0.105.6-alpha.4
But
grep -i QuotingMetrics ~/.local/share/safe/node/*/logs/safenode.log
finds nothing yet.
EDIT: In the past my nodes showed records instantly but this time it took minutes (not sure how many). Looking again after 46mins, all my nodes have sprung into life!
If anyone sees records in vdash with this new alpha, or can search for QuotingMetrics as above I’d be interested.
The log looks different than before but something still seems off @joshuef Here’s a log:
safenode-alpha.zip (22.7 KB)
Here’s the summary after 12m:
Installation of the client
and node
and operation of the faucet is lightning fast.
Time to test this marvel
No Safe, no wave.
safe wallet get-faucet 159.65.29.172:8000
les upload <directory-path>Logging to directory: "/home/ubuntu/.local/share/safe/client/logs/log_2024-04-04_19-19-10"
Built with git version: f682844 / alpha-tolerance /
f682844 / 2024-04-04
Instantiating a SAFE client...
Trying to fetch the bootstrap peers from https://sn-testnet.s3.eu-west-2.amazonaws.com/network-contacts
Connecting to the network with 50 peers
🔗 Connected to the Network
Requesting token for wallet address: 8daae2473bfdfdebaba7889f2209396bdfb2193254281ab9973b2d1308d92acd3bcb5620547c514817233ff3942ae090
Successfully parsed transfer.
Verifying transfer with the Network...
Failed to verify and redeem transfer: CouldNotReceiveMoney("InvalidTransfer(\"GetRecord Query Error RecordNotFound\")")
Error:
0: Failed to receive transfer due to InvalidTransfer("GetRecord Query Error RecordNotFound")
Location:
sn_cli/src/bin/subcommands/wallet/helpers.rs:141
Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
safe --version
sn_cli 0.90.4-alpha.5
@chriso I’m not sure whether this is a problem specific to the arm binary, but as you can see from above, for me the client tries to pull the peers from https://sn-testnet.s3.eu-west-2.amazonaws.com/network-contacts
rather than https://sn-testnet.s3.eu-west-2.amazonaws.com/alpha-tolerance-network-contacts
as @riddim pointed out. I seem to be the only one experiencing that, but I did a full clean up and installed the right binaries afaik.
Ah ok, thanks for pointing this out.
It will be specific to the ARM binary, and the reason is because that binary is built using a cross-compilation process that uses a Docker container. That container won’t be picking up the build-time environment variable that needs to be set to use the different network contacts file.
So, you will need to explicitly supply the --peer
argument on the ARM build, until we can fix this.
Cool, thanks. That said, is it possible to specify the peer with the faucet argument? I tried but failed…
Yeah, as far as I know you can use --peer
on any safe
command. It may have to come before the other arguments.
Edit: you can also use the SAFE_PEERS
environment variable to avoid setting it all the time.
are we meant to be cycling through peers like crazy?
These 3 are about 20 seconds apart.
PS C:\Windows\system32> safenode-manager status
=================================================
Safenode Services
=================================================
Refreshing the node registry...
Service Name Peer ID Status Connected Peers
safenode1 12D3KooWL6fy7Vc4GY8ec4ZY8mPfQ3oAmAPzRN7dyTBRyGgBzSfa ←[32mRUNNING←[0m 53
safenode2 12D3KooWNxiPV4MMm4ZXeT9FWsf2GABhEoHWP2LSQzzhqTTVc2Pm ←[32mRUNNING←[0m 20
safenode3 12D3KooWJrPqLu6HDpKQhmZN5Ra48C3CqfujAcb7HwUjUBDgUyJG ←[32mRUNNING←[0m 5
safenode4 12D3KooWCsWa3Tf4hiUEMxSbCdqiiKdoHSWhzhko64JNLy3y9r2H ←[32mRUNNING←[0m 9
safenode5 12D3KooWPKSnJyTR3ZjXZpzGYa56UceVyjg3j3yV4QymPh4pBF8d ←[32mRUNNING←[0m 62
PS C:\Windows\system32> safenode-manager status
=================================================
Safenode Services
=================================================
Refreshing the node registry...
Service Name Peer ID Status Connected Peers
safenode1 12D3KooWL6fy7Vc4GY8ec4ZY8mPfQ3oAmAPzRN7dyTBRyGgBzSfa ←[32mRUNNING←[0m 7
safenode2 12D3KooWNxiPV4MMm4ZXeT9FWsf2GABhEoHWP2LSQzzhqTTVc2Pm ←[32mRUNNING←[0m 32
safenode3 12D3KooWJrPqLu6HDpKQhmZN5Ra48C3CqfujAcb7HwUjUBDgUyJG ←[32mRUNNING←[0m 3
safenode4 12D3KooWCsWa3Tf4hiUEMxSbCdqiiKdoHSWhzhko64JNLy3y9r2H ←[32mRUNNING←[0m 8
safenode5 12D3KooWPKSnJyTR3ZjXZpzGYa56UceVyjg3j3yV4QymPh4pBF8d ←[32mRUNNING←[0m 66
PS C:\Windows\system32> safenode-manager status
=================================================
Safenode Services
=================================================
Refreshing the node registry...
Service Name Peer ID Status Connected Peers
safenode1 12D3KooWL6fy7Vc4GY8ec4ZY8mPfQ3oAmAPzRN7dyTBRyGgBzSfa ←[32mRUNNING←[0m 7
safenode2 12D3KooWNxiPV4MMm4ZXeT9FWsf2GABhEoHWP2LSQzzhqTTVc2Pm ←[32mRUNNING←[0m 22
safenode3 12D3KooWJrPqLu6HDpKQhmZN5Ra48C3CqfujAcb7HwUjUBDgUyJG ←[32mRUNNING←[0m 176
safenode4 12D3KooWCsWa3Tf4hiUEMxSbCdqiiKdoHSWhzhko64JNLy3y9r2H ←[32mRUNNING←[0m 7
safenode5 12D3KooWPKSnJyTR3ZjXZpzGYa56UceVyjg3j3yV4QymPh4pBF8d ←[32mRUNNING←[0m 12
That worked. For reference:
safe --peer /ip4/209.97.139.230/udp/59695/quic-v1/p2p/12D3KooWPTyqVdNzqcaeNGzjHA4nNnqJWkxU81gW7uBPYewu3jop wallet get-faucet 159.65.29.172:8000
Yeah, I’ve noticed this before, and not just on this testnet.
It’s not a question I can answer. Maybe one for @qi_ma ?
Just wanted to point out docker build args where the environment variable could be passed (without needing to rewrite the docker file again and again for replacing stuff)
Just ignore if known anyway
Yeah, thanks. The thing we use is actually a plugin for cargo
, called cross
, which then uses a container. So, we’ll need to supply the argument to that. I’m guessing it will probably support the scenario.
Edit: Actually, cross
is not a plugin, it’s a full program. But, I’m sure they will support it.
Kk - yeah I guess it’s a common scenario for build stuff
Looking again after 46mins, all my nodes have sprung into life! I don’t know exactly how long they took to receive records though.
Here they are after 50mins:
I am again at work, and again trying to upload the nmap install executable (version 7.94). I’ve yet to be able to upload this file from my work windows machine. I was able to sftp this file to my linux server and upload from there (on previous beta). However, the log does not look like it’s having an issue on my machine, but it’s not storing to the network. I think the windows machine thing is not causal, but just mentioning it. The office is behind a NAT, but no special setup or anything on the network.
Seemingly relevant log section that includes the phrase “this should not happen”
[2024-04-04T19:41:44.001735Z INFO sn_networking::event] Query task QueryId(187) NotFound record e45e68(a89e6a3a7955c71ccb38240041f863331ee6ac7a2ae73fdab1996fee4c52b6f7) among peers [PeerId("12D3KooWLD9BiWBmYbHPEvHTx8j3db7vDSnvb6F5eiXXCU1r8LgV"), PeerId("12D3KooWMqMsZ3R1EYndVcnkvGCk39RZYzMVDYAF6f8yhNCa1YML"), PeerId("12D3KooWM4bMfXrfC3tRHUuNdm5w5CiUjcEW2ykJih7annTs2Sgu"), PeerId("12D3KooWKaDj3UTKJp8F9UJ7zXM7BgrrdGiuw6zPxsiVVAaTzBFz"), PeerId("12D3KooWSRp3mueFQfWD4EYepUWEvqtqcZDt57GjBCHsA8bYkvUe")], QueryStats { requests: 22, success: 17, failure: 0, start: Some(Instant { t: 93186.4036362s }), end: Some(Instant { t: 93186.9574873s }) } - ProgressStep { count: 1, last: true }
[2024-04-04T19:41:44.001803Z DEBUG sn_networking::get_record_handler] Get record task QueryId(187) failed with {PeerId("12D3KooWSRp3mueFQfWD4EYepUWEvqtqcZDt57GjBCHsA8bYkvUe"), PeerId("12D3KooWMqMsZ3R1EYndVcnkvGCk39RZYzMVDYAF6f8yhNCa1YML"), PeerId("12D3KooWM4bMfXrfC3tRHUuNdm5w5CiUjcEW2ykJih7annTs2Sgu"), PeerId("12D3KooWLD9BiWBmYbHPEvHTx8j3db7vDSnvb6F5eiXXCU1r8LgV"), PeerId("12D3KooWKaDj3UTKJp8F9UJ7zXM7BgrrdGiuw6zPxsiVVAaTzBFz")} expected holders not responded, error NotFound { key: Key(b"\xe4^h\x8f\xe3.\x9b\xdf\x0b\xd3\xb3\x837\x99\xfb\x06\xd4H{\x15\\\xf1\xea\xe4N\xe3\xc2Q\xcf\x03,\x8d"), closest_peers: [PeerId("12D3KooWLD9BiWBmYbHPEvHTx8j3db7vDSnvb6F5eiXXCU1r8LgV"), PeerId("12D3KooWMqMsZ3R1EYndVcnkvGCk39RZYzMVDYAF6f8yhNCa1YML"), PeerId("12D3KooWM4bMfXrfC3tRHUuNdm5w5CiUjcEW2ykJih7annTs2Sgu"), PeerId("12D3KooWKaDj3UTKJp8F9UJ7zXM7BgrrdGiuw6zPxsiVVAaTzBFz"), PeerId("12D3KooWSRp3mueFQfWD4EYepUWEvqtqcZDt57GjBCHsA8bYkvUe")] }
[2024-04-04T19:41:44.001930Z WARN sn_networking] No holder of record 'e45e68(a89e6a3a7955c71ccb38240041f863331ee6ac7a2ae73fdab1996fee4c52b6f7)' found.
[2024-04-04T19:41:48.032165Z WARN sn_networking::record_store_api] Calling set_distance_range at Client. This should not happen
[2024-04-04T19:41:48.069167Z DEBUG sn_logging::metrics] {"physical_cpu_threads":8,"system_cpu_usage_percent":47.70044,"process":{"cpu_usage_percent":0.3121748,"memory_used_mb":49,"bytes_read":0,"bytes_written":6750,"total_mb_read":51,"total_mb_written":34}}
[2024-04-04T19:42:03.038709Z WARN sn_networking::record_store_api] Calling set_distance_range at Client. This should not happen
[2024-04-04T19:42:03.091175Z DEBUG sn_logging::metrics] {"physical_cpu_threads":8,"system_cpu_usage_percent":58.426815,"process":{"cpu_usage_percent":0.3121748,"memory_used_mb":50,"bytes_read":0,"bytes_written":386,"total_mb_read":51,"total_mb_written":34}}
[2024-04-04T19:42:04.083872Z INFO sn_networking] Getting record from network of e45e68(a89e6a3a7955c71ccb38240041f863331ee6ac7a2ae73fdab1996fee4c52b6f7). with cfg GetRecordCfg { get_quorum: Majority, retry_strategy: Some(Balanced), target_record: e45e68(a89e6a3a7955c71ccb38240041f863331ee6ac7a2ae73fdab1996fee4c52b6f7), expected_holders: {PeerId("12D3KooWSRp3mueFQfWD4EYepUWEvqtqcZDt57GjBCHsA8bYkvUe"), PeerId("12D3KooWMqMsZ3R1EYndVcnkvGCk39RZYzMVDYAF6f8yhNCa1YML"), PeerId("12D3KooWM4bMfXrfC3tRHUuNdm5w5CiUjcEW2ykJih7annTs2Sgu"), PeerId("12D3KooWLD9BiWBmYbHPEvHTx8j3db7vDSnvb6F5eiXXCU1r8LgV"), PeerId("12D3KooWKaDj3UTKJp8F9UJ7zXM7BgrrdGiuw6zPxsiVVAaTzBFz")} }
[2024-04-04T19:42:04.084134Z DEBUG sn_networking::cmd] Record e45e68(a89e6a3a7955c71ccb38240041f863331ee6ac7a2ae73fdab1996fee4c52b6f7) with task QueryId(188) expected to be held by {PeerId("12D3KooWSRp3mueFQfWD4EYepUWEvqtqcZDt57GjBCHsA8bYkvUe"), PeerId("12D3KooWMqMsZ3R1EYndVcnkvGCk39RZYzMVDYAF6f8yhNCa1YML"), PeerId("12D3KooWM4bMfXrfC3tRHUuNdm5w5CiUjcEW2ykJih7annTs2Sgu"), PeerId("12D3KooWLD9BiWBmYbHPEvHTx8j3db7vDSnvb6F5eiXXCU1r8LgV"), PeerId("12D3KooWKaDj3UTKJp8F9UJ7zXM7BgrrdGiuw6zPxsiVVAaTzBFz")}
[2024-04-04T19:42:04.084179Z INFO sn_networking::cmd] We now have 1 pending get record attempts and cached 0 fetched copies
[2024-04-04T19:42:04.640248Z INFO sn_networking::event] Query task QueryId(188) NotFound record e45e68(a89e6a3a7955c71ccb38240041f863331ee6ac7a2ae73fdab1996fee4c52b6f7) among peers [PeerId("12D3KooWLD9BiWBmYbHPEvHTx8j3db7vDSnvb6F5eiXXCU1r8LgV"), PeerId("12D3KooWMqMsZ3R1EYndVcnkvGCk39RZYzMVDYAF6f8yhNCa1YML"), PeerId("12D3KooWM4bMfXrfC3tRHUuNdm5w5CiUjcEW2ykJih7annTs2Sgu"), PeerId("12D3KooWKaDj3UTKJp8F9UJ7zXM7BgrrdGiuw6zPxsiVVAaTzBFz"), PeerId("12D3KooWSRp3mueFQfWD4EYepUWEvqtqcZDt57GjBCHsA8bYkvUe")], QueryStats { requests: 21, success: 13, failure: 0, start: Some(Instant { t: 93207.0399927s }), end: Some(Instant { t: 93207.5959174s }) } - ProgressStep { count: 1, last: true }
[2024-04-04T19:42:04.640316Z DEBUG sn_networking::get_record_handler] Get record task QueryId(188) failed with {PeerId("12D3KooWSRp3mueFQfWD4EYepUWEvqtqcZDt57GjBCHsA8bYkvUe"), PeerId("12D3KooWMqMsZ3R1EYndVcnkvGCk39RZYzMVDYAF6f8yhNCa1YML"), PeerId("12D3KooWM4bMfXrfC3tRHUuNdm5w5CiUjcEW2ykJih7annTs2Sgu"), PeerId("12D3KooWLD9BiWBmYbHPEvHTx8j3db7vDSnvb6F5eiXXCU1r8LgV"), PeerId("12D3KooWKaDj3UTKJp8F9UJ7zXM7BgrrdGiuw6zPxsiVVAaTzBFz")} expected holders not responded, error NotFound { key: Key(b"\xe4^h\x8f\xe3.\x9b\xdf\x0b\xd3\xb3\x837\x99\xfb\x06\xd4H{\x15\\\xf1\xea\xe4N\xe3\xc2Q\xcf\x03,\x8d"), closest_peers: [PeerId("12D3KooWLD9BiWBmYbHPEvHTx8j3db7vDSnvb6F5eiXXCU1r8LgV"), PeerId("12D3KooWMqMsZ3R1EYndVcnkvGCk39RZYzMVDYAF6f8yhNCa1YML"), PeerId("12D3KooWM4bMfXrfC3tRHUuNdm5w5CiUjcEW2ykJih7annTs2Sgu"), PeerId("12D3KooWKaDj3UTKJp8F9UJ7zXM7BgrrdGiuw6zPxsiVVAaTzBFz"), PeerId("12D3KooWSRp3mueFQfWD4EYepUWEvqtqcZDt57GjBCHsA8bYkvUe")] }
[2024-04-04T19:42:04.640409Z WARN sn_networking] No holder of record 'e45e68(a89e6a3a7955c71ccb38240041f863331ee6ac7a2ae73fdab1996fee4c52b6f7)' found.
[2024-04-04T19:42:04.640431Z WARN sn_networking] Record e45e68(a89e6a3a7955c71ccb38240041f863331ee6ac7a2ae73fdab1996fee4c52b6f7) not found after PUT, either rejected or not yet stored by nodes when we asked
If I remember correctly, it stalled out at chunk 32 last time as well (been sitting at chunk 32 for 10 minutes now)