I’ll keep at it - see how repeatable this might be…
In other questions, who hid the logs?
I can’t find them at all…
2 Likes
earlier
68.183.40.246: 1602MB
now
165.22.112.246: 607MB
1 Like
Seems same sort of memory rise as last time with same exact setup from my side, except the newer safe version binaries. This time I turned on logging on the safe cli as well. Memory of sn_node immediately starts increasing when safe cli container’s network traffic starts increasing at 21:29 UTC.
Note: the safe cli between 21:22:50 and 21:31:04 did not record any more logs other than ‘awaiting some network knowledge’, but during this time the sn_node’s memory started rising.
safe-cli-113: container hosting safe cli; command initiated at 21:22:49 UTC
safe-109: container hosting sn_node; command initiated at 21:17:45 UTC
safe-cli-113:~# RUST_LOG=safe_network=info,qp2p=info safe files put -r ./uploads/
Dec 20 21:22:49.489 INFO safe_network::client::config_handler: Client set to use a query timeout of 120s, and AE await post-put for 0ns
Dec 20 21:22:49.489 INFO safe_network::client::client_api: Client started for new randomly created pk: Ed25519(PublicKey(7e57..beea))
Dec 20 21:22:49.489 WARN safe_network::client::client_api: Could not read PrefixMap at '.safe/prefix_map': FailedToParse("No such file or directory (os error 2)")
Dec 20 21:22:49.489 INFO safe_network::client::client_api: Checking Client's root dir
Dec 20 21:22:49.489 WARN safe_network::client::client_api: Could not read PrefixMap at Client's root dir: FailedToParse("No such file or directory (os error 2)")
Dec 20 21:22:49.489 INFO safe_network::client::client_api: Defaulting to a fresh NetworkPrefixMap
Dec 20 21:22:50.085 WARN safe_network::client::connections::listeners: CmdError received is: Data(DataNotFound(Register(Public { name: 195cf3(00011001).., tag: 1 })))
Dec 20 21:22:50.183 WARN safe_network::client::connections::listeners: CmdError received is: Data(DataNotFound(Register(Public { name: 195cf3(00011001).., tag: 1 })))
Dec 20 21:22:50.301 WARN safe_network::client::connections::listeners: CmdError received is: Data(DataNotFound(Register(Public { name: 195cf3(00011001).., tag: 1 })))
Dec 20 21:22:50.463 INFO safe_network::client::connections::messaging: On client startup, awaiting for some network knowledge
Dec 20 21:31:04.846 WARN safe_network::client::client_api::blob_apis: Reading chunk 4a8425.. from network, resulted in error NoResponse.
Dec 20 21:31:04.846 WARN safe_network::client::client_api::blob_apis: Reading chunk 113222.. from network, resulted in error NoResponse.
Dec 20 21:31:04.847 WARN safe_network::client::client_api::blob_apis: Reading chunk 2739fc.. from network, resulted in error NoResponse.
...
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: Error during MessageId(4894..d448) send: Err(QuicP2p(Send(ConnectionLost(TimedOut))))
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: Sending the message (MessageId(4894..d448)) from 0.0.0.0:41465 to 3/3 of the elders failed: [Peer { name: 0f9c03(00001111).., addr: 68.183.40.246:12000, connection: Some(Connection { id: 93824998301552, remote_address: 68.183.40.246:12000, .. }) }, Peer { name: 15fd8f(00010101).., addr: 138.68.141.56:12000, connection: Some(Connection { id: 93825002614736, remote_address: 138.68.141.56:12000, .. }) }, Peer { name: 75846c(01110101).., addr: 134.122.104.128:12000, connection: Some(Connection { id: 93825003343696, remote_address: 134.122.104.128:12000, .. }) }]
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: More errors when sending a message than successess
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: Error during MessageId(7c79..5de7) send: Err(QuicP2p(Send(ConnectionLost(TimedOut))))
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: Error during MessageId(7c79..5de7) send: Err(QuicP2p(Send(ConnectionLost(TimedOut))))
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: Error during MessageId(7c79..5de7) send: Err(QuicP2p(Send(ConnectionLost(TimedOut))))
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: Sending the message (MessageId(7c79..5de7)) from 0.0.0.0:41465 to 3/3 of the elders failed: [Peer { name: 0f9c03(00001111).., addr: 68.183.40.246:12000, connection: Some(Connection { id: 93824998301552, remote_address: 68.183.40.246:12000, .. }) }, Peer { name: 15fd8f(00010101).., addr: 138.68.141.56:12000, connection: Some(Connection { id: 93825002614736, remote_address: 138.68.141.56:12000, .. }) }, Peer { name: 75846c(01110101).., addr: 134.122.104.128:12000, connection: Some(Connection { id: 93825003343696, remote_address: 134.122.104.128:12000, .. }) }]
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: More errors when sending a message than successess
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: Error during MessageId(393b..08f0) send: Err(QuicP2p(Send(ConnectionLost(TimedOut))))
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: Error during MessageId(393b..08f0) send: Err(QuicP2p(Send(ConnectionLost(TimedOut))))
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: Error during MessageId(393b..08f0) send: Err(QuicP2p(Send(ConnectionLost(TimedOut))))
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: Sending the message (MessageId(393b..08f0)) from 0.0.0.0:41465 to 3/3 of the elders failed: [Peer { name: 0f9c03(00001111).., addr: 68.183.40.246:12000, connection: Some(Connection { id: 93824998301552, remote_address: 68.183.40.246:12000, .. }) }, Peer { name: 15fd8f(00010101).., addr: 138.68.141.56:12000, connection: Some(Connection { id: 93825002614736, remote_address: 138.68.141.56:12000, .. }) }, Peer { name: 75846c(01110101).., addr: 134.122.104.128:12000, connection: Some(Connection { id: 93825003343696, remote_address: 134.122.104.128:12000, .. }) }]
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: More errors when sending a message than successess
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: Error during MessageId(3004..f1fc) send: Err(QuicP2p(Send(ConnectionLost(TimedOut))))
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: Error during MessageId(3004..f1fc) send: Err(QuicP2p(Send(ConnectionLost(TimedOut))))
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: Error during MessageId(3004..f1fc) send: Err(QuicP2p(Send(ConnectionLost(TimedOut))))
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: Sending the message (MessageId(3004..f1fc)) from 0.0.0.0:41465 to 3/3 of the elders failed: [Peer { name: 0f9c03(00001111).., addr: 68.183.40.246:12000, connection: Some(Connection { id: 93824998301552, remote_address: 68.183.40.246:12000, .. }) }, Peer { name: 15fd8f(00010101).., addr: 138.68.141.56:12000, connection: Some(Connection { id: 93825002614736, remote_address: 138.68.141.56:12000, .. }) }, Peer { name: 75846c(01110101).., addr: 134.122.104.128:12000, connection: Some(Connection { id: 93825003343696, remote_address: 134.122.104.128:12000, .. }) }]
Dec 20 21:31:44.917 ERROR safe_network::client::connections::messaging: More errors when sending a message than successess
...
Dec 20 21:37:30.548 ERROR safe_network::client::connections::messaging: More errors when sending a message than successess
Error:
0: NetDataError: Failed to prepare write to Register operation: NoResponse
Location:
/rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/result.rs:1914
3 Likes
If you use the command in the OP to start your node try adding --log-dir ~/.safe/node/logs
I have lost a couple instances, may be nearing the end.
3 Likes
Thank you
LEts see if I can kill this node and restart it again
Are they running out of RAM as before?
2 Likes
no not even close
2 Likes
@aatonnomicc did you manage to join with a Oracle instance?
1 Like
I can’t put anything now, even tiny files
<ClientError(NoResponse)>
2 Likes
Yeah you could be right…
willie@gagarin:~$ RUST_LOG=safe_network=info,qp2p=info ~/.safe/node/sn_node --skip-auto-port-forwarding --log-dir ~/.safe/node/logs
Starting logging to file: "/home/willie/.safe/node/logs"
Encountered a timeout while trying to join the network. Retrying after 3 minutes.
1 Like
ditto, damn yesterday held up much better.
1 Like
I’ve got 1.3G of chunks - I guess the uploading volume finished it off
4 Likes
I have just shy of 400MB chunkage but I was late to the party
2 Likes
As David said, we probably need paid put to remedy this.
4 Likes
2 Likes
I accuse @aatonnomicc, in the Caspian Sea, with 1.5Gb of photies…
1 Like
Isn’t it weird using time as a variable here? Is it a temporary solution? Can something other than time be used?
I also see on Github that sometimes time is measured in seconds, while other times it’s measured in minutes.
Better in what way? Do you just mean it stayed up longer, as in more hours/minutes?
yes oracle instances can join but a couple of steps @stout77 @JPL
- clear out iptables
- open port on oracle firewall
sudo rm /etc/iptables/*.*
sudo reboot
then you need to go to the instance in oracle cloud website and look under the Primary VNIC it will list a subnet click on the subnets name and then there will be a screen showing an option for editing default security list
edit the default security list and you need to open the port for safe network to connect to your node that is listed when you try to start the node
5 Likes
yea maybe my bad on that one I figured they were all small files nothing larger than a few mb so was hoping it would keep the nodes filing at a nice steady rate
2 Likes
Timeouts were I think introduced to prevent CLI commands from hanging indefinitely when things were not working, so they are a convenience but in early networks we can expect some tuning so they are not to short, not to long, but just right 
.
I expect them to remain but possibly be shorter in the long run when the network characteristics are more reliable and efficient, when we’d probably only see them due to connection issues.
3 Likes
Thanks @aatonnomicc! I did the security list part but forgot about the ip tables after I started a new instance…thanks heaps for the info!
3 Likes