PunchBowl [Testnet 09/05/2024] [Offline]

[happybeing]

3 days update on my ten cloud nodes. All look healthy. Some but not many more shuns so maybe shunning is happening more early on.

On the very high storage cost, the node that was very high is now charging 20 and a different node has a very high fee. Another went very high and has come down. So I guess as nodes join and churn happens this will even itself out, although people may still be tempted to speed up that process by killing nodes when their fees rise too high, which would be bad for network health.

[riddim]

hmmmmmmmmmm - i have one node with a record-count of 2068 …

last Logs

[2024-05-20T10:54:11.354841Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWNDBU6rGEUCW384zHMHcX4i4rNtuxQbV5uU4jTvd1D6RN"), found 135 beyond current farthest
[2024-05-20T10:54:11.760150Z INFO sn_networking::record_store] Retrieved record from disk! filename: ebab329881d237cfa61f6ec969dab1d0319fdc707bd75cf611b914ed377f8b99 after 1.81342ms
[2024-05-20T10:54:14.167481Z DEBUG sn_logging::metrics] {"physical_cpu_threads":4,"system_cpu_usage_percent":14.693816,"process":{"cpu_usage_percent":15.195259,"memory_used_mb":40,"bytes_read":53248,"bytes_written":8192,"total_mb_read":887,"total_mb_written":12}}
[2024-05-20T10:54:17.792628Z INFO sn_networking::record_store] Retrieved record from disk! filename: f174f6d316b393e62969c56386f10939df1e7c322ebfe6dd44d9eb15d7f60ada after 1.066779ms
[2024-05-20T10:54:17.923522Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWBZcsjMiWsYF1EtEepJk8uGktUNzwKXxxsYYbgZDmbZ5m"), found 135 beyond current farthest
[2024-05-20T10:54:17.989262Z INFO sn_networking::record_store] Retrieved record from disk! filename: 51dc45157a11179bf95e859dcd3d81b32f0be4b2933ec194bfa010cf63700c17 after 6.316282ms
[2024-05-20T10:54:21.589742Z INFO sn_networking::record_store] Retrieved record from disk! filename: 4be389ed822bc6b17560779f1be8b23081a1e7fc8a452254c4a2030423589a83 after 1.500547ms
[2024-05-20T10:54:25.888182Z INFO sn_node::log_markers] IntervalBadNodesCheckTriggered
[2024-05-20T10:54:25.889213Z DEBUG sn_node::node] Undertake bad_nodes check against bucket 247 having 7 peers, 7 candidates to be queried
[2024-05-20T10:54:26.064847Z INFO sn_networking::event::swarm] Peer PeerId("12D3KooWDeADzbUdvdtHkCLCP1z5MFmoY8wafejgUMbEuGamqEpz") is considered as bad, blocking it.
[2024-05-20T10:54:26.337943Z WARN sn_networking::event::swarm] OutgoingConnectionError to PeerId("12D3KooWDPuoyhZuo8ETDJLCQgQdhTmjGqRvWXTKdVU7t5ZPS8Hf") on ConnectionId(2491) - Transport([("/ip4/144.126.233.244/udp/49925/quic-v1/p2p/12D3KooWLM4hrJMZP7DYKQzk6phrSPDTfWvpKy8trmK6JKCukQ8P/p2p-circuit/p2p/12D3KooWDPuoyhZuo8ETDJLCQgQdhTmjGqRvWXTKdVU7t5ZPS8Hf", Other(Custom { kind: Other, error: Left(Left(Left(Connect(ResourceLimitExceeded)))) }))])
[2024-05-20T10:54:26.338106Z ERROR sn_networking::event::swarm] Dial errors len : 1
[2024-05-20T10:54:26.338110Z ERROR sn_networking::event::swarm] OutgoingTransport error : Other(Custom { kind: Other, error: Left(Left(Left(Connect(ResourceLimitExceeded)))) })
[2024-05-20T10:54:27.137595Z INFO sn_networking::event::kad] kad_event::RoutingUpdated 229: PeerId("12D3KooWKVRox7t4kRXkX9uxX4T4uUwWVLMvg7tn3quYFzEbDyd3"), is_new_peer: false old_peer: None
[2024-05-20T10:54:29.175831Z DEBUG sn_logging::metrics] {"physical_cpu_threads":4,"system_cpu_usage_percent":18.868757,"process":{"cpu_usage_percent":6.422153,"memory_used_mb":40,"bytes_read":0,"bytes_written":16384,"total_mb_read":887,"total_mb_written":12}}
[2024-05-20T10:54:29.181945Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWKvThpmZC3qNyFbPPdu7hsfE9zmYQJZTfJsXSSt2XrxiM"), found 135 beyond current farthest
[2024-05-20T10:54:29.222780Z INFO sn_networking::record_store] Retrieved record from disk! filename: b905b8629321390b2e91109444ac5316b6ea0cef05f5c83c516551f9058808e6 after 7.415119ms
[2024-05-20T10:54:36.204468Z INFO sn_networking::record_store] Retrieved record from disk! filename: f7654aba5b0e84f7556b2569bad0264f061efb20ac95ef195bf119caa9fdb086 after 330.125µs
[2024-05-20T10:54:40.129316Z INFO sn_networking::record_store] Retrieved record from disk! filename: 04432feb20f620f456c7ed12b9d1380c96092a2c221457fe847c3afcf17c1fca after 1.512218ms
[2024-05-20T10:54:44.412613Z DEBUG sn_logging::metrics] {"physical_cpu_threads":4,"system_cpu_usage_percent":16.717634,"process":{"cpu_usage_percent":5.184214,"memory_used_mb":39,"bytes_read":528384,"bytes_written":4096,"total_mb_read":888,"total_mb_written":12}}
[2024-05-20T10:54:44.883509Z INFO sn_node::log_markers] IntervalReplicationTriggered
[2024-05-20T10:54:45.119072Z INFO sn_networking::record_store] Retrieved record from disk! filename: 5230e16ad609538f523e90f3417c6d115a0f2023a451b41d1af26d3ee27b74fa after 17.440504ms
[2024-05-20T10:54:45.492943Z INFO sn_networking::record_store] Retrieved record from disk! filename: dd62745cc63084138ed9f273d3d6e8510b894e391db5188ab99544db95f5d5dc after 951.122µs
[2024-05-20T10:54:45.757988Z INFO sn_networking::record_store] Retrieved record from disk! filename: e1443011005218b4093463961f74cb7845b84149a9784ff31dbf93d36a5c8585 after 6.121358ms
[2024-05-20T10:54:46.234375Z INFO sn_networking::record_store] Retrieved record from disk! filename: 3546b1d4149d6ec2a1e2ec83fa9496906534c3e740c28633ea4aaa2134ab3b39 after 92.973µs
[2024-05-20T10:54:46.927185Z INFO sn_networking::record_store] Retrieved record from disk! filename: e6ec5503a819bfe00fd84c8484b90738a35c5023e5c080dcf6af666d2d089fc2 after 23.174522ms
[2024-05-20T10:54:47.221336Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWNDBU6rGEUCW384zHMHcX4i4rNtuxQbV5uU4jTvd1D6RN"), found 135 beyond current farthest
[2024-05-20T10:54:47.294529Z INFO sn_networking::record_store] Retrieved record from disk! filename: 1f273d0ec002ee5ff166f53041bc1bb927574c815c9f5b160396ea8597a9c6e1 after 159.557µs
[2024-05-20T10:54:48.162409Z INFO sn_networking::record_store] Retrieved record from disk! filename: f0bbb62de9d5ee80925e1ddf4f8c7f21d9a9c77145477aac8808599c19f75c27 after 8.04826ms
[2024-05-20T10:54:49.922908Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWBZcsjMiWsYF1EtEepJk8uGktUNzwKXxxsYYbgZDmbZ5m"), found 135 beyond current farthest
[2024-05-20T10:54:49.951504Z INFO sn_networking::record_store] Retrieved record from disk! filename: ce552da1a01e01434272cb8490300f925f55bc3546a0a1acb9f123254bbdfc36 after 4.59277ms
[2024-05-20T10:54:54.836512Z DEBUG safenode::rpc_service] RPC request received at 127.0.0.1:34421: NodeInfoRequest
[2024-05-20T10:54:54.841547Z INFO sn_transfers::wallet] Attempting to read wallet file
[2024-05-20T10:54:54.845739Z DEBUG sn_transfers::wallet::watch_only] Loaded wallet from "/var/safenode-manager/services/safenode18/wallet" with balance NanoTokens(40)
[2024-05-20T10:54:55.024524Z DEBUG safenode::rpc_service] RPC request received at 127.0.0.1:34421: RecordAddressesRequest
[2024-05-20T10:54:55.111491Z DEBUG safenode::rpc_service] RPC request received at 127.0.0.1:34421: NetworkInfoRequest
[2024-05-20T10:54:55.112712Z DEBUG safenode::rpc_service] RPC request received at 127.0.0.1:34421: KBucketsRequest
[2024-05-20T10:54:56.747549Z INFO sn_networking::record_store] Retrieved record from disk! filename: be5ac77404c4716065ab97d218b9fa793d84340f746cc10646afba26069389ae after 133.528µs
[2024-05-20T10:54:59.035813Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWKvThpmZC3qNyFbPPdu7hsfE9zmYQJZTfJsXSSt2XrxiM"), found 135 beyond current farthest
[2024-05-20T10:54:59.080246Z INFO sn_networking::record_store] Retrieved record from disk! filename: 763d6eff406d62a3372feb2580dd7989c5f11841ef24413b6729e53a981bc245 after 10.043007ms
[2024-05-20T10:54:59.415113Z DEBUG sn_logging::metrics] {"physical_cpu_threads":4,"system_cpu_usage_percent":14.229785,"process":{"cpu_usage_percent":11.263364,"memory_used_mb":39,"bytes_read":778240,"bytes_written":20480,"total_mb_read":889,"total_mb_written":12}}
[2024-05-20T10:55:07.692964Z INFO sn_networking::record_store] Retrieved record from disk! filename: d3b99f4021fd6ce18d9183eae77aedf6511dad2ea2d38757369e0e18f85d689c after 949.781µs
[2024-05-20T10:55:14.419606Z DEBUG sn_logging::metrics] {"physical_cpu_threads":4,"system_cpu_usage_percent":12.112932,"process":{"cpu_usage_percent":6.8002424,"memory_used_mb":39,"bytes_read":0,"bytes_written":4096,"total_mb_read":889,"total_mb_written":12}}
[2024-05-20T10:55:19.505379Z INFO sn_networking::record_store] Retrieved record from disk! filename: 421caa544c3c08d8cc5c1106114c00b10d0ccf10af7f1b04ebf5e661a0fb17b5 after 7.232719ms
[2024-05-20T10:55:21.786148Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWBZcsjMiWsYF1EtEepJk8uGktUNzwKXxxsYYbgZDmbZ5m"), found 135 beyond current farthest
[2024-05-20T10:55:21.817488Z INFO sn_networking::record_store] Retrieved record from disk! filename: 5e1ee8c48890991b849c3db370ea54879c1f549fcc00b3215270c74ec2abdf14 after 1.061197ms
[2024-05-20T10:55:23.261974Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWNDBU6rGEUCW384zHMHcX4i4rNtuxQbV5uU4jTvd1D6RN"), found 135 beyond current farthest
[2024-05-20T10:55:23.334924Z INFO sn_networking::record_store] Retrieved record from disk! filename: e3d8a820b28dff246a4b3e3c9752e17a0b8a7f07eea28a3f7c25fdb0a25e4128 after 886.993µs
[2024-05-20T10:55:25.936073Z INFO sn_node::log_markers] IntervalReplicationTriggered
[2024-05-20T10:55:26.306519Z INFO sn_networking::record_store] Retrieved record from disk! filename: 14714c8a878359c4ec410ec1de46876e5b11325850977e91576212220ad7e679 after 990.024µs
[2024-05-20T10:55:26.356765Z INFO sn_networking::record_store] Retrieved record from disk! filename: b7cc351ac24d53407320f3e63473b523f04a21f8489bb534933ca9104508dbf9 after 1.945005ms
[2024-05-20T10:55:26.999642Z INFO sn_networking::record_store] Retrieved record from disk! filename: 5c141142929bbd797994d1ce1792a15adf9ac686d18a425e5b9ba71b7d40e8d5 after 6.941658ms
[2024-05-20T10:55:27.182183Z WARN sn_networking::event::request_response] RequestResponse: OutboundFailure for request_id: OutboundRequestId(2388) and peer: PeerId("12D3KooWKVRox7t4kRXkX9uxX4T4uUwWVLMvg7tn3quYFzEbDyd3"), with error: Io(Custom { kind: NotConnected, error: ConnectionLost(ApplicationClosed(ApplicationClose { error_code: 0, reason: b"" })) })
[2024-05-20T10:55:27.182292Z WARN sn_networking::event::swarm] MsgReceivedError: ReceivedResponseDropped(OutboundRequestId(2388))
[2024-05-20T10:55:27.923587Z INFO sn_networking::record_store] Retrieved record from disk! filename: 3546b1d4149d6ec2a1e2ec83fa9496906534c3e740c28633ea4aaa2134ab3b39 after 72.515µs
[2024-05-20T10:55:28.273426Z INFO sn_networking::record_store] Retrieved record from disk! filename: 4b88a73b4d0fd08618ba3d3eaab26a64f286a403242b1c7a5699833add672089 after 1.433701ms
[2024-05-20T10:55:28.977305Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWKvThpmZC3qNyFbPPdu7hsfE9zmYQJZTfJsXSSt2XrxiM"), found 135 beyond current farthest
[2024-05-20T10:55:29.004317Z INFO sn_networking::record_store] Retrieved record from disk! filename: 5aca4f6514659b09351f67621d558c3991dceabf067d7d4aca2018d9bb9e3fcb after 1.381194ms
[2024-05-20T10:55:29.428911Z DEBUG sn_logging::metrics] {"physical_cpu_threads":4,"system_cpu_usage_percent":13.05981,"process":{"cpu_usage_percent":7.974574,"memory_used_mb":38,"bytes_read":1118208,"bytes_written":16384,"total_mb_read":890,"total_mb_written":12}}
[2024-05-20T10:55:30.920762Z INFO sn_networking::record_store] Retrieved record from disk! filename: 5168f9fd59d67a6b435b2d2ff8dd785712b6ad8201410e2081f66689ab7e69f2 after 24.074039ms
[2024-05-20T10:55:35.923373Z INFO sn_networking::record_store] Retrieved record from disk! filename: 7dda66d12e396391d8a0d76c9ebf2f4db020c1984dc18ee9b28c74886f6bdd20 after 1.47025ms
[2024-05-20T10:55:44.448933Z DEBUG sn_logging::metrics] {"physical_cpu_threads":4,"system_cpu_usage_percent":19.079737,"process":{"cpu_usage_percent":4.316547,"memory_used_mb":38,"bytes_read":8192,"bytes_written":4096,"total_mb_read":890,"total_mb_written":12}}
[2024-05-20T10:55:44.676192Z INFO sn_networking::event::swarm] Peer PeerId("12D3KooWDeADzbUdvdtHkCLCP1z5MFmoY8wafejgUMbEuGamqEpz") is considered as bad, blocking it.
[2024-05-20T10:55:53.672674Z INFO sn_networking::record_store] Retrieved record from disk! filename: b58e29ed564c2ec1dfbcf60259e0accc0577b96999326a5b8e91d73b88e62939 after 1.080674ms
[2024-05-20T10:55:53.932287Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWBZcsjMiWsYF1EtEepJk8uGktUNzwKXxxsYYbgZDmbZ5m"), found 135 beyond current farthest
[2024-05-20T10:55:54.158561Z INFO sn_networking::record_store] Retrieved record from disk! filename: ab61310597153c31da1e3fad837a2216b09ebbdef14f04c8b7362ac5eb647856 after 189.437351ms
[2024-05-20T10:55:55.364308Z INFO sn_networking::record_store] Retrieved record from disk! filename: 047faf60f9f0a6ee67eb4ac1337beaa5882406aac395fb6110e148772fb56721 after 5.170056ms
[2024-05-20T10:55:56.686445Z INFO sn_networking::record_store] Retrieved record from disk! filename: 8d6c7843f31d7badf9a03cc05c58b91cebfbb1a3cc3f926c05c34a336635eb31 after 8.347688ms
[2024-05-20T10:55:56.700325Z INFO sn_networking::record_store] Retrieved record from disk! filename: 6240c833a82e48050248c9becf52a6544652a7f53a1587c6ad18f4a22502a9ca after 7.881338ms
[2024-05-20T10:55:56.707394Z INFO sn_networking::record_store] Retrieved record from disk! filename: ab61310597153c31da1e3fad837a2216b09ebbdef14f04c8b7362ac5eb647856 after 1.86729ms
[2024-05-20T10:55:56.716408Z INFO sn_networking::record_store] Retrieved record from disk! filename: e3d8a820b28dff246a4b3e3c9752e17a0b8a7f07eea28a3f7c25fdb0a25e4128 after 2.67733ms
[2024-05-20T10:55:56.722729Z INFO sn_networking::record_store] Retrieved record from disk! filename: 77b21bbe3eca10e1494b3afca31103339ae249ecff9ba692136a67b610646881 after 6.219632ms
[2024-05-20T10:55:58.555338Z DEBUG safenode::rpc_service] RPC request received at 127.0.0.1:34421: NodeInfoRequest
[2024-05-20T10:55:58.556224Z INFO sn_transfers::wallet] Attempting to read wallet file
[2024-05-20T10:55:58.556600Z DEBUG sn_transfers::wallet::watch_only] Loaded wallet from "/var/safenode-manager/services/safenode18/wallet" with balance NanoTokens(40)
[2024-05-20T10:55:58.581256Z DEBUG safenode::rpc_service] RPC request received at 127.0.0.1:34421: RecordAddressesRequest
[2024-05-20T10:55:58.586225Z DEBUG safenode::rpc_service] RPC request received at 127.0.0.1:34421: NetworkInfoRequest
[2024-05-20T10:55:58.586949Z DEBUG safenode::rpc_service] RPC request received at 127.0.0.1:34421: KBucketsRequest
[2024-05-20T10:55:59.199832Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWNDBU6rGEUCW384zHMHcX4i4rNtuxQbV5uU4jTvd1D6RN"), found 135 beyond current farthest
[2024-05-20T10:55:59.408848Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWKvThpmZC3qNyFbPPdu7hsfE9zmYQJZTfJsXSSt2XrxiM"), found 135 beyond current farthest
[2024-05-20T10:55:59.473455Z DEBUG sn_logging::metrics] {"physical_cpu_threads":4,"system_cpu_usage_percent":13.470183,"process":{"cpu_usage_percent":10.908001,"memory_used_mb":41,"bytes_read":561152,"bytes_written":20480,"total_mb_read":890,"total_mb_written":12}}
[2024-05-20T10:55:59.623702Z INFO sn_networking::record_store] Retrieved record from disk! filename: a7baf8d7604fec5e91e932bac8715832ea8f4142c721c913c1f71c7edea0cb96 after 1.297689ms
[2024-05-20T10:55:59.638620Z INFO sn_networking::record_store] Retrieved record from disk! filename: 5838dbf7ebc3c439769b9c736494a69ad75694706c972f9d7bad934ad9846c63 after 10.584617ms
[2024-05-20T10:56:01.797528Z INFO sn_networking::record_store] Retrieved record from disk! filename: 81a1e9bbb6ea1ae44c5107146b16d0dcaafc911cd04028e587438ea241b4426f after 1.182364ms
[2024-05-20T10:56:06.902912Z INFO sn_node::log_markers] IntervalReplicationTriggered
[2024-05-20T10:56:07.344117Z INFO sn_networking::record_store] Retrieved record from disk! filename: 25f43cf5a8ad202f2097bdd3309036b963a1b9d41a10e9cc1a2e6deb7d9360a9 after 1.211678ms
[2024-05-20T10:56:07.563024Z INFO sn_networking::record_store] Retrieved record from disk! filename: 015ecbbe2d2d67b327ec50779bfafee219676a959d5dadf03a5b576c7f4afeef after 6.78703ms
[2024-05-20T10:56:08.141403Z INFO sn_networking::record_store] Retrieved record from disk! filename: 186ad951b5224055e567822c8babdd55d9e5c8b6de838778432d367fee7c382c after 1.064283ms
[2024-05-20T10:56:08.410911Z INFO sn_networking::record_store] Retrieved record from disk! filename: cdf198285e03b9a580bf52c2eeba08449dcca3a5e181569532f677911632dd17 after 1.410138ms
[2024-05-20T10:56:09.714690Z INFO sn_networking::record_store] Retrieved record from disk! filename: 403efb73d738030bc306c6d15acf617d07e8bb40bdcebf806eabecf92fd4b6ac after 874.5µs
[2024-05-20T10:56:09.969218Z INFO sn_networking::record_store] Retrieved record from disk! filename: 5a16a0c4f07ecc95928666fe4eba048dbd6f3eef2f97b909ec20bf7709c260f4 after 1.050127ms
[2024-05-20T10:56:10.379747Z INFO sn_networking::record_store] Retrieved record from disk! filename: 26cceef14e418a53087067a02fc17be6c002544d734a626ba3cabb0016df2410 after 889.848µs
[2024-05-20T10:56:14.481943Z DEBUG sn_logging::metrics] {"physical_cpu_threads":4,"system_cpu_usage_percent":13.196916,"process":{"cpu_usage_percent":8.956109,"memory_used_mb":37,"bytes_read":53248,"bytes_written":4096,"total_mb_read":890,"total_mb_written":12}}
[2024-05-20T10:56:14.915118Z INFO sn_networking::record_store] Retrieved record from disk! filename: efc9817e8495ef71aa4c7ebf12ecb373e607a850691c45bbc119d0dba3189912 after 418.409µs
[2024-05-20T10:56:16.598946Z INFO sn_networking::record_store] Retrieved record from disk! filename: 2ea391006347cc69f9c47f14cf40084db7af955ba9e24b9ef0355ffb52e4c268 after 3.242093ms
[2024-05-20T10:56:25.803595Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWBZcsjMiWsYF1EtEepJk8uGktUNzwKXxxsYYbgZDmbZ5m"), found 135 beyond current farthest
[2024-05-20T10:56:25.845254Z INFO sn_networking::record_store] Retrieved record from disk! filename: 3a220f0034a149f7173760b51d13574ecc74f06c9bf9b2e60c16eb8280df5fb8 after 323.674µs
[2024-05-20T10:56:29.240245Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWKvThpmZC3qNyFbPPdu7hsfE9zmYQJZTfJsXSSt2XrxiM"), found 135 beyond current farthest
[2024-05-20T10:56:29.562601Z INFO sn_networking::record_store] Retrieved record from disk! filename: 8d85ab4799bf6c0ff7de07e97b64ff45a4b0b68d1989c2ccfa84cf7442635441 after 1.140968ms
[2024-05-20T10:56:29.659594Z DEBUG sn_logging::metrics] {"physical_cpu_threads":4,"system_cpu_usage_percent":13.374851,"process":{"cpu_usage_percent":8.600237,"memory_used_mb":38,"bytes_read":0,"bytes_written":8192,"total_mb_read":890,"total_mb_written":12}}
[2024-05-20T10:56:35.438213Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWNDBU6rGEUCW384zHMHcX4i4rNtuxQbV5uU4jTvd1D6RN"), found 135 beyond current farthest
[2024-05-20T10:56:35.598143Z INFO sn_networking::record_store] Retrieved record from disk! filename: 679f527ac11c3e7ee5a47935ec632576771d72e79497fea3aa4938583b24a329 after 5.422897ms
[2024-05-20T10:56:35.886331Z INFO sn_node::log_markers] IntervalBadNodesCheckTriggered
[2024-05-20T10:56:35.911290Z DEBUG sn_node::node] Undertake bad_nodes check against bucket 247 having 7 peers, 7 candidates to be queried
[2024-05-20T10:56:36.437610Z WARN sn_networking::event::swarm] OutgoingConnectionError to PeerId("12D3KooWDPuoyhZuo8ETDJLCQgQdhTmjGqRvWXTKdVU7t5ZPS8Hf") on ConnectionId(2563) - Transport([("/ip4/144.126.233.244/udp/49925/quic-v1/p2p/12D3KooWLM4hrJMZP7DYKQzk6phrSPDTfWvpKy8trmK6JKCukQ8P/p2p-circuit/p2p/12D3KooWDPuoyhZuo8ETDJLCQgQdhTmjGqRvWXTKdVU7t5ZPS8Hf", Other(Custom { kind: Other, error: Left(Left(Left(Connect(ResourceLimitExceeded)))) }))])
[2024-05-20T10:56:36.437853Z ERROR sn_networking::event::swarm] Dial errors len : 1
[2024-05-20T10:56:36.437856Z ERROR sn_networking::event::swarm] OutgoingTransport error : Other(Custom { kind: Other, error: Left(Left(Left(Connect(ResourceLimitExceeded)))) })
[2024-05-20T10:56:38.073568Z INFO sn_networking::record_store] Retrieved record from disk! filename: cd181e8073c8c1ff60c56cce7976a8ef6041b844e99b9230659ba96a75dd621d after 2.287783ms
[2024-05-20T10:56:39.798853Z INFO sn_networking::record_store] Retrieved record from disk! filename: 4a6e7ac05f4e471b1beb8e7513a5380abcffb87ab4e1e8f239199f71db821feb after 962.824µs
[2024-05-20T10:56:44.693045Z DEBUG sn_logging::metrics] {"physical_cpu_threads":4,"system_cpu_usage_percent":21.673653,"process":{"cpu_usage_percent":10.557367,"memory_used_mb":37,"bytes_read":0,"bytes_written":8192,"total_mb_read":890,"total_mb_written":12}}
[2024-05-20T10:56:47.899155Z INFO sn_node::log_markers] IntervalReplicationTriggered
[2024-05-20T10:56:48.666245Z INFO sn_networking::record_store] Retrieved record from disk! filename: 8a0b4806faa8c519e8b29ed250dce21be24a9a6db2ae0ef075c74c516540e562 after 95.048µs
[2024-05-20T10:56:49.836863Z INFO sn_networking::record_store] Retrieved record from disk! filename: 9312b3e43fe54fede5a697fa5d5f2a9f459077f5512ffd3338af8dd52b693e1d after 2.675006ms
[2024-05-20T10:56:49.854864Z INFO sn_networking::record_store] Retrieved record from disk! filename: 3a96b6d08832a5d8ad6cbb1d2f5829c01d5a65ae391b0e5488daea1fa50d9064 after 1.718754ms
[2024-05-20T10:56:50.309760Z INFO sn_networking::record_store] Retrieved record from disk! filename: 66f90f0950aaad23445466a2bb1a8d80750c3663d388cccf44d47f9fe7af56f6 after 1.159782ms
[2024-05-20T10:56:53.769087Z INFO sn_networking::record_store] Retrieved record from disk! filename: 81a1e9bbb6ea1ae44c5107146b16d0dcaafc911cd04028e587438ea241b4426f after 863.59µs
[2024-05-20T10:56:56.877539Z INFO sn_networking::record_store] Retrieved record from disk! filename: 17a95aa8cb1b13cdd028518411cc618226739f042017f9f0b2399a4f4cf6d2bc after 948.93µs
[2024-05-20T10:56:57.879672Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWBZcsjMiWsYF1EtEepJk8uGktUNzwKXxxsYYbgZDmbZ5m"), found 135 beyond current farthest
[2024-05-20T10:56:58.128548Z INFO sn_networking::record_store] Retrieved record from disk! filename: 7681ecf28cbae1928a82827244e1d9cba51bbfd38402a5861048c80c7a502471 after 786.395µs
[2024-05-20T10:56:59.223219Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWKvThpmZC3qNyFbPPdu7hsfE9zmYQJZTfJsXSSt2XrxiM"), found 135 beyond current farthest
[2024-05-20T10:56:59.554167Z INFO sn_networking::record_store] Retrieved record from disk! filename: f9555368d367f166cd6caefe27d9e3f07e1cd99a92173ffb93eb621d9dac64f0 after 11.367507ms
[2024-05-20T10:56:59.733352Z DEBUG sn_logging::metrics] {"physical_cpu_threads":4,"system_cpu_usage_percent":19.836117,"process":{"cpu_usage_percent":9.647495,"memory_used_mb":38,"bytes_read":176128,"bytes_written":16384,"total_mb_read":891,"total_mb_written":12}}
[2024-05-20T10:57:00.830255Z INFO sn_networking::event::swarm] Peer PeerId("12D3KooWDeADzbUdvdtHkCLCP1z5MFmoY8wafejgUMbEuGamqEpz") is considered as bad, blocking it.
[2024-05-20T10:57:03.424066Z INFO sn_networking::record_store] Retrieved record from disk! filename: aa0e1ef1f37ca4e896203eb30cefce52dc24fbdb7a89e50afdd9c4f84d9bbeab after 1.08399ms
[2024-05-20T10:57:03.469022Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWBZcsjMiWsYF1EtEepJk8uGktUNzwKXxxsYYbgZDmbZ5m"), found 135 beyond current farthest
[2024-05-20T10:57:03.573398Z INFO sn_networking::record_store] Retrieved record from disk! filename: 5e1ee8c48890991b849c3db370ea54879c1f549fcc00b3215270c74ec2abdf14 after 36.538132ms
[2024-05-20T10:57:04.304112Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWKvThpmZC3qNyFbPPdu7hsfE9zmYQJZTfJsXSSt2XrxiM"), found 135 beyond current farthest
[2024-05-20T10:57:04.353558Z INFO sn_networking::record_store] Retrieved record from disk! filename: 80afe2d3b051544fde65c7b81fe0c553e90c4644257c9ac142f5e5b9dd91ae49 after 1.785157ms
[2024-05-20T10:57:08.740507Z INFO sn_networking::record_store] Retrieved record from disk! filename: e6ec5503a819bfe00fd84c8484b90738a35c5023e5c080dcf6af666d2d089fc2 after 1.178688ms
[2024-05-20T10:57:08.911776Z DEBUG safenode::rpc_service] RPC request received at 127.0.0.1:34421: NodeInfoRequest
[2024-05-20T10:57:08.913174Z INFO sn_transfers::wallet] Attempting to read wallet file
[2024-05-20T10:57:08.914022Z DEBUG sn_transfers::wallet::watch_only] Loaded wallet from "/var/safenode-manager/services/safenode18/wallet" with balance NanoTokens(40)
[2024-05-20T10:57:08.930671Z DEBUG safenode::rpc_service] RPC request received at 127.0.0.1:34421: RecordAddressesRequest
[2024-05-20T10:57:08.947453Z DEBUG safenode::rpc_service] RPC request received at 127.0.0.1:34421: NetworkInfoRequest
[2024-05-20T10:57:08.948262Z DEBUG safenode::rpc_service] RPC request received at 127.0.0.1:34421: KBucketsRequest
[2024-05-20T10:57:10.311770Z INFO sn_networking::record_store] Retrieved record from disk! filename: 67a33ede8d45ebc239f27300d1c9d238fbdfb2a60dffc8a62dfcf9db8f388e9d after 1.078991ms
[2024-05-20T10:57:14.736288Z DEBUG sn_logging::metrics] {"physical_cpu_threads":4,"system_cpu_usage_percent":11.658725,"process":{"cpu_usage_percent":8.87433,"memory_used_mb":37,"bytes_read":0,"bytes_written":8192,"total_mb_read":891,"total_mb_written":12}}
[2024-05-20T10:57:15.214500Z INFO sn_networking::record_store] Retrieved record from disk! filename: 9ba25f87dc0de3e1670dda78a79681f108f2ee86a8cb6d6cbfb41e1233222212 after 4.550822ms
[2024-05-20T10:57:16.383957Z INFO sn_networking::record_store] Retrieved record from disk! filename: 601485c990081c124a6de7d627992b9a06e194408660247fe314beded317a02f after 1.061478ms
[2024-05-20T10:57:25.665812Z INFO sn_networking::record_store] Retrieved record from disk! filename: 9cb488a36f601339364c02906f5ea80cfce0e4452254ea96674f6b26270a3114 after 1.619829ms
[2024-05-20T10:57:28.899382Z INFO sn_node::log_markers] IntervalReplicationTriggered
[2024-05-20T10:57:29.136655Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWKvThpmZC3qNyFbPPdu7hsfE9zmYQJZTfJsXSSt2XrxiM"), found 135 beyond current farthest
[2024-05-20T10:57:29.266746Z INFO sn_networking::record_store] Retrieved record from disk! filename: 7a6ada3ba4d0a5b623c7bde7155ff9c9869e99983e8d470194807217d73ce409 after 1.928174ms
[2024-05-20T10:57:29.271963Z INFO sn_networking::record_store] Retrieved record from disk! filename: 4168984aca87cede870d9b85d4e1375e2a9ce717b9bfb8415723939b5417fb1a after 1.041151ms
[2024-05-20T10:57:29.277765Z INFO sn_networking::record_store] Retrieved record from disk! filename: ee56ef8a787512ce4896446b6f46fafadbee3acaebf970e4fcef44307b40004d after 2.01791ms
[2024-05-20T10:57:29.838465Z DEBUG sn_logging::metrics] {"physical_cpu_threads":4,"system_cpu_usage_percent":16.358679,"process":{"cpu_usage_percent":5.089367,"memory_used_mb":38,"bytes_read":32768,"bytes_written":12288,"total_mb_read":891,"total_mb_written":12}}
[2024-05-20T10:57:30.211464Z INFO sn_networking::replication_fetcher] Node is full, among 135 incoming replications from PeerId("12D3KooWBZcsjMiWsYF1EtEepJk8uGktUNzwKXxxsYYbgZDmbZ5m"), found 135 beyond current farthest
[2024-05-20T10:57:30.317836Z INFO sn_networking::record_store] Retrieved record from disk! filename: 8d85ab4799bf6c0ff7de07e97b64ff45a4b0b68d1989c2ccfa84cf7442635441 after 2.796453ms
[2024-05-20T10:57:30.360945Z INFO sn_networking::record_store] Retrieved record from disk! filename: 536b31eaf87bebf6f62784e995ef07a9d4cbf118f43c9159763c12bb9989ab5b after 1.032976ms
[2024-05-20T10:57:32.133461Z INFO sn_networking::record_store] Retrieved record from disk! filename: 90870fecf326c1ddc68d542ec820a2935f38129aabe7d89e97a5d8c7599ee591 after 4.539862ms
[2024-05-20T10:57:33.135148Z INFO sn_networking::record_store] Retrieved record from disk! filename: 536b31eaf87bebf6f62784e995ef07a9d4cbf118f43c9159763c12bb9989ab5b after 550.877µs
[2024-05-20T10:57:34.902082Z INFO sn_networking::record_store] Retrieved record from disk! filename: 6ee27a67eb5c3f4af7f463f573ea5aba3ece39af9c4da4959a4e57f1b037d133 after 1.136588ms
[2024-05-20T10:57:37.775098Z INFO sn_networking::record_store] Retrieved record from disk! filename: ed9b783fa2b1bc23e7fbb367f10c503e2179bb55d98d64604523a7b5cf94624a after 89.977µ

and i have quite a couple of nodes at/above 2048 records (if i’m not doing it wrong …)

and the whole record distribution looks a bit skewed … either @storage_guy 's random data upload (which is super awesome!) is not perfectly random but fills up certain parts of the network more than others … or we might be facing the first attempt to target groups specifically (?) … or nodes leaving specific regions that are full? … or bad luck and some nodes are in trouble but not kicked out of the network?

[Josh]

This kind of thing is hopefully now going to quickly be ironed with graphmaster shu on the team

[R43]

Substituting two bad nodes per day cca. 22 running with home network.

[riddim]

hmhmmm - dumb question ahead …

safenode[3949]: No self-update supported yet.

this means the update command in the rpc interface is just a dummy and working - right? so using the node-manager is the only way to update a note atm (?)

[neo]

@riddim

couple of nodes at/above 2048 records

Are they all active chunks, or have some chunks stopped being active? Only the chunks you are a close node for will be in the active list. The inactive will be deleted when more chunks need to be stored.

This is why I wanted the active chunks figure because that is what the pricing algo should be using as the inactive ones do not count towards fullness

[neo]

@R43

Substituting two bad nodes

If you are shunned by one or two or 5 then you should be fine.

Or are you talking of the high store cost ones. People have seen the figure drop back down in time.

[riddim]

Problem is that I didn’t see it in the logs… I think my node doesn’t make quotes because it’s full and that’s where I’ve seen the active chunk count… But since it’s way beyond 2048 I assume this chunk count is only possible with all active (because I assume when reaching 2048 that’s the point when inactive chunks need to be deleted - when else would be a reasonable condition for chunk deletion if not when full?)
… It’s the count of chunks that are being reported through the rpc interface which is the same as the number of chunks in the folder … I suspect it’s the active chunks…

… And as you can see in the logs it repeatedly says it is full … So we’re at least at 2048…

[happybeing]

Anyone got a token or two to spare?

a72414784a15740f7f2553a98d74bec9d6f3571c9a98b5a64df25392aaec5dbe76f12c4669e20a91add4e25be919f1fb

[digipl]

@happybeing

a72414784a15740f7f2553a98d74bec9d6f3571c9a98b5a64df25392aaec5dbe76f12c4669e20a91add4e25be919f1fb

741e059dccc9cc2f756f30cfcca2ccc3cc42becce7cc776d041d22061ac1cc3f460394cc35aacca6cc2c81cc1d352b93cc6dd7cc9bccd5cc6584cc82cc8dcceccc48b1cc0053abcc95ccb7ccb6cc3a292a6eefcc215c83cc9bcc427576c0cc4d5bbdcca9ccf0cccdcc557392cc8acce3cc85ccf2cce9cca2cc6a96cccfcc0d8ccc62187161acccafcc36cdcc7992cc6000dcf4cce9ccedccf6cc738ecc6949ccccd3cc39d7cc5187ccfccc2d746b518eccd6cc6587cce7cc536dfdcc01a6ccabccd0cc5867ecccefcc50d6cc8ccc64fcccf9cc3b7b4eacccf4cc392099ccfbcc5e7b9ecc71f3cca5cc3adbcca3cc96ccd5cc11a0cc1f387ad3cc7bf0cc9ccca7ccdeccfdccbeccf9ccbecc048ecc96cc7b5be6cc44e7cc4f9fcc3d0c08ddccd9cc87cc4eb8ccf7cccfcc39f7ccc1cc90cce0ccdacc4c4eb4ccd9cc9dcc210f087de6cc6c7100dc6b98cc22d3cc025455decc57cbcc8bccfcccaccc98cc773f22b7ccfbcc6e120cc9ccedccfdcce2cc51ffcca4cc121d1c4f6003372af8ccdbcc9fccf4ccf8ccafcca2cc740befccaacc3000dc9391646574707972636e45a981

[aatonnomicc]

after an un unprofitable night nanos have started rolling in again

is that you doing your morning upload @Southside ?

[Josh]

@aatonnomicc

after an un unprofitable night nanos have started rolling in again

Unprofitable night? Are you suggesting profitable nights have occurred? Steer me sir, my nights are mostly spent dreaming of what not operating at a loss would be like.

[aatonnomicc]

I don’t think about it in money terms I think about it in terms of my happiness

[Southside]

I want to say this is working wonderfully - both the testnet and my shiny new fibre connection

but it has stalled at the last few chunks for >5 mins now

[aatonnomicc]

fingers crossed for those last few chunks.

[Southside]

Nope its not happening - still stalled and another smaller upload is sitting at 0/388 chunks after 30 mins

sad.

[Josh]

@Southside

my shiny new fibre connection

All dressed up and nowhere to go?

To be fair to PunchBowl it has done pretty well!

[Michael_Hills]

For me on the second node start: shunning not so bad, errors a little less, records more, but earnings suck, since i restarted my nodes.

All in all a good test-net.

I do think nodes, especially in the early days of the launch, need to earn a little more, in order to get the network up and running. Just a little Incentive.

The economics of the network will be very important to its success overall.

[aatonnomicc]

@Josh

Punch

its taken a few of them alright

[aatonnomicc]

network just made it up the hill to pass 7000 nodes