So the large file I’ve been trying to upload, and thought yesterday was successful with all but possibly one chunk remaining to upload, is still not finished, and worse, the chunks that were uploaded, upon verification weren’t and seem to have been lost off the network.
So still trying to get it uploaded.
log_2024-02-01_11-45-48.zip (2.8 MB)
got it, thx
looks like failure nodes keeps croping up
This looks like the error you get when there is an old wallet there. Try deleting the old wallet with the Windows client cleanup command:-
rmdir /s C:\Users\<username>\AppData\Roaming\safe
from the instructions and hitting the faucet again.
Unfortunately, all the 4 nodes in problem that causing the 5 chunks failed to upload are not run by us.
If anyone running the following 4 nodes, please terminate them, and share me the logs.
thank you very much.
12D3KooW9xcXG4pXdWtzxT5a4DujAwW2YuyECHCpTD9wGVwYm1nU
12D3KooW9rXZ2Z8cVziVAEwUzerh2S4f7t5q87CTzUQh187Rw5Te
12D3KooWN7Yt2n3uegfi7Atap7jnTPjAcps8vMGTfpjgTHoAHpMc
12D3KooWB8BQbp3cyBM4VksEnsxk1sdPC44CkuEpVGY4iNSXzVHk
Like others, I’ve had a node ramp in RAM usage and then die. I’m sure it wasn’t that it was killed by OOM. The VM has 8GB RAM with 5 nodes running on it with RAM in the range of 30-50MB at the time and this one went up to 1258MB before it was killed. Anyway, OOM would have killed a random process, not specifically this one.
Here are the zipped up logs (1.7MB) in case they are useful:-
"safenode.log_12D3KooWG9Htg7psVPCx7K8tfpCd8WrvV2ynkHmYk7SVL8wShwvc.gz" 2360d326dbfb51460320bb4a419da84d2c3b2935959df6c1f6cbc3e2f63495ed
It looks like it had a lot of peers at the time compared to other nodes and was removing some.
The last few lines from the logs are:-
[2024-01-31T17:53:18.193952Z INFO sn_networking::event] Current libp2p peers pool stats is NetworkInfo { num_peers: 99, connection_counters: ConnectionCounters { pending_incoming: 0, pending_outgoing: 0, established_incoming: 52, established_outgoing: 62 } }
[2024-01-31T17:53:18.194065Z INFO sn_networking::event] Removing 1 outdated live connections, still have 40 left.
[2024-01-31T17:53:18.261282Z INFO sn_networking::event] Current libp2p peers pool stats is NetworkInfo { num_peers: 98, connection_counters: ConnectionCounters { pending_incoming: 0, pending_outgoing: 0, established_incoming: 51, established_outgoing: 62 } }
[2024-01-31T17:53:18.261294Z INFO sn_networking::event] Removing 1 outdated live connections, still have 39 left.
[2024-01-31T17:53:18.481911Z INFO sn_networking::event] received identify info from undialed peer for not full kbucket Some(249), dail back to confirm external accesable peer_id=12D3KooWSS9RiS9TS7XNDb5iVACcbqDLWx9Mbs8mjwGMRus348pR addrs={"/ip4/134.209.24.43/udp/33848/quic-v1"}
[2024-01-31T17:53:18.986792Z INFO sn_networking::event] New peer added to routing table: PeerId("12D3KooWCQAQofjPUckMLqp8tek5qYWZwda7qRuFNf4cF9fvKcS9"), now we have #205 connected peers
[2024-01-31T17:53:18.986809Z INFO sn_networking::event] Peer PeerId("12D3KooWCQAQofjPUckMLqp8tek5qYWZwda7qRuFNf4cF9fvKcS9") has a Some(246) distance to us
[2024-01-31T17:53:18.986833Z INFO sn_networking::event] kBucketTable has 12 kbuckets 159 peers, [(0, 2, 243), (1, 2, 245), (2, 4, 246), (3, 5, 247), (4, 7, 248), (5, 19, 249), (6, 20, 250), (7, 20, 251), (8, 20, 252), (9, 20, 253), (10, 20, 254), (11, 20, 255)]
[2024-01-31T17:53:18.986841Z INFO sn_networking::event] kad_event::RoutingUpdated 205: PeerId("12D3KooWCQAQofjPUckMLqp8tek5qYWZwda7qRuFNf4cF9fvKcS9"), is_new_peer: true old_peer: None
[2024-01-31T17:53:18.986969Z INFO sn_node::log_markers] PeersInRoutingTable(205)
[2024-01-31T17:53:18.987273Z INFO sn_node::log_markers] PeerAddedToRoutingTable(PeerId("12D3KooWCQAQofjPUckMLqp8tek5qYWZwda7qRuFNf4cF9fvKcS9"))
[2024-01-31T17:53:18.987396Z INFO sn_node::log_markers] IntervalReplicationTriggered
[2024-01-31T17:53:21.378262Z INFO sn_networking::event] received identify info from undialed peer for not full kbucket Some(245), dail back to confirm external accesable peer_id=12D3KooWSEDxhDsF84Lyq17gUYnaX2UfPYAq9wmkhn5ug2CJFtbL addrs={"/ip4/68.183.37.8/udp/37223/quic-v1"}
I was looking in $HOME/.local/share/safe/node for the logs for my dead node and saw something odd. The contents of the directory shows a lot more ‘nodes’ than I thought. 25 directories where there should only be 5.
I thought I’d maybe forgotten to clear out the dir before starting the 5 nodes but I was sure I had.
Looking at the contents the only contents of the mystery ‘node’ directories is a ‘wallet’ directory. No logs and no records.
The creation dates of the directories are actually just after I started the nodes yesterday morning:-
ls -lrth
total 100K
drwxrwxr-x 5 safe safe 4.0K Jan 31 06:27 12D3KooWRxHn8qgRP6U2oLun2XgTMKmCW7dCiFo6gTZr97L8QqR1
drwxrwxr-x 5 safe safe 4.0K Jan 31 06:27 12D3KooWRyz18PLRSfWqi3cjsu4LjhSf4zCZbETmooJ6v28f1AsJ
drwxrwxr-x 5 safe safe 4.0K Jan 31 06:28 12D3KooWG9Htg7psVPCx7K8tfpCd8WrvV2ynkHmYk7SVL8wShwvc
drwxrwxr-x 5 safe safe 4.0K Jan 31 06:28 12D3KooWBsghVAY8ioezpBnodhw3uGk211HSQhpEAoqaAUFKye7M
drwxrwxr-x 5 safe safe 4.0K Jan 31 06:28 12D3KooWM3grG8G9maQNwKjNyvg3Qs3rRQ7Mc3PaLWDGbtxW51Hf
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWCMxrpDHydgnyrAxdQFSMRN3fRnyZ4KnyzNk7eqUmVShq
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWDApYDkLbJA2chkA4PPoupMnGpmzds7vnKD1po1FMSvyZ
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWDfuGYMpaV2L7ufafsXFjX64nSVCXPjosBL6Ky16H9v1z
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWFjJ4YhB8WqjmbWS5bqBBAKhUxxpXDvbZVm4gdGvXrpJ9
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWHST6ynC2rNZvTHdgRHqf2J6UEWzuFnamimYE8GUX8tv4
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWJN1mUK2fB7aSB3X8kPFgmf78zVBURH4JCbk7VMFg1dHa
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWKfQSEKVf7cqBN2UroCoVpBg6Kx22np7jf4UQpREXf2HX
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWLDP3ncXViVXQK2VT5fSL77KW331CXN8pmNGQFT4dm3Xc
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWMQRs7x2EYrp9wXFQjfTDucCWu93Gc48uW7CTXTrwoJdy
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWNJZjt9GG8C7xHpkv9XB8AAUwzC1MFLshQePu6n8sR84j
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooW9qAqA7mGmjzfvqVtFYhDcXmthNRxRQ7uSjnYtCFhv9VM
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWBq5CtjvHk4WusoTbpZJQfrmfr6SJe1yufQivSKhdPLmP
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWBxC8LdpFC8iQRdRRvan2AAnpG9CVe8NQ9iNAKV3FGnEN
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWC4mmud2fK9e8NhQ6qDQpcV25f1b3bxCcMGzQgwZcd9rX
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWDRGCwSwxPJpujNJU4M7Sqg12nqprofRH1BuELY4mDV63
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWEiGN4itWnDFQBBMYCXKwtQ4g9pexTfsobti12emn6ZdK
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWEMYY5oARiVyN71rzozte5ZnxykV5sdWZcZWcvFTQuX55
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWPFv4bfx9y34jgUjCxKioVwPtEh8pSdUHdY82n87Ccvcy
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWS2NgPSv3AmcUgSiTNKNnXanNXjrAB8wUrTnWfkcGLx1s
drwxrwxr-x 3 safe safe 4.0K Jan 31 06:30 12D3KooWSztVX9JF3jLrFXqbjQUZE6xVpM5o8yzG6yUd8edQPhVw
The top 5 are definitely the dirs of the actual nodes.
So it’s a bit of a mystery.
If anyone else has this and you want to easily find the directories of your actual nodes doing an ls -lrth will put the directories of your actual nodes at the top of the list so you can find them easily as they will always be the first created. Or ls -lth to put them at the bottom if you are that way inclined.
Or find them by the size of the directories:-
du -sh *
to show the sizes of the directories. The 20KB ones will just be the wallets.
du -sh * | sort $1
use to order them by size to put the real node directories at the bottom of the list.
two of them were mine and are now terminated
12D3KooW9rXZ2Z8cVziVAEwUzerh2S4f7t5q87CTzUQh187Rw5Te
12D3KooWB8BQbp3cyBM4VksEnsxk1sdPC44CkuEpVGY4iNSXzVHk
here are the logs
12D3KooW9rXZ2Z8cVziVAEwUzerh2S4f7t5q87CTzUQh187Rw5Te
12D3KooWB8BQbp3cyBM4VksEnsxk1sdPC44CkuEpVGY4iNSXzVHk
I have the same issue. Despite trying numerous times, almost all uploads fail due to one or several chunks.
This one due to one chunk.
log_2024-02-01_09-40-09.zip (45.6 KB)
And this one due to seven.
log_2024-02-01_09-21-56.zip (101.2 KB)
The client cleanup command shows an error and does not remove the old wallet:
PS C:\Users\gggg> rmdir /s C:\Users\gggg\AppData\Roaming\safe
Remove-Item : A positional parameter cannot be found that accepts argument 'C:\Users\gggg\AppData\Roaming\safe'.
At line:1 char:1
+ rmdir /s C:\Users\gggg\AppData\Roaming\safe
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Remove-Item], ParameterBindingException
+ FullyQualifiedErrorId : PositionalParameterNotFound,Microsoft.PowerShell.Commands.RemoveItemCommand
I have reported this in earlier tests:
@Chriso please provide guidance because we have some more complicated problem here.
looks like you are in Windows Powershell ?
if so, you can try
rm -r C:\Users\gggg\AppData\Roaming\safe
btw. it does more helpful to try chatgpt for this kind of OS cmd stuff 
It’s great to wake up and see 25 unread posts in a testnet topic. All this community energy and problem solving. Everyone here is building the Safe Network including those lurking, this is not software but wetware too.
Everyone 
rm -r C:\Users\gggg\AppData\Roaming\safe
Yes, I use Win PowerShell Terminal. Unfortunately, using the command does not produce any response:
PS C:\Users\gggg> rm -r C:\Users\gggg\AppData\Roaming\safe
PS C:\Users\gggg>
Sorry but I didn’t understand what you meant? 
How come 4 faulty nodes can cause a failure? Shouldn’t each of the 5 copies of a chunk end up in a different node, thus leaving at least 1 properly stored chunk?
Also, what kind percentage of the nodes have had this fault leading them to not store or deliver the chunks? It seems quite a few files and users are affected. I would expect it is more than just 4-5 nodes.
no response is a good sign.
does the safe folder still exists ?
when uploading, there is only one node got paid, and that node becomes the only one authorize the uploading.
once the upload is approved, it will then be replicated to other nodes.
so, at moment, if the node in trouble being picked as the one to be paid, it will block the upload.
Yes, folder safe exists, the safe.exe file was also not deleted by Windows.
Have you identified what the problem with these nodes is, is picking them off 1 by 1 as they are discovered a practical solution, wont more just keep popping up?
if you are using safeup, then safe.exe shall not appear at C:\Users\gggg\AppData\Roaming\safe
if it does, then maybe there is running safe or safenode process prevent the removal ?
also, as you are in Windows, why not just delete the folder in the explorer GUI ?
this will give you more info if the deletion failed
No. the current log is on info level, which does give much useful hints.
could be. however I prefer not to do any of this kind of resolvement at moment, as it will just cover up the root issue, and make it harder to be detected and track.
Unfortunately, yes, with the Quicnet.