If node performs wrong, most likely cause is problems with code.
Detecting and ignoring such node is wrong decision.
What needs to be done is figuring out what exact expectations by code were not met.
Detect can be useful only in case of outright malicious node.
But it may be hard to distinguish between bug and malice.
2 Likes
Yes I noticed that I have a winner 
. Will get them for you but it will be a couple hours before I can.
I don’t know if it is just early but it seems that memory is at a less aggressive curve.
@joshuef sent via DM.
4 Likes
Thanks Josh and @Anselme, I just saw that merge which will be ideal.
The design of vdash centres around the logfiles for various reasons so it doesn’t work to have it use APIs (CLI or system calls).
The data has to be in the logfile!
I will add that when this test finishes.
Fantastic work guys. This is a great stage to be part of.
13 Likes
the use of utxos reduces memory a looot. So that’ll probably help nodes there.
11 Likes
Just did my second batch of test files - uploaded sucessfully on the first try with c 200 batch-size 2 … Given my low processor speed and slow connection speed I think this combo (max CPU & tiny batches) works for me.
Will try with some other larger files soon, but I don’t think size will make any difference here.
First round of upload completed, verifying and repaying if required...
======= Verification: 233 chunks to be checked and repayed if required =============
======= Verification Completed! All chunks have been paid and stored! =============
Uploaded all chunks in 108 minutes 11 seconds
4 Likes
Oof, this testnet is slow. I managed to get a file of almost 10MB uploaded, but it was a struggle. It took more than 10 minutes. At first, all chunks had to be repaid, then another 9 because it was so slow, then it finally was successful:
pi@raspberrypi:~ $ safe files upload -c 10 --batch-size 40 rar/temp.part002.rar
Built with git version: 26c3d70 / main / 26c3d70
Instantiating a SAFE client...
🔗 Connected to the Network
Total number of chunks to be stored: 20
Transfers applied locally
After 23.835855079s, All transfers made for total payment of Token(13055) nano tokens for 20 chunks.
Successfully made payment of 0.000013055 for 20 chunks.
Successfully stored wallet with cached payment proofs, and new balance 199.999979260.
After 42.082598643s, uploaded 20 chunks, current progress is 20/20.
Uploaded chunk #674684.. in 26 seconds
Uploaded chunk #b3f312.. in 26 seconds
Uploaded chunk #190bf1.. in 26 seconds
Uploaded chunk #8f8037.. in 27 seconds
Uploaded chunk #44a54c.. in 26 seconds
Uploaded chunk #3a7d42.. in 27 seconds
Uploaded chunk #b3a65e.. in 27 seconds
Uploaded chunk #8c508d.. in 27 seconds
Uploaded chunk #abdb0a.. in 27 seconds
Uploaded chunk #59d4b2.. in 27 seconds
Uploaded chunk #b7a9f2.. in 27 seconds
Uploaded chunk #597ba2.. in 27 seconds
Uploaded chunk #0e2fe8.. in 26 seconds
Uploaded chunk #cc5228.. in 27 seconds
Uploaded chunk #d6edba.. in 27 seconds
Uploaded chunk #820680.. in 26 seconds
Uploaded chunk #f7ff0f.. in 26 seconds
Uploaded chunk #0e2c1b.. in 53 seconds
Uploaded chunk #79a0b6.. in 26 seconds
Uploaded chunk #f4a9a4.. in 27 seconds
First round of upload completed, verifying and repaying if required...
======= Verification: 20 chunks to be checked and repayed if required =============
Failed to fetch a chunk ChunkAddress(8f8037(10001111)..)
Failed to fetch a chunk ChunkAddress(b3f312(10110011)..)
Failed to fetch a chunk ChunkAddress(190bf1(00011001)..)
Failed to fetch a chunk ChunkAddress(59d4b2(01011001)..)
Failed to fetch a chunk ChunkAddress(674684(01100111)..)
Failed to fetch a chunk ChunkAddress(44a54c(01000100)..)
Failed to fetch a chunk ChunkAddress(3a7d42(00111010)..)
Failed to fetch a chunk ChunkAddress(b3a65e(10110011)..)
Failed to fetch a chunk ChunkAddress(8c508d(10001100)..)
Failed to fetch a chunk ChunkAddress(abdb0a(10101011)..)
Failed to fetch a chunk ChunkAddress(b7a9f2(10110111)..)
Failed to fetch a chunk ChunkAddress(597ba2(01011001)..)
Failed to fetch a chunk ChunkAddress(0e2fe8(00001110)..)
Failed to fetch a chunk ChunkAddress(d6edba(11010110)..)
Failed to fetch a chunk ChunkAddress(f7ff0f(11110111)..)
Failed to fetch a chunk ChunkAddress(cc5228(11001100)..)
Failed to fetch a chunk ChunkAddress(820680(10000010)..)
Failed to fetch a chunk ChunkAddress(79a0b6(01111001)..)
Failed to fetch a chunk ChunkAddress(f4a9a4(11110100)..)
Failed to fetch a chunk ChunkAddress(0e2c1b(00001110)..)
======= Verification: 20 chunks were not stored in the network, repaying them in batches =============
Failed to fetch 20 chunks, attempting to repay them
Transfers applied locally
After 28.912608211s, All transfers made for total payment of Token(13134) nano tokens for 20 chunks.
Uploaded chunk #190bf1.. in 37 seconds
Uploaded chunk #b3f312.. in 37 seconds
Uploaded chunk #59d4b2.. in 37 seconds
Uploaded chunk #8f8037.. in 37 seconds
Uploaded chunk #674684.. in 37 seconds
Uploaded chunk #3a7d42.. in 37 seconds
Uploaded chunk #44a54c.. in 37 seconds
Uploaded chunk #f7ff0f.. in 37 seconds
Uploaded chunk #d6edba.. in 37 seconds
Uploaded chunk #820680.. in 1 minutes 51 seconds
Uploaded chunk #cc5228.. in 37 seconds
Uploaded chunk #79a0b6.. in 1 minutes 14 seconds
Uploaded chunk #f4a9a4.. in 37 seconds
Uploaded chunk #0e2fe8.. in 37 seconds
Uploaded chunk #b3a65e.. in 37 seconds
Uploaded chunk #597ba2.. in 1 minutes 14 seconds
Uploaded chunk #0e2c1b.. in 37 seconds
Uploaded chunk #8c508d.. in 37 seconds
Uploaded chunk #abdb0a.. in 37 seconds
Uploaded chunk #b7a9f2.. in 37 seconds
After 250.919499339s, verified 20 chunks
20 failed chunks were found, repaid & re-uploaded.
======= Verification: 20 chunks to be checked and repayed if required =============
Failed to fetch a chunk ChunkAddress(8f8037(10001111)..)
Failed to fetch a chunk ChunkAddress(b3f312(10110011)..)
Failed to fetch a chunk ChunkAddress(190bf1(00011001)..)
Failed to fetch a chunk ChunkAddress(674684(01100111)..)
Failed to fetch a chunk ChunkAddress(44a54c(01000100)..)
Failed to fetch a chunk ChunkAddress(3a7d42(00111010)..)
Failed to fetch a chunk ChunkAddress(b3a65e(10110011)..)
Failed to fetch a chunk ChunkAddress(8c508d(10001100)..)
Failed to fetch a chunk ChunkAddress(abdb0a(10101011)..)
======= Verification: 9 chunks were not stored in the network, repaying them in batches =============
Failed to fetch 9 chunks, attempting to repay them
Transfers applied locally
After 23.948035261s, All transfers made for total payment of Token(6131) nano tokens for 9 chunks.
Uploaded chunk #8f8037.. in 27 seconds
Uploaded chunk #b3f312.. in 27 seconds
Uploaded chunk #44a54c.. in 27 seconds
Uploaded chunk #190bf1.. in 27 seconds
Uploaded chunk #3a7d42.. in 28 seconds
Uploaded chunk #674684.. in 28 seconds
Uploaded chunk #b3a65e.. in 28 seconds
Uploaded chunk #8c508d.. in 28 seconds
Uploaded chunk #abdb0a.. in 30 seconds
After 141.356566787s, verified 20 chunks
9 failed chunks were found, repaid & re-uploaded.
======= Verification: 9 chunks to be checked and repayed if required =============
======= Verification Completed! All chunks have been paid and stored! =============
Uploaded all chunks in 10 minutes 4 seconds
Writing 64 bytes to "/home/pi/.local/share/safe/client/uploaded_files/file_names_2023-09-19_12-41-03"
6 Likes
@Joop77 Have another go with -c 200 --batch-size 5
Those repayment after failure take a bit of time - get around it with smaller batches and processing as fast as possible.
2 Likes
Yes much better, no repayments, but still slow upload.
Same file size (10MB), different file, 5 minutes.
pi@raspberrypi:~ $ safe files upload -c 200 --batch-size 5 rar/temp.part004.rar
Built with git version: 26c3d70 / main / 26c3d70
Instantiating a SAFE client...
⠐ 1/8 initial peers found. ⠈ 1/8 in🔗 Connected to the Network
Total number of chunks to be stored: 20
Transfers applied locally
After 22.021972681s, All transfers made for total payment of Token(13397) nano tokens for 5 chunks.
Successfully made payment of 0.000013397 for 5 chunks.
Successfully stored wallet with cached payment proofs, and new balance 199.999928773.
After 43.576113073s, uploaded 5 chunks, current progress is 5/20.
Uploaded chunk #a5b6c1.. in 25 seconds
Uploaded chunk #565422.. in 25 seconds
Uploaded chunk #d798a3.. in 25 seconds
Uploaded chunk #2c85fc.. in 26 seconds
Transfers applied locally
Uploaded chunk #2064e1.. in 23 seconds
After 25.398777244s, All transfers made for total payment of Token(3521) nano tokens for 5 chunks.
Successfully made payment of 0.000003521 for 5 chunks.
Successfully stored wallet with cached payment proofs, and new balance 199.999925252.
After 60.070155727s, uploaded 5 chunks, current progress is 10/20.
Uploaded chunk #93c5d9.. in 23 seconds
Uploaded chunk #cd395e.. in 23 seconds
Uploaded chunk #3b6a68.. in 24 seconds
Uploaded chunk #4893fe.. in 24 seconds
Transfers applied locally
Uploaded chunk #5808af.. in 20 seconds
After 22.91481609s, All transfers made for total payment of Token(6974) nano tokens for 5 chunks.
Successfully made payment of 0.000006974 for 5 chunks.
Successfully stored wallet with cached payment proofs, and new balance 199.999918278.
After 60.667460329s, uploaded 5 chunks, current progress is 15/20.
Uploaded chunk #1ef420.. in 24 seconds
Uploaded chunk #6e8e60.. in 24 seconds
Uploaded chunk #fd776f.. in 24 seconds
Uploaded chunk #a1baab.. in 25 seconds
Transfers applied locally
Uploaded chunk #c038fc.. in 21 seconds
After 22.425064148s, All transfers made for total payment of Token(2818) nano tokens for 5 chunks.
Successfully made payment of 0.000002818 for 5 chunks.
Successfully stored wallet with cached payment proofs, and new balance 199.999915460.
After 54.908482543s, uploaded 5 chunks, current progress is 20/20.
Uploaded chunk #cfcba6.. in 21 seconds
Uploaded chunk #19614c.. in 21 seconds
Uploaded chunk #968142.. in 22 seconds
Uploaded chunk #074e21.. in 22 seconds
Uploaded chunk #0e6d17.. in 22 seconds
First round of upload completed, verifying and repaying if required...
======= Verification: 20 chunks to be checked and repayed if required =============
======= Verification Completed! All chunks have been paid and stored! =============
Uploaded all chunks in 5 minutes 9 seconds
I did another upload of a similar file, took 4 minutes. But hey, very happy that it works!! 
I’m sure the up/download speed is a temporary thing 
4 Likes
Maybe interesting as a benchmark, we can check how long it takes to show the balance of the wallet. Or does this not indicate anything useful? 
$ time safe wallet balance
Built with git version: 26c3d70 / main / 26c3d70
199.999915460
real 0m13.892s
user 0m13.959s
sys 0m0.140s
2 Likes
It might indicate that the Node you are using for the peer_id is already busy! I wonder if lots of people are using the one:-
SAFE_PEERS: /ip4/139.59.182.135/tcp/43953/p2p/12D3KooWFP9PfPoDeAtQU5Y9akqmciABu4AJqPsiA3WQTgDe5XHn
The first thing I do for these testnets is start up my own node (using one of the Safe ones as the peer_id obviously) and then point my client and any other nodes I start to mine as the peer_id. This is to test that node and also because I think that the Safe ones will be getting slammed. Probably especially the first one in the list that is referenced with the export command because it’s easy.
With mine I’m getting the wallet balance returned in about 7.3 seconds consistently.
Feel free to try mine:-
export SAFE_PEERS="/ip4/54.88.136.70/tcp/12000/p2p/12D3KooWAPhXYaEaFc3bdvbZZLGY7Pau2C1EMsxCJVKSEe7j4fQe"
and see if you can get a better time.
5 Likes
Same, does not matter:
$ export SAFE_PEERS="/ip4/54.88.136.70/tcp/12000/p2p/12D3KooWAPhXYaEaFc3bdvbZZLGY7Pau2C1EMsxCJVKSEe7j4fQe"
$ time safe wallet balance
Built with git version: 26c3d70 / main / 26c3d70
199.999915460
real 0m13.884s
user 0m13.887s
sys 0m0.210s
Could this be a client side issue? Running a Pi 2b here.
Where are those logs saved?
ahhh very astute. Could well be. And could well be why GetStoreCost suffers (it happens really fast on client init, when presumably our routing table is much smaller).
Perhaps we should reformat the OP to use more peers in SAFE_PEERS. It’s ugly but would likely help there a good amount.
2 Likes
the client output should include that for you (when you have logging set)
I just don’t get it… Did I set it right like this?
~$ time SN_LOG=all safe files upload -c 200 --batch-size 5 sn_node
Built with git version: 26c3d70 / main / 26c3d70
Instantiating a SAFE client...
🔗 Connected to the Network Total number of chunks to be stored: 70
Transfers applied locally
Error:
0: Transfer Error Failed to send tokens due to Network Error Could not retrieve the record after storing it: 4115f6629d50d8c9cdac427c1ff60324d40dae6f49224ecec695611c35914bca..
1: Failed to send tokens due to Network Error Could not retrieve the record after storing it: 4115f6629d50d8c9cdac427c1ff60324d40dae6f49224ecec695611c35914bca.
Location:
sn_cli/src/subcommands/files.rs:179
Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
real 1m10,712s
user 0m7,187s
sys 0m1,405s
I had the same, I needed to do
rm -rf ~/.local/share/safe
An then it worked. Could be coincidental though.
I did that before I installed these versions. And when the logs work, where are they then?
So on the well-rewarded node front, @josh
has one node responsible for a distance of:
Distance(988965900894403463792576163452432494906623158970102030562903985953690248533)
and the largest distance i’ve noted in our nodes is
Distance(732681025913819444983637574006628256964494215200619615477670434834007234416)
So a fair bit larger. Which would account for more records/rewards.
This itself seems to be related to very few close group churn events. 6, vs ~10-20 on maidsafe nodes.
So this doesn’t look to be statistically significant as yet. And would probably change a fair bit if the clients were to be a bit more choosy (and refuse to pay outlier prices eg; which we’ve had and may well come bacl).
4 Likes
I tried this now, and Connecting to The SAFE Network seems to be stuck at the phase where the dot is rolling around, right before it would usually start the n/8 process. Been like that for about 2-5 minutes already. I need to go soon, I’ll leave it trying. Will be an hour or so before I’m back.
My node is connected and working though, according to Vdash.
EDIT: Did it all wrong and that’s why it is stuck. Wrong IP. 
1 Like
it does look awfully like it. I suppose it could be network related and maybe you are a long way away from both my node and the one you were using but the time is really similar. I wouldn’t have thought that getting the wallet balance would be very CPU intensive but maybe it is.