just interesting that myself and southside are missing different chunks but consistently the same different chunks for each of us from home network connections.
latest client sn_cli 0.83.34 has been uploading fine for me reason for last couple of failures has been running out of coins
Wondering if there is any value in folks noting which particular chunk fails for them on any given download - and then repeating the attempted download to see if they get a consistent (to them) failure?
HOT NEWS!!!
Now I am getting a failure on the same chunk as you… f026db(11110000)
🔗 Connected to the Network Downloading AnarchyInTheSouthside.mp3 from eaa0b39813183323b491e6715a0b4ea9f3cfdeede6f04a73c5b849b5210ad20d
Client (read all) download progress 1/132
Client (read all) download progress 2/132
Error downloading "AnarchyInTheSouthside.mp3": Chunks error Chunk could not be retrieved from the network: f026db(11110000)...
the thott plickens…
i just downloaded it on another machine at a different location with better internet and it downloaded fine
Client (read all) download progress 129/132
Client (read all) download progress 130/132
Client (read all) download progress 131/132
Client (read all) download progress 132/132
Client downloaded file in 6.351981502s
Saved AnarchyInTheSouthside.mp3 at /home/ubuntu/.local/share/safe/client/AnarchyInTheSouthside.mp3
tried again on my home internet connection and failed again with the same chunk as last time f026db(11110000)
was this a cloud instance, presumably on the end of a fat pipe? Perhaps some footering with timeouts may help mitigate this?
was on a home connection virgin media fibre 200 up/down that is downloading successfuly
home connection that is failing is Vodafone broadband 50 down 10 up
worked for me
Saved AnarchyInTheSouthside.mp3 at /home/wyse/.local/share/safe/client/AnarchyInTheSouthside.mp3
Four hours and counting to verfiy my ubuntu.iso chunks.
Holy smokes, no, no verify was a mistake.
oooooh!!! posh WestEndies…
lol thats in Hamilton my home connection is the rubbish one west end still has the original cable put in for telegrams lol
Hyperinflation 
Input was split into 2931 chunks
Will now attempt to upload them...
Error:
0: Transfer Error Transfer error: Not enough balance, 93.696685750 available, 6348.449310932 required.
1: Transfer error: Not enough balance, 93.696685750 available, 6348.449310932 required
2: Not enough balance, 93.696685750 available, 6348.449310932 required
ZimbabWeimar Republic 
Likewise
🔗 Connected to the Network Chunking 187 files...
Input was split into 695 chunks
Will now attempt to upload them...
Error:
0: Transfer Error Transfer error: Not enough balance, 196.266851651 available, 6935.572078410 required.
1: Transfer error: Not enough balance, 196.266851651 available, 6935.572078410 required
2: Not enough balance, 196.266851651 available, 6935.572078410 required
So there wont be too many new chunks to store … 
But small uploads are cheap
🔗 Connected to the Network Chunking 1 files...
Input was split into 4 chunks
Will now attempt to upload them...
Uploaded 4 chunks in 29 seconds
**************************************
* Payment Details *
**************************************
Made payment of 0.000045452 for 4 chunks
New wallet balance: 12857.402647039
**************************************
* Verification *
**************************************
4 chunks to be checked and repaid if required
Verified 4 chunks in 912.224083ms
Verification complete: all chunks paid and stored
**************************************
* Uploaded Files *
**************************************
Uploaded desktop to 189c83a9110ec10abc7bc7b33af7a6c84aa6ced082d4782af2086a9f8945cf93
much prettier output, thanks @chriso 
🔗 Connected to the Network Chunking 7 files...
Input was split into 24 chunks
Will now attempt to upload them...
Uploaded 24 chunks in 1 minutes 47 seconds
**************************************
* Payment Details *
**************************************
Made payment of 3.340906046 for 24 chunks
New wallet balance: 17454.061493841
**************************************
* Verification *
**************************************
24 chunks to be checked and repaid if required
Verified 24 chunks in 22.157636669s
Verification complete: all chunks paid and stored
**************************************
* Uploaded Files *
**************************************
Uploaded upgrading-ubuntu-desktop to 37d5a4de2309a622bf85a80077ba4962b19b3eaa75b6f872b837aa5d5ee4a035
Uploaded tutorial-guidelines to 5f69233823bbda912161a73dc6e5df6bb49a6f4ed5f38e2c5ccd0a355df73a84
Uploaded how-to-install-ubuntu-desktop-on-raspberry-pi-4 to 9372df413127b8082a974e60f492b02387091a9230cfefc88e29e8320e59a302
Uploaded how-to-run-ubuntu-desktop-on-a-virtual-machine-using-virtualbox to 966321ac49041dff6a76a65c78508b289dd284f6588fcecf4cde3f0e3ee9c0f7
Uploaded tutorial-install-ubuntu-server to bce762440745503dc3e8a738bfecd93cede658e031326b382ca05277cd572010
Uploaded tutorial-install-ubuntu-desktop to f61c26a620143bc018a236fd92903506871eac6007b026ad1bd97c88b279ef93
but trying to push my luck to a massive 36 chunks failed
🔗 Connected to the Network Chunking 10 files...
Input was split into 36 chunks
Will now attempt to upload them...
Error:
0: Transfer Error Transfer error: Not enough balance, 20392.924066954 available, 624022578.218770155 required.
1: Transfer error: Not enough balance, 20392.924066954 available, 624022578.218770155 required
2: Not enough balance, 20392.924066954 available, 624022578.218770155 required
Location:
sn_cli/src/subcommands/files.rs:226
Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
Oh and I broke the faucet as well
🔗 Connected to the Network Requesting token for wallet address: b37d1540cde9ee228cce90cdebd467c74eb6b5df68639adeb2566aa042411fe9e53b2fd1f9c8990fb03b98b52b91f80e...
Successfully parsed transfer.
Verifying transfer with the Network...
Failed to verify and redeem transfer: CouldNotReceiveMoney("InvalidTransfer(\"Spend not found: SpendAddress(63fb22(01100011)..)\")")
Successfully got tokens from faucet.
[1]+ Done for i in {1..100};
do
safe wallet get-faucet 138.68.161.29:8000;
done
Now with every register I get this error
Connected to the Network Successfully made payment of 3.524154730 for a Register (At a cost per record of NanoTokens(3524154730).)
Successfully stored wallet with cached payment proofs, and new balance 2290.169100422.
Error:
0: Network Error Could not retrieve the record after storing it: efc9d5388c9ef916b16196e6f82f783001e23fa034c38bed0d0b8bb4bad9d2de.
1: Could not retrieve the record after storing it: efc9d5388c9ef916b16196e6f82f783001e23fa034c38bed0d0b8bb4bad9d2de
I get this with every faucet request
Connected to the Network Requesting token for wallet address: 941408db7f87d4e57c29bda3af02f9a9d7f59667fe266000eb30668c271f4584870d97448d865b1bada39c5e00def82b...
Failed to get tokens from faucet, server responded with: Failed to send tokens: Transfer Error Failed to send tokens due to The transfer was not successfully registered in the network: CouldNotSendMoney("Network Error Could not retrieve the record after storing it: f06e43801983b6210d201769022364420a196333896e6de20529e13c60250b27.")
46114213.837534351 required
Connected to the Network Chunking 1 files...
Input was split into 53 chunks
Will now attempt to upload them...
Error:
0: Transfer Error Transfer error: Not enough balance, 2090.172381004 available, 46114213.837534351 required.
1: Transfer error: Not enough balance, 2090.172381004 available, 46114213.837534351 required
2: Not enough balance, 2090.172381004 available, 46114213.837534351 required
Of course, connections will use some memory.
But what more important is that memory consumption also increases without increase of connections (first part of your chart).
Something is off with safeup. It does not give you the latest version as a default. I removed everything, run safeup without specifying the version, and the version I got was:
sn_cli 0.83.21
But a while ago @aatonnomicc already had:
I managed to download without problems:
Saved AnarchyInTheSouthside.mp3 at /home/topi/.local/share/safe/client/AnarchyInTheSouthside.mp3
real 0m18,873s
user 0m30,124s
sys 0m7,538s
That’s not working for me:
Downloading JoshsUbuntu from 6cfa28d385d5af711893744362aaa32e9116aacce06287614163c20e1b5064df
Client (read all) download progress 1/3666
Client (read all) download progress 2/3666
Client (read all) download progress 3/3666
Client (read all) download progress 4/3666
Client (read all) download progress 5/3666
Client (read all) download progress 6/3666
Client (read all) download progress 7/3666
Error downloading "JoshsUbuntu": Chunks error Chunk could not be retrieved from the network: ab2214(10101011).
On a second trial a different chunk failed:
Downloading JoshsUbuntu from 6cfa28d385d5af711893744362aaa32e9116aacce06287614163c20e1b5064df
Client (read all) download progress 1/3666
Error downloading "JoshsUbuntu": Chunks error Chunk could not be retrieved from the network: d8b678(11011000)...
real 6m1,599s
user 9m30,009s
sys 1m21,952s
I was checking my system monitor while downloading, and it seemed to download a whole lot of stuff. At the same time the information in the Terminal seemed to be lagging behind. I was receiving a long time at about 24MB/s, before nothing happened there.
Is it downloading in parallel, in whatever order, but updating the progress in order of chunks from smallest number to largest?
I think you’re hitting a hang there. It’s worth checking logs for any activity,
Yeh kind of. I think we’re only progressing the counter when we have the next chunk necessary at the moment.
edit:
sn_node 0.92.4 has removed the code that was aiming to prevent “lost nodes”. It’s suspected to be involved in the mem leak. If folk fancy restarting a couple of nodes to use that, it could be interesting to see what/if there’s any difference.
(it’s also worth noting, I’m wondering if the new network-contact setup, with many many more contacts than in the older testnets might well help with the ‘lost-node’ issues…)
Yes it’s the size of the Transaction. Since each cashnote contains the transaction it was created in:
batch 1: means 6 outputs (1*5 payments + 1 change) in the Transaction, meaning small transaction, resulting in smaller cashnotesbatch 21: means 105 outputs (21*5 payments + 1 change) in the Transaction, meaning bigger transaction, resulting in big cashnotesWe currently keep a local copy of the cashnotes we created for the nodes as payment. In both cases (any batch size) there will be the same amount of those as we ultimately pay the same nodes. The difference is that instead of having 105 small ones (with batch 1) we will have 105 BIG ones (batch 21)!
One transaction means 1 change output:
batch 21 would mean we add 1 change cashnote to our wallet because all is settled with only one transaction.batch 1 in means we need 21 transaction resulting in 21 change outputs.All in one transaction:
28K28K2k105 * 28k + 1* 28k + 2k make roughly 3.0M
Divided in 21 transactions:
4K4K2k105 * 4k + 21* 4k + 2k make roughly 528K
Now intuitively I would want to end up with a smaller cashnote dir when batching payments! The reason it is not is because we currently keep the cashnotes we created for the node payments for debug purposes. Ultimately those will not be kept anymore as we don’t need them (we don’t own them), resulting in smaller cashnote dir when batching! Which is indeed the intended behaviour!
Hopefully this has helped clear some confusion!
It worked pretty consistently, never hang so to speak I could watch it slowly work through verification. Completed verification after 8 hours.
It was only on the batch-size 1 upload that verification took this long, are they connected somehow?
With batch-size 1 the upload took, 21 hours but was only missing 1 chunk after the check, way less than other attempts, after batch-size 2838 (all chunks) the first check found 1700 missing chunks.
⠓ [08:06:14] [################################>-------] 2323/2838 ⠈ Verified 2838 chunks in 35191.643637228s
1 chunks were not stored. Repaying them in batches.
Failed to fetch 1 chunks. Attempting to repay them.
⠤ [00:00:30] [----------------------------------------] 0/2838 Repaid and re-uploaded 1 chunks in 30.687921004s
**************************************
* Verification *
**************************************
1 chunks to be checked and repaid if required
Verified 1 chunks in 44.79699842s
Verification complete: all chunks paid and stored
**************************************
* Uploaded Files *
**************************************
Uploaded ubuntu.iso to 3fb74b0c141f57f2028efee188c26e34d9b29e1b752782129fcfe4e73e5a19d1
I am failing to download it though.
How do I limit concurrency?
Thank you! @Anselme I much better understand now.
Started 8 under 5:x