True it’s definitely going up but it’s not going up as fast as it was before in previous test nets.
So hopefully the line will be nice and flat in the not to distant future
2 Likes
I suspect the fix may come from libp2p and we just need to be patient.
5 Likes
The speed of the tests is killer! MaidSafe does not slow down in the curves and it accelerates even more - for the first time (I do not know how) I messed up the tests and instead of posting them here I posted them on IntolerantNodeNet 
Below is the data from uploading the 94.2 MB file to the network:
Here is the result of uploading the same 94.2 MB file using safe files upload -c 10 --batch-size 40:
PS C:\Users\gggg> safe files upload -c 10 --batch-size 40 E:\gggg\Pictures\VID_20210505_113014.mp4
Built with git version: 2c47455 / main / 2c47455
Instantiating a SAFE client...
🔗 Connected to the Network
Total number of chunks to be stored: 190
Transfers applied locally
All transfers completed in 21.5237144s
Total payment: NanoTokens(1105007) nano tokens for 40 chunks
Made payment of 0.001105007 for 40 chunks
Stored wallet with cached payment proofs. New balance: 299.998791071
Uploaded chunk #41ad71.. in 3 seconds
Uploaded chunk #0c0c02.. in 3 seconds
Uploaded chunk #2c5814.. in 3 seconds
Uploaded chunk #dc0ddf.. in 10 seconds
Uploaded chunk #9822fb.. in 3 seconds
Uploaded chunk #ed6e20.. in 3 seconds
Uploaded chunk #ebe3d2.. in 11 seconds
Uploaded chunk #5a4112.. in 7 seconds
Uploaded chunk #dd38e6.. in 3 seconds
Uploaded chunk #f9852c.. in 7 seconds
Uploaded chunk #0ca297.. in 3 seconds
Uploaded chunk #6f9f18.. in 7 seconds
Uploaded chunk #eca2df.. in 18 seconds
Uploaded chunk #6535f0.. in 7 seconds
Uploaded chunk #26bad9.. in 15 seconds
Uploaded chunk #d3642d.. in 23 seconds
Uploaded chunk #e74708.. in 23 seconds
Uploaded chunk #ab646c.. in 19 seconds
Uploaded chunk #2557e3.. in 15 seconds
Uploaded chunk #581d52.. in 11 seconds
Uploaded chunk #681b43.. in 11 seconds
Uploaded chunk #ba7786.. in 10 seconds
Uploaded chunk #4e1d0a.. in 10 seconds
Uploaded chunk #383f2d.. in 7 seconds
Uploaded chunk #7a8c2c.. in 7 seconds
Uploaded chunk #3c6370.. in 7 seconds
Uploaded chunk #a930c6.. in 6 seconds
Uploaded chunk #8689ff.. in 3 seconds
Uploaded chunk #21fbd5.. in 18 seconds
Uploaded chunk #bc0f60.. in 3 seconds
Uploaded chunk #1d26f2.. in 3 seconds
Uploaded chunk #32d020.. in 2 seconds
Uploaded chunk #376b98.. in 6 seconds
Uploaded chunk #316ad0.. in 2 seconds
Uploaded chunk #fb9468.. in 2 seconds
Uploaded chunk #82e7ce.. in 5 seconds
Uploaded chunk #0f6c26.. in 2 seconds
Uploaded chunk #bd6a36.. in 5 seconds
Uploaded chunk #e7fc0d.. in 2 seconds
Uploaded chunk #fb3556.. in 2 seconds
Uploaded 40 chunks in 102.5828224s. Current progress is 40/190.
Cannot get store cost for NetworkAddress::ChunkAddress(a3e428(10100011).. - - a3e4281718ad85e08e6c45fcc72c6f8f38d090265a06bc5507c930870f4f40d4) with error CouldNotSendMoney("Network Error Not enough store cost prices returned from the network to ensure a valid fee is paid.")
Cannot get store cost for NetworkAddress::ChunkAddress(ca480d(11001010).. - - ca480d3c934ea77d13a4776da507ae09fbab6a88e8b8990b6be0705f90df5d0e) with error CouldNotSendMoney("Network Error Not enough store cost prices returned from the network to ensure a valid fee is paid.")
Cannot get store cost for NetworkAddress::ChunkAddress(f49535(11110100).. - - f49535bc1900e718abbd970659d4dbe6e371cdc5f1948b42938e0c64f460818e) with error CouldNotSendMoney("Network Error Not enough store cost prices returned from the network to ensure a valid fee is paid.")
Transfers applied locally
All transfers completed in 20.5847587s
Total payment: NanoTokens(552316) nano tokens for 37 chunks
Made payment of 0.000552316 for 40 chunks
Stored wallet with cached payment proofs. New balance: 299.998238755
Uploaded chunk #76a7fd.. in 3 seconds
Uploaded chunk #98c790.. in 3 seconds
Uploaded chunk #70ff9b.. in 3 seconds
Uploaded chunk #2449bc.. in 5 seconds
Uploaded chunk #dafc18.. in 3 seconds
Uploaded chunk #44c529.. in 3 seconds
Uploaded chunk #98ad7b.. in 4 seconds
Uploaded chunk #06834b.. in 4 seconds
Uploaded chunk #1c1b39.. in 7 seconds
Uploaded chunk #7c5ac9.. in 3 seconds
Uploaded chunk #3736e8.. in 17 seconds
Uploaded chunk #27827e.. in 4 seconds
Uploaded chunk #292dcc.. in 17 seconds
Uploaded chunk #a096f1.. in 3 seconds
Uploaded chunk #f49535.. in 2 seconds
Uploaded chunk #39a946.. in 3 seconds
Uploaded chunk #458ae6.. in 3 seconds
Uploaded chunk #9c30d0.. in 3 seconds
Uploaded chunk #351e1f.. in 3 seconds
Uploaded chunk #8b6ec3.. in 31 seconds
Uploaded chunk #17f145.. in 3 seconds
Uploaded chunk #680018.. in 3 seconds
Uploaded chunk #311739.. in 3 seconds
Uploaded chunk #c17622.. in 3 seconds
Uploaded chunk #ecc19c.. in 3 seconds
Uploaded chunk #e14ff2.. in 3 seconds
Uploaded chunk #3959d9.. in 15 seconds
Uploaded chunk #1b2b12.. in 35 seconds
Uploaded chunk #a3e428.. in 29 seconds
Uploaded chunk #d4f03b.. in 26 seconds
Uploaded chunk #9efbad.. in 22 seconds
Uploaded chunk #84d8e5.. in 18 seconds
Uploaded chunk #7a2964.. in 14 seconds
Uploaded chunk #9abd12.. in 11 seconds
Uploaded chunk #5d1821.. in 7 seconds
Uploaded chunk #26bdc8.. in 3 seconds
Uploaded chunk #279853.. in 25 seconds
Uploaded chunk #ca480d.. in 2 seconds
Uploaded chunk #69ee86.. in 3 seconds
Uploaded chunk #797979.. in 3 seconds
Uploaded 40 chunks in 131.730995s. Current progress is 80/190.
Transfers applied locally
All transfers completed in 19.9657908s
Total payment: NanoTokens(1197955) nano tokens for 40 chunks
Made payment of 0.001197955 for 40 chunks
Stored wallet with cached payment proofs. New balance: 299.997040800
Uploaded chunk #f1ee73.. in 3 seconds
Uploaded chunk #81ce9e.. in 3 seconds
Uploaded chunk #a16066.. in 3 seconds
Uploaded chunk #6bc3f1.. in 3 seconds
Uploaded chunk #33286b.. in 14 seconds
Uploaded chunk #9c81cc.. in 3 seconds
Uploaded chunk #733830.. in 14 seconds
Uploaded chunk #33d6f0.. in 14 seconds
Uploaded chunk #46dc77.. in 3 seconds
Uploaded chunk #85f112.. in 3 seconds
Uploaded chunk #a1fa1d.. in 3 seconds
Uploaded chunk #8d7369.. in 3 seconds
Uploaded chunk #e8fce6.. in 3 seconds
Uploaded chunk #f6aadf.. in 3 seconds
Uploaded chunk #50f5f2.. in 3 seconds
Uploaded chunk #5782c6.. in 29 seconds
Uploaded chunk #34aece.. in 3 seconds
Uploaded chunk #311080.. in 22 seconds
Uploaded chunk #04037b.. in 3 seconds
Uploaded chunk #46c99e.. in 26 seconds
Uploaded chunk #1720b1.. in 3 seconds
Uploaded chunk #40e2a6.. in 3 seconds
Uploaded chunk #91a017.. in 26 seconds
Uploaded chunk #572786.. in 3 seconds
Uploaded chunk #94269c.. in 3 seconds
Uploaded chunk #99edf4.. in 26 seconds
Uploaded chunk #be88d9.. in 19 seconds
Uploaded chunk #2ee493.. in 19 seconds
Uploaded chunk #b75d8b.. in 15 seconds
Uploaded chunk #156663.. in 11 seconds
Uploaded chunk #4bdd29.. in 7 seconds
Uploaded chunk #d5d770.. in 30 seconds
Uploaded chunk #a5ef1b.. in 22 seconds
Uploaded chunk #421c01.. in 3 seconds
Uploaded chunk #62b30c.. in 19 seconds
Uploaded chunk #8ce49a.. in 11 seconds
Uploaded chunk #6d0d42.. in 7 seconds
Uploaded chunk #99ad08.. in 26 seconds
Uploaded chunk #094edf.. in 30 seconds
Uploaded chunk #43c469.. in 3 seconds
Uploaded 40 chunks in 123.7981329s. Current progress is 120/190.
Cannot get store cost for NetworkAddress::ChunkAddress(dd00f7(11011101).. - - dd00f709bac1cddb75876aae6559e4d57b06baab032334290fe916feaa480934) with error CouldNotSendMoney("Network Error Not enough store cost prices returned from the network to ensure a valid fee is paid.")
Cannot get store cost for NetworkAddress::ChunkAddress(bb919c(10111011).. - - bb919c1f4472996814861b5d96046c5e9be559c89996367fda1754c422a899b6) with error CouldNotSendMoney("Network Error Not enough store cost prices returned from the network to ensure a valid fee is paid.")
Cannot get store cost for NetworkAddress::ChunkAddress(dd81e7(11011101).. - - dd81e7576610eed5244961e8446627aa97c92f9b41c330d1a34874458574e91c) with error CouldNotSendMoney("Network Error Not enough store cost prices returned from the network to ensure a valid fee is paid.")
Cannot get store cost for NetworkAddress::ChunkAddress(062f32(00000110).. - - 062f32ee798dc58ab4aa06d27f86b6965e571f50fcdbea58271f9d6303aff4f0) with error CouldNotSendMoney("Network Error Not enough store cost prices returned from the network to ensure a valid fee is paid.")
Transfers applied locally
All transfers completed in 23.8443828s
Total payment: NanoTokens(1244769) nano tokens for 36 chunks
Made payment of 0.001244769 for 40 chunks
Stored wallet with cached payment proofs. New balance: 299.995796031
Uploaded chunk #6b93cf.. in 3 seconds
Uploaded chunk #eb81f1.. in 3 seconds
Uploaded chunk #40537c.. in 3 seconds
Uploaded chunk #1e7093.. in 3 seconds
Uploaded chunk #ef026d.. in 7 seconds
Uploaded chunk #275cb1.. in 3 seconds
Uploaded chunk #c2e835.. in 3 seconds
Uploaded chunk #b9ccb6.. in 3 seconds
Uploaded chunk #bcee0c.. in 13 seconds
Uploaded chunk #bccc45.. in 3 seconds
Uploaded chunk #d14218.. in 14 seconds
Uploaded chunk #e268d5.. in 14 seconds
Uploaded chunk #20cae1.. in 3 seconds
Uploaded chunk #164ef3.. in 13 seconds
Uploaded chunk #dd00f7.. in 2 seconds
Uploaded chunk #ceb15d.. in 3 seconds
Uploaded chunk #e30ecf.. in 3 seconds
Uploaded chunk #fb34a0.. in 3 seconds
Uploaded chunk #465ef6.. in 3 seconds
Uploaded chunk #a6e451.. in 3 seconds
Uploaded chunk #771c44.. in 3 seconds
Uploaded chunk #ee0837.. in 3 seconds
Uploaded chunk #984527.. in 3 seconds
Uploaded chunk #8d73dc.. in 3 seconds
Uploaded chunk #858947.. in 3 seconds
Uploaded chunk #dd81e7.. in 28 seconds
Uploaded chunk #99697e.. in 22 seconds
Uploaded chunk #93e6a6.. in 11 seconds
Uploaded chunk #09dc94.. in 7 seconds
Uploaded chunk #3c1f95.. in 24 seconds
Uploaded chunk #bb919c.. in 21 seconds
Uploaded chunk #6afaab.. in 20 seconds
Uploaded chunk #0cccfe.. in 28 seconds
Uploaded chunk #ff078d.. in 18 seconds
Uploaded chunk #330fc7.. in 14 seconds
Uploaded chunk #364f38.. in 10 seconds
Uploaded chunk #a6179d.. in 28 seconds
Uploaded chunk #062f32.. in 3 seconds
Uploaded chunk #99839c.. in 3 seconds
Uploaded chunk #603694.. in 3 seconds
Uploaded 40 chunks in 137.643782s. Current progress is 160/190.
Cannot get store cost for NetworkAddress::ChunkAddress(09379f(00001001).. - - 09379fad4ac1aee45fd74edbff26ac8f16c2b08e8748061b50dac9bd62b40a7b) with error CouldNotSendMoney("Network Error Not enough store cost prices returned from the network to ensure a valid fee is paid.")
Transfers applied locally
All transfers completed in 28.005338s
Total payment: NanoTokens(573601) nano tokens for 29 chunks
Made payment of 0.000573601 for 30 chunks
Stored wallet with cached payment proofs. New balance: 299.995222430
Uploaded chunk #e28cc1.. in 3 seconds
Uploaded chunk #dd0ea2.. in 3 seconds
Uploaded chunk #e44428.. in 3 seconds
Uploaded chunk #446fe3.. in 3 seconds
Uploaded chunk #287f82.. in 3 seconds
Uploaded chunk #1a04f2.. in 3 seconds
Uploaded chunk #fdb823.. in 3 seconds
Uploaded chunk #65a2e4.. in 14 seconds
Uploaded chunk #09379f.. in 14 seconds
Uploaded chunk #8d9edd.. in 4 seconds
Uploaded chunk #3192ec.. in 14 seconds
Uploaded chunk #4e73bb.. in 4 seconds
Uploaded chunk #53843d.. in 3 seconds
Uploaded chunk #7ce6e5.. in 3 seconds
Uploaded chunk #1fc4a3.. in 3 seconds
Uploaded chunk #6a5010.. in 3 seconds
Uploaded chunk #03653b.. in 3 seconds
Uploaded chunk #6c8039.. in 14 seconds
Uploaded chunk #c13863.. in 11 seconds
Uploaded chunk #0dae9a.. in 14 seconds
Uploaded chunk #6eabc9.. in 21 seconds
Uploaded chunk #f1cd7d.. in 18 seconds
Uploaded chunk #08574d.. in 18 seconds
Uploaded chunk #b9d6e5.. in 15 seconds
Uploaded chunk #f4e20a.. in 11 seconds
Uploaded chunk #c07a80.. in 3 seconds
Uploaded chunk #103c31.. in 7 seconds
Uploaded chunk #e7f2ca.. in 4 seconds
Uploaded chunk #40ccdb.. in 3 seconds
Uploaded chunk #9c1eb9.. in 2 seconds
Uploaded 30 chunks in 127.1653976s. Current progress is 190/190.
First round of upload complete. Verifying and repaying if required...
======= Verification: 190 chunks to be checked and repaid if required =============
After 169.7906244s, verified 190 chunks
======= Verification complete: all chunks paid and stored =============
Uploaded all chunks in 13 minutes 22 seconds
Uploaded VID_20210505_113014.mp4 to 9c1eb9226ab2bb720a3f79eeaa9adc76476eedc81cf82a7410296b9b7d0c113c
The standard upload resulted in all chunks being uploaded in 36 minutes 27 seconds, while the non-standard upload concurrency and batch-size -c 10 --batch-size 40, uploaded all chunks in 13 minutes 22 seconds.
10 Likes
A tale of two clients.
My home client is not phased by any size of upload, but my cloud client turns its nose up at the smallest files with the ‘could not retrieve the record’ error.
[2023-09-24T12:53:51.293038Z WARN sn_client::wallet] The storage payment transfer was not successfully registered in the network: CouldNotSendMoney("Network Error Could not retrieve the record after storing it: 369c18d93160540ca9897959a7e41a8848bad082757d776856d7e0ac8b77b748."). It will be retried later.
Anyone have any idea why that might be? Disk is 91% full - could that be it?
Edit - found lots of chunks cached in /tmp. Cleared those out and disk is now 75% full - but still getting the same error
7 Likes
Does cloud client have a shared CPU? Maybe need high -c value?
1 Like
Tried up to 50. No dice
1 Like
Another tale of two clients
My home node cannot upload anything - fails with deserialized bytes don't encode a group element.
willie@gagarin:~$ SN_LOG=ALL safe --log-output-dest=data-dir files upload -c50 whatyoulookingat.jpg
Logging to directory: "/home/willie/.local/share/safe/client/logs"
Using SN_LOG=ALL
Built with git version: e8392d1 / main / e8392d1
Instantiating a SAFE client...
🔗 Connected to the Network Total number of chunks to be stored: 4
Error:
0: Transfer Error Bincode error:: deserialized bytes don't encode a group element.
1: Bincode error:: deserialized bytes don't encode a group element
2: deserialized bytes don't encode a group element
Location:
sn_cli/src/subcommands/files.rs:195
Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
My cloud client does just fine
safe@ClientImprovementNet-Southside01:~$ SN_LOG=ALL safe --log-output-dest=data-dir files upload -c50 myenv/
Logging to directory: "/home/safe/.local/share/safe/client/logs"
Using SN_LOG=ALL
Built with git version: e8392d1 / main / e8392d1
Instantiating a SAFE client...
🔗 Connected to the Network Total number of chunks to be stored: 1
Transfers applied locally
All transfers completed in 14.672849589s
Total payment: NanoTokens(2290) nano tokens for 1 chunks
Made payment of 0.000002290 for 1 chunks
Stored wallet with cached payment proofs. New balance: 99.998769509
Uploaded chunk #a939bd.. in 2 seconds
Uploaded 1 chunks in 25.053079281s. Current progress is 1/1.
First round of upload complete. Verifying and repaying if required...
======= Verification: 1 chunks to be checked and repaid if required =============
After 1.427723484s, verified 1 chunks
======= Verification complete: all chunks paid and stored =============
Uploaded all chunks in 29 seconds
Uploaded pyvenv.cfg to a939bd46ac6b8e9ea38dc58384024da67a59f398f4c035da2c61bfde6384445f
Its cramping my upload style as I am running this cloud instance hard with 16 nodes and have little CPU/RAM headroom left for the heavy client work. Disk space is tight but I am about to buy another 10Gb.
6 Likes
if anyone fancy’s trying to download a 2Gb movie file
time safe files download crow.mkv 0bce3acee278c78bd59f9ad673c6ae743cd5094af22b485d9d62d86dff53f06a
3 Likes
iv lost some nodes running on my home machine which are doing nothing and giving the not enough peers in the k bucket error.
but most of the nodes on that machine are doing fine think it may be ISP or router related
1 Like
Crow Road? Iain Banks?
1 Like
Crow road it’s just off great western road
Its a dashcam movie from Partick up through Broomhill?!?!?!
To me (the) Crow Road runs up out of Lennoxtown through the Campsies to Fintry. As kids we used to spend hours pushing our bikes up that hill for 5 mins of severe terror back down into Lennoxtown 
I have tried downloading some of the files I have uploaded these days. While some, especially the smaller ones, have not given any problems, others have given errors.
Error downloading “test01.mkv”: Chunks error Not all chunks were retrieved, expected 750, retrieved 748, missing [e8f7b6(11101000)…, f84701(11111000)…]…
1 Like
Are you certain the upload went fine - didn’t fail partway through?
just tried a heavy handed experiment and restarted my home router to see what would happen to the nodes on it and it looks like the nodes came right back to life when the connection was restored after the reboot
10 Likes
I’ve done that twice during this testnet. WiFi was in a coma, and I did it to bring it back alive. (It worked).
Node didn’t mind, but client ended it’s ongoing upload there.
8 Likes
Great to see this working now wondering if it’s possible to restart the box the node is running on
Any one got any ideas how to try and do a hard restart ?
1 Like
I don’t think that’s possible at this point but expect it will be.
2 Likes
@joshuef I just tried using the SAFE_PEER from the OP and it isn’t working so changed to the last one in the OP list and that is fine.
4 Likes
Yeh something like this is theoretically in there… But clearly it’s broken 
Bad connection / slow upload / lots of payments going on to target nodes. It kiiind of doesnt matter, it should not fail out as you say.
The good news is you could just try again and (once everything is in place), you’d only be paying for the missing chunks price difference eg).
(Looking at the logs there, if it did not try and repay them again, there’s actually just a logic bug as it should loop over repay/reupload).
Juicy indeed.
Nothing specific. I’d check out dmesg output on that machine and look for Killed to find out why (if they did not just error out… in which case you should have the error printed?
we do (assuming some set variables here and rg installed on the target machine)
ssh root@${ip} 'dmesg | rg "Killed"' > "workspace/${TESTNET_CHANNEL}/droplets/${name}__${ip}/killed.log"
Ah, good to know. I’m grabbing some monday morning stats now so will update if needed thanks 
edit: we;ve lost only 4 nodes so far, and the peer from the OP is working for me. Can you shoot me failed logs perchance @happybeing ? Would be good to know what your client is seeing
7 Likes