Money rooling in 
8 Likes
A rough look without any real analysis shows that there is much more even distribution of records over nodes. This is my AWS based 50 since Thu Sep 28 22:50:02 UTC 2023:-
172
315
217
201
165
641
213
184
572
189
316
205
376
152
227
152
104
268
165
359
169
606
153
296
161
283
654
133
182
151
148
332
225
178
321
320
142
159
181
238
252
127
342
314
276
201
426
180
228
321
On previous testnets I’d have expected to see a couple with a 1000+, some with only a couple of dozen and a few with 0.
The earnings for each one are a much less even distribution:-
0.000003656
0.000223882
0.000002632
0.000002440
0.000006432
0.085579385
0.000015359
0.000004171
0.000453683
0.000004425
0.000008692
0.000002767
0.000251537
0.000003879
0.000012315
0.000009717
0.000001374
0.000026320
0.000011514
0.000029774
0.000003032
0.000117855
0.000005204
0.000003343
0.000010933
0.000008800
0.007634453
0.000001652
0.000002551
0.000006361
0.000004845
0.000067242
0.000005817
0.000013263
0.000011021
0.000275788
0.000008055
0.000009374
0.000018520
0.000004793
0.000015987
0.000001635
0.000043840
0.000197710
0.000020256
0.000004145
0.000473851
0.000018073
0.000002458
0.000018637
I’m logging metrics with @Josh 's
SN-StatsNTracking
https://github.com/javages/SN-StatsNTracking
So I’ll try drawing the graphs now!
11 Likes
Ooooh! Much niceness with @Josh 's graphs!
The big purple one is my retirement plan!
It’s interesting to see the RAM usage of nodes trending downwards a bit. I don’t think we’ve seen that before.
13 Likes
any one fancies downloading an ubuntu image took a long time but i got there
safe files download ubuntu-23.04-desktop-amd64.iso cd5b376487b0dea610ba8de1e073341c3304e0bfd18d7bc6d65e9ddc2227d014
======= Verification complete: all chunks paid and stored =============
Uploaded all chunks in 1785 minutes 38 seconds
Uploaded ubuntu-23.04-desktop-amd64.iso to cd5b376487b0dea610ba8de1e073341c3304e0bfd18d7bc6d65e9ddc2227d014
9 Likes
Here is comparison with one of your older results (from this message).
(I just excluded “chunkless nodes” from older result to see shape better)
For me it looks like older result was more uniform. New result is more “spikey”.
2 Likes
Three attempts, and three identical errors with chunk #938
Client (read all) download progress 938/9408
Error downloading "ubuntu-23.04-desktop-amd64.iso":
Chunks error Chunk could not be retrieved from the network: c363a5(11000011)...
Did your upload have any errors?
5 Likes
Up load was a night mare kept failing to fetch chunks and retrying 
thought it got there in the end when it said
======= Verification complete: all chunks paid and stored =============
Uploaded all chunks in 1785 minutes 38 seconds
Uploaded ubuntu-23.04-desktop-amd64.iso to cd5b376487b0dea610ba8de1e073341c3304e0bfd18d7bc6d65e9ddc2227d014
======= Verification: 74 chunks to be checked and repaid if required =============
Failed to fetch a chunk ChunkAddress(e86f92(11101000)..)
Failed to fetch a chunk ChunkAddress(2c698d(00101100)..)
Failed to fetch a chunk ChunkAddress(fcfc58(11111100)..)
Failed to fetch a chunk ChunkAddress(301bdd(00110000)..)
Failed to fetch a chunk ChunkAddress(2a5a36(00101010)..)
Failed to fetch a chunk ChunkAddress(11d253(00010001)..)
Failed to fetch a chunk ChunkAddress(7e687a(01111110)..)
Failed to fetch a chunk ChunkAddress(2dcde9(00101101)..)
Failed to fetch a chunk ChunkAddress(357973(00110101)..)
Failed to fetch a chunk ChunkAddress(81389e(10000001)..)
Failed to fetch a chunk ChunkAddress(056675(00000101)..)
Failed to fetch a chunk ChunkAddress(80dfb9(10000000)..)
Failed to fetch a chunk ChunkAddress(198fbe(00011001)..)
Failed to fetch a chunk ChunkAddress(b6bd5e(10110110)..)
Failed to fetch a chunk ChunkAddress(7f0ac4(01111111)..)
Failed to fetch a chunk ChunkAddress(3e0757(00111110)..)
Failed to fetch a chunk ChunkAddress(f89426(11111000)..)
Failed to fetch a chunk ChunkAddress(1f8c9d(00011111)..)
Failed to fetch a chunk ChunkAddress(ffdb63(11111111)..)
Failed to fetch a chunk ChunkAddress(92ebe7(10010010)..)
Failed to fetch a chunk ChunkAddress(5e2e6f(01011110)..)
Failed to fetch a chunk ChunkAddress(87423d(10000111)..)
Failed to fetch a chunk ChunkAddress(18e8d8(00011000)..)
Failed to fetch a chunk ChunkAddress(e7baf6(11100111)..)
Failed to fetch a chunk ChunkAddress(58ddae(01011000)..)
Failed to fetch a chunk ChunkAddress(bef5bc(10111110)..)
Failed to fetch a chunk ChunkAddress(94b6b3(10010100)..)
Failed to fetch a chunk ChunkAddress(06271b(00000110)..)
Failed to fetch a chunk ChunkAddress(60a762(01100000)..)
Failed to fetch a chunk ChunkAddress(a3290c(10100011)..)
Failed to fetch a chunk ChunkAddress(ff19c1(11111111)..)
Failed to fetch a chunk ChunkAddress(036c49(00000011)..)
Failed to fetch a chunk ChunkAddress(7ced4c(01111100)..)
Failed to fetch a chunk ChunkAddress(18396e(00011000)..)
Failed to fetch a chunk ChunkAddress(c4f927(11000100)..)
Failed to fetch a chunk ChunkAddress(6cc321(01101100)..)
Failed to fetch a chunk ChunkAddress(e97c8f(11101001)..)
Failed to fetch a chunk ChunkAddress(738ed5(01110011)..)
Failed to fetch a chunk ChunkAddress(0eb9c5(00001110)..)
Failed to fetch a chunk ChunkAddress(15af40(00010101)..)
Failed to fetch a chunk ChunkAddress(f8408e(11111000)..)
Failed to fetch a chunk ChunkAddress(cdaf80(11001101)..)
Failed to fetch a chunk ChunkAddress(90c759(10010000)..)
Failed to fetch a chunk ChunkAddress(2a1b6f(00101010)..)
Failed to fetch a chunk ChunkAddress(a0017a(10100000)..)
Failed to fetch a chunk ChunkAddress(90af22(10010000)..)
Failed to fetch a chunk ChunkAddress(e90daf(11101001)..)
Failed to fetch a chunk ChunkAddress(ffd645(11111111)..)
After 7.000831689s, verified 74 chunks
======= Verification: 48 chunks were not stored. Repaying them in batches. =============
Failed to fetch 48 chunks. Attempting to repay them.
Transfers applied locally
All transfers completed in 13.302774369s
Total payment: NanoTokens(1582677) nano tokens for 48 chunks
Uploaded chunk #ff19c1.. in 4 minutes 17 seconds
Uploaded chunk #2c698d.. in 5 minutes 42 seconds
Uploaded chunk #301bdd.. in 5 minutes 42 seconds
Uploaded chunk #58ddae.. in 1 minutes 25 seconds
Uploaded chunk #fcfc58.. in 5 minutes 42 seconds
Uploaded chunk #92ebe7.. in 3 minutes 48 seconds
Uploaded chunk #7e687a.. in 3 minutes 48 seconds
Uploaded chunk #15af40.. in 3 minutes 48 seconds
Uploaded chunk #e86f92.. in 5 minutes 42 seconds
Uploaded chunk #90af22.. in 57 seconds
Uploaded chunk #2a5a36.. in 5 minutes 14 seconds
Uploaded chunk #036c49.. in 3 minutes 48 seconds
Uploaded chunk #7f0ac4.. in 57 seconds
Uploaded chunk #1f8c9d.. in 5 minutes 14 seconds
Uploaded chunk #f8408e.. in 3 minutes 20 seconds
Uploaded chunk #06271b.. in 5 minutes 14 seconds
Uploaded chunk #3e0757.. in 57 seconds
Uploaded chunk #5e2e6f.. in 3 minutes 20 seconds
Uploaded chunk #2dcde9.. in 3 minutes 20 seconds
Uploaded chunk #e90daf.. in 28 seconds
Uploaded chunk #b6bd5e.. in 28 seconds
Uploaded chunk #60a762.. in 5 minutes 14 seconds
Uploaded chunk #f89426.. in 28 seconds
Uploaded chunk #198fbe.. in 57 seconds
Uploaded chunk #7ced4c.. in 3 minutes 20 seconds
Uploaded chunk #e97c8f.. in 4 minutes 45 seconds
Uploaded chunk #cdaf80.. in 2 minutes 51 seconds
Uploaded chunk #c4f927.. in 2 minutes 23 seconds
Uploaded chunk #738ed5.. in 4 minutes 45 seconds
Uploaded chunk #2a1b6f.. in 1 minutes 54 seconds
Uploaded chunk #87423d.. in 2 minutes 51 seconds
Uploaded chunk #6cc321.. in 4 minutes 45 seconds
Uploaded chunk #e7baf6.. in 1 minutes 54 seconds
Uploaded chunk #357973.. in 2 minutes 51 seconds
Uploaded chunk #056675.. in 1 minutes 54 seconds
Uploaded chunk #ffd645.. in 28 seconds
Uploaded chunk #90c759.. in 2 minutes 23 seconds
Uploaded chunk #18e8d8.. in 2 minutes 23 seconds
Uploaded chunk #18396e.. in 2 minutes 51 seconds
Uploaded chunk #11d253.. in 4 minutes 17 seconds
Uploaded chunk #ffdb63.. in 4 minutes 17 seconds
Uploaded chunk #a3290c.. in 4 minutes 46 seconds
Uploaded chunk #94b6b3.. in 1 minutes 26 seconds
Uploaded chunk #a0017a.. in 1 minutes 26 seconds
Uploaded chunk #bef5bc.. in 1 minutes 54 seconds
Uploaded chunk #81389e.. in 2 minutes 23 seconds
Uploaded chunk #0eb9c5.. in 4 minutes 17 seconds
Uploaded chunk #80dfb9.. in 1 minutes 26 seconds
Repaid and re-uploaded 48 chunks in 385.582448515s
======= Verification: 48 chunks to be checked and repaid if required =============
Failed to fetch a chunk ChunkAddress(e86f92(11101000)..)
Failed to fetch a chunk ChunkAddress(2c698d(00101100)..)
Failed to fetch a chunk ChunkAddress(fcfc58(11111100)..)
Failed to fetch a chunk ChunkAddress(301bdd(00110000)..)
Failed to fetch a chunk ChunkAddress(2a5a36(00101010)..)
Failed to fetch a chunk ChunkAddress(11d253(00010001)..)
Failed to fetch a chunk ChunkAddress(7e687a(01111110)..)
Failed to fetch a chunk ChunkAddress(2dcde9(00101101)..)
Failed to fetch a chunk ChunkAddress(357973(00110101)..)
Failed to fetch a chunk ChunkAddress(81389e(10000001)..)
Failed to fetch a chunk ChunkAddress(056675(00000101)..)
Failed to fetch a chunk ChunkAddress(80dfb9(10000000)..)
Failed to fetch a chunk ChunkAddress(198fbe(00011001)..)
Failed to fetch a chunk ChunkAddress(b6bd5e(10110110)..)
Failed to fetch a chunk ChunkAddress(7f0ac4(01111111)..)
Failed to fetch a chunk ChunkAddress(3e0757(00111110)..)
Failed to fetch a chunk ChunkAddress(f89426(11111000)..)
Failed to fetch a chunk ChunkAddress(1f8c9d(00011111)..)
Failed to fetch a chunk ChunkAddress(ffdb63(11111111)..)
Failed to fetch a chunk ChunkAddress(92ebe7(10010010)..)
Failed to fetch a chunk ChunkAddress(5e2e6f(01011110)..)
Failed to fetch a chunk ChunkAddress(87423d(10000111)..)
Failed to fetch a chunk ChunkAddress(18e8d8(00011000)..)
Failed to fetch a chunk ChunkAddress(e7baf6(11100111)..)
Failed to fetch a chunk ChunkAddress(58ddae(01011000)..)
Failed to fetch a chunk ChunkAddress(bef5bc(10111110)..)
Failed to fetch a chunk ChunkAddress(94b6b3(10010100)..)
Failed to fetch a chunk ChunkAddress(06271b(00000110)..)
Failed to fetch a chunk ChunkAddress(60a762(01100000)..)
Failed to fetch a chunk ChunkAddress(a3290c(10100011)..)
Failed to fetch a chunk ChunkAddress(ff19c1(11111111)..)
Failed to fetch a chunk ChunkAddress(036c49(00000011)..)
Failed to fetch a chunk ChunkAddress(18396e(00011000)..)
Failed to fetch a chunk ChunkAddress(c4f927(11000100)..)
Failed to fetch a chunk ChunkAddress(6cc321(01101100)..)
Failed to fetch a chunk ChunkAddress(e97c8f(11101001)..)
Failed to fetch a chunk ChunkAddress(738ed5(01110011)..)
Failed to fetch a chunk ChunkAddress(0eb9c5(00001110)..)
Failed to fetch a chunk ChunkAddress(15af40(00010101)..)
Failed to fetch a chunk ChunkAddress(f8408e(11111000)..)
Failed to fetch a chunk ChunkAddress(cdaf80(11001101)..)
Failed to fetch a chunk ChunkAddress(90c759(10010000)..)
Failed to fetch a chunk ChunkAddress(2a1b6f(00101010)..)
Failed to fetch a chunk ChunkAddress(a0017a(10100000)..)
Failed to fetch a chunk ChunkAddress(90af22(10010000)..)
Failed to fetch a chunk ChunkAddress(e90daf(11101001)..)
Failed to fetch a chunk ChunkAddress(ffd645(11111111)..)
After 13.224102234s, verified 48 chunks
======= Verification: 47 chunks were not stored. Repaying them in batches. =============
Failed to fetch 47 chunks. Attempting to repay them.
Transfers applied locally
All transfers completed in 6.32748035s
Total payment: NanoTokens(1653740) nano tokens for 47 chunks
Uploaded chunk #2dcde9.. in 2 minutes 52 seconds
Uploaded chunk #18e8d8.. in 2 minutes 52 seconds
Uploaded chunk #18396e.. in 2 minutes 52 seconds
Uploaded chunk #bef5bc.. in 2 minutes 23 seconds
Uploaded chunk #357973.. in 2 minutes 23 seconds
Uploaded chunk #94b6b3.. in 2 minutes 23 seconds
Uploaded chunk #c4f927.. in 2 minutes 23 seconds
Uploaded chunk #e7baf6.. in 1 minutes 54 seconds
Uploaded chunk #81389e.. in 1 minutes 54 seconds
Uploaded chunk #06271b.. in 1 minutes 54 seconds
Uploaded chunk #6cc321.. in 1 minutes 54 seconds
Uploaded chunk #58ddae.. in 1 minutes 26 seconds
Uploaded chunk #056675.. in 1 minutes 26 seconds
Uploaded chunk #60a762.. in 1 minutes 26 seconds
Uploaded chunk #e97c8f.. in 1 minutes 26 seconds
Uploaded chunk #738ed5.. in 57 seconds
Uploaded chunk #80dfb9.. in 57 seconds
Uploaded chunk #b6bd5e.. in 57 seconds
Uploaded chunk #198fbe.. in 57 seconds
Uploaded chunk #0eb9c5.. in 29 seconds
Uploaded chunk #7f0ac4.. in 29 seconds
Uploaded chunk #15af40.. in 29 seconds
Uploaded chunk #90af22.. in 29 seconds
Uploaded chunk #3e0757.. in 4 minutes 18 seconds
Uploaded chunk #fcfc58.. in 4 minutes 18 seconds
Uploaded chunk #92ebe7.. in 4 minutes 18 seconds
Uploaded chunk #11d253.. in 4 minutes 18 seconds
Uploaded chunk #5e2e6f.. in 4 minutes 18 seconds
Uploaded chunk #ff19c1.. in 3 minutes 49 seconds
Uploaded chunk #a3290c.. in 3 minutes 49 seconds
Uploaded chunk #7e687a.. in 3 minutes 49 seconds
Uploaded chunk #87423d.. in 3 minutes 49 seconds
Uploaded chunk #036c49.. in 3 minutes 21 seconds
Uploaded chunk #e90daf.. in 28 seconds
Uploaded chunk #2a1b6f.. in 28 seconds
Uploaded chunk #90c759.. in 28 seconds
Uploaded chunk #2a5a36.. in 5 minutes 15 seconds
Uploaded chunk #301bdd.. in 5 minutes 15 seconds
Uploaded chunk #e86f92.. in 5 minutes 15 seconds
Uploaded chunk #2c698d.. in 5 minutes 16 seconds
Uploaded chunk #1f8c9d.. in 4 minutes 47 seconds
Uploaded chunk #f89426.. in 4 minutes 47 seconds
Uploaded chunk #ffdb63.. in 4 minutes 47 seconds
Uploaded chunk #ffd645.. in 28 seconds
Uploaded chunk #f8408e.. in 28 seconds
Uploaded chunk #a0017a.. in 28 seconds
Uploaded chunk #cdaf80.. in 28 seconds
Repaid and re-uploaded 47 chunks in 380.654256415s
======= Verification: 47 chunks to be checked and repaid if required =============
After 4.274545908s, verified 47 chunks
======= Verification complete: all chunks paid and stored =============
Uploaded all chunks in 1785 minutes 38 seconds
Uploaded ubuntu-23.04-desktop-amd64.iso to cd5b376487b0dea610ba8de1e073341c3304e0bfd18d7bc6d65e9ddc2227d014
4 Likes
For me it looks like older result was more uniform. New result is more “spikey”.
That is very interesting @Vort so thank you for doing that. Maybe it’s not as even as I thought. It’s got to be good that there are none with 0 though.
Those results were also from about a day and a bit into a network run so the age of the nodes is roughly comparable which is good. I wonder if this one might even out a bit as time goes by if it keeps running for a few days.
4 Likes
I decided to also calculate Mean[] and StandardDeviation[].
For new data mean is 258, standard deviation is 131.
For old data (excluding values < 16), mean is 573, standard deviation is 277.
Turned out standard deviation is approximately equal to 1/2 of mean in both cases.
Probably results are not much different in fact.
4 Likes
Got the same issue with 938
3 Likes
Client (read all) download progress 1001/9408
Error downloading "ubuntu-23.04-desktop-amd64.iso": Chunks error Chunk could not be retrieved from the network: 7becb0(01111011)...
Keep the log!
1 Like
Has any one got any tips on how to figure out which log it it is from all the logs in the client file ?
1 Like
If you still have the terminal window open, it will tell you the name of the file it is going to write on every command. It’s there soon after the command line itself.
2 Likes
it dosent scroll that far back
4 Likes
Has happened to me too 
1 Like
fail on @aatonnomicc’s ubuntu image
Client download progress 9407/9408
Client download progress 9408/9408
Client downloaded file in 6642.510147277s
Error downloading "ubuntu-23.04-desktop-amd64.iso": Chunks error Not all chunks were retrieved, expected 9408, retrieved 9390, missing [5495b8(01010100).., 7becb0(01111011).., ab542f(10101011).., 7631ad(01110110).., 21935a(00100001).., b4d3f4(10110100).., 6c962f(01101100).., e405ab(11100100).., cdff04(11001101).., 50721a(01010000).., 875a98(10000111).., b77c52(10110111).., 1fc115(00011111).., e0455d(11100000).., 15ae45(00010101).., 26a155(00100110).., 2412a3(00100100).., b04f1a(10110000)..]..
willie@gagarin:~$
durrrrr… I thought the latest client was automatically keeping logs…
willie@gagarin:~$ cd safehome/client/logs/
willie@gagarin:~/safehome/client/logs$ ll
total 0
-rw-rw-r-- 1 willie willie 0 Sep 24 00:40 safe.log
safehome is an alias for ~/.local/share/safe BTW
3 Likes
It is but it’s figuring out which one
1 Like
Where are they then?
Naive me thought ~/.local/share/safe/client/logs would be where to look for upload logs.
They are there. But in directories within that:-
pwd
/home/ubuntu/.local/share/safe/client/logs
ubuntu@ip-172-30-1-42:~/.local/share/safe/client/logs$ ls | tail
log_2023-09-30_13-05-21
log_2023-09-30_13-36-12
log_2023-09-30_14-10-10
log_2023-09-30_14-35-14
log_2023-09-30_15-06-22
log_2023-09-30_15-43-42
log_2023-09-30_16-11-17
log_2023-09-30_17-00-56
log_2023-09-30_17-36-48
log_2023-09-30_17-57-55
ubuntu@ip-172-30-1-42:~/.local/share/safe/client/logs$ cd log_2023-09-30_17-57-55
ubuntu@ip-172-30-1-42:~/.local/share/safe/client/logs/log_2023-09-30_17-57-55$ ls
safe.log
ubuntu@ip-172-30-1-42:~/.local/share/safe/client/logs/log_2023-09-30_17-57-55$ tail safe.log
[2023-09-30T18:13:01.566808Z ERROR sn_networking] RecordNotFound
[2023-09-30T18:13:01.566811Z WARN sn_networking] Did not retrieve Record '44f3b578b4f49713ae2dd15ba3759e7b0e61ae58a27f8401129162ffa7aa5649' from network!. Retrying...
[2023-09-30T18:13:04.567897Z INFO sn_networking] Getting record of 44f3b578b4f49713ae2dd15ba3759e7b0e61ae58a27f8401129162ffa7aa5649 attempts 2/3
[2023-09-30T18:13:04.888302Z DEBUG sn_logging::metrics] {"physical_cpu_threads":2,"system_cpu_usage_percent":0.6024096,"system_total_memory_mb":961.9497,"system_memory_used_mb":366.90738,"system_memory_usage_percent":38.142056,"process":{"cpu_usage_percent":1.8072288,"memory_used_mb":68.558846,"bytes_read":0,"bytes_written":0,"total_mb_read":0.098304,"total_mb_written":56.737793}}
[2023-09-30T18:13:05.726008Z INFO sn_networking] Record returned: 44f3b578b4f49713ae2dd15ba3759e7b0e61ae58a27f8401129162ffa7aa5649
[2023-09-30T18:13:05.726036Z INFO sn_client::wallet] Spend has completed: Ok(())
[2023-09-30T18:13:05.727746Z TRACE sn_client::file_apis/get_local_payment_and_upload_chunk] Client upload started for chunk: NetworkAddress::ChunkAddress(ef3739(11101111).. - - ef3739ecfdc6649bcc60ca01cf6ca8014b97d67a6ffaa4616a2ad985316a3b76)
[2023-09-30T18:13:09.837104Z DEBUG sn_client::api] Client inactivity... waiting for a network event
[2023-09-30T18:13:09.889853Z DEBUG sn_logging::metrics] {"physical_cpu_threads":2,"system_cpu_usage_percent":42.0,"system_total_memory_mb":961.9497,"system_memory_used_mb":370.2538,"system_memory_usage_percent":38.489933,"process":{"cpu_usage_percent":83.8,"memory_used_mb":77.2096,"bytes_read":0,"bytes_written":4096,"total_mb_read":0.098304,"total_mb_written":56.741886}}
[2023-09-30T18:13:14.891581Z DEBUG sn_logging::metrics] {"physical_cpu_threads":2,"system_cpu_usage_percent":50.1,"system_total_memory_mb":961.9497,"system_memory_used_mb":373.98938,"system_memory_usage_percent":38.87827,"process":{"cpu_usage_percent":99.8,"memory_used_mb":79.55661,"bytes_read":0,"bytes_written":0,"total_mb_read":0.098304,"total_mb_written":56.741886}}
2 Likes
But not here …
willie@gagarin:~/safehome/client/logs$ tree
.
└── safe.log
ELIF me again on this client logging automagically without the use of SN_LOG=all
Please do this quietly and methodically as I have a hangover that would kill a small sheep.
Well done the sheepies BTW <–Scottish fitba joke
1 Like