IntolerantNodeNet [18/09/23 Testnet] [ Offline ]

Development Releases
41

In further moaning…

After 56.648375908s, uploaded 40 chunks, current progress is 80/301. 
Uploaded chunk #b68b94.. in 2 seconds
Uploaded chunk #e9df5d.. in 1 seconds
Uploaded chunk #3bef09.. in 6 seconds
Uploaded chunk #60eee3.. in 1 seconds
Uploaded chunk #49282a.. in 9 seconds
Uploaded chunk #51be20.. in 1 seconds
Uploaded chunk #c38eef.. in 14 seconds
Uploaded chunk #f220c2.. in 1 seconds
Uploaded chunk #fce48f.. in 1 seconds
Uploaded chunk #3f2fa0.. in 15 seconds
Uploaded chunk #d08dbc.. in 18 seconds
Uploaded chunk #9b8d7b.. in 1 seconds
Uploaded chunk #1d271a.. in 1 seconds
Uploaded chunk #a60aac.. in 21 seconds
Uploaded chunk #1fdc05.. in 1 seconds
Uploaded chunk #5d55af.. in 1 seconds
Uploaded chunk #7f44a8.. in 1 seconds
Uploaded chunk #4eced6.. in 27 seconds
Uploaded chunk #72eec4.. in 1 seconds
Uploaded chunk #621d16.. in 1 seconds
Uploaded chunk #9d0cd7.. in 1 seconds
Uploaded chunk #d4c99c.. in 31 seconds
Uploaded chunk #caa1e7.. in 1 seconds
Uploaded chunk #62d8e6.. in 1 seconds
Uploaded chunk #685e59.. in 1 seconds
Uploaded chunk #5d461b.. in 33 seconds
Uploaded chunk #a6d747.. in 32 seconds
Uploaded chunk #b5e5b6.. in 30 seconds
Uploaded chunk #549b04.. in 26 seconds
Uploaded chunk #e8a1ce.. in 25 seconds
Uploaded chunk #d18243.. in 24 seconds
Uploaded chunk #b0d051.. in 23 seconds
Uploaded chunk #57610a.. in 22 seconds
Uploaded chunk #5b114d.. in 20 seconds
Uploaded chunk #012192.. in 17 seconds
Uploaded chunk #5b6373.. in 16 seconds
Uploaded chunk #f463aa.. in 13 seconds
Uploaded chunk #d262ba.. in 12 seconds
Uploaded chunk #5167c4.. in 10 seconds
Uploaded chunk #100680.. in 6 seconds

These times bear no relation to reality, each chunk seemed to take roughly the same time - 1-2 seconds each

2 Likes
42

Sharing what is happening with my nodes here.
Still ironing out creases but it seems to be working well enough.

(updates once every 20 min past the hour)

7 Likes
43

Had to take my Brave shields down to see these graphs.

image
image1920×1080 414 KB

I feel unsafe and will now go back under my shell

5 Likes
44

Probably it’s a messages from upload finish event and you don’t see upload start event to measure difference by yourself.

By the way, uploaded 40 chunks message looks wrong. It looks like it’s a point when upload is only started, not finished.

1 Like
45

Sorry - that was misleading, I grabbed a that paste out of a long list of uploaded chunks. Perhaps this paste illustrates it better?

Uploaded chunk #11b8f6.. in 2 seconds
Uploaded chunk #a453e2.. in 2 seconds
Uploaded chunk #7803cd.. in 2 seconds
Uploaded chunk #ca4e77.. in 2 seconds
Uploaded chunk #6c014b.. in 52 seconds
Uploaded chunk #709603.. in 2 seconds
Uploaded chunk #34ccfb.. in 1 seconds
Uploaded chunk #089f18.. in 1 seconds
Uploaded chunk #00827e.. in 1 seconds
Uploaded chunk #a15a92.. in 58 seconds
Uploaded chunk #72addc.. in 56 seconds
Uploaded chunk #f64f05.. in 54 seconds
Uploaded chunk #6de2b9.. in 52 seconds
Uploaded chunk #18544f.. in 48 seconds
Uploaded chunk #23932e.. in 46 seconds
Uploaded chunk #6e1094.. in 44 seconds
Uploaded chunk #931948.. in 42 seconds
Uploaded chunk #53beda.. in 38 seconds
Uploaded chunk #2c0511.. in 31 seconds
Uploaded chunk #9724e9.. in 27 seconds
Transfers applied locally
After 13.799855449s, All transfers made for total payment of Token(12421) nano tokens for 40 chunks. 
Successfully made payment of 0.000012421 for 40 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999856941.
After 111.007721135s, uploaded 40 chunks, current progress is 200/334.

Note that batch (of which I have only show a part) is alleged to be at the 111secs mark, but if you add up the times claimed for each chunk, its a LOT more than 111 secs :slight_smile:

Also note that we suddenly get a relatively large time for a chunk, then subsequent chunks are generally slightly smaller. Dunno if thats a wee clue to squash this bug…

ALSO - just where are the client logs stored? not seeing them in /local/share/safe/client

safe@IntolerantNetSouthside01:~/.local/share/safe/client$ ll
total 20
drwxrwxr-x 4 safe safe 4096 Sep 18 14:59 ./
drwxrwxr-x 5 safe safe 4096 Sep 18 14:40 ../
-rw-rw-r-- 1 safe safe   64 Sep 18 14:58 clientkey
drwxrwxr-x 2 safe safe 4096 Sep 18 16:43 uploaded_files/
drwxrwxr-x 4 safe safe 4096 Sep 18 14:44 wallet/
1 Like
46

agreed - my uploads from the cloud clients seem to be less error prone.

1 Like
47

What concurrency means is that uploads happen simultaneously.

1 Like
48

Yes I get that, however some of the times claimed for each chunk are greater than the total time for the batch.
Its not a showstopper but there is work to be done yet on displaying these times correctly - which may or may not have any long-term real world use in any case.

What IS encouraging is the retry of the failed chunks and the subsequent iterations until all are verified

here is the end of a largish upload

1 Like
49

What I saw and what I said in one of my previous mesages is that current progress messages looks desynced.
In your example, current progress is 334/334 is displayed and then progress messages Uploaded chunk #8c7a46, Uploaded chunk #e8815b, … continue to appear.

As for time: looks like you agree that times can’t be simply summed.
Which means that individual messages should be analyzed.
Do you have example when individual Uploaded chunk took more than whole batch?

2 Likes
50 
safe@IntolerantNetSouthside01:~/.local/share/safe/node$ safe files upload -c30 --batch-size 40 12D3KooWLWfvU6ToNqgSXsNheoVUmwHPZokVVtLyuhNSKAvyX2UZ/
Built with git version: 26c3d70 / main / 26c3d70
Instantiating a SAFE client...
🔗 Connected to the Network                                                                                                                                                  Total number of chunks to be stored: 334
Transfers applied locally
After 15.77442455s, All transfers made for total payment of Token(11340) nano tokens for 40 chunks. 
Successfully made payment of 0.000011340 for 40 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999903786.
After 28.731796235s, uploaded 40 chunks, current progress is 40/334. 
Uploaded chunk #365c12.. in 2 seconds
Uploaded chunk #c9f6d2.. in 3 seconds
Uploaded chunk #3901f0.. in 3 seconds
Uploaded chunk #0353cf.. in 4 seconds
Uploaded chunk #1104cc.. in 6 seconds
Uploaded chunk #e76263.. in 7 seconds
Uploaded chunk #9f1f60.. in 1 seconds
Uploaded chunk #0e82e0.. in 1 seconds
Uploaded chunk #cdfb65.. in 9 seconds
Uploaded chunk #62c2a3.. in 1 seconds
Uploaded chunk #049293.. in 11 seconds
Uploaded chunk #01d891.. in 12 seconds
Uploaded chunk #9ade0f.. in 1 seconds
Uploaded chunk #b17946.. in 1 seconds
Uploaded chunk #37c2e2.. in 15 seconds
Uploaded chunk #12c954.. in 1 seconds
Uploaded chunk #ee116d.. in 1 seconds
Uploaded chunk #65227f.. in 1 seconds
Uploaded chunk #17a48d.. in 1 seconds
Uploaded chunk #0f21f7.. in 16 seconds
Uploaded chunk #23b419.. in 1 seconds
Uploaded chunk #29bc53.. in 1 seconds
Uploaded chunk #1373bc.. in 17 seconds
Uploaded chunk #7fa4c7.. in 1 seconds
Uploaded chunk #8eee92.. in 1 seconds
Uploaded chunk #2cfc3d.. in 1 seconds
Uploaded chunk #cbf004.. in 1 seconds
Uploaded chunk #f23c47.. in 1 seconds
Uploaded chunk #20c2d6.. in 1 seconds
Uploaded chunk #2d70a2.. in 1 seconds
Uploaded chunk #c26fda.. in 1 seconds
Uploaded chunk #c1f286.. in 1 seconds
Uploaded chunk #30f176.. in 1 seconds
Uploaded chunk #2ef7ca.. in 1 seconds
Uploaded chunk #e76323.. in 1 seconds
Uploaded chunk #0e8bc4.. in 33 seconds
Uploaded chunk #de4449.. in 1 seconds
Uploaded chunk #ba27cc.. in 1 seconds
Uploaded chunk #a7adb7.. in 33 seconds
Uploaded chunk #228e4e.. in 24 seconds
Transfers applied locally
After 14.771578658s, All transfers made for total payment of Token(10515) nano tokens for 40 chunks. 
Successfully made payment of 0.000010515 for 40 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999893271.

Am I reading this incorrectly?
Should the summary times above say exactly what it is that is summarised?

  • Total time since command initiated?
  • Total time to store the current batch?
51

After 28.731796235s, uploaded 40 chunks, current progress is 40/334 message can’t measure future events. Which means this time is related to some events before upload is started.
After 14.771578658s, All transfers made probably have measurements only for payment part, which means that next message after this one may have measurements for the whole batch.

Probably it is better to wait for clarifications/fixes, it is hard to make conclusions from what we have now.

52

I suspect that client may ignore values larger than CPU core count.
If it is the case, such behaviour should be documented.
If I set -c 100, I expect it to be exactly 100.

1 Like
53

vdash now has a Storage Cost timeline showing min, mean and maximum values:

10 Likes
54

I can successfully put small files but anything over 10MB is throwing Not enough store cost quotes at me.

Concurrency is limitted by thread count correct? Is there a future where gpu’s can be used to blast uploads onto the network?

1 Like
55

Thank you for the updated version.

image
image1920×1080 289 KB

On various nodes I see Status shown as one of Connected, Started and Stopped. I think I understand Stopped but am confused about the difference between Connected and Started.

image
image1920×1080 169 KB

image
image1920×1080 300 KB

Think I have sussed it

Connected → the node is working OK
Stopped → the node can see other nodes but is not sure how to connect
Started → the safenode command was issued but the node hasnt quite plucked up the courage to try talking to other nodes yet.

EDIT: I am not entirely convinced I do have it sussed. Why does the Stopped screenshot above show that a storage fee was charged? Or does it signify that a storage fee was announced/decided upon?

Also noted that Chunk Storage is consistently 0 B across all nodes but I am storing chunks in several of my nodes.

safe@IntolerantNetSouthside01:~$ du -h ~/.local/share/safe/node/*/record_store
38M	/home/safe/.local/share/safe/node/12D3KooWA32YjMNhx8J9d39hCAwJsZTvtCjKC3G6fCqsbiLFHsmo/record_store
18M	/home/safe/.local/share/safe/node/12D3KooWByVBk1bzC1w2deS8BPAokPHKiS1WRK7ghxydpMPSCspF/record_store
9.1M	/home/safe/.local/share/safe/node/12D3KooWD9KdRb16M8xQhN4HLPMxoBfdzNGZG13Dv4EHbivKEHBZ/record_store
4.0K	/home/safe/.local/share/safe/node/12D3KooWEjKwH7wPDEhGmtifBC4dEEMd8Gyrdue5P8PdUPDVoWkD/record_store
19M	/home/safe/.local/share/safe/node/12D3KooWFvGeR8eDvNnoWinX21boJkZNmuL9Zw4TAGpKCDMszH5e/record_store
30M	/home/safe/.local/share/safe/node/12D3KooWGmUxkdYjst1goLYnamZWgnoXNp2Gwzz18mfK9y3ce8Bn/record_store
24M	/home/safe/.local/share/safe/node/12D3KooWGxdpbZguaNkdzSNfobDQzNqjAJAxEHBGoW3NKrURHVXC/record_store
26M	/home/safe/.local/share/safe/node/12D3KooWJ8rQnzytbicPdLzjdz1KDcZ2JjAaijkvsmC4NhTiPZU1/record_store
26M	/home/safe/.local/share/safe/node/12D3KooWLJy2u4wSauipwMsfdtAk2tshti7nMPp7BFm2aEEFCqtR/record_store
21M	/home/safe/.local/share/safe/node/12D3KooWLWfvU6ToNqgSXsNheoVUmwHPZokVVtLyuhNSKAvyX2UZ/record_store
27M	/home/safe/.local/share/safe/node/12D3KooWMDm2YMd17DpFkbPevGmCv77hNqSPY8r2wzqPs26XwFcq/record_store
34M	/home/safe/.local/share/safe/node/12D3KooWMc7Y5w4ejkXJKsHvc3NcfCpMhXWfJ1AbpixF6EaUrTHe/record_store
4.0K	/home/safe/.local/share/safe/node/12D3KooWMipdqY2rSMuQbU65iX2YLD9uGpM2YcwDpAoSpd94BSdL/record_store
26M	/home/safe/.local/share/safe/node/12D3KooWPmXovbiig4aMC34RUnzVWYTJGsNsQvWFECb5Xye2kCGx/record_store
4.0K	/home/safe/.local/share/safe/node/12D3KooWPwmVTc2ZDhYWV9DfZEsGtQhfK9LWgQ1dwC5fPYVfFoz4/record_store
15M	/home/safe/.local/share/safe/node/12D3KooWRWM2gCbjAXvDmNvnCpcwbm1ZiYSgCTvM4wyHMLEXfmFw/record_store
23M	/home/safe/.local/share/safe/node/12D3KooWS46Lym67PpzrStBSnbuTjLtwdMxm4WYMvQZiHi4Y76c6/record_store
5 Likes
56

For a bit of fun I decided to max things out a bit - I’m using an older 4-core CPU (perhaps that’s part of my problem?) but in earlier trials I didn’t come close to even 25% of my cpu.

This time I did -c 200 and -batch-size 200 (knowing that my total # of chunks was 197 from previous attempts I figured this would do it all in one go)

Results - maxed all cores of my CPU’s! So I think there was a seriously hard try here. But still failed - however I received messages I hadn’t received before:

First round of upload completed, verifying and repaying if required...
======= Verification: 197 chunks to be checked and repayed if required =============

then

======= Verification: 182 chunks were not stored in the network, repaying them in batches =============
Failed to fetch 182 chunks, attempting to repay them

then

Error: 
   0: Failed to send tokens due to Network Error Not enough store cost quotes returned from the network to ensure a valid fee is paid.

Location:
   sn_cli/src/subcommands/files.rs:365

So interesting, but I don’t know if anything useful was gleaned except you can really crank up those concurrency and batch numbers.

Those defaults are rookie numbers!

EDIT: ran again as I forgot to log the first go round.

This time it uploaded a few more chunks:

======= Verification: 174 chunks were not stored in the network, repaying them in batches =============
Failed to fetch 174 chunks, attempting to repay them

AND it actually restarted the upload (tried again after failing) - which I hadn’t seen it do in all my other attempts.

It then suceeded in putting a few more chunks:

======= Verification: 163 chunks were not stored in the network, repaying them in batches =============
Failed to fetch 163 chunks, attempting to repay them

But failed after that and didn’t restart, giving the same error as before:

Error: 
   0: Failed to send tokens due to Network Error Not enough store cost quotes returned from the network to ensure a valid fee is paid.

Location:
   sn_cli/src/subcommands/files.rs:365

Thinking now I might try with high concurrency but small (tiny) batch size.

2 Likes
57

im here in the back ground i have 100 nodes running across 2 cloud vps’s and a home mini pc all are are running nicely

uploading is going more or less okay on default settings.

Beg blag is up and i managed a 2Gb movie and am now uploading 40gb of mp3s and a some Ubuntu images that have been split up into 2Mb files with zip

most uploads are going fine but some are failing with the following error


Failed to fetch 13 chunks, attempting to repay them
Error:
   0: Failed to send tokens due to Network Error Not enough store cost quotes returned from the network to ensure a valid fee is paid.

Location:
   sn_cli/src/subcommands/files.rs:365
5 Likes
58

Where can I download some of your work?

Ideally for every upload we should have one or more downloads to validate. Maybe I should think harder aboiut a script to do just that

59

the new uploads print a xor url like the last test net and i saw something in the git hub pr’s about adding a ls command to make a ls command that got dropped

1 Like
60

looks like i jinxed it i tried opening a second terminal on the grey machine and doing a simultaneous upload and what had been going fine suddenly was not fine not sure if this means anything the blue vps carried on uploading without issue

image
image1920×1046 181 KB

4 Likes