IntolerantNodeNet [18/09/23 Testnet] [ Offline ]

Development Releases
21

with " -c 10 --batch-size 40" – failed with same error.
with " -c 10 --batch-size 30" – failed with same error.
with " -c 10 --batch-size 20" – failed with same error.
with " -c 10 --batch-size 30" – failed with same error.

with " -c 15 --batch-size 40" – failed with same error.
with " -c 20 --batch-size 40" – failed with same error.
with " -c 30 --batch-size 40" – failed with same error.

with " -c 20 --batch-size 10" – failed with same error.
with " -c 20 --batch-size 20" – failed with same error.
with " -c 20 --batch-size 30" – failed with same error.
with " -c 20 --batch-size 40" – failed with same error.

Error: 
   0: Transfer Error Failed to send tokens due to Network Error Not enough store cost quotes returned from the network to ensure a valid fee is paid..
   1: Failed to send tokens due to Network Error Not enough store cost quotes returned from the network to ensure a valid fee is paid.

Location:
   sn_cli/src/subcommands/files.rs:179

Searched logs for “Non store cost response received” and found nothing.

4 Likes
22

verrrry interessting.

Would you mind to send me your client logs for any one of those runs? Maybe the lowest batch size actually

3 Likes
23

Sorry, I got to go now, and likely can’t participate until tomorrow…

2 Likes
24

The log is all concatted together now … I can run again with specific -c -batch-size if you want?

Otherwise here is the whole output for all I’ve done so far.
safe.log.20230918T225352.zip (3.1 MB)

2 Likes
25

I tested only -c 100 for now, and upload was fine (37MB file).

4 Likes
26

I didn’t even think to try something that high. What is a reasonable range for us to be testing here @joshuef ?

3 Likes
27

No idea! Depends on your setup. That’s part of what I’m hoping to glean from the testnet :wink: .
More concurreny should yield more CPU usage… could cause some slowdown at higher values.

This seems to be the storecost issue, we’re timing out when sending reqs, so naturally don’t get responses for those. Now why is that happening?!

Got response for the req: Query(GetStoreCost(NetworkAddress::ChunkAddress(7b836b(01111011).. -  - 7b836be4b46b2aea8e60edbc0f5973c0027316813c38155086c0c8df518c9c11))), res: OutboundError(Timeout)
4 Likes
28

@TylerAbeoJordan did you try with default concurrency btw? Which is 5?

4 Likes
29

I initially uploaded without any settings there - so presume that would have been the default.

4 Likes
30

I tried -c 100 --batch-size 100 and it failed with Not enough store cost quotes. How to properly enable logs for client?
I remember client created log file in temp dir some time ago. But now it looks like no default logs are available.

4 Likes
31

Thank you for asking the question I was wondering about myself :slight_smile:

4 Likes
32

for unix: (windows will have to adapt cmds as the examples in the OP)

SN_LOG=all safe files upload <directory-path>

I think we may have a handle on that issue from @TylerAbeoJordan 's logs though. It appears to be connection overload somewhere.

I think we may be able to just not bail here, and instead save that chunk as one to try and reupload later. (As I suspect these errors are transient).

It’s under investigation right now.

5 Likes
33

Now we have another testnet to play with, @Josh and I would be dead chuffed if some of you had a shot at GitHub - javages/Rewards_plotting: Safe Network Rewards Tracking
One-click install (YMMV if you havent already got venv)
I know Rewards is not the main focus of this testnet but I doubt running this would hurt.

EDIT

hold off on this for a while. I spotted a stupid error and have submitted a quick PR to @Josh

OR workaround

chmod +x create_graphs.sh

6 Likes
34

I have slightly different result:
[2023-09-18T14:40:48.396697Z DEBUG sn_networking] Got response for the req: Query(GetStoreCost(NetworkAddress::ChunkAddress(5579a3(01010101).. - - 5579a324df5aff67e5381a331d7ce6a133dcfa5c8fe196a6dabbdc2956e07863))), res: OutboundError(DialFailure)

4 Likes
35

@joshuef I’m working on an update to vdash and would like to include a timeline and total for node earnings. Is there anything in the safenode log to show payments received and / or the total so far (or what is in the wallet)?

I’ve looked at my RewardNet logs and the latest source but couldn’t find anything.

My IntolerantNodeNet node is looking good, early days but so far:
PUTS: 26
GETS: 2
ERRORS: 17
safenode Memory: 45MB
Node’s rewards wallet balance (PeerId: 12D3KooWRZXC4XAiYDv4JXLeUuoNt3yHi7K2wAvVNUYbTXUpvvnC): 0.000000272

6 Likes
36

I get a strange message when I try to upload a file, what is the issue?

PS C:\Users\gggg> safe files upload <directory-E:\gggg\Pictures\VID_20210505_113014.mp4>
At line:1 char:19
+ safe files upload <directory-E:\gggg\Pictures\VID_20210505_113014.mp4 ...
+                   ~
The '<' operator is reserved for future use.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : RedirectionNotSupported
3 Likes
37

Failure, retry and success :clap: :+1: :confetti_ball:

safe@IntolerantNetSouthside01:~$ safe files upload -c 10 --batch-size 40  main.zip 
Built with git version: 26c3d70 / main / 26c3d70
Instantiating a SAFE client...
🔗 Connected to the Network                                                                                                                                                  Total number of chunks to be stored: 4
Transfers applied locally
After 14.456680026s, All transfers made for total payment of Token(842) nano tokens for 4 chunks. 
Successfully made payment of 0.000000842 for 4 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999999158.
After 20.740054734s, uploaded 4 chunks, current progress is 4/4. 
Uploaded chunk #85574b.. in 0 seconds
Uploaded chunk #b4f948.. in 0 seconds
Uploaded chunk #c716fa.. in 0 seconds
Uploaded chunk #485302.. in 0 seconds
First round of upload completed, verifying and repaying if required...
======= Verification: 4 chunks to be checked and repayed if required =============
Failed to fetch a chunk ChunkAddress(85574b(10000101)..)
Failed to fetch a chunk ChunkAddress(b4f948(10110100)..)
Failed to fetch a chunk ChunkAddress(c716fa(11000111)..)
Failed to fetch a chunk ChunkAddress(485302(01001000)..)
======= Verification: 4 chunks were not stored in the network, repaying them in batches =============
Failed to fetch 4 chunks, attempting to repay them
Transfers applied locally
After 15.081144771s, All transfers made for total payment of Token(842) nano tokens for 4 chunks. 
Uploaded chunk #85574b.. in 0 seconds
Uploaded chunk #b4f948.. in 0 seconds
Uploaded chunk #c716fa.. in 0 seconds
Uploaded chunk #485302.. in 0 seconds
After 19.731319734s, verified 4 chunks
4 failed chunks were found, repaid & re-uploaded.
======= Verification: 4 chunks to be checked and repayed if required =============
======= Verification Completed! All chunks have been paid and stored! =============
Uploaded all chunks in 48 seconds
Writing 56 bytes to "/home/safe/.local/share/safe/client/uploaded_files/file_names_2023-09-18_14-59-02"

Its a trivially small upload but all the same…

6 Likes
38

Uploading a large collection of audio. Still chugging away, no errors but quite slow, and definitely getting slower over time (client side resources I’d guess - didn’t set client log unfortunately). At the start 20 chunks took about 90 seconds, as reported in the output. Now it’s about 140s. Also a big variation in the reported storage time for individual chunks: 4s - 79s.


Uploaded chunk #d90a6b.. in 14 seconds
Uploaded chunk #954dfe.. in 24 seconds
Uploaded chunk #bea675.. in 4 seconds
Uploaded chunk #be241f.. in 39 seconds
Uploaded chunk #bee111.. in 4 seconds
Uploaded chunk #6c2e9e.. in 4 seconds
Uploaded chunk #98619e.. in 5 seconds
Uploaded chunk #96e845.. in 54 seconds
Uploaded chunk #b20d88.. in 4 seconds
Uploaded chunk #43e5e3.. in 4 seconds
Uploaded chunk #4177b2.. in 4 seconds
Uploaded chunk #27eecc.. in 4 seconds
Uploaded chunk #a8491a.. in 1 minutes 19 seconds
Uploaded chunk #a44cee.. in 1 minutes 14 seconds
Uploaded chunk #bb1e4c.. in 1 minutes 9 seconds
Uploaded chunk #1de2ce.. in 1 minutes 4 seconds
Uploaded chunk #0b52d0.. in 59 seconds
Uploaded chunk #64d7aa.. in 44 seconds
Uploaded chunk #aea814.. in 24 seconds
Uploaded chunk #4818be.. in 4 seconds
Transfers applied locally
After 18.543492699s, All transfers made for total payment of Token(3753) nano tokens for 20 chunks. 
Successfully made payment of 0.000003753 for 20 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999584666.
After 143.844150327s, uploaded 20 chunks, current progress is 2940/3789.
9 Likes
39

Getting these errors quite consistently now

After 25.547063969s, All transfers made for total payment of Token(2022) nano tokens for 10 chunks. 
Successfully made payment of 0.000002022 for 10 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999969779.
After 46.41418972s, uploaded 10 chunks, current progress is 60/214. 
Uploaded chunk #8d61cc.. in 0 seconds
Uploaded chunk #b4c164.. in 0 seconds
Uploaded chunk #8d4346.. in 0 seconds
Uploaded chunk #ea6b11.. in 0 seconds
Uploaded chunk #b5efa6.. in 0 seconds
Uploaded chunk #997735.. in 0 seconds
Uploaded chunk #6142b1.. in 0 seconds
Uploaded chunk #6067ee.. in 0 seconds
Uploaded chunk #ccb62e.. in 0 seconds
Uploaded chunk #18fcfc.. in 0 seconds
Error: 
   0: Transfer Error Failed to send tokens due to Network Error Not enough store cost quotes returned from the network to ensure a valid fee is paid..
   1: Failed to send tokens due to Network Error Not enough store cost quotes returned from the network to ensure a valid fee is paid.

Varying the batch size does not seem to change anything - at c 60 I can hardly detect any change in CPU from c 20

3 Likes
40

had a 2Gb movie uploading on a 200Mbs up/down connection and not seen any errors so far
on a home connection 56 down 10 up I am getting those same errors

could be connection related

6 Likes