IntolerantNodeNet [18/09/23 Testnet] [ Offline ]

Another week, another wee testnet. We’re looking to verify various changes we’ve worked on since the last…

Most notably the pricing curve has changed, it’s now much more granular (a new price at each step) and stays cheaper for longer.

We also have various client improvements to reduce memory when working with larger files, and offer more control during uploads. As well as the total concurrency limit, which you can set with -c 10 (e.g., to allow for 10 concurrent uploads/queries), we have a --batch-size 10 argument, which defines how many chunks are paid for an uploaded together (this should have more impact on larger files… hopefully improving the success rate there). More examples of that below.

Our objectives:

  1. How does the client tolerance (currently 50% over the asking price) hold up?
  2. How are client up/download performing? (How do concurrency, and batch-size affect errors here?)
  3. Check register payments are working smoothly

It should be noted that while the client will currently repay if a PUT failed… it does not only pay the difference as yet. That’s still to come.

Network Details

Node version: 0.90.14
Client version: 0.81.44
Faucet url: 139.59.182.135:8000
SAFE_PEERS: /ip4/139.59.182.135/tcp/43953/p2p/12D3KooWFP9PfPoDeAtQU5Y9akqmciABu4AJqPsiA3WQTgDe5XHn
Alternatives:

"/ip4/174.138.95.28/tcp/42445/p2p/12D3KooWH9KeGXY6xYxp9aPy2WNR48XWo4ce17i4UdzmiR2nWi86"
"/ip4/64.227.157.116/tcp/35321/p2p/12D3KooWHo1u5VZ4KxMiHiVxy5xH2j5AgYvk18M5J5bJNZDKN4y6"
"/ip4/137.184.188.173/tcp/44513/p2p/12D3KooWBCEhQiahMb6zexVaJVquZZWyui1Dzz44uHLUXpx9uU3Q"
"/ip4/142.93.214.164/tcp/40253/p2p/12D3KooWB2sxFRaSV77jvmXiqwVFFF6bAXr1V7L3rmbVEkHGJq8c"
"/ip4/137.184.188.122/tcp/44133/p2p/12D3KooWKre2WdD2ScmJ1j4Cxh1dRFsTze1WYYkMqc8tQn6CtM7H"
"/ip4/64.227.132.205/tcp/37671/p2p/12D3KooWQVWBExLhXkmoQzARcGMv96fa7nj1tiFai4boQUKh5j8G"
"/ip4/139.59.120.201/tcp/39227/p2p/12D3KooWDppSnhhtZJdagspz6Vu7AmTR5Vc1zdN29eMu1vHS3azx"
"/ip4/159.223.229.32/tcp/37469/p2p/12D3KooWDN4sZAUuc4WqUj2LHo3i4mPWvTt22PWA4jXfQ1wmgkV4"
"/ip4/167.71.170.90/tcp/44331/p2p/12D3KooWDAAviAzBfSQBTRoXuQZw9ge1PLDY24ar8EsYFXuvjirV"
"/ip4/161.35.178.141/tcp/33119/p2p/12D3KooWRUd5BmFbqBBzRWb9uvwy1dWBdhzk2BGV7g5kN1sLjyzb"]
...

We have 2001 droplets running a total of 100 nodes. One droplet has 2vcpu and 4GB of memory.


If you are a regular user, see the ‘quickstart’ section for getting up and running.

If you are a first-time user, or would like more information, see the ‘further information’ section.


Quickstart

If you already have safeup, you can obtain the client and node binaries:

safeup client --version 0.81.44
safeup node --version 0.90.14

Run a Node

Linux/macOS:

export SAFE_PEERS="/ip4/139.59.182.135/tcp/43953/p2p/12D3KooWFP9PfPoDeAtQU5Y9akqmciABu4AJqPsiA3WQTgDe5XHn"
SN_LOG=all safenode

Windows:

$env:SAFE_PEERS="/ip4/139.59.182.135/tcp/43953/p2p/12D3KooWFP9PfPoDeAtQU5Y9akqmciABu4AJqPsiA3WQTgDe5XHn"
$env:SN_LOG = "all"; safenode

Check local node’s reward balance

Your local node’s peer id will be printed to the terminal on startup with an example command). (You can also retrieve this from the node directory.)

safe wallet balance --peer-id="<local-node-peer-id>"

Connect to the Network as a Client

Linux/macOS:

export SAFE_PEERS="/ip4/139.59.182.135/tcp/43953/p2p/12D3KooWFP9PfPoDeAtQU5Y9akqmciABu4AJqPsiA3WQTgDe5XHn"
safe wallet get-faucet 139.59.182.135:8000
safe files upload <directory-path>

Windows:

$env:SAFE_PEERS = "/ip4/139.59.182.135/tcp/43953/p2p/12D3KooWFP9PfPoDeAtQU5Y9akqmciABu4AJqPsiA3WQTgDe5XHn"
safe wallet get-faucet 139.59.182.135:8000
safe files upload <directory-path>

To do this with non-default concurrency or batch-sizes (along with SAFE_PEERS set as above):

safe files upload -c 10 --batch-size 40 <directory-path> 

10 and 40 being the integer values you want to set for each

Further Information

You can participate in the testnet either by connecting as a client or running your own node.

Connecting as a client requires the safe client binary; running a node requires the safenode binary.

Obtaining Binaries

We have a tool named safeup which is intended to make it easy to obtain the client, node, and other utility binaries.

Installing Safeup

On Linux/macOS, run the following command in your terminal:

curl -sSL https://raw.githubusercontent.com/maidsafe/safeup/main/install.sh | bash

On Windows, run the following command in a Powershell session (be careful to use Powershell, not cmd.exe):

iex (Invoke-RestMethod -Uri "https://raw.githubusercontent.com/maidsafe/safeup/main/install.ps1")

On either platform, you may need to restart your shell session for safeup to become available.

Installing Binaries

After obtaining safeup, you can install binaries like so:

safeup client # get the latest version of the client
safeup client --version 0.81.44 # get a specific version

safeup node # get the latest version of the node
safeup node --version 0.90.14 # get a specific version

safeup update # update all installed components to latest versions

When participating in our testnets, it is recommended to use a specific version. In our project we release a new version of the binaries every time we merge new code. This happens frequently, so over the lifetime of a testnet, many new releases will probably occur. So for participating in this particular testnet, you may not want the latest version.

The binaries are installed to ~/.local/bin on Linux and macOS, and on Windows they go to C:\Users\<username>\safe. Windows doesn’t really have a standard location for binaries that doesn’t require elevated privileges.

The safeup tool will modify the PATH variable on Linux/macOS, or the user Path variable on Windows. The effect of this is that the installed binaries will then become available in your shell without having to refer to them with their full paths. However, if you’re installing for the first time, you may need to start a new shell session for this change to be picked up.

Running a Node

You can participate in the testnet by running your own node. At the moment, you may not be successful if you’re running the node from your home machine. This is a situation we are working on. If you run from a cloud provider like Digital Ocean or AWS, you should be able to participate.

You can run the node process like so:

# Linux/macOS
SN_LOG=all safenode

# Windows
$env:SN_LOG = "all"; safenode

This will output all the logs in the terminal.

Sometimes it will be preferable to output the logs to file. You can do this by running the node like so:

# Linux/macOS
SN_LOG=all safenode --log-output-dest data-dir

# Windows
$env:SN_LOG = "all"; safenode --log-output-dest data-dir

The location of data-dir is platform specific:

# Linux
~/.local/share/safe/node/<peer id>/logs

# macOS
/Users/<username>/Library/Application Support/safe/node/<peer id>/logs

# Windows
C:\Users\<username>\AppData\Roaming\safe\node\<peer-id>\logs

If you wish, you can also provide your own path:

# Linux/macOS
SN_LOG=all safenode --log-output-dest <path>

# Windows
$env:SN_LOG = "all"; safenode --log-output-dest <path>

The advantage of using the predefined data-dir location is you can run multiple nodes on one machine without having to specify your own unique path for each node and manage that overhead yourself.

Connecting as a Client

You can use the safe client binary to connect as a client and upload or download files to/from the network.

To connect, you must provide another peer, in the form of a multi address. You can find one in the ‘Network Details’ section at the top.

It is recommended to set the peer using the environment variable SAFE_PEERS. You can set this variable once and it will apply for the duration of your shell session:

# Linux/macOS
export SAFE_PEERS="/ip4/139.59.182.135/tcp/43953/p2p/12D3KooWFP9PfPoDeAtQU5Y9akqmciABu4AJqPsiA3WQTgDe5XHn"

# Windows
$env:SAFE_PEERS = "/ip4/139.59.182.135/tcp/43953/p2p/12D3KooWFP9PfPoDeAtQU5Y9akqmciABu4AJqPsiA3WQTgDe5XHn"

NOTE: If you close and/or start a new shell session, you will be required to redefine this environment variable in the new session.

As an alternative to the environment variable, it’s also possible to use the --peer argument:

safe --peer="/ip4/139.59.182.135/tcp/43953/p2p/12D3KooWFP9PfPoDeAtQU5Y9akqmciABu4AJqPsiA3WQTgDe5XHn" ...

However, this requires specifying the peer with each command.

Using the Client

You’ll first need to get some Safe Network Tokens:

safe wallet get-faucet 139.59.182.135:8000

You can now proceed to use the client, by, e.g., uploading files:

safe files upload <directory-path>

To download that same content:

safe files download

This will download the files to the default location, which is platform specific:

# Linux
~/.local/share/safe/client/downloaded_files

# macOS
/Users/<username>/Library/Application Support/safe/client/downloaded_files

# Windows
C:\Users\<username>\AppData\Roaming\safe\client\downloaded_files

To download to a particular file or directory:

safe file download [directory/filename] [XORURL]

Troubleshooting

Cleanup

If you’ve used previous versions of the network before and you find problems when running commands, you may want to consider clearing out previous data (worthless DBCs from previous runs, old logs, old keys, etc.).

# Linux
rm -rf ~/.local/share/safe

# macOS
rm -rf ~/Library/Application\ Support/safe

# Windows
rmdir /s C:\Users\<username>\AppData\Roaming\safe

If you encounter a problem running any of our binaries on Windows, it’s possible you need the Visual C++ Redistributable installed.

39 Likes

Amazing!!!

19 Likes

The next TestNet ? MaidSafe is faster than SpaceX! Well done!

Obviously I ended up outside of NAT when I tried to run the node.
However, I don’t know what the next message means?

Location:
   sn_node\src/bin/safenode/main.rs:328

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
15 Likes

first quick attempt failed -

   0: Transfer Error Failed to send tokens due to Network Error Not enough store cost quotes returned from the network to ensure a valid fee is paid..
   1: Failed to send tokens due to Network Error Not enough store cost quotes returned from the network to ensure a valid fee is paid.
willie@gagarin:~$ safe files upload -c 10 --batch-size 40 ~/trecem
Built with git version: 26c3d70 / main / 26c3d70
Instantiating a SAFE client...
🔗 Connected to the Network                                                                                                                                Total number of chunks to be stored: 104
Transfers applied locally
After 16.267167406s, All transfers made for total payment of Token(4995) nano tokens for 40 chunks. 
Successfully made payment of 0.000004995 for 40 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999995005.
After 45.825907694s, uploaded 40 chunks, current progress is 40/102. 
Uploaded chunk #06faab.. in 0 seconds
Uploaded chunk #73582f.. in 0 seconds
Uploaded chunk #5eff94.. in 0 seconds
Uploaded chunk #033efb.. in 0 seconds
Uploaded chunk #2c2035.. in 0 seconds
Uploaded chunk #c83d28.. in 0 seconds
Uploaded chunk #453db8.. in 0 seconds
Uploaded chunk #5e8e88.. in 0 seconds
Uploaded chunk #1a306b.. in 1 seconds
Uploaded chunk #bd1d41.. in 1 seconds
Uploaded chunk #ca8ea5.. in 1 seconds
Uploaded chunk #2666dd.. in 0 seconds
Uploaded chunk #47bd1c.. in 0 seconds
Uploaded chunk #2b269f.. in 0 seconds
Uploaded chunk #3fdcef.. in 0 seconds
Uploaded chunk #27912c.. in 0 seconds
Uploaded chunk #33346f.. in 0 seconds
Uploaded chunk #165825.. in 0 seconds
Uploaded chunk #bcb74d.. in 0 seconds
Uploaded chunk #036e38.. in 0 seconds
Uploaded chunk #21ed67.. in 0 seconds
Uploaded chunk #46b560.. in 0 seconds
Uploaded chunk #ec6016.. in 0 seconds
Uploaded chunk #a022b0.. in 0 seconds
Uploaded chunk #50fa54.. in 0 seconds
Uploaded chunk #4e4d94.. in 1 seconds
Uploaded chunk #21ea8c.. in 2 seconds
Uploaded chunk #af085c.. in 0 seconds
Uploaded chunk #4a18c5.. in 0 seconds
Uploaded chunk #4f7787.. in 0 seconds
Uploaded chunk #42114b.. in 0 seconds
Uploaded chunk #f84ce3.. in 0 seconds
Uploaded chunk #2a7d09.. in 0 seconds
Uploaded chunk #9cfb19.. in 0 seconds
Uploaded chunk #1ea502.. in 0 seconds
Uploaded chunk #3ec622.. in 0 seconds
Uploaded chunk #d64dc8.. in 0 seconds
Uploaded chunk #83b670.. in 0 seconds
Uploaded chunk #a63216.. in 0 seconds
Uploaded chunk #ca6d17.. in 0 seconds
Error: 
   0: Transfer Error Failed to send tokens due to Network Error Not enough store cost quotes returned from the network to ensure a valid fee is paid..
   1: Failed to send tokens due to Network Error Not enough store cost quotes returned from the network to ensure a valid fee is paid.

Location:
   sn_cli/src/subcommands/files.rs:179

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
willie@gagarin:~$

EDIT: totally repeatable error - got exactly the same on a retry

Now going again with batch size=20

yes a smaller batch size worked OK :slight_smile:

willie@gagarin:~$ safe files upload -c 10 --batch-size 20 ~/trecem
Built with git version: 26c3d70 / main / 26c3d70
Instantiating a SAFE client...
🔗 Connected to the Network                                                                                                                                Total number of chunks to be stored: 104
Transfers applied locally
After 17.962269256s, All transfers made for total payment of Token(2531) nano tokens for 20 chunks. 
Successfully made payment of 0.000002531 for 20 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999987408.
After 34.702342257s, uploaded 20 chunks, current progress is 20/102. 
Uploaded chunk #21ea8c.. in 0 seconds
Uploaded chunk #bd1d41.. in 0 seconds
Uploaded chunk #453db8.. in 0 seconds
Uploaded chunk #5e8e88.. in 0 seconds
Uploaded chunk #06faab.. in 0 seconds
Uploaded chunk #5eff94.. in 0 seconds
Uploaded chunk #c83d28.. in 0 seconds
Uploaded chunk #73582f.. in 0 seconds
Uploaded chunk #2c2035.. in 0 seconds
Uploaded chunk #1a306b.. in 0 seconds
Uploaded chunk #033efb.. in 0 seconds
Uploaded chunk #ca8ea5.. in 0 seconds
Uploaded chunk #33346f.. in 0 seconds
Uploaded chunk #27912c.. in 0 seconds
Uploaded chunk #21ed67.. in 0 seconds
Uploaded chunk #4e4d94.. in 0 seconds
Uploaded chunk #2666dd.. in 0 seconds
Uploaded chunk #2b269f.. in 0 seconds
Uploaded chunk #a63216.. in 0 seconds
Uploaded chunk #83b670.. in 0 seconds
Transfers applied locally
After 15.222129202s, All transfers made for total payment of Token(2555) nano tokens for 20 chunks. 
Successfully made payment of 0.000002555 for 20 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999984853.
After 58.748933037s, uploaded 20 chunks, current progress is 40/102. 
Uploaded chunk #3fdcef.. in 0 seconds
Uploaded chunk #d64dc8.. in 0 seconds
Uploaded chunk #2a7d09.. in 0 seconds
Uploaded chunk #47bd1c.. in 0 seconds
Uploaded chunk #3ec622.. in 0 seconds
Uploaded chunk #bcb74d.. in 0 seconds
Uploaded chunk #036e38.. in 0 seconds
Uploaded chunk #165825.. in 0 seconds
Uploaded chunk #1ea502.. in 0 seconds
Uploaded chunk #ca6d17.. in 0 seconds
Uploaded chunk #42114b.. in 0 seconds
Uploaded chunk #9cfb19.. in 0 seconds
Uploaded chunk #4f7787.. in 0 seconds
Uploaded chunk #f84ce3.. in 0 seconds
Uploaded chunk #4a18c5.. in 0 seconds
Uploaded chunk #ec6016.. in 0 seconds
Uploaded chunk #46b560.. in 0 seconds
Uploaded chunk #a022b0.. in 0 seconds
Uploaded chunk #af085c.. in 0 seconds
Uploaded chunk #50fa54.. in 0 seconds
Transfers applied locally
After 15.571866821s, All transfers made for total payment of Token(2609) nano tokens for 20 chunks. 
Successfully made payment of 0.000002609 for 20 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999982244.
After 32.669383506s, uploaded 20 chunks, current progress is 60/102. 
Uploaded chunk #6985cb.. in 0 seconds
Uploaded chunk #56e21f.. in 0 seconds
Uploaded chunk #54ad13.. in 0 seconds
Uploaded chunk #65f9d5.. in 0 seconds
Uploaded chunk #d66957.. in 0 seconds
Uploaded chunk #6b105b.. in 0 seconds
Uploaded chunk #bf588d.. in 0 seconds
Uploaded chunk #676116.. in 0 seconds
Uploaded chunk #e16b75.. in 0 seconds
Uploaded chunk #17c28c.. in 0 seconds
Uploaded chunk #7c29f7.. in 0 seconds
Uploaded chunk #726342.. in 0 seconds
Uploaded chunk #aee200.. in 0 seconds
Uploaded chunk #8a7942.. in 0 seconds
Uploaded chunk #55ae37.. in 0 seconds
Uploaded chunk #46f806.. in 0 seconds
Uploaded chunk #c71b05.. in 0 seconds
Uploaded chunk #88e6ea.. in 0 seconds
Uploaded chunk #886d2a.. in 0 seconds
Uploaded chunk #5bfd26.. in 0 seconds
Transfers applied locally
After 18.133343044s, All transfers made for total payment of Token(2575) nano tokens for 20 chunks. 
Successfully made payment of 0.000002575 for 20 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999979669.
After 33.877056284s, uploaded 20 chunks, current progress is 80/102. 
Uploaded chunk #cea9d4.. in 0 seconds
Uploaded chunk #31ed80.. in 0 seconds
Uploaded chunk #926d66.. in 0 seconds
Uploaded chunk #9c3f48.. in 0 seconds
Uploaded chunk #32a681.. in 0 seconds
Uploaded chunk #9eb7f1.. in 0 seconds
Uploaded chunk #dd9e5e.. in 0 seconds
Uploaded chunk #c25de8.. in 0 seconds
Uploaded chunk #a53573.. in 0 seconds
Uploaded chunk #25949d.. in 0 seconds
Uploaded chunk #ad2c45.. in 0 seconds
Uploaded chunk #947bf3.. in 0 seconds
Uploaded chunk #e313fb.. in 0 seconds
Uploaded chunk #e482e4.. in 0 seconds
Uploaded chunk #0b445e.. in 0 seconds
Uploaded chunk #c37a62.. in 0 seconds
Uploaded chunk #e48a49.. in 0 seconds
Uploaded chunk #aa91d1.. in 0 seconds
Uploaded chunk #ebe99e.. in 0 seconds
Uploaded chunk #bcc041.. in 0 seconds
Transfers applied locally
After 15.336392737s, All transfers made for total payment of Token(2697) nano tokens for 20 chunks. 
Successfully made payment of 0.000002697 for 20 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999976972.
After 34.018962757s, uploaded 20 chunks, current progress is 100/102. 
Uploaded chunk #efa702.. in 0 seconds
Uploaded chunk #e554b8.. in 0 seconds
Uploaded chunk #2afd0f.. in 0 seconds
Uploaded chunk #f44a8b.. in 0 seconds
Uploaded chunk #c0f3fe.. in 0 seconds
Uploaded chunk #e57618.. in 0 seconds
Uploaded chunk #f86a15.. in 0 seconds
Uploaded chunk #666f3f.. in 0 seconds
Uploaded chunk #f408f7.. in 0 seconds
Uploaded chunk #47800f.. in 0 seconds
Uploaded chunk #870676.. in 0 seconds
Uploaded chunk #4cc1ad.. in 0 seconds
Uploaded chunk #faebe7.. in 0 seconds
Uploaded chunk #faff6f.. in 0 seconds
Uploaded chunk #edab22.. in 0 seconds
Uploaded chunk #ed8a22.. in 0 seconds
Uploaded chunk #a196c5.. in 0 seconds
Uploaded chunk #f70069.. in 0 seconds
Uploaded chunk #b892f1.. in 0 seconds
Uploaded chunk #faecef.. in 0 seconds
Transfers applied locally
After 15.100284859s, All transfers made for total payment of Token(294) nano tokens for 2 chunks. 
Successfully made payment of 0.000000294 for 2 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999976678.
After 22.360688981s, uploaded 2 chunks, current progress is 102/102. 
Uploaded chunk #ffa702.. in 0 seconds
Uploaded chunk #fbd49b.. in 0 seconds
First round of upload completed, verifying and repaying if required...
======= Verification: 102 chunks to be checked and repayed if required =============
======= Verification Completed! All chunks have been paid and stored! =============
Uploaded all chunks in 4 minutes 26 seconds
Writing 2786 bytes to "/home/willie/.local/share/safe/client/uploaded_files/file_names_2023-09-18_13-21-27"

Thank you to all involved in getting us this far :man_bowing:

19 Likes

Node running fine.

Upload of a small jpg took a few rounds of retries, but succeeded in the end:

:~$ time safe files upload topibott.jpg
Built with git version: 26c3d70 / main / 26c3d70
Instantiating a SAFE client...
🔗 Connected to the Network                                                     Total number of chunks to be stored: 4
Transfers applied locally
After 15.293741075s, All transfers made for total payment of Token(530) nano tokens for 4 chunks. 
Successfully made payment of 0.000000530 for 4 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999999470.
After 25.779622412s, uploaded 4 chunks, current progress is 4/4. 
Uploaded chunk #07bb5b.. in 0 seconds
Uploaded chunk #cc128c.. in 0 seconds
Uploaded chunk #31a65a.. in 0 seconds
Uploaded chunk #f178ae.. in 0 seconds
First round of upload completed, verifying and repaying if required...
======= Verification: 4 chunks to be checked and repayed if required =============
Failed to fetch a chunk ChunkAddress(31a65a(00110001)..)
Failed to fetch a chunk ChunkAddress(cc128c(11001100)..)
Failed to fetch a chunk ChunkAddress(07bb5b(00000111)..)
Failed to fetch a chunk ChunkAddress(f178ae(11110001)..)
======= Verification: 4 chunks were not stored in the network, repaying them in batches =============
Failed to fetch 4 chunks, attempting to repay them
Transfers applied locally
After 10.301751728s, All transfers made for total payment of Token(535) nano tokens for 4 chunks. 
Uploaded chunk #cc128c.. in 0 seconds
Uploaded chunk #07bb5b.. in 0 seconds
Uploaded chunk #f178ae.. in 0 seconds
Uploaded chunk #31a65a.. in 0 seconds
After 18.079363738s, verified 4 chunks
4 failed chunks were found, repaid & re-uploaded.
======= Verification: 4 chunks to be checked and repayed if required =============
======= Verification Completed! All chunks have been paid and stored! =============
Uploaded all chunks in 52 seconds
Writing 60 bytes to "/home/topi/.local/share/safe/client/uploaded_files/file_names_2023-09-18_15-19-29"

real	0m58,549s
user	0m5,918s
sys	0m1,638s

15 Likes

Nice. Yeh, I suspect we’re still overly keen on pushing out too many reqs to the network, straining the client, of not remote nodes… And so causing this to fail. It shoudl still not fail… but somethign is off there.

You could try increasing concurrency further too. (it may be that it and batch size should just be linked… It’s something I’m hoping to get more of a handle on from this testnet, what are ideal values and how might they relate in more situations)

13 Likes

I am getting the same upload error as @Southside I think.


Error: 
   0: Transfer Error Failed to send tokens due to Network Error Not enough store cost quotes returned from the network to ensure a valid fee is paid..
   1: Failed to send tokens due to Network Error Not enough store cost quotes returned from the network to ensure a valid fee is paid.

Location:
   sn_cli/src/subcommands/files.rs:179

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
12 Likes

can yourself or @Southside give the logs a wee check for Non store cost response received i’m curious what we did get back from nodes here.

@TylerAbeoJordan can you try varying the concurrency and batch sizes while uploading and see if any combo there is more stable than another? :bowing_man:

12 Likes

I pushed the boat out in a different direction – this is a 1Gb directory with ~2500 files

willie@gagarin:~$ safe files upload -c 10 --batch-size 20 /fgfs/Aircraft/A320-family/
Built with git version: 26c3d70 / main / 26c3d70
Instantiating a SAFE client...
🔗 Connected to the Network                                                                                                                                Total number of chunks to be stored: 4584
Transfers applied locally
After 13.364884623s, All transfers made for total payment of Token(2494) nano tokens for 20 chunks. 
Successfully made payment of 0.000002494 for 20 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999974184.
After 28.848465996s, uploaded 20 chunks, current progress is 20/4468. 
Uploaded chunk #4e7a81.. in 0 seconds
Uploaded chunk #5c6bfa.. in 0 seconds
Uploaded chunk #e2dc06.. in 0 seconds
Uploaded chunk #87e06c.. in 0 seconds
Transfers applied locally
After 14.049525324s, All transfers made for total payment of Token(2740) nano tokens for 20 chunks. 
Successfully made payment of 0.000002740 for 20 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999950464.
After 33.888666425s, uploaded 20 chunks, current progress is 200/4468.

I have to out now for an hour or so. I’ll let this run and set up a few cloud nodes on Hetzner when I get back - Im still NATpositive at home :frowning:

11 Likes

doing that now.

5 Likes

20 seems to be the default batch size, but what is the default for concurrency @joshuef, if I don’t set anything? (Was too busy to start uploading before reading the whole post and setting the 10 and 40… :face_with_open_eyes_and_hand_over_mouth:)

4 Likes
PS C:\WINDOWS\system32> Measure-Command {safe --peer=/ip4/139.59.182.135/tcp/43953/p2p/12D3KooWFP9PfPoDeAtQU5Y9akqmciABu4AJqPsiA3WQTgDe5XHn files upload -c 10 --batch-size 40 -- C:\ud\yb.jpg | Out-Default}
Built with git version: 26c3d70 / main / 26c3d70
Instantiating a SAFE client...
🔗 Connected to the Network                                                                                             
Total number of chunks to be stored: 4
Transfers applied locally
After 16.7210569s, All transfers made for total payment of Token(489) nano tokens for 4 chunks.
Successfully made payment of 0.000000489 for 4 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999999511.
After 25.6102295s, uploaded 4 chunks, current progress is 4/4.
Uploaded chunk #d4513f.. in 0 seconds
Uploaded chunk #1a7fdb.. in 0 seconds
Uploaded chunk #5a712f.. in 0 seconds
Uploaded chunk #0b13ec.. in 0 seconds
First round of upload completed, verifying and repaying if required...
======= Verification: 4 chunks to be checked and repayed if required =============
Failed to fetch a chunk ChunkAddress(d4513f(11010100)..)
Failed to fetch a chunk ChunkAddress(1a7fdb(00011010)..)
Failed to fetch a chunk ChunkAddress(0b13ec(00001011)..)
Failed to fetch a chunk ChunkAddress(5a712f(01011010)..)
======= Verification: 4 chunks were not stored in the network, repaying them in batches =============
Failed to fetch 4 chunks, attempting to repay them
Transfers applied locally
After 17.5907044s, All transfers made for total payment of Token(492) nano tokens for 4 chunks.
Uploaded chunk #0b13ec.. in 0 seconds
Uploaded chunk #1a7fdb.. in 0 seconds
Uploaded chunk #d4513f.. in 0 seconds
Uploaded chunk #5a712f.. in 0 seconds
After 25.0004378s, verified 4 chunks
4 failed chunks were found, repaid & re-uploaded.
======= Verification: 4 chunks to be checked and repayed if required =============
======= Verification Completed! All chunks have been paid and stored! =============
Uploaded all chunks in 58 seconds
Writing 54 bytes to "C:\\Users\\Eric\\AppData\\Roaming\\safe\\client\\uploaded_files\\file_names_2023-09-18_07-33-29"


Days              : 0
Hours             : 0
Minutes           : 1
Seconds           : 3
Milliseconds      : 543
Ticks             : 635434944
TotalDays         : 0.000735457111111111
TotalHours        : 0.0176509706666667
TotalMinutes      : 1.05905824
TotalSeconds      : 63.5434944
TotalMilliseconds : 63543.4944

Looks good so far. I noticed the address the file is uploaded to isn’t displayed anymore like it used to during previous testnets i.e.

Successfully stored file "yb.jpg" to 5a712f6e9b46e0f2423b0768db838b5c51ab0b4d2b62dc08d8eca4806a42c172

How do I read the contents of the file_names_2023-09-18_07-33-29 file generated after the upload?

Edit: Nevermind. Figured it out. I opened the file in Notepad++ and converted the string from ASCII to HEX to get the address.

5 Likes

Can anyone explain what it means with simple example?
I see that repayment was made with difference lot less than 50% (1186 → 1233). Or it’s a different difference?

?? Connected to the Network
Total number of chunks to be stored: 9
Transfers applied locally
After 22.089585104s, All transfers made for total payment of Token(1186) nano to
kens for 9 chunks.
Successfully made payment of 0.000001186 for 9 chunks.
Successfully stored wallet with cached payment proofs, and new balance 99.999998
814.
After 44.171575681s, uploaded 9 chunks, current progress is 9/9.
Uploaded chunk #2335c8.. in 0 seconds
Uploaded chunk #8fe681.. in 0 seconds
Uploaded chunk #6273af.. in 0 seconds
Uploaded chunk #2fc024.. in 0 seconds
Uploaded chunk #31d419.. in 0 seconds
Uploaded chunk #378b75.. in 0 seconds
Uploaded chunk #294605.. in 0 seconds
Uploaded chunk #aa8784.. in 0 seconds
Uploaded chunk #d43536.. in 0 seconds
First round of upload completed, verifying and repaying if required...
======= Verification: 9 chunks to be checked and repayed if required ===========
==
Failed to fetch a chunk ChunkAddress(8fe681(10001111)..)
Failed to fetch a chunk ChunkAddress(6273af(01100010)..)
Failed to fetch a chunk ChunkAddress(2335c8(00100011)..)
Failed to fetch a chunk ChunkAddress(378b75(00110111)..)
Failed to fetch a chunk ChunkAddress(31d419(00110001)..)
Failed to fetch a chunk ChunkAddress(294605(00101001)..)
Failed to fetch a chunk ChunkAddress(2fc024(00101111)..)
Failed to fetch a chunk ChunkAddress(aa8784(10101010)..)
Failed to fetch a chunk ChunkAddress(d43536(11010100)..)
======= Verification: 9 chunks were not stored in the network, repaying them in
batches =============
Failed to fetch 9 chunks, attempting to repay them
Transfers applied locally
After 23.311739848s, All transfers made for total payment of Token(1233) nano to
kens for 9 chunks.
Uploaded chunk #2335c8.. in 0 seconds
Uploaded chunk #6273af.. in 0 seconds
Uploaded chunk #2fc024.. in 0 seconds
Uploaded chunk #8fe681.. in 0 seconds
Uploaded chunk #378b75.. in 0 seconds
Uploaded chunk #294605.. in 0 seconds
Uploaded chunk #aa8784.. in 0 seconds
Uploaded chunk #d43536.. in 0 seconds
Uploaded chunk #31d419.. in 0 seconds
After 43.783359486s, verified 9 chunks
9 failed chunks were found, repaid & re-uploaded.
======= Verification: 9 chunks to be checked and repayed if required ===========
==
======= Verification Completed! All chunks have been paid and stored! ==========
===
Uploaded all chunks in 1 minutes 46 seconds
3 Likes

5

Client tolerance is for any price regardless if repayment or not.

If we are given a store cost of 10, right now, we will pay 15 (50% over the cost). (again, down the line this should be configurable).

Repayment will ask for a new store cost, and so will pay that (and the 50% margin). So both those prices you have there have the 50% ‘tolerance’ included.


(Also worth reiterating from the OP. Right now repayments are made in full. They will soon enough be only paying the difference, but that’s not yet implemented).

9 Likes

Another testnet is a great way to start the week. :confetti_ball: :tada:

12 Likes

@joshuef I’m trying to understand if repayment was triggered because price was changed more than by 50%.
From command output it looks like price did not changed that much.
If that’s not the reason, what triggered it then?

5 Likes

Repayment is performed if validation fails. There’s currently no cost comparison of storecost to see if that has changed. It’s a shotgun approach right now.

So it looks like the price has changed (due to the higher price you’re paying).

It may just be that we validated too soon and failed to get a quorum eg. But as I mentioned we’re not checking what we’ve paid previously or checking if that falls within the prior tolerance yet. That will come down the line.


edit: for the interested the current store cost curve looks like this:

/// Table (amount of records stored):
///    1 =         0.000000010
///    2 =         0.000000010
///    4 =         0.000000011
///    8 =         0.000000012
///   16 =         0.000000014
///   32 =         0.000000018
///   64 =         0.000000033
///  128 =         0.000000111
///  256 =         0.000001238
///  512 =         0.000153173
/// 1024 =         2.346196716
/// 1280 =       290.372529764
/// 1536 =     35937.398370712
/// 1792 =   4447723.077333529
/// 2048 = 550463903.051128626 (about 13% of TOTAL_SUPPLY at moment of writing)```
6 Likes

For me, I tried to upload a 700MB movie:

It went fine with the default values, but was a bit on the slow side, so I cancelled it and retried increasing the values to 10 and 40. The I too got this:

Error: 
   0: Transfer Error Failed to send tokens due to Network Error Not enough store cost quotes returned from the network to ensure a valid fee is paid..
   1: Failed to send tokens due to Network Error Not enough store cost quotes returned from the network to ensure a valid fee is paid.
5 Likes

This error i thought was concurrency related. I’m struggling to repro it here. Could you rerun with logs added and grep for Non store cost response received to see if we have errors there.

(Otherwise we must be returning with less quotes than we should have asked for… which is definite bug somewhere)

3 Likes