Royalties2 [Testnet 13/11/23 ] [Offline]

After a couple of weeks of looking to improve the how we utilise gossip and surrounding memory use, we’re back with another testnet.

Again, we’re primarily looking to verify that royalty payments are getting back to maidsafe nodes.

Other interesting updates since the last testnet was launched:

  • We now only pay one node (the cheapest of 3 contacted)
  • Nodes replicate on uploads
  • Clients verify at least two nodes have data.
  • Clients can resume/reuse chunking from cancelled/unsuccessful uploads

Node runners, please note: nodes now will be running with higher memory requirements than in non-gossip testnets, possibly reaching ~200mb/node

Network Details

Node version: 0.98.4
Client version: 0.85.13
Faucet url: 206.189.125.36:8000

We have 101 droplets running a total of 2001 nodes. One droplet has 4vcpu and 8GB of memory.


If you are a regular user, see the ‘quickstart’ section for getting up and running.

If you are a first-time user, or would like more information, see the ‘further information’ section.


Quickstart

If you already have safeup, you can obtain the client and node binaries:

safeup client --version 0.85.13
safeup node --version 0.98.4

Run a Node

Linux/macOS:

safenode

Windows:

safenode

Check local node’s reward balance

Your local node’s peer id will be printed to the terminal on startup with an example command). (You can also retrieve this from the node directory.)

safe wallet balance --peer-id="<local-node-peer-id>"

Connect to the Network as a Client

Linux/macOS:

safe wallet get-faucet 206.189.125.36:8000
safe files upload <directory-path>

Windows:

safe wallet get-faucet 206.189.125.36:8000
safe files upload <directory-path>

To do this with non-default batch-sizes (along with SAFE_PEERS set as above):

safe files upload --batch-size 40 <directory-path> 

40 being the integer value you want to set


Further Information

You can participate in the testnet either by connecting as a client or running your own node.

Connecting as a client requires the safe client binary; running a node requires the safenode binary.

Obtaining Binaries

We have a tool named safeup which is intended to make it easy to obtain the client, node, and other utility binaries.

Installing Safeup

On Linux/macOS, run the following command in your terminal:

curl -sSL https://raw.githubusercontent.com/maidsafe/safeup/main/install.sh | bash

On Windows, run the following command in a Powershell session (be careful to use Powershell, not cmd.exe):

iex (Invoke-RestMethod -Uri "https://raw.githubusercontent.com/maidsafe/safeup/main/install.ps1")

On either platform, you may need to restart your shell session for safeup to become available.

Installing Binaries

After obtaining safeup, you can install binaries like so:

safeup client # get the latest version of the client
safeup client --version 0.85.13 # get a specific version

safeup node # get the latest version of the node
safeup node --version 0.98.4 # get a specific version

safeup update # update all installed components to latest versions

When participating in our testnets, it is recommended to use a specific version. In our project we release a new version of the binaries every time we merge new code. This happens frequently, so over the lifetime of a testnet, many new releases will probably occur. So for participating in this particular testnet, you may not want the latest version.

The binaries are installed to ~/.local/bin on Linux and macOS, and on Windows they go to C:\Users\<username>\safe. Windows doesn’t really have a standard location for binaries that doesn’t require elevated privileges.

The safeup tool will modify the PATH variable on Linux/macOS, or the user Path variable on Windows. The effect of this is that the installed binaries will then become available in your shell without having to refer to them with their full paths. However, if you’re installing for the first time, you may need to start a new shell session for this change to be picked up.

Running a Node

You can participate in the testnet by running your own node. At the moment, you may not be successful if you’re running the node from your home machine. This is a situation we are working on. If you run from a cloud provider like Digital Ocean or AWS, you should be able to participate.

You can run the node process like so:

# Linux/macOS
safenode

# Windows
safenode

This will output all the logs to the filesystem, with the location of logs being platform specific:

# Linux
~/.local/share/safe/node/<peer id>/logs

# macOS
/Users/<username>/Library/Application Support/safe/node/<peer id>/logs

# Windows
C:\Users\<username>\AppData\Roaming\safe\node\<peer-id>\logs

If you wish, you can also provide your own path:

# Linux/macOS
SN_LOG=all safenode --log-output-dest <path>

# Windows
$env:SN_LOG = "all"; safenode --log-output-dest <path>

The advantage of using the predefined data-dir location is you can run multiple nodes on one machine without having to specify your own unique path for each node and manage that overhead yourself.

Connecting as a Client

You can use the safe client binary to connect as a client and upload or download files to/from the network.

Using the Client

You’ll first need to get some Safe Network Tokens:

safe wallet get-faucet 206.189.125.36:8000

You can now proceed to use the client, by, e.g., uploading files:

safe files upload <directory-path>

To download that same content:

safe files download

This will download the files to the default location, which is platform specific:

# Linux
~/.local/share/safe/client/downloaded_files

# macOS
/Users/<username>/Library/Application Support/safe/client/downloaded_files

# Windows
C:\Users\<username>\AppData\Roaming\safe\client\downloaded_files

To download to a particular file or directory:

safe file download [directory/filename] [XORURL]

Troubleshooting

Cleanup

If you’ve used previous versions of the network before and you find problems when running commands, you may want to consider clearing out previous data (worthless DBCs from previous runs, old logs, old keys, etc.).

# Linux
rm -rf ~/.local/share/safe

# macOS
rm -rf ~/Library/Application\ Support/safe

# Windows
rmdir /s C:\Users\<username>\AppData\Roaming\safe
or
rmdir C:\Users\<username>\AppData\Roaming\safe -r -Force

If you encounter a problem running any of our binaries on Windows, it’s possible you need the Visual C++ Redistributable installed.

23 Likes

Because @aatonnomicc is busy elsewhere on the forum, I dived in and uploaded BegBlagandSteal before he did :slight_smile:


🔗 Connected to the Network                                                     Chunking 1 files...
Input was split into 32 chunks
Will now attempt to upload them...
Uploaded 32 chunks in 56 seconds
**************************************
*          Payment Details           *
**************************************
Made payment of 0.000000422 for 32 chunks
New wallet balance: 99.999999578
**************************************
*            Verification            *
**************************************
The current overall verify and repay iteration is 1
32 chunks to be checked and repaid if required
Verified 32 chunks in 40.189919998s
24 chunks were not stored. Repaying them in batches.
⠒ [00:00:57] [#################################>------] 20/24                                                                               Repaid and re-uploaded 32 chunks in 57.461815944s
Retrying chunk upload and payment verification in 2s (retry attempt 1)
24 chunks to be checked and repaid if required
Verified 24 chunks in 29.215615087s
3 chunks were not stored. Repaying them in batches.
⠂ [00:00:03] [----------------------------------------] 0/3                                                                                 Repaid and re-uploaded 24 chunks in 3.456294519s
Retrying chunk upload and payment verification in 2s (retry attempt 2)
3 chunks to be checked and repaid if required
Verified 3 chunks in 6.511840287s
2 chunks were not stored. Repaying them in batches.
⠁ [00:00:02] [----------------------------------------] 0/2                                                                                 Repaid and re-uploaded 3 chunks in 2.868294379s
Retrying chunk upload and payment verification in 2s (retry attempt 3)
2 chunks to be checked and repaid if required
Verified 2 chunks in 4.089073005s
2 chunks were not stored. Repaying them in batches.
⠈ [00:00:02] [----------------------------------------] 0/2                                                                                 Repaid and re-uploaded 2 chunks in 2.725423994s
Retrying chunk upload and payment verification in 2s (retry attempt 4)
2 chunks to be checked and repaid if required
Verified 2 chunks in 3.860698502s
2 chunks were not stored. Repaying them in batches.
⠁ [00:00:02] [----------------------------------------] 0/2                                                                                 Repaid and re-uploaded 2 chunks in 2.983494803s
Retrying chunk upload and payment verification in 2s (retry attempt 5)
2 chunks to be checked and repaid if required
Verified 2 chunks in 4.365174055s
1 chunks were not stored. Repaying them in batches.
⠓ [00:00:02] [----------------------------------------] 0/1                                                                                 Repaid and re-uploaded 2 chunks in 2.40325382s
Retrying chunk upload and payment verification in 2s (retry attempt 6)
1 chunks to be checked and repaid if required
Verified 1 chunks in 1.012402936s
Verification complete: all chunks paid and stored
**************************************
*          Uploaded Files            *
**************************************
"BegBlagAndSteal.mp3" 80744b3d25bab269cab54e8baccf4f54f1aa01615230b99171bc3576c1ca7230

Love to hear from all who try to download this

EDIT: #WorksForMe


......
Client (read all) download progress 31/31
Client downloaded file in 21.342770579s
Saved BegBlagAndSteal.mp3 at /home/willie/.local/share/safe/client/BegBlagAndSteal.mp3
12 Likes

was about to tell you to wind you neck in on another thread but that will have to wait now we have a test net :slight_smile:

8 Likes

Feel free but you have to buy me beer first :slight_smile: :beers: :beers:

7 Likes

A new testnet, a new vdash so you can add nodes without restarting.

10 Likes

Great!

Concurrency setting is back?

5 Likes

I’m running this simple script on a Hetzner cloud instance

#/bin/bash

baseport=15000


cd  ~/safe_network

for i in  {1..50}
do
        port=$(($baseport + $i))
        echo $port
        cargo run --release  --bin safenode  --features=network-contacts,open-metrics -- --port=$port &
        sleep 5
done

Getting lots of errors like

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
The application panicked (crashed).
Message:  cannot sample empty range
Location: /home/safe/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rand-0.8.5/src/rng.rs:134

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
The application panicked (crashed).
Message:  cannot sample empty range
Location: /home/safe/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rand-0.8.5/src/rng.rs:134

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

Should I forget about open-metrics for now?

1 Like

Ah no, that should not be in there :thinking: Odd copy paste fail from templates here.

edit: updated op

4 Likes

:tada: whooo back in action.

Noticing some nodes that are very well connected, those are also a little greedy when it comes to CPU.

Global (UTC) Timestamp: Mon Nov 13 15:31:37 UTC 2023
Number: 0
Node: 12D3KooWAiLhv4niRBt2NsHKjiezrqdKU3TJ9xE9UUm4pj5BfXiR
PID: 14390
Status: running
Memory used: 222.461MB
CPU usage: 20.1%
TCP connections (established): 2178
Bandwidth Usage:  KB/s
Records: 571
Disk usage: 208M
Rewards balance: 0.000000000

Some others that stand out:

Records: 1406 already nearly full?

Memory used: 210.734MB
CPU usage: 33.4%
TCP connections (established): 2035
7 Likes

NOT A BUG :blush:

I did this before - forgot to delete the client directory, so this is solved with rm -rf ~/.local/share/safe/client.

Again I wonder if the client can check for this and give a better error because I’m not smart enough to remember this is there’s not at least one testnet every two days! :man_shrugging:

First… BUG!!!

safe -V
sn_cli 0.85.13
safe wallet get-faucet 206.189.125.36:8000

[snip]

Connected to the Network                                                     Error: 
   0: Bincode error:: invalid value: integer `2803815042`, expected variant index 0 <= i < 2
   1: invalid value: integer `2803815042`, expected variant index 0 <= i < 2

Location:
   sn_cli/src/subcommands/wallet.rs:188

Note that on my cloud instance I used the earlier client and it reports an error and then claims to have worked. :thinking: But in fact it hasn’t increased the wallet balance, so that may be a second bug!

safe -V
sn_cli 0.83.45
equesting token for wallet address: a45fa814aa69afa96f40bd633766bfcc7ef2ff714b495fea6d8dc08b2da8abf6b4090969edb3d853e3c3a86cd3ce47f9...
Failed to parse transfer: TransferDeserializationFailed
Transfer: "ab25b4536d316e3771d614685d98d80c7f2aecf468cb5395582ea5701c4a65c99a7c8e7f89def3385e1c06d805ddf904613d4f1deac81a40509ae552e8d31118bcb07e70e64f223c2a6aaa9a512d21f7af2b4f2b1d2e6703ed870b1859a3fd863046f04d80738e36edb312fc7dfe1f2f1e1d24259c6cc09ab04b7996c2427b9ecf0e7dc4fb6d9b0bfdbaae35432678ff80f39f34531fe5684edd1732ec3c9f593711386e44144f602fc24b7f3c38a5486504c083b2b97ca2d20705dc919fddc3acf7875738e540af2c0dd6f1000000000000006c15a281ef43232f1500f75580f50577d126d7530f818bc192173c16135d8a80fc9423ed57a970fd3a8287fd35d62b3b84000000000000000100000000"
Successfully got tokens from faucet.
3 Likes

I am noticing big peaks in memory usage. One of four nodes on this machine got heavy like this and than returned to ~100 MB. On other machines I am seeing nodes go over 600 MB of RAM usage.

1 Like

I’m having problems with safeup.

On my laptop it installed the new versions correctly, but on my cloud instance it is failing to update the node with:

safeup node --version 0.98.4
**************************************
*                                    *
*          Installing safenode       *
*                                    *
**************************************
Installing safenode for x86_64-unknown-linux-musl at /home/safe/.local/bin...
  [########################################] 9.00 MiB/9.00 MiB
Error: Text file busy (os error 26)

SOLVED:
This happens if the safenode binary is in use (running) so is solved with pkill safenode.

@chriso maybe the error message can be improved?

5 Likes

Getting a wallet error on the client on my cloud instance

safe@slimming-aid:~$ safe wallet get-faucet 206.189.125.36:8000
Logging to directory: "/home/safe/.local/share/safe/client/logs/log_2023-11-13_16-00-22"
Built with git version: de52073 / main / de52073
Instantiating a SAFE client...
Trying to fetch the bootstrap peers from https://sn-testnet.s3.eu-west-2.amazonaws.com/network-contacts

snip


🔗 Connected to the Network                                                                                                                 Requesting token for wallet address: 8e18ea19a2c5cccbf486266cba96dc2622b525af19c388643ed994ae11851da78a38ac22a7026009894bd4c237072970...
Failed to get tokens from faucet, server responded with: Failed to send tokens: Transfer Error Failed to send tokens due to The transfer was not successfully registered in the network: CouldNotSendMoney("Network Error Could not retrieve the record after storing it: f664369850290eb1c83a2e200a271b649922388feb71121e88cb13617ea3970c(d42869ce9d2ffde662c41bf0d78082f35a7bd78eda282676c1c7a065e4eacc54).").

Worked fine from my client at home…

EDIT: tried again and it works as expected


🔗 Connected to the Network                                                                                                                 Requesting token for wallet address: 8e18ea19a2c5cccbf486266cba96dc2622b525af19c388643ed994ae11851da78a38ac22a7026009894bd4c237072970...
Successfully parsed transfer. 
Verifying transfer with the Network...
Successfully verified transfer.
Successfully stored cash_note to wallet dir. 
Old balance: 0.000000000
New balance: 100.000000000
Successfully got tokens from faucet.
2 Likes

Now some real problems, IDK :man_shrugging:

On my cloud node I tried the faucet a second time and it failed:

Successfully parsed transfer. 
Verifying transfer with the Network...
Failed to verify and redeem transfer: CouldNotReceiveMoney("InvalidTransfer(\"Spend not found: SpendAddress(8ec9af(10001110)..)\")")
Error: 
   0: Failed to receive transfer due to InvalidTransfer("Spend not found: SpendAddress(8ec9af(10001110)..)")
Location:
/rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/convert/mod.rs:716
Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

I’ve also lost five out of 20 nodes (which was an acceptable number in the past, except for the last testnet). Now seven of 20. They’re dying like flies!

All the dead nodes had received chunks though some with chunks are still running.

Dead nodes show between 78MB and 157MB RAM.
Live nodes with PUTS show 102MB to 132MB
Live nodes with zero PUTS show 72MB to 96MB

I’ve now lost 13/20 nodes. I also notice that my nodes with PUTS are either showing StoreCost of 0 or quite low (10 to 14). None of my nodes have been paid 75% of them having from 537 to 1395 PUTS.

@joshuef I’ve not looked in the logfiles but it appears that the storecost logging has been reduced, or something is not working as expected? See:

4 Likes

My upload seems to be looping, endlessly verifying the last 15 chunks

15 chunks were not stored. Repaying them in batches.
⠂ [00:01:42] [########>-------------------------------] 3/15                                                                                        Repaid and re-uploaded 15 chunks in 102.231246514s
Retrying chunk upload and payment verification in 2s (retry attempt 8)
15 chunks to be checked and repaid if required
Verified 15 chunks in 10.80776088s
15 chunks were not stored. Repaying them in batches.
⠠ [00:00:50] [----------------------------------------] 0/15                                                                                        Repaid and re-uploaded 15 chunks in 50.9745712s
Retrying chunk upload and payment verification in 2s (retry attempt 9)
15 chunks to be checked and repaid if required
Verified 15 chunks in 20.047964868s
15 chunks were not stored. Repaying them in batches.
⠁ [00:01:03] [##>-------------------------------------] 1/15                                                                                        Repaid and re-uploaded 15 chunks in 63.910622481s
Failed to verify and repay all chunks after 10 retries.
The current overall verify and repay iteration is 3
15 chunks to be checked and repaid if required
Verified 15 chunks in 10.478068962s
15 chunks were not stored. Repaying them in batches.
⠄ [00:00:59] [################>-----------------------] 6/15                                                                                        Repaid and re-uploaded 15 chunks in 59.210778409s
Retrying chunk upload and payment verification in 2s (retry attempt 1)
15 chunks to be checked and repaid if required
Verified 15 chunks in 18.111535523s
15 chunks were not stored. Repaying them in batches
5 Likes

@JPL Thank you for uploading, we need EVERYONE to upload as much as possible.

My 50 nodes in the cloud are not doing too well


I see have zero earnings but a max storage cost of 13 nanos. How does that happen?
This cloud box is running with the full grafana services. Correct me if Im wrong but grafana and monitoring in general is not the focus of this test so I am about to disable these services for now,

EDIT now down to 3 active nodes out of 50 and the load average is wild.
50 nodes seems to just fit into the 4GB RAM of this instance

4 Likes

Is 2 hour mark cursed? )
I can’t get-faucet now:

?? Connected to the Network
Requesting token for wallet address: b6d11c13b1621c4fe64c15d10c3a69accaae091b8d6
29d751dec9f86976191a7445c158a3bc4a42345f5b5a02bef354a...
Failed to get tokens from faucet, server responded with: Failed to send tokens:
Transfer Error Failed to send tokens due to The transfer was not successfully re
gistered in the network: CouldNotSendMoney("Network Error Could not retrieve the
 record after storing it: dd9dfd39e504915c5e60e5054e8dae63603135622ff998fdad1d5d
e905a4f2a2(e42fe9220e71ea44fdab27313de78165036e2a603be08400085f265f431dec24).").

Tried 2 times.

2 Likes

My upload from home may take some time…

You are getting far better RAM numbers there than me, so 40 in 4GB looking at my nodes is impossible, my average is ~500MB.

ps -p 14379 -o rss= | awk '{print $1/1024 " MB"}'
721.746 MB


This guy just got killed:
Number: 2
Node: 12D3KooWDr89k64nMDz2XtKwW9P8s1ZccgdShbsLz2tCY5pBCr2Q
PID: 14367
Status: running
Memory used: 1446.92MB
CPU usage: 32.3%
3 Likes