NoClientGossipDiscovery [24/11/23 Testnet] [Offline]

Hot on the heals of NoClientGossip, we’ve a tweaked version of the code which aims to improve the network discovery setup.

As mentioned in the update, it looks like some nodes did not know enough of the network, leading to assumptions of responsibility over more and more data, and so filling up and failing to store new data.

So here we want to keep testing uploads, royalties, and will be keeping an eye on how the kademlia network kbuckets fill up over time.

If you’re running a node, memory still looks to be in the 200-300 range.

And we’ve a sample 5gb file uploaded:

safe files download ubuntu.iso 3911604a5798de6e7f591a6c9e44097e9f4979c0a1ef439d02d17494d7f41886

Network Details

Node version: 0.98.32
Client version: 0.86.20
Faucet url: 167.71.141.88:8000

We have 101 droplets running a total of 2001 nodes. One droplet has 4vcpu and 8GB of memory.


If you are a regular user, see the ‘quickstart’ section for getting up and running.

If you are a first-time user, or would like more information, see the ‘further information’ section.


Quickstart

If you already have safeup, you can obtain the client and node binaries:

safeup client --version 0.86.20
safeup node --version 0.98.32

Run a Node

Linux/macOS:

safenode

Windows:

safenode

Check local node’s reward balance

Your local node’s peer id will be printed to the terminal on startup with an example command). (You can also retrieve this from the node directory.)

safe wallet balance --peer-id="<local-node-peer-id>"

Connect to the Network as a Client

Linux/macOS:

safe wallet get-faucet 167.71.141.88:8000
safe files upload <directory-path>

Windows:

safe wallet get-faucet 167.71.141.88:8000
safe files upload <directory-path>

To do this with non-default concurrency or batch-sizes (along with SAFE_PEERS set as above):

safe files upload -c 10 --batch-size 40 <directory-path> 

10 and 40 being the integer values you want to set for each


Further Information

You can participate in the testnet either by connecting as a client or running your own node.

Connecting as a client requires the safe client binary; running a node requires the safenode binary.

Obtaining Binaries

We have a tool named safeup which is intended to make it easy to obtain the client, node, and other utility binaries.

Installing Safeup

On Linux/macOS, run the following command in your terminal:

curl -sSL https://raw.githubusercontent.com/maidsafe/safeup/main/install.sh | bash

On Windows, run the following command in a Powershell session (be careful to use Powershell, not cmd.exe):

iex (Invoke-RestMethod -Uri "https://raw.githubusercontent.com/maidsafe/safeup/main/install.ps1")

On either platform, you may need to restart your shell session for safeup to become available.

Installing Binaries

After obtaining safeup, you can install binaries like so:

safeup client # get the latest version of the client
safeup client --version 0.86.20 # get a specific version

safeup node # get the latest version of the node
safeup node --version 0.98.32 # get a specific version

safeup update # update all installed components to latest versions

When participating in our testnets, it is recommended to use a specific version. In our project we release a new version of the binaries every time we merge new code. This happens frequently, so over the lifetime of a testnet, many new releases will probably occur. So for participating in this particular testnet, you may not want the latest version.

The binaries are installed to ~/.local/bin on Linux and macOS, and on Windows they go to C:\Users\<username>\safe. Windows doesn’t really have a standard location for binaries that doesn’t require elevated privileges.

The safeup tool will modify the PATH variable on Linux/macOS, or the user Path variable on Windows. The effect of this is that the installed binaries will then become available in your shell without having to refer to them with their full paths. However, if you’re installing for the first time, you may need to start a new shell session for this change to be picked up.

Running a Node

You can participate in the testnet by running your own node. At the moment, you may not be successful if you’re running the node from your home machine. This is a situation we are working on. If you run from a cloud provider like Digital Ocean or AWS, you should be able to participate.

You can run the node process like so:

# Linux/macOS
safenode

# Windows
safenode

This will output all the logs to the filesystem, with the location of logs being platform specific:

# Linux
~/.local/share/safe/node/<peer id>/logs

# macOS
/Users/<username>/Library/Application Support/safe/node/<peer id>/logs

# Windows
C:\Users\<username>\AppData\Roaming\safe\node\<peer-id>\logs

If you wish, you can also provide your own path:

# Linux/macOS
SN_LOG=all safenode --log-output-dest <path>

# Windows
$env:SN_LOG = "all"; safenode --log-output-dest <path>

The advantage of using the predefined data-dir location is you can run multiple nodes on one machine without having to specify your own unique path for each node and manage that overhead yourself.

Connecting as a Client

You can use the safe client binary to connect as a client and upload or download files to/from the network.

Using the Client

You’ll first need to get some Safe Network Tokens:

safe wallet get-faucet 167.71.141.88:8000

You can now proceed to use the client, by, e.g., uploading files:

safe files upload <directory-path>

To download that same content:

safe files download

This will download the files to the default location, which is platform specific:

# Linux
~/.local/share/safe/client/downloaded_files

# macOS
/Users/<username>/Library/Application Support/safe/client/downloaded_files

# Windows
C:\Users\<username>\AppData\Roaming\safe\client\downloaded_files

To download to a particular file or directory:

safe file download [directory/filename] [NetworkAddress]

Troubleshooting

Cleanup

If you’ve used previous versions of the network before and you find problems when running commands, you may want to consider clearing out previous data (worthless DBCs from previous runs, old logs, old keys, etc.).

# Linux
rm -rf ~/.local/share/safe

# macOS
rm -rf ~/Library/Application\ Support/safe

# Windows
rmdir /s C:\Users\<username>\AppData\Roaming\safe

If you encounter a problem running any of our binaries on Windows, it’s possible you need the Visual C++ Redistributable installed.

20 Likes

Jippy too short to reply :partying_face:

Now test :sweat_smile: :beers:

safe files download “catub.jpeg” 9693f3253c24c77395d4d5c3eb876bdfa432ddc9c77c25a142c1006b28be41e8 :beer:

Uploaded 434 chunks in 10 minutes 2 seconds
safe files download “Ourwealth.mp4” d085b50835d313aace0592f9ed127c8ab766126d1f91376fc1ca81224f677b31

14 Likes

Second incoming :slight_smile:

10 Likes

Beg blag test is a pass :slight_smile:

just joining nodes now

safe files download "BegBlagandSteal.mp3" 80744b3d25bab269cab54e8baccf4f54f1aa01615230b99171bc3576c1ca7230
7 Likes

2 Likes

Downloading Ourwealth.mp4 from d085b50835d313aace0592f9ed127c8ab766126d1f91376fc1ca81224f677b31 with batch-size 16
Client (read all) download progress 1/433

Client (read all) download progress 79/433
Error downloading “Ourwealth.mp4”: Chunks error Chunk could not be retrieved from the network: b7f59b(10110111)… :face_with_raised_eyebrow:

5 Likes

Same for me.

3 Likes

Another failing file:
BigBuckBunny_320x180.mp4 4196cde29e614825d51328a3a52e53721d180d481f12ef1f1d003573919dcd2e
up.zip (273.7 KB)
d.zip (43.5 KB)

Looks like no need to wait for 2 hours this time.

5 Likes

Downloading BigBuckBunny_320x180.mp4 from 4196cde29e614825d51328a3a52e53721d180d481f12ef1f1d003573919dcd2e with batch-size 16
Client (read all) download progress 1/124

Client (read all) download progress 122/124
Error downloading “BigBuckBunny_320x180.mp4”: Chunks error Chunk could not be retrieved from the network: e4d634(11100100)…

4 Likes

So far this testnet looks runs very smooth. I am currently running nodes on 3 machines (40+20+3) and CPU, RAM and bandwidth usage is much lover than last time.

9 Likes

Eehhhmm, I may have started some nodes with older versions and killed them, as well as newest versions and killed those too… :grimacing: Does this impact the network at an early stage?

3 Likes

Unless it was 1000 of them it shouldn’t do much.

4 Likes

No, just 2 or 3.

1 Like

I have the same errors while downloading the files linked above, at the same chunks.

Uploading and chunking process is pretty quick though :smiley:

3 Likes

Downloaded fine!

The files that work seem to download pretty fast. Also the faucet was much faster than in earlier testnets. Connectig to the network seems much faster too, and so does chunking. :+1:

There is again a mention of concurrency at the end of OP, @joshuef.

11 Likes

safe files download "Intro to Large Language Models.webm" 0cda933805d4baec9951d10ac1864cd36f5e000a09e8ae3af72219f415a056a7

1 Like

My 3 nodes are all connected to the network and are receiving chunks! :boom: Hooray :partying_face:

9 Likes

Downloaded fine :tada:

Client downloaded file in 174.059439215s
Saved Intro to Large Language Models.webm at /home/pi/.local/share/safe/client/Intro to Large Language Models.webm

And pretty fast too, 165MB in 174 seconds

8 Likes

On a pi? 11 sec on a cloud instance.

3 Likes