MoreStabilityNet [19/07/23 Testnet] [Offline]

Our next testnet, MoreStabilityNet, is looking to see how node stability has been affected by recent fixes.

The NodeDiscoveryNet has done well, surfacing data all the way through, even though we lost 12/20 nodes per droplet due to excessive memory.

We’ve a few fixes in and more pending that should improve things here. And we’d like to see how this goes for our MaidSafe Digital Ocean droplets and for those starting nodes in the community.

With all that, our objectives for this testnet are relatively simple:

  • Verify if we still have “lost nodes” and get more data there (we may be asking for some specific types of behaviour to be tested as we go foward).

For those wondering what a lost node might look like: It’s basically a node dying or being killed due to excessive memory usage (>400mb for a long time eg). If you’re only runnning a node at home, or one per machine, you should likely not see this. If you run many nodes per machine, it’s more likely that a spike might cause the operating system to kill your node.

Network Details

Node version: 0.86.25
Client version: 0.79.26
SAFE_PEERS: /ip4/176.58.124.107/tcp/44439/p2p/12D3KooWEwMPj64C1MXEKaCYHGgtMApNzzvoynfprLJRLfgfGo7G
Alternatives:

 "/ip4/161.35.39.103/tcp/38297/p2p/12D3KooWCgxHVhPfSndYod2KzRivZF5AnF125R5WoZt5GbX7JMtB"
 "/ip4/143.110.169.215/tcp/38247/p2p/12D3KooWHSrZhRZSstxdhHGU3Vfj6Czxp12zqepsLvhFEfGS9R5Q"
 "/ip4/161.35.39.22/tcp/36197/p2p/12D3KooWT1ywindpChtzfaWHfCFWexGfxYG1AVNQxwyEwR2cZwR4"
 "/ip4/165.232.101.63/tcp/46029/p2p/12D3KooWGEBCRAmekSwfa7yidNNYbXtRDh1JPP14mntgtJ1yGChb"
 "/ip4/178.62.91.4/tcp/39243/p2p/12D3KooWGXKWUfiCvEevBaT9VnmR7qVihyQD1PaJThzG4LLewJCU"
 "/ip4/165.232.101.62/tcp/34009/p2p/12D3KooWAUuNGrFXux6Fx9kJGSzRxM6fXr1ZBvjY64ujqx3MwjSv"
 "/ip4/161.35.39.101/tcp/40003/p2p/12D3KooWEPNrJmZyhRyEqvFkSng8v4yVuQ1ZJDu2PerLRpaQkELH"

We have 100 droplets running a total of ~2000 nodes. One droplet has 2vcpu and 4GB of memory.


If you are a regular user, see the ‘quickstart’ section for getting up and running.

If you are a first-time user, or would like more information, see the ‘further information’ section.


Quickstart

If you already have safeup, you can obtain the client and node binaries:

safeup client --version 0.79.26
safeup node --version 0.86.25

Run a Node

Linux/macOS:

SN_LOG=all safenode

Windows:

$env:SN_LOG = "all" ; safenode

Connect to the Network

Linux/macOS:

export SAFE_PEERS="/ip4/176.58.124.107/tcp/44439/p2p/12D3KooWEwMPj64C1MXEKaCYHGgtMApNzzvoynfprLJRLfgfGo7G"
safe files upload <directory-path>

Windows:

$env:SAFE_PEERS = "/ip4/176.58.124.107/tcp/44439/p2p/12D3KooWEwMPj64C1MXEKaCYHGgtMApNzzvoynfprLJRLfgfGo7G"
safe files upload <directory-path>

Further Information

You can participate in the testnet either by connecting as a client or running your own node.

Connecting as a client requires the safe client binary; running a node requires the safenode binary.

Obtaining Binaries

We have a tool named safeup which is intended to make it easy to obtain the client, node, and other utility binaries.

Installing Safeup

On Linux/macOS, run the following command in your terminal:

curl -sSL https://raw.githubusercontent.com/maidsafe/safeup/main/install.sh | bash

On Windows, run the following command in a Powershell session (be careful to use Powershell, not cmd.exe):

iex (Invoke-RestMethod -Uri "https://raw.githubusercontent.com/maidsafe/safeup/main/install.ps1")

On either platform, you may need to restart your shell session for safeup to become available.

Installing Binaries

After obtaining safeup, you can install binaries like so:

safeup client # get the latest version of the client
safeup client --version 0.79.26 # get a specific version

safeup node # get the latest version of the node
safeup node --version 0.86.25 # get a specific version

safeup update # update all installed components to latest versions

When participating in our testnets, it is recommended to use a specific version. In our project we release a new version of the binaries every time we merge new code. This happens frequently, so over the lifetime of a testnet, many new releases will probably occur. So for participating in this particular testnet, you may not want the latest version.

The binaries are installed to ~/.local/bin on Linux and macOS, and on Windows they go to C:\Users\<username>\safe. Windows doesn’t really have a standard location for binaries that doesn’t require elevated privileges.

The safeup tool will modify the PATH variable on Linux/macOS, or the user Path variable on Windows. The effect of this is that the installed binaries will then become available in your shell without having to refer to them with their full paths. However, if you’re installing for the first time, you may need to start a new shell session for this change to be picked up.

Running a Node

You can participate in the testnet by running your own node. At the moment, you may not be successful if you’re running the node from your home machine. This is a situation we are working on. If you run from a cloud provider like Digital Ocean or AWS, you should be able to participate.

You can run the node process like so:

# Linux/macOS
SN_LOG=all safenode

# Windows
$env:SN_LOG = "all"; safenode

This will output all the logs in the terminal.

Sometimes it will be preferable to output the logs to file. You can do this by running the node like so:

# Linux/macOS
SN_LOG=all safenode --log-output-dest data-dir

# Windows
$env:SN_LOG = "all"; safenode --log-output-dest data-dir

The location of data-dir is platform specific:

# Linux
~/.local/share/safe/node/<peer id>/logs

# macOS
/Users/<username>/Library/Application Support/safe/node/<peer id>/logs

# Windows
C:\Users\<username>\AppData\Roaming\safe\node\<peer-id>\logs

If you wish, you can also provide your own path:

# Linux/macOS
SN_LOG=all safenode --log-output-dest <path>

# Windows
$env:SN_LOG = "all"; safenode --log-output-dest <path>

The advantage of using the predefined data-dir location is you can run multiple nodes on one machine without having to specify your own unique path for each node and manage that overhead yourself.

Connecting as a Client

You can use the safe client binary to connect as a client and upload or download files to/from the network.

To connect, you must provide another peer, in the form of a multi address. You can find one in the ‘Network Details’ section at the top.

It is recommended to set the peer using the environment variable SAFE_PEERS. You can set this variable once and it will apply for the duration of your shell session:

# Linux/macOS
export SAFE_PEERS="/ip4/176.58.124.107/tcp/44439/p2p/12D3KooWEwMPj64C1MXEKaCYHGgtMApNzzvoynfprLJRLfgfGo7G"

# Windows
$env:SAFE_PEERS = "/ip4/176.58.124.107/tcp/44439/p2p/12D3KooWEwMPj64C1MXEKaCYHGgtMApNzzvoynfprLJRLfgfGo7G"

NOTE: If you close and/or start a new shell session, you will be required to redefine this environment variable in the new session.

As an alternative to the environment variable, it’s also possible to use the --peer argument:

safe --peer="/ip4/176.58.124.107/tcp/44439/p2p/12D3KooWEwMPj64C1MXEKaCYHGgtMApNzzvoynfprLJRLfgfGo7G" ...

However, this requires specifying the peer with each command.

Using the Client

You can now proceed to use the client, by, e.g., uploading files:

safe files upload <directory-path>

To download that same content:

safe files download

This will download the files to the default location, which is platform specific:

# Linux
~/.local/share/safe/client/downloaded_files

# macOS
/Users/<username>/Library/Application Support/safe/client/downloaded_files

# Windows
C:\Users\<username>\AppData\Roaming\safe\client\downloaded_files

To download to a particular file or directory:

safe file download [directory/filename] [XORURL]

Troubleshooting

If you encounter a problem running any of our binaries on Windows, it’s possible you need the Visual C++ Redistributable installed.

30 Likes

First :sunglasses: I like the name of this one !! Count me in.

Edit :
The safeup tool is neat !

But the network is not a big success for me so far, I could upload a small file but download failed

Storing file ‘safe.jpeg’ of 226719 bytes (4 chunk/s)…
Successfully stored ‘safe.jpeg’ to 89bf86508e7054781eb0db7cd32a11aa26cd9ea22d300d80a00965e252727ad7


Downloading file “safe2.jpeg” with address 89bf86508e7054781eb0db7cd32a11aa26cd9ea22d300d80a00965e252727ad7
Client download progress 2/3
Did not get file “safe2.jpeg” from the network! Chunks error Not all chunks were retrieved, expected 3, retrieved 2, missing [3f40e2(00111111)…]…

I could upload and download another jpeg of 4mb though!

I also tried to upload a 39mb file but probably too ambitious for now :

sudo safe --peer=“/ip4/143.110.169.218/tcp/38599/p2p/12D3KooWFUp78dnwMh6Hpne4ZPG5kH3Y11wDrcyAfkaXU8V4zdh5” files upload ~/safenetwork.mp4
Built with git version: 81a42f2 / main / 81a42f2
Instantiating a SAFE client…
⢀ Connecting to The SAFE Network… The client still does not know enough network nodes.
:link: Connected to the Network Preparing (chunking) files at ‘/Users/x/safenetwork.mp4’…
Storing file ‘safenetwork.mp4’ of 39372796 bytes (78 chunk/s)…
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
[…]
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Successfully stored ‘safenetwork.mp4’ to c7a3c7602369aba9b9721ceb998a68f2c9d74d8eb1dd269b9e2833a3bba6ec98
Writing 63 bytes to “/Users/x/Library/Application Support/safe/client/uploaded_files/file_names_2023-07-19_08-26-58”
Error: Too many open files (os error 24)
Location:
sn_cli/src/subcommands/files.rs:122:5

I didn’t have this “too many open files” issue on previous testnets (with same config)

13 Likes

Second!!! :fire:

13 Likes

God damn it, this is likely tcp not behaving. It should not be opening sockets like this. At least my suspicion, need to check logs

Thanks for pushing on this

12 Likes

Hmm, we should be opening less not as we only attempt to connect to publicly available IPs. So odd that would be happening, but yeh, that does sounds like TCP.

Odd that would be happening for the client though. Hmmm. How big is the file? Can you provide some client logs for this @Atom ?

10 Likes

The file is 39mb;
I couldn’t find a log file, should there be one in the client folder ?
CleanShot 2023-07-19 at 11.19.43

My full chaotic zsh session (I’m on mac) :

Summary

Last login: Wed Jul 19 10:20:20 on ttys004
➜ ~ safeup client
Installing safe
Installing safe for x86_64-apple-darwin at /usr/local/bin…
[########################################] 4.64 MiB/4.64 MiB Error: Permission denied (os error 13)
Location:
src/install.rs:198:39
➜ ~ sudo safeup client
Installing safe for x86_64-apple-darwin at /usr/local/bin…
[########################################] 4.64 MiB/4.64 MiB safe 0.79.26 is now available at /usr/local/bin/safe
➜ ~ export SAFE_PEERS=“/ip4/143.110.169.218/tcp/38599/p2p/12D3KooWFUp78dnwMh6Hpne4ZPG5kH3Y11wDrcyAfkaXU8V4zdh5”
➜ ~ safe files upload ~/1.jpeg
Built with git version: 81a42f2 / main / 81a42f2
Instantiating a SAFE client…
⠠ Connecting to The SAFE Network… The client still does not know enough network nodes.
:link: Connected to the Network Error: Unable to read wallet file in {path:?}
Caused by:
0: I/O error: Permission denied (os error 13)
1: Permission denied (os error 13)
Location:
sn_cli/src/subcommands/wallet.rs:188:10
➜ ~ sudo safe files upload ~/1.jpeg
Built with git version: 81a42f2 / main / 81a42f2
Instantiating a SAFE client…
Error: No peers given. As local-discovery feature is disabled, we will not be able to connect to the network.
Location:
sn_cli/src/main.rs:58:24
➜ ~ sudo safe files upload ~/1.jpg
Built with git version: 81a42f2 / main / 81a42f2
Instantiating a SAFE client…
Error: No peers given. As local-discovery feature is disabled, we will not be able to connect to the network.
Location:
sn_cli/src/main.rs:58:24
➜ ~ sudo safe --peer=“/ip4/143.110.169.218/tcp/38599/p2p/12D3KooWFUp78dnwMh6Hpne4ZPG5kH3Y11wDrcyAfkaXU8V4zdh5” files upload ~/1.jpg
Built with git version: 81a42f2 / main / 81a42f2
Instantiating a SAFE client…
⢀ Connecting to The SAFE Network… The client still does not know enough network nodes.
:link: Connected to the Network Preparing (chunking) files at ‘/Users/x/1.jpg’…
Storing file ‘1.jpg’ of 4775902 bytes (11 chunk/s)…
Successfully stored ‘1.jpg’ to 64e2de2752aa2cb227f4dcd8ec330dcc1229d1801f1f9c772cd263b99b41de2d
Writing 53 bytes to “/Users/x/Library/Application Support/safe/client/uploaded_files/file_names_2023-07-19_10-26-02”
➜ ~ sudo safe --peer=“/ip4/143.110.169.218/tcp/38599/p2p/12D3KooWFUp78dnwMh6Hpne4ZPG5kH3Y11wDrcyAfkaXU8V4zdh5” files upload ~/safenetwork.mp4
Built with git version: 81a42f2 / main / 81a42f2
Instantiating a SAFE client…
⢀ Connecting to The SAFE Network… The client still does not know enough network nodes.
:link: Connected to the Network Preparing (chunking) files at ‘/Users/x/safenetwork.mp4’…
Storing file ‘safenetwork.mp4’ of 39372796 bytes (78 chunk/s)…
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Successfully stored ‘safenetwork.mp4’ to c7a3c7602369aba9b9721ceb998a68f2c9d74d8eb1dd269b9e2833a3bba6ec98
Writing 63 bytes to “/Users/x/Library/Application Support/safe/client/uploaded_files/file_names_2023-07-19_08-26-58”
Error: Too many open files (os error 24)
Location:
sn_cli/src/subcommands/files.rs:122:5
➜ ~ sudo safe --peer=“/ip4/143.110.169.218/tcp/38599/p2p/12D3KooWFUp78dnwMh6Hpne4ZPG5kH3Y11wDrcyAfkaXU8V4zdh5” files download c7a3c7602369aba9b9721ceb998a68f2c9d74d8eb1dd269b9e2833a3bba6ec98 > testsafenetwork.mp4
:link: Connected to the Network % ➜ ~ sudo safe --peer=“/ip4/143.110.169.218/tcp/38599/p2p/12D3KooWFUp78dnwMh6Hpne4ZPG5kH3Y11wDrcyAfkaXU8V4zdh5” files download 64e2de2752aa2cb227f4dcd8ec330dcc1229d1801f1f9c772cd263b99b41de2d > test.jpeg
:link: Connected to the Network % ➜ ~ sudo safe --peer=“/ip4/143.110.169.218/tcp/38599/p2p/12D3KooWFUp78dnwMh6Hpne4ZPG5kH3Y11wDrcyAfkaXU8V4zdh5” files upload testimage.jpg
Built with git version: 81a42f2 / main / 81a42f2
Instantiating a SAFE client…
⠂ Connecting to The SAFE Network… ⢀ Connecting to The SAFE Network… The client still does not know enough network nodes.
:link: Connected to the Network Preparing (chunking) files at ‘testimage.jpg’…
Storing file ‘testimage.jpg’ of 6755742 bytes (15 chunk/s)…
Successfully stored ‘testimage.jpg’ to 60fcfc24c2593e50c95e3c8f4f0b047a88f80c372ebcaac9fb245583d47b776a
Writing 61 bytes to “/Users/x/Library/Application Support/safe/client/uploaded_files/file_names_2023-07-19_10-31-31”
➜ ~ sudo safe --peer=“/ip4/143.110.169.218/tcp/38599/p2p/12D3KooWFUp78dnwMh6Hpne4ZPG5kH3Y11wDrcyAfkaXU8V4zdh5” files download 60fcfc24c2593e50c95e3c8f4f0b047a88f80c372ebcaac9fb245583d47b776a > retest.jpg
:link: Connected to the Network Error: Too many open files (os error 24)
Location:
sn_cli/src/subcommands/files.rs:165:47
➜ ~ sudo safe --peer=“/ip4/143.110.169.218/tcp/38599/p2p/12D3KooWFUp78dnwMh6Hpne4ZPG5kH3Y11wDrcyAfkaXU8V4zdh5” files download test123.jpg 60fcfc24c2593e50c95e3c8f4f0b047a88f80c372ebcaac9fb245583d47b776a
Built with git version: 81a42f2 / main / 81a42f2
Instantiating a SAFE client…
⢀ Connecting to The SAFE Network… The client still does not know enough network nodes.
:link: Connected to the Network Downloading file “test123.jpg” with address 60fcfc24c2593e50c95e3c8f4f0b047a88f80c372ebcaac9fb245583d47b776a
Client download progress 9/14
Did not get file “test123.jpg” from the network! Chunks error Not all chunks were retrieved, expected 14, retrieved 9, missing [771aa1(01110111)…, f9bfce(11111001)…, 0f7675(00001111)…, 5e7e28(01011110)…, b4008a(10110100)…]…
➜ ~ sudo safe --peer=“/ip4/143.110.169.218/tcp/38599/p2p/12D3KooWFUp78dnwMh6Hpne4ZPG5kH3Y11wDrcyAfkaXU8V4zdh5” files download test.jpeg 64e2de2752aa2cb227f4dcd8ec330dcc1229d1801f1f9c772cd263b99b41de2d
Password:
Built with git version: 81a42f2 / main / 81a42f2
Instantiating a SAFE client…
⢀ Connecting to The SAFE Network… The client still does not know enough network nodes.
:link: Connected to the Network Downloading file “test.jpeg” with address 64e2de2752aa2cb227f4dcd8ec330dcc1229d1801f1f9c772cd263b99b41de2d
Client download progress 10/10
Successfully got file test.jpeg!
Writing 4775902 bytes to “/Users/x/Library/Application Support/safe/client/test.jpeg”
➜ ~ sudo safe --peer=“/ip4/143.110.169.218/tcp/38599/p2p/12D3KooWFUp78dnwMh6Hpne4ZPG5kH3Y11wDrcyAfkaXU8V4zdh5” files download safenetwork.mp4 c7a3c7602369aba9b9721ceb998a68f2c9d74d8eb1dd269b9e2833a3bba6ec98
Built with git version: 81a42f2 / main / 81a42f2
Instantiating a SAFE client…
⠠ Connecting to The SAFE Network… The client still does not know enough network nodes.
:link: Connected to the Network Downloading file “safenetwork.mp4” with address c7a3c7602369aba9b9721ceb998a68f2c9d74d8eb1dd269b9e2833a3bba6ec98
Did not get file “safenetwork.mp4” from the network! Network Error Record was not found locally.
➜ ~ sudo safe --peer=“/ip4/143.110.169.218/tcp/38599/p2p/12D3KooWFUp78dnwMh6Hpne4ZPG5kH3Y11wDrcyAfkaXU8V4zdh5” files upload ~/safenetwork.mp4
Built with git version: 81a42f2 / main / 81a42f2
Instantiating a SAFE client…
⢀ Connecting to The SAFE Network… The client still does not know enough network nodes.
:link: Connected to the Network Preparing (chunking) files at ‘/Users/x/safenetwork.mp4’…
Storing file ‘safenetwork.mp4’ of 39372796 bytes (78 chunk/s)…
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Could not upload generated chunk of file ‘safenetwork.mp4’: Too many open files (os error 24)
Successfully stored ‘safenetwork.mp4’ to c7a3c7602369aba9b9721ceb998a68f2c9d74d8eb1dd269b9e2833a3bba6ec98
Writing 63 bytes to “/Users/x/Library/Application Support/safe/client/uploaded_files/file_names_2023-07-19_08-49-17”
Error: Too many open files (os error 24)
Location:
sn_cli/src/subcommands/files.rs:122:5
➜ ~ sudo safe --peer=“/ip4/143.110.169.218/tcp/38599/p2p/12D3KooWFUp78dnwMh6Hpne4ZPG5kH3Y11wDrcyAfkaXU8V4zdh5” files upload ~/safe.jpeg
Built with git version: 81a42f2 / main / 81a42f2
Instantiating a SAFE client…
⢀ Connecting to The SAFE Network… The client still does not know enough network nodes.
:link: Connected to the Network Preparing (chunking) files at ‘/Users/x/safe.jpeg’…
Storing file ‘safe.jpeg’ of 226719 bytes (4 chunk/s)…
Successfully stored ‘safe.jpeg’ to 89bf86508e7054781eb0db7cd32a11aa26cd9ea22d300d80a00965e252727ad7
Writing 57 bytes to “/Users/x/Library/Application Support/safe/client/uploaded_files/file_names_2023-07-19_10-50-28”
➜ ~ sudo safe --peer=“/ip4/143.110.169.218/tcp/38599/p2p/12D3KooWFUp78dnwMh6Hpne4ZPG5kH3Y11wDrcyAfkaXU8V4zdh5” files download safe2.jpeg 89bf86508e7054781eb0db7cd32a11aa26cd9ea22d300d80a00965e252727ad7
Built with git version: 81a42f2 / main / 81a42f2
Instantiating a SAFE client…
⢀ Connecting to The SAFE Network… The client still does not know enough network nodes.
:link: Connected to the Network Downloading file “safe2.jpeg” with address 89bf86508e7054781eb0db7cd32a11aa26cd9ea22d300d80a00965e252727ad7
Client download progress 2/3
Did not get file “safe2.jpeg” from the network! Chunks error Not all chunks were retrieved, expected 3, retrieved 2, missing [3f40e2(00111111)…]…

4 Likes

Successfully running 100 nodes on one machine.

TCP:   21117 (estab 20370, closed 403, orphaned 2, timewait 403)

%Cpu(s):  6.1 us,  1.3 sy,  0.0 ni, 91.7 id,  0.2 wa,  0.0 hi,  0.7 si,  0.0 st
MiB Mem :  11905.1 total,    231.3 free,   3183.0 used,   8490.9 buff/cache
MiB Swap:   4096.0 total,   4094.0 free,      2.0 used.   8574.9 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                        
 155566 petr      20   0   80056  45736   8168 S  23.8   0.4   0:35.16 safenode                                       
 155299 petr      20   0   61812  36484   8152 S   7.0   0.3   0:25.47 safenode                                       
 155325 petr      20   0   59932  34172   7860 S   3.6   0.3   0:15.84 safenode                                       
 155686 petr      20   0   60812  35664   8196 S   3.0   0.3   0:25.53 safenode                                       
 154528 petr      20   0   54716  29364   8164 S   1.7   0.2   0:16.08 safenode                                       
 154581 petr      20   0   50304  24980   8056 S   1.7   0.2   0:08.86 safenode                                       
 154967 petr      20   0   57492  31480   8196 S   1.3   0.3   0:22.81 safenode                                       
 154447 petr      20   0   57088  31904   8076 S   1.0   0.3   0:20.27 safenode                                       
 154648 petr      20   0   57808  32072   8148 S   1.0   0.3   0:17.56 safenode                                       
 154794 petr      20   0   58604  33228   8124 S   1.0   0.3   0:22.64 safenode                                       
 155619 petr      20   0   56112  30776   8016 S   1.0   0.3   0:08.51 safenode                                       
 154489 petr      20   0   54828  29616   8148 S   0.7   0.2   0:13.55 safenode                                       
 154542 petr      20   0   57128  31928   8152 S   0.7   0.3   0:20.27 safenode                                       
 154927 petr      20   0   52300  27120   8116 S   0.7   0.2   0:19.26 safenode                                       
 155512 petr      20   0   57780  32712   8168 S   0.7   0.3   0:14.11 safenode                                       
 154381 petr      20   0   57868  32592   8116 S   0.3   0.3   0:28.50 safenode                                       
 154394 petr      20   0   50992  25592   8116 S   0.3   0.2   0:19.55 safenode                                       
 154474 petr      20   0   56784  31104   8084 S   0.3   0.3   0:17.58 safenode                                       
 154595 petr      20   0   52996  27548   8196 S   0.3   0.2   0:14.29 safenode                                       
 154621 petr      20   0   57376  31860   8124 S   0.3   0.3   0:21.16 safenode                                       
 154662 petr      20   0   52196  26936   8132 S   0.3   0.2   0:08.25 safenode                                       
 154755 petr      20   0   57620  32408   8196 S   0.3   0.3   0:16.03 safenode                                       
 154808 petr      20   0   57448  32024   8196 S   0.3   0.3   0:16.35 safenode                                       
 154847 petr      20   0   57216  32020   8184 S   0.3   0.3   0:14.67 safenode                                       
 154874 petr      20   0   54868  29628   8164 S   0.3   0.2   0:11.21 safenode                                       
 154887 petr      20   0   54684  29436   8100 S   0.3   0.2   0:11.32 safenode                                       
 155006 petr      20   0   57320  31652   8224 S   0.3   0.3   0:11.30 safenode     
...

EDIT:
Added 10 nodes on OrangePi PC. Everything running fine so far.

TCP:   2014 (estab 1833, closed 125, orphaned 5, synrecv 0, timewait 125/0), ports 0

%Cpu(s):  6.2 us,  2.0 sy,  0.0 ni, 91.5 id,  0.1 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem :  1023036 total,   581160 free,   229644 used,   212232 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   743140 avail Mem 

13 Likes

if you run the command with SN_LOG=all safe --log-output-dest data-dir files upload <dir> it should write a log file to disk and tell you where that has been written in command output.

(or you can provide a real folder location you want the log files written to in place of data-dir)

8 Likes

New testnet YAY

The year of the testnets continues

10 Likes

Woohoo again, this is becoming a habit. One testnet ends and the next begins. :clap:

I’ve not had a chance to turn this into a script to run multiple nodes yet so just posting here for anyone who wants to use with vdash and likes short paths. You can of course use the defaults or specify something else.

To capture logs for use with vdash I suggest this to start your node:

SN_LOG=all safenode --root-dir=/tmp/safenode1 --log-output-dest /tmp/safenode1

Not trying to wind you up @chriso honest, just my preference. And is there a missing ‘export’ in front of the first SAFE_PEERS setting in the OP?

6 Likes

Am I missing something? I am trying to upload some files a I am getting this erorr:

safe files upload "/home/petr/Stažené/Download/Paul Kalkbrenner - 7 (2015) Album @ MP3/"
Removed old logs from directory: "/tmp/safe-client"
Logging to directory: "/tmp/safe-client"
Built with git version: f725637 / main / f725637
Instantiating a SAFE client...
🔗 Connected to the Network                                                     Storing file "02. Cylence 412.mp3" of 10720877 bytes..
Did not store file "02. Cylence 412.mp3" to all nodes in the close group! Network Error Outbound Error.
safe files upload /home/petr/Stažené/playing-ranked.webp
Removed old logs from directory: "/tmp/safe-client"
Logging to directory: "/tmp/safe-client"
Built with git version: f725637 / main / f725637
Instantiating a SAFE client...
🔗 Connected to the Network                                                                                                                  Storing file "playing-ranked.webp" of 44434 bytes..
Did not store file "playing-ranked.webp" to all nodes in the close group! Network Error Outbound Error.
Writing 8 bytes to "/home/petr/.safe/client/uploaded_files/file_names_2023-07-19_12-21-54"

client-log

2 Likes

I got that one on the previous testnet too. Didn’t get any solution though.

1 Like

I haven’t even got around to stopping my node from the last testnet yet. :exploding_head: :joy:

4 Likes

Seems like a bug in the client

Error broadcasting inactive client event: Events sender error channel closed.

That should be properly reported but isn’t. @aed900 can you take a look at this see if we can report better and find out where that channel is being dropped? :bowing_man:

4 Likes

If it helps I just tried same thing on Windows machine (same network behind same router) and it works fine there.

BTW it is not related, but this command listed in first post is not working (command not found).

2 Likes

I had that problem on Windows.

1 Like

Joined with 100 nodes on an AWS t4g.medium Instance with 4GB.

RAM usage for the safenode processes is increasing so I might get some nodes being killed soon.

Just had a thought: this could actually be a good thing at the moment for seeing what effects churn has on the network. As long as new nodes are being started by someone.

Further thought: If the nodes are going to be killed by ‘oom-kill’ anyway why don’t I just pre-emptively kill the one with the highest RAM usage, remove its log and records directories and start a replacement? And script that to run when ‘available’ as shown by ‘free’ dips below a threshold like 200MB on a check every minute? Project!

4 Likes

Testing the safe client, I was able to upload/download successfully. Just one small file in this test:

➜ export SAFE_PEERS="/ip4/143.110.169.218/tcp/38599/p2p/12D3KooWFUp78dnwMh6Hpne4ZPG5kH3Y11wDrcyAfkaXU8V4zdh5"

➜ safe files upload Documents/SafeNetworkTest 
Built with git version: 81a42f2 / main / 81a42f2
Instantiating a SAFE client...
⠈ Connecting to The SAFE Network...                                                                                                                                                                                                                                                                                                                    The client still does not know enough network nodes.
🔗 Connected to the Network                                                                                                                                                                                                                                                                                                                            Preparing (chunking) files at 'Documents/SafeNetworkTest'...
Storing file 'alpaca1.jpeg' of 7012 bytes (4 chunk/s)..
Successfully stored 'alpaca1.jpeg' to 706631b764a4535f880ba1c9c8ff3f81b33a327d879313062617efccfb9bc9a6
Writing 60 bytes to "/home/mark/.local/share/safe/client/uploaded_files/file_names_2023-07-19_12-35-43"

➜ safe files download
Built with git version: 81a42f2 / main / 81a42f2
Instantiating a SAFE client...
⠈ Connecting to The SAFE Network...                                                                                                                                                                                                                                                                                                                    The client still does not know enough network nodes.
🔗 Connected to the Network                                                                                                                                                                                                                                                                                                                            Trying to download files recorded in uploaded_files folder
Loading file names from index doc "file_names_2023-07-19_12-35-43"
Downloading file "alpaca1.jpeg" with address 706631b764a4535f880ba1c9c8ff3f81b33a327d879313062617efccfb9bc9a6
Client download progress 3/3
Successfully got file alpaca1.jpeg!
Writing 7012 bytes to "/home/mark/.local/share/safe/client/downloaded_files/alpaca1.jpeg"
13 Likes

Two nodes running, a few chunks in each. Small test file uploaded and downloaded fine.

Here’s a reliable :crossed_fingers: connection node if anyone needs one. My client wouldn’t connect to the default choice. I won’t overload this VM.

"/ip4/176.58.124.107/tcp/44439/p2p/12D3KooWEwMPj64C1MXEKaCYHGgtMApNzzvoynfprLJRLfgfGo7G"

8 Likes