DataPaymentNet [26/7/23 Testnet] [Offline]

Edit:

Current status:

08/08

  • Only one more node lost.
  • Many nodes full
  • Mem appears to climb slowly across nodes

03/08

A spike in network activity appears to have killed nodes for maidsafe and community users.

  • Maidsafe right now has 1675 nodes alive and kicking
  • some of those nodes have reached max capacity
  • Investigations underway

(31/07)

  • still 2001 maidsafe nodes
  • largest droplets are about ~60% full

(28/07)

  • All maidsafe nodes still live. 2001 going strong (biiig improvement over last testnets).
  • Droplets are about ~40% full on random sampling (~400-600 records)
  • Client validation of PUTs is problematic for some users. We’re investigating. (Large files will be harder to store)
  • There’s no mechanism for actually sending the DBCs as yet. You’ll need to inform the other party atm (I think… @roland is double checking there).

Orignal Post

We’ve seen some good improvements to node memory usage, and things have been looking healthy on the data side for a few testnets now.

As such, we’re going to try something new. Actually paying for data! And with this comes a faucet which you’ll need to ask for Safe Network Tokens before you’re allowed to upload! (if all goes well).

So with this testnet we want to check:

  • Faucet stability / ux
  • DBC transfers work as expected (are timely, no double spends etc).
  • Data payments work as expected

(It should be noted that the faucet is a temporary distribution measure until we have nodes earning money.)

Network Details

Node version: 0.87.1
Client version: 0.80.1
Faucet url: http://178.62.79.112:8000
SAFE_PEERS: /ip4/167.99.195.200/tcp/41805/p2p/12D3KooWP7SR4kWdN2SdeBdnpkxTbk1Buomb5PEjZVaZU3fGkkpK
Alternatives:

 "/ip4/167.99.197.30/tcp/33243/p2p/12D3KooWFBYaQcFqM8GMW91ifuxgTXHxmdjns14XPGTQVcopnkqn"
 "/ip4/167.99.199.20/tcp/45197/p2p/12D3KooWBfZcLR3kCNFnqzJsEdNKSL2vA5MHg24jAcg2HNJJHjnq"
 "/ip4/167.99.199.231/tcp/33541/p2p/12D3KooWDPnfjR7LTSjcJgQGjhdN2bK6j3Pw345g2d8TbXU5yUQy"
 "/ip4/178.62.79.112/tcp/46803/p2p/12D3KooWJnPhTVEnv7rx9ZCZZZRgTxrts33m33sKWfDdFH4jBcGo"
 "/ip4/167.99.195.200/tcp/44037/p2p/12D3KooWSc6JmQE6yQaXxaBbiUSjm3YUgZpu44oLeUVL8yAs6zAL"
 "/ip4/167.99.205.124/tcp/46301/p2p/12D3KooWMWkpHQkjJHDqwZRHd8RyzcykmFHm7mszMy7syqKySCvp"
...

We have 100 droplets running a total of 2000 nodes. One droplet has 2vcpu and 4GB of memory.


If you are a regular user, see the ‘quickstart’ section for getting up and running.

If you are a first-time user, or would like more information, see the ‘further information’ section.


Quickstart

If you already have safeup, you can obtain the client and node binaries:

safeup client --version 0.80.1
safeup node --version 0.87.1

Run a Node

Linux/macOS:

SN_LOG=all safenode

Windows:

$env:SN_LOG  =  "all"; safenode

Connect to the Network

Linux/macOS:

export SAFE_PEERS="/ip4/167.99.195.200/tcp/41805/p2p/12D3KooWP7SR4kWdN2SdeBdnpkxTbk1Buomb5PEjZVaZU3fGkkpK"
safe wallet get-faucet http://178.62.79.112:8000
safe files upload <directory-path>

Windows:

$env:SAFE_PEERS = "/ip4/167.99.195.200/tcp/41805/p2p/12D3KooWP7SR4kWdN2SdeBdnpkxTbk1Buomb5PEjZVaZU3fGkkpK"
safe wallet get-faucet http://178.62.79.112:8000
safe files upload <directory-path>

Further Information

You can participate in the testnet either by connecting as a client or running your own node.

Connecting as a client requires the safe client binary; running a node requires the safenode binary.

Cleanup

If you’ve used previous versions of the Network before, you will want to cleanup the previous files (worthless DBCs from previous runs, old logs, old keys).

# Linux
rm -rf ~/.local/share/safe

# macOS
rm -rf ~/Library/Application\ Support/safe

# Windows
rmdir /s C:\Users\<username>\AppData\Roaming\safe

Obtaining Binaries

We have a tool named safeup which is intended to make it easy to obtain the client, node, and other utility binaries.

Installing Safeup

On Linux/macOS, run the following command in your terminal:

curl -sSL https://raw.githubusercontent.com/maidsafe/safeup/main/install.sh | bash

On Windows, run the following command in a Powershell session (be careful to use Powershell, not cmd.exe):

iex (Invoke-RestMethod -Uri "https://raw.githubusercontent.com/maidsafe/safeup/main/install.ps1")

On either platform, you may need to restart your shell session for safeup to become available.

Installing Binaries

After obtaining safeup, you can install binaries like so:

safeup client # get the latest version of the client
safeup client --version 0.80.1 # get a specific version

safeup node # get the latest version of the node
safeup node --version 0.87.1 # get a specific version

safeup update # update all installed components to latest versions

When participating in our testnets, it is recommended to use a specific version. In our project we release a new version of the binaries every time we merge new code. This happens frequently, so over the lifetime of a testnet, many new releases will probably occur. So for participating in this particular testnet, you may not want the latest version.

The binaries are installed to ~/.local/bin on Linux and macOS, and on Windows they go to C:\Users\<username>\safe. Windows doesn’t really have a standard location for binaries that doesn’t require elevated privileges.

The safeup tool will modify the PATH variable on Linux/macOS, or the user Path variable on Windows. The effect of this is that the installed binaries will then become available in your shell without having to refer to them with their full paths. However, if you’re installing for the first time, you may need to start a new shell session for this change to be picked up.

Running a Node

You can participate in the testnet by running your own node. At the moment, you may not be successful if you’re running the node from your home machine. This is a situation we are working on. If you run from a cloud provider like Digital Ocean or AWS, you should be able to participate.

You can run the node process like so:

# Linux/macOS
SN_LOG=all safenode

# Windows
$env:SN_LOG = all; safenode

This will output all the logs in the terminal.

Sometimes it will be preferable to output the logs to file. You can do this by running the node like so:

# Linux/macOS
SN_LOG=all safenode --log-output-dest data-dir

# Windows
$env:SN_LOG = "all"; safenode --log-output-dest data-dir

The location of data-dir is platform specific:

# Linux
~/.local/share/safe/node/<peer id>/logs

# macOS
/Users/<username>/Library/Application Support/safe/node/<peer id>/logs

# Windows
C:\Users\<username>\AppData\Roaming\safe\node\<peer-id>\logs

If you wish, you can also provide your own path:

# Linux/macOS
SN_LOG=all safenode --log-output-dest <path>

# Windows
$env:SN_LOG  =  "all"; safenode --log-output-dest <path>

The advantage of using the predefined data-dir location is you can run multiple nodes on one machine without having to specify your own unique path for each node and manage that overhead yourself.

Connecting as a Client

You can use the safe client binary to connect as a client and upload or download files to/from the network.

To connect, you must provide another peer, in the form of a multi address. You can find one in the ‘Network Details’ section at the top.

It is recommended to set the peer using the environment variable SAFE_PEERS. You can set this variable once and it will apply for the duration of your shell session:

# Linux/macOS
export SAFE_PEERS="/ip4/167.99.195.200/tcp/41805/p2p/12D3KooWP7SR4kWdN2SdeBdnpkxTbk1Buomb5PEjZVaZU3fGkkpK"

# Windows
$env:SAFE_PEERS = "/ip4/167.99.195.200/tcp/41805/p2p/12D3KooWP7SR4kWdN2SdeBdnpkxTbk1Buomb5PEjZVaZU3fGkkpK"

NOTE: If you close and/or start a new shell session, you will be required to redefine this environment variable in the new session.

As an alternative to the environment variable, it’s also possible to use the --peer argument:

safe --peer="/ip4/167.99.195.200/tcp/41805/p2p/12D3KooWP7SR4kWdN2SdeBdnpkxTbk1Buomb5PEjZVaZU3fGkkpK" ...

However, this requires specifying the peer with each command.

Using the Client

You’ll first need to get some Safe Network Tokens:

safe wallet get-faucet http://178.62.79.112:8000

You can now proceed to use the client, by, e.g., uploading files:

safe files upload <directory-path>

To download that same content:

safe files download

This will download the files to the default location, which is platform specific:

# Linux
~/.local/share/safe/client/downloaded_files

# macOS
/Users/<username>/Library/Application Support/safe/client/downloaded_files

# Windows
C:\Users\<username>\AppData\Roaming\safe\client\downloaded_files

To download to a particular file or directory:

safe file download [directory/filename] [XORURL]

Troubleshooting

If you encounter a problem running any of our binaries on Windows, it’s possible you need the Visual C++ Redistributable installed.

47 Likes

Basic test done: get-faucet, upload and download worked fine.

21 Likes

I’m at 200 Safe Token !!! :sparkler: :tada: :confetti_ball: :tada:

uhm - yes - and upload + download worked fine xD

18 Likes

Faucet, upload, download, all seem to be working :slight_smile:

One observation

$ safe wallet get-faucet http://178.62.79.112:8000
Built with git version: 1262368 / main / 1262368
Instantiating a SAFE client...
⠂ 0/20 initial peers found.                                                                                                                 ⠐ 0/20 initial peers found.                                                                                                                 The client still does not know enough network nodes.
🔗 Connected to the Network  

# long wait

Requesting token for wallet address: a69d2c25e0f46de73b8049b2ff5f340527c539dd2fe6b3ec4be9bb3d565d886ffb36c8e8d9c4c8d4c2bacc5b497ccea3...
Successfully stored dbc to wallet dir. 
Old balance: 0.000000000
New balance: 10.000000000
Successfully got tokens from faucet.

For all commands the output was at the long wait for about one minute and seemed a bit like it might have hung. Might be good to have some more feedback about what is happening at this point.

19 Likes

Just running client - where are the client logs stored?

I’m getting some errors on upload “Network error could not retrieve the record after storing it.” … it seems to indicate it stored successfully though. Will download and check next.

5 Likes

Curious - it seems to be downloading all the chunks to verify upload … that being the case seems no point in checking the download myself. The whole process is quite slow for me as I don’t have a speedy connection. Only ~90MB of data took about ten minutes to complete.

3 Likes

@TylerAbeoJordan if you run thus:

log location should be printed for you.

I suspect a lot of folk will be seeing this. We’ve shifted some underlying APIs here to validate the PUT (and actually compare the write) as opposed to just wating for an ACK.

That validation can be made optional (atm, it’s hard set on). So any failure to get a chunk after PUT e g(or dbc) will be reported there.

Just looking at better flows for this the now.

Yup, this will definitely be slower. But will actually validate. We can probs speed things up here w/r/t how we do this check (just check for record addr as opposed to the whole record eg).

11 Likes

I didn’t install safenode so that doesn’t work.

3 Likes

wrong quote sorry, just replace safenode with safe i’ll edit my response there

6 Likes

Have a node running OK I think, although no chunks as yet (perhaps to be expected).

I saw this error in the logs at one stage, which is new to me.

ConnectionClosed error

root@localhost:~# cat .local/share/safe/node/12D3KooWCrhiaJyQA9d1kTbXLi9QvHx5PDCKTpeAJ9rfa2UBC2E1/logs/safenode.log | tail
[2023-07-26T08:05:09.947467Z DEBUG sn_networking::event] ConnectionClosed: outgoing (/ip4/161.35.166.184/tcp/40699/p2p/12D3KooWMS1CWvHGKvAC9wu4Xm6DKWZz1xkD4zTiSDGWj21NWiuc) peer_id=12D3KooWMS1CWvHGKvAC9wu4Xm6DKWZz1xkD4zTiSDGWj21NWiuc connection_id=ConnectionId(531) cause=Some(IO(Custom { kind: Other, error: Error(Closed) })) num_established=0
[2023-07-26T08:05:10.081261Z DEBUG sn_networking::event] ConnectionClosed: outgoing (/ip4/188.166.144.176/tcp/40945/p2p/12D3KooWGKq3nrqke1Ni798fkeBdrGoeezu64guxjCs3URY2SUJS) peer_id=12D3KooWGKq3nrqke1Ni798fkeBdrGoeezu64guxjCs3URY2SUJS connection_id=ConnectionId(533) cause=Some(IO(Custom { kind: Other, error: Error(Closed) })) num_established=0
[2023-07-26T08:05:10.084977Z DEBUG sn_networking::event] ConnectionClosed: outgoing (/ip4/161.35.171.107/tcp/43745/p2p/12D3KooWQhQEQLJGsqbqDoZA1m6yXXVCwvrBagypGd1N6WmWYSu3) peer_id=12D3KooWQhQEQLJGsqbqDoZA1m6yXXVCwvrBagypGd1N6WmWYSu3 connection_id=ConnectionId(532) cause=Some(IO(Custom { kind: Other, error: Error(Closed) })) num_established=0
[2023-07-26T08:05:10.085520Z DEBUG sn_networking::event] ConnectionClosed: outgoing (/ip4/167.99.197.30/tcp/45009/p2p/12D3KooWG5AYU1E9bQyufxjVegSxaQ4pVur7wendGzQqwcmQsyUz) peer_id=12D3KooWG5AYU1E9bQyufxjVegSxaQ4pVur7wendGzQqwcmQsyUz connection_id=ConnectionId(536) cause=Some(IO(Custom { kind: Other, error: Error(Closed) })) num_established=0
[2023-07-26T08:05:10.086733Z DEBUG sn_networking::event] ConnectionClosed: outgoing (/ip4/167.99.197.30/tcp/36967/p2p/12D3KooWRsq7T9BGJND64iaJZpt9DwW67eAjXuBdisRn42K8KXZn) peer_id=12D3KooWRsq7T9BGJND64iaJZpt9DwW67eAjXuBdisRn42K8KXZn connection_id=ConnectionId(534) cause=Some(IO(Custom { kind: Other, error: Error(Closed) })) num_established=0
[2023-07-26T08:05:10.086958Z DEBUG sn_networking::event] ConnectionClosed: outgoing (/ip4/142.93.34.193/tcp/43611/p2p/12D3KooWSep3QsjbEUQeuxKqf5SUFUERvTMs4oL6Phu5d84dDTYF) peer_id=12D3KooWSep3QsjbEUQeuxKqf5SUFUERvTMs4oL6Phu5d84dDTYF connection_id=ConnectionId(535) cause=Some(IO(Custom { kind: Other, error: Error(Closed) })) num_established=0
[2023-07-26T08:05:10.091469Z DEBUG sn_networking::event] ConnectionClosed: outgoing (/ip4/46.101.20.233/tcp/43921/p2p/12D3KooWLvRvokuG5vkgVhZnZTA2e78k4VJGARh5WNVX2XvJGtxD) peer_id=12D3KooWLvRvokuG5vkgVhZnZTA2e78k4VJGARh5WNVX2XvJGtxD connection_id=ConnectionId(537) cause=Some(IO(Custom { kind: Other, error: Error(Closed) })) num_established=0
[2023-07-26T08:05:10.093048Z DEBUG sn_networking::event] ConnectionClosed: outgoing (/ip4/142.93.44.166/tcp/40795/p2p/12D3KooWA3qBp6BCcMJi3x7iV63k3AJ6HQ2ggUmn56qcGQVhisfR) peer_id=12D3KooWA3qBp6BCcMJi3x7iV63k3AJ6HQ2ggUmn56qcGQVhisfR connection_id=ConnectionId(538) cause=Some(IO(Custom { kind: Other, error: Error(Closed) })) num_established=0
[2023-07-26T08:05:10.173355Z DEBUG sn_networking::event] ConnectionClosed: outgoing (/ip4/142.93.38.131/tcp/39759/p2p/12D3KooWRYw3Pgj76LMhKQP6ScXT9JXh6vReKdNVYqMkYNNtG2wo) peer_id=12D3KooWRYw3Pgj76LMhKQP6ScXT9JXh6vReKdNVYqMkYNNtG2wo connection_id=ConnectionId(539) cause=Some(IO(Custom { kind: Other, error: Error(Closed) })) num_established=0
[2023-07-26T08:05:10.185677Z DEBUG sn_networking::event] ConnectionClosed: outgoing (/ip4/134.209.16.110/tcp/43687/p2p/12D3KooWLJ3eQvqfvxBNdJi5njLQpgaw6oBmHCimsptFRByYPx77) peer_id=12D3KooWLJ3eQvqfvxBNdJi5njLQpgaw6oBmHCimsptFRByYPx77 connection_id=ConnectionId(540) cause=Some(IO(Custom { kind: Other, error: Error(Closed) })) num_established=0

4 Likes

“error: safe requires a subcommand but none was given.”

lol, sorry but it’s being a pain.

3 Likes

No sweat!

What command were you running? you can just add that on the end there and it should go

eg:

    SN_LOG=all safe --log-output-dest data-dir files upload <dir>
5 Likes

Odd that’s reporting as error, probably just surfacing now we’ve simplified some of the conn error management. It doesn’t look like anything to worry about, i think.

5 Likes

I was running the command you gave me … :confused:

3 Likes

Not working for me either

root@localhost:~# SN_LOG=all safe files upload hello.text -- --log-output-dest data-dir 
error: unexpected argument '--log-output-dest' found

root@localhost:~# SN_LOG=all safe files upload hello.txt --log-output-dest data-dir 
error: unexpected argument '--log-output-dest' found

EDIT

But this works

root@localhost:~# SN_LOG=all safe --log-output-dest data-dir files upload hello.txt

4 Likes

yeh log output has to be a first flag on safe, the rest of your command will follow

@TylerAbeoJordan does the command with subcommands work when adding in a dir of files in the <dir> portion here?

SN_LOG=all safe --log-output-dest data-dir files upload <dir>

3 Likes

Probably comparison of states for different nodes can help to understand if such behaviour is expected or not.
If every node in a batch have 0 chunks, then probably this is what expected. If some have alot, and some 0, then probably this is not expected.

4 Likes

Great stuff!

Q. If you re-upload a file do you pay twice?

ie. Neil uploads begblag and I do, or I simply run safe files upload my_cat.jpeg 5 times.

8 Likes

No, payment proofs are cached and checked for pre-upload.

@JPL no chunks is odd. Current running nodes have ~100-300 so you shooould be getting some. If you’ve still none, can you save those logs and maybe restart your node?

11 Likes