CloserNet [13/12/23] Testnet [Offline]

ReduceConnectionsNet has been nice, but, it’s still not yet there. For the most part it’s not super noticable, but there is one thing underpinning some wild swings in price and probably related to some seemingly lost chunks: closeness.

There was a bug whereby our closest nodes were not actually ordered by closeness to a given thing (be it our node or some data).

So we have that licked. Alongside some other updates:

  • Even less connections!
  • Smoother uploads (chunk uploads roll along, no longer on waiting on a complete batch to finish before progressing)
  • We expose upload errors earlier. (But, you should now more reliably be able to redo the upload and try again; we’re working on making this nicer once more while still failing early for anything more critical).
  • More checks around partially written data (at times we were sending out half written data, for example).

And for some verification, you can grab an ubuntu.iso via

safe files download ubuntu.iso cd794073ac7f0e3c70688db4003e98ceed930ee5cbe38f18f3fabee2f5c09b15

Network Details

Node version: 0.100.3
Client version: 0.86.57
Faucet url: 178.128.33.82:8000

We have 101 droplets running a total of 2001 nodes. One droplet has 2vcpu and 4GB of memory.


If you are a regular user, see the ‘quickstart’ section for getting up and running.

If you are a first-time user, or would like more information, see the ‘further information’ section.


Quickstart

If you already have safeup, you can obtain the client and node binaries:

safeup client --version 0.86.57 
safeup node --version 0.100.3

Run a Node

Linux/macOS:

safenode

Windows:

safenode

Check local node’s reward balance

Your local node’s peer id will be printed to the terminal on startup with an example command). (You can also retrieve this from the node directory.)

safe wallet balance --peer-id="<local-node-peer-id>"

Connect to the Network as a Client

Linux/macOS:

safe wallet get-faucet 178.128.33.82:8000
safe files upload <directory-path>

Windows:

safe wallet get-faucet 178.128.33.82:8000
safe files upload <directory-path>

To do this with non-default batch-sizes (along with SAFE_PEERS set as above):

safe files upload --batch-size 40 <directory-path> 

40 being the integer value you want to set


Further Information

You can participate in the testnet either by connecting as a client or running your own node.

Connecting as a client requires the safe client binary; running a node requires the safenode binary.

Obtaining Binaries

We have a tool named safeup which is intended to make it easy to obtain the client, node, and other utility binaries.

Installing Safeup

On Linux/macOS, run the following command in your terminal:

curl -sSL https://raw.githubusercontent.com/maidsafe/safeup/main/install.sh | bash

On Windows, run the following command in a Powershell session (be careful to use Powershell, not cmd.exe):

iex (Invoke-RestMethod -Uri "https://raw.githubusercontent.com/maidsafe/safeup/main/install.ps1")

On either platform, you may need to restart your shell session for safeup to become available.

Installing Binaries

After obtaining safeup, you can install binaries like so:

safeup client # get the latest version of the client
safeup client --version 0.86.57  # get a specific version

safeup node # get the latest version of the node
safeup node --version 0.100.3 # get a specific version

safeup update # update all installed components to latest versions

When participating in our testnets, it is recommended to use a specific version. In our project we release a new version of the binaries every time we merge new code. This happens frequently, so over the lifetime of a testnet, many new releases will probably occur. So for participating in this particular testnet, you may not want the latest version.

The binaries are installed to ~/.local/bin on Linux and macOS, and on Windows they go to C:\Users\<username>\safe. Windows doesn’t really have a standard location for binaries that doesn’t require elevated privileges.

The safeup tool will modify the PATH variable on Linux/macOS, or the user Path variable on Windows. The effect of this is that the installed binaries will then become available in your shell without having to refer to them with their full paths. However, if you’re installing for the first time, you may need to start a new shell session for this change to be picked up.

Running a Node

You can participate in the testnet by running your own node. At the moment, you may not be successful if you’re running the node from your home machine. This is a situation we are working on. If you run from a cloud provider like Digital Ocean or AWS, you should be able to participate.

You can run the node process like so:

# Linux/macOS
safenode

# Windows
safenode

This will output all the logs to the filesystem, with the location of logs being platform specific:

# Linux
~/.local/share/safe/node/<peer id>/logs

# macOS
/Users/<username>/Library/Application Support/safe/node/<peer id>/logs

# Windows
C:\Users\<username>\AppData\Roaming\safe\node\<peer-id>\logs

If you wish, you can also provide your own path:

# Linux/macOS
SN_LOG=all safenode --log-output-dest <path>

# Windows
$env:SN_LOG = "all"; safenode --log-output-dest <path>

The advantage of using the predefined data-dir location is you can run multiple nodes on one machine without having to specify your own unique path for each node and manage that overhead yourself.

Connecting as a Client

You can use the safe client binary to connect as a client and upload or download files to/from the network.

Using the Client

You’ll first need to get some Safe Network Tokens:

safe wallet get-faucet 178.128.33.82:8000

You can now proceed to use the client, by, e.g., uploading files:

safe files upload <directory-path>

To download that same content:

safe files download

This will download the files to the default location, which is platform specific:

# Linux
~/.local/share/safe/client/downloaded_files

# macOS
/Users/<username>/Library/Application Support/safe/client/downloaded_files

# Windows
C:\Users\<username>\AppData\Roaming\safe\client\downloaded_files

To download to a particular file or directory:

safe file download [directory/filename] [NetworkAddress]

Troubleshooting

Cleanup

If you’ve used previous versions of the network before and you find problems when running commands, you may want to consider clearing out previous data (worthless DBCs from previous runs, old logs, old keys, etc.).

# Linux
rm -rf ~/.local/share/safe

# macOS
rm -rf ~/Library/Application\ Support/safe

# Windows
rmdir /s C:\Users\<username>\AppData\Roaming\safe

If you encounter a problem running any of our binaries on Windows, it’s possible you need the Visual C++ Redistributable installed.

35 Likes

Any closer and you’ll have to put a ring on it

Yay for the continuance of the year of the testnets

21 Likes

Fantastic stuff :star_struck:

13 Likes

Will start testing asap, thx 4 another 1 super ants :sweat_smile:

Happy testing folks :partying_face: :beers: :beer:

16 Likes

BegBlag test is a pass :slight_smile:

but had an error on 70mb AnarchyInTheSouthside.mp3

🔗 Connected to the Network                                                                                                                                                              Starting to chunk "AnarchyInTheSouthside.mp3" now.
Chunking 1 files...
Input was split into 133 chunks
Will now attempt to upload them...
Error: 
   0: Network Error Could not retrieve the record after storing it: ded00bb97626c62c04daf9f55291649b7a1ed8bacdff851909798cd485c5d4b8(1b48acd4d917f64fdd862ea9cb8f3712c72e24429cd7b7e5f44729c698fabcca).
   1: Could not retrieve the record after storing it: ded00bb97626c62c04daf9f55291649b7a1ed8bacdff851909798cd485c5d4b8(1b48acd4d917f64fdd862ea9cb8f3712c72e24429cd7b7e5f44729c698fabcca)

Location:
   sn_cli/src/subcommands/files/mod.rs:424

Log

"safe.log" 9731996c0d2c79908e705c97bd24500f7fd87529af92a66e27e1a6db68706da2

worked fine on second attempt :slight_smile:

"AnarchyInTheSouthside.mp3" c0893739a688b739aac5b75ac845b6c428c21b73dc782352bb1ad529495a87af
14 Likes

I think that’s to be expected on this testnet. There’s still some niggles around verification that we’re smoothing out (but the errors are no longer hidden and we’ve squashed a few bugs around that for this testnet).

So do not be shy about retrying to get the remaining chunks up. :+1:

14 Likes

As seen by netstat ?

3 Likes

just posting if its of use to to the team the Mp3 it went up second attempt and downloads fine.

just herding my nodes on line now then will get some more uploads on the go

10 Likes

Still rolling out like hot cakes? :heart_eyes:

12 Likes

Is it needed to upload logs of failures?

? [00:03:26] [####################################>---] 115/125
? [00:03:26] [####################################>---] 115/125
Error:
   0: ←[91mNetwork Error Could not retrieve the record after storing it: b98e64a
37c5c78d055de9db2f29528f7677c0211b0e1424fe8eeb34dc1c3684b(20ee8fb425675093e13671
fbf667526d4703fb3a3292bbaa3442191e70c9f420).←[0m
   1: ←[91mCould not retrieve the record after storing it: b98e64a37c5c78d055de9
db2f29528f7677c0211b0e1424fe8eeb34dc1c3684b(20ee8fb425675093e13671fbf667526d4703
fb3a3292bbaa3442191e70c9f420)←[0m

Location:
   ←[35msn_cli\src\subcommands\files\mod.rs←[0m:←[35m424←[0m
5 Likes

After twenty minutes here are my ten nodes.

First obvious thing to observer is more peers (>250 instead of <150) but presumably the reporting is more accurate?

[Also note vdash can now display records stored, but you’ll have to wait for a new safenode version before the data makes it into the logfiles]

10 Likes

I know its early doors but rewards are looking more evenly spread and ram usage is also looking pretty sweet :slight_smile:

http://132.145.53.195/NTracking.html

14 Likes

Kind of how it feels when you open your eyes at 5am and everyone is already bouncing around the playground. :man_cartwheeling:

:rofl: comiiiiiing…!!!

Edit: @neik you are right, this thing is svelte.

Edit2: I am up too https://javages.github.io/

10 Likes

On second upload attempt:


"500m" e2bf7cf5eec3c2e496b3a91c67fd32d53708ea20031954dd93306a88bde126ed
Uploaded 432 chunks (with 2 exist chunks) in 5 minutes 17 seconds

Should this be “2 missing chunks reuploaded”?

4 Likes

Yeah, I was working at office, but wrapping up now, to head home and start testing. :grin:

10 Likes

Both the the team and the testers :clap:t2::clap:t2::clap:t2:

11 Likes

Anyone getting problems paying for chunks - my laptop client isn’t finding any peers:

Starting to chunk "crdt" now.
Input was split into 422 chunks
Will now attempt to upload them...
Error: 
   0: Failed to pay for chunks: Transfer Error Failed to send tokens due to Could not get enough peers (5) to satisfy the request, found 0.

EDIT: I can reliably get tokens from the faucet though. :thinking: @joshuef, what difference is there that might help identify why paying for chunks fails but getting from the faucet succeeds?

Uploading from my cloud node seems fine.

From the laptop I also had the usual problem with safeup and wrote a script to retry until it worked. Maybe both are due to my connection, although web browsing, CLI interaction with github etc are all working fine. This is over a mobile hotspot on a phone with 4G mobile b/b.

3 Likes

Not sure vdash (latest version) is picking up all PUTs @happybeing. All my nodes have several dozen records but it’s only counting 1 or 2 in each case. Or is it now differentiating between replicas and paid chunks?

2 Likes

Working fine for me, home and cloud clients.

3 Likes