On the the total # of peer connections connected and closed in a fixed time frame with and without Request IDs or Chunks in flight to this node.
If 90% of all the nodes are backed by Maidsafe’s droplets (assumption), I am still kind of surprised to see this high level of peer connections closed, and then connected. I guess if its the norm, I will just have to get used to it, .
FWIW, The Connection closed to Peer is recording this internal error: Some(IO(Custom { kind: Other, error: Error(Closed) }))
Maybe the above message is a polite way of saying the connection closed gracefully? then again, it also says ‘Other’ for the kind of error. This all could be coming back from libp2p as is? .
This inner error message is at a rate off 2600 to 3100 per hour for past 12 hours.
FWIW, this was the # of reported ‘Connected Peers’ over past 12 hrs every 10 seconds:
Large files just create less chunks per spent traffic.
But if large file have the same amount of chunks as bunch of smaller files, it should spread in the same way I think.
Is this with --port X being provided as a argument to safenode and manual NAT port forwarding on the router?
OR
without the --port X provided as is, and they simply obtained a WAN IP etc?
using System;
using System.Diagnostics;
using System.IO;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
namespace ConsoleApplication116
{
class Program
{
static void Main(string[] args)
{
var loglock = new object();
var psi = new ProcessStartInfo("d:\\SN\\safe.exe");
psi.UseShellExecute = false;
psi.RedirectStandardOutput = true;
Parallel.For(0, 8000, new ParallelOptions { MaxDegreeOfParallelism = 32 }, i =>
{
string path = $"{i}.txt";
File.WriteAllText(path, i.ToString());
string peer = "/ip4/68.183.44.129/tcp/38921/p2p/12D3KooWN3P9exk23WW7naW7Kb1rdzZeLpZRHXtF5ntyQtJrsiF7";
psi.Arguments = $"--peer={peer} files upload -- {Path.GetFullPath(path)}";
psi.RedirectStandardError = true;
var process = Process.Start(psi);
var output = process.StandardOutput.ReadToEnd();
File.Delete(path);
var m = Regex.Match(output,
"Successfully stored file \"[0-9]+.txt\" to ([0-9a-f]+)");
var result = "Failure";
if (m.Success)
result = m.Groups[1].Value;
lock (loglock)
{
string logline = $"{path} - {result}\r\n";
File.AppendAllText("log.txt", logline);
Console.Write(logline);
}
});
}
}
}
Let’s see if it will help to spread chunks to nodes.
Right now only ~250 files uploaded, so it will take some time: log.zip (10.5 KB)
My node got one of my files: d9966ad7461f9bf5dc53c5fa2003f3b6f701bcc06cbaddc4c469c3fc60698a2b, so they are spreading. But how uniform - is another question.
So we’ve taken no real steps on the “lost node” issue we saw in the last testnet yet. We are getting more clarity on it in this one. Seems like a node can drop off the (routing) table of it’s neighbours, so to speak, and we don’t do enough to ensure it speaks up. So basically other nodes might not know they exist as yet.
I’ve also seen that those nodes do a looot of reconnecting/disconnecting, and there’s some errors there. That may account for what you’re seeing @Shu .
Ah, yeh,
This was the very error I’ve seen on those “lost” nodes.
Chunks are now stored by default at ~/.local/share/safe/node/record_store, but when running multiple nodes on one machine there don’t seem to be any sub-directories created. (eg ~/.local/share/safe/node/record_store/safenode1.. /safenode2 etc). Are all chunks dumped into the same pot?
Edit: Answered my own Q. Currently all chunks end up in the record_store folder. In future iterations it would be nice to have one subfolder per node created automatically.
Small files upload result: 4133 files uploaded (log4133.zip (167.3 KB)), 5 chunks appeared at my node.
If every file goes to 8 nodes at once, and right now approximately 2000 nodes are online, then every node should get approximately 16 of them.
5 is not much smaller than 16 (considering random nature of IDs), so results are relatively good.
Do anyone with large amount of nodes want to check uniformity of spreading?
I noticed strange thing:
I searched log files for GET requests and every request I found was followed by Record not found locally.
Is this a bug or normal behaviour?
Installing safe for x86_64-apple-darwin at /usr/local/bin…
[########################################] 4.75 MiB/4.75 MiB safe 0.77.43 is now available at /usr/local/bin/safe
Please run ‘safeup --help’ to see how to install network components.
I think it will be essential going forward to have each node’s records in it’s own folder so they can be cleared out if there is a problem with an individual node. I’m starting my nodes with this that is just a modification of what @southside was using for a previous testnet:-
cat start_safenodes
#!/bin/bash
for i in {1..100}
do
SN_LOG=all /home/ubuntu/.local/bin/safenode --log-dir=/tmp/safenode/$i --root-dir=/home/ubuntu/.local/share/safe/node/$i &
sleep 5
done
Just thinking out loud here - bear with me and please correct any misconceptions I may have…
Any and every file uploaded to the network >1kb is split into chunks, self-encrypted and the chunks stored at multiple (4,8, whatever) locations depending on the first few chars of the actual chunk content being closest to the node addresses.
It does not matter if we store a single 1Gb file or a few thousand files of several hundred kb each, we are storing approximately the same no of chunks in random locations, dependant solely on the first few bytes of the actual chunk content matching the relevant node addresses.
.
.
