NTracking


Node Tracking is a tool that I created to monitor my nodes across multiple machines.
The aim being that I want to easily spot anomalies across a large number of nodes, easily find high rewards balances or dead nodes etcetera at a glance with minimal searching.

It is very young if you try it, expect bugs!
If you feel that this should not be my day job… it’s not :slight_smile:
More functionality to come.
Works on :penguin: only.

Thanks to @Southside for the installation scripts.

If you clone the repo, it has some test log files and pregenerated graphs.
These are for testing purposes, the graphs will be overwritten but be sure to remove the log files before each run.

An online version can be seen here during test networks.

25 Likes

Hmm Ubuntu / Debian stuff - when I get back from Sydney I might try and convert the scripts into Fedora versions . .

3 Likes

how are you combing all the data from different systems into one graph on the public site?

iv got quite a few systems set up now and would like to recreate the same thing so i can keep an eye on them all without having to ssh in to them all the time.

There’s a Linux program to forward and receive live logfiles, rsyslog I think. I’m planning to use it with vdash but haven’t tried it yet.

1 Like

Going to have a look into that

1 Like

My setup is a little different to yours in that all my machines are on the same local network, not that it would make much of a difference if they were not.

Basically one of the machines has a script that pulls the logs from all other machines via scp, generates the combined graphs and pushes to github pages.

Pretty simple setup and really convenient to then be able to watch what is going on from my phone.

1 Like

Just thinking I could maybe wireguard all my systems together in there own network

Can you share the scripts for pulling all the logs and pushing to git hub ?

Guessing pulling the logs together would also be useful for vdash as well ?

Of course, I am away for a couple of days, more than likely will only be when I get back.

I am not 100% sure but I think that there is a rule that github pages is to host what you have in your repos whether or not that it is enforced idk, doubt it.

Not for live updates, which requires the local copies to be updated on the fly.

If you just want to run occasional analysis, then yes.

1 Like

Amazing I’ll look forward trying it out :slight_smile:

1 Like

Perhaps they could be mounted via sshfs? Again, typical Linux thingy…

1 Like

Wasn’t aware of that. :+1: I’m trying rsyslog.d atm but can’t get it working.

I once used it to send logs from a bunch of Besu nodes to Graylog monitoring server. Nodes have to be configured to log into syslog – it’s automatic when you start a node as a systemd service, and it writes to stdout instead of own log file, as safenode does (do we have a switch to log into stdout instead of log file? It could be useful here). Then it was as simple as adding a one-liner file /etc/rsyslog.d/90-graylog.conf like:

*.* @<remote-ip>:5140;RSYSLOG_SyslogProtocol23Format

The Graylog had a tcp endpoint for that on port 5140, and could read the data from nodes’ rsyslogs. Perhaps this is the main problem… to parse the rsyslog format? There seems to be a crate for this.

1 Like

I’ve tried rsyslog and sshfs but can’t get either working.

In both cases I suspect this is because my monitoring machine is connected over a mobile broadband hotspot, but can’t be sure.

I don’t really need this for myself but wanted to be able to suggest it if I could get it working. Maybe someone else will have a go.

1 Like

I gather someone is maintaining this package since it now uses metrics. I have a way to get accurate node balances now. The logs still at this time are reflecting the node attempting to double spend some earnings back to maidsafe and Jim confirmed its still happening.

Yes @aatonnomicc has taken this under his wing.

1 Like

@aatonnomicc can you tell me what metric value you are using to get earnings

1 Like

I’m greping the out put of the metrics port for the value labeled “sn_node_total_forwarded_rewards”
Seemed like the right choice :slight_smile:

What’s your method for getting accurate rewards?

1 Like

That metric has the bug still where the node is trying to forward back to Maidsafe double or more times. The double spends are rejected of course but the node doesn’t know,

I have been monitoring the wallet balance figure and it shows the correct earnings. Yes it is still kept even though the nanos are spent. It is the total earned by the node

I want NTracking to show the true earnings so there are less people worrying about rank being wrong

1 Like