Update 25th July, 2024

Development Updates
1

4
4830×430 42.7 KB

News this week is we’re working on a new release, which as usual is a mixture of new features, fixes and a few experiments. Users should not need to do anything more than update their nodes through the CLI, or a reset via the Node Launchpad. The release is slated for Monday, so keep an eye out for the full details.

Among the new features are CLI flags to provide a password to encrypt a newly created wallet with, to encrypt an existing wallet, and to set a derivation passphrase used with the mnemonic to create a new private key.

Fixes include more fixes to prevent the problem with mixed networks, discussed last week, reductions in messaging, preventing spam attacks on spends, enabling nodes to allow more relayed connections through them to ease connectivity problems, and improvements to the auditor.

On the experimental side, we’ve removed DCUtR (Direct Connection Upgrade through Relay) from the coming release. DCUtR is a libp2p’s protocol for establishing direct connections between nodes behind routers via hole punching. DCUtR has been causing connection errors by overzealously dialling addresses of a peer we want to hole punch to. We have turned this off now to observe what effect that has, and will reintroduce it once we have a better flow to ignore the errors it produces. At the same time, we are working to improve the hole-punching process.

On the even more experimental side, not in this release, @joshuef and @dirvine are refining our Sybil defences. Should be something to show you on that soon.

General progress

@rusty.spork has been doing sterling work fielding user queries and suggestions on Discord, among them continued issues with users on home networks. We’ll see if our latest changes improve things there. There’s also been some feedback about the Discord UX, which we have made note of. Some good news is that there is a slow but steady increase in the number of users earning nanos each day. Users on 4G and 5G networks are also reporting connectivity problems. All this feedback has also helped @jimcollinson keep rolling updates to our docs site, including to a new troubleshooting section.

@joshuef continues to work with @dirvine on Sybil resistance, and removed some unused close group checks. He’s also helping coordinate issues round the DCUtR removal, which is being led by @bzee and @roland.

Meanwhile, @Anselme has finally completed his massive overhaul of the spend and transaction process, which has removed a lot of unnecessary code. He’s now gone for a bit of a lie down.

@mick.vandijke has been on UPnP this week. He and found a glitch in a library called igd-next which libp2p depends on and put in a PR for a fix. He also raised a PR to give more control to users by allowing them to specify the node install path.

@qi_ma has been looking at the issue of reducing replication messaging, and raised a PR to improve upload speed.

@roland created a PR to remove DCUtR and worked with @qi_ma to reduce replication traffic.

@shu continues with his observability work, and @chriso has been coordinating the infrastructure aspects of the next release.

57 Likes
3

At last first I think…

16 Likes
4

second - and now to read

14 Likes
5

Yaaaaiiii…Yesssss…now let’s do the reading…

12 Likes
6

first and third thats just sheer greed :laughing: :laughing:

well done to all the team amazing to see everything coming together :slight_smile:

looking forward to trying an upgrade on running nodes also this means beg blag will hopefull survive into the next version :slight_smile:

14 Likes
7

Where is my awe? You promised!

A short update on awe (a website publisher and browser demo) which I was hoping to have ready to roll this morning but network issues have held me back a bit. So my scripts for setting up sites and builds have improved, and I have a community website in the index thanks to @aatonnomicc but am having to set things up again due to wallet probs.

A handful of people have played with it and it is still available at https://downloads.happybeing.com Those builds work, but I’m planning a new build so I can update the awe-some-sites index.

Hopefully the reset next week will improve things, but I’m hoping to have a go tomorrow morning anyway, as the network seems to perform better then - until our US cousins wake up? :thinking:

wallet problems

For interest, trying to upload over mobile broadband causes things to fail often, and for the wallet to end up with unconfirmed spends. Once this happens the wallet becomes useless (won’t pay or transfer out), but if I create a clean wallet my private key has changed so I cannot update the registers I created with that wallet (which is necessary for updating versioned websites).

@joshuef is there any way to recover a wallet without changing the private key? Or will unconfirmed spends exist in the DAG cause problems? Food for thought.

Anyway, great work Autonomi team - we don’t see the half or tenth of it now I think - have a good weekend and good luck with the reset. :muscle:

27 Likes
8

@joshuef would it be possible to add a comment at the top of the open metrics file with the node version and the node Peer ID ?

the open metrics file is fantastic and that would just finish off being able to track upgrades across lots of systems and have the ability to track down were a node is across multiple systems.

8 Likes
9

About the nano distribution/earnings for some, or lack thereof .
Do we really expect all nodes to earn “more or less equally” when uploads are happening from only a limited segment of a huge network?

Does it not sort itself out when people are uploading from all corners?

8 Likes
10

Aren’t the chunks random, no matter who and where they get unloaded?

6 Likes
11

In theory they should be but something seems to be skewing the distribution. Some nodes earn for several chunks while others started at the same time earn none, when the odds should be that several should earn before a node earns for more than one chunk.

9 Likes
12

Yes but I cannot see how it would be dependent on the uploaders. I mean

Is the network really organized so that uploads happen from some part of it? If there’s a chunk, does it end up in different nodes if it is uploaded by me in Finland or you in UK?

5 Likes
13

Thx 4 the update Maidsafe devs

Sweeeeeet, can’t wait to try this out

I’ve been running a 100 nodes, most have Connected peers: 0
Still have to figure out what that is all about

Keep coding/hacking/testing super ants

2 Likes
14

What file are you referring to?

2 Likes
15

the one I get when I wget or curl the open metrics on the node.

ie for a node with open metrics on port 13001

ubuntu@s00:~$ curl 127.0.0.1:13001/metrics

# HELP sn_node_put_record_ok Number of successful record PUTs.
# TYPE sn_node_put_record_ok counter
sn_node_put_record_ok_total{record_type="Chunk"} 73
sn_node_put_record_ok_total{record_type="Spend"} 93
# HELP sn_node_put_record_err Number of errors during record PUTs.
# TYPE sn_node_put_record_err counter
sn_node_put_record_err_total 0
# HELP sn_node_replication_triggered Number of time that replication has been triggered.
# TYPE sn_node_replication_triggered counter
sn_node_replication_triggered_total 0
# HELP sn_node_replication_keys_to_fetch Number of replication keys to fetch from the network.
# TYPE sn_node_replication_keys_to_fetch histogram
sn_node_replication_keys_to_fetch_sum 253.0
sn_node_replication_keys_to_fetch_count 133
sn_node_replication_keys_to_fetch_bucket{le="1.0"} 112
sn_node_replication_keys_to_fetch_bucket{le="2.0"} 122
sn_node_replication_keys_to_fetch_bucket{le="4.0"} 124
sn_node_replication_keys_to_fetch_bucket{le="8.0"} 126
sn_node_replication_keys_to_fetch_bucket{le="+Inf"} 133
# HELP sn_node_peer_added_to_routing_table Number of peers that have been added to the Routing Table.
# TYPE sn_node_peer_added_to_routing_table counter
sn_node_peer_added_to_routing_table_total 10972
# HELP sn_node_peer_removed_from_routing_table Number of peers that have been removed from the Routing Table.
# TYPE sn_node_peer_removed_from_routing_table counter
sn_node_peer_removed_from_routing_table_total 10714
# HELP sn_node_current_reward_wallet_balance The number of Nanos in the node reward wallet.
# TYPE sn_node_current_reward_wallet_balance gauge
sn_node_current_reward_wallet_balance 10
# HELP sn_node_total_forwarded_rewards The cumulative number of Nanos forwarded by the node.
# TYPE sn_node_total_forwarded_rewards gauge
sn_node_total_forwarded_rewards 10
# HELP sn_node_uptime The uptime of the node in seconds.
# TYPE sn_node_uptime gauge
sn_node_uptime 1418830
# HELP libp2p_dcutr_events Events emitted by the relay NetworkBehaviour.
# TYPE libp2p_dcutr_events counter
libp2p_dcutr_events_total{event="DirectConnectionUpgradeSucceeded"} 47743
libp2p_dcutr_events_total{event="DirectConnectionUpgradeFailed"} 763
# HELP libp2p_identify_errors Number of errors while attempting to identify the remote.
# TYPE libp2p_identify_errors counter
libp2p_identify_errors_total 40055
# HELP libp2p_identify_pushed Number of times identification information of the local node has been actively pushed to a peer..
# TYPE libp2p_identify_pushed counter
libp2p_identify_pushed_total 0
# HELP libp2p_identify_received Number of times identification information has been received from a peer.
# TYPE libp2p_identify_received counter
libp2p_identify_received_total 9767122
# HELP libp2p_identify_sent Number of times identification information of the local node has been sent to a peer in response to an identification request.
# TYPE libp2p_identify_sent counter
libp2p_identify_sent_total 9782763
# HELP libp2p_identify_remote_protocols Number of connected nodes supporting a specific protocol, with "unrecognized" for each peer supporting one or more unrecognized protocols
# TYPE libp2p_identify_remote_protocols gauge
libp2p_identify_remote_protocols{protocol="unrecognized"} 403
libp2p_identify_remote_protocols{protocol="/ipfs/id/1.0.0"} 403
libp2p_identify_remote_protocols{protocol="/ipfs/id/push/1.0.0"} 403
libp2p_identify_remote_protocols{protocol="/libp2p/circuit/relay/0.2.0/hop"} 403
libp2p_identify_remote_protocols{protocol="/libp2p/circuit/relay/0.2.0/stop"} 403
libp2p_identify_remote_protocols{protocol="/ipfs/kad/1.0.0"} 403
# HELP libp2p_identify_remote_listen_addresses Number of connected nodes advertising a specific listen address
# TYPE libp2p_identify_remote_listen_addresses gauge
libp2p_identify_remote_listen_addresses{listen_address="/ip4/udp/quic-v1/p2p"} 354
libp2p_identify_remote_listen_addresses{listen_address="/ip4/udp/quic-v1"} 403
libp2p_identify_remote_listen_addresses{listen_address="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 49
libp2p_identify_remote_listen_addresses{listen_address="/ip4/udp/quic-v1/p2p/p2p-circuit"} 49
# HELP libp2p_identify_local_observed_addresses Number of connected nodes observing the local node at a specific address
# TYPE libp2p_identify_local_observed_addresses gauge
libp2p_identify_local_observed_addresses{observed_address="/ip4/udp/quic-v1/p2p"} 377
libp2p_identify_local_observed_addresses{observed_address="/ip4/udp/quic-v1"} 26
# HELP libp2p_kad_query_result_get_record_ok Number of records returned by a successful Kademlia get record query.
# TYPE libp2p_kad_query_result_get_record_ok counter
libp2p_kad_query_result_get_record_ok_total 2458
# HELP libp2p_kad_query_result_get_record_error Number of failed Kademlia get record queries.
# TYPE libp2p_kad_query_result_get_record_error counter
libp2p_kad_query_result_get_record_error_total{error="Timeout"} 28
libp2p_kad_query_result_get_record_error_total{error="NotFound"} 262
# HELP libp2p_kad_query_result_get_closest_peers_ok Number of closest peers returned by a successful Kademlia get closest peers query.
# TYPE libp2p_kad_query_result_get_closest_peers_ok histogram
libp2p_kad_query_result_get_closest_peers_ok_sum 784845.0
libp2p_kad_query_result_get_closest_peers_ok_count 144087
libp2p_kad_query_result_get_closest_peers_ok_bucket{le="1.0"} 0
libp2p_kad_query_result_get_closest_peers_ok_bucket{le="2.0"} 0
libp2p_kad_query_result_get_closest_peers_ok_bucket{le="4.0"} 0
libp2p_kad_query_result_get_closest_peers_ok_bucket{le="8.0"} 143742
libp2p_kad_query_result_get_closest_peers_ok_bucket{le="16.0"} 144087
libp2p_kad_query_result_get_closest_peers_ok_bucket{le="32.0"} 144087
libp2p_kad_query_result_get_closest_peers_ok_bucket{le="64.0"} 144087
libp2p_kad_query_result_get_closest_peers_ok_bucket{le="128.0"} 144087
libp2p_kad_query_result_get_closest_peers_ok_bucket{le="256.0"} 144087
libp2p_kad_query_result_get_closest_peers_ok_bucket{le="512.0"} 144087
libp2p_kad_query_result_get_closest_peers_ok_bucket{le="+Inf"} 144087
# HELP libp2p_kad_query_result_get_closest_peers_error Number of failed Kademlia get closest peers queries.
# TYPE libp2p_kad_query_result_get_closest_peers_error counter
libp2p_kad_query_result_get_closest_peers_error_total{error="Timeout"} 9173
# HELP libp2p_kad_query_result_get_providers_ok Number of providers returned by a successful Kademlia get providers query.
# TYPE libp2p_kad_query_result_get_providers_ok histogram
libp2p_kad_query_result_get_providers_ok_sum 0.0
libp2p_kad_query_result_get_providers_ok_count 0
libp2p_kad_query_result_get_providers_ok_bucket{le="1.0"} 0
libp2p_kad_query_result_get_providers_ok_bucket{le="2.0"} 0
libp2p_kad_query_result_get_providers_ok_bucket{le="4.0"} 0
libp2p_kad_query_result_get_providers_ok_bucket{le="8.0"} 0
libp2p_kad_query_result_get_providers_ok_bucket{le="16.0"} 0
libp2p_kad_query_result_get_providers_ok_bucket{le="32.0"} 0
libp2p_kad_query_result_get_providers_ok_bucket{le="64.0"} 0
libp2p_kad_query_result_get_providers_ok_bucket{le="128.0"} 0
libp2p_kad_query_result_get_providers_ok_bucket{le="256.0"} 0
libp2p_kad_query_result_get_providers_ok_bucket{le="512.0"} 0
libp2p_kad_query_result_get_providers_ok_bucket{le="+Inf"} 0
# HELP libp2p_kad_query_result_get_providers_error Number of failed Kademlia get providers queries.
# TYPE libp2p_kad_query_result_get_providers_error counter
# HELP libp2p_kad_query_result_num_requests Number of requests started for a Kademlia query.
# TYPE libp2p_kad_query_result_num_requests histogram
libp2p_kad_query_result_num_requests_sum{type="GetRecord"} 105503.0
libp2p_kad_query_result_num_requests_count{type="GetRecord"} 3658
libp2p_kad_query_result_num_requests_bucket{le="1.0",type="GetRecord"} 0
libp2p_kad_query_result_num_requests_bucket{le="2.0",type="GetRecord"} 0
libp2p_kad_query_result_num_requests_bucket{le="4.0",type="GetRecord"} 121
libp2p_kad_query_result_num_requests_bucket{le="8.0",type="GetRecord"} 349
libp2p_kad_query_result_num_requests_bucket{le="16.0",type="GetRecord"} 900
libp2p_kad_query_result_num_requests_bucket{le="32.0",type="GetRecord"} 2732
libp2p_kad_query_result_num_requests_bucket{le="64.0",type="GetRecord"} 3378
libp2p_kad_query_result_num_requests_bucket{le="128.0",type="GetRecord"} 3638
libp2p_kad_query_result_num_requests_bucket{le="256.0",type="GetRecord"} 3658
libp2p_kad_query_result_num_requests_bucket{le="512.0",type="GetRecord"} 3658
libp2p_kad_query_result_num_requests_bucket{le="+Inf",type="GetRecord"} 3658
libp2p_kad_query_result_num_requests_sum{type="PutRecord"} 60.0
libp2p_kad_query_result_num_requests_count{type="PutRecord"} 1
libp2p_kad_query_result_num_requests_bucket{le="1.0",type="PutRecord"} 0
libp2p_kad_query_result_num_requests_bucket{le="2.0",type="PutRecord"} 0
libp2p_kad_query_result_num_requests_bucket{le="4.0",type="PutRecord"} 0
libp2p_kad_query_result_num_requests_bucket{le="8.0",type="PutRecord"} 0
libp2p_kad_query_result_num_requests_bucket{le="16.0",type="PutRecord"} 0
libp2p_kad_query_result_num_requests_bucket{le="32.0",type="PutRecord"} 0
libp2p_kad_query_result_num_requests_bucket{le="64.0",type="PutRecord"} 1
libp2p_kad_query_result_num_requests_bucket{le="128.0",type="PutRecord"} 1
libp2p_kad_query_result_num_requests_bucket{le="256.0",type="PutRecord"} 1
libp2p_kad_query_result_num_requests_bucket{le="512.0",type="PutRecord"} 1
libp2p_kad_query_result_num_requests_bucket{le="+Inf",type="PutRecord"} 1
libp2p_kad_query_result_num_requests_sum{type="GetClosestPeers"} 4976481.0
libp2p_kad_query_result_num_requests_count{type="GetClosestPeers"} 153260
libp2p_kad_query_result_num_requests_bucket{le="1.0",type="GetClosestPeers"} 0
libp2p_kad_query_result_num_requests_bucket{le="2.0",type="GetClosestPeers"} 0
libp2p_kad_query_result_num_requests_bucket{le="4.0",type="GetClosestPeers"} 0
libp2p_kad_query_result_num_requests_bucket{le="8.0",type="GetClosestPeers"} 593
libp2p_kad_query_result_num_requests_bucket{le="16.0",type="GetClosestPeers"} 37353
libp2p_kad_query_result_num_requests_bucket{le="32.0",type="GetClosestPeers"} 96188
libp2p_kad_query_result_num_requests_bucket{le="64.0",type="GetClosestPeers"} 140048
libp2p_kad_query_result_num_requests_bucket{le="128.0",type="GetClosestPeers"} 152752
libp2p_kad_query_result_num_requests_bucket{le="256.0",type="GetClosestPeers"} 153260
libp2p_kad_query_result_num_requests_bucket{le="512.0",type="GetClosestPeers"} 153260
libp2p_kad_query_result_num_requests_bucket{le="+Inf",type="GetClosestPeers"} 153260
# HELP libp2p_kad_query_result_num_success Number of successful requests of a Kademlia query.
# TYPE libp2p_kad_query_result_num_success histogram
libp2p_kad_query_result_num_success_sum{type="GetRecord"} 80632.0
libp2p_kad_query_result_num_success_count{type="GetRecord"} 3658
libp2p_kad_query_result_num_success_bucket{le="1.0",type="GetRecord"} 168
libp2p_kad_query_result_num_success_bucket{le="2.0",type="GetRecord"} 250
libp2p_kad_query_result_num_success_bucket{le="4.0",type="GetRecord"} 385
libp2p_kad_query_result_num_success_bucket{le="8.0",type="GetRecord"} 811
libp2p_kad_query_result_num_success_bucket{le="16.0",type="GetRecord"} 1843
libp2p_kad_query_result_num_success_bucket{le="32.0",type="GetRecord"} 3057
libp2p_kad_query_result_num_success_bucket{le="64.0",type="GetRecord"} 3461
libp2p_kad_query_result_num_success_bucket{le="128.0",type="GetRecord"} 3647
libp2p_kad_query_result_num_success_bucket{le="256.0",type="GetRecord"} 3658
libp2p_kad_query_result_num_success_bucket{le="512.0",type="GetRecord"} 3658
libp2p_kad_query_result_num_success_bucket{le="+Inf",type="GetRecord"} 3658
libp2p_kad_query_result_num_success_sum{type="PutRecord"} 48.0
libp2p_kad_query_result_num_success_count{type="PutRecord"} 1
libp2p_kad_query_result_num_success_bucket{le="1.0",type="PutRecord"} 0
libp2p_kad_query_result_num_success_bucket{le="2.0",type="PutRecord"} 0
libp2p_kad_query_result_num_success_bucket{le="4.0",type="PutRecord"} 0
libp2p_kad_query_result_num_success_bucket{le="8.0",type="PutRecord"} 0
libp2p_kad_query_result_num_success_bucket{le="16.0",type="PutRecord"} 0
libp2p_kad_query_result_num_success_bucket{le="32.0",type="PutRecord"} 0
libp2p_kad_query_result_num_success_bucket{le="64.0",type="PutRecord"} 1
libp2p_kad_query_result_num_success_bucket{le="128.0",type="PutRecord"} 1
libp2p_kad_query_result_num_success_bucket{le="256.0",type="PutRecord"} 1
libp2p_kad_query_result_num_success_bucket{le="512.0",type="PutRecord"} 1
libp2p_kad_query_result_num_success_bucket{le="+Inf",type="PutRecord"} 1
libp2p_kad_query_result_num_success_sum{type="GetClosestPeers"} 4135748.0
libp2p_kad_query_result_num_success_count{type="GetClosestPeers"} 153260
libp2p_kad_query_result_num_success_bucket{le="1.0",type="GetClosestPeers"} 0
libp2p_kad_query_result_num_success_bucket{le="2.0",type="GetClosestPeers"} 0
libp2p_kad_query_result_num_success_bucket{le="4.0",type="GetClosestPeers"} 6
libp2p_kad_query_result_num_success_bucket{le="8.0",type="GetClosestPeers"} 19565
libp2p_kad_query_result_num_success_bucket{le="16.0",type="GetClosestPeers"} 54843
libp2p_kad_query_result_num_success_bucket{le="32.0",type="GetClosestPeers"} 110591
libp2p_kad_query_result_num_success_bucket{le="64.0",type="GetClosestPeers"} 144257
libp2p_kad_query_result_num_success_bucket{le="128.0",type="GetClosestPeers"} 152949
libp2p_kad_query_result_num_success_bucket{le="256.0",type="GetClosestPeers"} 153260
libp2p_kad_query_result_num_success_bucket{le="512.0",type="GetClosestPeers"} 153260
libp2p_kad_query_result_num_success_bucket{le="+Inf",type="GetClosestPeers"} 153260
# HELP libp2p_kad_query_result_num_failure Number of failed requests of a Kademlia query.
# TYPE libp2p_kad_query_result_num_failure histogram
libp2p_kad_query_result_num_failure_sum{type="GetClosestPeers"} 109767.0
libp2p_kad_query_result_num_failure_count{type="GetClosestPeers"} 153260
libp2p_kad_query_result_num_failure_bucket{le="1.0",type="GetClosestPeers"} 123441
libp2p_kad_query_result_num_failure_bucket{le="2.0",type="GetClosestPeers"} 141125
libp2p_kad_query_result_num_failure_bucket{le="4.0",type="GetClosestPeers"} 152069
libp2p_kad_query_result_num_failure_bucket{le="8.0",type="GetClosestPeers"} 153259
libp2p_kad_query_result_num_failure_bucket{le="16.0",type="GetClosestPeers"} 153260
libp2p_kad_query_result_num_failure_bucket{le="32.0",type="GetClosestPeers"} 153260
libp2p_kad_query_result_num_failure_bucket{le="64.0",type="GetClosestPeers"} 153260
libp2p_kad_query_result_num_failure_bucket{le="128.0",type="GetClosestPeers"} 153260
libp2p_kad_query_result_num_failure_bucket{le="256.0",type="GetClosestPeers"} 153260
libp2p_kad_query_result_num_failure_bucket{le="512.0",type="GetClosestPeers"} 153260
libp2p_kad_query_result_num_failure_bucket{le="+Inf",type="GetClosestPeers"} 153260
libp2p_kad_query_result_num_failure_sum{type="GetRecord"} 1398.0
libp2p_kad_query_result_num_failure_count{type="GetRecord"} 3658
libp2p_kad_query_result_num_failure_bucket{le="1.0",type="GetRecord"} 3331
libp2p_kad_query_result_num_failure_bucket{le="2.0",type="GetRecord"} 3511
libp2p_kad_query_result_num_failure_bucket{le="4.0",type="GetRecord"} 3621
libp2p_kad_query_result_num_failure_bucket{le="8.0",type="GetRecord"} 3656
libp2p_kad_query_result_num_failure_bucket{le="16.0",type="GetRecord"} 3658
libp2p_kad_query_result_num_failure_bucket{le="32.0",type="GetRecord"} 3658
libp2p_kad_query_result_num_failure_bucket{le="64.0",type="GetRecord"} 3658
libp2p_kad_query_result_num_failure_bucket{le="128.0",type="GetRecord"} 3658
libp2p_kad_query_result_num_failure_bucket{le="256.0",type="GetRecord"} 3658
libp2p_kad_query_result_num_failure_bucket{le="512.0",type="GetRecord"} 3658
libp2p_kad_query_result_num_failure_bucket{le="+Inf",type="GetRecord"} 3658
libp2p_kad_query_result_num_failure_sum{type="PutRecord"} 2.0
libp2p_kad_query_result_num_failure_count{type="PutRecord"} 1
libp2p_kad_query_result_num_failure_bucket{le="1.0",type="PutRecord"} 0
libp2p_kad_query_result_num_failure_bucket{le="2.0",type="PutRecord"} 1
libp2p_kad_query_result_num_failure_bucket{le="4.0",type="PutRecord"} 1
libp2p_kad_query_result_num_failure_bucket{le="8.0",type="PutRecord"} 1
libp2p_kad_query_result_num_failure_bucket{le="16.0",type="PutRecord"} 1
libp2p_kad_query_result_num_failure_bucket{le="32.0",type="PutRecord"} 1
libp2p_kad_query_result_num_failure_bucket{le="64.0",type="PutRecord"} 1
libp2p_kad_query_result_num_failure_bucket{le="128.0",type="PutRecord"} 1
libp2p_kad_query_result_num_failure_bucket{le="256.0",type="PutRecord"} 1
libp2p_kad_query_result_num_failure_bucket{le="512.0",type="PutRecord"} 1
libp2p_kad_query_result_num_failure_bucket{le="+Inf",type="PutRecord"} 1
# HELP libp2p_kad_query_result_duration_seconds Duration of a Kademlia query.
# TYPE libp2p_kad_query_result_duration_seconds histogram
# UNIT libp2p_kad_query_result_duration_seconds seconds
libp2p_kad_query_result_duration_seconds_sum{type="PutRecord"} 6.201925756
libp2p_kad_query_result_duration_seconds_count{type="PutRecord"} 1
libp2p_kad_query_result_duration_seconds_bucket{le="0.1",type="PutRecord"} 0
libp2p_kad_query_result_duration_seconds_bucket{le="0.2",type="PutRecord"} 0
libp2p_kad_query_result_duration_seconds_bucket{le="0.4",type="PutRecord"} 0
libp2p_kad_query_result_duration_seconds_bucket{le="0.8",type="PutRecord"} 0
libp2p_kad_query_result_duration_seconds_bucket{le="1.6",type="PutRecord"} 0
libp2p_kad_query_result_duration_seconds_bucket{le="3.2",type="PutRecord"} 0
libp2p_kad_query_result_duration_seconds_bucket{le="6.4",type="PutRecord"} 1
libp2p_kad_query_result_duration_seconds_bucket{le="12.8",type="PutRecord"} 1
libp2p_kad_query_result_duration_seconds_bucket{le="25.6",type="PutRecord"} 1
libp2p_kad_query_result_duration_seconds_bucket{le="51.2",type="PutRecord"} 1
libp2p_kad_query_result_duration_seconds_bucket{le="+Inf",type="PutRecord"} 1
libp2p_kad_query_result_duration_seconds_sum{type="GetClosestPeers"} 595097.3499527855
libp2p_kad_query_result_duration_seconds_count{type="GetClosestPeers"} 153260
libp2p_kad_query_result_duration_seconds_bucket{le="0.1",type="GetClosestPeers"} 0
libp2p_kad_query_result_duration_seconds_bucket{le="0.2",type="GetClosestPeers"} 0
libp2p_kad_query_result_duration_seconds_bucket{le="0.4",type="GetClosestPeers"} 343
libp2p_kad_query_result_duration_seconds_bucket{le="0.8",type="GetClosestPeers"} 18200
libp2p_kad_query_result_duration_seconds_bucket{le="1.6",type="GetClosestPeers"} 48796
libp2p_kad_query_result_duration_seconds_bucket{le="3.2",type="GetClosestPeers"} 78788
libp2p_kad_query_result_duration_seconds_bucket{le="6.4",type="GetClosestPeers"} 122705
libp2p_kad_query_result_duration_seconds_bucket{le="12.8",type="GetClosestPeers"} 153260
libp2p_kad_query_result_duration_seconds_bucket{le="25.6",type="GetClosestPeers"} 153260
libp2p_kad_query_result_duration_seconds_bucket{le="51.2",type="GetClosestPeers"} 153260
libp2p_kad_query_result_duration_seconds_bucket{le="+Inf",type="GetClosestPeers"} 153260
libp2p_kad_query_result_duration_seconds_sum{type="GetRecord"} 6561.315472847991
libp2p_kad_query_result_duration_seconds_count{type="GetRecord"} 3658
libp2p_kad_query_result_duration_seconds_bucket{le="0.1",type="GetRecord"} 15
libp2p_kad_query_result_duration_seconds_bucket{le="0.2",type="GetRecord"} 56
libp2p_kad_query_result_duration_seconds_bucket{le="0.4",type="GetRecord"} 321
libp2p_kad_query_result_duration_seconds_bucket{le="0.8",type="GetRecord"} 1215
libp2p_kad_query_result_duration_seconds_bucket{le="1.6",type="GetRecord"} 2554
libp2p_kad_query_result_duration_seconds_bucket{le="3.2",type="GetRecord"} 3055
libp2p_kad_query_result_duration_seconds_bucket{le="6.4",type="GetRecord"} 3564
libp2p_kad_query_result_duration_seconds_bucket{le="12.8",type="GetRecord"} 3658
libp2p_kad_query_result_duration_seconds_bucket{le="25.6",type="GetRecord"} 3658
libp2p_kad_query_result_duration_seconds_bucket{le="51.2",type="GetRecord"} 3658
libp2p_kad_query_result_duration_seconds_bucket{le="+Inf",type="GetRecord"} 3658
# HELP libp2p_kad_routing_updated Number of peers added, updated or evicted to, in or from a specific kbucket in the routing table.
# TYPE libp2p_kad_routing_updated counter
libp2p_kad_routing_updated_total{action="Updated",bucket="243"} 3
libp2p_kad_routing_updated_total{action="Added",bucket="248"} 533
libp2p_kad_routing_updated_total{action="Evicted",bucket="246"} 233
libp2p_kad_routing_updated_total{action="Added",bucket="253"} 443
libp2p_kad_routing_updated_total{action="Updated",bucket="255"} 138
libp2p_kad_routing_updated_total{action="Added",bucket="242"} 471
libp2p_kad_routing_updated_total{action="Updated",bucket="245"} 1591
libp2p_kad_routing_updated_total{action="Evicted",bucket="254"} 221
libp2p_kad_routing_updated_total{action="Evicted",bucket="248"} 207
libp2p_kad_routing_updated_total{action="Evicted",bucket="244"} 4
libp2p_kad_routing_updated_total{action="Evicted",bucket="249"} 169
libp2p_kad_routing_updated_total{action="Added",bucket="252"} 443
libp2p_kad_routing_updated_total{action="Added",bucket="246"} 722
libp2p_kad_routing_updated_total{action="Evicted",bucket="251"} 163
libp2p_kad_routing_updated_total{action="Added",bucket="245"} 1553
libp2p_kad_routing_updated_total{action="Added",bucket="238"} 176
libp2p_kad_routing_updated_total{action="Evicted",bucket="247"} 224
libp2p_kad_routing_updated_total{action="Updated",bucket="250"} 56
libp2p_kad_routing_updated_total{action="Updated",bucket="251"} 74
libp2p_kad_routing_updated_total{action="Added",bucket="244"} 2263
libp2p_kad_routing_updated_total{action="Added",bucket="243"} 1172
libp2p_kad_routing_updated_total{action="Evicted",bucket="252"} 184
libp2p_kad_routing_updated_total{action="Added",bucket="251"} 421
libp2p_kad_routing_updated_total{action="Added",bucket="247"} 623
libp2p_kad_routing_updated_total{action="Updated",bucket="253"} 73
libp2p_kad_routing_updated_total{action="Updated",bucket="242"} 79
libp2p_kad_routing_updated_total{action="Updated",bucket="249"} 391
libp2p_kad_routing_updated_total{action="Added",bucket="239"} 1
libp2p_kad_routing_updated_total{action="Added",bucket="241"} 17
libp2p_kad_routing_updated_total{action="Updated",bucket="246"} 250
libp2p_kad_routing_updated_total{action="Updated",bucket="247"} 133
libp2p_kad_routing_updated_total{action="Added",bucket="250"} 442
libp2p_kad_routing_updated_total{action="Added",bucket="255"} 717
libp2p_kad_routing_updated_total{action="Evicted",bucket="255"} 275
libp2p_kad_routing_updated_total{action="Updated",bucket="248"} 156
libp2p_kad_routing_updated_total{action="Added",bucket="249"} 442
libp2p_kad_routing_updated_total{action="Evicted",bucket="245"} 426
libp2p_kad_routing_updated_total{action="Updated",bucket="244"} 3466
libp2p_kad_routing_updated_total{action="Updated",bucket="254"} 55
libp2p_kad_routing_updated_total{action="Updated",bucket="252"} 344
libp2p_kad_routing_updated_total{action="Evicted",bucket="250"} 172
libp2p_kad_routing_updated_total{action="Added",bucket="237"} 1
libp2p_kad_routing_updated_total{action="Added",bucket="254"} 532
libp2p_kad_routing_updated_total{action="Evicted",bucket="253"} 186
# HELP libp2p_kad_inbound_requests Number of inbound requests.
# TYPE libp2p_kad_inbound_requests counter
libp2p_kad_inbound_requests_total{request="FindNode"} 7398485
libp2p_kad_inbound_requests_total{request="GetRecord"} 900091
libp2p_kad_inbound_requests_total{request="PutRecord"} 16
# HELP libp2p_relay_events Events emitted by the relay NetworkBehaviour.
# TYPE libp2p_relay_events counter
libp2p_relay_events_total{event="CircuitReqAcceptFailed"} 1
libp2p_relay_events_total{event="CircuitReqAccepted"} 143543
libp2p_relay_events_total{event="CircuitReqDenyFailed"} 13
libp2p_relay_events_total{event="ReservationReqAccepted"} 350
libp2p_relay_events_total{event="CircuitClosed"} 143611
libp2p_relay_events_total{event="CircuitReqOutboundConnectFailed"} 455
libp2p_relay_events_total{event="CircuitReqDenied"} 307712
# HELP libp2p_swarm_connections_incoming Number of incoming connections per address stack.
# TYPE libp2p_swarm_connections_incoming counter
libp2p_swarm_connections_incoming_total{protocols="/ip4/udp/quic-v1"} 6323469
# HELP libp2p_swarm_connections_incoming_error Number of incoming connection errors.
# TYPE libp2p_swarm_connections_incoming_error counter
libp2p_swarm_connections_incoming_error_total{error="TransportErrorOther",protocols="/ip4/udp/quic-v1"} 193310
# HELP libp2p_swarm_new_listen_addr Number of new listen addresses.
# TYPE libp2p_swarm_new_listen_addr counter
libp2p_swarm_new_listen_addr_total{protocols="/ip4/udp/quic-v1"} 4
# HELP libp2p_swarm_expired_listen_addr Number of expired listen addresses.
# TYPE libp2p_swarm_expired_listen_addr counter
# HELP libp2p_swarm_external_addr_candidates Number of new external address candidates.
# TYPE libp2p_swarm_external_addr_candidates counter
libp2p_swarm_external_addr_candidates_total{protocols="/ip4/udp/quic-v1/p2p"} 6136115
libp2p_swarm_external_addr_candidates_total{protocols="/p2p"} 48714
libp2p_swarm_external_addr_candidates_total{protocols="/ip4/udp/quic-v1"} 3581446
# HELP libp2p_swarm_external_addr_confirmed Number of confirmed external addresses.
# TYPE libp2p_swarm_external_addr_confirmed counter
# HELP libp2p_swarm_external_addr_expired Number of expired external addresses.
# TYPE libp2p_swarm_external_addr_expired counter
# HELP libp2p_swarm_listener_closed Number of listeners closed.
# TYPE libp2p_swarm_listener_closed counter
# HELP libp2p_swarm_listener_error Number of listener errors.
# TYPE libp2p_swarm_listener_error counter
libp2p_swarm_listener_error_total 1161
# HELP libp2p_swarm_dial_attempt Number of dial attempts.
# TYPE libp2p_swarm_dial_attempt counter
libp2p_swarm_dial_attempt_total 3842910
# HELP libp2p_swarm_outgoing_connection_error Number outgoing connection errors.
# TYPE libp2p_swarm_outgoing_connection_error counter
libp2p_swarm_outgoing_connection_error_total{peer="Known",error="TransportOther"} 337353
libp2p_swarm_outgoing_connection_error_total{peer="Known",error="TransportMultiaddrNotSupported"} 10248
libp2p_swarm_outgoing_connection_error_total{peer="Known",error="WrongPeerId"} 1780
# HELP libp2p_swarm_connections_established Number of connections established.
# TYPE libp2p_swarm_connections_established counter
libp2p_swarm_connections_established_total{role="Listener",protocols="/ip4/udp/quic-v1"} 6130158
libp2p_swarm_connections_established_total{role="Dialer",protocols="/ip4/udp/quic-v1/p2p"} 3646639
libp2p_swarm_connections_established_total{role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 48809
# HELP libp2p_swarm_connections_establishment_duration Time it took (locally) to establish connections.
# TYPE libp2p_swarm_connections_establishment_duration histogram
libp2p_swarm_connections_establishment_duration_sum{role="Listener",protocols="/ip4/udp/quic-v1"} 1353841.5313917506
libp2p_swarm_connections_establishment_duration_count{role="Listener",protocols="/ip4/udp/quic-v1"} 6130158
libp2p_swarm_connections_establishment_duration_bucket{le="0.01",role="Listener",protocols="/ip4/udp/quic-v1"} 143781
libp2p_swarm_connections_establishment_duration_bucket{le="0.015",role="Listener",protocols="/ip4/udp/quic-v1"} 172793
libp2p_swarm_connections_establishment_duration_bucket{le="0.0225",role="Listener",protocols="/ip4/udp/quic-v1"} 219807
libp2p_swarm_connections_establishment_duration_bucket{le="0.03375",role="Listener",protocols="/ip4/udp/quic-v1"} 274004
libp2p_swarm_connections_establishment_duration_bucket{le="0.050625",role="Listener",protocols="/ip4/udp/quic-v1"} 338268
libp2p_swarm_connections_establishment_duration_bucket{le="0.0759375",role="Listener",protocols="/ip4/udp/quic-v1"} 531195
libp2p_swarm_connections_establishment_duration_bucket{le="0.11390625",role="Listener",protocols="/ip4/udp/quic-v1"} 2517964
libp2p_swarm_connections_establishment_duration_bucket{le="0.170859375",role="Listener",protocols="/ip4/udp/quic-v1"} 4615670
libp2p_swarm_connections_establishment_duration_bucket{le="0.2562890625",role="Listener",protocols="/ip4/udp/quic-v1"} 5493440
libp2p_swarm_connections_establishment_duration_bucket{le="0.38443359375",role="Listener",protocols="/ip4/udp/quic-v1"} 5655266
libp2p_swarm_connections_establishment_duration_bucket{le="0.576650390625",role="Listener",protocols="/ip4/udp/quic-v1"} 5750883
libp2p_swarm_connections_establishment_duration_bucket{le="0.8649755859375",role="Listener",protocols="/ip4/udp/quic-v1"} 5813439
libp2p_swarm_connections_establishment_duration_bucket{le="1.29746337890625",role="Listener",protocols="/ip4/udp/quic-v1"} 5947619
libp2p_swarm_connections_establishment_duration_bucket{le="1.946195068359375",role="Listener",protocols="/ip4/udp/quic-v1"} 6050508
libp2p_swarm_connections_establishment_duration_bucket{le="2.9192926025390628",role="Listener",protocols="/ip4/udp/quic-v1"} 6097763
libp2p_swarm_connections_establishment_duration_bucket{le="4.378938903808594",role="Listener",protocols="/ip4/udp/quic-v1"} 6124898
libp2p_swarm_connections_establishment_duration_bucket{le="6.568408355712891",role="Listener",protocols="/ip4/udp/quic-v1"} 6130158
libp2p_swarm_connections_establishment_duration_bucket{le="9.852612533569337",role="Listener",protocols="/ip4/udp/quic-v1"} 6130158
libp2p_swarm_connections_establishment_duration_bucket{le="14.778918800354005",role="Listener",protocols="/ip4/udp/quic-v1"} 6130158
libp2p_swarm_connections_establishment_duration_bucket{le="22.168378200531007",role="Listener",protocols="/ip4/udp/quic-v1"} 6130158
libp2p_swarm_connections_establishment_duration_bucket{le="+Inf",role="Listener",protocols="/ip4/udp/quic-v1"} 6130158
libp2p_swarm_connections_establishment_duration_sum{role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 104668.51793091954
libp2p_swarm_connections_establishment_duration_count{role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 48809
libp2p_swarm_connections_establishment_duration_bucket{le="0.01",role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 0
libp2p_swarm_connections_establishment_duration_bucket{le="0.015",role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 0
libp2p_swarm_connections_establishment_duration_bucket{le="0.0225",role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 0
libp2p_swarm_connections_establishment_duration_bucket{le="0.03375",role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 0
libp2p_swarm_connections_establishment_duration_bucket{le="0.050625",role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 0
libp2p_swarm_connections_establishment_duration_bucket{le="0.0759375",role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 0
libp2p_swarm_connections_establishment_duration_bucket{le="0.11390625",role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 4
libp2p_swarm_connections_establishment_duration_bucket{le="0.170859375",role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 7
libp2p_swarm_connections_establishment_duration_bucket{le="0.2562890625",role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 49
libp2p_swarm_connections_establishment_duration_bucket{le="0.38443359375",role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 300
libp2p_swarm_connections_establishment_duration_bucket{le="0.576650390625",role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 1893
libp2p_swarm_connections_establishment_duration_bucket{le="0.8649755859375",role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 8607
libp2p_swarm_connections_establishment_duration_bucket{le="1.29746337890625",role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 24372
libp2p_swarm_connections_establishment_duration_bucket{le="1.946195068359375",role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 39332
libp2p_swarm_connections_establishment_duration_bucket{le="2.9192926025390628",role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 43437
libp2p_swarm_connections_establishment_duration_bucket{le="4.378938903808594",role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 44910
libp2p_swarm_connections_establishment_duration_bucket{le="6.568408355712891",role="Dialer",protocols="/ip4/udp/quic-v1/p2p/p2p-circuit/p2p"} 45558
libp2p_swarm_connections_establishment_duration_bucket{le="9.852612533569337",role="Dial
2 Likes
16

Oh right. I think @Shu wasn’t a fan of that information being in the metrics because it was obtainable from safenode-manager status --details.

4 Likes
17

the issue I’m getting is running safe node manager on machines running lots of nodes is very resource intensive so trying to do it say 4 times an hour has a real affect on the load averages.

also with trying to share out the community dash board to new comers. being able to get everything from that file is supper easy and light on resources with a simple curl and grep with out going down the rpc or safe node manager route.

if it would be possible to just have a comment in the file I can grep it doesn’t need to be an actual metric it would be very much appreciated :slight_smile:

6 Likes
18

Thanks so much to the entire Autonomi team for all of your hard work! :man_factory_worker: :man_factory_worker: :man_factory_worker:

Also, good luck to everyone earning beta rewards! :four_leaf_clover:

5 Likes
19

Personally I don’t have a problem adding it in just for convenience. I’m not the one you’d need to convince, though :slight_smile: .

4 Likes
20

Each of the metrics on this endpoint that can be obtained are in OpenMetrics format. Each of the metrics if present have a valid value here. Internally, this endpoint is being processed on 2000+ safenodes services * # of different testnets, and harvesting these data points at a constant interval.

If this has to be added to this endpoint, it needs to maintain OpenMetrics format. I believe stuffing everything inside a comment line doesn’t make sense for this data point.

I realize it might be convenient for your scripts, but the consensus internally was to not put it in /metrics including that peer id and version aren’t in the same bucket as the other metrics that govern the network metrics in general found on this /metrics endpoint. A version and peer id aren’t going to be changing that often as the remaining metrics once the safenode pid is up and running, and one would be parsing and storing strings continuously outside of numeric values as well.

Lets go back to your root problem of the status --details being slow. Are you saying its taking more than 15 minutes to query the results (4 times a hour) per host? How many safenode services are you running on a single physical or virtual machine? Is this a typical use case where the slow down is excessive and constantly re-occurring?

The peer id and version from the status command is used internally as well to harvest this data and has not caused any issues thus far.

4 Likes