BasicEconomyTweaks [Early Technical Beta] [OFFLINE - see new beta test - part deux]

[anon99156678]

I only got through about half the messages on here, but kept a few nodes running all evening.

“Classic_shot_of_the_ENIAC.jpg” c53079c3bf0fdf83d518cc0bf40a0745874e82eddadf4d18a139d011400db462 available for download, if anyone is in the mood for a charming new desktop wallpaper, from a simpler time in computing.

Otherwise, I successfully downloaded @Southside’s “AnarchyInTheSouthside.mp3”, and it’s been a musical rollercaster. Some crunchy tunes, to say the least! Has anyone else indulged? Is that yourself abusing the decks, southside? Very enjoyable.

I spun up 5 nodes, but I’m just on my phone’s wifi hotspotting to the laptop, so maybe I’m just not connecting properly, and that’s of no use at this stage?

1. No records means I didn’t actually store anything?
2. Does that mean that everyone at this stage is either port forwarding which they’ve manually set up, or they’re renting VPNs or similar?
3. What is the difference between records, PUTs and GETs then? Why are PUTs and GETs showing up at all?
4. What command are people running to get a snapshot of the logs? Just $ echo ~/.local/blahblah/logs > dump-file.txt? Or something else?
5. Are there any neat solutions to avoid typing horrendously long filenames? I made symlinks for each node (like below), which worked fine, but I imagine there might be something niftier.

$ ln -s ~/.local/share/blah-blah/safenode.log ~/.n1

It was very exciting anyway doing some uploading and downloading and seeing it there. Also, I must say - vdash is wonderful! Thank you @happybeing! Very intuitive to figure out and nicely documented. Makes getting an idea what’s going on much easier.

[Shu]

@anon99156678

Does that mean that everyone at this stage is either port forwarding which they’ve manually set up, or they’re renting VPNs or similar?

NAT Port Forwarding setup properly, or public WAN IP itself on the VM etc.

@anon99156678

What is the difference between records, PUTs and GETs then? Why are PUTs and GETs showing up at all?

PUT → a request to store data to your local node
GET → a request to retrieve data from your local node

I can’t comment on some of the other questions etc regarding vdash and its internals.

---

By any chance are your errors happen to be with HandshakeTimedOut too?

If you could run the following against that safenode.log, it would add to the data points:

cat safenode.log | grep -Eo 'PeerId\(\"(\w{52})\")|(p2p\/\w{52})' | grep -Eo '\w{52}' | sort | uniq | wc -l

cat safenode.log | grep "OutgoingConnectionError" | grep "HandshakeTimedOut" | grep -Eo 'PeerId\(\"(\w{52})\")|(p2p\/\w{52})' | grep -Eo '\w{52}' | sort | uniq | wc -l

[Southside]

@anon99156678

Is that yourself abusing the decks, southside?

No me, I’m afraid. Long long way outside my skillset…

@aatonnomicc found the track [MemDebugNet] [4/10/23 Testnet] [Offline] - #210 by aatonnomicc

[Southside]

hyperinflation…

🔗 Connected to the Network                                                                                                                                  Chunking 173 files...
Splitting and uploading "/fgfs/Aircraft/Beechcraft-T6-Texan-II/" into 548 chunks
Error: 
   0: Not enough balance in wallet to pay for chunk. We have
 NanoTokens(5907527951) but need NanoTokens(42136938741940) to pay for the chunk

[Shu]

Wanted to note here, I am seeing non public IPs being carried over or conveyed to remote peers… this is triggering MultiAddr not supported messages, safe to ignore or possibly a concern?

safe-node-155-0:# cat safenode.log | grep 12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b
[2024-04-02T03:11:22.238562Z INFO sn_networking::event] received identify info from undialed peer for not full kbucket Some(254), dail back to confirm external accesable peer_id=12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b addrs={"/ip4/65.108.236.166/udp/10425/quic-v1"}
[2024-04-02T03:11:22.464108Z INFO sn_networking::event] New peer added to routing table: PeerId("12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b"), now we have #191 connected peers
[2024-04-02T03:11:22.464128Z INFO sn_networking::event] Peer PeerId("12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b") has a Some(254) distance to us
[2024-04-02T03:11:22.464158Z INFO sn_networking::event] kad_event::RoutingUpdated 191: PeerId("12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b"), is_new_peer: true old_peer: None
[2024-04-02T03:11:22.464272Z INFO sn_node::log_markers] PeerAddedToRoutingTable(PeerId("12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b"))
[2024-04-02T03:16:28.808829Z WARN sn_networking::event] OutgoingConnectionError to PeerId("12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b") on ConnectionId(1078) - Transport([("/ip4/127.0.0.1/udp/10425/quic-v1/p2p/12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b", MultiaddrNotSupported("/ip4/127.0.0.1/udp/10425/quic-v1/p2p/12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b")), ("/ip4/10.0.0.1/udp/10425/quic-v1/p2p/12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b", MultiaddrNotSupported("/ip4/10.0.0.1/udp/10425/quic-v1/p2p/12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b")), ("/ip4/192.168.9.1/udp/10425/quic-v1/p2p/12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b", MultiaddrNotSupported("/ip4/192.168.9.1/udp/10425/quic-v1/p2p/12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b")), ("/ip4/65.108.236.166/udp/10425/quic-v1/p2p/12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b", Other(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }))])
[2024-04-02T03:16:28.808885Z ERROR sn_networking::event] OutgoingTransport error : MultiaddrNotSupported("/ip4/127.0.0.1/udp/10425/quic-v1/p2p/12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b")
[2024-04-02T03:16:28.808892Z WARN sn_networking::event] Multiaddr not supported : "/ip4/127.0.0.1/udp/10425/quic-v1/p2p/12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b"
[2024-04-02T03:16:28.808900Z ERROR sn_networking::event] OutgoingTransport error : MultiaddrNotSupported("/ip4/10.0.0.1/udp/10425/quic-v1/p2p/12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b")
[2024-04-02T03:16:28.808907Z WARN sn_networking::event] Multiaddr not supported : "/ip4/10.0.0.1/udp/10425/quic-v1/p2p/12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b"
[2024-04-02T03:16:28.808913Z ERROR sn_networking::event] OutgoingTransport error : MultiaddrNotSupported("/ip4/192.168.9.1/udp/10425/quic-v1/p2p/12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b")
[2024-04-02T03:16:28.808920Z WARN sn_networking::event] Multiaddr not supported : "/ip4/192.168.9.1/udp/10425/quic-v1/p2p/12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b"
[2024-04-02T03:16:28.808932Z WARN sn_networking::event] Cleaning out peer PeerId("12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b")
[2024-04-02T03:16:28.808948Z INFO sn_networking::event] Peer PeerId("12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b") has a Some(254) distance to us
[2024-04-02T03:16:28.809474Z INFO sn_node::log_markers] PeerRemovedFromRoutingTable(PeerId("12D3KooWEScVfEESAf5dYKin3Ss1aMnHe2RAJNa4H85wro4FEF8b"))

For the Peer ID above, the node must have started with a binding on ::10425 port on all adapters.

127.0.0.1 ← INTERNAL
10.0.0.1 ← INTERNAL
192.168.9.1 ← INTERNAL
65.108.236.166 ← WAN IP

Is some of this HandshakeTimedOut also getting confused between the multiple addresses being delivered in the payload? Could it be a different issue, if so, should one probably scrub the internal IP/port that are reserved for LAN segments only, unless offcourse libp2p is intelligently is picking the right ip/port/peer id url to send outbound here? If so, never-mind …

Eventually in the example above, the peer was added only for it to be removed within 6 seconds.

---

Also, FWIW, I have up’ed my socket send/receive buffers size on a single Linux hosts as per this snippet of text found on the internet (not sure if the scenario below applies to libp2p with quic):

Experiments have shown that QUIC transfers on high-bandwidth connections can be limited by the size of the UDP receive and send buffer. The receive buffer holds packets that have been received by the kernel, but not yet read by the application (quic-go in this case). The send buffer holds packets that have been sent by quic-go, but not sent out by the kernel. In both cases, once these buffers fill up, the kernel will drop any new incoming packet.

sysctl -w net.core.rmem_max=2500000
sysctl -w net.core.wmem_max=2500000

---

I am experimenting with the wrapper startup script that continues to monitor the CPU so it stays less than 50% (4 core LXC), before triggering the next safenode pid to launch… as arbitrarily guessing different intervals wasn’t ideal to smooth the cpu usage curve when spinning up 50+ safenode pids. I felt it needed to take CPU load into account so to not max out the cpu.

I suspect some of the failed bootstraps might be due to CPU load with too many safenode pids initially ramping up or a QUIC related issue too… (no firm data here to back that theory up atm), but wanted to have a graceful ramp up in as timely off a manner as possible so pivoted based on CPU load levels for now instead off --interval.

No scientific results trial done on the above, but FWIW, with the above changes, 47 out of the 49 safenodes that were running did bootstrap properly (along with the DialPeerConditionFalse and continued HandshakeTimedout messages). It still was a lot better success rate on bootstrap than before.

[joshuef]

I think what you describe is where we’ll get towards in some form. I’d be keen to keep it as simple as possible. In Ye Olde Network we have a version of fault detectin with weights and averages and standard deviations etc… And it was pretty hard to reason about.

But certainly we’ll need more nuance. Will be good to KISS as much as possible there though.

@Shu

safe network’s code base that are currently used or its all default libp2p settings

It’s mostly default, there’s a couple changes (safe_network/sn_networking/src/driver.rs at main · maidsafe/safe_network · GitHub)

@Shu

I am seeing non public IPs being carried over or conveyed to remote peers…

Aha, we had that filtered previously. cc @bzee not sure if that should have changed?

@Shu

No scientific results trial done on the above, but FWIW, with the above changes, 47 out of the 49 safenodes that were running did bootstrap properly (along with the DialPeerConditionFalse and continued HandshakeTimedout messages). It still was a lot better success rate on bootstrap than before.

Oh very interesting!

[riddim]

badly needed i guess - not earning anything but nobody will be able to upload at current prices -.-" …

[joshuef]

yup! but that’s good. At least one point of the economics there is working as intended

So now we’ll see (“now”== “soon”) if the updated pricing algo (and potentially more nodes) will have the desired effect!

[happybeing]

Small files worked fine yesterday so don’t be so greedy

[riddim]

is there a possibility for vdash to set the earnings to n€ / µ€ / m€ ? those leading zeros look a bit depressing when you look at them long enough xD

ps: on a live network it can’t stay at current earning rates … so it’s more of a test network issue - but maybe it would be just a minor change for you and would look nice for the next couple of weeks

[joshuef]

@Shu

I suspect some of the failed bootstraps might be due to CPU load with too many safenode pids initially ramping up or a QUIC related issue too… (no firm data here to back that theory up atm), but wanted to have a graceful ramp up in as timely off a manner as possible so pivoted based on CPU load levels for now instead off --interval.

I think this may be why it’s all local/out.

fix(networking): don't report ConnectionIssues during initial bootstrap by joshuef · Pull Request #1550 · maidsafe/safe_network · GitHub Aims to solve this, basically saying we don’t treat any bootstrap node as bad until we our ourselves, bootstrapped.

That should allow for more CPU variation and comms issues during that initial phase…

[neo]

@joshuef

Network we have a version of fault detectin with weights and averages and standard deviations etc… And it was pretty hard to reason about.

Definitely. LOL

Yea I didn’t explain it great, but you’d already have some way of measuring the “badness” now because you have some tipping point where you declare the node bad. And all this is having something like

(Badness + RT emptiness) / 2
Where:-

• badness is 0-100
• RT emptiness is 0-100 Maybe even 100-RT size or similar normalised to 0-100 (negative value is zero)

and if above some tipping point then ignore the node, with 2 tests first for badness > x being immediate ignore and if badness < y then do not ignore.

2 if tests and if undecided then do the equation

This would be a simple and non precise, but hey its all imprecise anyhow so the fact the equation is not precise isn’t so much an issue.

[joshuef]

Yeh, that sounds like the method.

Atm it’s just “3 things in the last 10 mins?”

Which is nice. Converting to a score is alright. And probs helps clarify/adjust a bit easier too.

[neo]

Also if things can be easy to understand then hopefully the next person to delve into the code will understand it straight away and less likely to cause days of testing because it didn’t work properly

[happybeing]

Thanks for the suggestion. I’m to busy with a demo app to look at vdash except for critical bugs, but if you - or anyone else - have suggestions you feel strongly about please open an issue or they will get forgotten.

I got something I’m very excited about working yesterday but want to do a bit more before showing it. Just a demo, but IMHO will blow some socks off.

[riddim]

then nevermind! those small numbers will vanish in time anyway =)

[anon26713768]

@happybeing

but IMHO will blow some socks off.

Can I get a pole and a beer with that tease please.

@joshuef

you could make a github issue with repro steps

Sorry forgot yesterday, will do.

[Profess]

@MrKeepingItSafe

You can check your reward balance by running:
safe wallet balance --peer-id=12D3KooWB8pSqrcmoBokCozxM5iNwe3fvxY8v7DQGz3jLnuCJqy3

Powershell hangs after this? can’t type anything?

You are probably doing everything right, try opening a new session in the terminal and checking the balance again, alternatively check that you have the latest client and node binaries.

---

No Safe, no wave.

[Toivo]

Hey, @aatonnomicc, @Shu, @anon26713768, @storage_guy… and everybody who usually throw a lot of nodes to these testnets. I have an idea:

What if for the next testnet you would start with for example 30% of your normal count and only when the network is getting a bit fuller you would throw the rest 70%? I don’t have any particular reason to propose this other than I would just like to see the storage cost go down, and nodes getting emptier with more nodes added. I don’t know if there is really any value in testing this, just an idea crossing my head.

[anon26713768]

I haven’t been going crazy, only have 25 for this test.

Interesting idea though. Unfortunately it looks like Joshuef adding 500 didn’t change anything, but that was before the new pricing algo tweaks.