Heapnet2 summary of observations

[Toivo]

I skimmed through the Heapnet2 -thread and tried to collect the main observations, for folks not willing to go through all that. Feel free to add your own, I’m sure I have missed many things:

1. Problems with faucet were user errors. Using arrow up in terminal may give you wrong faucet address.
2. Nodes may stay empty for a long time and then start to get chunks.
3. Problems in downloading were likely mostly because of bugs in client, not loss of data. Mostly fixed now.
4. Uploads are quite slow and CPU intensive.
5. Downloads are extremely fast.
6. Nodes are very resilient to disturbances in connections.
7. Clients are very sensitive to disturbances in connections. Up/downloads break and don’t restart.
8. Wallet corruption was a typical reason for problems in uploading.
9. Node’s memory consumption is creeping up very slowly, but RAM can also decrease after it has increased. More information:
   HeapNet2 [Testnet 12/10/23] [Offline] - #153 by Josh
   HeapNet2 [Testnet 12/10/23] [Offline] - #170 by storage_guy
   HeapNet2 [Testnet 12/10/23] [Offline] - #239 by storage_guy
10. Nodes can be run behind NAT with manual port forwarding.
11. Nodes don’t die that easily anymore.

Everything in general is working quite well already!
And improvements are coming steadily!
Big thanks to the team!

Some future prospects:

Ease of use:

HeapNet2 [Testnet 12/10/23] [Offline]

@joshuef had a revelation today that might even make this considerably more so. Let’s see, some tests and even more code simplification possibly en route. The feeling we are onto something quite revolutionary here is high. Much like the simplicity of LLMs giving such credible output or, of course, the simplicity of a chaotic ant colony. It really feels like the simplicity here is going to be amounting and will very much shake the foundations of many a decentralised approach that uses total ord…

Upload resilience:

HeapNet2 [Testnet 12/10/23] [Offline]

it makes perfect sense we have a queue for chunks to be stored and one for stored. So we move the chunks over when stored. That way we should be able to restart where we left of no matter how many files are being uploaded in parallel. It’s a relatively easy UX fix we can do. cc @chriso @roland

Lowering non-essential network traffic:

HeapNet2 [Testnet 12/10/23] [Offline]

Because we didn’t implement another way yet really, it’s not been a priority as yet. There’s no wild technical reason we can’t avoid that for verification I think. Beyond

[happybeing]

@Toivo

Nodes don’t die that easily anymore.

Did anyone report nodes dying?

Nice summary

[Toivo]

@happybeing

Did anyone report nodes dying?

I didn’t see any reports. But because the net still goes on, it’s a bit premature to say if the slow memory increase will lead to that. (If it ever was the reason.)

[andreruigrok]

Toivo thanks for your great resume

I say every year that we will get our Beta on 23 December, lets hope this year we will !!

Will we have the Safe Network, before, on or after December 23rd?

• Before
• On
• After

[Toivo]

@andreruigrok

Toivo thanks for your great resume

Thanks! And thanks to @happybeing too!

I’m planning to make this a habit, to make some kind of summary after every testnet has run for a while. I just hate how all the nuggets get lost in the main thread. (Where would this World be without all the hate?)

[DeusNexus]

Great work guys! @Toivo, @happybeing, @Josh, @storage_guy and everyone else I didn’t mention.
Is the API ready yet to be used for development? The testnets itself sound utterly promising and telling me to get started with building apps.

Edit:
I’m now working with Dask for Machine Learning / AI and one of the benefits is that can handle huge datasets spread over different files. Now that the download speeds are looking good and consistent it would be cool to see if I can already port the distributed dataset chunks and feed them towards Dask to perform Machine Learning on it. What I imagine is public register of open-source large datasets, they can either be downloaded to client or some sort of streaming to decentralized clusters. I really think these kind of use-cases are what makes Safe appealing, storage cost can be cheaper and data should be redundant if everything works out.

How could would it be if things like Event Horizon Telescope can publish their data on here and big data projects, AI, machine learning can all benefit from this.

[dirvine]

@DeusNexus

Is the API ready yet to be used for development?

I would say honestly, no. However, making them so is simple code wise, but we will need to press the community again to help us help them create a simple-to-use API, and it’s on us to iterate towards providing that as quickly as possible. We all now know we can be very quick, and we will be.

[Toivo]

@Toivo

@happybeing

Did anyone report nodes dying?

I didn’t see any reports. But because the net still goes on, it’s a bit premature to say if the slow memory increase will lead to that. (If it ever was the reason.)

Nodes are starting to die:

HeapNet2 [Testnet 12/10/23] [Offline]

I have lost 7 of 15 nodes on my 2GB SBC. It has been confusing me as to why. The lost nodes had no significant spike at the time of being ejected but the free memory is low with plentyy used by buffers/cache. It appears to me that that “available” memory is not being released. I was under the impression it is and will always be rapidly released for running processes. After looking into this a bit I was mistaken and if the system is under high I/O load it will not or it could be slow at doin…