SAFE Network Dev Update - July 23, 2020

[Blindsite2k]

i mean it hasn’t opened at all, period. Like hours after I tried launching it. No splash screen, no error messsages, no nothing, zip, nada.

[KafkaLee]

It may help u

The safe browser doesnt open for me on windows 10

Ive tried the gui some years ago around the time the logo was being voted on and it loaded ok but there wasnt any network at the time to play with it. I thought to give it another go now that theres a working network but i installed it and opened and nothing happens. I thought maybe i needed the CLI installed to have it open and ran through the git terminal window to install it, then from there proceeded to install the auth server and everything else i could find to install that was written on …

[zeroflaw]

Do you run testnets with your nodes attached to the rust-lldb debugger (and debug build)? (I know the performance would be degraded) but if the application is crashing I have found sometimes it can provide extremely valuable information.

[dirvine]

We use heaptrace here to track any memory issues. We did find 2 hot spots and are looking into that right now.

[zeroflaw]

It’s a useful tool, but you have an advantage of many nodes and they are crashing, so if the debugger is attached you’ll have a high chance of the allocation causing the OOM to be where the debugger catches the exact action of code.

Heaptrace is useful for a general area but can still make tracking down what’s causing the allocations to be difficult.

Just saying might be an idea to run the next testnet with the debugger attached, it could save you some time, and only takes 30 seconds to setup.

[lionel.faber]

Heaptrack has a top down view that helps us identify the exact point of the memory leak as well
It has led us to the exact function where it is happening. Looking at the code the reason behind the leak isn’t obvious so we are looking at some of our dependencies (and the way we use them) to identify where the leak happens.

[joshuef]

There’s an issue on windows for some folk: The safe browser doesnt open for me on windows 10 - #59 by joshuef

It’s not clear why it’s happening as yet. Please have a read through there and fire in any logs/info you can

Also please be sure you have the auth daemon installed (via cli: safe auth install).

[GeekOverdose]

epic. very cool

[mav]

@maidsafe

After about 14 hours, the Elders of the network were killed by the operating system because of high memory usage. This seemed similar to the issues we identified in the previous testnet, so we restarted the testnet with memory profiling enabled. Thanks to the help of the community, we were able to collect meaningful results that have helped us identify the source of the issue. The reason behind this high memory usage is because of memory leaks in two places. We continue investigating this to identify the exact root cause - at the moment it doesn’t look like the memory leaks are as a result of our code, so we are also looking at some of our dependencies and their usage. We will keep you posted on any advancements on this front.

Any tips about how to trigger the high memory usage? I’ve been trying to reproduce this with vault v0.25.1 but no luck so far.

[davidpbrown]

This was reproducible from test network.
So,

safe vault run-baby-fleming -t

On Linux Mint without putting data and just creating an account, it maxed out x4 i3 CPUs and chewed through 14of16GB… watch the system monitor and it’s obvious as .1% of memory disappearing every 4/5secs. That’s my memory of it…

see [Status: offline] SAFE Network Testnet - Vaults from home with IGD - Iteration 2 - #129 by davidpbrown

reducing the logs had no effect but can be done with

$ export RUST_LOG=safe=trace

[Dimitar]

Thank you for the heavy work team MaidSafe! Together we will make the Internet what it might have been!

I add the translation into Bulgarian in the first post

SAFE Network новини 🇧🇬 23.7.2020

[lionel.faber]

Heavy churn with the lots of data already in the network triggered the OOM killer. It isn’t easy to reproduce took about 12+ hours to reproduce it in an internal testnet.

[davidpbrown]

That’s curious… seemed to trigger almost from the start on my test networks, with no data and simple account creation… wondering if it’s mine was resource limited to just i3 and all vaults on the same external SSD3.0 that in some way forces a problem for i/o. Disclaimer I know nothing !

Is it possible to query memory on what is filling it up??