I lost 3/10 nodes during night, according to logs memory usage of nodes more than doubled at the peak and machine run out of memory.
Btw I am seeing these in logs at random times, what does this mean?
[2023-08-03T06:25:02.483276Z DEBUG sn_networking::event] ConnectionClosed: incoming (/ip4/167.99.197.187/tcp/36748) peer_id=12D3KooWL2hvTUuFw2nCX4BvjK8VziuJQnHndsarhbNpNz6oE8bE connection_id=ConnectionId(731645) cause=Some(IO(Custom { kind: Other, error: Error(Closed) })) num_established=1
I was going to work on my script for downloading files I’ve previously uploaded and checking them against the md5sum file I uploaded but I’ve found that downloads seem to not be working. I’ve had a download of this file running for an hour with nothing happening:-
time safe files download f8d2f3d2d4378e20729c0fc1a415ef918958cac21c4eb14ceca3b544021bdc21 > 1MB_43
It’s a file that was uploaded a couple of few days ago. I’ve got:-
🔗 Connected to the Network
but nothing else
We log those, but they are not errors, well networking so it can mean a lot, but most likely other side closed the connection as it’s finished receiving etc.
My node continues to look good with PUTS, GETS and ERRORS shown in the vdash timelines.
Memory use was steady at 103 MB after the big churn event a couple of days ago but increased to 107 MB overnight.
So there may be an issue with a memory leak triggered when activity spikes.
Have some failed uploads again this morning.
There is a slight rise in open file descriptors over time, anyone else see the same?
idk if that is a cause for concern.
Your file downloaded fine for me (md5 aaa4f6c050c780cf6909882c827a9be5).
Also my file, which I uploaded at 2023-07-26, still downloads fine.
Thanks!
I’m still having issues…
I’m glad things are still working in general though.
This is not a weekend I can play but in a few days I’ll be able to work on a proper methodology for testing downloads and verifying them using the md5sums.
These graphs all look super promising! Great job community!
My node is still running but memory is still increasing, up 9MB to 116MB since yesterday morning and twice the previous day’s increase which was up 4MB to 107MB.
Looks like a memory leak 
I see all the same, with regards to the failed uploads I saw, I am baffled. Seems that only I noticed any upload fails, they are working again but there are now 2 distinct periods where uploads failed consecutively.
Iv got failing uploads as well hanging on making payment
Built with git version: 1262368 / main / 1262368
Instantiating a SAFE client...
⠐ 0/20 initial peers found. The client still does not know enough network nodes.
🔗 Connected to the Network Preparing (chunking) files at './safe-scripts/'...
Making payment for 84 Chunks that belong to 58 file/s.
Tried to upload a pic just for fun, now that it seems the network falters a bit at times for some. Went smooth, so did the download (I have also attempted downloading files I uploaded ten days ago. Curiously, nine days ago, two thirds of the download attempts failed. Now, all smooth. Retrieved all of them.)
I understand it’s just a quickly assembled testnet, wobbly as it should, but the overall experience is very encouraging for me as a user. I say this as someone who does not go around high-fiving everyone. This just gives me a good, good feeling. There’s something happening there and I believe this utopia may actually become a thing not too far from now.
I’m noticing that no Data Payment is made for creating and editing registers yet. Dunno if this just has not been implemented yet…
willie@gagarin:~/projects/GPT/GPT4all/bin$ safe register edit wee_black_book D.Ross
Built with git version: 1262368 / main / 1262368
Instantiating a SAFE client...
⠁ 0/20 initial peers found. The client still does not know enough network nodes.
🔗 Connected to the Network Trying to retrieve Register from 7812bc(01111000).., 3006
Successfully retrieved Register 'wee_black_book' from RegisterAddress { name: 7812bc(01111000).., tag: 3006 }!
Editing Register 'wee_black_book' with: D.Ross
willie@gagarin:~/projects/GPT/GPT4all/bin$ safe wallet balance
Built with git version: 1262368 / main / 1262368
Instantiating a SAFE client...
⠁ 0/20 initial peers found. The client still does not know enough network nodes.
🔗 Connected to the Network 109.999999585
willie@gagarin:~/projects/GPT/GPT4all/bin$ safe register edit wee_black_book M.Fraser
Built with git version: 1262368 / main / 1262368
Instantiating a SAFE client...
⠁ 0/20 initial peers found. The client still does not know enough network nodes.
🔗 Connected to the Network Trying to retrieve Register from 7812bc(01111000).., 3006
Successfully retrieved Register 'wee_black_book' from RegisterAddress { name: 7812bc(01111000).., tag: 3006 }!
Editing Register 'wee_black_book' with: M.Fraser
willie@gagarin:~/projects/GPT/GPT4all/bin$ safe wallet balance
Built with git version: 1262368 / main / 1262368
Instantiating a SAFE client...
⠁ 0/20 initial peers found. The client still does not know enough network nodes.
🔗 Connected to the Network 109.999999585
I take it all back, downloads are still working - just rather slowly. Less than 1KB a second.
while true; do date ; ls -l 1MB_68 ; sleep 1; done
Sat Aug 5 22:53:31 UTC 2023
-rw-rw-r-- 1 ubuntu ubuntu 82857 Aug 5 22:53 1MB_68
Sat Aug 5 22:53:32 UTC 2023
-rw-rw-r-- 1 ubuntu ubuntu 82857 Aug 5 22:53 1MB_68
Sat Aug 5 22:53:33 UTC 2023
-rw-rw-r-- 1 ubuntu ubuntu 83109 Aug 5 22:53 1MB_68
Sat Aug 5 22:53:34 UTC 2023
-rw-rw-r-- 1 ubuntu ubuntu 83109 Aug 5 22:53 1MB_68
Sat Aug 5 22:53:35 UTC 2023
-rw-rw-r-- 1 ubuntu ubuntu 83109 Aug 5 22:53 1MB_68
Sat Aug 5 22:53:36 UTC 2023
-rw-rw-r-- 1 ubuntu ubuntu 83360 Aug 5 22:53 1MB_68
Sat Aug 5 22:53:37 UTC 2023
-rw-rw-r-- 1 ubuntu ubuntu 83360 Aug 5 22:53 1MB_68
Sat Aug 5 22:53:38 UTC 2023
-rw-rw-r-- 1 ubuntu ubuntu 83360 Aug 5 22:53 1MB_68
Sat Aug 5 22:53:39 UTC 2023
-rw-rw-r-- 1 ubuntu ubuntu 83360 Aug 5 22:53 1MB_68
Sat Aug 5 22:53:40 UTC 2023
-rw-rw-r-- 1 ubuntu ubuntu 83610 Aug 5 22:53 1MB_68
The download of the 1MB file did eventually complete after about 9 hours.
At this point I wish I’d uploaded a whole load of 1KB files for the upload and download verification thing I want to do. But at the time part of the point was to get some more data in so that people would get some records.
If the network is still up at the end of this week when I have more time to play I’ll upload some 1KB files with their md5sums and write the verification script.
Overnight, memory use is up another 13Mb to 120MB.
I’m also seeing this: mine is on 137Mb now up from 125ish yesterday
It’s odd that you’ve lost a node without oom-kill killing it because of low RAM on your machine. I wonder if there is something in the logs at the end that would be useful?
You could find which safenode is missing by running:-
lsof | grep safenode.log
(assuming that is where you stored the logs)
and looking for which numbered safenode is missing.
I know which one was it, I just didn’t see what went wrong. I searched the logs again and there was another network activity spike at that time together with a memory usage spike and it died because of not enough memory.