NoEncryptionNet [07/02/24 Testnet] [Offline]

Chunking 52208 files…
Uploading 158668 chunks
⠈ [19:55:25] [#######################>----------------] 93196/158668
Upload terminated due to un-recoverable error Err(SequentialUploadPaymentError)
Error:
0: Failed to upload chunk batch: Too many sequential upload payment failures

*Directory of 52208, 5KB files totalling 7.34GB

*Always the same error and once it fails, no more uploads can be actioned.

*Reinstalled the client to upload the logs, but that also failed →

Uploading 1139 chunks
⠋ [00:49:03] [###################################>----] 997/1139
Retrying failed chunks 142 …
⠒ [01:09:57] [####################################>—] 1045/1139
Retrying failed chunks 94 …
⠁ [01:28:34] [#####################################>–] 1058/1139
Retrying failed chunks 81 …
⠁ [01:44:56] [#####################################>–] 1067/1139
Retrying failed chunks 72 …
⠂ [01:59:59] [#####################################>–] 1073/1139
Retrying failed chunks 66 …
⠖ [02:12:53] [#####################################>–] 1082/1139
Retrying failed chunks 57 …
⠓ [02:23:48] [######################################>-] 1091/1139
Retrying failed chunks 48 …
⠤ [02:33:41] [######################################>-] 1099/1139
Retrying failed chunks 40 …
⠙ [02:44:00] [######################################>-] 1101/1139
Retrying failed chunks 38 …
⠄ [02:53:46] [######################################>-] 1102/1139
Retrying failed chunks 37 …
⠁ [03:03:34] [######################################>-] 1103/1139
Retrying failed chunks 36 …
⠲ [03:10:32] [######################################>-] 1108/1139
Retrying failed chunks 31 …
⠒ [03:17:27] [######################################>-] 1109/1139
Retrying failed chunks 30 …
⠁ [03:24:13] [######################################>-] 1110/1139
Retrying failed chunks 29 …
⠄ [03:48:28] [######################################>-] 1110/1139
Upload terminated due to un-recoverable error Err(SequentialUploadPaymentError)
Error:
0: Failed to upload chunk batch: Too many sequential upload payment failures

3 Likes

A bit late into the party, but I finally fixed my issues with port forwarding. I was focused in the router, which was fine all the time, but did not notice the docker container had a default redirection as tcp, that I needed to explicitly change to udp.
It’s been behaving well overnight, one node a bit off in memory usage (please notice I used these parameters in the container for avoiding resources problems in my “production” RPI4:
--cpus=1.5 -m 800m):

5 Likes

I have been getting that as well. But after killing the process I can still do uploads of smaller files. Although it took 13 mins there to upload a 10 byte file so I think we are holed below the water line.

I also have a node with an outrageous RAM usage. 3GB just now.

Very early on (but I was a day late to this testnet) it was up at more than 400MB but dropped back down again. Then it went on another excursion up to about 2GB before recovering a bit. Now it’s got on a rampage up to 3044MB and still climbing.

I’ll post the logs when I get some more time.

The other 4 nodes on that VM are at 65 to 104MB and have spent most of their time below 60MB.

1 Like

It may be the above node mem explosions are bringing us down at the moment.

There are potential fixes in the pipeline… we’ll see if we manage to resuscitate this with those fixes.

4 Likes

20 hours of uploading and it fails @ 93196/158668 chunks

These are originally 5KB files…Safe seems to struggle with this scenario

2 Likes

Same thing here, my setting was 100 logs 0 archives and nodes have only 1-4 logfiles.

3 Likes

Yes, I know I can make a file public, I didn’t think about it when I sent it.
Now that I have uploaded a file, can I change its status to public or must it remain private?

2 Likes

you can just re-upload the same file with the -p option.

4 Likes

Is this expected behavior? Seems wrong, should there not be a bunch.

Also not sure if I understand the record limit removal, nodes appear capped?
newplot (3)

1 Like

This also seems suspicious. These are stats from a random node. According to logs, it was doing it even at the start of the testnet when network was working fine, but why is there this periodic oscillation?

5 Likes

If it’s about every twenty seconds (from memory) it’s been present for several tests.

1 Like

@bzee has been looking into reducing certain aspects of ndoe workload, which feel as if they’re doing too much. I wonder if you’ve any inkling there @bzee at what could be the cause?

3 Likes

I should have put bigger font in the graph description. Those peaks are 2-3 minutes and it repeats every 5 minutes.

5 Likes

It definitely seem nodes are capped at 2048 records and maybe 70% of my nodes is at that value.

I think what we are seeing is endless loop of nodes trying to replicate chunks and failing because either it cannot be finished or some other chunks are dropped instead.

3 Likes

I;m attempting a node upgrade command now. Let’s see how this process goes! (the newest node version has some fixes for suspected-mem spikes contributors.)

8 Likes

I think it’s unlikely to be related to the Identify behavior, as from the logs it looks very rhythmic/steady interval. It could be in theory though, I’m quite curious if there’s any hints in the logs that correlate to the spikes.

2 Likes

All logs from that node:
https://file.io/WEvXEzn68qx7

5 Likes

Just to let everyone know, because of the way this testnet was deployed–using a branch specification rather than a version number–the upgrade process will not work correctly. The issue was fixed when this PR was merged, but that was after this testnet had already been deployed.

So we will need to wait until the next testnet to try an upgrade.

10 Likes

Does this mean this testnet is close to death and there is no point keeping nodes on?

3 Likes