Update 14th November, 2024

Definitely need about 100 times the uploads AND 1000 times the downloading to get a realistic figure. As soon as that happens each node will be using a little more CPU and many people will lose a few/many nodes and we’ll see a better picture.

The error rate will rise dramatically (ie more volume of traffic due to retries) with more large chunks flowing rather than the vast majority being a few hundred byte protocol packets. Once data chunk traffic really exceeds protocol volume then it’ll be a real test of the network. At the moment “we are specifically asked not to just take larger node setups off line too fast”, “uploads are still slow”, “downloads almost non-existent” which adds up to treating the network with kids gloves.

The reason I say the error rate will rise with more usage is that the volume of traffic we see is mostly protocol traffic and the majority of it is one UDP packet in size back and forth adding up to retry volume very small. A chunk of any significant size is many udp packets in length and ALL packets have to arrive in good condition for the whole chunk to be considered valid, otherwise a retry of the whole chunk will occur. So while 1 in 1 million (to 100 million) UDP packets transfer through a switching/routing point in error, the error rate becomes significant once chunks are transferred.

EG for a 4MB chunk we have over 2,500 udp packets being transferred though at least 12 switch/routing points (2 only between nodes within datacentre like DO) the error rate figure used is increased by 32K. Thus the error rate is like 1 in 31 to 3100 chunks which for a setup with 100 nodes means you’ll see one chunk error every few time periods for an active network with uploading/churning/downloading occurring in volume. Of course this can be reduced with setting the QUIC window size down to 128K (1/32) which would reduce the error rate to 1 chunk in 1000 to 100K window size. The retry is the full window size sent again, max 4MB chunk currently. And if packets are dropped due to ISP router buffer size then its even worse.

10 Likes