There is an env setting of CHUNK_DOWNLOAD_BATCH_SIZE , which you can try set it to a lower number, say 8 or even 4, to see if download will become smoother.
6 Likes
I’ll have to double down on that. I tried downloading it — everything looked fine for the first few minutes, downloading at 3 Mb/s — and then my internet connection suddenly shut down. And now it seems I can’t recover internet after resetting my router. Don’t try this at home folks ! 
Ok that should probably be the default then (?)
EDIT: router not dead, I recovered internet. Ok let’s try again 
2 Likes
the default value is system::thread_parrallism (normally num_of_cpu) * 8.
8 Likes
Ive had some similar issues in the past, where it has taken out my wifi.
I wonder whether using CPU cores is just too aggressive for some home networks?
5 Likes
@home: my MacBook has 14 cores * 8 == 112 parallel download streams … behind one crappy router and a rather basic internet connection …
@dev_vps: 4 cores == 32 parallel stream on a 1gbit up and down connection …
not sure I need to say anymore … the parallel downloads should imho not be tied to the number of processors … the 2 things have nothing to do with each other imho … but good to know there is an env variable for it 
6 Likes
It probably makes sense as an upper limit, assuming there is CPU work to do for decryption etc. Just a lower limit, based on network performance, is needed too.
Perhaps a conservative default should be used, which can then be overridden by env variable/parameter/config.
6 Likes
oh well - but begblag seems not to be too many chunks for the router flying in in parallel and downloads in ~5 seconds =D so parallelism is a nice thing 
9 Likes
The limitation should not be connected to amount of concurrent threads. The implementation should be based on “window size”, ie define max amount of not-yet-replied request; do not send new request if the amount is greater than window size.
That would automatically adjust the speed to the maximum without filling the router buffers.
4 Likes
Actually limiting download by limiting number of threads is fundamentally wrong. If your download bandwidth equals to the average node upload speed, one thread is not enought and even with two it’s just matter of time, when the buffers are filled. If the file size is big enought, it will eventually fail.
For this I assume blocking threads. For nio it’s totally different.
Hmm.. actually, I might be wrong. if the thread is blocked during each request, then it actually limits the amount of resquest-waiting-response to the amount of threads, ie it’s like window size.
2 Likes
We might try to estimate:
Buffers needed: num_of_threads*chunk size, where chunk size is 4M
For optimal bandwidth: we should know average node upload speed? Currently it might be quite high since most nodes are on cloud server? But if majority of nodes are on home network, the lowest speed might be even 10M.
To get maximum speed, threads should be at least your speed/average node upload speed.
Latency plays also some role, but I assume relatively slow.
Currently I would say 2 threads will most likely give you quite close to max speed. Later maybe up to 10, assuming 100M average node upload bw, and you have 1 G download bw.
If your router has at least 40M buffers, 10 should not harm.
1 Like
If you have 1Gbps download or less with a LAN of 1Gbps or higher then the buffer usage in the router should be minimal.
The global average for upload speeds is 48Mbps according to a speed test site. And this average is very much skewed because each 1Gpbs or higher upload connection matches 40 more common 25Mbps upload speed connections. So most common does not match average.
Also the most common download speeds is also much less than 1Gbps
The issue then is that while you can work out the sums for max speed download and completely max out your download speed, is it desirable? Not really if your household/business is using the internet for anything else (eg others watching youtube, businesses doing business stuff, or other people downloading from Autonomi)
There has to be consideration for what the application is that is downloading. EG a video viewer only needs say 30 seconds of buffer, so no need to max out a connection trying to get whole file when only few chunks ever need to be buffered up.
2 Likes