Big file upload problem - discussion

Development
1

Considering the network type (millions of nodes, each block stored to different node), the behviour is quite different compared to normal upload/download. If the timing is not carefully thought, you might very easily accidentally call a DoS attack to your router.
Its quite simple, just send a request that will trigger million nodes to send you one UDP frame. If it happens simultaneously, no router will survive that.

I assume this is taken into account, but since large file uploads does not work, thought it might be good idea to ask.
There should be some kind of flow control to prevent different nodes to send data to the same address simultaneously, is there such thing?

2 Likes
2

Tell me this request first. I do not know of a request to do that.

1 Like
3

I don’t know, if there is none, then this is not the root cause and so no help of this.

But lets think possible causes:

  • How many nodes typically send response to request for quotes?
  • how many requests for quotes are send when uploading? Are they sent as fast as possible or are there some limit how many pending requests are allowed?
4

Then don’t assume far fetch operations.

You request a chunk and one node, yes one node will send that chunk to you because you asked that particular node to send it

You request peer list (eg close to an xor address) from a node and that node will return that list.

Not much different to sending a request to a server, except this is to a peer in a distributed network.

There is no operations where you broadcast a request and multiple nodes respond. All communications are node to node (peer to peer), since this is a p2p network.

Now the client uploader will send multiple chunks as part of the programming but it is still one chunk to one node. For some routers the number of chunks being sent in my opinion is too many (8 I think) but its 8 nodes not millions the data is sent to.

downloading is multiple chunks too but again like requesting from 4 or 8 nodes at this time. One different chunk from each node.

5

Good, there is a flow control! But since some users have problems, maybe it should be configurable?

6

Its apparently being worked on currently.

I was going to have some theory figures for the team last week but a combination of personal events have delayed this for a bit.

Been doing comms since the 70’s when flow control was an essential feature of comms

2 Likes
7

Sorry, I disagree with this. The network has a problem with large file upload, you should welcome any suggestion that has even remote chance to help solving the problem.

But maybe i first should read the documents (or source code) before posting. And maybe this should be on the dev forum.

8

Going to millions when this is extremely unlikely is far fetched. Something more reasonable would not have gotten such a response from me.

1 Like
9

You started 20 years before me, congrats!
Yes, back then knowing flow control was much more important than nowdays when almost all is TCP etc with ready made flow control. That’s why I raised this question, UDP for multiple endpoints is so different compared to TCP.

10

And why i am examining the flow control in quic. The team is super busy and maybe not enough time to try out different combinations of the 2 main parameters controlling flow. Imbalance of one to the other can just clog things down and it’ll take them time to try different combinations in the real world. That is why i want to do some graphs for them (theory) so hopefully guide them to combination that will work good for not only the potato routers but also high end setups. In that topic I did explain what was happening in the potato routers with buffer overflows due to UDP protocol.

Well I got to play with unix in 75 when it came here on a 11/4? computer. Going from RT11 to Unix required toggling in a bootloader to get the image from the removable rp05 (think I remember the model right)

1 Like