Hi Everyone. I’m stress testing and polishing the Colony GUI this week. I had the app download everything I’ve indexed on the network, about 15GB worth of stuff, all at once. It was able to get everything downloaded on the first pass, which was great! But it highlighted an issue, particularly with large files that I didn’t anticipate. It looks like the data_get_public() method that I’m using to pull stuff down buffers all of the data in memory before returning the whole blob at once to be written to a file.
I realize its doing a lot more under the hood than a simple HTTP data stream to reassemble the chunks, decrypt, etc. to get the final Bytes data object. Has someone come up with a streaming version of this method in their projects so far? I figure something that streams the chunks directly to disk in some temp work directory. Or maybe the MaidSafe folks are working on something for this?
Awesome, thank you @Traktion ! This looks like exactly what I need. Do you have a project somewhere that’s using this crate that I could see an example use case?
The feature for big blobs is to use the old X.25 method of d bit type of data frame, cntrl or data and m bit signalling… hey there is more coming in sequence to expand receive buffers and cycle the priority higher to get more cpu%
for developer inspiration here is a link I hunted down.. this via Brave AI Assisted Search:
X.25 D and More Bits Function
The X.25 protocol suite, an ITU-T standard for packet-switched data communication in wide area networks (WAN), includes specific features such as the D bit and more bit, which play roles in managing data transmission. The D bit, or “data” bit, is used to indicate whether a packet contains user data or control information. The more bit, on the other hand, is used to indicate whether more data packets follow in a sequence, allowing for efficient flow control and error handling. These bits work in conjunction with other features of X.25, such as virtual circuits and error detection mechanisms, to ensure reliable and efficient data transmission across the network.
D bit : The D bit in X.25 is used to distinguish between data packets and control packets. When the D bit is set to 1, it indicates that the packet contains user data; when set to 0, it indicates that the packet contains control information.
More bit : The more bit is used to signal whether additional data packets are expected in a sequence. This helps in managing the flow of data and ensures that the receiving end is prepared to handle incoming packets
Interesting. I didn’t know X.25 was still alive and kicking. I’ve done plenty of packet radio over the years (AX.25) as well as APRS. Similar, but better error handling for bad radio links and some packet tagging to follow FCC regulations.
