AntTP - Serving Autonomi data over HTTP

Traktion · May 26, 2025, 10:28pm

Yes, for sure. It just takes a lot of time to add that level of refinement. The way I see it, we must try to grab all the large, low hanging, fruit first. Then we can start spending quality time on nailing ease of use.

For example, I could imagine an installer that puts an icon on your menu thar launches a browser with the proxy already setup. At the same time, it starts the AntTP service, if it isn’t already. All from one icon.

I could also imagine browser extensions giving us tighter integration, so things like ant:// can be used (or ants:// with the padlock? ). Maybe that could avoid having to setup a proxy altogether too?

I’m chatting with @safemedia about adding websockets support earlier to help with extension integration too, so we’re trying to collaborate to make the whole thing smoother and more seamless to use.

The important thing is that this stuff is all solvable. It’s just a case of trying to find the most straightforward way to do it, with the least heavy lifting. If we choose wisely, we can go a long way, with relatively little. We just need to find the right giant’s shoulders to stand on!

loziniak · May 27, 2025, 6:31am

I would say it created a code summary, not a user manual, which I expected. It’s still very useful

Traktion · May 27, 2025, 9:02am

I wanted to link this debate here…

FWIW, I’m designing something like this into AntTP.

In v0.6.0, when you perform an API create/update (for pointers/registers), it currently just executes in the background and returns control immediately - it assumes success, but a get can be used to confirm success if needed.

However, the plan is to push these creates/updates into a queue/pool of tasks that don’t immediately get sent to the network. Instead, they will be pending signing by the creator. Once signed, they will be sent to the network.

In short, there will be a header which can be used to provide a public key associated with the operation (it can/should be different from other keys for improved security). When a new task is submitted, the user can then confirm and sign it with their private key, which will then trigger AntTP to process the task.

I expect this signing flow would be a separate app and/or browser plugin (or part of this browser extension? ). It would list / detail the tasks and allow management over batching tasks, etc, to avoid annoying noise (confirming everything could get painful!).

It would also open the door to the client encrypting the data with said public key. The signing flow could then decrypt, validate, then sign. This would improve security and avoid man-in-the-middle attacks (at least between client and remote AntTP, which is most public, but local AntTP doesn’t suffer MITM issues).

EDIT: Note, this could also potentially support multi-sig, where multiple parties would need to sign prior to the task being executed by AntTP.

Herodotos · May 27, 2025, 10:45am

I tried your latest AntTP 0.6.0 on Windows 11, default settings.[8 download threads]

I tested download of autonomi.mp4 video. Here is the result:

curl http://localhost:8080/91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309/to-autonomi.mp4 > autonomi.mp4
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 35.7M 0 35.7M 0 0 998k 0 --:–:-- 0:00:36 --:–:-- 9.8M

It took 36 seconds. Tried it multiple times, it ranged from 26-39 seconds.

Than I tried to download 1600MB ubuntu iso file.

curl http://localhost:8080/e7bb1b87c1f0e07cdb76ba5e82a425a8da712940c2d3553aa6791494e92aa54d/ubuntu-16.04.6-desktop-i386.iso > ubuntu-16.04.6-desktop-i386.iso
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1600M 0 1600M 0 0 836k 0 --:–:-- 0:32:39 --:–:-- 1406k

It took 32 minutes to download.

Than I tried the streaming of the same autonomi.mp4 file from browser, and it failed, this is the log with exceptions:

thread ‘actix-rt|system:0|arbiter:3’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_receiver.rs:33:43:
called Result::unwrap() on an Err value: JoinError::Panic(Id(2684), “attempt to subtract with overflow”, …)
[2025-05-27T09:22:50Z INFO actix_web::middleware::logger] 127.0.0.1 “GET /91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309/to-autonomi.mp4 HTTP/1.1” 206 0 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36” 0.041237
[2025-05-27T09:22:52Z INFO anttp::service::resolver_service] Found public archive at [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T09:22:52Z INFO anttp::controller::file_controller] Retrieving file from public archive [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T09:22:52Z INFO anttp::service::archive_helper] Resolved path [to-autonomi.mp4], path_buf [to-autonomi.mp4] to xor address [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3]
[2025-05-27T09:22:52Z INFO anttp::service::file_service] Streaming item [to-autonomi.mp4] at addr [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3], range_from [37224448], range_to [37487986]
[2025-05-27T09:22:52Z INFO autonomi::client::data_types::chunk] Getting chunk: 71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3

thread ‘actix-rt|system:0|arbiter:4’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_fetcher.rs:32:42:
attempt to subtract with overflow

thread ‘actix-rt|system:0|arbiter:4’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_receiver.rs:33:43:
called Result::unwrap() on an Err value: JoinError::Panic(Id(2689), “attempt to subtract with overflow”, …)
[2025-05-27T09:22:52Z INFO actix_web::middleware::logger] 127.0.0.1 “GET /91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309/to-autonomi.mp4 HTTP/1.1” 206 0 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36” 0.049763
[2025-05-27T09:22:54Z INFO anttp::service::resolver_service] Found public archive at [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T09:22:54Z INFO anttp::controller::file_controller] Retrieving file from public archive [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T09:22:54Z INFO anttp::service::archive_helper] Resolved path [to-autonomi.mp4], path_buf [to-autonomi.mp4] to xor address [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3]
[2025-05-27T09:22:54Z INFO anttp::service::file_service] Streaming item [to-autonomi.mp4] at addr [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3], range_from [37224448], range_to [37487986]
[2025-05-27T09:22:54Z INFO autonomi::client::data_types::chunk] Getting chunk: 71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3

Traktion · May 27, 2025, 11:18am

FYI, I didn’t notice that this didn’t update yesterday. I’ve just re-attempted the push and latest dockerhub should be up to date to v0.6.0 now too.

Traktion · May 27, 2025, 11:32am

Thanks for the bug report @Herodotos !

Do you have the log just before the first panic?

Judging by the error, I’m wondering if there is an overflow with the RANGE from/to some how, as CURL wouldn’t provide these.

A browser will sometimes ask for the first and last part of the file first, then start streaming (presumably to get some metadata).

It looks like the last RANGE worked, but the first one failed. Perhaps it is going negative, which could break a usize/u64 (as they are unsigned). I doubt it is getting too large, but that’s possible too.

I’ve not seen that error on Linux with Brave/Firefox/Chrome, but it’s possible Windows is handling the types differently.

EDIT: Apologies too - I’ve not got easy access to a Windows box to test this stuff on. I plan on getting hold of one soon though, as there have been some reported issues that are hard for me to reproduce.

Herodotos · May 27, 2025, 1:05pm

Here is the full log for that streaming test.

Summary

cargo run
Finished dev profile [unoptimized + debuginfo] target(s) in 0.64s
Running target\debug\anttp.exe
[2025-05-27T11:36:26Z INFO anttp::config::anttp_config] Listen address [0.0.0.0:8080]
[2025-05-27T11:36:26Z INFO anttp::config::anttp_config] Static file directory:
[2025-05-27T11:36:26Z INFO anttp::config::anttp_config] Wallet private key: [*****]
[2025-05-27T11:36:26Z INFO anttp::config::anttp_config] Download threads: [8]
[2025-05-27T11:36:26Z INFO anttp::config::anttp_config] Uploads disabled: [false]
[2025-05-27T11:36:26Z INFO anttp::config::anttp_config] No app private key provided. Try this one: [“36314943c39850652d33732adbfe35e2356ca11f341b9421b08a7ff16327dff1”]
[2025-05-27T11:36:26Z INFO anttp::config::anttp_config] Bookmarks [“traktion-blog=b70a146f95b3ff237fa8140c4175f6a302c8250fe268aacdb47c2783f2b2ee6af5575410d07f6eae9ac7fb9ce95995e4”, “imim=959c2ba5b84e1a68fedc14caaae96e97cfff19ff381127844586b2e0cdd2afdfb1687086a5668bced9f3dc35c03c9bd7”]
[2025-05-27T11:36:26Z INFO anttp::config::anttp_config] Cached mutable TTL 60
[2025-05-27T11:36:27Z INFO libp2p_swarm] local_peer_id=12D3KooWRUPE1naiLqdEJffHVXgkgpSmUpj9BKZyxC2EumsL6ve7
[2025-05-27T11:36:27Z INFO anttp] Starting listener
[2025-05-27T11:36:27Z INFO actix_server::builder] starting 28 workers
[2025-05-27T11:36:27Z INFO actix_server::server] Actix runtime found; starting in Actix runtime
[2025-05-27T11:36:27Z INFO actix_server::server] starting service: “actix-web-service-0.0.0.0:8080”, workers: 28, listening on: 0.0.0.0:8080
[2025-05-27T11:36:36Z INFO anttp::service::resolver_service] Found public archive at [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:36:36Z INFO anttp::controller::file_controller] Retrieving file from public archive [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:36:36Z INFO anttp::service::archive_helper] Resolved path [to-autonomi.mp4], path_buf [to-autonomi.mp4] to xor address [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3]
[2025-05-27T11:36:36Z INFO anttp::service::file_service] Streaming item [to-autonomi.mp4] at addr [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3], range_from [0], range_to [18446744073709551615]
[2025-05-27T11:36:36Z INFO autonomi::client::data_types::chunk] Getting chunk: 71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3
[2025-05-27T11:36:40Z INFO autonomi::client::data_types::chunk] Getting chunk: eccbf4cf2e9fbb77a229c1034239e88abd66307bbcf968b4dc585bec99b3d114
[2025-05-27T11:36:40Z INFO autonomi::client::data_types::chunk] Getting chunk: 04e00023bc250e264d0db3a146e0a92c2ec69637cd97c30692a75142f11ee174
[2025-05-27T11:36:40Z INFO autonomi::client::data_types::chunk] Getting chunk: 7fd15193d3bfe29dd71af97911b161fecc58a67c6807e3c4d13b315bd0979563
[2025-05-27T11:36:40Z INFO autonomi::client::data_types::chunk] Getting chunk: 81194febdf573d3c19878eb90a958ebd3b9233fd5faebb29338f260130ea6cbc
[2025-05-27T11:36:40Z INFO autonomi::client::data_types::chunk] Getting chunk: b0cf7085bfea983fb80107270875f6d298ea2131b1da25bb140ea0bf7af113ee
[2025-05-27T11:36:40Z INFO autonomi::client::data_types::chunk] Getting chunk: 86830fe6dd8dea22e272806e2ee32d8afab83900556e121c6b535acf238f46b5
[2025-05-27T11:36:40Z INFO autonomi::client::data_types::chunk] Getting chunk: bcf83693111d169e8dd4776f17105fad1b03e0daed9d9a0015a43b3fdcdd9ec6
[2025-05-27T11:36:40Z INFO autonomi::client::data_types::chunk] Getting chunk: 47786078a3c4b9f1a5e91cab414200f2748fbd8ee1e2ccafd23fb719b5957a70
[2025-05-27T11:36:40Z INFO autonomi::client::data_types::chunk] Getting chunk: 266c522c6f1285a29e4e4e8a513e3bf4d520bf9c5d7447d8ee5aae0de47c96cd
[2025-05-27T11:36:40Z WARN quinn_udp] sendmsg error: Os { code: 10040, kind: Uncategorized, message: “A message sent on a datagram socket was larger than the internal message buffer or some other network limit, or the buffer used to receive a datagram into was smaller than the datagram itself.” }, Transmit: { destination: 38.114.121.113:26509, src_ip: None, ecn: Some(Ect0), len: 1420, segment_size: None }
[2025-05-27T11:37:02Z INFO actix_web::middleware::logger] 127.0.0.1 “GET /91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309/to-autonomi.mp4 HTTP/1.1” 206 8388608 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36” 25.640284
[2025-05-27T11:37:02Z INFO anttp::service::resolver_service] Found public archive at [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:02Z INFO anttp::controller::file_controller] Retrieving file from public archive [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:02Z INFO anttp::service::archive_helper] Resolved path [to-autonomi.mp4], path_buf [to-autonomi.mp4] to xor address [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3]
[2025-05-27T11:37:02Z INFO anttp::service::file_service] Streaming item [to-autonomi.mp4] at addr [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3], range_from [37224448], range_to [37487986]
[2025-05-27T11:37:02Z INFO autonomi::client::data_types::chunk] Getting chunk: 71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3

thread ‘actix-rt|system:0|arbiter:2’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_fetcher.rs:32:42:
attempt to subtract with overflow
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

thread ‘actix-rt|system:0|arbiter:2’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_receiver.rs:33:43:
called Result::unwrap() on an Err value: JoinError::Panic(Id(2171), “attempt to subtract with overflow”, …)
[2025-05-27T11:37:03Z INFO actix_web::middleware::logger] 127.0.0.1 “GET /91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309/to-autonomi.mp4 HTTP/1.1” 206 0 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36” 0.537126
[2025-05-27T11:37:03Z INFO anttp::service::resolver_service] Found public archive at [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:03Z INFO anttp::controller::file_controller] Retrieving file from public archive [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:03Z INFO anttp::service::archive_helper] Resolved path [to-autonomi.mp4], path_buf [to-autonomi.mp4] to xor address [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3]
[2025-05-27T11:37:03Z INFO anttp::service::file_service] Streaming item [to-autonomi.mp4] at addr [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3], range_from [37224448], range_to [37487986]
[2025-05-27T11:37:03Z INFO autonomi::client::data_types::chunk] Getting chunk: 71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3

thread ‘actix-rt|system:0|arbiter:3’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_fetcher.rs:32:42:
attempt to subtract with overflow

thread ‘actix-rt|system:0|arbiter:3’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_receiver.rs:33:43:
called Result::unwrap() on an Err value: JoinError::Panic(Id(2174), “attempt to subtract with overflow”, …)
[2025-05-27T11:37:03Z INFO actix_web::middleware::logger] 127.0.0.1 “GET /91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309/to-autonomi.mp4 HTTP/1.1” 206 0 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36” 0.041432
[2025-05-27T11:37:04Z INFO anttp::service::resolver_service] Found public archive at [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:04Z INFO anttp::controller::file_controller] Retrieving file from public archive [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:04Z INFO anttp::service::archive_helper] Resolved path [to-autonomi.mp4], path_buf [to-autonomi.mp4] to xor address [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3]
[2025-05-27T11:37:04Z INFO anttp::service::file_service] Streaming item [to-autonomi.mp4] at addr [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3], range_from [37224448], range_to [37487986]
[2025-05-27T11:37:04Z INFO autonomi::client::data_types::chunk] Getting chunk: 71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3

thread ‘actix-rt|system:0|arbiter:4’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_fetcher.rs:32:42:
attempt to subtract with overflow

thread ‘actix-rt|system:0|arbiter:4’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_receiver.rs:33:43:
called Result::unwrap() on an Err value: JoinError::Panic(Id(2193), “attempt to subtract with overflow”, …)
[2025-05-27T11:37:05Z INFO actix_web::middleware::logger] 127.0.0.1 “GET /91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309/to-autonomi.mp4 HTTP/1.1” 206 0 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36” 0.638724
[2025-05-27T11:37:05Z INFO anttp::service::resolver_service] Found public archive at [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:05Z INFO anttp::controller::file_controller] Retrieving file from public archive [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:05Z INFO anttp::service::archive_helper] Resolved path [to-autonomi.mp4], path_buf [to-autonomi.mp4] to xor address [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3]
[2025-05-27T11:37:05Z INFO anttp::service::file_service] Streaming item [to-autonomi.mp4] at addr [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3], range_from [37224448], range_to [37487986]
[2025-05-27T11:37:05Z INFO autonomi::client::data_types::chunk] Getting chunk: 71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3

thread ‘actix-rt|system:0|arbiter:5’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_fetcher.rs:32:42:
attempt to subtract with overflow

thread ‘actix-rt|system:0|arbiter:5’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_receiver.rs:33:43:
called Result::unwrap() on an Err value: JoinError::Panic(Id(2199), “attempt to subtract with overflow”, …)
[2025-05-27T11:37:05Z INFO actix_web::middleware::logger] 127.0.0.1 “GET /91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309/to-autonomi.mp4 HTTP/1.1” 206 0 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36” 0.040093
[2025-05-27T11:37:06Z INFO anttp::service::resolver_service] Found public archive at [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:06Z INFO anttp::controller::file_controller] Retrieving file from public archive [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:06Z INFO anttp::service::archive_helper] Resolved path [to-autonomi.mp4], path_buf [to-autonomi.mp4] to xor address [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3]
[2025-05-27T11:37:06Z INFO anttp::service::file_service] Streaming item [to-autonomi.mp4] at addr [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3], range_from [37224448], range_to [37487986]
[2025-05-27T11:37:06Z INFO autonomi::client::data_types::chunk] Getting chunk: 71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3

thread ‘actix-rt|system:0|arbiter:6’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_fetcher.rs:32:42:
attempt to subtract with overflow

thread ‘actix-rt|system:0|arbiter:6’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_receiver.rs:33:43:
called Result::unwrap() on an Err value: JoinError::Panic(Id(2207), “attempt to subtract with overflow”, …)
[2025-05-27T11:37:06Z INFO actix_web::middleware::logger] 127.0.0.1 “GET /91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309/to-autonomi.mp4 HTTP/1.1” 206 0 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36” 0.045834
[2025-05-27T11:37:06Z INFO anttp::service::resolver_service] Found public archive at [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:06Z INFO anttp::controller::file_controller] Retrieving file from public archive [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:06Z INFO anttp::service::archive_helper] Resolved path [to-autonomi.mp4], path_buf [to-autonomi.mp4] to xor address [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3]
[2025-05-27T11:37:06Z INFO anttp::service::file_service] Streaming item [to-autonomi.mp4] at addr [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3], range_from [37224448], range_to [37487986]
[2025-05-27T11:37:06Z INFO autonomi::client::data_types::chunk] Getting chunk: 71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3

thread ‘actix-rt|system:0|arbiter:7’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_fetcher.rs:32:42:
attempt to subtract with overflow

thread ‘actix-rt|system:0|arbiter:7’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_receiver.rs:33:43:
called Result::unwrap() on an Err value: JoinError::Panic(Id(2211), “attempt to subtract with overflow”, …)
[2025-05-27T11:37:06Z INFO actix_web::middleware::logger] 127.0.0.1 “GET /91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309/to-autonomi.mp4 HTTP/1.1” 206 0 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36” 0.040814
[2025-05-27T11:37:07Z INFO anttp::service::resolver_service] Found public archive at [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:07Z INFO anttp::controller::file_controller] Retrieving file from public archive [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:07Z INFO anttp::service::archive_helper] Resolved path [to-autonomi.mp4], path_buf [to-autonomi.mp4] to xor address [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3]
[2025-05-27T11:37:07Z INFO anttp::service::file_service] Streaming item [to-autonomi.mp4] at addr [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3], range_from [37224448], range_to [37487986]
[2025-05-27T11:37:07Z INFO autonomi::client::data_types::chunk] Getting chunk: 71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3

thread ‘actix-rt|system:0|arbiter:8’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_fetcher.rs:32:42:
attempt to subtract with overflow

thread ‘actix-rt|system:0|arbiter:8’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_receiver.rs:33:43:
called Result::unwrap() on an Err value: JoinError::Panic(Id(2256), “attempt to subtract with overflow”, …)
[2025-05-27T11:37:07Z INFO actix_web::middleware::logger] 127.0.0.1 “GET /91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309/to-autonomi.mp4 HTTP/1.1” 206 0 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36” 0.061509
[2025-05-27T11:37:08Z INFO anttp::service::resolver_service] Found public archive at [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:08Z INFO anttp::controller::file_controller] Retrieving file from public archive [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:08Z INFO anttp::service::archive_helper] Resolved path [to-autonomi.mp4], path_buf [to-autonomi.mp4] to xor address [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3]
[2025-05-27T11:37:08Z INFO anttp::service::file_service] Streaming item [to-autonomi.mp4] at addr [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3], range_from [37224448], range_to [37487986]
[2025-05-27T11:37:08Z INFO autonomi::client::data_types::chunk] Getting chunk: 71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3

thread ‘actix-rt|system:0|arbiter:9’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_fetcher.rs:32:42:
attempt to subtract with overflow

thread ‘actix-rt|system:0|arbiter:9’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_receiver.rs:33:43:
called Result::unwrap() on an Err value: JoinError::Panic(Id(2266), “attempt to subtract with overflow”, …)
[2025-05-27T11:37:08Z INFO actix_web::middleware::logger] 127.0.0.1 “GET /91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309/to-autonomi.mp4 HTTP/1.1” 206 0 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36” 0.043068
[2025-05-27T11:37:09Z INFO anttp::service::resolver_service] Found public archive at [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:09Z INFO anttp::controller::file_controller] Retrieving file from public archive [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:09Z INFO anttp::service::archive_helper] Resolved path [to-autonomi.mp4], path_buf [to-autonomi.mp4] to xor address [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3]
[2025-05-27T11:37:09Z INFO anttp::service::file_service] Streaming item [to-autonomi.mp4] at addr [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3], range_from [37224448], range_to [37487986]
[2025-05-27T11:37:09Z INFO autonomi::client::data_types::chunk] Getting chunk: 71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3

thread ‘actix-rt|system:0|arbiter:10’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_fetcher.rs:32:42:
attempt to subtract with overflow

thread ‘actix-rt|system:0|arbiter:10’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_receiver.rs:33:43:
called Result::unwrap() on an Err value: JoinError::Panic(Id(2291), “attempt to subtract with overflow”, …)
[2025-05-27T11:37:09Z INFO actix_web::middleware::logger] 127.0.0.1 “GET /91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309/to-autonomi.mp4 HTTP/1.1” 206 0 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36” 0.071105
[2025-05-27T11:37:10Z INFO anttp::service::resolver_service] Found public archive at [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:10Z INFO anttp::controller::file_controller] Retrieving file from public archive [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:10Z INFO anttp::service::archive_helper] Resolved path [to-autonomi.mp4], path_buf [to-autonomi.mp4] to xor address [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3]
[2025-05-27T11:37:10Z INFO anttp::service::file_service] Streaming item [to-autonomi.mp4] at addr [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3], range_from [37224448], range_to [37487986]
[2025-05-27T11:37:10Z INFO autonomi::client::data_types::chunk] Getting chunk: 71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3

thread ‘actix-rt|system:0|arbiter:11’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_fetcher.rs:32:42:
attempt to subtract with overflow

thread ‘actix-rt|system:0|arbiter:11’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_receiver.rs:33:43:
called Result::unwrap() on an Err value: JoinError::Panic(Id(2298), “attempt to subtract with overflow”, …)
[2025-05-27T11:37:10Z INFO actix_web::middleware::logger] 127.0.0.1 “GET /91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309/to-autonomi.mp4 HTTP/1.1” 206 0 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36” 0.091297
[2025-05-27T11:37:11Z INFO anttp::service::resolver_service] Found public archive at [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:11Z INFO anttp::controller::file_controller] Retrieving file from public archive [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:11Z INFO anttp::service::archive_helper] Resolved path [to-autonomi.mp4], path_buf [to-autonomi.mp4] to xor address [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3]
[2025-05-27T11:37:11Z INFO anttp::service::file_service] Streaming item [to-autonomi.mp4] at addr [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3], range_from [37224448], range_to [37487986]
[2025-05-27T11:37:11Z INFO autonomi::client::data_types::chunk] Getting chunk: 71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3

thread ‘actix-rt|system:0|arbiter:12’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_fetcher.rs:32:42:
attempt to subtract with overflow

thread ‘actix-rt|system:0|arbiter:12’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_receiver.rs:33:43:
called Result::unwrap() on an Err value: JoinError::Panic(Id(2304), “attempt to subtract with overflow”, …)
[2025-05-27T11:37:11Z INFO actix_web::middleware::logger] 127.0.0.1 “GET /91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309/to-autonomi.mp4 HTTP/1.1” 206 0 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36” 0.050324
[2025-05-27T11:37:12Z INFO anttp::service::resolver_service] Found public archive at [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:12Z INFO anttp::controller::file_controller] Retrieving file from public archive [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:12Z INFO anttp::service::archive_helper] Resolved path [to-autonomi.mp4], path_buf [to-autonomi.mp4] to xor address [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3]
[2025-05-27T11:37:12Z INFO anttp::service::file_service] Streaming item [to-autonomi.mp4] at addr [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3], range_from [37224448], range_to [37487986]
[2025-05-27T11:37:12Z INFO autonomi::client::data_types::chunk] Getting chunk: 71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3

thread ‘actix-rt|system:0|arbiter:13’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_fetcher.rs:32:42:
attempt to subtract with overflow

thread ‘actix-rt|system:0|arbiter:13’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_receiver.rs:33:43:
called Result::unwrap() on an Err value: JoinError::Panic(Id(2308), “attempt to subtract with overflow”, …)
[2025-05-27T11:37:12Z INFO actix_web::middleware::logger] 127.0.0.1 “GET /91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309/to-autonomi.mp4 HTTP/1.1” 206 0 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36” 0.062962
[2025-05-27T11:37:13Z INFO anttp::service::resolver_service] Found public archive at [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:13Z INFO anttp::controller::file_controller] Retrieving file from public archive [91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309]
[2025-05-27T11:37:13Z INFO anttp::service::archive_helper] Resolved path [to-autonomi.mp4], path_buf [to-autonomi.mp4] to xor address [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3]
[2025-05-27T11:37:13Z INFO anttp::service::file_service] Streaming item [to-autonomi.mp4] at addr [71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3], range_from [37224448], range_to [37487986]
[2025-05-27T11:37:13Z INFO autonomi::client::data_types::chunk] Getting chunk: 71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3

thread ‘actix-rt|system:0|arbiter:14’ panicked at C:\Users\xxx.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\chunk-streamer-0.1.0\src\chunk_fetcher.rs:32:42:
attempt to subtract with overflow

Traktion · May 27, 2025, 1:15pm

Thanks - that’s a huge help!

The range values actually look ok, so something else funky is going on. I’ll do some tests and see if I can reproduce it later today.

[71723a4dddf484542c4a487a7089aaa6c30f3fec21c55a692996b6cc29a30fa3], range_from [37224448], range_to [37487986]

Traktion · May 27, 2025, 6:45pm

So, it looks like a cast from a u64 to a usize may be the cause. I can’t reproduce easily right now, but I’ve added safety to see if it fixes the issue. It will now round down to u32::MAX, rather than truncating to zero, which then could potentially be negated to -1 (which is an overflow for a unsigned type).

It’s a bit strange though, as Win64 should have 64-bit usizes and there shouldn’t be any truncation. However, maybe there is something funky going on with the build target, which is treating it the target as 32-bit.

Anyway, I’ve pushed a new v0.6.1 release with the above in. Please let me know if it helps!

Github: Release v0.6.1 · traktion/AntTP · GitHub
Dockerhub: https://hub.docker.com/r/traktion/anttp
Cargo: cargo install anttp

Herodotos · May 28, 2025, 9:36am

So, I tried your fix in version 0.6.1, it failed. The same error.

So I looked at the code it seems the error is in chunk_streamer library. In the chunk_fetcher.rs you do something like this:

let total_len = (position_end_usize - position_start_usize) + 1;
let derived_chunk_size = total_len.min(stream_chunk_size) - chunk_start_offset;

If you ever request a range that begins inside a chunk but is shorter than the offset into that chunk, you end up subtracting a larger number (chunk_start_offset) from a smaller one (total_len.min(...)), which on unsigned integers wraps around and in debug builds panics with “attempt to subtract with overflow.”

Concretely, imagine you ask for bytes 800–810 of a 4 096 -byte chunk:

total_len = (810 − 800) + 1 = 11
chunk_start_offset = 800
then derived_chunk_size = 11 − 800 → underflow → panic

Therefore I suggest this fix:

let total_requested = end.checked_sub(start).map(|d| d + 1).unwrap_or(0);
// total_requested = 11

let avail_in_chunk = stream_chunk_size.saturating_sub(chunk_start_offset);
// avail_in_chunk = 4096 - 800 = 3296

let derived_chunk_size = total_requested.min(avail_in_chunk);
// derived_chunk_size = min(11, 3296) = 11

What I find out it is not Windows related bug. I was testing it in debug mode.
However building in release mode disables Rust’s debug‐mode overflow checks, so that particular underflow won’t panic. Instead it will wrap around to a huge usize value, and code will continue running. That’s why on Linux you didn’t see a panic, you were almost certainly running the release binary there, so the subtraction silently wrapped instead of trapping.

I implemented those changes that I suggested and few minor tweaks. And build it from local repository and it works. So I was able to see working stream, even in Debug mode. I tested it on Chrome/Firefox/Edge..

Here is the final file:

use autonomi::{ChunkAddress, Client};
use bytes::{Bytes};
use log::{info};
use self_encryption::{DataMap, EncryptedChunk, Error};

#[derive(Clone)]
pub struct ChunkFetcher {
    autonomi_client: Client,
}

impl ChunkFetcher {
    pub fn new(autonomi_client: Client) -> Self {
        ChunkFetcher { autonomi_client }
    }

    pub async fn fetch_from_data_map_chunk(
        &self,
        data_map: DataMap,
        position_start: u64,
        position_end: u64
    ) -> Result<Bytes, Error> {
        info!("fetch from data map chunk");

        // note: range queries can be u64, but autonomi only expects usize, so safely convert
  	let position_start_usize = usize::try_from(position_start).ok().unwrap_or_else(|| usize::MAX);
        let position_end_usize = usize::try_from(position_end).ok().unwrap_or_else(|| usize::MAX);        
        
        let stream_chunk_size = data_map.infos().get(0).unwrap().src_size;
        let chunk_position = position_start_usize / stream_chunk_size; // chunk_info.src_size needed for exact size, as last chunk size varies
        let chunk_start_offset = position_start_usize % stream_chunk_size;

        info!("decrypt chunk in position=[{}] of [{}], offset=[{}], total_size=[{}]", chunk_position+1, data_map.infos().len(), chunk_start_offset, data_map.file_size());
        match data_map.infos().get(chunk_position) {
            Some(chunk_info) => {
                info!("get chunk from data map with hash {:?} and size {}", chunk_info.dst_hash, chunk_info.src_size);

                // Safely derive how many bytes to decrypt from this chunk
                 let derived_chunk_size = {
                    // total bytes requested in this range
                    let total_requested = position_end_usize
                        .checked_sub(position_start_usize)
                        .map(|d| d + 1)
                        .unwrap_or(0);
                    // bytes available in this chunk after the start offset
                    let avail_in_chunk = chunk_info.src_size.saturating_sub(chunk_start_offset);
                    // use the smaller of requested vs available
                    total_requested.min(avail_in_chunk)
                };

                let chunk = self.autonomi_client.chunk_get(&ChunkAddress::new(chunk_info.dst_hash)).await.expect("get chunk failed");

                info!("self decrypt chunk: {:?}", chunk_info.dst_hash);
                let encrypted_chunks = &[EncryptedChunk { index: chunk_position, content: chunk.value.clone()}];

                // Decrypt only the requested byte range of the chunk,
		//Note: Original is using match, which is literally just forwarding the Ok or Err from decrypt_range without doing anything extra. The expression decrypt_range already has the type Result<Bytes, Error>, exactly matching function’s return type. 
                self_encryption::decrypt_range(&data_map, encrypted_chunks, chunk_start_offset, derived_chunk_size)
            }
            None => Ok(Bytes::new()),
        }
    }
}

Traktion · May 28, 2025, 10:45am

Thanks! I did try running it in dev mode, as I thought that was supposed to flag these errors, but it still silently succeeded for me (with the problem range/file). I’ll give it a whirl in debug mode, now I know that is how the error manifests. I just couldn’t see the problem to fix it and it seemed to be silently working for me.

EDIT: I just realised I was building ‘dev’, but still running ‘release’, as I was in the wrong directory! I can reproduce the error now!

I should really add some unit tests around this and the main AntTP code too. I’ll ensure I run in debug mode for testing too (sounds like a good best practice).

Thanks for the detailed feedback.

BTW, out of curiosity, did it run ok for you in a release build before (despite the silent overflow)?

Herodotos · May 28, 2025, 11:41am

I tested both version 0.6.0 and 0.6.1 and they do work in release mode.

While testing your original implementation in release mode I see in logs, that it downloads way too many chunks when it starts streaming, I would expect 1-2 chunks to be enough to start playing. It looks like it tries to download whole file when it is at starting position. When I click random positions in the video, it simply download rest of the file from that position.

Than I tried my fixed version in release mode, and it does the same. Downloads whole file form position 0, and always downloads rest of the file when you click some position. So I think, there is another bug, or bad design or something causing this. As far as I understand streaming, I would expect to return just a portion of the file, and the client will request next part if required. Loading while file at once is not required.

I also did a test download via ‘curl’ command in release mode, and compared to debug mode, and the autonomi.mp4 file downloaded in test mode in on average over 30 seconds, in release mode it took 11-12 seconds.

But for the big Ubuntu ISO file, download in release mode did not improve much. Debug mode: 32 minuts, Release mode slightly better. Based on what I see there, I think there is some problem, it should download much faster. It looks like it is not reusing those 8 download threads correctly, I would expect all 8 threads to work all the time. .. But It is just empirical guess. Just simple math, autonomi.mp4 35MB in 11 seconds => more than 3MB/sec. But ISO file has 1MB or less per second.

Traktion · May 28, 2025, 12:25pm

Herodotos:

I tested both version 0.6.0 and 0.6.1 and they do work in release mode.

While testing your original implementation in release mode I see in logs, that it downloads way too many chunks when it starts streaming, I would expect 1-2 chunks to be enough to start playing. It looks like it tries to download whole file when it is at starting position. When I click random positions in the video, it simply download rest of the file from that position.

Than I tried my fixed version in release mode, and it does the same. Downloads whole file form position 0, and always downloads rest of the file when you click some position. So I think, there is another bug, or bad design or something causing this. As far as I understand streaming, I would expect to return just a portion of the file, and the client will request next part if required. Loading while file at once is not required.

Thanks for digging more into this!

Iirc, it does depend on the browser, the video player and how you open the file (whether inline or in its own tab).

In many cases, the client seems to assume the viewer will want the whole video, so just starts buffering the entire thing, up until it decides it has enough. Then the client seems to block, until more buffer is needed. Ive seen this happen in VLC especially, where the data rate drops to dribble, but keeps streaming smoothly.

In the above scenario, it manifests as what looks like slow channels through chunk streamer. However, I’m fairly certain this is some sort of back pressure from the client.

As you noticed, skipping ahead repeats the above, but from a new start position. It just assumes you want to watch the rest of the video, but may provide back pressure when the buffer is large enough. Again, VLC is most aggressive/obvious, but I believe browser players do similar at some point.

I suspect this is essentially behaviour inherited from when there is no range support. Perhaps it is more efficient too?

Ofc, there is nothing stopping the client iterating through multiple range requests, but it doesn’t seem to be default client behaviour.

Perhaps there is room for tuning with better clients though, etc?

Edit: Also, I’ve seen times where the browser asks for all, then drops the connection when it has enough, then asks for data in another range.

Thanks for confirming release worked too, as I thought I was losing my mind! Ha! I’m still learning Rust nuances and it’s good to understand this one!

It is strange that there is such a large difference. It could be that the connection is struggling on the larger file, which can result in throttling. I’ve seen quinn errors before on bigger files, where it has been more aggressive than the connection can handle. Reducing the download threads can actually improve throughput where this is the case (ironically!).

Also, another consideration is that threads will block to maintain ordering. So, if thread 1 is waiting on data, threads 2-8 can fill and block until 1 is ready, then the whole lot gets sent to the client (in order). Likewise, thread 7 may be slow, which is only blocking thread 8 (if it has already completed).

Over a larger data set, it usually just evens out, but if the odd thread is getting quinn errors, it could have a knock on effect of blocking the other threads.

Ofc, there may be optimisations around that side too. It’s a relatively naive implementation right now, although it usually does pretty well. The odd outlier may clog things up a bit, but it averages out pretty well (and much better than 1 thread).

Thanks again for taking the time to test this. I feel AntTP will become an important tool and the more eyes on improving it, the better!

Southside · May 28, 2025, 12:38pm

Seconded most heartily…

When you have time, could you give us some specific examples of what you would like tested and which parameters we should set to give you the best feedback please?

Traktion · May 28, 2025, 12:43pm

Thanks and will do!

So far, I’ve added the performance testing scripts, where more suits could be added to: AntTP/test/performance at master · traktion/AntTP · GitHub

Ideally, we could do with a suite of HTTP calls to the various endpoints, including the new REST ones. I use some local tests in my IDE, but want to share these (although I’m not sure if they are easy to run outside the IDE). Something like a Postman collection would be handy (with Newman to run them through CLI), but I’d rather use an open source tool (or one that doesn’t badger so much!).

In short, something to black box test for regressions would be very helpful.

Traktion · May 28, 2025, 4:33pm

I’ve patched chunk-streamer with the changes above and linked it into AntTP.

It seems to run well in dev and release profiles now. Big thanks @Herodotos !

Github: Release v0.6.2 · traktion/AntTP · GitHub
Dockerhub: https://hub.docker.com/r/traktion/anttp
Cargo: cargo install anttp

Herodotos · May 29, 2025, 1:00pm

I was trying to find out why streaming always returns the whole file, and it looks like the main problem is in the mp4 file itself. It has to have ‘moov atom’ at the front of the file. The available video I work with autonomi.mp4 has it at the end of the file.

We need a correct video, to test the functionality. I do not have ANT/ETH to play with uploads at the moment. It would be nice if someone uploads such test video, ideally much larger than current one. But even current one can be used, if someone runs conversion and re uploads it.

it can be converted like this:

ffmpeg -i input_file.mp4 -c copy -movflags +faststart output_file.mp4

ffmpeg can be installed like

sudo apt update
sudo apt install ffmpeg

I was testing this video:

91d16e58e9164bccd29a8fd8d25218a61d8253b51c26119791b2633ff4f6b309/to-autonomi.mp4

loziniak · May 29, 2025, 1:03pm

You can give your Arbitrum Sepolia address to @rusty.spork and he will send you ETH+ANT for alphanet.

Herodotos · May 29, 2025, 1:14pm

I still prefer to do only read only staff. I am not interested in doing crypto related tax fillings based on my few cents crypto payments. I wish there were non crypto way to do uploads… Like earning a non transferable upload receipts, that by nature are not crypto, since they can’t be sold or moved, only spend.

loziniak · May 29, 2025, 1:32pm

Sepolia is a testnet, so I don’t think you have to tax anything, these tokens are generally worthless.

Topic		Replies	Views
Community Public File Directory Community	85	1367	May 2, 2026
Ant upload: an app to share files to the world Apps	50	994	October 3, 2025
Update 18th September, 2025 Updates	52	971	September 22, 2025
:IF: Announcing - MutAnt: Mutable Key/Value storage (prev: Anthill) Apps impossible-futures-1	86	1140	September 3, 2025
How to publish fully function Autonomi Web browser launcher anonymously? Apps	43	433	July 11, 2025

AntTP - Serving Autonomi data over HTTP

Related topics