Current Limitations? Estimating ETH costs to upload files? Proper way to find answers?

Support
,
21

As far as I understand correctly, there are 2 types of earnings for nodes. One are direct payments for data uploads and the other one is 250 million ANT pool distributed over 50 years. I see, that the second rewarding system works well, there are not many bottlenecks and right now it covers more than 99.999% of rewards in absolute ANT terms. The first system, direct payments was designed for native currency and is not very limiting because of high fees.

Pardon me, I do not have very deep understanding of all the nuances. But would be the problem of high fees solved, if uploader did not pay directly to so many various node owners, but rather to the big reward pool, that is used for rewards? Or even better, another daily pool, where all uploders pay their part for uploaded files in a single transaction, and which is than used by the existing randomized rewards? So basically the random rewards transaction count would not change, but would deliver slightly higher rewards(original + what uploaders paid that day). This would speed up uploads. At least it could work until native currency is developed.

Yes, I understand, that this requires some prepaid receipt for uploads implementation. But since ownership of that receipt can’t be transferred it could be easier to implement than a native currency.

22

Hey there, welcome!

I may be wrong but I think that the ANT pool is just being distributed through PayForQuotes (same smart contract function that uploaders use to pay for storage). At least, that’s how I’ve always seen ANT come to me from running nodes (before and after Autonomi’s launch/TGE). I think it could be that they upload whatever random data, which (due to self-encryption) makes every node have an equal probability of being the one that’s closest to the address of chunks that are to be stored.

I’ve thought of similar approaches. A couple of concerns I have about your idea of one big pool is centralized management, the node not knowing whether the uploader is really paying their fair share for the particular upload, and also ensuring that nodes are getting rewarded for the time they’re online (and are not cheating/spoofing). Also currently, AFAIK there is no list of nodes.

One payments approach (which would require modifications to Autonomi’s source code) could be something akin to Bitcoin’s Lightning network. Rather than directly paying a node, there are trusted (staked) intermediaries that conduct lots of small transactions off-chain and settle periodically. Kind of like traditional finance, where there are banks (e.g. a buyer instructs his bank to transfer money from his account to a seller’s account at another bank, and the banks periodically settle their batches of transactions). There are other ideas being discussed in various threads (such as Autonomi Community Token Proposal).

Autonomi documentation mentions plans to have a “Receipt” form of payment, but currently details are scarse.

23

Any transfer has to be paid for by someone. So the nodes you upload to incur a transfer fee. Be it from a pool or the uploader.

Transactions can have up to 250 transfers which help reduce processing fees by the ARB-node

The whole concept of uploader paying the node being uploaded to is compensation, some might cal it paying, for the data being uploaded. To put it into a random pay pool removes the concept the network is using.

And yes until Native token exists there will be slow uploading

24

It is good that some visibility is coming to this issue, though my concern is much less about the impact on quoting/payment for storage although that is a big drag on performance.

My concern is about the enormous barrier this creates to adoption, and we have little attention given to that and no sense of whether that can be mitigated. We have speculation that native token will arrive in a few years and so far, not much more than wishful thinking about that solving the issue.

25

I’m not 100% and I would need to do some more research. But if I remember correct, the recent Pectra upgrade on ETH mainnet would allow for arbitrum one to adopt blobs and another new protocol that would allow a) for somewhere between 90 and 99% fee reduction and b) allows the user to not have ETH on their wallet and pay with (in our case) ANT for gas fees directly. Arbitrum one has stated to be upgrading as well, just no timeline yet. Does anyone know if my knowledge is up to date? That would help both the adoption and the blockchain cost issues.

26

My main concern though is the need to buy crypto. The plan was for anyone to spin up some nodes and off you go.

With crypto as is, our audience is a tiny fraction of “everyone” due to the barrier that creates.

27

Users wouldn’t need to buy crypto if anyone could earn ANT ERC20 from running nodes, and pay for storage + transactions with ANT ERC20 instead of transactions requiring Arbitrum ETH.

If that becomes possible, there will still be a massive ETH ‘tax’ on Autonomi services, but the user wouldn’t need to engage with the ETH side of it knowingly. It’d be a much smoother user experience for the non-crypto savvy.

To buy ANT on exchanges through would still require crypto, though that will probably be true with Native ANT as well.

28

I believe that long term, the need to buy crypto can go away if people are able to actually earn meaningful amounts of ANT by hosting nodes on their own computer hardware.

ERC-4337 enables paymaster contracts - paying gas with tokens. So if someone doesn’t want to pay a penny for crypto but are willing to install Autonomi on their computer, they can in theory earn ANT and later spend it.

29

yes, but when someone is uploading and paying for upload, he basically pays for 4MB block to each node. This means very tiny payment. On the other hand the rewarding algorithm is paying much higher ANT amounts. Since there is already this function, that rewards randomly nodes, it can simply be adjusted to reward each one slightly more by using that daily reward pool. This would drastically reduce number of transactions. I as uploader could prepay for 1 day let say 20 GB. And pay only single transaction to the daily pool. The random rewards mechanism would still use same number of transactions, but the amount they pay would be proportionally higher using that daily rewards pool in addition to existing emission.
Right now the upload is the bottleneck and in reality until we have a native currency random rewards will be still likely more than 90% of all rewards. So what is the point of using payment scheme that does not work well, does not reward nodes well and make main obstacle for adoption? What I suggest would instantly allow rapid, cheap uploading and everyone would benefit. The network with its current random rewards is centralized anyway. Does not matter much whether random rewards are 100% or 99% of income for node operators. Until there is a native currency, removing direct payment to nodes would solve the bottleneck that makes no good only harm at current environment.

30

I agree, that the lack of option to earn a currency for common user is a problem. The fact, that he requires ETH together with ANT is a big adoption problem. But maybe he could earn receipts instead of a currency. You don’t seem to worry about the fact, that user can’t move his coins to someone else. So what we need is a user earning a right for upload. Some prepaid receipt that can be used nativelly.
My suggestion was to remove direct payments, keep random rewards. introduce some receipt with upload rights. Than upload costs with ETH will not be a problem. And small node operators can be rewarded with ANT or if they wish with those prepaid receipt for uploads.

I do understand, that the “receipt” is a form of an internal currency, but does not require transfer of ownership and can’t be sold. So could be easier to implement and what is more important receipt is not a crypto, so it likely would not cause users problems with crypto tax laws.

31

Interesting. Figured I’d take a couple minutes to check into it…

ChatGPT o3’s says:

Analysts model an additional 30-40 % fee drop for L2s such as Arbitrum, not another 90-99 %

and in its “Bottom line”:

The post is half-remembering two different upgrades. Dencun/Atlas (2024) delivered the headline 10× fee cut, while Pectra (2025) layers on smaller but still meaningful savings and finally standardises wallet-level gas abstraction. Users will still need ETH under the hood unless a paymaster or a custom-gas Orbit chain is in place, but front-end dApps can soon make that invisible.

32

This is one of the problems I tried to tackle when designing Ark. I’m trying to make Autonomi an attractive option for people from outside the crypto sphere. Arks design gives the user the option to delegate / outsource the uploading part if they so choose - while still retaining full control.

One scenario where this approach shines is with business users: dealing with crypto payments is a no-go for many, so they could use the service of a middleman (I know, a bad word in here :sweat_smile:) and get a regular invoice that fulfills all the legal requirements of their jurisdiction (e.g. things like VAT) and pay it like any other supplier using their existing workflows. It’s one way to make Autonomi more “compatible” with the wider world.

I know this is slightly off-topic, but I just want to add this perspective to demonstrate a pragmatic approach - it’s certainly a compromise but it works today.

33

I tried again with a --max-gas-fee. Despite paying (again for the same chunks) it looks like I didn’t succeed in getting a single chunk uploaded.

You’re right - retrying costs. I ended up paying for 1034 chunks again. Unfortunately, all chunks failed to upload. I think it takes about a minute or so for each “A network error occurred.” line.

At the end, ant reported “Upload of 1 files completed in 390080.351990605s”. A bit misleading since the upload didn’t really work, but the duration seems right - this represents just over 4.5 days of the program running, until it eventually exited.

Here is some of the output (I omited a lot from the middle, since the output is over 4,100 lines):

$ ant file upload --public --max-fee-per-gas=20000000 ~/Downloads/kali-linux-2024.3-installer-amd64.iso
Logging to directory: "/home/sbosshardt/.local/share/autonomi/client/logs/log_2025-05-19_15-12-10"
🔗 Connected to the Network                                                                                          Uploading data to network...
Encrypting file: "/home/sbosshardt/Downloads/kali-linux-2024.3-installer-amd64.iso"..
Successfully encrypted file: "/home/sbosshardt/Downloads/kali-linux-2024.3-installer-amd64.iso"
Quoting for 1034 chunks..
Paying for 1034 chunks..
Chunk payments of 1034 chunks completed. 0 chunks were free / already paid for
Uploading file: /home/sbosshardt/Downloads/kali-linux-2024.3-installer-amd64.iso (1034 chunks)..
(1/1034) Chunk failed to be stored at: a74bd3fb833e9ee36c309889079b7f189026eae5386dc7a325c6e2362cbaaaa6 (A network error occurred.)
(2/1034) Chunk failed to be stored at: 096f2341670ebc036424adb40b18376cf358ab7ceb72598e80b914d8299ba0e7 (A network error occurred.)
(3/1034) Chunk failed to be stored at: 9616eab078ab376293d6a063bba39e1f7a4405c9abf3b017a26eb96e3a434c34 (A network error occurred.)
(4/1034) Chunk failed to be stored at: 8f03f001cd794f72fa65ae625e1b7ce9d947cf2ec57fe2ce92e4f074e89d5998 (A network error occurred.)
(5/1034) Chunk failed to be stored at: cfd94d816268451c58052991baa97b510d9893f375071273b02702fdb6a34e34 (A network error occurred.)
(6/1034) Chunk failed to be stored at: 3ffac67ecb9009d1a62967016336546e98bdbbd3e707198fc3c9876a8acea7d8 (A network error occurred.)
[...]
(Loops around a few times)
[...]
(1033/1034) Chunk failed to be stored at: 6aa2bed6e0eead31449d7bce1eb395f53a1b948e43bfcad00d0ec99f6434e7f9 (A network error occurred.)
(1034/1034) Chunk failed to be stored at: 661922aa2941f6fc1a900ac030556c06c91530ae7002aa934c179589e995fc6e (A network error occurred.)
Upload of 1 files completed in 390080.351990605s
Error uploading file /home/sbosshardt/Downloads/kali-linux-2024.3-installer-amd64.iso: PutError(Network(FailedToVerifyChunkProof(NetworkAddress::RecordKey("661922aa2941f6fc1a900ac030556c06c91530ae7002aa934c179589e995fc6e") - (738c3ca37b72cf5757be21025ffa03f75cef61f99419b4dae996b60ea5a5265a))))

   0: Failed to upload file
   1: Failed to upload file
   2: A network error occurred.
   3: Failed to verify the ChunkProof with the provided quorum

Location:
   ant-cli/src/commands/file.rs:86

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
sbosshardt@sam-desktop ~/Programming/autonomi $ ## The below "date" command runs immediately after "ant file upload" exits.

sbosshardt@sam-desktop ~/Programming/autonomi $ date
Sat May 24 03:33:36 AM PDT 2025

I see that Autonomi devs have looked at this post and made some changes based on the feedback - I appreciate it.

Now that ant exited (after 4.5 days), I’ve uploaded the “logs” folder it made during its run. It’s 242 MB on my computer, and that’s with two thirds of the log files (automatically) .gz compresed! In the folder I’ve also included a notes file I made that has the full output of the command (“2025-05-19 uploading 4 gb linux iso.txt”). A zip file is available here in case anyone wants it:
https://www.sbosshardt.com/test/autonomi/log_2025-05-19_15-12-10.zip

I did try using ant to upload those logs, but the Arbitrum gas fees went up significantly and caused the upload to fail after nearly an hour. At this stage of ant upload, I think there is quoting & payment to upload the datamap - after all the underlying chunks of the files are been saved. I’m guessing that if the datamap were to be paid up front, it might improve the odds of success (less time for gas fees to skyrocket).

It might be good if there were an argument to indicate to ant upload that you want it to not throw an exception when gas fees are too high - just wait a while and keep trying until the fees come down low enough to continue. Or similarly, have an argument to make it timeout after so long (and have the default behavior to patiently wait for as long as it takes and retry).

I uploaded (to a website) a zip file of the logs so I don’t have to keep checking Arbitrum’s gas price to come down.

In hindsight, if I had zipped the folder first, it might have succeeded (42MB instead of 242 MB). Here are logs of my failed attempt uploading the (above) logs - also including a notes file that has the command’s output:
https://www.sbosshardt.com/test/autonomi/log_2025-05-25_00-03-31.zip

34

If you publish a directory with dweb it will retry any failures forever at the moment. It has an argument to set the number of retries, but I set the default to unlimited for the time being.

cargo install dweb-cli --locked
dweb --help

The output includes more info than that from ant-cli at the moment so you should have everything you need.

35

Sounds good, I think I’ll try dweb for uploads.

Fingers crossed that it’ll let me upload that logs folder without trying to pay for & upload chunks that I’ve already uploaded.

36

Use filebin to upload logs that the devs might want to see

37

Re-payment is in the hands of the Autonomi API I’m afraid, so until they merge the latest PRs you will be re-paying.