Ah maybe not, it failed on the 0.3.8 as well
Trivial files that are already uploaded via both MissionCtrl and DWEB
My upload script is seeing 1 in every 2 attempts being successful. Actually a little better than the last couple of days which were 1 in 3 or 1 in 4.
using 0.3.8
3 Likes
Is there is anyone willing to use some ARB-ETH (for gas) to get some large chunks onto the network?
Here is the current version of upload-exercise.bash
upload-exercise.zip (4.4 KB)
Usage: ./upload-exercise.bash --rate rate --timeout timeout --fee maxGWEI --parallel n
Where
--rate | -r rate is the seconds between starts of each upload attempt
--timeout | -t timeout is the max seconds for attempting to upload the file
--fee | -f maxGWEI is the maximum GWEI per GAS
For maxGWEI of 0.1 (default) 4 chunks is approx
a maximum of 0.0001 ARB-ETH
--parallel | -p num is the maximum parallel uploads in progress
It is best to create a directory to unzip it into and run it from there.
mkdir upload-testing
cd upload-testing
Most systems will require the script to be given execute rights for the user
chmod 755 upload-exercise.bash
then run it. You will see the attempt number and some other info while running.
The success rate is not 100% and if you set the timeout too low it will never succeed. suggested values for most simple system is
-r 60
-f 0.015
-p 6
-t 900 OR -t 1200
Those options will set
- (-p 6) max of 6 uploads running in parallel
- (-r 60) attempt a new upload every 60 seconds, will skip if already at max parallel uploads happening
- (-t 900) 900 seconds maximum to allow upload to run. This should get the file up and a datamap address if it was going to succeed anyhow. But not enough time for the archive chunk which can take up to 2 hours
- (-f 0.015) is 0.015 GWEI max per gas which is 15000000 x 10^-18 ETH
- for the 3 chunk file results in around a max of 0.000015 ARB-ETH per upload. This is not exact and could be 25% higher or lower maximum.
- typically at this time the actual fee is less
- the upload will not be attempted or paid for if its cost is higher than the max
The datamap addresses are written to a file called ādatamap.listā and if any archive record get written then they are written to a file called ādataarchive.listā
I would ask if anyone does run this script that they zip up those 2 list files and post here so I can combine then to produce a list file for the download-exercise.bash script to get nodes coughing up chunks
2 Likes
Here is an update download list to use for the download-exercise.bash script. There are around 2000 datamap addresses to attempt a file download from. The files in general are 10MiB in size and quite random, except the few from the topic for people to put their uploaded files into
datamap.zip (77.8 KB)
See this post for the download script details Testing - #45 by neo
2 Likes
Update to datamap, now with over 5000 files. More to come with someone else doing some uploading
datamap.zip (200.9 KB)
Please consider running the download-exercise.bash script to make most node setups over 50 nodes do some work and exercise them. Thank you in advance.
4 Likes
Im glad someone is able to upload.
Iām failing
willie@gagarin:~/Music$ time ant file upload -p 'The Beat Farmers - Powder Finger-.mp3'
Logging to directory: "/home/willie/.local/share/autonomi/client/logs/log_2025-03-16_23-35-01"
š Connected to the Network Uploading data to network...
Encrypting file: "The Beat Farmers - Powder Finger-.mp3"..
Successfully encrypted file: "The Beat Farmers - Powder Finger-.mp3"
Paying for 4 chunks..
Uploading file: The Beat Farmers - Powder Finger-.mp3 (4 chunks)..
Upload of 1 files completed in 5126.458528586s
Error uploading file The Beat Farmers - Powder Finger-.mp3: PutError(Network(FailedToVerifyChunkProof(NetworkAddress::RecordKey("6207e204610bf2dc2ba5b84240ed6c8fda35a7277eb8eb03535ce9beb7b0dfb3" - b6592a0fe6587cb5b879a21b9a38d9b06867518d4c59cc0483734bf079958a14))))
Error:
0: Failed to upload file
1: Failed to upload file
2: A network error occurred.
3: Failed to verify the ChunkProof with the provided quorum
Location:
ant-cli/src/commands/file.rs:77
Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
real 85m26.940s
user 1m10.361s
sys 0m39.305s
1 Like
Which version? @aatonnomicc couldnāt with 0.3.9 but had success with 0.3.8 and when he has finished there will be even more files
These files are only 10MiB - I also could not get much success with a 500MiB file I tried
willie@gagarin:~/Music$ ant --version
Autonomi Client v0.3.8
Network version: ant/1.0/1
Package version: 2025.1.2.6
Git info: stable / c7c15e9 / 2025-02-27
I downgraded a few days back. Hasnāt made much difference.
We have to see a fix next week or there is a real problem.
2 Likes
Come one come all, lets get this network singing and dancing. Well at least geting the nodes to do some work instead of just sitting idle
I have updated the download script, its more exciting to watch while it downloads 
Now you can set the timeout and max parallel downloads. A lot kinder to your PC
The data map and the script. unzip them into the directory you created to run them from.
Usage: ./download-exercise.bash --rate rate --timeout timeout --parallel n
--rate | -r rate is the seconds between starts of each download attempt
--timeout | -t timeout is the max seconds for attempting to download the file
--parallel | -p num is the maximum parallel downloads in progress
datamap.zip (200.9 KB)
download-exercise.zip (3.0 KB)
5 Likes
Out of curiosity and to answer a niggle I have @neo Are you seeing downloading to be repeatable and consistent (i.e. always correct) and lastly any ideas on speed? (compared with a direct file download from the internet)
3 Likes
The script now leaves the capture of the output of the download in the currentdir/tmp directory and that can be queried
The files it is downloading had the -x flag set during uploading so it is possible that the occasional file will have had a missing chunk. But the upload did complete so its a good chance the file was all uploaded.
I am running the script on my PC with 1000 nodes running as well and it is downloading (in parallel) at the rate of one new file every 5 seconds and never has more than 25 downloads running in parallel. The files are 10MiB so 3 near 4MiB chunks and one datamap
4 Likes
One thing I would love to know is how many copies of a chunk we get each time. So if we are ask 5 nodes for the chunk I hope we can see 5 copies coming back as that number of copies should give us some visibility into the network structure being correct. i.e. we ask for 5 and we think we have the 5 closest, but do we? If they all return a chunk, itās likely we do and not only think we do, if that makes sense.
This is you becoming a core network developer analsis expert though 
I hope you donāt mind
1 Like
When i was doing the quote testing and also my little custom node ping script (using ant file cost) the nodes returning the quotes were not the closest but just fairly close.
I can say this because my ping a node uses the massive database I have of small file ā SE ā 3 chunks and that way I can find a chunk that is ultra close to the node I want to ping and when I do a quote with that small file it ends up getting quotes from nodes around the chunkās XOR address but often they are further away than the node I want to ping.
On early beta nets this worked perfectly. But with millions of nodes and many updates to the code it is not longer the closest nodes, just fairly close. So the ping a node is run on say the starlink to nodes on my NBN so there is no issues with hairpin routing not done right.
Said all that to say that I doubt the client is now uploading to 3 of the 5 closest nodes, but ones in the close neighbourhood and replication will eventually get the records to the closest 5
How do I get to see all the chunks being returned? The ant file download just gives me the files. Is that info in the logs? If so do you know the log entry I need to be looking for? Because I can certainly do that and give you a report for 5550 files that are in my list. If I can do it for one I can do it for 5000
I have done a fair amount of data mining on company data to help populate new data bases from incompatible ones they no longer can access with applications. Including electrical network infrastructure from their remote sensor protocols etc
1 Like
I am not sure @neo I need to ask the team and see if they know.
4 Likes
In the client log file, you can looking for the keyword of Record returned, which shall show you all fetched chunks.
btw. for chunk record, there is only one copy expected to be returned.
hence, as long as the first node enroute responded, the query will stop.
The client will try to upload to closest 5.
However, with the over 30M nodes, it may not have that accurate knowledge (especially with the current 1.2.11, that having a shorter time_out, which we are working on it).
Hence, the client might consider some close enough nodes as closest and upload to them.
5 Likes
Yea this is an area we need to measure accurately and make sure we get as close to 100% accuracy as possible, taking of course churn into account as the thing that we know will happen.
5 Likes
When you get the test method settled, can you share the necessary scripts so we can test from many different setups please?
2 Likes
Thanks for that
Yes this is lines up with my experience trying to ping a node. Also it is the case most of the time in my brief experience trying to ping nodes.
@qi_ma Just this morning, we had a power cut and I had a chance to run my downloading script without nodes running and saw something I thought was quite strange.
The script is in parallel running multiple downloads (up to 50 in this case this morning) And the downloads are started in sequence, not all at once and a new one started within 5 seconds when one completes.
The upload data rate was 50% higher than the download rate. I thought is was very strange since the script is only running ant file download The rates were like 35Mbps up and 20Mbps down. This upload traffic seems excessive. Any ideas/reasons?
2 Likes
Sorry, I donāt get it clearly.
If you mean for the uploader, the used traffic is 35Mbps; for the downloader, the used traffic is 20Mbps. Then it make some sense.
As the uploader actually contacts all 5 nodes to upload the record.
Meanwhile, the downloader doesnāt need to fetch ALL 5 copies, could be just couple among the 5 in most of time.
1 Like
This is
ant file download
And it does not make sense and I am trying to figure out why it would
The computerās system monitor and the router both agree that the traffic is around 35Mbps (32 to 40Mbps) to the internet and around 22Mbs (20-25Mbps) down from the internet [EDIT: while running the client download. Many instances of it with different datamap addresses]
Also the chunks being downloaded are 3 x 3.5MiB and one datamap record.
So in reality I expected the download to ba a lot higher than the upload traffic.
1 Like