Thanks - I found an example from ant-cli and grafted it into sn_httpd now. Seems to work! 
7 Likes
Was this one file or 3 files?
It would have to have been 3 files for 3 meta chunks wouldn’t it, considering they are small file(s) requiring only one datamap chunk (each).
Also @qi_ma the report said 8 chunk uploaded, not 5. If 3 meta chunks were created and not uploaded then it would be 5 chunks uploaded or should be 24 attos if 8 chunks uploaded and some meta chunks not uploaded.
Just a little confusing here. Is the 8 chunks uploaded message wrong and it was only 5. Is it 3 meta chunks per 5 chunks of the one 5 chunk file created?
1 Like
To update on my stability issues, there seems to be some serious regressions in my router stability.
I enabled sn_httpd with the new ant_bootstrap code and it kills my network when the client library connects.
I’ve also experienced similar issues with ant CLI when attempting uploads.
I’ve also stopped my nodes as they also take my router out periodically, not just on startup, but randomly.
I’m assuming this all relates to changes with node discovery. If I’m having these issues with a decent 500mb/70mb connection with a not terrible router, this could cause major headaches for others.
4 Likes
What is the block size being selected for uploads? With 4MB chunks and starting to upload too many chunks at once you could be initially overloading the router’s buffer space for buffering UDP packets.
You see the QUIC starts (attempts) to use its max window sizing of 10MB and since chunks are upto 4MB this means the uploader sends the whole 4MB at your local network speed to the router. So if block size is 8, its possible 32MB is sent to to the router at 1Gbps while the router is limited to 70Mbps on its uplink.
For most routers their buffers are limited to that or less. Even many commercial routers have such limits with only 64MB total RAM, but they are fine since the local net speed is like 1Gbps and the uplink is that or faster.
TCP doesn’t have this issue since TCP “window” size is already in KB only (1 packet of 1536 bytes or 7 or 10 packets)
3 Likes
I’m using defaults with sn_httpd and ant CLI. I don’t think they can be changed from the latter either (at least not using an argument).
Your prognosis sounds reasonable to me. The latest version of the libs seems to be much less stable than previous iterations though.
2 Likes
Hey,
I’m sorry to hear of the difficulties you’re having.
Would you mind sharing the logs for your uploads please?
Every client run as a log, which is available at ~/.local/share/autonomi/client/<timestamp>.
Also, you mentioned that bootstrapping in sn_httpd is killing your code. Is there any output you can share from that? Can you run it with ANT_LOG=all or some kind of debugging?
Also, what is your router/networking configuration? Are you using port forwarding?
I think it would be useful for us to have these logs. We haven’t experienced any problems like this with our uploaders.
5 Likes
QQ - Are nodes still being shunned if they don’t have 35GB of space to play in?
2 Likes
AIUI it is only node-launchpad that demands 35GB be allocated to each node.
I think you could start more nodes manually with antnode, ignoring the recommended 35GB restriction. You better be ready to add more disk space as they fill up though.
I dont think there is any mechanism whereby nodes can be shunned because they dont have the space allocated - until they actually fill up and fail.
I ve been distracted by OtherStuff this past while and so Im not 100% up to date.
5 Likes
Since autonomi 0.3.1 I’m able to connect to the network and everything works fine with a local test network - uploading multiple directories of files.
However, with the live network, running my client locally or on VPS it repeatedly fails to upload a directory of about a dozen small files due to a lack of quotes: Issue #2573
7 Likes
You are right here. On one device I certainly do not have 35GB for each node. Small setup but prob about 20GB per node. But since the nodes never get very full in these tests the nodes are fine. They’d have to be storing 4000 records at least to be an issue and the max I’ve seen in the tests is like 1500 records.
4 Likes
Mind you if the nodes are half full (8K records out of 16K) with the current batch of 4MB chunks then that is 32GB per node.
6 Likes
Hi all - for those who have active monitoring on antnodes in past 3 to 5 days since 2024.12.1.6 was released, i.e. v0.3.1 - can some of the community members provide some feedback on the average stats off their antnodes (bucketed by groups of antnodes running on same hardware specs), so we can get a sense of how the new hotfix is doing?
Granted, we are actively monitoring our portion of antnodes (all of them were upgraded as of yesterday), but are curious to get some feedback from the community on the hotfix.
If the feedback can come in the following template/format, we would appreciate it:
Number of Antnodes per Host: 100
Host CPU Usage: Before vs After
Per Antnode CPU Usage: Before vs After
Per Antnode Mem Usage: Before vs After (in MB)
Per Antnode Connected Peers: Before vs After
Per Antnode Open Connections: Before vs After
Per Antnode Avg Uptime: (Latest Version in seconds)
Type of Update Performed: <Reset> (new peer id) or <Upgrade> (maintained same peer id)
Type of Setup: Port Forwarded vs -home-network vs Automatic vs UPnP vs Public Cloud
This request is specific to antnodes, and not ant CLI activity (re: uploading & downloading) etc.
Note: If you don’t have the before the update (v0.3.0) metrics readily available, that’s okay, just post the recent / current version’s metrics.
3 Likes
I upgraded my router’s software to the latest version and it seems to have helped (Fritz!OS 8). Not had any connection drop outs since.
I’ve also set the upload/download rate, which is supposed to help shape the traffic.
Still having issues with absent quotes on uploads, but I can run nodes, etc, without trouble.
3 Likes
Sorry Shu I don’t have all the requested data to share but CPU is down significantly (30-40% improvement) across a range of devices from Ryzen 9, to some dual CPU intels all the way down to little 10 year old 4 core AMD GX-420CA’s.
2 Likes
I’ve been having this same difficulty with directory uploads. Repeated lack of quotes terminating attempts.
5 Likes
I have noticed a lack of earnings the last 24 hours after the earnings were steady at 3 to 4 a day for the three nodes I have run since the update to 0.3.1
I am MS Windows dumbed down so don’t have a good grasp on how to get you the stats you are looking for. Time to get back to linux…
1 Like
I have had the feeling for a long time that the distribution to nodes might be skewed, example if starting 100 nodes than it can take 4-5 days until all nodes get at least 1 atto while other nodes get 10 or above. It might be only a feeling, maybe I miss nodes stopped or something but thought it might be worth mentioning. So guestion becomes if the distribution is as close to random as possible?
Also when nodes stop because over 50% cpu usage, is there a way that they maybe can try to connect again when load drops? To every day have to check if nodes have stopped becomes work, to get as many ordinary people to run nodes I believe it has to be as close to, fire up and forget, as possible. Also nodes that have been stopped for a period of time, can you try and start them again or always have to reset those nodes?
1 Like
They seem similar.
Both still increased bandwidth (after initial usage) gradually for about a day till I had to reduce from 100 nodes to 80 nodes. And just now had to reduce to 75
Still seeing some nodes (rare) grow to GB in mem usage
Using port forwarding, MikroTik router 700Mbps down and nominal 40Mbps uplink. Uplink can go higher for short periods. The uplink for the node computer is smoothed to 37Mbps in the router and queue only needed sometimes, plenty of buffer space (>200MB) in router.
Upgrade was done via a new set of nodes.
2 Likes
I am having a hard time with gas fees for uploads.
4 Likes
Same here I’m guessing this might have something to do with why the uploaders keep running out of eth.
4 Likes