Update 21st November, 2024

Development Updates
124

Depends how quickly you want to find out whats going to break?
Cos something will break and it would be nice to find out about it sooner rather than much more publicly later, possibly trashing the whole project.

Its obvious there is not enough dev resources to do the width and depth of testing the network needs. If the community can take up some of that slack then we should.

The clever trick will be to do this in a way that causes the least hassle to the devs by bothering them about what to test and how and when. So Im hoping community members with the relevant testing experience can contribute to getting us a realistic test scheme. @rreive and others know a lot more than me
As for me I know the value of proper testing and the cost of doing and not doing it and I can follow a test schedule someone else has written.

7 Likes
125

I understand, but then it should be done privately, not publicly. Then i see the benefits of it.

1 Like
126

We need too many people to participate to do it quietly. And anyway this place is full of transparency zealots, we’d never get away with it :slight_smile:

4 Likes
127

Maybe that’s why it’s best to stay patient and trust the team to handle it

(I still do believe they’ll launch only when they’re confident the network can handle a serious hammering.)

2 Likes
128

That’s certainly a wise approach in general, but please note the depth and extent of @neo’s experience and analysis of this specific issue.

3 Likes
129

I didn’t know they had nuclear bow and arrows :joy:

4 Likes
130

How many lines of code is it? Is it more than the 10k or so lines it is said a human can really understand and debug? How large is the largest file needed to set it up on any network element or device? Is different code going to different devices? If so is that coming from or managed by, beyond launch, a single or concentrated set of failure points?

1 Like
131

Only in the original “Get Smart” shows

132

Testing, and builder, partner development activity needs its own test net parallel to production network. It can also be used to replicate problems, apply and test proposed fixes etc…, its just part of growing up the organization and community… production and test always run in parallel when that happens .

The Test net does not need to be tomorrow, aka QUIC, it just need to be acknowledge on the roadmap and at some point funded to get the community actively involved, its one of the best places to do deep know how transfer from the core team to the community…, and show the rest of the world we are on top of it like two big dogs, which instills confidence and get’s sign-up.

imo if we get there some time in 2025 after launch, that’s a logical progression, but also

This is star trek, we are going places no network has gone before, so we expect challenges and we are ready for them , together. . :wink:

5 Likes
133

Who has some wise advice on how to troubleshoot this issue.

I have a few servers dotted around multiple geographic locations all with the same provider.

At one specific location the nodes are seeing an average network size of 35K vs all other locations they see an average network size of 75K.

The problem location is getting rewards but at a far lower rate.

Support is hopeless as they say there is no Firewall or NAT at any of the locations but why would just this one be causing me grief.

Its in Canada if that helps the rest are not.

3 Likes
134

actually I don’t think we’re using the large default window size from what I’m seeing with Wireshark …

image
image1002×650 131 KB

this hints to 1.5KB frames …but the 10MB (at least to me) don’t seem to be active when using the autonomi client to download data from the network …

… I see somewhere between 80-160k of those packets when downloading BegBlag (15MB)

autonomi file download e731c67a6be8c6abc052ed17338f540ab650e969d5a86571c1cb9821738a7956 .

which is … kind of strange in a different way now I guess :smiley:

…and I think for bad internet connections/routers we need some kind of throttle that can be activated … I get absolutely hammered with data pkges as soon as I start a download … I could even imagine the amount of parallel TXs happening here might be the reason for failing downloads …

(for sure no networking expert here - just thought I’d start Wireshark and check some assumptions…)

EDIT:

where did the batch-size argument go?! … - I needed it many times in the past to make downloads succeed (locally - on my crappy internet line … not the cloud machines … those never had issues)!

6 Likes
135

I think you are mixing more things together.
Ethernet frames are HW limited to 1500 B (ignore Jumbo frames for simplicity), but can be lowered by things like PPTP, VPN and other encapsulations.

Window size is how many packects we send before we have at least some ACK response (picture is for 3 packets window size).

obrazek
obrazek771×485 67.6 KB

10 MB window size would be almost 6k+ packets.

Many parallel connections, you would have to filter one specific stream to see the window size effects.

Yes, there is the multiplication effect of many nodes sending you data in parallel. Basically you will get window-size * number-of-chunk-holders MB of data before anybody even detects something may be wrong and and your connection may be overloaded and dropping packets.

4 Likes
136

okay

image
image1205×799 140 KB

so here I see some kind of handshake after a couple hundred messages … so then this is probably the window size …

…still nowhere near 4MB … I see it (when filtering for 1 particular partner ip) between every 120-600 pkgs … which is a super rough diagnosis I know … but I think before changing settings in QUIC and thinking we resolved everything with it maybe it does make sense to have a look at the bytes we send around … I don’t have the feeling we know what we’re doing here atm :smiley:

5 Likes
137

QUIC is pretty smart actually. As a protocol spec it does do sliding window adjustments and so on. So it can likely go all the way. up to XMb as a MAX setting but that will likely be on a data centre Hugh bandwidth zero error type line. QUIC should try and find the best runtime settings for any links.

So perhaps what we are seeing is QUIC doing what QUIC should be doing and adjusting to the network connection types it sees and tests for the best runtime settings. It does a bunch of agreements with the remote to test some of these settings. It gets very deep very fast.

11 Likes
138

That would be pretty impressive black magic going on there - but since I indeed seemed to notice different lengths it is absolutely possible… :exploding_head:

4 Likes
139

That is the UDP packet size, thats all

Also the chunk might be being split up across multiple streams. QUIC allows multiple streams to be sent at once. On a potato ISP router this will be equivalent as one stream since the router’s buffer is loaded up at 1Gbps and unloading at most likely <50Mbps. For Datacentre then the packets sent through their routers as fast as they are received.

So if Autonomi is enabling QUIC to use multiple streams (connections) for sending chunks then the smaller initial windows size is more essential since a 1MB windows size could still see the whole 4MB chunk sent at once if 4 connections/streams are allowed. There is a setting for max streams as well.

Its like a machine has “oil here” or “Adjust for specific operation” and its ignored. There is a reason why the max window size can be set and that is to account for the connections/devices being used. Potato routers need plenty of oil and that equates to limiting the amount of concurrent&sequential packets being sent at once.

Its basic flow control, a (used to be) well understood concept used for systems with uneven connection (and/or device) speeds. Its essential to be implemented since we are not using QUIC in a server-client configuration but a peer to peer outside of datacentres where speeds are hugely different between nodes and the routers at the end points have severe buffer limitations that would be used to compensate for these speed differences.

Correct, but the initial window size is set to the max allowed. Thus if the initial block of (upto 4MB max chunk size) is rejected then it will negotiate down, maybe half (5MB window size) and it that fails then lower again.

If we start with a sensible size for home routers then we save plenty on retries and multiple chunks running through a potato ISP router. Its like oiling the squeaking wheel that sometimes jams.

10 Likes
140

Only in Canada eh? :wink: , Well it might be the NAT of the ISP is agressively firewalling unknown UDP sources incoming from other countries, because if the UDP traffic is coming from the USA, the current government is even mandating ISPs block news from the USA and the UK and some of those feeds fly across the border in UDP.

So if your Close Group peer connections in the USA for example are trying to send chunks over UDP to the near XOR address in Canada, good luck with that.

My three node earnings hosted here in the home office are at best modest, I look to be earning about 35% lower than the average for nodes my size per the leaderboard the past three weeks, so this ISP blocking action of UDP at the border probably explains that.

I am in Canada, its a CCP vassal state these days, its totally crazy,

The CDN Govt thanks to Turdeau have FB/Meta complying blocking any USA news to Canada as an example, Canadians on FB cannot repost USA news to FB Canada timelines. grrr.

This is not the only case, Google and Youtube also comply, we are actually fairly isolated as far as news goes from our traditional allies and neighbours in the USA and UK at the moment and, it will be like this,

till this turd gets punted in late 2025. His popularity is in the teens these days with Canadian voters, especially after the lockdown, the forced jabs on workers who lost their jobs when they refused (lots of nurses, doctors, police, fireman, Fed and prov. and muni. govt workers and military got fired for not taking the jab) , the excess death count here is really bad for people in their 30s-40s and elderly who took the jab, In Alberta they lost an estimated 40,000 people to escess deaths they are covering up, and the way they treated the CDN Trucker Roll for Freedom in early 2022 was disgraceful, locking their bank accounts (TD bank was the worst) and arresting and detaining them for almost a year. A bit on the Coutts 4 Jailed in 2022

Coutts Four Jail Status

Based on the provided search results, the Coutts Four were arrested in February 2022 and held in custody for nearly two years. However, as the trial progressed, the charges against them were dropped, and two of the men, Jerry Morin and Anthony Olienick, were released from prison after agreeing to plead guilty to lesser charges. The other two men, Chris Carbert and Tony Olienick, stood trial by jury and were ultimately acquitted of all charges, including conspiracy to murder, on August 7, 2024.

Therefore, as of August 2024, none of the Coutts Four are still in jail. Two had already been released, and the remaining two were acquitted and set free.

Total Canadian Govt BS with the major CDN PM muppet controlled by the Rockefeller Cadre.

My better half lost her son who was only 34 to this nonsense, it was three jabs and he was gone in lates 2022, 2 years just a day ago. Sad and really evil.

}:expressionless:

;/

4 Likes
141

Well that is insane. I asked to move out of that location but they are not having it.

Not in my wildest dreams did I think that geography would have an impact.

2 Likes
142

Yeah its crazy, that and the country has let in 4.5 million immigrants in under 4 years to raise the population from 37M to 41.5M, where local high school kids can’t get a part time job because the Federal CDN govt subsidizes the big Fast food corps to hire them all. Its population replacement.

One goes to a fast food place in my neighbourhood and just in the last two years all the native born kids and first nations kids ( I Live near a mohawk reserve) , the high school kids and retirees(who really need the extra income because the Canada pension plan is broke) have been replaced with young immigrants from Pakistan, India, Bangladesh largely where I live, they also get assisted living from the Federal Government and the Ontario Government, and preferred access to college and university over the local kids.

And now the Federal Govt has agreed to allow all the USA deportations coming after the Trump election, into Canada.

This place is so over. It was likely the original plan.

3 Likes
143

Geography always plays a role. But it doesn’t have to be problem of Canada and government regulations, it could simply be that your service provider had cheaped out on peering at that location or they have badly configured BGP there or main router that needs reboot.

1 Like