Possible vulnerabilities

Support
1

I wasn’t sure where to post this so I figured I’d make a topic for it. It’s been on my mind for a while now, but with TGE approaching rapidly I think it’s worth discussing: possible network vulnerabilities.

At the start of the beta I’ve mentioned that I think a bug hunting program is absolutelt neccesary, I believe the team has mentioned that they’ve got a few things planned, but again, TGE is approaching soon with the holidays in between. How do you guys feel about this?

Second question I have, how does quantum security work for files that are smaller than chunk size? Lets say I upload a textfile with my private key which is 1kb. This file isn’t getting cut up in multiple chunks, right?

Last, a question slightly related. If I understand correct, a node is not limited to 35GB, but to a number of chunks. If that’s that case, could I flood the nodes with small 1kb files that will fill up nodes?

Trying to fully understand some of the network fundamentals still so I would appreciate someone a bit more technical could try to make me understand.

3 Likes
2

it’s being chunked if you do it via the regular files api

I don’t really get it; the 35GB (or better currently 64GB) would be if the node would hold 4MB chunks (16k of them) - if you now upload 1kb files you would “fill” the nodes to a max of 16MB of used storage space while you pay the full price for chunks - for each chunk (and that will rise significantly the the more network fills)

I think there’s still a lot of obvious points the team knows about and not really a point in bug hunting if everyone finds stuff that is known anyway (and the team doesn’t have time to address it) … just my impression …

2 Likes
3

Even a 3 Bytes file is chunked now.

imagen
imagen1248×566 74.4 KB

5 Likes
4

It would seem like an expensive way to cause mischief.

That said, it definitely seems to be a down side of larger chunks. With the native currency, I wonder if this can/will change?

1 Like
5

imo the uploader ‘Dave’ or otherwise (cli and api) should rollup these little files into a bigger file, then upload as one meta file then these files get chunked, copied with a header list for the chunk, gets added to the Key Map , yadda,

more complicated? sure,

but way more efficient use of the node space,

all of which creates an argument for a NFR ‘new feature request’

in the future, for Autonomi to support smaller upload size and smaller chunks,

or a developer can can create the upload tool variant for ‘smaller file handling rolled up’ into a minimum bigger file size.

Lots of ways to solve this, and not let micro file uploads clog the network

That said,

its going to cost one to load that many 1K micro files big time in the current edition of Autonomi, most uploaders of modest means would go broke trying to attack the network and chew up capacity with such a planned attack …

That means this type of micro file upload attack (as posited) attempt is malicious and funded by big money/competitive/dark interests,

Just thinking out loud here

One way for the network to address this type of attack might be to employ a Smart Contract Security layer

ie- Have a Smart Contract doing statistical monitoring of posts to the DAG could catch and alarm such patterns of micro file mass upload behavior and

ID the wallet(s) spending trying to support this type of attack to rapidly chew up space in an automated fashion,

a security governance mechanism fashioned say with a Smart Contract which could instruct the DAG to not accept posts to the DL for node receive payments from such Distributed Wallet Upload MicroFile Attacks from those wallet addresses confirmed by the Security SC doing the governing in RT,

with say all distributed node operators have added special Security role/daemon process, and keep an automated blacklist for their close group, if you like top prevent these wallets posted to the DAG, that get stored in the Node as an encrypted chunk copied and distributed like other chunks, always available for reference…

so in the above hypthesized security layer it would defeat in theory such a ‘mass microfile consumption of capacity attack’ from multiple ‘wallet carrying’ uploaders,

plus the network in this supposed/hypothesized security layer would have the documentation to send to the authorities to go after the hackers.

Anyway lots to think about in this regard… all doable, its just time, brain power and a bit of effort to solve it…prioritized

6

The key of course is not upload, but people accessing the files not using Dave. Need to have individual data map and chunks for each file.

Now if you simply are uploading the chunks for multiple files in the one batch then that should be fine along with their data maps.

Just remember though that there is a 256 chunk limit per smart contract execution.

1 Like
7

This might work in some cases but not in most. Any application requiring real-time communication between one or more individuals, which constitutes a large part of our digital life (messaging, email, chats, forums…), will need upload files that, in most cases, are quite small in size.

Without its native token or a blockchain solution with exponentially lower fees than the current ones, Autonomi is extremely limited. In fact, from my point of view, it is currently economically unviable for most applications.

3 Likes
8

A major vulnerability that is emerging is the problem of having a large percentage of the nodes residing on one cloud provider’s infrastructure.

To date we’ve had reported publicly 6 cases of Hetzner reporting port scanning, probing by people on their infrastructure, and the person losing/penalised their ISP account. These are done by an abuse notice being sent to the people’s ISP provider and at least 2 cases on top of the 6 the notice was sent to another cloud provider resulting in action being taken.

The reason why Hetzner IDS is detecting a problem is to do with the concentration of nodes on Hetzner’s infrastructure and their over zealous IDS.

To break this down a bit, we had, my estimate, 30% or more nodes on Hetzner’s cloud infrastructure with both dedicated and non-dedicated servers running nodes. For this description I was use 1/3rd of the nodes, the analysis applies to other %ages just different amounts/degree.

  • A node outside of Hetzner will have on average 1/3 of their connections to a node somewhere within Hetzner’s cloud.
  • The number of Routing Table peers averages at around 200-240 peers
    • that is around 66 to 80 peers on Hetzner
  • The number of connections though can reach 1000 at times during churn and other events.
    • that is around 100 to 330 connections at any one instance to nodes on Hetzner
  • A user with 50 nodes will have up to 4000 peers in their combined routing tables and up to 16500 connections at any instance in time
  • The instantaneous connections is an ever changing figure with the added issue of connections happening to many different nodes over time. IE 15K from 50 nodes at one instance, but a second later it could be 10K with 2K the same as before but 8K different nodes for those combined 50 nodes the user is running.

When viewed by Hetzner’s IDS on the full infrastructure this could be considered abusive behaviour since they are very fast connections happening across a lot of IP addresses (different VPS) within Hetzner.

This is the vulnerability where a large %age of nodes are within the one cloud provider’s infrastructure.

The solution of course is decentralisation away from a few cloud providers. Like over 75% of nodes on home and other similar connections. And the other 25% more or less evenly distributed across cloud providers with no more than about 5% on any one provider.

6 Likes
9

Do you know if @Bux or anyone else from Autonomi has been in touch with Hetzner yet?

2 Likes
10

One vulnerability is the data just not surviving sudden losses of nodes. In recent testnets we have had quite many sudden drops of nodes.

Here is four different scenarios for a probability of data losing one or more chunks, in case of nodes dropping suddenly:

SCENARIO 1

Data size: 1TB
Chunk size: 4MB
Copies per chunk: 5
Sudden network shrinkage: 5%
Chance of survival for data: 0.92

SCENARIO 2

Data size: 1TB
Chunk size: 4MB
Copies per chunk: 5
Sudden network shrinkage: 10%
Chance of survival for data: 0.08

SCENARIO 3

Data size: 1TB
Chunk size: 4MB
Copies per chunk: 8
Sudden network shrinkage: 5%
Chance of survival for data: 0.99999023

SCENARIO 4

Data size: 1TB
Chunk size: 4MB
Copies per chunk: 8
Sudden network shrinkage: 10%
Chance of survival for data: 0.9975

With repeated sudden, too-quick-to-replicate, shrinkages the chance of survival drops of course.

Sudden drops are less likely, the more diverse the network is in all terms: decentralization of nodes (various cloud providers, geographical scattering, nodes from homes, ISP providers…), diversity of devices and operation systems, etc.

The other mitigation is more copies of the chunks, as the scenarios above shows.

Anyone can play with these calculations with this Google spreadsheet.

5 Likes
11

Though isn’t it a case now, that nodes that are cut off from the network temporarily still retain the chunks they have? If so, the temporary shut downs might actually increase the data’s chance of survival in the World.

Let’s say a temporary shutdown includes 4 of the 5 replicas. The 1 replica is still in the network and it is replicated so that there are 5 again. On top of that, there are the 4 replicas in the nodes that are temporarily cut off from the network, and the data’s replication count in the World is raised by that 4 that are in the nodes that are temporarily out of the network.

On the other hand if the network holds 35 EiB of data, the probability of all of the chunks remaining after 1% loss is 33%. After 2% loss it is 0.

These number are easy to whip up, but really meaningless if they are not compared to other systems. I have had some data corrupted in a cloud service, and I would not bet on the data on some old, random hard drives in my closet to have much higher chance of surviving. But that’s about the comparison I can make.

12

Already running with more than 5 copies. The code now has the “neighbourhood” holding copies. This can be up to like 70 copies depending on circumstances. It is still 5 nodes that are responsible but up to 70 holding a copy and able to deliver the record. I would expect for a 1/2 full network there is at least on average 10 copies.Thus for your 10% loss the test network just finished would have had a minimum chance of 0.999975 of surviving with 10% immediate loss.

The calculation that derives this “chance” is (1 minus % loss to the power of #copies) times the number of records

2 Likes
13

Hetzner did send the hetzner customers a retraction email for every wrongfully sent abusemail:

Dear Mr xxxxx,

Unfortunately you were mistakenly sent an abuse email. This was caused by an automated detection system. Please ignore the previous email, the abuse case has been closed.

Thank you for your understanding.

Important note:
When replying to us, please leave the abuse ID [AbuseID:EECCFD:1A] unchanged in the subject line.

Kind regards

I’m wondering if they sent that same mail to the non-hetzner customers.

If they did send that email to those other isp’s: there’s no hope if they don’t read the abuse mails carefully and only panic-react in a repressive way.

If they didn’t send that email to the other isp’s: well, someone at Hetzner has a bug to solve.

14

Doesn’t sound like it from the reports we’ve seen. And I can see that happening as the notices are from a different cause in their IDS algo. Incoming port scanning vs internal port scanning

15

I don’t understand what you mean.

This is the abuse email that was sent:

Dear xxxx,

We have indications that there was an attack from your server.
Please take all necessary measures to avoid this in the future and to solve the issue.

We also request that you send a short response to us. This response should contain information about how this could have happened and what you intend to do about it.
In the event that the following steps are not completed successfully, your server can be blocked at any time after the 2024-12-05 14:12:20 +0100.

How to proceed:
- Solve the issue
- Test if the issue still exists by using the following link: [https://statement-abuse.hetzner.com/retries/?token=f0dad73df25ab285d4e8e9d646ee0]
- After successfully testing that the issue is resolved, send us a statement by using the following link: [https://statement-abuse.hetzner.com/statements/?token=f0dad73df25ab285d4e8e9d646ee0]

Important note:
When replying to us, please leave the abuse ID [AbuseID:EECCFD:1A] unchanged in the subject line. Manual replies will only be handled in the event of a lock down. Should you have any questions relating to this, please contact our support staff at support@hetzner.com.
Please note that we do not provide telephone support in our department.
If you have any questions, please send them to us by responding to this email.

Kind regards

Network department

Hetzner Online GmbH
Industriestr. 25
91710 Gunzenhausen / Germany
Tel: +49 9831 505-0
Fax: +49 9831 505-3
abuse@hetzner.com
[www.hetzner.com](https://deref-gmx.com/mail/client/8RqjAALT1fc/dereferrer/?redirectUrl=http%3A%2F%2Fwww.hetzner.com&lm)

Register Court: Registergericht Ansbach, HRB 6089
CEO: Martin Hetzner, Stephan Konvickova, Günther Müller

For the purposes of this communication, we may save some
of your personal data. For information on our data privacy
policy, please see: [www.hetzner.com/datenschutzhinweis](https://deref-gmx.com/mail/client/7GghxKeI2mE/dereferrer/?redirectUrl=http%3A%2F%2Fwww.hetzner.com%2Fdatenschutzhinweis&lm)

> #############################################################################
> # Portscan detected from host 168.119.36.190 #
> #############################################################################
>
>
> TIME (UTC) SRC SRC-PORT -> DST DST-PORT SIZE PROT
> ----------------------------------------------------------------------------
> 2024-12-05 04:10:23 168.119.36.190 57646 -> 142.132.221.187 8571 1246 UDP
> 2024-12-05 04:10:24 168.119.36.190 57646 -> 142.132.221.187 8571 1246 UDP
> 2024-12-05 04:10:24 168.119.36.190 57646 -> 142.132.221.187 8571 1246 UDP
> 2024-12-05 04:10:23 168.119.36.190 57646 -> 142.132.221.187 9968 1246 UDP
> 2024-12-05 04:10:24 168.119.36.190 57646 -> 142.132.221.187 9968 1246 UDP
> 2024-12-05 04:10:24 168.119.36.190 57646 -> 142.132.221.187 9968 1246 UDP

...andsoforth...

Which is the exact same message as OVH has relayed to me.

16

The people who are using Hetzner’s VPS are getting emails about their VPS violating rules and getting retractions.

Then there are outside people getting abuse reports sent to their ISP without a retraction being sent.

I was under the impression the emails sent out were different for Hetzner VPS customers and to the ISPs, but even if same wording, they will be triggered by a different aspect of their detection system. Hetzner are looking after their customers by retraction of the warning, but not to external people for the Abuse notice sent out

17

good to know, ty, stay safe, this stuff needs to find its way into reference documentation at some point… :wink:

1 Like
18

Its mentioned as a cache. This males up part of the caching that was added recently. Mentioned a number of updates ago IIRC. Mentioned as more copies kept to reduce the sybil attack vector where a big operator tries to own a region of XOR space to hold chunks/records to ransom. And if mentioned in the WP it will appear as caching

1 Like
19

This topic was automatically closed after 59 days. New replies are no longer allowed.