Realtime database

[neo]

This is one reason I questioned the free nature of scratchpad updates. Even if the cost was 1/10 of storing immutable then it would even out the use of them

[Traktion]

@storage_guy

I thought it was stated that once the size of chunks had been set to 4MB it wouldn’t ever be possible to have a smaller size?

As @neo said, but I’d also add that small files already exercise this. If you upload a 5 KB files, it still gets chunked and stored… just as very small chunks.

The network has a lot of flexibility built-in in this regard. Right now, the gas price just means you would want 4 MB chunks whenever possible. Indeed, it led me to suggest the Tarchive idea (which I still want to play with).

Once we can substantially lower the gas price, many cool features become more accessible.

[Traktion]

@happybeing

I mused on that for about 5s when creating the ETag handling and saw references to server side caching among other things. I don’t do much of that yet, nothing for resources in fact (just archives and history metadata).

I don’t know much about that side of things, but expect there’s a protocol somewhere for an app to control this - perhaps by saying no older than (I think that may already exist in the ETag handling).

So we could have a sensible default for caching mutable data, but apps which need fine control can have it either through ETags and related headers, or some other existing protocol or something we come up with together.

I don’t see much need for this yet, so have not given it more than 5s thought but it seems feasible when apps begin to need this.

Yeah, etags are a very cool caching feature with HTTP, assuming your client has its own memory/file cache (like a browser does).

It would help to have an etag like implementation for mutable data requests at the network level too (rather than just the edge with HTTP). It is easier to confirm data is the same, than send it again. It does rely on the client being capable/willing to use them though (good actors).

For general caching of mutable data, it’s very context sensitive though (one apps, stale is another’s new enough), so it’s hard to have a one sized cache to fit all. We could possibly have multiple cache buckets with different TTLs within the network, with clients maybe hinting how stale the data can be, but ideally, you want caches to be invisible.

Ofc, we could have a short TTL cache for all mutables, just to stop thrashing. Something like a 5 second cache, to stop nodes getting hammered by repeated requests. It would manifest as a data change only propagating after 5 seconds, which may be undesirable in some cases, but fine for most.

Compared to being able to cache immutable data forever, 5 seconds looks pretty weak though. I think that’s why app designer should consider using immutable data whenever possible/feasible though.

[Traktion]

@neo

This is one reason I questioned the free nature of scratchpad updates. Even if the cost was 1/10 of storing immutable then it would even out the use of them

Yeah, I had similar concerns, but there was a thread (somewhere!) where one of the devs suggested throttling and then it started to feel very much like an app, rather than network issue.

In other words, scratchpads can make promises about being able to read/write for free (once created), but they only have limited guarantees about performance. If they are overused, they will need to be throttled, which just shouldn’t need to happen with immutable data (at least on a mature network).

[Hannu]

Did a bit research. No events, so polling is the only option, but polling the version number should be “cheap”. Some event relay might be useful, but it would be no longer decentralized.
Maybe there is natural support for events some day. But it is not an easy task and might easily open some attack vectors (dos).

[Hannu]

So not optimal for a shared database, but certainly useful.

Another useful option would be location based search. Autonomi network would be superb for such applications. For example a simple distributed second hand shop; upload the stuff you want to sell with location. See all nearby sellers and just pick up the things you want to buy. (Oops, a bit offtopic. but somehow related anyway)

[Traktion]

@Hannu

Did a bit research. No events, so polling is the only option, but polling the version number should be “cheap”. Some event relay might be useful, but it would be no longer decentralized.
Maybe there is natural support for events some day. But it is not an easy task and might easily open some attack vectors (dos).

It’s worth noting that kafka, probably the world’s most popular/famous distributed message broker, also relies on polling by clients.

If, as with kafka, you batch send records when polled, it can mask the latency pretty well and maintain throughout.

Fwiw, I always fancied having a crack at a message broker implementation on Autonomi. Maybe once AntTP settles down!

[Hannu]

@Hannu

location based search

Btw, this needs no special support from autonomi itself.

[bytes]

I think it’s possible to utilize Pears to develop a corestore that utilizes Autonomi for occasional retrieval (from a permanent storage archive) of Hypercore blocks. The freshest blocks need not be written to Autonomi immediately, so the buffer can grow so as to fully utilize the 4 MB chunk size. Writing / consensus logic could be implemented with an Autobase (and signatures could be generated by a threshold of peers). Basically, an eventually consistent append-only log that has entries added by multiple peers. I think this could provide lightweight and fast key/value storage (via a Hyperbee), among other things. Notably, queries would not use Autonomi directly - peers would share Hypercore blocks with each-other (and fall back to retrieving archived data from Autonomi if needed). Reading data from Autonomi would occur infrequently.

Or likewise with Peerbit (if you have your heart set on a RDBMS) - implement a persistence provider that utilizes Autonomi.

[Hannu]

Oh, thank! Interrsting tech! Maybe the Peerbit with autonomi as a persisting layer is what I’m looking for.

[Traktion]

I think it depends whether you want hybrid or native autonomi solutions. Both definitely have their place.

Native are great for use with native apps. Hybrid are good for any apps.

I’d love to see folks exploring this sort of thing though. So many interesting dev challenges out there with autonomi.

[Hannu]

Or maybe there just should be peerbit/firebase like solution on top of autonomi The only missing part is notifications of changes.

[Hannu]

Let me explain what I have in mind; a taxi calling application. A taxi driver would have an application to be able to put her price list and location to the realtime database. A customer has another application where he can see all nearby taxes, compare the prices and see the feedback they have got. He can call the one he wants.

When he calls it, he will see the map where the car is getting closer, until he can see it and jump in.

I have done this, with google firebase. No servers needed. It worked fine, but it was centralized (google).

The same could be done with autonomi, completely de-centralized.

And one day you might be able to pay with ANT’s

[bytes]

Whether centralized or decentralized, access control and security is important to plan for. This I think is one of the most significant barriers.

A decentralized solution has to have mechanisms to prevent invalid records from being written (by malicious or outdated clients), thwart spam attacks, ban malicious users, etc.

With a decentralized solution, most of the below “know your customer” verifications are more difficult, or nearly impossible - without some authority figure. Forsaking all of the following verifications is not really viable, because a “permissionless” decentralized app would fall prey to Sybil attacks, manipulated ratings, victimizing of users by criminals, etc. With centralized solutions, app developers consider KYC verifications like:

1. Who is allowed to create a taxi driver account? e.g. some or all of the following?..
   – Anyone who submits a registration form?
   – Clicks a confirmation link in an email message?
   – Recieves a SMS message with a verification code and provides it?
   – Claims to have a date of birth that’s 18 or more years old?
   – Uploads a photo of a driver’s license and be validated by a central authority (you)?
   – Provides insurance documentation and be validated by a central authority (you)?
   – etc.
2. Who is allowed to create a taxi passenger account?
   – Anyone who has a credit card? Prepays with crypto? Promises to pay the driver in cash (and the driver promises to pay a cut to the app developer)? etc.
   – Ticks a EULA checkbox?
   – Goes through ID verification?
   – Doesn’t have a violent felony criminal record?
   – etc.
3. How are disputes between drivers and passengers settled? What happens in the case of an accident? How are rules enforced? etc.

A DAO (“decentralized autonomous organization”) may be possible for you to set up, but governance can be tricky and messy. And it requires participation from humans (or their automated agents). It’s basically a committee of people who have roles and make decisions, subject to some governance rules that have been chosen.

Essentially, new users would be created by:

1. Fill out a registration form (and provide a public key)
2. The DAO would perform some review process and approve the KYC verifications. This could be done by one or more trusted “moderators”, “notaries”, “judges”, or whatever you want to call the members who have authority. Perhaps even with automated tools, AI, etc - but ultimately is under the authority of the DAO stakeholders.
3. Users would be able to write records to the database, unless the DAO chooses to revoke access.

Maybe you already have a DAO in mind. Or maybe you intend to let the taxi app be a “wild west” - but I believe that would be a non-starter (bots would destroy it). Autonomi does not currently have capability for DAOs itself, but several blockchains do (including Arbitrum which Autonomi currently uses for data payments). I don’t really know that much about participating in DAOs. I’ve done it on Internet Computer’s SNS DAOs.

It is possible to have a database/website/etc hosted on Internet Computer. I do know that one disadvantage of hosting on Internet Computer is that someone (i.e. you or the DAO) has to pay for the usage (data storage utilized, processing time, etc). Also, it can be a bit tricky to set up DDoS protection in order to thrwart resource-drain attacks. Still, that might be economically viable for your purposes.

With Autonomi, there are a lot of limitations currently but one of its major strengths is that it’s a pay-once model for data storage (and unlimited retrieval). It does not support DAOs itself. However, Arbitrum (the EVM-compatible blockchain Autonomi payments currently use) can have smart contracts that can provide DAO functionality, since a lot of DAO code is written for Ethereum compatible chains. That said, from what I’ve seen on the forums most people in the Autonomi ecosystem seem to want Autonomi to move away from its payments occurring on an EVM compatible chain. I seem to be in the minority on this issue - I believe it’s possible to greatly improve transaction fees for data storage by using techniques that are compatible with Arbitrum (or cheaper EVM-compatible chains). In other words, in case you’re thinking of launching a DAO, I wouldn’t count on Autonomi staying on Arbitrum long-term.

[Hannu]

Well, if you see a taxi on a road, wave your hand and the taxi stops, nothing like this is needed. Think this kind of application as just doing the waving digitally.

Of course, things get more complicated if the app is used to transfer money. But without that it just connects someone needing a ride to someone offering a ride.

[happybeing]

Except taxis are regulated/drivers vetted etc, and without that any app becomes a home for predators of all sorts and hardly anyone will use it.

This will be a challenge for lots of apps, but it’s especially important where the risks to users are so high: getting into a vehicle with a stranger.

[bytes]

Even if it’s just for “waving digitally”, you’d be trusting users to be real people who actually are where they say they are.

Thing is, it wouldn’t be that hard for someone malicious to create bots that spoof GPS coordinates. Could be done by somebody who gets a thrill, or a competitor (or a contractor of theirs), or someone who wants your business to pay them a ransom.

If you think there aren’t competitors, there really are: e.g. ridesharing apps like Uber, Lyft, and so many other apps (which are prevalent in various countries across the globe). If your app is benefiting taxis in an economically significant way, you are “taking business away from” ridesharing apps (and thus you are a competitor).

I suppose you could opt for a somewhat centralized solution that makes it costly for anonymous users to fake one’s location: code signing and attestation (via Apple and Android APIs). Ensures that the program output is generated by genuine hardware and software that isn’t jailbroken.

[Hannu]

There are ways to collect reliable data by community, for example waze makes it work. It is not necessarily easy, but it is possible. Does the data have any network provided timestamp? That could be used to prevent the most obvious frauds. With timestamp we can find out trustworthy users in the long term.

[neo]

There is no network timestamp. The network is made up of protocols at the base level, above udp, but below any NTP, or SMPT etc

[Southside]

@Hannu

Does the data have any network provided timestamp?

What heresy is this?