Caching Chunks (lru_time_cache)

Development
1

Caching makes it possible that data get’s faster over the network the moment more people start downloading the same file(s). If you ask your Client Managers (close group of 32 nodes) for a Chunk, they’ll forward your request to the closest address in XOR. This means it will take an x-number of steps to find the Chunk if it’s out there. To quote @Seneca :

The amount of hops a chunk takes to reach the recipient is O(log(n)) in a worst case scenario. For example, in a network with 1 million nodes that means about 20 hops.

Can we do better? You bet. Chunks are cached along the way. So the next time someone close to me in XOR requests that Chunk it won’t be served out of the Vault where it first came from. It can be in Cache only 2 hops away. Making the network faster and better. It also prevents popular files to create to many Safecoins in Farming.

A few days ago @dirvine was talking about the cache, and he said:

This is again down to XOR, yes cache will kick in fast on that route (closer to destination route becomes more unique, i.,e a single cache hit), but there are separate individual routes to a chunk. Each of these will cache for 10 mins (current) but the app needs to be super popular to hit a cache and then only for 10 mins or so (also based on number of cache hits per route).

Today I saw some activity on GitHub in the Lru-time-cache repo.

Time to talk cache :yum:

@dirvine said that the 10 min. limit for the cache is “current”. Does this mean it’s open for debate? Why not 5 minutes or 20 minutes? Should people be able to choose their own size for caching files? What’s the maximum size of the cache?

2

Cache will be automatically calculated and not time based. The cache for chunk caching will be size based. then it’s much harder to try cache defeat attacks. The size of these is not currently settable (we would have config files all over the place in this case, but it’s open for debate). It should be calculated on size (number of chunks) and also dynamically calculated as much as possible.

There are a few magic numbers still in the system and sometimes it won’t be wise to allow users to also set more :wink: Yes for sure needs further work to get down to absolute foundational types, but we will get there. This cache (after saying that) can have a large affect on your computer so maybe should be settable.

7 Likes
3

I can imagine times when no caching is desirable, especially if the data updates very frequently and must be precisely correct.

4

Yes we do not cache structured data for instance for this very reason.

1 Like