Pushed v0.11.0 with some exciting new features, mostly concentrating on performance and offline browsing.
Adds response compression (where applicable) to decrease bandwidth consumption between AntTP and clients.
Adds tarchive performance test.
Minor logging improvements.
Updates mutable gets (pointer, register, graph, scratchpad) to use the hybrid cache. This allows last known value to be returned immediately, then fetches the current value in the background. This provides a more responsive user experience for common use cases (more options will be added later).
Makes the autonomi client optional, allowing offline operation. If data exists in the cache, it will be served.
Refactors to simplify dependencies around the caching client and controllers.
Adds gimim (gateway imim) and gindex (gateway index) bookmarks for IMIM and index (respectively) to be viewed over a gateway server.
Adds server response header to provide server name/version.
Reduces the default mutable TTL from 60s to 5s now that there are last-known-value caches and background updates.
This one is a bug fix that was impacting streaming on Chrome (and related, e.g. Brave) browser. I think it was also causing some issues with Firefox munching through CPU power, but I’ll keep an eye on that one over the next few days.
Fixes issue where an incorrect ‘range to’ is returned in the ContentRange header. This primarily caused issues with Chrome (and related browsers). It also caused CPU spiking in Firefox.
I decided to try out antTP for first time as I wanted to repeatedly run benchmarks without any caching taking place, and see if results come back reliably if downloading near 5Gbps continously.
On my first attempt ended up causing a panic on AntTP:
@Shu, I’ve been experimenting with the cache tuning and the latest version of Foyer (v0.19) lets me turn off the in-memory and on-disk caching (without refactoring).
So, I was able to do some benchmarking with each of the 3 modes above.
I’ve not pushed a formal release yet, but master includes these changes, if they are still useful to you.
very nice this one is working much better with the streaming over my dodgey internet connection also nice bonus being able to set the cache size to zero
Adds idle disconnect from the autonomi network to decrease CPU usage when inactive (and save power).
Changed all mutexes to use Tokio to improve performance and standardise.
Added milliseconds to logs to help trace performance.
In summary, it’s now got a power saving mode. After a (configurable) idle timeout of 30s, the autonomi client is disconnected and deallocated.
From my testing in Linux, CPU usage falls from around 1.00% to 0.05% - a 20x improvement when idle. Obviously, this will also reduce power usage, which should reduce bills and increase battery time (especially handy on mobiles).
When a request comes in that can’t find what it wants in the cache, it will reconnect automatically. This is actually a very swift process these days and i hardly noticeable, so maybe the default could be decreased, but I’d like to monitor how it goes first.
Their knowledge of lower level sys calls is way beyond mine, but it is curius that they have used a tar file for performance reasons.
Fwiw, the AntTP implementation of tarchive stores the self encrypted chunks (in a hybrid memory/disk LRU cache) for the tar file, rather than the tar file itself.
It is done this way to optimise the data retrieval from the autonomi network (much longer than any sys calls to disk). Perhaps similar goals in some way, but different extremes of optimisation.
Why only store the chunks? Well, the data remains encrypted up until the point it is served. This does improve security, as no data is stored decrypted on the host. However, it does the chunks required in the response must be decrypted each time.
So, I’ve mused over having a mode which stores the decrypted content in the cache. It would certainly speed up response times pretty massively (orders of magnitude, I suspect), but it does come at a security/privacy cost.
Edit: btw, there is potential to cache decrypted chunks in memory (until rotated out), which could balance the performance vs security element. Maybe something to consider.
Good thoughts on the tradeoffs of when to decrypt chunks. (before caching, before [mem only] caching, after caching) If I were running AntTP on a server, I’d want it configurable so I support those musings.
Integrates Autonomi 0.6.0 and self_encryption 0.34.0 and addresses breaking changes with DataMaps and streaming in general.
Updates performance test URLs to correctly retrieve file (now that exact paths are used).
Removes dependency on self_encryption and uses Bytes to communicate with chunk-streamer.
Aligns boundaries with chunk-streamer and fixes issues with zero indexing since Autonomi 0.6.0 integration.
Improves range handling to return 416 RangeNotSatisfiable when the range cannot be processed.
Various range simplifications to reduce complexity.
Fixes bug with chrome not streaming due to the final byte not being present in the stream.
In short, this fixes the incompatibilities for v0.6.0 of Autonomi libs, including ant CLI, etc. You should now be able to upload data with the CLI and retrieve it via AntTP again.
Backwards compatibility is maintained, but there are some performance regressions. However, these are largely masked by the LRU cache, as some of regressions are due to reading the same chunks repeatedly, in different scerarios
With caching enabled, performance benchmarks show similar performance. I’ll update the performance thread with some results soon.
I’m working on getting the latest anttp library running in Colony. It does run and I can open up webapps and files using the REST API. I’m not sure how to open IMIM pages from the autonomi address (or maybe this isn’t how it works?). For example, In the IMIM README, it gives this link to your blog:
Which works fine if I open it manually in a web browser. However, I don’t really like encoding localhost and port into the metadata. I could remove the localhost stuff and assume that someone setup the proxy, but that is additional setup for the user. I want the links to ‘just work’ and open in the web browser. Is there a way I could pass an autonomi address directly?
For example, in the bookmarks, I see this as the address for your blog:
Adds x-cache-only header for uploading chunks and public data, allowing offline POSTs/PUTs to local storage. This allows data to be stored and accessed in the regular way, without uploading to autonomi and paying associated fees. Useful for testing data and/or persisting data that doesn’t need to be pushed to autonomi.
Updates README with improved roadmap.
Adds cache only uploads for pointers.
Adds cache only uploads for chunks.
Adds cache only uploads for public archives.
Updates chunk-streamer to 0.4.3 to include encryption support.
Adds memory/disk/none options for cache-only options.
Adds GET and POST for public data (e.g. tarchives!).
Refactors for CachingClient to break into several files for each category.
The big news in this release is adding cache-only uploads. These allow a couple of cool features:
Easy to upload new chunks, files, public archives, etc, then try them out locally before uploading to the Autonomi network. Once uploaded, the REST API will return a regular XOR address, just the same as when uploading to the network. AntTP will then serve these chunks too! For devs, this is really handy and shortens the cost/feedback loop. For users, devs can use this to provide previews, etc, easily.
More REST API endpoints are now cache first, upload after. This shortens the gap between the changes being sent and made available locally. It paves the way for upload queues too (more on that another time).
How can you upload files? I’ve improved the swagger UI integration and completed some refactors to make it easier to use. Running locally, you can hit swagger UI through this URL: http://localhost:18888/swagger-ui/#/
To upload a file (such as a tarchive), you can go to the post public data page (http://localhost:18888/swagger-ui/#/public_data_controller/post_public_data), hit ‘Try it out’, then choose your file from the picker, leave ‘x-cache-only’ as ‘memory’, then ‘Execute’. The response below the form will include ‘address’, which you can then navigate to.
Some screenshots with an example (using @safemedia 's work! :D):
Edit: for the eagle eyed, there are a few rough edges to be taken off the directory listings (weird duplicates, etc). It works ok, but just needs a bit of polish.
