You certainly can run AntTP in a container. I create a docker version, which I think some folks use.
Locally, I just run the binary directly, mind. It doesn’t have any special dependencies, so it’s pretty easy.
2 Likes
A useful reference link for those wrestling with HTTP3 QUIC over UDP to get it to perform properly… not for the feint of heart. 
read this one too, if you really want to go down the HTTP3 QUIC rabbit hole..
Just an update on the Proposal: Tarchive experiment.
I’ve implemented a POC locally, which uses a public archive to wrap an archive.tar and an archive.tar.idx file. The former is a regular tar file and the latter is an index file generated through tarindexer.
This builds on the LFU caching that was added in the prior release, so that reading the same chunk multiple times is very rapid within AntTP.
I used IMIM as the use case, which has about 12 key files to load when loading a blog. These include the public archive, the app-config.json (for routing), the CSS and JS files, the index.html, a couple of fonts. While Angular could do better at squashing these into fewer files, it does a decent job here.
For 12 files, at 4 chunks each (3x data, 1x datamap for all files under about 12 MB), that’s about 48 chunks to download.
Using the tarchive POC, a maximum of 12 chunks were downloaded - the public archive (4), the archive.tar (4) and the archive.tar.idx (4). Loading time was substantially reduced (especially while the network is rather slow).
While caching masks the subsequent requests, whether 12 or 48, the initial load is much more responsive. As soon as I have the index.html file, the rest take about 1ms each to serve (i.e. as fast as AntTP can pull them from cache).
So, when will it be released? Well, with the POC showing good results, I wanted to make some further improvements first.
Instead of using a public archive to contain the tar + idx, I want to just upload the whole tarchive as a single file. For ease of implementation, the strategy will be:
- Create a tar file for the target directory
- Run tarindex to generate the index
- Concatenate the index to the tar file to create a tar.idx.img (or whatever name)
- Upload tar.idx.img to autonomi
Why not just create a tar file with it all in? Well, there is no index at the start of the tar, which means I would need to navigate to the end of the file, then figure out where the index starts. I could reserve a specific index size or make the last line an index size or some such, but I figured I could make my life easier.
How? Well, a tar file includes its size in its header. It’s always in the same offset. So, it’s trivial to retrieve this value, then use it as an offset against the full/concatenated file, to return the index. No need to reinvent the wheel, right? The network doesn’t care what is in the chunk, so it should work well.
So, I just need to implement the above and then I can read everything from one set of chunks - that’s 4 in total (for archives up to about 12 MB). For larger archives, searching to the end of the file may result in pulling another 4 chunks, but that’s still pretty efficient. For IMIM, the whole app fits easily within 12 MB, so it should be very suitable.
What about modifications? Well, the original tar file could be appended to, then reindexed/concatenated. Rinse and repeat.
I have visitors over the next few days, so I may not have time until next week to look at it. However, I thought I’d share the results, as they’re pretty exciting for performance. Getting a whole app in a mere 4 chunks should result in speedy loading times!
Edit: realised that the size is only for the first file in the archive. No bother, I’ll read the last file of the archive instead!
8 Likes
Quick update. Latest AntTP isn’t compatible with main net (at least not for me).
I have some other changes to add, relating to the above and will built them together. I have guests staying, so apologies for my tardiness.
4 Likes
New v0.10.0 release is out!
- Tarchive integration, which allows either public archives or tar archives as wrappers for files.
- Public archive and tar indexes are loaded using chunk-streamer and the chunks are new cached between calls (improves performance).
- Optimisations and fixes for hybrid cache.
- Bumps autonomi client library to 0.5.3 and disables library cache (in favour of LFU cache in AntTP).
- Change traktion-blog to a pointer to improve reliability/performance.
- Fixed utopio path definition for updating registers.
As above, this version implements tarchive support, which is a big set of changes.
Why are tarchives interesting and what do they provide?
A tarchive is simply a tar file, which includes a tar index (archive.tar.idx) as the last entry. While this format can support any size of tar file, with any size of files within them, there are big advantages for small files.
How do small files benefit? Instead of uploading each file separately, as a minimum of 4 chunks (including data map), there is also the cost of the public archive chunk too. What does this mean? For IMIM, it reduces the number of chunks down from about 48 chunks to 4!
This means the upload price is about 10x cheaper! Not only that, but you only need to download 4 chunks to have the full IMIM application ready to use.
For larger archives, such as my IMIM blog (which includes lots of pictures and several videos), there is also a gain. As the article files and images are small, they benefit as above. This also saves money and improves performance too.
How about deduplication and modifying these tarchives? As the only rule is that the last file in the tar file must be archive.tar.idx (as created through tarindexer), anything else can also be appended too, without changing the rest of the file. This means large tar files are still deduplicated against your previous versions, allowing easy/cheap modifications.
To create a tar file, you use the regular syntax in Linux/Unix:
cd mydirectory
tar -cv ../archive.tar ./
cd ..
tarindex -i archive.tar archive.tar.idx
tar -rf archive.tar archive.tar.idx
ant file upload -p archive.tar
Then you use the last address returned by ant file upload as you would a regular public archive with AntTP - you just use it as part of your path.
To immediately dog food the changes, I’ve updated the pointers for imim and traktion-blog to point to their new tarchives. You should see the performance difference pretty quickly.
Without pointer resolution (to take them out of the equation), here is a direct comparison between public archives and tarchives for both IMIM and traktion blog (apologies for small text, but there are lots of files being loaded):
Public Archive (46 seconds from cold AntTP start with wiped immutable cache and no browser cache):
Tarchive (15 seconds from cold AntTP start with wiped immutable cache and no browser cache):
Obviously, this is worst case, as subsequent calls are either cached by the browser or AntTP. However, first loads set an impression and browsers will clear/rotate out caches. Having a solid performing base is very desirable.
I’ve very happy with a solid 3x speed improvement on a pretty comprehensive SPA (single page app). If the tar was only appended to with files grouped together for specific blog posts, I’d expect the speed to be even better, as they would all be within the same bunch of chunks.
Note that most of the images load in the background on the index and the user can use the page much more quickly.
If you want to try it out for yourself, download and install AntTP, install (say) Firefox and configure the proxy (see README), then navigate to http://imim/blog/traktion-blog*. Have fun!
EDIT: Updated to v0.10.1:
Github: Release v0.10.1 · traktion/AntTP · GitHub
Dockerhub: https://hub.docker.com/r/traktion/anttp
Cargo: cargo install anttp
** NOTE: IMIM doesn’t like creating blogs from short cut names (that is an IMIM issue that needs to be improved though! 
)
12 Likes
Nice to learn that about deduplication.
Be interested to know how your Pointers perform as you update over time.
Interesting work and nice to have more options for file handling.
7 Likes
Pushed v0.10.1 out to solve this issue:
- Fixed issues with range queries not returning correct from/to/size values.
- Fixed issues with overflows/conversions.
Github: Release v0.10.1 · traktion/AntTP · GitHub
Dockerhub: https://hub.docker.com/r/traktion/anttp
Cargo: cargo install anttp
8 Likes
Yes, I wanted to make sure deduplication was retained and given tar files are designed to be append only for writing to tapes, it works really well.
I’ll keep track of the pointers, but they are only used when I update the IMIM app or a blog article is added.
I’m really pleased with the tarchive results. IMIM actually flies now, as after that 15 seconds or so, all the core stuff is cached and any other assets load quickly in the background.
Even the pointers are only taking a few seconds to resolve atm and I want to add an improved last-known-value cache. The idea being it will return the last retrieved value, while fetching the latest in the background. This would be great for name/blog resolution duties on URLs. Ofc, it will be configurable to disable for other nuanced uses, etc.
6 Likes
This is how dweb uses them for publishing versioned websites.
2 Likes
WOW that is so much faster and cheeper mind officially blown!!! @chriso tagging you in case you missed this tarchive magic maybe this is worth adding to the ant client as default for archives ?
here is my first tarchive website upload
http://localhost:18888/2c2a8a13ddfe88ad5166d3d3035fe423f30c8f14c29a01db64c5a6797e2a2b15/index.html
8 Likes
Nice! That felt pretty much like a regular site, speed wise, even on an on-site wifi network! 
Literally, it took a moment to resolve, them boom, it was all loaded! 
Subsequent requests will also be cached using both eTags (client side) and immutable LFU cache (AntTP side). So, the first load will be the slowest.
7 Likes
you are here by awarded the BMF of the week 
yes that felt like a normal site loading and all packed into 4 chunks for a bargain price
9 Likes
you need to test @aatonnomicc’s website locally on your internet connection 
- it takes some moments to fetch the archive but then it’s 
for static stuff this is really a whole different playing field suddenly
(won’t speed up dynamic apps like friends a lot where you know the friend addresses only after fetching your account package and the profile pics of your friends after you fetched their profiles … etc etc … but for static sites it’s super awesome!)
6 Likes
It’s worth noting that there is a category of dynamic sites that don’t necessarily use mutable types, but still have a client side app loading data on demand. These also benefit greatly from tarchives.
For example, IMIM is a SPA (single page app), which loads some content via traditional static HTTP requests, but then the rest via JavaScript HTTP requests. The latter can also be immutable data, but the former can pick/choose/react to that data.
So, we’re not restricted to static, server side, page generation solutions to get these gains. We can have pretty rich sites (like IMIM), which is essentially a Typescript Angular app running in your browser, pulling resources depending on what the user asks for, etc.
Maybe the above is obvious, but I just wanted to point out the power of immutable types, how they can be grouped, cached, etc, while still giving a highly interactive user experience.
Obviously, pairing the above with mutables is a whole extra dimension. However, at least in the case of IMIM, each blog is just a tarchive too, containing articles and media which can be cached forever.
Note that as soon as I have last-known-value support for mutables, fully offline mode should be a step away too, with AntTP caching all content on your disk indefinitely (or at least until your space allocated runs out, which means stuff gets trimmed away). The client libs also include immutable caching now too, but afaik, Foyer (AntTP caching library) is more advanced and flexible.
The final step will be to allow AntTP to ignore client connectivity issues, i.e. the client becomes optional. I want to set an idle timer too, then close the client connection. This will save power, which will be great for hosting and especially mobile devices (as the client is quite busy maintaining connectivity). Coming in a version soon, hopefully!
10 Likes
yes you’re right - I guess this is worth to point out
I included the JS reactivity stuff in “non dynamic apps” but this doesn’t mean “static site” in the sense this term is typically used … in contrast to
Apps with client side routing.
…the non-speedup-effect is only where we explicitly traverse along mutables on autonomi (and I should e.g. move one write operation for session management that is being executed on friends login into the background too that is currently slowing down the login process significantly but wouldn’t need to be a foreground process …)
typical read times for mutables on mainnet are in the area of 1-5s from what I observe here … only the write stuff is taking forever (10+ seconds)…
5 Likes
Thanks.
I had seen some of these proposals before and I meant to pass them on. I’ve now done so with this one.
Qi thinks it’s a nice optimisation we could make, though he said he thinks it might be opt in rather than the default behaviour.
11 Likes
FWIW, the current implementation is very much ‘layer 2’, in so much as it just interprets files stored in a specific way. The uploads are just tar files with a convention of storing a tar index and the downloads are just the reverse.
You could create a more native solution where the data map contained the tar file list, offsets and sizes too. I’m assuming that would need a specific data type, which could make it simpler to integrate. Given it would keep the index within the data map chunk (instead of within the self-encrypted chunks), it would also be faster to lookup files (for existence within the tar), especially for larger tar files, where the ‘tail’ chunks also need to be read.
Still, I think it is good that we can do this sort of stuff without adding more data types. That shows we have a fairly flexible platform to build on and useful ideas can be more tightly integrated later on, if needed.
12 Likes
I would definitely make it opt-in and not default. It has consequences on dedup, and as @Traktion says is more of a higher layer than the normal Autonomi functioning.
8 Likes
For anyone who doesn’t have AntTP installed, you can see what this is like through the AntsNest gateway here:
Note that there is some stuff on this site that is still pulled over clear net, but the core content isn’t. It could certainly be changed to put the rest of it in the tarchive too.
Edit: It also illustrates the immutable data caching, as you will probably notice more! for public proxies, it is even more powerful.
10 Likes