New v0.10.0 release is out!
- Tarchive integration, which allows either public archives or tar archives as wrappers for files.
- Public archive and tar indexes are loaded using chunk-streamer and the chunks are new cached between calls (improves performance).
- Optimisations and fixes for hybrid cache.
- Bumps autonomi client library to 0.5.3 and disables library cache (in favour of LFU cache in AntTP).
- Change traktion-blog to a pointer to improve reliability/performance.
- Fixed utopio path definition for updating registers.
As above, this version implements tarchive support, which is a big set of changes.
Why are tarchives interesting and what do they provide?
A tarchive is simply a tar file, which includes a tar index (archive.tar.idx) as the last entry. While this format can support any size of tar file, with any size of files within them, there are big advantages for small files.
How do small files benefit? Instead of uploading each file separately, as a minimum of 4 chunks (including data map), there is also the cost of the public archive chunk too. What does this mean? For IMIM, it reduces the number of chunks down from about 48 chunks to 4!
This means the upload price is about 10x cheaper! Not only that, but you only need to download 4 chunks to have the full IMIM application ready to use.
For larger archives, such as my IMIM blog (which includes lots of pictures and several videos), there is also a gain. As the article files and images are small, they benefit as above. This also saves money and improves performance too.
How about deduplication and modifying these tarchives? As the only rule is that the last file in the tar file must be archive.tar.idx (as created through tarindexer), anything else can also be appended too, without changing the rest of the file. This means large tar files are still deduplicated against your previous versions, allowing easy/cheap modifications.
To create a tar file, you use the regular syntax in Linux/Unix:
cd mydirectory
tar -cv ../archive.tar ./
cd ..
tarindex -i archive.tar archive.tar.idx
tar -rf archive.tar archive.tar.idx
ant file upload -p archive.tar
Then you use the last address returned by ant file upload as you would a regular public archive with AntTP - you just use it as part of your path.
To immediately dog food the changes, I’ve updated the pointers for imim and traktion-blog to point to their new tarchives. You should see the performance difference pretty quickly.
Without pointer resolution (to take them out of the equation), here is a direct comparison between public archives and tarchives for both IMIM and traktion blog (apologies for small text, but there are lots of files being loaded):
Public Archive (46 seconds from cold AntTP start with wiped immutable cache and no browser cache):
Tarchive (15 seconds from cold AntTP start with wiped immutable cache and no browser cache):
Obviously, this is worst case, as subsequent calls are either cached by the browser or AntTP. However, first loads set an impression and browsers will clear/rotate out caches. Having a solid performing base is very desirable.
I’ve very happy with a solid 3x speed improvement on a pretty comprehensive SPA (single page app). If the tar was only appended to with files grouped together for specific blog posts, I’d expect the speed to be even better, as they would all be within the same bunch of chunks.
Note that most of the images load in the background on the index and the user can use the page much more quickly.
If you want to try it out for yourself, download and install AntTP, install (say) Firefox and configure the proxy (see README), then navigate to http://imim/blog/traktion-blog*. Have fun!
EDIT: Updated to v0.10.1:
Github: Release v0.10.1 · traktion/AntTP · GitHub
Dockerhub: https://hub.docker.com/r/traktion/anttp
Cargo: cargo install anttp
** NOTE: IMIM doesn’t like creating blogs from short cut names (that is an IMIM issue that needs to be improved though! 
)