I’ve just pushed a new version which:
- (sn_httpd) Tidied up the code since disabling streaming, which should improve download times
- (sn_httpd) Uses latest autonomi libs
- (docker) Silences noisy logs (sn_networking mostly)
7 Likes
Submitted a feature request: Feature(API): Allow data streaming for gets · Issue #2421 · maidsafe/safe_network · GitHub
7 Likes
Begone! I had some wrestling with streams in Rust, and I don’t wish this on anyone 
I’d like it to be more like an implementations of Read and Write.
Check out the Dev Forum
2 Likes
I’ve made a number of improvements and released a new version (v0.3.5) of sn_httpd.
sn_httpd binaries can be found here: Release v0.3.5 · traktion/sn_httpd · GitHub
docker images can be found here: https://hub.docker.com/repository/docker/traktion/sn_httpd/general
Docker is probably the easiest to get going!
Changes include:
- You can now upload an archive, then use this as a container for retrieving the named files within. See sn_httpd/README.md at master · traktion/sn_httpd · GitHub for instructions.
- Disabled app-conf.json, as this is mostly provided by archives now (more to come though).
- Renamed ‘safe’ to ‘xor’ for retrieving XOR addressed files directly, e.g. http://localhost:8080/xor/8f2c02dbee730807d93ee12495eed6e36d2376b75402f01ea580eca97c13dc0a (direct mode) or http://xor.autonomi/8f2c02dbee730807d93ee12495eed6e36d2376b75402f01ea580eca97c13dc0a (proxy mode)
- Various performance improvements, including DNS caching, not needing app-conf.json to load, etc.
- Streaming is still disabled (sadly!) until the API matures. Quick start videos can still start early though, I believe. This page still has a video (ignore the many broken links… I need to update the page! ha!): http://localhost:8080/387f61da64d2a4c5d2e02ca34660fa2ac4fa6b3604ed8b67a58a3cba6e8ae787/streaming.html (direct mode) or http://traktion.autonomi/streaming.html (proxy mode)
- Logs are still nice and quiet, so you can see what is going on!
Images and MP3s, etc, seem to load in a couple of seconds or so from my local docker instance now, which is pretty cool.
I hope you like it and more changes are to come! I want to have a play with directory listings and file uploads again too.
Some screenshots!
14 Likes
I’ve pushed a new release (v0.3.6) with the following release notes:
- Adds directory listings when only the archive address is provided (but not filename).
- Adds ETag caching for files, to boost performance. If a browser/client already has the file cached, it will just check the ETag to confirm it is unchanged, then render it.
- Temporarily updated archive lookup logic to do a brute force search of only the file name. The archive key currently includes the original upload directory, which makes it difficult to resolve directly.
Release binaries: Release v0.3.6 · traktion/sn_httpd · GitHub
Docker latest updated too: https://hub.docker.com/r/traktion/sn_httpd
I updated the test file to access some of the test files folks have uploaded too (assuming sn_httpd is running on localhost:8080): http://localhost:8080/c8430f24e51ad9225c48a7e9c4cb8fcd9fec19066c178fe1458847b85e670597/streaming.html
Obligatory screenshots:
(Yes, I know… too many tabs! I was in a rush! 
)
6 Likes
Fyi, I managed to get Foxyproxy working on android with Firefox. It lets me whitelist a specific *.autonomi URL pattern to send to sn_httpd.
As android proxy settings are usually system wide and exclude only, this is great news.
I can now dedicate firefox to autonomi only or let it do clearnet and autonomi. Then chrome (or whatever) can still be used for regular clearnet browsing.
Why does it matter? It is cool to use DNS style resolution instead or xor addresses. It is gives us an easy way to do this on android.
(Vader is just my wee server at home)
Also, as android can run sn_httpd in termux, out of the box (just turn hotspot on to get a dns resolver for seed servers), it gives us more options. Both the browser and server can run on the device, avoiding external hosting, etc.
In fact, we can also resolve everything in a path only way, e.g. http://autonomi/traktion/streaming.html
There could be security issues/benefits, but interesting option.
8 Likes
fantastic work @Traktion iv got a docker set up just having a play with it now 
4 Likes
Yes, fantastic work on that. I’ve been moving to a localhost server approach myself as a custom browser (awe style) is out of reach. So very interested in how you are handling this.
My new URL scheme seems clunky but was designed to support versioning in a regular browser. I’d like to have feedback on it sometime in case there’s a better way.
Great to see this @Traktion, very cool indeed.
BTW I have just published the first version [cough] of an API for handling, er… versioned data (see dweb / trove::TroveHistory on crates.io). Even though it’s going to need a rewrite for Transactions it should be fine pretty much as is, though I’ll make the terminology more generic (ie remove refs to Registers) so you might find it useful.
5 Likes
I’ve been meaning to take a look at AWE addressing approach. It definitely feels like an area to align on!
So far, sn_https just looks first 2 levels of hierarchy, with the first being the archive and the second being the file within it. It used to be a bit more conveluted, but with the archives available in the API, it makes life a lot easier.
So, http://localhost:8080/archive/filename is the same as http://autonomy/archive/filename or http://archive.autonomi/filename, etc.
Edit: fwiw, the middle one is neat… narrowing hierarchy is cool. If we don’t have to worry about CORS, certs, etc, it would be nice. I’ve not dedicated thought to those implications yet though.
Sub directories in an archive should slot into the middle easily enough, but not tested that yet! 
The archive itself can either be XOR or a ‘name’ stored in a register. The name lets us point to the latest archive xor address.
Under the hood, there is a public root name list (register) for pointing a another private app list (register). The latter can be appended (updated) to the latest archive one update.
So, the idea is, first come first serve for public root names. Then the names can be privately managed to point where they want to.
It’s pretty rudimentary, but works… sort of. I rely on CLI to update the registers atm and the history was funky last time I messed with it. I’m assuming transactions can solve that though.
Performance wise, sn_httpd now loads all names into a hashmap on startup. That may not scale, but ok for now. In theory, the list could be stored on a flat file (or archived there) and maybe the whole thing could be spun off into a resolver app… maybe even an existing DNS resolver could be bent to fit?
I’ll take a look at the crate too. Will be interesting to see! I need to do the crates thing at some point too… not enough time in the day!
6 Likes
Thanks for that. I won’t get into questions just yet. Ideally I’ll write an explainer for key design areas and you could comment on that, compare to your approach etc. I find it helps to write things up, because I gain a better understanding of what I’ve done!
Any feedback or questions you have for me will be welcome if you do look at dweb or awe.
awe now uses dweb but I’ve not pushed that change yet so you will see dweb within awe until I do. I chose not to use Archive but stick with my own type for now. I don’t think they have had time to think things through so my types are not much more complex but much more capable and shouldn’t need to change for a while, hopefully never. Famous last words. I use a single type to cater for both a file system and a website style application (so providing for extending metadata for files, web redirects etc).
On CORS, I’ve not looked either because I wasn’t needing to. Moving to the browser I will, but am hoping it might help plug leaks from Autonomi sites to the conventional web. I really don’t know if that is likely though!
Publishing to crates.io is fairly painless BTW, good instructions and helpful error messages make it straightforward.
4 Likes
Changes have now been tested and pushed to awe and dweb, including updates for the upcoming renamed Autonomi modules.
That means others can’t easily build them, but the code can be explored.
Next I’ll be moving awe functionality to dweb-cli at which point awe will be archived.
Versioning has been broken by the removal of the merkle_reg from Registers (in the upcoming Autonomi release) so I’m waiting for Transactions to become useable in order to restore that.
2 Likes