I’ve just pushed a new version which:
- (sn_httpd) Tidied up the code since disabling streaming, which should improve download times
- (sn_httpd) Uses latest autonomi libs
- (docker) Silences noisy logs (sn_networking mostly)
8 Likes
Submitted a feature request: Feature(API): Allow data streaming for gets · Issue #2421 · maidsafe/safe_network · GitHub
8 Likes
Begone! I had some wrestling with streams in Rust, and I don’t wish this on anyone 
I’d like it to be more like an implementations of Read and Write.
Check out the Dev Forum
2 Likes
I’ve made a number of improvements and released a new version (v0.3.5) of sn_httpd.
sn_httpd binaries can be found here: Release v0.3.5 · traktion/sn_httpd · GitHub
docker images can be found here: https://hub.docker.com/repository/docker/traktion/sn_httpd/general
Docker is probably the easiest to get going!
Changes include:
- You can now upload an archive, then use this as a container for retrieving the named files within. See sn_httpd/README.md at master · traktion/sn_httpd · GitHub for instructions.
- Disabled app-conf.json, as this is mostly provided by archives now (more to come though).
- Renamed ‘safe’ to ‘xor’ for retrieving XOR addressed files directly, e.g. http://localhost:8080/xor/8f2c02dbee730807d93ee12495eed6e36d2376b75402f01ea580eca97c13dc0a (direct mode) or http://xor.autonomi/8f2c02dbee730807d93ee12495eed6e36d2376b75402f01ea580eca97c13dc0a (proxy mode)
- Various performance improvements, including DNS caching, not needing app-conf.json to load, etc.
- Streaming is still disabled (sadly!) until the API matures. Quick start videos can still start early though, I believe. This page still has a video (ignore the many broken links… I need to update the page! ha!): http://localhost:8080/387f61da64d2a4c5d2e02ca34660fa2ac4fa6b3604ed8b67a58a3cba6e8ae787/streaming.html (direct mode) or http://traktion.autonomi/streaming.html (proxy mode)
- Logs are still nice and quiet, so you can see what is going on!
Images and MP3s, etc, seem to load in a couple of seconds or so from my local docker instance now, which is pretty cool.
I hope you like it and more changes are to come! I want to have a play with directory listings and file uploads again too.
Some screenshots!
16 Likes
I’ve pushed a new release (v0.3.6) with the following release notes:
- Adds directory listings when only the archive address is provided (but not filename).
- Adds ETag caching for files, to boost performance. If a browser/client already has the file cached, it will just check the ETag to confirm it is unchanged, then render it.
- Temporarily updated archive lookup logic to do a brute force search of only the file name. The archive key currently includes the original upload directory, which makes it difficult to resolve directly.
Release binaries: Release v0.3.6 · traktion/sn_httpd · GitHub
Docker latest updated too: https://hub.docker.com/r/traktion/sn_httpd
I updated the test file to access some of the test files folks have uploaded too (assuming sn_httpd is running on localhost:8080): http://localhost:8080/c8430f24e51ad9225c48a7e9c4cb8fcd9fec19066c178fe1458847b85e670597/streaming.html
Obligatory screenshots:
(Yes, I know… too many tabs! I was in a rush! 
)
8 Likes
Fyi, I managed to get Foxyproxy working on android with Firefox. It lets me whitelist a specific *.autonomi URL pattern to send to sn_httpd.
As android proxy settings are usually system wide and exclude only, this is great news.
I can now dedicate firefox to autonomi only or let it do clearnet and autonomi. Then chrome (or whatever) can still be used for regular clearnet browsing.
Why does it matter? It is cool to use DNS style resolution instead or xor addresses. It is gives us an easy way to do this on android.
(Vader is just my wee server at home)
Also, as android can run sn_httpd in termux, out of the box (just turn hotspot on to get a dns resolver for seed servers), it gives us more options. Both the browser and server can run on the device, avoiding external hosting, etc.
In fact, we can also resolve everything in a path only way, e.g. http://autonomi/traktion/streaming.html
There could be security issues/benefits, but interesting option.
9 Likes
fantastic work @Traktion iv got a docker set up just having a play with it now 
5 Likes
Yes, fantastic work on that. I’ve been moving to a localhost server approach myself as a custom browser (awe style) is out of reach. So very interested in how you are handling this.
My new URL scheme seems clunky but was designed to support versioning in a regular browser. I’d like to have feedback on it sometime in case there’s a better way.
Great to see this @Traktion, very cool indeed.
BTW I have just published the first version [cough] of an API for handling, er… versioned data (see dweb / trove::TroveHistory on crates.io). Even though it’s going to need a rewrite for Transactions it should be fine pretty much as is, though I’ll make the terminology more generic (ie remove refs to Registers) so you might find it useful.
7 Likes
I’ve been meaning to take a look at AWE addressing approach. It definitely feels like an area to align on!
So far, sn_https just looks first 2 levels of hierarchy, with the first being the archive and the second being the file within it. It used to be a bit more conveluted, but with the archives available in the API, it makes life a lot easier.
So, http://localhost:8080/archive/filename is the same as http://autonomy/archive/filename or http://archive.autonomi/filename, etc.
Edit: fwiw, the middle one is neat… narrowing hierarchy is cool. If we don’t have to worry about CORS, certs, etc, it would be nice. I’ve not dedicated thought to those implications yet though.
Sub directories in an archive should slot into the middle easily enough, but not tested that yet! 
The archive itself can either be XOR or a ‘name’ stored in a register. The name lets us point to the latest archive xor address.
Under the hood, there is a public root name list (register) for pointing a another private app list (register). The latter can be appended (updated) to the latest archive one update.
So, the idea is, first come first serve for public root names. Then the names can be privately managed to point where they want to.
It’s pretty rudimentary, but works… sort of. I rely on CLI to update the registers atm and the history was funky last time I messed with it. I’m assuming transactions can solve that though.
Performance wise, sn_httpd now loads all names into a hashmap on startup. That may not scale, but ok for now. In theory, the list could be stored on a flat file (or archived there) and maybe the whole thing could be spun off into a resolver app… maybe even an existing DNS resolver could be bent to fit?
I’ll take a look at the crate too. Will be interesting to see! I need to do the crates thing at some point too… not enough time in the day!
8 Likes
Thanks for that. I won’t get into questions just yet. Ideally I’ll write an explainer for key design areas and you could comment on that, compare to your approach etc. I find it helps to write things up, because I gain a better understanding of what I’ve done!
Any feedback or questions you have for me will be welcome if you do look at dweb or awe.
awe now uses dweb but I’ve not pushed that change yet so you will see dweb within awe until I do. I chose not to use Archive but stick with my own type for now. I don’t think they have had time to think things through so my types are not much more complex but much more capable and shouldn’t need to change for a while, hopefully never. Famous last words. I use a single type to cater for both a file system and a website style application (so providing for extending metadata for files, web redirects etc).
On CORS, I’ve not looked either because I wasn’t needing to. Moving to the browser I will, but am hoping it might help plug leaks from Autonomi sites to the conventional web. I really don’t know if that is likely though!
Publishing to crates.io is fairly painless BTW, good instructions and helpful error messages make it straightforward.
5 Likes
Changes have now been tested and pushed to awe and dweb, including updates for the upcoming renamed Autonomi modules.
That means others can’t easily build them, but the code can be explored.
Next I’ll be moving awe functionality to dweb-cli at which point awe will be archived.
Versioning has been broken by the removal of the merkle_reg from Registers (in the upcoming Autonomi release) so I’m waiting for Transactions to become useable in order to restore that.
3 Likes
I’d like to try this. I’m on a chromebook in Linux Developer mode and installed docker.io. Did docker run --rm -it -p 8080:8080 traktion/sn_http, and it seemed to be working until this line:
[2024-12-24T01:08:13Z ERROR ant_bootstrap::contacts] Failed to get bootstrap addrs from URL http://139.59.198.251/bootstrap_cache.json: reqwest::Error { kind: Request, url: “http://139.59.198.251/bootstrap_cache.json”, source: TimedOut }
Can you walk me through the steps I need to take to get there?
2 Likes
@Helen glad to see you are trying it out!
I was literally pushing new docker images last night, as the latest network was incompatible.
From looking at those log entries, you just need to wait a while until it logs that it has ‘Started listener’.
The autonomi libs seem to take a bit longer to bootstrap now, but it should get there.
3 Likes
Trying it now… lots of “Failed to dial” errors… just let it run?
Yup! It seems a bit noisy on the error logging in the new library. After about a minute, it should be ready.
Okay! Still waiting for it. It’s been hanging here for a while:
[2024-12-24T19:51:49Z INFO actix_web::middleware::logger] 172.17.0.1 “GET /autonomi/4b5efdacd5d87da24f72fe6ddb214ab9df7aa875b198a20ccdbdc5dc6d7a0d6a HTTP/1.1” 404 39 “-” “Mozilla/5.0 (X11; CrOS x86_64 14541.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36” 2.093666
Looks like it is listening if you’re getting those logs. You should get some messages about downloading chunks.
EDIT: Try this URL if that doesn’t work:
http://localhost:8080/7a8554af57f3664c1f7896cae2394119d8ead2d49e736b060d8a66fc6a2befa1/BegBlag.mp3
Yeah, it’s working! So cool. Now, how can I develop a web page that can be viewed by others?
3 Likes
If you want to start with a static site, you can upload a directory files as a ‘public archive’.
You may have figured it out from the logs, but the directory in the link above is the archive. If you go to the directory instead, you will see a listing (of only 1 file in this case):
http://localhost:8080/7a8554af57f3664c1f7896cae2394119d8ead2d49e736b060d8a66fc6a2befa1/
If you upload an archive, you will see a bunch of files in the archive and you can link any of them.
I was waiting for a stable network again to do a bit more dev to allow ‘index.html’ to auto-load when available. I’m also having issues uploading archives at the moment, as it’s taking out my router, which is adding to the dev challenge!
Anyway, if your uploads work, you should be able to see how it comes together.
EDIT: To add, you can upload a directory as a public archive using:
ant file upload -p -x mydirectory
5 Likes
It seems to not be allowing uploads due to not recognizing my wallet. I tried setting SECRET_KEY as well as importing it… any insight?