Sn_httpd - Serving Autonomi data over HTTP

I thought I should create a thread for sn_httpd, as I’m developing and testing various features on the test nets. A separate thread may help to consolidate sn_httpd related setup/config/testing results and also to keep track of new features I’m adding.

Note that this is not a Maidsafe project, but it is open source! It’s also highly experimental, as it’s my first Rust project, has zero unit testing and is a bubbling hotbed of me messing about to see what works!

So, what is sn_httpd? If you’ve not tried it out yet, it’s a HTTP daemon for Autonomi (aka, Safe Network - maybe a new prefix is needed? :smiley:). This allows regular web browsers to interact with Autonomic, including rendering web sites, images, streaming music, streaming video, etc.

You can run sn_httpd locally and have your own, private, web server to interact with. Or, you can run it on a remote instance, then allow access to yourself and others.

If you run it locally, it can also act as a forward web proxy for Autonomy. What is a forward web proxy? It is a service that sits between your browser and the Autonomy network. This allows any browser to interact with Autonomi and allows (experimental!) DNS to resolve names for Autonomi apps/sites.

If you would like to use the proxy feature, I’d recommend using a second browser for Autonomi and manually setting the HTTP proxy to 127.0.0.1:8080 (or whichever port you are running the local instance on).

For Firefox, the settings look like this:

As all requests on the browser using the proxy are forwarded to sn_httpd, the hostname can be pulled from the URL and used as a DNS style name to resolve. At this time (experimental!), Autonomi host names must end with ‘.autonomi’, e.g. http://traktion.autonomi/streaming.html.

Alternatively, localhost can be referred to directly to avoid and DNS style resolving, by using /safe/ URL formats, e.g. http://localhost:8080/safe/3509bad03dc869dec883c7b44662c3503d2517fa9e828bb64f4dbe719d3837bf.

If you would like to try it out, you can compile from source here: GitHub - traktion/sn_httpd: Safe Network httpd gateway server

Or download binaries here: Releases · traktion/sn_httpd · GitHub

Or pull the latest docker container here: https://hub.docker.com/repository/docker/traktion/sn_httpd/general

Q & A

Q: How do file names map to Autonomi XOR addresses?
A: The DNS style name (e.g. traktion.autonomi) points to a app-config.json file, which provides a list of path/XOR maps. There is more information here: GitHub - traktion/sn_httpd: Safe Network httpd gateway server . Note that this format is experimental and set to change (to add cache TTL, MIMI type overrides, etc), but it’s great for trying stuff out in POC form.

Q: Can I set the proxy on my regular browser and use it for both clear net and Autonomi browsing?
A: With the standard build, this is currently disabled. It can be enabled by toggling PROXY_ENABLED in code, but there are some build issues on Arm platforms. However, using a separate browser prevents leakage between clear net and Autonomy, which may be desirable from a security perspective.

Q: Can I stream video or music straight from Autonomi to my browser?
A: Yes! I’ve tested this with Chrome and Firefox on Ubuntu, as well as Chrome on Android mobile. It uses 206 partial content responses to communicate smoothly with the browsers and allows you to start watching/listening stuff almost immediately.

Q: I can run this on Android too?!
A: Yes! If you install Termux, you can run the sn_httpd Android binary directly from the console. No need to jail break phone, etc. You can then access localhost or set it up as a proxy (in a second browser). I plan to experiment with making sn_httpd and Android service that can just be run in the background easily, but it’s early days. Termux lets us play with the concept though! Some notes here: PunchNet [24/04/24 Testnet] [Offline] - #339 by Traktion

Q: Will there be a dedicated Autonomi web browser?
A: Yes, I’m sure there will! This offers a quick and simple way of getting content to a regular browser though and could help to feed in ideas on how both could live alongside one another. Building and maintaining a browser requires more effort than creating a HTTP service wrapper around Autonomi.

Q: Are there non-browser use cases for sn_httpd?
A: Yes! If you’re building out apps which exist in the clear net, but want to start using elements of Autonomi, sn_httpd can help to bridge the gap. HTTP is simple to integrate with, requiring no special libraries or languages. GETs and PUTs are supported, along with routing for SPAs (single page applications). In short, any backend could talk to a sn_httpd instance running nearby and store or retrieve data from Autonomi.

Q: How does SPA routing work?
A: See this example here: sn_httpd/app-config.json at 5abbee4948d65b10c65b99674ad9cbbf5b21e5ac · traktion/sn_httpd · GitHub . In short, URLs with pattern matches can be directed to your SPA root, allowing it to respond to requests directly, just as with the clear net. This allows Autonomi to be the backend to your SPA (e.g. Angular, React, etc) app/site, but communicating directly with Autonomi instead of needing a dedicated REST backend.

Q: Can I build an SPA app that doesn’t need a HTTP web service running somewhere?
A: Yes! The IMIM blogging platform is an example of this, which doesn’t use any clear net infrastructure other than sn_httpd. If sn_httpd is run on the client (e.g. as a proxy), then no clear net infrastructure is needed at all!

35 Likes

very impressive. bravo :clap: :clap:

12 Likes

Fantastic work.

I know opening content hosted on current test networks from my phone browser via your proxy was a highlight to me of how powerful this network will be. Well done and thanks for putting this out there!

8 Likes

Some minor updates to create 0.3.3 of sn_httpd, which works with the Beta Rewards Network:

I’ve also created a new docker hub repository, which is called sn_httpd (rather than i-am-immutable, which is something separate now):

https://hub.docker.com/r/traktion/sn_httpd

I don’t have any nanos to test uploads etc, but here is a sample uploaded by @aatonnomicc

Using a proxy:

Using direct (localhost):

12 Likes

hmmm - when trying to follow your dockerhub link it wants me to login to docker - that’s the link for everyone i guess:

https://hub.docker.com/r/traktion/sn_httpd

so here comes the docker run cmd:

docker run --rm -it -p 8080:8080 traktion/sn_httpd

if you then visit:

http://localhost:8080/safe/4b5efdacd5d87da24f72fe6ddb214ab9df7aa875b198a20ccdbdc5dc6d7a0d6a

you can play ANT INVASION on autonomi!

ps: yes my ants have green blood and no feet

10 Likes

Glad it’s working well! I’ve updated the link! :sweat_smile:

3 Likes

awesome work with the proxy! this really is cool!

3 Likes

I thought the proxy made it a bit more user friendly, especially as you can set one browser to use a proxy and keep another for clear net.

I’m not sure if I documented it yet, but it allows the root directory to be shifted into the URL too, e.g. http://localhost/safe can be addressed as http://safe.autonomi.

Annoyingly, not one Android browser lets you set the proxy. It’s only system wide. I was hoping install a second browser that just talked to the proxy running on my phone, but it isn’t that simple. Maybe some sort of VPN mode could make it work, but I’ve not dug into that yet.

The long game is to provide an easy alternative to a native autonomi browser, btw. If a proxy and a regular browser does the job, the rest is relatively simple.

8 Likes

There was a conversation on live video streaming recently and I was having a play with OBS for recording videos. I’ve confirmed it has an option to output a video to multiple files. I’m sure other apps can do the same, but OBS is free and open source, which is great!

It should be relatively trivial to setup a directory watcher, then upload new files (as chunks with XOR addresses) to Autonomi as they are produced. These could then be added to a register (or an alternative) to allow consumers to read the chunks as they become available.

Using an appropriate client (need to dig some more for that), it should be possible to stitch these files together to allow live streaming. Using sn_httpd and an appropriate web streaming client, it sounds like it should all be feasible.

Edit: Alternatively, sn_httpd could actually stitch the files together as a stream them. That would be simple for the client (just a regular stream). @happybeing once the POC works, we may want to align on an approach too, but should be easy to change.

I’ll start having a play with this next when I have some free time and the beta network allows be to experiment more easily.

10 Likes

I thought that too when I first read this. Hopefully it will be easier to think about next steps once we have apps on a usable network. I can either keep working on improving the browserness or I could separate the web stuff into a crate in case it is a useful basis for you and others, but I don’t plan to do that until I have something decent and others express interest in building alongside it.

Even just streaming video in pages will be impressive, so maybe we can start with me including that, or perhaps someone from the community who is inspired to build websites once they see how easy and effective it is.

7 Likes

@Toivo Example these can be streamed from the network through @Traktion proxy.

http://safe-ntracking.ddns.net:8080/safe/c0893739a688b739aac5b75ac845b6c428c21b73dc782352bb1ad529495a87af

http://safe-ntracking.ddns.net:8080/safe/3509bad03dc869dec883c7b44662c3503d2517fa9e828bb64f4dbe719d3837bf

2 Likes

The guys on the Media over QUIC discord channel might have something useful…

2 Likes

They are very nice as themselves, but I feel they need too much talk alongside to explain what’s really going on. Good case, but not enough show to be a good showcase for my target group.

2 Likes

Are you looking for something more like a step by step tutorial, so that anyone can set it up? The documentation is quite light at the moment, mostly just because I’ve been hacking away at it in my spare time. However, I could spend some more time on this if useful.

If it’s not a tutorial, could you be specific in what you need to impress others? Is it beyond the concept of sn_httpd or just examples of how it can be used?

Some use cases:

  • As a client proxy to easily point their web browser directly to Autonomi (per browser or per system). Allows regular browsers to access Autonomi sites/data.
  • As a companion service to run alongside a backend service (either within the same container, or adjacent to it). This removes the need to integrate apps directly with the Rust library, with HTTP being used instead.
  • Standalone web server to host data on Autonomi to traditional clear net web browsers (with no client changes needed).
5 Likes

For my audience it would need to be something like an app you can use to read Project Gutenberg books for example. Or a way to upload your own files.

I think you do very good job, but there’s not much to offer for my audience. With these I would say something like:

“I know this music is not for your taste, but it’s actually impressive, because behind the scenes there is a very cool autonomous network you can participate…”

Also the fact that it works in your normal browser is a hindrance, because I need to tell that “actually it’s not your normal internet even though it seems so…”

But again, there’s absolutely nothing wrong with your work, it’s just not promotion material for my target group. I would like to have an app that looks somehow distinctive. Maybe a Gutenberg reader with Autonomi backend?

2 Likes