Dweb - web publishing, RESTful web apps, versioned data and

[happybeing]

d-day is here

From the maker of vdash, the incredible awe and hopefully soon to be fought over by budding website and web app builders. Here’s one I built months ago but haven’t been able to upload to the live network yet. This all works perfectly on local testnets:

All earlier versions can be viewed because dweb publishes to a History data type.

dweb

Immediately after installing dweb with a single command (cargo install dweb-cli), you will be able to do the following:

• view websites
• publish websites*
• create a WordPress-like blog
• use web apps
• build RESTful web apps using standard web tooling such as Svelte, React, Vue etc. You can view the REST APIs using dweb openapi-docs (“Autonomi RESTful APIs” below).
• inspect Autonomi data using dweb inspect- commands
• there’s more, and a lot more potential which you can read about in Current Features and Future Plans or see Contents if you want to browse.

Thanks to an intrepid guinea pig there is at least one website you can view right now!

View vroom

1. Get Rust
2. Install dweb: cargo install --locked dweb-cli
3. Start the server: dweb serve

There are two ways to view vroom and both will take ages to load (or fail more quickly) so watch the terminal where you started the server while your hair greys and your skin wrinkles. (Those messages are very exciting to some, honest).

Ready? Now open vroom via an Archive (accessing the website directly, using the address of a PublicArchive):

dweb open 58f30e22caae7354c2039f4f2690f6205c93b41da5391fe45f75f2f17c559eec

Open the vroom History - most recent version (currently two versions I believe)…

Most recent version:

(Note: accessing versions will take a loooong time because of issues with Pointers not being updated. But it does work. When pointers are sorted this will be much faster).

dweb open b4d001a4b826cf9cfca6101f35da73d21c2464ab8abf4d054ea99ecd9a57858f9c74854ae1acf856d244aba5d4a0ed96

Open version 1 etc by inserting -v <VERSION> after ‘open’:

dweb open -v 1 b4d001a4b826cf9cfca6101f35da73d21c2464ab8abf4d054ea99ecd9a57858f9c74854ae1acf856d244aba5d4a0ed96

Can testnet?

Yes, the intrepid guinea pig (you know who you are) can try many more things out on a local testnet, including things like:

• Publii - build blogs and websites using this third party desktop app
• Names - list, create and delete memorable names for Autonomi websites
• Fileman - view versioned files on Autonomi by address or name

Builders too, can experience the exquisite pain of trying to build real things with it and tell me what the RESTful HTTP API is missing (hint, most things but easy for me to add on request).

When mainnet?

Recent improvements to the network by Autonomi combined with changes to make dweb repeat upload operations until they succeed, whatever the cost (so watch your wallet!) are looking promising (today 2/4/2025).

So have a go. Anyone can upload a static website, and if you are into building a web app that can access Autonomi check out the dweb RESTful APIs next…

Autonomi RESTful APIs

I’m exposing dweb and Autonomi APIs using OpenAPI and a great tool called SwaggerUI. Here’s a screenshot of SwaggerUI which provides details on every API, and can also be used to test each one with the live network or a local testnet:

To open the above in your browser, install dweb and start the server:

cargo install --locked dweb-cli
dweb serve

Then in another terminal:

dweb openapi-docs

To use with a local testnet start the server with dweb --local serve and upload some websites using dweb publish-new and dweb publish-update. You can build and test web apps exactly as you would normally with the setup (i.e. running your web app using npm run dev or yarn dev etc.)

The example web apps Fileman and Names (above) make it easy to check this out too.

Developers.. help!

Please let me know if you want a particular API or if you think I’m doing it wrong. I’m starting from zero with REST API design so welcome suggestions, feedback and requests.

More details please

A slightly longer version for those who are still reading but don’t want to click the link above.

• open websites on Autonomi in a standard browser, with no configuration of the browser or anything else
• find websites by typing dweb open awesome
• view websites and directories published with dweb, AntTP, ant or any Autonomi upload tool or app
• view every version of websites published using dweb (and those made for AntTP and website made of static HTML uploaded using ant or a third party app) with dweb open <ADDRESS>
• publish your own website with a single command, issued over and over until your wallet is emptied and refilled numerous times
• create and use short names from the command line or the Names demo app
• view the contents of Archives (uploaded directories) using the command line or the Fileman demo app
• open any version of a website going back as far as, er, last week (forever!), because dweb History started in March 2025 and guess who got there F I R S T !!! (P.S. not me).

Links

• dweb github (lots of info)
• Names - a static HTML app for creating short names for websites
• Fileman - a RESTful Svelte app for viewing files on Autonomi
• Publii - a third party app for creating websites and blogs

---

*some features require storing data for which you will first have to unlock your ERC20 badge, though plenty of help will be available from this wonderful community as always. So don’t let that stop you if you want to be second, or third, or let’s face it anything in the top 100 will be something to tell your grandchildren about!

We know how much people being first (ish) that here

[bridge]

“Your talent, passion, and dedication will be remembered forever.”

[Southside]

Are your excellent efforts being sabotaged by an uncoperative network?

willie@gagarin:~$ dweb serve
Dweb Autonomi client initialising...
🔗 Connected to the Network                                                                                                                                                  -> public network evm-arbitrum-one
Registered built-in DWEB-NAME: awesome -> 8650c4284430522a638a6fa37dd3e8d610c65b300f89f0199a95a1a9eab0455287f8c8d137fad390654bd9f19b868a5c
dweb main server listening on 127.0.0.1:8080
DEBUG serve with ports HttpRequest : GET /dweb-open/v/58f30e22caae7354c2039f4f2690f6205c93b41da5391fe45f75f2f17c559eec/
DEBUG /dweb-open/v/58f30e22caae7354c2039f4f2690f6205c93b41da5391fe45f75f2f17c559eec/
DEBUG parse_versioned_path_params_with_as_name() v/58f30e22caae7354c2039f4f2690f6205c93b41da5391fe45f75f2f17c559eec/
1:v 2: 58f30e22caae7354c2039f4f2690f6205c93b41da5391fe45f75f2f17c559eec r: 
version:None as_name: anonymous, address_or_name: 58f30e22caae7354c2039f4f2690f6205c93b41da5391fe45f75f2f17c559eec remote_path: 
DEBUG directory_tree_download() at 58f30e22caae7354c2039f4f2690f6205c93b41da5391fe45f75f2f17c559eec
FAILED: General networking error: GetRecordError(QueryTimeout)
DEBUG Failed to fetch archive from network - General networking error: GetRecordError(QueryTimeout)
DEBUG serve with ports HttpResponse: 502 Bad Gateway  (Bad Gateway)
DEBUG serve with ports HttpRequest : GET /favicon.ico
DEBUG www_handler(/favicon.ico)...
DEBUG serve with ports HttpResponse: 404 Not Found  (Not Found)
DEBUG serve with ports HttpRequest : GET /dweb-open/v/58f30e22caae7354c2039f4f2690f6205c93b41da5391fe45f75f2f17c559eec/
DEBUG /dweb-open/v/58f30e22caae7354c2039f4f2690f6205c93b41da5391fe45f75f2f17c559eec/
DEBUG parse_versioned_path_params_with_as_name() v/58f30e22caae7354c2039f4f2690f6205c93b41da5391fe45f75f2f17c559eec/
1:v 2: 58f30e22caae7354c2039f4f2690f6205c93b41da5391fe45f75f2f17c559eec r: 
version:None as_name: anonymous, address_or_name: 58f30e22caae7354c2039f4f2690f6205c93b41da5391fe45f75f2f17c559eec remote_path: 
DEBUG directory_tree_download() at 58f30e22caae7354c2039f4f2690f6205c93b41da5391fe45f75f2f17c559eec
FAILED: General networking error: GetRecordError(QueryTimeout)
DEBUG Failed to fetch archive from network - General networking error: GetRecordError(QueryTimeout)
DEBUG serve with ports HttpResponse: 502 Bad Gateway  (Bad Gateway)

OTOH I must admit to also running @neo 's download script
on this box - that may be a factor

[happybeing]

Yep, @riddim can confirm vroom fails to load due to a network error. Not unexpected. It shall be a good measure of improving network health when it begins to load.

[Southside]

If I thought I could get a clean upload, I could do various new versions of vroom that would demonstrate the history etc.

But I am not seeing any successful uploads recently. I believe @aatonnomicc has had some luck though.

I may even start paying Hetzner again for a small box I can try uploading from just to eliminate the possibility its something at this end.

[happybeing]

Yeh, we should get there thooon. I have another tweak to do to help with this and will be back on the VPS to try things out. Maybe I just need to pay more than £12/month or whatever it is. Or send @Josh a CD

[riddim]

My vps is 5.50 - maybe that’s the reason I encounter issues

[happybeing]

My old one is about that, but I have another. Oddly, the cheap one has performed better at running nodes and earning for a while now so maybe I should be using that one.

[Josh]

Amazing as always @happybeing.

[Josh]

I am likely the first person ever to read a README but this link does nothing Mark.

@happybeing
should you not have dweb serve in here?

[happybeing]

Thanks for looking through and reporting these Josh. Keep 'em coming if you can bare it!

EDIT: @Josh fixes applied, thanks again.

[Michael_Hills]

Amazing work!

[happybeing]

I’ve updated dweb to v0.3.1 which includes the latest Autonomi v0.4.2 updates to the client (just published) so by all means see if this works better for uploads and downloads before I get the chance (while I’m busy for a couple of days). So…

cargo install dweb-cli

…but watch your wallet if you leave this running! It will keep going and going until all the Autonomi APIs have returned success!

For info on what this includes see:

Upcoming fixes have improved downloads

Both @Traktion and I have tried downloading with the improved Autonomi client API which are in the latest release candidate but not yet the ‘stable’ release (so not yet in ant unless you build yourself).

I used dweb to download and inspect the content of a directory both with and without these tweaks.

I was downloading a small directory (which @Southside created for his website) and found that instead of 3 of 5 downloads failing, 5 out of 5 succeeded, although they took a loooong time (1 to 3 minutes). @Traktion reported something similar.

This improvement in reliability is very good to see. I still can’t do anything over mobile b/b hotspot, but if you have a good connection and on VPS things are improving.

Striving for uploads to publish websites

I haven’t had a chance to try uploads, but have also added code to make these more reliable in dweb. One is that when an Autonomi API call fails dweb will immediately try again and repeat this until it succeeds, or the universe ends. You can though set a limited number of tries if you don’t want to wait that long.

There is also a problem when trying to upload a directory. When this fails the Autonomi API (for putting a directory) starts again from scratch every time. This, and the fact you get charged for repeat uploads means that even a small error rate can mean it is very unlikely to succeed, plus, your wallet is being emptied much more than necessary every time you retry. Individual charges are small but will add up when dweb keeps repeating this over and over, so…

Upload file-by-file

I’ve also added the option (currently the default in dweb publish-new and dweb publish-update commands) for dweb to upload each file individually.

This, combined with the retry feature means that even if only a small percentage of PUT operations succeed, it will be possible to publish a directory or website (using dweb) if you wait long enough.

I’ll be trying this once the release candidate makes it into a release.

Inspecting a History and its Pointer

BTW dweb also has inspect commands, one of which shows you whether a pointer is pointing to the head of a graph for a History or is stuck pointing to an earlier entry.

This is another issue we have when using the History to implement versioned websites. But, again dweb now has a feature to work around this by ignoring the pointer if you want and scanning the graph from the Pointer target to the end (real head entry).

Uploads, downloads and web viewing may be close

All-in-all, it should be possible to publish directories and websites with dweb so long as a percentage of API calls are succeeding. This approach will mean upload and download will be very slow for now, but that’s better than not working well enough to publish and view websites at all.

This may be improved even more when nodes catch up with the last two releases.

Fingers crossed for next week that we’ll all be able to try out some of these features.

[Southside]

Sadly after an hour I still can’t see whats so awesome about this

🔗 Connected to the Network                                                                                                              -> public network evm-arbitrum-one
Registered built-in DWEB-NAME: awesome -> 8650c4284430522a638a6fa37dd3e8d610c65b300f89f0199a95a1a9eab0455287f8c8d137fad390654bd9f19b868a5c
dweb main server listening on 127.0.0.1:8080
DEBUG serve with ports HttpRequest : GET /dweb-open/v/awesome/
DEBUG /dweb-open/v/awesome/
DEBUG parse_versioned_path_params_with_as_name() v/awesome/
1:v 2: awesome r: 
version:None as_name: anonymous, address_or_name: awesome remote_path: 
>>TRYING pointer_get() unlimited times...
>>TRY 1 of unlimited
.......



> >>TRY 7 of unlimited
> failed to get pointer from the network - Network error
> >>TRY 8 of unlimited
> failed to get pointer from the network - Network error
> >>TRY 9 of unlimited
> failed to get pointer from the network - Network error
> >>TRY 10 of unlimited
> failed to get pointer from the network - Network error
> >>TRY 11 of unlimited

[happybeing]

I’ve not uploaded it yet. It’s in the README

[Southside]

Aw naw…

Premature compilation disappoints again…

[happybeing]

Thanks for blazing the trail as usual. Have a go uploading and see how your wallet does

[Southside]

I’m getting bored and my wallet is slightly dented

willie@gagarin:~$ dweb publish-update --files-root ~/projects/maidsafe/test-sites/Kwak
Dweb Autonomi client initialising...
🔗 Connected to the Network                                                                                                                                                  -> public network evm-arbitrum-one
DEBUG publish_or_update_files()...
Uploading files to network...
Uploading files from directory (file by file): "/home/willie/projects/maidsafe/test-sites/Kwak"
file: "check-layout.css"
>>TRYING file_content_upload_public() unlimited times...
>>TRY 1 of unlimited
>>OK
file: "style1.css"
>>TRYING file_content_upload_public() unlimited times...
>>TRY 1 of unlimited
Failed to upload file - Failed to upload file
>>TRY 2 of unlimited
Failed to upload file - Failed to upload file
>>TRY 3 of unlimited
Failed to upload file - Failed to upload file
>>TRY 4 of unlimited

Before

+---------------+--------------------+
| Token Balance | 500011984063271935 |
+---------------+--------------------+
| Gas Balance   | 1702287873059000   |
+---------------+--------------------+

After

+---------------+--------------------+
| Token Balance | 500011961709148699 |
+---------------+--------------------+
| Gas Balance   | 1693537093059000   |
+---------------+--------------------+

`

[happybeing]

This highlights the issue of being charged again and again for failed operations. This was solved prior to ERC20 but has come back with it. I don’t know if this can be fixed or if it is impractical due to the ERC20 changes. I filed a new issue a while back.

What connection are you using?

I don’t know of anything else I can done to work around this so we shall have to wait for the nodes to upgrade and then for the upstream changes David mentioned.

[InteXX]

Thanks for this, but I’m still trying to wrap my head around it. Can you assist?

What I’m trying to figure out is how we can run a web app (i.e. server-side code & logic) on Autonomi, a la ASP.NET on IIS, etc. I do understand the basic principle at work here, yes—secure and private file handling—but as we know there’s a lot more to a business application than just file storage & retrieval.

For some reason I’m not able to get my head past the notion that an Autonomi-based website can only be comprised of static files, with no backend logic allowed.

e.g. If this do that, otherwise get a value from the array

Actually, this segues into a related topic that’s been on my mind for some time now.

What about data? How do we build a multi-user application that warehouses its data in a single location?

We need that Single Source of Truth™ that a central relational database provides. How do we go about using something like Entity Framework? Optimistic concurrency? Autoincrementing primary keys? Cascading deletes?

It seems a thorny problem.

Are these types of apps going to be possible at all with Autonomi?