Web client devs - best way to do low level Autonomi REST APIs

[happybeing]

This is a topic for discussing what REST APIs for Autonomi are wanted, feedback on what exists and so on.

This is not just for dweb, although that’s what I will be focusing on. Any RESTful API is welcome here and if we can work towards common standards that would be great - but for now I just want to get some useful APIs in place so folk can try out building dynamic web apps for Autonomi.

Here’s my first question, but devs feel free to open discussion on any REST API for Autonomi.

dweb

I’m working on REST APIs for Autonomi functions and have POST/GET for a few types working (archives POST/GET, files multipart-POST/GET, data GET etc).

EDIT: For the moment I’m focused on web clients so expect JSON payloads and responses (although for /data which gets data using a datamap or data address the response is binary so you can load directly into a browser). If you need binary POST/GET let me know. I’ll be doing it anyway at some point but will prioritise based on what folk want.

So, with a JS web app client in mind here’s a simple example I made (literally yesterday):

/chunk POST/GET

I have just implemented Chunk POST/GET but to start with have the client supplying the content with a simple JSON object, which mirrors the following DwebChunk struct in Rust:

pub struct DwebChunk {
    content: String,
}

So in JSON the client would create a JSON object to POST to /chunk which looks like this:

{
  "content": "hi, I'm a chunk"
}

The response is a PutResult (in Rust) which for the above returns the following JSON to the client:

{
  "dweb_type": "Chunk",
  "status": 200,
  "status_message": "success",
  "cost_in_attos": "3",
  "file_name": "",
  "full_path": "",
  "data_address": "a31f27622645bb1a467d0e1d8e9b169de1a907f170291e1cbe28ad3c3fbfdc2c",
  "data_map": ""
}

I return similar JSON for other POST and PUT operations with content depending on the type of data being stored, and whether it succeeds.

Feedback

Comments, suggestions and requests on any of this are welcome but for now I’m wondering about the JSON used to POST a Chunk. Using a String is limited, but JSON support for binary also not great so any ideas?

What additional JSON options would be useful for the data and using what encoding? For binary it might be a base64 or IntelHex ecoding for example, both of which are discussed on StackOverflow here.

For now I will publish as is, but will update this depending on feedback. If you have requests or suggestions relating to any Autonomi data types, please reply here too.

Next I’m working on Pointer, Scratchpad and probably Vault but for now from a web JS client perspective. So if you need non-JS stuff shout!

Thanks, and if you want to try the dweb REST APIs, see here.

[Traktion]

A few thoughts I have:

• use HTTP status code to reflect the status, rather than embedding it in the payload
• post binary data directly instead of embedding it in json. REST as HTTP POST is fine for this
• the reverse as HTTP GET returning binary is good too and browsers expect it.
• Imo, it’s ok to have a mixture of JSON endpoints and those that request or respond with binary payloads.

[happybeing]

Very helpful comments @Traktion, thanks for taking time.

@Traktion

• use HTTP status code to reflect the status, rather than embedding it in the payload

Where APIs need to return something like a datamap or address I use a standard struct/JSON object (e.g. PutResult) even if there’s an error - so you spotted a BUG there. So I will extract that and return an error response if the put fails, thanks!

• post binary data directly instead of embedding it in json. REST as HTTP POST is fine for this

• the reverse as HTTP GET returning binary is good too and browsers expect it.

Indeed, /data GET (the first one I wrote) returns binary. So you can use that in the browser address bar with a datamap or data address to view content.

I’m using JSON for more complex responses such as the result of a POST/PUT. I’m still not sure if what I’m doing is the best though so hints and suggestions around these are very welcome. Being my first REST API I really appreciate your input.

• Imo, it’s ok to have a mixture of JSON endpoints and those that request or respond with binary payloads.

Good to know , that’s what I plan but am fumbling around in the dark a bit.

For now I’m mostly focused on web JS clients, so JS dominates the APIs for complex types. I do use binary for /data GET (not sure how to do binary PUT yet, but I do have multipart form upload of file/directory in the meantime).

Where a browser JS client needs to post binary data I’m still open to feedback (as per the OP), so let me know if you have any opinion on that. For multi-part forms (local file upload) it is handled by the browser so no problem, but what if a client wants to save binary data (e.g. to a chunk or as self-encrypted data)? I’m assuming they would create a JSON object, but maybe that’s not the only option there.

Over time I hope we can come up with a common set of APIs to cater for different needs so it is important to me to incorporate your ideas and requests, adapt to a common view etc. Thanks again.

[loziniak]

I think you should also consider using custom headers for metadata, then you can have pure payload in binary/json.

[happybeing]

I didn’t even know you could do that!

Could you help me understand what you have in mind, maybe take one API and suggest what headers etc?

Or point me to any example(s) of good practice.

I’m not sure of the pros and cons of the different approaches for different use cases. And for dweb this may be atypical because it is designed to be a local server on the same client rather than a remote.

I’m such a novice at this any good advice is invaluable.

[loziniak]

One advantage of the headers approach is that you can use them in any tools that understand http – cache, load balancer etc. Your chunk example could look like this in javascript:

const response = await fetch("http://localhost:12345/chunk", {
  method: "POST",
  body: new Blob(["hi, I'm a chunk"], {type: "text/plain"}),
});

console.log(response.status); // 200
console.log(response.statusText); // OK
console.log(response.headers.get("DWeb-Attos-Cost")); // 3
console.log(response.headers.get("DWeb-Type")); // "Chunk"
console.log(response.headers.get("Content-Length")); // 32
console.log(response.headers.get("Content-Type")); // "application/octet-stream"
console.log(await response.bytes()); // Uint8Array [ 163, 31, 39, ...

[Traktion]

I think the downside is that they are more opaque than a URL and a body though. Not a huge issue, but worth considering.

[riddim]

@Traktion

I think the downside is that they are more opaque than a URL and a body though.

true for answers - I have rarely come across responses holding basic return data as header-info - more the in-depth-stuff (or e.g. user info being injected that can be processed by a backend (behind a reverse proxy) that then uses it for authentication stuff … doesn’t get to the frontend/user but used for internal authentication flows without modifying the data itself)

and for using it on query side the upside is that you can even put e.g. a payment private keys / a key reference token into the header-data and can be pretty sure it’s not being logged/forwarded somewhere by accident (if behind https … that ofc isn’t the best idea when transferring the whole stuff via cleartext http …)

just as additional reference that’s how I have it in my proxy atm (public put of a chunk)

data/put/public as path variables (datatype, what to do with it, public/private)

write-token as header-data for authorization

returns the most important stuff as json

on error it returns a different struct containing the error

…now that we talk about it I should probably make it /data/public and then just have a put / get method there instead of writing it to the path …

the gas limit would e.g. be something I would add to the header section
… but I guess this is more very personal preference if that should be header data or query param …

---

really nice ofc would be too if we’d have a dweb client library in typescript that would put data, and the result of it would just be a “put chunk” object - possibly with async status update if the upload was executed correctly, how much was paid, which gas limit was used, the address, the content, …

I’m not entirely sure where this thought is meant to lead xD … but a immutable data wrapper / scratchpad, graphEntry, pointer wrapper that lets you create, save, update, read the objects in a nice way (while executing the calls/status updates asynchronously) sounds pretty nice in my head … ofc then there should be a notification to the user “you are trying to leave the site while uploads are still running - are you sure you want to close the window?”

[loziniak]

@riddim

holding basic return data as header

The most important data in this example is an address of created Chunk, which is returned as binary payload.

@riddim

logged/forwarded somewhere by accident (if behind https

Good point. I’m not sure if headers are encrypted in HTTPS, and it could be an important issue.

[happybeing]

Thanks all - really useful for me to hear these ideas.

@loziniak

Your chunk example could look like this in javascript:

Thanks for clarifying. For the response there’s no reason I can’t include headers even if there is a JSON PutResponse body.

But I think you were at least as keen for the headers to be in a POST/PUT request. For binary data (chunk, file) I’m not sure there’s much need, and think having URL parameters works well for that. For more complex PUTs I think best to have JSON for Archive and probably GraphEntry, so it may be best to keep to JSON body (POST/PUT/GET) for all those which aren’t just data (chunk/file).

I’m still a bit unclear how devs will construct and post binary data from JavaScript, say when putting a chunk (and handle the reverse again as a binary body). I guess they can put anything in the body, I’m just not used to this.

How about for structured types JSON body POST/PUT/GET and for data types binary body POST/PUT/GET?

In the mean time I’ll continue with JSON body for POST/PUT/GET of Archive, Pointer, GraphEntry, History/Register. Not sure about Scratchpad yet, it feels more like Chunk/File, and I’ve not looked at Vault so it depends how structured that is.

@loziniak you make a good point about exposing headers in REST tools, so URL params for request, headers and where appropriate JSON in response body. For chunk/data get no JSON, just binary. I’ve already changed this for POST/GET chunk now so just need to look at producing headers for things being returned in a PutResponse structure/object.

I’ll also add custom response headers based on the PutResponse to my ‘LATER’ todo list.

Anyway thanks all and don’t hold back with any suggestions.

[riddim]

@happybeing

I’m still a bit unclear how devs will construct and post binary data from JavaScript

doesn’t need to be javascript - dweb can be used from other unsupported languages too (ruby?)

[loziniak]

@happybeing

I’m still a bit unclear how devs will construct and post binary data from JavaScript

I think I’d ask perhaps more experienced and up-to-date frontend devs like @safemedia about all that, because my REST grasp/knowledge can be outdated already.

[riddim]

@loziniak

I’m not sure if headers are encrypted in HTTPS

they are - that’s why sending an auth-token via header data is pretty much industry standard

[happybeing]

@riddim

doesn’t need to be javascript - dweb can be used from other unsupported languages too (ruby?)

Yes, I heard of Ruby years ago as the great new thing and yet still have never used or learned any more about it! Showing great ignorance here, I’m starting on the area I am least ignorant of, which is in browser clients, so JS/JSON.

It would be great to find some REST API best practices or examples but searching isn’t the best way I think (having tried a little). We need some experts! Ideally with knowledge of the landscape rather than just particular ‘proprietary’ backends.

[happybeing]

@riddim

they are - that’s why sending an auth-token via header data is pretty much industry standard

I’ve also seen (on StackExchange) some people use POST for everything (including GET) for reasons of privacy, so that the metadata is not exposed in URL parameters.

That’s something to consider for anything that isn’t intended to be used in HTML links.

[riddim]

Oh well - get, put, delete and even post/update have their meaning.. I would try to stick with the meaning for intuitivity and if I want to avoid path/query param I’d stick it into header data… Using put for everything doesn’t sound right to me personally…

Edit: but the put/post difference is still not intuitive to me…

[happybeing]

I agree. I read an article by one of the REST gurus though where he was saying it was ok, in fact necessary, to diverge from POST=create, PUT=update. For the time being I’m sticking with that where it makes sense and yes, good point that we can always move stuff from URL params to request headers to keep the semantics of POST/PUT as clean as possible.

[Southside]

@happybeing

Yes, I heard of Ruby years ago as the great new thing

Sadly it went off the Rails

I loved this though

Why's (Poignant) Guide to Ruby

[riddim]

I used it for the first crappychat (bevause that was the easiest to use safenet library back then for me… and never again afterwards) … Many many years ago…

I think it was 4h of ruby in my life - pretty intuitive and pretty powerful but somehow didn’t stick with me

[Southside]

I swear no mushrooms were involved when I wrote this back in 2005 for some tutorial

-rwxr-xr-x 1 willie willie 1222 Apr  2  2005 /home/willie/dr_chams.rb
dr_chams.rb

def dr_chams_timeline( year )
  case year
  when 1894
    "Born."
  when 1895..1913
    "Childhood in Lousville, Winston Co., Mississippi."
  when 1914..1919
    "Worked at a pecan nursery; punched a Quaker."
  when 1920..1928
    "Sailed in the Brotherhood of River Wisdomming, which journeyed \
      the Mississippi River and engaged in thoughtful self-improvement, \
      where he finished 140 credit hours from their Oarniversity."
  when 1929
    "Returned to Louisville to pen a novel about time-travelling pheasant hunters."
  when 1930..1933
    "Took up a respectable career insuring pecan nurseries.   Financially stable, he \
      spent time in Brazil and New Mexico, buying up rare paper-shell pecan trees.   Just \
      as his notariety came to a crescendo: gosh, he tried to buried himself alive."
  when 1934
    "Went back to writing his novel.   Changed the hunters to insurance tycoons and the \
      pheasants to Quakers."
  when 1935..1940
    "Took Arthur Cone, the Headmaster of the Brotherhood of River Wisdomming, as a \
      houseguest.  Together for five years, engineering and inventing."
  when 1941
    "And this is where things got interesting."
  end
end
puts dr_chams_timeline( 1941 )