Putting Web Apps on SAFENetwork by Adapting Existing Frameworks

Development
1

@joshuef

@happybeing I’d be stoked to see what you’re working on. I’m a frontend dev (mostly, I’ve got some full stack node projects on the go also), so if there’s anything I can contribute to here, this sounds like it!

I missed this somehow! I will be in touch (away for a few days) to get your input on a few things in this area, but here’s a summary…

If anyone else wants to add to this, please jump in!

I was looking at CouchApp as a potential way to port simple web apps to SAFE and had just started analysing sample apps to isolate the CouchDB API dependencies when I came across the noBackend / offlinefirst tools, and was reminded about unhosted / remoteStorage. So now I’m looking at these, PouchDB, Hoodie and anything similar. Also, I just found REST + CORS and no doubt there is more to dig out of this rabbit hole :smile:

I think there is definitely going to be something we can do here, and it could make it very easy for anyone with basic web skills to make simple SAFE Apps that load from the network and run in the browser (with the SAFE plugin). Plus easy to port apps already built with these approaches - all we have to do is make bridges between their APIs and SAFE APIs :wink:

None of these approaches were familiar to me so I’d be interested to know if you’ve worked with any of them, know of others we could consider, and if you have any thoughts on this approach. I’ll be more available next week.

3 Likes
2

Hmmmm Unhosted I heard that name before

3

@happybeing No worries. I’ve been away the last days too!

So a quick glance around the services you mention are essentially just alternative or syncing (couchDB for example) services. Which should work fine, as would a more traditional API server etc, with any safe app pages in a browser.

I think where SAFE could get interesting is acting as the API itself (if I’m understanding what I’ve read elsewhere: that SAFE can essentially operate as a key/value store), then it could feasibly replace the backend itself.

Frontend paages would then still work the safe, querying the backend / some DB platform, and being loaded into the browser from the SAFE Network itself.

For me the stickler comes in getting SAFE into the browser easily. Via a chrome extension or some such. So far there’s nothing in the repos (that I know of, please tell me if someone is working on such a thing), that woull allow SAFE access inside of chrome.

My first experiments with making an extension were enlightening. Hijacking the omnibar is easy enough, and then you can access your own URLs, but sadly not the filesystem. Which would be crucial if safe is being loaded onto your computer as a virtual file system (as, I understand it to be).

The work around I didn’t get around to (ahem), is that chrome apps can access the local filesystem. So this could be an In there, which would simplify things a great deal.

4

For me the stickler comes in getting SAFE into the browser easily. Via a chrome extension or some such. So far there’s nothing in the repos (that I know of, please tell me if someone is working on such a thing), that woull allow SAFE access inside of chrome.

There’s nothing in the MaidSafe repos yet, but there is a proof of concept Firefox extension that shows we can intercept “safe:” URLs and handle them. So there will be support for this, and we can expect to have input into what it should do when the team get onto it.

I’m still not clear on the SAFE API, and particularly what “querying” the key/value store means. I think it just means get/put/post/delete of a document, not what we might think of as queries, so I’m imagining that we (you, me, etc!) would build or adapt a client side JavaScript framework to provide more complex operations.

The API is being worked on right now so hopefully will become clearer to me very soon!

Regarding client side storage, there are some interesting developments in this area (PouchDB for example) that we may be able to borrow from or adapt. Though I’m conscious that storing in the client degrades the security model provided by leaving traces or actual files on the client machine. I think we could still do it, but also look for ways to mitigate this.

FYI, I posted this on another thread today:

I’m currently investigating putting Web Apps on SAFENetwork by adapting existing frameworks which may start to give you some options before long. I’m still researching, but more hands looking into this and evaluating would be great.

I think there will be several options, so I envisage setting these out and then choosing one to have a go with. My guess at the moment is a JavaScript framework that offers different levels of API for different app capability. Example (only!):

  1. CRUD: simple storage i/f (RESTful access to JSON objects/documents). Not very useful, but easy to provide, and enough for simple apps.
  1. CouchApp: Cut down CouchDB API, with a subset of DB functions implemented on the client in JavaScript. Limited scalability, limited features, but easier to port existing apps expecting a NoSQL style DB, and especially some simple CouchApps, of which some are quite interesting.
  1. PouchDB: As 2, but with full NoSQL CouchDB style API for offline operation (auto sync with SAFE storage when connected).

The above its to illustrate my approach, NOT at this stage a plan! :smile:

Anyone who can add to my knowledge or wants to go investigate and try out ideas like this, let’s talk.

@joshuef if you are not on Project SAFE slack yet I suggest you do so as it makes code discussion easier. You might need an invite, let me know if you do.

5

Looks like supporting remoteStorage.js apps will be feasible, maybe even straightforward, because they operate as a key value store and only require us to provide a module with get/put/delete methods. They have alpha level examples for Dropbox and GoogleDrive, so we can start by modifying one of these.

All we need is a suitable JavaScript interface to the SAFE RESTful API from within the Browser plugin, though we should have a discussion about the features it needs to provide to support web apps (e.g. for collaborative editing, see below).

There aren’t many remoteStorage apps yet, but still some useful ones, and it does provide a very easy way for someone wanting to build a SAFE web app to do so, and to leverage some neat functionality provided by the remoteStorage client side framework (local storage for example).

One question is whether collaborative editing features can be supported - in remoteStorage these allow put/delete to specify a document version, so the client to detect when a clash occurs. Since the REST API has access to document versions maybe we can do this, but I’m not certain.

There’s a short topic over on the remoteStorage forum where you can find more about this: What apps work with remoteStorage?

FYI @Krishna_Kumar - no need to respond, just letting you know in preparation for browser plugin API discussions.

2 Likes
6

I will try to explain what can be expected from the first version of the maidsafe_client/SAFE-API, hopefully that should answer most of your questions @happybeing.

By the end of this sprint, the maidsafe_client will provide authentication API and NFS (REST styled) API.

By calling it NFS (REST styled) API, we are emulating an Cloud Storage Kind of an API interface for storage on MaidSafeNetwork. So the devs can be relating storage with Containers and Blobs to which most of the web developers are familiar. Once the examples are implemented next week, it should make it easier to understand, hopefully :).

We might have a NFS (POSIX styled) API too sooner :smiley:

To keep the data in Sync while colaborating, we need some kind of a messaging mechanism to keep the application informed about the changes, rather than polling. Once the application gets notified, then the application can get the updated data(latest version) and act accordingly. Combining Messaging-API and NFS-API, we should be able to achieve it.

5 Likes
7

Thanks @Krishna_Kumar that’s great news, I’ve been following your commits to the client so don’t want to interrupt your progress more than is helpful.

To keep the data in Sync while colaborating, we need some kind of a messaging mechanism to keep the application informed about the changes, rather than polling.

Did you see the comments I left on Jira yesterday, on task 1103?

For some uses polling may be effective, or at least an expected cloud behaviour, remoteStorage.js for example. So GET conditional on version match/mismatch would be good. Though I am sure a notification method would also be useful.

I can’t wait to have a go at coding it and trying out on a SAFE browser plug-in client! But I guess the latter is some way off :frowning:

BTW, RS offers three modes of cache behaviour (ALL, SEEN, and FLUSH) for file sync which might be worth you looking at (docs here: https://remotestorage.io/doc/code/files/caching-js.html).

Also, remoteStorage.js wants the ability for get/put/delete to supply the expected current document version, and the API to error if not met. They use PUT for PUT and UPDATE, so we need to consider how this and the other features can be mapped to SAFE REST API in the SAFE RS backend.

I haven’t looked into other frameworks adequately yet, but RS is so close to what we have that it should be easy to implement, which means anyone will be able to build simple web apps that load directly off SAFE with the SAFE browser plug-in, and if we can support collaboration, well, amazing! :smile:

Thanks for stepping out to answer me, I appreciate it and am aware the priority is to get the network to beta, and some of this may therefore not follow until afterwards. I’m hassling you partly because I’m excited about being able to do something constructive for the project, and because I hope it helps to get the a API right. I’m not sure if that’s true, but there you are :slight_smile:

2 Likes
8

Thanks again @Krishna_Kumar, I’ve read your response on Jira and also looked a bit more into the code while trying to understand how it fits together, and how it supports web apps.

On the latter, I see the capabilities web apps are likely to need as follows:

  • storgage and retrieval - obvs :slight_smile:
  • sharing and collaborative editing - options here (poll or notify), but one way or another, SAFE supports it.
  • small local search - needs client side support (e.g. build/maintain indexes), so not in SAFE, but achievable.
  • big data local search - the same as above but harder, will hit some limits. Not in SAFE but achievable.
  • big data distributed search (public or private) - ex-SAFE, probably best in non-web app, with indexes built and stored on nodes which choose to collaborate by running a custom index and search app. I think this will be one of the big killer apps of SAFE when there is a lot of content needing to be searched. So not part of SAFE, but I think feasible and a very exciting pattern to build. I suspect we’ll see @dirvine on this one :smile:!
  • messaging - this I haven’t thought about, but its a no-brainer that it will make apps far more capable, such as the distributed search, and much much more. So in SAFE, but I’m not sure about SAFE REST.

I’ve been trying to understand what the REST API will support and how to bridge the gap between this and the easiest app building frameworks we can support.

@Viv are you planning to stop at exposing the REST API in a plugin, or might you go further - adding it as a backend component for some frameworks? I’m interested in helping here but if you have things covered, or know your priorities so I can either contribute or focus elsewhere, that would be good. I’m not good enough to code production stuff in JavaScript or Rust, but I can build things that work, investigate, specify, test, and generally be a nuisance. Your call :slight_smile: I’d also love to build or port some useful apps with the easiest route I can devise, which is what got me going on CouchApps and now web apps in the first place.

Right now I’ve got a big list of client side frameworks that I know very little about, but I don’t want to spend ages learning about them unless its going to prove useful in creating more capable web apps, or easier development options. I’m happy to continue poking around like this and consuming what is pouring out of the sprints, but if I can be working in ways that dovetail with the team or complement what you plan, all the better.

In the mean time I think I’ll stop hassling @Krishna_Kumar, who has been very helpful, so he can get on and deliver the examples he’s mentioned. I think that will clarify a lot, so I should probably get on with some more welding instead :-). Its just this is so exciting!!!

9

depends on what these backend components are Mark :wink: The REST or POSIX API’s by itself are in rust, so exposing the REST API to a browser via a plugin is just “one use case”. Having a few different usage options is never a bad thing for such API’s. We can crack a few of these ourselves or if community members such as yourself are interested, we can certainly help in putting these together then :smile:

well things like meteor-js is always interesting in facilitating such requirements from client-side.

3 Likes
10

Not a problem at all @happybeing :smiley: I am happy to get your inputs on what you (user) would be expecting. Last week, I couldn’t keep up to your pace with the investigations you have been carrying out. But now, it is Weekend :smiley: Phew!

I have been churning all your inputs over the last two days, they have been really helpful indeed. Over the weekend I will check the remotestorage.js that you have mentioned above. Once I get to understand its potentials, then I am sure I can explain things better comparing our API or even related to usage for various use cases.

Absolutely, it does help a lot :smile:

4 Likes
11

@joshuef, you are right, the extensions cannot access the file system, because of security restrictions. But we don’t have to access the Virtual File System from the browser plugin.

To brief a little, Integrating applications make use of the MaidSafe Client API to access the data from the MaidSafe network. We can consider Virtual File System also as an integrating application with the MaidSafe Network using the API.

Now coming back to browser plugin, we were trying a put together a POC on firefox initially. Linking, here a short documentation on what we had done. At present, we have parked it and probably start again once the APIs are stable and ready.

.

1 Like
12

Thanks @Krishna_Kumar, I’m glad I’m not pure nuisance ;-). I’m really ignorant of the area though - not been active for over a decade so the software / app scene is very different, and I’ve forgotten everything I learned back then anyway. My fingers remember how to type tho :smile: . But don’t take anything I come up with too much trust/authority…

For example, Viv just mentioned meteor, and I hadn’t even heard of it. Yesterday it was Firebase, before that Angular etc etc. I have a BIG list of these frameworks and have not built anything with any of them. I’m just trying to get a sense of where to look first based on what I think I understand about all the issues (web app needs and limitations, capabilities of SAFE REST API etc.)

My question for you is what is the basis for the REST API features you are implementing? Do you have a spec in terms of high level capabilities, or low level Request/Response, or a similar REST API you are trying to match, or…? Don’t spend a long time clarifying, but if you have a quick answer I will sleep better :smile:

So far I’ve looked mostly at CouchApp, remoteStorage.js and Hoodie, with Angular and some other frameworks staring to look important. And PouchDB which is increasingly looking important!

  • PouchDB is compatible with CouchDB (NoSQL) but designed to work completely offline
  • the Hoodie guys are implemeting PouchDB. Hoodie will give PouchDB a very intuitive frontend API (compared to other client/offline approaches that are modelled on backend APIs and so not so nice for frontend devs)
  • some anecdotal reports that speak highly of PouchDB, (eg one StackExchange answer puts it above Meteor and Couchbase Lite), but this is not based on very much yet.
  • Meteor looks very slick to get into, and given @Viv suggested it I give it weight too, but still from very little knowledge or experience.

So please don’t spend too much of your weekend following my lead! Except if you want to of course :slight_smile:

1 Like
13

Features on a high level,

Operations on Container:

  • Create Container
  • Update Container metadata
  • Get versions
  • Get container based on a version
  • List Containers
  • Delete Container

Operations on Blob

  • List Blobs
  • Get versions of a Blob
  • Get Blob by Version
  • Read blob content
  • Update blob content
  • Delete Blob
  • Copy Blob

I will try to come up with a small documentation on API usage over the weekend and share it, @happybeing . Hopefully, that should give some clarity, even before the examples are ready. It is already on my list for the weekend. I will try to share it here at earliest I can :hourglass_flowing_sand:

5 Likes
14

Thanks again. I have been following your implementation of these and watching the todo list being ticked off. What I was wondering is how these were arrived at, the capabilities they are intended to support (such as collaborative working), or will not support, either for now, or forever. But please don’t produce docs just because of my interest, I can wait until you put them in the code (I’m following it closely). And you deserve your days off :smile:

1 Like
15

Not so hard, Three sprints and three dev bundles (API packs). There is a new roadmap coming out. Basically we choose to implement what we think is the most desirable parts first in line with what is foundational (so private data first as it’s very complex but essential), then messaging and public data (a technically close pairing) with initial safecoin farming rate (affects storage) etc.

The way these are chosen is not simple, but in a sprint we decide what we would like to deliver, then everyone specs out their libs requirements, then we guess what each task takes in terms of resources, then add all resources and decide what features we can provide.

This all gets added to the mix of what is coming down the line, for instance I am working on Pay the Producer, safecoin algorithms decentralised computing and some more (semantic data etc.). So this all gets added as we are sure how we will achieve them (while planning it’s very very hard to discuss these openly though, sometimes it needs a dark room and nothing else, taking into account what the community want and why). Some of these are very exciting though and now the team is really taking this network to themselves and running fast, I get a chance to research deeper issues down the line.When we are feature complete and apps launching (by the community and us) then the Engineers will take many of these tasks on board as well.

Viv, Nick and myself spend probably 30-40% of each day looking again at what we are putting out and when in relation to the community expectations and our ability (plus how we also survive as a team, so very keen on launch :wink: ). Lots of changes recently to get to this way of working, but it’s really shaping up nicely.

Hope that helps a wee bit (I am so glad we went rust and managed sprints, it’s a new day :slight_smile: )

7 Likes
16

@happybeing great to know you are following the code closely :slight_smile:

We are not supporting collaborative working for now and as you know this is just the first version of the client. In this iteration we are trying to get an implementation to perform basic storage and assert whether the APIs are easy to use.

But we can definitely support collaborative working in future, but how are we are going to support is debatable. If we look at the present cloud storage APIs they do provide features to register listeners, We can also come up with such listener\Event Subscribers, making use of our Messaging API, just an early thought.

I will put the document at the earliest explaining the usage of API for the current version under development. Which will help in getting early feedback. Would be helpful :slight_smile:

6 Likes
17

Thanks David, this does help. My hunger is to know the REST API targets you’re aiming at so its excruciating watching the repos for clues while not even knowing the questions I should ask (though I’m getting a bit of a handle on those at last!). My aim being to find and help create quick and easy ways of building apps, so as soon as the API is in place we can start buildling and porting to it. Not least so I can have a go myself, while sharing with everyone. Apologies if I’m distruptive, I’m happy to defer to your priorities, and of course there is welding. The welding has taken a back seat for too long :-).

3 Likes
18

That’s wonderful. I shall feed back if my meagre knowledge can be constructive!

Thanks also for clarifying the target and where you are with future API etc. That helps a lot. One of my unstated concerns was that an API, once devised, can be hard to extend. In my heart I know you guys are aware of these issues and capable of handling them, but my fear is does push me too, and fires up my wish to know, now, today… etc :smile:

Thanks very much for your help. Have a great weekend.

2 Likes
19

@Krishna_Kumar Great! I’ve requested access to have a look at your link re: firefox then. I’ll concentrate any learning I do into firefox then for the minute!

@happybeing Meteor/Angular/Backbone/React etc are all various ways of structuring your webapp code and dataflow. I think the more pertinent questions will come from the DB sort of angle. And then how we access this in the browser.

If we’re able, with the API to get/set as to a database, that would almost remove the NoSQL DB need. If not, we’d need to wrap our NoSQL database with something that translates these DB calls to the SAFE API.

Hoodie looks awesome. Ha. I’ve just been building a project and creating the CRUD backend was such a pain. A good learning experience, but for sure not the fun part. So yeh, that looks like it could be super handy for faster webApp development. But I think it doesn’t necessarily solve the problem as you need your data hosted somewhere (and then you make requests to it). So while your app / page etc can be stored as a static file in safe, it’s accessing data from without.

Meteor I’ve used before and didn’t love. But that’s just like… my opinion, man. (It’s not a way I like to work).

I guess to fully move everything, including the storing of data to safe, we’d need an ODM for SAFE (something to act like CouchDB or mongoose for the apps); which would handle the put/get of info from the network.

When I get more time I’ll try and read up on @Krishna_Kumar 's info on the API as right now Blob/container don’t really mean much to me.

20

Or… if I’m reading what @Viv was saying, correctly, we could have a default REST api for browsers (via a plugin etc), then that might simplify the whole thing.

If we’re able to make a GET request to the safe network on some uri (safe:yourappsacce/api/users) then the whole CouchDB etc stuff becomes moot. Your app, whatever you build it in, could access all info here, and then do all the local storage / offline first stuff as you want. Grabbing updated data as is needed (perhaps after updated messages / push notes from the network itself).

That would be cool.

1 Like