So sorry if this sounds like a dumb question, but practically speaking, how would a database for a website work on the SAFE network?
I’m not well versed in web design stuff, but I know that generally speaking you want to go with MySQL for databases.
So, for example, if I wanted to make a site where people could “upload” pictures to their “account”, how would I do this?
How I imagine it working is that the webapp would actually just have them upload their image to their own storage account, and then take a link to that resource, and present that instead - the user always has control over their data, but it’s viewable via the site. However, you would still need to store that link, and perhaps the name associated with it and so on, so how would you do that? Just files? I’m sorry if I’m not explaining this correctly (I almost never do, if you read through any of my posts, lol) but yeah.
Not a dumb question, a very important one that will be asked many times I’m sure.
It is pretty much as you imagine. I think that one way will be for the website to own /create mutable data objects, but allow others to do certain changes to them. Such as adding an entry.
So yes, a visitor would upload a picture and it would be stored in their account, and a link to it would be inserted into mutable data belonging to the website (maybe - not 100% about that so if someone knows better please correct me). So the website has control of what it links to (eg it could remove the link) but it can’t delete the picture itself. That’s what I think, but I haven’t looked into how to actually do it yet.
Yes, you will need to structure your app data like a database. In your app folder you will probably want a transaction log, and one or more indexes that keep track of where you write application data that you want to locate later.
If you want an API for sorting/querying data, you can consider using an in-memory database. I like to use TaffyDB.
Someone (maybe me once I get rolling) will probably produce a library to tie all this together to make it easier.
Okay, so for example, you make a website that people can upload images to - basically imgur or something like that. You allow them to tag the images, so they can be searched. With a normal site, (keeping in mind here I’m no expert webdev) you use the javascript for page interactivity and the like, and then at some point if you need to do a search, you have server-side code run (ie: php) to query the database, which is also running on the server. (Although I suppose it could also be done with Javascript?)
On the SAFE network, you have no server, so would you write javascript into the page to query a file-based database, or something like json? Does that mean that the user has to download that database every time? If so, how can you implement any kinds of limitations or security?
Of course, I’m probably misunderstanding this because of a lack of knowledge, but I’m finding this to be one of the hardest concepts to wrap my head around, especially trying to figure out how to build my own SAFE sites.
Yes, all the data management needs to occur in your client-side app, in JavaScript. No, you don’t want to load your entire database into memory at once.
Think of it more like a simple key/value store like you get in the browser itself (LocalStorage) or on the server with something like Redis. These databases don’t provide indexes, just basic read/write storage, so you need to figure out how to partition your data, and manage an index so you can lookup the data you need, and get the storage key to go read/write it, etc. You index strategy needs to accomodate for the various storage options in SAFE as well (mutable vs unmutable, public vs private, etc).
The SAFE network API allows you to read/write data locations, but for your application, you need to create and manage the MAP (indexes) with regard to how/where you store everything.
In my eyes it must be possible to create some ‘server db account’ eventually through multisig if implemented, and query against this db client side (website that serves your setup of the db) through react+graphQl if some of the multisig creators setup a mongoDB service on a server/servers reading/writing/routing the datebase queries to the ‘server db account’ public/private folders.
I think many of today server structure and even company owned websites with registered users could be reproduced and will get reproduced.
It’s all up to the user signing into this or not, eventually using another persona, but a total kyc etc could be applied if wanted imo…
I’m interested to see how this one pans out. Databases are central to the way pretty much every large application works right now. Can anyone tell me what sort of work would be involved to port something like Redis to SAFE, or is it more likely that some new database or database-like system will be built from the ground up?
I think (idiot warning here) the way to do this will be to implement NoSQL features such as map-reduce when compute is available.
In the mean time you can run something local and sync with the network using client side scripts (in JavaScript). Browsers already support client side databases these days, plus you have things like PouchDb.
This is how remoteStorage.js works for example, and some of you have seen and played with a demo app using this on the earlier test networks (see my favourite drinks and litewrite).
That’s interesting, and promising, and fits in with the general trend of moving compute to the edge. Good to see you’re already playing with this stuff. I guess for really big heavyweight apps - the equivalent of a CRM system or something - we’d either need to wait for compute on SAFE and some sort of native or ported distributed database, or find a way to interface with databases hosted off the network (which would presumably defeat the object of decentralisation). Or perhaps SAFE won’t be suitable for such applications anyway.
My understanding is that maidsafe is a disk on the pc, we can put a file dB on this disk and share the records in db through safe app. Safe app will maintain separate file db for each user of the app to store every users’ own data.
To display every user’s data on to the screen of the app, safe app need to query every relevant user’s db and find relevant records, display them on the screen.
Thanks for sharing this. I am, however, curious whether there is an application or framework in the works for the SAFE network that will make website generation a simpler process. That is, we do not want to start back in 1994 where website development was done manually in html and the vast majority of sites were static. Nowadays, for a rich Internet experience many sites are dynamic as there is an interaction with a database layer. This is necessary in order to pull in dynamic content rather than hard-coded text and images.
I realise that there have been some discussions on this forum surrounding database querying and how that will be accomplished in the application layer and not at the SAFE network core layer. So having a framework (e.g. Django or Rails, etc.) to assist in building modern and rich websites will be crucial. Otherwise, SAFE will mostly be used for file sharing (e.g. Napster style). Any veterans care to point us in the right direction?
How might this look, practically speaking? Say I have a DB for the website - if it’s public, every user has the ability to see everything in it.
If each user gets their own DB that is private to them, then the full database is never accessible by everyone… unless there’s some way to get semi-private sharing?
Data can be public, and most website data is public so that’s same as now.
Some data can be made accessible to individual users (like your private account on a website) and some can be made accessible to a group - more than one user - such as admins, or people who are members of a Facebook style closed group etc.
So at a superficial level at least, things are very similar.
You would be right to see it as more complicated - or at least different in some of the details - for example how do you arrive at data accessible to multiple users. How is it created and who pays to store it being the main question?
I think while these kind of subtle issues will make it hard to build websites that work exactly as the ones we currently use, but that would miss the point. I don’t have the answers, but think this will in part mean we’ll build websites that function slightly differently, in a more decentralised way, and so we’ll have to get used to things working slightly differently but with the advantages of decentralised data storage and control.
Developers definitely need to get their heads around the fact that SAFE is a decentralized server-less network designed to protect users privacy and security. In my mind, this significantly changes the rules of the game for a lot of how existing Internet applications work.
No server means no server-side web rendering frameworks like Express or Rails or Django. You need to build client-side applications. That could mean native desktop and mobile applications using the various Safe SDK’s (as they become available in various languages), or single-page web applications written in HTML.
The SAFE network IS the database. It provides an API that allows you to read and write data. Sure, it doesn’t provide indexing, query, sort, filter, etc., like more powerful centralized databases we are familiar with, but some of that can be done using client-side in-memory databases that are populated on-demand with data stored on the network when the user needs it. Big Data kinds of activities that need to work across large data sets will probably need to wait for compute resources to be introduced to the network.
User’s own their data, not the company that produces the application. Yes, data can have access rights granted for multiple users, but this is significantly different than the typical delegated security model with centralized database that many distributed applications are built with.
If you don’t like these features, then why use MaidSafe? It is supposed to be different. Companies are not supposed to control and own all our data. Apps are not supposed to be able to do anything they want with our data and treat it like it belongs to them instead of us.
I think compute would be really amazing for the SAFE network. If compute can be instituted, it truly can replace the internet.
Imagine no longer going to Google or AWS, and just using the SAFE network?
More on topic though, when you say the SAFE network is the database, how would you populate the client side database?
Like, a search, how does that work? If all the images are tagged with various things, and you want to search for the images with a tag or something, how would it work practically? I’m guessing that it would be JavaScript which would then issue a GET or something like that from the network? But the tags would have to be stored in some other database… which would be a file, or a series of files, which would then only partially be loaded?
So maybe like, folders: cats dogs kids: search for cats > pull the db file from folder cats?
But then who owns the database? Can others add to it?
I know I sound kind of retarded, but I’m really struggling for that a-ha moment here.
This is the key part. The owner of the databases structure would be the app owner/developer in many cases, the app publisher in others, but the database content will be owned by the user/person using the app/system. The use of the word database here is misleading, but can be thought of in that way to an extent. As many folks who are our target won’t even know what a database is, it’s best to perhaps just say your data belongs to you and always will, regardless of where you publish/post it.
If you make it public then it’s always ours (ours being society/humanity) until then it’s yours under your control. Of course you will be able to have private shares and then it belongs to that groups, but that’s another step down the rabbit hole.
So think of it like this, your data is yours, if published in an app it’s still yours, if public it is everyone’s. The database terminology is 90’s for organised data pools, in this network apps will organise network data in ways that make the app work, but the data is on the network, not in some apps container/ownership. Several apps could have your same piece of data displayed for instance.
If you have an image, then you would likely store that as immutable data. Then you would store the tags in some other object. The list of tags might just be ids though. To search you could then make indexes.
There could for example be an mutable data for each of cat’s, dogs and kids that would contain a list of all the images. If you want to find all images of cats and dogs you would then fetch those two indexes and compare them, so you would get the cat index [img1, img3] and the dog index [img1,img2] and compare them on the client side to get img1 as the result.
Multiple people or apps could create their own tag lists and indexes for an image. The tag list and indexes could be mutable data and different users could get permission to update them.
until then it’s yours under your control. Of course you will be able to have private shares and then it belongs to that groups, but that’s another step down the rabbit hole.