Serving dynamic content instead of static files

[bguiz]

Having watched the Maidsafe introduction videos and reading up about it so far, my understanding is that the Maidsafe network allows one to store files across many other nodes across the network.
Nodes are encouraged through the use of cryptocurency.

I can grok that. You are basically bartering some of your hard disk space for others’ hard disk space. OF course, there is a lot of neat and amazing technology that allows this to all work, but at a fundamental level, this is what it reduces to (in my understanding anyway).

OK, so far we have file storage. Next thing, how do we share files?

- YouTube

OK, that’s great, with this sample app, we effectively have a private Dropbox.
I’ll need to look into this further, as I am yet to understand how this works,
so if someone could explain how private shares work at a conceptual level,
that would be great.

… and after that rather long preamble, I come to my main question:
How would one be able to write their own web server and host it on the
Maidsafe network?

Let’s say I have a NodeJs app that connects to a mongodb database,
or a PHP app that connects to a mySql database.
What you would do in today’s pre-Maidsafe network would be
to get access to a computer enable TCP/IP on its operating system
register its IP address with a DNS registrar,
and run the database daemon and the webserver which accepts connections on
a particular port.
End users can then use a web browser that understands DNS protocols,
and the protocols that the webserver communicates with (HTTP),
and then they can use your app.

Now, let’s say I would like to do the same thing over the Maidsafe network.

• What is the equivalent for each of the steps described above?
• Are there currently any limitations preventing this from happening now?
      - e.g. Web servers need to be patched to enable them to communicate over rUDP instead of TCP/IP
• Are there any examples web apps that should run on the Maidsafe network to take a look at?

[chadrickm]

I asked/answered a similar question here… How to start - #3 by chadrickm

And welcome to our community. Great to have you here.

[bguiz]

@chadrickm Thanks for your reply - I have just revived that old thread

I’d like to know if you have made any progress since asking that question a couple of months ago.
While being able to serve static content is great, that limits the potentials uses cases for the Maidsafe network to essentially a decentralised and super-secure Dropbox.

Being able to have a program run and serve dynamic content from a distributed server,
would be extremely amazing; and I just would like to know if this is indeed possible to do.

At the moment it sounds like it is not - everyone needs to have the same software running on their nodes -
the Maidsafe program - and so long as they all adhere to the same protocol, static files can be served.

Serving dynamic content, OTOH, poses a completely different challenge.
This is because every single user, potentially, is writing their own programs, and distributing it to multiple nodes.
When someone accesses a URL that is to be served, they need to:

1. connect to one of the (four or more) nodes that contain this particular program,
2. execute the relevant parts of this program, and
3. return the results.

Points #1 and #3 have already been solved by the same parts of the Maidsafe network that enable it to serve static content.
However point #2 poses a slew of problems, such as:

• Sandbox the executing code, so that it does not run outside any other parts of the host operating system on the node
• Memory synchronisation & disk persistence.
  • If the server needs to maintain state (for example when dealing with sessions), memory across different nodes needs to be synchronised
  • If the server persists state to a database, the contents of the database needs to be synchronised across the different nodes. This is probably going to be different/ harder than synchronising flat files.

Anyway, I feel the need to ask these questions now, because there is a huge difference in the potential applications of the Maidsafe network, between being able to share static files, and being able to serve dynamic content.

[hamiltino]

SAFEspace browser

Project SAFEspace The original idea of the web was that it should be a collaborative space where you can communicate through sharing information. - Tim Berners Lee Sir TBL invented the world wide web and the hyperlink initially from a scientific point of view: hyperlinks refer to the original source (without having to know on which computer this source information is stored). Since then the web has evolved a lot, and with the advent of web applications we now see centralised applications th…

SAFE web browsing

Thank you for boosting this (important) discussion. People will need the smoothest transition possible. That might first be desktop applications to download, but the web-compatibility should not lag far behind, in my opinion. So from earlier messages I learnt that it will be hard to build a light weight client in JavaScript (even though I still love that idea; it would allow any page to connect to the SAFE network). A browser plugin, I agree, is probably one of the best starting points. Is …

[qi_ma]

@bguiz

Let’s say I have a NodeJs app that connects to a mongodb database,or a PHP app that connects to a mySql database.What you would do in today’s pre-Maidsafe network would beto get access to a computer enable TCP/IP on its operating systemregister its IP address with a DNS registrar,and run the database daemon and the webserver which accepts connections ona particular port.End users can then use a web browser that understands DNS protocols,and the protocols that the webserver communicates with (HTTP),and then they can use your app.
Now, let’s say I would like to do the same thing over the Maidsafe network.
What is the equivalent for each of the steps described above?

In short, the main difference will be :
1, use decentralized database instead of current centralised database
2, the client app may need to become little bit “heavier” as some previous work in server (such as data querying) now need to be carried out by client app

@bguiz

Are there currently any limitations preventing this from happening now?    - e.g. Web servers need to be patched to enable them to communicate over rUDP instead of TCP/IP

couple of main limitations :
1, Access control over decentralized database
The solution will be something similiar to the access control part described in this paper :
Building web applications on top of encrypted data
2, Browser support of a much heavier client code
This may eventually means a decentralized browser needs to be developped.
Though personally, I think a js wrapper shall be enough to enable current browser supports SAFE network.
3, Application developper
The decentralized app model will be quite different to the current centralized model.
This requires app developpers to change the way of thinking they used to (shifting work from server to client)

@bguiz

Are there any examples web apps that should run on the Maidsafe network to take a look at?

I am not quite familiar with web apps. We don’t have an in-house one yet.
Here is an interesting project may worth have a look
Decentralized URL

[yonam_nous]

When you speak about a de-centralised DB are there any plans to go down the Bitcloud route and allow for DB operations using SQL (or similar)?

If a sites code executes on the client how is access to the publishers content secured? For example; I’ve an online store, how do I prevent users from inspecting my code and then querying my DB?

Thanks

[dirvine]

The best way is to use key value stores rather than SQL as SQL centralises data for sure. It’s a shift for many db devs for sure, but it is worth it. Then you can have db’s that are enormous, but you may shift from immediate to eventual consistency as amazon, google, facebook etc. do at the moment.

SQL can be useful for smaller sites and uses, just as in the server based world really. I posted a video here recently by a chap called Fowler that explains it all really well. So it’s best we provide all those capabilities, but an SQL only solution is likely to be the wrong solution in my mind anyway.

[qi_ma]

@yonam_nous

When you speak about a de-centralised DB are there any plans to go down the Bitcloud route and allow for DB operations using SQL (or similar)?

@dirvine has answered this and I do agree with the points he listed.

@yonam_nous

If a sites code executes on the client how is access to the publishers content secured? For example; I’ve an online store, how do I prevent users from inspecting my code and then querying my DB?

This is the access control part I mentioned in the previous post.
You can’t & don’t need to protect your code.
You only need to encrypt the data in DB so others can only query your DB with proper decrypt key.

There is another way to handle dynamic content request :
Have a new Persona, call it BundlingManager.
It is a light weight vault, receive a request from client (call it AskingNode),
pass the request to user-defined code and then pass the result from user-defined code to client.
These BundlingManager / AskingNode provide another set of API for those traditional ICPs.

It will be much easier to to migrate those old apps / websites.
But this will bring back the centralized ICP.

[yonam_nous]

@qi_ma

This is the access control part I mentioned in the previous post.You can’t & don’t need to protect your code.You only need to encrypt the data in DB so others can only query your DB with proper decrypt key.

I didn’t read the entire paper on Mylar but from what I did read it appeared to be a way to prevent static analysis of code. I didn’t see how it would prevent end-users from looking at a sites JS (or whatever) when it’s executing in their browser. Appologies if it answers my questions later on in the paper, I’ll try and take a proper look later today.

The bit I was unclear on was how the JS (or whatever) code that’s executing on the client could be hidden away from the user so that they couldn’t compromise the service they’re using. I’m making up API calls here but the image in my mind is that the code that executes on the client would look something like:

order = safe.Get(authKey, objectKey);
order.PaymentReceived = 100;
safe.Save(authKey, objectKey, order);

If I develop a site that executes on the client how do I prevent users from determining what my key is for performing operations and being able to craft their own calls - maybe they want to say that they’ve paid 999 rather than 100.

I think you’re now saying though that it’ll be possible to have code executed on some machine which I can secure and this will get around this problem. Have I understood you correctly? Can I have a number of machines associated with my “Persona” and one of these will serve the request (allowing for the likes of load balancing and failover support)?

[qi_ma]

@yonam_nous

I didn’t read the entire paper on Mylar but from what I did read it appeared to be a way to prevent static analysis of code.  I didn’t see how it would prevent end-users from looking at a sites JS (or whatever) when it’s executing in their browser.  Appologies if it answers my questions later on in the paper, I’ll try and take a proper look later today.

Sorry, I shall make it clearer.
I refer to that paper just to give some sense that access control can be achieved by using multi-sig.
The paper itself is more about static data (supporting of dynamic data is possible but will be difficult).
Our approach of using multi-sig will be different, so dynamic data is supported much easier.

@yonam_nous

If I develop a site that executes on the client how do I prevent users from determining what my key is for performing operations and being able to craft their own calls - maybe they want to say that they’ve paid 999 rather than 100.

Here comes the big difference to the previous centralized model :
each order (transaction) shall be presented as a multi-sig SDV .
an order (transaction) won’t be considered as valid till the app approved authorizor signed the SDV as well.

Actually, this will be running much like SafeCoin.
From DB view, SAFE network provides a decentralized DB, access right control achieved by multi-sig.
SafeCoin is one of the things such DB can be used to.

@yonam_nous

I think you’re now saying though that it’ll be possible to have code executed on some machine which I can secure and this will get around this problem. Have I understood you correctly? Can I have a number of machines associated with my “Persona” and one of these will serve the request (allowing for the likes of load balancing and failover support)?

Yes, you can do whatever you want once the client request has been passed to your side, under this centralized model.

[yonam_nous]

Thanks very much

[bguiz]

The first couple of links from @hamiltino show projects under development that enable the “client” side of the equation.

The “server-replacement” side of the equation, described by @qi_ma - if I were to summarise - is to have the “client” talk directly to the “database”, but a database like this that aims to talk the Maidsafe protocol, currently does not exist, and currently has not begun development. These are currently academic papers. Can I confirm that I have understood this correctly?

[socketexception]

This is clear as mud to me. I can see how the data is handled, similar to nosql, access the data on SAFE as key values. What I am not seeing is how a node.js app would run, I do not envision users installing node on there machines and npm-ing dependencies. With no server the code has to run somewhere. I am curious to see more on what the ideas are for implementing this.

Now what I can envision is maybe a java based dynamic solution. An app could be a jar file could be stored on the network. It could be loaded dynamically with JarInputStream and ClassLoader. The constructor could make whatever calls it needs to for content retrieval. So a simple wrapper to handle this, then you could deploy java to handle the dynamic aspects of the app. The java could make the requests to any data in the SAFE network as well as access the user’s account info.

Ideally, there would be some kind of execution layer in the core, but that does not seem like a small task.

[bguiz]

This is clear as mud to me.

@socketexception I totally agree. From what I can tell of the discussion at the moment, the idea works in theory, but has yet to be implemented in practical terms.

While yes, it is possible to have a multi-user server running distributed across multiple system, this also assumes that all users have the same rights to all the data. Solving the problem of individual permissions without a central server acting as arbiter is not going to be easy, without, like you point out, some arbiter like functions built into the distribution layer - maidsafe - itself.

This introduces further complications, which need to be solved, before I we can build real applications with it.

[bluebird]

Which is the better way to serve dynamic content (such as email or social media) on the SAFE network:

1. Have a traditional server somewhere, and use SAFE network simply as a pass-through to make the server and clients hidden from each other and anyone else.

or…

2. Somehow build the “server” onto the network itself, or make the server distributed, or spread the “server” across all the application’s clients.

[cretz]

IMO the answer is most of the time the latter, but it completely depends on the problem space. Email is definitely the latter because it is data at rest. Social media is mostly the latter as well including real time messaging. The only reason the former would ever come into play IMO is when you need to do CPU or memory-intensive work (and even then, they aren’t a server per se, just a more powerful user) or if you must have something online all the time answering centralized requests (which again is just a more reliable user).

[bluebird]

I hope my comments here don’t seem too outlandish, but they are on topic, if speculative:

1. The SAFE network provides distributed storage, but not distributed computation - except as it applies to farming, but not for applications. Its farmers are rewarded for providing storage.

2. The Ethereum blockchain is a distributed computer. DEVCON1: Understanding the Ethereum Blockchain Protocol - Vitalik Buterin - YouTube That is, the Ethereum miners, collectively, are a distributed computer because they are rewarded for executing arbitrary scripts (smart contracts in Solidity, Ethereum’s scripting language) created by clients, and stored on the blockchain. The expectation, as I understand it, seems to be that any storage beyond the actual smart contract is provided by the client creating the smart contract.

3. Combining the two would give a fully distributed, general-purpose computer, whose capabilities would include delivery of dynamic content. All the client would need to do is provide, or call, the program. I can’t say how this might be done.

[cretz]

@bluebird

but not distributed computation

It is a goal at some point. But it turns out you rarely need it for most use cases. Computation can be distributed in many forms…this is what applications do.

@bluebird

The Ethereum blockchain is a distributed computer

A slow one tailored to specific auditability requirements. Not for general purpose computation.

@bluebird

Combining the two would give a fully distributed

Sure, there are discussions on the forum concerning this.

[Traktion]

@bluebird

The Ethereum blockchain is a distributed computer. https://www.youtube.com/watch?v=gjwr-7PgpN8 That is, the Ethereum miners, collectively, are a distributed computer because they are rewarded for executing arbitrary scripts (smart contracts in Solidity, Ethereum’s scripting language) created by clients, and stored on the blockchain. The expectation, as I understand it, seems to be that any storage beyond the actual smart contract is provided by the client creating the smart contract.

Technically, ethereum is decentralized, not distributed. All nodes repeat the same calculations, which makes the network less efficient as it grows.

However, when safe net gets compute features added, it will be distributed. Consequently, as the network gets bigger, efficiency will remain high and the computational power will grow proportionately.

Safe net has a chance to revolutionize distributed computing, once it has delivered on its foundations.

[bluebird]

Hi. The amount of messages on this forum is vast and I don’t have time to read them all. Could anyone please give me e link to previous discussions on adding computation to SAFE?