Enable creating and modifying of FilesContainers within the SAFE Browser environment

Development RFCs
1

Enable creating and modifying of FilesContainers within the SAFE Browser environment

Note: This RFC has been created as a pull request to the Maisafe RFC repository here, I have published this topic to get comments and feedback on the proposed API changes.

Summary

The NodeJS libraries work via the existing SAFE network CRUD conventions: we take files, we upload them to a network and generate an XORURL we can reference as a result. In the browser we do not have access to a native file sytem, so the underlying NodeJS libraries which the browser calls must allow the uploading of raw bytes, and the creation of empty FilesContainers.

Conventions

  • The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.

Motivation

At the moment, there is no browser native way to create an empty FilesContainer (since the previous convention was the creation of an NFS container which is no longer possible).

Detailed design

At the moment, the following API calls are available (and implemented in the NodeJS rust library) with regards to FilesContainers:

  • files_container_create
  • files_container_sync
  • files_container_get
  • files_container_add
  • files_container_add_from_raw

Logic would dictate that the browser would call files_container_create to create the files container, however first argument to this method location: &str requires a path to a directory or file which will be uploaded to the FilesContainer. The same issues also exist for files_container_sync and files_container_add. files_container_add_from_raw is usable.

I propose the addition of 1 new API to be added to the NodeJS library (and thereby allowed for use via the SAFE Browser):

  • files_container_create_empty

files_container_create_empty

This method will create an empty FilesContainer and return the XORURL of the created FilesContainer, or throw an Error.

It would make sense to implement this function here and expose it to the NodeJS libraries.

In NodeJS pseudocode, the interface would be:

function files_container_create_empty(
    bool dry_run
) : string;

Drawbacks

N/A

Alternatives

The alternative is that FilesContainers can not be created from within the browser, which will significantly limit the abilities of SAFE browser native web apps.

Unresolved questions

The actual code needs to be written by the Maidsafe team.

7 Likes
2

The ability to create empty FilesContainer is important as noted IMO, and there’s a related use case that we may want to consider: the ability to create empty (sub)containers/(sub)directories in general. This was not supported in the old NFS API and was a severe limitation. AFAIK this is anticipated but the detail not yet clear in terms of implementation or API.

Also, in addition to create from string, I think it is worth considering (if not necessarily implmementing yet) a way to supply a ReadableStream (whatwg rather than Nodejs, or both whatwg and Nodejs style). This will reduce memory use when uploading large files, and we can expect videos etc. will be an important user requirement - e.g. streaming video from mobile to the network. So maybe we can add a placeholder for say files_container_add_from_stream?

9 Likes
3

Agreed, the files_container_add_from_stream addition sounds reasonable.

Presumably the use case would be to pass the handle from a fetch response to files_container_add_from_stream and write the response stream directly to the SAFE network without it ever having to enter the application’s memory?

Do you want to add it to this RFC or shall I? I’m not wholly familiar with the ReadableStream interface so it’s probably best if it’s you, I don’t want to create an API where I don’t fully understand the subject area. If you fork this repo: GitHub - SAFEPublishing/rfcs: Request for Comment (RFC) papers and discussions on Project SAFE core libraries and APIs and submit a pull request to the master branch I’ll merge it in and update the upstream pull request in the maidsafe repo.

I’ll have a think about the subdirectory issue as well, I need to rationalise exactly how the API would work and will get the RFC updated later today. The more I think about it, the more I don’t know if directories even exist within the safe network, there’s a function here which generates normalised file paths and I’m not sure if the network itself has a concept of directories. Perhaps blah/blah2.png is treat at face value, just a raw string with no meaning, rather than an imploded directory path.

Perhaps the ever knowledgeable @bochaco could elucidate us?

2 Likes
4

Yes, or for uploading files etc. I’m not that familiar with streams either - just started using them - but happy to do a PR.

I already have a fork of maidsafe/rfcs and when I try to fork your repo github tells me that - so I can’t fork your repo. I’ll see if I can edit your PR and push back to that but I’m in uncharted territory! @Shane on second thoughts - I see it isn’t obvious how to contribute to your PR and don’t want to go down a rabbit hole. I’ll leave this for now.

3 Likes
5

I’m a bit confused here. @Shane the issue you’re facing is accessing files, right?

Can you not use an <input type=file> to securely get a file path, and then pass that to the node apis?

(I recognise the other APIs may still be useful, but just wondering about exactly what you’re facing w/ files)

Also, I think we have an add from raw data api in safe-api already. that @danda worked up? (or @bochaco). So one part may just be exposing that to the nodejs lib.

I’m not sure if that API is yet pipeable on the CLI side. But if so that’s be the start for any writeable stream API I’d guess.

I can see how adding a create_empty api may well be useful. :+1:

6

Can str parameter just be optional? Creating additional function in SAFE Api just to convert String to ArrayBuffer is superficial in my opinion. Streaming idea looks good although.

2 Likes
7

Hi Josh, this only works when the file is stored in the browsers PC, but not when it’s something that has been generated entirely within the current web page.

Imagine, for instance, a textarea containing a user’s comment on a Youtube video, where the resulting comment needs to be pushed to the SAFE network.

If the javascript is running entirely within the browser, it’s not going to have access to any sort of file system.

1 Like
8

Okay grand. Thanks for clarifying. :+1: I thought you were stuck on the former, so that’d get you going.

yup, all makes sense. So we need to expose the “from raw” type API, that I will hunt down now…

edit: Or as a string, which is also what you’re saying above. Understanding now :slight_smile:

3 Likes
9

That’s a reasonable comment and one I’m happy to take on board, it was just because it seems more native to the browser to use a string rather than an ArrayBuffer and would mean we wouldn’t have to be constantly be converting between the two.

There are pros and cons to both methods, but if it’s literally just a single method in the libraries which converts from a string to a buffer and then passes the buffer to the add_from_raw method, it would be better (in my opinion, of course) if that was implemented in one single place, rather than every developer having to reimplement it.

1 Like
10

It already is :slight_smile:

1 Like
11

Sure, but then every SAFE browser developer in the future has to include that method, rather than it being implemented in a single wrapper function in the safe-nodejs libs

12

I’m not sure about the from_string one either, also wouldn’t that be a problem for char encoding? this is what you can do in a single line if you want: electron-neon-safe/renderer.js at master · bochaco/electron-neon-safe · GitHub

1 Like
13

Fair enough, you’ve convinced me. Feedback taken on board (@bochaco, @loziniak) , I’ll remove the string method now and leave the create_empty call, and wait for @happybeing’s stream method to be added

14

I’m just concerned about the char encoding, it seems we’ll need to then provide mechanisms to specify it, the network only takes bytes, it’s up to the app and user to decide how those bytes will be represented/encoded when reading/writing those bytes?

2 Likes
15

@Shane on second thoughts - I see it isn’t obvious how to contribute to your PR and don’t want to go down a rabbit hole so I’ll leave this for now.

16

@happybeing here you go:

2 Likes
17

Github won’t let me fork your fork as I have already forked maidsafe/rfcs. I could probably achieve this by messing with remotes, but would rather not get into that.

18

Ahhhh, gotcha. In that case - no worries. I probably won’t have a chance to fiddle with the ReadableStream implementation tonight (going out for a meal with my fiance, woop woop, forgot to mention in my return post that we got engaged over Christmas!) so I’ll update the RFC tomorrow.

10 Likes
19

Congratulations @Shane - have a great night out. :partying_face:

3 Likes
20

Congratulations Shane, have a great meal tonight.

2 Likes