CLI queries

[davidpbrown]

So, I’m just working on a CLI GUI… encouraging to see I’m not the only one…

If someone who knows can answer any of these, it’ll help tick them off… mostly these are minor but useful to be sure of them.

================
I wonder missing from CLI safe cat is missing options that
safe files get has
-e, --exists
-p, --preserve

That is if the url is a file location, then get complains it’s not a Files Container; so, the option I wonder becomes to use safe cat?.. following through with those options if wanted.

================
Wondering trivially why a link required for nrs add subname?

safe nrs add [OPTIONS] <PUBLIC_NAME>

…
-l, --link
The safe:// URL to link to. Usually a FilesContainer for a website. This should be
wrapped in double quotes on bash based systems. A link must be provided for a subname.
If you don’t provide it with this argument, you will be prompted to provide it
interactively

================

What is safe files sync without a recursive option?

================

NRS Create replies
New NRS Map created for “safe://bard”
The container for the map is located at safe://hyryygyikx63mgr1ce9hwewnufhgw1dcweodaqgxqq71xkiq9wdswgxkcyon7a
The entry points to safe://hyryyryunq4tbtcfxcpnzucuwym9msggp7ewdrfnr3wgi73k6xitp1x8scwnra?v=hc6o64ip96acj98ijoeu9b4zjgc3bp9cy7refhi74wa7qo7gg7kuo

is that the same as the datamap
and how is a datamap read?

What is [The entry points to ]?..
if that is the 1st ?v=
how do those v iterate… how to I know the order of them?
expecting there is still default to latest.

Does tree suggest detail by version?

================

and then if anyone has had success using the API, that would be interesting to see as an example… I haven’t yet been able to get flutter-rust to link to it.

[davidpbrown]

Just as a tease…

[chriso]

I’ll look into this in some detail later tonight, if nobody else does in the mean time.

[davidpbrown]

No rush… I’m just checking for a last few bugs and it’ll need to evolve over time. Have a good weekend…

[Southside]

You have inspired me to click on my long-neglected DuoLingo app.

Ruigsinneachd tèarainte dha na h-uile !!!

Well done, is there a git repo yet?

[chriso]

files get

I think I reproduced your first issue.

Upload the test data directory from the safe_network repo:

cargo run --bin safe --release -- files put resources/testdata
FilesContainer created at: "safe://hyryyry1cwbf6ujer1sj9br4gf6681k5qw471zaqy4ngpiazcqzcu9z88oynra?v=hf1yhh3bpw7ohw5dtw9zyaw48x4sabnqgkea8mrnqiiwds9cotihy"
+---+-------------------------------------------+--------------------------------------------------------------------+
| + | resources/testdata/.hidden.txt            | safe://hy8oycymskorc9j35jtc7n5qrcft85m31j1qwyop3wgokokzfw71mzga3de |
|---+-------------------------------------------+--------------------------------------------------------------------|
| + | resources/testdata/another.md             | safe://hy8oyeypg8tphr18ukfeatfkw7p1zoxcw1wa19enckue138ugj5omnbcr5h |
|---+-------------------------------------------+--------------------------------------------------------------------|
| + | resources/testdata/large_markdown_file.md | safe://hy8oyeyxnyrf9jfa8817bjc5xwbhh78yqg7qqstaowcx38ij6s1wkmfqajc |
|---+-------------------------------------------+--------------------------------------------------------------------|
| + | resources/testdata/noextension            | safe://hyryyyym8qq7xqhyqhm7ar9wedde6br5wcro5whnh3fjqoeoiq384y1gofr |
|---+-------------------------------------------+--------------------------------------------------------------------|
| + | resources/testdata/test.md                | safe://hy8oyeypfsyugk8y1dygyxasc3cfmxu6zk8s66jyq6x6ndo1ncu7gy5yjeh |
+---+-------------------------------------------+--------------------------------------------------------------------+

If I try to use files get to retrieve one of the files directly:

Error:
   0: You can target files only by providing a FilesContainer with the file's path

Location:
   sn_cli/src/subcommands/files_get.rs:338

Intuitively, this behaviour seems wrong to me. Even though files get is more geared toward working with containers, I would expect to be able to use the command with a direct file reference.

I suspect the reason for this is because of the distinction between immutable and mutable data, but I’ll need to verify that.

I don’t think the solution is to change the behaviour of cat. The safe cat command is supposed to be analogous to the unix cat command, which prints the contents of a file, so it doesn’t seem right to apply arguments to that relating to files.

I’ll look into changing the behaviour of files get to support retrieving a file.

nrs add

If you add a subname to an NRS topname, it needs to link to something, so it makes sense that you would be required to provide a link. I actually feel here that it may be worth changing this command to make the link argument mandatory, because if you don’t supply it optionally, it’s going to prompt you for it anyway. I don’t really see much advantage in pasting the link with the original command as opposed to pasting it after you’ve been asked for input.

Can you let me know please, what was your intuition here? Why is it surprising to you that you would have to supply a link?

files sync

There is already a --recursive argument on the files sync command:

Sync files to the SAFE Network

USAGE:
    safe files sync [OPTIONS] <LOCATION> [TARGET]

ARGS:
    <LOCATION>
            The source location

    <TARGET>
            The target FilesContainer to sync up source files with, optionally including the
            destination path (default is '/')

OPTIONS:
        --config-dir-path <CONFIG_DIR_PATH>
            Set the location for the config directory.

            Defaults to $HOME/.safe on Linux/macOS or %USERPROFILE%\.safe on Windows

            [env: SAFE_CONFIG_DIR_PATH=]

    -d, --delete
            Delete files found at the target FilesContainer that are not in the source location.
            This is only allowed when --recursive is passed as well

    -h, --help
            Print help information

        --json
            Use JSON as output serialisation format (alias of '--output json')

    -l, --follow-links
            Follow symlinks

    -n, --dry-run
            Perform a dry run of the command. No data will be written

    -o, --output <OUTPUT_FMT>
            Output data serialisation: [json, jsoncompact, yaml]

    -r, --recursive
            Recursively sync folders and files found in the source location

    -u, --update-nrs
            Automatically update the NRS name to link to the new version of the FilesContainer. This
            is only allowed if an NRS URL was provided, and if the NRS name is currently linked to a
            specific version of the FilesContainer

        --xorurl <XORURL_BASE>
            Base encoding for XOR-URLs. Currently supported: base32z (default), base32 and base64

nrs register

I think there is confusion coming from the word “map” here. This is referring to an NRS map, not a “datamap”.

The NRS map contains the information about the registered top name. If you do a safe cat on the container for the NRS map, you will see something like this:

NRS Map Container at safe://hyryygyzm4jqxhbakskbnkbjuwey68drdr1qwtgt568rx1hcmiitsu7c31on7a
Listing NRS map contents:
another.testdata: safe://hy8oyeypg8tphr18ukfeatfkw7p1zoxcw1wa19enckue138ugj5omnbcr5h

So for the registered top name, it will list all the subnames and what they point to.

We don’t actually have a “datamap” type in the network at the moment. Not that I’m aware of anyway.

We should probably just make it completely clear that we’re referring to an “NRS map” here, rather than just a “map”. I’ll get that changed.

Regarding the version, it’s not simply an integer that gets incremented. Rather, it’s the hash of the content for the entry at that point in time. Using integers is too simplistic for dealing with scenarios where there are concurrent updates to the same container.

I’m not actually sure how to get all the versions of a container. I’ll need to look into that and get back to you.

API Usage

Finally, regarding the API, can you be a little more specific about what you’re trying to do? We may be able to point you in the right direction.

[davidpbrown]

Thanks very helpful answers there.

@chriso

I’ll look into changing the behaviour of files get to support retrieving a file.

and I don’t know but I wonder that handling all types would be ideal … I’ve not seen enough examples and instances to have checked what follows back from these types but some reply from files get useful… even if that is a prompt then to use dog for these:
// SafeData::Register
// SafeData::SafeKey
// SafeData::NrsMapContainer
// SafeData::NrsEntry
// SafeData::Multimap

@chriso

Can you let me know please, what was your intuition here? Why is it surprising to you that you would have to supply a link?

Simply checking for consistency… registering a topname does not require a link. No strong view on it.

@chriso

There is already a --recursive argument on the files sync command:

Yes - it’s trivial but I couldn’t think what sync would be doing that is not recursive… is that option needed or is it inherently what sync is. If user is sync a file perhaps that is ok though odd choice but sync of folder would be a recursive over the contents?..

@chriso

We don’t actually have a “datamap” type in the network at the moment.

There are questions that will take time to answer about how the user has choice and datamap is tempted as an option for blocking … tempting that the get action would iterate over the datamap and a block on one chunk could be useful to stop the download. That would be a choice by the user for that… so, I would have a list of spam… never gonna give up on this … until I can block that thought… visibility on datamap then not obvious atm and the alt real option for blocking would be a pina if it’s client side after download.

@chriso

I’m not actually sure how to get all the versions of a container. I’ll need to look into that and get back to you.

Yes, unclear what the outputs are from dog etc are atm but would expect perhaps one version detail links back to the previous or there is a list in order those were created. Just a option to pull on the versions, would be a nice to have and something to add later.

@chriso

regarding the API, can you be a little more specific

So, sn_api suggests a clear intent to make available

certain options:

// from sn_api/src/app/mod.rs
// ------ The following is what’s meant to be the public API -------
//
// pub mod files;
// pub mod keys;
// pub mod multimap;
// pub mod nrs;
// pub mod register;
// pub mod resolver;
// pub mod wallet;
//
// pub use crate::safeurl::*;
// pub use consts::DEFAULT_XORURL_BASE;
// pub use sn_client::DEFAULT_NETWORK_CONTACTS_FILE_NAME;
// pub use sn_interface::network_knowledge::SectionTree;
// pub use xor_name::{XorName, XOR_NAME_LEN};
//

The sn_api/examples are fn main() and I’ve tried switching that
but I could do with the simplest example of a rust patch that can engage with sn_api.

Flutter is not obvious just yet in its documentation the way that it can make use of a workspace. It is simple against a single api.rs in a src folder of a native folder in it’s structure but calling out from that to a crate that is a copy of the safe_network workspace is not obvious to me but close… I have some visibility on it and the mixture of types is not helping…

I think atm that is should be possible to have
one folder that is a working flutter-rust-bridge
alongside another folder that is safe_network clone.

Then, in the flutter native/Cargo.toml like this:

Cargo:

[package]
name = “native”
version = “0.1.0”
edition = “2021”

[lib]
crate-type = [“cdylib”, “staticlib”]

[dependencies]
anyhow = “1”
flutter_rust_bridge = “1”
sn_api = { path = “/mnt/vault/flutter/x_safe_api/ABC/safe_network/sn_api” }

and then native/src/api.rs

code:

use bytes::Buf;
use color_eyre::{eyre::eyre, Result};
use sn_api::{resolver::SafeData, Safe};
use std::{env::args, net::SocketAddr};

#[allow(dead_code)]
pub fn poc() → String {
let test = exampleget(“safe://test”.to_string());
let result = test.unwrap();
return result.to_string()
}

// To be executed passing Safe network contact address and file Safe URL, e.g.:
// $ cargo run --release --example fetch_file 127.0.0.1:12000 safe://hy8oyeyqhd1e8keggcjyb9zjyje1m7ihod1pyru6h5y6jkmmihdnym4ngdf
#[tokio::main]
async fn exampleget() -> Result<()> {
    env_logger::init();

    // Skip executable name form args
    let mut args_received = args();
    args_received.next();

    // Read the network contact socket address from first arg passed
    let network_contact = args_received
        .next()
        .ok_or_else(|| eyre!("No Safe network contact socket address provided"))?;
    let network_addr: SocketAddr = network_contact
        .parse()
        .map_err(|err| eyre!("Invalid Safe network contact socket address: {}", err))?;
    println!("Safe network to be contacted at {}", network_addr);

    // Read URL from second argument passed
    let url = args_received
        .next()
        .ok_or_else(|| eyre!("No Safe URL provided as argument"))?;
    println!("Fetching file from Safe with URL: {}", url);

    // The Safe instance is what will give us access to the network API.
    let safe = Safe::connected(None, None, None, None).await?;

    println!("Connected to Safe!");

    // Now we can simply fetch the file using `fetch` API,
    // it will return not only the content of the file
    // but its metadata too, so we can distinguish what has
    // been fetched from the provided Safe-URL.
    match safe.fetch(&url, None).await {
        Ok(SafeData::PublicFile { data, .. }) => {
            let data = String::from_utf8(data.chunk().to_vec())?;
            println!("File content retrieved:\n{}", data);
        }
        Ok(other) => println!("Failed to retrieve file, instead obtained: {:?}", other),
        Err(err) => println!("Failed to retrieve file: {:?}", err),
    }

    Ok(())
}

that is the sn_api example with main() renamed as exampleget()
and a simple pull on that should see fn poc() do what flutter does well, following through to a dart that just presents that result.

I’m just one step away… and perhaps will crack it but gone round a number of iterations already… that rust code being sat separately from flutter, is a lot better than what I was trying merging flutter on top of the workspace!.. flutter-rust-bridge really needs to setup to allow obvious integration into complex workspace, rather than as now seemingly expecting just one api.rs to host all the code it interacts with. Looks very possible but just the small matter of how to do it

[chriso]

@davidpbrown

and I don’t know but I wonder that handling all types would be ideal … I’ve not seen enough examples and instances to have checked what follows back from these types but some reply from files get useful… even if that is a prompt then to use dog for these:
// SafeData::Register
// SafeData::SafeKey
// SafeData::NrsMapContainer
// SafeData::NrsEntry
// SafeData::Multimap

Are you suggesting that a files get would apply to these different data types?

To be honest, I’m not sure I agree. I think files should stick to dealing with files containers.

@davidpbrown

Simply checking for consistency… registering a topname does not require a link. No strong view on it.

The reason for this is because a topname is going to be available on a first come, first served basis. So you can register a name you know you want, but without necessarily having any content to link it to at that point in time. However, if you add a subname, that has to link to something.

@davidpbrown

There are questions that will take time to answer about how the user has choice and datamap is tempted as an option for blocking … tempting that the get action would iterate over the datamap and a block on one chunk could be useful to stop the download. That would be a choice by the user for that… so, I would have a list of spam… never gonna give up on this … until I can block that thought… visibility on datamap then not obvious atm and the alt real option for blocking would be a pina if it’s client side after download.

Sorry, but I’m not really sure what you mean when you use the term “datamap”. As I mentioned, “datamap” is not a type in the network at the moment.

@davidpbrown

Yes, unclear what the outputs are from dog etc are atm but would expect perhaps one version detail links back to the previous or there is a list in order those were created. Just a option to pull on the versions, would be a nice to have and something to add later.

I am definitely going to look into this. I’m going to finish what I’m working on just now then ask Josh if I can get some time to address these various usability issues that have came up recently.

@davidpbrown

The sn_api/examples are fn main() and I’ve tried switching that
but I could do with the simplest example of a rust patch that can engage with sn_api.

What do you mean by a “rust patch”?

The examples in sn_api/examples don’t look too comprehensive. Another way you can see the API being used is to look at the CLI code. In sn_cli/src/subcommands/ there is a module that corresponds to each sub command that’s available in the CLI. I’m not sure if that might help a bit more?

I have to be honest, I don’t know anything about Flutter. It looks kind of interesting, but I’ve never really been a fan of GUI programming, so it’s not something I’m likely to look into.

[Southside]

@chriso

The reason for this is because a topname is going to be available on a first come, first served basis. So you can register a name you know you want, but without necessarily having any content to link it to at that point in time. However, if you add a subname, that has to link to something.

Ahhh thanks @chriso makes sense now.

[Southside]

I wish this made more sense to me.

I am trying to store a wallet url at a new topname so I can get a human-friendly wallet address
Everything is newly created so how can I know/calculate the version hash?

willie@gagarin:~/.safe/accounts$ safe wallet create
Wallet created at: "safe://hyryynyz1bhk3zjd58xu5ieay458f8zbzacpdsmktw3pqmtexbp9hscsy4hb6o"



willie@gagarin:~/.safe/accounts$ safe nrs add -y -l "safe://hyryynyz1bhk3zjd58xu5ieay458f8zbzacpdsmktw3pqmtexbp9hscsy4hb6o"  wallet.mytopdomain
Error: 
   0: The destination you're trying to link to is versionable content. When linking to versionable content, you must supply a version hash on the url. The requested topname was not registered.
   1: UnversionedContentError: Register content is versionable. NRS requires the supplied link to specify a version hash.

Location:
   /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/core/src/convert/mod.rs:726

Suggestion: Please run the command again with the version hash appended to the link. The link should have the form safe://<url>?v=<versionhash>.

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
willie@gagarin:~/.safe/accounts$ safe nrs add -y -l "safe://hyryynyz1bhk3zjd58xu5ieay458f8zbzacpdsmktw3pqmtexbp9hscsy4hb6o?v=0"  wallet.mytopdomain
Error: 
   0: InvalidInput: v param could not be parsed as VersionHash. invalid: '0'

Location:
   sn_cli/src/subcommands/helpers.rs:272

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

[davidpbrown]

@chriso

I think files should stick to dealing with files containers.

Happy to go with your thought - my suggestions sometimes are just a naive user expectation. If there’s something there then just useful to have a sense what response is received. Given that directly to a file was an error, unclear if instances of those others do the same. Just want to handle all flavours of output from files get.

@chriso

I’m not really sure what you mean when you use the term “datamap”.

What is the list of chunks that make a file.
What is the option to block download of known content like spam.
Another thread was SN for those who do not want content
which still talks as if the list of xorurls was “datamap”.

@neo

The XOR storage system can allow people to make lists of XOR addresses (hash/signature) for the sections of a malware file.

@neo

Without the datamap these chunks are useless and unable to be decoded.

It’s not obvious to me that safe xorurl spawns that level of detail of the xorurls of chunks of a file but if it could, that would be interesting to see.

@chriso

In sn_cli/src/subcommands/ there is a module that corresponds to each sub command that’s available in the CLI.

That’s a great idea - thanks

[neo]

@chriso

mean when you use the term “datamap”. As I mentioned, “datamap” is not a type in the network at the moment.

Datamap is a term from the early days and I thought it still existed. It was the “map” for the file containing the chunks in order that makes up the file.

How is that info (list of chunks) kept now?

[chriso]

0: The destination you're trying to link to is versionable content. When linking to versionable content, you must supply a version hash on the url. The requested topname was not registered.
1: UnversionedContentError: Register content is versionable. NRS requires the supplied link to specify a version hash.

Hmm, well, this is a confusing situation, because if you look closely, there’s actually two different errors here. It’s telling you at the end that you haven’t registered the topname. If you didn’t already run nrs register, you need to use the --register-top-name flag with nrs add to create the topname at the same time as adding a subname.

So perhaps the versionable content error here is a red herring, or possibly both might apply.

I honestly can’t remember if we supported assigning NRS names to wallets. I’ll need to get back to you on that one.

[chriso]

Hey @dirvine

Would you please be able to help here regarding the “datamap” term? It’s a term I’ve heard you use, but as far as I’m aware, we don’t actually represent this as a type anywhere in the safe_network code.

[dirvine]

The data map is what self encryption returns. It’s the map of pre and post enc chunks with original lengths. To decrypt a file, you get the chunks this data map shows you nd use the post-encrypt hashes to decrypt the chunks into a file.

[Southside]

So a “datamap” is an internal transitory data structure only used for self-enc and had no connection with the data types that are explicitly stored/retrieved from the network?

Sorry but sometimes I need things set out in very simple terms.

[JPL]

I believe it’s an immutable blob

[dirvine]

Yes, to read a public file you read that blob and it tells you where all the bits you need are.

For private files you just don’t store the data map in plain, but either encrypt it or keep it local.

[Southside]

@chriso

you need to use the --register-top-name flag with nrs add to create the topname at the same time as adding a subname

I thought that was what I was doing with the -y flag?

My thinking with the command

safe nrs add -y -l "safe://hyryynyz1bhk3zjd58xu5ieay458f8zbzacpdsmktw3pqmtexbp9hscsy4hb6o" wallet.mytopdomain

Add a link safe://wallet.mytopdomain to safe:/mytopdomain
Make that link point to "safe://hyryynyz1bhk3zjd58xu5ieay458f8zbzacpdsmktw3pqmtexbp9hscsy4hb6o" which is the wallet I just created with the previous command.
Oh and by the way, this is the first you have heard of .mytopdomain, Here is the -y flag to register that top domain so the wallet subname can get tacked on to that

Its a kind of bassackwards way of doing things but I can see the convenience.

I can imagine a conversation amongst a team of workies going like…

“Away and put the wallpaper up, son”
“BTW, here is a -y flag. That will let you put up the frame and gyproc so you have something to put the wallpaper on”

[davidpbrown]

@dirvine

you read that blob and it tells you where all the bits you need are.

Not obvious that option is available for a user to then make use of that blob and choose to block or filter, based on a known chunk - whether they are blocking spam or viruses or conservative manifestos, or AI bot content, matters not.

Knowing the full file xorurl, is different - that does not work for blacklists as people will not share their opinions; where the suggestion was that blocking a chunk might work. Still, there is a problem perhaps for one chunk files.

Obviously the need to see the network nodes blind to content matters most of all; but the next layer up where users start to introduce their interest, becomes either a block or a less efficient service that filters, or more difficult an expectation the user would download a file before managing it - that seems inefficient.

Perhaps this tempts the network will be driven largely by service providers that do cater for user interests where they don’t want raw data directly??

I guess the point is that once its resolved what the network must do, we will have to consider what users might want and the many options for bridging the differences.