:IF: Announcing - MutAnt: Mutable Key/Value storage (prev: Anthill)

Apps
21

Hey guys !

I’ve just released mutant v0.5.1.

You can install/update it using cargo install mutant.

This is a complete rewrite from the ground up, with a cleaner API, better performances and more stable features.

Main changes

  • Configurable Upload Size: Split the data into smaller of bigger chunks to optimize the upload speed and storage costs (default is 2MiB).

    When doing a put you can now provide the following flags to control how much data is uploaded per pads. Smaller pads are faster and more reliable but use more storage ($$$)
    Bigger pads will take a lot more time to upload and download (and might fail) but will use your precious storage space more efficiently
    - $> cat my_file | mutant put my_key --mode lightest Chunk size: 0.5MiB
    - $> cat my_file | mutant put my_key --mode light Chunk size: 1MiB
    - $> cat my_file | mutant put my_key --mode medium Chunk size: 2MiB [default]
    - $> cat my_file | mutant put my_key --mode heavy Chunk size: 3MiB
    - $> cat my_file | mutant put my_key --mode heaviest Chunk size: 4MiB

    The --heaviest option is the maximum size allowed for a pad, but will very likely fail because of the actual mainnet network state.

  • Recycling mechanism: Automatically recycle bad pads that take time to answer to stay performant.

    When doing a put, if a pad fails to be uploaded 3 times in a row, it will be recycled.
    If the verification fails repeatedly during 10 minutes, the pad is recycled.

    A recycled pad is put in the verification list in your index, to be later check by a mutant purge operation to discard it or add it back to the free list.
    It is replaced by a new pad either taken from the free list or generated.

  • Health Check: Perform a health check on scratchpads that should have been created but cannot be retrieved. Recycles the pads that are not found.

    When doing big uploads, it is more than likely that some pads will fail to be created or propagated properly.
    Now you can call mutant health-check my_key to check for such pads and mark them for retry, and the key marked as incomplete.
    You will have to call again the exact same put command you called with the exact same data and parameters to resume the upload for the specific pads that failed.

  • Better retry mechanisms: The library will now automatically retry operations that fail due to issues.

    I’m pleased to announce that with this release, the uploads/downloads are now more reliable than ever on the mainnet.
    It is still not perfect, but it is a good step forward, and most importantly you have now very little chance to actually lose the pads you pay for.

  • Import/Export: Import and export your index to a file to backup your data.

  • Overall performance improvements: We now don’t do redundant calls to check the status of the pads during uploads/downloads

What’s next ?

I’m am already working on the next version of the library that will bring many interesting features, like:

  • Daemon mode: Run the library in daemon mode to automatically upload/download and manage data in the background. You will be able to interact with it in JSON via HTTP or websocket.
  • Web UI: A web UI to visualize and manage your data. Will use the daemon mode to interact with the library.
  • Official Frontpage Index: An official index that will be shown by default in the web UI to allow you to discover new data posted by others. You will be able to add your own data to it and mutate it at will.
  • Some more experiments that I will disclose later :stuck_out_tongue:

Daily meme public upload

Every day from now on I will post a new meme on this public key: 80a9296e666d9f6d0dad8672e8a17fe15d00bae67e35ae4a471aabc2351ac85c4af654be0ea4c2ababf9a788b751ceac

You can download the meme of the day with the following command:

$> mutant get -p 80a9296e666d9f6d0dad8672e8a17fe15d00bae67e35ae4a471aabc2351ac85c4af654be0ea4c2ababf9a788b751ceac > daily_meme.jpg

Please try it yourself and let me know what you think !

8 Likes
22

I’m not generally a meme person but in the interests of science… :slight_smile:

image
image631×650 57.1 KB

Well done :slight_smile:

5 Likes
23

Alright,

So I realized that upload and download over autonomi on my machine/network are just broken.
I’ve been fighting to put the simpliest of data and fetch it afterwards.
Well so I’ve rented a DigitalOcean small VM to try it out and OH SURPRISE
Everything I throw at it works. Big files, 4MB chunks, not a single error in the logs.

So I’ve uploaded 2 big files, is anybody willing to try it out on their end to check it if works correctly ?

I’m particularly interested if the downloads completes correcly, and in how much time

~40MB
$> mutant get -p b192124a7b2c5ffa68a97793a72d89c1f884d4efd001d918c6ac034d06b88fb70ba6bfb9dbab376c9a4c7274bf1b6347 > TheSimpsonsS1E1.mp4

~140MB
$> mutant get -p 94392c4cd50a91f53155583593d50b17b7730e32b62c2a0e730d78c8db9de38cf1a2507e03e81a2c84d6c20baa77291f > AmericanDadS18E16.mkv

It should work with the version 0.5.1, but if not you can try the version on the develop branch
(With the new daemon and JSON websocket interface, native/WASM client, background tasks, and more)

4 Likes
24

Both worked for me on mutant v0.5.1, on first attempt :+1:
~20s for the simpsons episode and ~2min for the american dad one.
I like the ETA feature ! Well done ! :clap:

3 Likes
25

TheSimpsonsS1E1.mp4
AmericanDadS18E16.mkv

Both worked fine for me - thank you :slight_smile:
mutant 0.5.1

1 Like
26

Amazing guys, thank you for trying <3

This is so frustrating that I’ve been working all that time with both my hands tied in my back haha

But this is such a great news that it works on the first try and with such good performances !
This is even better good news that those result come from the 0.5.1, because I’ve made a s**t-ton of performances improvements in the next version like having a pool of 20 Autonomi Client instead of just one instance (configurable with ENV) that is CONSIDERABLY improving both upload and download.

For instance, if I recall correctly, the 140MB file got uploaded in ~13mn on brand new scratchpads (that counts the Ethereum payment time)
When overwriting the scratchpads with new data I got closer to the 5mn mark.

So, thank you again, this is really valuable insight :slight_smile:
I’m gonna go back to it now, I hope I can release that new version this week. This will be the last step before I start working on the WebUI and the Official Mutable Index™.
Exciting ^^

5 Likes
27

Oh and sadly I removed that ETA feature, if one step in the process is taking too long to complete, it starts showing crazy times in the Million Years range.
I must admit that I liked that feature too, at least it kept you entertained while waiting way too long for an upload that would never finish anyway (Can feel the salt in my tone ? :p)
I might consider implement a better one that actually show a meaningful time, but this is not my priority.
What do you think ? Should I reactivate the old ETA feature in between, even if it shows crazy times sometimes ?

2 Likes
28

We want our Microsoft Time back!!!

2 Likes
29

Hey guys!

I’m thrilled to announce the release of Mutant v0.6.0! This version marks a major leap forward, introducing a robust daemon architecture and a host of new features to make your decentralized storage experience smoother, more powerful, and more reliable.

You can install/update to the latest version using:

$> cargo install mutant-daemon
$> cargo install mutant

Note that mutant-daemon is a new component you’ll want to install alongside the mutant CLI.

Here’s a rundown of the exciting new changes since v0.5.1:

Main Changes

:rocket: Daemon Architecture & Background Operations

MutAnt now features a persistent daemon process (mutant-daemon) that manages all network interactions and operations. This is a game-changer!

  • Background Processing: Long-running tasks like uploads and downloads can now be sent to the background, freeing up your terminal.
    $> mutant put my_large_file big_archive.zip --background
  • Task Management: Keep an eye on and manage your background tasks with ease.
    $> mutant tasks list
$> mutant tasks get <task_id>
$> mutant tasks stop <task_id>
  • Persistent Connectivity: The daemon maintains a stable connection to the Autonomi network, improving reliability.
  • Client-Server Model: The mutant CLI (and your own apps) now communicates with the mutant-daemon via WebSockets using the new mutant-protocol. For library users, there’s a new mutant-client crate to interact with the daemon.
    # Start/check the daemon (Note: Starts automatically when needed)
$> mutant daemon start
$> mutant daemon status
  • High Performance: Under the hood, the daemon uses a sophisticated worker architecture (10 workers, 100 concurrent operations) with work stealing to maximize throughput.

:sparkles: Automatic Resumable Uploads

Interrupted uploads? No problem! Mutant now automatically picks up right where you left off, making large uploads much more resilient to network hiccups. This significantly improves upon the previous manual resume process.

###:counterclockwise_arrows_button: Sync with Remote Storage (mutant sync)
Keep your local cache index perfectly synchronized with the state of your data on the Autonomi network.

$> mutant sync

This ensures your local view is always up-to-date.

:scroll: Fetch History (mutant ls --history)

Ever fetched some public data and then struggled to find its address again? Now, Mutant keeps a history of the public data you’ve accessed.

$> mutant ls --history

:broom: Purge Command (mutant purge)

The new mutant purge command allows you to proactively manage your storage pads. It performs a check on pads that might have encountered issues during creation and removes those that are not found or failed verification, helping to keep your storage healthy.

$> mutant purge my_key
$> mutant purge --all # To purge for all keys

:stethoscope: Improved Health Check

The mutant health-check command is now more powerful. It not only identifies problematic pads but can also automatically recycle pads that are not found or consistently fail, ensuring your keys remain in good health.

$> mutant health-check mykey --recycle

:speaking_head: Enhanced Public Data Sharing UX

When you store public data, Mutant now immediately provides you with its unique address, making it easier to share right away:

$> mutant put -p my_public_notes notes.txt
Public Address: 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a086db65fdc485f1737814c81ab6d61dab0

:books: Library Enhancements

For Rust developers using mutant-lib:

  • Keyless Public Fetching: Initialize a lightweight MutAnt instance for fetching public data without needing a private key:
    // Example: Initialize a public fetcher (defaults to Mainnet)
let public_fetcher = MutAnt::init_public().await?;
  • Daemon Client (mutant-client): A new crate, mutant-client, is available to programmatically interact with the mutant-daemon from your Rust and Web applications (WASM/JS), enabling you to leverage the background processing and task management features.

What’s next ?

The journey doesn’t stop here! I’m already brewing up the next set of features:

  • Web UI: A user-friendly web interface to visualize and manage your data, powered by the new daemon.
  • Official Frontpage Index: An official, community-curated index to help you discover interesting public data shared by others. You’ll be able to submit your own data too!
  • And, of course, a few more secret experiments up my sleeve!

Daily Meme Public Upload

The daily meme tradition continues! I’ll be posting a new meme every day to this public key:
9620303d41cd65177e6763809f4780b1fa7d864a14d4ed0ed0322c5d4524fe406db65fdc485f1737814c81ab6d61dab0

You can download the meme of the day with:

$> mutant get -p a420224971527d61ce6ee21d850a07c243498c95808697e8fac23f461545656933016697d10b805c0fa26b50eb3532b2 daily_meme.jpg

Please try out Mutant v0.6.0 and share your feedback. Your support and suggestions are invaluable!

Happy Storing ! :slight_smile:

12 Likes
30

When I try to put a file it tells me I am missing the secret key

I try

willie@gagarin:~/Downloads$ mutant put  ~/Downloads/Professor_Oscar_Schwiglhofer_1923_2001.pdf
error: the following required arguments were not provided:
  <FILE>

Usage: mutant put <KEY> <FILE>

then this needs killed if I try to use $SECRET_KEY

willie@gagarin:~/Downloads$ mutant put  $SECRET_KEY ~/Downloads/Professor_Oscar_Schwiglhofer_1923_2001.pdf
Starting daemon...
Error: MutAnt(Network(InvalidKeyInput("Failed to create SecretKey from HASH: InvalidBytes")))

^C

What syntax should I use?

1 Like
31

Ow that’s a bug, since I’ve added the Daemon, it requires the wallet to start. I’m working on a fix to require the wallet only on the calls that need it.
Thank you for trying, I’m working on the fix right now, it will be available tomorrow eventually :slight_smile:

3 Likes
32

You are welcome :slight_smile:
Feel free to DM if there is anything you want (quietly) tested.

Thank you for all your efforts so far..

3 Likes
33 
willie@gagarin:~/Downloads$ mutant put  ~/Downloads/Professor_Oscar_Schwiglhofer_1923_2001.pdf
error: the following required arguments were not provided:
  <FILE>

Usage: mutant put <KEY> <FILE>

For this, you actually need to specify a name for the file you are uploading. It is actually required because the whole thing is a key/value system.
I don’t take the name of the file by default, because you can also use raw bytes instead of a file in your programs

I would have wrote

mutant put schwiglhofer.pdf ~/Downloads/Professor_Oscar_Schwiglhofer_1923_2001.pdf
1 Like
34

I might take your word on that :stuck_out_tongue: Thank you

2 Likes
35

I just released a minor update v0.6.2

Changelog:

  • Daemon does not require wallet for public operations anymore
  • Remove the short -h for ls, use --history instead
  • Better update mechanism with data checksum and pad reuse. We don’t reupload existing data now !
  • Better public index management

You can update with

$> cargo install mutant-daemon
$> cargo install mutant

Don’t forget to restart your daemon !

$> mutant daemon restart
4 Likes
36

I got this on 3 different machines - all 24.04


 Compiling env_logger v0.11.8
   Compiling hyper v0.14.32
   Compiling warp v0.3.7
   Compiling autonomi v0.4.6
   Compiling mutant-lib v0.6.2
error[E0560]: struct `InitialPeersConfig` has no field named `disable_mainnet_contacts`
  --> /home/willie/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/mutant-lib-0.6.2/src/network/client.rs:77:21
   |
77 |                     disable_mainnet_contacts: true,
   |                     ^^^^^^^^^^^^^^^^^^^^^^^^ `InitialPeersConfig` does not have this field
   |
   = note: all struct fields are already assigned

For more information about this error, try `rustc --explain E0560`.
error: could not compile `mutant-lib` (lib) due to 1 previous error
error: failed to compile `mutant-daemon v0.6.2`, intermediate artifacts can be found at `/tmp/cargo-installoEYpec`.
To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.

Rust version is rustc 1.86.0 (05f9846f8 2025-03-31)

37

Did you try with --locked?

2 Likes
38

No
cargo install mutant-daemon --locked then?

1 Like
39

Ah yes I got the same thing, here it works if I add the --locked attribute

cargo install mutant-daemon --locked
cargo install mutant --locked

I’m working to fix this for the next time, thanks for the report :slight_smile:

1 Like
40

Yes that works but then

  Compiling mutant-protocol v0.6.2
   Compiling mutant-client v0.6.2
   Compiling mutant v0.6.2
    Finished `release` profile [optimized] target(s) in 26.31s
   Replacing /home/willie/.cargo/bin/mutant
    Replaced package `mutant v0.6.1` with `mutant v0.6.2` (executable `mutant`)
willie@gagarin:~/projects/maidsafe/formicaio$ mutant daemon restart
Daemon not running
Starting daemon...
Error: MutAnt(Network(InvalidKeyInput("Failed to create SecretKey from HASH: InvalidBytes")))
willie@gagarin:~/projects/maidsafe/formicaio$ echo $SECRET_KEY 
232dd014doyawantacoffeebabeface21c69ed5b5f1

So $SECRET_KEY is set OK