:IF: Announcing - MutAnt: Mutable Key/Value storage (prev: Anthill)

I’m not generally a meme person but in the interests of science… :slight_smile:

Well done :slight_smile:

5 Likes

Alright,

So I realized that upload and download over autonomi on my machine/network are just broken.
I’ve been fighting to put the simpliest of data and fetch it afterwards.
Well so I’ve rented a DigitalOcean small VM to try it out and OH SURPRISE
Everything I throw at it works. Big files, 4MB chunks, not a single error in the logs.

So I’ve uploaded 2 big files, is anybody willing to try it out on their end to check it if works correctly ?

I’m particularly interested if the downloads completes correcly, and in how much time

~40MB
$> mutant get -p b192124a7b2c5ffa68a97793a72d89c1f884d4efd001d918c6ac034d06b88fb70ba6bfb9dbab376c9a4c7274bf1b6347 > TheSimpsonsS1E1.mp4

~140MB
$> mutant get -p 94392c4cd50a91f53155583593d50b17b7730e32b62c2a0e730d78c8db9de38cf1a2507e03e81a2c84d6c20baa77291f > AmericanDadS18E16.mkv

It should work with the version 0.5.1, but if not you can try the version on the develop branch
(With the new daemon and JSON websocket interface, native/WASM client, background tasks, and more)

5 Likes

Both worked for me on mutant v0.5.1, on first attempt :+1:
~20s for the simpsons episode and ~2min for the american dad one.
I like the ETA feature ! Well done ! :clap:

3 Likes

TheSimpsonsS1E1.mp4
AmericanDadS18E16.mkv

Both worked fine for me - thank you :slight_smile:
mutant 0.5.1

1 Like

Amazing guys, thank you for trying <3

This is so frustrating that I’ve been working all that time with both my hands tied in my back haha

But this is such a great news that it works on the first try and with such good performances !
This is even better good news that those result come from the 0.5.1, because I’ve made a s**t-ton of performances improvements in the next version like having a pool of 20 Autonomi Client instead of just one instance (configurable with ENV) that is CONSIDERABLY improving both upload and download.

For instance, if I recall correctly, the 140MB file got uploaded in ~13mn on brand new scratchpads (that counts the Ethereum payment time)
When overwriting the scratchpads with new data I got closer to the 5mn mark.

So, thank you again, this is really valuable insight :slight_smile:
I’m gonna go back to it now, I hope I can release that new version this week. This will be the last step before I start working on the WebUI and the Official Mutable Index™.
Exciting ^^

7 Likes

Oh and sadly I removed that ETA feature, if one step in the process is taking too long to complete, it starts showing crazy times in the Million Years range.
I must admit that I liked that feature too, at least it kept you entertained while waiting way too long for an upload that would never finish anyway (Can feel the salt in my tone ? :p)
I might consider implement a better one that actually show a meaningful time, but this is not my priority.
What do you think ? Should I reactivate the old ETA feature in between, even if it shows crazy times sometimes ?

2 Likes

We want our Microsoft Time back!!!

2 Likes

Hey guys!

I’m thrilled to announce the release of Mutant v0.6.0! This version marks a major leap forward, introducing a robust daemon architecture and a host of new features to make your decentralized storage experience smoother, more powerful, and more reliable.

You can install/update to the latest version using:

$> cargo install mutant-daemon
$> cargo install mutant

Note that mutant-daemon is a new component you’ll want to install alongside the mutant CLI.

Here’s a rundown of the exciting new changes since v0.5.1:

Main Changes

:rocket: Daemon Architecture & Background Operations

MutAnt now features a persistent daemon process (mutant-daemon) that manages all network interactions and operations. This is a game-changer!

  • Background Processing: Long-running tasks like uploads and downloads can now be sent to the background, freeing up your terminal.
    $> mutant put my_large_file big_archive.zip --background
    
  • Task Management: Keep an eye on and manage your background tasks with ease.
    $> mutant tasks list
    $> mutant tasks get <task_id>
    $> mutant tasks stop <task_id>
    
  • Persistent Connectivity: The daemon maintains a stable connection to the Autonomi network, improving reliability.
  • Client-Server Model: The mutant CLI (and your own apps) now communicates with the mutant-daemon via WebSockets using the new mutant-protocol. For library users, there’s a new mutant-client crate to interact with the daemon.
    # Start/check the daemon (Note: Starts automatically when needed)
    $> mutant daemon start
    $> mutant daemon status
    
  • High Performance: Under the hood, the daemon uses a sophisticated worker architecture (10 workers, 100 concurrent operations) with work stealing to maximize throughput.

:sparkles: Automatic Resumable Uploads

Interrupted uploads? No problem! Mutant now automatically picks up right where you left off, making large uploads much more resilient to network hiccups. This significantly improves upon the previous manual resume process.

###:counterclockwise_arrows_button: Sync with Remote Storage (mutant sync)
Keep your local cache index perfectly synchronized with the state of your data on the Autonomi network.

$> mutant sync

This ensures your local view is always up-to-date.

:scroll: Fetch History (mutant ls --history)

Ever fetched some public data and then struggled to find its address again? Now, Mutant keeps a history of the public data you’ve accessed.

$> mutant ls --history

:broom: Purge Command (mutant purge)

The new mutant purge command allows you to proactively manage your storage pads. It performs a check on pads that might have encountered issues during creation and removes those that are not found or failed verification, helping to keep your storage healthy.

$> mutant purge my_key
$> mutant purge --all # To purge for all keys

:stethoscope: Improved Health Check

The mutant health-check command is now more powerful. It not only identifies problematic pads but can also automatically recycle pads that are not found or consistently fail, ensuring your keys remain in good health.

$> mutant health-check mykey --recycle

:speaking_head: Enhanced Public Data Sharing UX

When you store public data, Mutant now immediately provides you with its unique address, making it easier to share right away:

$> mutant put -p my_public_notes notes.txt
Public Address: 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a086db65fdc485f1737814c81ab6d61dab0

:books: Library Enhancements

For Rust developers using mutant-lib:

  • Keyless Public Fetching: Initialize a lightweight MutAnt instance for fetching public data without needing a private key:
    // Example: Initialize a public fetcher (defaults to Mainnet)
    let public_fetcher = MutAnt::init_public().await?;
    
  • Daemon Client (mutant-client): A new crate, mutant-client, is available to programmatically interact with the mutant-daemon from your Rust and Web applications (WASM/JS), enabling you to leverage the background processing and task management features.

What’s next ?

The journey doesn’t stop here! I’m already brewing up the next set of features:

  • Web UI: A user-friendly web interface to visualize and manage your data, powered by the new daemon.
  • Official Frontpage Index: An official, community-curated index to help you discover interesting public data shared by others. You’ll be able to submit your own data too!
  • And, of course, a few more secret experiments up my sleeve!

Daily Meme Public Upload

The daily meme tradition continues! I’ll be posting a new meme every day to this public key:
9620303d41cd65177e6763809f4780b1fa7d864a14d4ed0ed0322c5d4524fe406db65fdc485f1737814c81ab6d61dab0

You can download the meme of the day with:

$> mutant get -p a420224971527d61ce6ee21d850a07c243498c95808697e8fac23f461545656933016697d10b805c0fa26b50eb3532b2 daily_meme.jpg

Please try out Mutant v0.6.0 and share your feedback. Your support and suggestions are invaluable!

Happy Storing ! :slight_smile:

13 Likes

When I try to put a file it tells me I am missing the secret key

I try

willie@gagarin:~/Downloads$ mutant put  ~/Downloads/Professor_Oscar_Schwiglhofer_1923_2001.pdf
error: the following required arguments were not provided:
  <FILE>

Usage: mutant put <KEY> <FILE>

then this needs killed if I try to use $SECRET_KEY

willie@gagarin:~/Downloads$ mutant put  $SECRET_KEY ~/Downloads/Professor_Oscar_Schwiglhofer_1923_2001.pdf
Starting daemon...
Error: MutAnt(Network(InvalidKeyInput("Failed to create SecretKey from HASH: InvalidBytes")))

^C

What syntax should I use?

1 Like

Ow that’s a bug, since I’ve added the Daemon, it requires the wallet to start. I’m working on a fix to require the wallet only on the calls that need it.
Thank you for trying, I’m working on the fix right now, it will be available tomorrow eventually :slight_smile:

3 Likes

You are welcome :slight_smile:
Feel free to DM if there is anything you want (quietly) tested.

Thank you for all your efforts so far..

3 Likes
willie@gagarin:~/Downloads$ mutant put  ~/Downloads/Professor_Oscar_Schwiglhofer_1923_2001.pdf
error: the following required arguments were not provided:
  <FILE>

Usage: mutant put <KEY> <FILE>

For this, you actually need to specify a name for the file you are uploading. It is actually required because the whole thing is a key/value system.
I don’t take the name of the file by default, because you can also use raw bytes instead of a file in your programs

I would have wrote

mutant put schwiglhofer.pdf ~/Downloads/Professor_Oscar_Schwiglhofer_1923_2001.pdf
1 Like

I might take your word on that :stuck_out_tongue: Thank you

2 Likes

I just released a minor update v0.6.2

Changelog:

  • Daemon does not require wallet for public operations anymore
  • Remove the short -h for ls, use --history instead
  • Better update mechanism with data checksum and pad reuse. We don’t reupload existing data now !
  • Better public index management

You can update with

$> cargo install mutant-daemon
$> cargo install mutant

Don’t forget to restart your daemon !

$> mutant daemon restart
4 Likes

I got this on 3 different machines - all 24.04


 Compiling env_logger v0.11.8
   Compiling hyper v0.14.32
   Compiling warp v0.3.7
   Compiling autonomi v0.4.6
   Compiling mutant-lib v0.6.2
error[E0560]: struct `InitialPeersConfig` has no field named `disable_mainnet_contacts`
  --> /home/willie/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/mutant-lib-0.6.2/src/network/client.rs:77:21
   |
77 |                     disable_mainnet_contacts: true,
   |                     ^^^^^^^^^^^^^^^^^^^^^^^^ `InitialPeersConfig` does not have this field
   |
   = note: all struct fields are already assigned

For more information about this error, try `rustc --explain E0560`.
error: could not compile `mutant-lib` (lib) due to 1 previous error
error: failed to compile `mutant-daemon v0.6.2`, intermediate artifacts can be found at `/tmp/cargo-installoEYpec`.
To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.

Rust version is rustc 1.86.0 (05f9846f8 2025-03-31)

Did you try with --locked?

2 Likes

No
cargo install mutant-daemon --locked then?

1 Like

Ah yes I got the same thing, here it works if I add the --locked attribute

cargo install mutant-daemon --locked
cargo install mutant --locked

I’m working to fix this for the next time, thanks for the report :slight_smile:

1 Like

Yes that works but then

  Compiling mutant-protocol v0.6.2
   Compiling mutant-client v0.6.2
   Compiling mutant v0.6.2
    Finished `release` profile [optimized] target(s) in 26.31s
   Replacing /home/willie/.cargo/bin/mutant
    Replaced package `mutant v0.6.1` with `mutant v0.6.2` (executable `mutant`)
willie@gagarin:~/projects/maidsafe/formicaio$ mutant daemon restart
Daemon not running
Starting daemon...
Error: MutAnt(Network(InvalidKeyInput("Failed to create SecretKey from HASH: InvalidBytes")))
willie@gagarin:~/projects/maidsafe/formicaio$ echo $SECRET_KEY 
232dd014doyawantacoffeebabeface21c69ed5b5f1

So $SECRET_KEY is set OK

mmh the secret key is normally read from the ant client’s wallet in .local/share/autonomi/client/wallets/0xMY_PUBLIC_KEY
That file contains the private key I use with Mutant.
Try to import it via ant wallet import YOUR_KEY ?