:IF: Colony - Command line edition. RELEASED, now in color!

[zettawatt]

Introducing Colony Utils!

After weeks of toiling away, I finally have an application that you all can play with. While my frontend guy is still working on the easy-to-use-for-normal-people GUI frontend for colonylib, I decided it would be good to have a CLI for us techy folks. Let’s dig in!

TL;DR, just want to try it on the main network (linux only):

wget https://github.com/zettawatt/colony-utils/releases/download/colony-daemon_v0.1.2/colony-daemon
wget https://github.com/zettawatt/colony-utils/releases/download/colony-daemon_v0.1.2/colony-cli
chmod +x colony-daemon
chmod +x colony-cli
sudo mv colony-daemon /usr/local/bin
sudo mv colony-cli /usr/local/bin
colony-daemon
# follow prompts to enter a password, generate a new seed phrase, and enter your ETH wallet private key to pay for uploads
# leave this running

In another terminal, run the following (takes ~4 minutes from a WiFi connection to populate the first time):

colony-cli add pod "main"
colony-cli add ref "main" 860b439c8b8cd01a7ee7f3b8791793963dfedac3bf0d5b998e57160256ef83a93ec010f8fa6d120ef3c5cbef9b59c43a
colony-cli refresh --depth 0
colony-cli search text "beg blag and steal"

First, GIFs!

Accessing Help

colony-cli --help
colony-cli sarch --help
colony-cli search --help
colony-cli search text --help

Search by text

colony-cli search text "beg blag"
colony-cli search text "ubuntu"
colony-cli search text "feyn"

Display info information about an address

colony-cli search subject e7bb1b87c1f0e07cdb76ba5e82a425a8da712940c2d3553aa6791494e92aa54d
colony-cli search subject c859818c623ce4fc0899c2ab43061b19caa0b0598eec35ef309dbe50c8af8d59

Add a pod and add new metadata

colony-cli search subject 142925b90fc5f6e38807713303fbdc91bfcd893a3c0f597e53764087fd70b123
colony-cli add pod "new pod"
colony-cli put "new pod" 142925b90fc5f6e38807713303fbdc91bfcd893a3c0f597e53764087fd70b123 '{\
  "@context": {"schema": "http://schema.org/"},\
  "@type": "schema:SoftwareApplication",\
  "@id": "ant://142925b90fc5f6e38807713303fbdc91bfcd893a3c0f597e53764087fd70b123",\
  "schema:name": "ubuntu-25.04-desktop-amd64.iso",\
  "schema:operatingSystem": "Ubuntu 25.04",\
  "schema:applicationCategory": "Operating System",\
  "schema:processorRequirements": "amd64",\
  "schema:contentSize": "5.7 GB"\
}'
colony-cli search subject 142925b90fc5f6e38807713303fbdc91bfcd893a3c0f597e53764087fd70b123

Modify and add metadata to an existing file in the graph

colony-cli search subject c859818c623ce4fc0899c2ab43061b19caa0b0598eec35ef309dbe50c8af8d59
colony-cli put "new pod" c859818c623ce4fc0899c2ab43061b19caa0b0598eec35ef309dbe50c8af8d59 '{\
  "@context": {"schema": "http://schema.org/"},\
  "@type": "schema:MediaObject",\
  "@id": "ant://c859818c623ce4fc0899c2ab43061b19caa0b0598eec35ef309dbe50c8af8d59",\
  "schema:description": "Beg Blag and Steal - Live",\
  "schema:contentSize": "4MB"\
}'
colony-cli search subject c859818c623ce4fc0899c2ab43061b19caa0b0598eec35ef309dbe50c8af8d59

Unfiltered SPARQL JSON Results

colony-cli search subject c859818c623ce4fc0899c2ab43061b19caa0b0598eec35ef309dbe50c8af8d59 --json

What is it?

This is a CLI for colonylib: a local first, metadata sharing and semantic search library for Autonomi. What it is not is a replacement for the Autonomi CLI. Use this for handling metadata and performing searches of the network. Use the Autonomi CLI to upload/download files.

The application and all of the documentation can be found on github here: colony-utils.

Here is an architectural overview:

┌─────────────────┐    HTTP/REST    ┌─────────────────┐
│   colony-cli    │ ──────────────► │  colony-daemon  │
│                 │                 │                 │
│ • CLI Interface │                 │ • REST API      │
│ • Progress Bars │                 │ • JWT Auth      │
│ • Colored Output│                 │ • Job Queue     │
└─────────────────┘                 └─────────────────┘
                                             │
                                             ▼
                                    ┌─────────────────┐
                                    │   colonylib     │
                                    │                 │
                                    │ • PodManager    │
                                    │ • DataStore     │
                                    │ • KeyStore      │
                                    └─────────────────┘
                                             │
                                             ▼
                                    ┌─────────────────┐
                                    │ Autonomi Network│
                                    │                 │
                                    │ • Decentralized │
                                    │ • Immutable     │
                                    │ • Secure        │
                                    └─────────────────┘
                                    

There are 2 applications: colony-daemon and colony-cli. The colony-daemon runs in the background and serves up REST endpoints for colonylib API functions. What dweb and anttp do for Autonomi operations, colony-daemon does for colonylib. The colony-cli interacts with these endpoints and provides a CLI that you can talk to from your favorite terminal.

It is designed to be cross platform and should compile on Linux, Mac, and Windows. I provide precompiled binaries for Linux based OSes on the github page for easy install. Windows/Mac users will need to compile from source or run cargo install to get these built (will build precompiled binaries in the future, haven’t gotten there yet).

What can it do today?

In short, everything! Upload metadata for files, share your pods with others, and perform semantic searches for everything it can find! All the things I talked about that Colony should do, it now does.

I have created initial metadata for all of the items on the Community Public File Directory post and uploaded them to a pod on the main network. After installing and running colony-daemon, you can run colony-cli commands to search for files using a fuzzy search, just like you would on Brave, DuckDuckGo, or those other search engines. It also follows the “web of trust” idea by sorting content on number of matches first and secondly on how “far away” that information is from you. In other words, metadata from you or your direct contact is ranked higher than metadata that is 5 layers deep in the friend of a friend of a friend graph.

How to run it

The top of the post had the quick start. Tons of info is in the README on the github page. Command specific info can be found by typing your standard --help argument. Subcommands in the CLI also have help info. I think it is pretty intuitive, hopefully that assumption stands up to public scrutiny

Why I went this route

Well for one, the fancy GUI is taking a while, and I needed a way to easily test colonylib. Writing tests for CLI is much easier to automate that button clicks. I found a TON of issues already and got those sorted. Secondly, the success of dweb and anttp showed that the community really likes the REST API concept for building applications. With colony-daemon and colony-cli, I was able to meet both goals in a relatively short amount of time. For the next 3 weeks, I’ll be testing testing testing. Any help from you guys would be greatly appreciated! And for anyone that wants to integrate the CLI or REST API into their application, let me know! I’m happy to help where I can. There is a TON of documentation here that describe pretty much everything there is to know about how this works.

What’s left and known issues:

There are couple known bugs in the CLI. The pod listing format hasn’t been cleaned up yet, so it spits out the raw SPARQL results format. The search by types doesn’t work yet. The search by text is supposed to search for exact matches within "", but doesn’t, so there is some work to do on the fuzzy matching logic. Probably more, let me know what you find!

I left error reporting on in the logger. On main net the Autonomi client connection is very chatty. You’ll probably see a ton of errors floating by in the daemon window, but as long as the app doesn’t crash, these can be ignored.

I need a method to change your password as well as change your ETH wallet key. Right now this is possible by deleting the data directory and calling a refresh to pull the data back down from Autonomi. It works, but it kind of clunky.

I’ll be making a video doing a walk through of how to use this and highlight some of the features, but its going to take me some time to get to that. I wanted to get this out there ASAP!

Costs and time

The application is local first, Autonomi is only used to download/upload from the network when you call the specific upoad or refresh commands. For downloading only, you never have to upload to Autonomi. The same is true for the Colony GUI, the idea being for many users, they just want to download stuff, and this gets them into the ecosystem without having to do all the crypto gymnastics. But if you never upload, you can’t save your preferences to the network and you can’t share your metadata with friends. Get them hooked and pull them in.

To upload the initial pods (1 configuration pod and one metadata pod) cost about 2.5 cents when I ran earlier today. Since pods are composed of scratchpads/pointers and the metadata itself is pretty small, this application will have a super low cost in the long run, nothing compared to the cost of uploading actual data!

For times, here is what I saw today on a WiFi connection at my house to the Autonomi main net:

• ~5 minutes for initial upload of 2 pods
• ~4 minutes for refresh (i.e. fetching from the network)
• ~30 seconds to upload updates to an existing pod

The vast majority of time for the initial refresh/upload is on a particular operation that I should be able to multithread. Obviously its going to just take longer as more data gets uploaded, so multithreading both of these will be key to getting performance up to a reasonable level.

[zettawatt]

Not sure it was clear above, if you want to search what I’ve added so far, you can reference the ‘genesis pod’ that I created with the addresses I was able to find. Do the install, run colony-daemon and run these commands to add your first pod and add the genesis pod address. Enjoy!

colony-cli add pod "main"
colony-cli add ref "main" 860b439c8b8cd01a7ee7f3b8791793963dfedac3bf0d5b998e57160256ef83a93ec010f8fa6d120ef3c5cbef9b59c43a
colony-cli refresh --depth 0

[Southside]

needs a chmod +x colony-*

[zettawatt]

Thanks, missed that one. I edited it above.

[Southside]

Reference added successfully!
Starting cache refresh…
⠄ Cache refresh with depth 0 - Starting refresh ref with depth 0 ⠠ Cache refresh with depth 0 - Starting refresh ref with depth 0 Cache refresh with depth 0 completed
Result:
{
“message”: “All pods refreshed successfully”,
“status”: “success”,
“timestamp”: “2025-06-23T01:10:47.798857646+00:00”
}
Searching text: beg blag and steal (limit: 50)
Text search completed
Search Results:
Name Description Address
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
BegBlag.mp3 Beg Blag and Steal

looks like ItJustWorks

noted that when asked to enter a passord, an “e” character was shown. I chickened out and simply entered a trivial password wit e as first char, Anybody else seeing this?

[zettawatt]

Thanks for trying it out! I’m not sure why that e character is there when entering a password, I see it on my side too. You can enter any password and it works fine. I’ll put that on my list of things to fix.

[Southside]

Just FYI – here’s another shortcut that doesn’t work — Booo!!

Enter KeyStore password: [hidden]
Generate a new BIP39 mnemonic? (No to enter existing) yes
Generated BIP39 12-word mnemonic: connect veteran effort next giant swift local couch horn rapid cactus media
Enter your Ethereum wallet private key: $SECRET_KEY
Error: "Failed to set wallet key: Odd number of digits"

[zettawatt]

Good point. I’ll put that enhancement on the list as well. I’ll make $SECRET_KEY the default so if set, you don’t have to put anything in there. Entering a key here would override the default.

[happybeing]

If you steal some ant-cli code you can have it pick up any cli wallet first, then look for SECRET_KEY if no local wallet.

[zettawatt]

That sounds like the right approach. I’ll do that.

[Southside]

@zettawatt

"@context": {"schema": "http://schema.org/"},\
  "@type": "schema:

Obviously some more reading req’d

willie@gagarin:~/projects/maidsafe/JAMS$ colony-cli put "The PoD pod" af0a27cc84d2979918ba197e3696fb4e0d0834227fd33a156e55fc5f3852099392515fa306e34d7181c81b320dcd1443 \
> "@context": {"schema": "http://schema.org/"},\
  "@type": "schema:Text",\
> "@id": "ant://af0a27cc84d2979918ba197e3696fb4e0d0834227fd33a156e55fc5f3852099392515fa306e34d7181c81b320dcd1443",\
> "schema:name": "The Phil o'Donnel PoD"}'
> '
error: unexpected argument '{schema:' found

Usage: colony-cli put [OPTIONS] <POD> <SUBJECT> <DATA>

For more information, try '--help'.

I promised no more Celtic references but when you say “pod”, I think “PoD” Phil O’Donnel who played for us and won some medals late 90s, went south and returned to Motherwell (his original club) Tragically died on the pitch about 20 years ago
Anyway I’m trying to make a pod for PoD that will have a link to his history , stats and some pics. When does a pod stop being a pod and becomes a mini website?

must RTFM: Person - Schema.org Type

[zettawatt]

I made an example document showing all the examples I used to make up the collateral I’d uploaded so far here: colonylib/metadata_examples at main · zettawatt/colonylib · GitHub. Check out the genesis.md file in that directory.

The syntax error looks like you’re missing a { after you put in the address.

Honestly, the best way I found to generate these was to throw this into your choice of LLM and tell it to generate a JSON-LD description using schema.org vocabulary for the following description object using that genesis.md file link as a list of examples.

@Southside

When does a pod stop being a pod and becomes a mini website?

Ah, you’re going down the RDF rabbit hole here . It kind of is. So you’re making a set of RDF statements about “Phil O’Donnel”. The person “Phil O’Donnel” isn’t an object on the network, but the other information may be, or it may be references to other things like the team he played on, which themselves have references to his team mates or match scores, etc. My main focus to start was to describe objects on the network so we could search for them, the second layer to this is to add contextual information about these disparate pieces of information, just like you describe, so we can understand them.

[Southside]

What is obvious is that I chose the wrong subject for my first pod.
OTOH if I work at this a bit I will learn a lot, so I might just continue in a naive trivial way.

Off to look at the example doc some more…

[zettawatt]

Thank you again for digging in and trying all this stuff out. I wish more people here would actually try out what we’re building vs complaining about “when native token? work fast! make rich quick!”

[Southside]

Aye - maybe we should ask the next “Wen Native” chancer just exactly what he has contributed so far …

We can’t all code - anything past basic scripting and trivial Python usually beats me - but we can all try what others have put on a plate for us.

Ok sometimes we need to go and cast new cutlery with which to eat it… but YKWIM

[Southside]

Slowly getting there - I may nearly have the beginnings of an outline for a template for footballers. Maybe

willie@gagarin:~$ colony-cli search subject af0a27cc84d2979918ba197e3696fb4e0d0834227fd33a156e55fc5f3852099392515fa306e34d7181c81b320dcd1443
🔍 Searching by subject: af0a27cc84d2979918ba197e3696fb4e0d0834227fd33a156e55fc5f3852099392515fa306e34d7181c81b320dcd1443
  ✅ Subject search completed                                                                                                                                                
📋 Search Results:
📦 Found in pods:
  916feb55209f84749c75d164262ad389e1049ea2747f0e71ad1cf813f887f969069b2c363529e54e07e90325e00424c9
  90848acbdb1d184551420f43f074703b98a42369c9ee6f17d7d437580e98172624f28156bf0cd8af6458a322740ecf76
  af0a27cc84d2979918ba197e3696fb4e0d0834227fd33a156e55fc5f3852099392515fa306e34d7181c81b320dcd1443

📋 Properties:
Predicate                                          Object
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
http://schema.org/givenName                        Phil
http://schema.org/memberOf                         d2713530c690c66d14d645ec4544351a
http://schema.org/memberOf                         f7784c8bd28f5ed4bce15762ca2a6ff2
http://schema.org/memberOf                         faad275c43b95c51115da95464b53977
http://schema.org/additionalName                   Pod
http://schema.org/deathPlace                       cb7de47448ddad427dd39f9c2b0bdf18
modified                                           2025-06-23T12:42:16.119963050+00:00
creation                                           2025-06-23T12:42:16.119963050+00:00
http://schema.org/careerEnd                        2007
http://schema.org/description                      A footballer from Motherwell, Scotland, who played for Motherwell, Celtic, and Sheffield Wednesday in the 1990s and
                                                   2000s.
http://www.w3.org/1999/02/22-rdf-syntax-ns#type    http://schema.org/Person
addr_type                                          ant://colonylib/vocabulary/0.1/object#pod
http://schema.org/careerStart                      1990
http://schema.org/birthPlace                       a6141aec65a665ca4379a748dc253eec
http://schema.org/affiliation                      cababf82afa1de0f76a18f3b569d153c
http://schema.org/birthDate                        1972-03-25
http://schema.org/familyName                       O'Donnell
http://schema.org/hasOccupation                    a3d7dcfad6e5e891cbad9e4ead4f14d9
http://schema.org/deathDate                        2007-12-29
http://schema.org/award                            b09bc9ff7abf806dd63919e9734dc34
http://schema.org/award                            b33f834bc86d0309d50bc8287590d9ed
http://schema.org/award                            b3a0a264de1add4c2951eb582c913a00
name                                               The PoD pod
http://schema.org/height                           f5ecc133dafb0f8a23d43f8bb40b0555
depth                                              0

[zettawatt]

Interesting. So the address of your created pod is what you’re using as the address for “Phil O’Donnell”? That’s a really clever way to go about it, I hadn’t thought about using the pod address for anything other than sharing it with others. This allows you to describe a ‘thing’ as an object on the network without there being an actual file, the pod is the file.

A user could make a pod their identity just like you did above and share that with people, include links to their stuff, etc. A one stop ‘this is the version of me I want to share with you’ address. And the beauty of it is that only you can change stuff about yourself. Other people can reference your ID pod and add attributes about you, but since it is in its own graph, you can prove that an attribute came from you and no one else.

@riddim, could you use this approach as a kind of portable ID for Friends? @Josh maybe this could also be used as an identity structure for Atlas? That would be so cool to have an ID that you could seemlessly port between applications and have your data follow you.

[Southside]

Now you have me excited, forget all about poor Phil, I’m doing me next

[zettawatt]

I love this idea so much. I wish I would have thought about using it this way! This is exactly what the SOMA guys are wanting to do with SOMA ID. They’re using colonylib on the backend, but I wasn’t sure how to give them an ID feature, THIS is the way.

I’m going to make an ID pod for myself as well.

[ambled]

Please do this the other way around, don’t grab the cli wallet unless SECRET_KEY is not defined.