Introducing Colony Utils!
After weeks of toiling away, I finally have an application that you all can play with. While my frontend guy is still working on the easy-to-use-for-normal-people GUI frontend for colonylib, I decided it would be good to have a CLI for us techy folks. Letβs dig in!
TL;DR, just want to try it on the main network (linux only):
wget https://github.com/zettawatt/colony-utils/releases/download/colony-daemon_v0.1.2/colony-daemon
wget https://github.com/zettawatt/colony-utils/releases/download/colony-daemon_v0.1.2/colony-cli
chmod +x colony-daemon
chmod +x colony-cli
sudo mv colony-daemon /usr/local/bin
sudo mv colony-cli /usr/local/bin
colony-daemon
# follow prompts to enter a password, generate a new seed phrase, and enter your ETH wallet private key to pay for uploads
# leave this running
In another terminal, run the following (takes ~4 minutes from a WiFi connection to populate the first time):
colony-cli add pod "main"
colony-cli add ref "main" 860b439c8b8cd01a7ee7f3b8791793963dfedac3bf0d5b998e57160256ef83a93ec010f8fa6d120ef3c5cbef9b59c43a
colony-cli refresh --depth 0
colony-cli search text "beg blag and steal"
First, GIFs!
Accessing Help
colony-cli --help
colony-cli sarch --help
colony-cli search --help
colony-cli search text --help
Search by text
colony-cli search text "beg blag"
colony-cli search text "ubuntu"
colony-cli search text "feyn"
Display info information about an address
colony-cli search subject e7bb1b87c1f0e07cdb76ba5e82a425a8da712940c2d3553aa6791494e92aa54d
colony-cli search subject c859818c623ce4fc0899c2ab43061b19caa0b0598eec35ef309dbe50c8af8d59
Add a pod and add new metadata
colony-cli search subject 142925b90fc5f6e38807713303fbdc91bfcd893a3c0f597e53764087fd70b123
colony-cli add pod "new pod"
colony-cli put "new pod" 142925b90fc5f6e38807713303fbdc91bfcd893a3c0f597e53764087fd70b123 '{\
"@context": {"schema": "http://schema.org/"},\
"@type": "schema:SoftwareApplication",\
"@id": "ant://142925b90fc5f6e38807713303fbdc91bfcd893a3c0f597e53764087fd70b123",\
"schema:name": "ubuntu-25.04-desktop-amd64.iso",\
"schema:operatingSystem": "Ubuntu 25.04",\
"schema:applicationCategory": "Operating System",\
"schema:processorRequirements": "amd64",\
"schema:contentSize": "5.7 GB"\
}'
colony-cli search subject 142925b90fc5f6e38807713303fbdc91bfcd893a3c0f597e53764087fd70b123
Modify and add metadata to an existing file in the graph
colony-cli search subject c859818c623ce4fc0899c2ab43061b19caa0b0598eec35ef309dbe50c8af8d59
colony-cli put "new pod" c859818c623ce4fc0899c2ab43061b19caa0b0598eec35ef309dbe50c8af8d59 '{\
"@context": {"schema": "http://schema.org/"},\
"@type": "schema:MediaObject",\
"@id": "ant://c859818c623ce4fc0899c2ab43061b19caa0b0598eec35ef309dbe50c8af8d59",\
"schema:description": "Beg Blag and Steal - Live",\
"schema:contentSize": "4MB"\
}'
colony-cli search subject c859818c623ce4fc0899c2ab43061b19caa0b0598eec35ef309dbe50c8af8d59
Unfiltered SPARQL JSON Results
colony-cli search subject c859818c623ce4fc0899c2ab43061b19caa0b0598eec35ef309dbe50c8af8d59 --json
What is it?
This is a CLI for colonylib: a local first, metadata sharing and semantic search library for Autonomi. What it is not is a replacement for the Autonomi CLI. Use this for handling metadata and performing searches of the network. Use the Autonomi CLI to upload/download files.
The application and all of the documentation can be found on github here: colony-utils.
Here is an architectural overview:
βββββββββββββββββββ HTTP/REST βββββββββββββββββββ
β colony-cli β βββββββββββββββΊ β colony-daemon β
β β β β
β β’ CLI Interface β β β’ REST API β
β β’ Progress Bars β β β’ JWT Auth β
β β’ Colored Outputβ β β’ Job Queue β
βββββββββββββββββββ βββββββββββββββββββ
β
βΌ
βββββββββββββββββββ
β colonylib β
β β
β β’ PodManager β
β β’ DataStore β
β β’ KeyStore β
βββββββββββββββββββ
β
βΌ
βββββββββββββββββββ
β Autonomi Networkβ
β β
β β’ Decentralized β
β β’ Immutable β
β β’ Secure β
βββββββββββββββββββ
There are 2 applications: colony-daemon and colony-cli. The colony-daemon runs in the background and serves up REST endpoints for colonylib API functions. What dweb and anttp do for Autonomi operations, colony-daemon does for colonylib. The colony-cli interacts with these endpoints and provides a CLI that you can talk to from your favorite terminal.
It is designed to be cross platform and should compile on Linux, Mac, and Windows. I provide precompiled binaries for Linux based OSes on the github page for easy install. Windows/Mac users will need to compile from source or run cargo install
to get these built (will build precompiled binaries in the future, havenβt gotten there yet).
What can it do today?
In short, everything! Upload metadata for files, share your pods with others, and perform semantic searches for everything it can find! All the things I talked about that Colony should do, it now does.
I have created initial metadata for all of the items on the Community Public File Directory post and uploaded them to a pod on the main network. After installing and running colony-daemon, you can run colony-cli commands to search for files using a fuzzy search, just like you would on Brave, DuckDuckGo, or those other search engines. It also follows the βweb of trustβ idea by sorting content on number of matches first and secondly on how βfar awayβ that information is from you. In other words, metadata from you or your direct contact is ranked higher than metadata that is 5 layers deep in the friend of a friend of a friend graph.
How to run it
The top of the post had the quick start. Tons of info is in the README on the github page. Command specific info can be found by typing your standard --help
argument. Subcommands in the CLI also have help info. I think it is pretty intuitive, hopefully that assumption stands up to public scrutiny
Why I went this route
Well for one, the fancy GUI is taking a while, and I needed a way to easily test colonylib. Writing tests for CLI is much easier to automate that button clicks. I found a TON of issues already and got those sorted. Secondly, the success of dweb and anttp showed that the community really likes the REST API concept for building applications. With colony-daemon and colony-cli, I was able to meet both goals in a relatively short amount of time. For the next 3 weeks, Iβll be testing testing testing. Any help from you guys would be greatly appreciated! And for anyone that wants to integrate the CLI or REST API into their application, let me know! Iβm happy to help where I can. There is a TON of documentation here that describe pretty much everything there is to know about how this works.
Whatβs left and known issues:
There are couple known bugs in the CLI. The pod listing format hasnβt been cleaned up yet, so it spits out the raw SPARQL results format. The search by types doesnβt work yet. The search by text is supposed to search for exact matches within ""
, but doesnβt, so there is some work to do on the fuzzy matching logic. Probably more, let me know what you find!
I left error reporting on in the logger. On main net the Autonomi client connection is very chatty. Youβll probably see a ton of errors floating by in the daemon window, but as long as the app doesnβt crash, these can be ignored.
I need a method to change your password as well as change your ETH wallet key. Right now this is possible by deleting the data directory and calling a refresh
to pull the data back down from Autonomi. It works, but it kind of clunky.
Iβll be making a video doing a walk through of how to use this and highlight some of the features, but its going to take me some time to get to that. I wanted to get this out there ASAP!
Costs and time
The application is local first, Autonomi is only used to download/upload from the network when you call the specific upoad
or refresh
commands. For downloading only, you never have to upload to Autonomi. The same is true for the Colony GUI, the idea being for many users, they just want to download stuff, and this gets them into the ecosystem without having to do all the crypto gymnastics. But if you never upload, you canβt save your preferences to the network and you canβt share your metadata with friends. Get them hooked and pull them in.
To upload the initial pods (1 configuration pod and one metadata pod) cost about 2.5 cents when I ran earlier today. Since pods are composed of scratchpads/pointers and the metadata itself is pretty small, this application will have a super low cost in the long run, nothing compared to the cost of uploading actual data!
For times, here is what I saw today on a WiFi connection at my house to the Autonomi main net:
- ~5 minutes for initial upload of 2 pods
- ~4 minutes for refresh (i.e. fetching from the network)
- ~30 seconds to upload updates to an existing pod
The vast majority of time for the initial refresh/upload is on a particular operation that I should be able to multithread. Obviously its going to just take longer as more data gets uploaded, so multithreading both of these will be key to getting performance up to a reasonable level.