:IF: Colony - Search Autonomi Simply

[zettawatt]

I wanted to share my project here officially, the ORIGINAL place for all things Safe Network, err, Autonomi. Colony is a cross platform (Linux, Windows, and Mac) desktop application written completely in Rust, using the Slint UI GUI framework. The main goal of this project is to make Autonomi successful by delivering to this network what made the WWW successful: a search bar. An interface so intuitive and easy to use, anyone can use it. The difference here is that search is done locally on the client so no middleman looking over your shoulder and no corporations or governments manipulating the underlying data.

For users only downloading files, you don’t need to have ANT/ETH tokens. All you need to do is download the app on any desktop computer and you’re up and running. Simple.

Want to run Colony on a different computer? Or maybe you lost access to the machine you originally installed Colony on? No problem! Input your 12 word seed phrase to restore all of your data directly from Autonomi.

More than an app, Colony is working with other IF projects like Stashbag and SOMA to leverage the same metadata and search framework. The end goal is to create a simple standard to enable seamless interoperability of your data between Autonomi applications. No lock-in, no problems.

I’ll be giving a short pitch on the discord stage tomorrow, but the longer form video where I dig into all the details can be found here: Introducing Colony

As for status, the app skeleton is in place. Honestly its about a third of the way done at this point, with the installation, configuration, key generation, and key management logic in place. Here’s the obligatory screenshot:

Looking forward to hearing everyone’s thoughts and seeing all the other projects on IF. After all these years we’ve finally got the network, let’s go use it!

[Nigel]

Love that you are collaborating with some other projects to create open standards. More of this!!! No lock-in is chef’s kiss. Best of luck!

[zettawatt]

Thanks! Good luck to you as well! I agree, community standards are going to be a major component to making Autonomi successful. After Colony, my goal is to leverage the learnings from the project to propose a first pass draft of a community metadata standard for Autonomi. Basically this whole exercise, on top of making a useful app, is to build a framework, or even a library, that any other project can use on Autonomi to automatically plug in to all the existing data. In other words, interoperability won’t be forced, it will happen naturally because it is the path of least resistance.

[Nigel]

I think you just spoke to everyone here. This is the way. You have my support!

[zettawatt]

Posting today’s update in case it doesn’t show up in the IF page:

Status

• Ending the mad dash marketing push week and starting to dig back into the code
• Performing some analysis of the demo app and determining exactly what needs to be completed for MVP.
• Created a a first pass functional requrements document and posted on github here: Requrements.md. These are mostly thoughts for myself, so may not make total sense, but wanted to share it anyway!

Shoutout to the community

If anyone has other critical features they think I should add to MVP, post a comment to the Colony topic on the Autonomi Forum or reach out to me on Discord in the general chat.

Thank you all who have voted for Colony so far! Let’s make Autonomi great!!

[zettawatt]

Today’s update:

Status

Its been a busy week, but I have some good news to share! Lets dig in:

New developer onboard!

Joining the Colony team is Maxx, a 10 year software engineer with an extensive JavaScript background. While my software development skills are akin to a toddler making macaroni art, Maxx is over here painting the Mona Lisa. His skills will be invaluable to making Colony not only functional, but a polished well tuned application

New development stack!

With 2 of us doing the work, we can divide and conquer. I spent much of the last month fighting Slint UI, drudging through type conversion hell. While it is OK, the JavaScript GUI world is leaps and bounds ahead. Based on a forum suggestion from loziniak, we dug into Tauri 2 which allows Maxx to focus on the frontend (which will be written using Svelte) and me to work solely on the Autonomi and crypto interactions in rust. Tauri 2 is fully cross platform, including for mobile, so one stack to rule them all. This is definitely the way to go.

But wait, isn’t this change going to slow things down? That’s what we’ve been working on this week!

Frontend Progress

Maxx basically rewrote the entire GUI demo that I had built in an afternoon and connected a couple buttons to some of my existing rust functions. So we’re up and running! No check-ins yet (still very messy), but we’re getting there.

Backend Progress

As for me, I’ve split out the core rust functions that I built in my demo, cleaned them up, and created a new repo on github called ColonyLib. As I’ve said in my pitches, I want to build up an Autonomi metadata standard. Colony is really just the vehicle to get us there. Colony will leverage ColonyLib to handle the master key creation, key derivation, pod creation, updates, and potentially RDF queries. This means any other Autonomi application can grab ColonyLib and know that whatever they write using their frontend application with be directly interoperable with Colony and its content discovery system.

Pods are being created!

Just this evening, I have successfully implemented a pointer/scratchpad combination in ColonyLib that I am calling a pod. The scratchpad is publicly readable (which is working!). The pointer points to the scratchpad and the counter field is updated for each update. The idea here is that when a user wants to refresh their pod cache (i.e. download all updates to pods) they just need to download the pointer, check if the count value is higher than their local cached copy, and then download the scratchpad vs downloading the scratchpad each time when often times, many of these will not change. The pointer can also be used to ‘hide’ data or move the pointer to something else. The concept here was for SOMA so that they could provide health information and when a user was done sharing that data with the provider, they could either point the pointer to itself or point it to some other data. Obviously, the provider could be caching that information, but I don’t think we can solve that in an ‘everything done client side’ world. Another problem for another day!

Thank you for your support!

Things are moving along nicely and we’re getting ready for the next phase. But we can’t do it without all of you! Thank you for all your support keeping Colony in the running. I am deeply honored. It makes the late nights and little sleep all worth it.

With your help, we can get this project finished and enable the world to Search Autonomi Simply!

[zettawatt]

Getting the band back together:

Status

It’s been a chaotic week here at the Colony headquarters (i.e. my computer desk). I needed to get a lot of details sorted out before I could give an update, so let’s get to it:

The contingency plan

After Colony dropped to 13 in the voting round, I accepted the loss and consulted with my frontend developer, Maxx. At that point, Colony became a solo project and without any deadlines, I figured I’d slowly work on it and get it out there eventually. So I started working on my next project for Autonomi: community standards! Like I mentioned in my pitch and video, the primary purpose of Colony is to make Autonomi successful, and if I wasn’t going to do an app, the least I could do was help coordinate and kick off some kind of community standards organization. So I dug into several different processes and finally settled on the BIP (Bitcoin Improvement Proposal) framework. I wrote an Autonomi version of this called AIP (Autonomi Improvement Proposal), which I’m pronoucing as ‘Ape’, because imagine the fun you could have here, anyway…

I published AIP-1 to an aips repo on the safenetforum-community github page that @Southside provided me access to (Thank you!). For any process nerds out there, the full AIP specification can be found here: AIP-1. I also setup a weekly Autonomi Developer Council and had a great conversation with @bytes. Great guy! I hadn’t interacted with him before on the forum, but he is really sharp! Thanks again for calling in! I hope to get some more attendees next week on Jitsi meet at 21:00 UTC Tuesday: Link If you have some time, please join! Its just a live call where we can talk shop about all things Autonomi, development, or anything at all! Just another way to build up the community. If the time doesn’t work, we can change it, its hard to meet everyone’s schedule when we’re scattered everywhere.

Wait, what? Colony is back on?

So anyway, by Monday I had already shifted gears. It was quite a shock when I checked Monday afternoon that Colony apparently made it to the top 12!? No messages, no warning, no information at all, just thrown back into the pool. Well, OK then, I guess I need to get back into this thing! I called Maxx and explained the whole thing. He was a good sport about it and we had a good laugh. He is now back onboard!

Frontend status

I had a 2 hour call with Maxx Tuesday night and we did a deep dive into what I envision for the frontend. His plan is to work on a skeleton over the upcoming 3 day weekend (Memorial Day here in the states). We’re still going to use Tauri 2 with Svelte as the JS framework. I gave him access to the Colony github so as soon as something is up, you’ll all see it.

Backend status

I’m back digging into the rust code from the Colony demo app. As I previously mentioned, I have created a secondary repo that will be used as a rust library for Colony or any other application that wants to leverage the same authentication and metadata storage mechanism. That repo lives here: colonylib. There is a lot of churn here as I hash out how I want the API to look. It will settle out in the next couple weeks as I get all the pieces roughed in. The reason for splitting this out was to enable the good folks at SOMA to be able to use this backend logic for their app and enable both apps to be interoperable.

I’ve been cleaning up the very ugly test code for pod creation and updates. The library now properly passes errors on to the end application and I implemented the tracing crate everywhere to enable easier debug. Its starting to look a lot more like a library!

I plan on taking a week vacation from my day job in the next 2 to 3 weeks to focus 100% on Colony and get the rust side up and running. I can take up to 2 weeks if necessary, but I think 1 60 hour slog should be able to really tidy this thing up.

Thank you!

Thank you again for everyone who voted for Colony in the first round and is backing Colony now! I had no idea I would get this much support! It is an amazing thing to see from where I’m sitting. Just a few more weeks until we can all search Autonomi simply!

[Traktion]

Sounds amazing - keep up the great work!

Top commitment to give up vacation time to work on this too!

[zettawatt]

Status

Lots of progress on the backend side this week. Starting to see results! Let’s get to it

Colonylib 0.1.0 crate released!

The colonylib crate version 0.1.0 has been uploaded to crates.io! Anyone can add this to their project by calling the standard cargo add colonylib. All warnings have been cleared. All errors are either handled internally or passed up to the end application, being wrapped into a custom Error type, in other words, no unwraps in the library!

No official docs yet and my testbench I haven’t checked in, that will be coming in the future.
That said, if anyone wants more info on how to leverage this library, let me know!

Running on the Alpha net!

That’s great, but what can this library do? There are lots of moving parts under the hood, but put together it can do this:

• Use a 12 word seed phrase or master key to generate a password protected key store to derive addresses for pods (composed of pointers and scratchpads)
• Create and modify pods as files in an on disk cache (automatically detects the OS to put them in the right place)
• When the user is happy with the local state of pods, call the upload_all() method to upload all the pods to Autonomi. There is logic in place to detect when a pod has been updated
  in the cache, so only new or modified pods are uploaded, no need to keep track of this yourself!

Then the magic:

• Delete the local cache or go to another computer
• Initialize a new key store with the same 12 word seed phrase or master key
• Call the refresh_cache() method, which will automatically walk through the list of
  derived keys, check if those addresses contain valid data on Autonomi,
  and populate the local cache of pod files with whatever it finds.

This is the worst case scenario obviously, but under normal operation, the refresh_cache()
method will walk through the pointers in the pods and compare their version vs the local
cache. The scratchpad data is only downloaded if there is an update. If there are missing
pods vs the local cache, they will automatically populate. Or if the key store was deleted, it will be repopulated as well. Even if an address gets populated with a broken pointer or scratchpad on Autonomi, the library will mark that address as bad in the key store so it knows not to use it again. In other words, lots and lots of exceptions handled here so that the user application doesn’t have to deal with this.

And best of all, this has been tested and all runs on the Alpha net!

A very Colony vacation

I officially put in my vacation request for next week to work full time on Colony.
Up to this point I’ve worked on this project solely in the evenings and a few multi hour
stints on weekends. With some serious time to devote to the project, I am expecting serious
results.

The main goal next week is to implement the RDF side of colonylib. At the end I’ll have
methods implemented like add_file_to_pod(), remove_file_from_pod(), and
get_info_from_pod(). By passing a pod address, an address to a file on Autonomi(or other object),
and a JSON encoded string, the frontend application will be able to read, write, and modify
pods from a per file entry perspective, while the library will handle all of the RDF
formating and scratchpad splitting (when data is larger than 4MB). In addition, there will be methods to add_pod_ref() and remove_pod_ref() to add references to other users’ pods, which is the key to this whole scheme.

Next will be to write a recursive function to walk through the external pod references and
download these into the local cache from Autonomi. With all the external pods downloaded,
we’re ready to implement search!

Thank you backers!

Thank you again to the backers keeping Colony at the top of the list! Honestly, seeing
this project up towards the top of the list is a major driver in keeping me motivated. I
won’t let you guys down! I hope you’re all as ready as I am to search Autonomi simply!

[zettawatt]

Status

We’ve got indexing! We’ve got search! Backend is MVP! Not only that, but colonylib is now being used to power content discovery in Mutant! Here are the details…

colonylib 0.2.0 crate released!

The colonylib crate version 0.2.0 has been uploaded to crates.io! I’ve added a ton of documentation for all of the public facing API’s as well as examples that you can run yourself with the standard cargo run --example framework. There is a full test suite for the internal functions. All tests pass. No errors and no warnings in the library. Docs for the public API can be found on docs.rs. The public facing API includes example code to get you started.

Want to get the full details? Check out the README and try running the examples yourself!

Shoutout to @Champii and Mutant

Before we dig into Colony, I want to recognize @Champii for his contributions to colonylib. We had a nice chat the other day and he introduced me to Augmentcode. He used this tool to build the rough cut for the following features. Instead of me hammering away at this by hand for a week or 2, I just needed to spend a day fixing what the AI agent didn’t understand or broke. @Champii literally told me this tool can take 3 weeks of work and compress it down to one day, he wasn’t kidding! He then plugged colonylib into Mutant to power its content discovery and file sharing mechanism. He built a frontend for colony before I had a chance! And he did all this in just a couple days! Truely amazing work. If you haven’t checked out Mutant, you should!

The best part is that content created on Mutant will be directly portable to Colony or any other frontend that leverages the colonylib library. This is what the semantic web is all about!

SPARQLy graphs!

So what does this new release do? Glad you asked.

Distributed Indexing

Colony at its core leverages a pod which stores metadata about objects stored on the network. Pods can reference each other. So the user, by referencing other users’ pods, can build a graph of all information on the network by recursively walking through all pods they are aware of and storing this index locally on their machine. This operation is done in colonylib by simply calling the refresh_ref() function. Go get a cup of coffee, and watch your index populate with everything it can find.

Fun with RDF

The colonylib library leverages RDF and the oxigraph rust library to build a local disk database. Originally, I thought this would only be needed for search, but what I found is that RDF is a super powerful concept that I leveraged for all sorts of things and coule be used for future applications:

• I built a custom vocabulary for colonylib itself to manage pods, pod references, and pod scratchpads. No extra code required, it just works.
• I split each pod, local and referenced, into their own named graphs with a depth attribute to build in the concept of a ‘web of trust’. The goal here being that nodes with less depth (i.e. less recursion required to reach) will have a higher relevance in search operations. No algorithm required here. Your search results are custom for you based on how close the nodes are to you. Pretty cool right! And this is all just baked into RDF graphs themselves.
• SPARQL queries could be used for graph metrics and visualizations by post processing the output.
• SPARQL queries allow you to INSERT and DELETE graphs and graph entries. This enables pruning the graph locally. Said another way content moderation. I envision groups or users making curated pods with content or even removal of content or pods. The difference here is it is the user’s choice to read in a pruning type pod. So if you want an internet with fluffy kitties and unicorns only, you can do that, or if you want to drink from the raw unfiltered feed of humanity’s online consciousness, you can do that too. But its your choice, no one elses.

The library itself doesn’t care about what RDF schema you use, it is completely agnostic to this. The frontend application might, depending on how it is built. As a recommendation, I would say use schema.org. This is the most complete vocabulary that I know of out there and if applications all use something similar, that makes portability better.

Loading metadata into the pods and graph database uses JSON-LD formatted strings, making it easy to build up metadata information.

Search!!!

And finally we get to search. With index building and all the RDF structure sorted out, it is trivial to search the data. The search() function issues a SPARQL query for you and will search by text, data type, predicate, or even a raw SPARQL query for the adventurous. Results are displayed in standard SPARQL JSON making it easy to plug into off the shelf libraries for parsing.

Collabs update

In addition to Mutant (have you checked this out yet?), I’m also working with the good folks at the SOMA project. They are planning to use colonylib as the backend for key management, file tracking, and search. I’m working with their team of developers so they can leverage this code base as much as possible, meaning their content will be portable as well!

A very Colony vacation

And that concludes day 3 of my very Colony vacation. I’ve got a couple days left and I won’t be slowing down! Maxx has a rough first pass on the Colony GUI, but he hasn’t hooked it up to the backend. My goal this week was to get the backend in a solid place, which I did! So tomorrow I will go back to the Colony repo, merge his edits over, and get the Tauri mappings in place so he can do his JavaScript magic. I’ll need to warn him though, Mutant already has a GUI using the library, so he needs to get busy! Other than that, there are enhancements that should be made, such as dynamic pod sizing (currently each pod can only accomodate 4MB) and threading the Autonomi download operations. Not required for IF, but we’ll want those before this is something I’d be proud to say is 1.0 level quality.

Thank you again backers!

Thank you again to the backers keeping Colony up in the rankings. The hard part is done, now we just need to make it pretty! And to the whales out there with ANT to spare for backing, please sir, can I have some more? After weeks of non stop work, we’re finally able to search Autonomi simply(ish)!!

[zettawatt]

Took all day yesterday, but I was finally able to get the Tauri plumbing all worked out for the main Colony app. I can now initialize the Autonomi client, initialize the colonylib pod manager, and add a pod to the local cache all with some quickly hacked in buttons in the frontend. Just need to map the rest of the colonylib API to Tauri commands. Then it’s all GUI development from here!

Screenshot after adding pod (yes it looks terrible right now, but I wanted to show something )

[riddim]

supercool!

congratulations - can’t wait to use colony and see what it’s capable of!

especially the content discovery part will be super exciting to watch I could imagine since that’s considered a major headache atm by many I think

[Southside]

@riddim

that’s considered a major headache atm by many I think

Exactly - I was pitching Autonomi the other night to a couple of local businessmen, the conversation slowed down after I had to admit that as yet we did not have a proper DNS.

“OK, its elegant,love using otherwise unused resources, we get the backups, but how can other folks find our site, these XOR addresses wont work for anyone but you geeks…”

And I did not have a proper answer for that…

[KafkaLee]

@zettawatt

. I can now initialize the Autonomi client, initialize the colonylib pod manager, and add a pod to the local cache all with some quickly hacked in buttons in the frontend.

Beautiful!!!

[happybeing]

@zettaw att

Took all day yesterday, but I was finally able to get the Tauri plumbing all worked out for the main Colony app.

Sorry to only think of mentioning this now, in case you aren’t aware my awe Autonomi browser demo is a Tauri app, so if you hit issues around plumbing again or using the APIs etc you can always take a look (code on Codeberg).

I’ve not run it for a while but it should still work and be compatible with dweb published websites as it was the forerunner for dweb-cli and much of its code has been migrated to dweb-lib`, which awe now uses itself.

PS That GUI looks great to me. Maybe take a look at ScratchChat to see why

[zettawatt]

Thanks everyone! Getting the backend code sorted has taken several months, but looking at lines in a terminal just doesn’t give much satisfaction. Being able to click a button makes the whole thing seem real

@happybeing

Sorry to only think of mentioning this now, in case you aren’t aware my awe Autonomi browser demo is a Tauri app

Thanks! I’ll take a look. The big headache for me was figuring out this whole AppState thing and dealing with mutable references to/from it. From here it looks like the pattern is just copy paste for the rest of the Tauri commands, so hopefully no more surprises

@happybeing

PS That GUI looks great to me.

The GUI code so far was all done by my fellow Colony hacker Maxx, and its pretty good. I was mostly talking about my sad vibe coded 3 buttons that are out of place. One day I’ll pick up JavaScript, but today is not that day, too much rust to learn

[zettawatt]

Quick update. I released colonylib 0.2.2. I added 2 new features in this release:

• Added a remove_pod_ref function to compliment the existing add_pod_ref function. This is used to add references to other pods in your local pods
• Major enhancement to enable pod sizes greater than 4MB. This took some doing, but now you can add as much metadata as you want to a pod and additional scratchpads will be provisioned automatically, decoupling the physical implementation from the high level concept of a pod.
• Maxx is busy plugging in the Tauri commands into a rough first pass of the Colony GUI so people can start playing with this thing. He should have something to show in the next day or 2.

Barring any bugs that show up, colonylib is now feature complete. The next 2 enhancements I’ll be tackling are:

• Sorting search results by depth. Should just be a tweak to the SPARQL query ORDER BY argument.
• Add multithreading support to Autonomi data fetch/put operations. This affects only a handful of functions and won’t change the API, but this will take some work to sort out. That said, this will be a huge performance enhancement that I can’t ignore. Onward!

[zettawatt]

Talked to Maxx on the frontend side a little bit ago. He’s able to use the Tauri commands I created to upload files and pay with the wallet in the key store. And he’s a Mac guy as well, so all the disk access stuff is working properly on Linux and Mac! He said he’s going to take the day off of work tomorrow to get a lot of the frontend hammered out. I’m looking forward to seeing what he comes up with I’ll get some screenshots and a demo video as soon as I’m able.

So as for me, I’m in a great spot on the backend side for IF, so I’m looking at other features to add. One thing I think would be really useful would be a CLI for colonylib. I’ve got a rough idea in my head for how to make it work. I don’t think it would take that long to put together and it would make testing colonylib a whole lot easier. Not to mention, batch uploading a bunch of files would be really nice to write a script for vs clicking buttons in a GUI.

What do you guys think? Worth the effort? Or put another way, if you were a judge, is this something that would help you in your decision making process? Figured I might as well ask

[zettawatt]

Maxx has been busy plugging in the Tauri commands to the frontend. I asked him to do a few screenshots of the progress that’s been made. The buttons are working, but not all the text jives, still hashing out how best to display the information. On to some screenshots!

File downloads:

File uploads:

Pod list page:

Wallets page:

[sam1]

Man, this feels like the real deal, awesome.