That does sound like a bug in sync for sure. cc @joshuef @qi_ma
When I did get it to compile from current source, updated rust, it runs gets peers, has PUTs happening and records stored.
But after a few minutes it panics with some error along the lines of unwrap and crashes. Reliably does this.
I am guessing it is some mismatch in libp2p and the current code. or another 3rd party package.
But no time at the moment to investigate further.
Maybe its a env var that needs setting when compiling.
The only other environment variable we set during our release process is the secret genesis key, but as far as Iām aware, that is only used in the faucet.
The only other thing you would need to be careful about is building with the network-contacts feature.
Otherwise I think you would need to post that error so we can take a look.
Thereās already an issue open but Iām just uploading a awe v0.5.0 to https://downloads.happybeing.com which helps show this. I donāt have a build for MacOS but there are instructions for that on the forum here.
The following command just syncs to get the register and then prints some info. You can list the entries too (use --help for options).
awe inspect-register --register-summary d6108b6090abc54159fad5e034473ec0a6264746001543a3b2e475474cc35da3a56cf326d1e1abbed387f8fd53e3fe6dfb3431cfb3f1c8c1bbd044cbddd1c08a2e4bf25521f909d9c18d0bd4a825a274
This gets the register and prints a summary for it such as:
Autonomi client initialising...
Connecting to the network using 50 peers
register: d6108b6090abc54159fad5e034473ec0a6264746001543a3b2e475474cc35da3a56cf326d1e1abbed387f8fd53e3fe6dfb3431cfb3f1c8c1bbd044cbddd1c08a2e4bf25521f909d9c18d0bd4a825a274
type: 1fd6ee..
size: 12
Most of the time however it reports size 2. Sometimes 12 (correct) and occasionally something else.
@qi_ma is on the trail of this one Mark. He is a debug guru.
replied in the raised issue bug: registers report incorrect numbers of entries Ā· Issue #2030 Ā· maidsafe/safe_network Ā· GitHub
and will keep it updated for any further progress.
More updates from @qi_ma for you @happybeing It seems replication flow overwrites values which of course for CRDT data (and transactions) is a disaster. So we will get that fixed.
This sounds like an issue I came across in range based get implementation.
Essentially that for registers, we had aimed to rely on user-space updates to divergent registers, and NOT sync at the node side.
This was then coupled with register verification only requiring ONE node to return the valid state. Which in turn led to potentially both states being seen as valid, never retried (both states exist), and the nodes never converging (they did not yet actively merge divergent data).
This had the unintended consequence that you could push updates, see both states as valid, depending on your validation level.
Another issue would be that on replication for the same key, we were NOT merging register states, and seemingly overwriting that there. (A bug)
Thereās changes and fixes in the pipeline for this.
(Iām just back off a couple weeks holiday; so catching up on the state of that now; iāll update if thereās more relevant details!).
Hope you had a fab break @joshuef, Iām sure you will have needed it! 
Can you elaborate in detail what behavior you are aiming for for Registers. Not having convergence on the network seems very hard for a client to use.
I was literally a single device, writing a single value with merge online one after another and in that situation I would expect always to get the most recent value back. And in this situation, always the same history.
Without that I think Registers would be very difficult to make useful in an app.
FYI Qi believes heās fixed that behaviour and has a PR waiting to be merged but it isnāt going to get into the latest update, so I wonāt be able to test this for weeks.
I donāt envisage many updates can usefully happen in the nine weeks allocated until launch, so this look quite risky and unsatisfactory, still to be finding bugs like this (and pretty much only me looking for them!)
The network cannot merge entries for you. That is all app dependent. Some apps may wish to use last edit as tip of tree or merge 2 tips or have some admin decide what entry to use (i.e. if they are changes to a doc, which change to accept and so on)
Hereās an example of a REAMD.md for registers to help.
sn_registers
Provides utilities for working with Conflict-free Replicated Data Type (CRDT) capable registers on the Safe Network.
Overview
sn_registers is a crate that implements a CRDT for registers in the Safe Network. It uses a Merkle tree-based Directed Acyclic Graph (DAG) internally to manage concurrent updates, allow for conflicts, and ensure eventual consistency across distributed nodes.
Key Features
- CRDT-based register allowing concurrent updates
- Signed entries for data integrity and ownership verification
- Conflict resolution through a tree-like structure (DAG)
- Flexible API for creating, updating, and merging registers
- Total size limit of 1MB per register
- Individual entry size limit of 100 bytes (plus signature)
- Support for storing pointers or links to data
Usage Guidelines
Due to the strict size limitations, it is strongly recommended to use registers primarily for storing pointers or links to data, rather than storing the data directly:
- Total Register Size: 1MB maximum
- Individual Entry Size: 100 bytes maximum (plus signature)
This approach allows for efficient use of the register while still providing access to larger datasets.
Best Practices
- Store pointers: Instead of storing full data, store pointers or links to where the data is actually stored on the network.
- Use content addressing: Store content-addressed links (e.g., hashes) that point to the actual data stored elsewhere on the network.
- Metadata storage: Use registers to store metadata about larger datasets, including version information, timestamps, or access control lists.
- Monitor register size: Keep track of the total register size to ensure it doesnāt exceed the 1MB limit.
Example of Storing a Pointer
let data_hash = "Qm..."; // Hash of the data stored elsewhere on the network
let entry = format!("data:{}", data_hash);
register.write(entry.as_bytes())?;
API
The crate provides a high-level API for working with registers:
Creating a Register
let register = Register::new(owner, name, tag, permissions);
Writing to a Register
register.write(pointer.as_bytes(), &secret_key)?;
Reading from a Register
let entries = register.read();
Syncing with the Network
register.sync(&mut wallet_client, verify_store, None).await?;
Size Management
Itās crucial to manage the size of your register to prevent exceeding the 1MB limit:
-
Regularly check the total size of your register:
let total_size = register.total_size(); if total_size > 1_000_000 { println!("Warning: Register size ({} bytes) exceeds 1MB limit", total_size); } -
When adding new entries, ensure they donāt push the total size over the limit:
let new_entry = "new_pointer:Qm..."; if register.total_size() + new_entry.len() <= 1_000_000 { register.write(new_entry.as_bytes(), &secret_key)?; } else { println!("Error: Adding this entry would exceed the 1MB limit"); } -
Consider implementing a cleanup strategy to remove old or less important entries when approaching the size limit.
Advanced Features
- Conflict Resolution: The Merkle tree DAG structure allows for automatic merging of concurrent updates.
- Offline Operations: Registers support offline writes with later synchronization.
- Cryptographic Security: Entries are signed for integrity and ownership verification.
For more detailed information and advanced usage, please refer to the API documentation.
License
This Safe Network Software is licensed under The General Public License (GPL), version 3 (LICENSE http://www.gnu.org/licenses/gpl-3.0.en.html).
I thought registers are append only? Oo
If itās possible to change a first version of a key and basically rewrite history for the register that would make the name resolution scheme we were discussing @happybeing not really feasibleā¦
Ok, I think it might be possible in principle but accepting that it doesnāt, we need detailed guidance on how torr use the API calls, what they do and donāt do.
For example, what about the āwrite merging onlineā function. Iām using that to write each value and from what I can see it should work but for the bug I found and helped Qi track down.
We canāt use or test this API yet because it isnāt clear what it does or how to use it. Other devs have mostly looked and given up, and then Iām the the one trying to explain it!
Iām fairly sure what Iām doing should work (and it did on local networks), but hearing what is going on on the network in discussions about the bug with Qi makes me unsure what to expect and what the app needs to do to thandle it. Thatās really what Iām asking for here rather than those guidelines.
They are. When talking about clean up it means merge branches really. So if you imagine lots of concurrent updates to a value gives lots of branches. The clean up means merge those to a new tip of tree.
I need to look at that badly named function. Itās a terrible API name so we will see what itās trying to do, but it sounds wrong.
Any node or client can merge the CRDT data types (although here we have only 1). The idea of some online merge is wrong. Any node can merge and write back to all others.
Thanks guys for sticking with this, we will get the API solid and more importantly documented. Itās about 12 months later than it should be to have that in place. It will get done.
The Register API sorely needs someone who understand both purpose, qualities and implementation to work through it.
I have looked in depth at the code and modified the API itself in order to get at the history, but I donāt claim to understand it well. It could do with two or three worked use cases that show how different structures of histories are created and can be accessed.
I chose the simplest possible, one client/device always writing a single new entry to the Register on top of the last. So in theory no branches, but because the network serves up the Register at different points in the overall history, that caused branches to exist, which obviously was confusing and prevented me accessing the whole history even in this simplest case.
Assuming thatās sorted with Qiās PR, thatās good but Iām still unsure how to do something more complex than this simple case. I came up with several ideas (now lost on the defunct dev forum along with related discussions) but am not confident in my understanding, and the API as it was doesnāt provide a way to access histories for set use cases. So I had it modified to expose the MerkeReg which I can now inspect directly. Thatās probably not ideal. 
But it has allowed me to access the history I needed for awe.
PR incoming with updated README.MD It will be followed with a load of tests and examples to help.
Itās been a tough ride for you here Mark and itās on us. So now letās focus on getting these questions answered finally in the readme and codebase examples.
If you can stick with this a few days we will get it slick and easy to read.
Critique welcomed and sought after here
I saw that just now and donāt think it is what we need - see above. That is apparently an AI generated intro which Qi has corrected and maybe added to. Iām not sure thatās overall an improvement.
I totally get that there are not enough hours in the day, so Iām not critical of the devs here.
My issue for a long time has been asking them to deliver something that has looked unrealistic in terms of time-scale and has become increasingly fraught and at odds with the roadmap. Iām very worried about this, and the lack of clarity over what is going to be launched on top of that, or how we go from there onwards. The network is going to launch unfinished as far as I can see, key areas unfinished and untested, serious bugs still being found (by one person doing something serious with the APIs) with only three or four network updates to go.
Hi, @happybeing , have to clarify that the branch of update history and branch of root content are two different stuff regarding the Register to be exposed.
branch of update history is mean to happen due to the nature of the crdt and p2pnetwork. and normally shall only be used for dubugging purpose, and not to be used for normal usage.
meanwhile, branch of root content, i.e. what register.read() so far returns, shall only happens when ther is un-resolvoable merge conflicts, which shall be rare.
Iām not sure, but think you are saying the the accessible history will always be a single sequence of values, each the result of a merge of one or more values written at the head until merged.
If thatās so, Registers in my mind just became much easier to understand (no really folks ). 
Although perhaps a little more limited in what they can represent than I thought. 
This goes back to ideas I present on the now disappeared dev forum for different ways to use a Register.
yeah, I feel that could be.
The Register is designed that a user only need to care about the head content.
With all merge/update history stuff all supposed to be hidden from the final user, (which to be undertaken by crdt data).
Itās unfortunately that due to the current network status and the un-purposed used kad::get_record, that let you experienced with cannāt fetch correct register, and have to access the update history to help us pin the issue. (I just realized that during the investigation )
In a word, for normal usage case, Register have new/write/read/merge shall be enough, with other facility functions such as sign/permissions/etc.
The merkle_reg shall only be used for rare situations like debugging, or really received multiple head contents (which means crdt cannāt resolve conflicts during merge) and would like to resolve conflict manually using some special rules .
I think that would be a shame and not what we have understood about this (and itās predecessors) for literally years.
Creating a versioned web with access to the history is in theory straightforward. Thatās the main point of awe really, to demonstrate that one quality and how it can be applied to the web on Autonomi.
If Registers are just a way to arrive at a single view, with no access to history weāve lost something really important. That would be a big deal which hasnāt been explained. cc @dirvine