Update 27 July, 2023

As you will have spotted, a new testnet is amongst us. Time to jump on board if you haven’t already! Download safe and upload some files, and see how payment for each upload is automatically subtracted from your wallet. All Monopoly money for now, of course, but it’s great to see it in action. Let us know of any errors you might see in the thread.

Payment for data is a big step forward, enabled by the fact that the network is now pretty well-behaved and stable (touch wood). Memory per node is hovering around the 50 - 60 MB range, whereas before it was in the hundreds of MB. But there’s still stuff to iron out. Thanks for the feedback so far, and keep it coming. Next stage will be to trial the basic payment flow to nodes and some sort of simple pricing mechanism.

There’s no solid NAT traversal as yet, unfortunately, but those with cloud nodes or who are happy with port forwarding can try running a node or two. Let us know how it goes.

Now that we are in a stable place, we’re also going to start picking apart the workings of the network. This week @roland is going to explain what happens during data replication.

General progress

@Chriso is working on centralised logging to make it easier to analyse errors in aggregate after testnets. Ultimately, we hope to enable community members to do this in real-time, but we’re keeping it simple for now.

Thomas Eizinger and Max Inden of the rust-libp2p team have merged a fix from @bzee, which was seeing new nodes continue to dial (message) other nodes even after they are connected to the network. @bzee is in regular contact with that team, who say that AutoNAT and hole punching should be working ‘soon’. One of the maintainers has a tentative solution that would fix the issue of port reuse - AutoNAT needs not to re-use ports on outgoing probes. Fingers crossed for some early progress on that.

@aed900 is looking at improving UX around unconnected nodes handling timeouts.

@bochaco is advancing a feature to fetch all spends for a payment proof concurrently to speed up the DBC verification process, and also an approach to make registers content-addressable.

@qi_ma has fixed a subtle bug with DBCs. When a client wants to spend a DBC to upload data, the nodes check if there is an existing spend proof at the relevant address (to stop double spend). The node holding that address will also be actively responding to GET requests, which may delay its response to the enquiring nodes, making them think there is no spent proof there. This is an extremely rare event but we’ve seen it in the wild and it would allow doublespend. We’re patching it by requiring 5 nodes to see a GET before it is acted upon.

And @joshuef is looking at dynamic payments and pricing, including the client asking nodes near the payment location what their storage cost is, and with that updating the payment for uploads. And also a basic store cost calculation based on the capacity of the record store.

Data replication

Data replication is a mechanism that allows nodes to distribute data to new peers as they join and transfer data from a “dead peer” to the other nodes. This ensures that data remains available in the network, even when nodes join or leave. However, the approach used by Libp2p relied on a node periodically flooding the network with all of its data. This came with some downsides, including heavy network traffic and high memory usage amongst nodes. Additionally, new nodes didn’t receive the data that they were responsible for, and data held by the dead nodes wasn’t replicated on time. To address these issues, we implemented a custom replication process that is triggered each time there is a change in a node’s close group.

Before delving into the replication process, let’s briefly discuss how data is stored in the network. The network operates like a Distributed Hash Table (DHT), enabling data retrieval using unique keys. When someone wants to store a file on the network, they’d first use their DBC to pay for the file and then upload the file to the network. This data, (DBC, Chunk or Register) is transformed into a series of 0s and 1s and placed inside something referred to as a Record. Although the records can be infinitely large, the nodes are restricted to handle records of size 1mb. Each Record has its own Key, which is the XorName of the data that it is storing.

To store the Record containing both the data and the Key in the network, the client calculates the Xor Distance between the Record Key and its nodes in its RT (Routing Table), sorting the peers by their closeness to the Key. Subsequently, the client asks these nodes to “calculate and share the peers they believe are closest to the Key”. The client then sorts the responses and identifies nodes that are even closer to the Key. This process continues until the eight responsible nodes for the record are found, and the record is stored with them. Libp2p abstracts this complex process away, enabling us to build our network on top of it.

Once a Record has been stored amongst the closest peers in the network, we want to maintain at least eight copies of the record, even if the nodes holding them go offline. Our current replication process works as follows: Each node keeps track of the peers close to it, i.e., its close group. Whenever a peer is added or removed from the RT, we check if this affects our close group. If there’s a change, we send the Record Keys to our close group peers who we believe should hold this record. Upon receiving the key list, if a peer does not have the corresponding record, it requests it from the node that initially sent the Key. In case it could not receive the record from the node that initially sent the key, it asks the network for the data.

Although further optimisations are possible, the current replication approach is significantly less resource-intensive than the Libp2p one and effectively prevents data loss.

Useful Links

Feel free to reply below with links to translations of this dev update and moderators will add them here:

:russia: Russian ; :germany: German ; :spain: Spanish ; :france: French; :bulgaria: Bulgarian

As an open source project, we’re always looking for feedback, comments and community contributions - so don’t be shy, join in and let’s create the Safe Network together!


First. :partying_face: :partying_face: :partying_face:

As you will have spotted, a new testnet is amongst us. Time to jump on board if you haven’t already!

Safenet usage just feels great.

Once we launch we are going to have a stampede of people running from the “dial up” world of legacy blockchain transactions and applications to a secure, fast, multipurpose, content rich new miracle crypto tech.

We’re probably going to see what they do in the business world with old systems. Bitcoin, etc… are going to be turned in to a “virtual machine” of sorts and placed on the Safenet to benefit from storing the blockchain on an actual distributed network. Why people would want to “Place a horse and cart in a Lambo” I don’t know, but they will for a while.


On the box, second! :))))


third,reading time


Honorable mention! Now to read.


How it reads! Wonderfully explained, excellent update, thank you MaidSafe and all the testers for a great job :point_left: :ok_hand: :blush:

Congratulations to the team, this is milestone information → how to effectively store the entire internet in 8 copies.

I’ve always been very curious about Safe’s data replication technology. :v: :grin:


Thx 4 the update Maidsafe devs

It was really super easy to get-faucet, upload and download a file, even send DBC with the current TestNet. :partying_face:

Thx 4 all the hard work and all those testing :clap: :clap: :clap:

Keep hacking super ants


Thanks so much to the entire Maidsafe team for all of your hard work! :horse_racing:

The testnets give us all hope! :horse_racing:


One stable test net after the other, and then another, and another, until we have forgotten that the Launch and come, and then it was gone; and now I’m using Safebook, and SafeX, and SafeSpend, and it will feel like all the years it took to get to this point was just a lot of sweat and tears and moments of joy, a long tome ago.

Splendid progress team!

I will drink a beer in your honour, cheers :beers:


Greatness is upon us :100: :pray:


Looks like it’s coming along, nice work team! Can’t wait to see this train start taking passengers :stuck_out_tongue_winking_eye::+1:


Great deep dive Roland, thank you :man_bowing:

Super work this week team … Seems like good progress is being made all around.

Won’t be long now! :tada: :partying_face:

Cheers :beers:


So ‘record’ is the new chunk?


No. Chunks still exist but are stored using records (as is everything handed over to libp2p).

I’m not sure if it’s quite correct, but maybe a record is a wrapper for whatever is being stored by libp2p.


Just to confirm what @happybeing said, when i asked it was that.

All the data types are stored in a record and for chunks the address is the same as the hash of the chunk but for other types its whatever the hash function makes of the item put in the record.


Record is a libp2p struct that the RecordStore (another libp2p concept, basically the storage API), stores :+1: . And yep, all our data is now validated and stored by the RecordStore.


Thank you for the heavy work team MaidSafe! I add the translations in the first post :dragon:

Privacy. Security. Freedom