Can someone explain the transformation used to go from datamap to xor addresses to fetch from network

neo · September 25, 2024, 12:36am

As per the title.

The datamap contains basic information on the file. It has a u64 file size and a table of entries where each entry contains chunk# u64, post-encryption-hash 32 bytes, pre-encryption-hash 32 bytes, and size of chunk u64

What I cannot seem to find from the code is the transformation done to go from the pre or post hash to the xor address that the chunk is stored at. The xor address of the chunk must be derived from the chunk entry in the datamap somehow and I wish to learn what it is.

Can someone help? Or do i need to tag a dev?

Thank you

dirvine · September 25, 2024, 9:16am

The pre hash is internal to the SE algo. The chunk is stored at the post hash, i.e. the hash of its contents. That and the xor address are 32byte arrays

neo · September 25, 2024, 9:49am

The post hash is the XorName and is not the xor address. The logs speak differently. The code also shows that the post hash of type XorName is not the xor address. The xor address is derived from the post hash and I’ve used that xor address to be able to get quotes from nodes around that xor address, so I have shown that the post hash is not the xor address

dirvine · September 25, 2024, 11:27am

I am not sure what you are saying here.

The post hash is the hash of the content of the chunk. That is where there chunk is stored.

Are you saying the chunks are not stored at that hash?

(BTW I have not checked the code for xorname verses post hash verses xor address, but in my mind these are all 32 byte addresses and all the same 32 byte array)

neo · September 25, 2024, 11:40am

The post hash in the datamap, and shown in the log files is not the xor address where it is stored.

The log file has both the post hash and the xor address listed on the one entry

…sn_networking] Getting the closest peers to NetworkAddress::ChunkAddress(post hash - xor address)

The quotes also show the xor address as different to the post hash. The post hash is what is stored in the datamap.

I am sure it is just a transformation.

Like you get the xor address of a peer from the peerID

echo -n “$peerID” | base58 -d | sha256sum

dirvine · September 25, 2024, 11:42am

cc @joshuef @qi_ma Can you guys chime in here. I am confused if we don’t store chunks at the hash of their contents address?

neo · September 25, 2024, 11:58am

The log file uses the one debug! with {key:?} to output the 2 values

debug!(“Getting the closest peers to {key:?}”);

This outputs the log line shown above

…sn_networking] Getting the closest peers to NetworkAddress::ChunkAddress(post hash - xor address)

where post hash is a shortened version of the XorName and full 32 bytes of the xor address. In this case the XorName is the post hash for the chunk which is stored in the datamap

qi_ma · September 25, 2024, 12:06pm

libp2p::kad using RecordKey to convert PeerId and content_key into universal XOR addresses to calculate distance.

The RecordKey is more like a hash of the input + type_bytes .

We internally wrap this into NetworkAddress, which accept all addresse types, like PeerId, XorName, etc.

neo · September 25, 2024, 12:08pm

Can you tell me please what is the transformation done to get xor address from the post encryption hash?

This has nothing to do with peerID since its only related to the chunk address XorName → xor address

I did try using sha256 hash but that doesn’t work

qi_ma · September 25, 2024, 12:13pm

github.com

maidsafe/safe_network/blob/main/sn_protocol/src/lib.rs#L169


      
              match self {
                  NetworkAddress::RecordKey(bytes) => Some(RecordKey::new(bytes)),
                  _ => None,
              }
          }
          
          /// Return the convertable `RecordKey`.
          pub fn to_record_key(&self) -> RecordKey {
              match self {
                  NetworkAddress::RecordKey(bytes) => RecordKey::new(bytes),
                  NetworkAddress::ChunkAddress(chunk_address) => RecordKey::new(chunk_address.xorname()),
                  NetworkAddress::RegisterAddress(register_address) => {
                      RecordKey::new(&register_address.xorname())
                  }
                  NetworkAddress::SpendAddress(cash_note_address) => {
                      RecordKey::new(cash_note_address.xorname())
                  }
                  NetworkAddress::ScratchpadAddress(addr) => RecordKey::new(&addr.xorname()),
                  NetworkAddress::PeerId(bytes) => RecordKey::new(bytes),
              }
          }

it’s actually not us doing the transformation for chunk’s XorName address.
It’s libp2p::RecordKey doing that, as I mentioned above:

neo · September 25, 2024, 12:14pm

@qi_ma

So you don’t know the transformation? I saw that but could not trace back to where the actual work is done

Guess I will just keep digging

qi_ma · September 25, 2024, 12:58pm

Ah, it’s actually KeyBytes internally carry out transformation for distance calculation.

github.com

libp2p/rust-libp2p/blob/master/protocols/kad/src/kbucket/key.rs#L77


      
          /// Converts the key into its preimage.
          pub fn into_preimage(self) -> T {
              self.preimage
          }
          
          /// Computes the distance of the keys according to the XOR metric.
          pub fn distance<U>(&self, other: &U) -> Distance
          where
              U: AsRef<KeyBytes>,
          {
              self.bytes.distance(other)
          }
          
          /// Exposing the hashed bytes.
          pub fn hashed_bytes(&self) -> &[u8] {
              &self.bytes.0
          }
          
          /// Returns the uniquely determined key with the given distance to `self`.
          ///
          /// This implements the following equivalence:

github.com

libp2p/rust-libp2p/blob/master/protocols/kad/src/kbucket/key.rs#L158


      
          #[derive(PartialEq, Eq, Clone, Copy, Debug)]
          pub struct KeyBytes(GenericArray<u8, U32>);
          
          impl KeyBytes {
              /// Creates a new key in the DHT keyspace by running the given
              /// value through a random oracle.
              pub fn new<T>(value: T) -> Self
              where
                  T: Borrow<[u8]>,
              {
                  KeyBytes(Sha256::digest(value.borrow()))
              }
          
              /// Computes the distance of the keys according to the XOR metric.
              pub fn distance<U>(&self, other: &U) -> Distance
              where
                  U: AsRef<KeyBytes>,
              {
                  let a = U256::from(self.0.as_slice());
                  let b = U256::from(other.as_ref().0.as_slice());
                  Distance(a ^ b)

using Sha256::digest function, then trasnform to U256 for the ease of ^

neo · September 25, 2024, 10:15pm

I tried doing sha256 on the XorName, but it did not give me the xor address. BTW this isn’t to do with a distance calculation. this is from the self encryption post encryption hash to give the xor address used to store the chunk.

Does digest do something to the bytes?

EDIT: Oh wait, I was doing sha256 on the hex representation and not the binary. Let me get onto that soon^tm

neo · September 26, 2024, 1:43am

@qi_ma Thank you for that

Found my problem was a fundamental one and one I am ashamed of doing, and that is applying the sha256 on the (character) hex representation of the post hash rather than using the binary version.

loziniak · September 27, 2024, 5:43pm

What are you cooking there @neo ?

neo · September 28, 2024, 12:38am

Learning how it all connects together. I was asked to help oversee a project that is taking a lot longer to get going than even I expected. (procurement etc). The company is dragging its feet getting the stuff

So I have been spending some time learning about how the network connects together at a deeper level. And testing out things we’ve assumed would be correct. Like is the distribution of self encrypted chunks really the random distribution we expected. And as it turns out it was pretty good.

Also in the process I am getting a handle on the rust language. I wasn’t going to learn yet another language in my 7th decade of life but here I am LOL It is an interesting one. How far I’ll go in learning it I am not sure, but I guess being able to read it is the easy part.

Topic		Replies	Views
How to reference a DataMap? Development	15	1733	October 29, 2015
What the size of the chunks after the data dispersal algorithm? Development	7	1270	March 5, 2015
Just finished a small analysis of the distribution of chunk xor hash/addresses Development	7	187	September 23, 2024
Corrupted Data Map? Features	3	857	May 11, 2015
Git Design: Pull and Push Features	6	789	March 7, 2016

Can someone explain the transformation used to go from datamap to xor addresses to fetch from network

Related topics