Weather report

[chriso]

OK, so are you saying any update at all is causing the fork here?

I think Anselme will probably be the best person to look at this. He will be able to reproduce the issue with the speed test script?

[riddim]

@chriso

OK, so are you saying any update at all is causing the fork here?

that can’t be the case either … I see raising counters when a read succeeds and I was able to update the speediest-scratchpad a few times I think and resolve the forking error with this … maybe … but it reappeared …

step 1 would be to expand the error message to include the counter values and first bytes of the forked scratchpads… so we see if it’s really forks with same counters … and what the counter values are etc … I really don’t understand why there should be so many forks and if they’re really forks then e.g. the python lib should return all the conflicting versions …

[chriso]

OK thanks for the info.

It looks like the Rust version returns a list of scratchpads, so I assume that would be all of them.

But it sounds like the Python library doesn’t have the same behaviour.

Are you using Python for Friends?

You also mentioned on Discord that there is no issue on alphanet. Have you confirmed this recently? The alpha network was updated with the latest version of the node.

[riddim]

@chriso

Are you using Python for Friends?

no - dweb - but that doesn’t handle the multiple versions it seems (and it’s not like it would have been communicated anywhere outside the IF channel … so @happybeing couldn’t have known in advance … and the change from “return first” to “returning multiple” then was discussed only in there too between @zettawatt and anselme)

@chriso

You also mentioned on Discord that there is no issue on alphanet. Have you confirmed this recently?

no and I did multiple writes on a new scratchpad from my cloud instance without issues and forks too now from python on mainnet … so I have no idea where those forks are coming from … maybe they tend to be created on worse internet connections (?)

ps:

on alpha I don’t see forks on friends

[Josh]

@riddim

discussed only in there

Going to pipe in only for a brief second.

This is a issue.

[Toivo]

@chriso

You also mentioned on Discord that there is no issue on alphanet. Have you confirmed this recently? The alpha network was updated with the latest version of the node.

Maybe too old a thing, but if I remember correctly, @happybeing reported last autumn pointers (or some such) working reliably on local testnet and failing equally reliably on the public testnet.

[riddim]

alpha is super robust and working like a charm at insane speeds

[happybeing]

@riddim

(while the biggest question really is why those forks are appearing … it’s not like they should appear … it’s not multiple concurrent writes that could create race conditions like with the speediest-script where it could be run in parallel by different people

@Chriso I accept your point [cough] about Pointers and priorities, but this seems very similar to the issue I originally reported there (getting different versions of a Pointer on different GETs), and which was ‘fixed’ (probably) but now results in some Pointers getting stuck sooner or later. So maybe not really fixed.

So addressing one may well be the key to addressing the other. In neither case have these been dues to concurrent updates, so figuring out what happens on update - on mainnet - over time, short gap and longer gap (days, weeks) might be the way to go forward.

@riddim

no - dweb - but that doesn’t handle the multiple versions it seems (and it’s not like it would have been communicated anywhere outside the IF channel … so @happybeing couldn’t have known in advance

I confirm dweb doesn’t know about forked Scratchpads. I recall there was some handling of Pointer forks in the original Register code and I may have simplified that - thinking that forks should not occur without concurrent updates. So my code may be part of the issue with Pointers. For Scratchpads though it just does what the early Scratchpad APIs supported.

[chriso]

@happybeing

I accept your point [cough] about Pointers and priorities, but this seems very similar to the issue I originally reported there (getting different versions of a Pointer on different GETs), and which was ‘fixed’ (probably) but now results in some Pointers getting stuck sooner or later. So maybe not really fixed.

OK thanks for the info.

I will pass this along also.

[zettawatt]

Do you guys sleep? I feel like I missed a lot here.

On the scratchpad forks, yes, the last fix from Anselme fixed the issue for me. I never saw this issue until the last round of updates where the network went flaky. Here is what I suspect is happening:

• I intend to make an update to an existing scratchpad, so I download it to get the current counter value, let’s call this version A
• I increment the counter, and push the updated scratchpad to the network, now version B
• I repeat this operation for a subsequent update, but due to the state of the network or just lack of time for the edit to propagate, I sometimes only get version A and its counter value again.

On a subsequent get, now there are 2 scratchpads with the same counter, hence the forked scratchpad error. In my scratchpad get function, I do this:

        let scratchpad = match self.client.scratchpad_get(&scratchpad_address).await {
            Ok(scratchpad) => scratchpad,
            Err(e) => match e {
                ScratchpadError::Fork(scratchpads) => Self::select_newest_scratchpad(scratchpads),
                _ => return Err(Error::Scratchpad(Box::new(e))),
            },
        };

The 'select_newest_scratchpadfunction looks at all of the returned scratchpads and compares the header, where I put a timestamp like this: #2025-07-20T01:53:13.629506885+00:00Then I take the newest one and return that scratchpad.

That’s how I fixed it, but I think the root cause is that the counter value (in my case) is arbitrary and relative to the last update. If I used some other increment value based on real time, I don’t think I would ever encounter this issue.

I never saw this issue on local or alpha, everything ‘just works’ on both of those. For pointers, I sometimes will get an error, which I’m guessing could be a similar scenario, but unlike @happybeing, I don’t have any issues. In my case, the pointer address never changes, it always points to the same scratchpad. I’m only using the counter value to tell me when an update occured on the scratchpad so I can skip downloading it when updating the local search cache.

[chriso]

Thanks for the detailed response here.

Hey @riddim is this useful for you? Could you take a similar approach with your app?

[riddim]

@chriso

Could you take a similar approach with your app?

No - I 100% depend on a dweb upgrade - without dweb supporting forks I cannot work with them … I can ofc look into creating a pr for dweb but in this moment (and when using python to interact with the network) this is not possible for me - right now both the official python lib and dweb just error out without giving me any info about the currently existing Scratchpad versions (not even the counter value without wich not even starting from scratch with an empty pad is possible…)

[happybeing]

I should have some time tomorrow to look at this. No promises, but copy and paste of @zettawatt’s code will probably do it I think. I just have to remember to have a look

[riddim]

thank you so much … had a lot of other stuff going on the last few days … so tomorrow night may be the first chance to look into “the important stuff” again for me

[riddim]

@chriso

Hey @riddim is this useful for you?

@chriso our awesome @happybeing did create a dweb patch (@zettawatt’s solution most certainly helped here to get faster to an implementation) that is not yet released but I tested it and it works like a charme

friends will soon work again as expected for everyone and I will release a new version with some enhancements too

so now only python is left behind but that’s not super important for friends in particular

[chriso]

Awesome, nice to hear you are back on track there.

Thanks to you both!

Maybe we need to speak to Victor on the Python bindings.

Oh and this is also working for the production network? Is there something that explains the difference you were seeing between production and alpha?

[riddim]

@chriso

Oh and this is also working for the production network?

I’m only talking about production network here with forks - yes

from the pads I see forked I’m pretty sure it’s something that happened while mainnet was pretty flaky … (so possibly an old version of the scratchpad returned to friends … and the network not rejecting another write with same version because of the overall state and data-propagation-delays)

alpha never was in this rather undesirable state

[Shu]

@Shu

Weather report off the network currently:

Isolated and scattered thunderstorms, though not severe, .

Hopefully, it heads towards lighter rain, and eventually sunny days.

@riddim

friends will soon work again as expected for everyone and I will release a new version with some enhancements too

Nice! Updated weather report:

Clearing skies. Mild rain. Temperature reported to be significantly warmer. .

[loziniak]

On forked scratchpads… (is there a Github issue perhaps? Maybe these technical talks would better fit there?)

Is this possible, that forked scratchpads are identical? From this code:

// these are scratchpads from ScratchpadError::Fork
scratchpads.iter().map(|sp| {
    println!("Scratchpad: {sp:?}");
    let bytes = sp.decrypt_data(sk).map_err(|e| format!("{e:?}"))?;
    println!("Data: {:x}", bytes);
    let wallet = rmp_serde::from_slice(&bytes).map_err(|e| format!("{e:?}"))?;
    println!("Wallet: {wallet:?}");
    Ok(wallet)
})

I get this output (added formatting for clarity):

Scratchpad: Scratchpad {
    address: b767c15841d7b59c212ab17e934b334566f241d332482bdf42659105cf720c68d1979f0f84833bc7cc7a3319a9c9c662,
    data_encoding: 0,
    encrypted_data: "(628 bytes of encrypted data)",
    counter: 8,
    signature: "8d23f599637de0cf66ddd18086dbd565de18afcdf04270b7631dcd30f527992aa36dc83f3e038554092ca5450a8a9e5813d10daa45b3f5760d153e279d56ca77d3b191a86ac2f3c738aad9d35a15ba6ba5fa6ba18cb1b37992fec617f9338a16"
}
Data: 93dc0030ccb767ccc15841ccd7ccb5cc9c212accb17ecc934b334566ccf241ccd332482bccdf4265cc9105cccf720c68ccd1cc97cc9f0fcc84cc833bccc7cccc7a3319cca9ccc9ccc66283dc0020cc84640237ccee78ccb5cce3cca85427cc8d0b0805cc94ccbc2036051eccc74fccc27532154ccce1ccc1ccc00092c420000000000000000000000000000000000000000000000000000000000000000490dc0020ccb778cc840accce4bccf9ccaf5669ccf1cc98cc87cc87ccb6ccb264392c32cce1cc8258cc896c417814ccacccde296e92c420000000000000000000000000000000000000000000000000000000000000000290dc002043ccd55d72ccb2cca97e4dcce7ccef64446bcc8bccb878ccdccc88cc9450cca9ccb8076bcce4cc89cc9422cce62b656492c42000000000000000000000000000000000000000000000000000000000000000039192dc0030cc88621dccbfcc8dccd0706ccc8c7761ccd47acce8cc91ccac0eccb3cc8f5406ccd3ccefcc8accb0cc954ecce2ccaacc82cccdcca517cc97cca3566d1ecc84ccedccbfcc88cc97cce4cce9ccae5711c4200000000000000000000000000000000000000000033b2e3c9fd0803ce8000000c4200000000000000000000000000000000000000000000000000000000000000004
Wallet: Wallet(
    PublicKey(1767..7d0a),
    {
        Some(846402(10000100)..): (4, []),
        Some(43d55d(01000011)..): (3, [(PublicKey(0862..9ca1), 1000000000000000000000000000)]),
        Some(b77884(10110111)..): (2, [])
    },
    4
)
Scratchpad: Scratchpad {
    address: b767c15841d7b59c212ab17e934b334566f241d332482bdf42659105cf720c68d1979f0f84833bc7cc7a3319a9c9c662,
    data_encoding: 0,
    encrypted_data: "(628 bytes of encrypted data)",
    counter: 8,
    signature: "8d23f599637de0cf66ddd18086dbd565de18afcdf04270b7631dcd30f527992aa36dc83f3e038554092ca5450a8a9e5813d10daa45b3f5760d153e279d56ca77d3b191a86ac2f3c738aad9d35a15ba6ba5fa6ba18cb1b37992fec617f9338a16"
}
Data: 93dc0030ccb767ccc15841ccd7ccb5cc9c212accb17ecc934b334566ccf241ccd332482bccdf4265cc9105cccf720c68ccd1cc97cc9f0fcc84cc833bccc7cccc7a3319cca9ccc9ccc66283dc0020cc84640237ccee78ccb5cce3cca85427cc8d0b0805cc94ccbc2036051eccc74fccc27532154ccce1ccc1ccc00092c420000000000000000000000000000000000000000000000000000000000000000490dc0020ccb778cc840accce4bccf9ccaf5669ccf1cc98cc87cc87ccb6ccb264392c32cce1cc8258cc896c417814ccacccde296e92c420000000000000000000000000000000000000000000000000000000000000000290dc002043ccd55d72ccb2cca97e4dcce7ccef64446bcc8bccb878ccdccc88cc9450cca9ccb8076bcce4cc89cc9422cce62b656492c42000000000000000000000000000000000000000000000000000000000000000039192dc0030cc88621dccbfcc8dccd0706ccc8c7761ccd47acce8cc91ccac0eccb3cc8f5406ccd3ccefcc8accb0cc954ecce2ccaacc82cccdcca517cc97cca3566d1ecc84ccedccbfcc88cc97cce4cce9ccae5711c4200000000000000000000000000000000000000000033b2e3c9fd0803ce8000000c4200000000000000000000000000000000000000000000000000000000000000004
Wallet: Wallet(
    PublicKey(1767..7d0a),
    {
        Some(b77884(10110111)..): (2, []),
        Some(846402(10000100)..): (4, []),
        Some(43d55d(01000011)..): (3, [(PublicKey(0862..9ca1), 1000000000000000000000000000)])
    },
    4
)

[Toivo]

Seems so:

fix: resolve failuring data_with_churn CI test