Memory and performance testing

Thanks for this @peca.

This seems like Cmds have failed to go through properly. Atm, if you see that “verification failed” error it generally means you’d need to try reuploading the file. I don’t think we do any checks on the cmd, beyond trying to verify by dlding. So a fail there means that some chunks were not put.

So to be clear, there’s a difference between not all chunks going up in one attempt (could well happen… just reupload and no worries; the ux of this should be improved down the line), and the nodes (or client) crashing.

It’d be good to see more about the crashing of nodes/client there.

This is interesting too. Seems like the nodes are unresponsive. Do you have logs from the client/nodes here @peca ?

I’d be intersting in seeing a message going out from the client (and it’s corresponding MsgId), and then seeing if that was received at Elders (or why not).

4 Likes

I am affraid the logs are gone I will look into it.

I will do more tests today and try to better watch the events and write down what happen when so it will be easier to pair it with logs.

4 Likes

I compiled new binary in the morning and here are todays results:

For files <250 MB everything works as expected, no missing chunks, uploads and download work on the first try. Memory usage is reasonable and steady.

For files >280MB put always finishes, mostly with “verification failed” (reuploading desn’t help), sometimes finishes cleanly, but downloading never works. Either it creates only empty 0 b file, or it throws this error:

./safe files get "safe://hyryyryo8m71h5qsccq61tqyhbjncxfte8tnw57j1hqzenjgrnuuegmpopanra?v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey"
Error: 
   0: Path '.' not found

Location:
   sn_cli/src/subcommands/files_get.rs:207

Also with these 280+ MB files it leaks memory a lot. After uploading 1.4 GB file it looks like this:


…but it didn’t crash or stall this time, It still runs and I am able to upload and then download <250 MB files at the same speed as before.

Maybe I understand something wrong, but I find this wierd:
What is this part of container address? Is it normal to be same for different files?
v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey

/safe files put /home/petr/Dokumenty/abcd.mkv
FilesContainer created at: "safe://hyryyrywa4pc3h6kuw1njuk4seer456te6zde1wayboupwqytiuhwjntmnynra?v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey"
+---+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| E | /home/petr/Dokumenty/abcd.mkv | <Content may have been correctly stored on the network, but verification failed: safe://hyryyyyj3xw18zkyscf7o74bweywdb61h4u68gsz19pi94nk3grgf8zfmse> |
+---+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+

./safe files put "/home/petr/Stažené/The Orville (2017) Season 3 S03 (1080p DSNP WEB-DL x265 HEVC 10bit DDP 5.1 Vyndros)/The.Orville.S03E01.Electric.Sheep.1080p.10bit.DSNP.WEB-DL.DDP5.1.HEVC-Vyndros.mkv"
FilesContainer created at: "safe://hyryyryo8m71h5qsccq61tqyhbjncxfte8tnw57j1hqzenjgrnuuegmpopanra?v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey"
+---+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| E | /home/petr/Stažené/The Orville (2017) Season 3 S03 (1080p DSNP WEB-DL x265 HEVC 10bit DDP 5.1 Vyndros)/The.Orville.S03E01.Electric.Sheep.1080p.10bit.DSNP.WEB-DL.DDP5.1.HEVC-Vyndros.mkv | <Content may have been correctly stored on the network, but verification failed: safe://hyryyyyxq38a9k3kapux143qj495ndetni1c6rsmfdra8pkbtxk3s4trjbr> |
+---+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+

Here are logs

10 Likes

The container is like a directory name so will be the same for all files in the same container.

4 Likes

I would expect that when uploading a folder, but these were separate single file uploads. Does this container work like a default upload folder for single files with unspecified folder?

2 Likes

Ah k, but if you look the container names are different!

2 Likes

I know, containers are different. I would expect this if it was same file in different folders, but these are different files from different folders. Or if the v=part was same for all uploaded files I would guess it is client id or something, but it wasn’t.
I feel like I saw this exact v=string many times today and also last time and it was only when when upload or download failed.

This is from @Southside 3 days ago, “verification failed”, same v=string:

willie@gagarin:~$ safe files put Videos/League_Of_Gentlemen_Xmas_Special_2000_DVDRip_XviD.avi 
FilesContainer created at: "safe://hyryyryuk61uid4wkahfmeh85nj3dyhf3ds344jjt711gyect9aobc7zq5cnra?v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey"
+---+--------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| E | Videos/League_Of_Gentlemen_Xmas_Special_2000_DVDRip_XviD.avi | <Content may have been correctly stored on the network, but verification failed: safe://hy8ayyyx8nubohs8c3wygxuc7n7pdwgh1yobo3naq4bz3f848on9jitwrby> |
+---+--------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
3 Likes

Until we get a UI, these mistakes will be frequent

safe://hyryyyyj3xw18zkyscf7o74bweywdb61h4u68gsz19pi94nk3grgf8zfmse etc is just not friendly for humans

I will go back and look for similar occurrences

1 Like

The v part is a version, not the container name so you have a container and a version.

5 Likes

But why would the version string be similar?

Do we start at a seemingly random no and increment?

1 Like

I’m curious myself. Not sure how version is generated but I recall is was once a sequential value but this was changed. I forget why!

2 Likes

It is somehow connected to the problem. I just looked at all files I tried to upload today and tried few more just to be sure:
files <250 MB get unique v=string and work
files >280 MB all get v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey and don’t work

9 Likes

Could be you isolated a key bug there, it certainly doesn’t look right. :clap:

@joshuef?

6 Likes

Grrr - an error in my flight sim scenery download has spewed thousands of lines into the terminal tab that had the verification failed messages and its wrapped round.

Only recent uploads I have evidence for were successful

willie@gagarin:~/Music$ safe files put PsychoKiller.mp3 
FilesContainer created at: "safe://hyryyryziaf96ejkoqj3t7de534gzy7apeo9xyt8r89ro1688ceko8fib5wnra?v=hhtbgynj44xc4qmiyfrdh1hh8gs9ht35eyyb5obgzhhhemjtt41mo"
+---+------------------+--------------------------------------------------------------------+
| + | PsychoKiller.mp3 | safe://hygyy1yp99wpd9it9xeqswx14sccs4835gpbeg8upaas9gy7h9g7fnaywgw |
+---+------------------+--------------------------------------------------------------------+

This is NOT your granda’s old Eddie Arnold version BTW

willie@gagarin:~/Music$ safe files put "I Really Don't Want To Know.mp3"
FilesContainer created at: "safe://hyryyrywf3ok7jcg8q11aewj61i9wfcnasxw1cbg88w7arpixingydeaqbonra?v=hiyjwinb1qocwt1ky8juzjqy4th3169k5t3d7ihicjiuegzs6kssy"
+---+---------------------------------+--------------------------------------------------------------------+
| + | I Really Don't Want To Know.mp3 | safe://hygyy1yce3zwfksekqy88hg4uk7kuwjb4yop75g4kpc4i9f5wg3qb94cjch |
+---+---------------------------------+--------------------------------------------------------------------+

so eagle-eyed @peca spotted the failure on the League of Gentlemen Xmas Special :clap: :clap: :clap:

Manky and knackered after physical work today. Going for a bath. If I feel awake after a good long soak, I will fire up a baby-fleming and see if this version anomaly can be recreated - No promises though…

2 Likes

Numbers are too close to 2^8 (or 2^18 or 2^28).
I doubt that it matters, but who knows.

3 Likes

Confirmation of this bug
Here is our old pal v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey again

willie@gagarin:~/Downloads$ ll logs2.zip 
-rw-rw-r-- 1 willie willie 466365415 Sep  4 13:33 logs2.zip


willie@gagarin:~/Downloads$ safe files put logs2.zip 
FilesContainer created at: "safe://hyryyrytbi441z81yhqetd81bzwjm68w48wjmezf1mhgqoik49sj5xiiogonra?v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey"
+---+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| E | logs2.zip | <Content may have been correctly stored on the network, but verification failed: safe://hyfenryxhwpoiu86e75axznzq9n1xgjzzzpfik6787nouzbtdpkaonozqur> |
+---+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------+

A 320MB file was OK however

willie@gagarin:~/Downloads$ ll latest-USB.img.gz 
-rw-r--r-- 1 willie willie 320284065 Jul  4  2015 latest-USB.img.gz

willie@gagarin:~/Downloads$ safe files put latest-USB.img.gz 
FilesContainer created at: "safe://hyryyryiwe6madrx5bhii1c4j53wxqhttryu7nu194xhcnewy7uwkauqm6anra?v=hhygmdpt8eh6wsfhtucn395k4bry4jndyohb5h7ky1xg9jddnrqso"
+---+-------------------+--------------------------------------------------------------------+
| + | latest-USB.img.gz | safe://hyryyyym49yack7odykkqpninb4kphnwfop69qus6eifrmz3irkn6r493gr |
+---+-------------------+--------------------------------------------------------------------+

This is with the last commit yesterday - before the new release I just noticed there…

EDIT: I’ll grab this release and try again

3 Likes

It was changed to be based upon registers, rather than serialised json for CRDT benefits. With that, a hash was needed to describe where in the tree things sit (I believe). (@Anselme I think you were last about here… not sure if you can shed any more light?)

I think the version should not be duplicated so… But i’m not sure. Will check and report back Bug or No Bug

9 Likes

I can now upload the 460MB logs2.zip file that failed earlier

willie@gagarin:~/Downloads$ safe files put logs2.zip 
FilesContainer created at: "safe://hyryyryu5xdsdq7uxxhnb5udwy4p5keiutfp6iikfzqbx7158hquy4yo1ahnra?v=huccsmbrzo8weumcukp8ww7x33gd89ihzxh78t9y9484faoj9euuy"
+---+-----------+--------------------------------------------------------------------+
| + | logs2.zip | safe://hyfenryxhwpoiu86e75axznzq9n1xgjzzzpfik6787nouzbtdpkaonozqur |
+---+-----------+--------------------------------------------------------------------+
willie@gagarin:~/Downloads$ ll logs2.zip 
-rw-rw-r-- 1 willie willie 466365415 Sep  4 13:33 logs2.zip

needs further testing but the new release looks promising

:slight_smile: Thanks guys

4 Likes

Versions used to be sequential values, but those weren’t resistant to concurrent writes. If two folks update v1 (to v2) for the same file at the same time, we can end up with a split view in the network were some nodes have one of the v2 and some have the other.

Sequences:

         v1(data)

when I insert 2 concurrent values after v1:

         v1(data) -> v2(data_2)
OR
         v1(data) -> v2(data_3)

GET v2: ??

So we replaced them with registers as they can deal with concurrent writes. Registers store values in a Merkle DAG, where each entry has a hash. If two nodes were to update the same entry with a different value, they would get two valid hashes that can both be inserted to the DAG.

Registers are a DAG:

                    hash_1(data)

when I insert 2 concurrent values after hash_1:

                    hash_1(data)
                     /        \
         hash_2(data_2)       hash_3(data_3)

GET hash_2: data_2
GET hash_3: data_3

The version is this hash.

11 Likes