This seems like Cmds have failed to go through properly. Atm, if you see that “verification failed” error it generally means you’d need to try reuploading the file. I don’t think we do any checks on the cmd, beyond trying to verify by dlding. So a fail there means that some chunks were not put.
So to be clear, there’s a difference between not all chunks going up in one attempt (could well happen… just reupload and no worries; the ux of this should be improved down the line), and the nodes (or client) crashing.
It’d be good to see more about the crashing of nodes/client there.
This is interesting too. Seems like the nodes are unresponsive. Do you have logs from the client/nodes here @peca ?
I’d be intersting in seeing a message going out from the client (and it’s corresponding MsgId), and then seeing if that was received at Elders (or why not).
I compiled new binary in the morning and here are todays results:
For files <250 MB everything works as expected, no missing chunks, uploads and download work on the first try. Memory usage is reasonable and steady.
For files >280MB put always finishes, mostly with “verification failed” (reuploading desn’t help), sometimes finishes cleanly, but downloading never works. Either it creates only empty 0 b file, or it throws this error:
./safe files get "safe://hyryyryo8m71h5qsccq61tqyhbjncxfte8tnw57j1hqzenjgrnuuegmpopanra?v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey"
Error:
0: Path '.' not found
Location:
sn_cli/src/subcommands/files_get.rs:207
Also with these 280+ MB files it leaks memory a lot. After uploading 1.4 GB file it looks like this:
…but it didn’t crash or stall this time, It still runs and I am able to upload and then download <250 MB files at the same speed as before.
Maybe I understand something wrong, but I find this wierd:
What is this part of container address? Is it normal to be same for different files? v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey
/safe files put /home/petr/Dokumenty/abcd.mkv
FilesContainer created at: "safe://hyryyrywa4pc3h6kuw1njuk4seer456te6zde1wayboupwqytiuhwjntmnynra?v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey"
+---+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| E | /home/petr/Dokumenty/abcd.mkv | <Content may have been correctly stored on the network, but verification failed: safe://hyryyyyj3xw18zkyscf7o74bweywdb61h4u68gsz19pi94nk3grgf8zfmse> |
+---+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
./safe files put "/home/petr/Stažené/The Orville (2017) Season 3 S03 (1080p DSNP WEB-DL x265 HEVC 10bit DDP 5.1 Vyndros)/The.Orville.S03E01.Electric.Sheep.1080p.10bit.DSNP.WEB-DL.DDP5.1.HEVC-Vyndros.mkv"
FilesContainer created at: "safe://hyryyryo8m71h5qsccq61tqyhbjncxfte8tnw57j1hqzenjgrnuuegmpopanra?v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey"
+---+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| E | /home/petr/Stažené/The Orville (2017) Season 3 S03 (1080p DSNP WEB-DL x265 HEVC 10bit DDP 5.1 Vyndros)/The.Orville.S03E01.Electric.Sheep.1080p.10bit.DSNP.WEB-DL.DDP5.1.HEVC-Vyndros.mkv | <Content may have been correctly stored on the network, but verification failed: safe://hyryyyyxq38a9k3kapux143qj495ndetni1c6rsmfdra8pkbtxk3s4trjbr> |
+---+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
I would expect that when uploading a folder, but these were separate single file uploads. Does this container work like a default upload folder for single files with unspecified folder?
I know, containers are different. I would expect this if it was same file in different folders, but these are different files from different folders. Or if the v=part was same for all uploaded files I would guess it is client id or something, but it wasn’t.
I feel like I saw this exact v=string many times today and also last time and it was only when when upload or download failed.
This is from @Southside 3 days ago, “verification failed”, same v=string:
willie@gagarin:~$ safe files put Videos/League_Of_Gentlemen_Xmas_Special_2000_DVDRip_XviD.avi
FilesContainer created at: "safe://hyryyryuk61uid4wkahfmeh85nj3dyhf3ds344jjt711gyect9aobc7zq5cnra?v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey"
+---+--------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| E | Videos/League_Of_Gentlemen_Xmas_Special_2000_DVDRip_XviD.avi | <Content may have been correctly stored on the network, but verification failed: safe://hy8ayyyx8nubohs8c3wygxuc7n7pdwgh1yobo3naq4bz3f848on9jitwrby> |
+---+--------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
It is somehow connected to the problem. I just looked at all files I tried to upload today and tried few more just to be sure:
files <250 MB get unique v=string and work
files >280 MB all get v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey and don’t work
Grrr - an error in my flight sim scenery download has spewed thousands of lines into the terminal tab that had the verification failed messages and its wrapped round.
Only recent uploads I have evidence for were successful
willie@gagarin:~/Music$ safe files put PsychoKiller.mp3
FilesContainer created at: "safe://hyryyryziaf96ejkoqj3t7de534gzy7apeo9xyt8r89ro1688ceko8fib5wnra?v=hhtbgynj44xc4qmiyfrdh1hh8gs9ht35eyyb5obgzhhhemjtt41mo"
+---+------------------+--------------------------------------------------------------------+
| + | PsychoKiller.mp3 | safe://hygyy1yp99wpd9it9xeqswx14sccs4835gpbeg8upaas9gy7h9g7fnaywgw |
+---+------------------+--------------------------------------------------------------------+
This is NOT your granda’s old Eddie Arnold version BTW
willie@gagarin:~/Music$ safe files put "I Really Don't Want To Know.mp3"
FilesContainer created at: "safe://hyryyrywf3ok7jcg8q11aewj61i9wfcnasxw1cbg88w7arpixingydeaqbonra?v=hiyjwinb1qocwt1ky8juzjqy4th3169k5t3d7ihicjiuegzs6kssy"
+---+---------------------------------+--------------------------------------------------------------------+
| + | I Really Don't Want To Know.mp3 | safe://hygyy1yce3zwfksekqy88hg4uk7kuwjb4yop75g4kpc4i9f5wg3qb94cjch |
+---+---------------------------------+--------------------------------------------------------------------+
so eagle-eyed @peca spotted the failure on the League of Gentlemen Xmas Special
Manky and knackered after physical work today. Going for a bath. If I feel awake after a good long soak, I will fire up a baby-fleming and see if this version anomaly can be recreated - No promises though…
Confirmation of this bug
Here is our old pal v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey again
willie@gagarin:~/Downloads$ ll logs2.zip
-rw-rw-r-- 1 willie willie 466365415 Sep 4 13:33 logs2.zip
willie@gagarin:~/Downloads$ safe files put logs2.zip
FilesContainer created at: "safe://hyryyrytbi441z81yhqetd81bzwjm68w48wjmezf1mhgqoik49sj5xiiogonra?v=hnrijhg7qx435iejzzjm1i5oc3qgw3s3b36ej9r7wwe8nmgxjnfey"
+---+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| E | logs2.zip | <Content may have been correctly stored on the network, but verification failed: safe://hyfenryxhwpoiu86e75axznzq9n1xgjzzzpfik6787nouzbtdpkaonozqur> |
+---+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------+
It was changed to be based upon registers, rather than serialised json for CRDT benefits. With that, a hash was needed to describe where in the tree things sit (I believe). (@Anselme I think you were last about here… not sure if you can shed any more light?)
I think the version should not be duplicated so… But i’m not sure. Will check and report back Bug or No Bug
Versions used to be sequential values, but those weren’t resistant to concurrent writes. If two folks update v1 (to v2) for the same file at the same time, we can end up with a split view in the network were some nodes have one of the v2 and some have the other.
Sequences:
v1(data)
when I insert 2 concurrent values after v1:
v1(data) -> v2(data_2)
OR
v1(data) -> v2(data_3)
GET v2: ??
So we replaced them with registers as they can deal with concurrent writes. Registers store values in a Merkle DAG, where each entry has a hash. If two nodes were to update the same entry with a different value, they would get two valid hashes that can both be inserted to the DAG.
Registers are a DAG:
hash_1(data)
when I insert 2 concurrent values after hash_1:
hash_1(data)
/ \
hash_2(data_2) hash_3(data_3)
GET hash_2: data_2
GET hash_3: data_3
Thanks guys