NewYearNewNet [04/01/2024 Testnet] [Offline]

[neo]

@dirvine

They decrypt and dev lier on request This means the on disk data is secured from everyone, but when the node reboots that is useless data now, which is fine for our small nodes.

replication?

do all chunks have to be decrypted with the temp key when being replicated? Not talking of or considering self encryption here.

my idea allows the client to do the work of decryption and the node does not have the keys in memory so even if the node is compromised at some time in future the one who compromised the machine cannot decrypt those chunks (eg a cracker/malware, machine seized, etc). Also the node owner can never be accused of knowingly storing the chunks at rest. If the node operator can at will decrypt the chunks at rest (even replicated to the node) then the knowingly storing can be used against them.

But yes the temp key is an easy and simple way to achieve most of whats needed. Only extra my suggestion gives is the responsibility (legally and doing it) to decrypt rests with the client retrieving the file/chunks.

[dirvine]

@neo

do all chunks have to be decrypted with the temp key when being replicated?

Yes when the chunk leaves disk it’s unencrypted and that would elso mean for replication.

@neo

my idea allows the client to do the work of decryption

Clients could encrypt the data, but then cannot share it. However, we cannot force clients to do anything and we must assume some will subvert the API

[neo]

@dirvine

Clients could encrypt the data, but then cannot share it.

Ah but having the client do that encryption only brings us back to the original problem of allowing clients the ability to cause unencrypted chunks to be stored on nodes.

As I said the temp key solves all but the node operator, or malware, or computer being seized while on being able to view chunks stored, even for those replicated to them.

Yes the node operator could copy the incoming chunks while in memory, but thats another legal separation. For the node operator that doesn’t do that they have the legal defence of not knowingly storing the chunk.

Also if malware then it requires the malware to be there when new chunks arrive which is a much lesser chance getting anything than the case of the temp key in memory opening all chunks not encrypted with self-encryption.

But as I also said the temp key in memory is a good compromise for these edge cases.

[dirvine]

So @joshuef For Registers I think it’s a really simple change.

Default

• On store, clients encrypt the Entry with the Register owners PubKey
• On read clients must have the SecretKey to decrypt.

So really simple and neat.

Public Data
The clients just miss out the step above.

Each Entry can point to a chunk and the first chunk is likely the file data map, which is also encrypted with the register owners key by default, unless the registers pointing to open data in which case the data map chunk is not encrypted.

I think as long as our Default settings ensure all data is fully encrypted by default and we also have the registers as directories (i.e. filesystems) then the client side apps we can build will be amazing. Of course depends on the account packet, which is a simple affair.

Nice chat folks, I think we got somewhere really simple and very powerful here. Best of all the code is client side impl, we know the network works, well when we fix the filename spaces And test a bit more.

Really nice place to be in here.

[Josh]

@dirvine

So @joshuef For Registers I think it’s a really simple change.

Josh when he comes to this thread after the weekend.

[happybeing]

Ok, new profile pic sorted

[scottefc86]

More like me when I try to follow the discussion

[storage_guy]

I hit the faucet 3 times while it was still up and I have plenty for my purposes. I’m also going to grab what I’ve earned from my nodes as well and use that first to see how far I can get in a self sustaining way.

So I have some to spread around in case people are short and want to upload more. I have to go to bed now but if anyone wants to post their addresses as found with:-

safe wallet address

I’ll spread what I have evenly amongst the addresses in the morning.

[mav]

Please send me some for testing

8f459c4e801ffb6014f1a320c4435a336fdefa925f9f17ada40263241563b7cf20f586643b0dc2fb752652f46ec12eda

Faucet error:

Requesting token for wallet address: 8f459c4e801ffb6014f1a320c4435a336fdefa925f9f17ada40263241563b7cf20f586643b0dc2fb752652f46ec12eda
Error: 
   0: error sending request for url (http://134.209.21.136:8000/8f459c4e801ffb6014f1a320c4435a336fdefa925f9f17ada40263241563b7cf20f586643b0dc2fb752652f46ec12eda): error trying to connect: tcp connect error: Connection refused (os error 111)
   1: error trying to connect: tcp connect error: Connection refused (os error 111)
   2: tcp connect error: Connection refused (os error 111)
   3: Connection refused (os error 111)

[storage_guy]

Yes, the faucet is confirmed to be down

You caught me just before I logged. Here’s 10 to be going on with:-

0b80cc9dccafcce6cc200a0ef2cccbccc8cc4d746f0b0fc1cc25f9ccb6cc39645b6e748ecc6a39fcccf8cc4ce9ccc2cc23ccccc6cc6070160cf7cc5831c3ccb5cc184701188bccfacca5ccc1ccdbcc1ecfccfcccb1ccd7ccbfcc17e3cce5cc91ccf3ccdacc1eb2cc9ccc02a0cc0532fccc9fcc37c4cc080aaccc15fecc65c9cc05a7cc3d2b1ab1cc002496ccfdcc91ccb7cc6000dcefcc623e2a7f78e5cc43a2cc41edcc264fcaccf9cc7a7d50179acc2889cc8fcc433dcbcc70588bcc3aa2ccbacc85cc52beccadcc31dacc81cce6cc4b403a89cc87cc3ff1cc5f592bb3ccc2cc2bd4ccd8cca0ccc5cc8dccb1cce9cc310c24770a244e4551f8ccfbcc411b6d42034bcecc3c1ec6cc484882ccbecc43abcc3594cc4f1fdccc0964159dcc3f7d9ccc15a5ccbfcc570b9ccc4796cc6b00dcd9cc4a095125aecc92cc8ccc6ea6cc22aaccb1ccbbcc85cc4596cc24becc3b5ac8cc89ccbcccd6cc6d27dcccc2cc6c0eaacc70f1cc6381cca4cc4995cc06d9cc3ff9ccd5ccd2cc6ec9ccb0cc3000dc9391646574707972636e45a981

which you can get into your wallet with:-

safe wallet receive <transfer>

[chrisfostertv]

50 sent

10b9cc7cfccc6612fecc7d1fafcc23c3ccddcce4cc9cccb1cc002dc7ccf4ccbecc1c48a7ccf9cc10bbcc96cc79b3cca2ccefccd0ccc0cc4e5576630dd3cc7dd5cccccc2f2136370ef2cccfcc02722f78c7cc19c5cc0b82cc32aacc7780cc88cc2fa1cc8acc60d2cc53d8cc1705e4cc4d736eefccdfcc4a5d9ccc6de2cc5a099dcc71c8cce2ccc2cc63c8cc693fa2cc6000dccccc73ddccfeccfcccb7cc72132c6168660bb3cc94cc83cc88cca9cc94ccb0cc53d5cc89cc83cc105067b3cc74c4cca3cc585bb5cc4bdacc8bcc722dbdcc51f7cc81cc3e15a6cc95cc70d6cc16a3cc321d192d7d5c3736113d5ccfcc84cc03aecc95cc95ccf1ccbacc1240f9cc6ab2cc43e4cca2cc94ccaecc73cecccecc57dbccbfcc57b5cc13f4cc377f2d10418bcc9bcc270becccc7cceacc6600dc6d7e4bc8cce9cc5d3deccce2cc50aecc37f7cc5377691735e4cc81ccd9cc0e0292ccdfccb0cc61515e6d45e1ccf0cc45e3cc1696cc83cc1ecdccb0cc0d1c66410b5181cc3000dc9391646574707972636e45a981

[mav]

Satisfying to be able to receive these at the same time as uploading a file, no conflict with simultaneous send and receive

[storage_guy]

I’m up and the bank is open for more disbursements. Anyone else short of chocolate coins?

[storage_guy]

I’ve found something odd. I started my random data file uploading script last night and it’s apparently uploaded a whole load of 1MB files in batches of 10 with a file containing the md5sums of each run. However, it’s reporting that for all except the first 11 files they are the same file but not going to upload them again! They are verifiably not the same file and they have diffferent addresses.

For all runs apart from the first one it reports:-

Files upload attempted previously, verifying 41 chunks
All files were already uploaded and verified

The only change I made to the script was to remove the ‘-c’ from the safe files upload command and add the ‘-p’ to make the files public.

Is it because the files in each run all have the same name as previous runs?

Or maybe it’s something to do with making the files public?

But the files in each run have different addresses to previous ones so I’m baffled.

I’ll try it without the ‘-p’ option and see if that makes a difference and then I’ll try again with something to make each file have a different name.

This is the start of the log showing the issue:-

Run 1
Mon  8 Jan 00:09:16 UTC 2024

Logging to directory: "/home/safe/.local/share/safe/client/logs/log_2024-01-08_00-09-16"
Built with git version: ee245b6 / main / ee245b6
Instantiating a SAFE client...
Trying to fetch the bootstrap peers from https://sn-testnet.s3.eu-west-2.amazonaws.com/network-contacts
Connecting to the network with 47 peers:
"/home/safe/safe_upload_stressor/temp" will be made public and linkable
Starting to chunk "/home/safe/safe_upload_stressor/temp" now.
Uploading 41 chunks
**************************************
*          Uploaded Files            *
**************************************
"1MB_4" a24eebfe068c85c80d408a3702a03986a677e7321f02ff031ee09cba8daa6b41
"1MB_10" 5f7aadeca72c4cc819dc2687244dc3f626de09196d5e1abd70f318437d0eef72
"1MB_7" b01d3ff2d0f4478dd1e1980ff44e773e514e7fc39045d680f3196241f227bf06
"1MB_3" c2993abc2f7a82d02d6666041966fd0726e8455c61234d44cd680818d84ae6c5
"1MB_6" da63df59c6e4eca9620de8a202f7fe9019158aa5197f3703f9f1d0a7ccc3fdcf
"1MB_9" e78bd40b980fa9e8342908d906dd49777437bce951e64afcab4737e3d3a1124f
"1MB_5" 389619fd60a666b90d9c1a17a280922e0fcec899b46bbb60c8b89adde5f94ac5
"1MB_2" 7cbe43c3e528d975321005182da849faec59d32e1c026043de141cf858946add
"1MB_8" eb98cc445b5feb44f8da28552bfef42c8d87bf7bc25c08543a6d0221077312d4
"1MB_1" 69101715e1a803b93f1fb32a00c9597b9c1d0f2d9bba462a8eba50dc40624abd
"md5sums_10x1MB" 2ae8ede7218f8afe7703983e80014d9a4e4985926379b3c29cc5aef096bb5754
Among 41 chunks, found 0 already existed in network, uploaded the leftover 41 chunks in 1 minutes 3 seconds
**************************************
*          Payment Details           *
**************************************
Made payment of 0.000007260 for 41 chunks
Made payment of 0.000001264 for royalties fees
New wallet balance: 9.999991347
--

Run 2
Mon  8 Jan 00:10:20 UTC 2024

Logging to directory: "/home/safe/.local/share/safe/client/logs/log_2024-01-08_00-10-20"
Built with git version: ee245b6 / main / ee245b6
Instantiating a SAFE client...
Trying to fetch the bootstrap peers from https://sn-testnet.s3.eu-west-2.amazonaws.com/network-contacts
Connecting to the network with 47 peers:
"/home/safe/safe_upload_stressor/temp" will be made public and linkable
Starting to chunk "/home/safe/safe_upload_stressor/temp" now.
Starting to chunk "/home/safe/safe_upload_stressor/temp" now.
Files upload attempted previously, verifying 41 chunks
All files were already uploaded and verified
**************************************
*          Uploaded Files            *
**************************************
"1MB_7" 96732b439d8241dbf389d570b97c99736bb1c9785e82a0778a3c0293bd22c3cc
"1MB_1" 58c248bdf17a9d8922c72199b55f6b7f41db740558121256a367be9db8af7602
"1MB_10" 7a56bae484fe07065d3eec53b979b749cb99ef5e1c96b8dacbc7b3002db2a735
"1MB_5" 24d4dbd8876d8103c95a3cc414a1db6beb1156dcbff5c83543e4927221c43e58
"1MB_6" d523ab2077d2fe1edbeab68c718615ecdb87237b839c6519cb69b97aca54e998
"1MB_8" dac8642ddeff1e6ee09b5e27950dc64f305ccaaf0a1847b9a93b8dfcd36c093f
"1MB_3" 684c7a7a8353c7960baa861256db6c747a6721cd7744295201e884d7571597ad
"1MB_2" 44ddcd76975e2e5e60226f1ad20ae14a1c8d6ec8f84762235df9d11845d92b3f
"1MB_4" 1b7f63cc937deceb0ccb4e4d704ca2515a27bcc994b0030dd11dcbe0c794215b
"1MB_9" 9c52c8bfccd4e68f9f4af39bef4525355ef0a56cd6aa4265f35664e27576c95f
"md5sums_10x1MB" c6ec406edf1df745dc860ff1d7e0e1eeb7bb15752870344cba1f3bcdad27ab37
--

EDIT
Removing the ‘-p’ didn’t help. The files in Run 2 are still reported as being the same as in Run 1.

–

EDIT
It’s the name of the file. It seems that with or without the ‘-p’ option if a file has the same name as one previously uploaded it will think it’s the same file again. This looks like a bug to me because the file is verifiably different and has produced a different address. I added something to make each file unique and the script is working correctly now. But I think there is something that needs fixing.

Mon  8 Jan 08:26:24 UTC 2024

Run 1
Mon  8 Jan 08:26:24 UTC 2024

Logging to directory: "/home/safe/.local/share/safe/client/logs/log_2024-01-08_08-26-24"
Built with git version: ee245b6 / main / ee245b6
Instantiating a SAFE client...
Trying to fetch the bootstrap peers from https://sn-testnet.s3.eu-west-2.amazonaws.com/network-contacts
Connecting to the network with 47 peers:
"/home/safe/safe_upload_stressor/temp" will be made public and linkable
Starting to chunk "/home/safe/safe_upload_stressor/temp" now.
Uploading 41 chunks
Retrying failed chunks 4, attempt 0/3...
**************************************
*          Uploaded Files            *
**************************************
"1_1MB_1" 67bcc22d1fec79c148ba64cf84fccfef22d5b582483d0d69713fa3bf44fd5c87
"1_1MB_4" fa673eebaf1c54e5e17c372d9519500c4552a0f637571ebdfa87c21c01043494
"1_1MB_5" 7fa2eabe0bc4ac502116a1a094b3a8db34ebe18c58c39d8892ee1945bcd69833
"1_1MB_8" 456034a1fbaac1e1e36cd498801e382a4e4f0106c6332db3a8c4efdda6213718
"1_1MB_3" 0c9c21923f77dd0ee41f55a271aa658d7e0ef767887b5e138af6d997ac126eb0
"1_1MB_10" 4cd6e48e4ee04b254562698d6ef69507a5fac166ede912591156beb003f263cf
"1_1MB_2" ee937b0a4253ec783c9a11c5bbcef0a61c236c8eba92d5d61f47847b0c559617
"1_1MB_6" eecde92c60e117c86ca21178723998e7f9de6c7270061476ec77bc4ed1bf1198
"1_md5sums_10x1MB" 451425ef0e66841c8a49d6114dd62778aeb90dfb024d22a9ef9778b74904f77f
"1_1MB_7" e653130f6b8c6300491c802ce595c3e054388ff590d9e66501ab3ff0f8847319
"1_1MB_9" 6c6510e1c665169fede7ff22a963dc0de056d9309b1f6227d1814e1f975059cd
Among 41 chunks, found 0 already existed in network, uploaded the leftover 41 chunks in 1 minutes 39 seconds
**************************************
*          Payment Details           *
**************************************
Made payment of 0.000007529 for 41 chunks
Made payment of 0.000001309 for royalties fees
New wallet balance: 9.999956330
--

Run 2
Mon  8 Jan 08:28:06 UTC 2024

Logging to directory: "/home/safe/.local/share/safe/client/logs/log_2024-01-08_08-28-06"
Built with git version: ee245b6 / main / ee245b6
Instantiating a SAFE client...
Trying to fetch the bootstrap peers from https://sn-testnet.s3.eu-west-2.amazonaws.com/network-contacts
Connecting to the network with 47 peers:
"/home/safe/safe_upload_stressor/temp" will be made public and linkable
Starting to chunk "/home/safe/safe_upload_stressor/temp" now.
Uploading 41 chunks
**************************************
*          Uploaded Files            *
**************************************
"2_1MB_8" aab7028f03fa9800b243e99b564440f9b1233c467448e65a4f175afe171ffde8
"2_1MB_3" e0989889124b799f55080f08d5922d08ee3a34daf3d6df5aea273f4b80c5f8f6
"2_1MB_10" dd247e7fa8b531189e603049e20778debf5accb328e97982e23cec0f0914de6f
"2_1MB_2" a1d1c902686665ae35aa2e0112c2fce7e116815af8e89a8d4bb8227e6de6e293
"2_1MB_5" e4839023f87b6ac1fa7451e276701ef926cda02ef5d99d95e48675ccf5359ea2
"2_1MB_1" a4644844cf3befb787759d2575db19a2842ddb0530777c014ba7ca5bd944d5cf
"2_md5sums_10x1MB" 999522bd62a6a5a8c96177d67a693db027474aec052c2b97fe47e867a0b108bf
"2_1MB_4" 68de6b6c8b5af59f3a513384e421e7777aa33ff0e9655df0a63af53e5ec01c13
"2_1MB_9" d27998b5f2e749e9d8d9fa13829f7607f7bc787d80cf4985a29ca22d47f2f1ee
"2_1MB_6" cc21e1106665cfcec792817d09f6fab47ec400812eeb9b9a32f38dfe4cff28d9
"2_1MB_7" 3a671967301b6b40d1354fabdd567eab7db201315407aa0712ad4224a3a91bbf
Among 41 chunks, found 0 already existed in network, uploaded the leftover 41 chunks in 1 minutes 7 seconds
**************************************
*          Payment Details           *
**************************************
Made payment of 0.000008238 for 41 chunks
Made payment of 0.000001430 for royalties fees
New wallet balance: 9.999946662
--

[Toivo]

@storage_guy

The only change I made to the script was to remove the ‘-c’ from the safe files upload command and add the ‘-p’ to make the files public.

I think -c flag is deprecated, or shouldn’t do anything for a few testnets already? Maybe it messes something?

Also, if anyone wants some tokens, I have many to share.

[storage_guy]

Yes, I removed the ‘-c’ because otherwise it just errors.

[raulillo]

root@e6cb40ba73ec:~# RUST_BACKTRACE=1 safe files upload -p openSUSE-Leap-15.5-DVD-x86_64-Build491.1-Media.iso 
Logging to directory: "/root/.local/share/safe/client/logs/log_2024-01-08_09-04-37"
Built with git version: ba2bb2b / main / ba2bb2b
Instantiating a SAFE client...
Trying to fetch the bootstrap peers from https://sn-testnet.s3.eu-west-2.amazonaws.com/network-contacts
Connecting to the network with 47 peers:
🔗 Connected to the Network                                                                                                                                                                                                            "openSUSE-Leap-15.5-DVD-x86_64-Build491.1-Media.iso" will be made public and linkable
Starting to chunk "openSUSE-Leap-15.5-DVD-x86_64-Build491.1-Media.iso" now.
Uploading 8058 chunks
Error: 
   0: Failed to upload chunk batch: Transfer Error Failed to send tokens due to Wallet has pre-unconfirmed tx, resent them and try later..

Location:
   sn_cli/src/subcommands/files/mod.rs:323

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                ⋮ 3 frames hidden ⋮                               
   4: safe::subcommands::files::upload_files::{{closure}}::ha9973f8b4ca2bf9f
      at <unknown source file>:<unknown line>
   5: safe::main::{{closure}}::h8bde188d10ac8c06
      at <unknown source file>:<unknown line>
   6: tokio::runtime::park::CachedParkThread::block_on::ha28e999aa1d6ba8c
      at <unknown source file>:<unknown line>
   7: tokio::runtime::context::runtime::enter_runtime::h8c4fb93bca637bea
      at <unknown source file>:<unknown line>
   8: tokio::runtime::runtime::Runtime::block_on::hf9ade62586aeaed2
      at <unknown source file>:<unknown line>
   9: safe::main::h64882c8b59f929cd
      at <unknown source file>:<unknown line>
  10: std::sys_common::backtrace::__rust_begin_short_backtrace::hb9f45c9e026884db
      at <unknown source file>:<unknown line>
  11: std::rt::lang_start::{{closure}}::ha36699d8aa4a5e1d
      at <unknown source file>:<unknown line>
  12: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h7d795fa8fec4facf
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284
  13: std::panicking::try::do_call::h5004c50ed4d8a610
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552
  14: std::panicking::try::hc97a06a6df5f8eaa
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516
  15: std::panic::catch_unwind::h32f9e1465b582567
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142
  16: std::rt::lang_start_internal::{{closure}}::h4173ab3d27e5e8a8
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148
  17: std::panicking::try::do_call::h207e219e0307211b
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552
  18: std::panicking::try::h5498160338cd80af
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516
  19: std::panic::catch_unwind::h8c320d4c7cb9f0db
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142
  20: std::rt::lang_start_internal::hd1ee6171f586dc89
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148
  21: std::rt::lang_start::h0347af31ee9df289
      at <unknown source file>:<unknown line>
  22: main<unknown>
      at <unknown source file>:<unknown line>

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
Run with RUST_BACKTRACE=full to include source snippets.

Not much more info with the full backtrace:

root@e6cb40ba73ec:~# RUST_BACKTRACE=full safe files upload -p openSUSE-Leap-15.5-DVD-x86_64-Build491.1-Media.iso 
Logging to directory: "/root/.local/share/safe/client/logs/log_2024-01-08_10-05-59"
Built with git version: ba2bb2b / main / ba2bb2b
Instantiating a SAFE client...
Trying to fetch the bootstrap peers from https://sn-testnet.s3.eu-west-2.amazonaws.com/network-contacts
Connecting to the network with 47 peers:
🔗 Connected to the Network                                                                                                                                                                                                            "openSUSE-Leap-15.5-DVD-x86_64-Build491.1-Media.iso" will be made public and linkable
Starting to chunk "openSUSE-Leap-15.5-DVD-x86_64-Build491.1-Media.iso" now.
Uploading 8058 chunks
Error: 
   0: Failed to upload chunk batch: Transfer Error Failed to send tokens due to Wallet has pre-unconfirmed tx, resent them and try later..

Location:
   sn_cli/src/subcommands/files/mod.rs:323

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                ⋮ 3 frames hidden ⋮                               
   4: safe::subcommands::files::upload_files::{{closure}}::ha9973f8b4ca2bf9f
      at <unknown source file>:<unknown line>
   5: safe::main::{{closure}}::h8bde188d10ac8c06
      at <unknown source file>:<unknown line>
   6: tokio::runtime::park::CachedParkThread::block_on::ha28e999aa1d6ba8c
      at <unknown source file>:<unknown line>
   7: tokio::runtime::context::runtime::enter_runtime::h8c4fb93bca637bea
      at <unknown source file>:<unknown line>
   8: tokio::runtime::runtime::Runtime::block_on::hf9ade62586aeaed2
      at <unknown source file>:<unknown line>
   9: safe::main::h64882c8b59f929cd
      at <unknown source file>:<unknown line>
  10: std::sys_common::backtrace::__rust_begin_short_backtrace::hb9f45c9e026884db
      at <unknown source file>:<unknown line>
  11: std::rt::lang_start::{{closure}}::ha36699d8aa4a5e1d
      at <unknown source file>:<unknown line>
  12: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h7d795fa8fec4facf
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284
  13: std::panicking::try::do_call::h5004c50ed4d8a610
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552
  14: std::panicking::try::hc97a06a6df5f8eaa
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516
  15: std::panic::catch_unwind::h32f9e1465b582567
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142
  16: std::rt::lang_start_internal::{{closure}}::h4173ab3d27e5e8a8
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148
  17: std::panicking::try::do_call::h207e219e0307211b
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552
  18: std::panicking::try::h5498160338cd80af
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516
  19: std::panic::catch_unwind::h8c320d4c7cb9f0db
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142
  20: std::rt::lang_start_internal::hd1ee6171f586dc89
      at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148
  21: std::rt::lang_start::h0347af31ee9df289
      at <unknown source file>:<unknown line>
  22: main<unknown>
      at <unknown source file>:<unknown line>

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.

[joshuef]

@happybeing

My point is it seems an imposition that apps treat their storage backend like a file system and don’t have access to a more basic API (which is secure by default).

Yeh, we need the secure by default and padding for small data, I think

@happybeing

I think of the personal Safe as an application and the question is whether to limit other applications to using an API which sits on top of that, or to allow them to use the core features directly.

@happybeing

And the contentious point is whether those packets should, by default, be encrypted regardless of size.

It is confusing to say that storing a packet above a certain size using the self encryption API results in encryption, but below it does not.

Agreed.

@dirvine

I think @joshuef intends this happens anyway, I mean encrypt what they store on disk. We can now as we have small nodes who on reboot just restart without causing tons of churn issues. So let’s see what he says, but I think this is a definite, unless we come up with an even simpler mechanism.

Aye, I think nods should be encrypting things yeh.

It’s true if that’s the case, you may upload a small file and feel safer. But it does still go against the fact that we say “all data is encrypted client side”… because it’s then not…

So we should do both. Any node could choose to subvert node-side encryption… we’d never know or be able to know (i think!?). Sos best that everyone is careful IMO.

@dirvine

Default

• On store, clients encrypt the Entry with the Register owners PubKey
• On read clients must have the SecretKey to decrypt.

So really simple and neat.

Public Data
The clients just miss out the step above.

Aye, that makes sense.

If we have that + padded chunks by default (so all data client encrypted) + node encryption before disk, I think we’re in a solid position (unless I missed something from the above?)

[dirvine]

@joshuef

padded chunks

We should keep in mind though. That if we use folders as a MUST. Then small files end up in the data_map and that data map ends up in a register (folder) and that is encrypted (per the above). [what happens is the link to the chunk that holds the data map is encrypted and that is then a normal chunk (hash content == name), the Entry is the name of that chunk and it’s also encrypted as per the above] So then all data in a file structure is encrypted and no need to pad.

So small files in chunks by themselves are not possible in our API and would mean nobody subverted it.

[joshuef]

Talking through this more with @dirvine just now:

It seems to the simplest way is for

1. low level client apis to just reject data they cannot encrypt (unless it’s being made public!)
2. high level APIs to perform encryption/managment/integration with registers and their encryption.

High level here == the CLI. Low level is the sn_client crate.

That’d get us in a predictable + reliable place w/r/t encryption. (With the vast majority of current users going through the high level APIs + so not noticing any real change beyond all data being encrypted there.