I’m worried about the loss of functionality that will happen with ‘large’ Chunk sizes of 2MB+. The currently plan of 4MB seems excessively large and 2MB not much better.
I realise it is possible to still upload much less but it will become much less attractive and maybe completely uneconomic. The price will be driven by the maximum chunk size because the bulk of data on the network will be in files or other data structures that are larger than whatever the chunk size is so that is the unit that storage will be purchased in.
I also realise that because of the use of a ERC20 L2 token to do the payments for storing data (hopefully not forever) it is more attractive to have a larger Chunk size.
You can have Object Storage / Content Addressed systems that can handle variable sized objects. Although it is much more frequent to have systems where a single size is set. But that is because of the way Erasure Coding works when fragments are spread across disks or nodes. Autonomi is a completely different architecture where there is no defined size of the network (or even knowledge of the size of the network) or physical tolerance zones based on disks, trays of disks, racks or sites that have to be taken into account.
At the moment I believe the network is using straightforward replication of 5 copied rather than EC and presumably that is coming later?
How about:-
512KB objects
2MB objects
4MB objects
… Other sizes in the future? 64MB is common is attractive for large uploads and downloads.
Objects up to 512KB = replication 5 ways because it’s hardly worth doing EC for that size. 3 chunks to still make the encryption work.
Objects larger than that would use whatever number of chunks needed to make the EC economic, resilient and performant.
If it’s not possible or too late to add variable size objects just now could it be added later? Or leave the door open to it somehow? Or does the chunk size have to be baked in for all time at the beginning?
I realise this question crosses with some others going on about encryption and the use of the ERC20 L2 token but it seems worthy of a separate question and I don’t think it’s been asked before. Sorry if that’s not the case!