you mean upload more data? It doesnt matter what type of data?
1 Like
Well keep adding data, preferably verifiable, images, music etc you dont mind sharing.
What I meant was we need to see more nodes and their reported capacities before we can make definitive statements about what is an actual outlier.
All the more reason that they should have little to no variance.
Anyway, I think a control test could be done just running all nodes and clients on a single machine, with client(s) on same machine or subnet. In other words, set up “perfect” conditions, and see if it performs within theoretical variance expectations.
3 Likes
I put to upload folder with 1155 files (8.72GB) to upload via cmd but I see this (nothing), its normal because it loading it?
also If some files have name XXXX - YYYY.mp3 it could be also uploaded? Because last time I tried it not work.
2 Likes
Yes - we eagerly await an upload progress bar for the put process
I don’t see why not - assuming the filenames are unique
1 Like
Again this is something we should be able to test in a controlled manner.
Perhaps we agree that each participant runs safe node run-baby-fleming --nodes=30 and then locally fires large numbers of <10Mb files at it for a few hours.
Then report results in here.
1 Like
Is it possible to get a single file from a container?
2 Likes
yes - but you have to ask nicely 
3 Likes
what upload speed I should wait, ± same as if I measure by some internet speed? If I upload 1150file of music how do u check if its corrupted or isnt?
I was going to but got distracted…
then I failed - but I KNOW it can be done
willie@gagarin:~$ safe files get "safe://hyryyryzyrzbym44gm75rr8yj1p6rnybtibnwgrbmsphy7hzaqs8njbaxhwnra/fgfs/Aircraft/Aichi-M6A/Models/Interior/Panel/Instruments/alt/alt.png" .
Error:
0: ContentError: Failed to obtain file map for path: /fgfs/Aircraft/Aichi-M6A/Models/Interior/Panel/Instruments/alt/alt.png, on FileContainer at: safe://hyryyryzyrzbym44gm75rr8yj1p6rnybtibnwgrbmsphy7hzaqs8njbaxhwnra/fgfs/Aircraft/Aichi-M6A/Models/Interior/Panel/Instruments/alt/alt.png, because: ContentError("no data found for path: /fgfs/Aircraft/Aichi-M6A/Models/Interior/Panel/Instruments/alt/alt.png/")
More later
1 Like
I think random doesn’t necessarily mean an even spread as someone noted above… I doubt it’s cloud-data center related if not just xorspace oddness?
Even if data is random, the node locations in xorspace are random, but finite… There’s only 18 of them (adults storing data), spread out over the whole space. And I doubt they are spread evenly (what would be the probability of that?) which would be a prerequisite for an even spread of data over these nodes, no?
@mav has previously done some work showing how xorspace doesn’t actually spread as we might expect
Overall, at the moment it’s something interesting to keep an eye on perhaps. More testing with more adults would be interesting to see if there is any statistically significant skew over many testnets.
14 Likes
That makes perfect sense! Looks like mystery solved to me
6 Likes
Would putting a stream (open-ended file) be different than a file?.. or is that a different case than put and not possible just yet?
Is data still unencrypted?.. is the delay uploading partly the fragmenting of files; I wonder the upload progress could do with some suggestion of what activity is local and what is the upload with how many fragments have been completely put.
1 Like
Sorry @joshuef that link doesnt work
ups. too many brackets. updated!
1 Like
If you safe cat the container, you’ll get a list of the files with relative safe:// addresses
3 Likes
cat the container
safe cat safe://hyryyryto6j66emsodsmwjqqrinmyr3ukn93oi9yhof51ojn4goir8h5b7anra?v=hyskyxto7sudu136uu3afr9sgzrfxj1rjtrap7mez11wpm67tqm4y
then you will see each file and it’s corresponding url
/.git/hooks/pre-rebase.sample | Raw | 4898 | 1671553979 | 1671553979 | safe://hyryyyyk59wufr4iw6penxxwc5zk4uwpzwct8y1pphbhs1yp3m9cytr95zr |
|----------------------------------------------------------------------+----------------------------+---------+------------+------------+--------------------------------------------------------------------|
| /.git/hooks/pre-receive.sample | Raw | 544 | 1671553979 | 1671553979 | safe://hyryyyym8crmnwgqwybik43985c8mhtejt3ueogs9p4i964fatmb3kxgt6a |
|----------------------------------------------------------------------+----------------------------+---------+------------+------------+--------------------------------------------------------------------|
| /.git/hooks/prepare-commit-msg.sample | Raw | 1492 | 1671553979 | 1671553979 | safe://hyryyyyko9jkdxsb3ddy8ripc18dd74pyam4fydbr7ji4izoaz91denucuh |
|----------------------------------------------------------------------+----------------------------+---------+------------+------------+--------------------------------------------------------------------|
| /.git/hooks/push-to-checkout.sample | Raw | 2783 | 1671553979 | 1671553979 | safe://hyryyyyce3kxnk88ejm96pa58pcmkbjzzfr8dkhx9cchyzypnongwhwxq1r |
|----------------------------------------------------------------------+----------------------------+---------+------------+------------+--------------------------------------------------------------------|
then you can cat whichever single file you like.
@stout77 ninja’d me
3 Likes
Uniform distribution of data across nodes is normally considered desirable, or even a requirement for sharded systems. hash functions are selected based on how uniformly they distribute data, etc.
Reading between the lines, I am guessing that other features were deemed more important for Safe?
Basically, I’m trying to figure out if this variance is “by design” and has been considered or is a surprise/bug/problem. And how it will affect section splits etc, if some nodes are filling up faster than others in every section. perhaps a q for @dirvine…
6 Likes