Its nearly later
1 Like
1 Like
Can’t get that to work with complex names/folder structures.
How would you cat that way one of these files:
safe://hyryyrysg7td9ozp5cmtn3u64qezgn371gcb1uho1b8f76rra8apuy8azgynra?v=h6us3m8ydkrkynycf35o9kdt7bp3jikoozzjdxek7p76f499cj4yo
there is a whitespace issue for sure? can the team correct the whitespace behavior?
e.g.
safe files get "safe://hyryyrysg7td9ozp5cmtn3u64qezgn371gcb1uho1b8f76rra8apuy8azgynra?v=h6us3m8ydkrkynycf35o9kdt7bp3jikoozzjdxek7p76f499cj4yo/_Made In The Shade_ (06.06.1975)/02. Tumbling Dice.mp3"
Error:
0: UrlError: InvalidInput: The URL cannot contain whitespace
1: InvalidInput: The URL cannot contain whitespace
Location:
sn_cli/src/subcommands/files_get.rs:329
You just need to make sure it’s URL-encoded when you have whitespaces or other special characters in the path, e.g. a whitespace can be URL-encoded as %20, so you can safe cat the content of the “You Got Me Rocking” folder with:
$ safe cat safe://hyryyrysg7td9ozp5cmtn3u64qezgn371gcb1uho1b8f76rra8apuy8azgynra/You%20Got%20Me%20Rocking
Files of FilesContainer (version h6us3m8ydkrkynycf35o9kdt7bp3jikoozzjdxek7p76f499cj4yo) at "safe://hyryyrysg7td9ozp5cmtn3u64qezgn371gcb1uho1b8f76rra8apuy8azgynra/You%20Got%20Me%20Rocking":
+-------------------------------------------------+-----------+---------+------------+------------+--------------------------------------------------------------------+
| Name | Type | Size | Created | Modified | Link |
|-------------------------------------------------+-----------+---------+------------+------------+--------------------------------------------------------------------|
| 01. You Got Me Rocking.m4a | audio/m4a | 3547267 | 1671589172 | 1671589172 | safe://hyryyyyxr4pctg7gcfr17dze45mpxpmq7chsb84yenpb1dwcp4ceusksb1w |
|-------------------------------------------------+-----------+---------+------------+------------+--------------------------------------------------------------------|
| 02. Jump On Top Of Me.m4a | audio/m4a | 4222942 | 1671589172 | 1671589172 | safe://hyryyyyewsz65acozjzfo5d4ree3e3geaghwobc5qqx4e89kfmytd3hg97y |
|-------------------------------------------------+-----------+---------+------------+------------+--------------------------------------------------------------------|
| 03. You Got Me Rocking [Perfecto Mix].m4a | audio/m4a | 5007725 | 1671589172 | 1671589172 | safe://hyryyyye89g5aw6j1k9chu96t8wejdxtg7iqr85qkjbzrjkihtum113ucjy |
|-------------------------------------------------+-----------+---------+------------+------------+--------------------------------------------------------------------|
| 04. You Got Me Rocking [Sexy Disco Dub Mix].m4a | audio/m4a | 6170983 | 1671589172 | 1671589172 | safe://hyryyyymdqbbbhtp3pi3ktpd5kc15f9ph19wgb7m8r5wmb8wfujbec9qq6o |
+-------------------------------------------------+-----------+---------+------------+------------+--------------------------------------------------------------------+
6 Likes
Here is container with 100 x 1MB files together with a file containing their md5sums.
safe://hyryyryue1nmb91gbzk4qnfn9syc4be3dw9987f6hc5uz3tp37z1kjf5rkhnra?v=hhfxy9hcxe97gwyx3gdchhyrso4wz74s6xsnqsybg4wn9uf4a9ijy
If you want to download the whole lot just do:-
safe files get safe://hyryyryue1nmb91gbzk4qnfn9syc4be3dw9987f6hc5uz3tp37z1kjf5rkhnra?v=hhfxy9hcxe97gwyx3gdchhyrso4wz74s6xsnqsybg4wn9uf4a9ijy
Then if you want to check the validity of the download you can do:-
Generate a file containing the md5sums of the downloaded files:-
md5sum 1MB* >> md5sums_downloaded
Then see if the md5sums are the same between what I uploaded and what you downloaded:-
diff md5sums md5sums_downloaded
The command prompt should return after printing nothing. If there are any differences in the md5sums for a file you’ll see output with the name of the file.
2 Likes
Sounds like a stable test net so far 
well done all
5 Likes
Like this
safe cat safe://hyryyrysg7td9ozp5cmtn3u64qezgn371gcb1uho1b8f76rra8apuy8azgynra/The%20Rolling%20Stones/05.%20Out%20Of%20Our%20Heads%20-%201965b/10%20Play%20With%20Fire.mp3 > 10.mp3
was waiting for someone to jinx it 
so far my 40gb upload has been going since inception and is still churning on im hoping it will finish uploading before we reach capacity on this test net.
but I am curious what will happen if after doing a safe files put ./ -r on a large folder
if the network reaches capacity before all the files are uploaded.
will it give me a container of what its got uploaded so far or will it fail and I wont get anything in my container ?
so I am looking forward to the end results
6 Likes
Time between this and previous update is 9 hours. About 6GB was loaded to the typical node during that time (12 to 18) so the pace seems the same 0.6GB/hour/adult.
Why just that? Are we the bottleneck, not giving more? Or is it the network? Or Digital Ocean?
3 Likes
holey moley as they say !!
I was wondering if my home internet connection was my bottle neck so I tried starting the same upload from my arm based vps and the results are mental !!!
home based Linux box i5 8Gb ram is maxing at 1.94 MiB/s upload and the cloud based arm box is maxing out at 63.9 MiB/s upload
I love Scotland because all the best things are invented in Scotland but its just a shame we have the original telephone cables installed by Alexander Graham Bell himself in Glasgow
14 Likes
Here is a container with 1000 x 1MB files:-safe://hyryyryts3xck4aseu1okgfurtp537fxx17qtptjkbim3mb8b6upd1kjj5rnra?v=h9fkq8tn8hxf56cis9nk948timempobo9zz6tkq46r9w5b616ba4y
Same thing as before. To download the whole lot do:-
safe files get safe://hyryyryts3xck4aseu1okgfurtp537fxx17qtptjkbim3mb8b6upd1kjj5rnra?v=h9fkq8tn8hxf56cis9nk948timempobo9zz6tkq46r9w5b616ba4y
Then if you want to check the validity of the download you can do:-
Generate a file containing the md5sums of the downloaded files:-
md5sum 1MB* >> md5sums_downloaded
Then see if the md5sums are the same between what I uploaded and what you downloaded:-
diff md5sums md5sums_downloaded
The command prompt should return after printing nothing. If there are any differences in the md5sums for a file you’ll see output with the name of the file.
4 Likes
Been uploading random 10MB files (pre-generated) for the past 3 hours via a pwsh script in an Alpine LXC with debug logging enabled. It has uploaded ~695 files (6+ GB) out of 5120 files (50GB total). No errors so far.
Seems the throughput and time it takes for the PUT has changed since 23:05 UTC by ~2x to 4x. This is a single LXC on a physical host without any other processes running. I know I am not bandwidth capped based on the current peak throughput, and current average throughput seen from this LXC.
Maybe something has changed in my environment that I am unaware off, or at the nodes’ server environment.
5 Likes
I have a fiberoptic dedicated line. Uploads downloads seem at times instantaneous with 10mb files. So the network will need to be resilient with many speeds I reckon.
2 Likes
Here is a container with 500 x 4MB files:-
safe://hyryyrys4a6c1mcedgbakubpmhzbs3s5r7g9gqysmoh85ypkw47ifi1638wnra?v=h6pe6iwbnezfz7c1xdz658fmn3dbjxnkf6wmf946x51r1kyesr66y
plus a file called ‘md5sums’ which contains the checksums of all the files.
To download the whole folder do:-
safe files get safe://hyryyrys4a6c1mcedgbakubpmhzbs3s5r7g9gqysmoh85ypkw47ifi1638wnra?v=h6pe6iwbnezfz7c1xdz658fmn3dbjxnkf6wmf946x51r1kyesr66y
Then if you want to check the validity of the download you can do:-
Generate a file containing the md5sums of the downloaded files:-
md5sum 4MB* >> m5sums_downloaded
Then see if the md5sums are the same between what I uploaded and what you downloaded:-
diff md5sums md5sums_downloaded
The command prompt should return after printing nothing. If there are any differences in the md5sums for a file you’ll see output with the name of the file.
8 Likes
using this script that I modified from Tyler
#!/bin/sh
# uncomment the verify_put call below to confirm uploaded data has no errors (untested & probably not necessary).
msg() {
printf '%s\n' "${0##*/}>> ${1}" | tee -a "test.log"
}
verify_put() {
safeContainer=$(grep -o ' | safe://[^ ]*' "${safeOutput}" | grep -o 'safe://[^ ]*')
msg "${safeContainer}"
safe cat ${3} >${1}
[ "$(md5sum ${1})" = "${2}" ] && msg "${3} has good checksum" || "${3} has BAD checksum"
rm ${1}
}
while true
do
filesize=$(shuf -i 10000-9000000 -n 1)
filename=$(cat /dev/urandom | tr -cd 'a-f0-9' | head -c 16)
head -c ${filesize} </dev/urandom >${filename}
checksum=$(md5sum ${filename})
msg "filename:${filename} -- ${filesize}"
errorBool=true
while $errorBool
do
safeOutput="$(safe files put ${filename})"
msg "${safeOutput}"
if echo $safeOutput | grep -q Error; then
echo "Got an error putting: retrying"
else
echo "No error so deleting the file"
errorBool=false
fi
done
rm ${filename}
#verify_put $filename $checksum $safeOutput &
#sleep $(shuf -i 1-10 -n 1)s
done
exit 0
I can run it up to 6 in parallel with no errors
if I run the script more than 6 in parallel I get errors in all instances
I am on Windows 11 using WSL 2 and Debian 11
the error I am getting is the following with different address in the end
<ClientError: Timeout after 90s when awaiting command ACK from Elders for data address 2cf5fe..>
ok so now 6 parallel uploads and 9 parallel downloads, everything seems smooth. will wait for the downloads to finish and will check the hash to see if it is indeed downloading without errors
scratch that. 6 uploads and 9 downloads lead to all uploads to error out. trying 6 up 6 down now to see if it will fix it
6 up 6 down and some up are erroring out. will try 4 up 4 down
4 up 4 down seems to be the sweetspot, no errors.
I guess this should be filed as a bug as people would like to do uploads/downloads in parallel! maybe a test/bug hunt for a future testnet!
3 Likes
around 24minutes 4 in parallel downloads of the above, will check hash now!
I get this:
diff: md5sums_downloaded: No such file or directory
what is going on @storage_guy ?
oh you got a type in your instructions m5 instead of md5
aaaand all hashes comfirmed correct!!! GG!!!
3 Likes
As josh pointed out above, the nodes will fill evenly only if the node addresses themselves are evenly distributed across the xor space. This points to a need for a simple optimization to randomly set the xor address of a new node to something that maximizes uniform spacing. Poisson Disk Sampling (PDS) is a well known technique that achieves this. So nothing to worry about (yet) imo. The issue can be confirmed by manually specifying the xor addresses to be exactly uniformly distributed with equal spacing across the xor space and see that all nodes fill up equally with negligible variance.
A refresher on PDS:
Fast Poisson disk sampling in arbitrary dimensions
https://dl.acm.org/doi/10.1145/1278780.1278807
11 Likes
You can use random generator to create 9~10 million characters per textfile which would be around 10mb depending on the encoding and characters used.
1 Like
I think we got a crashed network? I got in all my uploads error with elder 7 expected 5 received!
1 Like