The 6th on the list, yes corresponds to 5.jpg
2 Likes
Easy python script to test GET performance. Uses the current XOR URLS stated above stored in array.
Attempt for XOR `safe://hygoygym7tsj5hhyyykd1aqpw3djxea6om6xku568ahm7hy7gfn6q5gy7xr` stored as 0.jpg [SUCCESSFULL]
Total Time run in sec: 1.0012853145599365 Total Size in MB: 4.554655075073242 Avg Download Speed: 4.549 MB/s
Attempt for XOR `safe://hygoygym19bdbzg6jakfzkmacxurpkbuphgsxi6af3aecugfqjfaq66shww` stored as 1.jpg [SUCCESSFULL]
Total Time run in sec: 1.0016148090362549 Total Size in MB: 4.001064300537109 Avg Download Speed: 3.995 MB/s
Attempt for XOR `safe://hygoygyq1xiofs8kx11jwkcguzuocd1xa5qp767bjjz39yrh44eo1mjkjch` stored as 2.jpg [SUCCESSFULL]
Total Time run in sec: 1.0013666152954102 Total Size in MB: 1.1512479782104492 Avg Download Speed: 1.15 MB/s
Attempt for XOR `safe://hygoygypnrqirzo7r5w464mnda8s6sdpatyrco5pda1w8nh5appa3ycb98h` stored as 3.jpg [SUCCESSFULL]
Total Time run in sec: 1.0016982555389404 Total Size in MB: 3.801058769226074 Avg Download Speed: 3.795 MB/s
Attempt for XOR `safe://hygoygykajkeropyam6yihj9exdhn5o4b1pbjbeet4bazqo8m47ga3pw8mo` stored as 4.jpg [SUCCESSFULL]
Total Time run in sec: 1.0013902187347412 Total Size in MB: 3.083775520324707 Avg Download Speed: 3.079 MB/s
Failed to GET file / download!
Attempt for XOR `safe://hy8ayqyjy9obx5kr5g9f1r95hshhfswt7yqh6cy9t34sap7zqid4ox7ucuo` stored as 5.jpg [FAILED]
Total Time run in sec: 1.001300573348999 Total Size in MB: 0.0 Avg Download Speed: 0.0 MB/s
Attempt for XOR `safe://hygoygyp8c1wy7ijrpnesz9cj7enf4ms9oxhmp5m4r4yw4pjcemgywekf9r` stored as 6.jpg [SUCCESSFULL]
Total Time run in sec: 1.0010404586791992 Total Size in MB: 2.2845458984375 Avg Download Speed: 2.282 MB/s
Attempt for XOR `safe://hygoygye13iznm1yxzpxp9seh6j8weh84c5ib3fq9mxozz4h6hb95o3uduo` stored as 7.jpg [SUCCESSFULL]
Total Time run in sec: 1.0012643337249756 Total Size in MB: 0.9072551727294922 Avg Download Speed: 0.906 MB/s
Attempt for XOR `safe://hygoygycc11b4pmrcb97ro6ro3wxh3cfrgi7ctbiwyya8mkktq1j35ysmzh` stored as 8.jpg [SUCCESSFULL]
Total Time run in sec: 1.0010101795196533 Total Size in MB: 1.4225358963012695 Avg Download Speed: 1.421 MB/s
Attempt for XOR `safe://hygoygykzkguckqygm985f5jrhcw87wiskbc97bottab47tq6fm9zm85f7r` stored as 9.jpg [SUCCESSFULL]
Total Time run in sec: 1.0013463497161865 Total Size in MB: 2.681166648864746 Avg Download Speed: 2.678 MB/s
Attempt for XOR `safe://hygoygyewnaj8o7za3qycckfy95o5j9gex6s3b8jygr9j48aiyzthc36ffo` stored as 10.jpg [SUCCESSFULL]
Total Time run in sec: 1.0011723041534424 Total Size in MB: 4.035780906677246 Avg Download Speed: 4.031 MB/s
Attempt for XOR `safe://hygoygykime3s7rup1ezeiqeqo1zy1o4bqmckewhriun889c64n4umxcyih` stored as 11.jpg [SUCCESSFULL]
Total Time run in sec: 1.001028299331665 Total Size in MB: 1.3675289154052734 Avg Download Speed: 1.366 MB/s
Attempt for XOR `safe://hygoygyc98c1ozh1cn9edxeworctuagt465zyybr61m8judfry4a4zycf8c` stored as 12.jpg [SUCCESSFULL]
Total Time run in sec: 1.0016491413116455 Total Size in MB: 2.7273664474487305 Avg Download Speed: 2.723 MB/s
Failed to GET file / download!
Attempt for XOR `safe://hygoygyqcq8bift7urnftiqxzn7d877ckwtk85haawg7oigm3bhn3r1dx7e` stored as 13.jpg [FAILED]
Total Time run in sec: 1.0010106563568115 Total Size in MB: 0.0 Avg Download Speed: 0.0 MB/s
Attempt for XOR `safe://hygoygyk7dzmcqxm3q4np87unu6tb9d8qt44fowxgqrhjy1xs6xgt69hade` stored as 14.jpg [SUCCESSFULL]
Total Time run in sec: 1.001042127609253 Total Size in MB: 3.857426643371582 Avg Download Speed: 3.853 MB/s
Attempt for XOR `safe://hygoygymq98z86j1oagpctmg7pnixz94ckekpgytu1jkb45du8xw49qrr4w` stored as 15.jpg [SUCCESSFULL]
Total Time run in sec: 1.001164436340332 Total Size in MB: 1.0540924072265625 Avg Download Speed: 1.053 MB/s
Attempt for XOR `safe://hygoygypoq7y31uedr7c6q6e3jaxhd94id3r8bwn7od3f79hgdj7xbqibjr` stored as 16.jpg [SUCCESSFULL]
Total Time run in sec: 1.0010707378387451 Total Size in MB: 1.4187126159667969 Avg Download Speed: 1.417 MB/s
Attempt for XOR `safe://hygoygykw8iw7wt9o3c7w36hi5xc1coyjdmdfm6dima3y9ehefxipxkwg3w` stored as 17.jpg [SUCCESSFULL]
Total Time run in sec: 1.0010130405426025 Total Size in MB: 0.7175998687744141 Avg Download Speed: 0.717 MB/s
Attempt for XOR `safe://hygoygyc67ozi6m9i9o8xq5wti7ysa4g6nodxetmdgw64g5pw1qmucheeac` stored as 18.jpg [SUCCESSFULL]
Total Time run in sec: 1.0011467933654785 Total Size in MB: 2.7174530029296875 Avg Download Speed: 2.714 MB/s
Attempt for XOR `safe://hygoygyqsxzn4p6mjx7ggj98pqc4he53xk1c54kw3hq1967n4yga1nbo3iw` stored as 19.jpg [SUCCESSFULL]
Total Time run in sec: 1.0010757446289062 Total Size in MB: 4.411093711853027 Avg Download Speed: 4.406 MB/s
Attempt for XOR `safe://hygoygyxojetcr5553xfyjnfoh3ywge7grs8d7z66e8nf678stecagdkomw` stored as 20.jpg [SUCCESSFULL]
Total Time run in sec: 1.0010991096496582 Total Size in MB: 2.8568925857543945 Avg Download Speed: 2.854 MB/s
Attempt for XOR `safe://hygoygyqf4au3yf7ouyi7imu1hty5qrsjppy1mj5uba5tahnxkmocys5g7h` stored as 21.jpg [SUCCESSFULL]
Total Time run in sec: 1.0015888214111328 Total Size in MB: 2.688328742980957 Avg Download Speed: 2.684 MB/s
Attempt for XOR `safe://hygoygyx41a9samc35ep6h7y6zwku5ess3377f8awtuwn44r1pc5w3ragky` stored as 22.jpg [SUCCESSFULL]
Total Time run in sec: 1.0014371871948242 Total Size in MB: 6.316998481750488 Avg Download Speed: 6.308 MB/s
Attempt for XOR `safe://hygoygyej97u6irwmgq1od4s8m6h8epbkgmmwqtghhhy1hbcsntizpjmh3c` stored as 23.jpg [SUCCESSFULL]
Total Time run in sec: 1.0011541843414307 Total Size in MB: 4.228791236877441 Avg Download Speed: 4.224 MB/s
Attempt for XOR `safe://hygoygypn9urw4mhoqfabnke71e8usekjeg8m54t1fkoeigdyr5srgsqoia` stored as 24.jpg [SUCCESSFULL]
Total Time run in sec: 1.0011131763458252 Total Size in MB: 1.8393077850341797 Avg Download Speed: 1.837 MB/s
Tested on Ubuntu.
Feel free to modify it to your liking, you need to have safe installed and then the script will create the commands to execute for the given XOR URLS given (take note I put .jpg behind it standard so you can see images, but it’s possible a file is not an image):
import subprocess
import time
import os
xor_urls = [
"safe://hygoygym7tsj5hhyyykd1aqpw3djxea6om6xku568ahm7hy7gfn6q5gy7xr",
"safe://hygoygym19bdbzg6jakfzkmacxurpkbuphgsxi6af3aecugfqjfaq66shww",
"safe://hygoygyq1xiofs8kx11jwkcguzuocd1xa5qp767bjjz39yrh44eo1mjkjch",
"safe://hygoygypnrqirzo7r5w464mnda8s6sdpatyrco5pda1w8nh5appa3ycb98h",
"safe://hygoygykajkeropyam6yihj9exdhn5o4b1pbjbeet4bazqo8m47ga3pw8mo",
"safe://hy8ayqyjy9obx5kr5g9f1r95hshhfswt7yqh6cy9t34sap7zqid4ox7ucuo",
"safe://hygoygyp8c1wy7ijrpnesz9cj7enf4ms9oxhmp5m4r4yw4pjcemgywekf9r",
"safe://hygoygye13iznm1yxzpxp9seh6j8weh84c5ib3fq9mxozz4h6hb95o3uduo",
"safe://hygoygycc11b4pmrcb97ro6ro3wxh3cfrgi7ctbiwyya8mkktq1j35ysmzh",
"safe://hygoygykzkguckqygm985f5jrhcw87wiskbc97bottab47tq6fm9zm85f7r",
"safe://hygoygyewnaj8o7za3qycckfy95o5j9gex6s3b8jygr9j48aiyzthc36ffo",
"safe://hygoygykime3s7rup1ezeiqeqo1zy1o4bqmckewhriun889c64n4umxcyih",
"safe://hygoygyc98c1ozh1cn9edxeworctuagt465zyybr61m8judfry4a4zycf8c",
"safe://hygoygyqcq8bift7urnftiqxzn7d877ckwtk85haawg7oigm3bhn3r1dx7e",
"safe://hygoygyk7dzmcqxm3q4np87unu6tb9d8qt44fowxgqrhjy1xs6xgt69hade",
"safe://hygoygymq98z86j1oagpctmg7pnixz94ckekpgytu1jkb45du8xw49qrr4w",
"safe://hygoygypoq7y31uedr7c6q6e3jaxhd94id3r8bwn7od3f79hgdj7xbqibjr",
"safe://hygoygykw8iw7wt9o3c7w36hi5xc1coyjdmdfm6dima3y9ehefxipxkwg3w",
"safe://hygoygyc67ozi6m9i9o8xq5wti7ysa4g6nodxetmdgw64g5pw1qmucheeac",
"safe://hygoygyqsxzn4p6mjx7ggj98pqc4he53xk1c54kw3hq1967n4yga1nbo3iw",
"safe://hygoygyxojetcr5553xfyjnfoh3ywge7grs8d7z66e8nf678stecagdkomw",
"safe://hygoygyqf4au3yf7ouyi7imu1hty5qrsjppy1mj5uba5tahnxkmocys5g7h",
"safe://hygoygyx41a9samc35ep6h7y6zwku5ess3377f8awtuwn44r1pc5w3ragky",
"safe://hygoygyej97u6irwmgq1od4s8m6h8epbkgmmwqtghhhy1hbcsntizpjmh3c",
"safe://hygoygypn9urw4mhoqfabnke71e8usekjeg8m54t1fkoeigdyr5srgsqoia",
]
objs = []
for idx,xor in enumerate(xor_urls):
objs.append({"cmd":"safe cat {} > {}.jpg".format(xor,idx), "file_name":"{}.jpg".format(idx), "xor_url":xor})
def run_time(obj):
cmd = obj["cmd"]
file_name = obj["file_name"]
xor_url = obj["xor_url"]
timeoutInSeconds = 1 # Our timeout value.
cmd = "sleep 5" # Your desired command.
proc = subprocess.Popen(cmd,shell=True) # Starting main process.
timeStarted = time.time() # Save start time.
cmdTimer = "sleep "+str(timeoutInSeconds) # Waiting for timeout...
cmdKill = "kill "+str(proc.pid)+" 2>/dev/null" # And killing process.
cmdTimeout = cmdTimer+" && "+cmdKill # Combine commands above.
procTimeout = subprocess.Popen(cmdTimeout,shell=True) # Start timeout process.
proc.communicate() # Process is finished.
timeDelta = time.time() - timeStarted # Get execution time.
#print("Finished process in "+str(timeDelta)+" seconds.") # Output result.
file_stats = os.stat(file_name)
file_MB = file_stats.st_size / (1024 * 1024)
success="SUCCESSFULL"
if(file_MB <= 0):
print("Failed to GET file / download!")
success="FAILED"
down_speed_Mbs = (file_stats.st_size / (1024 * 1024)) / timeDelta
print(f"Attempt for XOR `{xor_url}` stored as {file_name} [{success}]")
print(f"Total Time run in sec: {str(timeDelta)} Total Size in MB: {str(file_MB)} Avg Download Speed: {str(round((file_MB/timeDelta),3) )} MB/s\n")
for obj in objs:
run_time(obj)
3 Likes
Can someone @maidsafe clarify where we are at with deduplication?
Putting the same file twice gives the same safe address but 2 different containers:
FilesContainer created at: "safe://hyryyryuigqpke8kb6fxikpsrcgx5m9uqcgcmbp9j9o8j78y13t4bjmd1menra?v=hfsc5pmp6n77uw4jxmscxzcn35kermuy3aqzmbu856wek33kmamuo"
+---+---------+--------------------------------------------------------------------+
| + | tuk.jpg | safe://hygoygyerg1azika5pwdwik5zu9eifku8dkstbfhxto3pjq7bj45et19ydy |
FilesContainer created at: "safe://hyryyrywdzdtbii3ebaaqoimt1obg7618ehtf5nmfr9idqjnxnpjneiz93enra?v=hgxqdsbkwkh37couzybmpzfkz8ietbightwqfsou1ozcyb1fssh8y"
+---+---------+--------------------------------------------------------------------+
| + | tuk.jpg | safe://hygoygyerg1azika5pwdwik5zu9eifku8dkstbfhxto3pjq7bj45et19ydy |
6 Likes
That should be the case right? Container is just a collection of references.
You can have 2 photo albums, one for public and one private (which would contain xor urls that are not included in public container).
8 Likes
maybe you load it to the gist?
You mean the python script should accept a file containing the xor urls as input?
1.7Gb of Rolling Stones
safe://hyryyrysg7td9ozp5cmtn3u64qezgn371gcb1uho1b8f76rra8apuy8azgynra?v=h6us3m8ydkrkynycf35o9kdt7bp3jikoozzjdxek7p76f499cj4yo
1 Like
loading your script to https://gist.github.com/ allows track changes like it could be a git repo
1 Like
How close to full now? I’ve not taken part in this testnet yet, maybe will later this evening my time if not over by then.
2 Likes
I’m loving how stable this is 
:
I’ve been trying to put a big folder with a bunch of random files…some over, most below 10mb. Additionally, doing this on the phone in a terminal ssh into my cloud instance, therefore every time I go out or come back in the session disconnects…so at least 5 interruptions/retries. Finally got enough time in one place and it went through, with god knows how many partial uploads before…anyhow, impressed!!
Now let’s kill this beast and let’s get into stable churn!!!
7 Likes
Ahhh, that should have been removed from the OP. That file is ~24mb, so has not been put. It was removed from a draft. But must have been the wrong draft 
.
Sorry for false alarm there folks. Will update the OP. (edit: updated).
@peca nice one bughunting. Could you report this on the github issues please 
So the underlying data would be deduplicated. The Files container would not at this point, I think.
Dataa update:
node-1: 32K total ─╯
node-24: 24K total
node-15: 24K total
node-23: 24K total
node-21: 24K total
node-16: 24K total
node-3: 24K total
node-19: 7.7G total
node-9: 7.8G total
node-22: 8.1G total
node-7: 8.8G total
node-12: 11G total
node-11: 8.8G total
node-17: 8.8G total
node-25: 12G total
node-18: 11G total
node-6: 12G total
node-5: 12G total
node-10: 12G total
node-8: 12G total
node-4: 12G total
node-2: 12G total
node-20: 12G total
node-13: 12G total
node-14: 12G total
And metricbeat is looking happy:
13 Likes
16 hours ago we had about 2,5GB, now 12Gb per node. That makes about 0,6GB / hour. If the pace is stable, nodes should be full in about 63 hours [(50GB - 12GB) / 0,6GB per hr] from now. That’s about 2,5 days. That overlaps quite badly with all the Christmas hassles.
Personally I cannot upload much more. I don’t have enough files, and creating meaningful number of slightly modificated images takes up too much resources.
I propose we try to fill it up with automatically generated junk, or what does @joshuef and @dirvine think?
4 Likes
Finished in:
real 265m7,911s
user 30m7,070s
sys 10m11,216s
5 Likes
I’m inclined to let it be for now. It’s good to check how long it remains up for too (w/o being at max capacity).
4 Likes
You could take a .jpg of 9.9 MB, copy it a thousand times and then let a tool add a unique watermark (with random text or a counter or something) to them. That would create verifiable and unique files.
I want to do this soon, when I have time for testing. Not sure what tool I need for that, but I’m sure they exist.
1 Like
Making unique files is easy.
Just take large random number, S, seed, 64 bits in size for example.
Then put into file S+i, where i is index of file, and fill rest of the file with zeros.
For example, with S = 2348527586160649322:
File #1: 2348527586160649323000000000000...000
File #2: 2348527586160649324000000000000...000
File #3: 2348527586160649325000000000000...000
…
2 Likes
Sorry but… This sounds to me like: Making a good cake is easy. Just mix the ingredients, put it in the oven, and make a nice topping!
1 Like
I can describe how to add two numbers, but why it is needed?
Or you mean that such file will be bad by some criterion?
If so, please tell exactly why it will be bad.
That would result in unreadable files, right?
We are testing data integrity so we would need to easily see if files are not corrupted. With a jpg that is easy to do.
3 Likes
Do you mean manually checking thousands of images?
1 Like