Merkel is here

Very encouraging to see the network performing well and improving over time. Great work testing & sharing results!

I’d like to upload a bunch of stuff, but I think the UX will be holding many back.

Could you lay out a simple list of the commands to upload and then how you can find & download files in the future?

I have loads of photos I’d like to batch upload, but don’t understand how I’d store keys / data maps to be able to find and download them in the future.

Hope the UX improves quickly (bring on indelible!), but in the meantime, clear instructions would certainly help me to get going with uploads, and probably others too.

6 Likes

I struggled at first to get started with Ant CLI, it really felt overwhelming to me by the look of it. But honestly, it’s super easy. Let me write a quick guide for people wanting to give it a try :slight_smile: gimme half an hour.

8 Likes

For the sake of time, I will not make a guide how to get antctl up and running, I’m sure there is enough of that on the forum already. Otherwise I can recommend copying the github url into chatgpt and asking for a step by step install guide. It works really well.

Uploading Files

First you need to attach your wallet, you do this by setting your private key. I set it as environmental value, that way, upon closing the powershell (windows) window the private key is no longer stored, this is a much safer way. Nonetheless, I recommend everyone to use a separate upload wallet with small amounts of ETH/ANT on it, you really don’t need much to get going. You set it as environment value by doing this

“$env:SECRET_KEY=”Your secret key here”

Uploading a single file is as easy as performing this command after”.

For a file:

ant --network-id 1 file upload -p --retry-failed 1000 --merkle “D:\yourfilename.txt"

For a folder:

ant --network-id 1 file upload -p --retry-failed 1000 --merkle “D:\yourfoldername"

These names are on a disk D. But it’s just the entire path to a file/folder, can be on other disks obviously. included it to make sure people don’t forget to include the disk path.

Uploading batches

Since uploading can still be a bit slow (chunk by chunk), I’ve created a script that can batch multiple files at once. It is slightly more costly, as smaller files will be treated individually, a file of 20 chunks will now have 1 merkle tree payment (±$0.09 at the moment). While if you upload a folder containing 10 files at once, it will merkle tree the payment of 200 chunks into one. Keep that in mind.

$secret = “YourPrivateKey”
$folder = “D:\FolderPath”
$files = Get-ChildItem -Path $folder -File

$secretEsc = $secret.Replace(“'”,“‘’”)

foreach ($f in $files) {
$path = $f.FullName.Replace(‘"’,‘“”’)
$cmd = “$env:SECRET_KEY='$secretEsc'; ant --network-id 1 file upload -p --retry-failed 1000 --merkle “$path`””
$enc = [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes($cmd))
Start-Process powershell.exe -ArgumentList “-NoExit -EncodedCommand $enc”
}

Add your private key on the top. At the path to the folder. It will create a new powershell window for each file in the folder and starting to upload that individual file. Keep in mind, it does require quite a bit of NAT connections, so for non-specialized home routers I wouldn’t recommend more than 5 files at once. I personally found that my PC really starts struggeling with 100 files+.

Downloading files:

ant file download “DataString” “D:\AutonomiDownloadFolder\namefile.extension”

Add the file data string. Then add the path where it should be downloaded, with namefile.extension being the file name and extension. For folders, dont add the extension.

Batch Downloading files:

This script below will look for a textfield downloads.txt that on each line contains the data string, followed by a space and “namefile.extension” So for example:

3837010a4d28a1f86948734743ad9c75db1d07e84bba939f07ed326f4949 “Testfile.iso”

For each line, it will create a powershell window, downloading the file onto the destination folder provided at the top of the script as well.

# Inputs

$downloadsFile = “D:\downloads.txt”
$destFolder    = “D:\AutonomiDownloadFolder”

# Optional, only if your downloads require it

# $secret = “key”

New-Item -ItemType Directory -Force -Path $destFolder | Out-Null

$lines = Get-Content -LiteralPath $downloadsFile

#If you set $secret above

if (Get-Variable secret -Scope Script -ErrorAction SilentlyContinue) {
$secretEsc = $secret.Replace(“'”,“‘’”)
}

foreach ($lineRaw in $lines) {
$line = $lineRaw.Trim()
if (-not $line) { continue }
if ($line.StartsWith(“#”)) { continue }

#Split into: first token = data key, rest of the line = filename (can contain spaces)

$parts = $line -split ‘\s+’, 2
if ($parts.Count -lt 2) {
Write-Warning “Skipping line (expected:  ): $line”
continue
}

$dataKey   = $parts[0].Trim()
$fileName  = $parts[1].Trim().Trim(‘"’)
$destPath  = Join-Path $destFolder $fileName

$cmd = “ant file download "$dataKey” "$destPath""
if (Get-Variable secret -Scope Script -ErrorAction SilentlyContinue) {
$cmd = “`$env:SECRET_KEY=‘$secretEsc’; $cmd”
}

$enc = [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes($cmd))
Start-Process powershell.exe -ArgumentList “-NoExit -EncodedCommand $enc”
}
8 Likes

Thanks. That’s very helpful, and also terrifying :laughing:

I may have a dabble if I find the time, but will probably wait for UX improvements.

3 Likes

Just start with a small folder, you only need this:

“$env:SECRET_KEY=”Your secret key here”

and then

ant --network-id 1 file upload -p --retry-failed 1000 --merkle “D:\yourfoldername"

Much less terrifying :heart_hands:t2:

3 Likes

That is certainly less daunting, thanks! :grinning_face_with_smiling_eyes:

1 Like

you can also play with these variables to speed things up :grinning_face:

#Uploads
export CHUNK_UPLOAD_BATCH_SIZE=25
export FILE_UPLOAD_BATCH_SIZE=25
export UPLOAD_FLOW_BATCH_SIZE=25
#Downloads
export CHUNK_DOWNLOAD_BATCH_SIZE=15
export MAX_IN_MEMORY_DOWNLOAD_SIZE=15
export STREAM_DECRYPT_BATCH_SIZE=15

1 Like

Does this allow for multiple chunk uploads/downloads simultaneously? That would be a lot easier for sure :joy:

1 Like

yes i think all defaults are one so try with 5 and work your way up till it brakes :slight_smile:

Sounds a little similar to the movie “Ralph breaks the internet” :thinking::wink:

1 Like

With these new settings I just managed to download a 600 MB file in less than a minute. Absolutely amazing.

10 Likes

great work with sharing your adventures in testing with the cli :slight_smile:

3 Likes

Really enjoying myself :slight_smile:

With these settings it does look like it’s uploading 25 items at once, then wait for all 25 to upload before batching another 25. Is there also a setting to make it 25 simultaneously? So when 1 of 25 is finished it starts 26.

I am guessing since merkel is new there is some more bits and bobs to be sorted out. with the old way of paying it was a lot faster to upload in large batches.

1 Like

How do you use / apply these settings?

Copy pasting in the terminal works. If you want to set them once you can set them as environmental value.

1 Like

On Linux I have them chucked in my ~/.bashrc for windows @anon75844067 can advise

1 Like

I gave this a shot with a small folder first, and it gave an error initially, but I added ./ before ant and it worked for me :smiley:

Uploading was very slow, with a 19mb folder taking around 20 minutes or something (though the CLI says it took 35s…).

I was able to download an individual file from the folder, and the whole folder in one go afterwards.

Once I figure out how to get better performance, I’ll try some bigger folders. Thanks for the help!

2 Likes

I just get an error if I try this with one of the commands, e.g. export CHUNK_DOWNLOAD_BATCH_SIZE=15 pasted into powershell yields:

export : The term ‘export’ is not recognized as the name of a cmdlet, function, script file, or operable program.

I use environment values, like this:

$env:CHUNK_UPLOAD_BATCH_SIZE=“25”
$env:FILE_UPLOAD_BATCH_SIZE=“25”
$env:UPLOAD_FLOW_BATCH_SIZE=“25”
$env:CHUNK_DOWNLOAD_BATCH_SIZE=“25”
$env:MAX_IN_MEMORY_DOWNLOAD_SIZE=“25”
$env:STREAM_DECRYPT_BATCH_SIZE=“25”

This does not safe when you close the terminal window, so you do it again for every time you open a new window. I prefer it that way, same with the secret key.

2 Likes