QuicNet [30/01/24Testnet] [Offline]

[DavidMc0]

Just re-tried my big 35.7GB upload again, and it worked

Trying to download it now.

It’s public if anyone else wants to try:

mixtral-8x7b-instruct-v0.1.Q6_K.gguf af4c83f1c16fbc213661f808a9fb1654a0a433617889d61a310ec3c86cec631b

[joshuef]

@happybeing

It would also help a lot if chunking / upload / delete of uploaded chunks were part of a batched process.

Can that be done with self encryption currently? It would need pause resume so maybe but supported yet

Right now, we dont start uploading until all chunks are done. But chunks themselves is a streaming process, so we could well start uploads before chunking is finished, eg.

It’s not an optimisation we’ll get into yet though, I think.

[happybeing]

@DavidMc0

mixtral-8x7b-instruct-v0.1.Q6_K.gguf af4c83f1c16fbc213661f808a9fb1654a0a433617889d61a310ec3c86cec631b

Something very wrong here. It downloads without error but is only 1.8K!

time safe files download mixtral-8x7b-instruct-v0.1.Q6_K.gguf af4c83f1c16fbc213661f808a9fb1654a0a433617889d61a310ec3c86cec631b
Logging to directory: "/home/peter/.local/share/safe/client/logs/log_2024-01-31_13-41-30"
Built with git version: a15b141 / main / a15b141
Instantiating a SAFE client...
Trying to fetch the bootstrap peers from https://sn-testnet.s3.eu-west-2.amazonaws.com/network-contacts
Connecting to the network with 43 peers
🔗 Connected to the Network                                                                                                               Downloading "mixtral-8x7b-instruct-v0.1.Q6_K.gguf" from af4c83f1c16fbc213661f808a9fb1654a0a433617889d61a310ec3c86cec631b with batch-size 16
Saved "mixtral-8x7b-instruct-v0.1.Q6_K.gguf" at /home/peter/Downloads/mixtral-8x7b-instruct-v0.1.Q6_K.gguf

real	0m23.035s
user	0m2.368s
sys	0m1.225s

$ safe -V
sn_cli 0.89.29

[TylerAbeoJordan]

@happybeing

downloads without error but is only 1.8K!

@DavidMc0

Just re-tried my big 35.7GB upload again, and it worked

Wow, SafeNetwork compression is legit – The Weissman score here must be off-the-charts!

[TylerAbeoJordan]

@qi_ma

could you have a try with re-upload again? thx

Seems to have worked? It’s trying to verify now, so I guess it finished all the chunk uploading?

[DavidMc0]

Hmm… I started the upload forgetting to make it public, and this only partially completed. I then re-uploaded remaining chunks with -p, but it seems it didn’t apply that to the whole file, so perhaps it’s part private and part public?

I’ll see if re-chunking and uploading from scratch helps with that.

[qi_ma]

@TylerAbeoJordan

Seems to have worked? It’s trying to verify now, so I guess it finished all the chunk uploading

yeah, shall be.
just give it sometime to complete the verification

[qi_ma]

@DavidMc0

I then re-uploaded remaining chunks with -p, but it seems it didn’t apply that to the whole file, so perhaps it’s part private and part public?

as long as the head chunk got uploaded when -p is set, all chunks will then become accessable.
so there is no part private and part public.

[Optimator]

I’m running Windows 10 and I had the same issue with Windows deleting my safe executable downloads a while back. I’ve now been running safe.exe and safenode.exe successfully by creating a directory exclusion from antivirus scans in Windows Security.

To do it, I went through the Virus and Threat Protection - Virus and Threat Protection Settings - Manage Settings - Add or Remove Exclusions - Add an Exclusion page in Windows Security.

I added an exclusion for the C:Users\<user>\safe directory, my default for safe downloads. Microsoft Defender Antivirus really doesn’t like safe.exe and safenode.exe.

[DavidMc0]

Ok, not sure what the issue is in that case. I’m downloading it successfully so far, with 4gb downloaded, so not sure why happybeing can’t download the full file.

Is the ‘Head chunk’ the first, or last to be uploaded?

[qi_ma]

@happybeing

Something very wrong here. It downloads without error but is only 1.8K!

tried to download it as well, which gives downloading a data_map contains 16 chunks and then downloading 73202 chunks.

⠲ [00:00:33] [########################################] 16/16
⠴ [00:00:33] [########################################] 16/16
⠁ [00:00:00] [----------------------------------------] 0/73202
⠉ [00:00:00] [----------------------------------------] 0/73202  

seems it is ok?
could you give me the log of that 1.8K download? thx

[qi_ma]

@DavidMc0

Is the ‘Head chunk’ the first, or last to be uploaded?

it’s randomized , could be in any position

[Toivo]

@Toivo

It didn’t solve them, but it did change the behaviour somewhat:

Previously:

• Lot’s of uploading made the whole LAN weird / slow / unusable for everyone in our home.

Now:

• It seems that only the CLI and nodes are affected. Even webpages on the same machine seem to be working fine

I was too positive there, it’s just the same.

[Profess]

@chriso

Hmm, that’s annoying that it’s marking safe as some kind of malicious file. You might need to look at the Windows firewall or anti-virus settings to add an exception for it. I don’t believe anyone else has ever hit that issue. I was wondering if you were using Windows 11, but you’ve already mentioned you’re on 10.

@Optimator

I’m running Windows 10 and I had the same issue with Windows deleting my safe executable downloads a while back. I’ve now been running safe.exe and safenode.exe successfully by creating a directory exclusion from antivirus scans in Windows Security.

I also just thought that other testers using Windows might have had a problem with the safe.exe file, but didn’t report it, just stopped testing.

That’s why I sent a message to @chriso that Windows treats this file as a Trojan, I checked Windows Security - Protection History, all attempts to run the client are marked as:

Blocked threat - serious status

Detected: Trojan:Win32/Wacatac.B!ml
Status: quarantine
These files will be deleted automatically.
Date: 31.01.2024 10:39
Details: This program is dangerous and executes the commands of the attacker.

Applies to items:
file: C:\Users\gggg\safe\safe.exe

I’m not sure, but I don’t think it’s enough to add an exclusion in the Windows security settings, because in a future network less advanced users will see the error and stop using SafeNet.

[digipl]

I downloaded the latest version of the client and reuploaded a directory several times (at first 19 chunks were failing).
17 chunks have been uploaded successfully eventually, but despite numerous attempts, two chunks still cannot be uploaded.
The last log
log_2024-01-31_15-22-02.zip (58.9 KB)

[chriso]

Honestly, this is the first I’ve heard about this issue. Thanks for the information.

I think we’re going to need to look into this. I guess we need to determine first of all, why does the OS flag our binaries, then what we can do it about it.

Might need to get in touch with Microsoft somehow.

[Josh]

After looking at RAM use a little more it seems to be released. the two nodes I lost were at ~3GB RAM when they got booted but I have others that got near there too and then dropped way back down.

So it would seem that if there is enough room for spikes it will level out again.

Are the faster and larger uploads just a bigger burden to bear?

[happybeing]

@qi_ma

could you give me the log of that 1.8K download? thx

Seems it’s failing to report the error at the end of the log, and saving the file regardless. See:
safe-client.zip (32.5 KB)

[qi_ma]

@digipl

but despite numerous attempts, two chunks still cannot be uploaded.

shall be ok now.
can have a re-upload.

[qi_ma]

@Josh

Are the faster and larger uploads just a bigger burden to bear?

yeah, could be a reason for it.
the better performed client does encouraging quite more uploads than previous testnet runs.