Files to test launch network with

Community
21

Thank you for getting some test files up, they are all working, but very slow :yawning_face: I’ve not got enough tokens to attempt an upload yet, hopefully tomorrow - assuming we have people saturating their upload bandwidth


autonomi client built with git version: stable / a44fd08 / 2024-10-29
🔗 Connected to the Network                                                                                                                                              Fetching file: "AnarchyInTheSouthside.mp3"...
Successfully downloaded data at: 78d772e35cc36964aa1ac37aac52de8b4678d150f29f90df5fac67843d52930a

real    12m16.576s
user    1m38.113s
sys     1m3.950s

autonomi client built with git version: stable / a44fd08 / 2024-10-29
🔗 Connected to the Network                                                                                                                                              Fetching file: "Best_CD_Ever.mp3"...
Successfully downloaded data at: 0fd628ef31ac942ba366a4c3fe6ac909f2762a956cede8ad9e041087cf7d8a21

real    9m42.505s
user    1m2.963s
sys     0m44.543s
2 Likes
22

Another possibility (probably easier short term) is renaming safeup to autonomi (instead of calling the client autonomi).

The commands could then be
autonomi install client
autonomi install node
autonomi install //ask user or install both
autonomi start //ask if client or node(s)
autonomi start client
autonomi start nodes 3 (open launchpad like interface)
etc.

The larger idea is to combine the launchpad and nodemanager with safeup and call it autonomi. Sounds more logical and intuitive to me than calling just the client autonomi (after all its the name of the whole network). It would also be a chance to get rid of 2-3 separate launchers.

CC: @JimCollinson

3 Likes
23

Yeah, these are all good suggestions folks.

We are having a consolodation of the CLI commands, names, apps, crates etc
 But we also wanted to let the dust settle a little and not throw it all in in the middle of launch. For Chris’ sanity more than anything!

But yeah will make sure this is considered. Thanks!

15 Likes
24

Throwing it into the middle of January launch dust doesn’t sound more reasonable tbh @JimCollinson :sweat_smile::thinking: and it we’re honest with ourselves you again came up with a timeline that just spreads across 3 months and includes Christmas/new year
 So a bit more than 2 months :sweat_smile:


 Doesn’t sound super relaxed when I look at the current status 


4 Likes
25

Here’s a wee toy self-encryption · PyPI This is about to get a breaking update mind you (it’s my side hussle for now) , but it’s ok to play with. I will try and get full client API soonish (I am working still sporadic and at half a tank :smiley: :smiley: ) Just wanna get some fast created apps to test API’s which I feel still need an overhaul and how best than trying them

Feedback welcome to me and not the team for this one. I am sporadic on line too mind you. Expect bugs and breaking changes in this python world (likely rust to) but we need to keep the underlying data format solid.

12 Likes
26

Is this the first failure to download:-

time autonomi file download ec3e0b2aa958befc9de1ad24b9336ac920088a1dfbfef07145092dbe2fda2324 Deep_Feelings_Mix.mp3
Logging to directory: "/home/ubuntu/.local/share/safe/autonomi/logs/log_2024-10-31_22-54-58"
autonomi client built with git version: stable / a44fd08 / 2024-10-29
🔗 Connected to the Network                                                                                             Fetching file: "Deep_Feelings_Mix.mp3"...
1 errors while downloading data at: ec3e0b2aa958befc9de1ad24b9336ac920088a1dfbfef07145092dbe2fda2324
[
    "Failed to fetch file \"Deep_Feelings_Mix.mp3\": General networking error: GetRecordError(RecordNotFound)",
]
Error: 
   0: Errors while downloading data

Location:
   autonomi-cli/src/actions/download.rs:124

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

real    22m35.068s
user    2m57.067s
sys     1m5.241s

The first attempt failed after less than a second. This one took 22 mins to error out. Is anyone else able to test it?

1 Like
27

When I run this command

autonomi file download 41eaab3ae781d1f394f135a374f97f26ac4deece5f220187560295741a6cb602 An_Introduction_To_The_SafeNetwork.mp4

I get the file in a directory like this:

$PWD/An_Introduction_To_The_SafeNetwork.mp4/An_Introduction_To_The_SafeNetwork.mp4

Is this just me or are others seeing this too? Was the original upload done like this with a file within a directory, or is this something the downloading client is doing?

Also the output doesn’t tell me where the file is located on my filesystem. The output is:

$  autonomi file download 41eaab3ae781d1f394f135a374f97f26ac4deece5f220187560295741a6cb602 An_Introduction_To_The_SafeNetwork.mp4
Logging to directory: "~/.local/share/safe/autonomi/logs/log_2024-11-01_10-31-26"
autonomi client built with git version: stable / a44fd08 / 2024-10-29
🔗 Connected to the Network
Fetching file: "An_Introduction_To_The_SafeNetwork.mp4"...
Successfully downloaded data at: 41eaab3ae781d1f394f135a374f97f26ac4deece5f220187560295741a6cb602

This doesn’t say where on the filesystem the file was saved.

6 Likes
28

Confirmed here as well

willie@gagarin:~$ ls -al An_Introduction_To_The_SafeNetwork.mp4/An_Introduction_To_The_SafeNetwork.mp4 
-rw-rw-r-- 1 willie willie 13439292 Nov  1 00:51 An_Introduction_To_The_SafeNetwork.mp4/An_Introduction_To_The_SafeNetwork.mp4

yep - should at least tell you its saved in a new dir in the current working dir

4 Likes
29

yeah - and the folder with name if the file including it’s file suffix doesn’t make sense to me - but maybe there is a good reason?

Maybe we’re just “doing it wrong” because we assume how it works
? And the passed parameter is always a folder and the filename comes from the network
?

 I’ll try


30

Yes, files that are downloaded go into a directory at your current location. Which is fair enough. Naming a folder with a suffix that is supposed to tell you what the file type though is confusing!

That was the behaviour with the last network as well. It’s not changed for this one.

ls
An_Introduction_To_The_SafeNetwork.mp4  BegBlag.mp3       Patosh-RoughNight.mp3
AnarchyInTheSouthside.mp3               Best_CD_Ever.mp3  autonomi.mp4
ubuntu@ip-172-30-1-42:~/downloads$ file *
An_Introduction_To_The_SafeNetwork.mp4: directory
AnarchyInTheSouthside.mp3:              directory
BegBlag.mp3:                            directory
Best_CD_Ever.mp3:                       directory
Patosh-RoughNight.mp3:                  directory
autonomi.mp4:                           directory
ubuntu@ip-172-30-1-42:~/downloads$ cd BegBlag.mp3
ubuntu@ip-172-30-1-42:~/downloads/BegBlag.mp3$ ls
BegBlag.mp3
ubuntu@ip-172-30-1-42:~/downloads/BegBlag.mp3$ file *
BegBlag.mp3: Audio file with ID3 version 2.2.0, contains: MPEG ADTS, layer III, v1, 320 kbps, 44.1 kHz, JntStereo
ubuntu@ip-172-30-1-42:~/downloads/BegBlag.mp3$

Maybe the slowness of downloads is to do with the extra CPU and bandwidth being used by nodes that will hopefully go away. Maybe it’s partly due to the relative smallness of the network at the moment.

More worrying is the failure to download the file at this early stage in the life of the network.

31

The behaviour is different from previous behaviour

The filename seems to be part of the data map now

When doing

autonomi file download 41eaab3ae781d1f394f135a374f97f26ac4deece5f220187560295741a6cb602 .

It downloads the mp4 and stores it with name An_Introduction_To_The_SafeNetwork.mp4 in the current working directory

When passing anything else it creates a folder with the passed name

I wonder what it does when a file with same name already exists


2 Likes
32

That is a nice spot! Yes, if you put a dot at the end you end up with just the file rather than a directory containing the file.

But the default behaviour of ending up with a directory named after the file which contains the file is the same as in the last network as well.

2 Likes
33

The dot really is a path (current directory)

When doing

autonomi file download 41eaab3ae781d1f394f135a374f97f26ac4deece5f220187560295741a6cb602 somefolder

You end up with a folder called somefolder in the current directory containing the file An_Introduction_To_The_SafeNetwork.mp4

So there is no way to influence the filename of the file you download (and I tested it - it just overwrites if a file with the same name exists already)


 I do think it’s a nice think to not needing to know the filename
 But somewhat the current solution is not intuitive (while better than the first version where the download just ended in the downloads folder and not the cwd)

3 Likes
34

I tried only one time:

Fetching file: "Deep_Feelings_Mix.mp3"...
1 errors while downloading data at: ec3e0b2aa958befc9de1ad24b9336ac920088a1dfbfef07145092dbe2fda2324
[
    "Failed to fetch file \"Deep_Feelings_Mix.mp3\": General networking error: GetRecordError(RecordNotFound)",
]
Error: 
   0: Errors while downloading data

Location:
   autonomi-cli/src/actions/download.rs:124

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

real	23m30,335s
user	1m16,363s
sys	0m33,902s
1 Like
35

one could get the impression we had faster and more reliable networks already :wink: 


iirc we upload our chunk to one node currently and the network takes care of replications and populates the data.
Could it be that for this file a node did accept the payment + data and then committed suicide due to >50% cpu? :smiley:


and guessing from current speed: the absence of many server-nodes with excellent connection speeds (and no incentive for node runners to run nodes on fast connections) doesn’t help with the current network experience :innocent:

3 Likes
36

I haven’t tested other things, but downloads are terribly slow compared to non-EVM testnets :cold_face:

2 Likes
37

grep’ing a log for trying a download is throwing lots of hand shake time out errors

Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) })
[2024-11-01T09:38:51.152954Z ERROR sn_networking::event::swarm] Dial errors len : 1
[2024-11-01T09:38:51.152959Z ERROR sn_networking::event::swarm] OutgoingTransport error : Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) })
[2024-11-01T09:38:51.153411Z ERROR sn_networking::event::swarm] Dial errors len : 1
[2024-11-01T09:38:51.153421Z ERROR sn_networking::event::swarm] OutgoingTransport error : Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) })
[2024-11-01T09:38:51.171053Z ERROR sn_networking::event::swarm] Dial errors len : 1
[2024-11-01T09:38:51.171058Z ERROR sn_networking::event::swarm] OutgoingTransport error : Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) })
[2024-11-01T09:38:51.215431Z ERROR sn_networking::event::swarm] Dial errors len : 1
[2024-11-01T09:38:51.215436Z ERROR sn_networking::event::swarm] OutgoingTransport error : Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) })
[2024-11-01T09:38:51.218713Z ERROR sn_networking::event::swarm] Dial errors len : 1
[2024-11-01T09:38:51.218914Z ERROR sn_networking::event::swarm] OutgoingTransport error : Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) })
[2024-11-01T09:38:51.219416Z ERROR sn_networking::event::swarm] Dial errors len : 1
[2024-11-01T09:38:51.219420Z ERROR sn_networking::event::swarm] OutgoingTransport error : Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) })
[2024-11-01T09:38:51.236883Z ERROR sn_networking::event::swarm] Dial errors len : 1
[2024-11-01T09:38:51.236888Z ERROR sn_networking::event::swarm] OutgoingTransport error : Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) })
[2024-11-01T09:38:51.243361Z ERROR sn_networking::event::swarm] Dial errors len : 1
[2024-11-01T09:38:51.243366Z ERROR sn_networking::event::swarm] OutgoingTransport error : Other(Custom { kind: Other, error: Left(Left(Left(Connect(NoReservation)))) })
[2024-11-01T09:38:51.308913Z ERROR sn_networking::event::swarm] Dial errors len : 1
[2024-11-01T09:38:51.308918Z ERROR sn_networking::event::swarm] OutgoingTransport error : Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) })
[2024-11-01T09:38:51.310530Z ERROR sn_networking::event::swarm] Dial errors len : 1
[2024-11-01T09:38:51.310535Z ERROR sn_networking::event::swarm] OutgoingTransport error : Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) })
[2024-11-01T09:38:51.361279Z ERROR sn_networking::event::swarm] Dial errors len : 1
[2024-11-01T09:38:51.361283Z ERROR sn_networking::event::swarm] OutgoingTransport error : Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) })
[2024-11-01T09:38:51.395616Z ERROR sn_networking::event::swarm] Dial errors len : 1
[2024-11-01T09:38:51.395623Z ERROR sn_networking::event::swarm] OutgoingTransport error : Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) })
[2024-11-01T09:38:51.396205Z ERROR sn_networking::event::swarm] Dial errors len : 1
[2024-11-01T09:38:51.396212Z ERROR sn_networking::event::swarm] OutgoingTransport error : Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) })
[2024-11-01T09:38:51.614944Z ERROR sn_networking::event::swarm] Dial errors len : 1
[2024-11-01T09:38:51.614947Z ERROR sn_networking::event::swarm] OutgoingTransport error : Other(Custom { kind: Other, error: Right(Custom { kind: Other, error: Custom { kind: Other, error: HandshakeTimedOut } }) })
3 Likes
38

Among the above files, all were available for download except for “autonomi file download ec3e0b2aa958befc9de1ad24b9336ac920088a1dfbfef07145092dbe2fda2324 Deep_Feelings_Mix.mp3”

2 Likes
39

Surely to goodness a PUT isn’t acknowledged and payment made until the data is safe (pun intended) on the total number of nodes that are supposed to hold it? Or at least a sensible proportion of them?

40

Considering replication is not instantaneous but could be many seconds to minutes (not sure on replication frequency) it would make for a very long time to upload.

But agree that at least the node that received the record should have it stored.

@joshuef A quick question if I may.

Since the encryption on disk has been implemented again, what, if any, process is there for nodes that went offline (say blackout in geo region) and when coming back online to offer back up the records they were holding? When the node restarts it cannot read the record_store since it was encrypted with a memory held key.

2 Likes