I have the same error as @davidpbrown but upload is slighly faster than last time
3 Likes
1.6GB file to upload:
$ safe files put ./to-upload/ivan.mkv
memory allocation of 1710792562 bytes failed
Upload is ~3x times faster. Did not measured however.
@Mendrit Right now upload command uses amount of RAM equal to ~2x of file size.
So for 1.6 GB file you need 3.2 GB of free RAM.
And look like it also uses 7x of upload traffic.
5 Likes
Sorry to see you go @StephenC, but your past (and future) contributions have been immense. And congratulations @bochaco and @joshuef . And welcome @Anselme and @chris.connelly! My heart usually sinks when seeing a personnel announcement, but 3 in and 1 (not completely) out is great 
Finally it is awesome to see an alumnus come back to SAFE!
24 Likes
To upload 118MB it takes 17min. About 1Mbps.
5 Likes
so we are losing @StephenC but gaining @Anselme and @chris.connelly. Plus Stephen will still be around. I read that as: the team is growing - which is great!
as for the testnets: I would encourage ppl to also keep posting improvements they observe.
this is surely almost as relevant as posting errors? It really makes my day reading sth like:
upload is 3x faster / finally I could connect / seems a lot more stable
24 Likes
This will be a lot more clear soon. Right now what is happening is this:
- File is chunked
- Each chunk compressed
- Each chunks AES encrypted
- Each chunk is xored with concatenated hash of previous 2 chunks
- Each chunk gets a StoreCost from the network (client asks 7 Elders)
- Each chunk then get paid for
- The payment is forwarded to store section
- Chunk uploaded to Elders as paid for then forwarded to data section Elders, then forwarded to Adults and stored
So for 118Mb we have at least 118 chunks that all go through this process. This will dramatically change in T7 though where we will pay for batches of chunks at a time and not do wallet xfers across the network and so on.
So upload speed for a decentralised network where we pay for data is really apples and oranges comparison. Downloads kinda also, but they should be much more similar as we are not doing 8 of the 9 steps here.
16 Likes
Besides speed and probably code complexity with so many less operations it just means less to go wrong! T7 is gonna be 
7 Likes
I was able to create coins but my upload looks like it is very slow or has hung uploading a total of 1.3M using this:
time safe files put --recursive /usr/lib/python3.9
EDIT: I don’t think this is a slow upload, last time was faster, and this has been going for 74 minutes on a cloud node with good connectivity. So I’m going to kill it and try something small…
Now uploading a single text file 1,321 bytes… also just sits there, doesn’t complete.
Started from scratch, made new coins (took ages) but now a different error uploading a 1,321 byte file:
time safe files put bin/setup-testnet
Error: NetDataError: Failed to PUT Public Blob: ErrorMessage { source: InvalidOperation("Not enough storage available on the network"), msg_id: 142dc901.. }
real 0m10.573s
user 0m0.149s
sys 0m0.048s
First attempt at downloading and this looks bad:
time safe cat safe://hygoyeyx768nkst7qqjjk8khjm67tr1n4otbctcts5j7dten43zae3ga83y > ~/safe-the-planet-6.1.png
Error: NetDataError: Failed to GET Public Blob: SelfEncryption(Storage("Generic error(Error received from the network: DataNotFound(Chunk(Public(c0189e(11000000)..))) MessageId: 79595a57..)"))
real 0m10.648s
user 0m0.225s
sys 0m0.074s
3 Likes
It is important to understand what makes subsequential chunk uploads slower and slower over time.
I do not see in your description why upload of last chunks of file can be slower than upload of first chunks.
I was going to write that it is incredible that you decided to set it all up so late in the Friday night, but… it’s actually Thursday! You see, I’m going to spend a week of holiday in an off grid(ish) island for a week beginning tomorrow, so calling it a week now… but good job anyway! I have a nagging suspicion that my participation had something to do with the all the problems of previous versions, so enjoy your v6.1, it will run smoothly until next Thu/Fri when I’m back in.
Sorry to hear @StephenC go. And happy to hear @Anselme @chris.connelly and Chris come in! Congratulations to @joshuef!
10 Likes
I should at this point give some more information here. Ths is a very quick dirty overview of a bigger picture.
There is a trick we have not employed yet and we don’t want to, just yet. So now we do a huge amount of work per store (way way too much) and transfers/pay per chunk per upload etc. is a killer. But imagine even with Dbc we batch pay (we will). As @oetyng we could do that for AT2 but we are going Dbc. So let’s say payments will become thousands of times less work for decent files, maybe more. But imagine it’s nothing!
So right now we send data to store and also Get requests through a ton of nodes, sometimes client → 7 Elders then each Elder → 7 more Elders then 7 Elders to 3 Adults for a put (it becomes mammoth traffic and also an amplification attack). For Get client → 3 Elders → 7Elders → 3 Adults → 3 Elders → 3 Elders → Client. Wow, that’s a lot.
However, bittorrent/storj/ipfs etc. don’t do this, they just go to a node and get data from it (possibly several repolica nodes). So why are we taking all this trouble. Well, we are doing first the fallback routes, the hard way. The way we are right now is the worse it can be. We treat every node as though it will be bad and not do it’s task and that’s great for a fallback. It has to work and we can catch bad nodes. Remember we are truly decentralised, no humans, no contracts between humans and so on.
So the big trick is, get this solid and hopefully never take this route, unless we discover a bad node by way of not getting our data.
This means put is client → Elder → Elder → 3 Adults.
Get is client → Elder → Elder → Adult(s) → Elder → Elder -Client
This is 100’s less messages per operation. Much more than 99% less work/traffic and serialise/deserialse plus signature checks etc. But how?
Well consider today’s Get and Put as unfailable (when we iron out bugs). The usual routes as just Put / Get, these are simple and very fast. So the process becomes:
- Get
Use fast path with a client-side Get timeout (not network, but app side) let’s call that T1.
If we Get the data then all is well. If we fail (T1 expires) we fall back to unfailable-get (the slow Get we see today.
- Put
Use Put then Get the data using a client-side timeout (not a network timeout). So let’s call that T2. If we Get the data then cool, otherwise we do unfailable-Get.
If that fails we do an unfailable-Put this is the Put we see today.
Why does this work?
A bad node that just does not store or get data will receive a ton of data when the unfailable call comes in. He must deliver then or be killed (with unfailable we detect bad behavior). So why would a node not deliver? It seems to only vandalise the network as trying to break it loses him money. The vandalisation is temporary and hurts the node that failed to deliver. So a silly attack/byzantine failure. Therefore it’s unlikely, but not impossible (why we need this unfailable-Get/Put working) that it can happen. So we have the tools to not only recover from it, but also flood that node with more traffic and if he still is not delivering, to kill him.
25 Likes
I was describing the upload process. This is a different question and would be a different answer if I knew it 
There can be a huge number of things, some bugs, some expected (like high num clients, floods of uploads in a short time), but this is why we must test to find all causes.
3 Likes
Do anyone except for me is uploading large files now?
Right now upload happens for me at rate of ~2 chunks / minute.
It was faster right after network start.
This is possibly the most helpful comment yet delivered on this. Amazing stuff and great insight into the level of planning.
12 Likes
still couldnt start a node igd error again but got a put up so tried a video but got the following error:
User_1@DESKTOP-4QAR72F MINGW64 ~
$ safe files put /c/Users/User_1/Pictures/GOPR0738.JPG
FilesContainer created at: "safe://hyryyrbqnqr56ccpr95jya9krqpk7gkiq76pm7ppqhe7qmb9ne17dur41jenra"
+ C:/Users/User_1/Pictures/GOPR0738.JPG safe://hygoygyq1oyj61hmpxu8ypbty99aaij69mfgifxdnxhaipn8kgarwshwhfo
User_1@DESKTOP-4QAR72F MINGW64 ~
$ safe cat safe://hygoyeyx768nkst7qqjjk8khjm67tr1n4otbctcts5j7dten43zae3ga83y > ~/safe-the-planet.png
User_1@DESKTOP-4QAR72F MINGW64 ~
$ safe files put /c/Users/User_1/Pictures/GOPR0727.MP4
[safe_network::client::connections::listeners] ERROR 2021-06-24T21:45:19.986529400 [C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\safe_network-0.2.17\src\client\connections\listeners.rs:307] Error handling transfer error. ErrorMessage could not be sent to handler: SendError((159.65.60.126:44800, InsufficientPayment))
[safe_network::client::connections::listeners] ERROR 2021-06-24T21:45:20.750560300 [C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\safe_network-0.2.17\src\client\connections\listeners.rs:307] Error handling transfer error. ErrorMessage could not be sent to handler: SendError((178.128.169.33:46106, InsufficientPayment))
[safe_network::client::connections::listeners] ERROR 2021-06-24T21:45:21.343518100 [C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\safe_network-0.2.17\src\client\connections\listeners.rs:307] Error handling transfer error. ErrorMessage could not be sent to handler: SendError((138.68.191.226:46181, InsufficientPayment))
[safe_network::client::connections::listeners] ERROR 2021-06-24T21:45:21.549366500 [C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\safe_network-0.2.17\src\client\connections\listeners.rs:307] Error handling transfer error. ErrorMessage could not be sent to handler: SendError((159.65.24.159:56465, InsufficientPayment))
[safe_network::client::connections::listeners] ERROR 2021-06-24T21:45:21.552816600 [C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\safe_network-0.2.17\src\client\connections\listeners.rs:307] Error handling transfer error. ErrorMessage could not be sent to handler: SendError((178.128.169.227:54173, InsufficientPayment))
[safe_network::client::connections::listeners] ERROR 2021-06-24T21:45:24.755145300 [C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\safe_network-0.2.17\src\client\connections\listeners.rs:307] Error handling transfer error. ErrorMessage could not be sent to handler: SendError((178.128.172.1:54636, InsufficientPayment))
[safe_network::client::connections::listeners] ERROR 2021-06-24T21:49:41.049078300 [C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\safe_network-0.2.17\src\client\connections\listeners.rs:307] Error handling transfer error. ErrorMessage could not be sent to handler: SendError((138.68.191.226:46181, InsufficientPayment))
[safe_network::client::connections::listeners] ERROR 2021-06-24T21:49:43.113170000 [C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\safe_network-0.2.17\src\client\connections\listeners.rs:307] Error handling transfer error. ErrorMessage could not be sent to handler: SendError((178.128.169.33:46106, InsufficientPayment))
[safe_network::client::connections::listeners] ERROR 2021-06-24T21:49:44.109185000 [C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\safe_network-0.2.17\src\client\connections\listeners.rs:307] Error handling transfer error. ErrorMessage could not be sent to handler: SendError((178.128.175.25:33290, InsufficientPayment))
FilesContainer created at: "safe://hyryyrbe6dt8tm4rxum51wcnin756dw54pcpzyentpj36kno1ohhetp4mqcnra"
+ C:/Users/User_1/Pictures/GOPR0727.MP4 safe://hy8ayqyetit1b5ejf1sgrnjaoxwkyqyserpssh7owb61ffsd49n8fab6wor
[safe_network::client::connections::listeners] ERROR 2021-06-24T21:50:48.390139000 [C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\safe_network-0.2.17\src\client\connections\listeners.rs:307] Error handling transfer error. ErrorMessage could not be sent to handler: SendError((159.65.60.126:44800, InsufficientPayment))
also tried to get my file and got the following error:
`User_1@DESKTOP-4QAR72F MINGW64 ~
$ safe cat safe://hygoygyq1oyj61hmpxu8ypbty99aaij69mfgifxdnxhaipn8kgarwshwhfo
Error: NetDataError: Failed to GET Public Blob: SelfEncryption(Storage("Generic error(Error received from the network: DataNotFound(Chunk(Public(873e01(10000111)..))) MessageId: f98bf236..)"))`
1 Like
Infallible speeds should also be optimized for a better or more seamless fallback experience but I agree this raises confidence on upload/download speeds and puts at least less stress over the emphasis of them at their current state.
4 Likes
Sadly, large file upload was failed after 3 hours because of split (probably) 
Error: NetDataError: Failed to PUT Public Blob: ErrorMessage { source: InvalidOp eration("Not enough storage available on the network"), msg_id: 75e9f873.. }
1 Like
Im still getting told the network is not accepting new nodes
The network is not accepting nodes right now. Retrying after 3 minutes
[sn_node] ERROR 2021-06-24T22:23:54.661742292+01:00 [src/routing/core/bootstrap/join.rs:139] Network is set to not taking any new joining node, try join later.
If one section is split, other one is not.
Or it is not split, but some failed node was replaced?