Was anybody else’s immediate thought when they first understood how the Safe Network was going to work is how it would speed up the whole Internet? That was the first thing that jumped out at me when I first started following this project and one of the main reasons I’ve stayed here so long. I want to see the Safe Network speed up the Internet!
12 Likes
I hope so! Can you imagine Coinbase exchange not crapping out at the exact worse time (Though I wonder if that’s an intentional excuse)? Instead the service running more smoothly. That’ll be the day.
Wonder when we’ll get opportunistic and deterministic caching. 
3 Likes
I depends on what you call speed of the Internet. Internet as the data transporting layer is in general very fast. What makes it feel slow is crappy website design and architecture. I would say 90+% you are waiting for page load is not waiting for what you want to see, but wainting for tracking and ad serving systems do their annoying job.
I am affraid that as soon as Safe will be mainstream, there will be lot people making big effort to push that crap in Safe Network too. It wont be the same, but they will find a way to make our lives miserable again 
11 Likes
You’re spot on here @peca, I posted a video here a week or so ago where a guy goes through some recipe websites which are averaging about 8 or 9 MB, he takes one and makes his own site with all the words and images from the original web page, and it’s around 900kB. So straight out of the gate we’re looking at a crazy difference there.
It will be necessary to foster a good design culture and non-ridiculous business practices to ensure things don’t get bloated, but we’ll be starting off in a very good place.
3 Likes
Could you dig that up please? I scrolled thru your recent activity, but no luck.
1 Like
Depends is the real answer in my opinion.
The small post appearing in a forum, or the text only email being sent, or the tweet, and so will take longer across the wire to reach you (on average).
Apart from what @peca points out, the tweets now take a lot more time for the server to process after the person sends it than the time across the wire. Maybe 10 to 1000 times more.
Safe while slower in transmitting a small packet of data across the wire, the time from the person successfully posting the data, or sending mail is nothing since the application is running on the person PC and successful posting/sending is when the app has done it.
Does that count as faster? Is the time measured from when the person hits post/send, or when the person sees the response saying it has been posted/sent?
So in the case of small packets of data, there is an issue of different measurements to compare and this would have to be defined. But in reality its all quick.
As @peca says web page loading is so slow now-a-days because of all the scripts for tracking/advertising/etc that are included in webpages. One advantage Safe has over current server based pages is that images will be coming down in parallel rather than queued in some server’s queue to process.
Now coming to bigger files (greater than say 10MB) Safe will be winning hands down because no longer are the parts of the file coming from the one server and done sequentially. The safe application can request as many chunks as will work for it and all the chunks will be delivered in parallel. So smart buffering for media will be much better performance without buffering if the App is properly written, albeit with a sub second delay on the first chunk due to longer over the wire time for it to arrive.
File download could easily max out your internet link if the download app requests enough chunks in parallel.
EDIT: The future.
As data increases and individual data elements increase in size, Safe is better suited to handle this than the current server-user model.
Imagine when webpages (ignoring the crap added) grow from small (KB to few 100KB) up to large (many MB) or even very large (100’s MB), Safe will provide much faster performance and response times. It is parallel whereas the server system is typically serial (internally and sending the data)
EDIT2: when I say Safe is parallel, I am meaning the requesting of multiple chunks is parallel since they will be coming from different nodes and across different routes (typically). It will still be coming down serially from your ISP’s connection to you.
7 Likes
Watch this video! :) - #439 by JayBird there you are
3 Likes
My website loads in a second. It depends on how you build the website.
1 Like
I should have included some of my reasoning in my initial post. If/When servers are no longer the backbone of the Internet, funds wouldn’t be sunk into specific areas to build huge pipelines and datacenters. The funds then could be used to build up the whole infrastructure and not just specific areas.
I also think a reduction in spam email traffic will be a reduction in Internet junk taking up bandwidth. Popular content being cached will also result in bandwidth being utilized more efficiently.
Will this be immediate? No, but in time I think with bandwidth being utilized in an such an efficient manner, how could it not speed everything up?
***I’ll probably have to edit something because no matter how many times I proofread, i’ll miss something I worded in a way that’s unclear. 
3 Likes
Safe will speed up the Internet in two other ways. No longer will smaller websites be tied down to a certain area. Someone accessing a US site from overseas will incur a lot of latency, and vice versa. The Googles/Facebooks of the world have a much easier time (and funds) setting up an international presence that many can’t afford.
In the same vein, smaller sites will have a much easier time reacting to a large influx of requests. Not having the requirement to have an IT team that can setup a cloud presence that flex’s with traffic needs will cut down costs for some, or provide those benefits to someone who couldn’t previously afford it. This will also speed things up, and may help with keeping sites from becoming overwhelmed and crashing.
4 Likes
Thank you. This is rarely talked about, maybe because plenty of the people who could and should talk about this recent phenomenon are the very people responsible for it.
How almost no-one questions why we can’t scroll down a simple page without glitches on a hardware that allows us to play 4k videos is a mystery.
This blog is not exactly new, but nails it for me:
3 Likes
I’d suggest that applications which optimise using asynchronous requests and lean heavily on immutable types will perform well.
Safe network, like all distributed networks that require multiple hops to reach data, will be subject to higher latencies. They also benefit from fewer bandwidth bottlenecks due to their dispersed nature. Therefore, performing many asynchronous requests to retrieve data simultaneously will be desirable. Likewise, retrieving larger data sets that can be held in the browser/application will mask latency and will benefit from higher bandwidth.
Using immutable types will also be desirable. These can be aggressively cached as they simply cannot change. They could be cached indefinitely in a browser or within the network itself and they will never go stale or need to be retrieved again. For storage space and privacy reasons, some mat choose not to cache locally, but performance would be maximised if aggressive local caching is also performed.
In contrast with the clear net, there is no concept of immutable data at the network layer. Anything cached will always require a TTL (time to live), so that it can periodically be refreshed, in case it has mutated (changed). This impacts on everything from browsers to servers. It results in a minimum of requests to ask if the source data has changed, but often this level optimisation isn’t present beyond static hosted data.
So, this would suggest minimal use of mutual data to reference immutable data where possible. Additionally, pulling dependent immutable data to use just in case it is required, would also improve application responsiveness, while filling caches locally and across the network. This will allow popular applications/sites to scale well.
Caching for mutable types could be interesting too. Even having a short TTL cache would speed things up substantially. If the application/site could hint at how aggressive the caching could be, it would assist further. E.g. a site config map may seldom change, but user comments sequential lists may change frequently.
Archiving mutable data as immutable data at the application level could also help. Replacing mutable lists with immutable lists for older data would give these the benefits described above, but with the obvious trade off (no changes possible).
I suspect we will see various clever designs, which maximise the use of immutable types, while retaining flexibility to change where required. These apps will surely scale very well, with much of their functionality being possible for extended periods with no connectivity to the network at all.
10 Likes
Nicely put and I agree with all of this. We are pushing that all mutable data holds only pointers or 32byte identifiers. All data itself would be immutable. This give us a few good points. I See it like this (and @bochaco / David Rusu are looking at this now, amongst many other points):
- Immutable data must go through self_encryption (bad apps might not do this, but they should be flagged as bad).
- Data is then obscured and nodes store what looks like junk (non identifiable data)
- People cannot “hide” data in mutable data elements
- As mutable data is CRDT then we can use state-based merges and that is powerful. So keeping mutable data small is a win win.
These are the headline points and folk should use mutable data in this manner. It keeps the mutable data small and means that data itself stays forever Safe. It’s an important point that we could easily miss, but we wont.
So a mutable data field could be a pointer to a data map (readable) and that data map points to chunks on the network. That way we stay secure.
11 Likes
@Traktion I’m not clear how mutable data (MD) using CRDTs and local-first capability will behave wrt latency. For example local-first means instant MD, but with latency to reach consensus. A bit like the latency from saving immutable data (ID) until it is actually stored on the network. So it very much depends on the nature of the application whether you experience any network latency at all with MD.
@dirvine Forcing metadata to be ID and for MD only to hold pointers is interesting and even less clear to me how this plays out!
My first thought is about FileTree CRDT although that in fact might be the most straightforward and a case where this works well. I’m not looking to spend time digging in at the moment, so just curious how this works in terms of behaviour. It feels like it will create a lot more small immutable chunks (eg for directory metadata), and inefficiencies that might be hard to optimise later. For example, more chunks to access a directory is more traffic and potentially more latency.
Having mutable data be only pointers has some obvious downsides that I see. Say I’m running a presentation and my presentation file is 1GB with embedded pictures and video. If I make a small change that updates the file, I then have to chunk out, encrypt, and upload another gigabyte and the previous presentation is just a wasted gig file. Now, one could say that the presentation itself should just have pointers to the file path of the pictures and video, but that comes with a host of other issues.
It may be, however, the parallelism of getting chunks will be a big help. We really need to test this, not a huge task to try a few tweaks here.
However if you replace your 1Gb file with some offensive videos, porn etc. and this is all in one large file then we have a few issues
- That 1Gb is gonna come from all 7 Elders in parallel to you
- The mutable data is huge and risking DOSing Elders
- Folk may stop using self_encrypt to upload chnks and just upload plain data to mutable data items. This is hard to spot as say encrypted data looks like jpeg data and so on, so we could end up with some pretty serious issues there.
In terms of small changes to large files and having to re-chunk the whole thing, that is an issue, but I don’t feel it is unsolvable really. There are ways to chunk data into block and each block holding data + padding etc. This could allow changes to data without rechunking. As it stands if you change the data, but not the size it will cost another 3 chunks (changes) per change. I think it’s a different story though.
However, if you had mutable data and stored the whole 1Gb in there as an entry and change it, you will certainly store the whole 1Gb there again and the raspi nodes are gonna die
3 Likes
YEs, this is true, but to be clear this is writes we are talking about, not reads as there is no consensus (NetworkAuthority) for reads.
It’s going to be an interesting few weeks playing with this, at long long last.
2 Likes
I suppose I was more thinking about read performance. While immutable data will certainly be cached readily, mutable data is more difficult. If the request needs to be routed across the network to the Elders responsible, then this will be prone to higher latency and more load on the Elders too.
I’m sure some caching will help here, but it is obviously much harder to cache something which may change. Some network level caching with low TTL may be necessary to prevent Elders becoming overwhelmed, but I suspect it is in the interest of app developers/users to avoid mutable types unless necessary for a good UX.
It depends. It’s been mentioned that MD will work local first, which sounds like it’s not as simple as that, but will have to see what it means in practice.
Maybe some clarity on local-first is needed? I can see local-first writes being feasible. Indeed, from the client code, I can see that the local transfer actor is updated immediately. Presumably, this is so it is available to local apps immediately too.
For writes that happen from other clients, presumably the client will need to request these changes from the Elders though. I’m not sure when/how the local version is invalidated though, but I’ve not gone looking through the code for that yet.
1 Like