This test is around performance of vaults with structured data. I admit to being slightly provocative because it may be compared to the bitcoin transaction rate (which is currently under some heavy contention).
The test is to modify structured data as rapidly as possible. This should hopefully find the maximum transaction rate for my personal safe network (which isn’t especially powerful).
Motivation
SafeCoin will be implemented as a structured data, and is intended to scale very efficiently, both in cost to the end user and load to the network.
Since the cost to update structured data is zero, it’s worth investigating the impact of high loads on the network, and what the transaction rate of SafeCoin may be.
Setup
Using Test 11 software versions, which is required for low level apis.
- safe_launcher 0.9.2
- safe_core 0.22.0
- safe_vault 0.12.1 (assumed from release date of 2016-10-19)
- routing 0.27.1
- custom script to modify structured data
Modifications were only to remove check for peers on LAN. Account limits don’t matter for this test, since updates to SD don’t count toward usage. Group size is 3 and Quorum size is 2.
Hardware for the network is same as all prior tests:
- Network of 7 pine64s running 4 vaults each on gigabit ethernet
- Client is laptop with quadcore i7-4500U @ 1.80GHz, 8GB RAM uploading on wifi at 300 Mbps
Method
The general idea is to rapidly modify structured data until an issue comes up.
The test script does the following:
- Create a structured data containing 30 random characters
- Measure the time taken to update that data X times with 30 new random characters
- Repeat several times to obtain an average update rate
Results
Update Rate
How long does it take to update SD, and how many updates / second can be achieved?
Whether updating 1 time or updating 100 times in rapid succession, the rate was usually between 120 - 130 updates per second.
However there was some variation where up to 300 u/s were seen.
The good news is that increasing the load did not cause any increase in update rate (as expected).
CPU load across all vaults was negligible.
CPU load on the client was 100% of one core (presumably due to the single-threaded nature of the launcher and safe_core).
Maximum Updates and Failure Point
The maximum number of updates before an error showed was about 1000 updates in a row.
The error returned is simply ‘EOF’, so the launcher server seems to drop requests (ie not the vaults flaking out). This is good news in one way (the launcher is a relatively easy fix) but bad in others (the launcher should be more stable than this). The other aspect to this bottleneck is it makes it difficult to test the limits of vaults with respect to DOS using SD updates.
I had intended to test many concurrent requests to the launcher, but it threw errors with only a low synchronous load so didn’t see much point pushing it further.
Hopefully the revamped authenticator and safe_core will expose a more robust interface to the network.
Conclusions
The update rate for structured data is between 100 - 300 updates per second per user (on my private safe network).
The current bottleneck to increasing this is the launcher, which throws errors when the load becomes too high (around 1000 updates in a row).
It costs nothing to update structured data, and it can be updated at a fairly fast rate. This leads to the notion that some vault rate limiting may be required when dealing with structured data updates.
I’m not terribly satisfied with this test, as it doesn’t really profile the vaults at all. But it satisfied a curiosity about the approximate order of magnitude to expect for structured data performance.
Asides
- The launcher UI froze after running these tests. The server was still running but the tabs wouldn’t change. I assume this is from the logging tab which has a lot of work to do in a very short time. Once the UI locked it never recovered, needing to restart the launcher to make the UI work again.
- The value of the SD was not checked, so it’s assumed it was updated to contain the correct value and there were no race conditions. This is a pretty big assumption, but these details were outside the scope of this test.
- I didn’t test this on the live test 11 network.
- The rates in this test are for a single user, and the overall network transaction rate would presumably be much higher with many concurrent users. I didn’t get into modifying the launcher to test this, mainly due to the imminent overhaul to safe_launcher > authenticator and structured_data > mutable_data.
- Other simultaneous upload activity such as immutable data may affect this rate in the real world.