@riddim reported the network speed increased yesterday so we have been wondering if someone running a lot of overloaded nodes has exited and improved the network performance?
Seems like certain people could help more by finding something to help with preferably another project 
Weāve recorded the dip at 0630 this morning, which was after riddimās observations. Iām not sure.
Last week, I observed a dip on the 8th at around midnight, which was then restored by about 0800. It could be that person brought more healthy nodes online, but they do seem to have restored the same number of nodes, so Iām not sure if they were over provisioned in the first place.
Do we know the root cause of why mainnet is slower? Is it due to the varied capacity of mainnet nodes, blockchain speed, something else, etc?
Not involved - pure Scratchpad upgrades
Alpha is significantly smaller too ofc as addition discrimination
hmmm - scratchpad update performance is still rather flaky ā¦
error on scratchpad update - time to error: 75.95 seconds
error on scratchpad update - time to error: 37.36 seconds
error on scratchpad update - time to error: 34.68 seconds
error on scratchpad update - time to error: 36.18 seconds
error on scratchpad update - time to error: 17.44 seconds
error on scratchpad update - time to error: 34.89 seconds
error on scratchpad update - time to error: 33.68 seconds
error on scratchpad update - time to error: 37.01 seconds
error on scratchpad update - time to error: 39.97 seconds
Scratchpad-Update time: 10.03 seconds
population time: 0.10 seconds
iterations: 1 - content: b'nKpR\x9c~\xb8\x11\xe1w'...
Scratchpad-Update time: 8.98 seconds
population time: 0.49 seconds
iterations: 1 - content: b'w\x92\x80N\x8cU"\x08\xfa\x02'...
Scratchpad-Update time: 5.16 seconds
population time: 0.09 seconds
iterations: 1 - content: b'\x1c$\x8b\x80\xf3:+\xb0"\x1e'...
Scratchpad-Update time: 24.92 seconds
population time: 0.30 seconds
iterations: 1 - content: b'\xc9\xa7\x0ex\xac\x19\x99\xba\xad\xe4'...
Scratchpad-Update time: 2.83 seconds
population time: 0.09 seconds
iterations: 1 - content: b'\x1f\x90\xec\xd7\x918.\xa5\x93\x9b'...
Scratchpad-Update time: 4.73 seconds
population time: 0.75 seconds
iterations: 1 - content: b'\xaf\xd36\xf4\xda\xc5\xe2\x81\xde\x14'...
error on scratchpad update - time to error: 35.42 seconds
error on scratchpad update - time to error: 33.26 seconds
Scratchpad-Update time: 1.48 seconds
population time: 0.08 seconds
iterations: 1 - content: b'\xc7\x07\xab\x90$\xbb\xd94\x1e\xc3'...
Scratchpad-Update time: 1.94 seconds
population time: 0.08 seconds
iterations: 1 - content: b'\xe7\x9d\x9b2}\xc5\xdfc\xf4\xb0'...
Scratchpad-Update time: 1.53 seconds
population time: 0.09 seconds
iterations: 1 - content: b'\x98\x1cT\xfb\xba=\x8fb}\x89'...
Scratchpad-Update time: 1.37 seconds
population time: 0.09 seconds
iterations: 1 - content: b'\xdagR\xf8\xb0\x90\xac\x9f43'...
somewhere between great speed <2s roundtrip time and multiple minutes to success ā¦
(I guess the <2s should make us excited; we just need to get rid of the repeated errors 
)
yep - results here are mixed - almost as if it forgot how to do it and then learned as it went alongā¦
(venv) willie@gagarin:~/projects/maidsafe/scratch-perf-test$ time python3 ./speedtest.py
Scratchpad-Update time: 88.54 seconds
population time: 3.89 seconds
iterations: 1 - content: b'\xdeC\xd8\xd6\xb1$\xafn\x17)'...
Scratchpad-Update time: 31.30 seconds
population time: 3.25 seconds
iterations: 1 - content: b'\x19Q\xfc\xc9K\xba\x1a%H!'...
Scratchpad-Update time: 15.26 seconds
population time: 0.53 seconds
iterations: 1 - content: b'\xab%\xcb\xd1\xcf(\x1d\x12\x10\xd0'...
Scratchpad-Update time: 3.27 seconds
population time: 0.52 seconds
iterations: 1 - content: b'\x18\xad\xb4\xb0(*\xfc\x8eOf'...
Scratchpad-Update time: 4.46 seconds
population time: 0.32 seconds
iterations: 1 - content: b'}q$\xa6\x13\xb3l\xf1\xe4\xe0'...
Scratchpad-Update time: 5.82 seconds
population time: 0.33 seconds
iterations: 1 - content: b'\x8b\xa8\xd9Z\x0b}n=\x04Y'...
Scratchpad-Update time: 4.45 seconds
population time: 0.28 seconds
iterations: 1 - content: b'\xbc\x04\x89\xd1\xc0R~=:]'...
Scratchpad-Update time: 3.40 seconds
population time: 0.28 seconds
iterations: 1 - content: b'\xf6\xf8\xa1\x8b\xbcHQ\xfc\x160'...
Scratchpad-Update time: 3.93 seconds
population time: 0.28 seconds
iterations: 1 - content: b'\x1ai\xac\x1f\xaeT\xa60P\x10'...
Scratchpad-Update time: 1.84 seconds
population time: 0.28 seconds
iterations: 1 - content: b'P\xdb\xfa\xaf\xcbmN\xbc\x11/'...
real 2m53.269s
user 0m3.313s
sys 0m1.544s
and then when I tried another run
(venv) willie@gagarin:~/projects/maidsafe/scratch-perf-test$ time python3 ./speedtest.py
Traceback (most recent call last):
File "/home/willie/projects/maidsafe/scratch-perf-test/./speedtest.py", line 40, in <module>
asyncio.run(main())
File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/home/willie/projects/maidsafe/scratch-perf-test/./speedtest.py", line 35, in main
await speedtest(500)
File "/home/willie/projects/maidsafe/scratch-perf-test/./speedtest.py", line 15, in speedtest
await client_mainnet.scratchpad_update(testpad_privatekey, 0, randomdata)
RuntimeError: Failed to update scratchpad: Failed to put scratchpad: A network error occurred for NetworkAddress::ScratchpadAddress(8e2d0725e1ef3847b21f1e8c2e37c28797717fbff4139d1fff74ff17162fc9f9473f96149c21f3b8f47d38847de3fbd7): Put verification failed: Peers have conflicting entries for this record: {PeerId("12D3KooWMFva5yQQrBxWVWGjgu5r6aMbWYsDVpyGsr86S9BDmUrD"): Record { key: Key(b"d\xab\xb0\xb8\xf2\x18Q\xd36\xb2`\x90&\xda\xab\xe5\x06\x05\xe7\xabA\xa8\xf6\xc2^\x17%1\x80e\xaa?"), value: [145, 3, 149, 220, 0, 48, 204, 142, 45, 7, 37, 204, 225, 204, 239, 56, 71, 204, 178, 31, 30, 204, 140, 46, 55, 204, 194, 204, 135, 204, 151, 113, 127, 204, 191, 204, 244, 19, 204, 157, 31, 204, 255, 116, 204, 255, 23, 22, 47, 204, 201, 204, 249, 71, 63, 204, 150, 20, 204, 156, 33, 204, 243, 204, 184, 204, 244, 125, 56, 204, 132, 125, 204, 227, 204, 251, 204, 215, 0, 196, 194, 166, 157, 123, 80, 126, 183, 118, 153, 150, 84, 22, 127, 31, 203, 193, 62, 204, 50, 216, 85, 168, 255, 201, 180, 236, 1, 66, 16, 252, 38, 0, 211, 54, 200, 252, 254, 89, 27, 18, 139, 23, 64, 118, 165, 164, 162, 2, 101, 133, 151, 7, 83, 21, 134, 231, 133, 170, 64, 185, 118, 107, 120, 193, 43, 26, 131, 218, 23, 110, 227, 14, 140, 49, 208, 219, 164, 116, 103, 234, 56, 44, 138, 207, 93, 229, 79, 211, 234, 126, 88, 175, 230, 127, 175, 247, 121, 18, 202, 45, 244, 70, 99, 77, 100, 69, 126, 232, 15, 239, 214, 100, 169, 26, 231, 30, 186, 7, 77, 73, 4, 201, 155, 86, 110, 137, 76, 123, 82, 68, 217, 187, 227, 34, 227, 199, 88, 152, 184, 88, 234, 95, 95, 173, 172, 57, 89, 240, 155, 40, 209, 141, 193, 201, 178, 19, 246, 148, 92, 228, 89, 42, 182, 169, 24, 44, 108, 140, 6, 63, 210, 169, 255, 112, 105, 94, 192, 218, 53, 169, 65, 118, 163, 246, 129, 121, 251, 206, 253, 45, 111, 76, 20, 32, 22, 45, 220, 0, 96, 204, 144, 204, 138, 204, 237, 88, 204, 169, 122, 204, 187, 204, 153, 204, 169, 204, 188, 69, 37, 14, 32, 112, 204, 209, 204, 164, 62, 204, 153, 95, 97, 204, 143, 7, 118, 204, 149, 204, 226, 30, 9, 110, 204, 208, 105, 204, 182, 204, 201, 127, 204, 232, 79, 107, 204, 225, 204, 221, 204, 232, 204, 214, 63, 28, 204, 224, 204, 216, 204, 202, 72, 106, 9, 204, 160, 47, 17, 204, 210, 204, 254, 88, 104, 204, 153, 204, 164, 204, 229, 72, 204, 168, 127, 204, 207, 204, 148, 34, 111, 106, 109, 119, 204, 243, 204, 174, 10, 204, 141, 32, 61, 114, 204, 167, 57, 49, 28, 204, 165, 95, 204, 185, 204, 248, 2, 5, 204, 229, 56, 204, 137, 99, 98, 66, 60, 5, 54, 204, 230], publisher: None, expires: None }, PeerId("12D3KooWPV6kpbp2SY8N2JQdUmyeVBCNCnMYqpiFbWkre6PpCrFy"): Record { key: Key(b"d\xab\xb0\xb8\xf2\x18Q\xd36\xb2`\x90&\xda\xab\xe5\x06\x05\xe7\xabA\xa8\xf6\xc2^\x17%1\x80e\xaa?"), value: [145, 3, 149, 220, 0, 48, 204, 142, 45, 7, 37, 204, 225, 204, 239, 56, 71, 204, 178, 31, 30, 204, 140, 46, 55, 204, 194, 204, 135, 204, 151, 113, 127, 204, 191, 204, 244, 19, 204, 157, 31, 204, 255, 116, 204, 255, 23, 22, 47, 204, 201, 204, 249, 71, 63, 204, 150, 20, 204, 156, 33, 204, 243, 204, 184, 204, 244, 125, 56, 204, 132, 125, 204, 227, 204, 251, 204, 215, 0, 197, 2, 132, 181, 62, 152, 98, 246, 4, 40, 0, 51, 184, 102, 166, 88, 44, 62, 78, 236, 147, 127, 181, 107, 66, 224, 69, 110, 175, 116, 101, 178, 82, 61, 109, 154, 164, 42, 244, 130, 173, 15, 104, 207, 53, 78, 169, 76, 166, 97, 138, 141, 151, 209, 63, 136, 93, 130, 168, 176, 180, 143, 241, 75, 102, 76, 110, 58, 38, 127, 132, 146, 25, 38, 182, 212, 239, 232, 18, 49, 149, 196, 10, 96, 74, 47, 203, 219, 7, 198, 16, 171, 11, 56, 191, 215, 35, 37, 34, 10, 106, 11, 86, 15, 116, 91, 96, 38, 63, 81, 157, 183, 30, 64, 75, 142, 70, 30, 68, 40, 96, 217, 39, 128, 158, 204, 124, 147, 11, 20, 160, 47, 7, 195, 146, 232, 97, 200, 22, 57, 104, 68, 23, 56, 168, 184, 66, 166, 118, 229, 236, 202, 250, 20, 45, 73, 189, 101, 133, 121, 253, 179, 67, 222, 37, 53, 79, 47, 152, 245, 86, 180, 68, 163, 101, 162, 111, 162, 23, 211, 138, 169, 64, 217, 106, 250, 145, 40, 32, 187, 165, 146, 244, 38, 129, 86, 34, 17, 255, 182, 135, 224, 205, 240, 144, 84, 39, 140, 5, 243, 221, 170, 168, 16, 93, 61, 242, 27, 191, 112, 214, 71, 5, 218, 3, 105, 0, 84, 229, 189, 104, 154, 166, 99, 161, 192, 152, 237, 81, 86, 129, 50, 46, 44, 226, 89, 195, 147, 189, 217, 134, 172, 23, 182, 62, 180, 115, 245, 197, 113, 123, 56, 176, 148, 199, 1, 211, 190, 101, 234, 171, 38, 202, 4, 253, 178, 64, 117, 4, 213, 96, 37, 224, 155, 255, 15, 237, 239, 192, 24, 115, 203, 135, 181, 217, 191, 195, 248, 211, 200, 70, 230, 41, 91, 219, 54, 245, 183, 144, 68, 53, 71, 101, 161, 230, 102, 103, 90, 69, 212, 153, 2, 77, 14, 254, 210, 223, 118, 236, 49, 52, 162, 129, 175, 209, 50, 18, 226, 62, 193, 65, 30, 69, 214, 167, 202, 118, 92, 118, 47, 171, 137, 249, 165, 104, 165, 21, 240, 242, 126, 217, 5, 47, 81, 252, 152, 56, 140, 172, 128, 226, 58, 106, 200, 39, 9, 14, 85, 57, 185, 19, 230, 95, 141, 69, 24, 255, 71, 163, 153, 56, 209, 83, 230, 219, 211, 193, 105, 239, 209, 33, 59, 199, 53, 173, 224, 176, 219, 47, 169, 196, 131, 80, 105, 164, 82, 95, 165, 217, 181, 43, 199, 54, 147, 63, 156, 157, 40, 99, 18, 56, 124, 93, 164, 179, 206, 103, 70, 137, 143, 35, 233, 113, 81, 238, 66, 18, 102, 83, 129, 27, 230, 83, 2, 58, 199, 48, 248, 213, 5, 15, 252, 153, 102, 112, 206, 140, 69, 31, 37, 190, 243, 239, 164, 39, 133, 217, 95, 4, 112, 106, 78, 117, 95, 58, 190, 213, 22, 106, 143, 10, 27, 66, 221, 235, 8, 26, 35, 59, 60, 5, 3, 159, 137, 171, 74, 125, 249, 85, 50, 62, 61, 91, 202, 68, 17, 198, 105, 251, 184, 241, 214, 183, 184, 202, 59, 21, 70, 48, 247, 69, 109, 169, 23, 146, 254, 2, 31, 129, 131, 130, 117, 144, 70, 68, 1, 102, 246, 109, 24, 86, 253, 245, 169, 88, 113, 105, 166, 99, 202, 6, 107, 100, 122, 172, 108, 126, 250, 12, 163, 118, 163, 78, 238, 164, 39, 225, 190, 48, 28, 76, 50, 89, 242, 189, 188, 128, 185, 125, 241, 203, 32, 158, 254, 37, 172, 252, 163, 135, 252, 91, 88, 217, 107, 18, 23, 160, 244, 50, 235, 239, 122, 138, 166, 88, 158, 82, 30, 115, 214, 231, 252, 246, 10, 26, 35, 177, 177, 120, 241, 205, 231, 184, 155, 133, 98, 218, 215, 8, 53, 158, 32, 98, 202, 34, 223, 194, 205, 1, 106, 220, 0, 96, 204, 146, 74, 204, 181, 204, 230, 204, 223, 204, 234, 204, 160, 111, 39, 1, 86, 204, 170, 35, 99, 204, 182, 37, 204, 139, 119, 204, 204, 54, 204, 158, 16, 204, 151, 22, 75, 204, 135, 55, 204, 147, 30, 204, 254, 204, 197, 125, 72, 204, 215, 80, 204, 228, 204, 215, 54, 83, 49, 118, 87, 204, 241, 111, 67, 204, 165, 204, 149, 52, 24, 204, 224, 6, 204, 182, 204, 188, 51, 204, 220, 204, 219, 127, 204, 180, 204, 152, 204, 185, 113, 25, 15, 204, 204, 204, 137, 204, 151, 204, 219, 204, 208, 31, 13, 54, 68, 204, 208, 204, 157, 204, 142, 36, 204, 138, 204, 229, 204, 244, 100, 204, 194, 31, 54, 124, 98, 67, 77, 204, 146, 48, 53, 91, 5, 204, 158, 204, 223, 15, 99], publisher: None, expires: None }}
real 0m40.130s
user 0m1.035s
sys 0m0.544s
That was followed by further errors
RuntimeError: Failed to update scratchpad: Failed to put scratchpad: A network error occurred for NetworkAddress::ScratchpadAddress(8e2d0725e1ef3847b21f1e8c2e37c28797717fbff4139d1fff74ff17162fc9f9473f96149c21f3b8f47d38847de3fbd7): Put verification failed: Peers have conflicting entries for this record: {PeerId("12D3KooWEYj9rCvF5Yuv5ufwy9vQQZoFNYk1VeumiHS61zzUJtse")
and a somewhat more successful but still pretty varied set of results
(venv) willie@gagarin:~/projects/maidsafe/scratch-perf-test$ time python3 ./speedtest.py
Scratchpad-Update time: 14.56 seconds
population time: 0.46 seconds
iterations: 1 - content: b'\xf9\xa3\x83\x10\xda\xe1#\x86\xb3\x85'...
Scratchpad-Update time: 36.04 seconds
population time: 0.80 seconds
iterations: 1 - content: b'L+\xc4\xd2\x8fs\xa5\x17fD'...
Scratchpad-Update time: 12.08 seconds
population time: 1.17 seconds
iterations: 1 - content: b'<SR\xd6\xe6a\xae\x1eC?'...
Scratchpad-Update time: 6.56 seconds
population time: 0.90 seconds
iterations: 1 - content: b'\xea\xa5/\xacw\xc8\xafl\xe36'...
Scratchpad-Update time: 9.86 seconds
population time: 0.82 seconds
iterations: 1 - content: b'GVH\xe9K\xaa\xdb\xba\xd8\xd0'...
Scratchpad-Update time: 6.80 seconds
population time: 1.11 seconds
iterations: 1 - content: b'\xa3\xc4\xdatZ4\xfb\xe1\xab\x12'...
Scratchpad-Update time: 3.51 seconds
population time: 0.48 seconds
iterations: 1 - content: b'\xe9\xae>,?CP\x90\xb3\xef'...
Scratchpad-Update time: 5.84 seconds
population time: 0.66 seconds
iterations: 1 - content: b'\xa6m\xd0\x8fdF\x1a\xee\x18\xcb'...
Scratchpad-Update time: 2.84 seconds
population time: 0.48 seconds
iterations: 1 - content: b'(\xdb2\xdf\x93\x00\xc8\xd5I]'...
Scratchpad-Update time: 3.53 seconds
population time: 0.66 seconds
iterations: 1 - content: b'\xb0z\x89\xcc\xda\xb2\x14\x9c1\x1a'...
real 1m49.494s
user 0m2.619s
sys 0m1.215s
yes because I got those errors too often Iāve a modded version of the script now that outputs time to error and doesnāt stop the process:
import asyncio
import time
from autonomi_client import SecretKey, Client, ScratchpadAddress
from secrets import token_bytes
async def main() -> None:
client_mainnet = await Client.init()
testpad_privatekey = SecretKey.from_hex("3a728f71282bd68fcbc6da9e335b46c1122e30eb4dc1ab4ba5c96ef65d365f47")
async def speedtest(byte_length: int) -> None:
randomdata: bytes = token_bytes(byte_length)
start_time: float = time.time()
try:
await client_mainnet.scratchpad_update(testpad_privatekey, 0, randomdata)
except:
update_time = time.time() - start_time
print(f"error on scratchpad update - time to error: {update_time:.2f} seconds")
await speedtest(byte_length)
return
update_time: float = time.time() - start_time
print(f"Scratchpad-Update time: {update_time:.2f} seconds")
tmp: int = 1
start_time = time.time()
while True:
try:
testpad = await client_mainnet.scratchpad_get(ScratchpadAddress.from_hex(testpad_privatekey.public_key().hex()))
except:
print('error fetching scratchpad')
continue
content: bytes = testpad.decrypt_data(testpad_privatekey)
if content == randomdata:
population_time: float = time.time() - start_time
print(f"population time: {population_time:.2f} seconds")
print(f"iterations: {tmp} - content: {content[:10]}...")
break
tmp += 1
for _ in range(10):
await speedtest(500)
print()
if __name__ == "__main__":
asyncio.run(main())
(copy+pasted in the mods - hope itās correct - my test didnāt return usable results - but possibly because we were running mods on the same scratchpad in parallel )
Iāll copy that script and run it on a cron job from a couple of different locations
but not the same moment from different locations - otherwise you need to put other scratchpads and use different secret-keys - if you write from 2 locations at the same time it may result in an endless-loop on reading (trying to get the written value - which was overwritten already by your other process)
Flaky is a good description of this performance right now. These results are all from home on 500Mb fibre
(venv) willie@gagarin:~/projects/maidsafe/scratch-perf-test$ time python3 ./speedtest-riddim.py
error on scratchpad update - time to error: 41.93 seconds
Scratchpad-Update time: 10.37 seconds
population time: 0.68 seconds
iterations: 1 - content: b')d*\x92\x14\x12\x999\xe2='...
Scratchpad-Update time: 4.75 seconds
population time: 0.75 seconds
iterations: 1 - content: b'\xcd}\x8b\xc9c\x9d&\x12\xa1\xd9'...
Scratchpad-Update time: 5.93 seconds
population time: 0.59 seconds
iterations: 1 - content: b'm\xeb\xbcP\xaa\xfdj\xd76\xb9'...
error on scratchpad update - time to error: 45.28 seconds
error on scratchpad update - time to error: 38.54 seconds
Scratchpad-Update time: 12.86 seconds
population time: 1.19 seconds
iterations: 1 - content: b'\xf4\xcfh\xa0wIr\x83\xa9\x99'...
Scratchpad-Update time: 11.94 seconds
population time: 0.66 seconds
iterations: 1 - content: b'L\x0b\x95\xcfb\x98;\x05RO'...
error on scratchpad update - time to error: 35.96 seconds
error on scratchpad update - time to error: 34.10 seconds
error on scratchpad update - time to error: 34.10 seconds
error on scratchpad update - time to error: 35.12 seconds
error on scratchpad update - time to error: 34.17 seconds
error on scratchpad update - time to error: 34.60 seconds
error on scratchpad update - time to error: 33.36 seconds
error on scratchpad update - time to error: 33.92 seconds
Scratchpad-Update time: 16.40 seconds
population time: 0.18 seconds
iterations: 1 - content: b'Lo\x84)\xb1\xd94\x03"\x7f'...
Scratchpad-Update time: 6.06 seconds
population time: 0.18 seconds
iterations: 1 - content: b'\xba\x99\xc5V@\x95\x1d\x8b\x0e\xf4'...
Scratchpad-Update time: 34.44 seconds
population time: 0.21 seconds
iterations: 1 - content: b'\x1b\x9c\xa2l\x1a2\xbf\xac&5'...
Scratchpad-Update time: 3.42 seconds
population time: 0.23 seconds
iterations: 1 - content: b'Y\\c\xa5\x14\x87,N\xe4\x81'...
Scratchpad-Update time: 2.87 seconds
population time: 0.29 seconds
iterations: 1 - content: b'\x9a\x91\xad\xc0\xcb\xe7\xb0o]\x06'...
real 8m35.370s
user 0m4.194s
sys 0m1.839s
About the only definitive result that can be drawn right now is that if it is going to fail, it will do so in 30-45 secs with a median of ~36 secs.
but when its fast , its bloody quick
Scratchpad-Update time: 2.03 seconds
population time: 0.14 seconds
iterations: 1 - content: b'\xa9F];\xec>\xb4}g\x7f'...
would love to see a mainnet with consistent speeds like this 
Err - this IS mainnetā¦
I said I was going to try Friends on alphanet. Never got around to it - too much fun to be had this weekend for that kind of malarkeyā¦
and itās not consistent - right? 
so mainnet may be awesome when we get rid of those errors! 
goooooood! It will be awesome!
Iāve testing an integration of a LRU cache for immutable chunks. Even with a sickly network, once you have the chunks in cache, performance then goes off the charts!
$ cat ~/dev/anttp/test/performance/src/localhost-autonomi-http.js; k6 run -u 10 -i 1000 ~/dev/anttp/test/performance/src/localhost-autonomi-http.js
import http from 'k6/http';
export default function () {
http.get('http://localhost:18888/cec7a9eb2c644b9a5de58bbcdf2e893db9f0b2acd7fc563fc849e19d1f6bd872/1_QdxdljdwBwR2QbAVr8scuw.png', { timeout: '600s' });
http.get('http://localhost:18888/cec7a9eb2c644b9a5de58bbcdf2e893db9f0b2acd7fc563fc849e19d1f6bd872/1_dH5Ce6neTHIfEkAbmsr1BQ.jpeg', { timeout: '600s' });
http.get('http://localhost:18888/cec7a9eb2c644b9a5de58bbcdf2e893db9f0b2acd7fc563fc849e19d1f6bd872/1_pt48p45dQmR5PBW8np1l8Q.png', { timeout: '600s' });
http.get('http://localhost:18888/cec7a9eb2c644b9a5de58bbcdf2e893db9f0b2acd7fc563fc849e19d1f6bd872/1_sWZ4OWGeQjWs6urcPwR6Yw.png', { timeout: '600s' });
http.get('http://localhost:18888/cec7a9eb2c644b9a5de58bbcdf2e893db9f0b2acd7fc563fc849e19d1f6bd872/1_ZT6qplX5Yt8PMCUqxq1lFQ.png', { timeout: '600s' });
http.get('http://localhost:18888/cec7a9eb2c644b9a5de58bbcdf2e893db9f0b2acd7fc563fc849e19d1f6bd872/1_SxkGLnSNsMtu0SDrsWW8Wg.jpeg', { timeout: '600s' });
http.get('http://localhost:18888/cec7a9eb2c644b9a5de58bbcdf2e893db9f0b2acd7fc563fc849e19d1f6bd872/1_bogEVpJvgx_gMHQoHMoSLg.jpeg', { timeout: '600s' });
http.get('http://localhost:18888/cec7a9eb2c644b9a5de58bbcdf2e893db9f0b2acd7fc563fc849e19d1f6bd872/1_LFEyRQMHmxRnZtJwMozW5w.jpeg', { timeout: '600s' });
}
/\ Grafana /ā¾ā¾/
/\ / \ |\ __ / /
/ \/ \ | |/ / / ā¾ā¾\
/ \ | ( | (ā¾) |
/ __________ \ |_|\_\ \_____/
execution: local
script: /home/paul/dev/anttp/test/performance/src/localhost-autonomi-http.js
output: -
scenarios: (100.00%) 1 scenario, 10 max VUs, 10m30s max duration (incl. graceful stop):
* default: 1000 iterations shared among 10 VUs (maxDuration: 10m0s, gracefulStop: 30s)
data_received..................: 23 MB 700 kB/s
data_sent......................: 1.4 MB 42 kB/s
http_req_blocked...............: avg=5.64Āµs min=1.32Āµs med=3.76Āµs max=947.23Āµs p(90)=7.86Āµs p(95)=9.65Āµs
http_req_connecting............: avg=589ns min=0s med=0s max=636.95Āµs p(90)=0s p(95)=0s
http_req_duration..............: avg=41.34ms min=2.59ms med=41.05ms max=65.51ms p(90)=42ms p(95)=42.44ms
{ expected_response:true }...: avg=41.34ms min=2.59ms med=41.05ms max=65.51ms p(90)=42ms p(95)=42.44ms
http_req_failed................: 0.00% 0 out of 8000
http_req_receiving.............: avg=40.76ms min=1.3ms med=40.66ms max=44.92ms p(90)=41.62ms p(95)=41.74ms
http_req_sending...............: avg=17.56Āµs min=5.1Āµs med=12.92Āµs max=1.53ms p(90)=31.52Āµs p(95)=37.61Āµs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=566.74Āµs min=147.66Āµs med=457.44Āµs max=25.35ms p(90)=875.39Āµs p(95)=1.01ms
http_reqs......................: 8000 241.022412/s
iteration_duration.............: avg=331.67ms min=295.75ms med=331.29ms max=386.84ms p(90)=334.08ms p(95)=335.79ms
iterations.....................: 1000 30.127802/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10
running (00m33.2s), 00/10 VUs, 1000 complete and 0 interrupted iterations
default ā [======================================] 10 VUs 00m33.2s/10m0s 1000/1000 shared iters
For reference, it was taking 10 minutes to to about 600 iterations before. With this change, it takes about 30 seconds to do 1000!
Ofc, it should be quick - the files are coming from memory (16 MB) and local disk (1 GB) with my testing defaults.
Still, given immutable data will never change, it would be nice to have a massive local cache to store them. Then you only need the network for stuff you donāt have already.
FYI, this LRU cache will cache every chunk that passes through chunk streamer, along with a few other sources too. For videos, this can make fast forward/rewind super quick and jumps to anywhere in the buffer become near instant. Pretty cool, right?
Iāll do some more testing and then get a new formal AntTP release out. For the brave, the source code is already committed, but Iām still testing for memory leaks, etc.
(EDIT: ofc, it also paves the way for tarchives, with the same chunks being read with different offset/limitsā¦ that will come next! )
(EDIT2: I think there is something going on with some public archives not deserialising too, from looking at the logs. Anyway, a taste of things to come, at least. Need a stable main-net to really try it out)
Sorry, can you please provide details on this?
Yes, will do when I have something repeatable. I almost posted on discord support earlier, but I wasnāt sure it it was my changes or the update. It was slow going to retry too! 
OK thanks. Among the tests for every release is a check that newer clients can download files that were uploaded with older versions, and I didnāt see any serialization problem, so Iām very keen to know what is going on here.