Yes we are in the droplets right now. We set a huge logging framework there and logs are hundreds of MBytes right now. It gives us the info we want though so we are debating bringing it down to stop the logging and allow us to trace all the calls etc.
Seems like disjoint sections with resource proof does work though, but crust is not traversing NAT properly maybe due to our late night changes on Thursday. We may restart next week, but first we need to take the time to analyse the massive logs and confirm candidate approval uses NAT traversal properly. We already identified a few things we need to attend to right now (not necessarily blockers).
- Routing missed sending a split event to vault (simple routing invalid log message)
- We have several
UnknownPeerStatemessages that we wish to clear. - We are currently able to see some droplets are tunnelling for 40 connections along with the other 60 they are holding. Tunnels should be at zero at this stage of test. This may require a tweak to resource proof to confirm connection is available to external machines.
We are not 100% comfortable to keep filling the logs up whilst these tunnels will stall the network. We think the network is stalling and repairing, but should not require to be repairing at this rate at this stage. We are unsure there is data loss but this is a part of the logs we want to confirm as well.
Thanks again everyone for the very quick test, just give us a wee while to analyse the result and keep on track with vaults from home and on to data republish. So we will shut this down at midnight tonight (UTC).