I’ve been finding it difficult to really get my hands dirty with safe_vault because the main interface we have is the log files, which are great but difficult to use especially when there are multiple of them as there always are.
So I made log_viewer_for_safe_vaults. Code is very prototype (read: messy).
I’m still not totally sure how it will be useful, and features are very thin just now but will be expanded as needed (so if you have suggestions I’d love to hear them).
Some things I’ve found from it so far that would be extremely difficult just from the logs.
Aggregated log lines
Having all log lines in chronological order is extremely helpful since it shows the connections of events between vaults much more clearly than side-by-side logs.
Visual Patterns
Seeing visually the density and arrangement of data on the timeline is very helpful.
Groups of points in a vertical line show ‘simultaneous’ events.
Single long horizontal lines show possible stalling points.
Dense clusters with large white space between show inactivity (sometimes intentional, sometimes not).
Vertical time marker shows which events happen before / after that time, which is unusually helpful when looking at lots of very closely spaced events.
Examples
From the example images above, some things that are seen which are not easily done from logs.
First image:
Vaults are seen to join at 20s intervals, one at a time, and each join event takes a little longer than the one before. This means that I probably don’t want to set a very rapid join rate or these would begin to overlap. This would be extremely hard to tell just by looking at logs.
The last vaults to join generate very little activity, suggesting elders + parsec may be the primary factor in workload during network start.
There’s a long gap of no activity. Vaults do not seem to be doing ‘idle work’, they really do seem to be at rest when nothing is happening. If I was seeing continuous log messages I would wonder why and be able to investigate.
At the end some burst of activity that only affects elders. Makes me wonder about the joining / status of the non-elder vaults. Again, very hard to see this easily from just log files.
Second image (zoomed into the time where the blue vault at the bottom of the timeline is joining):
The activity during vault joining is not dominated by any one vault.
It took about 2.5s (see the heading).
Messages seem to be mostly around 10-30 ms apart, there’s not a lot of parallel activity except just prior to the time marker.
There’s a few darker dots (which indicate multiple very closely timed events) which may be of interest.
Most of all, the tool is just interesting to play with. As I play with it more I’m sure other useful things will come up. Hopefully this tool is useful to others, and if you feel anything is missing I’m keen to add more stuff to make it extra useful.
I’m thinking a filter to show/hide specific messages will be essential?
Maybe some kind of lines joining related events so the flow of, say, one chunk upload becomes clearer?
Maybe the need for new log messages will become more apparent?
Maybe include client logs as well as vault logs?
Who knows what else…
I will be getting this set up locally!
and now they are super busy, I dont see this getting sorted soon.
