Notice in the graph above that during 36 minutes (from to 23:13 to 23:49) the written_bytes curve (magenta color) is flat at 1 MB. When I change the end point in Grafana, I observe that “Used space MB” remains unchanged at 13 MB all along this region even though written_bytes is not null.
Apparently the chunks written in this region are not counted.
I’m impressed by the shininess of your Grafana set up, Im also impressed by the power and ubquity of the ELK stack.
Are they mutually exclusive?
The reason I ask is that I am looking at adding an option to the comnet-install script that will add filebeat and a config.xml to point at an ELK stack that I’m told @yogesh is building. The downside to this is that filebeat needs java and this turns what was meant to be a simple wee script to into somewhat more involved installation exercise and casually dumping java onto a n00bs box is not good manners in my book. So the report back to Maidsafe with the Comnet logs has to optional. So really I want to look at all the alternatives before I possibly waste time on a dead end.
EDIT @chriso has suggested I look at a dockerised filebeat so thats being considered too which would simplify the install in some respects and may complicate in others.
They are different stacks but they may be interoperable at some points. For example there is a Telegraf plugin that can read metrics from Elastic Beat (=> reading logs your way and visualizing them my way) and there is another one that can write to Elasticsearch (=> reading logs my way and visualizing them your way).
Note that the complete stack I use is:
Initially I didn’t use Grafana but used InfluxDB own visualization solution but I didn’t find a file format which could be used to both export and import dashboards in both InfluxDB OSS and InfluxDB Cloud. Then I found out that Grafana could do that. And my experience with Grafana is better, specially its overrides systems which is more powerful.
The stack I use doesn’t need programming but only configuration files.
Its all getting a bit complex for what was meant to be a simple get-you-going script.
I’ll wait to hear what @yogesh has done so far. At the end of the day, the logs are primarily for the devs to analyse so its best (I think) to send them to a tool/stack they are comfortable with. They will be using Kibana for visualisation and while I dont have great experience of either Kibana nor Grafana they look pretty similar to me.
I need to read more on the Telegraf plugins that can write to Elasticsearch.
Then the question arises - If Im using docker to deliver Filebeat/Telegraf, what else can/should I be using it for in this context?
EDIT: And how much of this is valid for podman as well?
Although interesting, at the moment, I don’t really see much advantage in running sn_node in a container, so I’ve not really paid attention to what’s going on with Podman. I would guess though, just like Kubernetes, it will manage Docker containers. So any solution you developed for this, should be re-usable with that setup.
Since the network is implemented in Rust and compiled with musl, we have virtually no dependencies. Docker is great when you have to distribute something with dependencies, like say a Java application like Tomcat or Apache. It can also be helpful for distributing lots of config. The only real good use case I can see for distributing sn_node in a container at the moment would be to distribute permanent configuration with it. However, we don’t have any permanent networks at the moment.
In the future though, I’m sure people will want to run nodes on Kubernetes and whatnot, so it is a legitimate scenario. At this stage, in my opinion, it’s just a bit of a novelty. In general, many people run stuff in containers just for the sake of it.
Looking at the Dockerfile for Filebeat though, it does actually look like possibly a good case for distributing in a container, as there’s a bit of setup there. If you had a config file for it, you could just mount that in on your docker run. The fact that you can run Docker on Windows now could also mean you’d have a cross platform solution there too.
I would say Docker itself isn’t too much of a huge dependency, but I guess you’d need to make your own assessment. You can detect an install of it in your script and then do the Filebeat stuff if a user has it, but don’t bother with it otherwise.
Assuming we DON’T have a local Kibana install the config file should be fairly straightforward
# filestream is an input for collecting log messages from files.
- type: filestream
# Unique ID among all inputs, an ID is required.
id: my-filestream-id <-- need to work this out
# Change to true to enable this input configuration.
enabled: false
# Paths that should be crawled and fetched. Glob based paths.
paths:
~/.safe/node/local-node/sn_node.log
~/.safe/node/baby-fleming-nodes/sn_node*/sn_node.log
# windowws log paths here
We need to fill in the blanks here…
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["Yogesh-ELK-server-ip:9200"]
# Protocol - either `http` (default) or `https`.
#protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key" <-- Can @Yogesh share this?
#username: "elastic" <-- or do we need to get testers to register for an acct?
#password: "changeme"
Thats a neat option and simply provide a link to the many Docker Install HOWTOs for those who do want to join in the logging fun but dont yet have Docker.
Tagging @yogesh
Anyone who DOES have a local Kibana install probably doesnt need a script 
so I’ll just give that a bye…
When you say “Kibana” here I take it you meant Filebeat?
I’m sure we can provide API keys and get users setup on the server.
No - I meant Kibana for those who want to see the pretty graphs locally
Again anyone who has a local Filebeat install wont need a script and certainly wont want me trampling over their setup.
I don’t think running Kibana locally is a very common case. I haven’t seen that before. It’s generally setup along with Elastic on the server. Filebeat is generally what’s needed locally, for the log forwarding.
I might be out of touch though. Not sure how this telegraf and vdash setup works.
If it takes Joe Punter half the time it took me to gett it running locally then I’d agree that it wont be common at all
are two different things Telegraf is part of what @tfa is using with Grafana ( which I do have a little experience with) and vdash is @happybeing 's monitoring tool in Rust.
It’s realisitic for tests. A lot of the mem usage (I suspect) is client side. Nodes (for me locally) on main have tended to hover ~150mb. But the client with a lot of large files gets a bit mad and for sure needs improvement there.
The above statement is bollocks.
I was getting confused reading HOWTOs that set up the whole ELK stack which does require java for Elasticsearch. Thanks to all who pointed this out.
So we can install filebeat without too much impact. @Yogesh is working on an ELK stack that we should all be able to point our filebeat config at and export the logs for analysis. More later when that ELK stack becomes ready for our use.
@happybeing not sure if this is neccesarily related to your github issue but I have noticed that not all latest releases work, for instance last night latest wouldn’t work for me but my current go to from about 2 weeks back does without fail.
Thanks Josh, noted. I was able to start a network using the launch tool, just not using ‘run-baby-fleming’ so I’m good to go. I just want it to test the latest log files with vdash and see if I can make it useful again
excellent - thank you 
yay
Supper thumbs up