Do my homework for me - Bash help needed

Support
1

Right now manually extracting the log files from the baby-fleming-nodes dir structure is a proper PITA.
I’m failing to put a script together to scan the contents of $HOME/.safe/node/baby-fleming-nodes/ and copy all the log files to a set of timestamped directories and then tar that up so the devs can look harder.
I traverse baby-fleming-nodes/ looking for sn_node.log* and copy those logs into the correct sub-dir in $HOME/saved-logs but I am screwing up with cut at line 30. There must be a better way to do this and treat the output of $d as a string rather than a path so I can append the correct node number and store the log correctly

#! /bin/bash

# collect log files from each run of baby-fleming
# we have many files named sn_node.log, so we need to store them in seperate dirs
# finally tar up all the logs from each run

TIMESTAMP=$(/usr/bin/date +%Y%m%d_%H%M)
#Do this outside the loop so the timestamp is consistent even if the time changes while the loop is running


if [[ ! -d $HOME/.safe/node/baby-fleming-nodes ]];
	then
		echo "Cannot find your baby-fleming nodes"
		echo "Are you sure your baby-fleming network is/was running?"
		exit 1
fi	

cd $HOME/.safe/node/baby-fleming-nodes/
for i in  {1..11};
	do 
		mkdir -p $HOME/baby-fleming-logs/$TIMESTAMP/sn_node-$i/
	done
	mv 	$HOME/baby-fleming-logs/$TIMESTAMP/sn_node-1 $HOME/baby-fleming-logs/$TIMESTAMP/sn_node-genesis

for d in $HOME/.safe/node/baby-fleming-nodes/*;
	do
		cd $d
		NODE_STRING=$(echo $d)
		echo $NODE_STRING
		NODE_NO=$(cut -c -40 $NODE_STRING)
		echo $NODE_NO
		cp -v sn_node.log*  $HOME/baby-fleming-logs/$TIMESTAMP/$NODE_NO/
		cd ..
	done	

tree $HOME/baby-fleming-logs

#TODO compress the saved logs
tar -cvf $HOME/baby-fleming-logs

I think this is worth putting some effort into as if successful we get a consistent way of delivering logs until ELK gets sorted out. With a script it becomes a LOT easier for others to join in with testing the latest from github and the more eyes on a problem, the sooner bugs will be uncovered etc etc etc

So someone, please do my homework so I can extract all the log files from (say) $HOME/.safe/node/baby-fleming-nodes/sn_node8/ and copy them to $HOME/baby-fleming-logs/$TIMESTAMP/sn_node8/ etc.

Of course maybe I a going about this the wrong way and a different approach is needed. Tell me if Im making it too difficult.

2 Likes
2

We can give ELK a comnet spin tomorrow.
I played with it briefly a few days ago, seemed to work fine.

4 Likes
3

In that case, I’ll put this aside for now then. :smiley:

1 Like
4

We should still try to get a script together to increase participation

1 Like
5

I have been running around like a clown touching ground only briefly here and there. I can confirm that metrics were working on beats but I did not look to see where or if the logs were stored.

3 Likes
6

Yes a few months back- was it a year? when I first looked at ELK, I got beats working OK but after that it got a bit more difficult

EDIT: and since @davidrusu simplified the logs to one line entries, getting beats running right will be even easier.

2 Likes
7

You could do this?

safe_network on  main via 🐍 v3.10.5 (aws) via 🦀 v1.60.0 on ☁️  (eu-west-2)
❯ cp -r ~/.safe/node/baby-fleming-nodes/ /tmp/baby-fleming-logs

safe_network on  main via 🐍 v3.10.5 (aws) via 🦀 v1.60.0 on ☁️  (eu-west-2)
❯ find /tmp/baby-fleming-logs/ -type f -not -name "*.log" -exec rm {} \;

safe_network on  main via 🐍 v3.10.5 (aws) via 🦀 v1.60.0 on ☁️  (eu-west-2)
❯ tree /tmp/baby-fleming-logs/
 /tmp/baby-fleming-logs
├──  sn-node-2
│  └──  sn_node.log
├──  sn-node-3
│  └──  sn_node.log
├──  sn-node-4
│  └──  sn_node.log
├──  sn-node-5
│  └──  sn_node.log
├──  sn-node-6
│  └──  sn_node.log
├──  sn-node-7
│  └──  sn_node.log
├──  sn-node-8
│  └──  sn_node.log
├──  sn-node-9
│  └──  sn_node.log
├──  sn-node-10
│  └──  sn_node.log
├──  sn-node-11
│  └──  sn_node.log
└──  sn-node-genesis
   └──  sn_node.log

Sure you could get the timestamp in the first directory you copied to or something.

3 Likes
8

Thank you :smile:

I had to mess about a bit but that is much more concise
I settled on

cp -r ~/.safe/node/baby-fleming-nodes/ /tmp/baby-fleming-logs-$TIMESTAMP

find /tmp/baby-fleming-logs-$TIMESTAMP/ -type f -not -name "sn_node.*" -exec rm -vrf {} +;

find /tmp/baby-fleming-logs-$TIMESTAMP/ -type d -empty -delete

and now tree shows the result I wanted
…

├── sn-node-9
│   ├── sn_node.log
│   ├── sn_node.log.20220819T022906
│   └── sn_node.log.20220819T032401
└── sn-node-genesis
    ├── sn_node.log
    ├── sn_node.log.20220819T014829
    ├── sn_node.log.20220819T020419
    ├── sn_node.log.20220819T022006
    ├── sn_node.log.20220819T023447
    ├── sn_node.log.20220819T025001
    ├── sn_node.log.20220819T030549
    ├── sn_node.log.20220819T032059
    └── sn_node.log.20220819T033651

Thanks again - I had forgotten just how useful the find command is especially with -exec

4 Likes
9

This topic was automatically closed after 60 days. New replies are no longer allowed.