Nico's blog

(Ugly) script to log AlertManager events

I created this (very) ugly script as I wanted to keep track of the events generated by AlertManager.

YES it's ugly, but it's useful (at least to me), so here we go:

#!/bin/sh
#
# Script to log alerts sent by AlertManager
# Nico 2025-03-12

BASEDIR=/home/nc/scripts/alerts
TMPFILE=$BASEDIR/alert_output.txt
OLDTMPFILE=$BASEDIR/alert_output_old.txt
LOGFILE=$BASEDIR/alerts.log
TIMESTAMP=`date +%Y-%m-%d_%H:%M`

/usr/local/bin/amtool alert --alertmanager.url http://REPLACE_WITH_YOUR_HOST:9093 |grep -v ^Alertname > $TMPFILE

if [ -e $OLDTMPFILE ]   # if $OLDTMPFILE exists, there is an ongoing alert 
then
    if [ `wc -l < $TMPFILE` -eq 0 ]     # if $TPMFILE == 0, the alert is over
    then
        echo "$TIMESTAMP - ALERT OVER" >> $LOGFILE
        rm $OLDTMPFILE
    else
        if diff -q $TMPFILE $OLDTMPFILE >/dev/null 2>&1 # otherwise we check the diff between old and new tmpfile
        then
            echo "pas de difference"    # if there is no difference, we do nothing
        else
            cat $TMPFILE >> $LOGFILE    # if there is a difference (i.e. additional alerts), we write it to the logfile
            mv $TMPFILE $OLDTMPFILE
        fi
    fi
else
    if [ `wc -l < $TMPFILE` -ne 0 ] 
    then
        cat $TMPFILE >> $LOGFILE
        mv $TMPFILE $OLDTMPFILE
    fi
fi

Tags: IT