PIKT Log Scan Macro

The pikt_log_scan_alarms_macros.cfg is a script macro to scan PIKT log file output for noteworthy entries.

Note how--by means of the generalized script macro, tailored to the needs of each PIKT log file by the parameters we pass to the =piktlog_scan() script macro--note how this technique saves us from having to write dozens of ungeneralized log file scanning scripts specific to each PIKT log file.

piktlog_scan(A, B1, B2)
 
        init
                status =piktstatus
                level =piktlevel
                task "Scan the PIKT (A).log for noteworthy entries"
                input logfile "=logdir/(A).log"
        begin   
                =checkpoint(=lalim)

                // assume no crisis (yet)
                set #crisis = #false()
#if dbserver | prodserver
                // on production servers, only process non-emergency pikt
                // log files in the early morning, during production downtime
                if $alert() !~~ "red|emergency"
                        if #hour() > 5
                                quit
                        fi
                fi
#endif

        rule    // automatic bypasses   
                if    $inlin =~~ "==> =logdir/.+ <=="
                        next
                endif

        rule    // special alarm-specific bypasses, first bypass
                if $inlin =~~ "(B1)"
                        next
                endif

        rule    // bypass empty lines
                if ! #length($inlin)
                        next
                endif

        rule
                =set_hr($inlin)

        rule    // report ERROR entries
                if $inlin =~ "ERROR]"
                        output mail $inlin
                        next
                endif

        rule    // report inactive alarms
                if    $inlin =~ "INFO].+suspended status"
                // && =monday
                        output mail $inlin
                        next
                endif


        rule    // report stderr output from invoked commands
                if =notlogmsg
                        if $inlin !~~ "broken pipe"
                                output mail $inlin
                        endif
                        next
                endif

        rule    // report corrupted lines, then bypass
                if #hr == #err()
                        output mail $inlin
                        next
                endif

        rule    // report entries due to a possibly badly formed input
                // command that results in no input
                if    $inlin =~ "WARNING].+no input data"
                        =outputmail $inlin
                        next
                endif

        rule    // bypass commonplace stuff
                if $inlin =~ "(DEBUG]|NOTICE]|INFO]|
                              last line is improperly date/time-stamped|
                              no answer from)"
                        next
                endif

        rule    // special alarm-specific bypasses, second bypass
                if $inlin =~~ "(B2)"
                        next
                endif

        rule    // possibly report WARNING entries
                if $inlin =~ "WARNING]"
                        =outputmail $inlin
                        next
                endif

        rule    // report anything not filtered out
                output mail $inlin

        end
                if #crisis
                        // =page()
                endif
                quit

You might invoke the =piktlog_scan() macro in your alarms.cfg file thusly:

///////////////////////////////////////////////////////////////////////////////
//
// logs_pikt_alarms.cfg
//
///////////////////////////////////////////////////////////////////////////////
=nbsp
#if piktmaster
PIKTREDLogScan

        =piktlog_scan(RED, =nonesuch, =nonesuch)

PIKTREDTestLogScan

        =piktlog_scan(REDTest, =nonesuch, =nonesuch)

#endif

///////////////////////////////////////////////////////////////////////////////

PIKTEMERGENCYLogScan

        =piktlog_scan(EMERGENCY, runaway...procs, =nonesuch)

PIKTEMERGENCYTestLogScan

        =piktlog_scan(EMERGENCYTest, =nonesuch, =nonesuch)

///////////////////////////////////////////////////////////////////////////////

PIKTSysAdminsUrgentLogScan

        =piktlog_scan(SysAdminsUrgent, dmesgrf: no such file|stat source.+dmesg, =nonesuch)

PIKTSysAdminsUrgentTestLogScan

        =piktlog_scan(SysAdminsUrgentTest, dmesgrf: no such file|stat source.+dmesg, =nonesuch)

///////////////////////////////////////////////////////////////////////////////

PIKTCodersUrgentLogScan

        =piktlog_scan(CodersUrgent, dmesgrf: no such file|stat source.+dmesg, =nonesuch)

PIKTCodersUrgentTestLogScan

        =piktlog_scan(CodersUrgentTest, dmesgrf: no such file|stat source.+dmesg, =nonesuch)

///////////////////////////////////////////////////////////////////////////////

PIKTSysAdminsCriticalLogScan

        =piktlog_scan(SysAdminsCritical, =nonesuch, =nonesuch)

PIKTSysAdminsCriticalTestLogScan

        =piktlog_scan(SysAdminsCriticalTest, =nonesuch, =nonesuch)

///////////////////////////////////////////////////////////////////////////////

[...]

///////////////////////////////////////////////////////////////////////////////

PIKTCheckDatabaseLogScan

        =piktlog_scan(CheckDatabase, =nonesuch, =nonesuch)

///////////////////////////////////////////////////////////////////////////////

PIKTDownSystemsLogScan

        =piktlog_scan(DownSystems, =nonesuch, =nonesuch)

///////////////////////////////////////////////////////////////////////////////

PIKTDownRpcLogScan

        =piktlog_scan(DownRpc, =nonesuch, =nonesuch)

///////////////////////////////////////////////////////////////////////////////

PIKTSysRebootsLogScan

        =piktlog_scan(SysReboots, =nonesuch, =nonesuch)

///////////////////////////////////////////////////////////////////////////////

Output from these scripts might look like, for example:

DEBUG:
    PIKTSysAdminsUrgentLogScan
        Scan the PIKT SysAdminsUrgent.log for noteworthy entries

        ping: unknown host grenada
        ping: unknown host grenada

DEBUG:
    PIKTSysAdminsCriticalLogScan
        Scan the PIKT SysAdminsCritical.log for noteworthy entries

        telnet: grenada: Name or service not known
        grenada: Unknown host
        telnet: connect to address 10.10.5.73: Connection refused
        telnet: connect to address 10.10.5.34: Connection refused
        telnet: grenada: Name or service not known

For more examples, see Samples.