Sorting Alarm Messages

[posted 2004/11/16]

[Notes issued with the release of PIKT 1.18.0pre3, which introduced the new feature:  sorting alarm messages by syslog level.]

Until now, alarm output has been ordered within alert messages according to the sequence you specify in alerts.cfg.  If, for example, your alerts.cfg has

        ...

        alarms

                SysReboot
                PiktcSvcLogScan
                ProcessSystemDead
                LoadAverage
                ...

then the alert message might look like

CRITICAL:
    SysReboot
        Scan the 'last' command output for signs of recent system reboots

        ...

WARNING:
    PiktcSvcLogScan
        Scan the PIKT piktc_svc.log for noteworthy entries

        ...

EMERGENCY:
    ProcessSystemDead
        Report or restart 'dead' crucial system processes

        ...

URGENT:
    LoadAverage
        Report perilously high system load averages

        ...

Beginning with pikt-1.18.0pre3, by default and regardless of how you have ordered the alarms in alerts.cfg, we sort alarm messages by syslog level--emergency (first, highest), alert (in Pikt-Speak, "urgent"), critical, error, warning, notice, info, debug (last, lowest):

EMERGENCY:
    ProcessSystemDead
        Report or restart 'dead' crucial system processes

        ...

URGENT:
    LoadAverage
        Report perilously high system load averages

        ...

CRITICAL:
    SysReboot
        Scan the 'last' command output for signs of recent system reboots

        ...

WARNING:
    PiktcSvcLogScan
        Scan the PIKT piktc_svc.log for noteworthy entries

        ...

Recall that you specify an alarm's importance using the "level" statement in a Pikt script, for example:

SysReboot

        init
                status active
                level critical
                task "Scan the 'last' command output for signs of recent system
                      reboots"
                ...

The idea here is to have more important alarm output appear toward the beginning of alerts messages, and less important alarm output appear toward the end.

If for some reason you prefer the traditional behavior where the sequence in alerts.cfg determines the order of alarm output, you may override the default by adding this new directive in PIKT.conf:

sort_messages                 NO        [or: FALSE, OFF, ...;
                                         defaults to YES, ...]

Sorting alarm messages according to syslog level was part of the original PIKT design specification, but for some reason it got overlooked and was not implemented until now.

pikt-1.18.0pre3 introduces the '#setdef <ident> = [!] [<proc>]' and (in defines.cfg) '<ident> [<proc>]' variants, for example:

#setdef paranoid = [test `hostname` = "kerberos"]

and (in defines.cfg):

dst     // TRUE if Daylight Savings Time now applies, FALSE otherwise
        // at our site, `date +%Z` returns "CDT" if DST is in effect,
        // "CST" otherwise; substitute your own time zone string as needed
        [test `date +%Z` = "CDT"]

With these new variants, you no longer have to hard-code define values beforehand (e.g., TRUE, FALSE, ...); you can now set them dynamically according to some process output (typically a test whether some expression is true or false).

As always, we invite bug reports, suggestions, and feedback.

For more examples, see Developer's Notes.