Logfile Time Stamps
[posted 2000/02/04]
We have set up a new Pikt script for monitoring system log file date/time stamps--checking for regular and recent log file updates.
In recent months, we have had e-mail fail on our primary mail server, also httpd fail on our primary Web server machine. In both cases, the first and surest sign of trouble was the cessation of logging. And in both cases, the usual daemons (sendmail, syslogd, httpd) were still seen to be running.
So far, here are our entries in objects.cfg for a new objects set, UpToDateLogFiles:
// for UpToDateLogFiles, the number is the lineage (of the last line, in
// seconds) beyond which a log file is deemed out of date
UpToDateLogFiles
#ifndef generic
# if mailserver
=syslog 600 // 10 mins
# else
=syslog 3600 // 60 mins
# endif
#elsedef
=syslog 3600 // 60 mins
#endifdef // generic
# if warsaw
/opt/ns-home/logs/http/access 600 // 10 mins
# endif
So, on the mailservers, if syslog hasn't logged an entry in the last 10 minutes, PIKT will signal this. On other systems, the threshold is 60 minutes. On warsaw, PIKT looks for Web server activity within the last 10 minutes.
We will add other log files to UpToDateLogFiles over time. We also expect to adjust the seconds thresholds, especially for syslog on the non-mailserver machines.
Here is the new UpToDateLogChk alarm, which runs every half hour or hour (depending on the machine) in our Urgent alerts set:
UpToDateLogChkUrgent
init
status active
level urgent
task "Check to see that various log files are up-to-date"
input file "=uptodatelogfiles_obj"
dat $file 1
dat #secs 2
rule
=check_if_up_to_date($file, #secs)
=check_if_up_to_date() is a new macro, with macro arguments. If you want to make use of this new feature, you will have to download pikt-1.9.0pre.tar.gz from the PIKT Web site, or wait for the official 1.9.0 release (expected later this month). Anyway, here is the macro definition:
check_if_up_to_date(F, S) // warn if file (F) is not at least (S)
// seconds up-to-date
// remove pesky embedded dquote chars, found especially
// in http access logs
set $line = $substitute($command("=tail -1 (F) 2>/dev/null"),
$dquote(), $char(0))
if $line eq ""
output mail "(F) is empty, else not found!"
else
=set_lineage($line)
if #lineage == #err()
output piktlog "ERROR: For (F),
last line is improperly
date/time-stamped: $line"
elseif #lineage < 0
output piktlog "WARNING: For (F),
last line is in
the future: $line"
elseif #lineage >= (S)
output mail "The last (F) line is at
least $text((S))
seconds out-of-date: $line"
fi
fi
=check_if_up_to_date() refers to another macro, =set_lineage(). (Yeah, we know. We're not talking ancestries here!) Here is that macro:
set_lineage(L) // for the given line (L), determine its age, in seconds,
// according to its date/time stamp (if any)
// for syslog and many other log files; look for a date/time
// stamp like "Feb 2 10:25:20" at the beginning of a line
if #parse("(L)","^(=months)[[:space:]]+([[:digit:]]+)[[:space:]]+
([[:digit:]]+):([[:digit:]]+):([[:digit:]]+)") == 5
set #lineage = #now() - #datevalue(#year(),
#monthnumber($1),
#val($2))
- #timevalue(#val($3),
#val($4),
#val($5))
// for http access logs and others; look for a date/time
// stamp like "02/Feb/2000:10:42:03" anywhere within a line
elseif #parse("(L)","([[:digit:]]+)/(=months)/
([[:digit:]]+):([[:digit:]]+):
([[:digit:]]+):([[:digit:]]+)") == 6
set #lineage = #now() - #datevalue(#val($3),
#monthnumber($2),
#val($1))
- #timevalue(#val($4),
#val($5),
#val($6))
// for uptime output and others; look for date/time
// stamp like "Feb 25 16:19" anywhere within a line
elseif #parse("(L)",
"(=months)[[:space:]]+([[:digit:]]+)[[:space:]]+
([[:digit:]]+):([[:digit:]]+)") == 4
set #lineage = #now() - #datevalue(#year(),
#monthnumber($1),
#val($2))
- #timevalue(#val($3),
#val($4),
0)
// add other cases here in their own 'elseif' as needed
// else no, or unaccounted for, date/time stamp
else
set #lineage = #err()
fi
We are already using =set_lineage() in other alarms.
Macros-with-arguments goes part way toward offsetting the absence of formal, user-definable functions in Pikt. Although
=check_if_up_to_date($file, #secs)
looks like a function call, of course it's not. The =check_if_up_to_date() definition (all 22 lines of it, subtracting the comment lines, and adding the lines in the =set_lineage() definition) will be repeated in the processed script code every time it is referenced. Script files (.alt files) aren't very long, so this isn't much of a concern.
We've been toying with the idea of writing a built-in function, #lineage(), akin to the already existing #fileage(). But unlike #fileage(), which refers to a file's date/time stamp in an 'ls -l' display, there are many different date/time stamps one might encounter in the many different system log files. The =set_lineage() definition above includes the three we have encountered so far. We're sure there will be others. One virtue of having =set_lineage() be a macro rather than a built-in function is that you, the user, can easily add to and modify the macro definition yourself.
You could do the same, too, with user-definable formal functions. Although we could have implemented them by now, we have not for the reason that we still intend to replace pikt's engine with something more robust (GNU Guile? A modified Perl? Something of our own devising?) We are reluctant to extend the pikt script interpreter much further (beyond a few easy extensions, like adding more built-in functions), knowing that it is likely to be scrapped sooner or later. (We hope and expect for Pikt's surface to remain unchanged, or little changed. Note that the new macros-with-arguments feature is not part of pikt, the script interpreter. Rather, it's an extension of piktc, the PIKT preprocessor. Note, too, that you can use macros-with-arguments within any of the PIKT configuration files, including programs.cfg.)
Even in advance of 1.9.0's release, we'll continue to post interesting new macros-with-arguments uses when we happen to discover them.
For more examples, see Developer's Notes.
