Successfully running validation tests on new PIKT code is not enough (because your results are only as good as your tests). One has to run the new code in an actual production environment as well.
We're running the latest pikt-1.10.0pre5 code on our master machine and are suddenly seeing these alerts:
PIKT ALERT Wed May 10 12:55:02 2000 vienna DEBUG: AlertChkCritical Detect PIKT alert/service daemon failures/restarts/redundancies piktd appears to be hung/dead on rheims piktd appears to be hung/dead on nantes ...
It seems like the piktd daemon is dead everywhere!
I cp'ed Debug.alt to Test.alt, edited out every alarm script except for AlertChkCritical, then added another output line to the last rule of AlertChkCritical:
rule do #split ( $time , ":" ) [new -->] output "$time, $, $, $" set #t = #datevalue ( #yrnow - #if ( #monnow == 1 && $mon eq "Dec" , 1 , 0 ) , #monthnumber ( $mon ) , #date ) + #timevalue ( #val ( $ ) , #val ( $ ) , #val ( $ ) ) if #timenow - #t > #hrs * 60 * 60 output mail "piktd appears to be hung/dead on $sys" endif
After doing a 'pikt +A Test', I got these results:
12:41:00, :, ÿ, ÿ piktd appears to be hung/dead on rheims 12:42:00, :, ÿ, ÿ piktd appears to be hung/dead on nantes ...
I recognized this instantly as a bug in the revised #split() implementation. I'll fix the bug later this week. In the meantime, we don't want to continue seeing bogus messages about supposed piktd failure. So, I commented out the last rule of AlertChkCritical and added a =remind() macro to remind me a week from now to reactivate the rule:
rule // if it's been more than #hrs hours since the last piktd_log // entry, piktd is not logging, hence appears to be hung/dead; =remind(2000, 5, 17, "REACTIVATE FINAL ALERTCHKCRITICAL RULE AFTER DEBUG") // do #split($time, ":") // set #t = #datevalue(#yrnow - #if(#monnow == 1 && $mon eq "Dec", 1, 0), #monthnumber($mon), #date) // + #timevalue(#val($), #val($), #val($)) // if #timenow - #t > #hrs*60*60 // output mail "piktd appears to be hung/dead on $sys" // endif
(I then reinstalled with 'piktc -iv +A Debug +H vienna', it perhaps goes without saying.)
Consider doing something like this when faced with similar temporary glitches.
For more examples, see Developer's Notes.