alerts
Here are some sample PIKT alerts. These are only highlights. Examples of more routine and mundane problem reports and information messages are not shown.
-------------------------------------------------------------------------------
PIKT ALERT
Thu Sep 27 12:33:15 2002
trondheim2
CRITICAL:
AuthLogScanCritical
Scan the authlog for critical authorization incidents
Sep 27 12:24:40 trondheim2 statd[174]: [ID 462824 auth.error] statd:
attempt to create "/var/statmon/sm/^D...^D...^E...^F...^F...^G...^G...
%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %0242x
...
-------------------------------------------------------------------------------
PIKT ALERT
Tue Oct 2 15:49:25 2002
athens4
URGENT:
MessagesScanUrgent
Scan the system messages log for urgent entries
Oct 2 15:40:21 athens4 unix: WARNING:
/sbus,0/QLGC,isp,10000/sd,0 (sd11):
Oct 2 15:40:21 athens4 SCSI transport failed:
reason 'reset': retrying command
Oct 2 15:41:12 athens4 unix: WARNING:
/sbus,0/QLGC,isp,10000/sd,0 (sd11):
Oct 2 15:41:12 athens4 SCSI transport failed:
reason 'reset': retrying command
...
-------------------------------------------------------------------------------
PIKT ALERT
Wed Oct 3 04:30:06 2002
athens4
CRITICAL:
DiskCapCritical
Report critical filesystem full or near-full situations
Filesystem /ckp on /dev/md/dsk/d10 is 100% full, 0 Kb left
17370930 /ckp/ingres
8 /ckp/lost+found
-------------------------------------------------------------------------------
PIKT ALERT
Wed Oct 3 07:08:06 2002
athens2
URGENT:
ProcessSystemDeadUrgent
Report or restart 'dead' crucial system processes
The process 'syslogd' is not running
-------------------------------------------------------------------------------
PIKT ALERT
Sun Jul 28 02:00:02 2002
davao
WARNING:
MessagesScanWarning
Scan the system messages log for perhaps worrisome entries
Jul 27 14:19:37 handel sshd[5973]: Accepted publickey for root from
::ffff:11.22.33.44 port 42351 ssh2
Jul 27 21:14:52 handel sshd[16487]:
Illegal user test from ::ffff:12.34.56.78
Jul 27 21:14:52 handel sshd[16487]: input_userauth_request:
illegal user test
Jul 27 21:14:55 handel sshd[16487]:
Address 12.34.56.78 [gecko.blechcough.org.]
maps to gecko.blechcough.org, but this does not map back to the address
- POSSIBLE BREAKIN ATTEMPT!
Jul 27 21:14:55 handel sshd[16487]:
Failed password for illegal user test from
::ffff:12.34.56.78 port 33851 ssh2
Jul 27 21:14:55 handel sshd[16487]:
Received disconnect from ::ffff:12.34.56.78:11: Bye Bye
Jul 27 21:14:55 handel sshd[16488]:
Illegal user guest from ::ffff:12.34.56.78
Jul 27 21:14:55 handel sshd[16488]:
input_userauth_request: illegal user guest
Jul 27 21:14:55 handel sshd[16488]:
Address 12.34.56.78 [gecko.blechcough.org.]
maps to gecko.blechcough.org, but this does not map back to the address
- POSSIBLE BREAKIN ATTEMPT!
Jul 27 21:14:55 handel sshd[16488]:
Failed password for illegal user guest
from ::ffff:12.34.56.78 port 33974 ssh2
Jul 27 21:14:55 handel sshd[16488]:
Received disconnect from ::ffff:12.34.56.78:11: Bye Bye
...
-------------------------------------------------------------------------------
PIKT ALERT
Wed Apr 16 23:54:44 2003
vienna
EMERGENCY:
NetworkDownEmergency
Report if network is down
pikt.org is down
-------------------------------------------------------------------------------
PIKT ALERT
Wed Oct 3 13:51:14 2002
madrid
URGENT:
MessagesScanUrgent
Scan the system messages log for urgent entries
Oct 3 13:26:15 madrid su: [ID 8191 auth.crit] 'su root' failed for cokaka
on /dev/pts/6
-------------------------------------------------------------------------------
PIKT ALERT
Tue Oct 2 14:29:17 2002
moscow
CRITICAL:
ProcZombieTotalCountsCritical
Report unusually high number of zombie and other processes
Unusually high process count (458): 458 processes: 454 sleeping,
2 zombie, 2 on cpu
UID PID PPID C STIME TTY TIME CMD
root 0 0 0 Oct 01 ? 0:13 sched
root 1 0 0 Oct 01 ? 0:38 /etc/init -
...
rdupqah 4398 4397 0 14:27:51 ? 0:00 imapd
mckym3 2501 291 0 12:41:12 ? 0:00 imapd
mjcoltrn 4385 4382 0 14:27:50 ? 0:00 imapd
pilson0 22436 291 0 14:22:17 ? 0:00 imapd
...
-------------------------------------------------------------------------------
PIKT ALERT
Mon Nov 12 22:18:17 2002
kiev
URGENT:
MessagesScanUrgent
Scan the system messages log for urgent entries
Nov 12 21:40:44 kiev bsd-gw[29187]: Error reading from connection:
Bad file number
Nov 12 21:44:04 kiev bsd-gw[29193]:
Invalid protocol request (66): BBBXXXXXXX%
.156u%300$n%.21u%301$nsecurity%302$n%.192un1ECf]fE'MECC1?A^u1FEMU/bin/sh
.232u%300$n%.199u%301$nsecurity.i%302$n%.192un1]fE'MECC1?A^u1FEMU/bin/sh
...
-------------------------------------------------------------------------------
PIKT ALERT
Thu Oct 4 11:27:17 2002
moscow
URGENT:
MessagesScanUrgent
Scan the system messages log for urgent entries
Oct 4 11:16:30 moscow nfs:
[ID 174370 kern.notice] NFS write error on host sun:
No space left on device.
Oct 4 11:16:36 moscow nfs:
[ID 174370 kern.notice] NFS write error on host sun:
No space left on device.
...
-------------------------------------------------------------------------------
PIKT ALERT
Fri Oct 5 10:32:21 2002
moscow
URGENT:
MailQueueLengthyUrgent
Report worrisomely long mail queue
223 messages, 3885 lines in mail queue
URGENT:
NewSystemStartupFileUrgent
Report new system startup files
new system startup file: -rwxr--r-- 1 root sys 1471 Jan 5 2000
/etc/init.d/sendmail.011005
-------------------------------------------------------------------------------
PIKT ALERT
Sat Oct 6 14:31:05 2002
cologne
CRITICAL:
CksumDifferenceCritical
Report checksum differences for certain critical system files
/usr/bin/login checksums differ!
auth: 63605c254a6a9928b8075963c951dbbf 29144 /usr/bin/login
actual: 938a4ca4a6dd7b8665faa4cde6a83f73 29356 /usr/bin/login
-------------------------------------------------------------------------------
PIKT ALERT
Sun Oct 7 02:49:22 2002
cologne
WARNING:
FileCtimeChangeWarning
Report ctime-changed files/dirs in file systems that should be stationary
/usr/bin/login: ELF 32-bit MSB executable SPARC 1, dynamically linked
-r-sr-xr-x 1 root bin 29144 Dec 17 07:08 /usr/bin/login
-------------------------------------------------------------------------------
PIKT ALERT
Wed Oct 24 02:38:09 2002
paris6
WARNING:
DumpDatesProblemsWarning
Report backup problems as revealed by dumpdates
LAST RECORDED INCR BACKUP 16 DAYS OLD FOR /PUB/DISK40 (/DEV/DSK/C0T2D0S4)
last recorded full backup 15 days old for /pub/disk41 (/dev/dsk/c0t2d0s5)
LAST RECORDED INCR BACKUP 16 DAYS OLD FOR /PUB/DISK41 (/DEV/DSK/C0T2D0S5)
-------------------------------------------------------------------------------
PIKT ALERT
Sun Oct 7 08:15:49 2002
prague
URGENT:
RootCoreFileExistUrgent
Deal with /core files
/core: ELF 32-bit MSB core file SPARC Version 1, from 'syslogd'
-rw------- 1 root other 1078508 Oct 7 03:10 /core
moved /core file to /tmp, after the move:
/dev/dsk/c0t0d0s0 1984230 1426880 497824 75% /
-------------------------------------------------------------------------------
PIKT ALERT
Wed Apr 20 15:37:01 2005
ottawa
URGENT:
MessagesScanUrgent
Scan the system messages log for urgent entries
Apr 20 14:43:07 ottawa named[1374]: zone earlymusichicago.com/IN:
refresh: failure trying master 192.168.5.16 []#53: timed out
Apr 20 14:43:52 ottawa named[1374]: zone earlymusichicago.com/IN:
refresh: failure trying master 192.168.5.16 []#53: timed out
Apr 20 14:53:29 ottawa named[1374]: zone earlymusicchicago.org/IN:
refresh: failure trying master 192.168.5.16 []#53: timed out
Apr 20 14:54:14 ottawa named[1374]: zone earlymusicchicago.org/IN:
refresh: failure trying master 192.168.5.16 []#53: timed out
...
-------------------------------------------------------------------------------
PIKT ALERT
Sun Oct 7 08:19:23 2002
paris6
URGENT:
MessagesScanUrgent
Scan the system messages log for urgent entries
Oct 7 00:25:04 paris6 automountd[175]:
berlin2:/home server not responding: RPC: Timed out
-------------------------------------------------------------------------------
PIKT ALERT
Mon Oct 8 10:23:20 2002
moscow
EMERGENCY:
PerUserProcessCountsEmergency
Report unusually high counts of per-user procs.
150 esmith5 imapd
killed all esmith5 imapd processes
-------------------------------------------------------------------------------
PIKT ALERT
Thu Jan 31 10:21:07 2002
nantes
URGENT:
YPPasswdFileProblemsUrgent
Report problems with NIS passwd file requiring immediate attention
User boinha has no password!
-------------------------------------------------------------------------------
PIKT ALERT
Sun Oct 21 14:26:10 2002
warsaw
URGENT:
MessagesScanUrgent
Scan the system messages log for urgent entries
Oct 21 14:07:39 warsaw sshd[23866]: log: ROOT LOGIN as 'root'
from server1.underworld.org
-------------------------------------------------------------------------------
PIKT ALERT
Sun Jul 29 12:30:16 2002
moscow
CRITICAL:
AuthLogChkCritical
Report noteworthy authorization incidents
Jul 29 11:56:20 moscow imapd[4534]: [ID 210418 auth.alert] Login
SYSTEM BREAK-IN ATTEMPT
user=+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++auth=
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
host=murmansk.uppity.edu [123.111.130.35]
-------------------------------------------------------------------------------
PIKT ALERT
Fri Nov 9 02:41:50 2002
kiev
WARNING:
SuLogScanWarning
Scan the sulog for noteworthy su incidents
SU-TO-ROOT FAILURE: SU 11/08 18:06 - pts/2 fjring-root
SU-TO-ROOT FAILURE: SU 11/08 18:19 - pts/2 fjring-root
-------------------------------------------------------------------------------
PIKT ALERT
Mon Oct 22 08:17:48 2002
paris6
URGENT:
RunawayProcUrgent
Report runaway processes, and cpu-hogs
USER TT PID PPID TIME COMMAND
picoopal ? 17958 1 3-07:44:21 /opt/local/matlabr12/bin/sol2/matlab
URGENT:
CPUUsageUrgent
Report unusually high CPU percentage usage figures
17958 picooalr 4 -15 0 50M 4464K sleep 79.7H 63.76% matlab
-------------------------------------------------------------------------------
PIKT ALERT
Mon Oct 22 02:31:58 2002
padua
WARNING:
PasswdFileNISProblemsWarning
Report /etc/passwd file problems with NIS
User trylim not found in NIS passwd
User pywong doesn't match the NIS uname ywong for uid 673
User dregurt doesn't match the NIS uname gregorio for uid 5134
User andre6 doesn't match the NIS uname lrezez for uid 5135
...
WARNING:
ShadowFileNISProblemsWarning
Report /etc/shadow file problems with NIS
User trylim not found in NIS passwd
-------------------------------------------------------------------------------
PIKT ALERT
Tue Oct 23 18:20:07 2002
moscow
URGENT:
LogUpdatesUrgent
Check to see if various log files are being updated
The last /var/log/popauth.watcher.log line is at least
600 seconds out-of-date:
Oct 23 17:28:27 mkorman authenticating relaying for 172.144.122.230
CRITICAL:
FileUpdatesCritical
Check to see if various files are being updated
/var/log/popauth.watcher.log is outdated, is 3136 seconds old
-------------------------------------------------------------------------------
PIKT ALERT
Wed Oct 24 04:00:06 2002
moscow
NOTICE:
MailFileProblemsNotice
Check several aspects of mail spool files
claffm is 15 MB long
cpoul is 13 MB long
bad file ownership: -rw-rw---- 1 sibelius mail 2 Oct 11 16:39 sibelius09mO
rfarmed is 12 MB long
SCBASSO IS 22 MB LONG
SJVEERMA IS 21 MB LONG
VLI IS 22 MB LONG
-------------------------------------------------------------------------------
PIKT ALERT
Wed Oct 24 06:12:31 2002
athens2
URGENT:
NISNoBindingUrgent
Report failures in NIS binding (as reported by ypwhich)
Domain egbdf not bound on athens2.
-------------------------------------------------------------------------------
PIKT ALERT
Wed Oct 24 06:25:11 2002
berlin2
CRITICAL:
MetastatErrorsCritical
Report DiskSuite metastat errors
Submirror 0: d21, State: Resyncing
Submirror 1: d22, State: Resyncing
d21: Submirror of d20, State: Resyncing
d22: Submirror of d20, State: Resyncing
-------------------------------------------------------------------------------
PIKT ALERT
Thu Oct 25 12:26:37 2002
paris7
URGENT:
SwapLowUrgent
Report when swap use is high
swap utilization is 96%:
swapfile dev swaplo blocks free
/dev/dsk/c0t3d0s1 32,25 8 524712 0
/swap1/swap1 - 8 3870712 179888
/swap2/swap2 - 8 3870712 185520
swap 4236660 4026108 210552 96% /tmp
4025808 /tmp/SAS_worka00006EE4
296 /tmp/ups_data
8 /tmp/ps_data
4 /tmp/screens
contents of /tmp:
total 624
drwx------ 2 pkfitro perf 336 Oct 25 08:23 SAS_worka00006EE4
-rw-rw-r-- 1 root sys 6088 Oct 25 03:58 ps_data
drwxr-xr-x 2 root other 69 Oct 18 08:35 screens
-rw-rw-r-- 1 root sys 302884 Oct 21 19:40 ups_data
last pid: 3178; load averages: 4.66, 4.70, 4.53 12:26:47
40 processes: 36 sleeping, 2 running, 1 zombie, 1 on cpu
Memory: 224M real, 4264K free, 3859M swap in use, 177M swap free
PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
28388 pkfitro 4 0 0 23M 18M run 200:49 33.94% sas
6981 jartigoo 1 50 0 1676K 1056K run 29.3H 21.60% top
3178 root 1 50 0 1580K 996K cpu 0:01 4.52% top
2755 root 1 58 0 6240K 3888K sleep 0:12 1.70% pikt
...
-------------------------------------------------------------------------------
PIKT ALERT
Thu Oct 25 15:11:12 2002
moscow
EMERGENCY:
PerUserProcessCountsEmergency
Report unusually high counts of per-user procs.
329 root /usr/lib/sendmail
-------------------------------------------------------------------------------
PIKT ALERT
Thu Oct 25 23:54:44 2002
vienna
URGENT:
SysDownUrgent
Report systems down or off the network
paris6 is sick, possibly down, or off the network (rpc failure)
-------------------------------------------------------------------------------
PIKT ALERT
Fri Oct 26 14:21:28 2002
moscow
URGENT:
NewSystemStartupFileUrgent
Report new system startup files
new startup file:
-rwxr--r-- 1 root sys 5577 Aug 20 13:10 /etc/init.d/mdemon
new startup file:
-rwxr--r-- 1 root sys 7333 Aug 20 13:10 /etc/init.d/rdacct
-------------------------------------------------------------------------------
PIKT ALERT
Fri Oct 26 14:26:01 2002
moscow
EMERGENCY:
PerUserProcessCountsEmergency
Report unusually high counts of per-user procs.
683 root /usr/lib/sendmail
killed all root /usr/lib/sendmail processes
317 nobody /opt/local/bin/python
killed all nobody /opt/local/bin/python processes
-------------------------------------------------------------------------------
PIKT ALERT
Sun Oct 28 14:18:07 2002
trondheim2
URGENT:
SysRebootUrgent
Scan the 'last' command output for signs of recent system reboots
reboot system boot Sun Oct 28 13:55
-------------------------------------------------------------------------------
PIKT ALERT
Mon Oct 29 07:22:00 2002
moscow
DEBUG:
ForwardFileExistDebug
Check for existence of vital .forward files
/home/ives/.forward not found! recreating from backup
-------------------------------------------------------------------------------
PIKT ALERT
Mon Oct 29 10:51:04 2002
moscow
URGENT:
MessagesScanUrgent
Scan the system messages log for urgent entries
Oct 29 10:48:42 moscow scsi: [ID 107833 kern.warning] WARNING:
/pci,4000/scsi,1/sd,0 (sd100):
Oct 29 10:48:42 moscow corrupt label - wrong magic number
-------------------------------------------------------------------------------
PIKT ALERT
Mon Oct 29 07:19:22 2002
moscow
URGENT:
FileExistWarnUrgent
Warn about, or possibly do something with, certain files.
/etc/nologin found!
-------------------------------------------------------------------------------
PIKT ALERT
Fri Nov 2 16:50:56 2002
madrid
URGENT:
DiskCapUrgent
Report urgent filesystem full or near-full situations
Filesystem /pub/comp_disk_1 on /dev/dsk/c1t2d0s0 is 100% full,
37243 Kb left
51115 /pub/comp_disk_1/fstrelk
26975 /pub/comp_disk_1/kchelis
26327 /pub/comp_disk_1/dturnky5
...
-------------------------------------------------------------------------------
PIKT ALERT
Tue Nov 6 02:30:19 2002
trondheim2
WARNING:
PasswdFileProblemsWarning
Report /etc/passwd file problems
User webown not in /etc/shadow file
WARNING:
PasswdShadowCrosscheckWarning
Report /etc/passwd entries not in /etc/shadow and vice-versa
webown in /etc/passwd, not in /etc/shadow
webown not found in NIS passwd
-------------------------------------------------------------------------------
PIKT ALERT
Wed Nov 28 12:56:33 2002
vienna
EMERGENCY:
DirSystemNotExistEmergency
Report system directory disappearances
/opt/lib not found!
/opt/libexec not found!
/opt/man not found!
/opt/sbin not found!
-------------------------------------------------------------------------------
PIKT ALERT
Sat Dec 1 10:25:17 2002
nantes
URGENT:
YPPasswdFileProblemsUrgent
Report problems with NIS passwd file requiring immediate attention
Users nextuid and nbarlow3 have duplicate uids: 52634
-------------------------------------------------------------------------------
PIKT ALERT
Sat Dec 8 11:23:04 2002
munich
URGENT:
LpHungUrgent
Report problems with printing
For the c216ps queue, 10 entries, possibly jammed
-------------------------------------------------------------------------------
PIKT ALERT
Mon Dec 17 19:18:23 2002
trondheim2
CRITICAL:
ShadowFileProblemsUrgent
Report urgent /etc/shadow file problems
User mailsrv has NO PASSWORD!
-------------------------------------------------------------------------------
PIKT ALERT
Tue Dec 18 02:53:41 2002
milan
WARNING:
FileCtimeChangeWarning
Report ctime-changed files/dirs in file systems that should be stationary
/usr/bin/login: ELF 32-bit MSB executable SPARC 1, dynamically linked
-r-sr-xr-x 1 root bin 29144 Dec 17 07:08 /usr/bin/login
-------------------------------------------------------------------------------
PIKT ALERT
Tue Feb 26 16:54:09 2002
murmansk
EMERGENCY:
LoadAverageEmergency
Report perilously high system load averages
4:54pm up 13 min, 0 users, load average: 131.92, 99.22, 50.60
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 1.4 0.0 1020 460 ? S 16:40 0:11 init [2]
...
www-data 269 1.9 0.6 8548 6140 ? R 16:41 0:14 /usr/sbin/apache
mysql 271 0.0 0.8 33764 8608 ? S 16:41 0:00 /usr/sbin/mysqld
...
-------------------------------------------------------------------------------
PIKT ALERT
Thu Mar 7 10:07:07 2002
milan
EMERGENCY:
PerUserProcessCountsEmergency
Report unusually high counts of per-user procs.
158 webrun menu.cgi
-------------------------------------------------------------------------------
For more examples, see Samples.
