Report New Google-Related Httpd Log Entries

In this example, we report new httpd log entries for pages of special interest, in this case, newly created web pages relating to Google.

The GoogleHttpdLogEntries script might send an alert message like the following:

                                PIKT ALERT
                         Fri Sep  9 15:20:26 2005
                                 calgary

INFO:
    GoogleHttpdLogEntries
        Report new Google-related httpd log entries

        218.16.123.1 - - [09/Sep/2005:12:11:56 -0500]
          "GET /pikt/samples/httpd_google_googlebot_alarms.cfg.html HTTP/1.1"
          200 10719 "https://pikt.org/pikt/samples/alarms.cfg_include_files.html"
          "Mozilla/4.0 (compatible; MSIE 6.0; Win 9x 4.90; .NET CLR 1.1.4322)"
        218.16.123.1 - - [09/Sep/2005:12:12:12 -0500]
          "GET /pikt/samples/googlebot.pl.html HTTP/1.1" 200 9717
          "https://pikt.org/pikt/samples/httpd_google_googlebot_alarms.cfg.html"
          "Mozilla/4.0 (compatible; MSIE 6.0; Win 9x 4.90; .NET CLR 1.1.4322)"
        63.14.59.22 - - [09/Sep/2005:13:41:28 -0500]
          "GET /pikt/samples/httpd_google_mediapartners_alarms.cfg.html HTTP/1.1"
          200 10006 "https://pikt.org/pikt/samples/alarms.cfg_include_files.html"
          "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
        ...

The script follows.

GoogleHttpdLogEntries

        init
                status =piktstatus
                level =piktlevel
                task "Report new Google-related httpd log entries"
                input logfile "=httpd_logdir/access_log_pikt"
                filter "=httpd_cgibin_root/webstats.pl -f -p -sp"

        rule
                =bypass_local

        rule
                if    $inlin =~~ "get /samples/[[:graph:]]+google[[:graph:]]+\\.html"
                   || $inlin =~~ "get /pikt/samples/[[:graph:]]+google[[:graph:]]+\\.html"
                        output mail $inlin
                endif

The GoogleHttpdLogEntries script makes use of a special filter program, webstats.pl (not shown), to output log entries representing actual human accesses (i.e., filtering out accesses by automated processes like robots, spiders, crawlers, and the like). 

In the first script rule, we also use the =bypass_local macro

bypass_local    if    $inlin =~~ "^57\\.141\\.82\\."
                   || $inlin =~~ "^57\\.151\\.195\\."
                   || $inlin =~~ "^(pikt\\.org|www\\.pikt\\.org)"
                   || $inlin =~~ "^123\\.200\\.132\\.101"
                        next
                endif

to exclude local accesses--our accessing our own pages.  (We could have the webstats.pl program do the local filtering, but for various reasons we prefer to let the Pikt script do it.)

You could also write scripts to report new httpd log entries for other pages of special interest.  For example, at pikt.org, we use the DownloadPIKTHttpdLogEntries script (not shown) to report when site visitors download the PIKT software-and-documentation package (pikt-current.tar.gz).

For more examples, see Samples.