path: root/man/man1/pmdaweblog.1

'\"macro stdmacro
.\"
.\" Copyright (c) 2012 Red Hat.
.\" Copyright (c) 2000 Silicon Graphics, Inc.  All Rights Reserved.
.\" 
.\" This program is free software; you can redistribute it and/or modify it
.\" under the terms of the GNU General Public License as published by the
.\" Free Software Foundation; either version 2 of the License, or (at your
.\" option) any later version.
.\" 
.\" This program is distributed in the hope that it will be useful, but
.\" WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
.\" or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
.\" for more details.
.\" 
.\"
.TH PMDAWEBLOG 1 "PCP" "Performance Co-Pilot"
.SH NAME
\f3pmdaweblog\f1 \- performance metrics domain agent (PMDA) for Web server logs
.\" literals use .B or \f3
.\" arguments use .I or \f2
.SH SYNOPSIS
\f3$PCP_PMDAS_DIR/weblog/pmdaweblog\f1
[\f3\-Cp\f1]
[\f3\-d\f1 \f2domain\f1]
[\f3\-h\f1 \f2helpfile\f1]
[\f3\-i\f1 \f2port\f1]
[\f3\-l\f1 \f2logfile\f1]
[\f3\-n\f1 \f2idlesec\f1]
[\f3\-S\f1 \f2num\f1]
[\f3\-t\f1 \f2delay\f1]
[\f3\-u\f1 \f2socket\f1]
[\f3\-U\f1 \f2username\f1]
\f2configfile\f1
.SH DESCRIPTION
.B pmdaweblog
is a Performance Metrics Domain Agent
.RB ( PMDA (3))
that scans Web server logs
to extract metrics characterizing Web server activity.
These performance metrics are then made available through the infrastructure
of the Performance Co-Pilot (PCP).
.PP
The
.I configfile
specifies which Web servers are to be monitored, their associated access
logs and error logs, and a regular-expression based scheme for extracting
detailed information about each Web access.  This file is maintained as
part of the PMDA installation and/or de-installation by the scripts
.B Install
and
.B Remove
in the directory
.BR $PCP_PMDAS_DIR/weblog .
For more details, refer to the section below covering installation.
.PP
Once started,
.B pmdaweblog
monitors a set of log files and in response to a request for information,
will process any new information that has been appended to the log files,
similar to a
.BR tail (1).
There is also periodic "catch up" to process new information from all
log files, and a scheme to detect the rotation of log files.
.PP
Like all other PMDAs,
.B pmdaweblog
is launched by
.BR pmcd (1)
using command line options specified in
.I $PCP_PMCDCONF_PATH
\- the
.B Install
script will prompt for appropriate values for the command line options, and
update
.IR $PCP_PMCDCONF_PATH .
.PP
A brief description of the
.B pmdaweblog
command line options follows:
.TP
.B \-C
Check the configuration and exit.
.TP
.BI \-d " domain"
Specify the
.I domain
number.  It is absolutely crucial that the performance metrics
.I domain
number specified here is unique and consistent.  That is,
.I domain
should be different for every PMDA on the one host, and the same
.I domain
number should be used for the
.B pmdaweblog
PMDA on all hosts.
.RS
.P
For most installations, the default
.I domain
as encapsulated in the file
.B $PCP_PMDAS_DIR/weblog/domain.h
will suffice.  For alternate values, check
.I $PCP_PMCDCONF_PATH
for the
.I domain
values already in use on this host, and the file
.B $PCP_VAR_DIR/pmns/stdpmid
contains a repository of ``well known''
.I domain
assignments that probably should be avoided.
.RE
.TP
.BI \-h " helpfile"
Get the help text from the supplied
.I helpfile
rather than from the default location.
.TP
.BI \-i " port"
Communicate with
.BR pmcd (1)
on the specified Internet
.I port
(which may be a number or a name).
.TP
.BI \-l " logfile"
Location of the log file.  By default, a log file named
.I weblog.log
is written in the current directory of
.BR pmcd (1)
when
.B pmdaweblog
is started, i.e.
.BR $PCP_LOG_DIR/pmcd .
If the log file cannot
be created or is not writable, output is written to the standard error instead.
.TP
.BI \-n " idlesec"
If a Web server log file has not been modified for
.IR idlesec
seconds, then the file will be closed and re-opened.
This is the only way
.B pmdaweblog
can detect any asynchronous rotation of the logs by Web server
administrative scripts.
The default period is 20 seconds.
This value may be changed dynamically using
.BR pmstore (1)
to modify the value of the performance metric
.BR web.config.check .
.I
.TP
.B \-p
Communicate with
.BR pmcd (1)
via a pipe.
.TP
.BI \-S " num"
Specify the maximum number of Web servers per
.IR sproc .
It may be desirable (from a latency and load balancing perspective) or
necessary (due to file descriptor limits) to delegate responsibility
for scanning the Web server log files to several
.IR sprocs .
.B pmdaweblog
will ensure that each
.I sproc
handles the log files for at most
.I num
Web servers.
The default value is 80 Web servers per
.IR sproc .
.TP
.BI \-t " delay"
To avoid the need to scan a lot of information from the Web
server logs in response to a single request for performance
metrics, all log files will be checked at least once
every
.I delay
seconds.
The default is 15 seconds.
This value may by changed dynamically using
.BR pmstore (1)
to modify the value of the performance metric
.BR web.config.catchup .
.TP
.BI \-u " socket"
Communicate with
.BR pmcd (1)
via the given Unix domain
.IR socket .
.TP
.B \-U
User account under which to run the agent.
The default is the unprivileged "pcp" account in current versions of PCP,
but in older versions the superuser account ("root") was used by default.
.SH INSTALLATION
The PCP framework allows metrics to be collected on one host
and monitored from another.  These hosts are referred to as 
.I collector 
and 
.I monitor 
hosts, respectively.  A host may be both a collector and a monitor.
.PP
Collector hosts require the installation of the agent, while monitoring
hosts require no agent installation at all.
.PP
For collector hosts do the following as root:
.PP
.ft CW
.nf
.in +0.25i
# cd $PCP_PMDAS_DIR/weblog
# ./Install
.in
.fi
.ft 1
.PP
The installation procedure prompts for a default or non-default installation.
A default installation will search for known server configurations and
automatically configure the PMDA for any server log files that are found.
A non-default installation will step through each server, prompting the
user for other server configurations and arguments to
.BR pmdaweblog .
The end result of a collector installation
is to build a configuration file that is passed to
.B pmdaweblog
via the
.I configfile
argument.
.PP
If you want to undo the installation, do the following as root:
.PP
.ft CW
.nf
.in +0.25i
# cd $PCP_PMDAS_DIR/weblog
# ./Remove
.in
.fi
.ft 1
.PP
.B pmdaweblog
is launched by
.BR pmcd (1)
and should never be executed directly.
The
.B Install
and
.B Remove 
scripts notify
.BR pmcd (1)
when the agent is installed or removed.
.SH CONFIGURATION
The configuration file for the weblog PMDA is an ASCII file that can
be easily modified.
Empty lines and lines beginning with '\f3#\f1'
are ignored.
All other lines must be either a regular expression or server
specification.
.PP
Regular expressions, which are used on both the access and error log files,
must be of the form:
.PP
.in +0.25i
.B regex
.I regexName regexp
.in
.I or
.PP
.in +0.25i
.B regex_posix
.I regexName ordering regexp_posix
.in
.PP
The
.I regexName
is a word which uniquely identifies the regular expression.
This is the reference used in the server specification.
The
.I regexp
for access logs is in the format described for
.BR regcmp (3).
The
.I regexp_posix
for access logs is in the format described for
.BR regcomp (3).
The argument
.I ordering
is explained below. The
.B Posix
form should be available on all platforms.
.PP
The regular expression requires the specification of up to four arguments
to be extracted from each line of a Web server access log, depending on the
type of server. In the most common case there are two arguments representing
the method and the size.
.PP
For the non\-
.B Posix
version, argument
.I $0
should contain the method:
.BR GET ,
.B HEAD ,
.B POST
or
.BR PUT .
The method
.B PUT
is treated as a synonym for
.BR POST ,
and anything else is categorized as
.BR OTHER .
.PP
The second argument,
.IR $1 ,
should contain the size of the request.
A size of ``\f3\-\f1'' or `` '' is treated as unknown.
.PP
Argument 
.I $3
should contain the status code returned to the client browser and argument 
.I $4
should contain the status code returned to the server from a remote host.
These latter two arguments are used for caching servers and must be specified
as a pair (or 
.I $3
will be ignored). For further information on status codes, refer to the
web site
.B http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
.PP
Some legal non\-
.B Posix
regex expression specifications for monitoring an access log are:
.PP
.ft CW
.nf
.in +0.25i
# pattern for CERN, NCSA, Netscape etc Access Logs
regex CERN ] "([A\-Za\-z][\-A\-Za\-z]+)$0 .*" [\-0\-9]+ ([\-0\-9]+)$1

# pattern for FTP Server access logs (normally in SYSLOG)
regex SYSLOG_FTP ftpd[.*]: ([gp][\-A\-Za\-z]+)$0( )$1
.in
.fi
.ft 1
.PP
There is 1 special types of access logs with the 
.I RegexName
.I SQUID.
This formats extract 4 parameters but since the 
.B Squid
log file uses text-based status codes, it is handled as a special case.
.PP
In the examples below,
.I NS_PROXY
parses the Netscape/W3C
.I Common Extended Log Format
and
.I SQUID
parses the default Squid Object Cache format log file.
.PP
.ft CW
.nf
.in +0.25i
# pattern for Netscape Proxy Server Extended Logs
regex NS_PROXY ] "([A\-Za\-z][\-A\-Za\-z]+)$0 .*" ([\-0\-9]+)$2 \\
.in +0.5i
([\-0\-9]+)$1 ([\-0\-9]+)$3
.in

# pattern for Squid Cache logs
regex SQUID [0\-9]+\.[0\-9]+[ ]+[0\-9]+ [a\-zA\-Z0\-9\.]+ \\
.in +0.5i
([_A\-Z]+)$3\/([0\-9]+)$2 ([0\-9]+)$1 ([A\-Z]+)$0
.in
.in
.fi
.ft 1
.PP
The
.I regexp
for the error logs does not require any arguments, only a match.
Some legal
expressions are:
.PP
.ft CW
.nf
.in +0.25i
# pattern for CERN, NCSA, Netscape etc Error Logs
regex CERN_err .

# pattern for FTP Server error logs (normally in SYSLOG)
regex SYSLOG_FTP_err FTP LOGIN FAILED
.in
.fi
.ft 1
.PP
If
.B POSIX
compliant regular expressions are used, additional information is required 
since the order of parameters cannot be specified in the regular expression. 
For backwards compatibility, the common case of two parameters the order 
may be specified as
.I method,size
or
.I size,method
In the general case, the ordering is specified by one of the following
methods:
.TP 0.5in
n1,n2,n3,n4
where nX is a digit between 1 and 4. Each comma-seperated field represents
(in order) the argument number for
.I method,size,client_status,server_status
.TP 0.5in
-
Used for cases like the error logs where the content is ignored.
.PP
As for the non-
.B Posix
format, the 
.I SQUID
RegexName is treated as a special case to match the non-numerical status codes.
.PP
Some legal 
.B Posix
regex expression specifications for monitoring an access log are:
.PP
.ft CW
.nf
.in +0.25i
# pattern for CERN, NCSA, Netscape, Apache etc Access Logs
regex_posix CERN method,size ][ \\]+"([A\-Za\-z][\-A\-Za\-z]+) \\
.in +0.5i
[^"]*" [\-0\-9]+ ([\-0\-9]+)
.in

# pattern for CERN, NCSA, Netscape, Apache etc Access Logs
regex_posix CERN 1,2 ][ \\]+"([A\-Za\-z][\-A\-Za\-z]+) \\
.in +0.5i
[^"]*" [\-0\-9]+ ([\-0\-9]+)
.in

# pattern for FTP Server access logs (normally in SYSLOG)
regex_posix SYSLOG_FTP method,size ftpd[.*]: \\
.in +0.5i
([gp][\-A\-Za\-z]+)( )
.in

# pattern for Netscape Proxy Server Extended Logs
regex_posix NS_PROXY 1,3,2,4 ][ ]+"([A\-Za\-z][\-A\-Za\-z]+) \\
.in +0.5i
[^"]*" ([\-0\-9]+) ([\-0\-9]+) ([\-0\-9]+)
.in

# pattern for Squid Cache logs
regex_posix SQUID 4,3,2,1 [0\-9]+\.[0\-9]+[ ]+[0\-9]+ \\
.in +0.5i
[a\-zA\-Z0\-9\.]+ ([_A\-Z]+)\/([0\-9]+) ([0\-9]+) ([A\-Z]+)
.in

# pattern for CERN, NCSA, Netscape etc Error Logs
regex_posix CERN_err \- .

# pattern for FTP Server error logs (normally in SYSLOG)
regex_posix SYSLOG_FTP_err \- FTP LOGIN FAILED
.in
.fi
.ft 1

.PP
A Web server can be specified using this syntax:
.PP
.ft CW
.nf
.in +0.25i
\f3server \f2serverName \f3on\f2|\f3off \f2accessRegex accessFile errorRegex errorFile
.in
.fi
.ft 1
.PP
The
.I serverName
must be unique for each server, and is the name given to the instance
for the associated performance metrics.
See
.BR PMAPI (3)
for a discussion of PCP instance domains.
The
.B on
or
.B off
flag indicates whether the server is to be monitored when the PMDA is
installed.
This can altered dynamically using
.BR pmstore (1)
for the metric
.BR web.perserver.watched ,
which has one instance for each Web server named in
.IR configfile .
.PP
Two files are monitored for each Web server, the access and the error log.
Each file requires the name of a previously declared regular expression,
and a file name.
The log files specified for each server do not
have to exist when the weblog PMDA is installed.
The PMDA will continue
to check for non-existent log files and open them when possible.
Some legal server specifications are:
.PP
.ft CW
.nf
.in +0.25i
# Netscape Server on Port 80 at IP address 127.55.555.555
server 127.55.555.555:80 on CERN /logs/access CERN_err /logs/errors

# FTP Server.
server ftpd on SYSLOG_FTP /var/log/messages SYSLOG_FTP_err /var/log/messages
.in
.fi
.ft 1
.SH CAVEATS
Specifying regular expressions with an incorrect number of arguments, anything other
than 2 for access logs, and none for error logs, may cause the PMDA to behave 
incorrectly and even crash. This is due to limitations in the interface of
.BR regex (3).
.SH FILES
.TP 10
.B $PCP_PMDAS_DIR/weblog
installation directory for the weblog PMDA
.TP
.B $PCP_PMDAS_DIR/weblog/Install
installation script for the weblog PMDA
.TP
.B $PCP_PMDAS_DIR/weblog/Remove
de-installation script for the weblog PMDA
.TP
.B $PCP_LOG_DIR/pmcd/weblog.log
default log file for error reporting
.TP
.I $PCP_PMCDCONF_PATH
.B pmcd
configuration file that specifies the command line options
to be used when
.B pmdaweblog
is launched
.TP
.B $PCP_LOG_DIR/NOTICES
log of PMDA installations and removals
.TP
.B $PCP_VAR_DIR/config/web/weblog.conf
likely location of the weblog PMDA configuration file
.TP
.B $PCP_DOC_DIR/pcpweb/index.html
the online HTML documentation for PCPWEB
.SH "PCP ENVIRONMENT"
Environment variables with the prefix
.B PCP_
are used to parameterize the file and directory names
used by PCP.
On each installation, the file
.B /etc/pcp.conf
contains the local values for these variables.
The
.B $PCP_CONF
variable may be used to specify an alternative
configuration file,
as described in
.BR pcp.conf (5).
.SH SEE ALSO
.BR pmcd (1),
.BR pmchart (1),
.BR pmdawebping (1),
.BR pminfo (1),
.BR pmstore (1),
.BR pmview (1),
.BR tail (1),
.BR weblogvis (1),
.BR webvis (1),
.BR PMAPI (3),
.BR PMDA (3)
and
.BR regcmp (3).