diff options
Diffstat (limited to 'qa/README')
| -rw-r--r-- | qa/README | 227 |
1 files changed, 227 insertions, 0 deletions
diff --git a/qa/README b/qa/README new file mode 100644 index 0000000..8bb9edd --- /dev/null +++ b/qa/README @@ -0,0 +1,227 @@ +Notes on using the PCP QA Suite +=============================== + +Preliminaries +------------- + + PCP needs to be installed on the local host, with pmcd operational. + The sample PMDA needs to be installed. + +Basic getting started +--------------------- + + There is some local configuration needed ... check the file + "common.config" ... this script uses heuristics to set a number of + interesting variables, specifically: + + $PCPQA_CLOSE_X_SERVER + The $DISPLAY setting for an X server that is willing to accept + connections from X clients running on the local machine. + + $PCPQA_FAR_PMCD + The hostname for a host running pmcd, but the host is preferably a + long way away (over a WAN) for timing test. + + $PCPQA_HYPHEN_HOST + The hostname for a host running pmcd, with a hyphen (-) in the + hostname. + + Next, mk.qa_hosts is a script that includes heuristics for selecting and + sorting the list of potential remote PCP QA hosts (qa_hosts.master). + Refer to the comments in qa_hosts.master, and make appropriate changes. + + For each of the potential remote PCP QA hosts, the following must be + set up: + + (a) PCP installed, + (b) pmcd(1) running, + (c) a login for the user "pcpqa" needs to be created, and then set + up in such a way that ssh/scp will work without the need for any + password, i.e. these sorts of commands + $ ssh pcpqa@pcp-qa-host some-command + $ scp some-file pcpqa@pcp-qa-host:some-dir + must work correctly when run from the local host. + The "pcpqa" user's environment must also be initialized + so that their shell's path includes all of the PCP binary + directories, so that all PCP commands are executable without + full pathnames. Of most concern would be auxilliary directory + (usually /usr/pcp/bin, /usr/share/pcp/bin or /usr/libexec/pcp/bin) + where commands like pmlogger(1), pmhostname(1), mkaf(1), etc.) are + installed. + + Once you've modified common.config and qa_hosts.master, then run + "chk.setup" to validate the settings. + + For test 051 we need five local hostnames that are valid, although + PCP does not need to be installed there, nor pmcd(1) running. The + five hosts listed in 051.hosts (the comments at the start of this + file explain what is required) should suffice for most installations. + + The PCP QA tests are designed to be run by a non-root user. Where + root privileges are needed, e.g. to stop or start pmcd, install/remove + PMDAs, etc. the "sudo" application is used. When using sudo for QA, + your local or pcpqa user needs to be able to execute commands as root + without being prompted for a password. This can be achieved by adding + the following line to the /etc/sudoers file (or in more recent versions + of sudo, a /etc/sudoers.d/pcpqa file): + + pcpqa ALL=(ALL) NOPASSWD: ALL + + Some tests are graphical, and wish to make use of your display. + For authentication to success, you may find you need to perform + some access list updates, e.g. "xhost +local:" for such tests to + pass (e.g. test 325). + + You can now verify your QA setup, by running: + + ./check 000 + + The first time you run "check" (see below) it will descend into the + src directory (see below) and make all of the QA test programs and + dynamic PCP archives, so some patience may be required. + + If test 000 fails, it may be that you have locally developed PMDAs or + optional PMDAs installed. Edit common.filter, and modify the + _filter_top_pmns() procedure to strip the top-level name components for + any new metric names (there are lots of examples already there) ... if + these are distributed PMDAs, you should send patches back to + pcp@mail.performancecopilot.org. + + +Doing the Real Work +------------------- + + + check ... + This script runs tests and verifies the output. In general, + test NNN is expected to terminate with an exit status of 0, + no core file and produce output that matches that in the file + NNN.out ... failures leave the current output in NNN.out.bad, + and may leave a more verbose trace that is useful for diagnosing + failures in NNN.full. + + The command line options to check are: + + NNN run test NNN (leading zeros will be added as necessary + to the test sequence number, so 00N and N are equivalent) + + NNN- all tests >= NNN + + NNN-MMM all tests in the range NNN ... MMM + + -l diffs in line mode (the default is to use xdiff or similar) + + -n show me, do not run any tests + + -q quick mode, by-pass the initial setup integrity checks + (recommended that you do not use this the first time, nor + if the last run test failed) + + -g xxx include tests from a named group (xxx) ... refer to the + "groups" file + + -x xxx exclude tests from a named group (xxx) ... refer to the + "groups" file + + If none of the NNN variants or -g is specified, then the default + is to run all tests. + + Each of the NNN scripts that may be run by check follows the + same basic scheme: + + - include some optional shell procedures and set variables to + define the local configuration options + - optionally, check the run-time environment to see if it makes + sense to run the test at all, and if not echo the reason to the + file NNN.notrun and exit ... check will notice the NNN.notrun + file and skip any testing of the exit status or comparison + of output + - define $tmp as a prefix to be used for all temporary files, and + install a trap handler to remove temporary files when the scipt + exits + - optionally, check the run-time environment to choose one of + a number of expected output formats, and link the selected + file to NNN.out ... if the same output is expected in all + environments, the NNN.out file will already exist as part of + the PCP QA distribution + - run the test + - optionally save all the output in the file NNN.full ... this + is only useful for debugging test failures + - filter the output to produce deterministic output that will + match NNN.out if the test has been successful + + remake NNN + This script creates a new NNN.out file. Since the NNN.out files + are precious, and reflect the state of the qualified and expected + output, they should typically not be changed unless some change + has been made to the NNN script or the filters it uses. + + new + Make sure "group" is writeable, then run "new" to + create the skeletal framework of a new test. + + It is strongly suggested that you base your test on an existing test + ... pay particular attention to making the output deterministic so + the test uses the "not run" protocols (see 009 and check for + examples) to avoid running the test (and hence failing) if an + optional application, feature or platform is not available, and uses + appropriate filters (see common.filter for lots of useful filters + already packaged as shell procedures). + + show-me ... + Report differences between the NNN.out and NNN.out.bad files. + By default, uses all of the NNN.out.bad files in the current + directory, but can also specify test numbers or ranges of test + numbers on the command line. + + Other options may be used to fetch good and bad output files from + various exotic remote locations (refer to the script). + + +Make in the src Directory +----------------------------- + + The src directory contains a number of test applications that are + designed to exercise some of the more exotic corners of the PCP + functionality. + + In making these applications, you may see this ... + + Error: trace_dev.h and ../../src/include/trace_dev.h are different! + make: [trace_dev.h] Error 1 (ignored) + + this is caused by the source for the pcp_trace library being out of sync + with the src applications. If this happens, please ... + + 1. cd src + 2. diff -u trace_dev.h ../../src/include/trace_dev.h + and mail the differences to pcp@mail.performancecopilot.org so we can + refine the Makefiles to avoid cosmetic differences + 3. mv trace_dev.h trace_dev.h.orig + cp ../../src/include/trace_dev.h trace_dev.h + 4. make + + +008 Issues +---------- + + Test 008 depends on the local disk configuration, so you need to make + your own 008.out file (or rather a variant that 008 will link to 008.out + when the test is run). Refer to the 008 script, but here is the basic + recipe: + + $ touch touch 008.out.`hostname` + $ remake 008 + $ mv 008.out 008.out.`hostname` + + Be aware that it can be adversely influenced by temporary disks like USB + sticks, mobile phones, or other transient storage that may come and go in + your test systems. + + +Fixes +----- + + If you find something that does not work, and fix it, or create additional + QA tests, please send the details to pcp@mail.performancecopilot.org. + |