path: root/books/PCP_PG/pcp-programmers-guide.xml

<?xml version="1.0"?>
<!DOCTYPE book PUBLIC "-//Norman Walsh//DTD DocBk XML V4.1.2//EN" "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd">
<book status="final" security="public"><title>Performance Co-Pilot&#x2122; Programmer's Guide</title>
<bookinfo><edition>3</edition>

<othercredit>
<contrib>Maintained by</contrib>
<affiliation>
<orgname>The Performance Co-Pilot Development Team</orgname>
<address>
<email>pcp@mail.performancecopilot.org</email>
<otheraddr>
<ulink url="http://www.performancecopilot.org"/>
<inlinemediaobject><imageobject><imagedata fileref="figures/pcp.svg"/></imageobject></inlinemediaobject>
</otheraddr>
</address>
</affiliation>
</othercredit>

<copyright>
<year>2000</year>
<year>2013</year>
<holder>Silicon Graphics, Inc.</holder>
</copyright>

<copyright>
<year>2013</year>
<year>2014</year>
<holder>Red Hat, Inc.</holder>
</copyright>

<legalnotice>
<title>LICENSE</title>
<para>Permission is granted to copy, distribute, and/or modify this document under
the terms of the Creative Commons Attribution-Share Alike, Version 3.0 or any
later version published by the Creative Commons Corp.
A copy of the license is available at
<ulink url="http://creativecommons.org/licenses/by-sa/3.0/us/"/></para>
</legalnotice>

<legalnotice>
<title>TRADEMARKS AND ATTRIBUTIONS</title>
<para>Silicon Graphics, SGI and the SGI logo are registered trademarks
and Performance Co-Pilot is a trademark of Silicon Graphics, Inc.</para>
<para>Red Hat and the Shadowman logo are trademarks of Red Hat, Inc.,
registered in the United States and other countries.</para>
<para>Cisco is a registered trademark of Cisco Systems, Inc.
Linux is a registered trademark of Linus Torvalds, used with permission.
UNIX is a registered trademark of The Open Group.</para>
</legalnotice>

<revhistory>
<revision><revnumber>007</revnumber><date>September 2013</date><revremark>Revised to include event metrics and the MMV PMDA coverage.</revremark></revision>
<revision><revnumber>006</revnumber><date>August 2013</date><revremark>Revised to support the Performance Co-Pilot 3.8 release.</revremark></revision>
<revision><revnumber>005</revnumber><date>December 2002</date><revremark>Revised to support the Performance Co-Pilot 2.3 release.</revremark></revision>
<revision><revnumber>004</revnumber><date>March 2001</date><revremark>Revised to support the Performance Co-Pilot 2.2 release.</revremark></revision>
<revision><revnumber>003</revnumber><date>July 1999</date><revremark>Revised to support the Performance Co-Pilot 2.1 release.</revremark></revision>
</revhistory>

</bookinfo>
<toc/>


<preface id="id5178752">

<title>About This Guide</title>
<para>This guide describes how to program the Performance Co-Pilot (PCP) performance analysis toolkit.
PCP provides a systems-level suite of tools that cooperate to deliver
distributed performance monitoring and performance management services spanning
hardware platforms, operating systems, service layers, database internals,
user applications and distributed architectures.</para>
<para>PCP is an open source, cross-platform software package - customizations, extensions,
source code inspection, and tinkering in general is actively encouraged.</para>
<para>&#x201C;About This Guide&#x201D; includes short descriptions of the chapters
in this book, directs you to additional sources of information, and explains
typographical conventions.</para>
<section id="id5178771">

<title>What This Guide Contains</title>
<para>This guide contains the following chapters:</para>
<itemizedlist>
<listitem><para><xref linkend="LE21795-PARENT"/>, contains a thumbnail sketch of how to program the various PCP components.</para>
</listitem>
<listitem><para><xref linkend="LE98072-PARENT"/>, describes how to write Performance Metrics Domain Agents (PMDAs) for PCP.</para>
</listitem>
<listitem><para><xref linkend="LE97135-PARENT"/>, describes the interface that allows you to design custom performance monitoring tools.</para>
</listitem>
<listitem><para><xref linkend="LE25915-PARENT"/>, introduces techniques, tools and interfaces to assist with exporting performance data from within applications.</para>
</listitem>
<listitem><para><xref linkend="LE54271-PARENT"/>, provides a comprehensive list of the acronyms used in this guide, in the PCP man pages, and in the release notes.</para>
</listitem></itemizedlist>
</section>
<section id="id5178891">

<title>Audience for This Guide</title>
<para>The guide describes the programming interfaces to Performance Co-Pilot (PCP) for the following intended audience:</para>
<itemizedlist>
<listitem><para>Performance analysts or system administrators who want to extend or customize performance monitoring tools available with PCP</para>
</listitem>
<listitem><para>Developers who wish to integrate performance data from within their applications into the PCP framework</para>
</listitem></itemizedlist>
<para>This book is written for those who are competent with the C programming language, the UNIX or the Linux operating systems, and the target domain from which the desired performance metrics are to be extracted. Familiarity with the PCP tool suite is assumed.</para>
</section>
<section id="id5178933">

<title>Related Resources</title>
<para>The <citetitle>Performance Co-Pilot User's and Administrator's Guide</citetitle>
is a companion document to the <citetitle>Performance Co-Pilot Programmer's Guide</citetitle>,
and is intended for system administrators and performance analysts who are directly
using and administering PCP installations.</para>
<para>The <citetitle>Performance Co-Pilot Tutorials and Case Studies</citetitle>
provides a series of real-world examples of using various PCP tools, and
lessons learned from deploying the toolkit in production environments.
It serves to provide reinforcement of the general concepts discussed in the
other two books with additional case studies, and in some cases very detailed
discussion of specifics of individual tools.
</para>
<para>Additional resources include man pages and the project web site.</para>
</section>
<section id="id5178968">

<title>Man Pages</title>
<para>The operating system man pages provide concise reference information on the use of commands, subroutines, and system resources. There is usually a man page for each PCP command or subroutine. To see a list of all the PCP man pages, start from the following command:</para>
<literallayout class="monospaced"><userinput>man PCPIntro</userinput></literallayout>
<para>Each man page usually has a &quot;SEE ALSO&quot; section, linking to other, related entries.</para>
<para>To see a particular man page, supply its name to the <literal>man</literal> command, for example:</para>
<literallayout class="monospaced"><userinput>man pcp</userinput></literallayout>
<para>The man pages are arranged in different sections separating commands, programming interfaces, and so on.
For a complete list of manual sections on a platform enter the command:</para>
<literallayout class="monospaced"><userinput>man man</userinput></literallayout>
<para>When referring to man pages, this guide follows a standard convention: the section number in parentheses follows the item. For example, <command>pminfo(1)</command> refers to the man page in section 1 for the <command>pminfo</command> command.</para>
</section>
<section id="id5179157">

<title>Web Site</title>
<para>The following web site is accessible to everyone:</para>
<variablelist condition="sgi_termlength:wide">
<varlistentry>
<term><emphasis role="bold">URL</emphasis></term><listitem><para><emphasis role="bold">Description</emphasis></para></listitem></varlistentry>
<varlistentry>
<term><ulink url="http://www.performancecopilot.org">http://www.performancecopilot.org</ulink></term>
<listitem><para>PCP is open source software released under
the GNU General Public License (GPL) and GNU Lesser General Public License (LGPL)</para>
</listitem></varlistentry>
</variablelist>
</section>
<section id="id5179276">

<title>Conventions</title>
<para>The following conventions are used throughout this document:<variablelist>
<varlistentry>
<term><emphasis role="bold">Convention</emphasis></term><listitem><para><emphasis role="bold">Meaning</emphasis></para></listitem></varlistentry>
<varlistentry>
<term><literal>${PCP_VARIABLE}</literal></term>
<listitem><para>A brace-enclosed all-capital-letters syntax indicates a variable
that has been sourced from the global <filename>/etc/pcp.conf</filename> file.
These special variables indicate parameters that affect all PCP commands,
and are likely to be different between platforms.</para>
</listitem>
</varlistentry>

<varlistentry>
<term><literal>command</literal></term>
<listitem><para>This fixed-space font denotes literal items such as commands,
files, routines, path names, signals, messages, and programming language
structures. </para>
</listitem>
</varlistentry>

<varlistentry>
<term><replaceable>variable</replaceable></term>
<listitem><para>Italic typeface denotes variable entries and words or concepts being
defined.</para>
</listitem></varlistentry>

<varlistentry>
<term><userinput>user input</userinput></term>
<listitem><para>This bold, fixed-space font denotes literal  items that the user enters in interactive  sessions.  (Output is shown in nonbold,  fixed-space font.)</para>
</listitem></varlistentry>

<varlistentry>
<term>[ ]</term>
<listitem><para>Brackets enclose optional portions of a  command or directive line.</para>
</listitem></varlistentry>

<varlistentry>
<term>...</term>
<listitem><para>Ellipses indicate that a preceding element can be repeated.</para>
</listitem></varlistentry>
<varlistentry>
<term>ALL CAPS</term>
<listitem><para>All capital letters denote environment variables, operator names, directives, defined constants, and macros in C programs.</para>
</listitem></varlistentry>
<varlistentry>
<term>()</term>
<listitem><para>Parentheses that follow function names surround function arguments or are empty if the function has no arguments; parentheses that follow commands surround man page section numbers.</para>
</listitem></varlistentry>
</variablelist></para>

</section>
<section id="z825546061melby">

<title>Reader Comments</title>
<para>If you have comments about the technical accuracy, content, or organization of this document, contact the PCP maintainers using either the email address or the web site listed earlier.</para>
<para>We value your comments and will respond to them promptly.</para>
</section>
</preface>


<chapter id="LE21795-PARENT">

<title>Programming Performance Co-Pilot</title>
<para><indexterm id="IG313401770"><primary>PCP</primary><secondary>description</secondary></indexterm>Performance Co-Pilot (PCP) provides a systems-level suite of tools that cooperate to deliver distributed, integrated performance management services. PCP is designed for the in-depth analysis and sophisticated control that are needed to understand and manage the hardest performance problems in the most complex systems.</para>
<para>PCP provides unparalleled power to quickly isolate and understand performance behavior, resource utilization, activity levels and performance bottlenecks.</para>
<para>Performance data may be collected and exported from multiple sources, most notably the hardware platform, the operating system kernel, layered services, and end-user applications.</para>
<para><indexterm id="IG313401771"><primary>programming components</primary></indexterm> <indexterm id="IG313401772"><primary>audience</primary></indexterm>There are several ways to extend PCP by programming certain of its components:</para>
<itemizedlist>
<listitem><para><indexterm id="Z963349317sdc"><primary>Performance Metrics Domain Agent </primary><see>PMDA</see></indexterm><indexterm id="IG313401773"><primary>PMDA</primary><secondary>introduction</secondary></indexterm>By writing a Performance Metrics Domain Agent (PMDA) to collect performance metrics from an uncharted performance domain (<xref linkend="LE98072-PARENT"/>)</para>
</listitem>
<listitem><para><indexterm id="Z963349400sdc"><primary>Performance Metrics Application Programming Interface </primary><see>PMAPI</see></indexterm><indexterm id="IG313401774"><primary>PMAPI</primary><secondary>introduction</secondary></indexterm>By creating new analysis or visualization tools using documented functions from the Performance Metrics Application Programming Interface (PMAPI) (<xref linkend="LE97135-PARENT"/>)</para>
</listitem>
<listitem><para><indexterm id="IG313401775"><primary>performance instrumentation</primary></indexterm><indexterm id="IG313401776"><primary>trace facilities</primary></indexterm>By adding performance instrumentation to an application using facilities from PCP libraries, which offer both sampling and event tracing models.</para>
</listitem></itemizedlist>
<para><indexterm id="IG313401777"><primary>customization</primary></indexterm>Finally, the topic of customizing an installation is covered in the chapter on customizing and extending PCP service in the <citetitle>Performance Co-Pilot User's and Administrator's Guide</citetitle>.</para>
<section id="id5177140">

<title>PCP Architecture</title>
<para><indexterm id="IG313401778"><primary>architecture</primary></indexterm>This section gives a brief overview of PCP architecture. For an explanation of terms and acronyms, refer to <xref linkend="LE54271-PARENT"/>.</para>
<para><indexterm id="IG313401779"><primary>pmchart command</primary></indexterm><indexterm id="IG3134017711"><primary>monitoring tools</primary></indexterm><indexterm id="IG3134017712"><primary>collection tools</primary></indexterm>PCP consists of numerous monitoring and collecting tools. <literal>Monitoring tools</literal> such as <command>pmval</command> and <command>pminfo</command> report on metrics, but have minimal interaction with target systems. <literal>Collection tools</literal>, called PMDAs, extract performance values from target systems, but do not provide user interfaces.</para>
<para>Systems supporting PCP services are broadly classified into two categories:</para>
<variablelist>
<varlistentry>
<term>Collector</term>
<listitem><para>Hosts that have the PMCD and one or more PMDAs running to collect and export performance metrics</para>
</listitem></varlistentry>
<varlistentry>
<term>Monitor</term>
<listitem><para>Hosts that import performance metrics from one or more collector hosts to be consumed by tools to monitor, manage, or record the performance of the collector hosts</para>
</listitem></varlistentry>
</variablelist>
<para>Each PCP enabled host can operate as a collector, or a monitor, or both.</para>
<para><xref linkend="id5177328"/> shows the architecture of PCP. The monitoring tools consume and process performance data using a public interface, the Performance Metrics Application Programming Interface (PMAPI).</para>
<para><indexterm id="IG3134017713"><primary>PMCD</primary><secondary>overview</secondary></indexterm><indexterm id="IG3134017714"><primary>Performance Metrics Collection Daemon </primary><see>PMCD</see></indexterm>Below the PMAPI level is the PMCD process, which acts in a  coordinating role, accepting requests from clients, routing requests to one or more PMDAs, aggregating responses from the PMDAs, and responding to the requesting client.</para>
<para>Each performance metric domain (such as the operating system kernel or a database management system) has a well-defined name space for referring to the specific performance metrics it knows how to collect.</para>
<para><figure id="id5177328"><title>PCP Global Process Architecture</title><mediaobject><imageobject><imagedata fileref="figures/local-collector.svg"/></imageobject><textobject><phrase>PCP Global Process Architecture</phrase></textobject></mediaobject></figure></para>
<section id="id5177343">

<title>Distributed Collection</title>
<para><indexterm id="IG3134017715"><primary>distributed performance management</primary><secondary>metrics collection</secondary></indexterm><indexterm id="IG3134017716"><primary>Cisco PMDA</primary></indexterm><indexterm id="IG3134017717"><primary>Cluster PMDA</primary></indexterm>The performance metrics collection architecture is distributed, in the sense that any monitoring tool may be executing remotely. However, a PMDA is expected to be running on the operating system for which it is collecting performance measurements; there are some notable PMDAs such as Cisco and Cluster that are exceptions, and collect performance data from remote systems.</para>
<para><indexterm id="IG3134017718"><primary>PMCD</primary><secondary>distributed collection</secondary></indexterm><indexterm id="IG3134017719"><primary>collector hosts</primary></indexterm>As shown in <xref linkend="id5177408"/>, monitoring tools communicate only with PMCD. The PMDAs are controlled by PMCD and respond to requests from the monitoring tools that are forwarded by PMCD to the relevant PMDAs on the collector host.</para>
<para><figure id="id5177408"><title>Process Structure for Distributed Operation</title><mediaobject><imageobject><imagedata fileref="figures/remote-collector.svg"/></imageobject><textobject><phrase>Process Structure for Distributed Operation</phrase></textobject></mediaobject></figure></para>
<para>The host running the monitoring tools does not require any collection tools, including PMCD, since all requests for metrics are sent to the PMCD process on the collector host.</para>
<para><indexterm id="IG3134017720"><primary>PMDA </primary><secondary>man page</secondary></indexterm><indexterm id="IG3134017721"><primary>PMAPI</primary><secondary>man page</secondary></indexterm>The connections between monitoring tools and PMCD processes are managed in <filename>libpcp</filename>, below the PMAPI level; see the <command>PMAPI(3)</command> man page. Connections between PMDAs and PMCD are managed by the PMDA functions; see the <command>PMDA(3)</command> and <command>pmcd(1)</command> man pages. There can be multiple monitor clients and multiple PMDAs on the one host, but there may be only one PMCD process.</para>
</section>
<section id="id5177529">

<title>Name Space</title>
<para><indexterm id="IG3134017722"><primary>name space</primary></indexterm><indexterm id="IG3134017723"><primary>Performance Metric Identifier</primary><see>PMID</see></indexterm><indexterm id="IG3134017724"><primary>PMID</primary><secondary>introduction</secondary></indexterm>Each PMDA provides a domain of metrics, whether they be for the operating system, a database manager, a layered service, or an application module. These metrics are referred to by name inside the user interface, and with a numeric  Performance Metric Identifier (PMID) within the underlying PMAPI.</para>
<para><indexterm id="IG3134017725"><primary>clusters</primary></indexterm><indexterm id="IG3134017726"><primary>item numbers</primary></indexterm><indexterm id="IG3134017727"><primary>domains</primary><secondary>fields</secondary></indexterm>The PMID consists of three fields: the domain, the cluster, and the item number of the metric. The domain is a unique number assigned to each PMDA. For example, two metrics with the same domain number must be from the same PMDA. The cluster and item numbers allow metrics to be easily organized into groups within the PMDA, and provide a hierarchical taxonomy to guarantee uniqueness within each PMDA.</para>
<para><indexterm id="IG3134017728"><primary>Performance Metrics Name Space</primary><see>PMNS</see></indexterm>The Performance Metrics Name Space (PMNS) describes the exported performance metrics, in particular the mapping from PMID to external name, and vice-versa.</para>
</section>
<section id="id5177616">

<title>Distributed PMNS</title>
<para> <indexterm id="IG3134017729"><primary>PMNS</primary><secondary>distributed</secondary></indexterm> Performance metric namespace (PMNS) operations are directed by default to the host or archive that is the source of the desired performance metrics.</para>
<para>In <xref linkend="id5177408"/>, both Performance Metrics Collection Daemon (PMCD) processes would respond to PMNS queries from monitoring tools by referring to their local PMNS. If different PMDAs were installed on the two hosts, then the PMNS used by each PMCD would be different, to reflect variations in available metrics on the two hosts.</para>
<para>Although extremely rarely used, the <command>-n</command>&#xA0;<replaceable>pmnsfile</replaceable> command line option may be used with many PCP monitoring tools to force use of a local PMNS file in preference to the PMNS at the source of the metrics.</para>
</section>
<section id="id5177692">

<title>Retrospective Sources of Performance Metrics</title>
<para><indexterm id="IG3134017730"><primary>retrospective analysis</primary></indexterm>The distributed collection architecture described in the previous section is used when PMAPI clients are requesting performance metrics from a real-time or live source.</para>
<para><indexterm id="IG3134017731"><primary>archive logs</primary><secondary>retrospective sources</secondary></indexterm>The PMAPI also supports delivery of performance metrics from a historical source in the form of a PCP archive log. Archive logs  are created using the <command>pmlogger</command> utility, and are replayed in an architecture as shown in <xref linkend="id5177742"/>.</para>
<para><figure id="id5177742"><title>Architecture for Retrospective Analysis</title><mediaobject><imageobject><imagedata fileref="figures/retrospective-architecture.svg"/></imageobject><textobject><phrase>Architecture for Retrospective Analysis</phrase></textobject></mediaobject></figure></para>
</section>
</section>
<section id="LE13618-PARENT">

<title>Overview of Component Software</title>
<para><indexterm id="IG3134017732"><primary>software</primary></indexterm><indexterm id="IG3134017733"><primary>component software</primary></indexterm>Performance Co-Pilot (PCP) is composed of text-based tools, optional graphical tools, and related commands. Each tool or command is fully documented by a man page. These man pages are named after the tools or commands they describe, and are accessible through the <command>man</command> command. For example, to see the <command>pminfo(1)</command> man page for the <command>pminfo</command> command, enter this command:<literallayout class="monospaced"><userinput>man pminfo</userinput></literallayout></para>
<para>A list of PCP developer tools and commands, grouped by functionality, is provided in the following section.</para>
<section id="id5177879">

<title>Application and Agent Development</title>
<para><indexterm id="ITch01-114"><primary>application programs</primary></indexterm><indexterm id="IG3134017734"><primary>PCP</primary><secondary>tool summaries</secondary></indexterm>The following PCP tools aid the development of new programs to consume performance data, and new agents to export performance data within the PCP framework:</para>
<variablelist>
<varlistentry>
<term><command>chkhelp</command></term>
<listitem><para><indexterm id="IG3134017735"><primary>chkhelp tool</primary></indexterm>Checks the consistency of performance metrics help database files.</para>
</listitem></varlistentry>
<varlistentry>
<term><command>dbpmda</command></term>
<listitem><para><indexterm id="IG3134017736"><primary/></indexterm>Allows PMDA behavior to be exercised and tested. It is an interactive debugger for PMDAs.</para>
</listitem></varlistentry>
<varlistentry>
<term><command>mmv</command></term>
<listitem><para><indexterm id="IG3134017742nat"><primary>mmv library</primary><see>MMV</see></indexterm>Is used to instrument applications using Memory Mapped Values (MMV). These are values that are communicated with <command>pmcd</command> instantly, and very efficiently, using a shared memory mapping. It is a program instrumentation library.</para>
</listitem></varlistentry>
<varlistentry>
<term><command>newhelp</command></term>
<listitem><para><indexterm id="IG3134017737"><primary>newhelp tool</primary></indexterm>Generates the database files for one or more source files of PCP help text.</para>
</listitem></varlistentry>
<varlistentry>
<term><command>pmapi</command></term>
<listitem><para><indexterm id="IG3134017738"><primary>pmclient tool</primary><secondary>brief description</secondary></indexterm><indexterm id="IG3134017739"><primary>PMAPI</primary></indexterm><indexterm id="IG3134017740"><primary>Performance Metrics Application Programming Interface </primary><see>PMAPI</see></indexterm>Defines a procedural interface for developing PCP client applications. It is the Performance Metrics Application Programming Interface (PMAPI).</para>
</listitem></varlistentry>
<varlistentry>
<term><literal>pmclient</literal></term>
<listitem><para><indexterm id="IG3134017741"><primary>pmclient tool</primary></indexterm>Is a simple client that uses the PMAPI to report some high-level system performance metrics. The source code for <command>pmclient</command> is included in the distribution.</para>
</listitem></varlistentry>
<varlistentry>
<term><command>pmda</command></term>
<listitem><para><indexterm id="IG3134017742"><primary>pmda library</primary><see>PMDA</see></indexterm>Is a library used by many shipped PMDAs to communicate with a <command>pmcd</command> process. It can expedite the development of new and custom PMDAs.</para>
</listitem></varlistentry>
<varlistentry>
<term><command>pmgenmap</command></term>
<listitem><para><indexterm id="IG3134017743"><primary>pmgenmap tool</primary></indexterm>Generates C declarations and <literal>cpp</literal> macros to aid the development of customized programs that use the facilities of PCP. It is a program development tool.</para>
</listitem></varlistentry>
</variablelist>
</section>
</section>
<section id="LE16056-PARENT">

<title>PMDA Development</title>
<para><indexterm id="IG3134017744"><primary>PMDA</primary><secondary>development</secondary></indexterm>A collection of Performance Metrics Domain Agents (PMDAs) are provided with PCP to extract performance metrics. Each PMDA encapsulates domain-specific knowledge and methods about performance metrics that implement the uniform access protocols and functional semantics of the PCP. There is one PMDA for the operating system, another for process specific statistics, one each for common DBMS products, and so on. Thus, the range of performance metrics can be easily extended by implementing and integrating new PMDAs. <xref linkend="LE98072-PARENT"/>, is a step-by-step guide to writing your own PMDA.</para>
<section id="id5178205">

<title>Overview</title>
<para>Once you are familiar with the PCP and PMDA frameworks, you can quickly implement a new PMDA with only a few data structures and functions. This book contains detailed discussions of PMDA architecture and the integration of PMDAs into the PCP framework. This includes integration with PMCD. However, details of extracting performance metrics from the underlying instrumentation vary from one domain to another and are not covered in this book.</para>
<para>A PMDA is responsible for a set of performance metrics, in the sense that it must respond to requests from PMCD for information about performance metrics, instance domains, and instantiated values. The PMCD process generates requests on behalf of monitoring tools that make requests using PMAPI functions.</para>
<para>You can incorporate new performance metrics into the PCP framework by creating a PMDA, then reconfiguring PMCD to communicate with the new PMDA.</para>
</section>
<section id="id5178242">

<title>Building a PMDA</title>
<para>A PMDA interacts with PMCD across one of several well-defined interfaces and protocol mechanisms. These implementation options are described in the <citetitle>Performance Co-Pilot User's and Administrator's Guide</citetitle>.</para>
<note><para>It is strongly recommended that code for a new PMDA be based on the source of one of the existing PMDAs below the <filename>${PCP_PMDAS_DIR}</filename> directory.</para></note>
<section id="id5178295">

<title>In-Process (DSO) Method</title>
<para><indexterm id="IG3134017745"><primary>DSO</primary><secondary>PMDA building</secondary></indexterm><indexterm id="IG3134017746"><primary>dlopen man page</primary></indexterm><indexterm id="IG3134017747"><primary>dynamic shared object</primary><see>DSO</see></indexterm><indexterm id="IG3134017748"><primary>interprocess communication</primary><see>IPC</see></indexterm><indexterm id="IG3134017749"><primary>IPC</primary><secondary>DSO</secondary></indexterm>This method of building a PMDA uses a Dynamic Shared Object (DSO) that is attached by PMCD, using the platform-specific shared library manipulation interfaces such as <command>dlopen(3)</command>, at initialization time. This is the highest performance option (there is no context switching and no interprocess communication (IPC) between the PMCD and the PMDA), but is operationally intractable in some situations. For example, difficulties arise where special access permissions are required to read the instrumentation behind the performance metrics (<command>pmcd</command> does not run as root), or where the performance metrics are provided by an existing process with a different protocol interface. The DSO PMDA effectively executes as part of PMCD; so great care is required when crafting a PMDA in this manner. Calls to <command>exit(1)</command> in the PMDA, or a library it uses, would cause PMCD to exit and end monitoring of that host. Other implications are discussed in <xref linkend="id5188840"/>.</para>
</section>
<section id="id5178384">

<title>Daemon Process Method</title>
<para><indexterm id="IG3134017750"><primary>daemon process method</primary></indexterm>Functionally, this method may be thought of as a DSO implementation with a standard <command>main</command> routine conversion wrapper so that communication with PMCD uses message passing rather than direct procedure calls. For some very basic examples, see the <filename>${PCP_PMDAS_DIR}/trivial/trivial.c</filename> and <filename>${PCP_PMDAS_DIR}/simple/simple.c</filename> source files.</para>
<para>The daemon PMDA is actually the most common, because it allows multiple threads of control, greater (different user) privileges when executing, and provides more resilient error encapsulation than the DSO method.</para>
<note><para>Of particular interest for daemon PMDA writers, the <filename>${PCP_PMDAS_DIR}/simple</filename> PMDA has implementations in C, Perl and Python.</para></note>
</section>
</section>
</section>
<section id="id5178467">

<title>Client Development and PMAPI</title>
<para><indexterm id="IG3134017752"><primary>client development</primary></indexterm>Application developers are encouraged to create new PCP client applications to monitor, display, and analyze performance data in a manner suited to their particular site, application suite, or information processing environment.</para>
<para><indexterm id="IG3134017755"><primary>PMAPI</primary><secondary>client development</secondary></indexterm>PCP client applications are programmed using the Performance Metrics Application Programming Interface (PMAPI), documented in <xref linkend="LE97135-PARENT"/>. The PMAPI, which provides performance tool developers with access to all of the historical and live distributed services of PCP, is the interface used by the standard PCP utilities.</para>
</section>
<section id="id5187605">

<title>Library Reentrancy and Threaded Applications</title>
<para><indexterm id="IG3134017756"><primary>threaded applications</primary></indexterm><indexterm id="IG3134017757"><primary>multiple threads</primary></indexterm><indexterm id="IG3134017758"><primary>library reentrancy</primary></indexterm>While the core PCP library (<command>libpcp</command>) is thread safe, the layered PMDA library (<command>libpcp_pmda</command>) is not. This is a deliberate design decision to trade-off commonly required performance and efficiency against the less common requirement for multiple threads of control to call the PCP libraries.</para>
<para>The simplest and safest programming model is to designate at most one thread to make calls into the PCP PMDA library.</para>
</section>
</chapter>


<chapter id="LE98072-PARENT">

<title>Writing a PMDA</title>
<para><indexterm id="IG3134017759"><primary>PMDA</primary><secondary>writing</secondary></indexterm>This chapter constitutes a programmer's guide to writing a Performance Metrics Domain Agent (PMDA) for Performance Co-Pilot (PCP).</para>
<para>The presentation assumes the developer is using the standard PCP <filename>libpcp_pmda</filename> library, as documented in the <command>PMDA(3)</command> and associated man pages.</para>
<section id="id5187772">

<title>Implementing a PMDA</title>
<para><indexterm id="IG3134017760"><primary>implementation</primary></indexterm><indexterm id="IG3134017761"><primary>design requirements</primary></indexterm>The job of a PMDA is to gather performance data and report them to the  Performance Metrics Collection Daemon (PMCD) in response to requests from PCP monitoring tools routed to the PMDA via PMCD.</para>
<para>An important  requirement for any PMDA is that it have low latency response to requests from PMCD. Either the PMDA must use a quick access method and a single thread of control, or it must have asynchronous refresh and two threads of control: one for communicating with PMCD, the other for updating the performance data.</para>
<para><indexterm id="IG3134017762"><primary>target domain</primary></indexterm><indexterm id="IG3134017763"><primary>sequential log files</primary></indexterm><indexterm id="IG3134017764"><primary>snapshot files</primary></indexterm><indexterm id="IG3134017765"><primary>IPC</primary><secondary> PMDA</secondary></indexterm>The PMDA is typically acting as a gateway between the target domain (that is, the performance instrumentation in an application program or service) and the PCP framework. The PMDA may extract the information using one of a number of possible  export options that include a shared memory segment or <command>mmap</command> file; a sequential log  file (where the PMDA parses the tail of the log file to extract the information); a snapshot file (the PMDA rereads the file as required); or application-specific communication services (IPC).</para>
<note><para>The choice of export methodology is typically determined by the source of the instrumentation (the target domain) rather than by the PMDA.</para></note>
<para><indexterm id="IG3134017766"><primary>PMDA</primary><secondary> checklist</secondary></indexterm><xref linkend="id5187871"/> describes the suggested steps for designing and implementing a PMDA:</para>
<procedure id="id5187871">
<title>Creating a PMDA</title>
<step><para>Determine how to extract the metrics from the target domain.</para></step>
<step><para>Select an appropriate architecture for the PMDA (daemon or DSO, IPC, <command>pthreads</command> or single threaded).</para></step>
<step><para>Define the metrics and instances that the PMDA will support.</para></step>
<step><para>Implement the functionality to extract the metric values.</para></step>
<step><para>Assign Performance Metric Identifiers (PMIDs) for the metrics, along with names for the metrics in the Performance Metrics Name Space (PMNS). These concepts will be further expanded in <xref linkend="LE97285-PARENT"/></para></step>
<step><para><indexterm id="IG3134017767"><primary>help text</primary><secondary>structure specification</secondary></indexterm>Specify the help file and control data structures for metrics and instances that are required by the standard PMDA implementation library functions.</para></step>
<step><para>Write code to supply the metrics and associated information to PMCD.</para></step>
<step><para>Implement any PMDA-specific callbacks, and PMDA initialization functions.</para></step>
<step><para><indexterm id="IG3134017768"><primary>dbpmda man page</primary></indexterm>Exercise and test the PMDA with the purpose-built PMDA debugger; see the <command>dbpmda(1)</command> man page.</para></step>
<step><para>Install and connect the PMDA to a running PMCD process; see the <command>pmcd(1)</command> man page.</para></step>
<step><para><indexterm id="IG3134017769"><primary>pmchart command</primary></indexterm><indexterm id="IG3134017770"><primary>pmgadgets command</primary></indexterm><indexterm id="IG3134017774"><primary>examples</primary><secondary>visualization tools</secondary></indexterm>Configure or develop tools to use the new metrics. For examples of visualization tools, see the <command>pmchart(1)</command> and <command>pmgadgets(1)</command> man pages. For examples of text-based tools, see the <command>pminfo(1)</command> and <command>pmval(1)</command> man pages.</para></step>
<step><para><indexterm id="IG3134017772"><primary>pmie command</primary></indexterm><indexterm id="IG3134017773"><primary>pmieconf command</primary></indexterm><indexterm id="IG3134017775"><primary>examples</primary><secondary>alarm tools</secondary></indexterm>Where appropriate, define <command>pmie</command> rule templates suitable for alerting or notification systems. For more information, see the <command>pmie(1)</command> and <command>pmieconf(1)</command> man pages.</para></step>
<step><para><indexterm id="IG3134017776"><primary>pmlogger command</primary></indexterm>Where appropriate, define <command>pmlogger</command> configuration templates suitable for creating PCP archives containing the new metrics. For more information, see the <command>pmlogconf(1)</command> and <command>pmlogger(1)</command> man pages.</para></step>
</procedure>
</section>
<section id="id5188149">

<title>PMDA Architecture</title>
<para><indexterm id="IG3134017777"><primary>architecture</primary></indexterm><indexterm id="IG3134017778"><primary>PMDA</primary><secondary>architecture</secondary></indexterm> <indexterm id="IG3134017779"><primary>DSO</primary><secondary>architecture</secondary></indexterm>This section discusses the two methods of connecting a PMDA to a PMCD process:<itemizedlist>
<listitem><para>As a separate process using some interprocess communication (IPC) protocol.</para>
</listitem>
<listitem><para>As a dynamically attached library (that is, a dynamic shared object or DSO).</para>
</listitem></itemizedlist></para>
<section id="id5188221">

<title>Overview</title>
<para><indexterm id="IG3134017780"><primary>PDU</primary></indexterm><indexterm id="IG3134017781"><primary>protocol data units</primary><see>PDU</see></indexterm>All PMDAs are launched and controlled by the PMCD process on the local host. PMCD receives requests from the monitoring tools and forwards them to the PMDAs. Responses, when required, are returned through PMCD to the clients. The requests fall into a small number of categories, and the PMDA must handle each request type. For a DSO PMDA, each request type corresponds to a method in the agent. For a daemon PMDA, each request translates to a message or  protocol data unit (PDU) that may be sent to a PMDA from PMCD.</para>
<para>For a daemon PMDA, the following request PDUs must be supported:<variablelist>
<varlistentry>
<term><literal>PDU_FETCH</literal></term>
<listitem><para><indexterm id="IG3134017782"><primary>pmFetch man page</primary></indexterm><indexterm id="IG3134017783"><primary>PDU_FETCH</primary></indexterm>Request for metric values (see the <command>pmFetch(3)</command> man page.)</para>
</listitem></varlistentry>
<varlistentry>
<term><literal>PDU_PROFILE</literal></term>
<listitem><para><indexterm id="IG3134017784"><primary>pmAddProfile function</primary></indexterm><indexterm id="IG3134017785"><primary>PDU_PROFILE</primary></indexterm>A list of instances required for the corresponding metrics in subsequent fetches (see the <command>pmAddProfile(3)</command> man page).</para>
</listitem></varlistentry>
<varlistentry>
<term><literal>PDU_INSTANCE_REQ</literal></term>
<listitem><para><indexterm id="IG3134017786"><primary>pmGetInDom function</primary></indexterm><indexterm id="IG3134017787"><primary>PDU_INSTANCE_REQ</primary></indexterm> Request for a particular instance domain for instance descriptions (see the <command>pmGetInDom(3)</command> man page).</para>
</listitem></varlistentry>
<varlistentry>
<term><literal>PDU_DESC_REQ</literal></term>
<listitem><para><indexterm id="IG3134017788"><primary>pmLookupDesc function</primary></indexterm><indexterm id="IG3134017789"><primary>PDU_DESC_REQ</primary></indexterm>Request for metadata describing metrics (see the <command>pmLookupDesc(3)</command> man page).</para>
</listitem></varlistentry>
<varlistentry>
<term><literal>PDU_TEXT_REQ</literal></term>
<listitem><para><indexterm id="IG3134017790"><primary>pmLookupText function</primary></indexterm><indexterm id="IG3134017791"><primary>PDU_TEXT_REQ</primary></indexterm><indexterm id="IG3134017792"><primary>help text</primary><secondary>PDU_TEXT_REQ</secondary></indexterm>Request for metric help text (see the <command>pmLookupText(3)</command> man page).</para>
</listitem></varlistentry>
<varlistentry>
<term><literal>PDU_RESULT</literal></term>
<listitem><para><indexterm id="IG3134017793"><primary>pmstore function</primary></indexterm><indexterm id="IG3134017794"><primary>PDU_RESULT</primary></indexterm>Values to store into metrics (see the <command>pmStore(3)</command> man page).</para>
</listitem></varlistentry>
</variablelist></para>
<para>The following request PDUs may optionally be supported:<variablelist>
<varlistentry>
<term><literal>PDU_PMNS_NAMES</literal></term>
<listitem><para><indexterm id="IG3134017782nat"><primary>pmLookupName function</primary></indexterm><indexterm id="IG3134017783nat"><primary>PDU_PMNS_NAMES</primary></indexterm>Request for metric names, given one or more identifiers (see the <command>pmLookupName(3)</command> man page.)</para>
</listitem></varlistentry>
<varlistentry>
<term><literal>PDU_PMNS_CHILD</literal></term>
<listitem><para><indexterm id="IG3134017784nat"><primary>pmGetChildren function</primary></indexterm><indexterm id="IG3134017785nat"><primary>PDU_PMNS_CHILD</primary></indexterm>A list of immediate descendent nodes of a given namespace node (see the <command>pmGetChildren(3)</command> man page).</para>
</listitem></varlistentry>
<varlistentry>
<term><literal>PDU_PMNS_TRAVERSE</literal></term>
<listitem><para><indexterm id="IG3134017786nat"><primary>pmTraversePMNS function</primary></indexterm><indexterm id="IG3134017787nat"><primary>PDU_PMNS_TRAVERSE</primary></indexterm>Request for a particular sub-tree of a given namespace node (see the <command>pmTraversePMNS(3)</command> man page).</para>
</listitem></varlistentry>
<varlistentry>
<term><literal>PDU_PMNS_IDS</literal></term>
<listitem><para><indexterm id="IG3134017788nat"><primary>pmNameID function</primary></indexterm><indexterm id="IG3134017789nat"><primary>PDU_PMNS_IDS</primary></indexterm>Perform a reverse name lookup, mapping a metric identifier to a name (see the <command>pmNameID(3)</command> man page).</para>
</listitem></varlistentry>
<varlistentry>
<term><literal>PDU_AUTH</literal></term>
<listitem><para><indexterm id="IG3134017790nat"><primary>__pmParseHostAttrsSpec function</primary></indexterm><indexterm id="IG3134017791nat"><primary>PDU_AUTH</primary></indexterm>Handle connection attributes (key/value pairs), such as client credentials and other authentication information (see the <command>__pmParseHostAttrsSpec(3)</command> man page).</para>
</listitem></varlistentry>
</variablelist></para>
<para>Each PMDA is associated with a unique domain number that is encoded in the domain field of metric and instance identifiers, and PMCD uses the domain number to determine which PMDA can handle the components of any given client request.</para>
</section>
<section id="LE82676-PARENT">

<title>DSO PMDA</title>
<para><indexterm id="IG3134017795"><primary>DSO</primary><secondary>implementation</secondary></indexterm><indexterm id="IG3134017796"><primary>dlopen man page</primary></indexterm>Each PMDA is required to implement a function that handles each of the request types. By implementing these functions as library functions, a PMDA can be implemented as a dynamically shared object (DSO) and attached by PMCD at run time with a platform-specific call, such as <command>dlopen</command>; see the <command>dlopen(3)</command> man page. This eliminates the need for an IPC layer (typically a pipe) between each PMDA and PMCD, because each request becomes a function call rather than a message exchange. The required library functions are detailed in <xref linkend="LE21831-PARENT"/>.</para>
<para>A PMDA that interacts with PMCD in this fashion must abide by a formal initialization protocol so that PMCD can discover the location of the library functions that are subsequently called with function pointers. When a DSO PMDA is installed, the PMCD configuration file, <filename>${PCP_PMCDCONF_PATH}</filename>, is updated to reflect the domain and name of the PMDA, the location of the shared object, and the name of the initialization function. The initialization sequence is discussed in <xref linkend="LE19047-PARENT"/>.</para>
<para><indexterm id="IG3134017797"><primary>simple PMDA</primary><secondary>DSO</secondary></indexterm>As superuser, install the simple PMDA as a DSO, as shown in <xref linkend="Z1033502809tls"/>, and observe the changes in the PMCD configuration file. The output may differ slightly depending on the operating system you are using, any other PMDAs you have installed or any PMCD access controls you have in place.</para>
<para><example id="Z1033502809tls">
<title>Simple PMDA as a DSO</title>
<programlisting><userinput>cat ${PCP_PMCDCONF_PATH}</userinput>
# Performance Metrics Domain Specifications
# 
# This file is automatically generated during the build
# Name  Id      IPC     IPC Params      File/Cmd
pmcd    2       dso     pmcd_init       ${PCP_PMDAS_DIR}/pmcd/pmda_pmcd.so
linux   60      dso     linux_init      ${PCP_PMDAS_DIR}/linux/pmda_linux.so
proc    3       pipe    binary          ${PCP_PMDAS_DIR}/linux/pmda_proc.so -d 3
simple  254     dso     simple_init     ${PCP_PMDAS_DIR}/simple/pmda_simple.so</programlisting>
</example></para>
<para>As can be seen from the contents of <filename>${PCP_PMCDCONF_PATH}</filename>, the DSO version of the simple PMDA is in a library named <filename>pmda_simple.so</filename> and has an initialization function called <indexterm id="IG3134017799"><primary>simple_init function</primary></indexterm> <command>simple_init</command>. The domain of the simple PMDA is 254, as shown in the column headed <literal>Id</literal>.</para>
<note><para>For some platforms the DSO file name will not be <filename>pmda_simple.so</filename>. On Mac OS X it is <filename>pmda_simple.dylib</filename> and on Windows it is <filename>pmda_simple.dll</filename>.
</para></note>
</section>
<section id="id5188840">

<title>Daemon PMDA</title>
<para><indexterm id="IG31340177100"><primary>DSO</primary><secondary>disadvantages</secondary></indexterm>A DSO PMDA provides the most efficient communication between the PMDA and PMCD. This approach has some  disadvantages resulting from the DSO PMDA being the same process as PMCD:</para>
<itemizedlist>
<listitem><para>An error or bug that causes a DSO PMDA to exit also causes PMCD to exit, which affects all connected client tools.</para>
</listitem>
<listitem><para>There is only one thread of control in PMCD; as a result, a computationally expensive PMDA, or worse, a PMDA that blocks for I/O, adversely affects the performance of PMCD.</para>
</listitem>
<listitem><para>PMCD runs as the &quot;pcp&quot; user; so all DSO PMDAs must also run as this user.</para>
</listitem>
<listitem><para>A memory leak in a DSO PMDA also causes a memory leak for PMCD.</para>
</listitem></itemizedlist>
<para>Consequently, many PMDAs are implemented as a daemon process.</para>
<para>The <filename>libpcp_pmda</filename> library is designed to allow simple implementation of a PMDA that runs as a separate process. The library functions provide a message passing layer acting as a generic wrapper that accepts PDUs, makes library calls using the standard DSO PMDA interface, and sends PDUs. Therefore, you can implement a PMDA as a DSO and then install it as either a daemon or a DSO, depending on the presence or absence of the generic wrapper.</para>
<para><indexterm id="IG31340177101"><primary>fork system call</primary></indexterm><indexterm id="IG31340177102"><primary>execv system call</primary></indexterm><indexterm id="IG31340177103"><primary>pipe</primary></indexterm>The PMCD process launches a daemon PMDA with  <command>fork</command> and <command>execv</command> (or <command>CreateProcess</command> on Windows). You can easily connect a pipe to the PMDA using standard input and output. The PMCD process may also connect to a daemon PMDA using IPv4 or IPv6 TCP/IP, or UNIX domain sockets if the platform supports that; see the <command>tcp(7)</command>,  <command>ip(7)</command>, <command>ipv6(7)</command> or <command>unix(7)</command> man pages.</para>
<para><indexterm id="IG31340177104"><primary>pipe</primary></indexterm><indexterm id="IG31340177105"><primary>simple PMDA</primary><secondary> as daemon</secondary></indexterm>As superuser, install the  simple PMDA as a daemon process as shown in <xref linkend="Z1033576478tls"/>. Again, the output may differ due to operating system differences, other PMDAs already installed, or access control sections in the PMCD configuration file.</para>
<para><example id="Z1033576478tls">
<title>Simple PMDA as a Daemon</title>
<para>The specification for the simple PMDA now states the connection type of <command>pipe</command> to PMCD and the executable image for the PMDA is <filename>${PCP_PMDAS_DIR}/simple/pmdasimple</filename>, using domain number 253.<programlisting># <userinput>cd ${PCP_PMDAS_DIR}/simple</userinput>
# <userinput>./Install</userinput>
... 
Install simple as a daemon or dso agent? [daemon] daemon 
PMCD should communicate with the daemon via pipe or socket? [pipe] pipe
...
# <userinput>cat ${PCP_PMCDCONF_PATH}</userinput>
# Performance Metrics Domain Specifications
# 
# This file is automatically generated during the build
# Name  Id      IPC     IPC Params      File/Cmd
pmcd    2       dso     pmcd_init       ${PCP_PMDAS_DIR}/pmcd/pmda_pmcd.so
linux   60      dso     linux_init      ${PCP_PMDAS_DIR}/linux/pmda_linux.so
proc    3       pipe    binary          ${PCP_PMDAS_DIR}/linux/pmda_proc.so -d 3
simple  253     pipe    binary          ${PCP_PMDAS_DIR}/simple/pmdasimple -d 253</programlisting></para>
</example></para>
</section>
<section id="id5189181">

<title>Caching PMDA</title>
<para><indexterm id="IG31340177106"><primary>caching PMDA</primary></indexterm>When either the cost or latency associated with collecting performance metrics is high, the PMDA implementer may choose to trade off the currency of the performance data to reduce the PMDA resource demands or the fetch latency time.</para>
<para>One scheme for doing this is called a  caching PMDA, which periodically instantiates values for the performance metrics and responds to each request from PMCD with the most recently instantiated (or cached) values, as opposed to instantiating current values on demand when the PMCD asks for them.</para>
<para><indexterm id="IG31340177107"><primary>Cisco PMDA</primary></indexterm><indexterm id="IG31340177108"><primary>pmdacisco man page</primary></indexterm>The Cisco PMDA is an example of a caching PMDA. For additional information, see the contents of the <filename>${PCP_PMDAS_DIR}/cisco</filename> directory and the <command>pmdacisco(1)</command> man page.</para>
</section>
</section>
<section id="LE97285-PARENT">

<title>Domains, Metrics, and Instances</title>
<para>This section defines metrics and instances, discusses how they should be designed for a particular target domain, and shows how to implement support for them.</para>
<para><indexterm id="IG31340177109"><primary>examples</primary><secondary>simple and trivial PMDAs</secondary></indexterm>The examples in this section are drawn from the trivial and simple PMDAs. Refer to the <filename>${PCP_PMDAS_DIR}/trivial</filename> and <filename>${PCP_PMDAS_DIR}/simple</filename> directories, respectively, where both binaries and source code are available.</para>
<section id="id5189300">

<title>Overview</title>
<para><indexterm id="IG31340177110"><primary>domains</primary><secondary>definition</secondary></indexterm><indexterm id="IG31340177111"><primary>metrics</primary><secondary>definition</secondary></indexterm><firstterm>Domains</firstterm> are autonomous performance areas, such as the operating system or a layered service or a particular application. <firstterm>Metrics</firstterm> are raw performance data for a domain, and typically quantify activity levels, resource utilization or quality of service. <firstterm>Instances</firstterm> are sets of related metrics, as for multiple processors, or multiple service classes, or multiple transaction types.</para>
<para> PCP employs the following simple and uniform data model to accommodate the demands of performance metrics drawn from multiple domains:</para>
<itemizedlist>
<listitem><para>Each metric has an identifier that is unique across all metrics for all PMDAs on a particular host.</para>
</listitem>
<listitem><para>Externally, metrics are assigned names for user convenience--typically there is a 1:1 relationship between a metric name and a metric identifier.</para>
</listitem>
<listitem><para>The PMDA implementation determines if a particular metric has a singular value or a set of (zero or more) values. For instance, the metric <literal>hinv.ndisk</literal> counts the number of disks and has only one value on a host, whereas the metric <literal>disk.dev.total</literal> counts disk I/O operations and has one value for each disk on the host.</para>
</listitem>
<listitem><para>If a metric has a set of values, then members of the set are differentiated by instances. The set of instances associated with a metric is an <firstterm>instance domain</firstterm>. For example, the set of metrics <literal>disk.dev.total</literal> is defined over an instance domain that has one member per disk spindle.</para>
</listitem></itemizedlist>
<para><indexterm id="IG31340177112"><primary>metrics and instances</primary></indexterm>The  selection of metrics and instances is an important design decision for a PMDA implementer. The metrics and instances for a target domain should have the following qualities:</para>
<itemizedlist>
<listitem><para>Obvious to a user</para>
</listitem>
<listitem><para>Consistent across the domain</para>
</listitem>
<listitem><para>Accurately representative of the operational and functional aspects of the domain</para>
</listitem></itemizedlist>
<para>For each metric, you should also consider these questions:</para>
<itemizedlist>
<listitem><para>How useful is this value?</para>
</listitem>
<listitem><para>What units give a good sense of scale?</para>
</listitem>
<listitem><para>What name gives a good description of the metric's meaning?</para>
</listitem>
<listitem><para>Can this metric be combined with another to convey the same useful information?</para>
</listitem></itemizedlist>
<para>As with all programming tasks, expect to refine the choice of metrics and instances several times during the development of the PMDA.</para>
</section>
<section id="id5189538">

<title>Domains</title>
<para>Each PMDA must be uniquely identified by PMCD so that requests from clients can be efficiently routed to the appropriate PMDA. The unique identifier, the PMDA's domain, is encoded within the metrics and instance domain identifiers so that they are associated with the correct PMDA, and so that they are unique, regardless of the number of PMDAs that are connected to the PMCD process.</para>
<para><indexterm id="IG31340177113"><primary>domains</primary><secondary>numbers</secondary></indexterm>The default  domain number for each PMDA is defined in <filename>${PCP_VAR_DIR}/pmns/stdpmid</filename>. This file is a simple table of PMDA names and their corresponding domain number. However, a PMDA does not have to use this domain number--the file is only a guide to help avoid domain number clashes when PMDAs are installed and activated.</para>
<para>The domain number a PMDA uses is passed to the PMDA by PMCD when the PMDA is launched. Therefore, any data structures that require the PMDA's domain number must be set up when the PMDA is initialized, rather than declared statically. The protocol for PMDA initialization provides a standard way for a PMDA to implement this run-time initialization.</para>
<tip><para>Although uniqueness of the domain number in the <filename>${PCP_PMCDCONF_PATH}</filename> control file used by PMCD is all that is required for successful starting of PMCD and the associated PMDAs, the developer of a new PMDA is encouraged to add the default domain number for each new PMDA to the <filename>${PCP_VAR_DIR}/pmns/stdpmid.local</filename> file and then to run the  <filename>Make.stdpmid</filename> script in <filename>${PCP_VAR_DIR}/pmns</filename> to recreate <filename>${PCP_VAR_DIR}/pmns/stdpmid</filename>; this file acts as a repository for documenting the known default domain numbers.</para>
</tip>
</section>
<section id="LE98565-PARENT">

<title>Metrics</title>
<para><indexterm id="IG31340177114"><primary>metrics</primary><secondary>definition</secondary></indexterm><indexterm id="IG31340177115"><primary>target domain</primary></indexterm>A PMDA provides support for a collection of metrics. In addition to the obvious performance metrics, and the measures of time, activity and resource utilization, the metrics should also describe how the target domain has been configured, as this can greatly affect the correct interpretation of the observed performance. For example, metrics that describe network transfer rates should also describe the number and type of network interfaces connected to the host (<literal>hinv.ninterface</literal>, <literal>network.interface.speed</literal>, <literal>network.interface.duplex</literal>, and so on)</para>
<para><indexterm id="IG31340177116"><primary>storage of metrics</primary></indexterm><indexterm id="IG31340177117"><primary>pmstore function</primary></indexterm>In addition, the metrics should describe how the PMDA has been configured. For example, if the PMDA was periodically probing a system to measure quality of service, there should be metrics for the delay between probes, the number of probes attempted, plus probe success and failure counters. It may also be appropriate to allow values to be  stored (see the <command>pmstore(1)</command> man page) into the delay metric, so that the delay used by the PMDA can be altered dynamically.</para>
<section id="id5189689">

<title>Data Structures</title>
<para><indexterm id="IG31340177118"><primary>data structures</primary></indexterm><indexterm id="IG31340177119"><primary>pmDesc structure</primary></indexterm><indexterm id="IG31340177120"><primary>pmLookupDesc function</primary></indexterm>Each metric must be described in a <filename>pmDesc</filename> structure; see the <command>pmLookupDesc(3)</command> man page:</para>
<programlisting>typedef struct { 
    pmID        pmid;           /* unique identifier */ 
    int         type;           /* base data type */ 
    pmInDom     indom;          /* instance domain */ 
    int         sem;            /* semantics of value */ 
    pmUnits     units;          /* dimension and units */ 
} pmDesc;</programlisting>
<para>This structure contains the following fields:</para>
<variablelist condition="sgi_termlength:narrow">
<varlistentry>
<term><literal>pmid</literal></term>
<listitem><para>A unique identifier, Performance Metric Identifier (PMID), that differentiates this metric from other metrics across the union of all PMDAs</para>
</listitem></varlistentry>
<varlistentry>
<term><literal>type</literal></term>
<listitem><para>A data type indicator showing whether the format is an integer (32 or 64 bit, signed or unsigned); float; double; string; or arbitrary aggregate of binary data</para>
</listitem></varlistentry>
<varlistentry>
<term><literal>indom</literal></term>
<listitem><para>An instance domain identifier that links this metric to an instance domain</para>
</listitem></varlistentry>
<varlistentry>
<term><literal>sem</literal></term>
<listitem><para>An encoding of the value's semantics (counter, instantaneous, or discrete)</para>
</listitem></varlistentry>
<varlistentry>
<term><literal>units</literal></term>
<listitem><para>A description of the value's units based on dimension and scale in the three orthogonal dimensions of space, time, and count (or events)</para>
</listitem></varlistentry>
</variablelist>
<note><para>This information can be observed for metrics from any active PMDA using <command>pminfo</command> command line options, for example:
<literallayout class="monospaced"> $ <userinput>pminfo -d -m network.interface.out.drops</userinput>

 network.interface.out.drops PMID: 60.3.11
     Data Type: 64-bit unsigned int  InDom: 60.3 0xf000003
     Semantics: counter  Units: count</literallayout>
</para></note>
<para><indexterm id="IG31340177121"><primary>__pmID_int structure</primary></indexterm>Symbolic constants of the form <literal>PM_TYPE_*</literal>, <literal>PM_SEM_*</literal>, <literal>PM_SPACE_*</literal>, <literal>PM_TIME_*</literal>, and <literal>PM_COUNT_*</literal> are defined in the <filename>&lt;pcp/pmapi.h&gt;</filename> header file. You may use them to initialize the elements of a <literal>pmDesc</literal> structure. The <literal>pmID</literal>  type is an unsigned integer that can be safely cast to a <filename>__pmID_int</filename> structure, which contains fields defining the metric's (PMDA's) domain, cluster, and item number as shown in <xref linkend="Z1033577630tls"/>:</para>
<para><example id="Z1033577630tls">
<title><filename>__pmID_int</filename> Structure</title>
<programlisting>typedef struct { 
        int             flag:1;
        unsigned int    domain:9;
        unsigned int    cluster:12;
        unsigned int    item:10;
} __pmID_int;</programlisting>
</example></para>
<para>For additional information, see the <filename>&lt;pcp/impl.h&gt;</filename> file.</para>
<para><indexterm id="IG31340177122"><primary>PMDA_PMID macro</primary></indexterm>The <literal>flag</literal> field should be ignored. The <literal>domain</literal> number should be set at run time when the PMDA is initialized. The <literal>PMDA_PMID</literal> macro defined in <filename>&lt;pcp/pmapi.h&gt;</filename> can be used to set the <literal>cluster</literal> and <literal>item</literal> fields at compile time, as these should always be known and fixed for a particular metric.</para>
<note><para>The three components of the PMID should correspond exactly to the three-part definition of the PMID for the corresponding metric in the PMNS described in <xref linkend="LE83854-PARENT"/>.</para>
</note>
<para><indexterm id="IG31340177123"><primary> pmdaMetric structure</primary></indexterm>A table of <filename>pmdaMetric</filename> structures should be defined within the PMDA, with one structure per metric as shown in <xref linkend="Z976036629sdc"/>. </para>
<example id="Z976036629sdc">
<title><filename>pmdaMetric</filename> Structure</title>
<programlisting>typedef struct { 
    void        *m_user;        /* for users external use */ 
    pmDesc      m_desc;         /* metric description */ 
} pmdaMetric;</programlisting>
</example>
<para>This structure contains a <filename>pmDesc</filename> structure and a handle that allows PMDA-specific structures to be associated with each metric. For example, <literal>m_user</literal> could be a pointer to a global variable containing the metric value, or a pointer to a function that may be called to instantiate the metric's value.</para>
<para><indexterm id="IG31340177124"><primary>trivial PMDA</primary><secondary>singular metric</secondary></indexterm>The trivial PMDA, shown in <xref linkend="Z963521873sdc"/>, has only a singular metric (that is, no instance domain):</para>
<example id="Z963521873sdc">
<title>Trivial PMDA</title>
<programlisting>static pmdaMetric metrictab[] = {
/* time */
 { NULL,
   { PMDA_PMID(0, 1), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_INSTANT,
     PMDA_PMUNITS(0, 1, 0, 0, PM_TIME_SEC, 0) }, },
}; </programlisting>
<para>This single metric (<literal>trivial.time</literal>) has the following:</para>
<itemizedlist>
<listitem><para>A PMID with a cluster of 0 and an item of 1. Note that this is not yet a complete PMID, the domain number which identifies the PMDA will be combined with it at runtime.</para>
</listitem>
<listitem><para>An unsigned 32-bit integer (<literal>PM_TYPE_U32</literal>)</para>
</listitem>
<listitem><para><indexterm id="IG31340177125"><primary>PM_INDOM_NULL instance domain</primary><secondary>data structures</secondary></indexterm>A singular value and hence no instance domain (<literal>PM_INDOM_NULL</literal>)</para>
</listitem>
<listitem><para><indexterm id="IG31340177126"><primary>PM_SEM_INSTANT semantic type</primary></indexterm>An instantaneous semantic value (<literal>PM_SEM_INSTANT</literal>)</para>
</listitem>
<listitem><para>Dimension &#x201C;time&#x201D; and the units &#x201C;seconds&#x201D;</para>
</listitem></itemizedlist>
</example>
</section>
<section id="id5190312">

<title>Semantics</title>
<para><indexterm id="IG31340177127"><primary>semantic types</primary></indexterm>The metric's semantics describe how PCP tools should interpret the metric's value. The following are the possible  semantic types:</para>
<itemizedlist>
<listitem><para><indexterm id="IG31340177128"><primary>PM_SEM_COUNTER semantic type</primary></indexterm>Counter (<literal>PM_SEM_COUNTER</literal>)</para>
</listitem>
<listitem><para><indexterm id="IG31340177129"><primary>PM_SEM_INSTANT semantic type</primary></indexterm> Instantaneous value (<literal>PM_SEM_INSTANT</literal>)</para>
</listitem>
<listitem><para><indexterm id="IG31340177130"><primary>PM_SEM_DISCRETE semantic type</primary></indexterm>Discrete value (<literal>PM_SEM_DISCRETE</literal>)</para>
</listitem></itemizedlist>
<para>A counter should be a value that monotonically increases (or monotonically decreases, which is less likely) with respect to time, so that the rate of change should be used in preference to the actual value. Rate conversion is not appropriate for metrics with instantaneous values, as the value is a snapshot and there is no basis for assuming any values that might have been observed between snapshots. Discrete is similar to instantaneous; however, once observed it is presumed the value will persist for an extended period (for example, system configuration, static tuning parameters and most metrics with non-numeric values).</para>
<para><indexterm id="IG31340177131"><primary>counter semantics</primary></indexterm> <indexterm id="IG31340177132"><primary> instantaneous semantics</primary></indexterm> <indexterm id="IG31340177133"><primary>discrete semantics</primary></indexterm>For a given time interval covering six consecutive timestamps, each spanning two units of time, the metric values in <xref linkend="Z963522765sdc"/> are exported from a PMDA (&#x201C;N/A&#x201D; implies no value is available): </para>
<example id="Z963522765sdc">
<title>Effect of Semantics on a Metric</title>
<programlisting>Timestamps:         1   3   5   7   9  11 
Value:             10  30  60  80  90 N/A</programlisting>
<para>The default display of the values would be as follows:</para>
<programlisting>Timestamps:         1   3   5   7   9  11 
Semantics: 
Counter           N/A  10  15  10   5 N/A 
Instantaneous      10  30  60  80  90 N/A 
Discrete           10  30  60  80  90  90</programlisting>
</example>
<para>Note that these interpretations of metric semantics are performed by
the monitor tool, automatically, before displaying a value and they are not
transformations that the PMDA performs.</para>
</section>
</section>
<section id="id5190481">

<title>Instances</title>
<para>Singular metrics have only one value and no associated instance domain. Some metrics contain a set of values that share a common set of semantics for a specific instance, such as one value per processor, or one value per disk spindle, and so on.</para>
<note><para>The PMDA implementation is solely responsible for choosing the instance identifiers that differentiate instances within the instance domain. The PMDA is also responsible for ensuring the uniqueness of instance identifiers in any instance domain, as described in <xref linkend="id5190505"/>.</para>
</note>
<section id="id5190505">
<title>Instance Identification</title>
<para>Consistent interpretation of instances and instance domains require a few simple rules to be followed by PMDA authors. The PMDA library provides a series of <command>pmdaCache</command> routines to assist.</para>
<itemizedlist>
<listitem><para>Each internal instance identifier (numeric) must be a unique 31-bit number.</para>
</listitem>
<listitem><para>The external instance name (string) must be unique.</para>
</listitem>
<listitem><para>When the instance name contains a space, the name to the left of the first space (the short name) must also be unique.</para>
</listitem>
<listitem><para>Where an external instance name corresponds to some object or entity, there is an expectation that the association between the name and the object is fixed.</para>
</listitem>
<listitem><para>It is preferable, although not mandatory, for the association between and external instance name (string) and internal instance identifier (numeric) to be persistent.</para>
</listitem></itemizedlist>
</section>
<section id="id5190506">

<title>N Dimensional Data</title>
<para><indexterm id="IG31340177134"><primary>arrays</primary><secondary>N dimensional data</secondary></indexterm>Where the performance data can be represented as scalar values (singular metrics) or one-dimensional arrays or lists (metrics with an instance domain), the PCP framework is more than adequate. In the case of metrics with an instance domain, each array or list element is associated with an instance from the instance domain.</para>
<para><indexterm id="IG31340177135"><primary>two or three dimensional arrays</primary></indexterm><indexterm id="IG31340177136"><primary> multidimensional arrays</primary></indexterm>To represent two or more dimensional arrays, the coordinates must be one of the following: </para>
<itemizedlist>
<listitem><para>Mapped onto one dimensional coordinates.</para>
</listitem>
<listitem><para>Enumerated into the Performance Metrics Name Space (PMNS).</para>
</listitem></itemizedlist>
<para>For example, this 2 x 3 array of values called M can be represented as instances 1,..., 6 for a metric M:</para>
<programlisting>  M[1]   M[2]   M[3] 
  M[4]   M[5]   M[6]</programlisting>
<para>Or  they can be represented as instances 1, 2, 3 for metric M1 and instances 1, 2, 3 for metric M2:</para>
<programlisting>  M1[1]  M1[2]  M1[3] 
  M2[1]  M2[2]  M2[3]</programlisting>
<para>The PMDA implementer must decide and consistently export this encoding from the N-dimensional instrumentation to the 1-dimensional data model of the PCP.</para>
<para>In certain special cases (for example, such as for a histogram), it may be appropriate to export an array of values as raw binary data (the type encoding in the descriptor is <literal>PM_TYPE_AGGREGATE</literal>). However, this requires the development of special PMAPI client tools, because the standard PCP tools have no knowledge of the structure and interpretation of the binary data. The usual issues of platform-depdendence must also be kept in mind for this case - endianness, word-size, alignment and so on - the (possibly remote) special PMAPI client tools may need this information in order to decode the data successfully.</para>
</section>
<section id="id5190626">

<title>Data Structures</title>
<para><indexterm id="IG31340177137"><primary>data structures</primary></indexterm><indexterm id="IG31340177138"><primary>pmdaInstid structure</primary></indexterm>If the PMDA is required to support instance domains, then for each instance domain the unique internal instance identifier and external instance identifier should be defined using a <filename>pmdaInstid</filename> structure as shown in <xref linkend="Z975964618sdc"/>:</para>
<example id="Z975964618sdc">
<title><filename>pmdaInstid</filename> Structure</title>
<programlisting>typedef struct { 
    int         i_inst;         /* internal instance identifier */ 
    char        *i_name;        /* external instance identifier */ 
} pmdaInstid;</programlisting>
<para>The <literal>i_inst</literal> instance identifier  must be a unique integer within a particular instance domain.</para>
</example>
<para><indexterm id="IG31340177139"><primary>pmdaIndom structure</primary></indexterm>The complete instance domain description is specified in a <filename>pmdaIndom</filename>  structure as shown in <xref linkend="Z975964773sdc"/>:</para>
<example id="Z975964773sdc">
<title><filename>pmdaIndom</filename>  Structure</title>
<programlisting>typedef struct { 
    pmInDom     it_indom;       /* indom, filled in */ 
    int         it_numinst;     /* number of instances */ 
    pmdaInstid  *it_set;        /* instance identifiers */ 
} pmdaIndom;</programlisting>
</example>
<para><indexterm id="IG31340177140"><primary>arrays</primary><secondary>instance description</secondary></indexterm><indexterm id="IG31340177141"><primary>__pmInDom_int structure</primary></indexterm>The <literal>it_indom</literal> element contains a <literal>pmInDom</literal> that must be unique across every PMDA. The other fields of the <filename>pmdaIndom</filename> structure are the number of instances in the instance domain and a pointer to an array of instance descriptions.</para>
<para><xref linkend="Z1033578294tls"/> shows that the <literal>pmInDom</literal> can be safely cast to <literal>__pmInDom_int</literal>, which specifies the PMDA's domain and the instance number  within the PMDA:</para>
<para><example id="Z1033578294tls">
<title><literal>__pmInDom_int</literal> Structure</title>
<programlisting>typedef struct { 
        int             flag:1;
        unsigned int    domain:9;   /* the administrative PMD */ 
        unsigned int    serial:22;  /* unique within PMD */         
} __pmInDom_int;</programlisting>
</example></para>
<para>As with metrics, the PMDA domain number is not necessarily known until run time; so the <literal>domain</literal> field must be set up when the PMDA is initialized.</para>
<para><indexterm id="IG31340177142"><primary>pmdaInit man page</primary></indexterm>For information about how an instance domain may also be associated with more than one metric, see the <command>pmdaInit(3)</command> man page.</para>
<para>The  simple PMDA, shown in <xref linkend="Z963524114sdc"/>, has five metrics and two instance domains of three instances.</para>
<example id="Z963524114sdc">
<title>Simple PMDA</title>
<programlisting>/* 
 * list of instances 
 */ 
static pmdaInstid color[] = {
    { 0, &#x201C;red&#x201D; }, { 1, &#x201C;green&#x201D; }, { 2, &#x201C;blue&#x201D; }
};
static pmdaInstid       *timenow = NULL;
static unsigned int     timesize = 0;
/*
 * list of instance domains
 */
static pmdaIndom indomtab[] = {
#define COLOR_INDOM     0
    { COLOR_INDOM, 3, color },
#define NOW_INDOM       1
    { NOW_INDOM, 0, NULL },
};
/*
 * all metrics supported in this PMDA - one table entry for each
 */
static pmdaMetric metrictab[] = {
/* numfetch */
    { NULL,
      { PMDA_PMID(0, 0), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_INSTANT,
       PMDA_PMUNITS(0, 0, 0, 0, 0, 0) }, },
/* color */
    { NULL,
      { PMDA_PMID(0, 1), PM_TYPE_32, COLOR_INDOM, PM_SEM_INSTANT,
        PMDA_PMUNITS(0, 0, 0, 0, 0, 0) }, },
/* time.user */
    { NULL,
      { PMDA_PMID(1, 2), PM_TYPE_DOUBLE, PM_INDOM_NULL, PM_SEM_COUNTER,
        PMDA_PMUNITS(0, 1, 0, 0, PM_TIME_SEC, 0) }, },
/* time.sys */
    { NULL,
      { PMDA_PMID(1,3), PM_TYPE_DOUBLE, PM_INDOM_NULL, PM_SEM_COUNTER,
        PMDA_PMUNITS(0, 1, 0, 0, PM_TIME_SEC, 0) }, },
/* now */
    { NULL,
      { PMDA_PMID(2,4), PM_TYPE_U32, NOW_INDOM, PM_SEM_INSTANT,
        PMDA_PMUNITS(0, 0, 0, 0, 0, 0) }, },
};</programlisting>
</example>
<para><indexterm id="IG31340177143"><primary>COLOR_INDOM</primary></indexterm>The metric <literal>simple.color</literal> is associated, via <literal>COLOR_INDOM</literal>,  with the first instance domain listed in <filename>indomtab</filename>. PMDA initialization assigns the correct domain portion of the instance domain identifier in <filename>indomtab[0].it_indom</filename> and <filename>metrictab[1].m_desc.indom</filename>. This instance domain has three instances: red, green, and blue.</para>
<para>The metric <literal>simple.now</literal> is associated, via <literal>NOW_INDOM</literal><indexterm id="IG31340177144"><primary>NOW_INDOM</primary></indexterm>, with the second instance domain listed in <filename>indomtab</filename>. PMDA initialization assigns the correct domain portion of the instance domain identifier in <filename>indomtab[1].it_indom</filename> and <filename>metrictab[4].m_desc.indom</filename>. This instance domain is dynamic and initially has no instances.</para>
<para><indexterm id="IG31340177145"><primary>PM_INDOM_NULL instance domain</primary><secondary>data structures</secondary></indexterm>All other metrics are singular, as specified by <literal>PM_INDOM_NULL</literal>.</para>
<para>In some cases an instance domain may vary dynamically after PMDA initialization (for example, <literal>simple.now</literal>), and this requires some refinement of the default functions and data structures of the <filename>libpcp_pmda</filename> library. Briefly, this involves providing new functions that act as wrappers for <command>pmdaInstance</command> and <command>pmdaFetch</command> while understanding the dynamics of the instance domain, and then overriding the instance and fetch methods in the <filename>pmdaInterface</filename> structure during PMDA initialization.</para>
<para>For the simple PMDA, the wrapper functions are <command>simple_fetch</command> and <command>simple_instance</command>, and defaults are over-ridden by the following assignments in the <command>simple_init</command> function:</para>
<programlisting>dp-&gt;version.any.fetch = simple_fetch;
dp-&gt;version.any.instance = simple_instance;</programlisting>
</section>
</section>
</section>
<section id="id5191164">

<title>Other Issues</title>
<para>Other issues include extracting the information, latency and threads of control, Name Space, PMDA help text, and management of evolution within a PMDA.</para>
<section id="id5191177">

<title>Extracting the Information</title>
<para><indexterm id="IG31340177146"><primary>information extraction</primary></indexterm>A suggested approach to writing a PMDA is to write a standalone program to extract the values from the target domain and then incorporate this program into the PMDA framework. This approach avoids concurrent debugging of two distinct problems:
<itemizedlist>
<listitem><para><indexterm id="IG31340177147"><primary>exporting data</primary></indexterm> Extraction of the data</para>
</listitem>
<listitem><para>Communication with PMCD</para>
</listitem></itemizedlist></para>
<para><indexterm id="IG31340177148"><primary>target domain</primary></indexterm>These are some possible ways of exporting the data from the target domain:</para>
<itemizedlist>
<listitem><para>Accumulate the performance data in a public shared memory segment.</para>
</listitem>
<listitem><para>Write the performance data to the end of a log file.</para>
</listitem>
<listitem><para>Periodically rewrite a file with the most recent values for the performance data.</para>
</listitem>
<listitem><para>Implement a protocol that allows a third party to connect to the target application, send a request, and receive new performance data.</para>
</listitem>
<listitem><para>If the data is in the operating system kernel, provide a kernel interface (preferred) to export the performance data.</para>
</listitem></itemizedlist>
<para>Most of these approaches require some further data processing by the PMDA.</para>
</section>
<section id="id5191305">

<title>Latency and Threads of Control</title>
<para><indexterm id="IG31340177149"><primary>latency</primary></indexterm><indexterm id="IG31340177150"><primary>control threads</primary></indexterm><indexterm id="IG31340177151"><primary>delays</primary></indexterm>The PCP protocols expect PMDAs to return the current values for performance metrics when requested, and with short delay (low latency). For some target domains, access to the underlying instrumentation may be costly or involve unpredictable  delays (for example, if the real performance data is stored on some remote host or network device). In these cases, it may be necessary to separate probing for new performance data from servicing PMCD requests.</para>
<para><indexterm id="IG31340177152"><primary>pthreads man page</primary></indexterm>An architecture that has been used successfully for several PMDAs is to create one or more child processes to obtain information while the main process communicates with PMCD.</para>
<para>At the simplest deployment of this arrangement, the two processes may execute without synchronization. Threads have also been used as a more portable multithreading mechanism; see the <command>pthreads(7)</command> man page.</para>
<para>By contrast, a complex deployment would be one in which the refreshing of the metric values must be atomic, and this may require double buffering of the data structures. It also requires coordination between parent and child processes.</para>
<warning><para>Since certain data structures used by the PMDA library are not thread-aware, only one PMDA thread of control should call PMDA library functions - this would typically be the thread servicing requests from PMCD.</para>
</warning>
<para><indexterm id="IG31340177153"><primary>caching PMDA</primary></indexterm>One caveat about this style of caching PMDA--in this (special) case it is better if the PMDA converts counts to rates based upon consecutive periodic sampling from the underlying instrumentation. By exporting precomputed rate metrics with instantaneous semantics, the PMDA prevents the PCP monitor tools from computing their own rates upon consecutive PMCD fetches (which are likely to return identical values from a caching PMDA). The finer points of metric semantics are discussed in <xref linkend="id5190312"/></para>
</section>
<section id="LE83854-PARENT">

<title>Name Space</title>
<para><indexterm id="IG31340177154"><primary>pmns man page</primary></indexterm><indexterm id="IG31340177155"><primary>name space</primary></indexterm>The PMNS file defines the name space of the PMDA. It is a simple text file that is used during installation to expand the Name Space of the PMCD process. The format of this file is described by the <command>pmns(5)</command> man page and its hierarchical nature, syntax, and helper tools are further described in the <citetitle>Performance Co-Pilot User's and Administrator's Guide</citetitle>.</para>
<para>Client processes will not be able to access the PMDA metrics if the PMNS file is not installed as part of the PMDA installation procedure on the collector host. The installed list of metric names and their corresponding PMIDs can be found in <filename>${PCP_VAR_DIR}/pmns/root</filename>.</para>
<para><indexterm id="IG31340177156"><primary>simple PMDA</primary><secondary> 2 branches, 4 metrics</secondary></indexterm><xref linkend="Z963526380sdc"/> shows the simple PMDA, which has five metrics:<itemizedlist>
<listitem><para>Three metrics immediately under the <literal>simple</literal> node</para>
</listitem>
<listitem><para>Two metrics under another non-terminal node called <literal>simple.time</literal></para>
</listitem></itemizedlist></para>
<example id="Z963526380sdc">
<title><filename>pmns</filename> File for the Simple PMDA</title>
<programlisting>simple {
    numfetch    SIMPLE:0:0
    color       SIMPLE:0:1
    time
    now         SIMPLE:2:4
}
simple.time {
    user        SIMPLE:1:2
    sys         SIMPLE:1:3
}</programlisting>
</example>
<para>Metrics that have different clusters do not have to be specified in different subtrees of the PMNS.  <xref linkend="Z976046292sdc"/> shows an alternative PMNS for the simple PMDA:</para>
<example id="Z976046292sdc">
<title>Alternate <filename>pmns</filename> File for the Simple PMDA</title>
<programlisting>simple { 
    numfetch    SIMPLE:0:0 
    color       SIMPLE:0:1 
    usertime    SIMPLE:1:2 
    systime     SIMPLE:1:3 
}</programlisting>
<para>In this example, the <literal>SIMPLE</literal> macro is replaced by the domain number listed in <filename>${PCP_VAR_DIR}/pmns/stdpmid</filename> for the corresponding PMDA during installation (for the simple PMDA, this would normally be the value 253).</para>
</example>
<para>If the PMDA implementer so chooses, all or a subset of the metric names and identifiers can be specified programatically. In this situation, a special asterisk syntax is used to denote those subtrees which are to be handles this way.   <xref linkend="Z976046292nat"/> shows this dynamic namespace syntax, for all metrics in the simple PMDA:</para>
<example id="Z976046292nat">
<title>Dynamic metrics <filename>pmns</filename> File for the Simple PMDA</title>
<programlisting>simple         SIMPLE:*:*</programlisting>
<para>In this example, like the one before, the <literal>SIMPLE</literal> macro is replaced by the domain number, and all (simple.*) metric namespace operations must be handled by the PMDA. This is in contrast to the static metric name model earlier, where the host-wide PMNS file is updated and used by PMCD, acting on behalf of the agent.</para>
</example>
</section>
<section id="LE72473-PARENT">

<title>PMDA Help Text</title>
<para><indexterm id="IG31340177157"><primary>PMDA</primary><secondary>help text</secondary></indexterm><indexterm id="IG31340177158"><primary>help text</primary><secondary>terse and extended descriptions</secondary></indexterm>For each metric defined within a PMDA, the PMDA developer is strongly encouraged to provide both terse and extended help text to describe the metric, and perhaps provide hints about the expected value ranges.</para>
<para><indexterm id="IG31340177159"><primary>newhelp man page</primary></indexterm>The help text is used to describe each metric in the visualization tools and <command>pminfo</command> with the <command>-T</command> option. The help text, such as the help text for the simple PMDA in <xref linkend="Z963526754sdc"/>, is specified in a specially formatted file, normally called <filename>help</filename>. This file is converted to the expected run-time format using the <command>newhelp</command> command; see the <command>newhelp(1)</command> man page. Converted help text files are usually placed in the PMDA's directory below <filename>${PCP_PMDAS_DIR}</filename> as part of the PMDA installation procedure.</para>
<example id="Z963526754sdc">
<title>Help Text for the Simple PMDA</title>
<para>The two instance domains and five metrics have a short and a verbose description. Each entry begins with a line that starts with the character &#x201C;@&#x201D; and is followed by either the metric name (<literal>simple.numfetch</literal>) or a symbolic reference to the instance domain number (<literal>SIMPLE.1</literal>), followed by the short description. The verbose description is on the following lines, terminated by the next line starting with &#x201C;@&#x201D; or end of file:</para>
<programlisting>@ SIMPLE.0 Instance domain &#x201C;colour&#x201D; for simple PMDA
Universally 3 instances, &#x201C;red&#x201D; (0), &#x201C;green&#x201D; (1) and &#x201C;blue&#x201D; (3).

@ SIMPLE.1 Dynamic instance domain &#x201C;time&#x201D; for simple PMDA
An instance domain is computed on-the-fly for exporting current time
information. Refer to the help text for simple.now for more details.

@ simple.numfetch Number of pmFetch operations.
The cumulative number of pmFetch operations directed to &#x201C;simple&#x201D; PMDA.

This counter may be modified with pmstore(1).

@ simple.color Metrics which increment with each fetch
This metric has 3 instances, designated &#x201C;red&#x201D;, &#x201C;green&#x201D; and &#x201C;blue&#x201D;.

The value of the metric is monotonic increasing in the range 0 to
255, then back to 0.  The different instances have different starting
values, namely 0 (red), 100 (green) and 200 (blue).

The metric values my be altered using pmstore(1).

@ simple.time.user Time agent has spent executing user code
The time in seconds that the CPU has spent executing agent user code.

@ simple.time.sys Time agent has spent executing system code
The time in seconds that the CPU has spent executing agent system code.

@ simple.now Time of day with a configurable instance domain
The value reflects the current time of day through a dynamically
reconfigurable instance domain.  On each metric value fetch request,
the agent checks to see whether the configuration file in
${PCP_PMDAS_DIR}/simple/simple.conf has been modified - if it has then
the file is re-parsed and the instance domain for this metric is again
constructed according to its contents.

This configuration file contains a single line of comma-separated time
tokens from this set:
  &#x201C;sec&#x201D;  (seconds after the minute),
  &#x201C;min&#x201D;  (minutes after the hour),
  &#x201C;hour&#x201D; (hour since midnight).

An example configuration file could be:  sec,min,hour
and in this case the simple.now metric would export values for the
three instances &#x201C;sec&#x201D;, &#x201C;min&#x201D; and &#x201C;hour&#x201D; corresponding respectively to
the components seconds, minutes and hours of the current time of day.

The instance domain reflects each token present in the file, and the
values reflect the time at which the PMDA processes the fetch.</programlisting>
</example>
</section>
<section id="id5191869">

<title>Management of Evolution within a PMDA</title>
<para><indexterm id="IG31340177160"><primary>new metrics</primary></indexterm><indexterm id="IG31340177161"><primary>PMDA</primary><secondary>evolution</secondary></indexterm>Evolution of a PMDA, or more particularly the underlying instrumentation to which it provides access, over time naturally results in the appearance of new metrics and the disappearance of old metrics. This  creates potential problems for PMAPI clients and PCP tools that may be required to interact with both new and former versions of the PMDA.</para>
<para>The following guidelines are intended to help reduce the complexity of implementing a PMDA in the face of evolutionary change, while maintaining predictability and semantic coherence for tools using the PMAPI, and for end users of those tools.</para>
<itemizedlist>
<listitem><para><indexterm id="IG31340177162"><primary>unavailable metrics support</primary></indexterm>Try to  support as full a range of metrics as possible in every version of the PMDA. In this context, <firstterm>support</firstterm> means responding sensibly to requests, even if the underlying instrumentation is not available.</para>
</listitem>
<listitem><para><indexterm id="IG31340177163"><primary>pmLookupDesc function</primary></indexterm><indexterm id="IG31340177164"><primary>pmDesc structure</primary></indexterm><indexterm id="IG31340177165"><primary>PM_TYPE_NOSUPPORT value</primary></indexterm>If a metric is not supported in a given version of the underlying instrumentation, the PMDA should respond to <command>pmLookupDesc</command> requests with a <filename>pmDesc</filename> structure whose <literal>type</literal> field has the special value  <literal>PM_TYPE_NOSUPPORT</literal>. Values of fields other than <literal>pmid</literal> and <literal>type</literal> are immaterial, but <xref linkend="Z976047129sdc"/> is typically benign:</para>
<example id="Z976047129sdc">
<title> Setting Values</title>
<programlisting>pmDesc dummy = { 
     .pmid  = PMDA_PMID(3,0),           /* pmid, fill this in */
     .type  = PM_TYPE_NOSUPPORT,        /* this is the important part */
     .indom = PM_INDOM_NULL,            /* singular,causes no problems */
     .sem   = 0,                        /* no semantics */
     .units = PMDA_PMUNITS(0,0,0,0,0,0) /* no units */
};</programlisting>
</example>
</listitem>
<listitem><para><indexterm id="IG31340177166"><primary>pmFetch man page</primary></indexterm><indexterm id="IG31340177167"><primary>PM_ERR_PMID error code</primary></indexterm>If a metric lacks support in a particular version of the underlying instrumentation, the PMDA should respond to <command>pmFetch</command> requests with a <command>pmResult</command> in which no values are returned for the unsupported metric. This is marginally friendlier than the other semantically acceptable option of returning an illegal PMID error or <literal>PM_ERR_PMID</literal>.</para>
</listitem>
<listitem><para><indexterm id="IG31340177168"><primary>pmLookupText function</primary></indexterm><indexterm id="IG31340177169"><primary>help text</primary><secondary>pmLookupText function</secondary></indexterm>Help text should be updated with annotations to describe different versions of the underlying product, or product configuration options, for which a specific metric is available. This is so <command>pmLookupText</command> can always respond correctly.</para>
</listitem>
<listitem><para><indexterm id="IG31340177170"><primary>type field</primary></indexterm><indexterm id="IG31340177171"><primary>pmStore function</primary></indexterm>The <command>pmStore</command> operation should fail with return status of <literal>PM_ERR_PERMISSION</literal> if a user or application tries to amend the value of an unsupported metric.</para>
</listitem>
<listitem><para><indexterm id="IG31340177172"><primary>pmExtractValue function</primary></indexterm><indexterm id="IG31340177173"><primary>pmConvScale function</primary></indexterm><indexterm id="IG31340177174"><primary>pmAtomStr function</primary></indexterm><indexterm id="IG31340177175"><primary>pmTypeStr function</primary></indexterm><indexterm id="IG31340177176"><primary>pmPrintValue function</primary></indexterm><indexterm id="IG31340177177"><primary>PM_ERR_CONV error code</primary></indexterm> The value extraction, conversion, and printing functions (<command>pmExtractValue</command>, <command>pmConvScale</command>, <command>pmAtomStr</command>, <command>pmTypeStr</command>, and <command>pmPrintValue</command>) return the  <literal>PM_ERR_CONV</literal> error or an appropriate diagnostic string, if an attempt is made to operate on a value for which <literal>type</literal> is <literal>PM_TYPE_NOSUPPORT</literal>.</para>
<para>If performance tools take note of the <literal>type</literal> field in the <filename>pmDesc</filename> structure, they should not manipulate values for unsupported metrics. Even if tools ignore <literal>type</literal> in the metric's description, following these development guidelines ensures that no misleading value is ever returned; so there is no reason to call the extraction, conversion, and printing functions.</para>
</listitem></itemizedlist>
</section>
</section>
<section id="LE21831-PARENT">

<title>PMDA Interface</title>
<para><indexterm id="IG31340177178"><primary>PMDA</primary><secondary>interface</secondary></indexterm>This section describes an interface for the request handling callbacks in a PMDA. This interface is used by PMCD for communicating with DSO PMDAs and is also used by daemon PMDAs with <command>pmdaMain</command>.</para>
<section id="id5192358">

<title>Overview</title>
<para>Both daemon and DSO PMDAs must handle multiple request types from PMCD. A daemon PMDA communicates with PMCD using the PDU protocol, while a DSO PMDA defines callbacks for each request type. To avoid duplicating this PDU processing (in the case of a PMDA that can be installed either as a daemon or as a DSO), and to allow a consistent framework, <command>pmdaMain</command> can be used by a daemon PMDA as a wrapper to handle the communication protocol using the same callbacks as a DSO PMDA. This allows a PMDA to be built as both a daemon and a DSO, and then to be installed as either.</para>
<para>To further simplify matters, default callbacks are declared in <filename>&lt;pcp/pmda.h&gt;</filename>:</para>
<itemizedlist>
<listitem><para><indexterm id="IG31340177179"><primary>pmdaFetch callback</primary></indexterm><command>pmdaFetch</command></para>
</listitem>
<listitem><para><indexterm id="IG31340177180"><primary>pmdaProfile callback</primary></indexterm><command>pmdaProfile</command></para>
</listitem>
<listitem><para><indexterm id="IG31340177181"><primary>pmdaInstance callback</primary></indexterm> <command>pmdaInstance</command></para>
</listitem>
<listitem><para><indexterm id="IG31340177182"><primary>pmdaDesc callback</primary></indexterm><command>pmdaDesc</command></para>
</listitem>
<listitem><para><indexterm id="IG31340177183"><primary>pmdaText callback</primary></indexterm> <command>pmdaText</command></para>
</listitem>
<listitem><para><indexterm id="IG31340177184"><primary>pmdaStore callback</primary></indexterm> <command>pmdaStore</command></para>
</listitem>
<listitem><para><indexterm id="IG31340177nat"><primary>pmdaPMID callback</primary></indexterm> <command>pmdaPMID</command></para>
</listitem>
<listitem><para><indexterm id="IG31340178nat"><primary>pmdaName callback</primary></indexterm> <command>pmdaName</command></para>
</listitem>
<listitem><para><indexterm id="IG31340179nat"><primary>pmdaChildren callback</primary></indexterm> <command>pmdaChildren</command></para>
</listitem>
<listitem><para><indexterm id="IG31340180nat"><primary>pmdaAttribute callback</primary></indexterm> <command>pmdaAttribute</command></para>
</listitem></itemizedlist>
<para><indexterm id="IG31340177185"><primary>pmdaExt structure</primary></indexterm>Each callback takes a <filename>pmdaExt</filename> structure as its last argument. This structure contains all the information that is required by the default callbacks in most cases. The one exception is <command>pmdaFetch</command>, which needs an additional callback to instantiate the current value for each supported combination of a performance metric and an instance.</para>
<para>Therefore, for most PMDAs all the communication with PMCD is automatically handled by functions in <filename>libpcp.so</filename> and <filename>libpcp_pmda.so</filename>.</para>
<section id="id5192570">

<title>Trivial PMDA</title>
<para><indexterm id="IG31340177186"><primary>trivial PMDA</primary><secondary>callbacks</secondary></indexterm>The trivial PMDA uses all of the default callbacks as shown in <xref linkend="Z964109292sdc"/>. The additional callback for <command>pmdaFetch</command> is defined as <command>trivial_fetchCallBack</command>:</para>
<example id="Z964109292sdc">
<title>Request Handling Callbacks in the Trivial PMDA</title>
<programlisting>static int
trivial_fetchCallBack(pmdaMetric *mdesc, unsigned int inst, pmAtomValue *atom)
{
   __pmID_int      *idp = (__pmID_int *)&amp;(mdesc-&gt;m_desc.pmid);

   if (idp-&gt;cluster != 0 || idp-&gt;item != 0)
       return PM_ERR_PMID;
   if (inst != PM_IN_NULL)
       return PM_ERR_INST;
   atom-&gt;l = time(NULL);
   return 0;
}</programlisting>
<para><indexterm id="IG31340177187"><primary>trivial_init function</primary></indexterm>This function checks that the PMID and instance are valid, and then places the metric value for the current time into the <filename>pmAtomValue</filename> structure.</para>
</example>
<para>The callback is set up by a call to <command>pmdaSetFetchCallBack</command> in <command>trivial_init</command>. As a rule of thumb, the API routines with named ending with <command>CallBack</command> are helpers for the higher PDU handling routines like <command>pmdaFetch</command>. The latter are set directly using the PMDA Interface Structures, as described in <xref linkend="id5193658"/>.</para>
</section>
<section id="id5192681">

<title>Simple PMDA</title>
<para><indexterm id="IG31340177188"><primary>simple PMDA</primary><secondary>pmdaFetch callback</secondary></indexterm><indexterm id="IG31340177189"><primary>simple_init function</primary></indexterm>The simple PMDA callback for <command>pmdaFetch</command> is more complicated because it supports more metrics, some metrics are instantiated with each fetch, and one instance domain is dynamic. The default <command>pmdaFetch</command> callback, shown in <xref linkend="Z964110950sdc"/>, is replaced by <command>simple_fetch</command> in <command>simple_init</command>, which increments the number of fetches and updates the instance domain for <literal>INDOM_NOW</literal> before calling <command>pmdaFetch</command>:</para>
<example id="Z964110950sdc">
<title>Request Handling Callbacks in the Simple PMDA</title>
<programlisting>static int
simple_fetch(int numpmid, pmID pmidlist[], pmResult **resp, pmdaExt *pmda)
{
    numfetch++;
    simple_timenow_check();
    simple_timenow_refresh();
    return pmdaFetch(numpmid, pmidlist, resp, pmda);
}</programlisting>
</example>
<para><indexterm id="IG31340177190"><primary>pmAtomValue structure</primary></indexterm>The callback for <command>pmdaFetch</command> is defined as <command>simple_fetchCallBack</command>. The PMID is extracted from the <filename>pmdaMetric</filename> structure, and if valid, the appropriate field in the <filename>pmAtomValue</filename> structure is set. The available types and associated fields are described further in <xref linkend="LE11914-PARENT"/> and <xref linkend="Z976562908sdc"/>.</para>
<note><para>Note that PMID validity checking need only check the cluster and item numbers, the domain number is guaranteed to be valid and the PMDA should make no assumptions about the actual domain number being used at this point.</para></note>
<para>The <literal>simple.numfetch</literal> metric has no instance domain and is easily handled first as shown in <xref linkend="Z976306482sdc"/>:</para>
<example id="Z976306482sdc">
<title><literal>simple.numfetch</literal> Metric</title>
<programlisting>static int
simple_fetchCallBack(pmdaMetric *mdesc, unsigned int inst, pmAtomValue *atom)
{
   int             i;
   static int      oldfetch;
   static double   usr, sys;
   __pmID_int      *idp = (__pmID_int *)&amp;(mdesc-&gt;m_desc.pmid);

   if (inst != PM_IN_NULL &amp;&amp;
       !(idp-&gt;cluster == 0 &amp;&amp; idp-&gt;item == 1) &amp;&amp;
       !(idp-&gt;cluster == 2 &amp;&amp; idp-&gt;item == 4))
       return PM_ERR_INST;
   if (idp-&gt;cluster == 0) {
       if (idp-&gt;item == 0) {                   /* simple.numfetch */
           atom-&gt;l = numfetch;
       }</programlisting>
</example>
<para><indexterm id="IG31340177191"><primary>simple.color metric</primary></indexterm>In <xref linkend="Z976049747sdc"/>, the <literal>inst</literal> parameter is used to specify which instance is required for the <literal>simple.color</literal> metric:</para>
<example id="Z976049747sdc">
<title><literal>simple.color</literal> Metric</title>
<programlisting>       else if (idp-&gt;item == 1) {              /* simple.color */
            switch (inst) {
            case 0:                             /* red */
                red = (red + 1) % 256;
                atom-&gt;l = red;
                break;
            case 1:                             /* green */
                green = (green + 1) % 256;
                atom-&gt;l = green;
                break;
            case 2:                             /* blue */
                blue = (blue + 1) % 256;
                atom-&gt;l = blue;
                break;
            default:
                return PM_ERR_INST;
            }
       }
       else
           return PM_ERR_PMID;</programlisting>
</example>
<para><indexterm id="IG31340177192"><primary>simple.time metric</primary></indexterm>In <xref linkend="Z976049353sdc"/>, the <literal>simple.time</literal> metric is in a second cluster and has a simple optimization to reduce the overhead of calling <command>times</command> twice on the same fetch and return consistent values from a single call to <command>times</command> when both metrics <literal>simple.time.user</literal> and <literal>simple.time.sys</literal> are requested in a single <command>pmFetch</command>. The previous fetch count is used to determine if the <filename>usr</filename> and <filename>sys</filename> values should be updated:</para>
<example id="Z976049353sdc">
<title><literal>simple.time</literal> Metric</title>
<programlisting>   else if (idp-&gt;cluster == 1) {               /* simple.time */
       if (oldfetch &lt; numfetch) {
           __pmProcessRunTimes(&amp;usr, &amp;sys);
           oldfetch = numfetch;
       }
       if (idp-&gt;item == 2)                     /* simple.time.user */
           atom-&gt;d = usr;
       else if (idp-&gt;item == 3)                /* simple.time.sys */
           atom-&gt;d = sys;
       else
           return PM_ERR_PMID;
    }</programlisting>
</example>
<para><indexterm id="IG31340177193"><primary>simple.now metric</primary></indexterm>In <xref linkend="Z976049020sdc"/>, the <literal>simple.now</literal> metric is in a third cluster and uses <literal>inst</literal> again to select a specific instance from the <literal>INDOM_NOW</literal> instance domain. The values associated with instances in this instance domain are managed using the <command>pmdaCache(3)</command> helper routines, which provide efficient interfaces for managing more complex instance domains:</para>
<example id="Z976049020sdc">
<title><literal>simple.now</literal> Metric</title>
<programlisting>    else if (idp-&gt;cluster == 2) {
        if (idp-&gt;item == 4) {                 /* simple.now */
            struct timeslice *tsp;
            sts = pmdaCacheLookup(*now_indom, inst, NULL, (void *)&amp;tsp);
            if (sts != PMDA_CACHE_ACTIVE) {
                if (sts &lt; 0)
                    __pmNotifyErr(LOG_ERR, "pmdaCacheLookup failed: inst=%d: %s",
                                  inst, pmErrStr(sts));
                return PM_ERR_INST;
            }
            atom-&gt;l = tsp-&gt;tm_field;
        }
        else 
            return PM_ERR_PMID;
    }</programlisting>
</example>
</section>
<section id="id5193108">
<title><literal>simple_store</literal> in the Simple PMDA</title>
<para><indexterm id="IG31340177194"><primary>simple.store metric</primary></indexterm><indexterm id="IG31340177195"><primary id="Z980104423sdc">pmstore function</primary></indexterm>The simple PMDA permits some of the metrics it supports to be modified by <command>pmStore</command> as shown in <xref linkend="Z964111850sdc"/>. For additional information, see the <command>pmstore(1)</command> and <command>pmStore(3)</command> man pages.</para>
<example id="Z964111850sdc">
<title><literal>simple_store</literal> in the Simple PMDA</title>
<para><indexterm id="IG31340177196"><primary>pmdaStore callback</primary></indexterm>The <command>pmdaStore</command> callback (which returns <literal>PM_ERR_PERMISSION</literal> to indicate no metrics can be altered) is replaced by <command>simple_store</command> in <command>simple_init</command>. This replacement function must take the same arguments so that it can be assigned to the function pointer in the <filename>pmdaInterface</filename> structure.</para>
<para>The function traverses the <literal>pmResult</literal> and checks the cluster and unit of each PMID to ensure that it corresponds to a metric that can be changed. Checks are made on the values to ensure they are within range before being assigned to variables in the PMDA that hold the current values for exported metrics:</para>
<programlisting>static int
simple_store(pmResult *result, pmdaExt *pmda)
{
    int         i, j, val, sts = 0;
    pmAtomValue av;
    pmValueSet  *vsp = NULL;
    __pmID_int  *pmidp = NULL;

    /* a store request may affect multiple metrics at once */
    for (i = 0; i &lt; result-&gt;numpmid; i++) {
        vsp = result-&gt;vset[i];
        pmidp = (__pmID_int *)&amp;vsp-&gt;pmid;
        if (pmidp-&gt;cluster == 0) {  /* storable metrics are cluster 0 */
            switch (pmidp-&gt;item) {
            case 0:                           /* simple.numfetch */
                val = vsp-&gt;vlist[0].value.lval;
                if (val &lt; 0) {
                    sts = PM_ERR_SIGN;
                    val = 0;
                }
                numfetch = val;
                break;
            case 1:                             /* simple.color */
                /* a store request may affect multiple instances at once */
                for (j = 0; j &lt; vsp-&gt;numval &amp;&amp; sts == 0; j++) {
                    val = vsp-&gt;vlist[j].value.lval;
                    if (val &lt; 0) {
                        sts = PM_ERR_SIGN;
                        val = 0;
                    } if (val &gt; 255) {
                        sts = PM_ERR_CONV;
                        val = 255;
                    }</programlisting>
</example>
<para><indexterm id="IG31340177197"><primary>PM_ERR_INST error code</primary></indexterm>The <literal>simple.color</literal> metric has an instance domain that must be searched because any or all instances may be specified. Any instances that are not supported in this instance domain should cause an error value of <literal>PM_ERR_INST</literal> to be returned as shown in <xref linkend="Z976051081sdc"/>:</para>
<example id="Z976051081sdc">
<title><literal>simple.color</literal> and <literal>PM_ERR_INST</literal> Errors</title>
<programlisting>                       switch (vsp-&gt;vlist[j].inst) {
                       case 0:                         /* red */
                           red = val;
                           break;
                       case 1:                         /* green */
                           green = val;
                           break;
                       case 2:                         /* blue */
                           blue = val;
                           break;
                       default:
                           sts = PM_ERR_INST;
                       }</programlisting>
</example>
<para><indexterm id="IG31340177198"><primary>PM_ERR_PMID error code</primary></indexterm>Any other PMIDs in cluster 0 that are not supported by the simple PMDA should result in an error value  of <literal>PM_ERR_PMID</literal> as shown in <xref linkend="Z976307148sdc"/>:</para>
<example id="Z976307148sdc">
<title><literal>PM_ERR_PMID</literal> Errors</title>
<programlisting>                default:
                    sts = PM_ERR_PMID;
                    break;
            }
        }</programlisting>
</example>
<para>Any metrics that cannot be altered should generate an error value of <literal>PM_ERR_PERMISSION</literal>, and metrics not supported by the PMDA should result in an error value of <literal>PM_ERR_PMID</literal> as shown in <xref linkend="Z976050822sdc"/>:</para>
<example id="Z976050822sdc">
<title><literal>PM_ERR_PERMISSION</literal> and <literal>PM_ERR_PMID</literal> Errors</title>
<programlisting>        else if ((pmidp-&gt;cluster == 1 &amp;&amp;
                 (pmidp-&gt;item == 2 || pmidp-&gt;item == 3)) ||
                 (pmidp-&gt;cluster == 2 &amp;&amp; pmidp-&gt;item == 4)) {
            sts = PM_ERR_PERMISSION;
            break;
        }
        else {
            sts = PM_ERR_PMID;
            break;
        }
    }
    return sts;
}</programlisting>
<para>The structure <literal>pmdaExt</literal> <replaceable>pmda</replaceable> argument is not used by the <command>simple_store</command> function above.</para>
</example>
<note><para>When using storable metrics, it is important to consider the implications. It is possible <command>pmlogger</command> is actively sampling the metric being modified, for example, which may cause unexpected results to be persisted in an archive. Consider also the use of client credentials, available via the <command>attribute</command> callback of the <filename>pmdaInterface</filename> structure, to appropriately limit access to any modifications that might be made via your storable metrics.</para></note>
</section>
<section id="id5193469">

<title>Return Codes for <command>pmdaFetch</command> Callbacks</title>
<para>In <literal>PMDA_INTERFACE_1</literal> and <literal>PMDA_INTERFACE_2</literal>, the return codes for the <command>pmdaFetch</command> callback function are defined:</para>
<variablelist>
<varlistentry>
<term>Value</term>
<listitem><para>Meaning</para>
</listitem></varlistentry>
<varlistentry>
<term>&lt; 0</term>
<listitem><para>Error code (for example, <literal>PM_ERR_PMID</literal>, <literal>PM_ERR_INST</literal> or <literal>PM_ERR_AGAIN</literal>)</para>
</listitem></varlistentry>
<varlistentry>
<term>0</term>
<listitem><para>Success</para>
</listitem></varlistentry>
</variablelist>
<para>In <literal>PMDA_INTERFACE_3</literal> and all later versions, the return codes for the <command>pmdaFetch</command> callback function are defined:</para>
<variablelist>
<varlistentry>
<term>Value</term>
<listitem><para>Meaning</para>
</listitem></varlistentry>
<varlistentry>
<term>&lt; 0</term>
<listitem><para>Error code (for example, <literal>PM_ERR_PMID</literal>, <literal>PM_ERR_INST</literal>)</para>
</listitem></varlistentry>
<varlistentry>
<term>0</term>
<listitem><para>Metric value not currently available</para>
</listitem></varlistentry>
<varlistentry>
<term>&gt; 0</term>
<listitem><para>Success</para>
</listitem></varlistentry>
</variablelist>
</section>
</section>
<section id="id5193658">

<title>PMDA Structures</title>
<para><indexterm id="IG31340177199"><primary>PMDA</primary><secondary>structures</secondary></indexterm>PMDA structures used with the <filename>pcp_pmda</filename> library are defined in <filename>&lt;pcp/pmda.h&gt;</filename>.  <xref linkend="Z964112160sdc"/> and <xref linkend="Z964117744sdc"/> describe the <literal>pmdaInterface</literal> and <literal>pmdaExt</literal> structures.</para>
<example id="Z964112160sdc">
<title><literal>pmdaInterface</literal> Structure Header</title>
<para><indexterm id="IG31340177200"><primary>pmdaInterface structure</primary></indexterm>The callbacks must be specified in a <filename>pmdaInterface</filename> structure:</para>
<programlisting>typedef struct {
    int domain;     /* set/return performance metrics domain id here */
    struct {
        unsigned int pmda_interface : 8;  /* PMDA DSO version */
        unsigned int pmapi_version : 8;   /* PMAPI version */
        unsigned int flags : 16;          /* optional feature flags */
    } comm;             /* set/return communication and version info */
    int status;         /* return initialization status here */
    union {
        ...
</programlisting>
</example>
<para>This structure is passed by PMCD to a DSO PMDA as an argument to the initialization function. This structure supports multiple (binary-compatible) versions--the second and subsequent versions have support for the <filename>pmdaExt</filename> structure. Protocol version one is for backwards compatibility only, and should not be used in any new PMDA.</para>
<para>To date there have been six revisions of the interface structure:</para>
<itemizedlist>
<listitem><para>Version two added the <literal>pmdaExt</literal> structure, as mentioned above.</para>
</listitem>
<listitem><para>Version three changed the fetch callback return code semantics, as mentioned in <xref linkend="id5193469"/>.</para>
</listitem>
<listitem><para>Version four added support for dynamic metric names, where the PMDA is able to create and remove metric names on-the-fly in response to changes in the performance domain (<command>pmdaPMID</command>, <command>pmdaName</command>, <command>pmdaChildren</command> interfaces)</para>
</listitem>
<listitem><para>Version five added support for per-client contexts, where the PMDA is able to track arrival and disconnection of PMAPI client tools via PMCD (<command>pmdaGetContext</command> helper routine). At the same time, support for <literal>PM_TYPE_EVENT</literal> metrics was implemented, which relies on the per-client context concepts (<command>pmdaEvent*</command> helper routines).</para>
</listitem>
<listitem><para>Version six added support for authenticated client contexts, where the PMDA is informed of user credentials and other PMCD attributes of the connection between individual PMAPI clients and PMCD (<command>pmdaAttribute</command> interface)</para>
</listitem>
</itemizedlist>
<example id="Z964112160nat">
<title><literal>pmdaInterface</literal> Structure, Latest Version</title>
<programlisting>   ...
    union {
        ...
        /*
         * PMDA_INTERFACE6
         */ 
        struct {
            pmdaExt *ext;
            int     (*profile)(pmdaInProfile *, pmdaExt *);
            int     (*fetch)(int, pmID *, pmResult **, pmdaExt *);
            int     (*desc)(pmID, pmDesc *, pmdaExt *);
            int     (*instance)(pmInDom, int, char *, pmdaInResult **, pmdaExt *);
            int     (*text)(int, int, char **, pmdaExt *);
            int     (*store)(pmResult *, pmdaExt *);
            int     (*pmid)(const char *, pmID *, pmdaExt *);
            int     (*name)(pmID, char ***, pmdaExt *);
            int     (*children)(const char *, int, char ***, int **, pmdaExt *);
            int     (*attribute)(int, int, const char *, int, pmdaExt *);
        } six;
    } version;
} pmdaInterface;</programlisting>
</example>
<note><para>Each new interface version is always defined as a superset of those that preceded it, only adds fields at the end of the new structure in the union, and is always binary backwards-compatible. <literal>And thus it shall remain.</literal>
For brevity, we have shown only the latest interface version (six) above, but all prior versions still exist, build, and function. In other words, PMDAs built against earlier versions of this header structure (and PMDA library) function correctly with the latest version of the PMDA library.
</para></note>

<example id="Z964117744sdc">
<title><literal>pmdaExt</literal> Stucture</title>
<para><indexterm id="IG31340177201"><primary>pmdaExt structure</primary></indexterm>Additional PMDA information must be specified in a <filename>pmdaExt</filename> structure:</para>
<programlisting id="Z964117899sdc">typedef struct {
    unsigned int e_flags;       /* PMDA_EXT_FLAG_* bit field */
    void        *e_ext;         /* used internally within libpcp_pmda */
    char        *e_sockname;    /* socket name to pmcd */
    char        *e_name;        /* name of this pmda */
    char        *e_logfile;     /* path to log file */
    char        *e_helptext;    /* path to help text */
    int         e_status;       /* =0 is OK */
    int         e_infd;         /* input file descriptor from pmcd */
    int         e_outfd;        /* output file descriptor to pmcd */
    int         e_port;         /* port to pmcd */
    int         e_singular;     /* =0 for singular values */
    int         e_ordinal;      /* &gt;=0 for non-singular values */
    int         e_direct;       /* =1 if pmid map to meta table */
    int         e_domain;       /* metrics domain */
    int         e_nmetrics;     /* number of metrics */
    int         e_nindoms;      /* number of instance domains */
    int         e_help;         /* help text comes via this handle */
    __pmProfile *e_prof;        /* last received profile */
    pmdaIoType  e_io;           /* connection type to pmcd */
    pmdaIndom   *e_indoms;      /* instance domain table */
    pmdaIndom   *e_idp;         /* instance domain expansion */
    pmdaMetric  *e_metrics;     /* metric description table */
    pmdaResultCallBack e_resultCallBack; /* to clean up pmResult after fetch */
    pmdaFetchCallBack  e_fetchCallBack;  /* to assign metric values in fetch */
    pmdaCheckCallBack  e_checkCallBack;  /* callback on receipt of a PDU */
    pmdaDoneCallBack   e_doneCallBack;   /* callback after PDU is processed */
    /* added for PMDA_INTERFACE_5 */
    int         e_context;      /* client context id from pmcd */
    pmdaEndContextCallBack e_endCallBack;  /* callback after client context closed */
} pmdaExt;</programlisting>
</example>
<para><indexterm id="IG31340177202"><primary>pmdaConnect man page</primary></indexterm><indexterm id="IG31340177203"><primary>pmdaDSO man page</primary></indexterm><indexterm id="IG31340177204"><primary>pmdaDaemon man page</primary></indexterm><indexterm id="IG31340177205"><primary>pmdaGetOptions man page</primary></indexterm><indexterm id="IG31340177206"><primary>pmdaInit man page</primary></indexterm>The <filename>pmdaExt</filename> structure contains filenames, pointers to tables, and some variables shared by several functions in the <filename>pcp_pmda </filename>library. All fields of the <filename>pmdaInterface</filename> and <filename>pmdaExt</filename> structures can be correctly set by PMDA initialization functions; see the <command>pmdaDaemon(3)</command>, <command>pmdaDSO(3)</command>, <command>pmdaGetOptions(3)</command>, <command>pmdaInit(3)</command>, and <command>pmdaConnect(3)</command> man pages for a full description of how various fields in these structures may be set or used by <filename>pcp_pmda</filename> library functions.</para>
</section>
</section>
<section id="LE19047-PARENT">

<title>Initializing a PMDA</title>
<para> <indexterm id="IG31340177207"><primary>PMDA</primary><secondary>initialization</secondary></indexterm>Several functions are provided to simplify the initialization of a PMDA. These functions, if used, must be called in a strict order so that the PMDA can operate correctly.</para>
<section id="id5194056">

<title>Overview</title>
<para><indexterm id="IG31340177208"><primary>pmdaInterface structure</primary></indexterm>The initialization process for a PMDA involves opening help text files, assigning callback function pointers, adjusting the metric and instance identifiers to the correct domains, and much more. The initialization of a daemon PMDA also differs significantly from a DSO PMDA, since the <literal>pmdaInterface</literal> structure is initialized by <command>main</command> or the PMCD process, respectively.</para>
</section>
<section id="id5194087">

<title>Common Initialization</title>
<para><indexterm id="IG31340177209"><primary>DSO</primary><secondary>PMDA initialization</secondary></indexterm>As described in <xref linkend="LE82676-PARENT"/>, an initialization function is provided by a DSO&#xA0;PMDA and called by PMCD. Using the standard PMDA wrappers, the same function can also be used as part of the daemon PMDA initialization. This PMDA initialization function performs the following tasks:</para>
<itemizedlist>
<listitem><para>Assigning callback functions to the function pointer interface of <filename>pmdaInterface</filename></para>
</listitem>
<listitem><para>Assigning pointers to the metric and instance tables from <filename>pmdaExt</filename></para>
</listitem>
<listitem><para><indexterm id="IG31340177210"><primary>help text</primary><secondary>initialization</secondary></indexterm>Opening the help text files</para>
</listitem>
<listitem><para>Assigning the domain number to the instance domains</para>
</listitem>
<listitem><para>Correlating metrics with their instance domains</para>
</listitem></itemizedlist>
<para><indexterm id="IG31340177211"><primary>pmdaInit man page</primary></indexterm><indexterm id="IG31340177212"><primary>pmdaInit man page</primary></indexterm>If the PMDA uses the common data structures defined for the <filename>pcp_pmda</filename> library, most of these requirements can be handled by the default <command>pmdaInit</command> function; see the <command>pmdaInit(3)</command> man page.</para>
<para>Because the initialization function is the only initialization opportunity for a DSO PMDA, the common initialization function should also perform any DSO-specific functions that are required. A default implementation of this functionality is provided by the <command>pmdaDSO</command> function; see the <command>pmdaDSO(3)</command> man page.</para>
<section id="id5194313">

<title>Trivial PMDA</title>
<para><indexterm id="IG31340177213"><primary>trivial PMDA</primary><secondary>initialization</secondary></indexterm><indexterm id="IG31340177214"><primary>pmdaFetch callback</primary></indexterm><indexterm id="IG31340177215"><primary>trivial_init function</primary></indexterm><xref linkend="Z976058585sdc"/> shows the trivial PMDA, which has no instances (that is, all metrics have singular values) and a single callback. This callback is for the <command>pmdaFetch</command> function called <command>trivial_fetchCallBack</command>; see the <command>pmdaFetch(3)</command> man page:</para>
<example id="Z976058585sdc">
<title>Initialization in the Trivial PMDA</title>
<programlisting>static char     *username;
static int      isDSO = 1;              /* ==0 if I am a daemon */

void trivial_init(pmdaInterface *dp)
{
    if (isDSO)
        pmdaDSO(dp, PMDA_INTERFACE_2, &#x201C;trivial DSO&#x201D;,
                &#x201C;${PCP_PMDAS_DIR}/trivial/help&#x201D;);
    else
        __pmSetProcessIdentity(username);

    if (dp-&gt;status != 0)
        return;

    pmdaSetFetchCallBack(dp, trivial_fetchCallBack);
    pmdaInit(dp, NULL, 0,
             metrictab, sizeof(metrictab)/sizeof(metrictab[0]));
}</programlisting>
</example>
<para>The trivial PMDA can execute as either a DSO or daemon PMDA. A default
installation installs it as a daemon, however, and the <command>main</command>
routine clears <replaceable>isDSO</replaceable> and sets <replaceable>username</replaceable>
accordingly.</para>
<para>The <command>trivial_init</command> routine provides the opportunity to do
any extra DSO or daemon setup before calling the library <command>pmdaInit</command>.
In the example, the help text is setup for DSO mode and the daemon is
switched to run as an unprivileged user (default is <literal>root</literal>,
but it is generally good form for PMDAs to run with the least privileges possible).
If <literal>dp-&gt;status</literal> is non-zero after the <command>pmdaDSO</command>
call, the PMDA will be removed by PMCD and cannot safely continue to use the
<command>pmdaInterface</command> structure.</para>
</section>
<section id="id5194416">

<title>Simple PMDA</title>
<para><indexterm id="IG31340177216"><primary>simple PMDA</primary><secondary>initialization</secondary></indexterm><indexterm id="IG31340177217"><primary>simple_init function</primary></indexterm><indexterm id="IG31340177218"><primary>PDU_RESULT</primary></indexterm><indexterm id="IG31340177219"><primary>PDU_FETCH</primary></indexterm>In <xref linkend="Z976058770sdc"/>, the simple PMDA uses its own callbacks to handle <literal>PDU_FETCH</literal> and <literal>PDU_RESULT</literal> request PDUs (for <command>pmFetch</command> and <command>pmStore</command> operations respectively), as well as providing <command>pmdaFetch</command> with the callback <command>simple_fetchCallBack</command>.</para>
<example id="Z976058770sdc">
<title>Initialization in the Simple PMDA</title>
<programlisting>static int      isDSO = 1;              /* =0 I am a daemon */
static char     *username;

void simple_init(pmdaInterface *dp)
{
    if (isDSO)
        pmdaDSO(dp, PMDA_INTERFACE_2, &#x201C;simple DSO&#x201D;,
                &#x201C;${PCP_PMDAS_DIR}/simple/help&#x201D;);
    else
        __pmSetProcessIdentity(username);

    if (dp-&gt;status != 0)
        return;

    dp-&gt;version.any.fetch = simple_fetch;
    dp-&gt;version.any.store = simple_store;
    dp-&gt;version.any.instance = simple_instance;
    pmdaSetFetchCallBack(dp, simple_fetchCallBack);
    pmdaInit(dp, indomtab, sizeof(indomtab)/sizeof(indomtab[0]),
             metrictab, sizeof(metrictab)/sizeof(metrictab[0]));
}</programlisting>
</example>
<para>Once again, the simple PMDA may be installed either as a daemon PMDA or a DSO PMDA. The static variable <replaceable>isDSO</replaceable> indicates whether the PMDA is running as a DSO or as a daemon. A daemon PMDA always changes the value of this variable to 0 in <literal>main</literal>, for PMDAs that can operate in both modes.</para>
<para>Remember also, as described earlier, <command>simple_fetch</command> is dealing with a single request for (possibly many) values for metrics from the PMDA, and <command>simple_fetchCallBack</command> is its little helper, dealing with just one metric and one instance (optionally, if the metric happens to have an instance domain) within that larger request.</para>
</section>
</section>
<section id="id5194563">

<title>Daemon Initialization</title>
<para>In addition to the initialization function that can be shared by a DSO and a daemon PMDA, a daemon PMDA must also meet the following requirements:</para>
<itemizedlist>
<listitem><para>Create the <filename>pmdaInterface</filename> structure that is passed to the initialization function</para>
</listitem>
<listitem><para>Parse any command-line arguments</para>
</listitem>
<listitem><para>Open a log file (a DSO PMDA uses PMCD's log file)</para>
</listitem>
<listitem><para>Set up the IPC connection between the PMDA and the PMCD process</para>
</listitem>
<listitem><para>Handle incoming PDUs</para>
</listitem></itemizedlist>
<para><indexterm id="IG31340177220"><primary>pmdaDaemon man page</primary></indexterm><indexterm id="IG31340177221"><primary>pmdaGetOptions man page</primary></indexterm><indexterm id="IG31340177222"><primary>pmdaOpenLog man page</primary></indexterm><indexterm id="IG31340177223"><primary>pmdaConnect man page</primary></indexterm><indexterm id="IG31340177224"><primary>pmdaMain man page</primary></indexterm>All these requirements can be handled by default initialization functions in the <filename>pcp_pmda</filename> library; see the <command>pmdaDaemon(3)</command>, <command>pmdaGetOptions(3)</command>, <command>pmdaOpenLog(3)</command>, <command>pmdaConnect(3)</command>, and <command>pmdaMain(3)</command> man pages.</para>
<note><para>Optionally, a daemon PMDA may wish to reduce or change its privilege level, as seen in <xref linkend="Z976058585sdc"/> and <xref linkend="Z976058770sdc"/>. Some performance domains <literal>require</literal> the extraction process to run as a specific user in order to access the instrumentation. Many domains require the default <literal>root</literal> level of access for a daemon PMDA.</para></note>
<para><indexterm id="IG31340177225"><primary>pmdaGetOptions man page</primary></indexterm>The simple PMDA specifies the command-line arguments it accepts using <command>pmdaGetOptions</command>, as shown in <xref linkend="Z964110483sdc"/>. For additional information, see the <command>pmdaGetOptions(3)</command> man page.</para>
<example id="Z964110483sdc">
<title><literal>main</literal> in the Simple PMDA</title>
<programlisting>static pmLongOptions longopts[] = {
    PMDA_OPTIONS_HEADER(&#x201C;Options&#x201D;),
    PMOPT_DEBUG,
    PMDAOPT_DOMAIN,
    PMDAOPT_LOGFILE,
    PMDAOPT_USERNAME,
    PMOPT_HELP,
    PMDA_OPTIONS_TEXT(&#x201C;\nExactly one of the following options may appear:&#x201D;),
    PMDAOPT_INET,
    PMDAOPT_PIPE,
    PMDAOPT_UNIX,
    PMDAOPT_IPV6,
    PMDA_OPTIONS_END
};
static pmdaOptions opts = {
    .short_options = &#x201C;D:d:i:l:pu:U:6:?&#x201D;,
    .long_options = longopts,
};

int
main(int argc, char **argv)
{
    pmdaInterface       dispatch;

    isDSO = 0;
    __pmSetProgname(argv[0]);
    __pmGetUsername(&amp;username);
    pmdaDaemon(&amp;dispatch, PMDA_INTERFACE_2, pmProgname, SIMPLE,
               &#x201C;simple.log&#x201D;, &#x201C;${PCP_PMDAS_DIR}/simple/help&#x201D;);

    pmdaGetOptions(argc, argv, &amp;opts, &amp;dispatch);
    if (opts.errors) {
        pmdaUsageMessage(&amp;opts);
        exit(1);
    }
    if (opts.username)
        username = opts.username;

    pmdaOpenLog(&amp;dispatch);
    simple_init(&amp;dispatch);
    simple_timenow_check();
    pmdaConnect(&amp;dispatch);
    pmdaMain(&amp;dispatch);

    exit(0);
}</programlisting>
</example>
<para>The conditions under which <command>pmdaMain</command> will return are either unexpected error conditions (often from failed initialisation, which would already have been logged), or when PMCD closes the connection to the PMDA. In all cases the correct action to take is simply to exit cleanly, possibly after any final cleanup the PMDA may need to perform.</para>
</section>
</section>
<section id="id5194770">

<title>Testing and Debugging a PMDA</title>
<para>Ensuring the correct operation of a PMDA can be difficult, because the responsibility of providing metrics to the requesting PMCD process and simultaneously retrieving values from the target domain requires nearly real-time communication with two modules beyond the PMDA's control. Some tools are available to assist in this important task. <indexterm id="IG31340177226"><primary>testing and debugging</primary></indexterm> <indexterm id="IG31340177227"><primary> debugging and testing</primary></indexterm></para>
<section id="id5194836">

<title>Overview</title>
<para><indexterm id="IG31340177228"><primary>dbx man page</primary></indexterm>Thoroughly testing a PMDA with PMCD is difficult, although testing a daemon PMDA is marginally simpler than testing a DSO PMDA. If a DSO PMDA exits, PMCD also exits because they share a single address space and control thread.</para>
<para>The difficulty in using PMCD to test a daemon PMDA results from PMCD requiring timely replies from the PMDA in response to request PDUs. Although a timeout period can be set in <filename>${PCP_PMCDOPTIONS_PATH}</filename>, attaching a debugger (such as <command>gdb</command>) to the PMDA process might cause an already running PMCD to close its connection with the PMDA. If timeouts are disabled, PMCD could wait forever to connect with the PMDA.</para>
<para>If you suspect a PMDA has been terminated due to a timeout failure, check the PMCD log file, usually <filename>${PCP_LOG_DIR}/pmcd/pmcd.log</filename>.</para>
<para><indexterm id="IG31340177229"><primary>dbpmda man page</primary></indexterm>A more robust way of testing a PMDA is to use the <command>dbpmda</command> tool, which is similar to PMCD except that <command>dbpmda</command> provides complete control over the PDUs that are sent to the PMDA, and there are no time limits--it is essentially an interactive debugger for exercising a PMDA. See the <command>dbpmda(3)</command> man page for details.</para>
<para><indexterm id="IG31340177230"><primary>pmdbg man page</primary></indexterm>In addition, careful use of PCP debugging flags can produce useful information concerning a PMDA's behavior; see the <command>PMAPI(3)</command> and <command>pmdbg(1)</command> man pages for a discussion of the PCP debugging and tracing framework.</para>
</section>
<section id="id5195016">

<title>Debugging Information</title>
<para><indexterm id="IG31340177231"><primary>pmdbg man page</primary></indexterm><indexterm id="IG31340177232"><primary>debugging flags</primary><see>flags</see></indexterm><indexterm id="IG31340177233"><primary>flags</primary><secondary>debugging</secondary></indexterm>You can activate  debugging flags in PMCD and most other PCP tools with the <literal>-D</literal> command-line option. Supported flags can be listed with the <command>pmdbg</command> command; see the <command>pmdbg(1)</command> man page. Setting the debug flag for PMCD in <filename>${PCP_PMCDOPTIONS_PATH}</filename> might generate too much information to be useful, especially if there are other clients and PMDAs connected to the PMCD process.</para>
<para>The PMCD debugging flag can also be changed dynamically by storing a new value into the metric <literal>pmcd.control.debug</literal>:</para>
<programlisting># <emphasis role="bold">pmstore pmcd.control.debug 5</emphasis></programlisting>
<para>Most of the <filename>pcp_pmda</filename> library functions log additional information if the <literal>DBG_TRACE_LIBPMDA</literal> flag is set within the PMDA; see the <command>PMDA(3)</command> man page. The command-line argument <command>-D</command> is trapped by <command>pmdaGetOptions</command> to set the global debugging control variable <literal>pmDebug</literal>. Adding tests within the PMDA for the <literal>DBG_TRACE_APPL0</literal>, <literal>DBG_TRACE_APPL1</literal>, and <literal>DBG_TRACE_APPL2</literal> trace flags permits different levels of information to be logged to the PMDA's log file.</para>
<para>All diagnostic, debugging, and tracing output from a PMDA should be written to the standard error stream. By convention, all debugging information is enclosed by preprocessor <literal>#ifdef</literal> <command>PCP_DEBUG</command> statements so that they can be compiled out of the program at a later stage, if required, although this is rarely done in practice.</para>
<para><indexterm id="IG31340177234"><primary>simple_store function</primary></indexterm>Adding this segment of code to the <command>simple_store</command> metric causes a timestamped log message to be sent to the current log file whenever <command>pmstore</command> attempts to change <literal>simple.numfetch</literal> and <command>pmDebug</command> has the <literal>DBG_TRACE_APPL0</literal> flag set as shown in <xref linkend="Z976060060sdc"/>:</para>
<example id="Z976060060sdc">
<title><literal>simple.numfetch</literal> in the Simple PMDA</title>
<programlisting>   case 0: /* simple.numfetch */ 
       	x
        val = vsp-&gt;vlist[0].value.lval; 
        if (val &lt; 0) { 
            sts = PM_ERR_SIGN; 
            val = 0; 
        } 
#ifdef PCP_DEBUG 
        if (pmDebug &amp; DBG_TRACE_APPL0) { 
            __pmNotifyErr(LOG_DEBUG,
                  &quot;simple: %d stored into numfetch&quot;, val); 
        } 
#endif 
        numfetch = val; 
        break;</programlisting>
</example>
<para><indexterm id="IG31340177235"><primary>pmstore function</primary></indexterm>For a description of <command>pmstore</command>, see the <command>pmstore(1)</command> man page.</para>
</section>
<section id="id5195283">

<title><command>dbpmda</command> Debug Utility</title>
<para><indexterm id="IG31340177236"><primary>dbpmda man page</primary></indexterm>The <command>dbpmda</command> utility provides a simple interface to the PDU communication protocol. It allows daemon and DSO PMDAs to be tested with most request types, while the PMDA process may be monitored with a debugger, tracing utilities, and other diagnostic tools. The <command>dbpmda(1)</command> man page contains a sample session with the <filename>simple</filename> PMDA.</para>
</section>
</section>
<section id="id5195340">

<title>Integration of a PMDA</title>
<para><indexterm id="IG31340177237"><primary>PMDA</primary><secondary>integration</secondary></indexterm>Several steps are required to install (or remove) a PMDA from a production PMCD environment without affecting the operation of other PMDAs or related visualization and logging tools.<indexterm id="IG31340177238"><primary>integrating a PMDA</primary></indexterm></para>
<para>The PMDA typically would have its own directory below <filename>${PCP_PMDAS_DIR}</filename> into which several files would be installed. In the description in <xref linkend="LE55181-PARENT"/>, the PMDA of interest is assumed to be known by the name <filename>newbie</filename>, hence the PMDA directory would be <filename>${PCP_PMDAS_DIR}/newbie</filename>.</para>
<note><para>Any installation or removal of a PMDA involves updating files and directories that are typically well protected. Hence the procedures described in this section must be executed as the superuser.</para>
</note>
<section id="LE55181-PARENT">

<title>Installing a PMDA</title>
<para>A PMDA is fully installed when these tasks are completed:</para>
<itemizedlist>
<listitem><para><indexterm id="IG31340177239"><primary>help text</primary><secondary>location</secondary></indexterm>Help text has been installed in a place where the PMDA can find it, usually in the PMDA directory <filename>${PCP_PMDAS_DIR}/newbie</filename>.</para>
</listitem>
<listitem><para>The name space has been updated in the <filename>${PCP_VAR_DIR}/pmns</filename> directory.</para>
</listitem>
<listitem><para>The PMDA binary has been installed, usually in the directory <filename>${PCP_PMDAS_DIR}/newbie</filename>.</para>
</listitem>
<listitem><para>The <filename>${PCP_PMCDCONF_PATH}</filename> file has been updated.</para>
</listitem>
<listitem><para>The PMCD process has been restarted or notified (with a <literal>SIGHUP</literal> signal) that the new PMDA exists.</para>
</listitem></itemizedlist>
<para>The <filename>Makefile</filename> should include an <literal>install</literal> target to compile and link the PMDA (as a DSO, or a daemon or both) in the PMDA directory. The <literal>clobber</literal> target should remove any files created as a by-product of the <literal>install</literal> target.<indexterm id="IG31340177240"><primary>PMDA</primary><secondary>Install script</secondary></indexterm></para>
<para>You may wish to use <filename>${PCP_PMDAS_DIR}/simple/Makefile</filename> as a template for constructing a new PMDA <filename>Makefile</filename>; changing the assignment of <literal>IAM</literal> from <literal>simple</literal> to <literal>newbie</literal> would account for most of the required changes.</para>
<para>The <filename>Install</filename> script should make use of the generic procedures defined in the script <filename>${PCP_SHARE_DIR}/lib/pmdaproc.sh</filename>, and may be as straightforward as the one used for the trivial PMDA, shown in <xref linkend="Z976309325sdc"/>:</para>
<example id="Z976309325sdc">
<title><filename>Install</filename> Script for the Trivial PMDA</title>
<programlisting>. ${PCP_DIR}/etc/pcp.env
. ${PCP_SHARE_DIR}/lib/pmdaproc.sh

iam=trivial
pmda_interface=2
forced_restart=false

pmdaSetup
pmdainstall
exit 0</programlisting>
</example>
<para>The variables, shown in <xref linkend="id5195779"/>, may be assigned values to modify the behavior of the <literal>pmdaSetup</literal> and <literal>pmdainstall</literal> procedures from <filename>${PCP_SHARE_DIR}/lib/pmdaproc.sh</filename>.</para>
<table id="id5195779" frame="topbot" pgwide="wide">

<title>Variables to Control Behavior of Generic <filename>pmdaproc.sh</filename> Procedures</title>
<tgroup cols="3" colsep="0" rowsep="0">
<colspec colwidth="113*"/>
<colspec colwidth="213*"/>
<colspec colwidth="67*"/>
<thead>
<row rowsep="1" valign="top"><entry align="left" valign="bottom"><para>Shell Variable</para></entry><entry align="left" valign="bottom"><para>Use</para></entry><entry align="left" valign="bottom"><para>Default</para></entry></row></thead>
<tbody>
<row valign="top">
<entry align="left" valign="top"><para><literal>$iam</literal></para></entry>
<entry align="left" valign="top"><para>Name of the PMDA; assignment to this variable is mandatory.</para><para>Example: <literal>iam=newbie</literal></para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$dso_opt</literal></para></entry>
<entry align="left" valign="top"><para>Can this PMDA be installed as a DSO?</para></entry>
<entry align="left" valign="top"><para><literal>false</literal></para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$daemon_opt</literal></para></entry>
<entry align="left" valign="top"><para>Can this PMDA be installed as a daemon?</para></entry>
<entry align="left" valign="top"><para><literal>true</literal></para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$perl_opt</literal></para></entry>
<entry align="left" valign="top"><para>Is this PMDA a perl script?</para></entry>
<entry align="left" valign="top"><para><literal>false</literal></para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$python_opt</literal></para></entry>
<entry align="left" valign="top"><para>Is this PMDA a python script?</para></entry>
<entry align="left" valign="top"><para><literal>false</literal></para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$forced_restart</literal></para></entry>
<entry align="left" valign="top"><para>Must this PMDA run as <literal>root</literal> or some other non-default user? (requires PMCD restart)</para></entry>
<entry align="left" valign="top"><para><literal>true</literal></para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$pipe_opt</literal></para></entry>
<entry align="left" valign="top"><para>If installed as a daemon PMDA, is the default IPC via pipes?</para></entry>
<entry align="left" valign="top"><para><literal>true</literal></para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$socket_opt</literal></para></entry>
<entry align="left" valign="top"><para>If installed as a daemon PMDA, is the default IPC via an Internet socket?</para></entry>
<entry align="left" valign="top"><para><literal>false</literal></para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$socket_inet_def</literal></para></entry>
<entry align="left" valign="top"><para>If installed as a daemon PMDA, and the IPC method uses an Internet socket, the default port number.</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$ipc_prot</literal></para></entry>
<entry align="left" valign="top"><para>IPC style for PDU exchanges involving a daemon PMDA; <literal>binary</literal> or <literal>text</literal>.</para></entry>
<entry align="left" valign="top"><para><literal>binary</literal></para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$check_delay</literal></para></entry>
<entry align="left" valign="top"><para>Delay in seconds between installing PMDA and checking if metrics are available.</para></entry>
<entry align="left" valign="top"><para><literal>3</literal></para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$args</literal></para></entry>
<entry align="left" valign="top"><para>Additional command-line arguments passed to a daemon PMDA.</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$pmda_interface</literal></para></entry>
<entry align="left" valign="top"><para>Version of the <filename>libpcp_pmda</filename> library required, used to determine the version for generating help text files.</para></entry>
<entry align="left" valign="top"><para><literal>1</literal></para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$pmns_source</literal></para></entry>
<entry align="left" valign="top"><para>The name of the PMNS file (by default relative to the PMDA directory).</para></entry>
<entry align="left" valign="top"><para><literal>pmns</literal></para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$pmns_name</literal></para></entry>
<entry align="left" valign="top"><para>First-level name for this PMDA's metrics in the PMNS.</para></entry>
<entry align="left" valign="top"><para><literal>$iam</literal></para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$help_source</literal></para></entry>
<entry align="left" valign="top"><para>The name of the help file (by default relative to the PMDA directory).</para></entry>
<entry align="left" valign="top"><para><literal>help</literal></para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$pmda_name</literal></para></entry>
<entry align="left" valign="top"><para>The name of the executable for a daemon PMDA.</para></entry>
<entry align="left" valign="top"><para><literal>pmda$iam</literal></para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$dso_name</literal></para></entry>
<entry align="left" valign="top"><para>The name of the shared library for a DSO PMDA.</para></entry>
<entry align="left" valign="top"><para><literal>pmda$iam.$dso_suffix</literal></para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$dso_entry</literal></para></entry>
<entry align="left" valign="top"><para>The name of the initialization function for a DSO PMDA.</para></entry>
<entry align="left" valign="top"><para><literal>${iam}_init</literal></para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$domain</literal></para></entry>
<entry align="left" valign="top"><para>The numerical PMDA domain number (from <filename>domain.h</filename>).</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>$SYMDOM</literal></para></entry>
<entry align="left" valign="top"><para>The symbolic name of the PMDA domain number (from <filename>domain.h</filename>).</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row></tbody></tgroup></table>
<para>In addition, the variables <literal>do_pmda</literal> and <literal>do_check</literal> will be set to reflect the intention to install the PMDA (as opposed to install just the PMNS) and to check the availability of the metrics once the PMDA is installed. By default, each variable is <literal>true</literal>; however, the command-line options <literal>-N</literal> and <literal>-Q</literal> to <filename>Install</filename> may be used to set the variables to <literal>false</literal>, as follows: <literal>do_pmda</literal> (<command>-N</command>) and <literal>do_check</literal> (<literal>-N</literal> or <literal>-Q</literal>).</para>
<para>The variables may also have their assignments changed by the user's response to the common prompt as shown in <xref linkend="Z976309844sdc"/>:</para>
<example id="Z976309844sdc">
<title>Changing Variable Assignments</title>
<programlisting>You will need to choose an appropriate configuration for installation 
of the ... Performance Metrics Domain Agent (PMDA).
  collector   collect performance statistics on this system
  monitor     allow this system to monitor local and/or remote systems
  both        collector and monitor configuration for this system</programlisting>
</example>
<para>Obviously, for anything but the most trivial PMDA, after calling the <filename>pmdaSetup</filename> procedure, the <filename>Install</filename> script should also prompt for any PMDA-specific parameters, which are typically accumulated in the <replaceable>args</replaceable> variable and used by the <literal>pmdainstall</literal> procedure.</para>
<para>The detailed operation of the <filename>pmdainstall</filename> procedure involves the following tasks:</para>
<itemizedlist>
<listitem><para>Using default assignments, and interaction where ambiguity exists, determine the PMDA type (DSO or daemon) and the IPC parameters, if any.</para>
</listitem>
<listitem><para>Copy the <filename>$pmns_source</filename> file, replacing symbolic references to <literal>SYMDOM</literal> by the desired numeric domain number from <literal>domain.</literal></para>
</listitem>
<listitem><para>Merge the PMDA's name space into the PCP name space at the non-leaf node identified by <filename>$pmns_name</filename>.</para>
</listitem>
<listitem><para>If any <command>pmchart</command> views can be found (files with names ending in &#x201C;.pmchart&#x201D;), copy these to the standard directory (<filename>${PCP_VAR_DIR}/config/pmchart</filename>) with the &#x201C;.pmchart&#x201D; suffix removed.</para>
</listitem>
<listitem><para><indexterm id="IG31340177241"><primary>help text</primary><secondary>creation</secondary></indexterm>Create new help files from <literal>$help_source</literal> after replacing symbolic references to <literal>SYMDOM</literal> by the desired numeric domain number from <literal>domain</literal>.</para>
</listitem>
<listitem><para>Terminate the old daemon PMDA, if any.</para>
</listitem>
<listitem><para>Use the <filename>Makefile</filename> to build the appropriate executables.</para>
</listitem>
<listitem><para>Add the PMDA specification to PMCD's configuration file (<filename>${PCP_PMCDCONF_PATH}</filename>).</para>
</listitem>
<listitem><para>Notify PMCD. To minimize the impact on the services PMCD provides, sending a <literal>SIGHUP</literal> to PMCD forces it to reread the configuration file and start, restart, or remove any PMDAs that have changed since the file was last read. However, if the newly installed PMDA must run using a different privilege level to PMCD then PMCD must be restarted. This is because PMCD runs unprivileged after initially starting the PMDAs.</para>
</listitem>
<listitem><para>Check that the metrics from the new PMDA are available.</para>
</listitem></itemizedlist>
<para>There are some PMDA changes that may trick PMCD into thinking nothing has changed, and not restarting the PMDA. Most notable are changes to the PMDA executable. In these cases, you may need to explicitly remove the PMDA as described in <xref linkend="Z976310185sdc"/>, or more drastically, restart PMCD as follows: <indexterm id="IG31340177242"><primary>restarting pmcd</primary></indexterm></para>
<literallayout class="monospaced"># <userinput>${PCP_RC_DIR}/pcp start</userinput></literallayout>
<para><indexterm id="IG31340177243"><primary>examples</primary><secondary>Install script</secondary></indexterm>The files <filename>${PCP_PMDAS_DIR}/*/Install</filename> provide a wealth of examples that may be used to construct a new PMDA <filename>Install</filename> script.</para>
</section>
<section id="id5197100">

<title>Upgrading a PMNS to Include Metrics from a New PMDA</title>
<para><indexterm id="IG31340177244"><primary>new PMDA</primary></indexterm><indexterm id="IG31340177245"><primary>PMNS</primary><secondary>upgrade</secondary></indexterm><indexterm id="IG31340177246"><primary>PMDA</primary><secondary> Install script</secondary></indexterm>When invoked with a <literal>-N</literal> command-line option, the PMDA <filename>Install</filename> script may be used to update the PMNS without installing the PMDA. This functionality is rarely, if ever, used in modern versions of PCP, but allows one to populate the local PMNS with the names of the performance metrics from a PMDA installed on a remote host. The <literal>-N</literal> option can also install <command>pmchart</command> views useful on a monitoring system, although this also is rarely used now with each platforms package management tools handling this task.</para>
</section>
<section id="Z976310185sdc">

<title>Removing a PMDA</title>
<para><indexterm id="IG31340177247"><primary>PMDA</primary><secondary>removal</secondary></indexterm>The simplest way to stop a PMDA from running, apart from killing the process, is to remove the entry from <filename>${PCP_PMCDCONF_PATH}</filename> and signal PMCD (with <literal>SIGHUP</literal>) to reread its configuration file. To completely remove a PMDA requires the reverse process of the installation, including an update of the Performance Metrics Name Space (PMNS).</para>
<para><indexterm id="IG31340177248"><primary>removal script</primary></indexterm>This typically involves a <filename>Remove</filename> script in the PMDA directory that uses the same common procedures as the <filename>Install</filename> script described <xref linkend="LE55181-PARENT"/>.</para>
<para><indexterm id="IG31340177249"><primary>examples</primary><secondary>Remove script</secondary></indexterm>The <filename>${PCP_PMDAS_DIR}/*/Remove</filename> files provide a wealth of examples that may be used to construct a new PMDA <filename>Remove</filename> script.</para>
</section>
<section id="id5197301">

<title>Configuring PCP Tools</title>
<para><indexterm id="IG31340177250"><primary>tool configuration</primary></indexterm><indexterm id="IG31340177251"><primary>configuration</primary></indexterm>Most PCP tools have their own configuration file format for specifying which metrics to view or to log. By using canned configuration files that monitor key metrics of the new PMDA, users can quickly see the performance of the target system, as characterized by key metrics in the new PMDA. </para>
<para>Any configuration files that are created should be kept with the PMDA and installed into the appropriate directories when the PMDA is installed.</para>
<para>As with all PCP customization, some of the most valuable tools can be created by defining views, scenes, and control-panel layouts that combine related performance metrics from multiple PMDAs or multiple hosts.</para>
<para><indexterm id="IG31340177254"><primary>pmlogger command</primary></indexterm><indexterm id="IG31340177255"><primary>pmlogconf command</primary></indexterm>Metrics suitable for on-going logging can be specified in templated <command>pmlogger</command> configuration files for <command>pmlogconf</command> to automatically add to the <command>pmlogger_daily</command> recorded set; see the <command>pmlogger(1)</command>, <command>pmlogconf(1)</command> and <command>pmlogger_daily(1)</command> man pages.</para>
<para><indexterm id="IG31340177252"><primary>pmie command</primary></indexterm><indexterm id="IG31340177253"><primary>pmieconf command</primary></indexterm>Parameterized alarm configurations can be created using the <command>pmieconf</command> facilities; see the <command>pmieconf(1)</command> and <command>pmie(1)</command> man pages.</para>
</section>
</section>
</chapter>


<chapter id="LE97135-PARENT">

<title>PMAPI--The Performance Metrics API</title>
<para><indexterm id="IG31340177256"><primary>PMAPI</primary><secondary>description</secondary></indexterm><indexterm id="IG31340177257"><primary>Application Programming Interface</primary></indexterm>This chapter describes the Performance Metrics Application Programming Interface (PMAPI) provided with Performance Co-Pilot (PCP).</para>
<para><indexterm id="IG31340177258"><primary>archive logs</primary><secondary>performance data</secondary></indexterm>The PMAPI is a set of functions and data structure definitions that allow client applications to access performance data from one or more Performance Metrics Collection Daemons (PMCDs) or from PCP archive logs. The PCP utilities are all written using the PMAPI.</para>
<para>The most common use of PCP includes running performance monitoring utilities on a workstation (the monitoring system) while performance data is retrieved from one or more remote collector systems by a number of PCP processes. These processes execute on both the monitoring system and the collector systems. The collector systems are typically servers, and are the targets for the performance investigations.</para>
<para>In the development of the PMAPI the most important question has been, &#x201C;How easily and quickly will this API enable the user to build new performance tools, or exploit existing tools for newly available performance metrics?&#x201D; The PMAPI and the standard tools that use the PMAPI have enjoyed a symbiotic evolution throughout the development of PCP.</para>
<para>It will be convenient to differentiate between code that uses the PMAPI and code that implements the services of the PMAPI. The former will be termed &#x201C;above the PMAPI&#x201D; and the latter &#x201C;below the PMAPI.&#x201D;</para>
<section id="LE87626-PARENT">

<title>Naming and Identifying Performance Metrics</title>
<para><indexterm id="IG31340177259"><primary>performance metrics</primary><see>metrics</see></indexterm><indexterm id="IG31340177260"><primary>metrics</primary><secondary>API</secondary></indexterm><indexterm id="IG31340177261"><primary>PMAPI</primary><secondary>identifying metrics</secondary></indexterm>Across all of the supported performance metric domains, there are a large number of performance metrics. Each metric has its own description, format, and semantics. PCP presents a uniform interface to these metrics above the PMAPI, independent of the source of the underlying metric data. For example, the performance metric <literal>hinv.physmem</literal> has a single 32-bit unsigned integer value, representing the number of megabytes of physical memory in the system, while the performance metric <literal>disk.dev.total</literal> has one 32-bit unsigned
integer value per disk spindle, representing the cumulative count of I/O operations involving each associated disk spindle. These concepts are described in greater detail in <xref linkend="LE97285-PARENT"/>.</para>
<para>For brevity and efficiency, internally PCP avoids using names for performance metrics, and instead uses an identification scheme that unambiguously associates a single integer with each known performance metric. This integer is known as a Performance Metric Identifier, or PMID. For functions using the PMAPI, a PMID is defined and manipulated with the typedef <literal>pmID</literal>.</para>
<para>Below the PMAPI, the integer value of the PMID has an internal structure that reflects the details of the PMCD and PMDA architecture, as described in <xref linkend="LE98565-PARENT"/>.</para>
<para>Above the PMAPI, a Performance Metrics Name Space (PMNS) is used to provide a hierarchic classification of external metric names, and a one-to-one mapping of external names to internal PMIDs. A more detailed description of the PMNS can be found in the <citetitle>Performance Co-Pilot User's and Administrator's Guide</citetitle>.</para>
<para>The default PMNS comes from the performance metrics source, either a PMCD process or a PCP archive. This PMNS always reflects the available metrics from the performance metrics source</para>
</section>
<section id="id5197718">

<title>Performance Metric Instances</title>
<para><indexterm id="IG31340177262"><primary>PMAPI</primary><seealso>metrics</seealso></indexterm>When performance metric values are returned across the PMAPI to a requesting application, there may be more than one value for a particular metric; for example, independent counts for each CPU, or each process, or each disk, or each system call type, and so on. This multiplicity of values is not enumerated in the Name Space, but rather when performance metrics are delivered across the PMAPI.</para>
<para>The notion of <literal>metric instances</literal> is really a number of related concepts, as follows:</para>
<itemizedlist>
<listitem><para>A particular performance metric may have a set of associated values or instances.</para>
</listitem>
<listitem><para>The instances are differentiated by an instance identifier.</para>
</listitem>
<listitem><para>An instance identifier has an internal encoding (an integer value) and an external encoding (a corresponding external name or label).</para>
</listitem>
<listitem><para>The set of all possible instance identifiers associated with a performance metric on a particular host constitutes an <firstterm>instance domain</firstterm>.</para>
</listitem>
<listitem><para>Several performance metrics may share the same instance domain.</para>
</listitem></itemizedlist>
<para>Consider <xref linkend="Z976548024sdc"/>:</para>
<example id="Z976548024sdc">
<title>Metrics Sharing the Same Instance Domain</title>
<programlisting><userinput>$ pminfo -f filesys.free</userinput>

filesys.free
    inst [1 or &#x201C;/dev/disk0&#x201D;] value 1803
    inst [2 or &#x201C;/dev/disk1&#x201D;] value 22140
    inst [3 or &#x201C;/dev/disk2&#x201D;] value 157938</programlisting>
</example>
<para>The metric <literal>filesys.free</literal> has three values, currently 1803, 22140, and 157938. These values are respectively associated with the instances identified by the internal identifiers 1, 2 and 3, and the external identifiers <filename>/dev/disk0</filename>, <filename>/dev/disk1</filename>, and <filename>/dev/disk2</filename>. These instances form an instance domain that is shared by the performance metrics <literal>filesys.capacity</literal>, <literal>filesys.used</literal>, <literal>filesys.free</literal>, <literal>filesys.mountdir</literal>, and so on.</para>
<para>Each performance metric is associated with an instance domain, while each instance domain may be associated with many performance metrics. Each instance domain is identified by a unique value, as defined by the following <literal>typedef</literal> declaration:</para>
<literallayout class="monospaced"><literal>typedef unsigned long pmInDom;</literal></literallayout>
<para><indexterm id="IG31340177263"><primary>PM_INDOM_NULL instance domain</primary><secondary>description</secondary></indexterm>The special instance domain <literal>PM_INDOM_NULL</literal> is reserved to indicate that the metric has a single value (a singular instance domain). For example, the performance metric <literal>mem.freemem</literal> always has exactly one value. Note that this is semantically different to a performance metric like <literal>kernel.percpu.cpu.sys</literal> that has a non-singular instance domain, but may have only one value available; for example, on a system with a single processor.</para>
<para><indexterm id="IG31340177264"><primary>PM_IN_NULL instance identifier</primary></indexterm>In the results returned above the PMAPI, each individual instance within an instance domain is identified by an internal integer instance identifier. The special instance identifier <literal>PM_IN_NULL</literal> is reserved for the single value in a singular instance domain. Performance metric values are delivered across the PMAPI as a set of instance identifier and value pairs.</para>
<para>The instance domain of a metric may change with time. For example, a machine may be shut down, have several disks added, and be rebooted. All performance metrics associated with the instance domain of disk devices would contain additional values after the reboot. The difficult issue of transient performance metrics means that repeated requests for the same PMID may return different numbers of values, or some changes in the particular instance identifiers returned. This means applications need to be aware that metric instantiation is guaranteed to be valid only at the time of collection.</para>
<note><para>Some instance domains are more dynamic than others. For example, consider the instance domains behind the performance metrics <literal>proc.memory.rss</literal> (one instance per process), <literal>swap.free</literal> (one instance per swap partition) and <literal>kernel.percpu.cpu.intr</literal> (one instance per CPU).</para>
</note>
</section>
<section id="id5198049">

<title>Current PMAPI Context</title>
<para><indexterm id="IG31340177265"><primary>PMAPI</primary><secondary>current context</secondary></indexterm>When performance metrics are retrieved across the PMAPI, they are delivered in the context of a particular source of metrics, a point in time, and a profile of desired instances. This means that the application making the request has already negotiated across the PMAPI to establish the context in which the request should be executed.</para>
<para><indexterm id="IG31340177266"><primary>archive logs</primary><secondary>performance data</secondary></indexterm>A metric's source may be the current performance data from a particular host (a live or real-time source), or an archive log of performance data collected by <command>pmlogger</command> at some remote host or earlier time (a retrospective or archive source). The metric's source is specified when the PMAPI context is created by calling the <command>pmNewContext</command> function. This function returns an opaque handle which can be used to identify the context.</para>
<para><indexterm id="IG31340177267"><primary>collection time</primary></indexterm>The collection time for a performance metric is always the current time of day for a real-time source, or current position for an archive source. For archives, the collection time may be set to an arbitrary time within the bounds of the archive log by calling the <command>pmSetMode</command> function.</para>
<para>The last component of a PMAPI context is an instance profile that may be used to control which particular instances from an instance domain should be retrieved. When a new PMAPI context is created, the initial state expresses an interest in all possible instances, to be collected at the current time. The instance profile can be manipulated using the <command>pmAddProfile</command> and <command>pmDelProfile</command> functions.</para>
<para>The current context can be changed by passing a context handle to <command>pmUseContext</command>. If a live context connection fails, the <command>pmReconnectContext</command> function can be used to attempt to reconnect it.</para>
</section>
<section id="LE11914-PARENT">

<title>Performance Metric Descriptions</title>
<para><indexterm id="IG31340177268"><primary>pmDesc structure</primary></indexterm>For each defined performance metric, there is associated metadata encoded in a performance metric description (<command>pmDesc</command> structure) that describes the format and semantics of the performance metric. The <command>pmDesc</command> structure, in <xref linkend="Z976548425sdc"/>, provides all of the information required to interpret and manipulate a performance metric through the PMAPI. It has the following declaration:</para>
<example id="Z976548425sdc">
<title><command>pmDesc</command> Structure</title>
<programlisting>/* Performance Metric Descriptor */
typedef struct {
    pmID    pmid;   /* unique identifier */
    int     type;   /* base data type (see below) */
    pmInDom indom;  /* instance domain */
    int     sem;    /* semantics of value (see below) */
    pmUnits units;  /* dimension and units (see below) */
} pmDesc;</programlisting>
</example>
<para>The <literal>type</literal> field in the <filename>pmDesc</filename> structure describes various encodings of a metric's value. Its value will be one of the following constants:</para>
<programlisting>/* pmDesc.type - data type of metric values */
#define PM_TYPE_NOSUPPORT -1   /* not in this version */
#define PM_TYPE_32        0    /* 32-bit signed integer */
#define PM_TYPE_U32       1    /* 32-bit unsigned integer */
#define PM_TYPE_64        2    /* 64-bit signed integer */
#define PM_TYPE_U64       3    /* 64-bit unsigned integer */
#define PM_TYPE_FLOAT     4    /* 32-bit floating point */
#define PM_TYPE_DOUBLE    5    /* 64-bit floating point */
#define PM_TYPE_STRING    6    /* array of char */
#define PM_TYPE_AGGREGATE 7    /* arbitrary binary data */
#define PM_TYPE_AGGREGATE_STATIC 8 /* static pointer to aggregate */
#define PM_TYPE_EVENT     9    /* packed pmEventArray */
#define PM_TYPE_UNKNOWN   255  /* used in pmValueBlock not pmDesc */</programlisting>
<para><indexterm id="IG31340177269"><primary>PM_TYPE_STRING type</primary></indexterm>By convention <literal>PM_TYPE_STRING</literal> is interpreted as a classic C-style null byte terminated string.</para>
<para><indexterm id="IG31340177269nat"><primary>PM_TYPE_EVENT type</primary></indexterm>Event records are encoded as a packed array of strongly-typed, well-defined records within a <literal>pmResult</literal> structure, using a container metric with a value of type <literal>PM_TYPE_EVENT</literal>.</para>
<para><indexterm id="IG31340177270"><primary>PM_TYPE_AGGREGATE type</primary></indexterm>If the value of a performance metric is of type <literal>PM_TYPE_STRING</literal>, <literal>PM_TYPE_AGGREGATE</literal>, <literal>PM_TYPE_AGGREGATE_STATIC</literal>, or <literal>PM_TYPE_EVENT</literal>, the interpretation of that value is unknown to many PCP components. In the case of the aggregate types, the application using the value and the Performance Metrics Domain Agent (PMDA) providing the value must have some common understanding about how the value is structured and interpreted. Strings can be manipulated using the standard C libraries. Event records contain timestamps, event flags and event parameters, and the PMAPI provides support for unpacking an event record - see the <command>pmUnpackEventRecords(3)</command> man page for details. Further discussion on event metrics and event records can be found in <xref linkend="id5199202"/>.</para>
<para><indexterm id="IG31340177271"><primary>PM_TYPE_NOSUPPORT value</primary></indexterm><literal>PM_TYPE_NOSUPPORT</literal> indicates that the PCP collection framework knows about the metric, but the corresponding service or application is either not configured or is at a revision level that does not provide support for this performance metric.</para>
<para>The semantics of the performance metric is described by the <literal>sem</literal> field of a <filename>pmDesc</filename> structure and uses the following constants:</para>
<programlisting>/* pmDesc.sem - semantics of metric values */
#define PM_SEM_COUNTER  1  /* cumulative count, monotonic increasing */
#define PM_SEM_INSTANT  3  /* instantaneous value continuous domain */
#define PM_SEM_DISCRETE 4  /* instantaneous value discrete domain */</programlisting>
<para><indexterm id="IG31340177272"><primary>dimensionality and scale</primary></indexterm><indexterm id="IG31340177273"><primary>scale and dimensionality</primary></indexterm>Each value for a performance metric is assumed to be drawn from a set of values that can be described in terms of their dimensionality and scale by a compact encoding, as follows:</para>
<itemizedlist>
<listitem><para>The dimensionality is defined by a power, or index, in each of three orthogonal dimensions: Space, Time, and Count (dimensionless). For example, I/O throughput is Space<superscript>1</superscript>.Time<superscript>-1</superscript>, while the running total of system calls is Count<superscript>1</superscript>, memory allocation is Space<superscript>1</superscript>, and average service time per event is Time<superscript>1</superscript>.Count<superscript>-1</superscript>.</para>
</listitem>
<listitem><para>In each dimension, a number of common scale values are defined that may be used to better encode ranges that might otherwise exhaust the precision of a 32-bit value. For example, a metric with dimension Space<superscript>1</superscript>.Time<superscript>-1</superscript> may have values encoded using the scale megabytes per second.</para>
</listitem></itemizedlist>
<para><indexterm id="IG31340177274"><primary>pmDesc structure</primary></indexterm>This information is encoded in the <filename>pmUnits</filename> data structure, shown in <xref linkend="Z1034792511tls"/>. It is embedded in the <filename>pmDesc</filename> structure :</para>
<para>The structures are as follows:</para>
<example id="Z1034792511tls">
<title><filename>pmUnits</filename> and <filename>pmDesc</filename> Structures</title>
<programlisting>/*
 * Encoding for the units (dimensions and
 * scale) for Performance Metric Values
 *
 * For example, a pmUnits struct of
 * { 1, -1, 0, PM_SPACE_MBYTE, PM_TIME_SEC, 0 }
 * represents Mbytes/sec, while 
 * { 0, 1, -1, 0, PM_TIME_HOUR, 6 }
 * represents hours/million-events
 */
typedef struct {
    int pad:8;
    int scaleCount:4; /* one of PM_COUNT_* below */
    int scaleTime:4;  /* one of PM_TIME_* below */
    int scaleSpace:4; /* one of PM_SPACE_* below */    
    int dimCount:4;   /* event dimension */
    int dimTime:4;    /* time dimension */
    int dimSpace:4;   /* space dimension
} pmUnits;   /* dimensional units and scale of value */
/* pmUnits.scaleSpace */
#define PM_SPACE_BYTE 0  /* bytes */
#define PM_SPACE_KBYTE 1 /* Kilobytes (1024) */
#define PM_SPACE_MBYTE 2 /* Megabytes (1024^2) */
#define PM_SPACE_GBYTE 3 /* Gigabytes (1024^3) */
#define PM_SPACE_TBYTE 4 /* Terabytes (1024^4) */
/* pmUnits.scaleTime */
#define PM_TIME_NSEC 0   /* nanoseconds */
#define PM_TIME_USEC 1   /* microseconds */
#define PM_TIME_MSEC 2   /* milliseconds */
#define PM_TIME_SEC 3    /* seconds */
#define PM_TIME_MIN 4    /* minutes */
#define PM_TIME_HOUR 5   /* hours */
/*
 * pmUnits.scaleCount (e.g. count events, syscalls,
 * interrupts, etc.) -- these are simply powers of 10,
 * and not enumerated here.
 * e.g. 6 for 10^6, or -3 for 10^-3
 */
#define PM_COUNT_ONE 0 /* 1 */</programlisting>
</example>
</section>
<section id="LE82331-PARENT">

<title>Performance Metrics Values</title>
<para><indexterm id="IG31340177275"><primary>pmFetch function</primary></indexterm><indexterm id="IG31340177276"><primary>pmStore function</primary></indexterm>An application may fetch (or store) values for a set of performance metrics, each with a set of associated instances, using a single <command>pmFetch</command> (or <command>pmStore</command>) function call. To accommodate this, values are delivered across the PMAPI in the form of a tree data structure, rooted at a <filename>pmResult</filename> structure. This encoding is illustrated in <xref linkend="id5198718"/>, and uses the component data structures in <xref linkend="Z976557818sdc"/>:</para>
<example id="Z976557818sdc">
<title><command>pmValueBlock</command> and <command>pmValue</command> Structures</title>
<programlisting>typedef struct {
    int inst;                 /* instance identifier */
    union {
        pmValueBlock *pval;   /* pointer to value-block */
        int           lval;   /* integer value insitu */
    } value;
} pmValue;</programlisting>
</example>
<para><figure id="id5198718"><title>A Structured Result for Performance Metrics from <command>pmFetch</command></title><mediaobject><imageobject><imagedata fileref="figures/pmresult.svg"/></imageobject><textobject><phrase>A Structured Result for Performance Metrics from pmFetch</phrase></textobject></mediaobject></figure></para>
<para><indexterm id="IG31340177277"><primary>internal instance identifier</primary></indexterm>The internal instance identifier is stored in the <literal>inst</literal> element. If a value for a particular metric-instance pair is a 32-bit integer (signed or unsigned), then it will be stored in the <literal>lval</literal> element. If not, the value will be in a <literal>pmValueBlock</literal> structure, as shown in <xref linkend="Z1034793414tls"/>, and will be located via <literal>pval</literal>:</para>
<para>The <literal>pmValueBlock</literal> structure is as follows:<example id="Z1034793414tls">
<title><literal>pmValueBlock</literal> Structure</title>
<programlisting>typedef struct {
    unsigned int    vlen : 24;    /* bytes for vtype/vlen + vbuf */
    unsigned int    vtype : 8;    /* value type */
    char            vbuf[1];      /* the value */
} pmValueBlock;</programlisting>
</example></para>
<para><indexterm id="IG31340177278"><primary>arrays</primary><secondary>performance metrics</secondary></indexterm>The length of the <filename>pmValueBlock</filename> (including the <literal>vtype</literal> and <literal>vlen</literal> fields) is stored in <literal>vlen</literal>. Despite the prototype declaration of <literal>vbuf</literal>, this array really accommodates <literal>vlen</literal> minus <command>sizeof</command>(<literal>vlen</literal>) bytes. The <literal>vtype</literal> field encodes the type of the value in the <literal>vbuf[]</literal> array, and is one of the <command>PM_TYPE_*</command> macros defined in <filename>&lt;pcp/pmapi.h&gt;</filename>.</para>
<para><indexterm id="IG31340177279"><primary>PM_VAL_INSITU value</primary></indexterm>A <filename>pmValueSet</filename> structure, as shown in <xref linkend="Z976549488sdc"/>, contains all of the values to be returned from <command>pmFetch</command> for a single performance metric identified by the <literal>pmid</literal> field.</para>
<example id="Z976549488sdc">
<title><filename>pmValueSet</filename> Structure</title>
<programlisting>typedef struct {
    pmID    pmid;          /* metric identifier */
    int     numval;        /* number of values */
    int     valfmt;        /* value style, insitu or ptr */
    pmValue vlist[1];      /* set of instances/values */ 
} pmValueSet;</programlisting>
</example>
<para>If positive, the <literal>numval</literal> field identifies the number of value-instance pairs in the <literal>vlist</literal> array (despite the prototype declaration of size 1). If <literal>numval</literal> is zero, there are no values available for the associated performance metric and <literal>vlist</literal>[0] is undefined. A negative value for <literal>numval</literal> indicates an error condition (see the <command>pmErrStr(3)</command> man page) and <literal>vlist</literal>[0] is undefined. The <literal>valfmt</literal> field has the value <literal>PM_VAL_INSITU</literal> to indicate that the values for the performance metrics should be located directly via the <literal>lval</literal> member of the <literal>value</literal> union embedded in the elements of <literal>vlist</literal>; otherwise, metric values are located indirectly via the <literal>pval</literal> member of the elements of <literal>vlist</literal>.</para>
<para><indexterm id="IG31340177280"><primary>pmFetch function</primary></indexterm>The <filename>pmResult</filename> structure, as shown in <xref linkend="Z976549833sdc"/>, contains a time stamp and an array of <literal>numpmid</literal> pointers to <filename>pmValueSet</filename>.</para>
<example id="Z976549833sdc">
<title><filename>pmResult</filename> Structure</title>
<programlisting>/* Result returned by pmFetch() */
typedef struct {
    struct timeval timestamp;    /* stamped by collector */
    int            numpmid;      /* number of PMIDs */
    pmValueSet     *vset[1];     /* set of value sets */
} pmResult</programlisting>
</example>
<para>There is one <literal>pmValueSet</literal> pointer per PMID, with a one-to-one correspondence to the set of requested PMIDs passed to <command>pmFetch</command>.</para>
<para>Along with the metric values, the PMAPI returns a time stamp with each <filename>pmResult</filename> that serves to identify when the performance metric values were collected. The time is in the format returned by <command>gettimeofday</command> and is typically very close to the time when the metric values were extracted from their respective domains.</para>
<note><para>There is a question of exactly when individual metrics may have been collected, especially given their origin in potentially different performance metric domains, and variability in metric updating frequency by individual PMDAs. PCP uses a pragmatic approach, in which the PMAPI implementation returns all metrics with values accurate as of the time stamp, to the maximum degree possible, and PMCD demands that all PMDAs deliver values within a small realtime window. The resulting inaccuracy is small, and the additional burden of accurate individual timestamping for each returned metric value is neither warranted nor practical (from an implementation viewpoint).</para>
</note>
<para>The PMAPI provides functions to extract, rescale, and print values from the above structures; refer to <xref linkend="LE44064-PARENT"/>.</para>
</section>
<section id="id5199202">

<title>Performance Event Metrics</title>
<para>In addition to performance metric values which are sampled by monitor tools,
PCP supports the notion of performance event metrics which occur independently to
any sampling frequency. These event metrics (PM_TYPE_EVENT) are delivered using a
novel approach which allows both sampled and event trace data to be delivered via
the same live wire protocol, the same on-disk archive format, and fundamentally
using the same PMAPI services. In other words, a monitor tool may be sample and
trace, simultaneously, using the PMAPI services discussed here.</para>
<para>Event metrics are characterised by certain key properties, distinguishing
them from the other metric types (counters, instantaneous, and discrete):</para>
<itemizedlist>
<listitem><para>Occur at times outside of any monitor tools control, and often
have a fine-grained timestamp associated with each event.
</para></listitem>
<listitem><para>Often have parameters associated with the event, which further
describe each individual event, as shown in <xref linkend="id5198719nat"/>.
</para></listitem>
<listitem><para>May occur in very rapid succession, at rates such that both the
collector and monitor sides may not be able to track all events. This property
requires the PCP protocol to support the notion of "dropped" or "missed" events.
</para></listitem>
<listitem><para>There may be inherent relationships between events, for example
the start and commit (or rollback) of a database transaction could be separate
events, linked by a common transaction identifier (which would likely also be
one of the parameters to each event).
Begin-end and parent-child relationships are relatively common, and these
properties require the PCP protocol to support the notion of "flags" that
can be associated with events.
</para></listitem>
</itemizedlist>
<para>These differences aside, the representation of event metrics within PCP shares
many aspects of the other metric types - event metrics appear in the Name Space (as
do each of the event parameters), each has an associated Performance Metric Identifier
and Descriptor, may have an instance domain associated with them, and may be recorded
by <command>pmlogger</command> for subsequent replay.</para>
<para><figure id="id5198719nat"><title>Sample <command>write(2)</command> syscall entry point encoding</title><mediaobject><imageobject><imagedata fileref="figures/syscallevent.svg"/></imageobject><textobject><phrase>Sample syscall entry point encoding</phrase></textobject></mediaobject></figure></para>
<para>Event metrics and their associated information (parameters, timestamps, flags,
and so on) are delivered to monitoring tools alongside sampled metrics as part of the
<command>pmResult</command> structure seen previously in <xref linkend="Z976549833sdc"/>.
</para>
<para>The semantics of <command>pmFetch(3)</command> specifying an event metric PMID
are such that all events observed on the collector since the previous fetch (by this
specific monitor client) are to transfered to the monitor. Each event will have the
metadata described earlier encoded with it (timestamps, flags, and so on) for each
event. The encoding of the series of events involves a compound data structure within
the <command>pmValueSet</command> associated with the event metric PMID, as illustrated
in <xref linkend="id5198719"/>.</para>
<para><figure id="id5198719"><title>Result Format for Event Performance Metrics from <command>pmFetch</command></title><mediaobject><imageobject><imagedata fileref="figures/pmevents.svg"/></imageobject><textobject><phrase>Result Format for Event Performance Metrics from pmFetch</phrase></textobject></mediaobject></figure></para>
<para>At the highest level, the "series of events" is encapsulated within a
<command>pmEventArray</command> structure, as in <xref linkend="Z976557818nat"/>:</para>
<example id="Z976557818nat">
<title><command>pmEventArray</command> and <command>pmEventRecord</command> Structures</title>
<programlisting>typedef struct {
    __pmTimeval      er_timestamp;  /* 2 x 32-bit timestamp format */
    unsigned int     er_flags;      /* event record characteristics */
    int              er_nparams;    /* number of ea_param[] entries */
    pmEventParameter er_param[1];
} pmEventRecord;

typedef struct {
    unsigned int     ea_len :  24;  /* bytes for type/len + records */
    unsigned int     ea_type : 8;   /* value type */
    int              ea_nrecords;   /* number of ea_record entries */
    pmEventRecord    ea_record[1];
} pmEventArray;</programlisting>
</example>
<para>Note that in the case of dropped events, the <command>pmEventRecord</command>
structure is used to convey the number of events dropped - <replaceable>er_flags</replaceable> is used to
indicate the presence of dropped events, and <replaceable>er_nparams</replaceable> is used to hold a count.
Unsurprisingly, the parameters (<replaceable>er_param</replaceable>) will be empty in this situation.</para>
<para>The <literal>pmEventParameter</literal> structure is as follows:</para>
<example id="Z1034793415nat">
<title><literal>pmEventParameter</literal> Structure</title>
<programlisting>typedef struct {
    pmID             ep_pmid;       /* parameter identifier */
    unsigned int     ep_type;       /* value type */
    int              ep_len;        /* bytes for type/len + vbuf */
    /* actual value (vbuf) here */
} pmEventParameter;</programlisting>
</example>

<section id="id5199203nat">

<title>Event Monitor Considerations</title>
<para><indexterm id="IG31340177280nat"><primary>pmUnpackEventRecords function</primary></indexterm>In order to simplify
the decoding of event record arrays, the PMAPI provides the <command>pmUnpackEventRecords</command> function
for monitor tools. This function is passed a pointer to a <command>pmValueSet</command> associated with an event
metric (within a <command>pmResult</command>) from a <command>pmFetch(3)</command>. For a given instance of
that event metric, it returns an array of "unpacked" <command>pmResult</command> structures for each event.</para>
<para>The control information (flags and optionally dropped events) is included as derived metrics
within each result structure. As such, these values can be queried similarly to other metrics, using
their names - <literal>event.flags</literal> and <literal>event.missed</literal>. Note that these
metrics will only exist after the first call to <command>pmUnpackEventRecords</command>.</para>
<para>An example of decoding event metrics in this way is presented in <xref linkend="Z976557819nat"/>:</para>
<example id="Z976557819nat">
<title>Unpacking Event Records from an Event Metric <literal>pmValueSet</literal></title>
<programlisting>enum { event_flags = 0, event_missed = 1 };
static char *metadata[] = { "event.flags", "event.missed" };
static pmID metapmid[2];

void dump_event(pmValueSet *vsp, int idx)
{
    pmResult    **res;
    int	        r, sts, nrecords;

    nrecords = pmUnpackEventRecords(vsp, idx, &amp;res);
    if (nrecords &lt; 0)
        fprintf(stderr, " pmUnpackEventRecords: %s\n", pmErrStr(nrecords));
    else
        printf(" %d event records\n", nrecords);

    if ((sts = pmLookupName(2, &amp;metadata, &amp;metapmid)) &lt; 0) {
        fprintf(stderr, "Event metadata error: %s\n", pmErrStr(sts));
        exit(1);
    }

    for (r = 0; r &lt; nrecords; r++)
        dump_event_record(res, r);

    if (nrecords &gt;= 0)
        pmFreeEventResult(res);
}

void dump_event_record(pmResult *res, int r)
{
    int         p;

    __pmPrintStamp(stdout, &amp;res[r]-&gt;timestamp);
    if (res[r]-&gt;numpmid == 0)
       	printf(" ==&gt; No parameters\n");
    for (p = 0; p &lt; res[r]-&gt;numpmid; p++) {
        pmValueSet  *vsp = res[r]-&gt;vset[p];

        if (vsp-&gt;numval &lt; 0) {
            int error = vsp-&gt;numval;
            printf("%s: %s\n", pmIDStr(vsp-&gt;pmid), pmErrStr(error));
        } else if (vsp-&gt;pmid == metapmid[event_flags]) {
            int flags = vsp-&gt;vlist[0].value.lval;
            printf(" flags 0x%x (%s)\n", flags, pmEventFlagsStr(flags));
        } else if (vsp-&gt;pmid == metapmid[event_missed]) {
            int count = vsp-&gt;vlist[0].value.lval;
            printf(" ==&gt; %d missed event records\n", count);
        } else {
            dump_event_record_parameters(vsp);
        }
    }
}

void dump_event_record_parameters(pmValueSet *vsp)
{
    pmDesc      desc;
    char        *name;
    int         sts, j;

    if ((sts = pmLookupDesc(vsp-&gt;pmid, &amp;desc)) &lt; 0) {
        fprintf(stderr, "pmLookupDesc: %s\n", pmErrStr(sts));
    } else
    if ((sts = pmNameID(vsp-&gt;pmid, &amp;name)) &lt; 0) {
        fprintf(stderr, "pmNameID: %s\n", pmErrStr(sts));
    } else {
        printf("parameter %s", name);
        for (j = 0; j &lt; vsp-&gt;numval; j++) {
            pmValue *vp = &amp;vsp-&gt;vlist[j];
            if (vsp-&gt;numval &gt; 1) {
                printf("[%d]", vp-&gt;inst);
                pmPrintValue(stdout, vsp-&gt;valfmt, desc.type, vp, 1);
                putchar('\n');
            }
        }
        free(name);
    }
}</programlisting>
</example>

</section>
<section id="id5199204nat">

<title>Event Collector Considerations</title>
<para>There is a feedback mechanism that is inherent in the design of the PCP
monitor-collector event metric value exchange, which protects both monitor
and collector components from becoming overrun by high frequency event arrivals.
It is important that PMDA developers are aware of this mechanism and all of
its implications.</para>
<para>Monitor tools can query new event arrival on whatever schedule they
choose. There are no guarantees that this is a fixed interval, and no way
for the PMDA to attempt to dictate this interval (nor should there be).</para>
<para>As a result, a PMDA that provides event metrics must:</para>
<itemizedlist>
<listitem><para>Track individual client connections using the per-client PMDA
extensions (PMDA_INTERFACE_5 or later).</para>
</listitem>
<listitem><para>Queue events, preferably in a memory-efficient manner, such
that each interested monitor tool (there may be more than one) is informed
of those events that arrived since their last request.</para>
</listitem>
<listitem><para>Control the memory allocated to in-memory event storage.
If monitors are requesting new events too slowly, compared to event arrival
on the collector, the "missed events" feedback mechanism must be used to
inform the monitor.
This mechanism is also part of the model by which a PMDA can fix the amount
of memory it uses. Once a fixed space is consumed, events can be dropped
from the tail of the queue for each client, provided a counter is incremented
and the client is subsequently informed.</para>
</listitem>
</itemizedlist>
<note><para>It is important that PMDAs are part of the performance solution,
and not part of the performance problem! With event metrics, this is much
more difficult to achieve than with counters or other sampled values.</para></note>

<para>There is certainly elegance to this approach for event metrics, and
the way they dovetail with other, sampled performance metrics is unique to
PCP. Notice also how the scheme naturally allows multiple monitor tools to
consume the same events, no matter what the source of events is.
The downside to this flexibility is increased complexity in the PMDA when
event metrics are used.</para>
<para>This complexity comes in the form of event queueing and memory
management, as well as per-client state tracking. Routines are available as
part of the <command>pcp_pmda</command> library to assist, however - refer
to the man page entries for <command>pmdaEventNewQueue(3)</command> and
<command>pmdaEventNewClient(3)</command> for further details.</para>

<para>One final set of helper APIs is available to PMDA developers who
incorporate event metrics.
There is a need to build the <command>pmEventArray</command> structure,
introduced in <xref linkend="Z976557818nat"/>. This can be done directly,
or using the helper routine <command>pmdaEventNewArray(3)</command>.
If the latter, simpler model is chosen, the closely related routines
<command>pmdaEventAddRecord</command>, <command>pmdaEventAddParam</command>
and <command>pmdaEventAddMissedRecord</command> would also usually be used.</para>

<para>Depending on the nature of the events being exported by a PMDA, it
can be desirable to perform <emphasis role="bold">filtering</emphasis>
of events on the collector system. This reduces the amount of event traffic
between monitor and collector systems (which may be filtered further on the
monitor system, before presenting results). Some PMDAs have had success using
the <command>pmStore(3)</command> mechanism to allow monitor tools to send a
filter to the PMDA - using either a special control metric for the store
operation, or the event metric itself. The filter sent will depend on the
event metric, but it might be a regular expression, or a tracing script,
or something else.</para>
<para>This technique has also been used to <emphasis role="bold">enable</emphasis>
and <emphasis role="bold">disable</emphasis> event tracing entirely. It is often
appropriate to make use of authentication and user credentials when providing
such a facility (PMDA_INTERFACE_6 or later).</para>
</section>
</section>
<section id="id5199203">

<title>PMAPI Programming Style and Interaction</title>
<para><indexterm id="IG31340177281"><primary>PMAPI</primary><secondary>programming style</secondary></indexterm>The following sections describe the PMAPI programming style:</para>
<itemizedlist>
<listitem><para>Variable length argument and results lists</para>
</listitem>
<listitem><para>Python specific issues</para>
</listitem>
<listitem><para>PMAPI error handling</para>
</listitem></itemizedlist>
<section id="LE37655-PARENT">

<title>Variable Length Argument and Results Lists</title>
<para><indexterm id="IG31340177282"><primary>PMAPI</primary><secondary>variable length arguments</secondary></indexterm><indexterm id="IG31340177283"><primary>arrays</primary><secondary>performance metrics</secondary></indexterm>All arguments and results involving a &#x201C;list of something&#x201D; are encoded as an array with an associated argument or function value to identify the number of elements in the array. This encoding scheme avoids both the <literal>varargs</literal> approach and sentinel-terminated lists. Where the size of a result is known at the time of a call, it is the caller's responsibility to allocate (and possibly free) the storage, and the called function assumes that the resulting argument is of an appropriate size.</para>
<para><indexterm id="IG31340177284"><primary>pmNameInDom function</primary></indexterm><indexterm id="IG31340177285"><primary>pmLookupText function</primary></indexterm><indexterm id="IG31340177286"><primary>pmGetInDom function</primary></indexterm><indexterm id="IG31340177287"><primary>pmNameID function</primary></indexterm><indexterm id="IG31340177288"><primary>pmGetChildren function</primary></indexterm><indexterm id="IG31340177289"><primary>pmFetch function</primary></indexterm><indexterm id="IG31340177290"><primary>pmFreeResult function</primary></indexterm>Where a result is of variable size and that size cannot be known in advance (for example, <command>pmGetChildren</command>, <command>pmGetInDom</command>,  <command>pmNameInDom</command>, <command>pmNameID</command>, <command>pmLookupText</command>, and <command>pmFetch</command>), the underlying implementation uses dynamic allocation through <command>malloc</command>
in the called function, with the caller responsible for subsequently calling <command>free</command> to release the storage when no longer required. In the case of the result from <command>pmFetch</command>, there is a function (<command>pmFreeResult</command>) to release the storage, due to the complexity of the data structure and the need to make multiple calls to <command>free</command> in the correct sequence. As a general rule, if the called function returns an error status, then no allocation is done, the pointer to the variable sized result is undefined, and <command>free</command> or <command>pmFreeResult</command> should not be called.</para>
</section>

<section>

<title>Python Specific Issues</title>
<para><indexterm id="python-issues"><primary>PMAPI</primary><secondary>python issues</secondary></indexterm>
A pcp client may be written in the python language by making use of the python bindings for PMAPI.  The bindings use the python ctypes module to provide an interface to the PMAPI C language data structures.  The primary imports that are needed by a client are:
<itemizedlist>
<listitem><para>cpmapi which provides access to PMAPI constants
<programlisting>import cpmapi as c_api</programlisting></para></listitem>
<listitem><para>pmapi which provides access to PMAPI functions and data structures
<programlisting>from pcp import pmapi</programlisting></para></listitem>
<listitem><para>pmErr which provides access to the python bindings exception handler
<programlisting>from pcp.pmapi import pmErr</programlisting></para></listitem>
<listitem><para>pmgui which provides access to PMAPI record mode functions
<programlisting>from pcp import ppmgui</programlisting></para></listitem>
</itemizedlist>
Creating and destroying a PMAPI context in the python environment is done by creating and destroying an object of the pmapi class.  This is done in one of two ways, either directly:
<programlisting>    context = pmapi.pmContext()</programlisting>
or by automated processing of the command line arguments (refer to the <command>pmGetOptions</command> man page for greater detail).
<programlisting>    options = pmapi.pmOptions(...)
    context = pmapi.pmContext.fromOptions(options, sys.argv)</programlisting>
Most PMAPI C functions have python equivalents with similar, although
not identical, call signatures.  Some of the python functions do not
return native python types, but instead return native C types wrapped
by the ctypes library.  In most cases these types are opaque, or
nearly so; for example <replaceable>pmid</replaceable>:
<programlisting>   pmid = context.pmLookupName("mem.freemem")
   desc = context.pmLookupDescs(pmid)
   result = context.pmFetch(pmid)
   ...</programlisting>
See the comparison of a standalone C and python client application in <xref linkend="id5212839"/>.
</para>
</section>

<section id="LE62826-PARENT">

<title>PMAPI Error Handling</title>
<para><indexterm id="IG31340177291"><primary>PMAPI</primary><secondary>error handling</secondary></indexterm>Where error conditions may arise, the functions that compose the PMAPI conform to a single, simple error notification scheme, as follows:</para>
<itemizedlist>
<listitem><para>The function returns an <literal>int</literal>. Values greater than or equal to zero indicate no error, and perhaps some positive status: for example, the number of items processed.</para>
</listitem>
<listitem><para>Values less than zero indicate an error, as determined by a global table of error conditions and messages.</para>
</listitem></itemizedlist>
<para>A PMAPI library function along the lines of <command>strerror</command> is provided to translate error conditions into error messages; see the <command>pmErrStr(3)</command> and <command>pmErrStr_r(3)</command> man pages. The error condition is returned as the function value from a previous PMAPI call; there is no global error indicator (unlike <literal>errno</literal>). This is to accommodate multi-threaded performance tools.</para>
<para>The available error codes may be displayed with the following command:</para>
<programlisting><userinput>pmerr -l</userinput></programlisting>
<para>Where possible, PMAPI routines are made as tolerant to failure as possible. In particular, routines which deal with compound data structures - results structures, multiple name lookups in one call and so on, will attempt to return all data that can be returned successfully, and errors embedded in the result where there were (partial) failures. In such cases a negative failure return code from the routine indicates catastrophic failure, otherwise success is returned and indicators for the partial failures are returned embedded in the results.</para>
</section>
</section>
<section id="id5199561">

<title>PMAPI Procedural Interface</title>
<para>The following sections describe all of the PMAPI functions that provide access to the PCP infrastructure on behalf of a client application:</para>
<itemizedlist>
<listitem><para>PMAPI Name Space services</para>
</listitem>
<listitem><para>PMAPI metric description services</para>
</listitem>
<listitem><para>PMAPI instance domain services</para>
</listitem>
<listitem><para>PMAPI context services</para>
</listitem>
<listitem><para>PMAPI timezone services</para>
</listitem>
<listitem><para>PMAPI metrics services</para>
</listitem>
<listitem><para>PMAPI record-mode services</para>
</listitem>
<listitem><para>PMAPI archive-specific services</para>
</listitem>
<listitem><para>PMAPI time control services</para>
</listitem>
<listitem><para>PMAPI ancillary support services</para>
</listitem></itemizedlist>
<section id="LE32034-PARENT">

<title>PMAPI Name Space Services</title>
<para>The functions described in this section provide Performance Metrics Application Programming Interface (PMAPI) Name Space services.</para>
<section id="id5199677">

<title><command>pmGetChildren</command> Function</title>
<literallayout class="monospaced">int pmGetChildren(const char*<replaceable>name</replaceable>, char***<replaceable>offspring</replaceable>)
<command>Python:</command>
[name1, name2...] = pmGetChildren(<replaceable>name</replaceable>)</literallayout>
<para><indexterm id="IG31340177292"><primary>PMAPI</primary><secondary>Name Space services</secondary></indexterm><indexterm id="IG31340177293"><primary>pmGetChildren function</primary></indexterm>Given a full pathname to a node in the current PMNS, as identified by <replaceable>name</replaceable>, return through <replaceable>offspring</replaceable> a list of the relative names of all the immediate descendents of <replaceable>name</replaceable> in the current PMNS. As a special case, if <replaceable>name</replaceable> is an empty string, (that is, <literal>&quot;&quot;</literal> but not <literal>NULL</literal> or <literal>(char *)0</literal>), the immediate descendents of the root node in the PMNS are returned.</para>

<para>For the python bindings a tuple containing the relative names of all the immediate descendents of <replaceable>name</replaceable> in the current PMNS is returned.
</para>
<para>Normally, <command>pmGetChildren</command> returns the number of descendent names discovered, or a value less than zero for an error. The value zero indicates that the <replaceable>name</replaceable> is valid, and associated with a leaf node in the PMNS.</para>
<para>The resulting list of pointers (<replaceable>offspring</replaceable>) and the values (relative metric names) that the pointers reference are allocated by <command>pmGetChildren</command> with a single call to <command>malloc</command>, and it is the responsibility of the caller to issue a <command>free</command><replaceable>(offspring)</replaceable> system call to release the space when it is no longer required. When the result of <command>pmGetChildren</command> is less than one, <replaceable>offspring</replaceable> is undefined (no space is allocated, and so calling <command>free</command> is counterproductive).</para>

<para>The python bindings return a tuple containing the relative names of all the immediate descendents of <replaceable>name</replaceable>, where <replaceable>name</replaceable> is a full pathname to a node in the current PMNS.</para>
</section>
<section id="id5199872">

<title><command>pmGetChildrenStatus</command> Function</title>
<literallayout class="monospaced">int pmGetChildrenStatus(const char *<replaceable>name</replaceable>, char ***<replaceable>offspring</replaceable>, int **<replaceable>status</replaceable>)
<command>Python:</command>
([name1, name2...],[status1, status2...]) = pmGetChildrenStatus(<replaceable>name</replaceable>)</literallayout>
<para><indexterm id="IG31340177294"><primary>pmGetChildren function</primary></indexterm>The <command>pmGetChildrenStatus</command> function is an extension of <command>pmGetChildren</command> that optionally returns status information about each of the descendent names.</para>
<para>Given a fully qualified pathname to a node in the current PMNS, as identified by <replaceable>name</replaceable>, <command>pmGetChildrenStatus</command> returns by means of <replaceable>offspring</replaceable> a list of the relative names of all of the immediate descendent nodes of <replaceable>name</replaceable> in the current PMNS. If <replaceable>name</replaceable> is the empty string (&#x201D;&#x201D;), it returns the immediate descendents of the root node in the PMNS.</para>
<para>If <replaceable>status</replaceable> is not NULL, then <command>pmGetChildrenStatus</command> also returns the status of each child by means of <replaceable>status</replaceable>. This refers to either a leaf node (with value <literal>PMNS_LEAF_STATUS</literal>) or a non-leaf node (with value <literal>PMNS_NONLEAF_STATUS</literal>).</para>
<para>Normally, <command>pmGetChildrenStatus</command> returns the number of descendent names discovered, or else a value less than zero to indicate an error. The value zero indicates that name is a valid metric name, being associated with a leaf node in the PMNS.</para>
<para>The resulting list of pointers (<replaceable>offspring</replaceable>) and the values (relative metric names) that the pointers reference are allocated by <command>pmGetChildrenStatus</command> with a single call to <command>malloc</command>, and it is the responsibility of the caller to <command>free</command>(<replaceable>offspring</replaceable>) to release the space when it is no longer required. The same holds true for the <replaceable>status</replaceable> array.</para>

<para>The python bindings return a tuple containing the relative names and statuses of all the immediate descendents of <replaceable>name</replaceable>, where <replaceable>name</replaceable> is a full pathname to a node in the current PMNS.</para>
</section>
<section id="id5200032">

<title><command>pmGetPMNSLocation</command> Function</title>
<literallayout class="monospaced">int pmGetPMNSLocation(void)
<command>Python:</command>
int loc = pmGetPMNSLocation()</literallayout>
<para><indexterm id="IG31340177295"><primary>pmGetPMNSLocation function</primary></indexterm>If an application needs to know where the origin of a PMNS is, <command>pmGetPMNSLocation</command> returns whether it is an archive (<filename>PMNS_ARCHIVE</filename>), a local PMNS file (<filename>PMNS_LOCAL</filename>), or a remote PMCD (<filename>PMNS_REMOTE</filename>). This information may be useful in determining an appropriate error message depending on PMNS location.</para>

<para>The python bindings return whether a PMNS is an archive <replaceable>cpmapi.PMNS_ARCHIVE</replaceable>, a local PMNS file <replaceable>cpmapi.PMNS_LOCAL</replaceable>, or a remote PMCD <replaceable>cpmapi.PMNS_REMOTE</replaceable>.  The constants are available by importing cpmapi.</para>  
</section>
<section id="id5200074">

<title><command>pmLoadNameSpace</command> Function</title>
<literallayout class="monospaced">int pmLoadNameSpace(const char *<replaceable>filename</replaceable>)
<command>Python:</command>
int <replaceable>status</replaceable> = pmLoadNameSpace(<replaceable>filename</replaceable>)</literallayout>
<para><indexterm id="IG31340177296"><primary>pmLoadNameSpace function</primary></indexterm>In the highly unusual situation that an application wants to force using a local Performance Metrics Name Space (PMNS), the application can load the PMNS using <command>pmLoadNameSpace</command>.</para>
<para>The <replaceable>filename</replaceable> argument designates the PMNS of interest. For applications that do not require a tailored Name Space, the special value <literal>PM_NS_DEFAULT</literal> may be used for <replaceable>filename</replaceable>, to force a default local PMNS to be established. Externally, a PMNS is stored in an ASCII format.</para>
<para>The python bindings load a local tailored Name Space from <replaceable>filename</replaceable>.  </para>
<note><para>Do not use this routine in monitor tools. The distributed PMNS services avoid the need for a local PMNS; so applications should <emphasis role="bold">not</emphasis> use <command>pmLoadNameSpace</command>. Without this call, the default PMNS is the one at the source of the performance metrics (PMCD or an archive).</para>
</note>
</section>
<section id="id5200342">

<title><command>pmLookupName</command> Function</title>
<literallayout class="monospaced">int pmLookupName(int <replaceable>numpmid</replaceable>, char *<replaceable>namelist</replaceable>[], pmID <replaceable>pmidlist</replaceable>[])
<command>Python:</command>
c_uint <replaceable>pmid</replaceable> [] = pmLookupName(<replaceable>"MetricName"</replaceable>)
c_uint <replaceable>pmid</replaceable> [] = pmLookupName((<replaceable>"MetricName1"</replaceable>, <replaceable>"MetricName2"</replaceable>, ...))</literallayout>
<para><indexterm id="IG31340177298"><primary>pmLookupName function</primary></indexterm>Given a list in <replaceable>namelist</replaceable> containing <replaceable>numpmid</replaceable> full pathnames for performance metrics from the current PMNS, <command>pmLookupName</command> returns the list of associated PMIDs through the <replaceable>pmidlist</replaceable> parameter. Invalid metrics names are translated to the error PMID value of <literal>PM_ID_NULL</literal>.</para>
<para>The result from <command>pmLookupName</command> is the number of names translated in the absence of errors, or an error indication. Note that argument definition and the error protocol guarantee a one-to-one relationship between the elements of <replaceable>namelist</replaceable> and <replaceable>pmidlist</replaceable>; both lists contain exactly <replaceable>numpmid</replaceable> elements.</para>
<para>The python bindings return an array of associated PMIDs corresponding to a tuple of <replaceable>MetricNames</replaceable>.  The returned <replaceable>pmid</replaceable> tuple is passed to <command>pmLookupDescs</command> and <command>pmFetch</command>.</para>
</section>
<section id="id5200423">

<title><command>pmNameAll</command> Function</title>
<literallayout class="monospaced">int pmNameAll(pmID <replaceable>pmid</replaceable>, char ***<replaceable>nameset</replaceable>)
<command>Python:</command>
[name1, name2...] = pmNameAll(<replaceable>pmid</replaceable>)</literallayout>
<para><indexterm id="IG31340177299"><primary>pmNameAll function</primary></indexterm>Given a performance metric ID in <replaceable>pmid</replaceable>, <command>pmNameAll</command> determines all the corresponding metric names, if any, in the PMNS, and returns these through <replaceable>nameset</replaceable>.</para>
<para>The resulting list of pointers <replaceable>nameset</replaceable> and the values (relative names) that the pointers reference are allocated by <command>pmNameAll</command> with a single call to <command>malloc</command>. It is the caller's responsibility to call <command>free</command> and release the space when it is no longer required.</para>
<para>In the absence of errors, <command>pmNameAll</command> returns the number of names in <command>nameset</command>.</para>
<para>For many PMNS instances, there is a 1:1 mapping between a name and a PMID, and under these circumstances, <command>pmNameID</command> provides a simpler interface in the absence of duplicate names for a particular PMID.</para>
<para>The python bindings return a tuple of all metric names having this identical <replaceable>pmid</replaceable>.</para>
</section>
<section id="id5200532">

<title><command>pmNameID</command> Function</title>
<literallayout class="monospaced">int pmNameID(pmID <replaceable>pmid</replaceable>, char **<replaceable>name</replaceable>)
<command>Python:</command>
"metric name" = pmNameID(<replaceable>pmid</replaceable>)</literallayout>
<para><indexterm id="IG31340177300"><primary>pmNameID function</primary></indexterm>Given a performance metric ID in <replaceable>pmid</replaceable>, <command>pmNameID</command> determines the corresponding metric name, if any, in the current PMNS, and returns this through <replaceable>name</replaceable>.</para>
<para>In the absence of errors, <command>pmNameID</command> returns zero. The <replaceable>name</replaceable> argument is a null byte terminated string, allocated by <command>pmNameID</command> using <command>malloc</command>. It is the caller's responsibility to call <command>free</command> and release the space when it is no longer required.</para>
<para>The python bindings return a metric name corresponding to a <replaceable>pmid</replaceable>.</para>
</section>
<section id="id5200683">

<title><command>pmTraversePMNS</command> Function</title>
<literallayout class="monospaced">int pmTraversePMNS(const char *<replaceable>name</replaceable>, void (*<command>dometric</command>)(const char *))
<command>Python:</command>
int <replaceable>status</replaceable> = pmTraversePMNS(<replaceable>name</replaceable>, <replaceable>traverse_callback</replaceable>)</literallayout>
<para><indexterm id="IG31340177301"><primary>pmTraversePMNS function</primary></indexterm>The function <command>pmTraversePMNS</command> may be used to perform a depth-first traversal of the PMNS. The traversal starts at the node identified by <replaceable>name</replaceable> --if <replaceable>name</replaceable> is an empty string, the traversal starts at the root of the PMNS. Usually, <replaceable>name</replaceable> would be the pathname of a non-leaf node in the PMNS.</para>
<para><indexterm id="IG31340177302"><primary>dometric function</primary></indexterm><indexterm id="IG31340177303"><primary>leaf node</primary></indexterm>For each leaf node (actual performance metrics) found in the traversal, the user-supplied function <command>dometric</command> is called with the full pathname of that metric in the PMNS as the single argument; this argument is a null byte-terminated string, and is constructed from a buffer that is managed internally to <command>pmTraversePMNS</command>. Consequently, the value is valid only during the call to <command>dometric</command>--if the pathname needs to be retained, it should be copied using <command>strdup</command> before returning from <command>dometric</command>; see the <command>strdup(3)</command> man page.</para>
<para>The python bindings perform a depth first traversal of the PMNS by scanning <replaceable>namespace</replaceable>, depth first, and call a python function <replaceable>traverse_callback</replaceable> for each node.  
</para>
</section>
<section id="id5200806">

<title><command>pmUnloadNameSpace</command> Function</title>
<literallayout class="monospaced">int pmUnloadNameSpace(void)
<command>Python:</command>
pmUnLoadNameSpace(<replaceable>"NameSpace"</replaceable>)</literallayout>
<para><indexterm id="IG31340177304"><primary>pmUnloadNameSpace function</primary></indexterm>If a local PMNS was loaded with <command>pmLoadNameSpace</command>, calling <command>pmUnloadNameSpace</command> frees up the memory associated with the PMNS and force all subsequent Name Space functions to use the distributed PMNS. If <command>pmUnloadNameSpace</command> is called before calling <command>pmLoadNameSpace</command>, it has no effect.</para>
<para>As discussed in <xref linkend="id5200074"/> there are few if any situations where clients need to call this routine in modern versions of PCP.</para>
</section>
</section>
<section id="LE89521-PARENT">

<title>PMAPI Metrics Description Services</title>
<para>The functions described in this section provide Performance Metrics Application Programming Interface (PMAPI) metric description services.</para>
<section id="id5200868">

<title><command>pmLookupDesc</command> Function</title>
<literallayout class="monospaced">int pmLookupDesc(pmID <replaceable>pmid</replaceable>, pmDesc *<replaceable>desc</replaceable>)
<command>Python:</command>
pmDesc* <replaceable>pmdesc</replaceable> = pmLookupDesc(c_uint <replaceable>pmid</replaceable>)
(pmDesc* <replaceable>pmdesc</replaceable>)[] = pmLookupDescs(c_uint <replaceable>pmids</replaceable>[N])
(pmDesc* <replaceable>pmdesc</replaceable>)[] = pmLookupDescs(c_uint <replaceable>pmid</replaceable>)</literallayout>
<para><indexterm id="IG31340177305"><primary>metrics description services</primary></indexterm><indexterm id="IG31340177306"><primary>metric description services</primary></indexterm><indexterm id="IG31340177307"><primary>pmLookupDesc function</primary></indexterm><indexterm id="IG31340177308"><primary>PMAPI</primary><secondary>description services</secondary></indexterm>Given a Performance Metric Identifier (PMID) as <replaceable>pmid</replaceable>, <command>pmLookupDesc</command> returns the associated <literal>pmDesc</literal> structure through the parameter <replaceable>desc</replaceable> from the current PMAPI context. For more information about <literal>pmDesc</literal>, see <xref linkend="LE11914-PARENT"/>.</para>
<para>
The python bindings return the metric description structure  <literal>pmDesc</literal> corresponding to <replaceable>pmid</replaceable>.  The returned <replaceable>pmdesc</replaceable> is passed to <command>pmExtractValue</command> and <command>pmLookupInDom</command>. The python bindings provide an entry <command>pmLookupDescs</command> that is similar to pmLookupDesc but does a metric description lookup for each element in a PMID array <replaceable>pmids</replaceable>.</para>
</section>
<section id="id5200999">

<title><command>pmLookupInDomText</command> Function</title>
<programlisting>int pmLookupInDomText(pmInDom <replaceable>indom</replaceable>, int <replaceable>level</replaceable>, char **<replaceable>buffer</replaceable>)
<command>Python:</command>
"metric description" = pmGetInDomText(pmDesc <replaceable>pmdesc</replaceable>)</programlisting>
<para><indexterm id="IG31340177309"><primary>indom instance domain</primary></indexterm><indexterm id="IG31340177310"><primary>pmLookupInDomText function</primary></indexterm>Provided the source of metrics from the current PMAPI context is a host, retrieve descriptive text about the performance metrics instance domain identified by <replaceable>indom</replaceable>.</para>
<para><indexterm id="IG31340177311"><primary>help text</primary><secondary>pmLookupInDomText function</secondary></indexterm>The <replaceable>level</replaceable> argument should be <literal>PM_TEXT_ONELINE</literal> for a one-line summary, or <literal>PM_TEXT_HELP</literal> for a more verbose description suited to a help dialogue. The space pointed to by <replaceable>buffer</replaceable> is allocated in <command>pmLookupInDomText</command> with <command>malloc</command>, and it is the responsibility of the caller to free unneeded space; see the <command>malloc(3)</command> and<command>&#xA0;free(3)</command> man pages.</para>
<para>The help text files used to implement <filename>pmLookupInDomText</filename> are often created using <command>newhelp</command> and accessed by the appropriate PMDA response to requests forwarded to the PMDA by PMCD. Further details may be found in <xref linkend="LE72473-PARENT"/>.</para>
<para>
The python bindings lookup the description text about the performance metrics pmDesc <replaceable>pmdesc</replaceable>.  The default is a one line summary; for a more verbose description add an optional second parameter <replaceable>cpmapi.PM_TEXT_HELP</replaceable>.  The constant is available by importing cpmapi.  
</para>
</section>
<section id="id5201140">

<title><command>pmLookupText</command> Function</title>
<programlisting>int pmLookupText(pmID <replaceable>pmid</replaceable>, int <replaceable>level</replaceable>, char **<replaceable>buffer</replaceable>)
<command>Python:</command>
"metric description" = pmLookupText(c_uint <replaceable>pmid</replaceable>)</programlisting>
<para><indexterm id="IG31340177312"><primary>pmLookupText function</primary></indexterm><indexterm id="IG31340177313"><primary>help text</primary><secondary>pmLookupText function</secondary></indexterm>Provided the source of metrics from the current PMAPI context is a host, retrieve descriptive text about the performance metric identified by <replaceable>pmid</replaceable>. The argument <replaceable>level</replaceable> should be <literal>PM_TEXT_ONELINE</literal> for a one-line summary, or <literal>PM_TEXT_HELP</literal> for a more verbose description, suited to a help dialogue.</para>
<para>The space pointed to by <replaceable>buffer</replaceable> is allocated in <command>pmLookupText</command> with <command>malloc</command>, and it is the responsibility of the caller to <command>free</command> the space when it is no longer required; see the <command>malloc(3)</command> and <command>free(3)</command> man pages.</para>
<para>The help text files used to implement <command>pmLookupText</command> are created using <command>newhelp</command> and accessed by the appropriate PMDA in response to requests forwarded to the PMDA by PMCD. Further details may be found in <xref linkend="LE72473-PARENT"/>.</para>
<para>
The python bindings lookup the description text about the performance metrics pmID <replaceable>pmid</replaceable>.  The default is a one line summary; for a more verbose description add an optional second parameter <replaceable>cpmapi.PM_TEXT_HELP</replaceable>.  The constant is available by importing cpmapi.   
</para>
</section>
</section>
<section id="LE27200-PARENT">

<title>PMAPI Instance Domain Services</title>
<para>The functions described in this section provide Performance Metrics Application Programming Interface (PMAPI) instance domain services.</para>
<section id="id5201278">

<title><command>pmGetInDom</command> Function</title>
<literallayout class="monospaced">int pmGetInDom(pmInDom <replaceable>indom</replaceable>, int **<replaceable>instlist</replaceable>, char ***<replaceable>namelist</replaceable>)
<command>Python:</command>
([instance1, instance2...] [name1, name2...]) pmGetInDom(pmDesc <replaceable>pmdesc</replaceable>)</literallayout>
<para><indexterm id="IG31340177314"><primary>instance domain services</primary></indexterm><indexterm id="IG31340177315"><primary>instlist argument</primary></indexterm><indexterm id="IG31340177316"><primary>pmGetInDom function</primary></indexterm><indexterm id="IG31340177317"><primary>PMAPI</primary><secondary>instance domain services</secondary></indexterm>In the current PMAPI context, locate the description of the instance domain <replaceable>indom</replaceable>, and return through <replaceable>instlist</replaceable> the internal instance identifiers for all instances, and through <replaceable>namelist</replaceable> the full external identifiers for all instances. The number of instances found is returned as the function value (or less than zero to indicate an error).</para>
<para>The resulting lists of instance identifiers (<replaceable>instlist</replaceable> and <replaceable>namelist</replaceable>), and the names that the elements of <replaceable>namelist</replaceable> point to, are allocated by <command>pmGetInDom</command> with two calls to <command>malloc</command>, and it is the responsibility of the caller to use <command>free</command><replaceable>(instlist)</replaceable> and <command>free</command><replaceable>(namelist) </replaceable>to release the space when it is no longer required. When the result of <command>pmGetInDom</command> is less than one, both <replaceable>instlist</replaceable> and <replaceable>namelist</replaceable> are undefined (no space is allocated, and so calling <command>free</command> is a bad idea); see the <command>
malloc(3)</command> and <command>free(3)</command> man pages.</para>
<para>
The python bindings return a tuple of the instance identifiers and instance names for an instance domain <replaceable>pmdesc</replaceable>.  </para>
</section>
<section id="id5201511">

<title><command>pmLookupInDom</command> Function</title>
<programlisting>int pmLookupInDom(pmInDom <replaceable>indom</replaceable>, const char *<replaceable>name</replaceable>)
<command>Python:</command>
int <replaceable>instid</replaceable> = pmLookupInDom(pmDesc <replaceable>pmdesc</replaceable>, <replaceable>"Instance"</replaceable>)</programlisting>
<para><indexterm id="IG31340177318"><primary>pmLookupInDom function</primary></indexterm>For the instance domain <replaceable>indom</replaceable>, in the current PMAPI context, locate the instance with the external identification given by <replaceable>name</replaceable>, and return the internal instance identifier.</para>
<para>
The python bindings return the instance id corresponding to <replaceable>"Instance"</replaceable> in the instance domain <replaceable>pmdesc</replaceable>.  
</para>
</section>
<section id="id5201553">

<title><command>pmNameInDom</command> Function</title>
<programlisting>int pmNameInDom(pmInDom <replaceable>indom</replaceable>, int <replaceable>inst</replaceable>, char **<replaceable>name</replaceable>)
<command>Python:</command>
"instance id" = pmNameInDom(pmDesc <replaceable>pmdesc</replaceable>, c_uint <replaceable>instid</replaceable>)</programlisting>
<para><indexterm id="IG31340177319"><primary>pmNameInDom function</primary></indexterm>For the instance domain <replaceable>indom</replaceable>, in the current PMAPI context, locate the instance with the internal instance identifier given by <replaceable>inst</replaceable>, and return the full external identification through <replaceable>name</replaceable>. The space for the value of <replaceable>name</replaceable> is allocated in <command>pmNameInDom</command> with <command>malloc</command>, and it is the responsibility of the caller to free the space when it is no longer required; see the <command>malloc(3)</command> and <command>free(3)</command> man pages.</para>
<para>
The python bindings return the text name of an instance corresponding to an instance domain <replaceable>pmdesc</replaceable> with instance identifier <replaceable>instid</replaceable>.  
</para>
</section>
</section>
<section id="LE94187-PARENT">

<title>PMAPI Context Services</title>
<para><indexterm id="IG31340177320"><primary>context services</primary></indexterm><indexterm id="IG31340177321"><primary>PMAPI</primary><secondary>context services</secondary></indexterm><xref linkend="id5201726"/> shows which of the three components of a PMAPI context (metrics source, instance profile, and collection time) are relevant for various PMAPI functions. Those PMAPI functions not shown in this table either manipulate the PMAPI context directly, or are executed independently of the current PMAPI context.</para>
<table id="id5201726" frame="topbot">

<title>Context Components of PMAPI Functions </title>
<tgroup cols="5" colsep="0" rowsep="0">
<colspec colwidth="132*"/>
<colspec colwidth="75*"/>
<colspec colwidth="77*"/>
<colspec colwidth="77*"/>
<colspec colwidth="35*"/>
<thead>
<row rowsep="1" valign="top"><entry align="left" valign="bottom"><para>Function Name</para></entry><entry align="left" valign="bottom"><para>Metrics Source</para></entry><entry align="left" valign="bottom"><para>Instance Profile</para></entry><entry align="left" valign="bottom"><para>Collection Time</para></entry><entry align="left" valign="bottom"><para>Notes</para></entry></row></thead>
<tbody>
<row valign="top">
<entry align="left" valign="top"><para><command>pmAddProfile</command><indexterm id="IG31340177322"><primary>pmAddProfile function</primary></indexterm></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmDelProfile<indexterm id="IG31340177323"><primary>pmDelProfile function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmDupContext<indexterm id="IG31340177324"><primary>pmDupContext function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmFetch<indexterm id="IG31340177325"><primary>pmFetch function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmFetchArchive<indexterm id="IG31340177326"><primary>pmFetchArchive function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>(1)</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmGetArchiveEnd<indexterm id="IG31340177327"><primary>pmGetArchiveEnd function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>(1)</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmGetArchiveLabel<indexterm id="IG31340177328"><primary>pmGetArchiveLabel function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>(1)</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmGetChildren<indexterm id="IG31340177329"><primary>pmGetChildren function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmGetChildrenStatus<indexterm id="IG31340177330"><primary>pmGetChildrenStatus function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmGetContextHostName<indexterm id="IG31340177330nat"><primary>pmGetContextHostName function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmGetPMNSLocation<indexterm id="IG31340177331"><primary>pmGetPMNSLocation function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmGetInDom<indexterm id="IG31340177332"><primary>pmGetInDom function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>(2)</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmGetInDomArchive<indexterm id="IG31340177333"><primary>pmGetInDomArchive function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>(1)</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmLookupDesc<indexterm id="IG31340177334"><primary>pmLookupDesc function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>(3)</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmLookupInDom<indexterm id="IG31340177335"><primary>pmLookupInDom function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>(2)</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmLookupInDomArchive<indexterm id="IG31340177336"><primary>pmLookupInDomArchive function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>(1,2)</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmLookupInDomText<indexterm id="IG31340177337"><primary>pmLookupInDomText function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>(4)</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmLookupName<indexterm id="IG31340177338"><primary>pmLookupName function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmLookupText<indexterm id="IG31340177339"><primary>pmLookupText function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>(4)</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmNameAll</command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmNameID<indexterm id="IG31340177340"><primary>pmNameID function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmNameInDom<indexterm id="IG31340177341"><primary>pmNameInDom function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>(2)</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmNameInDomArchive<indexterm id="IG31340177342"><primary>pmNameInDomArchive function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>(1,2)</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmSetMode<indexterm id="IG31340177343"><primary>pmSetMode function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmStore<indexterm id="IG31340177344"><primary>pmStore function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>(5)</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmTraversePMNS<indexterm id="IG31340177345"><primary>pmTraversePMNS function</primary></indexterm></command></para></entry>
<entry align="left" valign="top"><para>Yes</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>&#xA0;</para></entry></row></tbody></tgroup></table>
<para>Notes:</para>
<orderedlist><listitem><para>Operation supported only for PMAPI contexts where the source of metrics is an archive.</para>
</listitem><listitem><para><indexterm id="IG31340177346"><primary>specific instance domain</primary></indexterm>A specific instance domain is included in the arguments to these functions, and the result is independent of the instance profile for any PMAPI context.</para>
</listitem><listitem><para>The metadata that describes a performance metric is sensitive to the source of the metrics, but independent of any instance profile and of the collection time.</para>
</listitem><listitem><para><indexterm id="IG31340177347"><primary>archive logs</primary><secondary>context services</secondary></indexterm>Operation is supported only for PMAPI contexts where the source of metrics is a host. The text associated with a metric is assumed to be invariant with time and is definitely insensitive to the current members of the instance domain. In all cases this information is unavailable from an archive context (it is not included in the archive logs), and is directly available from a PMDA via PMCD in the other cases.</para>
</listitem><listitem><para>This operation is supported only for contexts where the source of the metrics is a host. Further, the instance identifiers are included in the argument to the function, and the effects upon the current values of the metrics are immediate (retrospective changes are not allowed). Consequently, from the current PMAPI context, neither the instance profile nor the collection time influence the result of this function.</para>
</listitem></orderedlist>
<section id="id5203779">

<title><command>pmNewContext</command> Function</title>
<programlisting>int pmNewContext(int <replaceable>type</replaceable>, const char *<replaceable>name</replaceable>)</programlisting>
<para><indexterm id="IG31340177348"><primary>PM_CONTEXT_ARCHIVE type</primary></indexterm><indexterm id="IG31340177349"><primary>PM_CONTEXT_HOST type</primary></indexterm><indexterm id="IG31340177350"><primary>pmNewContext function</primary></indexterm>The <command>pmNewContext</command> function may be used to establish a new PMAPI context. The source of metrics is identified by <replaceable>name</replaceable>, and may be a host specification (<replaceable>type</replaceable> is <literal>PM_CONTEXT_HOST</literal>) or the basename of an archive log (<replaceable>type</replaceable> is <literal>PM_CONTEXT_ARCHIVE</literal>).</para>
<para>A host specification usually contains a simple hostname, an internet address (IPv4 or IPv6), or the path to the PMCD Unix domain socket. It can also specify properties of the connection to PMCD, such as the protocol to use (secure and encrypted, or native) and whether PMCD should be reached via a <command>pmproxy</command> host. Various other connection attributes, such as authentication information (user name, password, authentication method, and so on) can also be specified. Further details can be found in the <command>PCPIntro(3)</command> man page, and the companion <citetitle>Performance Co-Pilot Tutorials and Case Studies</citetitle> document.</para>
<para>In the case where <replaceable>type</replaceable> is <literal>PM_CONTEXT_LOCAL</literal>, <replaceable>name</replaceable> is ignored, and the context uses a stand-alone connection to the PMDA methods used by PMCD. When this type of context is in effect, the range of accessible performance metrics is constrained to DSO PMDAs listed in the <command>pmcd</command> configuration file <filename>${PCP_PMCDCONF_PATH}</filename>. The reason this is done, as opposed to all of the DSO PMDAs found below <filename>${PCP_PMDAS_DIR}</filename> for example, is that DSO PMDAs listed there are very likely to have their metric names reflected in the local Name Space file, which will be loaded for this class of context.</para>
<para><indexterm id="IG31340177351"><primary>pmFetch function</primary></indexterm><indexterm id="IG31340177352"><primary>collection time</primary></indexterm>The initial instance profile is set up to select all instances in all instance domains, and the initial collection time is the current time at the time of each request for a host, or the time at the start of the log for an archive. In the case of archives, the initial collection time results in the earliest set of metrics being returned from the archive at the first <command>pmFetch</command>.</para>
<para>Once established, the association between a PMAPI context and a source of metrics is fixed for the life of the context; however, functions are provided to independently manipulate both the instance profile and the collection time components of a context.</para>
<para><indexterm id="IG31340177353"><primary>pmUseContext function</primary></indexterm>The function returns a &#x201C;handle&#x201D; that may be used in subsequent calls to <command>pmUseContext</command>. This new PMAPI context stays in effect for all subsequent context sensitive calls across the PMAPI until another call to <command>pmNewContext</command> is made, or the context is explicitly changed with a call to <command>pmDupContext</command> or <command>pmUseContext</command>.</para>
<para>For the python bindings creating and destroying a PMAPI context is done by creating and destroying an object of the pmapi class.</para>
</section>
<section id="id5203964">

<title><command>pmDestroyContext</command> Function</title>
<programlisting>int pmDestroyContext(int <replaceable>handle</replaceable>)</programlisting>
<para><indexterm id="IG31340177354"><primary>pmDestroyContext function</primary></indexterm>The PMAPI context identified by <replaceable>handle</replaceable> is destroyed. Typically, this implies terminating a connection to PMCD or closing an archive file, and orderly clean-up. The PMAPI context must have been previously created using <command>pmNewContext</command> or <command>pmDupContext</command>.</para>
<para>On success, <command>pmDestroyContext</command> returns zero. If <replaceable>handle</replaceable> was the current PMAPI context, then the current context becomes undefined. This means the application must explicitly re-establish a valid PMAPI context with <command>pmUseContext</command>, or create a new context with <command>pmNewContext</command> or <command>pmDupContext</command>, before the next PMAPI operation requiring a PMAPI context.</para>
<para>For the python bindings creating and destroying a PMAPI context is done by creating and destroying an object of the pmapi class.</para>

</section>
<section id="id5204063">

<title><command>pmDupContext</command> Function</title>
<programlisting>int pmDupContext(void)</programlisting>
<programlisting></programlisting>
<para><indexterm id="IG31340177355"><primary>pmDupContext function</primary></indexterm>Replicate the current PMAPI context (source, instance profile, and collection time). This function returns a handle for the new context, which may be used with subsequent calls to <command>pmUseContext</command>. The newly replicated PMAPI context becomes the current context.</para>
</section>
<section id="id5204116">

<title><command>pmUseContext</command> Function</title>
<programlisting>int pmUseContext(int <replaceable>handle</replaceable>)</programlisting>
<para><indexterm id="IG31340177356"><primary>pmUseContext function</primary></indexterm>Calling <command>pmUseContext</command> causes the current PMAPI context to be set to the context identified by <replaceable>handle</replaceable>. The value of <replaceable>handle</replaceable> must be one returned from an earlier call to <command>pmNewContext</command> or <command>pmDupContext</command>.</para>
<para>Below the PMAPI, all contexts used by an application are saved in their most recently modified state, so <command>pmUseContext</command> restores the context to the state it was in the last time the context was used, not the state of the context when it was established.</para>
</section>
<section id="id5204177">

<title><command>pmWhichContext</command> Function</title>
<programlisting>int pmWhichContext(void)
<command>Python:</command>
int <replaceable>ctx</replaceable>_idx = pmWhichContext()</programlisting>
<para><indexterm id="IG31340177357"><primary>collection time</primary></indexterm><indexterm id="IG31340177358"><primary>pmWhichContext function</primary></indexterm>Returns the handle for the current PMAPI context (source, instance profile, and collection time).</para>
<para>
The python bindings return the handle of the current PMAPI context.  
</para>
</section>
<section id="id5204210">

<title><command>pmAddProfile</command> Function</title>
<programlisting>int pmAddProfile(pmInDom <replaceable>indom</replaceable>, int <replaceable>numinst</replaceable>, int <replaceable>instlist</replaceable>[])
<command>Python:</command>
int <replaceable>status</replaceable> = pmAddProfile(pmDesc pmdesc, [c_uint instid])</programlisting>
<para><indexterm id="IG31340177359"><primary>indom instance domain</primary></indexterm><indexterm id="IG31340177360"><primary>instlist argument</primary></indexterm><indexterm id="IG31340177361"><primary>pmAddProfile function</primary></indexterm>Add new instance specifications to the instance profile of the current PMAPI context. At its simplest, instances identified by the <replaceable>instlist</replaceable> argument for the <replaceable>indom</replaceable> instance domain are added to the instance profile. The list of instance identifiers contains <replaceable>numinst</replaceable> values.</para>
<para><indexterm id="IG31340177362"><primary>PM_INDOM_NULL instance domain</primary><secondary>pmAddProfile function</secondary></indexterm>If <replaceable>indom</replaceable> equals <literal>PM_INDOM_NULL</literal>, or <replaceable>numinst</replaceable> is zero, then all instance domains are selected. If <replaceable>instlist</replaceable> is NULL, then all instances are selected. To enable all available instances in all domains, use this syntax:</para>
<programlisting>pmAddProfile(PM_INDOM_NULL,&#xA0;0, NULL).</programlisting>
<para>
The python bindings add the list of instances <replaceable>instid</replaceable> to the instance profile of the instance <replaceable>pmdesc</replaceable>.  
</para>
</section>
<section id="id5204360">

<title><command>pmDelProfile</command> Function</title>
<programlisting>int pmDelProfile(pmInDom <replaceable>indom</replaceable>, int <replaceable>numinst</replaceable>, int <replaceable>instlist</replaceable>[])
<command>Python:</command>
int <replaceable>status</replaceable> = pmDelProfile(pmDesc pmdesc, c_uint instid)
int <replaceable>status</replaceable> = pmDelProfile(pmDesc pmdesc, [c_uint instid])
</programlisting>
<para><indexterm id="IG31340177363"><primary>PM_INDOM_NULL instance domain</primary><secondary>pmDelProfile function</secondary></indexterm><indexterm id="IG31340177364"><primary>pmDelProfile function</primary></indexterm>Delete instance specifications from the instance profile of the current PMAPI context. In the simplest variant, the list of instances identified by the <replaceable>instlist</replaceable> argument for the <replaceable>indom</replaceable> instance domain is removed from the instance profile. The list of instance identifiers contains <replaceable>numinst</replaceable> values.</para>
<para>If <replaceable>indom</replaceable> equals <literal>PM_INDOM_NULL</literal>, then all instance domains are selected for deletion. If <replaceable>instlist</replaceable> is NULL, then all instances in the selected domains are removed from the profile. To disable all available instances in all domains, use this syntax:</para>
<programlisting>pmDelProfile(PM_INDOM_NULL, 0, NULL) </programlisting>
<para>
The python bindings delete the list of instances <replaceable>instid</replaceable> from the instance profile of the instance domain <replaceable>pmdesc</replaceable>.  
</para>
</section>
<section id="id5204444">

<title><command>pmSetMode</command> Function</title>
<programlisting>int pmSetMode(int <replaceable>mode</replaceable>, const struct timeval *<replaceable>when</replaceable>, int <replaceable>delta</replaceable>)
<command>Python:</command>
int <replaceable>status</replaceable> = pmSetMode(mode, timeVal <replaceable>timeval</replaceable>, int <replaceable>delta</replaceable>)</programlisting>
<para><indexterm id="IG31340177365"><primary>pmNameInDom function</primary></indexterm><indexterm id="IG31340177366"><primary>pmLookupInDom function</primary></indexterm><indexterm id="IG31340177367"><primary>pmGetInDom function</primary></indexterm><indexterm id="IG31340177368"><primary>pmLookupDesc function</primary></indexterm><indexterm id="IG31340177369"><primary>pmFetchArchive function</primary></indexterm><indexterm id="IG31340177370"><primary>pmFetch function</primary></indexterm><indexterm id="IG31340177371"><primary>pmSetMode function</primary></indexterm>This function defines the collection time and mode for accessing performance metrics and metadata in the current PMAPI context. This mode affects the semantics of subsequent calls to the following PMAPI functions: <command>pmFetch</command>, <command>pmFetchArchive</command>, <command>pmLookupDesc</command>, <command>pmGetInDom</command>, <command>pmLookupInDom
</command>, and <command>pmNameInDom</command>.</para>
<para>The <command>pmSetMode</command> function requires the current PMAPI context to be of type <literal>PM_CONTEXT_ARCHIVE</literal>.</para>
<para>The <replaceable>when</replaceable> parameter defines a time origin, and all requests for metadata (metrics descriptions and instance identifiers from the instance domains) are processed to reflect the state of the metadata as of the time origin. For example, use the last state of this information at, or before, the time origin.</para>
<para>If the <replaceable>mode</replaceable> is <literal>PM_MODE_INTERP</literal> then, in the case of <command>pmFetch</command>, the underlying code uses an interpolation scheme to compute the values of the metrics from the values recorded for times in the proximity of the time origin.</para>
<para>If the <replaceable>mode</replaceable> is <literal>PM_MODE_FORW</literal>, then, in the case of <command>pmFetch</command>, the collection of recorded metric values is scanned forward, until values for at least one of the requested metrics is located after the time origin. Then all requested metrics stored in the PCP archive at that time are returned with a corresponding time stamp. This is the default mode when an archive context is first established with <command>pmNewContext</command>.</para>
<para>If the <replaceable>mode</replaceable> is <literal>PM_MODE_BACK</literal>, then the situation is the same as for <literal>PM_MODE_FORW</literal>, except a <command>pmFetch</command> is serviced by scanning the collection of recorded metrics backward for metrics before the time origin.</para>
<para>After each successful <command>pmFetch</command>, the time origin is reset to the time stamp returned through the <filename>pmResult</filename>.</para>
<para>The <command>pmSetMode</command> parameter <replaceable>delta</replaceable> defines an additional number of time unit that should be used to adjust the time origin (forward or backward) after the new time origin from the <filename>pmResult</filename> has been determined. This is useful when moving through archives with a mode of <literal>PM_MODE_INTERP</literal>. The high-order bits of the <replaceable>mode</replaceable> parameter field is also used to optionally set the units of time for the <filename>delta</filename> field. To specify the units of time, use the <literal>PM_XTB_SET</literal> macro with one of the values <literal>PM_TIME_NSEC</literal>, <literal>PM_TIME_MSEC</literal>, <literal>PM_TIME_SEC</literal>, or so on as follows:</para>
<literallayout class="monospaced">PM_MODE_INTERP | PM_XTB_SET(PM_TIME_<replaceable>XXXX)</replaceable></literallayout>
<para>If no units are specified, the default is to interpret <replaceable>delta</replaceable> as milliseconds.</para>
<para>Using these mode options, an application can implement replay, playback, fast forward, or reverse for performance metric values held in a PCP archive log by alternating calls to <command>pmSetMode</command> and <command>pmFetch</command>.</para>
<para>In <xref linkend="Z976827209sdc"/>, the code fragment may be used to dump only those values stored in correct temporal sequence, for the specified performance metric <filename>my.metric.name</filename>:</para>
<example id="Z976827209sdc">
<title>Dumping Values in Temporal Sequence</title>
<programlisting>    int     sts;
    pmID    pmid;
    char    *name = &#x201C;my.metric.name&#x201D;;

    sts = pmNewContext(PM_CONTEXT_ARCHIVE, &#x201C;myarchive&#x201D;);
    sts = pmLookupName(1, &amp;name, &amp;pmid);
    for ( ; ; ) {
        sts = pmFetch(1, &amp;pmid, &amp;result);
        if (sts &lt; 0)
            break;
        /* dump value(s) from result-&gt;vset[0]-&gt;vlist[] */
        pmFreeResult(result);
   }</programlisting>
</example>
<para><indexterm id="IG31340177372"><primary>interpolated metrics</primary></indexterm>Alternatively, the code fragment in <xref linkend="Z976827219sdc"/> may be used to replay interpolated metrics from an archive in reverse chronological order, at ten-second intervals (of recorded time):</para>
<example id="Z976827219sdc">
<title>Replaying Interpolated Metrics</title>
<programlisting>    int             sts;
    pmID            pmid;
    char            *name = &#x201C;my.metric.name&#x201D;;
    struct timeval  endtime;

    sts = pmNewContext(PM_CONTEXT_ARCHIVE, &#x201C;myarchive&#x201D;);
    sts = pmLookupName(1, &amp;name, &amp;pmid);
    sts = pmGetArchiveEnd(&amp;endtime);
    sts = pmSetMode(PM_MODE_INTERP, &amp;endtime, -10000);
    while (pmFetch(1, &amp;pmid, &amp;result) != PM_ERR_EOL) {
        /*
         * process interpolated metric values as of result-&gt;timestamp
         */
        pmFreeResult(result);
   }</programlisting>
</example>
<para>
The python bindings define the collection
<replaceable>time</replaceable> and <replaceable>mode</replaceable>
for reading archive files.  <replaceable>mode</replaceable> can be one
of:  c_api.PM_MODE_LIVE, c_api.PM_MODE_INTERP, c_api.FORW,
c_api.BACK. wjocj are available by importing cpmapi.
</para>
</section>
<section id="id5204926">

<title><command>pmReconnectContext</command> Function</title>
<programlisting>int pmReconnectContext(int <replaceable>handle</replaceable>)
<command>Python:</command>
int <replaceable>status</replaceable> = pmReconnectContext()
</programlisting>
<para><indexterm id="IG31340177373"><primary>PMCD</primary><secondary>pmReconnectContext function</secondary></indexterm><indexterm id="IG31340177374"><primary>pmReconnectContext function</primary></indexterm>As a result of network, host, or PMCD (Performance Metrics Collection Daemon) failure, an application's connection to PMCD may be established and then lost.</para>
<para><indexterm id="IG31340177375"><primary>handle context</primary></indexterm>The function <command>pmReconnectContext</command> allows an application to request that the PMAPI context identified by <replaceable>handle</replaceable> be re-established, provided the associated PMCD is accessible.</para>
<note><para><replaceable>handle</replaceable> may or may not be the current context.</para>
</note>
<para><indexterm id="IG31340177376"><primary>PMCD_RECONNECT_TIMEOUT variable</primary></indexterm>To avoid flooding the system with reconnect requests, <command>pmReconnectContext</command> attempts a reconnection only after a suitable delay from the previous attempt. This imposed restriction on the reconnect re-try time interval uses a default exponential back-off so that the initial delay is 5 seconds after the first unsuccessful attempt, then 10 seconds, then 20 seconds, then 40 seconds, and then 80 seconds thereafter. The intervals between reconnection attempts may be modified using the environment variable <literal>PMCD_RECONNECT_TIMEOUT</literal> and the time to wait before an attempted connection is deemed to have failed is controlled by the <literal>PMCD_CONNECT_TIMEOUT</literal> environment variable;
see the <command>PCPIntro(1)</command> man page.</para>
<para>If the reconnection succeeds, <command>pmReconnectContext</command> returns <replaceable>handle</replaceable>. Note that even in the case of a successful reconnection, <command>pmReconnectContext</command> does not change the current PMAPI context.</para>
<para>
The python bindings reestablish the connection for the context.
</para>
</section>
<section id="id5205074">

<title><command>pmGetContextHostName</command> Function</title>
<programlisting>const char *pmGetContextHostName(int <replaceable>id</replaceable>)
char *pmGetContextHostName_r(int <replaceable>id</replaceable>, char *<replaceable>buf</replaceable>, int <replaceable>buflen</replaceable>)
<command>Python:</command>
"hostname" = pmGetContextHostName()
</programlisting>
<para>Given a valid PCP context identifier previously created with <command>pmNewContext</command> or <command>pmDupContext</command>, the <command>pmGetContextHostName</command> function provides a possibility to retrieve a host name associated with a context regardless of the context type.</para>
<para>This function will use the <literal>pmcd.hostname</literal> metric if it is available, and so is able to provide an accurate hostname in the presence of connection tunnelling and port forwarding.</para>
<para>If <replaceable>id</replaceable> is not a valid PCP context identifier, this function returns a zero length string and therefore never fails.</para>
<para>In the case of <command>pmGetContextHostName</command>, the string value is held in a single static buffer, so concurrent calls may not produce the desired results. The <command>pmGetContextHostName_r</command> function allows a buffer and length to be passed in, into which the message is stored; this variant uses no shared storage and can be used in a thread-safe manner.</para>
<para>
The python bindings query the current context hostname.
</para>
</section>
</section>
<section id="LE34685-PARENT">

<title>PMAPI Timezone Services</title>
<para>The functions described in this section provide Performance Metrics Application Programming Interface (PMAPI) timezone services.</para>
<section id="id5205140">

<title><command>pmNewContextZone</command> Function</title>
<literallayout class="monospaced">int pmNewContextZone(void)
<command>Python:</command>
pmNewContextZone()
</literallayout>
<para><indexterm id="IG31340177377"><primary>PMAPI</primary><secondary>timezone services</secondary></indexterm><indexterm id="IG31340177378"><primary>timezone services</primary></indexterm><indexterm id="IG31340177379"><primary>pmNewContextZone function</primary></indexterm>If the current PMAPI context is an archive, the <command>pmNewContextZone</command> function uses the timezone from the archive label record to set the current reporting timezone. The current reporting timezone affects the timezone used by <command>pmCtime</command> and <command>pmLocaltime</command>.</para>
<para>If the current PMAPI context corresponds to a host source of metrics, <command>pmNewContextZone</command> executes a <command>pmFetch</command> to retrieve the value for the metric <literal>pmcd.timezone</literal> and uses that to set the current reporting timezone.</para>
<para>In both cases, the function returns a value to identify the current reporting timezone that may be used in a subsequent call to <command>pmUseZone</command> to restore this reporting timezone.</para>
<para><literal>PM_ERR_NOCONTEXT</literal> indicates the current PMAPI context is not valid. A return value less than zero indicates a fatal error from a system call, most likely <command>malloc</command>.</para>
</section>
<section id="id5205286">

<title><command>pmNewZone</command> Function</title>
<literallayout class="monospaced">int pmNewZone(const char *<replaceable>tz</replaceable>)
<command>Python:</command>
int <replaceable>tz_handle</replaceable> = pmNewZone(int <replaceable>tz</replaceable>)</literallayout>
<para><indexterm id="IG31340177380"><primary>pmNewZone function</primary></indexterm>The <command>pmNewZone</command> function sets the current reporting timezone, and returns a value that may be used in a subsequent call to <command>pmUseZone</command> to restore this reporting timezone. The current reporting timezone affects the timezone used by <command>pmCtime</command> and <command>pmLocaltime</command>.</para>
<para>The <replaceable>tz</replaceable> argument defines a timezone string, in the format described for the <literal>TZ</literal> environment variable. See the <command>environ(7)</command> man page.</para>
<para>A return value less than zero indicates a fatal error from a system call, most likely <command>malloc</command>.</para>
<para>
The python bindings create a new zone handle and set reporting timezone for the timezone defined by <replaceable>tz</replaceable>.
</para>
</section>
<section id="id5205362">

<title><command>pmUseZone</command> Function</title>
<literallayout class="monospaced">int pmUseZone(const int <replaceable>tz_handle</replaceable>)
<command>Python:</command>
int <replaceable>status</replaceable> = pmUseZone(int <replaceable>tz_handle</replaceable>)
</literallayout>
<para><indexterm id="IG31340177381"><primary>pmUseZone function</primary></indexterm>In the <command>pmUseZone</command> function, <replaceable>tz_handle</replaceable> identifies a reporting timezone as previously established by a call to <command>pmNewZone</command> or <command>pmNewContextZone</command>, and this becomes the current reporting timezone. The current reporting timezone effects the timezone used by <command>pmCtime</command> and <command>pmLocaltime</command>).</para>
<para>A return value less than zero indicates the value of <replaceable>tz_handle</replaceable> is not legal.</para>
<para>
The python bindings set the current reporting timezone defined by timezone <replaceable>tz_handle</replaceable>.  
</para>
</section>
<section id="id5205444">

<title><command>pmWhichZone</command> Function</title>
<literallayout class="monospaced">int pmWhichZone(char **<replaceable>tz</replaceable>)
<command>Python:</command>
"zone string" = pmWhichZone()</literallayout>
<para><indexterm id="IG31340177382"><primary>pmWhichZone function</primary></indexterm>The <command>pmWhichZone</command> function returns the handle of the current timezone, as previously established by a call to <command>pmNewZone</command> or <command>pmNewContextZone</command>. If the call is successful (that is, there exists a current reporting timezone), a non-negative integer is returned and <replaceable>tz</replaceable> is set to point to a static buffer containing the timezone string itself. The current reporting timezone effects the timezone used by <command>pmCtime</command> and <command>pmLocaltime</command>.</para>
<para>A return value less than zero indicates there is no current reporting timezone.</para>
<para>
The python bindings return the current reporting timezone.  
</para>
</section>
</section>
<section id="LE25844-PARENT">

<title>PMAPI Metrics Services</title>
<para>The functions described in this section provide Performance Metrics Application Programming Interface (PMAPI) metrics  services.</para>
<section id="Z1034802678tls">

<title><command>pmFetch</command> Function</title>
<programlisting>int pmFetch(int <replaceable>numpmid</replaceable>, pmID <replaceable>pmidlist</replaceable>[], pmResult **<replaceable>result</replaceable>)
<command>Python:</command>
pmResult* <replaceable>pmresult</replaceable> = pmFetch(c_uint <replaceable>pmid[]</replaceable>)</programlisting>
<para><indexterm id="IG31340177383"><primary>PMAPI </primary><secondary>metrics services</secondary></indexterm><indexterm id="IG31340177384"><primary>metrics services</primary></indexterm><indexterm id="IG31340177385"><primary>pmFetch function</primary></indexterm>The most common PMAPI operation is likely to be calls to <command>pmFetch</command>, specifying a list of PMIDs (for example, as constructed by <command>pmLookupName</command>) through <replaceable>pmidlist</replaceable> and <replaceable>numpmid</replaceable>. The call to <command>pmFetch</command> is executed in the context of a source of metrics, instance profile, and collection time, previously established by calls to the functions described in <xref linkend="LE94187-PARENT"/>.</para>
<para>The principal result from <command>pmFetch</command> is returned as a tree structured <replaceable>result</replaceable>, described in the <xref linkend="LE82331-PARENT"/>.</para>
<para>If one value (for example, associated with a particular instance) for a requested metric is unavailable at the requested time, then there is no associated <command>pmValue</command> structure in the result. If there are no available values for a metric, then <replaceable>numval</replaceable> is zero and the associated <command>pmValue</command>[] instance is empty; <replaceable>valfmt</replaceable> is undefined in these circumstances, but <replaceable>pmid</replaceable> is correctly set to the PMID of the metric with no values.</para>
<para>If the source of the performance metrics is able to provide a reason why no values are available for a particular metric, this reason is encoded as a standard error code in the corresponding <replaceable>numval</replaceable>; see the <command>pmerr(1)</command> and <command>pmErrStr(3)</command> man pages. Since all error codes are negative, values for a requested metric are unavailable if <replaceable>numval</replaceable> is less than or equal to zero.</para>
<para>The argument definition and the result specifications have been constructed to ensure that for each PMID in the requested <replaceable>pmidlist</replaceable> there is exactly one <command>pmValueSet</command> in the result, and that the PMIDs appear in exactly the same sequence in both <replaceable>pmidlist</replaceable> and <replaceable>result</replaceable>. This makes the number and order of entries in <replaceable>result</replaceable> completely deterministic, and greatly simplifies the application programming logic after the call to <command>pmFetch</command>.</para>
<para><indexterm id="IG31340177386"><primary>pmFreeResult function</primary></indexterm>The result structure returned by <command>pmFetch</command> is dynamically allocated using one or more calls to <command>malloc</command> and specialized allocation strategies, and should be released when no longer required by calling <command>pmFreeResult</command>. Under no circumstances should <command>free</command> be called directly to release this space.</para>
<para>As common error conditions are encoded in the result data structure, only serious events (such as loss of connection to PMCD, <command>malloc</command> failure, and so on) would cause an error value to be returned by <command><indexterm id="IG31340177387"><primary>pmFetch function</primary></indexterm>pmFetch</command>. Otherwise, the value returned by the <command>pmFetch</command> function is zero.</para>
<para>In <xref linkend="Z976559487sdc"/>, the code fragment dumps the values (assumed to be stored in the <replaceable>lval</replaceable> element of the <filename>pmValue</filename> structure) of selected performance metrics once every 10 seconds:</para>
<example id="Z976559487sdc">
<title>PMAPI Metrics Services</title>
<programlisting>    int       i, j, sts;
    pmID      pmidlist[10];
    pmResult  *result;
    time_t    now;

    /* set up PMAPI context, numpmid and pmidlist[] ... */
    while ((sts = pmFetch(10, pmidlist, &amp;result)) &gt;= 0) {
        now = (time_t)result-&gt;timestamp.tv_sec;
        printf(&quot;\n@ %s&quot;, ctime(&amp;now));
        for (i = 0; i &lt; result-&gt;numpmid; i++) {
            printf(&quot;PMID: %s&quot;, pmIDStr(result-&gt;vset[i]-&gt;pmid));
            for (j = 0; j &lt; result-&gt;vset[i]-&gt;numval; j++) {
                printf(&quot; 0x%x&quot;, result-&gt;vset[i]-&gt;vlist[j].value.lval);
                putchar('\n');
            }
        }
        pmFreeResult(result);
        sleep(10);
    }</programlisting>
</example>
<note><para><indexterm id="IG31340177388"><primary>PM_ERR_TIMEOUT error code</primary></indexterm><indexterm id="IG31340177389"><primary>PMCD_REQUEST_TIMOUT variable</primary></indexterm>If a response is not received back from PMCD within 10 seconds, the <command>pmFetch</command> times out and returns <literal>PM_ERR_TIMEOUT</literal>. This is most likely to occur when the PMAPI client and PMCD are communicating over a slow network connection, but may also occur when one of the hosts is extremely busy. The time out period may be modified using the <literal>PMCD_REQUEST_TIMEOUT</literal> environment variable; see the <command>PCPIntro(1)</command> man page.</para>
</note>
<para>
The python bindings fetch a pmResult corresponding to a <replaceable>pmid</replaceable> list, which is returned from <command>pmLookupName</command>.  The returned <replaceable>pmresult</replaceable> is passed to <command>pmExtractValue</command>.   
</para>
</section>
<section id="id5205941">

<title><command>pmFreeResult</command> Function</title>
<programlisting>void pmFreeResult(pmResult *<replaceable>result</replaceable>)
<command>Python:</command>
pmFreeResult(pmResult* <replaceable>pmresult</replaceable>)</programlisting>
<para><indexterm id="IG31340177390"><primary>pmFetch function</primary></indexterm><indexterm id="IG31340177391"><primary>pmFreeResult function</primary></indexterm>Release the storage previously allocated for a result by <command>pmFetch</command>.</para>
<para>
THe python bindings free a <replaceable>pmresult</replaceable> previously allocated by <command>pmFetch</command>.  
</para>
</section>
<section id="id5206030">

<title><command>pmStore</command> Function</title>
<programlisting>int pmStore(const pmResult *<replaceable>request</replaceable>)
<command>Python:</command>
pmResult* <replaceable>pmresult</replaceable> = pmStore(pmResult* <replaceable>pmresult</replaceable>)</programlisting>
<para><indexterm id="IG31340177392"><primary>pmStore function</primary></indexterm>In some special cases it may be helpful to modify the current values of performance metrics in one or more underlying domains, for example to reset a counter to zero, or to modify a <firstterm>metric</firstterm>, which is a control variable within a Performance Metric Domain.</para>
<para><indexterm id="IG31340177393"><primary>pmStore function</primary></indexterm>The <command>pmStore</command> function is a lightweight inverse of <command>pmFetch</command>. The caller must build the <filename>pmResult</filename> data structure (which could have been returned from an earlier <command>pmFetch</command> call) and then call <command>pmStore</command>. It is an error to pass a <replaceable>request</replaceable> to <command>pmStore</command> in which the <literal>numval</literal> field within any of the <command>pmValueSet</command> structure has a value less than one.</para>
<para>The current PMAPI context must be one with a host as the source of metrics, and the current value of the nominated metrics is changed. For example, <command>pmStore</command> cannot be used to make retrospective changes to information in a PCP archive log.</para>
</section>
</section>
<section id="LE40692-PARENT">

<title>PMAPI Record-Mode Services</title>
<para>The functions described in this section provide Performance Metrics Application Programming Interface (PMAPI) record-mode services. These services allow a monitor tool to establish connections to <command>pmlogger</command> co-processes, which they create and control for the purposes of recording live performance data from (possibly) multiple hosts. Since <command>pmlogger</command> records for one host only, these services can administer a group of loggers, and set up archive folios to track the logs. Tools like <command>pmafm</command> can subsequently use those folios to replay recorded data with the initiating tool. <command>pmchart</command> uses these concepts when providing its Record mode functionality.</para>
<section id="LE13213-PARENT">

<title><command>pmRecordAddHost</command> Function</title>
<literallayout class="monospaced">int pmRecordAddHost(const char *<replaceable>host</replaceable>, int <replaceable>isdefault</replaceable>, pmRecordHost **<replaceable>rhp</replaceable>)
<command>Python:</command>
(int <replaceable>status</replaceable>, pmRecordHost* <replaceable>rhp</replaceable>) = pmRecordAddHost("host string", 1, "configure string")</literallayout>
<para><indexterm id="IG31340177394"><primary>pmRecordAddHost function</primary></indexterm><indexterm id="IG31340177395"><primary>PMAPI</primary><secondary>record-mode services</secondary></indexterm><indexterm id="IG31340177396"><primary>record-mode services</primary></indexterm>The <command>pmRecordAddHost</command> function adds hosts once <command>pmRecordSetup</command> has established a new recording session. The <command>pmRecordAddHost</command> function along with the <command>pmRecordSetup</command> and <command>pmRecordControl</command> functions are used to create a PCP archive.</para>
<para><command>pmRecordAddHost</command> is called for each host that is to be included in the recording session. A new <filename>pmRecordHost</filename> structure is returned via <replaceable>rhp</replaceable>. It is assumed that PMCD is running on the host as this is how <command>pmlogger</command> retrieves the required performance metrics.</para>
<para>If this host is the default host for the recording session, <replaceable>isdefault</replaceable> is nonzero. This ensures that the corresponding archive appears first in the PCP archive <replaceable>folio</replaceable>. Hence the tools used to replay the archive <replaceable>folio</replaceable> make the correct determination of the archive associated with the default host. At most one host per recording session may be nominated as the default host.</para>
<para>The calling application writes the desired <command>pmlogger</command> configuration onto the stdio stream returned via the <literal>f_config</literal> field in the <filename>pmRecordHost</filename> structure.</para>
<para><command>pmRecordAddHost</command> returns 0 on success and a value less than 0 suitable for decoding with <command>pmErrStr</command> on failure. The value <literal>EINVAL</literal> has the same interpretation as <literal>errno</literal> being set to <literal>EINVAL</literal>.</para>
</section>
<section id="LE35969-PARENT">

<title><command>pmRecordControl</command> Function</title>
<literallayout class="monospaced">int pmRecordControl(pmRecordHost *<replaceable>rhp</replaceable>, int <replaceable>request</replaceable>, const char *<replaceable>options</replaceable>)
<command>Python:</command>
int <replaceable>status</replaceable> = pmRecordControl("host string", 1, "configure string")</literallayout>
<para><indexterm id="IG31340177397"><primary>pmRecordControl function</primary></indexterm>Arguments may be optionally added to the command line that is used to launch <command>pmlogger</command> by calling the <command>pmRecordControl</command> function with a request of <literal>PM_REC_SETARG</literal>. The <command>pmRecordControl</command> along with the <command>pmRecordSetup</command> and <command>pmRecordAddHost</command> functions are used to create a PCP archive.</para>
<para>The argument is passed via <replaceable>options</replaceable> and one call to <command>pmRecordControl</command> is required for each distinct argument. An argument may be added for a particular <command>pmlogger</command> instance identified by <replaceable>rhp</replaceable>. If the <replaceable>rhp</replaceable> argument is NULL, the argument is added for all <command>pmlogger</command> instances that are launched in the current recording session.</para>
<para>Independent of any calls to <command>pmRecordControl</command> with a request of <literal>PM_REC_SETARG</literal>, each <command>pmlogger</command> instance is automatically launched with the following arguments: <literal>-c</literal>, <literal>-h</literal>, <literal>-l</literal>, <literal>-x</literal>, and the basename for the PCP archive log.</para>
<para>To commence the recording session, call <command>pmRecordControl</command> with a request of <literal>PM_REC_ON</literal>, and <replaceable>rhp</replaceable> must be NULL. This launches one <command>pmlogger</command> process for each host in the recording session and initializes the <literal>fd_ipc</literal>, <literal>logfile</literal>, <literal>pid</literal>, and <literal>status</literal> fields in the associated <filename>pmRecordHost</filename> structure(s).</para>
<para>To terminate a <command>pmlogger</command> instance identified by <replaceable>rhp</replaceable>, call <command>pmRecordControl</command> with a request of <literal>PM_REC_OFF</literal>. If the <replaceable>rhp</replaceable> argument to <command>pmRecordControl</command> is NULL, the termination request is broadcast to all <command>pmlogger</command> processes in the current recording session. An informative dialogue is generated directly by each <command>pmlogger</command> process.</para>
<para>To display the current status of the <command>pmlogger</command> instance identified by <replaceable>rhp</replaceable>, call <command>pmRecordControl</command> with a request of <literal>PM_REC_STATUS</literal>. If the <replaceable>rhp</replaceable> argument to <command>pmRecordControl</command> is NULL, the status request is broadcast to all <command>pmlogger</command> processes in the current recording session. The display is generated directly by each <command>pmlogger</command> process.</para>
<para>To detach a <command>pmlogger</command> instance identified by <replaceable>rhp</replaceable>, allow it to continue independent of the application that launched the recording session and call <command>pmRecordControl</command> with a request of <literal>PM_REC_DETACH</literal>. If the <replaceable>rhp</replaceable> argument to <command>pmRecordControl</command> is NULL, the detach request is broadcast to all <command>pmlogger</command> processes in the current recording session.</para>
<para><command>pmRecordControl</command> returns 0 on success and a value less than 0 suitable for decoding with <command>pmErrStr</command> on failure. The value <literal>EINVAL</literal> has the same interpretation as <literal>errno</literal> being set to <literal>EINVAL</literal>.</para>
<para><command>pmRecordControl</command> returns <literal>PM_ERR_IPC</literal> if the associated <command>pmlogger</command> process has already exited.</para>
</section>
<section id="id5206798">

<title><command>pmRecordSetup</command> Function</title>
<literallayout class="monospaced">FILE *pmRecordSetup(const char *<replaceable>folio</replaceable>, const char *<replaceable>creator</replaceable>, int <replaceable>replay</replaceable>)
<command>Python:</command>
int <replaceable>status</replaceable> = pmRecordSetup("folio string", "creator string", int replay)</literallayout>
<para><indexterm id="IG31340177398"><primary>pmRecordSetup function</primary></indexterm>The <command>pmRecordSetup</command> function along with the <command>pmRecordAddHost</command> and <command>pmRecordControl</command> functions may be used to create a PCP archive on the fly to support record-mode services for PMAPI client applications.</para>
<para>Each record mode session involves one or more PCP archive logs; each is created using a dedicated <command>pmlogger</command> process, with an overall Archive Folio format as understood by the <command>pmafm</command> command, to name and collect all of the archive logs associated with a single recording session.</para>
<para>The <filename>pmRecordHost</filename> structure is used to maintain state information between the creator of the recording session and the associated <command>pmlogger</command> process(es). The structure, shown in <xref linkend="Z976560662sdc"/>, is defined as:</para>
<example id="Z976560662sdc">
<title><filename>pmRecordHost</filename> Structure</title>
<programlisting>typedef struct {
    FILE   *f_config;    /* caller writes pmlogger configuration here */
    int    fd_ipc;       /* IPC channel to pmlogger */
    char   *logfile;     /* full pathname for pmlogger error logfile */
    pid_t  pid;          /* process id for pmlogger */
    int    status;       /* exit status, -1 if unknown */
} pmRecordHost;</programlisting>
</example>
<para>In <xref linkend="id5206969"/>, the functions are used in combination to create a recording session.</para>
<procedure id="id5206969">
<title>Creating a Recording Session</title>
<step><para>Call <command>pmRecordSetup</command> to establish a new recording session. A new Archive Folio is created using the name <replaceable>folio</replaceable>. If the <replaceable>folio</replaceable> file or directory already exists, or if it cannot be created, this is an error. The application that is creating the session is identified by creator (most often this would be the same as the global PMAPI application name, <literal>pmProgname</literal>). If the application knows how to create its own configuration file to replay the recorded session, replay should be nonzero. The <command>pmRecordSetup</command> function returns a stdio stream onto which the application writes the text of any required replay configuration file.</para></step>
<step><para>For each host that is to be included in the recording session, call <command>pmRecordAddHost</command>. A new <filename>pmRecordHost</filename> structure is returned via <replaceable>rhp</replaceable>. It is assumed that PMCD is running on the host as this is how <command>pmlogger</command> retrieves the required performance metrics. See <xref linkend="LE13213-PARENT"/> for more information.</para></step>
<step><para>Optionally, add arguments to the command line that is used to launch <command>pmlogger</command> by calling <command>pmRecordControl</command> with a request of <literal>PM_REC_SETARG</literal>. The argument is passed via options and one call to <command>pmRecordControl</command> is required for each distinct argument. See <xref linkend="LE35969-PARENT"/> for more information.</para></step>
<step><para>To commence the recording session, call <command>pmRecordControl</command> with a request of <literal>PM_REC_ON</literal>, and <replaceable>rhp</replaceable> must be NULL.</para></step>
<step><para>To terminate a <command>pmlogger</command> instance identified by <replaceable>rhp</replaceable>, call <command>pmRecordControl</command> with a request of <literal>PM_REC_OFF</literal>.</para></step>
<step><para>To display the current status of the <command>pmlogger</command> instance identified by <replaceable>rhp</replaceable>, call <command>pmRecordControl</command> with a request of <literal>PM_REC_STATUS</literal>.</para></step>
<step><para>To detach a <command>pmlogger</command> instance identified by <replaceable>rhp</replaceable>, allow it to continue independent of the application that launched the recording session, call <command>pmRecordControl</command> with a request of <literal>PM_REC_DETACH</literal>.</para></step>
</procedure>
<para>The calling application should not close any of the returned stdio streams; <command>pmRecordControl</command> performs this task when recording is commenced.</para>
<para>Once <command>pmlogger</command> has been started for a recording session, <command>pmlogger</command> assumes responsibility for any dialogue with the user in the event that the application that launched the recording session should exit, particularly without terminating the recording session.</para>
<para>By default, information and dialogues from <command>pmlogger</command> is displayed using <command>pmconfirm</command>. This default is based on the assumption that most applications launching a recording session are GUI-based. In the event that <command>pmconfirm</command> fails to display the information (for example, because the <literal>DISPLAY</literal> environment variable is not set), <command>pmlogger</command> writes on its own stderr stream (not the stderr stream of the launching process). The output is assigned to the <filename><replaceable>xxxxxx</replaceable>.host.log</filename> file. For convenience, the full pathname to this file is provided via the <literal>logfile</literal> field in the <filename>pmRecordHost</filename> structure.</para>
<para>If the <replaceable>options</replaceable> argument to <command>pmRecordControl</command> is not NULL, this string may be used to pass additional arguments to <command>pmconfirm</command> in those cases where a dialogue is to be displayed. One use of this capability is to provide a -geometry string to control the placement of the dialogue.</para>
<para>Premature termination of a launched <command>pmlogger</command> process may be determined using the <filename>pmRecordHost</filename> structure, by calling <command>select</command> on the <literal>fd_ipc</literal> field or polling the <literal>status</literal> field that will contain the termination status from <command>waitpid</command> if known, or -1.</para>
<para>These functions create a number of files in the same directory as the <replaceable>folio</replaceable> file named in the call to <command>pmRecordSetup</command>. In all cases, the <replaceable>xxxxxx</replaceable> component is the result of calling <command>mkstemp</command>.</para>
<itemizedlist>
<listitem><para>If replay is nonzero, <replaceable>xxxxxx</replaceable> is the creator's replay configuration file, else an empty control file, used to guarantee uniqueness.</para>
</listitem>
<listitem><para>The <replaceable>folio</replaceable> file is the PCP Archive Folio, suitable for use with the <command>pmafm</command> command.</para>
</listitem>
<listitem><para>The <filename><replaceable>xxxxxx</replaceable>.host.confi</filename>g file is the <command>pmlogger</command> configuration for each host. If the same host is used in different calls to <command>pmRecordAddHost</command> within the same recording session, one of the letters 'a' through 'z' is appended to the <replaceable>xxxxxx</replaceable> part of all associated file names to ensure uniqueness.</para>
</listitem>
<listitem><para><filename><replaceable>xxxxxx</replaceable>.host.log</filename> is stdout and stderr for the <command>pmlogger</command> instance for each host.</para>
</listitem>
<listitem><para>The <filename><replaceable>xxxxxx</replaceable>.host.{0,meta,index}</filename> files comprise a single PCP archive for each host.</para>
</listitem></itemizedlist>
<para><command>pmRecordSetup</command> may return NULL in the event of an error. Check <literal>errno</literal> for the real cause. The value <literal>EINVAL</literal> typically means that the order of calls to these functions is not correct; that is, there is an obvious state associated with the current recording session that is maintained across calls to the functions.</para>
<para>For example, calling <command>pmRecordControl</command> before calling <command>pmRecordAddHost</command> at least once, or calling <command>pmRecordAddHost</command> before calling <command>pmRecordSetup</command> would produce an <literal>EINVAL</literal> error.</para>
</section>
</section>
<section id="LE85604-PARENT">

<title>PMAPI Archive-Specific Services</title>
<para>The functions described in this section provide archive-specific services.</para>
<section id="id5207510">

<title><command>pmGetArchiveLabel</command> Function</title>
<programlisting>int pmGetArchiveLabel(pmLogLabel *<replaceable>lp</replaceable>)
<command>Python:</command>
pmLogLabel <replaceable>loglabel</replaceable> = pmGetArchiveLabel()</programlisting>
<para><indexterm id="IG31340177399"><primary>pmGetArchiveLabel function</primary></indexterm><indexterm id="IG31340177400"><primary>PMAPI</primary><secondary>archive-specific services</secondary></indexterm><indexterm id="IG31340177401"><primary>archive-specific services</primary></indexterm>Provided the current PMAPI context is associated with a PCP archive log, the <command>pmGetArchiveLabel</command> function may be used to fetch the label record from the archive. The structure returned through <replaceable>lp</replaceable> is as shown in <xref linkend="Z976561683sdc"/>:</para>
<example id="Z976561683sdc">
<title><filename>pmLogLabel</filename> Structure</title>
<programlisting>/*
 * Label Record at the start of every log file - as exported above the PMAPI ...
 */
#define PM_TZ_MAXLEN    40
#define PM_LOG_MAXHOSTLEN   64
#define PM_LOG_MAGIC    0x50052600
#define PM_LOG_VERS01   0x1
#define PM_LOG_VERS02   0x2
#define PM_LOG_VOL_TI   -2      /* temporal index */
#define PM_LOG_VOL_META -1      /* meta data */
typedef struct {
    int            ll_magic;          /* PM_LOG_MAGIC | log format version no. */
    pid_t          ll_pid;            /* PID of logger */
    struct timeval ll_start;          /* start of this log */
    char           ll_hostname[PM_LOG_MAXHOSTLEN]; /* name of collection host */
    char           ll_tz[PM_TZ_MAXLEN];            /* $TZ at collection host */
} pmLogLabel;</programlisting>
</example>
<para>
The python bindings get the label record from the archive.  
</para>
</section>
<section id="id5207673">

<title><command>pmGetArchiveEnd</command> Function</title>
<programlisting>int pmGetArchiveEnd(struct timeval *<replaceable>tvp</replaceable>)
<command>Python:</command>
timeval <replaceable>tv</replaceable> = <replaceable>status</replaceable> = pmGetArchiveEnd()</programlisting>
<para><indexterm id="IG31340177402"><primary>archive logs</primary><secondary>pmGetArchiveEnd function</secondary></indexterm><indexterm id="IG31340177403"><primary>pmGetArchiveEnd function</primary></indexterm><indexterm id="IG31340177404"><primary>pmSetMode function</primary></indexterm>Provided the current PMAPI context is associated with a PCP archive log, <command>pmGetArchiveEnd</command> finds the logical end of file (after the last complete record in the archive), and returns the last recorded time stamp with <replaceable>tvp</replaceable>. This timestamp may be passed to <command>pmSetMode</command> to reliably position the context at the last valid log record, for example, in preparation for subsequent reading in reverse chronological order.</para>
<para>For archive logs that are not concurrently being written, the physical end of file and the logical end of file are co-incident. However, if an archive log is being written by <command>pmlogger</command> at the same time that an application is trying to read the archive, the logical end of file may be before the physical end of file due to write buffering that is not aligned with the logical record boundaries.</para>
<para>
  The python bindings get the last recorded timestamp from the archive.  
</para>
</section>
<section id="id5207751">

<title><command>pmGetInDomArchive</command> Function</title>
<programlisting>int pmGetInDomArchive(pmInDom <replaceable>indom</replaceable>, int **<replaceable>instlist</replaceable>, char ***<replaceable>namelist</replaceable> )
<command>Python:</command>
((instance1, instance2...) (name1, name2...)) pmGetInDom(pmDesc <replaceable>pmdesc</replaceable>)</programlisting>
<para><indexterm id="IG31340177405"><primary>archive logs</primary><secondary>pmGetInDomArchive function</secondary></indexterm><indexterm id="IG31340177406"><primary>indom instance domain</primary></indexterm><indexterm id="IG31340177407"><primary>pmGetInDomArchive function</primary></indexterm>Provided the current PMAPI context is associated with a PCP archive log, <command>pmGetInDomArchive</command> scans the metadata to generate the union of all instances for the instance domain <replaceable>indom</replaceable> that can be found in the archive log, and returns through <replaceable>instlist</replaceable> the internal instance identifiers, and through <replaceable>namelist</replaceable> the full external identifiers.</para>
<para><indexterm id="IG31340177408"><primary>pmGetInDom function</primary></indexterm>This function is a specialized version of the more general PMAPI function <command>pmGetInDom</command>.</para>
<para>The function returns the number of instances found (a value less than zero indicates an error).</para>
<para>The resulting lists of instance identifiers (<replaceable>instlist</replaceable> and <replaceable>namelist</replaceable>), and the names that the elements of <replaceable>namelist</replaceable> point to, are allocated by <command>pmGetInDomArchive</command> with two calls to <command>malloc</command>, and it is the responsibility of the caller to use <command>free</command><replaceable>(instlist)</replaceable> and <command>free</command><replaceable>(namelist)</replaceable> to release the space when it is no longer required; see the <command>malloc(3)</command> and <command>free(3)</command> man pages.</para>
<para>When the result of <command>pmGetInDomArchive</command> is less than one, both <replaceable>instlist</replaceable> and <replaceable>namelist</replaceable> are undefined (no space is allocated; so calling <command>free</command> is a singularly bad idea).</para>
<para>
  The python bindings return a tuple of the instance IDs and names for the union of all instances for the instance domain <replaceable>pmdesc</replaceable> that can be found in the archive log.  
</para>
</section>
<section id="id5207937">

<title><command>pmLookupInDomArchive</command> Function</title>
<literallayout class="monospaced">int pmLookupInDomArchive(pmInDom <replaceable>indom</replaceable>, const char *<replaceable>name</replaceable>)
<command>Python:</command>
c_uint <replaceable>instid</replaceable> = pmLookupInDomArchive(pmDesc <replaceable>pmdesc</replaceable>, "Instance")</literallayout>
<para><indexterm id="IG31340177409"><primary>pmLookupInDomArchive function</primary></indexterm>Provided the current PMAPI context is associated with a PCP archive log, <command>pmLookupInDomArchive</command> scans the metadata for the instance domain <replaceable>indom</replaceable>, locates the first instance with the external identification given by <replaceable>name</replaceable>, and returns the internal instance identifier.</para>
<para>This function is a specialized version of the more general PMAPI function <command>pmLookupInDom</command>.</para>
<para>The <command>pmLookupInDomArchive</command> function returns a positive instance identifier on success.</para>
<para>
  The python bindings return the instance id in <replaceable>pmdesc</replaceable> corresponding to <replaceable>Instance</replaceable>.  
</para>
</section>
<section id="id5208032">

<title><command>pmNameInDomArchive</command> Function</title>
<literallayout class="monospaced">int pmNameInDomArchive(pmInDom <replaceable>indom</replaceable>, int <replaceable>inst</replaceable>, char **<replaceable>name</replaceable>)
<command>Python:</command>
"instance id" = pmNameInDomArchive(pmDesc <replaceable>pmdesc</replaceable>, c_uint <replaceable>instid</replaceable>)</literallayout>
<para><indexterm id="IG31340177410"><primary>pmNameInDomArchive function</primary></indexterm>Provided the current PMAPI context is associated with a PCP archive log, <command>pmNameInDomArchive</command> scans the metadata for the instance domain <replaceable>indom</replaceable>, locates the first instance with the internal instance identifier given by <command>inst</command>, and returns the full external instance identification through <replaceable>name</replaceable>. This function is a specialized version of the more general PMAPI function <command>pmNameInDom</command>.</para>
<para>The space for the value of <replaceable>name</replaceable> is allocated in <command>pmNameInDomArchive</command> with <command>malloc</command>, and it is the responsibility of the caller to free the space when it is no longer required; see the <command>malloc(3)</command> and<command>free(3)</command> man pages.</para>
<para>
  The python bindings return the text name of an instance corresponding to an instance domain <replaceable>pmdesc</replaceable> with instance identifier <replaceable>instid</replaceable>.  
</para>
</section>
<section id="id5208144">

<title><command>pmFetchArchive</command> Function</title>
<literallayout class="monospaced">int pmFetchArchive(pmResult **<replaceable>result</replaceable>)
<command>Python:</command>
pmResult* <replaceable>pmresult</replaceable> = pmFetchArchive()</literallayout>
<para><indexterm id="IG31340177411"><primary>pmFetch function</primary></indexterm><indexterm id="IG31340177412"><primary>pmFetchArchive function</primary></indexterm>This is a variant of <command>pmFetch</command> that may be used only when the current PMAPI context is associated with a PCP archive log. The <replaceable>result</replaceable> is instantiated with all of the metrics (and instances) from the next archive record; consequently, there is no notion of a list of desired metrics, and the instance profile is ignored.</para>
<para>It is expected that <command>pmFetchArchive</command> would be used to create utilities that scan archive logs (for example, <command>pmdumplog</command> and <command>pmlogsummary</command>), and the more common access to the archives would be through the <command>pmFetch</command> interface.</para>
</section>
</section>
<section id="LE73955-PARENT">

<title>PMAPI Time Control Services</title>
<para><indexterm id="IG31340177413"><primary>PMAPI</primary><secondary>time control services</secondary></indexterm><indexterm id="IG31340177414"><primary>time control services</primary></indexterm>The PMAPI provides a common framework for client applications to control time and to synchronize time with other applications. The user interface component of this service is fully described in the companion <citetitle>Performance Co-Pilot User's and Administrator's Guide</citetitle>. See also the <command>pmtime(1)</command> man page.</para>
<para><indexterm id="IG31340177415"><primary>archive logs</primary><secondary>time control services</secondary></indexterm>This service is most useful when processing PCP archive logs, to control parameters such as the current archive position, update interval, replay rate, and timezone, but it can also be used in live mode to control a subset of these parameters. Applications such as <command>pmchart</command>, <command>pmgadgets</command>, <command>pmstat</command>, and <command>pmval</command> use the time control services to connect to an instance of the time control server process, <command>pmtime</command>, which provides a uniform graphical user interface to the time control services.</para>
<para><indexterm id="IG31340177416"><primary>examples</primary><secondary>time control functions </secondary></indexterm>A full description of the PMAPI time control functions along with code examples can be found in man pages as listed in <xref linkend="id5208351"/>:</para>
<table id="id5208351" frame="topbot" pgwide="wide">

<title>Time Control Functions in PMAPI</title>
<tgroup cols="2" colsep="0" rowsep="0">
<colspec colwidth="137*"/>
<colspec colwidth="259*"/>
<thead>
<row rowsep="1" valign="top"><entry align="left" valign="bottom"><para>Man Page</para></entry><entry align="left" valign="bottom"><para>Synopsis of Time Control Function</para></entry></row></thead>
<tbody>
<row valign="top">
<entry align="left" valign="top"><para><command>pmCtime(3)</command></para></entry>
<entry align="left" valign="top"><para>Formats the date and time for a reporting timezone.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmLocaltime(3)</command></para></entry>
<entry align="left" valign="top"><para>Converts the date and time for a reporting timezone.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmParseTimeWindow(3)</command></para></entry>
<entry align="left" valign="top"><para>Parses time window command line arguments.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmTimeConnect(3)</command></para></entry>
<entry align="left" valign="top"><para>Connects to a time control server via a command socket.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmTimeDisconnect(3)</command></para></entry>
<entry align="left" valign="top"><para>Closes the command socket to the time control server.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmTimeGetPort(3)</command></para></entry>
<entry align="left" valign="top"><para>Obtains the port name of the current time control server.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmTimeRecv(3)</command></para></entry>
<entry align="left" valign="top"><para>Blocks until the time control server sends a command message.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmTimeSendAck(3)</command></para></entry>
<entry align="left" valign="top"><para>Acknowledges completion of the step command.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmTimeSendBounds(3)</command></para></entry>
<entry align="left" valign="top"><para>Specifies beginning and end of archive time period.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmTimeSendMode(3)</command></para></entry>
<entry align="left" valign="top"><para>Requests time control server to change to a new VCR mode.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmTimeSendPosition(3)</command></para></entry>
<entry align="left" valign="top"><para>Requests time control server to change position or update intervals.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmTimeSendTimezone(3)</command></para></entry>
<entry align="left" valign="top"><para>Requests time control server to change timezones.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmTimeShowDialog(3)</command></para></entry>
<entry align="left" valign="top"><para>Changes the visibility of the time control dialogue.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><command>pmTimeGetStatePixmap(3)</command></para></entry>
<entry align="left" valign="top"><para>Returns array of pixmaps representing supplied time control state.</para></entry></row></tbody></tgroup></table>
</section>
<section id="LE44064-PARENT">

<title>PMAPI Ancillary Support Services</title>
<para><indexterm id="IG31340177417"><primary>ancillary support services</primary></indexterm><indexterm id="IG31340177418"><primary>PMAPI</primary><secondary>ancillary support services</secondary></indexterm>The functions described in this section provide services that are complementary to, but not necessarily a part of, the distributed manipulation of performance metrics delivered by the PCP components.</para>
<section id="id5208965">

<title><command>pmGetConfig</command> Function</title>
<literallayout class="monospaced">char *pmGetConfig(const char <replaceable>*variable</replaceable>)
<command>Python:</command>
"env variable value = pmGetConfig("env variable")</literallayout>
<para>The <command>pmGetConfig</command> function searches for a variable first in the environment and then, if one is not found, in the PCP configuration file and returns the string result. If a variable is not already in the environment, it is added with a call to the <command>putenv</command> function before returning.</para>
<para>The default location of the PCP configuration file is <filename>/etc/pcp.conf</filename>, but this location may be changed by setting <literal>PCP_CONF</literal> in the environment to a new location, as described in the <command>pcp.conf(5)</command> man page.</para>
<para>If the variable is not found in either the environment or the PCP configuration file (or the PCP configuration file is not found and <literal>PCP_CONF</literal> is not set in the environment), then a fatal error message is printed and the process will exit. Although this sounds drastic, it is the only course of action available because the PCP configuration or installation is fatally flawed.</para>
<para>If this function returns, the returned value points to a string in the environment; and so although the function returns the same type as the <command>getenv</command> function (which should probably be a <literal>const char *</literal>), changing the content of the string is not recommended.</para>
<para>
The python bindings return a value for environment variable <replaceable>"env variable"</replaceable> from environment or pcp config file.
</para>
</section>
<section id="id5209050">

<title><command>pmErrStr</command> Function</title>
<literallayout class="monospaced">const char *pmErrStr(int <replaceable>code</replaceable>)
char *pmErrStr_r(int <replaceable>code</replaceable>, char *<replaceable>buf</replaceable>, int <replaceable>buflen</replaceable>);
<command>Python:</command>
"error string text" = pmErrStr(int <replaceable>error_code</replaceable>)</literallayout>
<para><indexterm id="IG31340177419"><primary>pmErrStr function</primary></indexterm>This function translates an error code into a text string, suitable for generating a diagnostic message. By convention within PCP, all error codes are negative. The small values are assumed to be negated versions of the platform error codes as defined in <filename>errno.h</filename>, and the strings returned are according to <command>strerror</command>. The large, negative error codes are PMAPI error conditions, and <command>pmErrStr</command> returns an appropriate PMAPI error string, as determined by <replaceable>code</replaceable>.</para>
<para>In the case of <command>pmErrStr</command>, the string value is held in a single static buffer, so concurrent calls may not produce the desired results. The <command>pmErrStr_r</command> function allows a buffer and length to be passed in, into which the message is stored; this variant uses no shared storage and can be used in a thread-safe manner.</para>
<para>
The python bindings return the error string corresponding to the <replaceable>error code</replaceable>.  
</para>
</section>
<section id="id5209112">

<title><command>pmExtractValue</command> Function</title>
<literallayout class="monospaced">int pmExtractValue(int <replaceable>valfmt</replaceable>, const pmValue *<replaceable>ival</replaceable>, int <replaceable>itype</replaceable>,
pmAtomValue *<replaceable>oval</replaceable>, int <replaceable>otype</replaceable>)
<command>Python:</command>
pmAtomValue <replaceable>atomval</replaceable> = pmExtractValue(int <replaceable>valfmt</replaceable>, const pmValue * <replaceable>ival</replaceable>,
	  int <replaceable>itype</replaceable>,
	  pmAtomValue *<replaceable>oval</replaceable>,
	  int <replaceable>otype</replaceable>)</literallayout>
<para><indexterm id="IG31340177420"><primary>pmExtractValue function</primary></indexterm>The <command>pmValue</command> structure is embedded within the <filename>pmResult</filename> structure, which is used to return one or more performance metrics; see the <command>pmFetch</command> man page.</para>
<para>All performance metric values may be encoded in a <filename>pmAtomValue</filename> union, defined in <xref linkend="Z976562908sdc"/>:</para>
<example id="Z976562908sdc">
<title><filename>pmAtomValue</filename> Structure</title>
<programlisting>/* Generic Union for Value-Type conversions */
typedef union {
    __int32_t  l;     /* 32-bit signed */
    __uint32_t ul;    /* 32-bit unsigned */
    __int64_t  ll;    /* 64-bit signed */
    __uint64_t ull;   /* 64-bit unsigned */
    float      f;     /* 32-bit floating point */
    double     d;     /* 64-bit floating point */
    char       *cp;   /* char ptr */
    void       *vp;   /* void ptr */
} pmAtomValue;</programlisting>
</example>
<para>The <command>pmExtractValue</command> function provides a convenient mechanism for extracting values from the <command>pmValue</command> part of a <filename>pmResult</filename> structure, optionally converting the data type, and making the result available to the application programmer.</para>
<para><indexterm id="IG31340177421"><primary>pmLookupDesc function</primary></indexterm>The <replaceable>itype</replaceable> argument defines the data type of the input value held in <replaceable>ival</replaceable> according to the storage format defined by <replaceable>valfmt</replaceable> (see the <command>pmFetch</command> man page). The <replaceable>otype</replaceable> argument defines the data type of the result to be placed in <replaceable>oval</replaceable>. The value for <replaceable>itype</replaceable> is typically extracted from a <filename>pmDesc</filename> structure, following a call to  <command>pmLookupDesc</command> for a particular performance metric.</para>
<para><xref linkend="id5209524"/> defines the various possibilities for the type conversion. The input type (<replaceable>itype</replaceable>) is shown vertically, and the output type (<replaceable>otype</replaceable>) horizontally. The following rules apply:</para>
<itemizedlist>
<listitem><para>Y means the conversion is always acceptable.</para>
</listitem>
<listitem><para><indexterm id="IG31340177422"><primary>PM_ERR_CONV error code</primary></indexterm>N means conversion can never be performed (function returns <literal>PM_ERR_CONV</literal>).</para>
</listitem>
<listitem><para>P means the conversion may lose accuracy (but no error status is returned).</para>
</listitem>
<listitem><para><indexterm id="IG31340177423"><primary>PM_ERR_TRUNC error code</primary></indexterm>T means the result may be subject to high-order truncation (if this occurs the function returns <literal>PM_ERR_TRUNC</literal>).</para>
</listitem>
<listitem><para><indexterm id="IG31340177424"><primary>PM_ERR_SIGN error code</primary></indexterm>S means the conversion may be impossible due to the sign of the input value (if this occurs the function returns <literal>PM_ERR_SIGN</literal>).</para>
</listitem></itemizedlist>
<para><indexterm id="IG31340177425"><primary>PM_TYPE_STRING type</primary></indexterm>If an error occurs, <replaceable>oval</replaceable> is set to zero (or NULL).</para>
<para><note><para> Note that some of the conversions involving the <literal>PM_TYPE_STRING</literal> and <literal>PM_TYPE_AGGREGATE</literal> types are indeed possible, but are marked N; the rationale is that <command>pmExtractValue</command> should not attempt to duplicate functionality already available in the C library through <command>sscanf</command> and <command>sprintf</command>. No conversion involving the type <literal>PM_TYPE_EVENT</literal> is supported.</para>
</note></para>
<table id="id5209524" frame="topbot">

<title>PMAPI Type Conversion</title>
<tgroup cols="10" colsep="0" rowsep="0">
<thead>
<row rowsep="1" valign="top"><entry align="left" valign="bottom"><para>TYPE</para></entry><entry align="left" valign="bottom"><para>32</para></entry><entry align="left" valign="bottom"><para>U32</para></entry><entry align="left" valign="bottom"><para>64</para></entry><entry align="left" valign="bottom"><para>U64</para></entry><entry align="left" valign="bottom"><para>FLOAT</para></entry><entry align="left" valign="bottom"><para>DBLE</para></entry><entry align="left" valign="bottom"><para>STRING</para></entry><entry align="left" valign="bottom"><para>AGGR</para></entry><entry align="left" valign="bottom"><para>EVENT</para></entry></row></thead>
<tbody>
<row valign="top">
<entry align="left" valign="top"><para><literal>32</literal></para></entry>
<entry align="left" valign="top"><para>Y</para></entry>
<entry align="left" valign="top"><para>S</para></entry>
<entry align="left" valign="top"><para>Y</para></entry>
<entry align="left" valign="top"><para>S</para></entry>
<entry align="left" valign="top"><para>P</para></entry>
<entry align="left" valign="top"><para>P</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>U32</literal></para></entry>
<entry align="left" valign="top"><para>T</para></entry>
<entry align="left" valign="top"><para>Y</para></entry>
<entry align="left" valign="top"><para>Y</para></entry>
<entry align="left" valign="top"><para>Y</para></entry>
<entry align="left" valign="top"><para>P</para></entry>
<entry align="left" valign="top"><para>P</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>64</literal></para></entry>
<entry align="left" valign="top"><para>T</para></entry>
<entry align="left" valign="top"><para>T,S</para></entry>
<entry align="left" valign="top"><para>Y</para></entry>
<entry align="left" valign="top"><para>S</para></entry>
<entry align="left" valign="top"><para>P</para></entry>
<entry align="left" valign="top"><para>P</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>u64</literal></para></entry>
<entry align="left" valign="top"><para>T</para></entry>
<entry align="left" valign="top"><para>T</para></entry>
<entry align="left" valign="top"><para>T</para></entry>
<entry align="left" valign="top"><para>Y</para></entry>
<entry align="left" valign="top"><para>P</para></entry>
<entry align="left" valign="top"><para>P</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>FLOAT</literal></para></entry>
<entry align="left" valign="top"><para>P, T</para></entry>
<entry align="left" valign="top"><para>P, T, S</para></entry>
<entry align="left" valign="top"><para>P, T</para></entry>
<entry align="left" valign="top"><para>P, T, S</para></entry>
<entry align="left" valign="top"><para>Y</para></entry>
<entry align="left" valign="top"><para>Y</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>DBLE</literal></para></entry>
<entry align="left" valign="top"><para>P, T</para></entry>
<entry align="left" valign="top"><para>P, T, S</para></entry>
<entry align="left" valign="top"><para>P, T</para></entry>
<entry align="left" valign="top"><para>P, T, S</para></entry>
<entry align="left" valign="top"><para>P</para></entry>
<entry align="left" valign="top"><para>Y</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>STRING</literal></para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>Y</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>AGGR</literal></para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>Y</para></entry>
<entry align="left" valign="top"><para>N</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>EVENT</literal></para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry>
<entry align="left" valign="top"><para>N</para></entry></row></tbody></tgroup></table>
<para>In the cases where multiple conversion errors could occur, the first encountered error is returned, and the order of checking is not defined.</para>
<para>If the output conversion is to one of the pointer types, such as <replaceable>otype</replaceable> <literal>PM_TYPE_STRING</literal> or <literal>PM_TYPE_AGGREGATE</literal>, then the value buffer is allocated by <command>pmExtractValue</command> using <command>malloc</command>, and it is the caller's responsibility to free the space when it is no longer required; see the <command>malloc(3)</command> and <command>free(3)</command> man pages.</para>
<para>Although this function appears rather complex, it has been constructed to assist the development of performance tools that convert values, whose type is known only through the <literal>type</literal> field in a <command>pmDesc</command> structure, into a canonical type for local processing.</para>
<para>
The python bindings extract a value from a pmValue struct <replaceable>ival</replaceable> stored in format <replaceable>valfmt</replaceable> (see <command>pmFetch</command>),  and convert its type from <replaceable>itype</replaceable> to <replaceable>otype</replaceable>.  
</para>
</section>
<section id="id5210752">

<title><command>pmConvScale</command> Function</title>
<programlisting>int
pmConvScale(int <replaceable>type</replaceable>, const pmAtomValue *<replaceable>ival</replaceable>, const pmUnits *<replaceable>iunit</replaceable>,
pmAtomValue *<replaceable>oval</replaceable>, pmUnits *<replaceable>ounit</replaceable>)
<command>Python:</command>
pmAtomValue <replaceable>atomval</replaceable> = pmConvScale(int <replaceable>itype</replaceable>, pmAtomValue value,
	  pmDesc* pmdesc , int descidx, int otype)</programlisting>
<para><indexterm id="IG31340177426"><primary>pmConvScale function</primary></indexterm>Given a performance metric value pointed to by <replaceable>ival</replaceable>, multiply it by a scale factor and return the value in <replaceable>oval</replaceable>. The scaling takes place from the units defined by <replaceable>iunit</replaceable> into the units defined by <replaceable>ounit</replaceable>. Both input and output units must have the same dimensionality.</para>
<para><indexterm id="IG31340177427"><primary>pmLookupDesc function</primary></indexterm>The performance metric type for both input and output values is determined by <replaceable>type</replaceable>, the value for which is typically extracted from a <filename>pmDesc</filename> structure, following a call to <command>pmLookupDesc</command> for a particular performance metric.</para>
<para><indexterm id="IG31340177428"><primary>pmExtractValue function</primary></indexterm><command>pmConvScale</command> is most useful when values returned through <command>pmFetch</command> (and possibly extracted using <command>pmExtractValue</command>) need to be normalized into some canonical scale and units for the purposes of computation.</para>
<para>
The python bindings convert a <replaceable>value</replaceable> pointed to by <replaceable>pmdesc</replaceable> entry <replaceable>descidx</replaceable> to a different scale <replaceable>otype</replaceable>.  
</para>
</section>
<section id="id5210895">

<title><command>pmUnitsStr</command> Function</title>
<literallayout class="monospaced">const char *pmUnitsStr(const pmUnits *<replaceable>pu</replaceable>)
char *pmUnitsStr_r(const pmUnits *<replaceable>pu</replaceable>, char *<replaceable>buf</replaceable>, int <replaceable>buflen</replaceable>)
<command>Python:</command>
"units string" = pmUnitsStr(pmUnits <replaceable>pmunits</replaceable>)</literallayout>
<para><indexterm id="IG31340177429"><primary>pmUnitsStr function</primary></indexterm>As an aid to labeling graphs and tables, or for error messages, <command>pmUnitsStr</command> takes a dimension and scale specification as per <replaceable>pu</replaceable>, and returns the corresponding text string.</para>
<para><replaceable>pu</replaceable> is typically from a <filename>pmDesc</filename> structure, for example, as returned by <command>pmLookupDesc</command>.</para>
<para>If <replaceable>*pu</replaceable> were <literal>{1, -2, 0, PM_SPACE_MBYTE, PM_TIME_MSEC, 0}</literal>, then the result string would be <literal>Mbyte/sec^2</literal>.</para>
<para>In the case of <command>pmUnitsStr</command>, the string value is held in a single static buffer; so concurrent calls may not produce the desired results. The <command>pmUnitsStr_r</command> function allows a buffer and length to be passed in, into which the units are stored; this variant uses no shared storage and can be used in a thread-safe manner.</para>
<para>
The python bindings translate a pmUnits struct <replaceable>pmunits</replaceable> to a readable string.  
</para>
</section>
<section id="id5210994">

<title><command>pmIDStr</command> Function</title>
<literallayout class="monospaced">const char *pmIDStr(pmID <replaceable>pmid</replaceable>)
char *pmIDStr_r(pmID <replaceable>pmid</replaceable>, char *<replaceable>buf</replaceable>, int <replaceable>buflen</replaceable>)
<command>Python:</command>
"ID string" = pmIDStr(int <replaceable>pmID</replaceable>)</literallayout>
<para><indexterm id="IG31340177430"><primary>pmIDStr function</primary></indexterm>For use in error and diagnostic messages, return a human readable version of the specified PMID, with each of the internal <literal>domain</literal>, <literal>cluster</literal>, and <literal>item</literal> subfields appearing as decimal numbers, separated by periods.</para>
<para>In the case of <command>pmIDStr</command>, the string value is held in a single static buffer; so concurrent calls may not produce the desired results. The <command>pmIDStr_r</command> function allows a buffer and length to be passed in, into which the identifier is stored; this variant uses no shared storage and can be used in a thread-safe manner.</para>
<para>
The python bindings translate a pmID <replaceable>pmid</replaceable> to a readable string.  
</para>
</section>
<section id="id5211046">

<title><command>pmInDomStr</command> Function</title>
<literallayout class="monospaced">const char *pmInDomStr(pmInDom <replaceable>indom</replaceable>)
char *pmInDomStr_r(pmInDom <replaceable>indom</replaceable>, char *<replaceable>buf</replaceable>, int <replaceable>buflen</replaceable>)
<command>Python:</command>
"indom" = pmGetInDom(pmDesc <replaceable>pmdesc</replaceable>)</literallayout>
<para><indexterm id="IG31340177431"><primary>pmInDomStr function</primary></indexterm>For use in error and diagnostic messages, return a human readable version of the specified instance domain identifier, with each of the internal <literal>domain</literal> and <literal>serial</literal> subfields appearing as decimal numbers, separated by periods.</para>
<para>In the case of <command>pmInDomStrr</command>, the string value is held in a single static buffer; so concurrent calls may not produce the desired results. The <command>pmInDomStr_r</command> function allows a buffer and length to be passed in, into which the identifier is stored; this variant uses no shared storage and can be used in a thread-safe manner.</para>
<para>
The python bindings translate an instance domain ID pointed to by a pmDesc <replaceable>pmdesc</replaceable> to a readable string.  
</para>
</section>
<section id="id5211095">

<title><command>pmTypeStr</command> Function</title>
<literallayout class="monospaced">const char *pmTypeStr(int <replaceable>type</replaceable>)
char *pmTypeStr_r(int <replaceable>type</replaceable>, char *<replaceable>buf</replaceable>, int <replaceable>buflen</replaceable>)
<command>Python:</command>
"type" = pmTypeStr(int <replaceable>type</replaceable>)</literallayout>
<para><indexterm id="IG31340177432"><primary>pmTypeStr function</primary></indexterm>Given a performance metric type, produce a terse ASCII equivalent, appropriate for use in error and diagnostic messages.</para>
<para>Examples are &#x201C;32&#x201D; (for <literal>PM_TYPE_32</literal>), &#x201C;U64&#x201D; (for <literal>PM_TYPE_U64</literal>), &#x201C;AGGREGATE&#x201D; (for <literal>PM_TYPE_AGGREGATE</literal>), and so on.</para>
<para>In the case of <command>pmTypeStr</command>, the string value is held in a single static buffer; so concurrent calls may not produce the desired results. The <command>pmTypeStr_r</command> function allows a buffer and length to be passed in, into which the identifier is stored; this variant uses no shared storage and can be used in a thread-safe manner.</para>
<para>
The python bindings translate a performance metric type to a readable string.  Constants are available for the types, e.g. c_api.PM_TYPE_FLOAT, by importing cpmapi.  
</para>
</section>
<section id="id5211169">

<title><command>pmAtomStr</command> Function</title>
<literallayout class="monospaced">const char *pmAtomStr(const pmAtomValue *<replaceable>avp</replaceable>, int <replaceable>type</replaceable>)
char *pmAtomStr_r(const pmAtomValue *<replaceable>avp</replaceable>, int <replaceable>type</replaceable>char *<replaceable>buf</replaceable>, int <replaceable>buflen</replaceable>)
<command>Python:</command>
"value" = pmAtomStr(<replaceable>atom</replaceable>, <replaceable>type</replaceable>)</literallayout>
<para><indexterm id="IG31340177433"><primary>pmAtomStr function</primary></indexterm>Given the <command>pmAtomValue</command> identified by <replaceable>avp</replaceable>, and a performance metric <replaceable>type</replaceable>, generate the corresponding metric value as a string, suitable for diagnostic or report output.</para>
<para>In the case of <command>pmAtomStr</command>, the string value is held in a single static buffer; so concurrent calls may not produce the desired results. The <command>pmAtomStr_r</command> function allows a buffer and length to be passed in, into which the identifier is stored; this variant uses no shared storage and can be used in a thread-safe manner.</para>
<para>
The python bindings translate a pmAtomValue <replaceable>atom</replaceable> having performance metric <replaceable>type</replaceable> to a readable string.  Constants are available for the types, e.g. c_api.PM_TYPE_U32, by importing cpmapi.  
</para>
</section>
<section id="id5211303">

<title><command>pmNumberStr</command> Function</title>
<literallayout class="monospaced">const char *pmNumberStr(double <replaceable>value</replaceable>)
char *pmNumberStr_r(double <replaceable>value</replaceable>, char *<replaceable>buf</replaceable>, int <replaceable>buflen</replaceable>)</literallayout>
<para><indexterm id="IG31340177434"><primary>pmNumberStr function</primary></indexterm>The <command>pmNumberStr</command> function returns the address of a static 8-byte buffer that holds a null-byte terminated representation of value suitable for output with fixed-width fields.</para>
<para>The value is scaled using multipliers in powers of one thousand (the decimal kilo) and has a bias that provides greater precision for positive numbers as opposed to negative numbers. The format depends on the sign and magnitude of <replaceable>value</replaceable>.</para>
</section>
<section id="id5211350">

<title><command>pmPrintValue</command> Function</title>
<literallayout class="monospaced">void pmPrintValue(FILE *<replaceable>f</replaceable>, int <replaceable>valfmt</replaceable>, int <replaceable>type</replaceable>, const pmValue *<replaceable>val</replaceable>,
int <replaceable>minwidth</replaceable>)
<command>Python:</command>
pmPrintValue(FILE* <replaceable>file</replaceable>, pmResult <replaceable>pmresult</replaceable>, <replaceable>pmdesc</replaceable>, <replaceable>vset_index</replaceable>, <replaceable>vlist_index</replaceable>, <replaceable>min_width</replaceable>)</literallayout>
<para><indexterm id="IG31340177435"><primary>pmFetch function</primary></indexterm><indexterm id="IG31340177436"><primary>pmPrintValue function</primary></indexterm>The value of a single performance metric (as identified by <replaceable>val</replaceable>) is printed on the standard I/O stream identified by <replaceable>f</replaceable>. The value of the performance metric is interpreted according to the format of <replaceable>val</replaceable> as defined by <replaceable>valfmt</replaceable> (from a <command>pmValueSet</command> within a <filename>pmResult</filename>) and the generic description of the metric's type from a <command>pmDesc</command> structure, passed in through.</para>
<para>If the converted value is less than <replaceable>minwidth</replaceable> characters wide, it will have leading spaces to pad the output to a width of <replaceable>minwidth</replaceable> characters.</para>
<para><xref linkend="Z976565414sdc"/> illustrates using <command>pmPrintValue</command> to print the values from a <filename>pmResult</filename> structure returned via <command>pmFetch</command>:</para>
<example id="Z976565414sdc">
<title>Using <command>pmPrintValue</command> to Print Values</title>
<programlisting>    int         numpmid, i, j, sts;
    pmID        pmidlist[10];
    pmDesc      desc[10];
    pmResult    *result;

    /* set up PMAPI context, numpmid and pmidlist[] ... */
    /* get metric descriptors */
    for (i = 0; i &lt; numpmid; i++) {
        if ((sts = pmLookupDesc(pmidlist[i], &amp;desc[i])) &lt; 0) {
            printf(&quot;pmLookupDesc(pmid=%s): %s\n&quot;,
                          pmIDStr(pmidlist[i]), pmErrStr(sts));
            exit(1);
        }
    }
    if ((sts = pmFetch(numpmid, pmidlist, &amp;result)) &gt;= 0) {
        /* once per metric */
        for (i = 0; i &lt; result-&gt;numpmid; i++) {
            printf(&quot;PMID: %s&quot;, pmIDStr(result-&gt;vset[i]-&gt;pmid));
            /* once per instance for this metric */
            for (j = 0; j &lt; result-&gt;vset[i]-&gt;numval; j++) {
                printf(&quot; [%d]&quot;, result-&gt;vset[i]-&gt;vlist[j].inst);
                pmPrintValue(stdout, result-&gt;vset[i]-&gt;valfmt,
                             desc[i].type,
                             &amp;result-&gt;vset[i]-&gt;vlist[j],
                             8);
            }
            putchar('\n');
        }
        pmFreeResult(result);
    }
    else
        printf(&quot;pmFetch: %s\n&quot;, pmErrStr(sts));</programlisting>
</example>
<para>
Print the value of a <replaceable>pmresult</replaceable> pointed to by <replaceable>vset_index</replaceable>/<replaceable>vlist_index</replaceable> and described by <replaceable>pmdesc</replaceable>.  The format of a pmResult is described in <link linkend="IG31340177277"><emphasis>pmResult</emphasis></link>  

The python bindings can use sys.__stdout__ as a value for <replaceable>file</replaceable> to display to stdout.
</para>
</section>
<section id="id5211504">

<title><command>pmflush</command> Function</title>
<literallayout class="monospaced">int pmflush(void);
<command>Python:</command>
int <replaceable>status</replaceable> = pmflush()</literallayout>
<para><indexterm id="IG31340177437"><primary>pmflush function</primary></indexterm>The <command>pmflush</command> function causes the internal buffer which is shared with <command>pmprintf</command> to be either displayed in a window, printed on standard error, or flushed to a file and the internal buffer to be cleared.</para>
<para>The <literal>PCP_STDERR</literal> environment variable controls the output technique used by <command>pmflush</command>:</para>
<itemizedlist>
<listitem><para>If <literal>PCP_STDERR</literal> is unset, the text is written onto the stderr stream of the caller.</para>
</listitem>
<listitem><para>If <literal>PCP_STDERR</literal> is set to the literal reserved word <literal>DISPLAY</literal>, then the text is displayed as a GUI dialogue using <command>pmconfirm</command>.</para>
</listitem></itemizedlist>
<para>The <command>pmflush</command> function returns a value of zero on successful completion. A negative value is returned if an error was encountered, and this can be passed to <command>pmErrStr</command> to obtain the associated error message.</para>
</section>
<section id="id5211641">

<title><command>pmprintf</command> Function</title>
<literallayout class="monospaced">int pmprintf(const char *<replaceable>fmt</replaceable>, ... /*<replaceable>args</replaceable>*/);
<command>Python:</command>
pmprintf("fmt", ... /*<replaceable>args</replaceable>*/);</literallayout>
<para><indexterm id="IG31340177438"><primary>pmprintf function</primary></indexterm>The <command>pmprintf</command> function appends the formatted message string to an internal buffer shared by the <command>pmprintf</command> and <command>pmflush </command> functions, without actually producing any output. The <replaceable>fmt</replaceable> argument is used to control the conversion, formatting, and printing of the variable length <replaceable>args</replaceable> list.</para>
<para>The <command>pmprintf</command> function uses the <command>mkstemp</command> function to securely create a <literal>pcp</literal>-prefixed temporary file in <filename>${PCP_TMP_DIR}</filename>. This temporary file is deleted when <command>pmflush</command> is called.</para>
<para>On successful completion, <command>pmprintf</command> returns the number of characters transmitted. A negative value is returned if an error was encountered, and this can be passed to <command>pmErrStr</command> to obtain the associated error message.</para>
</section>
<section id="id5211762">

<title><command>pmSortInstances</command> Function</title>
<literallayout class="monospaced">void pmSortInstances(pmResult *<replaceable>result</replaceable>)
<command>Python:</command>
pmSortInstances (pmResult* pmresult)</literallayout>
<para><indexterm id="IG31340177439"><primary>pmSortInstances function</primary></indexterm><indexterm id="IG31340177440"><primary>pmFetch function</primary></indexterm>The <command>pmSortInstances</command>  function may be used to guarantee that for each performance metric in the result from <command>pmFetch</command>, the instances are in ascending internal instance identifier sequence. This is useful when trying to compute rates from two consecutive <command>pmFetch</command> results, where the underlying instance domain or metric availability is not static.</para>
</section>
<section id="id5211834">

<title><command>pmParseInterval</command> Function</title>
<literallayout class="monospaced">int pmParseInterval(const char *<replaceable>string</replaceable>, struct timeval *<replaceable>rslt</replaceable>, char **<replaceable>errmsg</replaceable>)
<command>Python:</command>
(struct <replaceable>timeval</replaceable>, "error message") = pmParseInterval("time string")</literallayout>
<para><indexterm id="IG31340177441"><primary>pmParseInterval function</primary></indexterm>The <command>pmParseInterval</command> function parses the argument string specifying an interval of time and fills in the <literal>tv_sec</literal> and <literal>tv_usec</literal> components of the <filename>rslt</filename> structure to represent that interval. The input string is most commonly the argument following a <literal>-t</literal> command line option to a PCP application, and the syntax is fully described in the <command>PCPIntro(1)</command> man page.</para>
<para><command>pmParseInterval</command> returns 0 and <replaceable>errmsg</replaceable> is undefined if the parsing is successful. If the given string does not conform to the required syntax, the function returns -1 and a dynamically allocated error message string in <replaceable>errmsg</replaceable>.</para>
<para>The error message is terminated with a newline and includes the text of the input string along with an indicator of the position at which the error was detected as shown in the following example:</para>
<programlisting>              4minutes 30mumble
                         ^ -- unexpected value</programlisting>
<para>In the case of an error, the caller is responsible for calling <command>free</command> to release the space allocated for <replaceable>errmsg</replaceable>.</para>
</section>
<section id="id5211943">

<title><command>pmParseMetricSpec</command> Function</title>
<literallayout class="monospaced">int pmParseMetricSpec(const char *<replaceable>string</replaceable>, int <replaceable>isarch</replaceable>, char *<replaceable>source</replaceable>,
                      pmMetricSpec **<replaceable>rsltp</replaceable>, char **<replaceable>errmsg</replaceable>)
<command>Python:</command>
(pmMetricSpec <replaceable>metricspec</replaceable>, "error message") = 
                      pmParseMetricSpec("metric specification", isarch, source)</literallayout>
<para><indexterm id="IG31340177442"><primary>pmParseMetricSpec function</primary></indexterm>The <command>pmParseMetricSpec</command> function accepts a <replaceable>string</replaceable> specifying the name of a PCP performance metric, and optionally the source (either a hostname, a PCP archive log filename, or a local context) and instances for that metric. The syntax is described in the <command>PCPIntro(1)</command> man page.</para>
<para>If neither host nor archive component of the metric specification is provided, the <literal>isarch</literal> and <literal>source</literal> arguments are used to fill in the returned <filename>pmMetricSpec</filename> structure. In <xref linkend="Z976566126sdc"/>, the <filename>pmMetricSpec</filename> structure, which is returned via <replaceable>rsltp,</replaceable> represents the parsed string.</para>
<example id="Z976566126sdc">
<title><filename>pmMetricSpec</filename> Structure</title>
<programlisting>typedef struct {
    int     isarch;      /* source type: 0 -&gt; host, 1 -&gt; archive, 2 -&gt; local context */
    char    *source;     /* name of source host or archive */
    char    *metric;     /* name of metric */
    int     ninst;       /* number of instances, 0 -&gt; all */
    char    *inst[1];    /* array of instance names */
} pmMetricSpec;</programlisting>
</example>
<para>The <command>pmParseMetricSpec</command> function returns 0 if the given string was successfully parsed. In this case, all the storage allocated by <command>pmParseMetricSpec</command> can be released by a single call to the <command>free</command> function by using the address returned from <command>pmMetricSpec</command> via <replaceable>rsltp</replaceable>. The convenience macro <command>pmFreeMetricSpec</command> is a thinly disguised wrapper for <command>free</command>.</para>
<para>The <command>pmParseMetricSpec</command> function returns 0 if the given string was successfully parsed. It returns <literal>PM_ERR_GENERIC</literal> and a dynamically allocated error message string in <replaceable>errmsg</replaceable> if the given string does not parse.  In this situation, the error message string can be released with the <command>free</command> function.</para>
<para>In the case of an error, <replaceable>rsltp</replaceable> is undefined. In the case of success, <replaceable>errmsg</replaceable> is undefined. If <replaceable>rsltp</replaceable>-&gt;<replaceable>ninst</replaceable> is 0, then <replaceable>rsltp</replaceable>-&gt;<replaceable>inst</replaceable>[0] is undefined.</para>
</section>
</section>
</section>

<section id="id5212196">

<title>PMAPI Programming Issues and Examples</title>
<para><indexterm id="IG31340177443"><primary>PMAPI</primary><secondary>programming issues</secondary></indexterm><indexterm id="IG31340177444"><primary>PMAPI</primary><secondary>programming issues</secondary></indexterm><indexterm id="IG31340177445"><primary>examples</primary><secondary>programming issues</secondary></indexterm>The following issues and examples are provided to enable you to create better custom performance monitoring tools.</para>
<para>The source code for a sample client (<command>pmclient</command>) using the PMAPI is shipped as part of the PCP package. See the <command>pmclient(1)</command> man page, and the source code, located in <filename>${PCP_DEMOS_DIR}/pmclient</filename>.</para>
<section id="id5212297">

<title>Symbolic Association between a Metric's Name and Value</title>
<para><indexterm id="IG31340177446"><primary>symbolic association</primary></indexterm><indexterm id="IG31340177447"><primary>metrics</primary><secondary>name and value</secondary></indexterm>A common problem in building specific performance tools is how to maintain the association between a performance metric's name, its access (instantiation) method, and the application program variable that contains the metric's value. Generally this results in code that is easily broken by bug fixes or changes in the underlying data structures. The PMAPI provides a uniform method for instantiating and accessing the values independent of the underlying implementation, although it does not solve the name-variable association problem. However, it does provide a framework within which a manageable solution may be developed.</para>
<para>Fundamentally, the goal is to be able to name a metric and reference the metric's value in a manner that is independent of the order of operations on other metrics; for example, to associate the <command>LOADAV</command> macro with the name <command>kernel.all.load</command>, and then be able to use <command>LOADAV</command> to get at the value of the corresponding metric.</para>
<para><indexterm id="IG31340177448"><primary>pmLookupName function</primary></indexterm><indexterm id="IG31340177449"><primary>pmFetch function</primary></indexterm>The one-to-one association between the ordinal position of the metric names is input to <command>pmLookupName</command> and the PMIDs returned by this function, and the one-to-one association between the PMIDs input to <command>pmFetch</command> and the values returned by this function provide the basis for an automated solution.</para>
<para>The tool <command>pmgenmap</command> takes the specification of a list of metric names and symbolic tags, in the order they should be passed to <command>pmLookupName</command> and <command>pmFetch</command>. For example, <command>pmclient</command>:</para>
<programlisting><userinput>cat ${PCP_DEMOS_DIR}/pmclient/pmnsmap.spec</userinput>
pmclient_init {
    hinv.ncpu			NUMCPU
}

pmclient_sample {
    kernel.all.load	LOADAV
    kernel.percpu.cpu.user	CPU_USR
    kernel.percpu.cpu.sys	CPU_SYS
    mem.freemem		FREEMEM
    disk.all.total		DKIOPS
}</programlisting>
<para>This <command>pmgenmap</command> input produces the C code in <xref linkend="Z976566536sdc"/>. It is suitable for including with the <literal>#include</literal> statement:</para>
<example id="Z976566536sdc">
<title>C Code Produced by <command>pmgenmap</command> Input</title>
<programlisting>/*
 * Performance Metrics Name Space Map
 * Built by runme.sh from the file
 * pmnsmap.spec
 * on Thu Jan  9 14:13:49 EST 2014
 *
 * Do not edit this file!
 */

char *pmclient_init[] = {
#define NUMCPU	0
	"hinv.ncpu",

};


char *pmclient_sample[] = {
#define LOADAV	0
	"kernel.all.load",
#define CPU_USR	1
	"kernel.percpu.cpu.user",
#define CPU_SYS	2
	"kernel.percpu.cpu.sys",
#define FREEMEM	3
	"mem.freemem",
#define DKIOPS	4
	"disk.all.total",

};</programlisting>
</example>
</section>
<section id="id5212519">

<title>Initializing New Metrics</title>
<para><indexterm id="IG31340177450"><primary>PMAPI</primary><secondary>initializing new metrics</secondary></indexterm><indexterm id="IG31340177451"><primary>initialization</primary></indexterm><indexterm id="IG31340177452"><primary>new metrics</primary></indexterm>Using the code generated by <command>pmgenmap</command>, you are now able to easily initialize the application's metric specifications as shown in <xref linkend="Z976566793sdc"/>:</para>
<example id="Z976566793sdc">
<title>Initializing Metric Specifications</title>
<programlisting>/* C code fragment from pmclient.c */
numpmid = sizeof(pmclient_sample) / sizeof(char *);
if ((pmidlist = (pmID *)malloc(numpmid * sizeof(pmidlist[0]))) == NULL) {...}
if ((sts = pmLookupName(numpmid, pmclient_sample, pmidlist)) &lt; 0) {...}
</programlisting>
<programlisting># The equivalent python code would be
pmclient_sample = ("kernel.all.load", "kernel.percpu.cpu.user",
	"kernel.percpu.cpu.sys", "mem.freemem", "disk.all.total")
pmidlist = context.pmLookupName(pmclient_sample)
</programlisting>
</example>
<para>At this stage, <command>pmidlist</command> contains the PMID for the five metrics of interest.</para>
</section>
<section id="id5212595">

<title>Iterative Processing of Values</title>
<para><indexterm id="IG31340177453"><primary>PMAPI</primary><secondary>iterative processing</secondary></indexterm><indexterm id="IG31340177454"><primary>iterative processing</primary></indexterm>Assuming the tool is required to report values every <replaceable>delta</replaceable> seconds, use code similar to that in <xref linkend="Z976567058sdc"/>:</para>
<example id="Z976567058sdc">
<title>Iterative Processing</title>
<programlisting>/* censored C code fragment from pmclient.c */
while (samples == -1 || samples-- &gt; 0) {
   if ((sts = pmFetch(numpmid, pmidlist, &amp;crp)) &lt; 0) { ... }
   for (i = 0; i &lt; numpmid; i++)
      if ((sts = pmLookupDesc(pmidlist[i], &amp;desclist[i])) &lt; 0) { ... }
   ...
   pmExtractValue(crp-&gt;vset[FREEMEM]-&gt;valfmt, crp-&gt;vset[FREEMEM]-&gt;vlist,
                  desclist[FREEMEM].type, &amp;tmp, PM_TYPE_FLOAT);
   pmConvScale(PM_TYPE_FLOAT, &amp;tmp, &amp;desclist[FREEMEM].units,
                    &amp;atom, &amp;mbyte_scale);
   ip-&gt;freemem = atom.f;
   ...
   __pmtimevalSleep(delta);
}</programlisting>
<programlisting># The equivalent python code would be
FREEMEM = 3
desclist = context.pmLookupDescs(metric_names)
while (samples &gt; 0):
    crp = context.pmFetch(metric_names)
    val = context.pmExtractValue(crp.contents.get_valfmt(FREEMEM),
                             crp.contents.get_vlist(FREEMEM, 0),
                             desclist[FREEMEM].contents.type,
                             c_api.PM_TYPE_FLOAT)
    atom = ctx.pmConvScale(c_api.PM_TYPE_FLOAT, val, desclist, FREEMEM,
    			   c_api.PM_SPACE_MBYTE)
    (tvdelta, errmsg) = c_api.pmParseInterval(delta)
    c_api.pmtimevalSleep(delta)			 
</programlisting>
</example>
</section>
<section id="id5212682">

<title>Accommodating Program Evolution</title>
<para><indexterm id="IG31340177455"><primary>PMAPI</primary><secondary>program evolution</secondary></indexterm><indexterm id="IG31340177456"><primary>program evolution</primary></indexterm>The flexibility provided by the PMAPI and the <command>pmgenmap</command> utility is demonstrated by <xref linkend="Z976567218sdc"/>. Consider the requirement for reporting a third metric <literal>mem.physmem</literal>.  This example shows how to add the line to the specification file:</para>
<example id="Z976567218sdc">
<title>Adding a Metric</title>
<literallayout class="monospaced">mem.freemem PHYSMEM </literallayout>
<para>Then regenerate the <filename>#include</filename> file, and augment pmclient.c:</para>
<programlisting>   pmExtractValue(crp-&gt;vset[PHYSMEM]-&gt;valfmt, crp-&gt;vset[PHYSMEM]-&gt;vlist,
                  desclist[PHYSMEM].type, &amp;tmp, PM_TYPE_FLOAT);
   pmConvScale(PM_TYPE_FLOAT, &amp;tmp, &amp;desclist[PHYSMEM].units,
                    &amp;atom, &amp;mbyte_scale);
</programlisting>
<programlisting># The equivalent python code would be:
val = context.pmExtractValue(crp.contents.get_valfmt(PHYSMEM),
			crp.contents.get_vlist(PHYSMEM, 0),
			desclist[PHYSMEM].contents.type,
			c_api.PM_TYPE_FLOAT);
</programlisting>
</example>
</section>
<section id="id5212805">

<title>Handling PMAPI Errors</title>
<para><indexterm id="IG31340177457"><primary>PMAPI</primary><secondary>error handling</secondary></indexterm><indexterm id="IG31340177458"><primary>error handling</primary></indexterm>In <xref linkend="id5212839"/>, the simple but complete PMAPI application demonstrates the recommended style for handling PMAPI error conditions.  The python bindings use the exception mechanism to raise an exception in error cases.  The python client can handle this condition by catching the <literal>pmErr</literal> exception.  For simplicity, no command line argument processing is shown here - in practice most tools use the <command>pmGetOptions</command> helper interface to assist with initial context creation and setup.
</para>
<example id="id5212839">
<title>PMAPI Error Handling</title>
<programlisting>#include &lt;pcp/pmapi.h&gt;

int
main(int argc, char* argv[])
{
    int                 sts = 0;
    char                *host = &quot;local:&quot;;
    char                *metric = &quot;mem.freemem&quot;;
    pmID                pmid;
    pmDesc              desc;
    pmResult            *result;

    sts = pmNewContext(PM_CONTEXT_HOST, host);
    if (sts &lt; 0) {
        fprintf(stderr, &quot;Error connecting to pmcd on %s: %s\n&quot;,
                host, pmErrStr(sts));
        exit(1);
    }
    sts = pmLookupName(1, &amp;metric, &amp;pmid);
    if (sts &lt; 0) {
        fprintf(stderr, &quot;Error looking up %s: %s\n&quot;, metric,
                pmErrStr(sts));
        exit(1);
    }
    sts = pmLookupDesc(pmid, &amp;desc);
    if (sts &lt; 0) {
        fprintf(stderr, &quot;Error getting descriptor for %s:%s: %s\n&quot;,
                host, metric, pmErrStr(sts));
        exit(1);
    }
    sts = pmFetch(1, &amp;pmid, &amp;result);
    if (sts &lt; 0) {
        fprintf(stderr, &quot;Error fetching %s:%s: %s\n&quot;, host, metric,
                pmErrStr(sts));
        exit(1);
    }
    sts = result-&gt;vset[0]-&gt;numval;
    if (sts &lt; 0) {
        fprintf(stderr, &quot;Error fetching %s:%s: %s\n&quot;, host, metric,
                pmErrStr(sts));
        exit(1);
    }
    fprintf(stdout, &quot;%s:%s = &quot;, host, metric);
    if (sts == 0)
        puts(&quot;(no value)&quot;);
    else {
        pmValueSet      *vsp = result-&gt;vset[0];
        pmPrintValue(stdout, vsp-&gt;valfmt, desc.type,
                             &amp;vsp-&gt;vlist[0], 5);
        printf(&quot; %s\n&quot;, pmUnitsStr(&amp;desc.units));
    }
    return 0;
}</programlisting>
<programlisting># The equivalent python code would be:
import sys
import traceback
from pcp import pmapi
from cpmapi import PM_TYPE_U32

try:
    context = pmapi.pmContext()
    pmid = context.pmLookupName("mem.freemem")
    desc = context.pmLookupDescs(pmid)
    result = context.pmFetch(pmid)
    freemem = context.pmExtractValue(result.contents.get_valfmt(0),
                                 result.contents.get_vlist(0, 0),
                                 desc[0].contents.type,
                                 PM_TYPE_U32)
    print "freemem is " + str(int(freemem.ul))

except pmapi.pmErr, error:
    print "%s: %s" % (sys.argv[0], error.message())
except Exception, error:
    sys.stderr.write(str(error) + "\n")
    sys.stderr.write(traceback.format_exc() + "\n")
</programlisting>
</example>
</section>
<section id="id5212855">

<title>Compiling and Linking PMAPI Applications</title>
<para><indexterm id="IG31340177459"><primary>PMAPI</primary><secondary>application compiling </secondary></indexterm><indexterm id="IG31340177460"><primary>applications</primary><secondary>compiling </secondary></indexterm><indexterm id="IG31340177461"><primary>compiling and linking</primary></indexterm>Typical PMAPI applications require the following line to include the function prototype and data structure definitions used by the PMAPI. </para>
<programlisting>#include &lt;pcp/pmapi.h&gt;</programlisting>
<para>Some applications may also require these header files: <filename>&lt;pcp/impl.h&gt;</filename> and <filename>&lt;pcp/pmda.h&gt;</filename>.</para>
<para>The run-time environment of the PMAPI is mostly found in the <filename>libpcp</filename> library; so to link a generic PMAPI application requires something akin to the following command:</para>
<literallayout class="monospaced"><userinput>cc</userinput> <replaceable>mycode</replaceable><userinput>.c -lpcp</userinput></literallayout>
</section>
</section>
</chapter>


<chapter id="LE25915-PARENT">

<title>Instrumenting Applications</title>
<para><indexterm id="IG31340177462nat"><primary>MMV PMDA</primary><secondary>description</secondary></indexterm><indexterm id="IG31340177462"><primary>trace PMDA</primary><secondary>description</secondary></indexterm><indexterm id="IG31340177463"><primary>PMDA</primary><secondary>trace</secondary></indexterm>This chapter provides an introduction to ways of instrumenting applications using PCP.</para>
<para><indexterm id="IG31340177565"><primary>customization</primary></indexterm> The first section covers the use of the Memory Mapped Value (MMV) Performance Metrics Domain Agent (PMDA) to generate customized metrics from an application. This provides a robust, extremely efficient mechanism for transferring custom instrumentation into the PCP infrastructure. It has been successfully deployed in production environments for many years, has proven immensely valuable in these situations, and can be used to instrument applications written in a number of programming languages.</para>
<para>The Memory Mapped Value library and PMDA is supported on every PCP platform, and is enabled by default.</para>
<note><para>A particularly expansive Java API is available from the separate
<ulink url="http://code.google.com/p/parfait/">Parfait</ulink> 
project. It supports both the existing JVM instrumentation, and custom application metric extensions.</para></note>
<para> <indexterm id="IG31340177464nat"><primary>libpcp_mmv library</primary><secondary>instrumenting applications</secondary></indexterm><indexterm id="IG31340177465nat"><primary>examples</primary><secondary>MMV PMDA</secondary></indexterm> The chapter also includes information on how to use the MMV library (<filename>libpcp_mmv</filename>) for instrumenting an application. The example programs are installed in <filename>${PCP_DEMOS_DIR}/mmv</filename>.</para>
<para>The second section covers the design of the Trace PMDA, in an effort to explain how to configure the agent optimally for a particular problem domain. This information supplements the functional coverage which the man pages provide to both the agent and the library interfaces.</para>
<para> <indexterm id="IG31340177464"><primary>libpcp_trace library</primary><secondary>instrumenting applications</secondary></indexterm><indexterm id="IG31340177465"><primary>examples</primary><secondary>trace PMDA</secondary></indexterm> This part of the chapter also includes information on how to use the Trace PMDA and its associated library (<filename>libpcp_trace</filename>) for instrumenting applications. The example programs are installed in <filename>${PCP_DEMOS_DIR}/trace</filename>.</para>
<warning><para>The current PCP trace library is a relatively heavy-weight solution, issuing multiple system calls per trace point, runs over a TCP/IP socket even locally and performs no event batching. As such it is not appropriate for production application instrumentation at this stage.</para></warning>
<para>A revised application tracing library and PMDA are planned which will be light-weight, suitable for production system tracing, and support event metrics and other advances in end-to-end distributed application tracing.</para>
<para><indexterm id="IG31340177554"><primary>application developers</primary></indexterm> <indexterm id="IG31340177555"><primary>embedded calls</primary></indexterm>The application instrumentation libraries are designed to encourage application developers to embed calls in their code that enable application performance data to be exported. When combined with system-level performance data, this feature allows total performance and resource demands of an application to be correlated with application activity.</para>
<para>For example, developers can provide the following application performance metrics:</para>
<itemizedlist>
<listitem><para><indexterm id="IG31340177556"><primary>computation state</primary></indexterm>Computation state (especially for codes with major shifts in resource demands between phases of their execution)</para>
</listitem>
<listitem><para><indexterm id="IG31340177557"><primary>parallelism</primary></indexterm>Problem size and parameters, that is, degree of parallelism throughput in terms of sub-problems solved, iteration count, transactions, data sets inspected, and so on</para>
</listitem>
<listitem><para><indexterm id="IG31340177558"><primary>service time</primary></indexterm>Service time by operation type</para>
</listitem></itemizedlist>
<section id="id5213105">

<title>Application and Performance Co-Pilot Relationship</title>
<para><indexterm id="IG31340177550"><primary>applications</primary><secondary>instrumentation</secondary></indexterm><indexterm id="IG31340177551"><primary>instrumentation</primary></indexterm> <indexterm id="IG31340177552"><primary>data export</primary></indexterm>The relationship between an application, the <filename>pcp_mmv</filename> and <filename>pcp_trace</filename> instrumentation libraries, the MMV and Trace PMDAs, and the rest of the PCP infrastructure is shown in <xref linkend="id5216302"/>:</para>
<para><figure id="id5216302"><title>Application and PCP Relationship</title><mediaobject><imageobject><imagedata fileref="figures/instrumentation.svg"/></imageobject><textobject><phrase>Application and PCP Relationship</phrase></textobject></mediaobject></figure></para>
<para>Once the application performance metrics are exported into the PCP framework, all of the PCP tools may be leveraged to provide performance monitoring and management, including:</para>
<itemizedlist>
<listitem><para>Two- and three-dimensional visualization of resource demands and performance, showing concurrent system activity and application activity.</para>
</listitem>
<listitem><para><indexterm id="IG31340177561"><primary>distributed performance management</primary><secondary>data transportation</secondary></indexterm>Transport of performance data over the network for distributed performance management.</para>
</listitem>
<listitem><para><indexterm id="IG31340177562"><primary>archive logs</primary><secondary>performance management</secondary></indexterm>Archive logging for historical records of performance, most useful for problem diagnosis, postmortem analysis, performance regression testing, capacity planning, and benchmarking.</para>
</listitem>
<listitem><para><indexterm id="IG31340177563"><primary>automated alarms</primary></indexterm>Automated alarms when bad performance is observed. These apply both in real-time or when scanning archives of historical application performance.</para>
</listitem></itemizedlist>
</section>
<section id="id5213104">

<title>Performance Instrumentation and Sampling</title>
<para><indexterm id="IG31340177467nat"><primary>performance instrumentation</primary></indexterm><indexterm id="IG31340177468nat"><primary>instrumentation</primary></indexterm>The <filename>pcp_mmv</filename> library provides function calls to assist with extracing important performance metrics from a program into a shared, in-memory location such that the MMV PMDA can examine and serve that information on behalf of PCP client tool requests. The <filename>pcp_mmv</filename> library is described in the <command>mmv_stats_init(3)</command>, <command>mmv_lookup_value_desc(3)</command>, <command>mmv_inc_value(3)</command> man pages. Additionally, the format of the shared memory mappings is described in detail in <command>mmv(5)</command>.</para>
</section>
<section id="id5213287nat">

<title>MMV PMDA Design</title>
<para><indexterm id="IG31340177470nat"><primary>MMV PMDA</primary><secondary>design</secondary></indexterm>An application instrumented with memory mapped values directly updates the memory that backs the metric values it exports. The MMV PMDA reads those values directly, from the <literal>same</literal> memory that the application is updating, when current values are sampled on behalf of PMAPI client tools. This relationship, and a simplified MMV API, are shown in <xref linkend="id5214410nat"/>.</para>
<para><figure id="id5214410nat"><title>Memory Mapped Page Sharing</title><mediaobject><imageobject><imagedata fileref="figures/pmdammv.svg"/></imageobject></mediaobject></figure></para>
<para>It is worth noting that once the metrics of an application have been registered via the <filename>pcp_mmv</filename> library initialisation API, subsequent interactions with the library are not intrusive to the instrumented application. At the points where values are updated, the only cost involved is the memory mapping update, which is a single memory store operation. There is no need to explicitly transfer control to the MMV PMDA, nor allocate memory, nor make system or library calls. The PMDA will only sample the values at times driven by PMAPI client tools, and this places no overhead on the instrumented application.</para>
</section>
<section id="id5213288nat">

<title>Memory Mapped Values API</title>
<para><indexterm id="IG31340177502nat"><primary>Application Programming Interface</primary></indexterm> <indexterm id="IG31340177503nat"><primary>libpcp_mmv library</primary><secondary>Application Programming Interface</secondary></indexterm>The <filename>libpcp_mmv</filename> Application Programming Interface (API) can be called from C, C++, Perl and Python (a separate project, Parfait, services the needs of Java applications). Each language has access to the complete set of functionality offered by <filename>libpcp_mmv</filename>. In most cases, the calling conventions differ only slightly between languages - in the case of Java and Parfait, they differ significantly however.</para>
<section id="id5214320nat">

<title>Starting and Stopping Instrumentation</title>
<para><indexterm id="IG31340177504nat"><primary>mmv_stats_init function</primary></indexterm> <indexterm id="IG31340177505nat"><primary>mmv_stats_stop function</primary></indexterm>
Instrumentation is begun with an initial call to <literal>mmv_stats_init</literal>, and ended with a call to <literal>mmv_stats_stop</literal>. These calls manipulate global state shared by the library and application. These are the only calls requiring synchonization and a single call to each is typically performed early and late in the life of the application (although they can be used to reset the library state as well, at any time). As such, the choice of synchonization primitive is left to the application, and none is currently performed by the library.</para>
<literallayout class="monospaced">void *mmv_stats_init(const char *<replaceable>name</replaceable>, int <replaceable>cluster</replaceable>, mmv_stats_flags_t <replaceable>flags</replaceable>,
                     const mmv_metric_t *<replaceable>stats</replaceable>, int <replaceable>nstats</replaceable>,
                     const mmv_indom_t *<replaceable>indoms</replaceable>, int <replaceable>nindoms</replaceable>)</literallayout>
<para>The <replaceable>name</replaceable> should be a simple symbolic name identifying the application. It is usually used as the first application-specific part of the exported metric names, as seen from the MMV PMDA. This behavior can be overriden using the <replaceable>flags</replaceable> parameter, with the MMV_FLAG_NOPREFIX flag. In the example below, full metric names such as <literal>mmv.acme.products.count</literal> will be created by the MMV PMDA. With the MMV_FLAG_NOPREFIX flag set, that would instead become <literal>mmv.products.count</literal>. It is recommended to not disable the prefix - doing so requires the applications to ensure naming conflicts do not arise in the MMV PMDA metric names.</para>
<para>The <replaceable>cluster</replaceable> identifier is used by the MMV PMDA to further distinguish different applications, and is directly used for the MMV PMDA PMID cluster field described in <xref linkend="Z1033577630tls"/>, for all MMV PMDA metrics.</para>
<para>All remaining parameters to <literal>mmv_stats_init</literal> define the metrics and instance domains that exist within the application. These are somewhat analagous to the final parameters of <literal>pmdaInit(3)</literal>, and are best explained using <xref linkend="Z1033577630nat"/> and <xref linkend="Z1033577631nat"/>. As mentioned earlier, the full source code for this example instrumented application can be found in <filename>${PCP_DEMOS_DIR}/mmv</filename>.</para>
<para><example id="Z1033577630nat">
<title>Memory Mapped Value Instance Structures</title>
<programlisting>#include &lt;pcp/pmapi.h&gt;
#include &lt;pcp/mmv_stats.h&gt;

static mmv_instances_t products[] = {
    {   .internal = 0, .external = "Anvils" },
    {   .internal = 1, .external = "Rockets" },
    {   .internal = 2, .external = "Giant_Rubber_Bands" },
};
#define ACME_PRODUCTS_INDOM 61
#define ACME_PRODUCTS_COUNT (sizeof(products)/sizeof(products[0]))

static mmv_indom_t indoms[] = {
    {   .serial = ACME_PRODUCTS_INDOM,
        .count = ACME_PRODUCTS_COUNT,
        .instances = products,
        .shorttext = "Acme products",
        .helptext = "Most popular products produced by the Acme Corporation",
    },
};</programlisting>
</example></para>
<para>The above data structures initialize an instance domain
of the set of products produced in a factory by the fictional
"Acme Corporation". These structures are directly comparable to
several concepts we have seen already (and for good reason - the
MMV PMDA must interpret the applications intentions and properly
export instances on its behalf):</para>
<itemizedlist>
<listitem><para>mmv_instances_t maps to pmdaInstid, as in <xref linkend="Z975964618sdc"/></para>
</listitem>
<listitem><para>mmv_indom_t maps to pmdaIndom, as in <xref linkend="Z975964773sdc"/> - the
major difference is the addition of oneline and long help text, the purpose of which should
be self-explanatory at this stage.</para>
</listitem>
<listitem><para><replaceable>serial</replaceable> numbers, as in <xref linkend="Z1033578294tls"/></para>
</listitem>
</itemizedlist>
<para>Next, we shall create three metrics, all of which use this instance domain.
These are the <literal>mmv.acme.products</literal> metrics, and they reflect the
rates at which products are built by the machines in the factory, how long these
builds take for each product, and how long each product type spends queued (while
waiting for factory capacity to become available).</para>
<para><example id="Z1033577631nat">
<title>Memory Mapped Value Metrics Structures</title>
<programlisting>static mmv_metric_t metrics[] = {
    {   .name = "products.count",
        .item = 7,
        .type = MMV_TYPE_U64,
        .semantics = MMV_SEM_COUNTER,
        .dimension = MMV_UNITS(0,0,1,0,0,PM_COUNT_ONE),
        .indom = ACME_PRODUCTS_INDOM,
        .shorttext = "Acme factory product throughput",
        .helptext =
"Monotonic increasing counter of products produced in the Acme Corporation\n"
"factory since starting the Acme production application.  Quality guaranteed.",
    },
    {   .name = "products.time",
        .item = 8,
        .type = MMV_TYPE_U64,
        .semantics = MMV_SEM_COUNTER,
        .dimension = MMV_UNITS(0,1,0,0,PM_TIME_USEC,0),
        .indom = ACME_PRODUCTS_INDOM,
        .shorttext = "Machine time spent producing Acme products",
        .helptext =
"Machine time spent producing Acme Corporation products.  Does not include\n"
"time in queues waiting for production machinery.",
    },
    {   .name = "products.queuetime",
        .item = 10,
        .type = MMV_TYPE_U64,
        .semantics = MMV_SEM_COUNTER,
        .dimension = MMV_UNITS(0,1,0,0,PM_TIME_USEC,0),
        .indom = ACME_PRODUCTS_INDOM,
        .shorttext = "Queued time while producing Acme products",
        .helptext =
"Time spent in the queue waiting to build Acme Corporation products,\n"
"while some other Acme product was being built instead of this one.",
    },
};
#define INDOM_COUNT (sizeof(indoms)/sizeof(indoms[0]))
#define METRIC_COUNT (sizeof(metrics)/sizeof(metrics[0]))</programlisting>
</example></para>
<para>As was the case with the "products" instance domain before, these metric-defining
data structures are directly comparable to PMDA data structures described earlier:</para>
<itemizedlist>
<listitem><para>mmv_metric_t maps to a pmDesc structure, as in <xref linkend="Z975964618sdc"/></para>
</listitem>
<listitem><para>MMV_TYPE, MMV_SEM, and MMV_UNITS map to PMAPI constructs for type, semantics, dimensionality and scale, as in <xref linkend="Z1034792511tls"/></para>
</listitem>
<listitem><para><replaceable>item</replaceable> number, as in <xref linkend="Z1033577630tls"/></para>
</listitem>
</itemizedlist>
<para>For the most part, all types and macros map directly to their core PCP counterparts, which the MMV PMDA will use when exporting the metrics. One important exception is the introduction of the metric type MMV_TYPE_ELAPSED, which is discussed further in <xref linkend="id5214734"/>.</para>
<para>The compound metric types - aggregate and event type metrics - are not supported by the MMV format.</para>
</section>

<section>
<title>Getting a Handle on Mapped Values</title>
<para><indexterm id="Z1033577633nat"><primary>mmv_lookup_value_desc function</primary></indexterm>Once metrics (and the instance domains they use) have been registered, the memory mapped file has been created and is ready for use. In order to be able to update the individual metric values, however, we must find get a handle to the value. This is done using the <command>mmv_lookup_value_desc</command> function, as shown in <xref linkend="Z1033577632nat"/>.</para>
<para><example id="Z1033577632nat">
<title>Memory Mapped Value Handles</title>
<programlisting>#define ACME_CLUSTER 321        /* PMID cluster identifier */

int
main(int argc, char * argv[])
{
    void *base;
    pmAtomValue *count[ACME_PRODUCTS_COUNT];
    pmAtomValue *machine[ACME_PRODUCTS_COUNT];
    pmAtomValue *inqueue[ACME_PRODUCTS_COUNT];
    unsigned int working;
    unsigned int product;
    unsigned int i;

    base = mmv_stats_init("acme", ACME_CLUSTER, 0,
                          metrics, METRIC_COUNT, indoms, INDOM_COUNT);
    if (!base) {
        perror("mmv_stats_init");
        return 1;
    }

    for (i = 0; i &lt; ACME_PRODUCTS_COUNT; i++) {
        count[i] = mmv_lookup_value_desc(base,
                        "products.count", products[i].external);
        machine[i] = mmv_lookup_value_desc(base,
                        "products.time", products[i].external);
        inqueue[i] = mmv_lookup_value_desc(base,
                        "products.queuetime", products[i].external);
    }</programlisting>
</example></para>
<para>Space in the mapping file for every value is set aside at initialization time (by the <command>mmv_stats_init</command> function) - that is, space for each and every metric, and each value (instance) of each metric when an instance domain is used. To find the handle to the space set aside for one individual value requires the tuple of base memory address of the mapping, metric name, and instance name. In the case of metrics with no instance domain, the final instance name parameter should be either NULL or the empty string.</para>
</section>
<section>

<title>Updating Mapped Values</title>
<para><indexterm id="Z1033577634nat"><primary>mmv_stats_inc function</primary></indexterm>At this stage we have individual handles (pointers) to each instrumentation point, we can now start modifying these values and observing changes through the PCP infrastructure. Notice that each handle is simply the canonical <command>pmAtomValue</command> pointer, as defined in <xref linkend="Z976562908sdc"/>, which is a union providing sufficient space to hold any single value.</para>
<para>This pointer can be either manipulated directly, or using helper functions provided by the <command>pcp_mmv</command> API, such as the <command>mmv_stats_inc</command> and <command>mmv_stats_set</command> functions.</para>
<para><example id="Z1033577636">
<title>Memory Mapped Value Updates</title>
<programlisting>    while (1) {
        /* choose a random number between 0-N -&gt; product */
        product = rand() % ACME_PRODUCTS_COUNT;

        /* assign a time spent "working" on this product */
        working = rand() % 50000;

        /* pretend to "work" so process doesn't burn CPU */
        usleep(working);

        /* update the memory mapped values for this one: */
        /* one more product produced and work time spent */
        mmv_inc_value(base, machine[product], working); /* API */
        count[product]-&gt;ull += 1;     /* or direct mmap update */

        /* all other products are "queued" for this time */
        for (i = 0; i &lt; ACME_PRODUCTS_COUNT; i++)
            if (i != product)
                mmv_inc_value(base, inqueue[i], working);
    }</programlisting>
</example></para>
<para>At this stage, it will be informative to compile and run the complete example program, which can be found in <filename>${PCP_DEMOS_DIR}/mmv/acme.c</filename>. There is an associated <filename>Makefile</filename> to build it, in the same directory. Running the <filename>acme</filename> binary creates the instrumentation shown in <xref linkend="Z1033577637"/>, with live values letting us explore simple queueing effects in products being created on the ACME factory floor.</para>
<para><example id="Z1033577637">
<title>Memory Mapped Value Reports</title>
<programlisting><userinput>pminfo -m mmv.acme</userinput>
mmv.acme.products.queuetime PMID: 70.321.10
mmv.acme.products.time PMID: 70.321.8
mmv.acme.products.count PMID: 70.321.7

<userinput>pmval -f2 -s3 mmv.acme.products.time</userinput>
metric:    mmv.acme.products.time
host:      localhost
semantics: cumulative counter (converting to rate)
units:     microsec (converting to time utilization)
samples:   3
interval:  1.00 sec

               Anvils               Rockets    Giant_Rubber_Bands 
                 0.37                  0.12                  0.50 
                 0.35                  0.25                  0.38 
                 0.57                  0.20                  0.23</programlisting>
</example></para>
<para>Experimentation with the algorithm from <xref linkend="Z1033577636"/> is encouraged. In particular, observe the effects of rate conversion (counter metric type) of a metric with units of "time" (PM_TIME_*). The reported values are calculated over a sampling interval, which also has units of "time", forming a utilization. This is extremely valuable performance analysis currency - comparable metrics would include processor utilization, disk spindle utilization, and so forth.</para>
</section>

<section id="id5214734">
<title>Elapsed Time Measures</title>
<para><indexterm id="Z1033577635nat"><primary>mmv_stats_interval_start function</primary></indexterm><indexterm id="Z1033577636nat"><primary>mmv_stats_interval_end function</primary></indexterm>One problem with the instrumentation model embodied by the <command>pcp_mmv</command> library is providing timing information for long-running operations. For instrumenting long-running operations, like uploading downloading a file, the overall operation may be broken into smaller, discrete units of work which can be easily instrumented in terms of operations and througput measures. In other cases, there are no divisible units for long-running operations (for example a black-box library call) and instrumenting these operations presents a challenge. Sometimes the best that can be done is adding the instrumentation point at the completion of the operation, and simply accept the "bursty" nature of this approach. In these problematic cases, the work completed in one sampling-interval may have begun several intervals before, from the point of view of the monitoring tool, which can lead to misleading results.</para>
<para>One technique that is available to combat this is through use of the MMV_TYPE_ELAPSED metric type, which provides the concept of a "timed section" of code. This mechanism stores the start time of an operation along with the mapped metric value (an "elapsed time" counter), via the <command>mmv_stats_interval_start</command> instrumentation function. Then, with help from the MMV PMDA which recognizes this type, the act of sampling the metric value causes an <emphasis role="bold">interim</emphasis> timestamp to be taken (by the MMV PMDA, not the application) and <emphasis role="bold">combined</emphasis> with the initial timestamp to form a more accurate reflection of time spent within the timed section, which effectively smooths out the bursty nature of the instrumentation.</para>
<para>The completion of each timed section of code is marked by a call to <command>mmv_stats_interval_end</command> which signifies to the MMV PMDA that the operation is not active, and no extra "in-progress" time should be applied to the exported value. At that time, the elapsed time for the entire operation is calculated and accounted toward metrics value.</para>
</section>

</section>
<section id="id5213106">

<title>Performance Instrumentation and Tracing</title>
<para><indexterm id="IG31340177466"><primary>pmdatrace man page</primary></indexterm><indexterm id="IG31340177467"><primary>performance instrumentation</primary></indexterm><indexterm id="IG31340177468"><primary>instrumentation</primary></indexterm>The <filename>pcp_trace</filename> library provides function calls for identifying sections of a program as transactions or events for examination by the trace PMDA, a user command called <command>pmdatrace</command>. The <filename>pcp_trace</filename> library is described in the <command>pmdatrace(3)</command> man page</para>
<para>The monitoring of transactions using the Performance Co-Pilot (PCP) infrastructure begins with a <command>pmtracebegin</command> call. Time is recorded from there to the corresponding <command>pmtraceend</command> call (with matching tag identifier). A transaction in progress can be cancelled by calling <command>pmtraceabort</command>.</para>
<para>A second form of program instrumentation is available with the <command>pmtracepoint</command> function. This is a simpler form of monitoring that exports only the number of times a particular point in a program is passed. The <command>pmtraceobs</command> and <command>pmtracecount</command> functions have similar semantics, but the former allows an arbitrary numeric value to be passed to the trace PMDA.</para>
<para><indexterm id="IG31340177469"><primary>pmdatrace man page</primary></indexterm>The <command>pmdatrace</command> command is a PMDA that exports transaction performance metrics from application processes using the <filename>pcp_trace</filename> library; see the <command>pmdatrace(1)</command> man page for details.</para>
</section>
<section id="id5213287">

<title>Trace PMDA Design</title>
<para><indexterm id="IG31340177470"><primary>trace PMDA</primary><secondary>design</secondary></indexterm>Trace PMDA design covers application interaction, sampling techniques, and configuring the trace PMDA.</para>
<section id="id5213308">

<title>Application Interaction</title>
<para><indexterm id="IG31340177471"><primary>applications</primary><secondary>interaction</secondary></indexterm><xref linkend="id5213335"/> describes the general state maintained within the trace PMDA.</para>
<para><figure id="id5213335"><title>Trace PMDA Overview</title><mediaobject><imageobject><imagedata fileref="figures/trace.svg"/></imageobject><textobject><phrase>Trace PMDA Overview</phrase></textobject></mediaobject></figure></para>
<para><indexterm id="IG31340177472"><primary>Application Programming Interface</primary></indexterm> <indexterm id="IG31340177473"><primary> identification tags</primary></indexterm> <indexterm id="IG31340177474"><primary>IPC</primary><secondary>trace API</secondary></indexterm>Applications that are linked with the <filename>libpcp_trace</filename> library make calls through the trace Application Programming Interface (API). These calls result in interprocess communication of trace data between the application and the trace PMDA. This data consists of an identification tag and the performance data associated with that particular tag. The trace PMDA aggregates the incoming information and periodically updates the exported summary information to describe activity in the recent past.</para>
<para><indexterm id="IG31340177475"><primary>PDU</primary></indexterm> <indexterm id="IG31340177476"><primary>working buffers</primary></indexterm>As each protocol data unit (PDU) is received, its data is stored in the current working buffer. At the same time, the global counter associated with the particular tag contained within the PDU is incremented. The working buffer contains all performance data that has arrived since the previous time interval elapsed. For additional information about the working buffer, see <xref linkend="LE42586-PARENT"/>.</para>
</section>
<section id="id5213449">

<title>Sampling Techniques</title>
<para><indexterm id="IG31340177477"><primary>rolling-window sampling</primary></indexterm> <indexterm id="IG31340177478"><primary>sampling techniques</primary></indexterm>The trace PMDA employs a rolling-window periodic sampling technique. The arrival time of the data at the trace PMDA in conjunction with the length of the sampling period being maintained by the PMDA determines the recency of the data exported by the PMDA. Through the use of rolling-window sampling, the trace PMDA is able to present a more accurate representation of the available trace data at any given time than it could through use of simple periodic sampling.</para>
<para> <indexterm id="IG31340177479"><primary>trace.observe.rate metric</primary></indexterm> <indexterm id="IG31340177480"><primary>trace.point.rate metric</primary></indexterm> <indexterm id="IG31340177481"><primary>trace.transact.ave_time metric</primary></indexterm> <indexterm id="IG31340177482"><primary>trace.transact.max_time metric</primary></indexterm> <indexterm id="IG31340177483"><primary>trace.transact.min_time metric</primary></indexterm> <indexterm id="IG31340177484"><primary>trace.transact.rate metric</primary></indexterm>The rolling-window sampling technique affects the metrics in <xref linkend="Z976568222sdc"/>:</para>
<example id="Z976568222sdc">
<title>Rolling-Window Sampling Technique</title>
<literallayout class="monospaced">trace.observe.rate
trace.counter.rate
trace.point.rate
trace.transact.ave_time
trace.transact.max_time
trace.transact.min_time
trace.transact.rate</literallayout>
</example>
<para>The remaining metrics are either global counters, control metrics, or the last seen observation value. <xref linkend="LE26087-PARENT"/>, documents in more detail all metrics exported by the trace PMDA.</para>
<section id="id5213576">

<title>Simple Periodic Sampling</title>
<para><indexterm id="IG31340177485"><primary>periodic sampling</primary></indexterm><indexterm id="IG31340177486"><primary>simple periodic sampling</primary></indexterm> <indexterm id="IG31340177487"><primary>historical buffers</primary></indexterm>The simple periodic sampling technique uses a single historical buffer to store the history of events that have occurred over the sampling interval. As events occur, they are recorded in the working buffer. At the end of each sampling interval, the working buffer (which at that time holds the historical data for the sampling interval just finished) is copied into the historical buffer, and the working buffer is cleared. It is ready to hold new events from the sampling interval now starting.</para>
</section>
<section id="LE42586-PARENT">

<title>Rolling-Window Periodic Sampling</title>
<para><indexterm id="IG31340177488"><primary>rolling-window sampling</primary></indexterm>In contrast to simple periodic sampling with its single historical buffer, the rolling-window periodic sampling technique maintains a number of separate buffers. One buffer is marked as the current working buffer, and the remainder of the buffers hold historical data. As each event occurs, the current working buffer is updated to reflect it.</para>
<para>At a specified interval, the current working buffer and the accumulated data that it holds is moved into the set of historical buffers, and a new working buffer is used. The specified interval is a function of the number of historical buffers maintained.</para>
<para>The primary advantage of the rolling-window sampling technique is seen at the point where data is actually exported. At this point, the data has a higher probability of reflecting a more recent sampling period than the data exported using simple periodic sampling.</para>
<para><indexterm id="IG31340177489"><primary>sample duration</primary></indexterm>The data collected over each sample duration and exported using the rolling-window sampling technique provides a more up-to-date representation of the activity during the most recently completed sample duration than simple periodic sampling as shown in <xref linkend="id5213739"/>.</para>
<para><figure id="id5213739"><title>Sample Duration Comparison</title><mediaobject><imageobject><imagedata fileref="figures/trace-sampling.svg"/></imageobject><textobject><phrase>Sample Duration Comparison</phrase></textobject></mediaobject></figure></para>
<para><indexterm id="IG31340177490"><primary>ring buffers</primary></indexterm>The trace PMDA allows the length of the sample duration to be configured, as well as the number of historical buffers that are maintained. The rolling-window approach is implemented in the trace PMDA as a ring buffer (see <xref linkend="id5213335"/>).</para>
<para><indexterm id="IG31340177491"><primary>working buffers</primary></indexterm> <indexterm id="IG31340177492"><primary>historical buffers</primary></indexterm>When the current working buffer is moved into the set of historical buffers, the least recent historical buffer is cleared of data and becomes the new working buffer.</para>
</section>
<section id="id5213803">

<title>Rolling-Window Periodic Sampling Example</title>
<para><indexterm id="IG31340177493"><primary>examples</primary><secondary>rolling-window sampling</secondary></indexterm>Consider the scenario where you want to know the rate of transactions over the last 10 seconds. You set the sampling rate for the trace PMDA to 10 seconds and fetch the metric <literal>trace.transact.rate</literal>. So if in the last 10 seconds, 8 transactions took place, the transaction rate would be 8/10 or 0.8 transactions per second.</para>
<para>The trace PMDA does not actually do this. It instead does its calculations automatically at a subinterval of the sampling interval. Reconsider the 10-second scenario. It has a calculation subinterval of 2 seconds as shown in <xref linkend="id5213848"/>.</para>
<para><figure id="id5213848"><title>Sampling Intervals</title><mediaobject><imageobject><imagedata fileref="figures/trace-example.svg"/></imageobject><textobject><phrase>Sampling Intervals</phrase></textobject></mediaobject></figure></para>
<para>If at 13.5 seconds, you request the transaction rate, you receive a value of 0.7 transactions per second. In actual fact, the transaction rate was 0.8, but the trace PMDA did its calculations on the sampling interval from 2 seconds to 12 seconds, and not from 3.5 seconds to 13.5 seconds. For efficiency, the trace PMDA calculates the metrics on the last 10 seconds every 2 seconds. As a result, the PMDA is not driven each time a fetch request is received to do a calculation.</para>
</section>
</section>
<section id="id5213903">

<title>Configuring the Trace PMDA</title>
<para><indexterm id="IG31340177494"><primary>trace PMDA</primary><secondary>command-line options</secondary></indexterm>The trace PMDA is configurable primarily through command-line options. The list of command-line options in <xref linkend="id5213936"/> is not exhaustive, but it identifies those options which are particularly relevant to tuning the manner in which performance data is collected.</para>
<table id="id5213936" frame="topbot">

<title>Selected Command-Line Options</title>
<tgroup cols="2" colsep="0" rowsep="0">
<colspec colwidth="149*"/>
<colspec colwidth="247*"/>
<thead>
<row rowsep="1" valign="top"><entry align="left" valign="bottom"><para>Option</para></entry><entry align="left" valign="bottom"><para>Description</para></entry></row></thead>
<tbody>
<row valign="top">
<entry align="left" valign="top"><para>Access controls<indexterm id="IG31340177495"><primary>access controls</primary></indexterm></para></entry>
<entry align="left" valign="top"><para>The trace PMDA offers host-based access control. This control allows and disallows connections from instrumented applications running on specified hosts or groups of hosts. Limits to the number of connections allowed from individual hosts can also be mandated.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para>Sample duration<indexterm id="IG31340177496"><primary>sample duration</primary></indexterm></para></entry>
<entry align="left" valign="top"><para>The interval over which metrics are to be maintained before being discarded is called the sample duration.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para>Number of historical buffers<indexterm id="IG31340177497"><primary>historical buffers</primary></indexterm></para></entry>
<entry align="left" valign="top"><para>The data maintained for the sample duration is held in a number of internal buffers within the trace PMDA. These are referred to as historical buffers. This number is configurable so that the rolling window effect can be tuned within the sample duration.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para>Counter and observation metric units<indexterm id="IG31340177498"><primary>observation metric units</primary></indexterm></para></entry>
<entry align="left" valign="top"><para><indexterm id="IG31340177499"><primary>pmchart command</primary></indexterm> Since the data being exported by the <literal>trace.observe.value</literal> and <literal>trace.counter.count</literal> metrics are user-defined, the trace PMDA by default exports these metrics with a type of &#x201C;none.&#x201D; A framework is provided that allows the user to make the type more specific (for example, bytes per second) and allows the exported values to be plotted along with other performance metrics of similar units by tools like <command>pmchart</command>.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para>Instance domain refresh<indexterm id="IG31340177500"><primary>instance domain refresh</primary></indexterm></para></entry>
<entry align="left" valign="top"><para>The set of instances exported for each of the <literal>trace</literal> metrics can be cleared through the storable <literal><indexterm id="IG31340177501"><primary>trace.control.reset metric</primary></indexterm> trace.control.reset</literal> metric.</para></entry></row></tbody></tgroup></table>
</section>
</section>
<section id="LE26087-PARENT">

<title>Trace API</title>
<para><indexterm id="IG31340177502"><primary>Application Programming Interface</primary></indexterm> <indexterm id="IG31340177503"><primary>libpcp_trace library</primary><secondary>Application Programming Interface</secondary></indexterm>The <filename>libpcp_trace</filename> Application Programming Interface (API) is called from C, C++, Fortran, and Java. Each language has access to the complete set of functionality offered by <filename>libpcp_trace</filename>. In some cases, the calling conventions differ slightly between languages. This section presents an overview of each of the different tracing mechanisms offered by the API, as well as an explanation of their mappings to the actual performance metrics exported by the trace PMDA.</para>
<section id="id5214320">

<title>Transactions </title>
<para><indexterm id="IG31340177504"><primary>pmtracebegin function</primary></indexterm> <indexterm id="IG31340177505"><primary>pmtracend function</primary></indexterm> <indexterm id="IG31340177506"><primary>transactions</primary></indexterm> <indexterm id="IG31340177507"><primary>pmtraceabort function</primary></indexterm>Paired calls to the <command>pmtracebegin</command> and <command>pmtraceend</command> API functions result in transaction data being sent to the trace PMDA with a measure of the time interval between the two calls. This interval is the transaction service time. Using the <command>pmtraceabort</command> call causes data for that particular transaction to be discarded. The trace PMDA exports transaction data through the following <literal>trace.transact</literal> metrics listed in <xref linkend="id5214410"/>:</para>
<table id="id5214410" frame="topbot">

<title><literal>trace.transact</literal> Metrics</title>
<tgroup cols="2" colsep="0" rowsep="0">
<colspec colwidth="174*"/>
<colspec colwidth="222*"/>
<thead>
<row rowsep="1" valign="top"><entry align="left" valign="bottom"><para>Metric</para></entry><entry align="left" valign="bottom"><para>Description</para></entry></row></thead>
<tbody>
<row valign="top">
<entry align="left" valign="top"><para><literal><indexterm id="IG31340177508"><primary>trace.transact.ave_time metric</primary></indexterm> trace.transact.ave_time</literal></para></entry>
<entry align="left" valign="top"><para>The average service time per transaction type. This time is calculated over the last sample duration.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal><indexterm id="IG31340177509"><primary>trace.transact.count metric</primary></indexterm> trace.transact.count</literal></para></entry>
<entry align="left" valign="top"><para>The running count for each transaction type seen since the trace PMDA started.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal><indexterm id="IG31340177510"><primary>trace.transact.max_time metric</primary></indexterm> trace.transact.max_time</literal></para><para><literal/></para></entry>
<entry align="left" valign="top"><para>The maximum service time per transaction type within the last sample duration.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal><indexterm id="IG31340177511"><primary>trace.transact.min_time metric</primary></indexterm> trace.transact.min_time</literal></para></entry>
<entry align="left" valign="top"><para>The minimum service time per transaction type within the last sample duration.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal><indexterm id="IG31340177512"><primary>trace.transact.rate metric</primary></indexterm> trace.transact.rate</literal></para><para>&#xA0;</para></entry>
<entry align="left" valign="top"><para>The average rate at which each transaction type is completed. The rate is calculated over the last sample duration.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal><indexterm id="IG31340177513"><primary>trace.transact.total_time metric</primary></indexterm> trace.transact.total_time</literal></para></entry>
<entry align="left" valign="top"><para>The cumulative time spent processing each transaction since the trace PMDA started running.</para></entry></row></tbody></tgroup></table>
</section>
<section id="id5214714">

<title>Point Tracing </title>
<para><indexterm id="IG31340177514"><primary>point tracing</primary></indexterm> <indexterm id="IG31340177515"><primary>pmtracepoint function</primary></indexterm>Point tracing allows the application programmer to export metrics related to salient events. The <command>pmtracepoint</command> function is most useful when start and end points are not well defined. For example, this function is useful when the code branches in such a way that a transaction cannot be clearly identified, or when processing does not follow a transactional model, or when the desired instrumentation is akin to event rates rather than event service times. This data is exported through the <literal>trace.point</literal> metrics listed in <xref linkend="id5214762"/>:<table id="id5214762" frame="topbot">

<title><literal>trace.point</literal> Metrics</title>
<tgroup cols="2" colsep="0" rowsep="1">
<colspec colwidth="149*"/>
<colspec colwidth="247*"/>
<thead>
<row valign="top"><entry align="left" valign="bottom"><para>Metric</para></entry><entry align="left" valign="bottom"><para>Description</para></entry></row></thead>
<tbody>
<row rowsep="0" valign="top">
<entry align="left" valign="top"><para><literal>trace.point.count<indexterm id="IG31340177516"><primary>trace.point.count metric</primary></indexterm></literal></para></entry>
<entry align="left" valign="top"><para>Running count of point observations for each tag seen since the trace PMDA started.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>trace.point.rate<indexterm id="IG31340177517"><primary>trace.point.rate metric</primary></indexterm></literal></para></entry>
<entry align="left" valign="top"><para>The average rate at which observation points occur for each tag within the last sample duration.</para></entry></row></tbody></tgroup></table></para>
</section>
<section id="id5214931">

<title>Observations and Counters</title>
<para><indexterm id="IG31340177518"><primary>pmtraceobs function</primary></indexterm> <indexterm id="IG31340177519"><primary>pmtracepoint function</primary></indexterm> <indexterm id="IG31340177520"><primary>trace.observe metrics</primary></indexterm>The <command>pmtraceobs</command> and <command>pmtracecount</command> functions have similar semantics to <command>pmtracepoint</command>, but also allow an arbitrary numeric value to be passed to the trace PMDA. The most recent value for each tag is then immediately available from the PMDA. Observation data is exported through the <literal>trace.observe</literal> metrics listed in <xref linkend="id5215021"/>:</para>
<table id="id5215021" frame="topbot">

<title><literal>trace.observe</literal> Metrics</title>
<tgroup cols="2" colsep="0" rowsep="0">
<colspec colwidth="149*"/>
<colspec colwidth="247*"/>
<thead>
<row rowsep="1" valign="top"><entry align="left" valign="bottom"><para>Metric</para></entry><entry align="left" valign="bottom"><para>Description</para></entry></row></thead>
<tbody>
<row valign="top">
<entry align="left" valign="top"><para><literal>trace.observe.count</literal></para></entry>
<entry align="left" valign="top"><para>Running count of observations seen since the trace PMDA started.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>trace.observe.rate</literal></para></entry>
<entry align="left" valign="top"><para>The average rate at which observations for each tag occur. This rate is calculated over the last sample duration.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>trace.observe.value</literal></para></entry>
<entry align="left" valign="top"><para>The numeric value associated with the observation last seen by the trace PMDA.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>trace.counter</literal></para></entry>
<entry align="left" valign="top"><para>Counter data is exported through the <literal>trace.counter</literal> metrics. The only difference between <literal>trace.counter</literal> and <literal>trace.observe</literal> metrics is that the numeric value of <literal>trace.counter</literal> must be a monotonic increasing count.</para></entry></row></tbody></tgroup></table>
</section>
<section id="id5215241">

<title>Configuring the Trace Library</title>
<para><indexterm id="IG31340177521"><primary>configuration</primary></indexterm><indexterm id="IG31340177522"><primary>environment variables</primary></indexterm> <indexterm id="IG31340177523"><primary>state flags</primary></indexterm> <indexterm id="IG31340177524"><primary>diagnostic output</primary></indexterm>The trace library is configurable through the use of environment variables listed in <xref linkend="id5215299"/> as well as through the state flags listed in <xref linkend="id5215745"/>. Both provide diagnostic output and enable or disable the configurable functionality within the library.</para>
<table id="id5215299" frame="topbot">

<title>Environment Variables</title>
<tgroup cols="2" colsep="0" rowsep="0">
<colspec colwidth="149*"/>
<colspec colwidth="247*"/>
<thead>
<row rowsep="1" valign="top"><entry align="left" valign="bottom"><para>Name</para></entry><entry align="left" valign="bottom"><para>Description</para></entry></row></thead>
<tbody>
<row valign="top">
<entry align="left" valign="top"><para><literal>PCP_TRACE_HOST</literal><indexterm id="IG31340177525"><primary>PCP_TRACE_HOST variable</primary></indexterm></para></entry>
<entry align="left" valign="top"><para>The name of the host where the trace PMDA is running.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>PCP_TRACE_PORT</literal><indexterm id="IG31340177526"><primary>PCP_TRACE_PORT variable</primary></indexterm></para></entry>
<entry align="left" valign="top"><para><indexterm id="IG31340177527"><primary>TCP/IP</primary></indexterm> TCP/IP port number on which the trace PMDA is accepting client connections.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>PCP_TRACE_TIMEOUT</literal><indexterm id="IG31340177528"><primary>PCP_TRACE_TIMEOUT variable</primary></indexterm></para></entry>
<entry align="left" valign="top"><para>The number of seconds to wait until assuming that the initial connection is not going to be made, and timeout will occur. The default is three seconds.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>PCP_TRACE_REQTIMEOUT</literal><indexterm id="IG31340177529"><primary>PCP_TRACE_REQTIMEOUT variable</primary></indexterm></para></entry>
<entry align="left" valign="top"><para><indexterm id="IG31340177530"><primary>asynchronous trace protocol</primary></indexterm> The number of seconds to allow before timing out on awaiting acknowledgment from the trace PMDA after trace data has been sent to it. This variable has no effect in the asynchronous trace protocol (refer to <xref linkend="id5215745"/>).</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>PCP_TRACE_RECONNECT</literal><indexterm id="IG31340177531"><primary>PCP_TRACE_RECONNECT variable</primary></indexterm></para></entry>
<entry align="left" valign="top"><para>A list of values which represents the backoff approach that the <filename>libpcp_trace</filename> library routines take when attempting to reconnect to the trace PMDA after a connection has been lost. The list of values should be a positive number of seconds for the application to delay before making the next reconnection attempt. When the final value in the list is reached, that value is used for all subsequent reconnection attempts.</para></entry></row></tbody></tgroup></table>
<para><indexterm id="IG31340177532"><primary>state flags</primary></indexterm> <indexterm id="IG31340177533"><primary>flags</primary><secondary>state</secondary></indexterm> <indexterm id="IG31340177534"><primary>pmtracestate call</primary></indexterm> <indexterm id="IG31340177535"><primary>libpcp_trace library</primary><secondary id="Z980104750sdc">functions</secondary></indexterm>The <xref linkend="id5215745"/> are used to customize the operation of the <filename>libpcp_trace</filename> routines. These are registered through the <command>pmtracestate</command> call, and they can be set either individually or together.</para>
<table id="id5215745" frame="topbot">

<title>State Flags</title>
<tgroup cols="2" colsep="0" rowsep="0">
<colspec colwidth="170*"/>
<colspec colwidth="226*"/>
<thead>
<row rowsep="1" valign="top"><entry align="left" valign="bottom"><para>Flag</para></entry><entry align="left" valign="bottom"><para>Description</para></entry></row></thead>
<tbody>
<row valign="top">
<entry align="left" valign="top"><para><literal>PMTRACE_STATE_NONE<indexterm id="IG31340177536"><primary>PMTRACE_STATE_NONE flag</primary></indexterm></literal></para></entry>
<entry align="left" valign="top"><para><indexterm id="IG31340177537"><primary>synchronous protocol</primary></indexterm> The default. No state flags have been set, the fault-tolerant, synchronous protocol is used for communicating with the trace PMDA, and no diagnostic messages are displayed by the <filename/><filename>libpcp_trace</filename> routines.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>PMTRACE_STATE_API<indexterm id="IG31340177538"><primary>PMTRACE_STATE_API flag</primary></indexterm></literal></para></entry>
<entry align="left" valign="top"><para>High-level diagnostics. This flag simply displays entry into each of the API routines.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>PMTRACE_STATE_COMMS<indexterm id="IG31340177539"><primary>PMTRACE_STATE_COMMS flag</primary></indexterm></literal></para></entry>
<entry align="left" valign="top"><para>Diagnostic messages related to establishing and maintaining the communication channel between application and PMDA.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>PMTRACE_STATE_PDU<indexterm id="IG31340177540"><primary>PMTRACE_STATE_PDU flag</primary></indexterm> <indexterm id="IG31340177541"><primary>PMTRACE_STATE_PDUBUF flag</primary></indexterm></literal></para></entry>
<entry align="left" valign="top"><para><indexterm id="IG31340177542"><primary>PDU</primary></indexterm> The low-level details of the trace protocol data units (PDU) is displayed as each PDU is transmitted or received.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>PMTRACE_STATE_PDUBUF<indexterm id="IG31340177543"><primary>PMTRACE_STATE_NOAGENT flag</primary></indexterm></literal></para></entry>
<entry align="left" valign="top"><para>The full contents of the PDU buffers are dumped as PDUs are transmitted and received.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>PMTRACE_STATE_NOAGENT<indexterm id="IG31340177544"><primary>PMTRACE_STATE_NOAGENT flag</primary></indexterm></literal></para></entry>
<entry align="left" valign="top"><para><indexterm id="IG31340177545"><primary>interprocess communication</primary><secondary>PMTRACE_STATE_NOAGENT flag</secondary></indexterm> <indexterm id="IG31340177546"><primary>debugging and testing</primary></indexterm> Interprocess communication control. If this flag is set, it causes interprocess communication between the instrumented application and the trace PMDA to be skipped. This flag is a debugging aid for applications using <filename>libpcp_trace</filename>.</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><literal>PMTRACE_STATE_ASYNC<indexterm id="IG31340177547"><primary>PMTRACE_STATE_ASYNC flag</primary></indexterm></literal></para></entry>
<entry align="left" valign="top"><para><indexterm id="IG31340177548"><primary>asynchronous trace protocol</primary></indexterm> <indexterm id="IG31340177549"><primary>libpcp_trace library</primary><secondary>entry points</secondary></indexterm> Asynchronous trace protocol. This flag enables the asynchronous trace protocol so that the application does not block awaiting acknowledgment PDUs from the trace PMDA. In order for the flag to be effective, it must be set before using the other <filename/><filename>libpcp_trace</filename> entry points.</para></entry></row></tbody></tgroup></table>
</section>
</section>








</chapter>


<appendix id="LE54271-PARENT">

<title>Acronyms</title>
<para><xref linkend="id5216717"/> provides a glossary of the acronyms used in the Performance Co-Pilot (PCP) documentation, help cards, man pages, and user interface.<indexterm id="IG31340177566"><primary>glossary</primary></indexterm><indexterm id="IG31340177567"><primary>acronyms</primary></indexterm></para>
<table id="id5216717" frame="topbot">

<title>Performance Co-Pilot Acronyms and Their Meanings</title>
<tgroup cols="2" colsep="0" rowsep="0">
<colspec colwidth="92*"/>
<colspec colwidth="304*"/>
<thead>
<row rowsep="1" valign="top"><entry align="left" valign="bottom"><para>Acronym</para></entry><entry align="left" valign="bottom"><para>Meaning</para></entry></row></thead>
<tbody>
<row valign="top">
<entry align="left" valign="top"><para>API</para></entry>
<entry align="left" valign="top"><para>Application Programming Interface</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para>DBMS</para></entry>
<entry align="left" valign="top"><para>Database Management System</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><indexterm id="IG31340177568"><primary>DNS</primary></indexterm> DNS</para></entry>
<entry align="left" valign="top"><para>Domain Name Service</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><indexterm id="IG31340177569"><primary>DSO</primary></indexterm> DSO</para></entry>
<entry align="left" valign="top"><para>Dynamic Shared Object</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para>I/O</para></entry>
<entry align="left" valign="top"><para>Input/Output</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para>IPC</para></entry>
<entry align="left" valign="top"><para>Interprocess Communication</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><indexterm id="IG31340177570"><primary>PCP</primary><secondary>acronym</secondary></indexterm> PCP</para></entry>
<entry align="left" valign="top"><para>Performance Co-Pilot</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><indexterm id="IG31340177571"><primary>PDU</primary></indexterm> PDU</para></entry>
<entry align="left" valign="top"><para>Protocol data unit</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><indexterm id="IG31340177572"><primary>PMAPI</primary><secondary>acronym</secondary></indexterm> PMAPI</para></entry>
<entry align="left" valign="top"><para>Performance Metrics Application Programming Interface</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><indexterm id="IG31340177573"><primary>PMCD</primary><secondary>acronym</secondary></indexterm> PMCD</para></entry>
<entry align="left" valign="top"><para>Performance Metrics Collection Daemon</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><indexterm id="IG31340177575"><primary>PMDA</primary><secondary>acronym</secondary></indexterm> PMDA</para></entry>
<entry align="left" valign="top"><para>Performance Metrics Domain Agent</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><indexterm id="IG31340177576"><primary>PMID</primary><secondary>acronym</secondary></indexterm> PMID</para></entry>
<entry align="left" valign="top"><para>Performance Metric Identifier</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><indexterm id="IG31340177577"><primary>PMNS</primary><secondary>acronym</secondary></indexterm> PMNS</para></entry>
<entry align="left" valign="top"><para>Performance Metrics Name Space</para></entry></row>
<row valign="top">
<entry align="left" valign="top"><para><indexterm id="IG31340177578"><primary>TCP/IP</primary></indexterm> TCP/IP</para></entry>
<entry align="left" valign="top"><para>Transmission Control Protocol/Internet Protocol</para></entry></row></tbody></tgroup></table>
</appendix>
<index id="sgi-index">

<indexentry>
   <primaryie>__pmID_int structure <link linkend="IG31340177121">Data Structures</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>__pmInDom_int structure <link linkend="IG31340177141">Data Structures</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>access controls <link linkend="IG31340177495">Configuring the Trace PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>acronyms <link linkend="IG31340177567">Acronyms</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>ancillary support services <link linkend="IG31340177417">PMAPI Ancillary Support Services</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>Application Programming Interface <link linkend="IG31340177257">PMAPI--The Performance Metrics API</link> <link linkend="IG31340177502nat">Memory Mapped Values API</link> <link linkend="IG31340177502">Trace API</link> <link linkend="IG31340177472">Application Interaction</link> <link linkend="IG31340177502">Trace API</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>application developers<link linkend="IG31340177554">Instrumenting Applications</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>application programs <link linkend="ITch01-114">Application and Agent Development</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>applications</primaryie>
      <secondaryie>compiling <link linkend="IG31340177460">Compiling and Linking PMAPI Applications</link></secondaryie>
      <secondaryie>instrumentation <link linkend="IG31340177550">Application and PCP Relationship</link></secondaryie>
      <secondaryie>interaction <link linkend="IG31340177471">Application Interaction</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>architecture <link linkend="IG313401778">PCP Architecture</link> <link linkend="IG3134017777">PMDA Architecture</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>archive logs</primaryie>
      <secondaryie>context services <link linkend="IG31340177347">PMAPI Context Services</link></secondaryie>
      <secondaryie>performance data <link linkend="IG31340177258">PMAPI--The Performance Metrics API</link> <link linkend="IG31340177266">Current PMAPI Context</link></secondaryie>
      <secondaryie>performance management <link linkend="IG31340177562">Application and PCP Relationship</link></secondaryie>
      <secondaryie>pmGetArchiveEnd function <link linkend="IG31340177402"><command>pmGetArchiveEnd</command> Function</link></secondaryie>
      <secondaryie>pmGetInDomArchive function <link linkend="IG31340177405"><command>pmGetInDomArchive</command> Function</link></secondaryie>
      <secondaryie>retrospective sources <link linkend="IG3134017731">Retrospective Sources of Performance Metrics</link></secondaryie>
      <secondaryie>time control services <link linkend="IG31340177415">PMAPI Time Control Services</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>archive-specific services <link linkend="IG31340177401"><command>pmGetArchiveLabel</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>Cluster PMDA <link linkend="IG3134017717">Distributed Collection</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>arrays</primaryie>
      <secondaryie>instance description <link linkend="IG31340177140">Data Structures</link></secondaryie>
      <secondaryie>N dimensional data <link linkend="IG31340177134">N Dimensional Data</link></secondaryie>
      <secondaryie>performance metrics <link linkend="IG31340177278">Performance Metrics Values</link> <link linkend="IG31340177283">Variable Length Argument and Results Lists</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>asynchronous trace protocol <link linkend="IG31340177530">Configuring the Trace Library</link> <link linkend="IG31340177548">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>audience <link linkend="IG313401772">Programming Performance Co-Pilot</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>automated alarms <link linkend="IG31340177563">Application and PCP Relationship</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>caching PMDA <link linkend="IG31340177106">Caching PMDA</link> <link linkend="IG31340177153">Latency and Threads of Control</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>chkhelp tool <link linkend="IG3134017735">Application and Agent Development</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>Cisco PMDA <link linkend="IG3134017716">Distributed Collection</link> <link linkend="IG31340177107">Caching PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>client development <link linkend="IG3134017752">Client Development and PMAPI</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>clusters <link linkend="IG3134017725">Name Space</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>collection time <link linkend="IG31340177267">Current PMAPI Context</link> <link linkend="IG31340177352"><command>pmNewContext</command> Function</link> <link linkend="IG31340177357"><command>pmWhichContext</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>collection tools <link linkend="IG3134017712">PCP Architecture</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>collector hosts <link linkend="IG3134017719">Distributed Collection</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>COLOR_INDOM <link linkend="IG31340177143">Data Structures</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>compiling and linking <link linkend="IG31340177461">Compiling and Linking PMAPI Applications</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>component software <link linkend="IG3134017733">Overview of Component Software</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>computation state <link linkend="IG31340177556">Instrumenting Applications</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>configuration <link linkend="IG31340177521">Configuring the Trace Library</link> <link linkend="IG31340177251">Configuring PCP Tools</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>context services <link linkend="IG31340177320">PMAPI Context Services</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>control threads <link linkend="IG31340177150">Latency and Threads of Control</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>counter semantics <link linkend="IG31340177131">Semantics</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>customization <link linkend="IG313401777">Programming Performance Co-Pilot</link> <link linkend="IG31340177565">Instrumenting Applications</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>daemon process method <link linkend="IG3134017750">Daemon Process Method</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>data export <link linkend="IG31340177552">Application and PCP Relationship</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>data structures <link linkend="IG31340177118">Data Structures</link> <link linkend="IG31340177137">Data Structures</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>dbpmda man page <link linkend="IG3134017768">Implementing a PMDA</link> <link linkend="IG31340177229">Overview</link> <link linkend="IG31340177236"><command>dbpmda</command> Debug Utility</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>dbx man page <link linkend="IG31340177228">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>debugging and testing <link linkend="IG31340177227">Testing and Debugging a PMDA</link> <link linkend="IG31340177546">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>debugging flags</primaryie>
      <seeie>flags</seeie>
</indexentry>

<indexentry>
   <primaryie>delays <link linkend="IG31340177151">Latency and Threads of Control</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>design requirements <link linkend="IG3134017761">Implementing a PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>diagnostic output <link linkend="IG31340177524">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>dimensionality and scale <link linkend="IG31340177272">Performance Metric Descriptions</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>discrete semantics <link linkend="IG31340177133">Semantics</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>distributed performance management</primaryie>
      <secondaryie>data transportation <link linkend="IG31340177561">Application and PCP Relationship</link></secondaryie>
      <secondaryie>metrics collection <link linkend="IG3134017715">Distributed Collection</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>dlopen man page <link linkend="IG3134017746">In-Process (DSO) Method</link> <link linkend="IG3134017796">DSO PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>DNS <link linkend="IG31340177568">Acronyms</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>domains</primaryie>
      <secondaryie>definition <link linkend="IG31340177110">Overview</link></secondaryie>
      <secondaryie>fields <link linkend="IG3134017727">Name Space</link></secondaryie>
      <secondaryie>numbers <link linkend="IG31340177113">Domains</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>dometric function <link linkend="IG31340177302"><command>pmTraversePMNS</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>DSO <link linkend="IG31340177569">Acronyms</link></primaryie>
      <secondaryie>architecture <link linkend="IG3134017779">PMDA Architecture</link></secondaryie>
      <secondaryie>disadvantages <link linkend="IG31340177100">Daemon PMDA</link></secondaryie>
      <secondaryie>implementation <link linkend="IG3134017795">DSO PMDA</link></secondaryie>
      <secondaryie>interface <link linkend="IG31340177178">PMDA Interface</link></secondaryie>
      <secondaryie>PMDA building <link linkend="IG3134017745">In-Process (DSO) Method</link></secondaryie>
      <secondaryie>PMDA initialization <link linkend="IG31340177209">Common Initialization</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>dynamic shared object</primaryie>
      <seeie>DSO</seeie>
</indexentry>

<indexentry>
   <primaryie>embedded calls <link linkend="IG31340177555">Instrumenting Applications</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>environment variables <link linkend="IG31340177522">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>error handling <link linkend="IG31340177458">Handling PMAPI Errors</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>examples</primaryie>
      <secondaryie>alarm tools <link linkend="IG3134017775">Implementing a PMDA</link></secondaryie>
      <secondaryie>Install script <link linkend="IG31340177243">Installing a PMDA</link></secondaryie>
      <secondaryie>MMV PMDA <link linkend="IG31340177465nat">Instrumenting Applications</link></secondaryie>
      <secondaryie>programming issues <link linkend="IG31340177445">PMAPI Programming Issues and Examples</link></secondaryie>
      <secondaryie>Remove script <link linkend="IG31340177249">Removing a PMDA</link></secondaryie>
      <secondaryie>rolling-window sampling <link linkend="IG31340177493">Rolling-Window Periodic Sampling Example</link></secondaryie>
      <secondaryie>simple and trivial PMDAs <link linkend="IG31340177109">Domains, Metrics, and Instances</link></secondaryie>
      <secondaryie>time control functions <link linkend="IG31340177416">PMAPI Time Control Services</link></secondaryie>
      <secondaryie>trace PMDA <link linkend="IG31340177465">Instrumenting Applications</link></secondaryie>
      <secondaryie>visualization tools <link linkend="IG3134017774">Implementing a PMDA</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>execv system call <link linkend="IG31340177102">Daemon PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>exporting data <link linkend="IG31340177147">Extracting the Information</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>flags</primaryie>
      <secondaryie>debugging <link linkend="IG31340177233">Debugging Information</link></secondaryie>
      <secondaryie>state <link linkend="IG31340177533">Configuring the Trace Library</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>fork system call <link linkend="IG31340177101">Daemon PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>glossary <link linkend="IG31340177566">Acronyms</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>handle context <link linkend="IG31340177375"><command>pmReconnectContext</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>help text</primaryie>
      <secondaryie>creation <link linkend="IG31340177241">Installing a PMDA</link></secondaryie>
      <secondaryie>initialization <link linkend="IG31340177210">Common Initialization</link></secondaryie>
      <secondaryie>location <link linkend="IG31340177239">Installing a PMDA</link></secondaryie>
      <secondaryie>PDU_TEXT_REQ <link linkend="IG3134017792">Overview</link></secondaryie>
      <secondaryie>pmLookupInDomText function <link linkend="IG31340177311"><command>pmLookupInDomText</command> Function</link></secondaryie>
      <secondaryie>pmLookupText function <link linkend="IG31340177169">Management of Evolution within a PMDA</link> <link linkend="IG31340177313"><command>pmLookupText</command> Function</link></secondaryie>
      <secondaryie>structure specification <link linkend="IG3134017767">Implementing a PMDA</link></secondaryie>
      <secondaryie>terse and extended descriptions <link linkend="IG31340177158">PMDA Help Text</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>historical buffers <link linkend="IG31340177487">Simple Periodic Sampling</link> <link linkend="IG31340177492">Rolling-Window Periodic Sampling</link> <link linkend="IG31340177497">Configuring the Trace PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>identification tags <link linkend="IG31340177473">Application Interaction</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>implementation <link linkend="IG3134017760">Implementing a PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>indom instance domain <link linkend="IG31340177309"><command>pmLookupInDomText</command> Function</link> <link linkend="IG31340177359"><command>pmAddProfile</command> Function</link> <link linkend="IG31340177406"><command>pmGetInDomArchive</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>information extraction <link linkend="IG31340177146">Extracting the Information</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>initialization <link linkend="IG31340177451">Initializing New Metrics</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>instance domain refresh <link linkend="IG31340177500">Configuring the Trace PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>instance domain services <link linkend="IG31340177314"><command>pmGetInDom</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>instantaneous semantics <link linkend="IG31340177132">Semantics</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>instlist argument <link linkend="IG31340177315"><command>pmGetInDom</command> Function</link> <link linkend="IG31340177360"><command>pmAddProfile</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>instrumentation <link linkend="IG31340177468nat">Performance Instrumentation and Sampling</link> <link linkend="IG31340177468">Performance Instrumentation and Tracing</link> <link linkend="IG31340177551">Application and PCP Relationship</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>integrating a PMDA <link linkend="IG31340177238">Integration of a PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>internal instance identifier <link linkend="IG31340177277">Performance Metrics Values</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>interpolated metrics <link linkend="IG31340177372"><command>pmSetMode</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>interprocess communication</primaryie>
      <seeie>IPC</seeie>
      <secondaryie>PMTRACE_STATE_NOAGENT flag <link linkend="IG31340177545">Configuring the Trace Library</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>IPC</primaryie>
      <secondaryie>DSO <link linkend="IG3134017749">In-Process (DSO) Method</link></secondaryie>
      <secondaryie>PMDA <link linkend="IG3134017765">Implementing a PMDA</link></secondaryie>
      <secondaryie>trace API <link linkend="IG31340177474">Application Interaction</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>item numbers <link linkend="IG3134017726">Name Space</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>iterative processing <link linkend="IG31340177454">Iterative Processing of Values</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>latency <link linkend="IG31340177149">Latency and Threads of Control</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>leaf node <link linkend="IG31340177303"><command>pmTraversePMNS</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>libpcp_mmv library</primaryie>
      <secondaryie>Application Programming Interface <link linkend="IG31340177503nat">Memory Mapped Values API</link></secondaryie>
      <secondaryie>instrumenting applications <link linkend="IG31340177464nat">Instrumenting Applications</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>libpcp_trace library</primaryie>
      <secondaryie>Application Programming Interface <link linkend="IG31340177503">Trace API</link></secondaryie>
      <secondaryie>entry points <link linkend="IG31340177549">Configuring the Trace Library</link></secondaryie>
      <secondaryie>functions <link linkend="IG31340177535">Configuring the Trace Library</link></secondaryie>
      <secondaryie>instrumenting applications <link linkend="IG31340177464">Instrumenting Applications</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>library reentrancy <link linkend="IG3134017758">Library Reentrancy and Threaded Applications</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>metric description services <link linkend="IG31340177306"><command>pmLookupDesc</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>metrics</primaryie>
      <secondaryie>API <link linkend="IG31340177260">Naming and Identifying Performance Metrics</link></secondaryie>
      <secondaryie>definition <link linkend="IG31340177111">Overview</link> <link linkend="IG31340177114">Metrics</link></secondaryie>
      <secondaryie>name and value <link linkend="IG31340177447">Symbolic Association between a Metric's Name and Value</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>metrics and instances <link linkend="IG31340177112">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>metrics description services <link linkend="IG31340177305"><command>pmLookupDesc</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>metrics services <link linkend="IG31340177384"><command>pmFetch</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>mmv_lookup_value_desc function <link linkend="Z1033577633nat">Getting a Handle on Mapped Values</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>mmv_stats_init function <link linkend="IG31340177504nat">Starting and Stopping Instrumentation</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>mmv_stats_stop function <link linkend="IG31340177505nat">Starting and Stopping Instrumentation</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>mmv_stats_inc function <link linkend="Z1033577634nat">Updating Mapped Values</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>mmv_stats_interval_start function <link linkend="Z1033577635nat">Elapsed Time Measures</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>mmv_stats_interval_end function <link linkend="Z1033577636nat">Elapsed Time Measures</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>monitoring tools <link linkend="IG3134017711">PCP Architecture</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>multidimensional arrays <link linkend="IG31340177136">N Dimensional Data</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>multiple threads <link linkend="IG3134017757">Library Reentrancy and Threaded Applications</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>MMV PMDA</primaryie>
      <secondaryie>description <link linkend="IG31340177462nat">Instrumenting Applications</link></secondaryie>
      <secondaryie>design <link linkend="IG31340177470nat">MMV PMDA Design</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>name space <link linkend="IG3134017722">Name Space</link> <link linkend="IG31340177155">Name Space</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>new metrics <link linkend="IG31340177160">Management of Evolution within a PMDA</link> <link linkend="IG31340177452">Initializing New Metrics</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>new PMDA <link linkend="IG31340177244">Upgrading a PMNS to Include Metrics from a New PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>newhelp man page <link linkend="IG31340177159">PMDA Help Text</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>newhelp tool <link linkend="IG3134017737">Application and Agent Development</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>NOW_INDOM <link linkend="IG31340177144">Data Structures</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>observation metric units <link linkend="IG31340177498">Configuring the Trace PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>parallelism <link linkend="IG31340177557">Instrumenting Applications</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PCP</primaryie>
      <secondaryie>acronym <link linkend="IG31340177570">Acronyms</link></secondaryie>
      <secondaryie>description <link linkend="IG313401770">Programming Performance Co-Pilot</link></secondaryie>
      <secondaryie>tool summaries <link linkend="IG3134017734">Application and Agent Development</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>PCP_TRACE_HOST variable <link linkend="IG31340177525">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PCP_TRACE_PORT variable <link linkend="IG31340177526">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PCP_TRACE_RECONNECT variable <link linkend="IG31340177531">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PCP_TRACE_REQTIMEOUT variable <link linkend="IG31340177529">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PCP_TRACE_TIMEOUT variable <link linkend="IG31340177528">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PDU <link linkend="IG3134017780">Overview</link> <link linkend="IG31340177475">Application Interaction</link> <link linkend="IG31340177542">Configuring the Trace Library</link> <link linkend="IG31340177571">Acronyms</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PDU_AUTH <link linkend="IG3134017791nat">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PDU_DESC_REQ <link linkend="IG3134017789">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PDU_FETCH <link linkend="IG3134017783">Overview</link> <link linkend="IG31340177219">Simple PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PDU_INSTANCE_REQ <link linkend="IG3134017787">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PDU_PMNS_CHILD <link linkend="IG3134017785nat">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PDU_PMNS_NAMES <link linkend="IG3134017783nat">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PDU_PMNS_TRAVERSE <link linkend="IG3134017787nat">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PDU_PMNS_IDS <link linkend="IG3134017789nat">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PDU_PROFILE <link linkend="IG3134017785">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PDU_RESULT <link linkend="IG3134017794">Overview</link> <link linkend="IG31340177218">Simple PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PDU_TEXT_REQ <link linkend="IG3134017791">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>performance instrumentation <link linkend="IG313401775">Programming Performance Co-Pilot</link> <link linkend="IG31340177467nat">Performance Instrumentation and Sampling</link> <link linkend="IG31340177467">Performance Instrumentation and Tracing</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>Performance Metric Identifier</primaryie>
      <seeie>PMID</seeie>
</indexentry>

<indexentry>
   <primaryie>performance metrics</primaryie>
      <seeie>metrics</seeie>
</indexentry>

<indexentry>
   <primaryie>Performance Metrics Application Programming Interface</primaryie>
      <seeie>PMAPI</seeie>
</indexentry>

<indexentry>
   <primaryie>Performance Metrics Collection Daemon</primaryie>
      <seeie>PMCD</seeie>
</indexentry>

<indexentry>
   <primaryie>Performance Metrics Domain Agent</primaryie>
      <seeie>PMDA</seeie>
</indexentry>

<indexentry>
   <primaryie>Performance Metrics Name Space</primaryie>
      <seeie>PMNS</seeie>
</indexentry>

<indexentry>
   <primaryie>periodic sampling <link linkend="IG31340177485">Simple Periodic Sampling</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pipe <link linkend="IG31340177103">Daemon PMDA</link> <link linkend="IG31340177104">Daemon PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PM_CONTEXT_ARCHIVE type <link linkend="IG31340177348"><command>pmNewContext</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PM_CONTEXT_HOST type <link linkend="IG31340177349"><command>pmNewContext</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PM_ERR_CONV error code <link linkend="IG31340177177">Management of Evolution within a PMDA</link> <link linkend="IG31340177422"><command>pmExtractValue</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PM_ERR_INST error code <link linkend="IG31340177197"><literal>simple_store</literal> in the Simple PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PM_ERR_PMID error code <link linkend="IG31340177167">Management of Evolution within a PMDA</link> <link linkend="IG31340177198"><literal>simple_store</literal> in the Simple PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PM_ERR_SIGN error code <link linkend="IG31340177424"><command>pmExtractValue</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PM_ERR_TIMEOUT error code <link linkend="IG31340177388"><command>pmFetch</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PM_ERR_TRUNC error code <link linkend="IG31340177423"><command>pmExtractValue</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PM_IN_NULL instance identifier <link linkend="IG31340177264">Performance Metric Instances</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PM_INDOM_NULL instance domain</primaryie>
      <secondaryie>data structures <link linkend="IG31340177125">Data Structures</link> <link linkend="IG31340177145">Data Structures</link></secondaryie>
      <secondaryie>description <link linkend="IG31340177263">Performance Metric Instances</link></secondaryie>
      <secondaryie>pmAddProfile function <link linkend="IG31340177362"><command>pmAddProfile</command> Function</link></secondaryie>
      <secondaryie>pmDelProfile function <link linkend="IG31340177363"><command>pmDelProfile</command> Function</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>PM_SEM_COUNTER semantic type <link linkend="IG31340177128">Semantics</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PM_SEM_DISCRETE semantic type <link linkend="IG31340177130">Semantics</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PM_SEM_INSTANT semantic type <link linkend="IG31340177126">Data Structures</link> <link linkend="IG31340177129">Semantics</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PM_TYPE_AGGREGATE type <link linkend="IG31340177270">Performance Metric Descriptions</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PM_TYPE_NOSUPPORT value <link linkend="IG31340177165">Management of Evolution within a PMDA</link> <link linkend="IG31340177271">Performance Metric Descriptions</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PM_TYPE_STRING type <link linkend="IG31340177269">Performance Metric Descriptions</link> <link linkend="IG31340177425"><command>pmExtractValue</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PM_TYPE_EVENT type <link linkend="IG31340177269nat">Performance Metric Descriptions</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PM_VAL_INSITU value <link linkend="IG31340177279">Performance Metrics Values</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmAddProfile function <link linkend="IG3134017784">Overview</link> <link linkend="IG31340177322">PMAPI Context Services</link> <link linkend="IG31340177361"><command>pmAddProfile</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PMAPI <link linkend="IG3134017739">Application and Agent Development</link> <link linkend="IG31340177262">Performance Metric Instances</link></primaryie>
      <seealsoie>metrics</seealsoie>
      <secondaryie>acronym <link linkend="IG31340177572">Acronyms</link></secondaryie>
      <secondaryie>ancillary support services <link linkend="IG31340177418">PMAPI Ancillary Support Services</link></secondaryie>
      <secondaryie>application compiling <link linkend="IG31340177459">Compiling and Linking PMAPI Applications</link></secondaryie>
      <secondaryie>archive-specific services <link linkend="IG31340177400"><command>pmGetArchiveLabel</command> Function</link></secondaryie>
      <secondaryie>client development <link linkend="IG3134017755">Client Development and PMAPI</link></secondaryie>
      <secondaryie>context services <link linkend="IG31340177321">PMAPI Context Services</link></secondaryie>
      <secondaryie>current context <link linkend="IG31340177265">Current PMAPI Context</link></secondaryie>
      <secondaryie>description <link linkend="IG31340177256">PMAPI--The Performance Metrics API</link></secondaryie>
      <secondaryie>description services <link linkend="IG31340177308"><command>pmLookupDesc</command> Function</link></secondaryie>
      <secondaryie>error handling <link linkend="IG31340177291">PMAPI Error Handling</link> <link linkend="IG31340177457">Handling PMAPI Errors</link></secondaryie>
      <secondaryie>identifying metrics <link linkend="IG31340177261">Naming and Identifying Performance Metrics</link></secondaryie>
      <secondaryie>initializing new metrics <link linkend="IG31340177450">Initializing New Metrics</link></secondaryie>
      <secondaryie>instance domain services <link linkend="IG31340177317"><command>pmGetInDom</command> Function</link></secondaryie>
      <secondaryie>introduction <link linkend="IG313401774">Programming Performance Co-Pilot</link></secondaryie>
      <secondaryie>iterative processing <link linkend="IG31340177453">Iterative Processing of Values</link></secondaryie>
      <secondaryie>man page <link linkend="IG3134017721">Distributed Collection</link></secondaryie>
      <secondaryie>metrics services <link linkend="IG31340177383"><command>pmFetch</command> Function</link></secondaryie>
      <secondaryie>Name Space services <link linkend="IG31340177292"><command>pmGetChildren</command> Function</link></secondaryie>
      <secondaryie>program evolution <link linkend="IG31340177455">Accommodating Program Evolution</link></secondaryie>
      <secondaryie>programming issues <link linkend="IG31340177443">PMAPI Programming Issues and Examples</link> <link linkend="IG31340177444">PMAPI Programming Issues and Examples</link></secondaryie>
      <secondaryie>programming style <link linkend="IG31340177281">PMAPI Programming Style and Interaction</link></secondaryie>
      <secondaryie>record-mode services <link linkend="IG31340177395"><command>pmRecordAddHost</command> Function</link></secondaryie>
      <secondaryie>time control services <link linkend="IG31340177413">PMAPI Time Control Services</link></secondaryie>
      <secondaryie>timezone services <link linkend="IG31340177377"><command>pmNewContextZone</command> Function</link></secondaryie>
      <secondaryie>variable length arguments <link linkend="IG31340177282">Variable Length Argument and Results Lists</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>pmAtomStr function <link linkend="IG31340177174">Management of Evolution within a PMDA</link> <link linkend="IG31340177433"><command>pmAtomStr</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmAtomValue structure <link linkend="IG31340177190">Simple PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PMCD</primaryie>
      <secondaryie>acronym <link linkend="IG31340177573">Acronyms</link></secondaryie>
      <secondaryie>distributed collection <link linkend="IG3134017718">Distributed Collection</link></secondaryie>
      <secondaryie>overview <link linkend="IG3134017713">PCP Architecture</link></secondaryie>
      <secondaryie>pmReconnectContext function <link linkend="IG31340177373"><command>pmReconnectContext</command> Function</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>PMCD_RECONNECT_TIMEOUT variable <link linkend="IG31340177376"><command>pmReconnectContext</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PMCD_REQUEST_TIMOUT variable <link linkend="IG31340177389"><command>pmFetch</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmchart command <link linkend="IG313401779">PCP Architecture</link> <link linkend="IG3134017769">Implementing a PMDA</link> <link linkend="IG31340177499">Configuring the Trace PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmclient tool <link linkend="IG3134017741">Application and Agent Development</link></primaryie>
      <secondaryie>brief description <link linkend="IG3134017738">Application and Agent Development</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>pmConvScale function <link linkend="IG31340177173">Management of Evolution within a PMDA</link> <link linkend="IG31340177426"><command>pmConvScale</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PMDA</primaryie>
      <secondaryie>acronym <link linkend="IG31340177575">Acronyms</link></secondaryie>
      <secondaryie>architecture <link linkend="IG3134017778">PMDA Architecture</link></secondaryie>
      <secondaryie>checklist <link linkend="IG3134017766">Implementing a PMDA</link></secondaryie>
      <secondaryie>development <link linkend="IG3134017744">PMDA Development</link></secondaryie>
      <secondaryie>evolution <link linkend="IG31340177161">Management of Evolution within a PMDA</link></secondaryie>
      <secondaryie>help text <link linkend="IG31340177157">PMDA Help Text</link></secondaryie>
      <secondaryie>initialization <link linkend="IG31340177207">Initializing a PMDA</link></secondaryie>
      <secondaryie>Install script <link linkend="IG31340177240">Installing a PMDA</link> <link linkend="IG31340177246">Upgrading a PMNS to Include Metrics from a New PMDA</link></secondaryie>
      <secondaryie>integration <link linkend="IG31340177237">Integration of a PMDA</link></secondaryie>
      <secondaryie>interface <link linkend="IG31340177178">PMDA Interface</link></secondaryie>
      <secondaryie>introduction <link linkend="IG313401773">Programming Performance Co-Pilot</link></secondaryie>
      <secondaryie>man page <link linkend="IG3134017720">Distributed Collection</link></secondaryie>
      <secondaryie>removal <link linkend="IG31340177247">Removing a PMDA</link></secondaryie>
      <secondaryie>structures <link linkend="IG31340177199">PMDA Structures</link></secondaryie>
      <secondaryie>trace <link linkend="IG31340177463">Instrumenting Applications</link></secondaryie>
      <secondaryie>writing <link linkend="IG3134017759">Writing a PMDA</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>pmda library <link linkend="IG3134017742">Application and Agent Development</link></primaryie>
      <seeie>PMDA</seeie>
</indexentry>

<indexentry>
   <primaryie>mmv library <link linkend="IG3134017742nat">Application and Agent Development</link></primaryie>
      <seeie>MMV</seeie>
</indexentry>

<indexentry>
   <primaryie>PMDA_PMID macro <link linkend="IG31340177122">Data Structures</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaAttribute callback <link linkend="IG31340180nat">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaChildren callback <link linkend="IG31340179nat">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdacisco man page <link linkend="IG31340177108">Caching PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaConnect man page <link linkend="IG31340177202">PMDA Structures</link> <link linkend="IG31340177223">Daemon Initialization</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaDaemon man page <link linkend="IG31340177204">PMDA Structures</link> <link linkend="IG31340177220">Daemon Initialization</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaDesc callback <link linkend="IG31340177182">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaDSO man page <link linkend="IG31340177203">PMDA Structures</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaExt structure <link linkend="IG31340177185">Overview</link> <link linkend="IG31340177201">PMDA Structures</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaFetch callback <link linkend="IG31340177179">Overview</link> <link linkend="IG31340177214">Trivial PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaGetOptions man page <link linkend="IG31340177205">PMDA Structures</link> <link linkend="IG31340177221">Daemon Initialization</link> <link linkend="IG31340177225">Daemon Initialization</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaIndom structure <link linkend="IG31340177139">Data Structures</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaInit man page <link linkend="IG31340177142">Data Structures</link> <link linkend="IG31340177206">PMDA Structures</link> <link linkend="IG31340177211">Common Initialization</link> <link linkend="IG31340177212">Common Initialization</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaInstance callback <link linkend="IG31340177181">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaInstid structure <link linkend="IG31340177138">Data Structures</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaInterface structure <link linkend="IG31340177200">PMDA Structures</link> <link linkend="IG31340177208">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaMain man page <link linkend="IG31340177224">Daemon Initialization</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaMetric structure <link linkend="IG31340177123">Data Structures</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaName callback <link linkend="IG31340178nat">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaOpenLog man page <link linkend="IG31340177222">Daemon Initialization</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaPMID callback <link linkend="IG31340177nat">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaProfile callback <link linkend="IG31340177180">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaStore callback <link linkend="IG31340177184">Overview</link> <link linkend="IG31340177196"><literal>simple_store</literal> in the Simple PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdaText callback <link linkend="IG31340177183">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdatrace man page <link linkend="IG31340177466">Performance Instrumentation and Tracing</link> <link linkend="IG31340177469">Performance Instrumentation and Tracing</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmdbg man page <link linkend="IG31340177230">Overview</link> <link linkend="IG31340177231">Debugging Information</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmDelProfile function <link linkend="IG31340177323">PMAPI Context Services</link> <link linkend="IG31340177364"><command>pmDelProfile</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmDesc structure <link linkend="IG31340177119">Data Structures</link> <link linkend="IG31340177164">Management of Evolution within a PMDA</link> <link linkend="IG31340177268">Performance Metric Descriptions</link> <link linkend="IG31340177274">Performance Metric Descriptions</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmDestroyContext function <link linkend="IG31340177354"><command>pmDestroyContext</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmDupContext function <link linkend="IG31340177324">PMAPI Context Services</link> <link linkend="IG31340177355"><command>pmDupContext</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmErrStr function <link linkend="IG31340177419"><command>pmErrStr</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmExtractValue function <link linkend="IG31340177172">Management of Evolution within a PMDA</link> <link linkend="IG31340177420"><command>pmExtractValue</command> Function</link> <link linkend="IG31340177428"><command>pmConvScale</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmFetch function <link linkend="IG31340177275">Performance Metrics Values</link> <link linkend="IG31340177280">Performance Metrics Values</link> <link linkend="IG31340177289">Variable Length Argument and Results Lists</link> <link linkend="IG31340177325">PMAPI Context Services</link> <link linkend="IG31340177351"><command>pmNewContext</command> Function</link> <link linkend="IG31340177370"><command>pmSetMode</command> Function</link> <link linkend="IG31340177385"><command>pmFetch</command> Function</link> <link linkend="IG31340177387"><command>pmFetch</command> Function</link> <link linkend="IG31340177390"><command>pmFreeResult</command> Function</link> <link linkend="IG31340177411"><command>pmFetchArchive</command> Function</link> <link linkend="IG31340177435"><command>pmPrintValue</command> Function</link> <link linkend="IG31340177440"><command>pmSortInstances</command> Function</link> <link linkend="IG31340177449">Symbolic Association between a Metric's Name and Value</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmFetch man page <link linkend="IG3134017782">Overview</link> <link linkend="IG31340177166">Management of Evolution within a PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmFetchArchive function <link linkend="IG31340177326">PMAPI Context Services</link> <link linkend="IG31340177369"><command>pmSetMode</command> Function</link> <link linkend="IG31340177412"><command>pmFetchArchive</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmflush function <link linkend="IG31340177437"><command>pmflush</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmFreeResult function <link linkend="IG31340177290">Variable Length Argument and Results Lists</link> <link linkend="IG31340177386"><command>pmFetch</command> Function</link> <link linkend="IG31340177391"><command>pmFreeResult</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmgadgets command <link linkend="IG3134017770">Implementing a PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmgenmap tool <link linkend="IG3134017743">Application and Agent Development</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmGetArchiveEnd function <link linkend="IG31340177327">PMAPI Context Services</link> <link linkend="IG31340177403"><command>pmGetArchiveEnd</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmGetArchiveLabel function <link linkend="IG31340177328">PMAPI Context Services</link> <link linkend="IG31340177399"><command>pmGetArchiveLabel</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmGetChildren function <link linkend="IG3134017785nat">Overview</link> <link linkend="IG31340177288">Variable Length Argument and Results Lists</link> <link linkend="IG31340177293"><command>pmGetChildren</command> Function</link> <link linkend="IG31340177294"><command>pmGetChildrenStatus</command> Function</link> <link linkend="IG31340177329">PMAPI Context Services</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmGetChildrenStatus function <link linkend="IG31340177330">PMAPI Context Services</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmGetContextHostName function <link linkend="IG31340177330nat">PMAPI Context Services</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmGetInDom function <link linkend="IG3134017786">Overview</link> <link linkend="IG31340177286">Variable Length Argument and Results Lists</link> <link linkend="IG31340177316"><command>pmGetInDom</command> Function</link> <link linkend="IG31340177332">PMAPI Context Services</link> <link linkend="IG31340177367"><command>pmSetMode</command> Function</link> <link linkend="IG31340177408"><command>pmGetInDomArchive</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmGetInDomArchive function <link linkend="IG31340177333">PMAPI Context Services</link> <link linkend="IG31340177407"><command>pmGetInDomArchive</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmGetPMNSLocation function <link linkend="IG31340177295"><command>pmGetPMNSLocation</command> Function</link> <link linkend="IG31340177331">PMAPI Context Services</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PMID</primaryie>
      <secondaryie>acronym <link linkend="IG31340177576">Acronyms</link></secondaryie>
      <secondaryie>introduction <link linkend="IG3134017724">Name Space</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>pmIDStr function <link linkend="IG31340177430"><command>pmIDStr</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmie command <link linkend="IG3134017772">Implementing a PMDA</link> <link linkend="IG31340177252">Configuring PCP Tools</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmieconf command <link linkend="IG3134017773">Implementing a PMDA</link> <link linkend="IG31340177253">Configuring PCP Tools</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmInDomStr function <link linkend="IG31340177431"><command>pmInDomStr</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmLoadNameSpace function <link linkend="IG31340177296"><command>pmLoadNameSpace</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmlogconf command <link linkend="IG31340177255">Configuring PCP Tools</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmlogger command <link linkend="IG3134017776">Implementing a PMDA</link> <link linkend="IG31340177254">Configuring PCP Tools</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmLookupDesc function <link linkend="IG3134017788">Overview</link> <link linkend="IG31340177120">Data Structures</link> <link linkend="IG31340177163">Management of Evolution within a PMDA</link> <link linkend="IG31340177307"><command>pmLookupDesc</command> Function</link> <link linkend="IG31340177334">PMAPI Context Services</link> <link linkend="IG31340177368"><command>pmSetMode</command> Function</link> <link linkend="IG31340177421"><command>pmExtractValue</command> Function</link> <link linkend="IG31340177427"><command>pmConvScale</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmLookupInDom function <link linkend="IG31340177318"><command>pmLookupInDom</command> Function</link> <link linkend="IG31340177335">PMAPI Context Services</link> <link linkend="IG31340177366"><command>pmSetMode</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmLookupInDomArchive function <link linkend="IG31340177336">PMAPI Context Services</link> <link linkend="IG31340177409"><command>pmLookupInDomArchive</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmLookupInDomText function <link linkend="IG31340177310"><command>pmLookupInDomText</command> Function</link> <link linkend="IG31340177337">PMAPI Context Services</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmLookupName function <link linkend="IG3134017782nat">Overview</link> <link linkend="IG31340177298"><command>pmLookupName</command> Function</link> <link linkend="IG31340177338">PMAPI Context Services</link> <link linkend="IG31340177448">Symbolic Association between a Metric's Name and Value</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmLookupText function <link linkend="IG3134017790">Overview</link> <link linkend="IG31340177168">Management of Evolution within a PMDA</link> <link linkend="IG31340177285">Variable Length Argument and Results Lists</link> <link linkend="IG31340177312"><command>pmLookupText</command> Function</link> <link linkend="IG31340177339">PMAPI Context Services</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmNameAll function <link linkend="IG31340177299"><command>pmNameAll</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmNameID function <link linkend="IG31340177287">Variable Length Argument and Results Lists</link> <link linkend="IG31340177300"><command>pmNameID</command> Function</link> <link linkend="IG31340177340">PMAPI Context Services</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmNameInDom function <link linkend="IG31340177284">Variable Length Argument and Results Lists</link> <link linkend="IG31340177319"><command>pmNameInDom</command> Function</link> <link linkend="IG31340177341">PMAPI Context Services</link> <link linkend="IG31340177365"><command>pmSetMode</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmNameInDomArchive function <link linkend="IG31340177342">PMAPI Context Services</link> <link linkend="IG31340177410"><command>pmNameInDomArchive</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmNewContext function <link linkend="IG31340177350"><command>pmNewContext</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmNewContextZone function <link linkend="IG31340177379"><command>pmNewContextZone</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmNewZone function <link linkend="IG31340177380"><command>pmNewZone</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PMNS</primaryie>
      <secondaryie>acronym <link linkend="IG31340177577">Acronyms</link></secondaryie>
      <secondaryie>distributed <link linkend="IG3134017729">Distributed PMNS</link></secondaryie>
      <secondaryie>upgrade <link linkend="IG31340177245">Upgrading a PMNS to Include Metrics from a New PMDA</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>pmns man page <link linkend="IG31340177154">Name Space</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmNumberStr function <link linkend="IG31340177434"><command>pmNumberStr</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmParseInterval function <link linkend="IG31340177441"><command>pmParseInterval</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmParseMetricSpec function <link linkend="IG31340177442"><command>pmParseMetricSpec</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmprintf function <link linkend="IG31340177438"><command>pmprintf</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmPrintValue function <link linkend="IG31340177176">Management of Evolution within a PMDA</link> <link linkend="IG31340177436"><command>pmPrintValue</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmReconnectContext function <link linkend="IG31340177374"><command>pmReconnectContext</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmRecordAddHost function <link linkend="IG31340177394"><command>pmRecordAddHost</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmRecordControl function <link linkend="IG31340177397"><command>pmRecordControl</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmRecordSetup function <link linkend="IG31340177398"><command>pmRecordSetup</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmSetMode function <link linkend="IG31340177343">PMAPI Context Services</link> <link linkend="IG31340177371"><command>pmSetMode</command> Function</link> <link linkend="IG31340177404"><command>pmGetArchiveEnd</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmSortInstances function <link linkend="IG31340177439"><command>pmSortInstances</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmstore function <link linkend="IG3134017793">Overview</link> <link linkend="IG31340177117">Metrics</link> <link linkend="IG31340177171">Management of Evolution within a PMDA</link> <link linkend="IG31340177195"><literal>simple_store</literal> in the Simple PMDA</link> <link linkend="IG31340177235">Debugging Information</link> <link linkend="IG31340177276">Performance Metrics Values</link> <link linkend="IG31340177344">PMAPI Context Services</link> <link linkend="IG31340177392"><command>pmStore</command> Function</link> <link linkend="IG31340177393"><command>pmStore</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PMTRACE_STATE_API flag <link linkend="IG31340177538">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PMTRACE_STATE_ASYNC flag <link linkend="IG31340177547">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PMTRACE_STATE_COMMS flag <link linkend="IG31340177539">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PMTRACE_STATE_NOAGENT flag <link linkend="IG31340177543">Configuring the Trace Library</link> <link linkend="IG31340177544">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PMTRACE_STATE_NONE flag <link linkend="IG31340177536">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PMTRACE_STATE_PDU flag <link linkend="IG31340177540">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>PMTRACE_STATE_PDUBUF flag <link linkend="IG31340177541">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmtraceabort function <link linkend="IG31340177507">Transactions </link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmtracebegin function <link linkend="IG31340177504">Transactions </link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmtracend function <link linkend="IG31340177505">Transactions </link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmtraceobs function <link linkend="IG31340177518">Observations and Counters</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmtracepoint function <link linkend="IG31340177515">Point Tracing </link> <link linkend="IG31340177519">Observations and Counters</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmtracestate call <link linkend="IG31340177534">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmTraversePMNS function <link linkend="IG3134017786nat">Overview</link> <link linkend="IG31340177301"><command>pmTraversePMNS</command> Function</link> <link linkend="IG31340177345">PMAPI Context Services</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>__pmParseHostAttrsSpec function <link linkend="IG3134017790nat">Overview</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmTypeStr function <link linkend="IG31340177175">Management of Evolution within a PMDA</link> <link linkend="IG31340177432"><command>pmTypeStr</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmUnitsStr function <link linkend="IG31340177429"><command>pmUnitsStr</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmUnloadNameSpace function <link linkend="IG31340177304"><command>pmUnloadNameSpace</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmUnpackEventRecords function <link linkend="IG31340177280nat">Event Monitor Considerations</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmUseContext function <link linkend="IG31340177353"><command>pmNewContext</command> Function</link> <link linkend="IG31340177356"><command>pmUseContext</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmUseZone function <link linkend="IG31340177381"><command>pmUseZone</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmWhichContext function <link linkend="IG31340177358"><command>pmWhichContext</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>pmWhichZone function <link linkend="IG31340177382"><command>pmWhichZone</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>point tracing <link linkend="IG31340177514">Point Tracing </link></primaryie>
</indexentry>

<indexentry>
   <primaryie>program evolution <link linkend="IG31340177456">Accommodating Program Evolution</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>programming components <link linkend="IG313401771">Programming Performance Co-Pilot</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>protocol data units</primaryie>
      <seeie>PDU</seeie>
</indexentry>

<indexentry>
   <primaryie>pthreads man page <link linkend="IG31340177152">Latency and Threads of Control</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>record-mode services <link linkend="IG31340177396"><command>pmRecordAddHost</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>removal script <link linkend="IG31340177248">Removing a PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>restarting pmcd <link linkend="IG31340177242">Installing a PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>retrospective analysis <link linkend="IG3134017730">Retrospective Sources of Performance Metrics</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>ring buffers <link linkend="IG31340177490">Rolling-Window Periodic Sampling</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>rolling-window sampling <link linkend="IG31340177477">Sampling Techniques</link> <link linkend="IG31340177488">Rolling-Window Periodic Sampling</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>sample duration <link linkend="IG31340177489">Rolling-Window Periodic Sampling</link> <link linkend="IG31340177496">Configuring the Trace PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>sampling techniques <link linkend="IG31340177478">Sampling Techniques</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>scale and dimensionality <link linkend="IG31340177273">Performance Metric Descriptions</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>semantic types <link linkend="IG31340177127">Semantics</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>sequential log files <link linkend="IG3134017763">Implementing a PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>service time <link linkend="IG31340177558">Instrumenting Applications</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>simple periodic sampling <link linkend="IG31340177486">Simple Periodic Sampling</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>simple PMDA</primaryie>
      <secondaryie>2 branches, 4 metrics <link linkend="IG31340177156">Name Space</link></secondaryie>
      <secondaryie>as daemon <link linkend="IG31340177105">Daemon PMDA</link></secondaryie>
      <secondaryie>DSO <link linkend="IG3134017797">DSO PMDA</link></secondaryie>
      <secondaryie>initialization <link linkend="IG31340177216">Simple PMDA</link></secondaryie>
      <secondaryie>pmdaFetch callback <link linkend="IG31340177188">Simple PMDA</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>simple_init function <link linkend="IG3134017799">DSO PMDA</link> <link linkend="IG31340177189">Simple PMDA</link> <link linkend="IG31340177217">Simple PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>simple_store function <link linkend="IG31340177234">Debugging Information</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>simple.color metric <link linkend="IG31340177191">Simple PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>simple.now metric <link linkend="IG31340177193">Simple PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>simple.store metric <link linkend="IG31340177194"><literal>simple_store</literal> in the Simple PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>simple.time metric <link linkend="IG31340177192">Simple PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>snapshot files <link linkend="IG3134017764">Implementing a PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>software <link linkend="IG3134017732">Overview of Component Software</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>specific instance domain <link linkend="IG31340177346">PMAPI Context Services</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>state flags <link linkend="IG31340177523">Configuring the Trace Library</link> <link linkend="IG31340177532">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>storage of metrics <link linkend="IG31340177116">Metrics</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>symbolic association <link linkend="IG31340177446">Symbolic Association between a Metric's Name and Value</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>synchronous protocol <link linkend="IG31340177537">Configuring the Trace Library</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>target domain <link linkend="IG3134017762">Implementing a PMDA</link> <link linkend="IG31340177115">Metrics</link> <link linkend="IG31340177148">Extracting the Information</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>TCP/IP <link linkend="IG31340177527">Configuring the Trace Library</link> <link linkend="IG31340177578">Acronyms</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>testing and debugging <link linkend="IG31340177226">Testing and Debugging a PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>threaded applications <link linkend="IG3134017756">Library Reentrancy and Threaded Applications</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>time control services <link linkend="IG31340177414">PMAPI Time Control Services</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>timezone services <link linkend="IG31340177378"><command>pmNewContextZone</command> Function</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>tool configuration <link linkend="IG31340177250">Configuring PCP Tools</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>trace facilities <link linkend="IG313401776">Programming Performance Co-Pilot</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>trace PMDA</primaryie>
      <secondaryie>command-line options <link linkend="IG31340177494">Configuring the Trace PMDA</link></secondaryie>
      <secondaryie>description <link linkend="IG31340177462">Instrumenting Applications</link></secondaryie>
      <secondaryie>design <link linkend="IG31340177470">Trace PMDA Design</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>trace.control.reset metric <link linkend="IG31340177501">Configuring the Trace PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>trace.observe metrics <link linkend="IG31340177520">Observations and Counters</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>trace.observe.rate metric <link linkend="IG31340177479">Sampling Techniques</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>trace.point.count metric <link linkend="IG31340177516">Point Tracing </link></primaryie>
</indexentry>

<indexentry>
   <primaryie>trace.point.rate metric <link linkend="IG31340177517">Point Tracing </link> <link linkend="IG31340177480">Sampling Techniques</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>trace.transact.ave_time metric <link linkend="IG31340177481">Sampling Techniques</link> <link linkend="IG31340177508">Transactions </link></primaryie>
</indexentry>

<indexentry>
   <primaryie>trace.transact.count metric <link linkend="IG31340177509">Transactions </link></primaryie>
</indexentry>

<indexentry>
   <primaryie>trace.transact.max_time metric <link linkend="IG31340177482">Sampling Techniques</link> <link linkend="IG31340177510">Transactions </link></primaryie>
</indexentry>

<indexentry>
   <primaryie>trace.transact.min_time metric <link linkend="IG31340177483">Sampling Techniques</link> <link linkend="IG31340177511">Transactions </link></primaryie>
</indexentry>

<indexentry>
   <primaryie>trace.transact.rate metric <link linkend="IG31340177484">Sampling Techniques</link> <link linkend="IG31340177512">Transactions </link></primaryie>
</indexentry>

<indexentry>
   <primaryie>trace.transact.total_time metric <link linkend="IG31340177513">Transactions </link></primaryie>
</indexentry>

<indexentry>
   <primaryie>transactions <link linkend="IG31340177506">Transactions </link></primaryie>
</indexentry>

<indexentry>
   <primaryie>trivial PMDA</primaryie>
      <secondaryie>callbacks <link linkend="IG31340177186">Trivial PMDA</link></secondaryie>
      <secondaryie>initialization <link linkend="IG31340177213">Trivial PMDA</link></secondaryie>
      <secondaryie>singular metric <link linkend="IG31340177124">Data Structures</link></secondaryie>
</indexentry>

<indexentry>
   <primaryie>trivial_init function <link linkend="IG31340177187">Trivial PMDA</link> <link linkend="IG31340177215">Trivial PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>two or three dimensional arrays <link linkend="IG31340177135">N Dimensional Data</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>type field <link linkend="IG31340177170">Management of Evolution within a PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>unavailable metrics support <link linkend="IG31340177162">Management of Evolution within a PMDA</link></primaryie>
</indexentry>

<indexentry>
   <primaryie>working buffers <link linkend="IG31340177476">Application Interaction</link> <link linkend="IG31340177491">Rolling-Window Periodic Sampling</link></primaryie>
</indexentry>

</index>

</book>