path: root/doc/guide/files/infr.design.xml

<!-- $NetBSD: infr.design.xml,v 1.10 2016/07/09 16:19:15 rillig Exp $ -->

<chapter id="infr.design"> <?dbhtml filename="infr.design.html"?>
<title>Design of the pkgsrc infrastructure</title>

	<para>The pkgsrc infrastructure consists of many small Makefile
	fragments. Each such fragment needs a properly specified
	interface. This chapter explains how such an interface looks
	like.</para>

<!--
<sect1 id="infr.intro">
<title>Introduction</title>

</sect1>
-->

<sect1 id="infr.vardef">
<title>The meaning of variable definitions</title>

	<para>Whenever a variable is defined in the pkgsrc
	infrastructure, the location and the way of definition provide
	much information about the intended use of that variable.
	Additionally, more documentation may be found in a header
	comment or in this pkgsrc guide.</para>

	<para>A special file is
	<filename>mk/defaults/mk.conf</filename>, which lists all
	variables that are intended to be user-defined. They are either
	defined using the <literal>?=</literal> operator or they are
	left undefined because defining them to anything would
	effectively mean <quote>yes</quote>. All these variables may be
	overridden by the pkgsrc user in the <varname>MAKECONF</varname>
	file.</para>

	<para>Outside this file, the following conventions apply:
	Variables that are defined using the <literal>?=</literal>
	operator may be overridden by a package.</para>

	<para>Variables that are defined using the <literal>=</literal>
	operator may be used read-only at run-time.</para>

	<para>Variables whose name starts with an underscore must not be
	accessed outside the pkgsrc infrastructure at all. They may
	change without further notice.</para>

	<note><para>These conventions are currently not applied
	consistently to the complete pkgsrc
	infrastructure.</para></note>

</sect1>

<sect1 id="infr.vardef.problems">
<title>Avoiding problems before they arise</title>

	<para>All variables that contain lists of things should default
	to being empty. Two examples that do not follow this rule are
	<varname>USE_LANGUAGES</varname> and
	<varname>DISTFILES</varname>. These variables cannot simply be
	modified using the <literal>+=</literal> operator in package
	<filename>Makefile</filename>s (or other files included by
	them), since there is no guarantee whether the variable is
	already set or not, and what its value is. In the case of
	<varname>DISTFILES</varname>, the packages <quote>know</quote>
	the default value and just define it as in the following
	example.</para>

<programlisting>
DISTFILES=      ${DISTNAME}${EXTRACT_SUFX} additional-files.tar.gz
</programlisting>

	<para>Because of the selection of this default value, the same
	value appears in many package Makefiles. Similarly for
	<varname>USE_LANGUAGES</varname>, but in this case the default
	value (<quote><literal>c</literal></quote>) is so short that it
	doesn't stand out. Nevertheless it is mentioned in many
	files.</para>

</sect1>

<sect1 id="infr.var">
<title>Variable evaluation</title>

<sect2 id="infr.var.load">
<title>At load time</title>

	<para>Variable evaluation takes place either at load time or at
	runtime, depending on the context in which they occur. The
	contexts where variables are evaluated at load time are:</para>

	<itemizedlist>

	<listitem><para>The right hand side of the <literal>:=</literal>
	and <literal>!=</literal> operators,</para></listitem>

	<listitem><para>Make directives like <literal>.if</literal> or
	<literal>.for</literal>,</para></listitem>

	<listitem><para>Dependency lines.</para></listitem>

	</itemizedlist>

	<para>A special exception are references to the iteration
	variables of <literal>.for</literal> loops, which are expanded
	inline, no matter in which context they appear.</para>

	<para>As the values of variables may change during load time,
	care must be taken not to evaluate them by accident. Typical
	examples for variables that should not be evaluated at load time
	are <varname>DEPENDS</varname> and
	<varname>CONFIGURE_ARGS</varname>. To make the effect more
	clear, here is an example:</para>

<programlisting>
CONFIGURE_ARGS=         # none
CFLAGS=                 -O
CONFIGURE_ARGS+=        CFLAGS=${CFLAGS:Q}

CONFIGURE_ARGS:=        ${CONFIGURE_ARGS}

CFLAGS+=                -Wall
</programlisting>

	<para>This code shows how the use of the <literal>:=</literal>
	operator can quickly lead to unexpected results. The first
	paragraph is fairly common code. The second paragraph evaluates
	the <varname>CONFIGURE_ARGS</varname> variable, which results in
	<literal>CFLAGS=-O</literal>. In the third paragraph, the
	<literal>-Wall</literal> is appended to the
	<varname>CFLAGS</varname>, but this addition will not appear in
	<varname>CONFIGURE_ARGS</varname>. In actual code, the three
	paragraphs from above typically occur in completely unrelated
	files.</para>

</sect2>
<sect2 id="infr.var.run">
<title>At runtime</title>

	<para>After all the files have been loaded, the values of the
	variables cannot be changed anymore. Variables that are used in
	the shell commands are expanded at this point.</para>

</sect2>
</sect1>

<sect1 id="infr.varspec">
<title>How can variables be specified?</title>

	<para>There are many ways in which the definition and use of a
	variable can be restricted in order to detect bugs and violations
	of the (mostly unwritten) policies. A package can be checked with
	<literal>pkglint -Wall</literal> to see whether it meets these
	rules.</para>

</sect1>

<sect1 id="infr.design.intf">
<title>Designing interfaces for Makefile fragments</title>

	<para>Most of the <filename>.mk</filename> files fall into one
	of the following classes. Cases where a file falls into more
	than one class should be avoided as it often leads to subtle
	bugs.</para>

<sect2 id="infr.design.intf.proc">
<title>Procedures with parameters</title>

	<para>In a traditional imperative programming language some of
	the <filename>.mk</filename> files could be described as
	procedures. They take some input parameters and&mdash;after
	inclusion&mdash;provide a result in output parameters. Since all
	variables in <filename>Makefile</filename>s have global scope
	care must be taken not to use parameter names that have already
	another meaning. For example, <varname>PKGNAME</varname> is a
	bad choice for a parameter name.</para>

	<para>Procedures are completely evaluated at preprocessing time.
	That is, when calling a procedure all input parameters must be
	completely resolvable. For example,
	<varname>CONFIGURE_ARGS</varname> should never be an input
	parameter since it is very likely that further text will be
	added after calling the procedure, which would effectively apply
	the procedure to only a part of the variable. Also, references
	to other variables will be modified after calling the
	procedure.</para>

	<para>A procedure can declare its output parameters either as
	suitable for use in preprocessing directives or as only
	available at runtime. The latter alternative is for variables
	that contain references to other runtime variables.</para>

	<para>Procedures shall be written such that it is possible to
	call the procedure more than once. That is, the file must not
	contain multiple-inclusion guards.</para>

	<para>Examples for procedures are
	<filename>mk/bsd.options.mk</filename> and
	<filename>mk/buildlink3/bsd.builtin.mk</filename>. To express
	that the parameters are evaluated at load time, they should be
	assigned using the <literal>:=</literal> operator, which should
	be used only for this purpose.</para>

</sect2>
<sect2 id="infr.design.intf.action">
<title>Actions taken on behalf of parameters</title>

	<para>Action files take some input parameters and may define
	runtime variables. They shall not define loadtime variables.
	There are action files that are included implicitly by the
	pkgsrc infrastructure, while other must be included
	explicitly.</para>

	<para>An example for action files is
	<filename>mk/subst.mk</filename>.</para>

</sect2>
</sect1>

<sect1 id="infr.order">
<title>The order in which files are loaded</title>

	<para>Package <filename>Makefile</filename>s usually consist of
	a set of variable definitions, and include the file
	<filename>../../mk/bsd.pkg.mk</filename> in the very last line.
	Before that, they may also include various other
	<filename>*.mk</filename> files if they need to query the
	availability of certain features like the type of compiler or
	the X11 implementation. Due to the heavy use of preprocessor
	directives like <literal>.if</literal> and
	<literal>.for</literal>, the order in which the files are loaded
	matters.</para>

	<para>This section describes at which point the various files
	are loaded and gives reasons for that order.</para>

<sect2 id="infr.order.prefs">
<title>The order in <filename>bsd.prefs.mk</filename></title>

	<para>The very first action in <filename>bsd.prefs.mk</filename>
	is to define some essential variables like
	<varname>OPSYS</varname>, <varname>OS_VERSION</varname> and
	<varname>MACHINE_ARCH</varname>.</para>

	<para>Then, the user settings are loaded from the file specified
	in <varname>MAKECONF</varname>, which is usually &mk.conf;.
	After that, those variables
	that have not been overridden by the user are loaded from
	<filename>mk/defaults/mk.conf</filename>.</para>

	<para>After the user settings, the system settings and platform
	settings are loaded, which may override the user
	settings.</para>

	<para>Then, the tool definitions are loaded. The tool wrappers
	are not yet in effect. This only happens when building a
	package, so the proper variables must be used instead of the
	direct tool names.</para>

	<para>As the last steps, some essential variables from the
	wrapper and the package system flavor are loaded, as well as the
	variables that have been cached in earlier phases of a package
	build.</para>

</sect2>

<sect2 id="infr.order.pkg">
<title>The order in <filename>bsd.pkg.mk</filename></title>

	<para>First, <filename>bsd.prefs.mk</filename> is loaded.</para>

	<para>Then, the various <filename>*-vars.mk</filename> files are
	loaded, which fill default values for those variables that have
	not been defined by the package. These variables may later
	be used even in unrelated files.</para>

	<para>Then, the file <filename>bsd.pkg.error.mk</filename>
	provides the target <literal>error-check</literal> that is added
	as a special dependency to all other targets that use
	<varname>DELAYED_ERROR_MSG</varname> or
	<varname>DELAYED_WARNING_MSG</varname>.</para>

	<para>Then, the package-specific hacks from
	<filename>hacks.mk</filename> are included.</para>

	<!-- bsd.pkg.use.mk -->

	<para>Then, various other files follow. Most of them don't have
	any dependencies on what they need to have included before or
	after them, though some do.</para>

	<para>The code to check <varname>PKG_FAIL_REASON</varname> and
	<varname>PKG_SKIP_REASON</varname> is then executed, which
	restricts the use of these variables to all the files that have
	been included before. Appearances in later files will be
	silently ignored.</para>

	<para>Then, the files for the main targets are included, in the
	order of later execution, though the actual order should not
	matter.</para>

	<para>At last, some more files are included that don't set any
	interesting variables but rather just define make targets to be
	executed.</para>

</sect2>
</sect1>
</chapter>