path: root/doc/guide/files/bulk.xml

<!-- $NetBSD: bulk.xml,v 1.35 2021/08/03 08:16:44 nia Exp $ -->

<chapter id="bulk">
<title>Creating binary packages for everything in pkgsrc (bulk
builds)</title>

<para>For a number of reasons, you may want to build binary packages
for a large selected set of packages in pkgsrc, or even for all pkgsrc packages.
For instance, when you have multiple machines that should run the same software,
it is wasted time if they all build their packages themselves from source.
Or you may want to build a list of packages you want and check them before
deploying onto production systems.
There is a way of getting a set of binary packages:
the bulk build system, or pbulk ("p" stands for "parallel").
This chapter describes how to set it up.</para>

<sect1 id="bulk.pre">
<title>Preparations</title>

<para>First of all, you have to decide whether you build all packages
or a limited set of them. Full bulk builds usually consume a lot more resources,
both space and time, than builds for some practical sets of packages.
A number of particularly heavy packages exist that are not actually
interesting to a wide audience. (The approximate resource consumption for a
full bulk build is given in section <xref linkend="bulk.req" />.)
For limited bulk builds you need to make a list of packages you want to build.
Note that all their dependencies will be built, so you don't need to track them manually.
</para>

<para>During bulk builds various packages are installed and deinstalled
in <filename>/usr/pkg</filename> (or whatever <filename>LOCALBASE</filename> is),
so make sure that you don't need any package during the builds.
Essentially, you should provide a fresh system, either a chroot environment
or something even more restrictive, depending on what the operating system provides,
or dedicate the whole physical machine.
As a useful side effect this makes sure that bulk builds cannot
break anything in your system. There have been numerous cases where
certain packages tried to install files outside the
<filename>LOCALBASE</filename> or wanted to edit some files in
<filename>/etc</filename>.
</para>

<!-- Developer-only information follows:
<para>Since a bulk build takes several days or even weeks to finish, you
should think about the setup before you start everything. Pay attention
to at least the following points:</para>

<itemizedlist>

<listitem><para>If you want to upload the binary packages to
ftp.NetBSD.org, make sure the setup complies to the requirements for binary
packages:</para>

<itemizedlist>

<listitem><para>To end up on ftp.NetBSD.org, the packages must be built
by a NetBSD developer on a trusted machine (that is, where you and only
you have root access).</para></listitem>

<listitem><para>Packages on ftp.NetBSD.org should only be created from
the stable branches (like &pkgsrc.version.latest;), so that users browsing the available
collections can see at a glance how old the packages
are.</para></listitem>

<listitem><para>The packages must be built as root, since some packages
require set-uid binaries at runtime, and creating those packages as
unprivileged user doesn't work well at the moment.</para></listitem>

</itemizedlist>
</listitem>

</itemizedlist>
-->
</sect1>

<sect1 id="bulk.pbulk">
<title>Running a bulk build</title>

<para>Running a bulk build works roughly as follows:</para>

<itemizedlist>
<listitem><para>First, build the pbulk infrastructure in a fresh pkgsrc location.</para></listitem>
<listitem><para>Then, build each of the packages from a clean installation directory using the infrastructure.</para></listitem>
</itemizedlist>

<sect2 id="bulk.pbulk.conf">
<title>Configuration</title>

<para>To simplify configuration, we provide the helper script <filename>mk/pbulk/pbulk.sh</filename>.</para>

<para>In order to use it, prepare a clear system (real one, chroot environment, jail, zone, virtual machine).
Configure network access to fetch distribution files.
Create a user with name "pbulk".</para>

<para>Fetch and extract pkgsrc. Use a command like one of these:</para>

<screen>
&rprompt; <userinput>(cd /usr &amp;&amp; ftp -o - https://cdn.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)</userinput>
&rprompt; <userinput>(cd /usr &amp;&amp; fetch -o - https://cdn.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)</userinput>
&rprompt; <userinput>(cd /usr &amp;&amp; cvs -Q -z3 -d anoncvs@anoncvs.NetBSD.org:/cvsroot get -P pkgsrc)</userinput>
</screen>

<para>Or any other way that fits (e.g., curl, wget).</para>

<para>Deploy and configure pbulk tools, e.g.:</para>

<screen>
&rprompt; <userinput>sh pbulk.sh -n # use native make, no bootstrap kit needed (for use on NetBSD)</userinput>
&rprompt; <userinput>sh pbulk.sh -n -c mk.conf.frag # native, apply settings from given mk.conf fragment</userinput>
&rprompt; <userinput>sh pbulk.sh -nlc mk.conf.frag # native, apply settings, configure for limited build</userinput>
</screen>

<note><para><filename>mk.conf.frag</filename> is a fragment of
<filename>mk.conf</filename> that contains settings you want to
apply to packages you build. For instance,</para>

<programlisting>
PKG_DEVELOPER=          yes     # perform more checks
X11_TYPE=               modular # use pkgsrc X11
SKIP_LICENSE_CHECK=     yes     # accept all licences (useful
                                # when building all packages)
</programlisting>
</note>
<!-- Think how to merge this or maintain short reference of useful settings.
<itemizedlist>
<listitem><para><literal><varname>WRKOBJDIR</varname>=/tmp/pbulk-outer</literal>, to keep <filename>/usr/pkgsrc</filename> free from any modifications,</para></listitem>
<listitem><para><literal><varname>DISTDIR</varname>=/distfiles</literal>, to have only one directory in which all distfiles (for the infrastructure and for the actual packages) are downloaded,</para></listitem>
<listitem><para><literal><varname>ACCEPTABLE_LICENSES</varname>+=...</literal>, to select some licenses additional to the usual Free/Open Source licenses that are acceptable to you,</para></listitem>
</itemizedlist>
-->

<para>If configured for limited list, replace the list in <filename>/usr/pbulk/etc/pbulk.list</filename>
with your list of packages, one per line without empty lines or comments. E.g.:</para>

<programlisting>
www/firefox
mail/thunderbird
misc/libreoffice4
</programlisting>

<para>At this point you can also review configuration in <filename>/usr/pbulk/etc</filename>
and make final amendments, if wanted.</para>

<para>Start it:</para>

<screen>
&rprompt; <userinput>/usr/pbulk/bin/bulkbuild</userinput>
</screen>

<para>After it finishes, you'll have <filename>/mnt</filename> filled with distribution files, binary packages, and reports,
plain text summary in <filename>/mnt/bulklog/meta/report.txt</filename>
</para>

<note><para>The <filename>pbulk.sh</filename> script does not cover all possible use cases.
While being ready to run, it serves as a good starting point to understand and build more complex setups.
The script is kept small enough for better understanding.</para>
</note>

<note><para>The <filename>pbulk.sh</filename> script supports running
unprivileged bulk build and helps configuring distributed bulk builds.
Distributed bulk builds support either building in worker chroots
(each node is a path to a different chroot)
that replicate the target system, including the pbulk prefix,
or remote machines (each node is
an IP address that must be accessible over SSH without a password).</para>
</note>

</sect2>

</sect1>

<sect1 id="bulk.req">
<title>Requirements of a full bulk build</title>

<para>A complete bulk build requires lots of disk space. Some of the
disk space can be read-only, some other must be writable. Some can be on
remote filesystems (such as NFS) and some should be local. Some can be
temporary filesystems, others must survive a sudden reboot.</para>

<itemizedlist>

<listitem><para>70 GB for the distfiles (read-write, remote, temporary)</para></listitem>

<listitem><para>60 GB for the binary packages (read-write, remote, permanent)</para></listitem>

<listitem><para>1 GB for the pkgsrc tree (read-only, remote, permanent)</para></listitem>

<listitem><para>5 GB for <filename>LOCALBASE</filename> (read-write, local, temporary)</para></listitem>

<listitem><para>10 GB for the log files (read-write, remote, permanent)</para></listitem>

<listitem><para>5 GB for temporary files (read-write, local, temporary)</para></listitem>

</itemizedlist>

</sect1>

<sect1 id="bulk.var">
<title>Bulk build variants</title>

<para>To ensure that pkgsrc packages work in different configurations, it
makes sense to run non-default bulk builds from time to time. This
section lists some ideas for bulk builds that intentionally let packages
fail if they don't follow the pkgsrc style.</para>

<!--
Needs to be implemented first, so that the remaining work is just
setting a variable in mk.conf, like in the other scenarios.

<sect2 id="bulk.var.subst_noop">
<title>Strict SUBST blocks</title>

<para>Up to May 2020, the SUBST blocks ignored files that didn't exist,
as well as substitutions that didn't have any effect. There were quite a
few SUBST blocks that were redundant, and these have been removed
already.</para>

<para>The next step would be to not only check that each filename pattern
has an effect but also that each substitution in SUBST_SED or SUBST_VARS
applies to at least one file.</para>

<para>To do this, <filename>mk/subst.mk</filename> would have to be
adjusted, in a similar way as the check for no-op SUBST_FILES. There are
several regression tests in <filename>regress/infra-unittests</filename>
that help to get all edge cases correct.</para>

<para>When a package fails this additional check, there are various
possible causes why the <varname>SUBST_SED</varname> became a
no-op:</para>

<orderedlist>

<listitem><para>The pattern used to be found in a former version of the
package, but is not needed anymore. In that case, just remove
it.</para></listitem>

<listitem><para>The pattern contains a typo. In that case, fix the typo
and bump <varname>PKGREVISION</varname>, since the fixed typo will
probably modify the resulting binary package.</para></listitem>

<listitem><para>There is a patch that is applied before the SUBST block,
and the patch accidentally contains the change that was intended for the
SUBST block. In that case, remove the respective hunk from the
patch.</para></listitem>

</orderedlist>

</sect2>
-->

<sect2 id="bulk.var.confopt">
<title>Detect unknown configure options</title>

<para>Add the following line to &mk.conf;.</para>

<programlisting>
GNU_CONFIGURE_STRICT=   yes
</programlisting>

<para>When a package fails this additional check, the most common cause
is that the configure option was valid for an older version of the
package but does not apply anymore. In that case, just remove it.</para>

</sect2>

<sect2 id="bulk.var.comperr">
<title>Detect classes of bugs by forcing compiler warnings</title>

<para>The job of a compiler is not restricted to producing executable
code, most compilers also detect typical programming mistakes. The pkgsrc
compiler wrappers make it easy to force compiler options when the package
is built. This can be used to find typical bugs across all packages that
are in pkgsrc. By reporting these bugs upstream, the packages will be
more reliable with the next updates.</para>

<para>Add some of the following lines to &mk.conf;:</para>

<programlisting>
CFLAGS+=        -Werror=char-subscripts
CFLAGS+=        -Werror=implicit-function-declaration
</programlisting>

<para>When a package fails to build using these stricter compiler
options, document the circumstances in which the compiler produced the
error message. This includes:</para>

<itemizedlist>

<listitem><para>The platform
(<varname>MACHINE_PLATFORM</varname>)</para></listitem>

<listitem><para>The source file</para></listitem>

<listitem><para>An excerpt of the code. GCC and Clang already do this as
part of the diagnostic.</para></listitem>

<listitem><para>The exact error message from the compiler.</para></listitem>

</itemizedlist>

<para>If a package produces these error messages, but the package is
fine, record this in your local &mk.conf;, like this, to skip this check
in the next builds:</para>

<programlisting>
.if ${PKGPATH} == category/package
# Version ${VERSION} failed on ${MACHINE_PLATFORM}:
# error message
# code
# Reason why the code does not need to be fixed.
BUILDLINK_TRANSFORM+=   rm:-Werror=char-subscripts
.endif
</programlisting>

<para>If the error messages from the compiler are valid and the code
needs to be fixed, prepare a local patch (see
<varname>LOCALPATCHES</varname>) and report the bug to the upstream
authors of the package, providing them with the information you collected
above.</para>

<para>Patches that are not essential for the package to work should only
be reported upstream but not committed to pkgsrc, to make future updates
easier.</para>

</sect2>

<sect2 id="bulk.var.builderr">
<title>Force compiler options only in the build phase</title>

<para>When adding custom compiler flags via <varname>CFLAGS</varname>,
these apply to all phases of the package build process. Especially in the
configure phase, adding <literal>-Werror</literal> leads to wrong
decisions. The GNU configure scripts feed many small test programs to the
C compiler to see whether certain headers are available, functions are
defined in a library and programs can be run. In many cases these
programs would not survive a strict compiler run with <literal>-Wall
-Wextra -Werror</literal>.</para>

<para>The pkgsrc infrastructure is flexible enough to support compiler
options being added between the <literal>configure</literal> and
<literal>build</literal> phases. It's a little more complicated than the
other examples in this section but still easy enough.</para>

<para>The basic idea is to use the pkgsrc compiler wrapper to inject the
desired compiler options. The compiler wrapper's original task is to hide
unwanted directories of include files and to normalize compiler options.
It does this by wrapping the compiler command and rewriting the command
line. To see this in action, run <command>bmake patch</command> in a
package directory and examine the
<filename>work/.cwrappers/config</filename> directory. It contains
individual configurations for the C compiler and the related tools. The
plan is to find a hook between the configure and build phases, and to
modify these configuration files at that point.</para>

<para>To find this hook, have a look at
<filename>mk/build/build.mk</filename>, which contains among others the
<literal>pre-build-checks-hook</literal>. The word
<literal>checks</literal> doesn't quite fit, but the
<literal>pre-build-hook</literal> sounds good enough.</para>

<para>The code to be included in &mk.conf; is:</para>

<programlisting>
# Just a few example options.
BUILD_ONLY_CFLAGS=      -Wall -Werror -O2 -DTEMPDIR='"/tmp"'

.if ${BUILD_ONLY_CFLAGS:U:M*}
pre-build-checks-hook: add-build-only-cflags

add-build-only-cflags: .PHONY
        ${RUN} cd ${CWRAPPERS_CONFIG_DIR};      \
        ${TEST} ! -f ${.TARGET} || exit 0;      \
        for flag in ${BUILD_ONLY_CFLAGS}; do    \
                ${ECHO} "append=$$flag" >> cc;  \
        done;                                   \
        > ${.TARGET}
.endif
</programlisting>

<para>(When editing the &mk.conf;, make sure that the commands of the
<literal>add-build-only-cflags</literal> target are indented with a tab,
not with spaces.)</para>

<para>The condition in the <literal>.if</literal> statement contains the
<literal>:U</literal> modifier to prevent parse errors if the variable
should be undefined, possibly because it is only defined for a subset of
the packages. The <literal>:M*</literal> modifier ensures that there is
at least one compiler option, to prevent a syntax error in the shell
parser.</para>

<para>The code around the <literal>${.TARGET}</literal> variable ensures
that the additional compiler options are only appended once, even if
<command>bmake build</command> is run multiple times. To do this, it
creates a marker file.</para>

<para>To verify that this setup works, run <command>bmake
configure</command> in a package directory. Up to now, everything works
as usual. Examine the directory
<filename>work/.cwrappers/config</filename> to see that the compiler
options from <varname>BUILD_ONLY_CFLAGS</varname> are not yet added to
the file <filename>cc</filename>. Examine the tail of the
<filename>work/.work.log</filename> file to see that the normal compiler
options are used.</para>

<para>Now run <command>bmake build</command>. This will append the
options to the file <filename>cc</filename> and will create the marker
file in the same directory. After that, the build starts as usual, but
with the added compiler options. Examine the tail of the file
<filename>work/.work.log</filename> to see that the lines starting with
<literal>[*]</literal> don't contain the compiler options, but the
corresponding lines starting with <literal>&lt;.&gt;</literal> do end
with these options.</para>

<para>Building packages using this setup variant and fixing the resulting
bugs is the same as in <xref linkend="bulk.var.comperr"/>.</para>

</sect2>

<sect2 id="bulk.var.dirs">
<title>Use custom directories</title>

<para>Some directories like <varname>PREFIX</varname>,
<varname>VARBASE</varname>, <varname>PKG_SYSCONFDIR</varname>,
<varname>PKGMANDIR</varname>, <varname>PKG_INFODIR</varname> can be
configured in pkgsrc. Set these to arbitrary paths during bootstrap or
afterwards in &mk.conf;.</para>

<programlisting>
PREFIX=         /a-random-uuid
PKG_SYSCONFDIR= /a-random-uuid
VARBASE=        /a-random-uuid
PKGMANDIR=	a-random-uuid
PKG_INFODIR=	a-random-uuid
</programlisting>

</sect2>

<sect2 id="bulk.var.warn">
<title>Turn warnings into errors</title>

<para>When building a package, warnings are typically ignored since they
just flow by and do not cause the build to fail immediately. To find
these warnings, redefine them to errors in &mk.conf;.</para>

<programlisting>
DELAYED_WARNING_MSG=    ${DELAYED_ERROR_MSG} "(was warning)"
WARNING_MSG=            ${FAIL_MSG} "(was warning)"
</programlisting>

<para>(There are many more classes of warnings in pkgsrc, and most of
them can be redefined with a simple definition like above.</para>

<para>If a package suggests to add <varname>USE_TOOLS+=perl</varname> to
the package Makefile, research whether the package actually needs Perl.
If it does, add <varname>USE_TOOLS+=perl</varname> to the package
Makefile, and if it doesn't, add
<varname>TOOLS_BROKEN+=perl</varname>.</para>

</sect2>

<sect2 id="bulk.var.pkglint">
<title>Reject packages for which pkglint reports errors</title>

<para>Using pkglint as part of the regular build process is mostly a
waste of time. If you want to fix some of the warnings, just run pkglint
recursively on the whole pkgsrc tree. This will take a few minutes (up to
10), which is much faster than a complete bulk build.</para>

</sect2>

<sect2 id="bulk.var.strings">
<title>Reject packages that contain forbidden strings</title>

<para>To ensure that the binary packages don't contain references to the
build directory, there is already <varname>CHECK_WRKREF</varname>. If
that variable includes the item <literal>extra</literal>, it is
possible to define additional patterns that must not appear in any
installed file. This is specified in &mk.conf;.</para>

<programlisting>
CHECK_WRKREF=                   extra
CHECK_WRKREF_EXTRA_DIRS+=       /usr/local
CHECK_WRKREF_EXTRA_DIRS+=       /usr/pkg
CHECK_WRKREF_EXTRA_DIRS+=	@[A-Z][A-Z]*@
</programlisting>

<para>The above patterns will probably generate many false positives,
therefore the results need to be taken with a grain of salt.</para>

</sect2>

<sect2 id="bulk.var.test">
<title>Reject packages whose self-test fails</title>

<para>To run the test suites that come with each package, add this line
to &mk.conf;.</para>

<programlisting>
PKGSRC_RUN_TEST=        yes
</programlisting>

<para>Be prepared that even the most basic packages fail this test. When
doing a bulk build with this, it will often abort in the early phase
where the packages are scanned for their dependencies since there are
cyclic dependencies. There is still a lot to do in this area.</para>

</sect2>

<sect2 id="bulk.var.shvar">
<title>Reject packages that use undefined shell variables</title>

<para>To catch typos in the shell snippets from the Makefile fragments,
add the <literal>-u</literal> flag to most of the commands by adding this
line to &mk.conf;.</para>

<programlisting>
RUN=    @set -eu;
</programlisting>

<para>After that, ensure that none of the bulk build log files (even
those for successfully built packages) contains the string
<literal>parameter not set</literal> or whatever error message the
command <command>sh -ceu '$undefined'</command> outputs.</para>

<para>See <filename>mk/misc/common.mk</filename> for the existing
definition.</para>

</sect2>

<sect2 id="bulk.var.quiet">
<title>Turn off verbose logging</title>

<para>The build logs of a package are often quite long. This allows error
messages or other interesting details to hide between the noise. To make
the actual error message stand out more, add these lines to
&mk.conf;.</para>

<programlisting>
GNU_CONFIGURE_QUIET=    yes
MAKE_FLAGS+=            -s
</programlisting>

<para>The <literal>-s</literal> option works for both GNU Make and BSD
Make. On exotic platforms with their own make, it may be a little
different.</para>

</sect2>

<!--
Needs to be implemented first, so that the remaining work is just
setting a variable in mk.conf, like in the other scenarios.

<sect2 id="bulk.var.options">
<title>Select random sets of options</title>

<para>Most bulk builds run with the default package options. This means
that other combinations of options are not regularly tested. To do this,
run a bulk build with these configurations.</para>

<itemizedlist>
<listitem><para>no options enabled</para></listitem>
<listitem><para>all options enabled</para></listitem>
<listitem><para>2n + 0</para></listitem>
<listitem><para>2n + 1</para></listitem>
<listitem><para>4n + 0..1</para></listitem>
<listitem><para>4n + 2..3</para></listitem>
<listitem><para>8n + 0..3</para></listitem>
<listitem><para>8n + 4..7</para></listitem>
<listitem><para>2048n + 0..1023</para></listitem>
<listitem><para>2048n + 1024..2047</para></listitem>
</itemizedlist>

<para>Open questions are:</para>

<itemizedlist>

<listitem><para>how to collect all options from the entire
pkgsrc</para></listitem>

<listitem><para>how to handle mutually exclusive
options</para></listitem>

<listitem><para>the sets of mutually exclusive options are defined
per-package</para></listitem>

<listitem><para>the sets of nonempty sets are defined
per-package</para></listitem>

</itemizedlist>

</sect2>
-->

<!--
Refers to bulk.var.options, which is not yet implemented.

<sect2 id="bulk.var.build_defs">
<title>Select random configurations of BUILD_DEFS</title>

<para>Just like the <varname>PKG_OPTIONS</varname>, the
<varname>BUILD_DEFS</varname> also allow different variants of pkgsrc to
be created. The same ideas as in <xref linkend="bulk.var.options"/>
apply.</para>

</sect2>
-->

</sect1>

  <sect1 id="creating-cdroms">
    <title>Creating a multiple CD-ROM packages collection</title>

    <para>After your pkgsrc bulk-build has completed, you may wish to
    create a CD-ROM set of the resulting binary packages to assist
    in installing packages on other machines.  The
    <filename role="pkg">pkgtools/cdpack</filename> package provides
    a simple tool for creating the ISO 9660 images.
    <command>cdpack</command> arranges the packages on the CD-ROMs in a
    way that keeps all the dependencies for a given package on the same
    CD as that package.</para>

    <sect2 id="cdpack-example">
      <title>Example of cdpack</title>

      <para>Complete documentation for cdpack is found in the cdpack(1)
      man page. The following short example assumes that the binary
      packages are left in
      <filename>/usr/pkgsrc/packages/All</filename> and that
      sufficient disk space exists in <filename>/u2</filename> to
      hold the ISO 9660 images.</para>

      <screen>
&rprompt; <userinput>mkdir /u2/images</userinput>
&rprompt; <userinput>pkg_add /usr/pkgsrc/packages/All/cdpack</userinput>
&rprompt; <userinput>cdpack /usr/pkgsrc/packages/All /u2/images</userinput>
      </screen>

      <para>If you wish to include a common set of files
      (<filename>COPYRIGHT</filename>, <filename>README</filename>,
      etc.) on each CD in the collection, then you need to create a
      directory which contains these files, e.g.:</para>

      <screen>
&rprompt; <userinput>mkdir /tmp/common</userinput>
&rprompt; <userinput>echo "This is a README" &gt; /tmp/common/README</userinput>
&rprompt; <userinput>echo "Another file" &gt; /tmp/common/COPYING</userinput>
&rprompt; <userinput>mkdir /tmp/common/bin</userinput>
&rprompt; <userinput>echo "#!/bin/sh" &gt; /tmp/common/bin/myscript</userinput>
&rprompt; <userinput>echo "echo Hello world" &gt;&gt; /tmp/common/bin/myscript</userinput>
&rprompt; <userinput>chmod 755 /tmp/common/bin/myscript</userinput>
      </screen>

      <para>Now create the images:</para>

      <screen>&rprompt; <userinput>cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images</userinput></screen>

      <para>Each image will contain <filename>README</filename>,
      <filename>COPYING</filename>, and <filename>bin/myscript</filename>
      in their root directories.</para>
    </sect2>
  </sect1>
</chapter>