path: root/doc/guide/files/bulk.xml

<!-- $NetBSD: bulk.xml,v 1.16 2014/07/16 21:12:33 asau Exp $ -->

<chapter id="bulk">
<title>Creating binary packages for everything in pkgsrc (bulk
builds)</title>

<para>When you have multiple machines that should run the same packages,
it is wasted time if they all build their packages themselves from
source. There is a ways of getting a set of binary packages:
The bulk build system, or pbulk ("p" stands for "parallel).
This chapter describes how to set it up so that the packages
are most likely to be usable later.</para>

<sect1 id="bulk.pre">
<title>Think first, build later</title>

<para>Since a bulk build takes several days or even weeks to finish, you
should think about the setup before you start everything. Pay attention
to at least the following points:</para>

<itemizedlist>

<listitem><para>If you want to upload the binary packages to
ftp.NetBSD.org, make sure the setup complies to the requirements for binary
packages:</para>

<itemizedlist>

<listitem><para>To end up on ftp.NetBSD.org, the packages must be built
by a NetBSD developer on a trusted machine (that is, where you and only
you have root access).</para></listitem>

<listitem><para>Packages on ftp.NetBSD.org should only be created from
the stable branches (like 2009Q1), so that users browsing the available
collections can see at a glance how old the packages
are.</para></listitem>

<listitem><para>The packages must be built as root, since some packages
require set-uid binaries at runtime, and creating those packages as
unprivileged user doesn't work well at the moment.</para></listitem>

</itemizedlist>
</listitem>

<listitem><para>Make sure that the bulk build cannot break anything in
your system. Most bulk builds run as root, so they should be run at least
in a chroot environment or something even more restrictive, depending on
what the operating system provides. There have been numerous cases where
certain packages tried to install files outside the
<filename>LOCALBASE</filename> or wanted to edit some files in
<filename>/etc</filename>. Furthermore, the bulk builds install and
deinstall packages in <filename>/usr/pkg</filename> (or whatever
<filename>LOCALBASE</filename> is) during their operation, so be sure
that you don't need any package during the build.</para></listitem>

</itemizedlist>
</sect1>

<sect1 id="bulk.req">
<title>Requirements of a bulk build</title>

<para>A complete bulk build requires lots of disk space. Some of the
disk space can be read-only, some other must be writable. Some can be on
remote filesystems (such as NFS) and some should be local. Some can be
temporary filesystems, others must survive a sudden reboot.</para>

<itemizedlist>

<listitem><para>40 GB for the distfiles (read-write, remote, temporary)</para></listitem>

<listitem><para>30 GB for the binary packages (read-write, remote, permanent)</para></listitem>

<listitem><para>1 GB for the pkgsrc tree (read-only, remote, permanent)</para></listitem>

<listitem><para>5 GB for <filename>LOCALBASE</filename> (read-write, local, temporary)</para></listitem>

<listitem><para>10 GB for the log files (read-write, remote, permanent)</para></listitem>

<listitem><para>5 GB for temporary files (read-write, local, temporary)</para></listitem>

</itemizedlist>

</sect1>

<sect1 id="bulk.pbulk">
<title>Running a pbulk-style bulk build</title>

<para>Running a pbulk-style bulk build works roughly as follows:</para>

<itemizedlist>
<listitem><para>First, build the pbulk infrastructure in a fresh pkgsrc location.</para></listitem>
<listitem><para>Then, build each of the packages from a clean installation directory using the infrastructure.</para></listitem>
</itemizedlist>

<sect2 id="bulk.pbulk.prepare">
<title>Preparation</title>

<para>First, you need to create a pkgsrc installation for the pbulk infrastructure. No matter on which platform you are (even on NetBSD), you should bootstrap into its own directory. Let's take the directory <filename>/usr/pbulk</filename> or <filename>$HOME/pbulk</filename> for it. This installation will be bootstrapped and all the tools that are required for the bulk build will be installed there.</para>

<screen>
$ <userinput>cd /usr/pkgsrc</userinput>
$ <userinput>./bootstrap/bootstrap --prefix=/usr/pbulk --varbase=/usr/pbulk/var --workdir=/tmp/pbulk-bootstrap</userinput>
$ <userinput>rm -rf /tmp/pbulk-bootstrap</userinput>
</screen>

<para>Now the basic environment for the pbulk infrastructure is installed. The specific tools are still missing. This is a good time to edit the pkgsrc configuration file <filename>/usr/pbulk/etc/mk.conf</filename> to fit your needs. Typical things you might set now are:</para>

<itemizedlist>
<listitem><para><literal><varname>PKG_DEVELOPER</varname>=yes</literal>, to enable many consistency checks,</para></listitem>
<listitem><para><literal><varname>WRKOBJDIR</varname>=/tmp/pbulk-outer</literal>, to keep <filename>/usr/pkgsrc</filename> free from any modifications,</para></listitem>
<listitem><para><literal><varname>DISTDIR</varname>=/distfiles</literal>, to have only one directory in which all distfiles (for the infrastructure and for the actual packages) are downloaded,</para></listitem>
<listitem><para><literal><varname>ACCEPTABLE_LICENSES</varname>+=...</literal>, to select some licenses additional to the usual Free/Open Source licenses that are acceptable to you,</para></listitem>
<listitem><para><literal><varname>SKIP_LICENSE_CHECK</varname>=yes</literal>, to bypass the license checks.</para></listitem>
</itemizedlist>

<para>Now you are ready to build the rest of the pbulk infrastructure.</para>

<screen>
$ <userinput>cd pkgtools/pbulk</userinput>
$ <userinput>/usr/pbulk/bin/bmake install</userinput>
$ <userinput>rm -rf /tmp/pbulk-outer</userinput>
</screen>

<para>Now the pbulk infrastructure is built and installed. It still needs to be configured, and after some more preparation, we will be able to start the real bulk build.</para>
</sect2>

<sect2 id="bulk.pbulk.conf">
<title>Configuration</title>

<para>To simplify configuration we provide helper script <filename>mk/pbulk/pbulk.sh</filename>.</para>

<para>In order to use it, prepare a clear system (real one, chroot environment, jail, zone, virtual machine).
Configure network access to fetch distribution files.
Create user with name "pbulk".</para>

<para>Fetch and extract pkgsrc. Use a command like one of these:</para>

<screen>
&rprompt; <userinput>(cd /usr &amp;&amp; ftp -o - http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)</userinput>
&rprompt; <userinput>(cd /usr &amp;&amp; fetch -o - http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)</userinput>
&rprompt; <userinput>(cd /usr &amp;&amp; cvs -Q -z3 -d anoncvs@anoncvs.netbsd.org:/cvsroot get -P pkgsrc)</userinput>
</screen>

<para>Or any other way that fits (e.g., curl, wget).</para>

<para>Deploy and configure pbulk tools, e.g.:</para>

<screen>
&rprompt; <userinput>sh pbulk.sh -n # native (NetBSD)</userinput>
&rprompt; <userinput>sh pbulk.sh -n -c mk.conf.frag # native, apply settings from given mk.conf fragment</userinput>
&rprompt; <userinput>sh pbulk.sh -nlc mk.conf.frag # native, apply settings, configure for limited build</userinput>
</screen>

<note><para><filename>mk.conf.frag</filename> is a fragment of
<filename>mk.conf</filename> that contains settings you want to
apply to packages you build. For instance,</para>

<programlisting>
PKG_DEVELOPER=		yes		# perform more checks
X11_TYPE=		modular		# use pkgsrc X11
SKIP_LICENSE_CHECK=	yes		# accept all licences (useful when building all packages)
</programlisting>
</note>

<para>If configured for limited list, replace the list in <filename>/usr/pbulk/etc/pbulk.list</filename>
with your list of packages one per line without empty lines or comments. E.g.:</para>

<programlisting>
www/firefox
mail/thunderbird
misc/libreoffice4
</programlisting>

<para>At this point you can also review configuration in <filename>/usr/pbulk/etc</filename>
and make final amendments, if wanted.</para>

<para>Start it:</para>

<screen>
&rprompt; <userinput>/usr/pbulk/bin/bulkbuild</userinput>
</screen>

<para>After it finishes, you'll have <filename>/mnt</filename> filled with distribution files, binary packages, and reports,
plain text summary in <filename>/mnt/bulklog/meta/report.txt</filename>
</para>

<note><para>The <filename>pbulk.sh</filename> script does not cover all possible use cases.
While being ready to run, it serves as a good starting point to understand and build more complex setups.
The script is kept small enough for better understanding.</para>
</note>

<note><para>The <filename>pbulk.sh</filename> script supports running
unprivileged bulk build and helps configuring distributed bulk builds.</para>
</note>

</sect2>

</sect1>

  <sect1 id="creating-cdroms">
    <title>Creating a multiple CD-ROM packages collection</title>

    <para>After your pkgsrc bulk-build has completed, you may wish to
    create a CD-ROM set of the resulting binary packages to assist
    in installing packages on other machines.  The
    <filename role="pkg">pkgtools/cdpack</filename> package provides
    a simple tool for creating the ISO 9660 images.
    <command>cdpack</command> arranges the packages on the CD-ROMs in a
    way that keeps all the dependencies for a given package on the same
    CD as that package.</para>

    <sect2 id="cdpack-example">
      <title>Example of cdpack</title>

      <para>Complete documentation for cdpack is found in the cdpack(1)
      man page. The following short example assumes that the binary
      packages are left in
      <filename>/usr/pkgsrc/packages/All</filename> and that
      sufficient disk space exists in <filename>/u2</filename> to
      hold the ISO 9660 images.</para>

      <screen>
&rprompt; <userinput>mkdir /u2/images</userinput>
&rprompt; <userinput>pkg_add /usr/pkgsrc/packages/All/cdpack</userinput>
&rprompt; <userinput>cdpack /usr/pkgsrc/packages/All /u2/images</userinput>
      </screen>

      <para>If you wish to include a common set of files
      (<filename>COPYRIGHT</filename>, <filename>README</filename>,
      etc.) on each CD in the collection, then you need to create a
      directory which contains these files. e.g.</para>

      <screen>
&rprompt; <userinput>mkdir /tmp/common</userinput>
&rprompt; <userinput>echo "This is a README" &gt; /tmp/common/README</userinput>
&rprompt; <userinput>echo "Another file" &gt; /tmp/common/COPYING</userinput>
&rprompt; <userinput>mkdir /tmp/common/bin</userinput>
&rprompt; <userinput>echo "#!/bin/sh" &gt; /tmp/common/bin/myscript</userinput>
&rprompt; <userinput>echo "echo Hello world" &gt;&gt; /tmp/common/bin/myscript</userinput>
&rprompt; <userinput>chmod 755 /tmp/common/bin/myscript</userinput>
      </screen>

      <para>Now create the images:</para>

      <screen>&rprompt; <userinput>cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images</userinput></screen>

      <para>Each image will contain <filename>README</filename>,
      <filename>COPYING</filename>, and <filename>bin/myscript</filename>
      in their root directories.</para>
    </sect2>
  </sect1>
</chapter>