For a number of reasons, you may want to build binary packages for a large selected set of packages in pkgsrc, or even for all pkgsrc packages. For instance, when you have multiple machines that should run the same software, it is wasted time if they all build their packages themselves from source. Or you may want to build a list of packages you want and check them before deploying onto production systems. There is a way of getting a set of binary packages: the bulk build system, or pbulk ("p" stands for "parallel"). This chapter describes how to set it up. First of all, you have to decide whether you build all packages or a limited set of them. Full bulk builds usually consume a lot more resources, both space and time, than builds for some practical sets of packages. A number of particularly heavy packages exist that are not actually interesting to a wide audience. (The approximate resource consumption for a full bulk build is given in section .) For limited bulk builds you need to make a list of packages you want to build. Note that all their dependencies will be built, so you don't need to track them manually. During bulk builds various packages are installed and deinstalled in /usr/pkg (or whatever LOCALBASE is), so make sure that you don't need any package during the builds. Essentially, you should provide a fresh system, either a chroot environment or something even more restrictive, depending on what the operating system provides, or dedicate the whole physical machine. As a useful side effect this makes sure that bulk builds cannot break anything in your system. There have been numerous cases where certain packages tried to install files outside the LOCALBASE or wanted to edit some files in /etc. Running a bulk build works roughly as follows: First, build the pbulk infrastructure in a fresh pkgsrc location. Then, build each of the packages from a clean installation directory using the infrastructure. To simplify configuration, we provide the helper script mk/pbulk/pbulk.sh. In order to use it, prepare a clear system (real one, chroot environment, jail, zone, virtual machine). Configure network access to fetch distribution files. Create a user with name "pbulk". Fetch and extract pkgsrc. Use a command like one of these: &rprompt; (cd /usr && ftp -o - https://cdn.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-) &rprompt; (cd /usr && fetch -o - https://cdn.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-) &rprompt; (cd /usr && cvs -Q -z3 -d anoncvs@anoncvs.NetBSD.org:/cvsroot get -P pkgsrc) Or any other way that fits (e.g., curl, wget). Deploy and configure pbulk tools, e.g.: &rprompt; sh pbulk.sh -n # use native make, no bootstrap kit needed (for use on NetBSD) &rprompt; sh pbulk.sh -n -c mk.conf.frag # native, apply settings from given mk.conf fragment &rprompt; sh pbulk.sh -nlc mk.conf.frag # native, apply settings, configure for limited build mk.conf.frag is a fragment of mk.conf that contains settings you want to apply to packages you build. For instance, PKG_DEVELOPER= yes # perform more checks X11_TYPE= modular # use pkgsrc X11 SKIP_LICENSE_CHECK= yes # accept all licences (useful # when building all packages) If configured for limited list, replace the list in /usr/pbulk/etc/pbulk.list with your list of packages, one per line without empty lines or comments. E.g.: www/firefox mail/thunderbird misc/libreoffice4 At this point you can also review configuration in /usr/pbulk/etc and make final amendments, if wanted. Start it: &rprompt; /usr/pbulk/bin/bulkbuild After it finishes, you'll have /mnt filled with distribution files, binary packages, and reports, plain text summary in /mnt/bulklog/meta/report.txt The pbulk.sh script does not cover all possible use cases. While being ready to run, it serves as a good starting point to understand and build more complex setups. The script is kept small enough for better understanding. The pbulk.sh script supports running unprivileged bulk build and helps configuring distributed bulk builds. Distributed bulk builds support either building in worker chroots (each node is a path to a different chroot) that replicate the target system, including the pbulk prefix, or remote machines (each node is an IP address that must be accessible over SSH without a password). A complete bulk build requires lots of disk space. Some of the disk space can be read-only, some other must be writable. Some can be on remote filesystems (such as NFS) and some should be local. Some can be temporary filesystems, others must survive a sudden reboot. 70 GB for the distfiles (read-write, remote, temporary) 60 GB for the binary packages (read-write, remote, permanent) 1 GB for the pkgsrc tree (read-only, remote, permanent) 5 GB for LOCALBASE (read-write, local, temporary) 10 GB for the log files (read-write, remote, permanent) 5 GB for temporary files (read-write, local, temporary) To ensure that pkgsrc packages work in different configurations, it makes sense to run non-default bulk builds from time to time. This section lists some ideas for bulk builds that intentionally let packages fail if they don't follow the pkgsrc style. Add the following line to &mk.conf;. GNU_CONFIGURE_STRICT= yes When a package fails this additional check, the most common cause is that the configure option was valid for an older version of the package but does not apply anymore. In that case, just remove it. The job of a compiler is not restricted to producing executable code, most compilers also detect typical programming mistakes. The pkgsrc compiler wrappers make it easy to force compiler options when the package is built. This can be used to find typical bugs across all packages that are in pkgsrc. By reporting these bugs upstream, the packages will be more reliable with the next updates. Add some of the following lines to &mk.conf;: CFLAGS+= -Werror=char-subscripts CFLAGS+= -Werror=implicit-function-declaration When a package fails to build using these stricter compiler options, document the circumstances in which the compiler produced the error message. This includes: The platform (MACHINE_PLATFORM) The source file An excerpt of the code. GCC and Clang already do this as part of the diagnostic. The exact error message from the compiler. If a package produces these error messages, but the package is fine, record this in your local &mk.conf;, like this, to skip this check in the next builds: .if ${PKGPATH} == category/package # Version ${VERSION} failed on ${MACHINE_PLATFORM}: # error message # code # Reason why the code does not need to be fixed. BUILDLINK_TRANSFORM+= rm:-Werror=char-subscripts .endif If the error messages from the compiler are valid and the code needs to be fixed, prepare a local patch (see LOCALPATCHES) and report the bug to the upstream authors of the package, providing them with the information you collected above. Patches that are not essential for the package to work should only be reported upstream but not committed to pkgsrc, to make future updates easier. When adding custom compiler flags via CFLAGS, these apply to all phases of the package build process. Especially in the configure phase, adding -Werror leads to wrong decisions. The GNU configure scripts feed many small test programs to the C compiler to see whether certain headers are available, functions are defined in a library and programs can be run. In many cases these programs would not survive a strict compiler run with -Wall -Wextra -Werror. The pkgsrc infrastructure is flexible enough to support compiler options being added between the configure and build phases. It's a little more complicated than the other examples in this section but still easy enough. The basic idea is to use the pkgsrc compiler wrapper to inject the desired compiler options. The compiler wrapper's original task is to hide unwanted directories of include files and to normalize compiler options. It does this by wrapping the compiler command and rewriting the command line. To see this in action, run bmake patch in a package directory and examine the work/.cwrappers/config directory. It contains individual configurations for the C compiler and the related tools. The plan is to find a hook between the configure and build phases, and to modify these configuration files at that point. To find this hook, have a look at mk/build/build.mk, which contains among others the pre-build-checks-hook. The word checks doesn't quite fit, but the pre-build-hook sounds good enough. The code to be included in &mk.conf; is: # Just a few example options. BUILD_ONLY_CFLAGS= -Wall -Werror -O2 -DTEMPDIR='"/tmp"' .if ${BUILD_ONLY_CFLAGS:U:M*} pre-build-checks-hook: add-build-only-cflags add-build-only-cflags: .PHONY ${RUN} cd ${CWRAPPERS_CONFIG_DIR}; \ ${TEST} ! -f ${.TARGET} || exit 0; \ for flag in ${BUILD_ONLY_CFLAGS}; do \ ${ECHO} "append=$$flag" >> cc; \ done; \ > ${.TARGET} .endif (When editing the &mk.conf;, make sure that the commands of the add-build-only-cflags target are indented with a tab, not with spaces.) The condition in the .if statement contains the :U modifier to prevent parse errors if the variable should be undefined, possibly because it is only defined for a subset of the packages. The :M* modifier ensures that there is at least one compiler option, to prevent a syntax error in the shell parser. The code around the ${.TARGET} variable ensures that the additional compiler options are only appended once, even if bmake build is run multiple times. To do this, it creates a marker file. To verify that this setup works, run bmake configure in a package directory. Up to now, everything works as usual. Examine the directory work/.cwrappers/config to see that the compiler options from BUILD_ONLY_CFLAGS are not yet added to the file cc. Examine the tail of the work/.work.log file to see that the normal compiler options are used. Now run bmake build. This will append the options to the file cc and will create the marker file in the same directory. After that, the build starts as usual, but with the added compiler options. Examine the tail of the file work/.work.log to see that the lines starting with [*] don't contain the compiler options, but the corresponding lines starting with <.> do end with these options. Building packages using this setup variant and fixing the resulting bugs is the same as in . Some directories like PREFIX, VARBASE, PKG_SYSCONFDIR, PKGMANDIR, PKG_INFODIR can be configured in pkgsrc. Set these to arbitrary paths during bootstrap or afterwards in &mk.conf;. PREFIX= /a-random-uuid PKG_SYSCONFDIR= /a-random-uuid VARBASE= /a-random-uuid PKGMANDIR= a-random-uuid PKG_INFODIR= a-random-uuid When building a package, warnings are typically ignored since they just flow by and do not cause the build to fail immediately. To find these warnings, redefine them to errors in &mk.conf;. DELAYED_WARNING_MSG= ${DELAYED_ERROR_MSG} "(was warning)" WARNING_MSG= ${FAIL_MSG} "(was warning)" (There are many more classes of warnings in pkgsrc, and most of them can be redefined with a simple definition like above. If a package suggests to add USE_TOOLS+=perl to the package Makefile, research whether the package actually needs Perl. If it does, add USE_TOOLS+=perl to the package Makefile, and if it doesn't, add TOOLS_BROKEN+=perl. Using pkglint as part of the regular build process is mostly a waste of time. If you want to fix some of the warnings, just run pkglint recursively on the whole pkgsrc tree. This will take a few minutes (up to 10), which is much faster than a complete bulk build. To ensure that the binary packages don't contain references to the build directory, there is already CHECK_WRKREF. If that variable includes the item extra, it is possible to define additional patterns that must not appear in any installed file. This is specified in &mk.conf;. CHECK_WRKREF= extra CHECK_WRKREF_EXTRA_DIRS+= /usr/local CHECK_WRKREF_EXTRA_DIRS+= /usr/pkg CHECK_WRKREF_EXTRA_DIRS+= @[A-Z][A-Z]*@ The above patterns will probably generate many false positives, therefore the results need to be taken with a grain of salt. To run the test suites that come with each package, add this line to &mk.conf;. PKGSRC_RUN_TEST= yes Be prepared that even the most basic packages fail this test. When doing a bulk build with this, it will often abort in the early phase where the packages are scanned for their dependencies since there are cyclic dependencies. There is still a lot to do in this area. To catch typos in the shell snippets from the Makefile fragments, add the -u flag to most of the commands by adding this line to &mk.conf;. RUN= @set -eu; After that, ensure that none of the bulk build log files (even those for successfully built packages) contains the string parameter not set or whatever error message the command sh -ceu '$undefined' outputs. See mk/misc/common.mk for the existing definition. The build logs of a package are often quite long. This allows error messages or other interesting details to hide between the noise. To make the actual error message stand out more, add these lines to &mk.conf;. GNU_CONFIGURE_QUIET= yes MAKE_FLAGS+= -s The -s option works for both GNU Make and BSD Make. On exotic platforms with their own make, it may be a little different. After your pkgsrc bulk-build has completed, you may wish to create a CD-ROM set of the resulting binary packages to assist in installing packages on other machines. The pkgtools/cdpack package provides a simple tool for creating the ISO 9660 images. cdpack arranges the packages on the CD-ROMs in a way that keeps all the dependencies for a given package on the same CD as that package. Complete documentation for cdpack is found in the cdpack(1) man page. The following short example assumes that the binary packages are left in /usr/pkgsrc/packages/All and that sufficient disk space exists in /u2 to hold the ISO 9660 images. &rprompt; mkdir /u2/images &rprompt; pkg_add /usr/pkgsrc/packages/All/cdpack &rprompt; cdpack /usr/pkgsrc/packages/All /u2/images If you wish to include a common set of files (COPYRIGHT, README, etc.) on each CD in the collection, then you need to create a directory which contains these files, e.g.: &rprompt; mkdir /tmp/common &rprompt; echo "This is a README" > /tmp/common/README &rprompt; echo "Another file" > /tmp/common/COPYING &rprompt; mkdir /tmp/common/bin &rprompt; echo "#!/bin/sh" > /tmp/common/bin/myscript &rprompt; echo "echo Hello world" >> /tmp/common/bin/myscript &rprompt; chmod 755 /tmp/common/bin/myscript Now create the images: &rprompt; cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images Each image will contain README, COPYING, and bin/myscript in their root directories.