When you have multiple machines that should run the same packages, it is wasted time if they all build their packages themselves from source. There are two ways of getting a set of binary packages: The old bulk build system, or the new (as of 2007) parallel bulk build (pbulk) system. This chapter describes how to set them up so that the packages are most likely to be usable later. Since a bulk build takes several days or even weeks to finish, you should think about the setup before you start everything. Pay attention to at least the following points: If you want to upload the binary packages to ftp.NetBSD.org, make sure the setup complies to the requirements for binary packages: To end up on ftp.NetBSD.org, the packages must be built by a NetBSD developer on a trusted machine (that is, where you and only you have root access). Packages on ftp.NetBSD.org should only be created from the stable branches (like 2009Q1), so that users browsing the available collections can see at a glance how old the packages are. The packages must be built as root, since some packages require set-uid binaries at runtime, and creating those packages as unprivileged user doesn't work well at the moment. Make sure that the bulk build cannot break anything in your system. Most bulk builds run as root, so they should be run at least in a chroot environment or something even more restrictive, depending on what the operating system provides. There have been numerous cases where certain packages tried to install files outside the LOCALBASE or wanted to edit some files in /etc. Furthermore, the bulk builds install and deinstall packages in /usr/pkg (or whatever LOCALBASE is) during their operation, so be sure that you don't need any package during the build. A complete bulk build requires lots of disk space. Some of the disk space can be read-only, some other must be writable. Some can be on remote filesystems (such as NFS) and some should be local. Some can be temporary filesystems, others must survive a sudden reboot. 10 GB for the distfiles (read-write, remote, temporary) 10 GB for the binary packages (read-write, remote, permanent) 400 MB for the pkgsrc tree (read-only, remote, permanent) 5 GB for LOCALBASE (read-write, local, temporary for pbulk, permanent for old-bulk) 5 GB for the log files (read-write, remote, permanent) 5 GB for temporary files (read-write, local, temporary) There are two ways of doing a bulk build. The old-style one and the new-style pbulk. The latter is the recommended way. The build.conf file is the main configuration file for bulk builds. You can configure how your copy of pkgsrc is kept up to date, how the distfiles are downloaded, how the packages are built and how the report is generated. You can find an annotated example file in pkgsrc/mk/bulk/build.conf-example. To use it, copy build.conf-example to build.conf and edit it, following the comments in that file. You may want to set variables in &mk.conf;. Look at pkgsrc/mk/defaults/mk.conf for details of the default settings. You will want to ensure that ACCEPTABLE_LICENSES meet your local policy. As used in this example, SKIP_LICENSE_CHECK=yes completely bypasses the license check. PACKAGES?= ${_PKGSRCDIR}/packages/${MACHINE_ARCH} WRKOBJDIR?= /usr/tmp/pkgsrc # build here instead of in pkgsrc BSDSRCDIR= /usr/src BSDXSRCDIR= /usr/xsrc # for x11/xservers OBJHOSTNAME?= yes # use work.`hostname` FAILOVER_FETCH= yes # insist on the correct checksum PKG_DEVELOPER?= yes SKIP_LICENSE_CHECK= yes Some options that are especially useful for bulk builds can be found at the top lines of the file mk/bulk/bsd.bulk-pkg.mk. The most useful options of these are briefly described here. If you are on a slow machine, you may want to set USE_BULK_BROKEN_CHECK to no. If you are doing bulk builds from a read-only copy of pkgsrc, you have to set BULKFILESDIR to the directory where all log files are created. Otherwise the log files are created in the pkgsrc directory. Another important variable is BULK_PREREQ, which is a list of packages that should be always available while building other packages. Some other options are scattered in the pkgsrc infrastructure: ALLOW_VULNERABLE_PACKAGES should be set to yes. The purpose of the bulk builds is creating binary packages, no matter if they are vulnerable or not. Leaving this variable unset would prevent the bulk build system from even trying to build them, so possible building errors would not show up. CHECK_FILES (pkgsrc/mk/check/check-files.mk) can be set to yes to check that the installed set of files matches the PLIST. CHECK_INTERPRETER (pkgsrc/mk/check/check-interpreter.mk) can be set to yes to check that the installed #!-scripts will find their interpreter. PKGSRC_RUN_TEST can be set to yes to run each package's self-test before installing it. Note that some packages make heavy use of good random numbers, so you need to assure that the machine on which you are doing the bulk builds is not completely idle. Otherwise some test programs will seem to hang, while they are just waiting for new random data to be available. It is possible to configure the bulk build to perform certain site-specific tasks at the end of the pre-build stage. If the file pre-build.local exists in /usr/pkgsrc/mk/bulk, it will be executed (as a &man.sh.1; script) at the end of the usual pre-build stage. An example use of pre-build.local is to have the line: echo "I do not have enough disk space to build this pig." \ > misc/openoffice/$BROKENF to prevent the system from trying to build a particular package which requires nearly 3 GB of disk space. As /usr/pkg will be completely deleted at the start of bulk builds, make sure your login shell is placed somewhere else. Either drop it into /usr/local/bin (and adjust your login shell in the passwd file), or (re-)install it via &man.pkg.add.1; from /etc/rc.local, so you can login after a reboot (remember that your current process won't die if the package is removed, you just can't start any new instances of the shell any more). Also, if you use &os; earlier than 1.5, or you still want to use the pkgsrc version of ssh for some reason, be sure to install ssh before starting it from rc.local: (cd /usr/pkgsrc/security/ssh && make bulk-install) if [ -f /usr/pkg/etc/rc.d/sshd ]; then /usr/pkg/etc/rc.d/sshd fi Not doing so will result in you being not able to log in via ssh after the bulk build is finished or if the machine gets rebooted or crashes. You have been warned! :) Make sure you don't need any of the packages still installed. During the bulk build, all packages, their configuration files and some more files from /var, /home and possibly other locations will be removed! So don't run a bulk build with privileges that might harm your system. Be sure to remove all other things that might interfere with builds, like some libs installed in /usr/local, etc. then become root and type: &rprompt; cd /usr/pkgsrc &rprompt; sh mk/bulk/build If for some reason your last build didn't complete (power failure, system panic, ...), you can continue it by running: &rprompt; sh mk/bulk/build restart At the end of the bulk build, you will get a summary via mail, and find build logs in the directory specified by FTP in the build.conf file. The bulk builds consist of three steps: 1. pre-build The script updates your pkgsrc tree via (anon)cvs, then cleans out any broken distfiles, and removes all packages installed. 2. the bulk build This is basically make bulk-package with an optimised order in which packages will be built. Packages that don't require other packages will be built first, and packages with many dependencies will be built later. 3. post-build Generates a report that's placed in the directory specified in the build.conf file named broken.html, a short version of that report will also be mailed to the build's admin. During the build, a list of broken packages will be compiled in /usr/pkgsrc/.broken (or .../.broken.${MACHINE} if OBJMACHINE is set), individual build logs of broken builds can be found in the package's directory. These files are used by the bulk-targets to mark broken builds to not waste time trying to rebuild them, and they can be used to debug these broken package builds later. Currently, roughly the following requirements are valid for NetBSD 2.0/i386: 10 GB - distfiles (NFS ok) 8 GB - full set of all binaries (NFS ok) 5 GB - temp space for compiling (local disk recommended) Note that all pkgs will be de-installed as soon as they are turned into a binary package, and that sources are removed, so there is no excessively huge demand to disk space. Afterwards, if the package is needed again, it will be installed via &man.pkg.add.1; instead of building again, so there are no cycles wasted by recompiling. If you don't want all the packages nuked from a machine (rendering it useless for anything but pkg compiling), there is the possibility of doing the package bulk build inside a chroot environment. The first step is to set up a chroot sandbox, e.g. /usr/sandbox. This can be done by using null mounts, or manually. There is a shell script called pkgsrc/mk/bulk/mksandbox which will set up the sandbox environment using null mounts. It will also create a script called sandbox in the root of the sandbox environment, which will allow the null mounts to be activated using the sandbox mount command and deactivated using the sandbox umount command. To set up a sandbox environment by hand, after extracting all the sets from a &os; installation or doing a make distribution DESTDIR=/usr/sandbox in /usr/src/etc, be sure the following items are present and properly configured: Kernel &rprompt; cp /netbsd /usr/sandbox /dev/* &rprompt; cd /usr/sandbox/dev ; sh MAKEDEV all /etc/resolv.conf (for security/smtpd and mail): &rprompt; cp /etc/resolv.conf /usr/sandbox/etc Working(!) mail config (hostname, sendmail.cf): &rprompt; cp /etc/mail/sendmail.cf /usr/sandbox/etc/mail /etc/localtime (for security/smtpd): &rprompt; ln -sf /usr/share/zoneinfo/UTC /usr/sandbox/etc/localtime /usr/src (system sources, e. g. for sysutils/aperture): &rprompt; ln -s ../disk1/cvs . &rprompt; ln -s cvs/src-2.0 src Create /var/db/pkg (not part of default install): &rprompt; mkdir /usr/sandbox/var/db/pkg Create /usr/pkg (not part of default install): &rprompt; mkdir /usr/sandbox/usr/pkg Checkout pkgsrc via cvs into /usr/sandbox/usr/pkgsrc: &rprompt; cd /usr/sandbox/usr &rprompt; cvs -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -d -P pkgsrc Do not mount/link this to the copy of your pkgsrc tree you do development in, as this will likely cause problems! Make /usr/sandbox/usr/pkgsrc/packages and .../distfiles point somewhere appropriate. NFS- and/or nullfs-mounts may come in handy! Edit &mk.conf;, see . Adjust mk/bulk/build.conf to suit your needs. When the chroot sandbox is set up, you can start the build with the following steps: &rprompt; cd /usr/sandbox/usr/pkgsrc &rprompt; sh mk/bulk/do-sandbox-build This will just jump inside the sandbox and start building. At the end of the build, mail will be sent with the results of the build. Created binary pkgs will be in /usr/sandbox/usr/pkgsrc/packages (wherever that points/mounts to/from). In addition to building a complete set of all packages in pkgsrc, the pkgsrc/mk/bulk/build script may be used to build a subset of the packages contained in pkgsrc. By setting SPECIFIC_PKGS in &mk.conf;, the variables SITE_SPECIFIC_PKGS HOST_SPECIFIC_PKGS GROUP_SPECIFIC_PKGS USER_SPECIFIC_PKGS will define the set of packages which should be built. The bulk build code will also include any packages which are needed as dependencies for the explicitly listed packages. One use of this is to do a bulk build with SPECIFIC_PKGS in a chroot sandbox periodically to have a complete set of the binary packages needed for your site available without the overhead of building extra packages that are not needed. This section describes how pkgsrc developers can upload binary pkgs built by bulk builds to ftp.NetBSD.org. If you would like to automatically create checksum files for the binary packages you intend to upload, remember to set MKSUMS=yes in your mk/bulk/build.conf. If you would like to PGP sign the checksum files (highly recommended!), remember to set SIGN_AS=username@NetBSD.org in your mk/bulk/build.conf. This will prompt you for your GPG password to sign the files before uploading everything. Then, make sure that you have RSYNC_DST set properly in your mk/bulk/build.conf file, i.e. adjust it to something like one of the following: RSYNC_DST=ftp.NetBSD.org:/pub/NetBSD/packages/packages-20xxQy/NetBSD-a.b.c/arch/upload Please use appropriate values for "packages-20xxQy", "NetBSD-a.b.c" and "arch" here. If your login on ftp.NetBSD.org is different from your local login, write your login directly into the variable, e.g. my local account is "feyrer", but for my login "hubertf", I use: RSYNC_DST=hubertf@ftp.NetBSD.org:/pub/NetBSD/packages/packages-20xxQy/NetBSD-a.b.c/arch/upload A separate upload directory is used here to allow "closing" the directory during upload. To do so, run the following command on ftp.NetBSD.org next: nbftp% mkdir -p -m 750 /pub/NetBSD/packages/packages-20xxQy/NetBSD-a.b.c/arch/upload Please note that /pub/NetBSD/packages is only appropriate for packages for the NetBSD operating system. Binary packages for other operating systems should go into /pub/pkgsrc. Before uploading the binary pkgs, ssh authentication needs to be set up. This example shows how to set up temporary keys for the root account inside the sandbox (assuming that no keys should be present there usually): &rprompt; chroot /usr/sandbox chroot-&rprompt; rm $HOME/.ssh/id-dsa* chroot-&rprompt; ssh-keygen -t dsa chroot-&rprompt; cat $HOME/.ssh/id-dsa.pub Now take the output of id-dsa.pub and append it to your ~/.ssh/authorized_keys file on ftp.NetBSD.org. You can remove the key after the upload is done! Next, test if your ssh connection really works: chroot-&rprompt; ssh ftp.NetBSD.org date Use "-l yourNetBSDlogin" here as appropriate! Now after all this works, you can exit the sandbox and start the upload: chroot-&rprompt; exit &rprompt; cd /usr/sandbox/usr/pkgsrc &rprompt; sh mk/bulk/do-sandbox-upload The upload process may take quite some time. Use &man.ls.1; or &man.du.1; on the FTP server to monitor progress of the upload. The upload script will take care of not uploading restricted packages. After the upload has ended, first thing is to revoke ssh access: nbftp% vi ~/.ssh/authorized_keys Gdd:x! Use whatever is needed to remove the key you've entered before! Last, move the uploaded packages out of the upload directory to have them accessible to everyone: nbftp% cd /pub/NetBSD/packages/packages-20xxQy/NetBSD-a.b.c/arch nbftp% mv upload/* . nbftp% rmdir upload nbftp% chmod 755 . Running a pbulk-style bulk build works roughly as follows: First, build the pbulk infrastructure in a fresh pkgsrc location. Then, build each of the packages from a clean installation directory using the infrastructure. First, you need to create a pkgsrc installation for the pbulk infrastructure. No matter on which platform you are (even on NetBSD), you should bootstrap into its own directory. Let's take the directory /usr/pbulk or $HOME/pbulk for it. This installation will be bootstrapped and all the tools that are required for the bulk build will be installed there. $ cd /usr/pkgsrc $ ./bootstrap/bootstrap --prefix=/usr/pbulk --varbase=/usr/pbulk/var --workdir=/tmp/pbulk-bootstrap $ rm -rf /tmp/pbulk-bootstrap Now the basic environment for the pbulk infrastructure is installed. The specific tools are still missing. This is a good time to edit the pkgsrc configuration file /usr/pbulk/etc/mk.conf to fit your needs. Typical things you might set now are: PKG_DEVELOPER=yes, to enable many consistency checks, WRKOBJDIR=/tmp/pbulk-outer, to keep /usr/pkgsrc free from any modifications, DISTDIR=/distfiles, to have only one directory in which all distfiles (for the infrastructure and for the actual packages) are downloaded, ACCEPTABLE_LICENSES+=..., to select some licenses additional to the usual Free/Open Source licenses that are acceptable to you, SKIP_LICENSE_CHECK=yes, to bypass the license checks. Now you are ready to build the rest of the pbulk infrastructure. $ cd pkgtools/pbulk $ /usr/pbulk/bin/bmake install $ rm -rf /tmp/pbulk-outer Now the pbulk infrastructure is built and installed. It still needs to be configured, and after some more preparation, we will be able to start the real bulk build. TODO; see pkgsrc/doc/HOWTO-pbulk for more information. TODO: continue writing After your pkgsrc bulk-build has completed, you may wish to create a CD-ROM set of the resulting binary packages to assist in installing packages on other machines. The pkgtools/cdpack package provides a simple tool for creating the ISO 9660 images. cdpack arranges the packages on the CD-ROMs in a way that keeps all the dependencies for a given package on the same CD as that package. Complete documentation for cdpack is found in the cdpack(1) man page. The following short example assumes that the binary packages are left in /usr/pkgsrc/packages/All and that sufficient disk space exists in /u2 to hold the ISO 9660 images. &rprompt; mkdir /u2/images &rprompt; pkg_add /usr/pkgsrc/packages/All/cdpack &rprompt; cdpack /usr/pkgsrc/packages/All /u2/images If you wish to include a common set of files (COPYRIGHT, README, etc.) on each CD in the collection, then you need to create a directory which contains these files. e.g. &rprompt; mkdir /tmp/common &rprompt; echo "This is a README" > /tmp/common/README &rprompt; echo "Another file" > /tmp/common/COPYING &rprompt; mkdir /tmp/common/bin &rprompt; echo "#!/bin/sh" > /tmp/common/bin/myscript &rprompt; echo "echo Hello world" >> /tmp/common/bin/myscript &rprompt; chmod 755 /tmp/common/bin/myscript Now create the images: &rprompt; cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images Each image will contain README, COPYING, and bin/myscript in their root directories.