The primary download location for all pkgsrc files is http://ftp.NetBSD.org/pub/pkgsrc/ or ftp://ftp.NetBSD.org/pub/pkgsrc/ (it points to the same location). There are a number of subdirectories for different purposes, which are described in detail in Appendix C, Directory layout of the pkgsrc FTP server.

The tar archive for the current branch is in the directory current and is called pkgsrc.tar.gz. It is autogenerated daily.

To save download time we provide bzip2- and xz-compressed archives which are published at pkgsrc.tar.bz2 and pkgsrc.tar.xz respectively.

You can fetch the same files using FTP.

The tar file for the stable branch 2009Q1 is in the directory pkgsrc-2009Q1 and is also called pkgsrc-2009Q1.tar.gz.

To download a pkgsrc stable tarball, run:

$ ftp ftp://ftp.NetBSD.org/pub/pkgsrc/pkgsrc-20xxQy/pkgsrc-20xxQy.tar.gz

Where pkgsrc-20xxQy is the stable branch to be downloaded, for example, “pkgsrc-2009Q1”.

You can use fetch it also using "wget", "curl", or your web browser.

Then, extract it with:

$ tar -xzf pkgsrc-20xxQy.tar.gz -C /usr

This will create the directory pkgsrc/ in /usr/ and all the package source will be stored under /usr/pkgsrc/.

To download pkgsrc-current, run:

$ ftp ftp://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz

To fetch a specific pkgsrc stable branch, run:

$ cd /usr && cvs -q -z2 -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -r pkgsrc-20xxQy -P pkgsrc

Where pkgsrc-20xxQy is the stable branch to be checked out, for example, “pkgsrc-2009Q1”

This will create the directory pkgsrc/ in your /usr/ directory and all the package source will be stored under /usr/pkgsrc/.

To fetch the pkgsrc current branch, run:

$ cd /usr && cvs -q -z2 -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -P pkgsrc

Refer to the list of available mirrors to choose a faster CVS mirror, if needed.

If you get error messages from rsh, you need to set CVS_RSH variable. E.g.:

$ cd /usr && env CVS_RSH=ssh cvs -q -z2 -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -P pkgsrc

Refer to documentation on your command shell how to set CVS_RSH=ssh permanently. For Bourne shells, you can set it in your .profile or better globally in /etc/profile:

# set CVS remote shell command
CVS_RSH=ssh
export CVS_RSH

By default, CVS doesn't do things like most people would expect it to do. But there is a way to convince CVS, by creating a file called .cvsrc in your home directory and saving the following lines to it. This file will save you lots of headache and some bug reports, so we strongly recommend it. You can find an explanation of this file in the CVS documentation.

# recommended CVS configuration file from the pkgsrc guide
cvs -q -z2
checkout -P
update -dP
diff -upN
rdiff -u
release -d

Note that by default the distfiles and the binary packages are saved in the pkgsrc tree, so don't forget to rescue them before updating. You can also configure pkgsrc to use other than the default directories by setting the DISTDIR and PACKAGES variables. See Chapter 5, Configuring pkgsrc for the details.

To update pkgsrc from a tar file, download the tar file as explained above. Then, make sure that you have not made any changes to the files in the pkgsrc directory. Remove the pkgsrc directory and extract the new tar file. Done.

To update pkgsrc via CVS, change to the pkgsrc directory and run cvs:

$ cd /usr/pkgsrc && cvs update -dP

If you get error messages from rsh, you need to set CVS_RSH variable as described above. E.g.:

$ cd /usr/pkgsrc && env CVS_RSH=ssh cvs up -dP

Cygwin 1.7.x and later are supported.

You need to install minimal base packages in `Base' category plus any of compiler, gcc, gcc4, and/or clang. For gcc and gcc4, C and C++ compiler will be installed by default, but you can install Fortran compiler additionally because it will be required to use libtool. If it is not installed (or too old), Fortran compiler will be installed with pkgsrc automatically.

As noted in Cygwin FAQ: `Why doesn't su work?', su(1) command has been in Cygwin distribution, but it has never worked. Unless you bootstrap pkgsrc with the --unprivileged option, workaround is:

Darwin 5.x and up are supported.

Before you start, you need to download and install the Mac OS X Developer Tools from Apple's Developer Connection. This requires (free) membership. See http://developer.apple.com/macosx/ for details. Also, make sure you install X11 (an optional package included with the Developer Tools) if you intend to build packages that use the X11 Window System. (If you don't want or need the full Xcode GUI, download and install Command Line Tools for Xcode.)

FreeBSD 8.3 and 9.0 have been tested and are supported, other versions may work.

Care should be taken so that the tools that this kit installs do not conflict with the FreeBSD userland tools. There are several steps:

Interix is a POSIX-compatible subsystem for the Windows NT kernel, providing a Unix-like environment with a tighter kernel integration than available with Cygwin. It is part of the Windows Services for Unix package, available for free for any licensed copy of Windows 2000, XP (not including XP Home), or 2003. SFU can be downloaded from http://www.microsoft.com/windows/sfu/.

Services for Unix 3.5 has been tested. 3.0 or 3.1 may work, but are not officially supported. (The main difference in 3.0/3.1 is lack of pthreads, but other parts of libc may also be lacking.)

Services for Unix Applications (aka SUA) is an integrated component of Windows Server 2003 R2 (5.2), Windows Vista and Windows Server 2008 (6.0), Windows 7 and Windows Server 2008 R2 (6.1). As of this writing, the SUA's Interix 6.0 (32bit) and 6.1 (64bit) subsystems have been tested. Other versions may work as well. The Interix 5.x subsystem has not yet been tested with pkgsrc.

interix:kP=\E[S:kN=\E[T:kH=\E[U:dc@:DC@:tc=pcansi:
	

# chmod -R g+w $PKG_DBDIR
	

You will need a working C compiler, either gcc or SGI's MIPS and MIPSpro compiler (cc/c89). Please set the CC environment variable according to your preference. If you do not have a license for the MIPSpro compiler suite, you can download a gcc tardist file from http://freeware.sgi.com/.

Please note that you will need IRIX 6.5.17 or higher, as this is the earliest version of IRIX providing support for if_indextoname(3), if_nametoindex(3), etc.

At this point in time, pkgsrc only supports one ABI at a time. That is, you cannot switch between the old 32-bit ABI, the new 32-bit ABI and the 64-bit ABI. If you start out using "abi=n32", that's what all your packages will be built with.

Therefore, please make sure that you have no conflicting CFLAGS in your environment or the mk.conf. Particularly, make sure that you do not try to link n32 object files with lib64 or vice versa. Check your /etc/compiler.defaults!

If you have the actual pkgsrc tree mounted via NFS from a different host, please make sure to set WRKOBJDIR to a local directory, as it appears that IRIX linker occasionally runs into issues when trying to link over a network-mounted file system.

The bootstrapping process should set all the right options for programs such as imake(1), but you may want to set some options depending on your local setup. Please see pkgsrc/mk/defaults/mk.conf and, of course, your compiler's man pages for details.

If you are using SGI's MIPSPro compiler, please set

PKGSRC_COMPILER=        mipspro
      

in mk.conf. Otherwise, pkgsrc will assume you are using gcc and may end up passing invalid flags to the compiler. Note that bootstrap should create an appropriate mk.conf.example by default.

If you have both the MIPSPro compiler chain installed as well as gcc, but want to make sure that MIPSPro is used, please set your PATH to not include the location of gcc (often /usr/freeware/bin), and (important) pass the '--preserve-path' flag.

Some versions of Linux (for example Debian GNU/Linux) need either libtermcap or libcurses (libncurses). Installing the distributions libncurses-dev package (or equivalent) should fix the problem.

pkgsrc supports both gcc (GNU Compiler Collection) and icc (Intel C++ Compiler). gcc is the default. icc 8.0 and 8.1 on i386 have been tested.

To bootstrap using icc, assuming the default icc installation directory:

env ICCBASE=/opt/intel/cc/10.1.008 ./bootstrap --compiler=icc
      

Use a value for ICCBASE that corresponds to the directory where icc is installed. After bootstrapping, set ICCBASE in mk.conf:

ICCBASE=                /opt/intel/cc/10.1.008
      

The pkgsrc default for ICCBASE is /opt/intel_cc_80. This is the default install directory for icc 8.0. If you are using a more recent version, be sure to set the correct path explicitly.

pkgsrc uses the static linking method of the runtime libraries provided by icc, so binaries can be run on other systems which do not have the shared libraries installed.

Libtool, however, extracts a list of libraries from the ld(1) command run when linking a C++ shared library and records it, throwing away the -Bstatic and -Bdynamic options interspersed between the libraries. This means that libtool-linked C++ shared libraries will have a runtime dependency on the icc libraries until this is fixed in libtool.

OpenBSD 5.1 has been tested and supported, other versions may work.

Care should be taken so that the tools that this kit installs do not conflict with the OpenBSD userland tools. There are several steps:

Solaris 2.6 through 10 are supported on both x86 and sparc. You will need a working C compiler. Both gcc 4.5.3 and Sun WorkShop 5 have been tested.

The following packages are required on Solaris 8 for the bootstrap process and to build packages.

Please note that the use of GNU binutils on Solaris is not supported, as of June 2006.

Whichever compiler you use, please ensure the compiler tools and your $prefix are in your PATH. This includes /usr/ccs/{bin,lib} and e.g. /usr/pkg/{bin,sbin}.

CC=     cc
CXX=    CC
CPP=    cc -E
CXXCPP= CC -E

PKGSRC_COMPILER=        sunpro
ABI=                    64

CONFIG_SHELL=   ${LOCALBASE}/bin/bash
WRAPPER_SHELL=  ${LOCALBASE}/bin/bash
      

To install binary packages, you first need to know from where to get them. The first place where you should look is on the main pkgsrc FTP server in the directory /pub/pkgsrc/packages.

This directory contains binary packages for multiple platforms. First, select your operating system. (Ignore the directories with version numbers attached to it, they just exist for legacy reasons.) Then, select your hardware architecture, and in the third step, the OS version and the “version” of pkgsrc.

In this directory, you often find a file called bootstrap.tar.gz which contains the package management tools. If the file is missing, it is likely that your operating system already provides those tools. Download the file and extract it in the / directory. It will create the directories /usr/pkg (containing the tools for managing binary packages) and /var/db/pkg (the database of installed packages).

In the directory from the last section, there is a subdirectory called All, which contains all the binary packages that are available for the platform, excluding those that may not be distributed via FTP or CDROM (depending on which medium you are using).

To install packages directly from an FTP or HTTP server, run the following commands in a Bourne-compatible shell (be sure to su to root first):

# PATH="/usr/pkg/sbin:$PATH"
# PKG_PATH="ftp://ftp.NetBSD.org/pub/pkgsrc/packages/OPSYS/ARCH/VERSIONS/All"
# export PATH PKG_PATH

Instead of URLs, you can also use local paths, for example if you are installing from a set of CDROMs, DVDs or an NFS-mounted repository. If you want to install packages from multiple sources, you can separate them by a semicolon in PKG_PATH.

After these preparations, installing a package is very easy:

# pkg_add openoffice2
# pkg_add kde-3.5.7
# pkg_add ap2-php5-*

Note that any prerequisite packages needed to run the package in question will be installed, too, assuming they are present where you install from.

Adding packages might install vulnerable packages. Thus you should run pkg_admin audit regularly, especially after installing new packages, and verify that the vulnerabilities are acceptable for your configuration.

After you've installed packages, be sure to have /usr/pkg/bin and /usr/pkg/sbin in your PATH so you can actually start the just installed program.

To deinstall a package, it does not matter whether it was installed from source code or from a binary package. The pkg_delete command does not know it anyway. To delete a package, you can just run pkg_delete package-name. The package name can be given with or without version number. Wildcards can also be used to deinstall a set of packages, for example *emacs*. Be sure to include them in quotes, so that the shell does not expand them before pkg_delete sees them.

The -r option is very powerful: it removes all the packages that require the package in question and then removes the package itself. For example:

# pkg_delete -r jpeg
    

will remove jpeg and all the packages that used it; this allows upgrading the jpeg package.

The pkg_info shows information about installed packages or binary package files.

The NetBSD Security-Officer and Packages Groups maintain a list of known security vulnerabilities to packages which are (or have been) included in pkgsrc. The list is available from the NetBSD FTP site at ftp://ftp.NetBSD.org/pub/pkgsrc/distfiles/vulnerabilities.

Through pkg_admin fetch-pkg-vulnerabilities, this list can be downloaded automatically, and a security audit of all packages installed on a system can take place.

There are two components to auditing. The first step, pkg_admin fetch-pkg-vulnerabilities, is for downloading the list of vulnerabilities from the NetBSD FTP site. The second step, pkg_admin audit, checks to see if any of your installed packages are vulnerable. If a package is vulnerable, you will see output similar to the following:

Package samba-2.0.9 has a local-root-shell vulnerability, see
    http://www.samba.org/samba/whatsnew/macroexploit.html

You may wish to have the vulnerabilities file downloaded daily so that it remains current. This may be done by adding an appropriate entry to the root users crontab(5) entry. For example the entry

# download vulnerabilities file
0 3 * * * /usr/sbin/pkg_admin fetch-pkg-vulnerabilities >/dev/null 2>&1
      

will update the vulnerability list every day at 3AM. You may wish to do this more often than once a day. In addition, you may wish to run the package audit from the daily security script. This may be accomplished by adding the following line to /etc/security.local:

/usr/sbin/pkg_admin audit
      

Install pkgtools/lintpkgsrc and run lintpkgsrc with the “-i” argument to check if your packages are up-to-date, e.g.

% lintpkgsrc -i
...
Version mismatch: 'tcsh' 6.09.00 vs 6.10.00
    

You can then use make update to update the package on your system and rebuild any dependencies.

The pkg_admin executes various administrative functions on the package system.

Please pay very careful attention to the warnings expressed in the pkg_add(1) manual page about the inherent dangers of installing binary packages which you did not create yourself, and the security holes that can be introduced onto your system by indiscriminate adding of such files.

The same warning of course applies to every package you install from source when you haven't completely read and understood the source code of the package, the compiler that is used to build the package and all the other tools that are involved.

To build packages from source, you need a working C compiler. On NetBSD, you need to install the “comp” and the “text” distribution sets. If you want to build X11-related packages, the “xbase” and “xcomp” distribution sets are required, too.

The first step for building a package is downloading the distfiles (i.e. the unmodified source). If they have not yet been downloaded, pkgsrc will fetch them automatically.

If you have all files that you need in the distfiles directory, you don't need to connect. If the distfiles are on CD-ROM, you can mount the CD-ROM on /cdrom and add:

DISTDIR=/cdrom/pkgsrc/distfiles

to your mk.conf.

By default a list of distribution sites will be randomly intermixed to prevent huge load on servers which holding popular packages (for example, SourceForge.net mirrors). Thus, every time when you need to fetch yet another distfile all the mirrors will be tried in new (random) order. You can turn this feature off by setting MASTER_SORT_RANDOM=NO (for PKG_DEVELOPERs it's already disabled).

You can overwrite some of the major distribution sites to fit to sites that are close to your own. By setting one or two variables you can modify the order in which the master sites are accessed. MASTER_SORT contains a whitespace delimited list of domain suffixes. MASTER_SORT_REGEX is even more flexible, it contains a whitespace delimited list of regular expressions. It has higher priority than MASTER_SORT. Have a look at pkgsrc/mk/defaults/mk.conf to find some examples. This may save some of your bandwidth and time.

You can change these settings either in your shell's environment, or, if you want to keep the settings, by editing the mk.conf file, and adding the definitions there.

If a package depends on many other packages (such as meta-pkgs/kde3), the build process may alternate between periods of downloading source, and compiling. To ensure you have all the source downloaded initially you can run the command:

% make fetch-list | sh

which will output and run a set of shell commands to fetch the necessary files into the distfiles directory. You can also choose to download the files manually.

Once the software has downloaded, any patches will be applied, then it will be compiled for you. This may take some time depending on your computer, and how many other packages the software depends on and their compile time.

For example, type

% cd misc/figlet
% make
    

at the shell prompt to build the various components of the package.

The next stage is to actually install the newly compiled program onto your system. Do this by entering:

% make install
    

while you are still in the directory for whatever package you are installing.

Installing the package on your system may require you to be root. However, pkgsrc has a just-in-time-su feature, which allows you to only become root for the actual installation step.

That's it, the software should now be installed and setup for use. You can now enter:

% make clean
    

to remove the compiled files in the work directory, as you shouldn't need them any more. If other packages were also added to your system (dependencies) to allow your program to compile, you can tidy these up also with the command:

% make clean-depends
    

Taking the figlet utility as an example, we can install it on our system by building as shown in Appendix B, Build logs.

The program is installed under the default root of the packages tree - /usr/pkg. Should this not conform to your tastes, set the LOCALBASE variable in your environment, and it will use that value as the root of your packages tree. So, to use /usr/local, set LOCALBASE=/usr/local in your environment. Please note that you should use a directory which is dedicated to packages and not shared with other programs (i.e., do not try and use LOCALBASE=/usr). Also, you should not try to add any of your own files or directories (such as src/, obj/, or pkgsrc/) below the LOCALBASE tree. This is to prevent possible conflicts between programs and other files installed by the package system and whatever else may have been installed there.

Some packages look in mk.conf to alter some configuration options at build time. Have a look at pkgsrc/mk/defaults/mk.conf to get an overview of what will be set there by default. Environment variables such as LOCALBASE can be set in mk.conf to save having to remember to set them each time you want to use pkgsrc.

Occasionally, people want to “look under the covers” to see what is going on when a package is building or being installed. This may be for debugging purposes, or out of simple curiosity. A number of utility values have been added to help with this.

If you want to install a binary package that you've either created yourself (see next section), that you put into pkgsrc/packages manually or that is located on a remote FTP server, you can use the "bin-install" target. This target will install a binary package - if available - via pkg_add(1), else do a make package. The list of remote FTP sites searched is kept in the variable BINPKG_SITES, which defaults to ftp.NetBSD.org. Any flags that should be added to pkg_add(1) can be put into BIN_INSTALL_FLAGS. See pkgsrc/mk/defaults/mk.conf for more details.

A final word of warning: If you set up a system that has a non-standard setting for LOCALBASE, be sure to set that before any packages are installed, as you cannot use several directories for the same purpose. Doing so will result in pkgsrc not being able to properly detect your installed packages, and fail miserably. Note also that precompiled binary packages are usually built with the default LOCALBASE of /usr/pkg, and that you should not install any if you use a non-standard LOCALBASE.

By default, pkgsrc will use GCC to build packages. This may be overridden by setting the following variables in /etc/mk.conf:

If you wish to set the CFLAGS variable, please make sure to use the += operator instead of the = operator:

CFLAGS+=        -your -flags

Using CFLAGS= (i.e. without the “+”) may lead to problems with packages that need to add their own flags. You may want to take a look at the devel/cpuflags package if you're interested in optimization specifically for the current CPU.

If you want to pass flags to the linker, both in the configure step and the build step, you can do this in two ways. Either set LDFLAGS or LIBS. The difference between the two is that LIBS will be appended to the command line, while LDFLAGS come earlier. LDFLAGS is pre-loaded with rpath settings for ELF machines depending on the setting of USE_IMAKE or the inclusion of mk/x11.buildlink3.mk. As with CFLAGS, if you do not wish to override these settings, use the += operator:

LDFLAGS+=        -your -linkerflags

As /usr/pkg will be completely deleted at the start of bulk builds, make sure your login shell is placed somewhere else. Either drop it into /usr/local/bin (and adjust your login shell in the passwd file), or (re-)install it via pkg_add(1) from /etc/rc.local, so you can login after a reboot (remember that your current process won't die if the package is removed, you just can't start any new instances of the shell any more). Also, if you use NetBSD earlier than 1.5, or you still want to use the pkgsrc version of ssh for some reason, be sure to install ssh before starting it from rc.local:

(cd /usr/pkgsrc/security/ssh && make bulk-install)
if [ -f /usr/pkg/etc/rc.d/sshd ]; then
  /usr/pkg/etc/rc.d/sshd
fi

Not doing so will result in you being not able to log in via ssh after the bulk build is finished or if the machine gets rebooted or crashes. You have been warned! :)

Make sure you don't need any of the packages still installed.

Be sure to remove all other things that might interfere with builds, like some libs installed in /usr/local, etc. then become root and type:

# cd /usr/pkgsrc
# sh mk/bulk/build
      

If for some reason your last build didn't complete (power failure, system panic, ...), you can continue it by running:

# sh mk/bulk/build restart

At the end of the bulk build, you will get a summary via mail, and find build logs in the directory specified by FTP in the build.conf file.

The bulk builds consist of three steps:

During the build, a list of broken packages will be compiled in /usr/pkgsrc/.broken (or .../.broken.${MACHINE} if OBJMACHINE is set), individual build logs of broken builds can be found in the package's directory. These files are used by the bulk-targets to mark broken builds to not waste time trying to rebuild them, and they can be used to debug these broken package builds later.

Currently, roughly the following requirements are valid for NetBSD 2.0/i386:

Note that all pkgs will be de-installed as soon as they are turned into a binary package, and that sources are removed, so there is no excessively huge demand to disk space. Afterwards, if the package is needed again, it will be installed via pkg_add(1) instead of building again, so there are no cycles wasted by recompiling.

If you don't want all the packages nuked from a machine (rendering it useless for anything but pkg compiling), there is the possibility of doing the package bulk build inside a chroot environment.

The first step is to set up a chroot sandbox, e.g. /usr/sandbox. This can be done by using null mounts, or manually.

There is a shell script called mksandbox installed by the pkgtools/mksandbox package, which will set up the sandbox environment using null mounts. It will also create a script called sandbox in the root of the sandbox environment, which will allow the null mounts to be activated using the sandbox mount command and deactivated using the sandbox umount command.

To set up a sandbox environment by hand, after extracting all the sets from a NetBSD installation or doing a make distribution DESTDIR=/usr/sandbox in /usr/src/etc, be sure the following items are present and properly configured:

When the chroot sandbox is set up, you can start the build with the following steps:

# cd /usr/sandbox/usr/pkgsrc
# sh mk/bulk/do-sandbox-build
      

This will just jump inside the sandbox and start building. At the end of the build, mail will be sent with the results of the build. Created binary pkgs will be in /usr/sandbox/usr/pkgsrc/packages (wherever that points/mounts to/from).

In addition to building a complete set of all packages in pkgsrc, the pkgsrc/mk/bulk/build script may be used to build a subset of the packages contained in pkgsrc. By setting SPECIFIC_PKGS in mk.conf, the variables

will define the set of packages which should be built. The bulk build code will also include any packages which are needed as dependencies for the explicitly listed packages.

One use of this is to do a bulk build with SPECIFIC_PKGS in a chroot sandbox periodically to have a complete set of the binary packages needed for your site available without the overhead of building extra packages that are not needed.

This section describes how pkgsrc developers can upload binary pkgs built by bulk builds to ftp.NetBSD.org.

If you would like to automatically create checksum files for the binary packages you intend to upload, remember to set MKSUMS=yes in your mk/bulk/build.conf.

If you would like to PGP sign the checksum files (highly recommended!), remember to set SIGN_AS=username@NetBSD.org in your mk/bulk/build.conf. This will prompt you for your GPG password to sign the files before uploading everything.

Then, make sure that you have RSYNC_DST set properly in your mk/bulk/build.conf file, i.e. adjust it to something like one of the following:

RSYNC_DST=ftp.NetBSD.org:/pub/pkgsrc/packages/NetBSD/arch/a.b.c-20xxQy/upload

Please use appropriate values for "20xxQy" (the branch), "a.b.c" (the OS version) and "arch" here. If your login on ftp.NetBSD.org is different from your local login, write your login directly into the variable, e.g. my local account is "feyrer", but for my login "hubertf", I use:

RSYNC_DST=hubertf@ftp.NetBSD.org:/pub/pkgsrc/packages/NetBSD/arch/a.b.c-20xxQy/upload

A separate upload directory is used here to allow "closing" the directory during upload. To do so, run the following command on ftp.NetBSD.org next:

nbftp% mkdir -p -m 750 /pub/pkgsrc/packages/NetBSD/arch/a.b.c-20xxQy/upload

Before uploading the binary pkgs, ssh authentication needs to be set up. This example shows how to set up temporary keys for the root account inside the sandbox (assuming that no keys should be present there usually):

# chroot /usr/sandbox
chroot-# rm $HOME/.ssh/id-dsa*
chroot-# ssh-keygen -t rsa
chroot-# cat $HOME/.ssh/id-rsa.pub
      

Now take the output of id-rsa.pub and append it to your ~/.ssh/authorized_keys file on ftp.NetBSD.org. You should remove the key after the upload is done!

Next, test if your ssh connection really works:

chroot-# ssh ftp.NetBSD.org date 

Use "-l yourNetBSDlogin" here as appropriate!

Now after all this works, you can exit the sandbox and start the upload:

chroot-# exit
# cd /usr/sandbox/usr/pkgsrc
# sh mk/bulk/do-sandbox-upload
      

The upload process may take quite some time. Use ls(1) or du(1) on the FTP server to monitor progress of the upload. The upload script will take care of not uploading restricted packages.

After the upload has ended, first thing is to revoke ssh access:

nbftp% vi ~/.ssh/authorized_keys
      Gdd:x! 

Use whatever is needed to remove the key you've entered before! Last, move the uploaded packages out of the upload directory to have them accessible to everyone:

nbftp% cd /pub/pkgsrc/packages/NetBSD/arch/a.b.c-20xxQy
nbftp% mv upload/* .
nbftp% rmdir upload
nbftp% chgrp -R netbsd .
nbftp% find . -type d | xargs chmod 775
      

First, you need to create a pkgsrc installation for the pbulk infrastructure. No matter on which platform you are (even on NetBSD), you should bootstrap into its own directory. Let's take the directory /usr/pbulk or $HOME/pbulk for it. This installation will be bootstrapped and all the tools that are required for the bulk build will be installed there.

$ cd /usr/pkgsrc
$ ./bootstrap/bootstrap --prefix=/usr/pbulk --varbase=/usr/pbulk/var --workdir=/tmp/pbulk-bootstrap
$ rm -rf /tmp/pbulk-bootstrap

Now the basic environment for the pbulk infrastructure is installed. The specific tools are still missing. This is a good time to edit the pkgsrc configuration file /usr/pbulk/etc/mk.conf to fit your needs. Typical things you might set now are:

Now you are ready to build the rest of the pbulk infrastructure.

$ cd pkgtools/pbulk
$ /usr/pbulk/bin/bmake install
$ rm -rf /tmp/pbulk-outer

Now the pbulk infrastructure is built and installed. It still needs to be configured, and after some more preparation, we will be able to start the real bulk build.

TODO; see pkgsrc/doc/HOWTO-pbulk for more information.

TODO: continue writing

Complete documentation for cdpack is found in the cdpack(1) man page. The following short example assumes that the binary packages are left in /usr/pkgsrc/packages/All and that sufficient disk space exists in /u2 to hold the ISO 9660 images.

# mkdir /u2/images
# pkg_add /usr/pkgsrc/packages/All/cdpack
# cdpack /usr/pkgsrc/packages/All /u2/images
      

If you wish to include a common set of files (COPYRIGHT, README, etc.) on each CD in the collection, then you need to create a directory which contains these files. e.g.

# mkdir /tmp/common
# echo "This is a README" > /tmp/common/README
# echo "Another file" > /tmp/common/COPYING
# mkdir /tmp/common/bin
# echo "#!/bin/sh" > /tmp/common/bin/myscript
# echo "echo Hello world" >> /tmp/common/bin/myscript
# chmod 755 /tmp/common/bin/myscript
      

Now create the images:

# cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images

Each image will contain README, COPYING, and bin/myscript in their root directories.

Simple Perl modules are handled automatically by url2pkg, including dependencies.

KDE applications should always include meta-pkgs/kde3/kde3.mk, which contains numerous settings that are typical of KDE packages.

Python modules and programs packages are easily created using a set of predefined variables.

Most Python packages use either “distutils” or easy-setup (“eggs”). If the software uses “distutils”, set the PYDISTUTILSPKG variable to “yes” so pkgsrc will make use of this framework. “distutils” uses a script called setup.py, if the “distutils” driver is not called setup.py, set the PYSETUP variable to the name of the script.

If the default Python versions are not supported by the software, set the PYTHON_VERSIONS_ACCEPTED variable to the Python versions the software is known to work with, from the most recent to the older one, e.g.

PYTHON_VERSIONS_ACCEPTED=       31 27 26

If the packaged software is a Python module, include “../../lang/python/extension.mk”. In this case, the package directory should be called “py-software” and PKGNAME should be set to “${PYPKGPREFIX}-${DISTNAME}”, e.g.

DISTNAME=   foopymodule-1.2.10
PKGNAME=    ${PYPKGPREFIX}-${DISTNAME}

If it is an application, also include “../../lang/python/application.mk” before “extension.mk”.

If the packaged software, either it is an application or a module, is egg-aware, you only need to include “../../lang/python/egg.mk”.

In order to correctly set the path to the Python interpreter, use the REPLACE_PYTHON variable and set it to the list of files (paths relative to WRKSRC) that must be corrected. For example :

REPLACE_PYTHON=   *.py

The patch-* files should be in diff -bu format, and apply without a fuzz to avoid problems. (To force patches to apply with fuzz you can set PATCH_FUZZ_FACTOR=-F2). Furthermore, each patch should contain only changes for a single file, and no file should be patched by more than one patch file. This helps to keep future modifications simple.

Each patch file is structured as follows: In the first line, there is the RCS Id of the patch itself. The second line should be empty for aesthetic reasons. After that, there should be a comment for each change that the patch does. There are a number of standard cases:

In all, the patch should be commented so that any developer who knows the code of the application can make some use of the patch. Special care should be taken for the upstream developers, since we generally want that they accept our patches, so we have less work in the future.

One important thing to mention is to pay attention that no RCS IDs get stored in the patch files, as these will cause problems when later checked into the NetBSD CVS tree. Use the pkgdiff command from the pkgtools/pkgdiff package to avoid these problems.

For even more automation, we recommend using mkpatches from the same package to make a whole set of patches. You just have to backup files before you edit them to filename.orig, e.g. with cp -p filename filename.orig or, easier, by using pkgvi again from the same package. If you upgrade a package this way, you can easily compare the new set of patches with the previously existing one with patchdiff. The files in patches are replaced by new files, so carefully check if you want to take all the changes.

When you have finished a package, remember to generate the checksums for the patch files by using the make makepatchsum command, see Section 11.2, “distinfo”.

When adding a patch that corrects a problem in the distfile (rather than e.g. enforcing pkgsrc's view of where man pages should go), send the patch as a bug report to the maintainer. This benefits non-pkgsrc users of the package, and usually makes it possible to remove the patch in future version.

The file names of the patch files are usually of the form patch-path_to_file__with__underscores.c. Many packages still use the previous convention patch-[a-z][a-z], but new patches should be of the form containing the filename. mkpatches included in pkgtools/pkgdiff takes care of the name automatically.

If you want to share patches between multiple packages in pkgsrc, e.g. because they use the same distfiles, set PATCHDIR to the path where the patch files can be found, e.g.:

PATCHDIR= ${.CURDIR}/../xemacs/patches

Patch files that are distributed by the author or other maintainers can be listed in PATCHFILES.

If it is desired to store any patches that should not be committed into pkgsrc, they can be kept outside the pkgsrc tree in the $LOCALPATCHES directory. The directory tree there is expected to have the same “category/package” structure as pkgsrc, and patches are expected to be stored inside these dirs (also known as $LOCALPATCHES/$PKGPATH). For example, if you want to keep a private patch for pkgsrc/graphics/png, keep it in $LOCALPATCHES/graphics/png/mypatch. All files in the named directory are expected to be patch files, and they are applied after pkgsrc patches are applied.

When fixing a portability issue in the code do not use preprocessor magic to check for the current operating system nor platform. Doing so hurts portability to other platforms because the OS-specific details are not abstracted appropriately.

The general rule to follow is: instead of checking for the operating system the application is being built on, check for the specific features you need. For example, instead of assuming that kqueue is available under NetBSD and using the __NetBSD__ macro to conditionalize kqueue support, add a check that detects kqueue itself — yes, this generally involves patching the configure script. There is absolutely nothing that prevents some OSes from adopting interfaces from other OSes (e.g. Linux implementing kqueue), something that the above checks cannot take into account.

Of course, checking for features generally involves more work on the developer's side, but the resulting changes are cleaner and there are chances they will work on many other platforms. Not to mention that there are higher chances of being later integrated into the mainstream sources. Remember: It doesn't work unless it is right!

Some typical examples:

case ${target_os} in
netbsd*) have_kvm=yes ;;
*)       have_kvm=no  ;;
esac

AC_CHECK_LIB(kvm, kvm_open, have_kvm=yes, have_kvm=no)

#if defined(__NetBSD__)
#  include <sys/event.h>
#endif

#if defined(HAVE_SYS_EVENT_H)
#  include <sys/event.h>
#endif

int
monitor_file(...)
{
#if defined(__NetBSD__)
        int fd = kqueue();
        ...
#else
        ...
#endif
}

int
monitor_file(...)
{
#if defined(HAVE_KQUEUE)
        int fd = kqueue();
        ...
#else
	...
#endif
}

For more information, please read the Making packager-friendly software article (part 1, part 2). It summarizes multiple details on how to make software easier to package; all the suggestions in it were collected from our experience in pkgsrc work, so they are possibly helpful when creating patches too.

Always, always, always feed back any portability fixes or improvements you do to a package to the mainstream developers. This is the only way to get their attention on portability issues and to ensure that future versions can be built out-of-the box on NetBSD. Furthermore, any user that gets newer distfiles will get the fixes straight from the packaged code.

This generally involves cleaning up the patches (because sometimes the patches that are added to pkgsrc are quick hacks), filing bug reports in the appropriate trackers for the projects and working with the mainstream authors to accept your changes. It is extremely important that you do it so that the packages in pkgsrc are kept simple and thus further changes can be done without much hassle.

When you have done this, please add a URL to the upstream bug report to the patch comment.

Support the idea of free software!

STRING=                 foo * bar `date`
INT_LIST=               # empty
ANOTHER_INT_LIST=       apache-[0-9]*:../../www/apache
EXT_LIST=               # empty
ANOTHER_EXT_LIST=       a=b c=d

INT_LIST+=              ${STRING}               # 1
INT_LIST+=              ${ANOTHER_INT_LIST}     # 2
EXT_LIST+=              ${STRING:Q}             # 3
EXT_LIST+=              ${ANOTHER_EXT_LIST}     # 4

When you add a string to an external list (example 3), it must be quoted. In all other cases, you must not add a quoting level. You must not merge internal and external lists, unless you are sure that all entries are correctly interpreted in both lists.

EXT_LIST=       # empty
.for i in ${INT_LIST}
EXT_LIST+=      ${i:Q}""
.endfor

This code converts the internal list INT_LIST into the external list EXT_LIST. As the elements of an internal list are unquoted they must be quoted here. The reason for appending "" is explained below.

Sometimes you may want to print an arbitrary string. There are many ways to get it wrong and only few that can handle every nastiness.

STRING=         foo bar <    > * `date` $$HOME ' "
EXT_LIST=       string=${STRING:Q} x=second\ item

all:
        echo ${STRING}                  # 1
        echo "${STRING}"                # 2
        echo "${STRING:Q}"              # 3
        echo ${STRING:Q}                # 4
        echo x${STRING:Q} | sed 1s,.,,  # 5
        printf "%s\\n" ${STRING:Q}""    # 6
        env ${EXT_LIST} /bin/sh -c 'echo "$$string"; echo "$$x"'

Example 1 leads to a syntax error in the shell, as the characters are just copied.

Example 2 leads to a syntax error too, and if you leave out the last " character from ${STRING}, date(1) will be executed. The $HOME shell variable would be evaluated, too.

Example 3 outputs each space character preceded by a backslash (or not), depending on the implementation of the echo(1) command.

Example 4 handles correctly every string that does not start with a dash. In that case, the result depends on the implementation of the echo(1) command. As long as you can guarantee that your input does not start with a dash, this form is appropriate.

Example 5 handles even the case of a leading dash correctly.

Example 6 also works with every string and is the light-weight solution, since it does not involve a pipe, which has its own problems.

The EXT_LIST does not need to be quoted because the quoting has already been done when adding elements to the list.

As internal lists shall not be passed to the shell, there is no example for it.

There are many possible sources of wrongly quoted variables. This section lists some of the commonly known ones.

The pkgsrc bmake program does not handle the following assignment correctly. In case _othervar_ contains a ``-'' character, one of the closing braces is included in ${VAR} after this code executes.

VAR:=   ${VAR:N${_othervar_:C/-//}}

For a more complex code snippet and a workaround, see the package regress/make-quoting, testcase bug1.

The following real-life example buildlink3.mk is taken from pkgsrc/graphics/tiff:

# $NetBSD: buildlink3.mk,v 1.16 2009/03/20 19:24:45 joerg Exp $

BUILDLINK_TREE+=	tiff

.if !defined(TIFF_BUILDLINK3_MK)
TIFF_BUILDLINK3_MK:=

BUILDLINK_API_DEPENDS.tiff+=	tiff>=3.6.1
BUILDLINK_ABI_DEPENDS.tiff+=	tiff>=3.7.2nb1
BUILDLINK_PKGSRCDIR.tiff?=	../../graphics/tiff

.include "../../devel/zlib/buildlink3.mk"
.include "../../graphics/jpeg/buildlink3.mk"
.endif # TIFF_BUILDLINK3_MK

BUILDLINK_TREE+=	-tiff

The header and footer manipulate BUILDLINK_TREE, which is common across all buildlink3.mk files and is used to track the dependency tree.

The main section is protected from multiple inclusion and controls how the dependency on pkg is added. Several important variables are set in the section:

The following variables are all optionally defined within this second section (protected against multiple inclusion) and control which package files are symlinked into ${BUILDLINK_DIR} and how their names are transformed during the symlinking:

This section can additionally include any buildlink3.mk needed for pkg's library dependencies. Including these buildlink3.mk files means that the headers and libraries for these dependencies are also symlinked into ${BUILDLINK_DIR} whenever the pkg buildlink3.mk file is included. Dependencies are only added for directly include buildlink3.mk files.

When providing a buildlink3.mk and including other buildlink3.mk files in it, please only add necessary ones, i.e., those whose libraries or header files are automatically exposed when the package is use.

In particular, if only an executable (bin/foo) is linked against a library, that library does not need to be propagated in the buildlink3.mk file.

The following steps should help you decide if a buildlink3.mk file needs to be included:

These two variables differ in that one describes source compatibility (API) and the other binary compatibility (ABI). The difference is that a change in the API breaks compilation of programs while changes in the ABI stop compiled programs from running.

Changes to the BUILDLINK_API_DEPENDS.pkg variable in a buildlink3.mk file happen very rarely. One possible reason is that all packages depending on this already need a newer version. In case it is bumped see the description below.

The most common example of an ABI change is that the major version of a shared library is increased. In this case, BUILDLINK_ABI_DEPENDS.pkg should be adjusted to require at least the new package version. Then the packages that depend on this package need their PKGREVISIONs increased and, if they have buildlink3.mk files, their BUILDLINK_ABI_DEPENDS.pkg adjusted, too. This is needed so pkgsrc will require the correct package dependency and not settle for an older one when building the source.

See Section 19.1.6, “Handling dependencies” for more information about dependencies on other packages, including the BUILDLINK_ABI_DEPENDS and ABI_DEPENDS definitions.

Please take careful consideration before adjusting BUILDLINK_API_DEPENDS.pkg or BUILDLINK_ABI_DEPENDS.pkg as we don't want to cause unneeded package deletions and rebuilds. In many cases, new versions of packages work just fine with older dependencies.

Also it is not needed to set BUILDLINK_ABI_DEPENDS.pkg when it is identical to BUILDLINK_API_DEPENDS.pkg.

The following is the recommended template for builtin.mk files:

.if !defined(IS_BUILTIN.foo)
#
# IS_BUILTIN.foo is set to "yes" or "no" depending on whether "foo"
# genuinely exists in the system or not.
#
IS_BUILTIN.foo?=        no

# BUILTIN_PKG.foo should be set here if "foo" is built-in and its package
# version can be determined.
#
.  if !empty(IS_BUILTIN.foo:M[yY][eE][sS])
BUILTIN_PKG.foo?=       foo-1.0
.  endif
.endif  # IS_BUILTIN.foo

.if !defined(USE_BUILTIN.foo)
USE_BUILTIN.foo?=       ${IS_BUILTIN.foo}
.  if defined(BUILTIN_PKG.foo)
.    for _depend_ in ${BUILDLINK_API_DEPENDS.foo}
.      if !empty(USE_BUILTIN.foo:M[yY][eE][sS])
USE_BUILTIN.foo!=                                                       \
        ${PKG_ADMIN} pmatch '${_depend_}' ${BUILTIN_PKG.foo}            \
        && ${ECHO} "yes" || ${ECHO} "no"
.      endif
.    endfor
.  endif
.endif  # USE_BUILTIN.foo

CHECK_BUILTIN.foo?=     no
.if !empty(CHECK_BUILTIN.foo:M[nN][oO])
#
# Here we place code that depends on whether USE_BUILTIN.foo is set to
# "yes" or "no".
#
.endif  # CHECK_BUILTIN.foo

The first section sets IS_BUILTIN.pkg depending on if pkg really exists in the base system. This should not be a base system software with similar functionality to pkg; it should only be “yes” if the actual package is included as part of the base system. This variable is only used internally within the builtin.mk file.

The second section sets BUILTIN_PKG.pkg to the version of pkg in the base system if it exists (if IS_BUILTIN.pkg is “yes”). This variable is only used internally within the builtin.mk file.

The third section sets USE_BUILTIN.pkg and is required in all builtin.mk files. The code in this section must make the determination whether the built-in software is adequate to satisfy the dependencies listed in BUILDLINK_API_DEPENDS.pkg. This is typically done by comparing BUILTIN_PKG.pkg against each of the dependencies in BUILDLINK_API_DEPENDS.pkg. USE_BUILTIN.pkg must be set to the correct value by the end of the builtin.mk file. Note that USE_BUILTIN.pkg may be “yes” even if IS_BUILTIN.pkg is “no” because we may make the determination that the built-in version of the software is similar enough to be used as a replacement.

The last section is guarded by CHECK_BUILTIN.pkg, and includes code that uses the value of USE_BUILTIN.pkg set in the previous section. This typically includes, e.g., adding additional dependency restrictions and listing additional files to symlink into ${BUILDLINK_DIR} (via BUILDLINK_FILES.pkg).

When building packages, it's possible to choose whether to set a global preference for using either the built-in (native) version or the pkgsrc version of software to satisfy a dependency. This is controlled by setting PREFER_PKGSRC and PREFER_NATIVE. These variables take values of either “yes”, “no”, or a list of packages. PREFER_PKGSRC tells pkgsrc to use the pkgsrc versions of software, while PREFER_NATIVE tells pkgsrc to use the built-in versions. Preferences are determined by the most specific instance of the package in either PREFER_PKGSRC or PREFER_NATIVE. If a package is specified in neither or in both variables, then PREFER_PKGSRC has precedence over PREFER_NATIVE. For example, to require using pkgsrc versions of software for all but the most basic bits on a NetBSD system, you can set:

PREFER_PKGSRC=  yes
PREFER_NATIVE=  getopt skey tcp_wrappers

A package must have a builtin.mk file to be listed in PREFER_NATIVE, otherwise it is simply ignored in that list.

The following variables can be set to request the creation of directories anywhere in the file system:

Creating non-empty files outside the installation prefix is tricky because the PLIST forces all files to be inside it. To overcome this problem, the only solution is to extract the file in the known place (i.e., inside the installation prefix) and copy it to the appropriate location during installation (done by the installation scripts generated by pkginstall). We will call the former the master file in the following paragraphs, which describe the variables that can be used to automatically and consistently handle files outside the installation prefix:

As said before, the PKG_SYSCONFDIR variable specifies where configuration files shall be installed. Its contents are set based upon the following variables:

Based on the above variables, pkginstall determines the value of PKG_SYSCONFDIR, which is the only variable that can be used within a package to refer to its configuration directory. The algorithm used to set its value is basically the following:

It is worth mentioning that ${PKG_SYSCONFDIR} is automatically added to OWN_DIRS. See Section 15.1.1, “Directory manipulation” what this means. This does not apply to subdirectories of ${PKG_SYSCONFDIR}, they still have to be created with OWN_DIRS or MAKE_DIRS.

Given that pkgsrc (and users!) expect configuration files to be in a known place, you need to teach each package where it shall install its files. In some cases you will have to patch the package Makefiles to achieve it. If you are lucky, though, it may be as easy as passing an extra flag to the configuration script; this is the case of GNU Autoconf- generated files:

CONFIGURE_ARGS+= --sysconfdir=${PKG_SYSCONFDIR}

Note that this specifies where the package has to look for its configuration files, not where they will be originally installed (although the difference is never explicit, unfortunately).

As said before, pkginstall automatically handles configuration files. This means that the packages themselves must not touch the contents of ${PKG_SYSCONFDIR} directly. Bad news is that many software installation scripts will, out of the box, mess with the contents of that directory. So what is the correct procedure to fix this issue?

You must teach the package (usually by manually patching it) to install any configuration files under the examples hierarchy, share/examples/${PKGBASE}/. This way, the PLIST registers them and the administrator always has the original copies available.

Once the required configuration files are in place (i.e., under the examples hierarchy), the pkginstall framework can use them as master copies during the package installation to update what is in ${PKG_SYSCONFDIR}. To achieve this, the variables CONF_FILES and CONF_FILES_PERMS are used. Check out Section 15.1.2, “File manipulation” for information about their syntax and their purpose. Here is an example, taken from the mail/mutt package:

EGDIR=        ${PREFIX}/share/doc/mutt/samples
CONF_FILES=   ${EGDIR}/Muttrc ${PKG_SYSCONFDIR}/Muttrc

Note that the EGDIR variable is specific to that package and has no meaning outside it.

The automatic copying of config files can be toggled by setting the environment variable PKG_CONFIG prior to package installation.

The automatic copying of config files can be toggled by setting the environment variable PKG_RCD_SCRIPTS prior to package installation. Note that the scripts will be always copied inside the examples hierarchy, ${PREFIX}/share/examples/rc.d/, no matter what the value of this variable is.

The automatic registration of shell interpreters can be disabled by the administrator by setting the PKG_REGISTER_SHELLS environment variable to NO.

The automatic update of fonts databases can be disabled by the administrator by setting the PKG_UPDATE_FONTS_DB environment variable to NO.