The primary download location for all pkgsrc files is ftp://ftp.NetBSD.org/pub/pkgsrc/. There are a number of subdirectories for different purposes, which are described in detail in Appendix C, Directory layout of the pkgsrc FTP server.

The tar file for the current branch is in the directory current and is called pkgsrc.tar.gz. It is autogenerated daily.

The tar file for the stable branch 2007Q4 is in the directory pkgsrc-2007Q4 and is also called pkgsrc-2007Q4.tar.gz.

To download a pkgsrc stable tarball, run:

$ ftp ftp://ftp.NetBSD.org/pub/pkgsrc/pkgsrc-200xQy/pkgsrc-200xQy.tar.gz

Where pkgsrc-200xQy is the stable branch to be downloaded, for example, “pkgsrc-2007Q4”.

Then, extract it with:

$ tar -xzf pkgsrc-200xQy.tar.gz -C /usr

This will create the directory pkgsrc/ in /usr/ and all the package source will be stored under /usr/pkgsrc/.

To download pkgsrc-current, run:

$ ftp ftp://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz

As an alternative to the tar file, you can get pkgsrc via the Software Update Protocol, SUP. To do so, make sure your supfile has a line

release=pkgsrc

in it, see the examples in /usr/share/examples/supfiles, and that the /usr/pkgsrc directory exists. Then, simply run:

$ sup -v /path/to/your/supfile.

To do an initial (full) checkout of pkgsrc, you first have to set some environment variables. For the C-Shell, type:

% setenv CVSROOT anoncvs@anoncvs.NetBSD.org:/cvsroot
% setenv CVS_RSH ssh

Or, the same for the bourne shell:

$ export CVSROOT="anoncvs@anoncvs.NetBSD.org:/cvsroot"
$ export CVS_RSH="ssh"

By default, CVS doesn't do things like most people would expect it to do. But there is a way to convince CVS, by creating a file called .cvsrc in your home directory and saving the following lines to it. This file will save you lots of headache and some bug reports, so we strongly recommend it. You can find an explanation of this file in the CVS documentation.

# recommended CVS configuration file from the pkgsrc guide
checkout -P
update -dP
release -d
diff -upN
cvs -q -z3
rdiff -u

To fetch a specific pkgsrc stable branch from scratch, run:

$ cd /usr
$ cvs checkout -r pkgsrc-200xQy -P pkgsrc

Where pkgsrc-200xQy is the stable branch to be checked out, for example, “pkgsrc-2007Q4”

This will create the directory pkgsrc/ in your /usr/ directory and all the package source will be stored under /usr/pkgsrc/.

To fetch the pkgsrc current branch, run:

$ cd /usr
$ cvs checkout -P pkgsrc

Note that by default the distfiles and the binary packages are saved in the pkgsrc tree, so don't forget to rescue them before updating. You can also configure pkgsrc to use other than the default directories by setting the DISTDIR and PACKAGES variables. See Chapter 5, Configuring pkgsrc for the details.

To update pkgsrc from a tar file, download the tar file as explained above. Then, make sure that you have not made any changes to the files in the pkgsrc directory. Remove the pkgsrc directory and extract the new tar file. Done.

To update pkgsrc via CVS, make sure the environment variable CVS_RSH is set as above. Then, change to the pkgsrc directory and run cvs:

$ cd /usr/pkgsrc
$ cvs update -dP

Darwin 5.x and up are supported. There are two methods of using pkgsrc on Mac OS X, by using a disk image, or a UFS or HFSX partition.

Before you start, you will need to download and install the Mac OS X Developer Tools from Apple's Developer Connection. See http://developer.apple.com/macosx/ for details. Also, make sure you install X11 for Mac OS X and the X11 SDK from http://www.apple.com/macosx/x11/download/ if you intend to build packages that use the X11 Window System.

If you already have a UFS or HFSX partition, or have a spare partition that you can format as UFS or HFSX, it is recommended to use that instead of the disk image. It'll be somewhat faster and will mount automatically at boot time, where you must manually mount a disk image.

# cd pkgsrc/bootstrap
# ./darwindiskimage create ~/Documents/NetBSD 1024 # megabytes - season to taste
# ./darwindiskimage mount ~/Documents/NetBSD
# sudo chown `id -u`:`id -g` /Volumes/NetBSD

# sudo mount -u -o suid /Volumes/NetBSD

	  WRKOBJDIR?=		/tmp/pkgsrc         # build here instead of in pkgsrc
	

# ./bootstrap --pkgdbdir /usr/pkg/pkgdb

FreeBSD 4.7 and 5.0 have been tested and are supported, other versions may work.

Care should be taken so that the tools that this kit installs do not conflict with the FreeBSD userland tools. There are several steps:

Interix is a POSIX-compatible subsystem for the Windows NT kernel, providing a Unix-like environment with a tighter kernel integration than available with Cygwin. It is part of the Windows Services for Unix package, available for free for any licensed copy of Windows 2000, XP (not including XP Home), or 2003. SFU can be downloaded from http://www.microsoft.com/windows/sfu/.

Services for Unix 3.5 has been tested. 3.0 or 3.1 may work, but are not officially supported. (The main difference in 3.0/3.1 is lack of pthreads, but other parts of libc may also be lacking.)

Services for Unix Applications (aka SUA) is an integrated component of Windows Server 2003 R2 and Windows Vista. As of this writing, SUA's Interix 5.x subsystem has not yet been tested with pkgsrc.

interix:kP=\E[S:kN=\E[T:kH=\E[U:dc@:DC@:tc=pcansi:
	

# chmod -R g+w $PKG_DBDIR
	

You will need a working C compiler, either gcc or SGI's MIPS and MIPSpro compiler (cc/c89). Please set the CC environment variable according to your preference. If you do not have a license for the MIPSpro compiler suite, you can download a gcc tardist file from http://freeware.sgi.com/.

Please note that you will need IRIX 6.5.17 or higher, as this is the earliest version of IRIX providing support for if_indextoname(3), if_nametoindex(3), etc.

At this point in time, pkgsrc only supports one ABI at a time. That is, you cannot switch between the old 32-bit ABI, the new 32-bit ABI and the 64-bit ABI. If you start out using "abi=n32", that's what all your packages will be built with.

Therefore, please make sure that you have no conflicting CFLAGS in your environment or the mk.conf. Particularly, make sure that you do not try to link n32 object files with lib64 or vice versa. Check your /etc/compiler.defaults!

If you have the actual pkgsrc tree mounted via NFS from a different host, please make sure to set WRKOBJDIR to a local directory, as it appears that IRIX linker occasionally runs into issues when trying to link over a network-mounted file system.

The bootstrapping process should set all the right options for programs such as imake(1), but you may want to set some options depending on your local setup. Please see pkgsrc/mk/defaults/mk.conf and, of course, your compiler's man pages for details.

If you are using SGI's MIPSPro compiler, please set

PKGSRC_COMPILER=        mipspro
      

in mk.conf. Otherwise, pkgsrc will assume you are using gcc and may end up passing invalid flags to the compiler. Note that bootstrap should create an appropriate mk.conf.example by default.

If you have both the MIPSPro compiler chain installed as well as gcc, but want to make sure that MIPSPro is used, please set your PATH to not include the location of gcc (often /usr/freeware/bin), and (important) pass the '--preserve-path' flag.

Some versions of Linux (for example Debian GNU/Linux) need either libtermcap or libcurses (libncurses). Installing the distributions libncurses-dev package (or equivalent) should fix the problem.

pkgsrc supports both gcc (GNU Compiler Collection) and icc (Intel C++ Compiler). gcc is the default. icc 8.0 and 8.1 on i386 have been tested.

To bootstrap using icc, assuming the default icc installation directory:

env CC=/opt/intel_cc_80/bin/icc LDFLAGS=-static-libcxa \
ac_cv___attribute__=yes ./bootstrap
      

icc supports __attribute__, but the GNU configure test uses a nested function, which icc does not support. #undef'ing __attribute__ has the unfortunate side-effect of breaking many of the Linux header files, which cannot be compiled properly without __attribute__. The test must be overridden so that __attribute__ is assumed supported by the compiler.

After bootstrapping, you should set PKGSRC_COMPILER in mk.conf:

PKGSRC_COMPILER=        icc
      

The default installation directory for icc is /opt/intel_cc_80, which is also the pkgsrc default. If you have installed it into a different directory, set ICCBASE in mk.conf:

ICCBASE=                /opt/icc
      

pkgsrc uses the static linking method of the runtime libraries provided by icc, so binaries can be run on other systems which do not have the shared libraries installed.

Libtool, however, extracts a list of libraries from the ld(1) command run when linking a C++ shared library and records it, throwing away the -Bstatic and -Bdynamic options interspersed between the libraries. This means that libtool-linked C++ shared libraries will have a runtime dependency on the icc libraries until this is fixed in libtool.

OpenBSD 3.0 and 3.2 are tested and supported.

Care should be taken so that the tools that this kit installs do not conflict with the OpenBSD userland tools. There are several steps:

Solaris 2.6 through 9 are supported on both x86 and sparc. You will need a working C compiler. Both gcc 2.95.3 and Sun WorkShop 5 have been tested.

The following packages are required on Solaris 8 for the bootstrap process and to build packages.

Please note that the use of GNU binutils on Solaris is not supported, as of June 2006.

Whichever compiler you use, please ensure the compiler tools and your $prefix are in your PATH. This includes /usr/ccs/{bin,lib} and e.g. /usr/pkg/{bin,sbin}.

CC=     cc
CXX=    CC
CPP=    cc -E
CXXCPP= CC -E

PKGSRC_COMPILER=        sunpro
ABI=                    64

CONFIG_SHELL=   ${LOCALBASE}/bin/bash
WRAPPER_SHELL=  ${LOCALBASE}/bin/bash
      

To install binary packages, you first need to know from where to get them. The first place where you should look is on the main pkgsrc FTP server in the directory /pub/pkgsrc/packages.

This directory contains binary packages for multiple platforms. First, select your operating system. (Ignore the directories with version numbers attached to it, they just exist for legacy reasons.) Then, select your hardware architecture, and in the third step, the OS version and the “version” of pkgsrc.

In this directory, you often find a file called bootstrap.tar.gz which contains the package management tools. If the file is missing, it is likely that your operating system already provides those tools. Download the file and extract it in the / directory. It will create the directories /usr/pkg (containing the tools for managing binary packages) and /var/db/pkg (the database of installed packages).

In the directory from the last section, there is a subdirectory called All, which contains all the binary packages that are available for the platform, excluding those that may not be distributed via FTP or CDROM (depending on which medium you are using), and the ones that have vulnerabilities and therefore are considered insecure to install without thinking before.

To install packages directly from an FTP or HTTP server, run the following commands in a Bourne-compatible shell (be sure to su to root first):

# PATH="/usr/pkg/sbin:$PATH"
# PKG_PATH="ftp://ftp.NetBSD.org/pub/pkgsrc/packages/OPSYS/ARCH/VERSIONS/All"
# export PATH PKG_PATH

Instead of URLs, you can also use local paths, for example if you are installing from a set of CDROMs, DVDs or an NFS-mounted repository. If you want to install packages from multiple sources, you can separate them by a semicolon in PKG_PATH.

After these preparations, installing a package is very easy:

# pkg_add openoffice2
# pkg_add kde-3.5.7
# pkg_add ap2-php5-*

Note that any prerequisite packages needed to run the package in question will be installed, too, assuming they are present where you install from.

As mentioned above, packages for which vulnerabilities get known are not stored in the All subdirectory. They don't get deleted since that could be very frustrating if many other packages depend on it. Instead, they are moved to the vulnerable subdirectory. So you may need to add this directory to the PKG_PATH variable. However, you should run audit-packages regularly, especially after installing new packages, and verify that the vulnerabilities are acceptable for your configuration.

After you've installed packages, be sure to have /usr/pkg/bin and /usr/pkg/sbin in your PATH so you can actually start the just installed program.

To deinstall a package, it does not matter whether it was installed from source code or from a binary package. The pkg_delete command does not know it anyway. To delete a package, you can just run pkg_delete package-name. The package name can be given with or without version number. Wildcards can also be used to deinstall a set of packages, for example *emacs*. Be sure to include them in quotes, so that the shell does not expand them before pkg_delete sees them.

The -r option is very powerful: it removes all the packages that require the package in question and then removes the package itself. For example:

# pkg_delete -r jpeg
    

will remove jpeg and all the packages that used it; this allows upgrading the jpeg package.

The pkg_info shows information about installed packages or binary package files.

The NetBSD Security-Officer and Packages Groups maintain a list of known security vulnerabilities to packages which are (or have been) included in pkgsrc. The list is available from the NetBSD FTP site at ftp://ftp.NetBSD.org/pub/pkgsrc/distfiles/vulnerabilities.

Through security/audit-packages, this list can be downloaded automatically, and a security audit of all packages installed on a system can take place.

There are two components to security/audit-packages. The first component, “download-vulnerability-list”, is for downloading the list of vulnerabilities from the NetBSD FTP site. The second component, “audit-packages”, checks to see if any of your installed packages are vulnerable. If a package is vulnerable, you will see output similar to the following:

Package samba-2.0.9 has a local-root-shell vulnerability, see
    http://www.samba.org/samba/whatsnew/macroexploit.html

One can set up security/audit-packages to download the vulnerabilities file daily, and include a package audit in the daily security script. Details on this are located in the MESSAGE file for security/audit-packages.

Install pkgtools/lintpkgsrc and run lintpkgsrc with the “-i” argument to check if your packages are up-to-date, e.g.

% lintpkgsrc -i
...
Version mismatch: 'tcsh' 6.09.00 vs 6.10.00
    

You can then use make update to update the package on your system and rebuild any dependencies.

The pkg_admin executes various administrative functions on the package system.

Please pay very careful attention to the warnings expressed in the pkg_add(1) manual page about the inherent dangers of installing binary packages which you did not create yourself, and the security holes that can be introduced onto your system by indiscriminate adding of such files.

The same warning of course applies to every package you install from source when you haven't completely read and understood the source code of the package, the compiler that is used to build the package and all the other tools that are involved.

To build packages from source, you need a working C compiler. On NetBSD, you need to install the “comp” and the “text” distribution sets. If you want to build X11-related packages, the “xbase” and “xcomp” distribution sets are required, too.

The first step for building a package is downloading the distfiles (i.e. the unmodified source). If they have not yet been downloaded, pkgsrc will fetch them automatically.

If you have all files that you need in the distfiles directory, you don't need to connect. If the distfiles are on CD-ROM, you can mount the CD-ROM on /cdrom and add:

DISTDIR=/cdrom/pkgsrc/distfiles

to your mk.conf.

By default a list of distribution sites will be randomly intermixed to prevent huge load on servers which holding popular packages (for example, SourceForge.net mirrors). Thus, every time when you need to fetch yet another distfile all the mirrors will be tried in new (random) order. You can turn this feature off by setting MASTER_SORT_RANDOM=NO (for PKG_DEVELOPERs it's already disabled).

You can overwrite some of the major distribution sites to fit to sites that are close to your own. By setting one or two variables you can modify the order in which the master sites are accessed. MASTER_SORT contains a whitespace delimited list of domain suffixes. MASTER_SORT_REGEX is even more flexible, it contains a whitespace delimited list of regular expressions. It has higher priority than MASTER_SORT. Have a look at pkgsrc/mk/defaults/mk.conf to find some examples. This may save some of your bandwidth and time.

You can change these settings either in your shell's environment, or, if you want to keep the settings, by editing the mk.conf file, and adding the definitions there.

If a package depends on many other packages (such as meta-pkgs/kde3), the build process may alternate between periods of downloading source, and compiling. To ensure you have all the source downloaded initially you can run the command:

% make fetch-list | sh

which will output and run a set of shell commands to fetch the necessary files into the distfiles directory. You can also choose to download the files manually.

Once the software has downloaded, any patches will be applied, then it will be compiled for you. This may take some time depending on your computer, and how many other packages the software depends on and their compile time.

For example, type

% cd misc/figlet
% make
    

at the shell prompt to build the various components of the package.

The next stage is to actually install the newly compiled program onto your system. Do this by entering:

% make install
    

while you are still in the directory for whatever package you are installing.

Installing the package on your system may require you to be root. However, pkgsrc has a just-in-time-su feature, which allows you to only become root for the actual installation step.

That's it, the software should now be installed and setup for use. You can now enter:

% make clean
    

to remove the compiled files in the work directory, as you shouldn't need them any more. If other packages were also added to your system (dependencies) to allow your program to compile, you can tidy these up also with the command:

% make clean-depends
    

Taking the figlet utility as an example, we can install it on our system by building as shown in Appendix B, Build logs.

The program is installed under the default root of the packages tree - /usr/pkg. Should this not conform to your tastes, set the LOCALBASE variable in your environment, and it will use that value as the root of your packages tree. So, to use /usr/local, set LOCALBASE=/usr/local in your environment. Please note that you should use a directory which is dedicated to packages and not shared with other programs (i.e., do not try and use LOCALBASE=/usr). Also, you should not try to add any of your own files or directories (such as src/, obj/, or pkgsrc/) below the LOCALBASE tree. This is to prevent possible conflicts between programs and other files installed by the package system and whatever else may have been installed there.

Some packages look in mk.conf to alter some configuration options at build time. Have a look at pkgsrc/mk/defaults/mk.conf to get an overview of what will be set there by default. Environment variables such as LOCALBASE can be set in mk.conf to save having to remember to set them each time you want to use pkgsrc.

Occasionally, people want to “look under the covers” to see what is going on when a package is building or being installed. This may be for debugging purposes, or out of simple curiosity. A number of utility values have been added to help with this.

If you want to install a binary package that you've either created yourself (see next section), that you put into pkgsrc/packages manually or that is located on a remote FTP server, you can use the "bin-install" target. This target will install a binary package - if available - via pkg_add(1), else do a make package. The list of remote FTP sites searched is kept in the variable BINPKG_SITES, which defaults to ftp.NetBSD.org. Any flags that should be added to pkg_add(1) can be put into BIN_INSTALL_FLAGS. See pkgsrc/mk/defaults/mk.conf for more details.

A final word of warning: If you set up a system that has a non-standard setting for LOCALBASE, be sure to set that before any packages are installed, as you cannot use several directories for the same purpose. Doing so will result in pkgsrc not being able to properly detect your installed packages, and fail miserably. Note also that precompiled binary packages are usually built with the default LOCALBASE of /usr/pkg, and that you should not install any if you use a non-standard LOCALBASE.

By default, pkgsrc will use GCC to build packages. This may be overridden by setting the following variables in /etc/mk.conf:

If you wish to set the CFLAGS variable, please make sure to use the += operator instead of the = operator:

CFLAGS+=        -your -flags

Using CFLAGS= (i.e. without the “+”) may lead to problems with packages that need to add their own flags. Also, you may want to take a look at the devel/cpuflags package if you're interested in optimization for the current CPU.

If you want to pass flags to the linker, both in the configure step and the build step, you can do this in two ways. Either set LDFLAGS or LIBS. The difference between the two is that LIBS will be appended to the command line, while LDFLAGS come earlier. LDFLAGS is pre-loaded with rpath settings for ELF machines depending on the setting of USE_IMAKE or the inclusion of mk/x11.buildlink3.mk. As with CFLAGS, if you do not wish to override these settings, use the += operator:

LDFLAGS+=        -your -linkerflags

As /usr/pkg will be completely deleted at the start of bulk builds, make sure your login shell is placed somewhere else. Either drop it into /usr/local/bin (and adjust your login shell in the passwd file), or (re-)install it via pkg_add(1) from /etc/rc.local, so you can login after a reboot (remember that your current process won't die if the package is removed, you just can't start any new instances of the shell any more). Also, if you use NetBSD earlier than 1.5, or you still want to use the pkgsrc version of ssh for some reason, be sure to install ssh before starting it from rc.local:

(cd /usr/pkgsrc/security/ssh && make bulk-install)
if [ -f /usr/pkg/etc/rc.d/sshd ]; then
  /usr/pkg/etc/rc.d/sshd
fi

Not doing so will result in you being not able to log in via ssh after the bulk build is finished or if the machine gets rebooted or crashes. You have been warned! :)

Make sure you don't need any of the packages still installed.

Be sure to remove all other things that might interfere with builds, like some libs installed in /usr/local, etc. then become root and type:

# cd /usr/pkgsrc
# sh mk/bulk/build
      

If for some reason your last build didn't complete (power failure, system panic, ...), you can continue it by running:

# sh mk/bulk/build restart

At the end of the bulk build, you will get a summary via mail, and find build logs in the directory specified by FTP in the build.conf file.

The bulk builds consist of three steps:

During the build, a list of broken packages will be compiled in /usr/pkgsrc/.broken (or .../.broken.${MACHINE} if OBJMACHINE is set), individual build logs of broken builds can be found in the package's directory. These files are used by the bulk-targets to mark broken builds to not waste time trying to rebuild them, and they can be used to debug these broken package builds later.

Currently, roughly the following requirements are valid for NetBSD 2.0/i386:

Note that all pkgs will be de-installed as soon as they are turned into a binary package, and that sources are removed, so there is no excessively huge demand to disk space. Afterwards, if the package is needed again, it will be installed via pkg_add(1) instead of building again, so there are no cycles wasted by recompiling.

If you don't want all the packages nuked from a machine (rendering it useless for anything but pkg compiling), there is the possibility of doing the package bulk build inside a chroot environment.

The first step is to set up a chroot sandbox, e.g. /usr/sandbox. This can be done by using null mounts, or manually.

There is a shell script called pkgsrc/mk/bulk/mksandbox which will set up the sandbox environment using null mounts. It will also create a script called sandbox in the root of the sandbox environment, which will allow the null mounts to be activated using the sandbox mount command and deactivated using the sandbox umount command.

To set up a sandbox environment by hand, after extracting all the sets from a NetBSD installation or doing a make distribution DESTDIR=/usr/sandbox in /usr/src/etc, be sure the following items are present and properly configured:

When the chroot sandbox is set up, you can start the build with the following steps:

# cd /usr/sandbox/usr/pkgsrc
# sh mk/bulk/do-sandbox-build
      

This will just jump inside the sandbox and start building. At the end of the build, mail will be sent with the results of the build. Created binary pkgs will be in /usr/sandbox/usr/pkgsrc/packages (wherever that points/mounts to/from).

In addition to building a complete set of all packages in pkgsrc, the pkgsrc/mk/bulk/build script may be used to build a subset of the packages contained in pkgsrc. By setting SPECIFIC_PKGS in mk.conf, the variables

will define the set of packages which should be built. The bulk build code will also include any packages which are needed as dependencies for the explicitly listed packages.

One use of this is to do a bulk build with SPECIFIC_PKGS in a chroot sandbox periodically to have a complete set of the binary packages needed for your site available without the overhead of building extra packages that are not needed.

This section describes how pkgsrc developers can upload binary pkgs built by bulk builds to ftp.NetBSD.org.

If you would like to automatically create checksum files for the binary packages you intend to upload, remember to set MKSUMS=yes in your mk/bulk/build.conf.

If you would like to PGP sign the checksum files (highly recommended!), remember to set SIGN_AS=username@NetBSD.org in your mk/bulk/build.conf. This will prompt you for your GPG password to sign the files before uploading everything.

Then, make sure that you have RSYNC_DST set properly in your mk/bulk/build.conf file, i.e. adjust it to something like one of the following:

RSYNC_DST=ftp.NetBSD.org:/pub/NetBSD/packages/packages-200xQy/NetBSD-a.b.c/arch/upload 

Please use appropriate values for "packages-200xQy", "NetBSD-a.b.c" and "arch" here. If your login on ftp.NetBSD.org is different from your local login, write your login directly into the variable, e.g. my local account is "feyrer", but for my login "hubertf", I use:

RSYNC_DST=hubertf@ftp.NetBSD.org:/pub/NetBSD/packages/packages-200xQy/NetBSD-a.b.c/arch/upload

A separate upload directory is used here to allow "closing" the directory during upload. To do so, run the following command on ftp.NetBSD.org next:

nbftp% mkdir -p -m 750 /pub/NetBSD/packages/packages-200xQy/NetBSD-a.b.c/arch/upload

Please note that /pub/NetBSD/packages is only appropriate for packages for the NetBSD operating system. Binary packages for other operating systems should go into /pub/pkgsrc.

Before uploading the binary pkgs, ssh authentication needs to be set up. This example shows how to set up temporary keys for the root account inside the sandbox (assuming that no keys should be present there usually):

# chroot /usr/sandbox
chroot-# rm $HOME/.ssh/id-dsa*
chroot-# ssh-keygen -t dsa
chroot-# cat $HOME/.ssh/id-dsa.pub
      

Now take the output of id-dsa.pub and append it to your ~/.ssh/authorized_keys file on ftp.NetBSD.org. You can remove the key after the upload is done!

Next, test if your ssh connection really works:

chroot-# ssh ftp.NetBSD.org date 

Use "-l yourNetBSDlogin" here as appropriate!

Now after all this works, you can exit the sandbox and start the upload:

chroot-# exit
# cd /usr/sandbox/usr/pkgsrc
# sh mk/bulk/do-sandbox-upload
      

The upload process may take quite some time. Use ls(1) or du(1) on the FTP server to monitor progress of the upload. The upload script will take care of not uploading restricted packages and putting vulnerable packages into the vulnerable subdirectory.

After the upload has ended, first thing is to revoke ssh access:

nbftp% vi ~/.ssh/authorized_keys
      Gdd:x! 

Use whatever is needed to remove the key you've entered before! Last, move the uploaded packages out of the upload directory to have them accessible to everyone:

nbftp% cd /pub/NetBSD/packages/packages-200xQy/NetBSD-a.b.c/arch
nbftp% mv upload/* .
nbftp% rmdir upload
nbftp% chmod 755 .
      

TODO; see pkgsrc/doc/HOWTO-pbulk for more information.

Complete documentation for cdpack is found in the cdpack(1) man page. The following short example assumes that the binary packages are left in /usr/pkgsrc/packages/All and that sufficient disk space exists in /u2 to hold the ISO 9660 images.

# mkdir /u2/images
# pkg_add /usr/pkgsrc/packages/All/cdpack
# cdpack /usr/pkgsrc/packages/All /u2/images
      

If you wish to include a common set of files (COPYRIGHT, README, etc.) on each CD in the collection, then you need to create a directory which contains these files. e.g.

# mkdir /tmp/common
# echo "This is a README" > /tmp/common/README
# echo "Another file" > /tmp/common/COPYING
# mkdir /tmp/common/bin
# echo "#!/bin/sh" > /tmp/common/bin/myscript
# echo "echo Hello world" >> /tmp/common/bin/myscript
# chmod 755 /tmp/common/bin/myscript
      

Now create the images:

# cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images

Each image will contain README, COPYING, and bin/myscript in their root directories.

Simple Perl modules are handled automatically by url2pkg, including dependencies.

KDE applications should always include meta-pkgs/kde3/kde3.mk, which contains numerous settings that are typical of KDE packages.

The patch-* files should be in diff -bu format, and apply without a fuzz to avoid problems. (To force patches to apply with fuzz you can set PATCH_FUZZ_FACTOR=-F2). Furthermore, each patch should contain only changes for a single file, and no file should be patched by more than one patch file. This helps to keep future modifications simple.

Each patch file is structured as follows: In the first line, there is the RCS Id of the patch itself. The second line should be empty for aesthetic reasons. After that, there should be a comment for each change that the patch does. There are a number of standard cases:

In all other cases, the patch should be commented so that any developer who knows the code of the application can make some use of the patch. Special care should be taken for the upstream developers, since we generally want that they accept our patches, so we have less work in the future.

One important thing to mention is to pay attention that no RCS IDs get stored in the patch files, as these will cause problems when later checked into the NetBSD CVS tree. Use the pkgdiff command from the pkgtools/pkgdiff package to avoid these problems.

For even more automation, we recommend using mkpatches from the same package to make a whole set of patches. You just have to backup files before you edit them to filename.orig, e.g. with cp -p filename filename.orig or, easier, by using pkgvi again from the same package. If you upgrade a package this way, you can easily compare the new set of patches with the previously existing one with patchdiff. Copy the patches you want to use or update from the work/.newpatches directory to patches/.

When you have finished a package, remember to generate the checksums for the patch files by using the make makepatchsum command, see Section 11.2, “distinfo”.

When adding a patch that corrects a problem in the distfile (rather than e.g. enforcing pkgsrc's view of where man pages should go), send the patch as a bug report to the maintainer. This benefits non-pkgsrc users of the package, and usually makes it possible to remove the patch in future version.

The file names of the patch files are usually of the form patch-[a-z][a-z].

If you want to share patches between multiple packages in pkgsrc, e.g. because they use the same distfiles, set PATCHDIR to the path where the patch files can be found, e.g.:

PATCHDIR= ${.CURDIR}/../xemacs/patches

Patch files that are distributed by the author or other maintainers can be listed in PATCHFILES.

If it is desired to store any patches that should not be committed into pkgsrc, they can be kept outside the pkgsrc tree in the $LOCALPATCHES directory. The directory tree there is expected to have the same “category/package” structure as pkgsrc, and patches are expected to be stored inside these dirs (also known as $LOCALPATCHES/$PKGPATH). For example, if you want to keep a private patch for pkgsrc/graphics/png, keep it in $LOCALPATCHES/graphics/png/mypatch. All files in the named directory are expected to be patch files, and they are applied after pkgsrc patches are applied.

When fixing a portability issue in the code do not use preprocessor magic to check for the current operating system nor platform. Doing so hurts portability to other platforms because the OS-specific details are not abstracted appropriately.

The general rule to follow is: instead of checking for the operating system the application is being built on, check for the specific features you need. For example, instead of assuming that kqueue is available under NetBSD and using the __NetBSD__ macro to conditionalize kqueue support, add a check that detects kqueue itself — yes, this generally involves patching the configure script. There is absolutely nothing that prevents some OSes from adopting interfaces from other OSes (e.g. Linux implementing kqueue), something that the above checks cannot take into account.

Of course, checking for features generally involves more work on the developer's side, but the resulting changes are cleaner and there are chances they will work on many other platforms. Not to mention that there are higher chances of being later integrated into the mainstream sources. Remember: It doesn't work unless it is right!

Some typical examples:

case ${target_os} in
netbsd*) have_kvm=yes ;;
*)       have_kvm=no  ;;
esac

AC_CHECK_LIB(kvm, kvm_open, have_kvm=yes, have_kvm=no)

#if defined(__NetBSD__)
#  include <sys/event.h>
#endif

#if defined(HAVE_SYS_EVENT_H)
#  include <sys/event.h>
#endif

int
monitor_file(...)
{
#if defined(__NetBSD__)
        int fd = kqueue();
        ...
#else
        ...
#endif
}

int
monitor_file(...)
{
#if defined(HAVE_KQUEUE)
        int fd = kqueue();
        ...
#else
	...
#endif
}

For more information, please read the Making packager-friendly software article (part 1, part 2). It summarizes multiple details on how to make software easier to package; all the suggestions in it were collected from our experience in pkgsrc work, so they are possibly helpful when creating patches too.

Always, always, always feed back any portability fixes or improvements you do to a package to the mainstream developers. This is the only way to get their attention on portability issues and to ensure that future versions can be built out-of-the box on NetBSD. Furthermore, any user that gets newer distfiles will get the fixes straight from the packaged code.

This generally involves cleaning up the patches (because sometimes the patches that are added to pkgsrc are quick hacks), filling bug reports in the appropriate trackers for the projects and working with the mainstream authors to accept your changes. It is extremely important that you do it so that the packages in pkgsrc are kept simple and thus further changes can be done without much hassle.

Support the idea of free software!

STRING=                 foo * bar `date`
INT_LIST=               # empty
ANOTHER_INT_LIST=       apache-[0-9]*:../../www/apache
EXT_LIST=               # empty
ANOTHER_EXT_LIST=       a=b c=d

INT_LIST+=              ${STRING}               # 1
INT_LIST+=              ${ANOTHER_INT_LIST}     # 2
EXT_LIST+=              ${STRING:Q}             # 3
EXT_LIST+=              ${ANOTHER_EXT_LIST}     # 4

When you add a string to an external list (example 3), it must be quoted. In all other cases, you must not add a quoting level. You must not merge internal and external lists, unless you are sure that all entries are correctly interpreted in both lists.

EXT_LIST=       # empty
.for i in ${INT_LIST}
EXT_LIST+=      ${i:Q}""
.endfor

This code converts the internal list INT_LIST into the external list EXT_LIST. As the elements of an internal list are unquoted they must be quoted here. The reason for appending "" is explained below.

Sometimes you may want to print an arbitrary string. There are many ways to get it wrong and only few that can handle every nastiness.

STRING=         foo bar <    > * `date` $$HOME ' "
EXT_LIST=       string=${STRING:Q} x=second\ item

all:
        echo ${STRING}                  # 1
        echo "${STRING}"                # 2
        echo "${STRING:Q}"              # 3
        echo ${STRING:Q}                # 4
        echo x${STRING:Q} | sed 1s,.,,  # 5
        printf "%s\\n" ${STRING:Q}""    # 6
        env ${EXT_LIST} /bin/sh -c 'echo "$$string"; echo "$$x"'

Example 1 leads to a syntax error in the shell, as the characters are just copied.

Example 2 leads to a syntax error too, and if you leave out the last " character from ${STRING}, date(1) will be executed. The $HOME shell variable would be evaluated, too.

Example 3 outputs each space character preceded by a backslash (or not), depending on the implementation of the echo(1) command.

Example 4 handles correctly every string that does not start with a dash. In that case, the result depends on the implementation of the echo(1) command. As long as you can guarantee that your input does not start with a dash, this form is appropriate.

Example 5 handles even the case of a leading dash correctly.

Example 6 also works with every string and is the light-weight solution, since it does not involve a pipe, which has its own problems.

The EXT_LIST does not need to be quoted because the quoting has already been done when adding elements to the list.

As internal lists shall not be passed to the shell, there is no example for it.

There are many possible sources of wrongly quoted variables. This section lists some of the commonly known ones.

The pkgsrc bmake program does not handle the following assignment correctly. In case _othervar_ contains a ``-'' character, one of the closing braces is included in ${VAR} after this code executes.

VAR:=   ${VAR:N${_othervar_:C/-//}}

For a more complex code snippet and a workaround, see the package regress/make-quoting, testcase bug1.

The following real-life example buildlink3.mk is taken from pkgsrc/graphics/tiff:

# $NetBSD: buildlink3.mk,v 1.7 2004/03/18 09:12:12 jlam Exp $

BUILDLINK_DEPTH:=       ${BUILDLINK_DEPTH}+
TIFF_BUILDLINK3_MK:=    ${TIFF_BUILDLINK3_MK}+

.if ${BUILDLINK_DEPTH} == "+"
BUILDLINK_DEPENDS+=     tiff
.endif

BUILDLINK_PACKAGES:=    ${BUILDLINK_PACKAGES:Ntiff}
BUILDLINK_PACKAGES+=    tiff

.if ${TIFF_BUILDLINK3_MK} == "+"
BUILDLINK_API_DEPENDS.tiff+=        tiff>=3.6.1
BUILDLINK_PKGSRCDIR.tiff?=      ../../graphics/tiff
.endif  # TIFF_BUILDLINK3_MK

.include "../../devel/zlib/buildlink3.mk"
.include "../../graphics/jpeg/buildlink3.mk"

BUILDLINK_DEPTH:=       ${BUILDLINK_DEPTH:S/+$//}

The header and footer manipulate BUILDLINK_DEPTH, which is common across all buildlink3.mk files and is used to track at what depth we are including buildlink3.mk files.

The first section controls if the dependency on pkg is added. BUILDLINK_DEPENDS is the global list of packages for which dependencies are added by buildlink3.

The second section advises pkgsrc that the buildlink3.mk file for pkg has been included at some point. BUILDLINK_PACKAGES is the global list of packages for which buildlink3.mk files have been included. It must always be appended to within a buildlink3.mk file.

The third section is protected from multiple inclusion and controls how the dependency on pkg is added. Several important variables are set in the section:

The following variables are all optionally defined within this second section (protected against multiple inclusion) and control which package files are symlinked into ${BUILDLINK_DIR} and how their names are transformed during the symlinking:

The last section includes any buildlink3.mk needed for pkg's library dependencies. Including these buildlink3.mk files means that the headers and libraries for these dependencies are also symlinked into ${BUILDLINK_DIR} whenever the pkg buildlink3.mk file is included.

The situation that requires increasing the dependency listed in BUILDLINK_API_DEPENDS.pkg after a package update is when the API or interface to the header files change.

In this case, BUILDLINK_API_DEPENDS.pkg should be adjusted to require at least the new package version. In some cases, the packages that depend on this new version may need their PKGREVISIONs increased and, if they have buildlink3.mk files, their BUILDLINK_API_DEPENDS.pkg adjusted, too. This is needed so pkgsrc will require the correct package dependency and not settle for an older one when building the source.

BUILDLINK_ABI_DEPENDS.pkg should be increased when the binary interface or sonames (major number of the library version) of any installed shared libraries change. This is needed so that binary packages made using it will require the correct package dependency and not settle for an older one which will not contain the necessary shared libraries.

See Section 19.1.6, “Handling dependencies” for more information about dependencies on other packages, including the BUILDLINK_ABI_DEPENDS and ABI_DEPENDS definitions.

Please take careful consideration before adjusting BUILDLINK_API_DEPENDS.pkg or BUILDLINK_ABI_DEPENDS.pkg as we don't want to cause unneeded package deletions and rebuilds. In many cases, new versions of packages work just fine with older dependencies.

Also it is not needed to set BUILDLINK_ABI_DEPENDS.pkg when it is identical to BUILDLINK_API_DEPENDS.pkg.

The following is the recommended template for builtin.mk files:

.if !defined(IS_BUILTIN.foo)
#
# IS_BUILTIN.foo is set to "yes" or "no" depending on whether "foo"
# genuinely exists in the system or not.
#
IS_BUILTIN.foo?=        no

# BUILTIN_PKG.foo should be set here if "foo" is built-in and its package
# version can be determined.
#
.  if !empty(IS_BUILTIN.foo:M[yY][eE][sS])
BUILTIN_PKG.foo?=       foo-1.0
.  endif
.endif  # IS_BUILTIN.foo

.if !defined(USE_BUILTIN.foo)
USE_BUILTIN.foo?=       ${IS_BUILTIN.foo}
.  if defined(BUILTIN_PKG.foo)
.    for _depend_ in ${BUILDLINK_API_DEPENDS.foo}
.      if !empty(USE_BUILTIN.foo:M[yY][eE][sS])
USE_BUILTIN.foo!=                                                       \
        ${PKG_ADMIN} pmatch '${_depend_}' ${BUILTIN_PKG.foo}            \
        && ${ECHO} "yes" || ${ECHO} "no"
.      endif
.    endfor
.  endif
.endif  # USE_BUILTIN.foo

CHECK_BUILTIN.foo?=     no
.if !empty(CHECK_BUILTIN.foo:M[nN][oO])
#
# Here we place code that depends on whether USE_BUILTIN.foo is set to
# "yes" or "no".
#
.endif  # CHECK_BUILTIN.foo

The first section sets IS_BUILTIN.pkg depending on if pkg really exists in the base system. This should not be a base system software with similar functionality to pkg; it should only be “yes” if the actual package is included as part of the base system. This variable is only used internally within the builtin.mk file.

The second section sets BUILTIN_PKG.pkg to the version of pkg in the base system if it exists (if IS_BUILTIN.pkg is “yes”). This variable is only used internally within the builtin.mk file.

The third section sets USE_BUILTIN.pkg and is required in all builtin.mk files. The code in this section must make the determination whether the built-in software is adequate to satisfy the dependencies listed in BUILDLINK_API_DEPENDS.pkg. This is typically done by comparing BUILTIN_PKG.pkg against each of the dependencies in BUILDLINK_API_DEPENDS.pkg. USE_BUILTIN.pkg must be set to the correct value by the end of the builtin.mk file. Note that USE_BUILTIN.pkg may be “yes” even if IS_BUILTIN.pkg is “no” because we may make the determination that the built-in version of the software is similar enough to be used as a replacement.

The last section is guarded by CHECK_BUILTIN.pkg, and includes code that uses the value of USE_BUILTIN.pkg set in the previous section. This typically includes, e.g., adding additional dependency restrictions and listing additional files to symlink into ${BUILDLINK_DIR} (via BUILDLINK_FILES.pkg).

When building packages, it's possible to choose whether to set a global preference for using either the built-in (native) version or the pkgsrc version of software to satisfy a dependency. This is controlled by setting PREFER_PKGSRC and PREFER_NATIVE. These variables take values of either “yes”, “no”, or a list of packages. PREFER_PKGSRC tells pkgsrc to use the pkgsrc versions of software, while PREFER_NATIVE tells pkgsrc to use the built-in versions. Preferences are determined by the most specific instance of the package in either PREFER_PKGSRC or PREFER_NATIVE. If a package is specified in neither or in both variables, then PREFER_PKGSRC has precedence over PREFER_NATIVE. For example, to require using pkgsrc versions of software for all but the most basic bits on a NetBSD system, you can set:

PREFER_PKGSRC=  yes
PREFER_NATIVE=  getopt skey tcp_wrappers

A package must have a builtin.mk file to be listed in PREFER_NATIVE, otherwise it is simply ignored in that list.

The following variables can be set to request the creation of directories anywhere in the file system:

Creating non-empty files outside the installation prefix is tricky because the PLIST forces all files to be inside it. To overcome this problem, the only solution is to extract the file in the known place (i.e., inside the installation prefix) and copy it to the appropriate location during installation (done by the installation scripts generated by pkginstall). We will call the former the master file in the following paragraphs, which describe the variables that can be used to automatically and consistently handle files outside the installation prefix:

As said before, the PKG_SYSCONFDIR variable specifies where configuration files shall be installed. Its contents are set based upon the following variables:

Based on the above variables, pkginstall determines the value of PKG_SYSCONFDIR, which is the only variable that can be used within a package to refer to its configuration directory. The algorithm used to set its value is basically the following:

It is worth mentioning that ${PKG_SYSCONFDIR} is automatically added to OWN_DIRS. See Section 15.1.1, “Directory manipulation” what this means.

Given that pkgsrc (and users!) expect configuration files to be in a known place, you need to teach each package where it shall install its files. In some cases you will have to patch the package Makefiles to achieve it. If you are lucky, though, it may be as easy as passing an extra flag to the configuration script; this is the case of GNU Autoconf- generated files:

CONFIGURE_ARGS+= --sysconfdir=${PKG_SYSCONFDIR}

Note that this specifies where the package has to look for its configuration files, not where they will be originally installed (although the difference is never explicit, unfortunately).

As said before, pkginstall automatically handles configuration files. This means that the packages themselves must not touch the contents of ${PKG_SYSCONFDIR} directly. Bad news is that many software installation scripts will, out of the box, mess with the contents of that directory. So what is the correct procedure to fix this issue?

You must teach the package (usually by manually patching it) to install any configuration files under the examples hierarchy, share/examples/${PKGBASE}/. This way, the PLIST registers them and the administrator always has the original copies available.

Once the required configuration files are in place (i.e., under the examples hierarchy), the pkginstall framework can use them as master copies during the package installation to update what is in ${PKG_SYSCONFDIR}. To achieve this, the variables CONF_FILES and CONF_FILES_PERMS are used. Check out Section 15.1.2, “File manipulation” for information about their syntax and their purpose. Here is an example, taken from the mail/mutt package:

EGDIR=        ${PREFIX}/share/doc/mutt/samples
CONF_FILES=   ${EGDIR}/Muttrc ${PKG_SYSCONFDIR}/Muttrc

Note that the EGDIR variable is specific to that package and has no meaning outside it.

The automatic copying of config files can be toggled by setting the environment variable PKG_CONFIG prior to package installation.

The automatic copying of config files can be toggled by setting the environment variable PKG_RCD_SCRIPTS prior to package installation. Note that the scripts will be always copied inside the examples hierarchy, ${PREFIX}/share/examples/rc.d/, no matter what the value of this variable is.

The automatic registration of shell interpreters can be disabled by the administrator by setting the PKG_REGISTER_SHELLS environment variable to NO.

The automatic update of fonts databases can be disabled by the administrator by setting the PKG_UPDATE_FONTS_DB environment variable to NO.

In simple cases, MASTER_SITES defines all URLs from where the distfile, whose name is derived from the DISTNAME variable, is fetched. The more complicated cases are described below.

The variable DISTFILES specifies the list of distfiles that have to be fetched. Its value defaults to ${DISTNAME}${EXTRACT_SUFX}, so that most packages don't need to define it at all. EXTRACT_SUFX is .tar.gz by default, but can be changed freely. Note that if your package requires additional distfiles to the default one, you cannot just append the additional filenames using the += operator, but you have write for example:

DISTFILES=      ${DISTNAME}${EXTRACT_SUFX} additional-files.tar.gz

Each distfile is fetched from a list of sites, usually MASTER_SITES. If the package has multiple DISTFILES or multiple PATCHFILES from different sites, you can set SITES.distfile to the list of URLs where the file distfile (including the suffix) can be found.

DISTFILES=      ${DISTNAME}${EXTRACT_SUFX}
DISTFILES+=     foo-file.tar.gz
SITES.foo-file.tar.gz= \
http://www.somewhere.com/somehow/ \
http://www.somewhereelse.com/mirror/somehow/

When actually fetching the distfiles, each item from MASTER_SITES or SITES.* gets the name of each distfile appended to it, without an intermediate slash. Therefore, all site values have to end with a slash or other separator character. This allows for example to set MASTER_SITES to a URL of a CGI script that gets the name of the distfile as a parameter. In this case, the definition would look like:

MASTER_SITES=   http://www.example.com/download.cgi?file=

There are some predefined values for MASTER_SITES, which can be used in packages. The names of the variables should speak for themselves.

${MASTER_SITE_APACHE}
${MASTER_SITE_BACKUP}
${MASTER_SITE_CYGWIN}
${MASTER_SITE_DEBIAN}
${MASTER_SITE_FREEBSD}
${MASTER_SITE_FREEBSD_LOCAL}
${MASTER_SITE_GENTOO}
${MASTER_SITE_GNOME}
${MASTER_SITE_GNU}
${MASTER_SITE_GNUSTEP}
${MASTER_SITE_IFARCHIVE}
${MASTER_SITE_KDE}
${MASTER_SITE_MOZILLA}
${MASTER_SITE_MYSQL}
${MASTER_SITE_OPENOFFICE}
${MASTER_SITE_PERL_CPAN}
${MASTER_SITE_PGSQL}
${MASTER_SITE_R_CRAN}
${MASTER_SITE_SOURCEFORGE}
${MASTER_SITE_SOURCEFORGE_JP}
${MASTER_SITE_SUNSITE}
${MASTER_SITE_SUSE}
${MASTER_SITE_TEX_CTAN}
${MASTER_SITE_XCONTRIB}
${MASTER_SITE_XEMACS}

Some explanations for the less self-explaining ones: MASTER_SITE_BACKUP contains backup sites for packages that are maintained in ftp://ftp.NetBSD.org/pub/NetBSD/packages/distfiles/${DIST_SUBDIR}. MASTER_SITE_LOCAL contains local package source distributions that are maintained in ftp://ftp.NetBSD.org/pub/NetBSD/packages/distfiles/LOCAL_PORTS/.

If you choose one of these predefined sites, you may want to specify a subdirectory of that site. Since these macros may expand to more than one actual site, you must use the following construct to specify a subdirectory:

MASTER_SITES=   ${MASTER_SITE_GNU:=subdirectory/name/}
MASTER_SITES=   ${MASTER_SITE_SOURCEFORGE:=project_name/}

Note the trailing slash after the subdirectory name.

The fetch phase makes sure that all the distfiles exist in a local directory (DISTDIR, which can be set by the pkgsrc user). If the files do not exist, they are fetched using commands of the form

${FETCH_CMD} ${FETCH_BEFORE_ARGS} ${site}${file} ${FETCH_AFTER_ARGS}

where ${site} varies through several possibilities in turn: first, MASTER_SITE_OVERRIDE is tried, then the sites specified in either SITES.file if defined, else MASTER_SITES or PATCH_SITES, as applies, then finally the value of MASTER_SITE_BACKUP. The order of all except the first and the last can be optionally sorted by the user, via setting either MASTER_SORT_RANDOM, and MASTER_SORT_AWK or MASTER_SORT_REGEX.

The distfiles mirror run by the NetBSD Foundation uses the mirror-distfiles target to mirror the distfiles, if they are freely distributable. Packages setting NO_SRC_ON_FTP (usually to “${RESTRICTED}”) will not have their distfiles mirrored.