The primary download location for all pkgsrc files is https://cdn.NetBSD.org/pub/pkgsrc/ or ftp://ftp.NetBSD.org/pub/pkgsrc/ (it points to the same location). There are a number of subdirectories for different purposes, which are described in detail in Appendix D, Directory layout of the pkgsrc FTP server.

The tar archive for the current branch is in the directory current and is called pkgsrc.tar.gz. It is autogenerated weekly.

To save download time we provide bzip2- and xz-compressed archives which are published at pkgsrc.tar.bz2 and pkgsrc.tar.xz respectively.

You can fetch the same files using FTP.

The tar file for the stable branch 2021Q4 is in the directory pkgsrc-2021Q4 and is also called pkgsrc.tar.gz.

To download the latest pkgsrc stable tarball, run:

$ ftp ftp://ftp.NetBSD.org/pub/pkgsrc/pkgsrc-2021Q4/pkgsrc.tar.gz

If you prefer, you can also fetch it using "wget", "curl", or your web browser.

Then, extract it with:

$ tar -xzf pkgsrc.tar.gz -C /usr

This will create the directory pkgsrc/ in /usr/ and all the package source will be stored under /usr/pkgsrc/.

To download pkgsrc-current, run:

$ ftp ftp://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz

To fetch a specific pkgsrc stable branch, run:

$ cd /usr && cvs -q -z2 -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -r pkgsrc-2021Q4 -P pkgsrc

This will create the directory pkgsrc/ in your /usr/ directory and all the package source will be stored under /usr/pkgsrc/.

To fetch the pkgsrc current branch, run:

$ cd /usr && cvs -q -z2 -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -P pkgsrc

Refer to the list of available mirrors to choose a faster CVS mirror, if needed.

If you get error messages from rsh, you need to set CVS_RSH variable. E.g.:

$ cd /usr && env CVS_RSH=ssh cvs -q -z2 -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -P pkgsrc

Refer to documentation on your command shell how to set CVS_RSH=ssh permanently. For Bourne shells, you can set it in your .profile or better globally in /etc/profile:

# set CVS remote shell command
CVS_RSH=ssh
export CVS_RSH

By default, CVS doesn't do things like most people would expect it to do. But there is a way to convince CVS, by creating a file called .cvsrc in your home directory and saving the following lines to it. This file will save you lots of headache and some bug reports, so we strongly recommend it. You can find an explanation of this file in the CVS documentation.

# recommended CVS configuration file from the pkgsrc guide
cvs -q
checkout -P
update -dP
diff -upN
rdiff -u
release -d

Note that by default the distfiles and the binary packages are saved in the pkgsrc tree, so don't forget to rescue them before updating. You can also configure pkgsrc to store distfiles and packages in directories outside the pkgsrc tree by setting the DISTDIR and PACKAGES variables. See Chapter 6, Configuring pkgsrc for the details.

To update pkgsrc from a tar file, download the tar file as explained above. Then, make sure that you have not made any changes to the files in the pkgsrc directory. Remove the pkgsrc directory and extract the new tar file. Done.

To update pkgsrc via CVS, change to the pkgsrc directory and run cvs:

$ cd /usr/pkgsrc && cvs update -dP

If you get error messages from rsh, you need to set CVS_RSH variable as described above. E.g.:

$ cd /usr/pkgsrc && env CVS_RSH=ssh cvs up -dP

To install binary packages, you first need to know from where to get them. The first place where you should look is on the main pkgsrc CDN in the directory /pub/pkgsrc/packages.

This directory contains binary packages for multiple platforms. First, select your operating system. (Ignore the directories with version numbers attached to it, they just exist for legacy reasons.) Then, select your hardware architecture, and in the third step, the OS version and the “version” of pkgsrc.

In this directory, you often find a file called bootstrap.tar.gz which contains the package management tools. If the file is missing, it is likely that your operating system already provides those tools. Download the file and extract it in the / directory. It will create the directories /usr/pkg (containing the tools for managing binary packages and the database of installed packages).

In the directory from the last section, there is a subdirectory called All/, which contains all the binary packages that are available for the platform, excluding those that may not be distributed via HTTP or FTP.

To install packages directly from an FTP or HTTP server, run the following commands in a Bourne-compatible shell (be sure to su to root first):

# PATH="/usr/pkg/sbin:/usr/pkg/bin:$PATH"
# PKG_PATH="https://cdn.NetBSD.org/pub/pkgsrc/packages"
# PKG_PATH="$PKG_PATH/OPSYS/ARCH/VERSIONS/All/"
# export PATH PKG_PATH
# pkg_add pkgin

Instead of URLs, you can also use local paths, for example if you are installing from a set of CDROMs, DVDs or an NFS-mounted repository. If you want to install packages from multiple sources, you can separate them by a semicolon in PKG_PATH.

After these preparations, installing a package is very easy:

# pkgin search nginx
nginx-1.19.6         Lightweight HTTP server and mail proxy server
nginx-1.18.0nb8      Lightweight HTTP server and mail proxy server
# pkgin install zsh nginx-1.19.6 vim

Note that pkgin is a user-friendly frontend to the pkg_* tools.

Any prerequisite packages needed to run the package in question will be installed, too, assuming they are present in the repository.

After you've installed packages, be sure to have /usr/pkg/bin and /usr/pkg/sbin in your PATH so you can actually start the just installed program.

To update binary packages, it is recommended that you use pkgin upgrade. This will compare the remote package repository to your locally installed packages and safely replace any older packages.

Note that pkgsrc is released as quarterly branches. If you are updating to a newer quarterly branch of pkgsrc, you may need to adjust the repository in /usr/pkg/etc/pkgin/repositories.conf.

To deinstall a package, it does not matter whether it was installed from source code or from a binary package. Neither the pkgin or the pkg_delete command need to know.

To delete a package, you can just run pkgin remove package-name. The package name can be given with or without version number.

The pkg_info shows information about installed packages or binary package files. As with other management tools, it works with packages installed from source or binaries.

The pkgsrc Security Team and Packages Groups maintain a list of known vulnerabilities to packages which are (or have been) included in pkgsrc. The list is available from the NetBSD CDN at https://cdn.NetBSD.org/pub/NetBSD/packages/vulns/pkg-vulnerabilities.

Please note that not every "vulnerability" with a CVE assignment is exploitable in every configuration. Some bugs are marked as active simply because an fix was not marked as such. Operating system specific hardening and mitigation features may also reduce the impact of bugs.

Through pkg_admin fetch-pkg-vulnerabilities, this list can be downloaded automatically, and a security audit of all packages installed on a system can take place.

There are two components to auditing. The first step, pkg_admin fetch-pkg-vulnerabilities, is for downloading the list of vulnerabilities from the NetBSD FTP site. The second step, pkg_admin audit, checks to see if any of your installed packages are vulnerable. If a package is vulnerable, you will see output similar to the following:

Package samba-2.0.9 has a local-root-shell vulnerability, see
    https://www.samba.org/samba/whatsnew/macroexploit.html

You may wish to have the vulnerabilities file downloaded daily so that it remains current. This may be done by adding an appropriate entry to the root users crontab(5) entry. For example the entry

# Download vulnerabilities file
0 3 * * * /usr/pkg/sbin/pkg_admin fetch-pkg-vulnerabilities >/dev/null 2>&1
# Audit the installed packages and email results to root
9 3 * * * /usr/pkg/sbin/pkg_admin audit |mail -s "Installed package audit result" \
	    root >/dev/null 2>&1
      

will update the vulnerability list every day at 3AM, followed by an audit at 3:09AM. The result of the audit are then emailed to root. On NetBSD this may be accomplished instead by adding the following line to /etc/daily.conf:

fetch_pkg_vulnerabilities=YES
      

to fetch the vulnerability list from the daily security script. The system is set to audit the packages by default but can be set explicitly, if desired (not required), by adding the following line to /etc/security.conf:

check_pkg_vulnerabilities=YES
      

see daily.conf(5) and security.conf(5) for more details.

Install pkgtools/lintpkgsrc and run lintpkgsrc with the “-i” argument to check if any packages are stale, e.g.

% lintpkgsrc -i
...
Version mismatch: 'tcsh' 6.09.00 vs 6.10.00
    

The pkg_admin executes various administrative functions on the package system.

To build packages from source, you need a working C compiler. On NetBSD, you need to install the “comp” and the “text” distribution sets. If you want to build X11-related packages, the “xbase” and “xcomp” distribution sets are required, too.

The first step for building a package is downloading the distfiles (i.e. the unmodified source). If they have not yet been downloaded, pkgsrc will fetch them automatically.

If you have all files that you need in the distfiles directory, you don't need to connect. If the distfiles are on CD-ROM, you can mount the CD-ROM on /cdrom and add:

DISTDIR=/cdrom/pkgsrc/distfiles

to your mk.conf.

By default a list of distribution sites will be randomly intermixed to prevent huge load on servers which holding popular packages (for example, SourceForge.net mirrors). Thus, every time when you need to fetch yet another distfile all the mirrors will be tried in new (random) order. You can turn this feature off by setting MASTER_SORT_RANDOM=NO (for PKG_DEVELOPERs it's already disabled).

You can overwrite some of the major distribution sites to fit to sites that are close to your own. By setting one or two variables you can modify the order in which the master sites are accessed. MASTER_SORT contains a whitespace delimited list of domain suffixes. MASTER_SORT_REGEX is even more flexible, it contains a whitespace delimited list of regular expressions. It has higher priority than MASTER_SORT. Have a look at pkgsrc/mk/defaults/mk.conf to find some examples. This may save some of your bandwidth and time.

You can change these settings either in your shell's environment, or, if you want to keep the settings, by editing the mk.conf file, and adding the definitions there.

If a package depends on many other packages (such as meta-pkgs/kde4), the build process may alternate between periods of downloading source, and compiling. To ensure you have all the source downloaded initially you can run the command:

% make fetch-list | sh

which will output and run a set of shell commands to fetch the necessary files into the distfiles directory. You can also choose to download the files manually.

Once the software has downloaded, any patches will be applied, then it will be compiled for you. This may take some time depending on your computer, and how many other packages the software depends on and their compile time.

For example, type

% cd misc/figlet
% make
    

at the shell prompt to build the various components of the package.

The next stage is to actually install the newly compiled program onto your system. Do this by entering:

% make install
    

while you are still in the directory for whatever package you are installing.

Installing the package on your system may require you to be root. However, pkgsrc has a just-in-time-su feature, which allows you to only become root for the actual installation step.

That's it, the software should now be installed and setup for use. You can now enter:

% make clean
    

to remove the compiled files in the work directory, as you shouldn't need them any more. If other packages were also added to your system (dependencies) to allow your program to compile, you can tidy these up also with the command:

% make clean-depends
    

Taking the figlet utility as an example, we can install it on our system by building as shown in Appendix C, Build logs.

The program is installed under the default root of the packages tree - /usr/pkg. Should this not conform to your tastes, set the LOCALBASE variable in your environment, and it will use that value as the root of your packages tree. So, to use /usr/local, set LOCALBASE=/usr/local in your environment. Please note that you should use a directory which is dedicated to packages and not shared with other programs (i.e., do not try and use LOCALBASE=/usr). Also, you should not try to add any of your own files or directories (such as src/, obj/, or pkgsrc/) below the LOCALBASE tree. This is to prevent possible conflicts between programs and other files installed by the package system and whatever else may have been installed there.

Some packages look in mk.conf to alter some configuration options at build time. Have a look at pkgsrc/mk/defaults/mk.conf to get an overview of what will be set there by default. Environment variables such as LOCALBASE can be set in mk.conf to save having to remember to set them each time you want to use pkgsrc.

Occasionally, people want to “look under the covers” to see what is going on when a package is building or being installed. This may be for debugging purposes, or out of simple curiosity. A number of utility values have been added to help with this.

If you want to install a binary package that you've either created yourself (see next section), that you put into pkgsrc/packages manually or that is located on a remote FTP server, you can use the "bin-install" target. This target will install a binary package - if available - via pkg_add(1), else do a make package. The list of remote FTP sites searched is kept in the variable BINPKG_SITES, which defaults to ftp.NetBSD.org. Any flags that should be added to pkg_add(1) can be put into BIN_INSTALL_FLAGS. See pkgsrc/mk/defaults/mk.conf for more details.

A final word of warning: If you set up a system that has a non-standard setting for LOCALBASE, be sure to set that before any packages are installed, as you cannot use several directories for the same purpose. Doing so will result in pkgsrc not being able to properly detect your installed packages, and fail miserably. Note also that precompiled binary packages are usually built with the default LOCALBASE of /usr/pkg, and that you should not install any if you use a non-standard LOCALBASE.

By default, pkgsrc will use GCC to build packages. This may be overridden by setting the following variables in /etc/mk.conf:

If you wish to set the CFLAGS variable, please make sure to use the += operator instead of the = operator:

CFLAGS+=        -your -flags

Using CFLAGS= (i.e. without the “+”) may lead to problems with packages that need to add their own flags. You may want to take a look at the devel/cpuflags package if you're interested in optimization specifically for the current CPU.

If you want to pass flags to the linker, both in the configure step and the build step, you can do this in two ways. Either set LDFLAGS or LIBS. The difference between the two is that LIBS will be appended to the command line, while LDFLAGS come earlier. LDFLAGS is pre-loaded with rpath settings for ELF machines depending on the setting of USE_IMAKE or the inclusion of mk/x11.buildlink3.mk. As with CFLAGS, if you do not wish to override these settings, use the += operator:

LDFLAGS+=        -your -linkerflags

To simplify configuration, we provide the helper script mk/pbulk/pbulk.sh.

In order to use it, prepare a clear system (real one, chroot environment, jail, zone, virtual machine). Configure network access to fetch distribution files. Create a user with name "pbulk".

Fetch and extract pkgsrc. Use a command like one of these:

# (cd /usr && ftp -o - https://cdn.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)
# (cd /usr && fetch -o - https://cdn.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)
# (cd /usr && cvs -Q -z3 -d anoncvs@anoncvs.NetBSD.org:/cvsroot get -P pkgsrc)

Or any other way that fits (e.g., curl, wget).

Deploy and configure pbulk tools, e.g.:

# sh pbulk.sh -n # use native make, no bootstrap kit needed (for use on NetBSD)
# sh pbulk.sh -n -c mk.conf.frag # native, apply settings from given mk.conf fragment
# sh pbulk.sh -nlc mk.conf.frag # native, apply settings, configure for limited build

PKG_DEVELOPER=          yes     # perform more checks
X11_TYPE=               modular # use pkgsrc X11
SKIP_LICENSE_CHECK=     yes     # accept all licences (useful
                                # when building all packages)

If configured for limited list, replace the list in /usr/pbulk/etc/pbulk.list with your list of packages, one per line without empty lines or comments. E.g.:

www/firefox
mail/thunderbird
misc/libreoffice4

At this point you can also review configuration in /usr/pbulk/etc and make final amendments, if wanted.

Start it:

# /usr/pbulk/bin/bulkbuild

After it finishes, you'll have /mnt filled with distribution files, binary packages, and reports, plain text summary in /mnt/bulklog/meta/report.txt

Add the following line to mk.conf.

GNU_CONFIGURE_STRICT=   yes

When a package fails this additional check, the most common cause is that the configure option was valid for an older version of the package but does not apply anymore. In that case, just remove it.

The job of a compiler is not restricted to producing executable code, most compilers also detect typical programming mistakes. The pkgsrc compiler wrappers make it easy to force compiler options when the package is built. This can be used to find typical bugs across all packages that are in pkgsrc. By reporting these bugs upstream, the packages will be more reliable with the next updates.

Add some of the following lines to mk.conf:

CFLAGS+=        -Werror=char-subscripts
CFLAGS+=        -Werror=implicit-function-declaration

When a package fails to build using these stricter compiler options, document the circumstances in which the compiler produced the error message. This includes:

If a package produces these error messages, but the package is fine, record this in your local mk.conf, like this, to skip this check in the next builds:

.if ${PKGPATH} == category/package
# Version ${VERSION} failed on ${MACHINE_PLATFORM}:
# error message
# code
# Reason why the code does not need to be fixed.
BUILDLINK_TRANSFORM+=   rm:-Werror=char-subscripts
.endif

If the error messages from the compiler are valid and the code needs to be fixed, prepare a local patch (see LOCALPATCHES) and report the bug to the upstream authors of the package, providing them with the information you collected above.

Patches that are not essential for the package to work should only be reported upstream but not committed to pkgsrc, to make future updates easier.

When adding custom compiler flags via CFLAGS, these apply to all phases of the package build process. Especially in the configure phase, adding -Werror leads to wrong decisions. The GNU configure scripts feed many small test programs to the C compiler to see whether certain headers are available, functions are defined in a library and programs can be run. In many cases these programs would not survive a strict compiler run with -Wall -Wextra -Werror.

The pkgsrc infrastructure is flexible enough to support compiler options being added between the configure and build phases. It's a little more complicated than the other examples in this section but still easy enough.

The basic idea is to use the pkgsrc compiler wrapper to inject the desired compiler options. The compiler wrapper's original task is to hide unwanted directories of include files and to normalize compiler options. It does this by wrapping the compiler command and rewriting the command line. To see this in action, run bmake patch in a package directory and examine the work/.cwrappers/config directory. It contains individual configurations for the C compiler and the related tools. The plan is to find a hook between the configure and build phases, and to modify these configuration files at that point.

To find this hook, have a look at mk/build/build.mk, which contains among others the pre-build-checks-hook. The word checks doesn't quite fit, but the pre-build-hook sounds good enough.

The code to be included in mk.conf is:

# Just a few example options.
BUILD_ONLY_CFLAGS=      -Wall -Werror -O2 -DTEMPDIR='"/tmp"'

.if ${BUILD_ONLY_CFLAGS:U:M*}
pre-build-checks-hook: add-build-only-cflags

add-build-only-cflags: .PHONY
        ${RUN} cd ${CWRAPPERS_CONFIG_DIR};      \
        ${TEST} ! -f ${.TARGET} || exit 0;      \
        for flag in ${BUILD_ONLY_CFLAGS}; do    \
                ${ECHO} "append=$$flag" >> cc;  \
        done;                                   \
        > ${.TARGET}
.endif

(When editing the mk.conf, make sure that the commands of the add-build-only-cflags target are indented with a tab, not with spaces.)

The condition in the .if statement contains the :U modifier to prevent parse errors if the variable should be undefined, possibly because it is only defined for a subset of the packages. The :M* modifier ensures that there is at least one compiler option, to prevent a syntax error in the shell parser.

The code around the ${.TARGET} variable ensures that the additional compiler options are only appended once, even if bmake build is run multiple times. To do this, it creates a marker file.

To verify that this setup works, run bmake configure in a package directory. Up to now, everything works as usual. Examine the directory work/.cwrappers/config to see that the compiler options from BUILD_ONLY_CFLAGS are not yet added to the file cc. Examine the tail of the work/.work.log file to see that the normal compiler options are used.

Now run bmake build. This will append the options to the file cc and will create the marker file in the same directory. After that, the build starts as usual, but with the added compiler options. Examine the tail of the file work/.work.log to see that the lines starting with [*] don't contain the compiler options, but the corresponding lines starting with <.> do end with these options.

Building packages using this setup variant and fixing the resulting bugs is the same as in Section 8.4.2, “Detect classes of bugs by forcing compiler warnings”.

Some directories like PREFIX, VARBASE, PKG_SYSCONFDIR, PKGMANDIR, PKG_INFODIR can be configured in pkgsrc. Set these to arbitrary paths during bootstrap or afterwards in mk.conf.

PREFIX=         /a-random-uuid
PKG_SYSCONFDIR= /a-random-uuid
VARBASE=        /a-random-uuid
PKGMANDIR=	a-random-uuid
PKG_INFODIR=	a-random-uuid

When building a package, warnings are typically ignored since they just flow by and do not cause the build to fail immediately. To find these warnings, redefine them to errors in mk.conf.

DELAYED_WARNING_MSG=    ${DELAYED_ERROR_MSG} "(was warning)"
WARNING_MSG=            ${FAIL_MSG} "(was warning)"

(There are many more classes of warnings in pkgsrc, and most of them can be redefined with a simple definition like above.

If a package suggests to add USE_TOOLS+=perl to the package Makefile, research whether the package actually needs Perl. If it does, add USE_TOOLS+=perl to the package Makefile, and if it doesn't, add TOOLS_BROKEN+=perl.

Using pkglint as part of the regular build process is mostly a waste of time. If you want to fix some of the warnings, just run pkglint recursively on the whole pkgsrc tree. This will take a few minutes (up to 10), which is much faster than a complete bulk build.

To ensure that the binary packages don't contain references to the build directory, there is already CHECK_WRKREF. If that variable includes the item extra, it is possible to define additional patterns that must not appear in any installed file. This is specified in mk.conf.

CHECK_WRKREF=                   extra
CHECK_WRKREF_EXTRA_DIRS+=       /usr/local
CHECK_WRKREF_EXTRA_DIRS+=       /usr/pkg
CHECK_WRKREF_EXTRA_DIRS+=	@[A-Z][A-Z]*@

The above patterns will probably generate many false positives, therefore the results need to be taken with a grain of salt.

To run the test suites that come with each package, add this line to mk.conf.

PKGSRC_RUN_TEST=        yes

Be prepared that even the most basic packages fail this test. When doing a bulk build with this, it will often abort in the early phase where the packages are scanned for their dependencies since there are cyclic dependencies. There is still a lot to do in this area.

To catch typos in the shell snippets from the Makefile fragments, add the -u flag to most of the commands by adding this line to mk.conf.

RUN=    @set -eu;

After that, ensure that none of the bulk build log files (even those for successfully built packages) contains the string parameter not set or whatever error message the command sh -ceu '$undefined' outputs.

See mk/misc/common.mk for the existing definition.

The build logs of a package are often quite long. This allows error messages or other interesting details to hide between the noise. To make the actual error message stand out more, add these lines to mk.conf.

GNU_CONFIGURE_QUIET=    yes
MAKE_FLAGS+=            -s

The -s option works for both GNU Make and BSD Make. On exotic platforms with their own make, it may be a little different.

Complete documentation for cdpack is found in the cdpack(1) man page. The following short example assumes that the binary packages are left in /usr/pkgsrc/packages/All and that sufficient disk space exists in /u2 to hold the ISO 9660 images.

# mkdir /u2/images
# pkg_add /usr/pkgsrc/packages/All/cdpack
# cdpack /usr/pkgsrc/packages/All /u2/images
      

If you wish to include a common set of files (COPYRIGHT, README, etc.) on each CD in the collection, then you need to create a directory which contains these files, e.g.:

# mkdir /tmp/common
# echo "This is a README" > /tmp/common/README
# echo "Another file" > /tmp/common/COPYING
# mkdir /tmp/common/bin
# echo "#!/bin/sh" > /tmp/common/bin/myscript
# echo "echo Hello world" >> /tmp/common/bin/myscript
# chmod 755 /tmp/common/bin/myscript
      

Now create the images:

# cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images

Each image will contain README, COPYING, and bin/myscript in their root directories.

The patch-* files should be in diff -bu format, and apply without a fuzz to avoid problems. (To force patches to apply with fuzz you can set PATCH_FUZZ_FACTOR=-F2). Furthermore, each patch should contain only changes for a single file, and no file should be patched by more than one patch file. This helps to keep future modifications simple.

Each patch file is structured as follows: In the first line, there is the RCS Id of the patch itself. The second line should be empty for aesthetic reasons. After that, there should be a comment for each change that the patch does. There are a number of standard cases:

The patch should be commented so that any developer who knows the code of the application can make some use of the patch. Special care should be taken for the upstream developers, since we generally want that they accept our patches, so we have less work in the future.

One important thing to mention is to pay attention that no RCS IDs get stored in the patch files, as these will cause problems when later checked into the NetBSD CVS tree. Use the pkgdiff command from the pkgtools/pkgdiff package to avoid these problems.

For even more automation, we recommend using mkpatches from the same package to make a whole set of patches. You just have to backup files before you edit them to filename.orig, e.g. with cp -p filename filename.orig or, easier, by using pkgvi again from the same package. If you upgrade a package this way, you can easily compare the new set of patches with the previously existing one with patchdiff. The files in patches are replaced by new files, so carefully check if you want to take all the changes.

When you have finished a package, remember to generate the checksums for the patch files by using the make makepatchsum command, see Section 12.2, “distinfo”.

When adding a patch that corrects a problem in the distfile (rather than e.g. enforcing pkgsrc's view of where man pages should go), send the patch as a bug report to the maintainer. This benefits non-pkgsrc users of the package, and usually makes it possible to remove the patch in future version.

The file names of the patch files are usually of the form patch-path_to_file__with__underscores.c. Many packages still use the previous convention patch-[a-z][a-z], but new patches should be of the form containing the filename. mkpatches included in pkgtools/pkgdiff takes care of the name automatically.

If you want to share patches between multiple packages in pkgsrc, e.g. because they use the same distfiles, set PATCHDIR to the path where the patch files can be found, e.g.:

PATCHDIR=       ../../editors/xemacs/patches

Patch files that are distributed by the author or other maintainers can be listed in PATCHFILES.

If it is desired to store any patches that should not be committed into pkgsrc, they can be kept outside the pkgsrc tree in the $LOCALPATCHES directory. The directory tree there is expected to have the same “category/package” structure as pkgsrc, and patches are expected to be stored inside these dirs (also known as $LOCALPATCHES/$PKGPATH). For example, if you want to keep a private patch for pkgsrc/graphics/png, keep it in $LOCALPATCHES/graphics/png/mypatch. All files in the named directory are expected to be patch files, and they are applied after pkgsrc patches are applied.

When fixing a portability issue in the code do not use preprocessor magic to check for the current operating system nor platform. Doing so hurts portability to other platforms because the OS-specific details are not abstracted appropriately.

The general rule to follow is: instead of checking for the operating system the application is being built on, check for the specific features you need. For example, instead of assuming that kqueue is available under NetBSD and using the __NetBSD__ macro to conditionalize kqueue support, add a check that detects kqueue itself — yes, this generally involves patching the configure script. There is absolutely nothing that prevents some OSes from adopting interfaces from other OSes (e.g. Linux implementing kqueue), something that the above checks cannot take into account.

Of course, checking for features generally involves more work on the developer's side, but the resulting changes are cleaner and there are chances they will work on many other platforms. Not to mention that there are higher chances of being later integrated into the mainstream sources. Remember: It doesn't work unless it is right!

Some typical examples:

case ${target_os} in
netbsd*) have_kvm=yes ;;
*)       have_kvm=no  ;;
esac

AC_CHECK_LIB(kvm, kvm_open, have_kvm=yes, have_kvm=no)

#if defined(__NetBSD__)
#  include <sys/event.h>
#endif

#if defined(HAVE_SYS_EVENT_H)
#  include <sys/event.h>
#endif

int
monitor_file(...)
{
#if defined(__NetBSD__)
        int fd = kqueue();
        ...
#else
        ...
#endif
}

int
monitor_file(...)
{
#if defined(HAVE_KQUEUE)
        int fd = kqueue();
        ...
#else
        ...
#endif
}

Always, always, always feed back any portability fixes or improvements you do to a package to the mainstream developers. This is the only way to get their attention on portability issues and to ensure that future versions can be built out-of-the box on NetBSD. Furthermore, any user that gets newer distfiles will get the fixes straight from the packaged code.

This generally involves cleaning up the patches (because sometimes the patches that are added to pkgsrc are quick hacks), filing bug reports in the appropriate trackers for the projects and working with the mainstream authors to accept your changes. It is extremely important that you do it so that the packages in pkgsrc are kept simple and thus further changes can be done without much hassle.

When you have done this, please add a URL to the upstream bug report to the patch comment.

Support the idea of free software!

In simple cases, MASTER_SITES defines all URLs from where the distfile, whose name is derived from the DISTNAME variable, is fetched. The more complicated cases are described below.

The variable DISTFILES specifies the list of distfiles that have to be fetched. Its value defaults to ${DEFAULT_DISTFILES} and its value is ${DISTNAME}${EXTRACT_SUFX}, so that most packages don't need to define it at all. EXTRACT_SUFX is .tar.gz by default, but can be changed freely. Note that if your package requires additional distfiles to the default one, you cannot just append the additional filenames using the += operator, but you have write for example:

DISTFILES=      ${DEFAULT_DISTFILES} additional-files.tar.gz

Each distfile is fetched from a list of sites, usually MASTER_SITES. If the package has multiple DISTFILES or multiple PATCHFILES from different sites, you can set SITES.distfile to the list of URLs where the file distfile (including the suffix) can be found.

DISTFILES=      ${DISTNAME}${EXTRACT_SUFX}
DISTFILES+=     foo-file.tar.gz
SITES.foo-file.tar.gz= \
https://www.somewhere.com/somehow/ \
https://www.somewhereelse.com/mirror/somehow/

When actually fetching the distfiles, each item from MASTER_SITES or SITES.* gets the name of each distfile appended to it, without an intermediate slash. Therefore, all site values have to end with a slash or other separator character. This allows for example to set MASTER_SITES to a URL of a CGI script that gets the name of the distfile as a parameter. In this case, the definition would look like:

MASTER_SITES=   https://www.example.com/download.cgi?file=

The exception to this rule are URLs starting with a dash. In that case the URL is taken as is, fetched and the result stored under the name of the distfile. You can use this style for the case when the download URL style does not match the above common case. For example, if permanent download URL is a redirector to the real download URL, or the download file name is offered by an HTTP Content-Disposition header. In the following example, foo-1.0.0.tar.gz will be created instead of the default v1.0.0.tar.gz.

DISTNAME=       foo-1.0.0
MASTER_SITES=   -https://www.example.com/archive/v1.0.0.tar.gz

There are some predefined values for MASTER_SITES, which can be used in packages. The names of the variables should speak for themselves.

Some explanations for the less self-explaining ones: MASTER_SITE_BACKUP contains backup sites for packages that are maintained in ftp://ftp.NetBSD.org/pub/pkgsrc/distfiles/${DIST_SUBDIR}. MASTER_SITE_LOCAL contains local package source distributions that are maintained in ftp://ftp.NetBSD.org/pub/pkgsrc/distfiles/LOCAL_PORTS/.

If you choose one of these predefined sites, you may want to specify a subdirectory of that site. Since these macros may expand to more than one actual site, you must use the following construct to specify a subdirectory:

MASTER_SITES=   ${MASTER_SITE_GNU:=subdirectory/name/}
MASTER_SITES=   ${MASTER_SITE_SOURCEFORGE:=project_name/}

Note the trailing slash after the subdirectory name.

The fetch phase makes sure that all the distfiles exist in a local directory (DISTDIR, which can be set by the pkgsrc user). If the files do not exist, they are fetched using commands of the form

${FETCH_CMD} ${FETCH_BEFORE_ARGS} ${site}${file} ${FETCH_AFTER_ARGS}

where ${site} varies through several possibilities in turn: first, MASTER_SITE_OVERRIDE is tried, then the sites specified in either SITES.file if defined, else MASTER_SITES or PATCH_SITES, as applies, then finally the value of MASTER_SITE_BACKUP. The order of all except the first and the last can be optionally sorted by the user, via setting either MASTER_SORT_RANDOM, and MASTER_SORT_AWK or MASTER_SORT_REGEX.

The specific command and arguments used depend on the FETCH_USING parameter. The example above is for FETCH_USING=custom.

The distfiles mirror run by the NetBSD Foundation uses the mirror-distfiles target to mirror the distfiles, if they are freely distributable. Packages setting NO_SRC_ON_FTP (usually to “${RESTRICTED}”) will not have their distfiles mirrored.

Python modules and programs packages are easily created using a set of predefined variables.

If some Python versions are not supported by the software, set the PYTHON_VERSIONS_INCOMPATIBLE variable to the Python versions that are not supported, e.g.

PYTHON_VERSIONS_INCOMPATIBLE=       27

If the packaged software is a Python module, include one of ../../lang/python/egg.mk or ../../lang/python/extension.mk.

Most Python packages use setuptools (“eggs”). If the packaged software is using setuptools, you only need to include “../../lang/python/egg.mk”.

Either way, the package directory should be called “py-software” and PKGNAME should be set to “${PYPKGPREFIX}-${DISTNAME}”, e.g.

DISTNAME=   foopymodule-1.2.10
PKGNAME=    ${PYPKGPREFIX}-${DISTNAME}

If it is an application, include “../../lang/python/application.mk”. In order to correctly set the path to the Python interpreter, use the REPLACE_PYTHON variable and set it to the list of files (paths relative to WRKSRC) that must be corrected. For example:

REPLACE_PYTHON=   *.py

Some Python modules have separate distributions for Python-2.x and Python-3.x support. In pkgsrc this is handled by the versioned_dependencies.mk file. Set PYTHON_VERSIONED_DEPENDENCIES to the list of packages that should be depended upon and include “../../lang/python/versioned_dependencies.mk”, then the pkgsrc infrastructure will depend on the appropriate package version. For example:

PYTHON_VERSIONED_DEPENDENCIES=dialog

Look inside versioned_dependencies.mk for a list of supported packages.

Simple R packages from CRAN are handled automatically by R2pkg, which is available in pkgtools/R2pkg. Individual packages (and optionally their dependencies) may be created and updated. R packages generally follow the same form, and most of the relevant information needed is contained in a DESCRIPTION file as part of each R package on CRAN. Consequently, R2pkg downloads that information and creates or updates a package in the canonical form. The resulting package should be reviewed for correctness.

TeXlive packages from CTAN are handled automatically by texlive2pkg, which is available in pkgtools/texlive2pkg.

If the TeXlive package name is not known, it may be useful to search CTAN. A “Contained in” field on the package page typically identifies the basename of the package file in the TeXlive archive.

If the TeXlive package name is known, download the files from the TeXlive archive. For package foo, you will need to download foo.tar.xz. Most TeXlive packages also have associated documentation packages, so download foo.doc.tar.xz at the same time. These files should be placed in the appropriate category directory, which is often but not always print. Then run the following command in the category directory.

texlive2pkg foo.tar.xz foo.doc.tar.xz

This will create two packages, tex-foo and tex-foo-doc. Be sure to check that both packages are correct.

Finally, CTAN currently does not include version information in package filenames and changes their contents periodically when updates occur. Consequently, pkgsrc avoids downloading distfiles directly from CTAN and instead relies on the pkgsrc archives. For each new or updated TeXlive package, e.g., the main one and the corresponding documentation, upload the distfiles with the following command in each package directory.

make upload-distfiles

When adding a string that possibly contains whitespace or quotes to a list (example 1), it must be quoted using the :Q modifier.

When adding another list to a list (example 2), it must not be quoted, since its elements are already quoted.

STRING=         foo * bar `date`
LIST=           # empty
ANOTHER_LIST=   a=b c=d

LIST+=          ${STRING:Q}       # 1
LIST+=          ${ANOTHER_LIST}   # 2

Echoing a string containing special characters needs special work.

STRING=         foo bar <    > * `date` $$HOME ' "
EXAMPLE_ENV=    string=${STRING:Q} x=multiple\ quoted\ words

all:
        echo ${STRING}                  # 1
        echo ${STRING:Q}                # 2
        printf '%s\n' ${STRING:Q}''     # 3
        env ${EXAMPLE_ENV} sh -c 'echo "$$string"; echo "$$x"'   # 4

Example 1 leads to a syntax error in the shell, as the characters are just copied.

Example 2 quotes the string so that the shell interprets it correctly. But the echo command may additionally interpret strings with a leading dash or those containing backslashes.

Example 3 can handle arbitrary strings, since printf(1) only interprets the format string, but not the next argument. The trailing single quotes handle the case when the string is empty. In that case, the :Q modifier would result in an empty string too, which would then be skipped by the shell. For printf(1) this doesn't make a difference, but other programs may care.

In example 4, the EXAMPLE_ENV does not need to be quoted because the quoting has already been done when adding elements to the list.

When passing CFLAGS or similar variables to a GNU-style configure script (especially those that call other configure scripts), it must not have leading or trailing whitespace, since otherwise the configure script gets confused. To trim leading and trailing whitespace, use the :M modifier, as in the following example:

CPPFLAGS=               # empty
CPPFLAGS+=              -Wundef -DPREFIX=\"${PREFIX}\"
CPPFLAGS+=              ${MY_CPPFLAGS}

CONFIGURE_ARGS+=        CPPFLAGS=${CPPFLAGS:M*:Q}

all:
        echo x${CPPFLAGS:Q}x            # leading and trailing whitespace
        echo x${CONFIGURE_ARGS:Q}x      # properly trimmed

In this example, CPPFLAGS has both leading and trailing whitespace because the += operator always adds a space.

When a possibly empty variable is used in a shell program, it may lead to a syntax error.

EGFILES=        # empty

install-examples:   # produces a syntax error in the shell
        for egfile in ${EGFILES}; do            \
                echo "Installing $$egfile";     \
        done

The shell only sees the text for egfile in ; do, since ${EGFILES} is replaced with an empty string by make(1). To fix this syntax error, use one of the snippets below.

EMPTY=          # empty

install-examples:
        for egfile in ${EGFILES} ""; do         \
                [ -n "$$egfile" ] || continue;  \
                echo "Installing $$egfile";     \
        done

In this case, an empty string is appended to the iteration list (to prevent the syntax error) and filtered out later.

EGFILES=        # empty

install-examples:
.for egfile in ${EGFILES}
        echo "Installing ${egfile}"
.endfor

If one of the filenames contains special characters, it should be enclosed in single or double quotes.

To have a shell command test whether a make variable is empty, use the following code: ${TEST} -z ${POSSIBLY_EMPTY:Q}"".

The following real-life example buildlink3.mk is taken from pkgsrc/graphics/tiff:

# $NetBSD: buildlink3.mk,v 1.16 2009/03/20 19:24:45 joerg Exp $

BUILDLINK_TREE+=        tiff

.if !defined(TIFF_BUILDLINK3_MK)
TIFF_BUILDLINK3_MK:=

BUILDLINK_API_DEPENDS.tiff+=    tiff>=3.6.1
BUILDLINK_ABI_DEPENDS.tiff+=    tiff>=3.7.2nb1
BUILDLINK_PKGSRCDIR.tiff?=      ../../graphics/tiff

.include "../../devel/zlib/buildlink3.mk"
.include "../../graphics/jpeg/buildlink3.mk"
.endif # TIFF_BUILDLINK3_MK

BUILDLINK_TREE+=        -tiff

The header and footer manipulate BUILDLINK_TREE, which is common across all buildlink3.mk files and is used to track the dependency tree.

The main section is protected from multiple inclusion and controls how the dependency on pkg is added. Several important variables are set in the section:

The following variables are all optionally defined within this second section (protected against multiple inclusion) and control which package files are symlinked into ${BUILDLINK_DIR} and how their names are transformed during the symlinking:

This section can additionally include any buildlink3.mk needed for pkg's library dependencies. Including these buildlink3.mk files means that the headers and libraries for these dependencies are also symlinked into ${BUILDLINK_DIR} whenever the pkg buildlink3.mk file is included. Dependencies are only added for directly include buildlink3.mk files.

When providing a buildlink3.mk and including other buildlink3.mk files in it, please only add necessary ones, i.e., those whose libraries or header files are automatically exposed when the package is use.

In particular, if only an executable (bin/foo) is linked against a library, that library does not need to be propagated in the buildlink3.mk file.

The following steps should help you decide if a buildlink3.mk file needs to be included:

Both variables set lower bounds for a version of this package. The two variables differ in that one describes source compatibility (API) and the other binary compatibility (ABI). The difference is that a change in the API breaks compilation of programs while changes in the ABI stop compiled programs from running.

The BUILDLINK_API_DEPENDS.pkg variable in a buildlink3.mk should be changed very rarely. (One possible scenario: If all packages using this package need a higher version than defined in the buildlink3.mk, BUILDLINK_API_DEPENDS.pkg could be updated to that higher version.)

On the other hand, changes to BUILDLINK_ABI_DEPENDS.pkg are more common. The variable will need to be updated every time the major version of one of its shared libraries is changed, or any other change where a binary built against the previous version of the package will not run against the new version any longer.

In such a case, the package's BUILDLINK_ABI_DEPENDS.pkg must be increased to require the new package version. Then the PKGREVISION of all packages foo that depend on this package need to be increased, and if they have buildlink3.mk files, BUILDLINK_ABI_DEPENDS.foo in their buildlink3.mk files must be increased to the new version as well. This is required so that a package will pull in the versions of the packages that use the new ABI and that the packages' PKGREVISIONs uniquely identify the packages built against the new ABI. The pkgtools/revbump package can help with these updates.

See Section 21.1.5, “Handling dependencies” for more information about dependencies on other packages, including the BUILDLINK_API_DEPENDS definitions.

Please take careful consideration before adjusting BUILDLINK_API_DEPENDS.pkg or BUILDLINK_ABI_DEPENDS.pkg in a buildlink3.mk file as we don't want to cause unneeded package deletions and rebuilds. In many cases, new versions of packages work just fine with older dependencies.

Also, it is not needed to set BUILDLINK_ABI_DEPENDS.pkg when it is identical to BUILDLINK_API_DEPENDS.pkg.

Note there is also the distinction that users are able to disable enforcement of ABI dependencies using the USE_ABI_DEPENDS variable, but there is no equivalent option for API dependencies.

The following is the recommended template for builtin.mk files:

.if !defined(IS_BUILTIN.foo)
#
# IS_BUILTIN.foo is set to "yes" or "no" depending on whether "foo"
# genuinely exists in the system or not.
#
IS_BUILTIN.foo?=        no

# BUILTIN_PKG.foo should be set here if "foo" is built-in and its package
# version can be determined.
#
.  if !empty(IS_BUILTIN.foo:M[yY][eE][sS])
BUILTIN_PKG.foo?=       foo-1.0
.  endif
.endif  # IS_BUILTIN.foo

.if !defined(USE_BUILTIN.foo)
USE_BUILTIN.foo?=       ${IS_BUILTIN.foo}
.  if defined(BUILTIN_PKG.foo)
.    for _depend_ in ${BUILDLINK_API_DEPENDS.foo}
.      if !empty(USE_BUILTIN.foo:M[yY][eE][sS])
USE_BUILTIN.foo!=                                                       \
        ${PKG_ADMIN} pmatch '${_depend_}' ${BUILTIN_PKG.foo}            \
        && ${ECHO} "yes" || ${ECHO} "no"
.      endif
.    endfor
.  endif
.endif  # USE_BUILTIN.foo

CHECK_BUILTIN.foo?=     no
.if !empty(CHECK_BUILTIN.foo:M[nN][oO])
#
# Here we place code that depends on whether USE_BUILTIN.foo is set to
# "yes" or "no".
#
.endif  # CHECK_BUILTIN.foo

The first section sets IS_BUILTIN.pkg depending on if pkg really exists in the base system. This should not be a base system software with similar functionality to pkg; it should only be “yes” if the actual package is included as part of the base system. This variable is only used internally within the builtin.mk file.

The second section sets BUILTIN_PKG.pkg to the version of pkg in the base system if it exists (if IS_BUILTIN.pkg is “yes”). This variable is only used internally within the builtin.mk file.

The third section sets USE_BUILTIN.pkg and is required in all builtin.mk files. The code in this section must make the determination whether the built-in software is adequate to satisfy the dependencies listed in BUILDLINK_API_DEPENDS.pkg. This is typically done by comparing BUILTIN_PKG.pkg against each of the dependencies in BUILDLINK_API_DEPENDS.pkg. USE_BUILTIN.pkg must be set to the correct value by the end of the builtin.mk file. Note that USE_BUILTIN.pkg may be “yes” even if IS_BUILTIN.pkg is “no” because we may make the determination that the built-in version of the software is similar enough to be used as a replacement.

The last section is guarded by CHECK_BUILTIN.pkg, and includes code that uses the value of USE_BUILTIN.pkg set in the previous section. This typically includes, e.g., adding additional dependency restrictions and listing additional files to symlink into ${BUILDLINK_DIR} (via BUILDLINK_FILES.pkg).

When building packages, it's possible to choose whether to set a global preference for using either the built-in (native) version or the pkgsrc version of software to satisfy a dependency. This is controlled by setting PREFER_PKGSRC and PREFER_NATIVE. These variables take values of either “yes”, “no”, or a list of packages. PREFER_PKGSRC tells pkgsrc to use the pkgsrc versions of software, while PREFER_NATIVE tells pkgsrc to use the built-in versions. Preferences are determined by the most specific instance of the package in either PREFER_PKGSRC or PREFER_NATIVE. If a package is specified in neither or in both variables, then PREFER_PKGSRC has precedence over PREFER_NATIVE. For example, to require using pkgsrc versions of software for all but the most basic bits on a NetBSD system, you can set:

PREFER_PKGSRC=  yes
PREFER_NATIVE=  getopt skey tcp_wrappers

A package must have a builtin.mk file to be listed in PREFER_NATIVE, otherwise it is simply ignored in that list.

PREFER_PKGSRC and PREFER_NATIVE should be set during bootstrap to ensure that the bootstrap process does not use inapropriate native tools as dependencies for core packages.

# ./bootstrap --prefer-pkgsrc yes --prefer-native openssl

Switching between settings globally at a later date can introduce complications with dependency resolution. This is caused by packages built with the opposite preference being installed alongside each other. Hence, any changes to these variables after bootstrap will necessitate rebuilding all packages depending on one whose preference has been changed. This is not trivial and should be avoided.

When using pkgsrc on Linux systems, there is high risk of "leakage", where programs installed by pkgsrc may inadvertently use a command or library not installed by pkgsrc, e.g. those installed by yum or apt. Such foreign dependencies may be installed, removed, or upgraded to a version incompatible with the pkgsrc package at any time, causing pkgsrc packages to subsequently malfunction. Pkgsrc cannot prevent this, as it has no control over other package managers. Another potential problem is that under Redhat Enterprise and related Linux systems, yum packages are only patched and never upgraded, so eventually they may become too outdated for use by pkgsrc. Even intentionally using foreign dependencies, not considered leakage, can lead to these problems, so it is generally discouraged. In order to minimize such problems, PREFER_PKGSRC defaults to "yes" on Linux systems. This ensures that pkgsrc is aware of any changes to dependency packages and can rebuild or upgrade the entire dependency tree as needed. This default can be overridden by setting --prefer-pkgsrc to "no" or a list of packages, or by setting --prefer-native to "yes".