Commit the buildlink3 developer's guide and the pkgviews user's guide until

they're polished enough to be included into Packages.txt.

2 files changed, 453 insertions, 0 deletions

diff --git a/mk/buildlink3/BUILDLINK3_DG b/mk/buildlink3/BUILDLINK3_DG
new file mode 100644
index 00000000000..a878984c48e
--- /dev/null
+++ b/mk/buildlink3/BUILDLINK3_DG
@@ -0,0 +1,231 @@
+$NetBSD: BUILDLINK3_DG,v 1.1 2004/01/29 06:03:15 jlam Exp $
+
+ 0 Developer's guide to buildlink3
+ =================================
+
+This is a tutorial for pkgsrc developers to understand and to use the
+buildlink3 framework in pkgsrc.
+
+
+ 1 Changes between buildlink2 and buildlink3
+ ===========================================
+
+The buildlink3 framework is a evolutionary descendant of the
+buildlink2 framework that does a better job of adhering to the
+fundamental buildlink principle: only allow the software build
+process to see what we choose to allow it to see.
+
+
+ 1.1 Better behavior with libtool
+ ================================
+
+One of the biggest problems in buildlink2 is handling packages that
+install libtool archive files for libraries that are also present in
+the base system.  buildlink3 is significantly better at this as it
+more tightly controls where libtool can find libtool archives.  One
+side effect of this is that we no longer need to create fake libtool
+archives to work around cases where the pkgsrc libraries were being
+used instead of the system libraries if they shared the same name.
+
+
+ 1.3 Support for native compilers
+ ================================
+
+The buildlink3 wrapper scripts have better support for using SunPro
+and MIPSpro compilers to build pkgsrc software.  For the most part,
+packages can use any compiler, but some third-party software is
+written assuming that it will be compiled using GCC.  The buildlink3
+wrapper scripts can capture some common GCC options and convert them
+into native toolchain equivalents.
+
+
+ 1.4 New buildlink3.mk file structure
+ ====================================
+
+buildlink3.mk files have two major differences over buildlink2.mk
+files.  The first, most noticeable difference is that buildlink3.mk
+generally don't have contain a BUILDLINK_FILES definition.  This is
+because buildlink3 automatically determines which files to symlink
+into ${BUILDLINK_DIR} by examining the PLIST of the installed package.
+The second difference is that buildlink3.mk files keep track of how
+"deep" we are in including buildlink3.mk files, and only creates
+dependencies on packages encountered at depth 1.  This means that
+packages that want to add a dependency must directly include the
+buildlink3.mk file for that dependency.
+
+
+ 1.5 Support for pkgviews
+ ========================
+
+When building pkgviews packages, buildlink3 doesn't symlink files
+into ${BUILDLINK_DIR} since it can safely refer to only a specific
+package's files by passing the appropriate -I<dir> and -L<dir> flags
+to the compiler, where <dir> points to a location in the package's
+depot directory.  When building "overwrite" packages, buildlink3 will
+act and feel very much like buildlink2 but with more advanced wrapper
+scripts, and there are provisions for allowing an "overwrite" package
+to build against the viewed instance of a depoted package.
+
+
+ 2 Writing buildlink3.mk files
+ =============================
+
+A package's buildlink3.mk file is included by Makefiles to indicate
+the need to compile and link against header files and libraries
+provided by the package.  A buildlink3.mk file should always provide
+enough information to add the correct type of dependency relationship
+and include any other buildlink3.mk files that it needs to find
+headers and libraries that it needs in turn.
+
+
+ 2.1 Simple packages
+ ===================
+
+To generate an initial buildlink3.mk file for further editting, Rene
+Hexel's pkgtools/createbuildlink package is highly recommended.  For
+most packages, the following command will generate a good starting
+point for buildlink3.mk files:
+
+    cd pkgsrc/category/pkgdir; createbuildlink -3 > buildlink3.mk
+
+The following real-life example buildlink3.mk is taken from
+graphics/tiff:
+
+------------8<------------8<------------8<------------8<------------
+# $NetBSD: BUILDLINK3_DG,v 1.1 2004/01/29 06:03:15 jlam Exp $
+
+BUILDLINK_DEPTH:=	${BUILDLINK_DEPTH}+
+TIFF_BUILDLINK3_MK:=	${TIFF_BUILDLINK3_MK}+
+
+.if !empty(BUILDLINK_DEPTH:M+)
+BUILDLINK_DEPENDS+=	tiff
+.endif
+
+.if !empty(TIFF_BUILDLINK3_MK:M+)
+BUILDLINK_PACKAGES+=		tiff
+BUILDLINK_DEPENDS.tiff+=	tiff>=3.5.4
+BUILDLINK_PKGSRCDIR.tiff?=	../../graphics/tiff
+
+.  include "../../devel/zlib/buildlink3.mk"
+.  include "../../graphics/jpeg/buildlink3.mk"
+.endif	# TIFF_BUILDLINK3_MK
+
+BUILDLINK_DEPTH:=	${BUILDLINK_DEPTH:S/+$//}
+------------8<------------8<------------8<------------8<------------
+
+The header and footer manipulate BUILDLINK_DEPTH, which is common
+across all buildlink3.mk files and is used to track at what depth we
+are in including buildlink3.mk files.
+
+The first section controls if the dependency on tiff is added.
+BUILDLINK_DEPENDS is the global list of packages for which
+dependencies are added by buildlink3.  The tiff package is only
+appended to this list if the buildlink3.mk is included directly by a
+package Makefile.
+
+The second section is protected from multiple inclusion and control
+how the dependency on tiff is added.  Several important variables
+are set in the section:
+
+ (1) BUILDLINK_PACKAGES is the global list of packages for which
+     buildlink3.mk files have been included.  It must _always_ be
+     appended to within a buildlink3.mk;
+
+ (2) BUILDLINK_DEPENDS.tiff is the actual dependency recorded in the
+     installed package; this should always be set using += to ensure
+     that we're appending to any pre-existing list of values.
+
+ (3) BUILDLINK_PKGSRCDIR.tiff is the location of the tiff pkgsrc
+     directory;
+
+ (4) BUILDLINK_DEPMETHOD.tiff (not shown above) controls whether we
+     use BUILD_DEPENDS or DEPENDS to add the dependency on tiff.  The
+     build dependency is selected by setting BUILDLINK_DEPMETHOD.tiff
+     to "build".  By default, the full dependency is used.
+
+ (5) BUILDLINK_INCDIRS.tiff and BUILDLINK_LIBDIRS.tiff (not shown
+     above) are lists of subdirectories of ${BUILDLINK_PREFIX.tiff} to
+     add the header and library search paths.  These default to
+     "include" and "lib" respectively.
+
+ (6) BUILDLINK_CPPFLAGS.tiff is the list of preprocessor flags to add
+     CPPFLAGS, which are passed on to the configure and build phases.
+     -I should be avoided and instead be added using
+     BUILDLINK_INCDIRS.tiff as above.
+
+Any buildlink3.mk for tiff's dependencies are also included at this
+point.  Including these buildlink3.mk files means that the headers
+and libraries for these dependencies are also symlinked into
+${BUILDLINK_DIR} whenever the tiff buildlink3.mk file is included.
+
+There are several considerations that arise when figuring out how
+to set BUILDLINK_DEPENDS.<pkg> correctly:
+
+ (1) If the package has a pre-existing buildlink2.mk file, then
+     match the BUILDLINK_DEPENDS.<pkg> lines between the
+     buildlink2.mk file and the newly-created buildlink3.mk file.
+
+ (2) If there is no pre-existing buildlink2.mk file, then set
+     BUILDLINK_DEPENDS.foo to the first version of the package that
+     had the last change in the major number of a shared library or
+     that had a major API change.
+
+
+ 2.1 Packages that coincide with base system software
+ ====================================================
+
+Some packages in pkgsrc install headers and libraries that coincide
+with headers and libraries present in the base system.  The best
+recommendation for writing buildlink3.mk files for these packages is
+to use graphics/MesaLib/buildlink3.mk as a template.
+
+
+ 3 bl3ifying a package
+ =====================
+
+The process of "bl3ifying" a package, or converting a package to use
+the buildlink3 framework, is surprisingly easy.  The things to keep in
+mind are:
+
+  (1) Set USE_BUILDLINK3=yes.
+
+  (2) Change references to buildlink2.mk files into buildlink3.mk
+      files.
+
+  (3) Ensure that the build always calls the wrapper scripts instead of
+      the actual toolchain.  Some are tricky, e.g. openssl, cdrecord,
+      ocaml, and the only way to know for sure is the check .work.log
+      to see if the wrappers are being invoked.
+
+  (4) Don't override PREFIX from within the package Makefile, e.g.
+      Java VMs, standalone shells, etc., because the code to symlink
+      files into ${BUILDLINK_DIR} looks for files relative to
+      "pkg_info -qp <pkgname>".
+
+
+ 4 Troubleshooting
+ =================
+
+Q1: I'm trying to bl3ify a package but I get an error that looks like:
+
+	make: don't know how to make _BUILDLINK_USE. Stop
+
+A1: You forgot to change a reference to a buildlink2.mk file into a
+    buildlink3.mk file.
+
+
+Q2: Dependencies are added for every single buildlink3.mk file I
+    include, including for when it's supposed to use the base system
+    software.  What's going on?
+
+A2: You forgot to change USE_BUILDLINK2 to USE_BUILDLINK3 in the
+    package Makefile.
+
+
+Q3: Where can I see the actual command executed by the wrapper
+    scripts?
+
+A3: You should examine the contents of the ${WRKDIR}/.work.log file.
+    The lines preceded with [*] are the commands that are intercepted
+    by the wrapper scripts, and the lines preceded with <.> are the
+    commands that are executed by the wrapper scripts.
diff --git a/mk/buildlink3/PKGVIEWS_UG b/mk/buildlink3/PKGVIEWS_UG
new file mode 100644
index 00000000000..6e586e3d898
--- /dev/null
+++ b/mk/buildlink3/PKGVIEWS_UG
@@ -0,0 +1,222 @@
+$NetBSD: PKGVIEWS_UG,v 1.1 2004/01/29 06:03:15 jlam Exp $
+
+ 0 User's Guide to pkgviews
+ ==========================
+
+This is a tutorial for pkgsrc users who wish to experiment with the
+new "pkgviews" implementation in pkgsrc.  More information about
+pkgviews may be found in pkgsrc/mk/buildlink3/README.  That document
+also explains why you might want to use pkgviews.  Some reasons
+include:
+
+	* fully dynamic PLISTs
+	* multiple version of the same package can co-exist
+	* no or non-fatal conflicting packages
+
+
+ 0.1 CAVEAT (USE AT YOUR OWN RISK!)
+ ==================================
+
+Pkgviews is *completely experimental* at this point in time.  Bug
+reports on pkgviews will be treated with a fairly low priority by the
+general pkgsrc developers, though I may personally be more responsive.
+However, the major thrust of the next several weeks of pkgsrc
+development for me will revolve around testing and integrating the
+buildlink3 framework into pkgsrc and deprecating buildlink2.  Any
+work on pkgviews during that time is strictly happenstance.
+
+
+ 0.1 Preparing your system to use pkgviews
+ =========================================
+
+You will need to start with a clean system to use pkgviews.  Depoted
+packages, a.k.a "pkgviews" packages (packages that are built using
+pkgviews) cannot depend on non-depoted packages, a.k.a.  "overwrite"
+packages, although the reverse is possible.  If you have _any_
+packages installed, you will need to pkg_delete them before you can
+start building depoted packages.  In fact, it's best to completely
+nuke /usr/pkg (or wherever you choose for your ${LOCALBASE}) as
+pkgviews manages all of its own directories.
+
+Next you will need to add the following line to /etc/mk.conf:
+
+	PKG_INSTALLATION_PREFS=	pkgviews overwrite
+
+This creates pkgviews packages instead of overwrite packages for any
+packages that support it.  The packages that do support pkgviews
+can be identified by searching the package Makefile for the the
+following line:
+
+	PKG_INSTALLATION_TYPES=	overwrite pkgviews
+
+
+ 0.2 Installing your first pkgviews package
+ ==========================================
+
+The first package you will need to install is pkgsrc/pkgtools/digest.
+
+===> Checking for vulnerabilities in digest-20021220
+===> Extracting for digest-20021220
+===> Checking for vulnerabilities in digest-20021220
+===> Patching for digest-20021220
+===> Overriding tools for digest-20021220
+===> Buildlinking for digest-20021220
+===> Configuring for digest-20021220
+configure: WARNING: If you wanted to set the --build type, don't use --host.
+    If a cross compiler is detected then cross compile mode will be used.
+checking build system type... i386-unknown-netbsdelf1.6.2.
+checking host system type... i386--netbsdelf
+...
+configure: creating ./config.status
+config.status: creating Makefile
+config.status: creating config.h
+===> Building for digest-20021220
+cc -DHAVE_CONFIG_H -I. -I. -DHOST=\"i386--netbsdelf\" -DVERSION=\"20021220\"  -O2 -mcpu=pentiumpro -c bits.c -o bits.o
+...
+cc -O2 -mcpu=pentiumpro  -o digest digest.o md5c.o rmd160.o rmd160hl.o sha2.o sha2hl.o  md5hl.o sha1.o sha1hl.o 
+=> Fixing buildlink references in files-to-be-installed.
+===> Installing for digest-20021220
+===> Becoming root@blossom.hq.williamlam.com to install digest.
+sudo /bin/sh ./mkinstalldirs /usr/pkg/packages/digest-20021220/bin
+mkdir /usr/pkg/packages/digest-20021220/bin
+install -c -s -o root -g wheel -m 555 digest /usr/pkg/packages/digest-20021220/bin/digest
+/bin/sh ./mkinstalldirs /usr/pkg/packages/digest-20021220/man/man1
+mkdir /usr/pkg/packages/digest-20021220/man
+mkdir /usr/pkg/packages/digest-20021220/man/man1
+install -c -o root -g wheel -m 444 digest.1 /usr/pkg/packages/digest-20021220/man/man1/digest.1
+===> Registering installation for digest-20021220
+===> Building views for digest-20021220
+=> Performing package view clash check for digest-20021220 in standard view
+=> Performing package view overwrite check for digest-20021220 in standard view
+=> Linking package into standard view
+
+Notice in the transcript that the digest package has been installed
+into /usr/pkg/packages/digest-20021220 and then subsequently linked
+into the "standard view".  The standard view is simply /usr/pkg, which
+means that the digest executable is accessible as /usr/pkg/bin/digest.
+
+
+ 0.3 The contents of /usr/pkg (${LOCALBASE})
+ ===========================================
+
+There is a directory /usr/pkg/packages that contains all of the
+depoted packages.  The contents of /usr/pkg/packages after installing
+digest are:
+
+/usr/pkg/packages/digest-20021220/+BUILD_INFO
+/usr/pkg/packages/digest-20021220/+BUILD_VERSION
+/usr/pkg/packages/digest-20021220/+COMMENT
+/usr/pkg/packages/digest-20021220/+CONTENTS
+/usr/pkg/packages/digest-20021220/+DESC
+/usr/pkg/packages/digest-20021220/+SIZE_ALL
+/usr/pkg/packages/digest-20021220/+SIZE_PKG
+/usr/pkg/packages/digest-20021220/+VIEWS
+/usr/pkg/packages/digest-20021220/bin/digest
+/usr/pkg/packages/digest-20021220/man/man1/digest.1
+
+Note that all of the files related to the package, including the
+package metadata files used by the pkg_* tools, are stored in the
+"depot directory" for digest (/usr/pkg/packages/digest-20021220).
+
+The digest package was also linked into the standard view.  Looking
+inside /usr/pkg shows us the following symlinks:
+
+/usr/pkg/bin/digest -> /usr/pkg/packages/digest-20021220/bin/digest
+/usr/pkg/man/man1/digest.1 -> /usr/pkg/packages/digest-20021220/man/man1/digest.1
+
+Since digest is present in /usr/pkg/bin, it can be used as always
+without any changes needed by the typical user with /usr/pkg/bin in
+his PATH, and "man digest" will continue to work as expected.
+
+
+ 0.4 Creating new views
+ ======================
+
+The following commands will install both pkgsrc/www/lynx and
+pkgsrc/www/lynx-current onto the same system using pkgviews:
+
+	cd /usr/pkgsrc/www/lynx; make install
+	cd /usr/pkgsrc/www/lynx-current; make install PKGVIEWS=devel
+
+The stable version of lynx (2.8.4.1nb2) is linked into the standard
+view, while the -current version of lynx (2.8.5.0.7nb3) is linked into
+the "devel" view.  The lynx-related files in /usr/pkg are:
+
+/usr/pkg/bin/lynx -> /usr/pkg/packages/lynx-2.8.4.1nb2/bin/lynx
+/usr/pkg/man/man1/lynx.1 -> /usr/pkg/packages/lynx-2.8.4.1nb2/man/man1/lynx.1
+/usr/pkg/share/locale/cs/LC_MESSAGES/lynx.mo -> ...
+...
+
+/usr/pkg/devel/bin/lynx -> /usr/pkg/packages/lynx-2.8.5.0.7nb3/bin/lynx
+/usr/pkg/devel/man/man1/lynx.1 -> /usr/pkg/packages/lynx-2.8.5.0.7nb3/man/man1/lynx.1
+/usr/pkg/devel/share/locale/cs/LC_MESSAGES/lynx.mo -> ...
+...
+
+Note that the files for pkgsrc/www/lynx have been symlinked into
+/usr/pkg as usual, but the files for pkgsrc/www/lynx-current have been
+symlinked into the "devel" subdirectory of /usr/pkg.  The files in
+in the "devel" view can be accessed by adding /usr/pkg/devel/bin to
+the PATH and adding /usr/pkg/devel/man to MANPATH in the appropriate
+place.
+
+
+ 0.5 Managing views using pkg_* tools
+ ====================================
+
+Adding and removing package instances to and from views is always a
+safe operation as the true package files are left untouched.  All that
+happens is that linkfarms are being added or removed, thus changes are
+always easy to reverse.
+
+Adding a depoted package into a view may be accomplished with:
+
+	pkg_view -w devel add gmake-3.80nb2
+
+This adds the gmake-3.80nb2 package (already present on the system) to
+the "devel" view.  The "devel" view is created if it doesn't already
+exist.
+
+Removing a package from a view may be accomplished with either of the
+following commands:
+
+	pkg_view -w devel delete gmake-3.80nb2
+	pkg_delete -K /usr/pkg/devel/.pkgdb gmake
+
+Either of these will remove the gmake-3.80nb2 package from the "devel"
+view if the package exists in the view.
+
+Removing a package from the standard view may be accomplished with
+either of the following commands
+
+	pkg_view delete gmake-3.80nb2
+	pkg_delete gmake
+
+The latter one is recommended for speed.
+
+
+ 0.6 Completely removing a package
+ =================================
+
+Removing a depoted package from the system (removing it from all views
+and removing the depot directory) is done with:
+
+	pkg_delete -K /usr/pkg/packages gmake-3.80nb2
+
+If you know that only a single gmake is present on your system, then
+you can specify "gmake" instead of the full package name to
+pkg_delete(1).  However, I recommend being explicit about the package
+you are removing to avoid any surprises.
+
+
+ 0.6 Where to learn more about pkgviews
+ ======================================
+
+There are several shell environment variables that may be set to
+affect the default behaviour of the pkg_* tools.  Please read the
+pkg_view(1) man page for more information.
+
+For a more complete understanding of the motivations and principles
+behind pkgviews, the following paper by Alistair Crooks is highly
+recommended:
+
+	http://www.NetBSD.org/Documentation/software/pkgviews.pdf