path: root/doc/guide/files/buildlink.xml

<!-- $NetBSD: buildlink.xml,v 1.37 2014/12/30 15:13:19 wiz Exp $ -->

<chapter id="buildlink">
  <title>Buildlink methodology</title>

  <para>Buildlink is a framework in pkgsrc that controls what headers and libraries
  are seen by a package's configure and build processes.  This is implemented
  in a two step process:</para>

  <orderedlist>
    <listitem>
      <para>Symlink headers and libraries for dependencies into
      <varname>BUILDLINK_DIR</varname>, which by default is a subdirectory
      of <varname>WRKDIR</varname>.</para>
    </listitem>

    <listitem>
      <para>Create wrapper scripts that are used in place of the normal compiler
      tools that translate <option>-I${LOCALBASE}/include</option> and
      <option>-L${LOCALBASE}/lib</option> into references to
      <varname>BUILDLINK_DIR</varname>. The wrapper scripts also make
      native compiler on some operating systems look like GCC, so that
      packages that expect GCC won't require modifications to build with
      those native compilers.</para>
    </listitem>
  </orderedlist>

  <para>This normalizes the environment in which a package is built so that the
  package may be built consistently despite what other software may be
  installed. Please note that the normal system header and library paths,
  e.g. <filename>/usr/include</filename>,
  <filename>/usr/lib</filename>, etc., are always searched -- buildlink3 is
  designed to insulate the package build from non-system-supplied
  software.</para>

  <sect1 id="converting-to-buildlink3">
    <title>Converting packages to use buildlink3</title>

    <para>The process of converting packages to use the buildlink3
    framework (<quote>bl3ifying</quote>) is fairly straightforward.
    The things to keep in mind are:</para>

    <orderedlist>
      <listitem>
	<para>Ensure that the build always calls the wrapper scripts
	instead of the actual toolchain.  Some packages are tricky,
	and the only way to know for sure is the check
	<filename>${WRKDIR}/.work.log</filename> to see if the
	wrappers are being invoked.</para>
      </listitem>

      <listitem>
	<para>Don't override <varname>PREFIX</varname> from within
	the package Makefile, e.g. Java VMs, standalone shells,
	etc., because the code to symlink files into
	<filename>${BUILDLINK_DIR}</filename> looks for files
	relative to <quote>pkg_info -qp <replaceable>pkgname</replaceable></quote>.
	</para>
      </listitem>

      <listitem>
	<para>Remember that <emphasis>only</emphasis> the
	<filename>buildlink3.mk</filename> files that you list in a
	package's Makefile are added as dependencies for that package.
	</para>
      </listitem>
    </orderedlist>

    <para>If a dependency on a particular package is required for its libraries and
    headers, then we replace:</para>

<programlisting>
DEPENDS+=       foo>=1.1.0:../../category/foo
</programlisting>

    <para>with</para>

<programlisting>
.include "../../category/foo/buildlink3.mk"
</programlisting>

    <para>The buildlink3.mk files usually define the required dependencies.
    If you need a newer version of the dependency when using buildlink3.mk
    files, then you can define it in your Makefile; for example:</para>

<programlisting>
BUILDLINK_API_DEPENDS.foo+=   foo>=1.1.0
.include "../../category/foo/buildlink3.mk"
</programlisting>

    <para>There are several <filename>buildlink3.mk</filename>
    files in <filename>pkgsrc/mk</filename>
    that handle special package issues:</para>

    <itemizedlist>
      <listitem>
	<para><filename>bdb.buildlink3.mk</filename> chooses either
	the native or a pkgsrc Berkeley DB implementation based on
	the values of <varname>BDB_ACCEPTED</varname> and
	<varname>BDB_DEFAULT</varname>.</para>
      </listitem>

      <listitem>
	<para><filename>curses.buildlink3.mk</filename>: If the system
	comes with neither Curses nor NCurses, this will take care
	to install the <filename
	role="pkg">devel/ncurses</filename> package.</para>
      </listitem>

      <listitem>
	<para><filename>krb5.buildlink3.mk</filename> uses the value
	of <varname>KRB5_ACCEPTED</varname> to choose between
	adding a dependency on Heimdal or MIT-krb5 for packages that
	require a Kerberos 5 implementation.</para>
      </listitem>

      <listitem>
	<para><filename>motif.buildlink3.mk</filename> checks for a
	system-provided Motif installation or adds a dependency on
	<filename role="pkg">x11/lesstif</filename>, <filename
	role="pkg">x11/motif</filename> or <filename
	role="pkg">x11/openmotif</filename>.  The user can set
	<varname>MOTIF_TYPE</varname> to <quote>dt</quote>,
	<quote>lesstif</quote>, <quote>motif</quote> or
	<quote>openmotif</quote> to choose
	which Motif version will be used.</para>
      </listitem>

      <listitem>
	<para><filename>readline.buildlink3.mk</filename> checks for a
	system-provided GNU readline or editline (libedit) installation,
	or adds a dependency on <filename role="pkg">devel/readline</filename>,
	<filename role="pkg">devel/editline</filename>. The user can set
	<varname>READLINE_DEFAULT</varname> to choose readline implementation.
	If your package really needs GNU readline library, its Makefile
	should include <filename>devel/readline/buildlink3.mk</filename>
	instead of <filename>readline.buildlink3.mk</filename>.</para>
      </listitem>

      <listitem>
	<para><filename>oss.buildlink3.mk</filename> defines several
	variables that may be used by packages that use the
	Open Sound System (OSS) API.</para>
      </listitem>

      <listitem>
	<para><filename>pgsql.buildlink3.mk</filename> will accept
	any of the Postgres versions in the variable
	<varname>PGSQL_VERSIONS_ACCEPTED</varname> and default to
	the version <varname>PGSQL_VERSION_DEFAULT</varname>. See
	the file for more information.</para>
      </listitem>

      <listitem>
	<para><filename>pthread.buildlink3.mk</filename> uses the value of
	<varname>PTHREAD_OPTS</varname> and checks for native pthreads or adds
	a dependency on <filename
	role="pkg">devel/pth</filename> as needed.</para>
      </listitem>

      <listitem>
	<para><filename>xaw.buildlink3.mk</filename> uses the value of
	<varname>XAW_TYPE</varname> to choose a particular Athena widgets
	library.</para>
      </listitem>
    </itemizedlist>

    <para>The comments in those <filename>buildlink3.mk</filename>
    files provide a more complete
    description of how to use them properly.</para>
  </sect1>

  <sect1 id="creating-buildlink3.mk">
    <title>Writing <filename>buildlink3.mk</filename> files</title>

<anchor id="buildlink3.mk"/>

    <para>A package's <filename>buildlink3.mk</filename> file is
    included by Makefiles to indicate the need to compile and link
    against header files and libraries provided by the package.  A
    <filename>buildlink3.mk</filename> file should always provide
    enough information to add the correct type of dependency
    relationship and include any other
    <filename>buildlink3.mk</filename> files that it needs to find
    headers and libraries that it needs in turn.</para>

    <para>To generate an initial <filename>buildlink3.mk</filename>
    file for further editing, Rene Hexel's <filename
    role="pkg">pkgtools/createbuildlink</filename>
    package is highly recommended.  For most packages, the following
    command will generate a good starting point for
    <filename>buildlink3.mk</filename> files:</para>

    <screen>
&cprompt; <userinput>cd pkgsrc/<replaceable>category</replaceable>/<replaceable>pkgdir</replaceable>
&cprompt; createbuildlink &gt;buildlink3.mk</userinput>
    </screen>

    <sect2 id="anatomy-of-bl3">
      <title>Anatomy of a buildlink3.mk file</title>

      <para>The following real-life example
      <filename>buildlink3.mk</filename> is taken
      from <filename>pkgsrc/graphics/tiff</filename>:</para>

<programlisting>
# &#36;NetBSD: buildlink3.mk,v 1.16 2009/03/20 19:24:45 joerg Exp &#36;

BUILDLINK_TREE+=	tiff

.if !defined(TIFF_BUILDLINK3_MK)
TIFF_BUILDLINK3_MK:=

BUILDLINK_API_DEPENDS.tiff+=	tiff>=3.6.1
BUILDLINK_ABI_DEPENDS.tiff+=	tiff>=3.7.2nb1
BUILDLINK_PKGSRCDIR.tiff?=	../../graphics/tiff

.include "../../devel/zlib/buildlink3.mk"
.include "../../graphics/jpeg/buildlink3.mk"
.endif # TIFF_BUILDLINK3_MK

BUILDLINK_TREE+=	-tiff
</programlisting>

      <para>The header and footer manipulate
      <varname>BUILDLINK_TREE</varname>, which is common across all
      <filename>buildlink3.mk</filename> files and is used to track
      the dependency tree.</para>

      <para>The main section is protected from multiple inclusion
      and controls how the dependency on <replaceable>pkg</replaceable> is
      added.  Several important variables are set in the section:</para>

      <itemizedlist>
	<listitem>
	  <para><varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>
	  is the actual dependency recorded in the installed
	  package; this should always be set using
	  <command>+=</command> to ensure that
	  we're appending to any pre-existing list of values.  This
	  variable should be set to the first version of the
	  package that had an backwards-incompatible API change.
	  </para>
	</listitem>

	<listitem>
	  <para><varname>BUILDLINK_PKGSRCDIR.<replaceable>pkg</replaceable></varname>
	  is the location of the <replaceable>pkg</replaceable>
	  pkgsrc directory.</para>
	</listitem>

	<listitem>
	  <para><varname>BUILDLINK_DEPMETHOD.<replaceable>pkg</replaceable></varname>
	  (not shown above) controls whether we use
	  <varname>BUILD_DEPENDS</varname> or
	  <varname>DEPENDS</varname> to add the dependency on
	  <replaceable>pkg</replaceable>.  The build dependency is
	  selected by setting
	  <varname>BUILDLINK_DEPMETHOD.<replaceable>pkg</replaceable></varname>
	  to <quote>build</quote>.  By default, the full dependency is
	  used.</para>
	</listitem>

	<listitem>
	  <para><varname>BUILDLINK_INCDIRS.<replaceable>pkg</replaceable></varname>
	    and
	    <varname>BUILDLINK_LIBDIRS.<replaceable>pkg</replaceable></varname>
	    (not shown above) are lists of subdirectories of
	    <filename>${BUILDLINK_PREFIX.<replaceable>pkg</replaceable>}</filename>
	    to add to the header and library search paths.  These
	    default to <quote>include</quote> and <quote>lib</quote>
	  respectively.</para>
	</listitem>

	<listitem>
	  <para><varname>BUILDLINK_CPPFLAGS.<replaceable>pkg</replaceable></varname>
	    (not shown above) is the list of preprocessor flags to add
	    to <varname>CPPFLAGS</varname>, which are passed on to the
	    configure and build phases.  The <quote>-I</quote> option
	    should be avoided and instead be handled using
	    <varname>BUILDLINK_INCDIRS.<replaceable>pkg</replaceable></varname> as
	  above.</para>
	</listitem>
      </itemizedlist>

      <para>The following variables are all optionally defined within
      this second section (protected against multiple inclusion) and
      control which package files are symlinked into
      <filename>${BUILDLINK_DIR}</filename> and how their names are
      transformed during the symlinking:</para>

      <itemizedlist>
	<listitem>
	  <para><varname>BUILDLINK_FILES.<replaceable>pkg</replaceable></varname>
	    (not shown above) is a shell glob pattern relative to
	    <filename>${BUILDLINK_PREFIX.<replaceable>pkg</replaceable>}</filename>
	    to be symlinked into
	    <filename>${BUILDLINK_DIR}</filename>,
	  e.g. <filename>include/*.h</filename>.</para>
	</listitem>

	<listitem>
	  <para><varname>BUILDLINK_FILES_CMD.<replaceable>pkg</replaceable></varname>
	    (not shown above) is a shell pipeline that
	    outputs to stdout a list of files relative to
	    <filename>${BUILDLINK_PREFIX.<replaceable>pkg</replaceable>}</filename>.
	    The resulting files are to be symlinked
	    into <filename>${BUILDLINK_DIR}</filename>.  By default,
	    this takes the <filename>+CONTENTS</filename> of a
	    <replaceable>pkg</replaceable> and filters it through
	    <varname>${BUILDLINK_CONTENTS_FILTER.<replaceable>pkg</replaceable>}</varname>.</para>
	</listitem>

	<listitem>
	  <para><varname>BUILDLINK_CONTENTS_FILTER.<replaceable>pkg</replaceable></varname>
	    (not shown above) is a filter command that filters
	    <filename>+CONTENTS</filename> input into a list of files
	    relative to
	    <filename>${BUILDLINK_PREFIX.<replaceable>pkg</replaceable>}</filename>
	    on stdout.  By default,
	    <varname>BUILDLINK_CONTENTS_FILTER.<replaceable>pkg</replaceable></varname>
	    outputs the contents of the <filename>include</filename>
	    and <filename>lib</filename> directories in the package
	    <filename>+CONTENTS</filename>.
	</listitem>

	<listitem>
	  <para><varname>BUILDLINK_FNAME_TRANSFORM.<replaceable>pkg</replaceable></varname>
	    (not shown above) is a list of sed arguments used to
	    transform the name of the source filename into a
	    destination filename, e.g. <command>-e
	    "s|/curses.h|/ncurses.h|g"</command>.</para>
	</listitem>
      </itemizedlist>

      <para>This section can additionally include any
      <filename>buildlink3.mk</filename> needed for
      <replaceable>pkg</replaceable>'s library dependencies.
      Including these <filename>buildlink3.mk</filename> files
      means that the headers and libraries for these
      dependencies are also symlinked into
      <filename>${BUILDLINK_DIR}</filename>
      whenever the <replaceable>pkg</replaceable>
      <filename>buildlink3.mk</filename>
      file is included. Dependencies are only added for directly
      include <filename>buildlink3.mk</filename> files.</para>

      <para>When providing a <filename>buildlink3.mk</filename> and
      including other <filename>buildlink3.mk</filename> files in it,
      please only add necessary ones, i.e., those whose libraries or
      header files are automatically exposed when the package is
      use.</para>

      <para>In particular, if only an executable
      (<filename>bin/foo</filename>) is linked against a library, that
      library does not need to be propagated in the
      <filename>buildlink3.mk</filename> file.</para>

      <para>The following steps should help you decide if a
      <filename>buildlink3.mk</filename> file needs to be included:
      <itemizedlist>
	<listitem><para>Look at the installed header files: What
	headers do they include? The packages providing these files
	must be buildlinked.</para></listitem>
	<listitem><para>Run <filename>ldd</filename> on all installed
	libraries and look against what other libraries they link.
	Some of the packages providing these probably need to be
	buildlinked; however, it's not automatic, since e.g. GTK on
	some systems pulls in the X libraries, so they will show up in
	the <filename>ldd</filename> output, while on others (like OS
	X) it won't. <filename>ldd</filename> output can thus only be
	used as a hint.</para></listitem>
      </itemizedlist>
      </para>
    </sect2>

    <sect2 id="updating-buildlink-depends">
      <title>Updating
      <varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>
      and
      <varname>BUILDLINK_ABI_DEPENDS.<replaceable>pkg</replaceable></varname>
      in <filename>buildlink3.mk</filename> files</title>

      <para>These two variables differ in that one describes source
      compatibility (API) and the other binary compatibility (ABI).
      The difference is that a change in the API breaks compilation of
      programs while changes in the ABI stop compiled programs from
      running.</para>

      <para>Changes to the
      <varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>
      variable in a <filename>buildlink3.mk</filename> file happen
      very rarely. One possible reason is that all packages depending
      on this already need a newer version. In case it is bumped see
      the description below.</para>

      <para>The most common example of an ABI change is that the major
      version of a shared library is increased. In this case,
      <varname>BUILDLINK_ABI_DEPENDS.<replaceable>pkg</replaceable></varname>
      should be adjusted to require at least the new package version.
      Then the packages that depend on this package need their
      <varname>PKGREVISION</varname>s increased and, if they have
      <filename>buildlink3.mk</filename> files, their
      <varname>BUILDLINK_ABI_DEPENDS.<replaceable>pkg</replaceable></varname>
      adjusted, too. This is needed so pkgsrc will require the correct
      package dependency and not settle for an older one when building
      the source.</para>

      <para>See <xref linkend="dependencies"/> for
      more information about dependencies on other packages,
      including the <varname>BUILDLINK_ABI_DEPENDS</varname> and
      <varname>ABI_DEPENDS</varname> definitions.</para>

      <para>Please take careful consideration before adjusting
      <varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>
      or
      <varname>BUILDLINK_ABI_DEPENDS.<replaceable>pkg</replaceable></varname>
      as we don't want to cause unneeded package deletions and
      rebuilds.  In many cases, new versions of packages work just
      fine with older dependencies.</para>

      <para>Also it is not needed to set
      <varname>BUILDLINK_ABI_DEPENDS.<replaceable>pkg</replaceable></varname>
      when it is identical to
      <varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>.	</para>
    </sect2>
  </sect1>

  <sect1 id="writing-builtin.mk">
    <title>Writing <filename>builtin.mk</filename> files</title>

    <para>Some packages in pkgsrc install headers and libraries that
      coincide with headers and libraries present in the base system.
      Aside from a <filename>buildlink3.mk</filename> file, these
      packages should also include a <filename>builtin.mk</filename>
      file that includes the necessary checks to decide whether using
      the built-in software or the pkgsrc software is
    appropriate.</para>

    <para>The only requirements of a builtin.mk file for
    <replaceable>pkg</replaceable> are:</para>

    <orderedlist>
      <listitem>
	<para>It should set
	<varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
	to either <quote>yes</quote> or <quote>no</quote>
	after it is included.</para>
      </listitem>

      <listitem>
	<para>It should <emphasis>not</emphasis> override any
	<varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
	which is already set before the
	<filename>builtin.mk</filename> file is included.</para>
      </listitem>

      <listitem>
	<para>It should be written to allow multiple inclusion.  This
	is <emphasis>very</emphasis> important and takes careful
	attention to <filename>Makefile</filename> coding.</para>
      </listitem>
    </orderedlist>

    <sect2 id="anatomy-of-builtin.mk">
      <title>Anatomy of a <filename>builtin.mk</filename> file</title>

      <para>The following is the recommended template for builtin.mk
      files:</para>

<programlisting>
.if !defined(IS_BUILTIN.foo)
#
# IS_BUILTIN.foo is set to "yes" or "no" depending on whether "foo"
# genuinely exists in the system or not.
#
IS_BUILTIN.foo?=        no

# BUILTIN_PKG.foo should be set here if "foo" is built-in and its package
# version can be determined.
#
.  if !empty(IS_BUILTIN.foo:M[yY][eE][sS])
BUILTIN_PKG.foo?=       foo-1.0
.  endif
.endif  # IS_BUILTIN.foo

.if !defined(USE_BUILTIN.foo)
USE_BUILTIN.foo?=       ${IS_BUILTIN.foo}
.  if defined(BUILTIN_PKG.foo)
.    for _depend_ in ${BUILDLINK_API_DEPENDS.foo}
.      if !empty(USE_BUILTIN.foo:M[yY][eE][sS])
USE_BUILTIN.foo!=                                                       \
        ${PKG_ADMIN} pmatch '${_depend_}' ${BUILTIN_PKG.foo}            \
        &amp;&amp; ${ECHO} "yes" || ${ECHO} "no"
.      endif
.    endfor
.  endif
.endif  # USE_BUILTIN.foo

CHECK_BUILTIN.foo?=     no
.if !empty(CHECK_BUILTIN.foo:M[nN][oO])
#
# Here we place code that depends on whether USE_BUILTIN.foo is set to
# "yes" or "no".
#
.endif  # CHECK_BUILTIN.foo
</programlisting>

      <para>The first section sets
      <varname>IS_BUILTIN.<replaceable>pkg</replaceable></varname>
      depending on if <replaceable>pkg</replaceable> really exists
      in the base system.  This should not be a base system software
      with similar functionality to <replaceable>pkg</replaceable>;
      it should only be <quote>yes</quote> if the actual package is
      included as part of the base system.  This variable is only
      used internally within the <filename>builtin.mk</filename>
      file.</para>

      <para>The second section sets
      <varname>BUILTIN_PKG.<replaceable>pkg</replaceable></varname>
      to the version of <replaceable>pkg</replaceable> in the base
      system if it exists (if
      <varname>IS_BUILTIN.<replaceable>pkg</replaceable></varname>
      is <quote>yes</quote>).  This variable is only used internally
      within the <filename>builtin.mk</filename> file.</para>

      <para>The third section sets
      <varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
      and is <emphasis>required</emphasis> in all
      <filename>builtin.mk</filename> files.  The code in this
      section must make the determination whether the built-in
      software is adequate to satisfy the dependencies listed in
      <varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>.
      This is typically done by comparing
      <varname>BUILTIN_PKG.<replaceable>pkg</replaceable></varname>
      against each of the dependencies in
      <varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>.
      <varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
      <emphasis>must</emphasis> be set to the correct value by the
      end of the <filename>builtin.mk</filename> file.  Note that
      <varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
      may be <quote>yes</quote> even if
      <varname>IS_BUILTIN.<replaceable>pkg</replaceable></varname>
      is <quote>no</quote> because we may make the determination
      that the built-in version of the software is similar enough to
      be used as a replacement.</para>

      <para>The last section is guarded by
      <varname>CHECK_BUILTIN.<replaceable>pkg</replaceable></varname>,
      and includes code that uses the value of
      <varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
      set in the previous section.  This typically includes, e.g.,
      adding additional dependency restrictions and listing additional
      files to symlink into <filename>${BUILDLINK_DIR}</filename> (via
      <varname>BUILDLINK_FILES.<replaceable>pkg</replaceable></varname>).</para>
    </sect2>

    <sect2 id="native-or-pkgsrc-preference">
      <title>Global preferences for native or pkgsrc software</title>

      <para>When building packages, it's possible to choose whether to set
	a global preference for using either the built-in (native)
	version or the pkgsrc version of software to satisfy a
	dependency.  This is controlled by setting
	<varname>PREFER_PKGSRC</varname> and
	<varname>PREFER_NATIVE</varname>.  These variables take values
	of either <quote>yes</quote>, <quote>no</quote>, or a list of
	packages.  <varname>PREFER_PKGSRC</varname> tells pkgsrc to
	use the pkgsrc versions of software, while
	<varname>PREFER_NATIVE</varname> tells pkgsrc to use the
	built-in versions.  Preferences are determined by the most
	specific instance of the package in either
	<varname>PREFER_PKGSRC</varname> or
	<varname>PREFER_NATIVE</varname>.  If a package is specified
	in neither or in both variables, then
	<varname>PREFER_PKGSRC</varname> has precedence over
	<varname>PREFER_NATIVE</varname>.  For example, to require
	using pkgsrc versions of software for all but the most basic
      bits on a NetBSD system, you can set:</para>

<programlisting>
PREFER_PKGSRC=  yes
PREFER_NATIVE=  getopt skey tcp_wrappers
</programlisting>

      <para>A package <emphasis>must</emphasis> have a
      <filename>builtin.mk</filename>
      file to be listed in <varname>PREFER_NATIVE</varname>,
      otherwise it is simply ignored in that list.</para>
    </sect2>
  </sect1>
</chapter>