path: root/doc/guide/files/buildlink.xml

<!-- $NetBSD: buildlink.xml,v 1.14 2006/04/06 06:21:57 reed Exp $ -->

<chapter id="buildlink">
  <title>Buildlink methodology</title>

  <para>Buildlink is a framework in pkgsrc that controls what headers and libraries
    are seen by a package's configure and build processes.  This is implemented
    in a two step process:</para>

  <orderedlist>
    <listitem>
      <para>Symlink headers and libraries for dependencies into
	<varname>BUILDLINK_DIR</varname>, which by default is a subdirectory
	of <varname>WRKDIR</varname>.</para>
    </listitem>

    <listitem>
      <para>Create wrapper scripts that are used in place of the normal compiler
	tools that translate <option>-I${LOCALBASE}/include</option> and
	<option>-L${LOCALBASE}/lib</option> into references to
	<varname>BUILDLINK_DIR</varname>. The wrapper scripts also make
        native compiler on some operating systems look like GCC, so that
        packages that expect GCC won't require modifications to build with
        those native compilers.</para>
    </listitem>
  </orderedlist>

  <para>This normalizes the environment in which a package is built so that the
    package may be built consistently despite what other software may be
    installed. Please note that the normal system header and library paths,
    e.g. <filename>/usr/include</filename>,
    <filename>/usr/lib</filename>, etc., are always searched -- buildlink3 is
    designed to insulate the package build from non-system-supplied
    software. </para>

  <sect1 id="converting-to-buildlink3">
    <title>Converting packages to use buildlink3</title>

    <para>The process of converting packages to use the buildlink3
      framework (<quote>bl3ifying</quote>) is fairly straightforward.
      The things to keep in mind are:</para>

    <orderedlist>
      <listitem>
        <para> Ensure that the build always calls the wrapper scripts
          instead of the actual toolchain.  Some packages are tricky,
          and the only way to know for sure is the check
          <filename>${WRKDIR}/.work.log</filename> to see if the
          wrappers are being invoked. </para>
      </listitem>

      <listitem>
        <para> Don't override <varname>PREFIX</varname> from within
          the package Makefile, e.g. Java VMs, standalone shells,
          etc., because the code to symlink files into
          <filename>${BUILDLINK_DIR}</filename> looks for files
          relative to <quote>pkg_info -qp <replaceable>pkgname</replaceable></quote>.
	  </para>
      </listitem>

      <listitem>
        <para> Remember that <emphasis>only</emphasis> the
          <filename>buildlink3.mk</filename> files that you list in a
          package's Makefile are added as dependencies for that package.
	  </para>
      </listitem>
    </orderedlist>

    <para> If a dependency on a particular package is required for its libraries and
      headers, then we replace: </para>

<programlisting>
    DEPENDS+=   foo>=1.1.0:../../category/foo
</programlisting>

    <para>with</para>

<programlisting>
    .include "../../category/foo/buildlink3.mk"
</programlisting>

    <para>The buildlink3.mk files usually define the required dependencies.
    If you need a newer version of the dependency when using buildlink3.mk
    files, then you can define it in your Makefile; for example:
    </para>

<programlisting>
    BUILDLINK_API_DEPENDS.foo+=   foo>=1.1.0
    .include "../../category/foo/buildlink3.mk"
</programlisting>

    <para>There are several <filename>buildlink3.mk</filename>
      files in <filename>pkgsrc/mk</filename>
      that handle special package issues:</para>

    <itemizedlist>
      <listitem>
	<para><filename>bdb.buildlink3.mk</filename> chooses either
          the native or a pkgsrc Berkeley DB implementation based on
          the values of <varname>BDB_ACCEPTED</varname> and
          <varname>BDB_DEFAULT</varname>.</para>
      </listitem>

      <listitem>
	<para><filename>curses.buildlink3.mk</filename>: If the system
          comes with neither Curses nor NCurses, this will take care
	  to install the <filename
	    role="pkg">devel/ncurses</filename> package.</para>
      </listitem>

      <listitem>
	<para><filename>krb5.buildlink3.mk</filename> uses the value
          of <varname>KRB5_ACCEPTED</varname> to choose between
	  adding a dependency on Heimdal or MIT-krb5 for packages that
          require a Kerberos 5 implementation.</para>
      </listitem>

      <listitem>
	<para><filename>motif.buildlink3.mk</filename> checks
	  for a system-provided
	  Motif installation or adds a dependency on <filename
	    role="pkg">x11/lesstif</filename> or
	  <filename role="pkg">x11/openmotif</filename>.</para>
      </listitem>

      <listitem>
	<para><filename>oss.buildlink3.mk</filename> defines several
	  variables that may be used by packages that use the
	  Open Sound System (OSS) API.</para>
      </listitem>

      <listitem>
	<para><filename>pgsql.buildlink3.mk</filename> will accept
        either Postgres 7.3 or 7.4, whichever is found installed. See
        the file for more information. </para>
      </listitem>

      <listitem>
	<para><filename>pthread.buildlink3.mk</filename> uses the value of
	  <varname>PTHREAD_OPTS</varname> and checks for native pthreads or adds
	  a dependency on <filename
	    role="pkg">devel/pth</filename> as needed.</para>
      </listitem>

      <listitem>
	<para><filename>xaw.buildlink3.mk</filename> uses the value of
	  <varname>XAW_TYPE</varname> to choose a particular Athena widgets
	  library.</para>
      </listitem>
    </itemizedlist>

    <para>The comments in those <filename>buildlink3.mk</filename>
      files provide a more complete
      description of how to use them properly.</para>
  </sect1>

  <sect1 id="creating-buildlink3.mk">
    <title>Writing <filename>buildlink3.mk</filename> files</title>

    <para> A package's <filename>buildlink3.mk</filename> file is
      included by Makefiles to indicate the need to compile and link
      against header files and libraries provided by the package.  A
      <filename>buildlink3.mk</filename> file should always provide
      enough information to add the correct type of dependency
      relationship and include any other
      <filename>buildlink3.mk</filename> files that it needs to find
      headers and libraries that it needs in turn.</para>

    <para> To generate an initial <filename>buildlink3.mk</filename>
      file for further editing, Rene Hexel's <filename
        role="pkg">pkgtools/createbuildlink</filename>
      package is highly recommended.  For most packages, the following
      command will generate a good starting point for
      <filename>buildlink3.mk</filename> files:</para>

    <screen>&cprompt; <userinput>cd pkgsrc/<replaceable>category</replaceable>/<replaceable>pkgdir</replaceable>
&cprompt; createbuildlink &gt;buildlink3.mk</userinput></screen>

    <sect2 id="anatomy-of-bl3">
      <title>Anatomy of a buildlink3.mk file</title>

      <para>The following real-life example
        <filename>buildlink3.mk</filename> is taken
        from <filename>pkgsrc/graphics/tiff</filename>:</para>

<programlisting>
    # &#36;NetBSD: buildlink3.mk,v 1.7 2004/03/18 09:12:12 jlam Exp &#36;

    BUILDLINK_DEPTH:=       ${BUILDLINK_DEPTH}+
    TIFF_BUILDLINK3_MK:=    ${TIFF_BUILDLINK3_MK}+

    .if !empty(BUILDLINK_DEPTH:M+)
    BUILDLINK_DEPENDS+=     tiff
    .endif

    BUILDLINK_PACKAGES:=    ${BUILDLINK_PACKAGES:Ntiff}
    BUILDLINK_PACKAGES+=    tiff

    .if !empty(TIFF_BUILDLINK3_MK:M+)
    BUILDLINK_API_DEPENDS.tiff+=        tiff>=3.6.1
    BUILDLINK_PKGSRCDIR.tiff?=      ../../graphics/tiff
    .endif  # TIFF_BUILDLINK3_MK

    .include "../../devel/zlib/buildlink3.mk"
    .include "../../graphics/jpeg/buildlink3.mk"

    BUILDLINK_DEPTH:=       ${BUILDLINK_DEPTH:S/+$//}
</programlisting>

      <para> The header and footer manipulate
        <varname>BUILDLINK_DEPTH</varname>, which is common across all
        <filename>buildlink3.mk</filename> files and is used to track
        at what depth we are including
        <filename>buildlink3.mk</filename> files.</para>

      <para> The first section controls if the dependency on
        <replaceable>pkg</replaceable> is added.
        <varname>BUILDLINK_DEPENDS</varname> is the global list of
        packages for which dependencies are added by
        buildlink3.</para>

      <para> The second section advises pkgsrc that the
        <filename>buildlink3.mk</filename> file for
        <replaceable>pkg</replaceable> has been included at some point.
        <varname>BUILDLINK_PACKAGES</varname> is the global list of
        packages for which <filename>buildlink3.mk</filename> files
        have been included.  It must <emphasis>always</emphasis> be
        appended to within a <filename>buildlink3.mk</filename>
        file. </para>

      <para> The third section is protected from multiple inclusion
        and controls how the dependency on <replaceable>pkg</replaceable> is
        added.  Several important variables are set in the section:
        </para>

      <itemizedlist>
        <listitem>
	  <para> <varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>
	    is the actual dependency recorded in the installed
            package; this should always be set using
	    <command>+=</command> to ensure that
            we're appending to any pre-existing list of values.  This
            variable should be set to the first version of the
            package that had the last change in the major number of a
            shared library or that had a major API change.
	    </para>
	</listitem>

        <listitem>
	  <para> <varname>BUILDLINK_PKGSRCDIR.<replaceable>pkg</replaceable></varname>
	    is the location of the <replaceable>pkg</replaceable>
	    pkgsrc directory.</para>
	</listitem>

        <listitem>
	  <para>
	    <varname>BUILDLINK_DEPMETHOD.<replaceable>pkg</replaceable></varname>
	    (not shown above) controls whether we use
	    <varname>BUILD_DEPENDS</varname> or
	    <varname>DEPENDS</varname> to add the dependency on
	    <replaceable>pkg</replaceable>.
	    The build dependency is selected by setting
	    <varname>BUILDLINK_DEPMETHOD.<replaceable>pkg</replaceable></varname>
	    to <quote>build</quote>.  By default, the
	    full dependency is used.
	    </para>
	</listitem>

        <listitem>
	  <para>
	    <varname>BUILDLINK_INCDIRS.<replaceable>pkg</replaceable></varname>
	    and
	    <varname>BUILDLINK_LIBDIRS.<replaceable>pkg</replaceable></varname>
	    (not shown above) are lists of subdirectories of
	    <filename>${BUILDLINK_PREFIX.<replaceable>pkg</replaceable>}</filename>
	    to add to the header and library search paths.  These
	    default to <quote>include</quote> and <quote>lib</quote>
	    respectively. </para>
	</listitem>

        <listitem>
	  <para>
	    <varname>BUILDLINK_CPPFLAGS.<replaceable>pkg</replaceable></varname>
	    (not shown above) is the list of preprocessor flags to add
	    to <varname>CPPFLAGS</varname>, which are passed on to the
	    configure and build phases.  The <quote>-I</quote> option
	    should be avoided and instead be handled using
	    <varname>BUILDLINK_INCDIRS.<replaceable>pkg</replaceable></varname> as
	    above.</para>
	</listitem>
      </itemizedlist>

      <para> The following variables are all optionally defined within
	this second section (protected against multiple inclusion) and
	control which package files are symlinked into
	<filename>${BUILDLINK_DIR}</filename> and how their names are
	transformed during the symlinking: </para>

      <itemizedlist>
        <listitem>
	  <para>
	    <varname>BUILDLINK_FILES.<replaceable>pkg</replaceable></varname>
	    (not shown above) is a shell glob pattern relative to
	    <filename>${BUILDLINK_PREFIX.<replaceable>pkg</replaceable>}</filename>
	    to be symlinked into
	    <filename>${BUILDLINK_DIR}</filename>,
	    e.g. <filename>include/*.h</filename>. </para>
	</listitem>

        <listitem>
	  <para>
	    <varname>BUILDLINK_FILES_CMD.<replaceable>pkg</replaceable></varname>
	      (not shown above) is a shell pipeline that
	      outputs to stdout a list of files relative to
	      <filename>${BUILDLINK_PREFIX.<replaceable>pkg</replaceable>}</filename>.
	      The resulting files are to be symlinked
	      into <filename>${BUILDLINK_DIR}</filename>.  By default,
	      this takes the <filename>+CONTENTS</filename> of a
	      <replaceable>pkg</replaceable> and filters it through
	      <varname>${BUILDLINK_CONTENTS_FILTER.<replaceable>pkg</replaceable>}</varname>.
	    </para>
	</listitem>

        <listitem>
	  <para>
	    <varname>BUILDLINK_CONTENTS_FILTER.<replaceable>pkg</replaceable></varname>
	    (not shown above) is a filter command that filters
	    <filename>+CONTENTS</filename> input into a list of files
	    relative to
	    <filename>${BUILDLINK_PREFIX.<replaceable>pkg</replaceable>}</filename>
	    on stdout.  By default for overwrite packages,
	    <varname>BUILDLINK_CONTENTS_FILTER.<replaceable>pkg</replaceable></varname>
	    outputs the contents of the <filename>include</filename>
	    and <filename>lib</filename> directories in the package
	    <filename>+CONTENTS</filename>, and for pkgviews packages,
	    it outputs any libtool archives in
	    <filename>lib</filename> directories.
	    </para>
	</listitem>

        <listitem>
	  <para>
	    <varname>BUILDLINK_TRANSFORM.<replaceable>pkg</replaceable></varname>
	    (not shown above) is a list of sed arguments used to
	    transform the name of the source filename into a
	    destination filename, e.g. <command>-e
	    "s|/curses.h|/ncurses.h|g"</command>.
	    </para>
	</listitem>
      </itemizedlist>

      <para> The last section includes any
	    <filename>buildlink3.mk</filename> needed for
	    <replaceable>pkg</replaceable>'s library dependencies.
	    Including these <filename>buildlink3.mk</filename> files
	    means that the headers and libraries for these
	    dependencies are also symlinked into
	    <filename>${BUILDLINK_DIR}</filename>
	    whenever the <replaceable>pkg</replaceable>
	    <filename>buildlink3.mk</filename>
	    file is included.
	    </para>
    </sect2>

    <sect2 id="updating-buildlink-depends">
      <title>Updating <varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname> in <filename>buildlink3.mk</filename> files</title>

      <para>
        There are two situations that require increasing the
	dependency listed in
	<varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>
	after a package update: </para>

      <orderedlist>
        <listitem>
	  <para> if the sonames (major number of the library version)
	    of any installed shared libraries change. </para>
        </listitem>

        <listitem>
	  <para> if the API or interface to the header files change. </para>
        </listitem>
      </orderedlist>

      <para> In these cases,
	<varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>
	should be adjusted to require at least the new package
	version.  In some cases, the packages that depend on this new
	version may need their <varname>PKGREVISION</varname>s
	increased and, if they have <filename>buildlink3.mk</filename>
	files, their
	<varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>
	adjusted, too. This is needed so that binary packages made
	using it will require the correct package dependency and not
	settle for an older one which will not contain the necessary
	shared libraries. </para>

      <para> Please take careful consideration before adjusting
	<varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>
	as we don't want to cause unneeded package deletions and
	rebuilds.  In many cases, new versions of packages work just
	fine with older dependencies.  See <xref
	linkend="dependencies"/> for
	more information about dependencies on other packages,
	including the <varname>BUILDLINK_ABI_DEPENDS</varname> and
	<varname>ABI_DEPENDS</varname> definitions. </para>
    </sect2>
  </sect1>

  <sect1 id="writing-builtin.mk">
    <title>Writing <filename>builtin.mk</filename> files</title>

    <para>
      Some packages in pkgsrc install headers and libraries that
      coincide with headers and libraries present in the base system.
      Aside from a <filename>buildlink3.mk</filename> file, these
      packages should also include a <filename>builtin.mk</filename>
      file that includes the necessary checks to decide whether using
      the built-in software or the pkgsrc software is
      appropriate. </para>

    <para> The only requirements of a builtin.mk file for
      <replaceable>pkg</replaceable> are:
      </para>

    <orderedlist>
      <listitem>
        <para> It should set
          <varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
          to either <quote>yes</quote> or <quote>no</quote>
          after it is included.
	</para>
      </listitem>

      <listitem>
        <para> It should <emphasis>not</emphasis> override any
          <varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
          which is already set before the
          <filename>builtin.mk</filename> file is included.
	</para>
      </listitem>

      <listitem>
        <para> It should be written to allow multiple inclusion.  This
          is <emphasis>very</emphasis> important and takes careful
          attention to <filename>Makefile</filename> coding.
	</para>
      </listitem>
    </orderedlist>

    <sect2 id="anatomy-of-builtin.mk">
      <title>Anatomy of a <filename>builtin.mk</filename> file</title>

      <para>The following is the recommended template for builtin.mk
        files: </para>

<programlisting>
    .if !defined(IS_BUILTIN.foo)
    #
    # IS_BUILTIN.foo is set to "yes" or "no" depending on whether "foo"
    # genuinely exists in the system or not.
    #
    IS_BUILTIN.foo?=        no

    # BUILTIN_PKG.foo should be set here if "foo" is built-in and its package
    # version can be determined.
    #
    .  if !empty(IS_BUILTIN.foo:M[yY][eE][sS])
    BUILTIN_PKG.foo?=       foo-1.0
    .  endif
    .endif  # IS_BUILTIN.foo

    .if !defined(USE_BUILTIN.foo)
    USE_BUILTIN.foo?=       ${IS_BUILTIN.foo}
    .  if defined(BUILTIN_PKG.foo)
    .    for _depend_ in ${BUILDLINK_API_DEPENDS.foo}
    .      if !empty(USE_BUILTIN.foo:M[yY][eE][sS])
    USE_BUILTIN.foo!=                                                       \
          if ${PKG_ADMIN} pmatch '${_depend_}' ${BUILTIN_PKG.foo}; then     \
                  ${ECHO} "yes";                                            \
          else                                                              \
                  ${ECHO} "no";                                             \
          fi
    .      endif
    .    endfor
    .  endif
    .endif  # USE_BUILTIN.foo

    CHECK_BUILTIN.foo?=     no
    .if !empty(CHECK_BUILTIN.foo:M[nN][oO])
    #
    # Here we place code that depends on whether USE_BUILTIN.foo is set to
    # "yes" or "no".
    #
    .endif  # CHECK_BUILTIN.foo
</programlisting>

      <para> The first section sets
        <varname>IS_BUILTIN.<replaceable>pkg</replaceable></varname>
        depending on if <replaceable>pkg</replaceable> really exists
        in the base system.  This should not be a base system software
        with similar functionality to <replaceable>pkg</replaceable>;
        it should only be <quote>yes</quote> if the actual package is
        included as part of the base system.  This variable is only
        used internally within the <filename>builtin.mk</filename>
        file. </para>

      <para> The second section sets
        <varname>BUILTIN_PKG.<replaceable>pkg</replaceable></varname>
        to the version of <replaceable>pkg</replaceable> in the base
        system if it exists (if
        <varname>IS_BUILTIN.<replaceable>pkg</replaceable></varname>
        is <quote>yes</quote>).  This variable is only used internally
        within the <filename>builtin.mk</filename> file. </para>

      <para> The third section sets
        <varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
        and is <emphasis>required</emphasis> in all
        <filename>builtin.mk</filename> files.  The code in this
        section must make the determination whether the built-in
        software is adequate to satisfy the dependencies listed in
        <varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>.
        This is typically done by comparing
        <varname>BUILTIN_PKG.<replaceable>pkg</replaceable></varname>
        against each of the dependencies in
        <varname>BUILDLINK_API_DEPENDS.<replaceable>pkg</replaceable></varname>.
        <varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
        <emphasis>must</emphasis> be set to the correct value by the
        end of the <filename>builtin.mk</filename> file.  Note that
        <varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
        may be <quote>yes</quote> even if
        <varname>IS_BUILTIN.<replaceable>pkg</replaceable></varname>
        is <quote>no</quote> because we may make the determination
        that the built-in version of the software is similar enough to
        be used as a replacement. </para>

      <para> The last section is guarded by
        <varname>CHECK_BUILTIN.<replaceable>pkg</replaceable></varname>,
        and includes code that uses the value of
        <varname>USE_BUILTIN.<replaceable>pkg</replaceable></varname>
        set in the previous section.  This typically includes, e.g.,
        adding additional dependency restrictions and listing
        additional files to symlink into
        <filename>${BUILDLINK_DIR}</filename> (via
        <varname>BUILDLINK_FILES.<replaceable>pkg</replaceable></varname>).
        </para>
    </sect2>

    <sect2 id="native-or-pkgsrc-preference">
      <title>Global preferences for native or pkgsrc software</title>

      <para>
        When building packages, it's possible to choose whether to set
        a global preference for using either the built-in (native)
        version or the pkgsrc version of software to satisfy a
        dependency.  This is controlled by setting
        <varname>PREFER_PKGSRC</varname> and
        <varname>PREFER_NATIVE</varname>.  These variables take values
        of either <quote>yes</quote>, <quote>no</quote>, or a list of
        packages.  <varname>PREFER_PKGSRC</varname> tells pkgsrc to
        use the pkgsrc versions of software, while
        <varname>PREFER_NATIVE</varname> tells pkgsrc to use the
        built-in versions.  Preferences are determined by the most
        specific instance of the package in either
        <varname>PREFER_PKGSRC</varname> or
        <varname>PREFER_NATIVE</varname>.  If a package is specified
        in neither or in both variables, then
        <varname>PREFER_PKGSRC</varname> has precedence over
        <varname>PREFER_NATIVE</varname>.  For example, to require
        using pkgsrc versions of software for all but the most basic
        bits on a NetBSD system, you can set: </para>

<programlisting>
    PREFER_PKGSRC=  yes
    PREFER_NATIVE=  getopt skey tcp_wrappers
</programlisting>

      <para> A package <emphasis>must</emphasis> have a
        <filename>builtin.mk</filename>
        file to be listed in <varname>PREFER_NATIVE</varname>,
        otherwise it is simply ignored in that list.
	</para>
    </sect2>
  </sect1>
</chapter>