diff options
author | dmcmahill <dmcmahill> | 2001-02-01 08:47:29 +0000 |
---|---|---|
committer | dmcmahill <dmcmahill> | 2001-02-01 08:47:29 +0000 |
commit | ced09b5cbfe9356cb2476d70fb5b9b7a7565db4f (patch) | |
tree | de8d5c677a8c586e5c26ba9701a518c1adc06408 /mk | |
parent | 2dc5bfe4fb2c99d201f64ce06a232cfac7ccee47 (diff) | |
download | pkgsrc-ced09b5cbfe9356cb2476d70fb5b9b7a7565db4f.tar.gz |
This commit incorporates several speed improvements which have been tested
over several bulk builds on an alpha:
- At the start of a bulk build, 4 files are created to allow fast lookup of
various dependency tree things. These files are
.index == maps package name (foo-2.3) to directory (bar/foo)
.dependstree == contains the complete pkgsrc dependency tree in a tsort(1)
compatible format.
.depends == contains 1 line per package and lists all the build depends
for the package.
.supports == contains 1 line per package and lists all packages which
depend on this package.
- When a package fails to build, the list of all packages which depend upon
it is read from .supports. Each of those packages is immediately marked
as broken. This prevents us from trying to build those packages which can
have significant overhead if lots of other depends are installed before the
system notices the broken one.
In addition, the post-build postprocessing will now indicate that a package
is 'truely broken' (ie, bad PLIST, failed compilation) vs. a package which
is broken because it depends on a failed package. This assists in determining
where to focus our efforts in fixing broken packages.
- In the old approach, all packages are removed after each one is built. The
purpose was to a) conserve disk space, b) verify that all required dependencies
are in fact listed, and c) prevent conflicts. The problem was that often times
several packages in a row have similar depends. For example we might want to
build several perl packages in a row. In the old approach, we would install perl,
build the pkg, deinstall perl and continue with the next one.
In the new approach, when it is decided that a pkg is out of date and should be
rebuilt, the .depends file is used to obtain a list of pkgs we require. Then
and pkgs which are no longer needed are removed. This helps to minimize the total
number of pkg_add/pkg_delete's in a bulk build.
- Since the order of the build is controlled by the depends tree, all depends for a
given package will have been built by the time the pkg in question needs them.
If any of the depends failed to build, then the pkg which needs the failed one will
have been marked as broken. Given this, the complete depends list for a pkg is
read from .depends and the depends installed via pkg_add rather than relying on
recursive make calls to install the depends.
- while here, fix a few minor bugs
x - don't leave .make log files around when the build succeeds
x - make sure we refer to the correct report file in the email
x - use '.order' for the build order file instead of '.l'
x - use 'grep -c' instead of 'grep | wc -l'
Diffstat (limited to 'mk')
-rw-r--r-- | mk/bulk/bsd.bulk-pkg.mk | 168 | ||||
-rw-r--r-- | mk/bulk/build | 43 | ||||
-rw-r--r-- | mk/bulk/post-build | 36 |
3 files changed, 193 insertions, 54 deletions
diff --git a/mk/bulk/bsd.bulk-pkg.mk b/mk/bulk/bsd.bulk-pkg.mk index 0bec34740f7..b61622da54c 100644 --- a/mk/bulk/bsd.bulk-pkg.mk +++ b/mk/bulk/bsd.bulk-pkg.mk @@ -1,4 +1,4 @@ -# $Id: bsd.bulk-pkg.mk,v 1.16 2001/01/28 21:20:48 dmcmahill Exp $ +# $NetBSD: bsd.bulk-pkg.mk,v 1.17 2001/02/01 08:47:29 dmcmahill Exp $ # # Copyright (c) 1999, 2000 Hubert Feyrer <hubertf@netbsd.org> @@ -45,17 +45,52 @@ LS?= ls WC?= wc -# Shall we keep the package installed after a "make bulk-package"? -# Set to "yes" by bulk-install. -KEEP?= no +# Shall we remove any packages which are installed, but not required +# to build this package +PRECLEAN?= yes -# This file exists to mark a package as broken .ifdef OBJMACHINE +# This file exists to mark a package as broken BROKENFILE?= .broken.${MACHINE} + +# This file is where the log of the build goes BUILDLOG?= .make.${MACHINE} + +# This is a top level file which lists the entire pkgsrc depends tree in the format: +# foo/bar devel/libfoo +# meaning 'foo/bar' is requied to build 'devel/libfoo' +# this is in the format needed by tsort(1) +DEPENDSTREEFILE?= ${PKGSRCDIR}/.dependstree.${MACHINE} + +# This is a top level file which lists the entire pkgsrc depends tree in the format: +# foo/bar depends on: devel/libfoo devel/libbar devel/baz ..... +# ie, to build foo/bar we need devel/libfoo devel/libbar devel/baz ... installed +DEPENDSFILE?= ${PKGSRCDIR}/.depends.${MACHINE} + +# This is a top level file which lists the entire pkgsrc depends tree in the format: +# devel/libfoo is depended upon by: foo/bar graphics/gtkfoo ... +# ie, to build foo/bar we need devel/libfoo to be installed. +# to build graphics/gtkfoo we need devel/libfoo to be installed +SUPPORTSFILE?= ${PKGSRCDIR}/.supports.${MACHINE} + +# This is a top level file which cross-references each package name and pkg directory +# in the format: +# devel/libfoo libfoo-1.3 +INDEXFILE?= ${PKGSRCDIR}/.index.${MACHINE} + +# file containing a list of all the packages in the correct order for a bulk build. +# the correct order is one where packages that are required by others are built +# before the packages which require them. +ORDERFILE?= ${PKGSRCDIR}/.order.${MACHINE} + .else -BROKENFILE?= .broken -BUILDLOG?= .make +BROKENFILE?= .broken +BUILDLOG?= .make +DEPENDSTREEFILE?= ${PKGSRCDIR}/.dependstree +DEPENDSFILE?= ${PKGSRCDIR}/.depends +SUPPORTSFILE?= ${PKGSRCDIR}/.supports +INDEXFILE?= ${PKGSRCDIR}/.index +ORDERFILE?= ${PKGSRCDIR}/.order .endif @@ -108,7 +143,7 @@ bulk-package: exit 1; \ fi @( \ - if [ "${KEEP}" = "no" ]; then \ + if [ "${PRECLEAN}" = "yes" ]; then \ ${ECHO_MSG} '' ; \ ${ECHO_MSG} '' ; \ ${ECHO_MSG} '###' ; \ @@ -126,9 +161,6 @@ bulk-package: if [ $$uptodate = 1 ]; then \ ( if [ $$installed = 1 ]; then \ ${ECHO_MSG} "BULK> Package ${PKGNAME} is upto-date, and still installed" ; \ - ${ECHO_MSG} " removing installed package." ; \ - ${ECHO_MSG} ${MAKE} deinstall DEINSTALLDEPENDS=YES ; \ - ${DO} ${MAKE} deinstall DEINSTALLDEPENDS=YES ; \ else \ ${ECHO_MSG} "BULK> Nothing to be done." ; \ fi \ @@ -136,47 +168,109 @@ bulk-package: else \ ( if [ $$installed = 1 ]; then \ ${ECHO_MSG} "BULK> Removing outdated (installed) package ${PKGNAME} first." ; \ - ${ECHO_MSG} ${MAKE} deinstall DEINSTALLDEPENDS=YES ; \ - ${DO} ${MAKE} deinstall DEINSTALLDEPENDS=YES ; \ + ${ECHO_MSG} ${MAKE} deinstall ; \ + ${DO} ${MAKE} deinstall ; \ fi ; \ - if [ -f ${BUILDLOG} ]; then \ - ${ECHO_MSG} "BULK> Continuing previous build..." ; \ - else \ + if [ -f ${PKGFILE} ]; then \ ${ECHO_MSG} "BULK> Removing old binary package..." ; \ ${ECHO_MSG} ${RM} -f ${PKGFILE} ; \ ${DO} ${RM} -f ${PKGFILE} ; \ + fi; \ ${ECHO_MSG} "BULK> Full rebuild in progress..." ; \ - ${ECHO_MSG} ${MAKE} -k clean CLEANDEPENDS=YES ; \ - ${DO} ${MAKE} -k clean CLEANDEPENDS=YES ; \ - fi ; \ - ${ECHO_MSG} ${MAKE} install-depends '(${PKGNAME})' 2>&1; \ - ${DO} ( ${MAKE} install-depends 2>&1 ); \ + ${ECHO_MSG} "BULK> Cleaning package and its depends" ;\ + thisdir=`${GREP} " ${PKGNAME} " ${INDEXFILE} | ${AWK} '{print $$1}'` ;\ + for pkgdir in $$thisdir `${GREP} "^$$thisdir " ${DEPENDSFILE} | ${SED} -e 's;^.*:;;g'`; do \ + ${ECHO_MSG} "(cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} clean)" ; \ + ${DO} (cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} clean) ; \ + done ;\ + if [ "${PRECLEAN}" = "yes" ]; then \ + ${ECHO_MSG} "BULK> Removing installed packages which are not needed to build ${PKGNAME}" ; \ + thisdir=`${GREP} " ${PKGNAME} " ${INDEXFILE} | ${AWK} '{print $$1}'` ;\ + for pkgname in `${PKG_INFO} | ${AWK} '{print $$1}'` ; \ + do \ + pkgdir=`${GREP} " $$pkgname " ${INDEXFILE} | ${AWK} '{print $$1}'` ;\ + if `${PKG_INFO} -qe $$pkgname` ; then \ + if ! `${EGREP} "^$$thisdir.* $$pkgdir( |$$)" ${DEPENDSFILE} >/dev/null 2>&1` ; then \ + ${ECHO_MSG} "BULK> ${PKGNAME} does not require installed package $$pkgname ($$pkgdir) to build." ;\ + ${ECHO_MSG} "BULK> Deinstalling $$pkgname" ;\ + ${ECHO_MSG} ${PKG_DELETE} -r $$pkgname ; \ + ${DO} ${PKG_DELETE} -r $$pkgname || true ; \ + if `${PKG_INFO} -qe $$pkgname` ; then \ + ${ECHO_MSG} "BULK> $$pkgname ($$pkgdir) did not nicely deinstall. Forcing the deinstall." ;\ + ${ECHO_MSG} ${PKG_DELETE} -f $$pkgname ; \ + ${DO} ${PKG_DELETE} -f $$pkgname || true ; \ + fi ;\ + else \ + ${ECHO_MSG} "BULK> ${PKGNAME} requires installed package $$pkgname ($$pkgdir) to build." ;\ + ${ECHO_MSG} "BULK> Keeping $$pkgname" ;\ + fi ;\ + else \ + ${ECHO_MSG} "BULK> package $$pkgname ($$pkgdir) is no longer installed" ;\ + fi ;\ + done ; \ + fi ;\ + thisdir=`${GREP} " ${PKGNAME} " ${INDEXFILE} | ${AWK} '{print $$1}'` ;\ + ${ECHO_MSG} "BULK> Installing packages which are required to build ${PKGNAME}." ;\ + for pkgdir in `${GREP} "^$$thisdir " ${DEPENDSFILE} | ${SED} -e 's;^.*:;;g'`; do \ + pkgname=`${GREP} "^$$pkgdir " ${INDEXFILE} | ${AWK} '{print $$2}'` ; \ + pkgfile=${PACKAGES}/All/$${pkgname}.tgz ;\ + if ! `${PKG_INFO} -qe $$pkgname` ; then \ + ${ECHO_MSG} "BULK> " ${PKG_ADD} $$pkgfile ; \ + ${DO} ${PKG_ADD} $$pkgfile ; \ + else \ + ${ECHO_MSG} "BULK> Required package $$pkgname ($$pkgdir) is already installed" ; \ + fi ;\ + done ;\ ${ECHO_MSG} ${MAKE} package '(${PKGNAME})' 2>&1 ; \ ${DO} ( ${MAKE} package 2>&1 ); \ ) 2>&1 | tee -a ${BUILDLOG} ; \ if [ -f ${PKGFILE} ]; then \ ${RM} ${BUILDLOG} ; \ else \ - ( \ ${MV} ${BUILDLOG} ${BROKENFILE} ; \ + ( \ ${ECHO_MSG} "BULK> ${PKGNAME} was marked as broken:" ; \ ${LS} -la ${BROKENFILE} ; \ - nerrors=`${GREP} '^\*\*\* Error code' ${BROKENFILE} | ${WC} -l`; \ - ${ECHO_MSG} "$$nerrors ${PKGPATH}/${BROKENFILE}" >>../../${BROKENFILE}; \ - ) 2>&1 | tee -a ${BUILDLOG}; \ - fi ; \ - ${ECHO_MSG} ${MAKE} clean CLEANDEPENDS=YES ; \ - ${DO} ${MAKE} clean CLEANDEPENDS=YES ; \ - if [ "${KEEP}" = "no" ]; then \ - ${ECHO_MSG} ${MAKE} deinstall DEINSTALLDEPENDS=YES ; \ - ${DO} ${MAKE} deinstall DEINSTALLDEPENDS=YES ; \ - ${ECHO_MSG} ${PKG_DELETE} -rR \\* ; \ - ${DO} ${PKG_DELETE} -rR \* >/dev/null 2>&1 || ${TRUE} ; \ + ${ECHO_MSG} ${MAKE} deinstall ; \ + ${DO} ${MAKE} deinstall ; \ + nerrors=`${GREP} -c '^\*\*\* Error code' ${BROKENFILE} || true`; \ + ${ECHO_MSG} " $$nerrors ${PKGPATH}/${BROKENFILE}" >> ${PKGSRCDIR}/${BROKENFILE}; \ + ${ECHO_MSG} "BULK> Marking all packages which depend upon ${PKGNAME} as broken:"; \ + thisdir=`${GREP} " ${PKGNAME} " ${INDEXFILE} | ${AWK} '{print $$1}'` ;\ + for pkgdir in `${GREP} "^$$thisdir " ${SUPPORTSFILE} | ${SED} -e 's;^.*:;;g'`; do \ + pkgname=`${GREP} "^$$pkgdir " ${INDEXFILE} | ${AWK} '{print $$2}'` ;\ + ${ECHO_MSG} "BULK> marking package that requires ${PKGNAME} as broken: $$pkgname ($$pkgdir)";\ + pkgerr="-1"; \ + pkgignore=`(cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} show-var VARNAME=IGNORE)`; \ + if [ ! -z "$$pkgignore" -a ! -f ${PKGSRCDIR}/$$pkgdir/${BROKENFILE} ]; then \ + ${ECHO_MSG} "BULK> $$pkgname ($$pkgdir) may not be packaged because:" >> ${PKGSRCDIR}/$$pkgdir/${BROKENFILE};\ + ${ECHO_MSG} "BULK> $$pkgignore" >> ${PKGSRCDIR}/$$pkgdir/${BROKENFILE};\ + if [ -z "`(cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} show-var VARNAME=BROKEN)`" ]; then \ + pkgerr="0"; \ + else \ + pkgerr="1"; \ + fi; \ + fi; \ + ${ECHO_MSG} "BULK> $$pkgname ($$pkgdir) is broken because it depends upon ${PKGNAME} ($$thisdir) which is broken." \ + >> ${PKGSRCDIR}/$$pkgdir/${BROKENFILE};\ + if ! `${GREP} " $$pkgdir/${BROKENFILE}" ${PKGSRCDIR}/${BROKENFILE} >/dev/null 2>&1` ; then \ + ${ECHO} " $$pkgerr $$pkgdir/${BROKENFILE}" >> ${PKGSRCDIR}/${BROKENFILE} ;\ + fi ;\ + done \ + ) 2>&1 | tee -a ${BROKENFILE}; \ fi ; \ + ${ECHO_MSG} "BULK> Cleaning packages and its depends" ;\ + thisdir=`${GREP} " ${PKGNAME} " ${INDEXFILE} | ${AWK} '{print $$1}'` ;\ + for pkgdir in $$thisdir `${GREP} "^$$thisdir " ${DEPENDSFILE} | ${SED} -e 's;^.*:;;g'`; do \ + ${ECHO_MSG} "(cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} clean)" ; \ + ${DO} (cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} clean) ; \ + done ;\ fi @if [ ! -f ${PKGFILE} ]; then \ - ${ECHO_MSG} "BULK>" Build for ${PKGNAME} was not successful, aborting. | tee -a ${BUILDLOG} ; \ + ${ECHO_MSG} "BULK>" Build for ${PKGNAME} was not successful, aborting. | tee -a ${BROKENFILE} ; \ false; \ + else \ + ${RM} -f ${BUILDLOG} ;\ fi # Install pkg - if possible from binary pkg (i.e. available & upto date) @@ -185,12 +279,12 @@ bulk-package: # been modified and need rebuilding. bulk-install: @if [ `${MAKE} bulk-check-uptodate REF=${PKGFILE}` = 1 ]; then \ - if ! ${PKG_INFO} -e ${PKGNAME} ; then \ + if ! ${PKG_INFO} -qe ${PKGNAME} ; then \ ${DO} ${MAKE} install-depends ; \ ${ECHO_MSG} "BULK> " ${PKG_ADD} ${PKGFILE} ; \ ${DO} ${PKG_ADD} ${PKGFILE} ; \ fi ; \ else \ - ${ECHO_MSG} ${MAKE} bulk-package KEEP=yes; \ - ${DO} ${MAKE} bulk-package KEEP=yes; \ + ${ECHO_MSG} ${MAKE} bulk-package PRECLEAN=no; \ + ${DO} ${MAKE} bulk-package PRECLEAN=no; \ fi diff --git a/mk/bulk/build b/mk/bulk/build index e31cf0bea2e..f831e50143a 100644 --- a/mk/bulk/build +++ b/mk/bulk/build @@ -1,5 +1,5 @@ #!/bin/sh -# $NetBSD: build,v 1.7 2000/12/31 17:38:32 dmcmahill Exp $ +# $NetBSD: build,v 1.8 2001/02/01 08:47:29 dmcmahill Exp $ # # Do builk build # @@ -42,15 +42,50 @@ else sh mk/bulk/pre-build # make veryveryclean :) fi +cd pkgtools/pkglint +DEPENDSTREEFILE=`make show-var VARNAME=DEPENDSTREEFILE` +DEPENDSFILE=`make show-var VARNAME=DEPENDSFILE` +SUPPORTSFILE=`make show-var VARNAME=SUPPORTSFILE` +INDEXFILE=`make show-var VARNAME=INDEXFILE` +ORDERFILE=`make show-var VARNAME=ORDERFILE` +cd ${USR_PKGSRC} + # Figure out optimal build order, and build if [ "$1" != "restart" ]; then - sh mk/bulk/printdepends | tsort | tee .l + echo "Building dependency tree (this may take a while)..." + sh mk/bulk/printdepends > $DEPENDSTREEFILE + + echo "Sorting build order..." + tsort $DEPENDSTREEFILE > $ORDERFILE + + echo "Generating up and down dependency files..." + ./mk/bulk/tflat -u $DEPENDSTREEFILE > $SUPPORTSFILE + ./mk/bulk/tflat -d $DEPENDSTREEFILE > $DEPENDSFILE + + echo "Generating package name <=> package directory cross reference file (this may take a while)..." + sh mk/bulk/printindex > $INDEXFILE + fi + +echo "Starting actual build using the order specified in $ORDERFILE..." nice -n 20 make \ SPECIFIC_PKGS=1 \ - GROUP_SPECIFIC_PKGS="`cat .l | tr '\012' ' '`" \ + GROUP_SPECIFIC_PKGS="`cat $ORDERFILE | tr '\012' ' '`" \ bulk-package | sed 's/^/'`uname -p`'> /g' -rm .l + +echo "Build finished. Removing all installed packages left over from build..." + +for pkg in `pkg_info` +do + if `pkg_info -qe $pkg`; then + echo pkg_delete -r $pkg + pkg_delete -rR $pkg + fi +done + +echo "Post processing bulk build results..." + +#rm $DEPENDSTREEFILE $DEPENDSFILE $SUPPORTSFILE $INDEXFILE $ORDERFILE # Perl was wiped, reinstall it! ( cd lang/perl5-base ; make bulk-install ) diff --git a/mk/bulk/post-build b/mk/bulk/post-build index 62e0b6838bc..f24c26b5983 100644 --- a/mk/bulk/post-build +++ b/mk/bulk/post-build @@ -1,5 +1,5 @@ #!/usr/pkg/bin/perl -# $NetBSD: post-build,v 1.8 2000/12/31 17:38:32 dmcmahill Exp $ +# $NetBSD: post-build,v 1.9 2001/02/01 08:47:29 dmcmahill Exp $ # # Collect stuff after a pkg bulk build # @@ -35,6 +35,7 @@ getconf("USR_PKGSRC"); # "/usr/pkgsrc" getconf("osrev"); # `uname -r` getconf("arch"); # `uname -m` chomp($date=`date`); +$reportf=basename($REPORT); # extract the name of the files used for the build log and broken build log. # these have defaults set by bsd.bulk-pkg.mk and may be overridden in /etc/mk.conf @@ -83,7 +84,7 @@ EOHTML ; # some nifty postprocessing of $FTP/.broken* -> HTML here -$nunpackaged = $nbroken = 0; +$nunpackaged = $nbroken = $nbrokendep = 0; chdir("$FTP"); open(BF,"sort +1 $BROKENFILE |") or die "can't open .broken-file '$BROKENFILE'"; while (<BF>) { @@ -98,7 +99,11 @@ while (<BF>) { $color = "red"; $state = "build broken"; $nbroken++; - } else { + } elsif ($nerrors == -1) { + $color = "orange"; + $state = "broken depends"; + $nbrokendep++; + } else { $color = "yellow"; $state = "not packaged"; $nunpackaged++; @@ -122,7 +127,7 @@ while (<BF>) { ($category, $dbfeed_pkg) = split('/', $pkg); - if ($nerrors > 0) { + if ($nerrors != 0) { printf("%-23s $state\t($who, $maintainer)\n", "$pkg:") if $verbose; } @@ -130,16 +135,19 @@ while (<BF>) { close(BF); -$ntotal=$nunpackaged+$nbroken; +$nbrokentot=$nbroken+$nbrokendep; +$ntotal=$nunpackaged+$nbroken+$nbrokendep; print HTML <<EOHTML </TABLE> <HR> <TABLE> -<TR><TD> Packages not packaged: <TD ALIGHT=RIGHT> <FONT COLOR=yellow>$nunpackaged</FONT> -<TR><TD> Packages broken: <TD ALIGHT=RIGHT> <FONT COLOR=red>$nbroken</FONT> -<TR><TD> Total: <TD ALIGHT=RIGHT> $ntotal +<TR><TD> Packages not packaged: <TD ALIGHT=RIGHT> <FONT COLOR=yellow>$nunpackaged</FONT> +<TR><TD> Packages really broken: <TD ALIGHT=RIGHT> <FONT COLOR=red>$nbroken</FONT> +<TR><TD> Pkgs broken due to them: <TD ALIGHT=RIGHT> <FONT COLOR=blue>$nbrokendep</FONT> +<TR><TD> Total broken: <TD ALIGHT=RIGHT> $nbrokentot +<TR><TD> Total: <TD ALIGHT=RIGHT> $ntotal </TABLE> <HR> @@ -205,7 +213,7 @@ close(HTML); EOOUT ; while (<IN>) { - print OUT "<A HREF=\"/${FTPURL}/leftovers-${arch}$_\">$_</A>\n"; + print OUT "<A HREF=\"${FTPHOST}/${FTPURL}/leftovers-${arch}$_\">$_</A>\n"; } print OUT <<EOOUT2 </PRE> @@ -219,11 +227,13 @@ EOOUT2 if ($verbose) { print "\n"; - print "Packages not packaged: $nunpackaged\n"; - print "Packages broken: $nbroken\n"; - print "Total: $ntotal\n"; + print "Packages not packaged: $nunpackaged\n"; + print "Packages really broken: $nbroken\n"; + print "Pkgs broken due to them: $nbrokendep\n"; + print "Total broken: $nbrokentot\n"; + print "Total: $ntotal\n"; print "\n"; - print "See $FTPHOST/$FTPURL/broken.html\n"; + print "See $FTPHOST/$FTPURL/$reportf\n"; print "for logs of builds broken or not resulting in a binary pkg.\n"; print "\n"; print "\n"; |