diff options
Diffstat (limited to 'doc/guide/files/regression.xml')
| -rw-r--r-- | doc/guide/files/regression.xml | 111 |
1 files changed, 111 insertions, 0 deletions
diff --git a/doc/guide/files/regression.xml b/doc/guide/files/regression.xml new file mode 100644 index 00000000000..48a9b35f0ef --- /dev/null +++ b/doc/guide/files/regression.xml @@ -0,0 +1,111 @@ +<!-- $NetBSD: regression.xml,v 1.1 2006/05/10 20:56:00 rillig Exp $ --> + +<chapter id="regression"> <?dbhtml filename="regression.html"?> +<title>Regression tests</title> + + <para>The pkgsrc infrastructure consists of a large codebase, + and there are many corners where every little bit of a file is + well thought out, making pkgsrc likely to fail as soon as + anything is changed near those parts. To prevent most changes + from breaking anything, a suite of regression tests should go + along with every important part of the pkgsrc infrastructure. + This chapter describes how regression tests work in pkgsrc and + how you can add new tests.</para> + +<sect1 id="regression.descr"> +<title>The regression tests framework</title> + +<para></para> + +</sect1> + +<sect1 id="regression.run"> +<title>Running the regression tests</title> + + <para>You first need to install the <filename + role="pkg">pkgtools/pkg_regress</filename> package, which + provides the <command>pkg_regress</command> command. Then you + can simply run that command, which will run all tests in the + <filename>regress</filename> category.</para> + +</sect1> + +<sect1 id="regression.new"> +<title>Adding a new regression test</title> + + <para>Every directory in the <filename>regress</filename> + category that contains a file called <filename>spec</filename> + is considered a regression test. This file is a shell program + that is included by the <command>pkg_regress</command> command. + The following functions can be overridden to suit your + needs.</para> + +<sect2 id="regression.fun.override"> +<title>Overridable functions</title> + + <para>These functions do not take any parameters. They are all + called in <quote>set -e</quote> mode, so you should be careful + to check the exitcodes of any commands you run in the + test.</para> + + <variablelist> + + <varlistentry><term><varname>do_setup()</varname></term> + <listitem><para>This function prepares the environment for the + test. By default it does nothing.</para></listitem> + </varlistentry> + + <varlistentry><term><varname>do_test()</varname></term> + <listitem><para>This function runs the actual test. By default, + it calls <varname>TEST_MAKE</varname> with the arguments + <varname>MAKEARGS_TEST</varname> and writes its output including + error messages into the file + <varname>TEST_OUTFILE</varname>.</para></listitem> + </varlistentry> + + <varlistentry><term><varname>check_result()</varname></term> + <listitem><para>This function is run after the test and is + typically used to compare the actual output from the one that is + expected. It can make use of the various helper functions from + the next section.</para></listitem> + </varlistentry> + + <varlistentry><term><varname>do_cleanup()</varname></term> + <listitem><para>This function cleans everything up after the + test has been run. By default it does nothing.</para></listitem> + </varlistentry> + + </variablelist> +</sect2> + +<sect2 id="regression.fun.helper"> +<title>Helper functions</title> + +<variablelist> + + <varlistentry><term><varname>exit_status(expected)</varname></term> + <listitem><para>This function compares the exitcode of the + <command>do_test()</command> function with its first parameter. + If they differ, the test will fail.</para></listitem> + </varlistentry> + + <varlistentry><term><varname>output_require(regex...)</varname></term> + <listitem><para>This function checks for each of its parameters + if the output from <command>do_test()</command> matches the + extended regular expression. If it does not, the test will + fail.</para></listitem> + </varlistentry> + + <varlistentry><term><varname>output_prohibit(regex...)</varname></term> + <listitem><para>This function checks for each of its parameters + if the output from <command>do_test()</command> does + <emphasis>not</emphasis> match the extended regular expression. + If any of the regular expressions matches, the test will + fail.</para></listitem> + </varlistentry> + + </variablelist> + +</sect2> +</sect1> +</chapter> |