blob: 29c667e1eb69d6caabe243381503f1f05bded7c0 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
|
<!-- $NetBSD: regression.xml,v 1.2 2006/12/15 13:22:14 martti Exp $ -->
<chapter id="regression"> <?dbhtml filename="regression.html"?>
<title>Regression tests</title>
<para>The pkgsrc infrastructure consists of a large codebase,
and there are many corners where every little bit of a file is
well thought out, making pkgsrc likely to fail as soon as
anything is changed near those parts. To prevent most changes
from breaking anything, a suite of regression tests should go
along with every important part of the pkgsrc infrastructure.
This chapter describes how regression tests work in pkgsrc and
how you can add new tests.</para>
<sect1 id="regression.descr">
<title>The regression tests framework</title>
<para></para>
</sect1>
<sect1 id="regression.run">
<title>Running the regression tests</title>
<para>You first need to install the <filename
role="pkg">pkgtools/pkg_regress</filename> package, which
provides the <command>pkg_regress</command> command. Then you
can simply run that command, which will run all tests in the
<filename>regress</filename> category.</para>
</sect1>
<sect1 id="regression.new">
<title>Adding a new regression test</title>
<para>Every directory in the <filename>regress</filename>
category that contains a file called <filename>spec</filename>
is considered a regression test. This file is a shell program
that is included by the <command>pkg_regress</command> command.
The following functions can be overridden to suit your
needs.</para>
<sect2 id="regression.fun.override">
<title>Overridable functions</title>
<para>These functions do not take any parameters. They are all
called in <quote>set -e</quote> mode, so you should be careful
to check the exitcodes of any commands you run in the
test.</para>
<variablelist>
<varlistentry><term><varname>do_setup()</varname></term>
<listitem><para>This function prepares the environment for the
test. By default it does nothing.</para></listitem>
</varlistentry>
<varlistentry><term><varname>do_test()</varname></term>
<listitem><para>This function runs the actual test. By default,
it calls <varname>TEST_MAKE</varname> with the arguments
<varname>MAKEARGS_TEST</varname> and writes its output including
error messages into the file
<varname>TEST_OUTFILE</varname>.</para></listitem>
</varlistentry>
<varlistentry><term><varname>check_result()</varname></term>
<listitem><para>This function is run after the test and is
typically used to compare the actual output from the one that is
expected. It can make use of the various helper functions from
the next section.</para></listitem>
</varlistentry>
<varlistentry><term><varname>do_cleanup()</varname></term>
<listitem><para>This function cleans everything up after the
test has been run. By default it does nothing.</para></listitem>
</varlistentry>
</variablelist>
</sect2>
<sect2 id="regression.fun.helper">
<title>Helper functions</title>
<variablelist>
<varlistentry><term><varname>exit_status(expected)</varname></term>
<listitem><para>This function compares the exitcode of the
<command>do_test()</command> function with its first parameter.
If they differ, the test will fail.</para></listitem>
</varlistentry>
<varlistentry><term><varname>output_require(regex...)</varname></term>
<listitem><para>This function checks for each of its parameters
if the output from <command>do_test()</command> matches the
extended regular expression. If it does not, the test will
fail.</para></listitem>
</varlistentry>
<varlistentry><term><varname>output_prohibit(regex...)</varname></term>
<listitem><para>This function checks for each of its parameters
if the output from <command>do_test()</command> does
<emphasis>not</emphasis> match the extended regular expression.
If any of the regular expressions matches, the test will
fail.</para></listitem>
</varlistentry>
</variablelist>
</sect2>
</sect1>
</chapter>
|
