1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
|
<!-- $NetBSD: regression.xml,v 1.4 2016/07/10 07:46:29 rillig Exp $ -->
<chapter id="regression"> <?dbhtml filename="regression.html"?>
<title>Regression tests</title>
<para>The pkgsrc infrastructure consists of a large codebase,
and there are many corners where every little bit of a file is
well thought out, making pkgsrc likely to fail as soon as
anything is changed near those parts. To prevent most changes
from breaking anything, a suite of regression tests should go
along with every important part of the pkgsrc infrastructure.
This chapter describes how regression tests work in pkgsrc and
how you can add new tests.</para>
<sect1 id="regression.run">
<title>Running the regression tests</title>
<para>You first need to install the <filename
role="pkg">pkgtools/pkg_regress</filename> package, which
provides the <command>pkg_regress</command> command. Then you
can simply run that command, which will run all tests in the
<filename>regress</filename> category.</para>
</sect1>
<sect1 id="regression.new">
<title>Adding a new regression test</title>
<para>Every directory in the <filename>regress</filename>
category that contains a file called <filename>spec</filename>
is considered a regression test. This file is a shell program
that is included by the <command>pkg_regress</command> command.
The following functions can be overridden to suit your
needs.</para>
<sect2 id="regression.fun.override">
<title>Overridable functions</title>
<para>These functions do not take any parameters. Although they
are called in <quote>set -e</quote> mode, they don't stop at the
first failing command. See <ulink
url="http://stackoverflow.com/q/4072984">this StackOverflow
question</ulink> for details.</para>
<variablelist>
<varlistentry><term><varname>do_setup</varname></term>
<listitem><para>This function prepares the environment for the
test. By default it does nothing.</para></listitem>
</varlistentry>
<varlistentry><term><varname>do_test</varname></term>
<listitem><para>This function runs the actual test. By default,
it calls <varname>TEST_MAKE</varname> with the arguments
<varname>MAKEARGS_TEST</varname> and writes its output including
error messages into the file
<varname>TEST_OUTFILE</varname>.</para>
<para>When defining this function, make sure that all output that
needs to be checked is written to the correct output file.
Example:</para>
<programlisting>
do_test() {
echo "Example output"
} 1>$TEST_OUTFILE 2>&1
</programlisting>
</listitem>
</varlistentry>
<varlistentry><term><varname>check_result</varname></term>
<listitem><para>This function is run after the test and is
typically used to compare the actual output from the one that is
expected. It can make use of the various helper functions from
the next section. Example:</para>
<programlisting>
check_result() {
exit_status 0
output_require "Example"
output_require "^[[:alpha:]+[[:space:]][[:alpha:]]{6}$"
output_prohibit "no such file or directory"
}
</programlisting>
</listitem>
</varlistentry>
<varlistentry><term><varname>do_cleanup</varname></term>
<listitem><para>This function cleans everything up after the
test has been run. By default it does nothing.</para></listitem>
</varlistentry>
</variablelist>
</sect2>
<sect2 id="regression.fun.helper">
<title>Helper functions</title>
<variablelist>
<varlistentry><term><varname>exit_status expected</varname></term>
<listitem><para>This function compares the exitcode of the
<command>do_test</command> function with its first parameter.
If they differ, the test will fail.</para></listitem>
</varlistentry>
<varlistentry><term><varname>output_require regex...</varname></term>
<listitem><para>This function checks for each of its parameters
if the output from <command>do_test</command> matches the
extended regular expression. If it does not, the test will
fail. Example:</para>
<programlisting>
output_require "looks fine"
output_require "^[[:alpha:]+[[:space:]][[:alpha:]]{6}$"
</programlisting>
</listitem>
</varlistentry>
<varlistentry><term><varname>output_prohibit(regex...)</varname></term>
<listitem><para>This function checks for each of its parameters
if the output from <command>do_test()</command> does
<emphasis>not</emphasis> match the extended regular expression.
If any of the regular expressions matches, the test will
fail.</para></listitem>
</varlistentry>
</variablelist>
</sect2>
</sect1>
</chapter>
|
