1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="Content-Language" content="en">
<title>Rsyslog Debug Support</title></head>
<body>
<h1>Rsyslog Debug Support</h1>
<p>
Rsyslog provides a number of debug aides. Some of them are activated by
adding the --enable-rtinst ./configure option ("rtinst" means runtime
instrumentation). Turning debugging on obviously costs some performance
(in some cases considerable).
</p>
<p>This is document is just being created and thus terse.</p>
<h2>Signals supported</h2>
<p><b>SIGUSR1</b> - turns debug messages on and off. Note that for this
signal to work, rsyslogd must be running with debugging enabled, either
via the -d command line switch or the environment options specified below.
It is <b>not</b> required that rsyslog was compiled with debugging enabled
(but depending on the settings this may lead to better debug info).
<p><b>SIGUSR2</b> - outputs debug information (including active threads
and a call stack) for the state when SIGUSR2 was received. This is a
one-time output. Can be sent as often as the user likes.
<p><b>Note:</b> this signal <b>may go away</b> in later releases and may
be replaced by something else.</p>
<h2>Environment Variables</h2>
<p>There are two environment variables that set several debug settings:
<ul>
<li>The "RSYSLOG_DEBUGLOG" (sample: RSYSLOG_DEBUGLOG="/path/to/debuglog/")
writes (almost)
all debug message to the specified log file in addition to stdout. Some
system messages (e.g. segfault or abort message) are not written to the
file as we can not capture them.
<li>Runtime debug support is controlled by "RSYSLOG_DEBUG".
<p>The "RSYSLOG_DEBUG" environment variable contains an option string with the following
options possible (all are case insensitive):</p>
<ul>
<li><b>LogFuncFlow</b> - print out the logical flow of functions (entering and exiting them)</li>
<li><b>FileTrace</b> - specifies which files to trace LogFuncFlow. If <b>not</b>
set (the default), a LogFuncFlow trace is provided for all files. Set
to limit it to the files specified. FileTrace may be specified multiple
times, one file each (e.g. export RSYSLOG_DEBUG="LogFuncFlow
FileTrace=vm.c FileTrace=expr.c"</li>
<li><b>PrintFuncDB</b> - print the content of the debug function database whenever debug information is printed (e.g. abort case)!</li>
<li><b>PrintAllDebugInfoOnExit</b> - print all debug information immediately before rsyslogd exits (<span style="font-weight: bold; font-style: italic;">currently not implemented!</span>)</li>
<li><b>PrintMutexAction</b> - print mutex action as it happens. Useful for finding deadlocks and such.</li>
<li><b>NoLogTimeStamp</b> - do not prefix log lines with a timestamp (default is to do that).</li>
<li><b>NoStdOut</b> - do not emit debug messages to stdout. If RSYSLOG_DEBUGLOG is not set, this means no messages will be displayed at all.</li>
<li><b>Debug</b> - if present, turns on the debug system and enables debug output
<li><b>DebugOnDemand</b> - if present, turns on the debug system but does not enable
debug output itself. You need to send SIGUSR1 to turn it on when desired.
<li><b>OutputTidToStderr</b> - if present, makes rsyslog output information about
the thread id (tid) of newly create processesto stderr. Note that not necessarily
all new threads are reported (depends on the code, e.g. of plugins). This is
only available under Linux. This usually does NOT work when privileges have
been dropped (that's not a bug, but the way it is).
<li><b>help</b> - display a very short list of commands - hopefully a life saver if you can't access the documentation...</li>
</ul>
<p>Individual options are separated by spaces.</p>
</ul>
<h3>Why Environment Variables?</h3>
<p>You may ask why we use environment variables for debug-system parameters and not
the usual rsyslog.conf configuration commands. After all, environment variables force one
to change distro-specific configuration files, whereas regular configuration directives
would fit nicely into the one central rsyslog.conf.
<p>The problem here is that many settings of the debug system must be initialized
before the full rsyslog engine starts up. At that point, there is no such thing like
rsyslog.conf or the objects needed to process it present in an running instance.
And even if we would enable to change settings some time later, that would mean that
we can not correctly monitor (and debug) the initial startup phase of rsyslogd. What
makes matters worse is that during this startup phase (and never again later!) some
of the base debug structure needs to be created, at least if the build is
configured for that (many of these things only happen in --enable-rtinst mode). So
if we do not initialize the debug system <b>before</b> actually startig up the
rsyslog core, we get a number of data structures wrong.
<p>For these reasons, we utilize environment variables to initialize and configure
the debugging system. We understand this may be somewhat painful, but now you know
there are at least some good reasons for doing so.
<p>HOWEVER, if you have a too hard time to set debug instructions using the environment
variables, there is a cure, described in the next paragraph.
<h2>Enabling Debug via rsyslog.conf</h2>
<p>As described in the previous paragraph, enabling debug via rsyslog.conf
may not be perfect for some debugging needs, but basic debug output will work - and
that is what most often is requried. There are limited options available, but these
cover the most important use cases.
<p>Debug processing is done via legacy config statements. There currently
is no plan to move these over to the v6+ config system. Availabe settings are
<ul>
<li>$DebugFile <filename> - sets the debug file name
<li>$DebugLevel <0|1|2> - sets the respective debug level, where
0 means debug off, 1 is debug on demand activated (but debug mode off)
and 2 is full debug mode.
</ul>
<p>Note that in theory it is forbidden to specify these parameters more
than once. However, we do not enforce that and if it happens results
are undefined.
<h2>Getting debug information from a running Instance</h2>
<p>It is possible to obtain debugging information from a running instance, but this requires
some setup. We assume that the instance runs in the background, so debug output to
stdout is not desired. As such, all debug information needs to go into a log file.
<p>To create this setup, you need to
<ul>
<li>point the RSYSLOG_DEBUGLOG environment variable to a file that is accessible
during the while runtime (we strongly suggest a file in the local file system!)
<li>set RSYSLOG_DEBUG at least to "DebugOnDeman NoStdOut"
<li>make sure these environment variables are set in the correct (distro-specifc)
startup script if you do not run rsyslogd interactively
</ul>
<p>These settings enable the capability to react to SIGUSR1. The signal will toggle
debug status when received. So send it one to turn debug loggin on, and send it again
to turn debug logging off again. The third time it will be turned on again ... and so on.
<p>On a typical system, you can signal rsyslogd as follows:
<pre>
kill -USR1 `cat /var/run/rsyslogd.pid`
</pre>
Important: there are backticks around the "cat"-command. If you use the regular
quote it won't work. The debug log will show whether debug logging has been turned
on or off. There is no other indication of the status.
<p>Note: running with DebugOnDemand by itself does in practice not have any performance
toll. However, switching debug logging on has a severe performance toll. Also, debug
logging synchronizes much of the code, removing a lot of concurrency and thus
potential race conditions. As such, the very same running instance may behave
very differently with debug logging turned on vs. off. The on-demand debug log
functionality is considered to be very valuable to analyze hard-to-find bugs that
only manifest after a long runtime. Turning debug logging on a failing instance
may reveal the cause of the failure. However, depending on the failure, debug logging
may not even be successfully be turned on. Also note that with this rsyslog version we cannot
obtain any debug information on events that happened <i>before</i> debug logging was
turned on.
<p>If an instance hangs, it is possible to obtain some useful information about the current
threads and their calling stack by sending SIGUSR2. However, the usefulness of that
information is very much depending on rsyslog compile-time settings, must importantly
the --enable-rtinst configure flag. Note that activating this option causes additional overhead
and slows down rsyslogd considerable. So if you do that, you need to check if it is
capable to handle the workload. Also, threading behavior is modified by the
runtime instrumentation.
<p>Sending SIGUSR2 writes new process state information to the log file each time
it is sent. So it may be useful to do that from time to time. It probably is most
useful if the process seems to hang, in which case it may (may!) be able to output
some diagnostic information on the current processing state. In that case, turning
on the mutex debugging options (see above) is probably useful.
<h2>Interpreting the Logs</h2>
<p>Debug logs are primarily meant for rsyslog developers. But they may still provide valuable
information to users. Just be warned that logs sometimes contains information the looks like
an error, but actually is none. We put a lot of extra information into the logs, and there
are some cases where it is OK for an error to happen, we just wanted to record it inside
the log. The code handles many cases automatically. So, in short, the log may not make sense to
you, but it (hopefully) makes sense to a developer. Note that we developers often need
many lines of the log file, it is relatively rare that a problem can be diagnosed by
looking at just a couple of (hundred) log records.
<h2>Security Risks</h2>
<p>The debug log will reveal potentially sensible information, including user accounts and
passwords, to anyone able to read the log file. As such, it is recommended to properly
guard access to the log file. Also, an instance running with debug log enabled runs much
slower than one without. An attacker may use this to place carry out a denial-of-service
attack or try to hide some information from the log file. As such, it is suggested to
enable DebugOnDemand mode only for a reason. Note that when no debug mode is enabled,
SIGUSR1 and SIGUSR2 are completely ignored.
<p>When running in any of the debug modes (including on demand mode), an interactive
instance of rsyslogd can be aborted by pressing ctrl-c.
<p>
<h2>See Also</h2>
<ul>
<li><a href="http://www.rsyslog.com/how-to-use-debug-on-demand/">How to use debug on demand</a></li>
</ul>
</p>
<p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p>
<p><font size="2">This documentation is part of the
<a href="http://www.rsyslog.com/">rsyslog</a> project.<br>
Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and
<a href="http://www.adiscon.com/">Adiscon</a>.
Released under the GNU GPL version 3 or higher.</font></p>
</body>
</html>
|
