diff options
Diffstat (limited to 'doc')
65 files changed, 2567 insertions, 179 deletions
diff --git a/doc/Makefile.am b/doc/Makefile.am index 56176d1..6e521c7 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -37,6 +37,15 @@ html_files = \ omjournal.html \ imjournal.html \ mmanon.html \ + mmsequence.html \ + mmcount.html \ + mmfields.html \ + mmrfc5424addhmac.html \ + mmutf8fix.html \ + omelasticsearch.html \ + ommongodb.html \ + queue_parameters.html \ + mmpstrucdata.html \ omusrmsg.html \ omstdout.html \ omudpspoof.html \ @@ -61,6 +70,15 @@ html_files = \ src/queueWorkerLogic.dia \ queueWorkerLogic.jpg \ queueWorkerLogic_small.jpg \ + rsyslog_secure_tls.html \ + tls_cert_ca.html \ + tls_cert_client.html \ + tls_cert_errmsgs.html \ + tls_cert_machine.html \ + tls_cert_scenario.html \ + tls_cert_server.html \ + tls_cert_summary.html \ + tls_cert_udp_relay.html \ tls_cert_100.jpg \ tls_cert_ca.jpg \ tls_cert.jpg \ @@ -146,6 +164,7 @@ html_files = \ rsyslog_conf_basic_structure.html \ rsyslog_conf_sysklogd_compatibility.html \ imkmsg.html \ + rainerscript_call.html \ src/classes.dia grfx_files = \ diff --git a/doc/Makefile.in b/doc/Makefile.in index ce66409..4b0a551 100644 --- a/doc/Makefile.in +++ b/doc/Makefile.in @@ -121,6 +121,8 @@ LIBGCRYPT_CFLAGS = @LIBGCRYPT_CFLAGS@ LIBGCRYPT_LIBS = @LIBGCRYPT_LIBS@ LIBLOGGING_CFLAGS = @LIBLOGGING_CFLAGS@ LIBLOGGING_LIBS = @LIBLOGGING_LIBS@ +LIBLOGGING_STDLOG_CFLAGS = @LIBLOGGING_STDLOG_CFLAGS@ +LIBLOGGING_STDLOG_LIBS = @LIBLOGGING_STDLOG_LIBS@ LIBLOGNORM_CFLAGS = @LIBLOGNORM_CFLAGS@ LIBLOGNORM_LIBS = @LIBLOGNORM_LIBS@ LIBM = @LIBM@ @@ -145,6 +147,8 @@ NM = @NM@ NMEDIT = @NMEDIT@ OBJDUMP = @OBJDUMP@ OBJEXT = @OBJEXT@ +OPENSSL_CFLAGS = @OPENSSL_CFLAGS@ +OPENSSL_LIBS = @OPENSSL_LIBS@ ORACLE_CFLAGS = @ORACLE_CFLAGS@ ORACLE_LIBS = @ORACLE_LIBS@ OTOOL = @OTOOL@ @@ -282,6 +286,15 @@ html_files = \ omjournal.html \ imjournal.html \ mmanon.html \ + mmsequence.html \ + mmcount.html \ + mmfields.html \ + mmrfc5424addhmac.html \ + mmutf8fix.html \ + omelasticsearch.html \ + ommongodb.html \ + queue_parameters.html \ + mmpstrucdata.html \ omusrmsg.html \ omstdout.html \ omudpspoof.html \ @@ -306,6 +319,15 @@ html_files = \ src/queueWorkerLogic.dia \ queueWorkerLogic.jpg \ queueWorkerLogic_small.jpg \ + rsyslog_secure_tls.html \ + tls_cert_ca.html \ + tls_cert_client.html \ + tls_cert_errmsgs.html \ + tls_cert_machine.html \ + tls_cert_scenario.html \ + tls_cert_server.html \ + tls_cert_summary.html \ + tls_cert_udp_relay.html \ tls_cert_100.jpg \ tls_cert_ca.jpg \ tls_cert.jpg \ @@ -391,6 +413,7 @@ html_files = \ rsyslog_conf_basic_structure.html \ rsyslog_conf_sysklogd_compatibility.html \ imkmsg.html \ + rainerscript_call.html \ src/classes.dia grfx_files = \ diff --git a/doc/build_from_repo.html b/doc/build_from_repo.html index 6e018a5..121db57 100644 --- a/doc/build_from_repo.html +++ b/doc/build_from_repo.html @@ -8,7 +8,7 @@ distribution tarball to generate it. But there may be situations where it is des to build directly from the source repository. This is useful for people who would like to participate in development or who would like to use the latest, not-yet-released code. The later may especially be the case if you are asked to try out an experimental version. -<p>Building from the repsitory is not much different than building from the source +<p>Building from the repository is not much different than building from the source tarball, but some files are missing because they are output files and thus do not belong into the repository. <h2>Obtaining the Source</h2> diff --git a/doc/debug.html b/doc/debug.html index 229aeb0..9f05c68 100644 --- a/doc/debug.html +++ b/doc/debug.html @@ -27,7 +27,7 @@ be replaced by something else.</p> <p>There are two environment variables that set several debug settings: <ul> <li>The "RSYSLOG_DEBUGLOG" (sample: RSYSLOG_DEBUGLOG="/path/to/debuglog/") -writes (allmost) +writes (almost) all debug message to the specified log file in addition to stdout. Some system messages (e.g. segfault or abort message) are not written to the file as we can not capture them. @@ -133,7 +133,7 @@ turned on. threads and their calling stack by sending SIGUSR2. However, the usefulness of that information is very much depending on rsyslog compile-time settings, must importantly the --enable-rtinst configure flag. Note that activating this option causes additional overhead -and slows down rsyslgod considerable. So if you do that, you need to check if it is +and slows down rsyslogd considerable. So if you do that, you need to check if it is capable to handle the workload. Also, threading behavior is modified by the runtime instrumentation. <p>Sending SIGUSR2 writes new process state information to the log file each time @@ -143,13 +143,13 @@ some diagnostic information on the current processing state. In that case, turni on the mutex debugging options (see above) is probably useful. <h2>Interpreting the Logs</h2> <p>Debug logs are primarily meant for rsyslog developers. But they may still provide valuable -information to users. Just be warned that logs sometimes contains informaton the looks like +information to users. Just be warned that logs sometimes contains information the looks like an error, but actually is none. We put a lot of extra information into the logs, and there are some cases where it is OK for an error to happen, we just wanted to record it inside the log. The code handles many cases automatically. So, in short, the log may not make sense to you, but it (hopefully) makes sense to a developer. Note that we developers often need many lines of the log file, it is relatively rare that a problem can be diagnosed by -looking at just a couple of (hundered) log records. +looking at just a couple of (hundred) log records. <h2>Security Risks</h2> <p>The debug log will reveal potentially sensible information, including user accounts and passwords, to anyone able to read the log file. As such, it is recommended to properly @@ -159,8 +159,9 @@ attack or try to hide some information from the log file. As such, it is suggest enable DebugOnDemand mode only for a reason. Note that when no debug mode is enabled, SIGUSR1 and SIGUSR2 are completely ignored. <p>When running in any of the debug modes (including on demand mode), an interactive -instance of rsyslogd can be aborted by pressing ctl-c. -<p><b>See Also</b> +instance of rsyslogd can be aborted by pressing ctrl-c. +<p> +<h2>See Also</h2> <ul> <li><a href="http://www.rsyslog.com/how-to-use-debug-on-demand/">How to use debug on demand</a></li> </ul> diff --git a/doc/dev_oplugins.html b/doc/dev_oplugins.html index 4a9cd15..bd2bfc3 100644 --- a/doc/dev_oplugins.html +++ b/doc/dev_oplugins.html @@ -15,13 +15,15 @@ and pointers than to have nothing. <p>The best to get started with rsyslog plugin development is by looking at existing plugins. All that start with "om" are <b>o</b>utput <b>m</b>odules. That means they are primarily thought of being message sinks. In theory, however, output -plugins may aggergate other functionality, too. Nobody has taken this route so far +plugins may aggregate other functionality, too. Nobody has taken this route so far so if you would like to do that, it is highly suggested to post your plan on the rsyslog mailing list, first (so that we can offer advise). <p>The rsyslog distribution tarball contains the omstdout plugin which is extremely well targeted for getting started. Just note that this plugin itself is not meant for production use. But it is very simplistic and so a really good starting point to -grasp the core ideas. +grasp the core ideas. Also, it supports two different parameter-passing modes and +offers some light functionality. Note, however, that in order to use omstdout as is, you +need to run rsyslog interactively as otherwise stdout is redirected. <p>In any case, you should also read the comments in ./runtime/module-template.h. Output plugins are build based on a large set of code-generating macros. These macros handle most of the plumbing needed by the interface. As long as no @@ -44,7 +46,7 @@ that shares a single instanceData structure. <p>So as long as you do not mess around with global data, you do not need to think about multithreading (and can apply a purely sequential programming methodology). -<p>Please note that duringt the configuraton parsing stage of execution, access to +<p>Please note that during the configuration parsing stage of execution, access to global variables for the configuration system is safe. In that stage, the core will only call sequentially into the plugin. <h3>Getting Message Data</h3> @@ -69,7 +71,7 @@ get it into the core (so far, we could accept all such suggestions - no promise, request access to the template components. The typical use case seems to be databases, where you would like to access properties via specific fields. With that mode, you receive a char ** array, where each array element points to one field from the template (from -left to right). Fields start at arrray index 0 and a NULL pointer means you have +left to right). Fields start at array index 0 and a NULL pointer means you have reached the end of the array (the typical Unix "poor man's linked list in an array" design). Note, however, that each of the individual components is a string. It is not a date stamp, number or whatever, but a string. This is because rsyslog processes @@ -132,19 +134,19 @@ for example in MongoDB or ElasticSearch. a single-message interface was supported. <p>With the <b>single message</b> plugin interface, each message is passed via a separate call to the plugin. Most importantly, the rsyslog engine assumes that each call to the plugin is a complete transaction -and as such assumes that messages be properly commited after the plugin returns to the engine. +and as such assumes that messages be properly committed after the plugin returns to the engine. <p>With the <b>batching</b> interface, rsyslog employs something along the line of "transactions". Obviously, the rsyslog core can not make non-transactional outputs to be fully transactional. But what it can is support that the output tells the core which -messages have been commited by the output and which not yet. The core can than take care -of those uncommited messages when problems occur. For example, if a plugin has received -50 messages but not yet told the core that it commited them, and then returns an error state, the +messages have been committed by the output and which not yet. The core can than take care +of those uncommitted messages when problems occur. For example, if a plugin has received +50 messages but not yet told the core that it committed them, and then returns an error state, the core assumes that all these 50 messages were <b>not</b> written to the output. The core then -requeues all 50 messages and does the usual retry processing. Once the output plugin tells the +re-queues all 50 messages and does the usual retry processing. Once the output plugin tells the core that it is ready again to accept messages, the rsyslog core will provide it with these 50 -not yet commited messages again (actually, at this point, the rsyslog core no longer knows that -it is re-submiting the messages). If, in contrary, the plugin had told rsyslog that 40 of these 50 -messages were commited (before it failed), then only 10 would have been requeued and resubmitted. +not yet committed messages again (actually, at this point, the rsyslog core no longer knows that +it is re-submitting the messages). If, in contrary, the plugin had told rsyslog that 40 of these 50 +messages were committed (before it failed), then only 10 would have been re-queued and resubmitted. <p>In order to provide an efficient implementation, there are some (mild) constraints in that transactional model: first of all, rsyslog itself specifies the ultimate transaction boundaries. That is, it tells the plugin when a transaction begins and when it must finish. The plugin @@ -155,7 +157,7 @@ transaction support. Note that batch sizes are variable within the range of 1 to maximum limit. Most importantly, that means that plugins may receive batches of single messages, so they are required to commit each message individually. If the plugin tries to be "smarter" than the rsyslog engine and does not commit messages in those cases (for example), the plugin -puts message stream integrity at risk: once rsyslog has notified the plugin of transacton end, +puts message stream integrity at risk: once rsyslog has notified the plugin of transaction end, it discards all messages as it considers them committed and save. If now something goes wrong, the rsyslog core does not try to recover lost messages (and keep in mind that "goes wrong" includes such uncontrollable things like connection loss to a database server). So it is @@ -170,8 +172,8 @@ This is also under evaluation and, once decided, the core will offer an interfac to preserve message stream integrity for properly-crafted plugins). <p>The second restriction is that if a plugin makes commits in between (what is perfectly legal) those commits must be in-order. So if a commit is made for message ten out of 50, -this means that messages one to nine are also commited. It would be possible to remove -this restriction, but we have decided to deliberately introduce it to simpify things. +this means that messages one to nine are also committed. It would be possible to remove +this restriction, but we have decided to deliberately introduce it to simplify things. <h3>Output Plugin Transaction Interface</h3> <p>In order to keep compatible with existing output plugins (and because it introduces no complexity), the transactional plugin interface is build on the traditional @@ -231,7 +233,7 @@ But they convey additional information about the commit status as follows: <table border="0"> <tr> <td valign="top"><i>RS_RET_OK</i></td> -<td>The record and all previous inside the batch has been commited. +<td>The record and all previous inside the batch has been committed. <i>Note:</i> this definition is what makes integrating plugins without the transaction being/end calls so easy - this is the traditional "success" return state and if every call returns it, there is no need for actually calling @@ -239,7 +241,7 @@ state and if every call returns it, there is no need for actually calling </tr> <tr> <td valign="top"><i>RS_RET_DEFER_COMMIT</i></td> -<td>The record has been processed, but is not yet commited. This is the +<td>The record has been processed, but is not yet committed. This is the expected state for transactional-aware plugins.</td> </tr> <tr> @@ -248,7 +250,7 @@ expected state for transactional-aware plugins.</td> current one not yet. This state is introduced to support sources that fill up buffers and commit once a buffer is completely filled. That may occur halfway in the next record, so it may be important to be able to tell the -engine the everything up to the previouos record is commited</td> +engine the everything up to the previous record is committed</td> </tr> </table> <p>Note that the typical <b>calling cycle</b> is <code>beginTransaction()</code>, @@ -269,7 +271,7 @@ exists. So we introduce it with that release. What the means is if a rsyslog cor not provide this query interface, it is a core that was build before batching support was available. So the absence of a query interface indicates that the transactional interface is not available. One might now be tempted the think there is no need to do -the actual check, but is is recommended to ask the rsyslog engine explicitely if +the actual check, but is is recommended to ask the rsyslog engine explicitly if the transactional interface is present and will be honored. This enables us to create versions in the future which have, for whatever reason we do not yet know, no support for this interface. diff --git a/doc/dev_queue.html b/doc/dev_queue.html index bf2af7f..6d5fe73 100644 --- a/doc/dev_queue.html +++ b/doc/dev_queue.html @@ -80,7 +80,7 @@ and terminating while waiting on the primary queue to fill. In practice, this is highly unlikely (but only for the main message queue) because rsyslog issues a startup message. HOWEVER, we can not rely on that, it would introduce a race. If the primary rsyslog thread (the one that issues the message) is scheduled very -late and there is a low inactivty timeout for queue workers, the queue worker +late and there is a low inactivity timeout for queue workers, the queue worker may terminate before the startup message is issued. And if the on-disk queue holds only a few messages, it may become empty before the DA worker is re-initiated again. So it is possible that the DA run mode termination criteria @@ -105,7 +105,7 @@ clean shutdown of the DA queue).</p> <p>One might think that it would be more natural for the DA queue to detect being idle and shut down itself. However, there are some issues associated with that. Most importantly, all queue worker threads need to be shut down during -queue destruction. Only after that has happend, final destruction steps can +queue destruction. Only after that has happened, final destruction steps can happen (else we would have a myriad of races). However, it is the DA queues worker thread that detects it is empty (empty queue detection always happens at the consumer side and must so). That would lead to the DA queue worker thread to @@ -115,7 +115,7 @@ destructed). Obviously, this does not work out (and I didn't even mention the other issues - so let's forget about it). As such, the thread that enqueues messages must destruct the queue - and that is the primary queue's DA worker thread.</p> -<p>There are some subleties due to thread synchronization and the fact that the +<p>There are some subtleties due to thread synchronization and the fact that the DA consumer may not be running (in a <b>case-2 startup</b>). So it is not trivial to reliably change the queue back from DA run mode to regular run mode. The priority is a clean switch. We accept the fact that there may be situations @@ -132,7 +132,7 @@ most probably even lead to worse performance under regular conditions).</p> </ol> <p>Case 2 is unlikely, but may happen (see info above on a case 2 startup).</p> <p><b>The DA worker may also not wait at all,</b> because it is actively -executing and shuffeling messages between the queues. In that case, however, the +executing and shuffling messages between the queues. In that case, however, the program flow passes both of the two wait conditions but simply does not wait.</p> <p><b>Finally, the DA worker may be inactive </b>(again, with a case-2 startup). In that case no work(er) at all is executed. Most importantly, without the DA @@ -247,4 +247,4 @@ no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be viewed at <a href="http://www.gnu.org/copyleft/fdl.html"> http://www.gnu.org/copyleft/fdl.html</a>.</p> </body> -</html>
\ No newline at end of file +</html> diff --git a/doc/droppriv.html b/doc/droppriv.html index 7293e87..75773e6 100644 --- a/doc/droppriv.html +++ b/doc/droppriv.html @@ -18,7 +18,7 @@ rsyslogd needs to start up as root. user. That is probably the safest way of operations. However, if a startup as root is required, you can use the $PrivDropToGroup and $PrivDropToUser config directives to specify a group and/or user that rsyslogd should drop to after initialization. -Once this happend, the daemon runs without high privileges (depending, of +Once this happens, the daemon runs without high privileges (depending, of course, on the permissions of the user account you specified). <p>There is some additional information available in the <a href="http://wiki.rsyslog.com/index.php/Security#Dropping_Privileges">rsyslog wiki</a>. diff --git a/doc/features.html b/doc/features.html index 626ff65..ecf6a01 100644 --- a/doc/features.html +++ b/doc/features.html @@ -42,7 +42,7 @@ into syslog messages (one per line)</li> <li>ability to configure backup syslog/database servers - if the primary fails, control is switched to a prioritized list of backups</li> <li>support for receiving messages via reliable <a href="http://www.monitorware.com/Common/en/glossary/rfc3195.php"> -RFC 3195</a> delivery (a bit clumpsy to build right now...)</li> +RFC 3195</a> delivery (a bit clumsy to build right now...)</li> <li>ability to generate file names and directories (log targets) dynamically, based on many different properties</li> <li>control of log output format, including ability to present @@ -95,7 +95,7 @@ via custom plugins</li> <li> an easy-to-write to plugin interface</li> <li> ability to send SNMP trap messages</li> <li> ability to filter out messages based on sequence of arrival</li> -<li>support for comma-seperated-values (CSV) output generation +<li>support for comma-separated-values (CSV) output generation (via the "csv" property replace option). The CSV format supported is that from RFC 4180.</li> <li>support for arbitrary complex boolean, string and diff --git a/doc/free_support.html b/doc/free_support.html index 182a82c..a3a9a69 100644 --- a/doc/free_support.html +++ b/doc/free_support.html @@ -18,7 +18,7 @@ reason is quite simple: If I do personal support, you gain some advantage withou contributing something back. Think about it: if you ask your question on the public forum or mailing list, other with the same problem can you and, most importantly, even years later find your post (and the answer) and get the problem solved. So by -solving your issue in public, you help create a great community ressource and also +solving your issue in public, you help create a great community resource and also help your fellow users finding solutions quicker. In the long term, this also contributes to improved code because the more questions users can find solutions to themselves, the fewer I need to look at. diff --git a/doc/global.html b/doc/global.html index a58f5c6..70f3b9c 100644 --- a/doc/global.html +++ b/doc/global.html @@ -13,6 +13,18 @@ unpredictable. <p>The following paramters can be set: <ul> +<li><b>action.reportSuspension</b> - binary, default "on", v7.5.8+<br> +If enabled ("on") action will log message under syslog.* when an action +suspends or resume itself. This usually happens when there are problems +connecting to backend systems. If disabled ("off"), these messages are +not generated. These messages can be useful in detecting problems with +backend systems. Most importantly, frequent suspension and resumption +points to a problem area. +<li><b>action.reportSuspensionContinuation</b> - binary, default "off", v7.6.1+, v8.2.0+<br> +If enabled ("on") the action will not only report the first suspension but +each time the suspension is prolonged. Otherwise, the follow-up messages +are not logged. If this setting is set to "on", action.reportSuspension is +also automaticaly turned "on". <li>workDirectory <li>dropMsgsWithMaliciousDNSPtrRecords <li>localHostname @@ -21,13 +33,34 @@ unpredictable. <li>defaultNetstreamDriverKeyFile <li>defaultNetstreamDriver <li>maxMessageSize +<li><b>debug.onShutdown</b> binary (on/off), available in 7.5.8+<br> +If enabled ("on"), rsyslog will log debug messages when a system shutdown +is requested. This can be used to track issues that happen only during +shutdown. During normal operations, system performance is NOT affected. +<br>Note that for this option to be useful, the debug.logFile parameter +must also be set (or the respective environment variable). +<li><b>debug.logFile</b> available in 7.5.8+<br> +This is used to specify the debug log file name. It is used for all debug +output. Please note that the RSYSLOG_DEBUGLOG environment variable always +<b>overrides</b> the value of debug.logFile. + +<li><b>processInternalMessages</b> binary (on/off), available in v7.4.9+, 7.6.0+, 8.1.5+<br> +This tell rsyslog if it shall process internal messages itself. This +is the default mode of operations ("on") and usually the best. However, +if this (instance) of rsyslog is not the main instance and there is +another main logging system, rsyslog internal messages will not be +inserted into the main instance's syslog stream. To do this, set +the parameter to "off", in which case rsyslog will send messages to +the system log sink (and if it is the only instance, receive them back +from there). This also works with systemd journal and will make +rsyslog messages show up in the systemd status control information. </ul> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="rainerscript.html">RainerScript reference</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2013-2014 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under ASL 2.0 or higher.</font></p> </body> diff --git a/doc/history.html b/doc/history.html index 57b6400..a071461 100644 --- a/doc/history.html +++ b/doc/history.html @@ -117,7 +117,7 @@ the need to have complex expression support, which was also the first use case. On February, 28th rsyslog 3.12.0 was released, the first version to contain expression support. This also meant that rsyslog from that date on supported all syslog-ng major features, but had a -number of major features exlusive to it. With 3.12.0, I consider +number of major features exclusive to it. With 3.12.0, I consider rsyslog fully superior to syslog-ng (except for platform support).</p> <p>Following the Fedora Developer's conference in Brno <b>2012</b>, rsyslog diff --git a/doc/im3195.html b/doc/im3195.html index aad9f3d..317ab84 100644 --- a/doc/im3195.html +++ b/doc/im3195.html @@ -25,7 +25,7 @@ The port on which imklog listens for RFC 3195 messages. The default port is 601 to this input module, but we have NOT conducted any testing. Also, the module does not yet properly handle the recovery case. If someone intends to put this module into production, good testing should be -cunducted. It also is a good idea to notify the rsyslog project that you intend to use +conducted. It also is a good idea to notify the rsyslog project that you intend to use it in production. In this case, we'll probably give the module another cleanup. We don't do this now because so far it looks just like a big waste of time. diff --git a/doc/imfile.html b/doc/imfile.html index a69f62e..8c9a425 100644 --- a/doc/imfile.html +++ b/doc/imfile.html @@ -106,6 +106,15 @@ This mode should defined when having multiline messages. The value can range fro <br>0 (<strong>default</strong>) - line based (Each line is a new message) <br>1 - paragraph (There is a blank line between log messages) <br>2 - indented (New log messages start at the beginning of a line. If a line starts with a space it is part of the log message before it) +<li><b>escapeLF</b> [<b>on</b>/off] (requires v7.5.3+)<br> +This is only meaningful if multi-line messages are to be processed. LF characters embedded into +syslog messages cause a lot of trouble, as most tools and even the legacy syslog TCP protocol +do not expect these. If set to "on", this option avoid this trouble by properly escaping +LF characters to the 4-byte sequence "#012". This is consistent with other rsyslog control character +escaping. By default, escaping is turned on. If you turn it off, make sure you test very carefully +with all associated tools. Please note that if you intend to use plain TCP syslog with embedded +LF characters, you need to enable octet-counted framing. For more details, see Rainer's blog posting on +<a href="http://blog.gerhards.net/2013/09/imfile-multi-line-messages.html">imfile LF escaping</a>. <li><b>MaxLinesAtOnce</b> [number]</b> <br> This is useful if multiple files need to be monitored. If set to 0, each file @@ -160,6 +169,11 @@ input(type="imfile" File="/path/to/file2" <p><b>Legacy Configuration Directives</b>:</p> +<p>Note: in order to preserve compatibility with previous versions, the +LF escaping in multi-line messages is turned off for legacy-configured +file monitors (the "escapeLF" input parameter). This can cause serious problems. +So it is highly suggested that new deployments use the new input() statement +and keep LF escaping turned on. <ul> <li><strong>$InputFileName /path/to/file</strong><br> equivalent to: File </li> diff --git a/doc/imjournal.html b/doc/imjournal.html index 8f29169..1831695 100644 --- a/doc/imjournal.html +++ b/doc/imjournal.html @@ -69,6 +69,18 @@ journal and read only new messages. This option is only used when there is no StateFile to avoid message loss. </ul> +<li><b>DefaultSeverity</b> <severity><br> +Some messages comming from journald don't have the SYSLOG_PRIORITY +field. These are typically the messages logged through journald's +native API. This option specifies the default severity for these +messages. Can be given either as a name or a number. Defaults to 'notice'. + +<li><b>DefaultFacility</b> <facility><br> +Some messages comming from journald don't have the SYSLOG_FACILITY +field. These are typically the messages logged through journald's +native API. This option specifies the default facility for these +messages. Can be given either as a name or a number. Defaults to 'user'. + <b>Caveats/Known Bugs:</b> <p> <ul> @@ -105,6 +117,10 @@ Equivalent to: ratelimit.interval</li> Equivalent to: ratelimit.burst</li> <li><strong>$ImjournalIgnorePreviousMessages</strong><br> Equivalent to: ignorePreviousMessages</li> +<li><strong>$ImjournalDefaultSeverity</strong><br> +Equivalent to: DefaultSeverity</li> +<li><strong>$ImjournalDefaultFacility</strong><br> +Equivalent to: DefaultFacility</li> </ul> </body> diff --git a/doc/imklog.html b/doc/imklog.html index 1f195b1..f2e36fd 100644 --- a/doc/imklog.html +++ b/doc/imklog.html @@ -25,7 +25,7 @@ imklog generates some messages of itself (e.g. on problems, startup and shutdown) and these do not stem from the kernel. Historically, under Linux, these too have "kern" facility. Thus, on Linux platforms the default is "kern" while on others it is "syslogd". You usually do not -need to specify this configuratin directive - it is included primarily +need to specify this configuration directive - it is included primarily for few limited cases where it is needed for good reason. Bottom line: if you don't have a good idea why you should use this setting, do not touch it.</li> diff --git a/doc/imkmsg.html b/doc/imkmsg.html index 23b9614..ba73715 100644 --- a/doc/imkmsg.html +++ b/doc/imkmsg.html @@ -16,7 +16,7 @@ Milan Bartos <p>Reads messages from the /dev/kmsg structured kernel log and submits them to the syslog engine.</p> <p> -The printk log buffer constains log records. These records are exported by /dev/kmsg +The printk log buffer contains log records. These records are exported by /dev/kmsg device as structured data in the following format:<br /> "level,sequnum,timestamp;<message text>\n"<br /> There could be continuation lines starting with space that contains key/value pairs.<br /> diff --git a/doc/impstats.html b/doc/impstats.html index 770f67a..c768dbf 100644 --- a/doc/impstats.html +++ b/doc/impstats.html @@ -12,7 +12,7 @@ <p><b>Description</b>:</p> <p>This module provides periodic output of rsyslog internal counters. Note that the whole statistics system is currently under development. So -availabilty and format of counters may change and is not yet stable (so be +availability and format of counters may change and is not yet stable (so be prepared to change your trending scripts when you upgrade to a newer rsyslog version). <p>The set of available counters will be output as a set of syslog messages. This output is periodic, with the interval being configurable (default is 5 minutes). @@ -20,7 +20,7 @@ Be sure that your configuration records the counter messages (default is syslog. Besides logging to the regular syslog stream, the module can also be configured to write statistics data into a (local) file. <p>Note that loading this module has impact on rsyslog performance. Depending on -settings, this impact may be noticable (for high-load environments). +settings, this impact may be noticeable (for high-load environments). <p>The rsyslog website has an updated overview of available <a href="http://rsyslog.com/rsyslog-statistic-counter/">rsyslog statistic counters</a>. </p> @@ -34,34 +34,48 @@ to understand and gain information from the pstats logs. <p><b>Module Confguration Parameters</b>:</p> <p>This module supports module parameters, only. <ul> - <li><strong>interval </strong>[seconds] (default 300 [5minutes])<br> + <li><b>interval </b>[seconds] (default 300 [5minutes])<br> Sets the interval, in <b>seconds</b> at which messages are generated. Please note that the actual interval may be a bit longer. We do not try to be precise and so the interval is actually a sleep period which is entered after generating all messages. So the actual interval is what is configured here plus the actual time required to generate messages. In general, the difference should not really matter. <br></li> - <li><strong>facility </strong>[templateName]<br> + <li><b>facility </b>[facility number]<br> The numerical syslog facility code to be used for generated messages. Default is 5 (syslog). This is useful for filtering messages. <br></li> - <li><strong>severity </strong>[templateName]<br> + <li><b>severity </b>[severity number]<br> The numerical syslog severity code to be used for generated messages. Default is 6 (info).This is useful for filtering messages. <br></li> - <li><strong>format </strong>[json/cee/<b>legacy</b>](rsyslog v6.3.8+ only)<br> + <li><b>resetCounters </b>[<b>off</b>/on] - available since 7.5.3<br> + When set to "on", counters are automatically reset after they are emitted. In that + case, the contain only deltas to the last value emitted. When set + to "off", counters always accumulate their values. + Note that in auto-reset mode not all counters can be reset. Some counters (like queue size) + are directly obtained from internal object and cannot be modified. Also, auto-resetting + introduces some additional slight inaccuracies due to the multi-threaded nature of + rsyslog and the fact that for performance reasons it cannot serialize access + to counter variables.</br> + As an alternative to auto-reset mode, you can use rsyslog's statistics + manipulation scripts to create delta values from the regular statistic + logs. This is the suggested method if deltas are not necessarily needed in + real-time. + <br></li> + <li><b>format </b>[json/cee/<b>legacy</b>] - available since 6.3.8<br> Specifies the format of emitted stats messages. The default of "legacy" is compatible with pre v6-rsyslog. The other options provide support for structured formats (note the "cee" is actually "project lumberack" logging). <br></li> - <li><strong>log.syslog </strong>[<b>on</b>/off] - available since 7.3.6<br> + <li><b>log.syslog </b>[<b>on</b>/off] - available since 7.3.6<br> This is a boolean setting specifying if data should be sent to the usual syslog stream. This is useful if custom formatting or more elaborate processing is desired. However, output is placed under the same restrictions as regular syslog data, especially in regard to the queue position (stats data may sit for an extended period of time in queues if they are full).<br></li> - <li><strong>log.file </strong>[file name] - available since 7.3.6<br> + <li><b>log.file </b>[file name] - available since 7.3.6<br> If specified, statistics data is written the specified file. For robustness, this should be a local file. The file format cannot be customized, it consists of a date header, followed by a colon, @@ -73,6 +87,8 @@ to understand and gain information from the pstats logs. on file logging does NOT turn of syslog logging. If that is desired log.syslog="off" must be explicitely set. <br></li> + <li><b>Ruleset</b> [ruleset] - available since 7.5.6<br> + Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> </ul> <p><b>Legacx Configuration Directives</b>:</p> diff --git a/doc/imptcp.html b/doc/imptcp.html index aece428..e1e3523 100644 --- a/doc/imptcp.html +++ b/doc/imptcp.html @@ -77,7 +77,7 @@ The default, 0, means that the operating system defaults are used. This has only effect if keep-alive is enabled. The functionality may not be available on all platforms. <li><b>KeepAlive.Interval</b> <number><br> -The interval between subsequential keepalive probes, regardless of what the connection has exchanged in the meantime. +The interval between subsequent keepalive probes, regardless of what the connection has exchanged in the meantime. The default, 0, means that the operating system defaults are used. This has only effect if keep-alive is enabled. The functionality may not be available on all platforms. @@ -96,12 +96,28 @@ the message was received from. Binds specified ruleset to next server defined. <li><b>Address</b> <name><br> On multi-homed machines, specifies to which local address the listerner should be bound. +<li><b>defaultTZ</b> <timezone-info><br> +This is an <b>experimental</b> parameter; details may change at any time and it may +also be discoutinued without any early warning.<br> +Permits to set a default timezone for this listener. This is useful when working with +legacy syslog (RFC3164 et al) residing in different timezones. If set it will be used as +timezone for all messages <b>that do not contain timezone info</b>. +Currently, the format <b>must</b> be "+/-hh:mm", e.g. "-05:00", "+01:30". Other formats, +including TZ names (like EST) are NOT yet supported. Note that consequently no daylight +saving settings are evaluated when working with timezones. If an invalid format is used, +"interesting" things can happen, among them malformed timestamps and rsyslogd segfaults. +This will obviously be changed at the time this feature becomes non-experimental.</li> <li><b>RateLimit.Interval</b> [number] - (available since 7.3.1) specifies the rate-limiting interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number of seconds (5 recommended) to activate rate-limiting. </li> <li><b>RateLimit.Burst</b> [number] - (available since 7.3.1) specifies the rate-limiting burst in number of messages. Default is 10,000. +<li><b>compression.mode</b><i>mode</i><br> +<i>mode</i> is one of "none" or "stream:always". +It is the counterpart to the compression modes set in +<a href="omfile.html">omfile</a>. +Please see it's documentation for details. </li> </ul> <b>Caveats/Known Bugs:</b> diff --git a/doc/imrelp.html b/doc/imrelp.html index f7fcc4b..8382166 100644 --- a/doc/imrelp.html +++ b/doc/imrelp.html @@ -28,14 +28,84 @@ nits outlined above, is a much more reliable solution than plain tcp syslog and so it is highly suggested to use RELP instead of plain tcp. Clients send messages to the RELP server via omrelp.</p> -<p><b>Configuration Directives</b>:</p> +<p><b>Module Parameters</b>:</p> +<ul> + <li><b>Ruleset</b> <name> (requires v7.5.0+)</br> + Binds the specified ruleset to <b>all</b> RELP listeners. +</ul> +<p><b>Input Parameters</b>:</p> <ul> <li><b>Port</b> <port><br> Starts a RELP server on selected port</li> +<li><b>tls</b> (not mandatory, values "on","off", default "off")<br> +If set to "on", the RELP connection will be encrypted by TLS, +so that the data is protected against observers. Please note +that both the client and the server must have set TLS to +either "on" or "off". Other combinations lead to unpredictable +results. +</li> +<li><b>tls.compression</b> (not mandatory, values "on","off", default "off")<br> +The controls if the TLS stream should be compressed (zipped). While this +increases CPU use, the network bandwidth should be reduced. Note that +typical text-based log records usually compress rather well. +</li> +<li><b>tls.dhbits</b> (not mandatory, integer)<br> +This setting controls how many bits are used for Diffie-Hellman key +generation. If not set, the librelp default is used. For secrity +reasons, at least 1024 bits should be used. Please note that the number +of bits must be supported by GnuTLS. If an invalid number is given, rsyslog +will report an error when the listener is started. We do this to be transparent +to changes/upgrades in GnuTLS (to check at config processing time, we would need +to hardcode the supported bits and keep them in sync with GnuTLS - this is +even impossible when custom GnuTLS changes are made...). +</li> +<li><b>tls.permittedPeer</b> peer</br> +Places access restrictions on this listener. Only peers which +have been listed in this parameter may connect. The validation +bases on the certificate the remote peer presents.<br> +The <i>peer</i> parameter lists permitted certificate +fingerprints. Note that it is an array parameter, so either +a single or multiple fingerprints can be listed. When a +non-permitted peer connects, the refusal is logged together +with it's fingerprint. So if the administrator knows this was +a valid request, he can simple add the fingerprint by copy and +paste from the logfile to rsyslog.conf. +<br>To specify multiple fingerprints, just enclose them +in braces like this: +<br>tls.permittedPeer=["SHA1:...1", "SHA1:....2"] +<br>To specify just a single peer, you can either +specify the string directly or enclose it in braces. +</li> +<li><b>tls.authMode</b> mode</br> +Sets the mode used for mutual authentication. Supported values are +either "<i>fingerprint</i>" or "<i>name"</i>. +<br>Fingerprint mode basically is what SSH +does. It does not require a full PKI to be present, instead self-signed +certs can be used on all peers. Even if a CA certificate is given, the +validity of the peer cert is NOT verified against it. Only the +certificate fingerprint counts. +<br>In "name" mode, certificate validation happens. Here, the matching +is done against the certificate's subjectAltName and, as a fallback, +the subject common name. If the certificate contains multiple names, +a match on any one of these names is considered good and permits the +peer to talk to rsyslog. +<li><b>tls.prioritystring</b> (not mandatory, string)<br> +This parameter permits to specify the so-called "priority string" to +GnuTLS. This string gives complete control over all crypto parameters, +including compression setting. For this reason, when the prioritystring +is specified, the "tls.compression" parameter has no effect and is +ignored. +<br>Full information about how to construct a priority string can be +found in the GnuTLS manual. At the time of this writing, this +information was contained in +<a href="http://gnutls.org/manual/html_node/Priority-Strings.html">section 6.10 of the GnuTLS manual</a>. +<br><b>Note: this is an expert parameter.</b> Do not use if you do +not exactly know what you are doing. +</li> </ul> <b>Caveats/Known Bugs:</b> <ul> -<li>ruleset can only be bound via legacy configuration format</li> +<li>see description</li> <li>To obtain the remote system's IP address, you need to have at least librelp 1.0.0 installed. Versions below it return the hostname instead of the IP address.</li> @@ -45,20 +115,19 @@ not specific ones. This is due to a currently existing limitation in librelp. <p><b>Sample:</b></p> <p>This sets up a RELP server on port 20514.<br> </p> -<textarea rows="15" cols="60">module(load="imrelp") # needs to be done just once +<textarea rows="5" cols="60">module(load="imrelp") # needs to be done just once input(type="imrelp" port="20514") </textarea> <p><b>Legacy Configuration Directives</b>:</p> <ul> <li>InputRELPServerBindRuleset <name> (available in 6.3.6+)</br> -Binds the specified ruleset to all RELP listeners. +equivalent to: RuleSet <li>InputRELPServerRun <port><br> equivalent to: Port</li> </ul> <b>Caveats/Known Bugs:</b> <ul> -<li>see description</li> <li>To obtain the remote system's IP address, you need to have at least librelp 1.0.0 installed. Versions below it return the hostname instead of the IP address.</li> @@ -68,14 +137,14 @@ not specific ones. This is due to a currently existing limitation in librelp. <p><b>Sample:</b></p> <p>This sets up a RELP server on port 20514.<br> </p> -<textarea rows="15" cols="60">$ModLoad imrelp # needs to be done just once +<textarea rows="5" cols="60">$ModLoad imrelp # needs to be done just once $InputRELPServerRun 20514 </textarea> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2011 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/imsolaris.html b/doc/imsolaris.html index ce0e7e8..cc9a745 100644 --- a/doc/imsolaris.html +++ b/doc/imsolaris.html @@ -18,7 +18,7 @@ is no special kernel input device. Instead, both kernel messages as well as messages emitted via syslog() are received from a single source. <p>This module obeys the Solaris door() mechanism to detect a running syslogd instance. As such, only one can be active at one time. If it detects another -active intance at startup, the module disables itself, but rsyslog will +active instance at startup, the module disables itself, but rsyslog will continue to run. <p><b>Configuration Directives</b>:</p> <ul> diff --git a/doc/imtcp.html b/doc/imtcp.html index b9f0b05..1323252 100644 --- a/doc/imtcp.html +++ b/doc/imtcp.html @@ -14,7 +14,7 @@ <p><b>Multi-Ruleset Support: </b>since 4.5.0 and 5.1.1 <p><b>Description</b>:</p> <p>Provides the ability to receive syslog messages via TCP. -Encryption is natively provided by selecting the approprioate network stream driver and +Encryption is natively provided by selecting the appropriate network stream driver and can also be provided by using <a href="rsyslog_stunnel.html">stunnel</a> (an alternative is the use the <a href="imgssapi.html">imgssapi</a> module).</p> @@ -63,13 +63,22 @@ the sender is throttled a bit when the queue becomes near-full. This is done in to preserve some queue space for inputs that can not throttle (like UDP), but it may have some undesired effect in some configurations. Still, we consider this as a useful setting and thus it is the default. To turn the handling off, simply -configure that explicitely. +configure that explicitly. </li> <li><b>MaxListeners</b> <number><br> Sets the maximum number of listeners (server ports) supported. Default is 20. This must be set before the first $InputTCPServerRun directive.</li> <li><b>MaxSessions</b> <number><br> Sets the maximum number of sessions supported. Default is 200. This must be set before the first $InputTCPServerRun directive</li> +<li><b>StreamDriver.Name</b> <name><br> +Sets the driver name and overrides the system default. This enables e.g. to +define a system default of "gtls" (for TLS transmission) and override it to +"ptcp" (traditional unprotected plain tcp). Note, however, that this is a module +parameter. Currently, imtcp does not support mixed TLS/non-TLS listeners. If this +is desired, use imtcp for TLS, and imptcp for non-TLS. However, setting the +stream driver enables you to use e.g. plain tcp for the imtcp listeners while +setting the system default to TLS, which is then used by multiple forwarding (omfwd) +actions. <li><b>StreamDriver.Mode</b> <number><br> -Sets the driver mode for the currently selected <a href="netstream.html">network stream driver</a>. <number> is driver specifc.</li> +Sets the driver mode for the currently selected <a href="netstream.html">network stream driver</a>. <number> is driver specific.</li> <li><b>StreamDriver.AuthMode</b> <mode-string><br> Sets the authentication mode for the currently selected <a href="netstream.html">network stream driver</a>. <mode-string> is driver specifc.</li> <li><b>PermittedPeer</b> <id-string><br> @@ -80,7 +89,7 @@ AuthMode and <a href="netstream.html">network stream driver</a>. Permitted <br>Single peer: PermittedPeer="127.0.0.1" <br>Array of peers: PermittedPeer=["test1.example.net","10.1.2.3","test2.example.net","..."]</li> </ul> -<p><b>Action Directives</b>:</p> +<p><b>Input Parameters</b>:</p> <ul> <li><b>Port</b> <port><br> Starts a TCP server on selected port</li> @@ -96,6 +105,17 @@ activated. This is the default and should be left unchanged until you know very well what you do. It may be useful to turn it off, if you know this framing is not used and some senders emit multi-line messages into the message stream. </li> +<li><b>defaultTZ</b> <timezone-info><br> +This is an <b>experimental</b> parameter; details may change at any time and it may +also be discoutinued without any early warning.<br> +Permits to set a default timezone for this listener. This is useful when working with +legacy syslog (RFC3164 et al) residing in different timezones. If set it will be used as +timezone for all messages <b>that do not contain timezone info</b>. +Currently, the format <b>must</b> be "+/-hh:mm", e.g. "-05:00", "+01:30". Other formats, +including TZ names (like EST) are NOT yet supported. Note that consequently no daylight +saving settings are evaluated when working with timezones. If an invalid format is used, +"interesting" things can happen, among them malformed timestamps and rsyslogd segfaults. +This will obviously be changed at the time this feature becomes non-experimental.</li> <li><b>RateLimit.Interval</b> [number] - (available since 7.3.1) specifies the rate-limiting interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number of seconds (5 recommended) to activate rate-limiting. diff --git a/doc/imudp.html b/doc/imudp.html index a8dbca3..b57eb71 100644 --- a/doc/imudp.html +++ b/doc/imudp.html @@ -29,13 +29,42 @@ only activated if messages come in at a very fast rate, so doing less frequent time calls should usually be acceptable. The default value is two, because we have seen that even without optimization the kernel often returns twice the identical time. You can set this value as high as you like, but do so at your own risk. The higher -the value, the less precise the timestamp. +the value, the less precise the timestamp.<br> +<b>Note:</b> the timeRequery is done based on executed system calls (<b>not</b> +messages received). So when batch +sizes are used, multiple messages are received with one system call. All of these +messages always receive the same timestamp, as they are effectively received at the +same time. When there is very high traffic and successive system calls immediately +return the next batch of messages, the time requery logic kicks in, which means that +by default time is only queried for every second batch. Again, this should not cause +a too-much deviation as it requires messages to come in very rapidly. However, we advise +not to set the "timeRequery" parameter to a large value (larger than 10) if input +batches are used. <li><b>SchedulingPolicy</b> <rr/fifo/other><br> Can be used the set the scheduler priority, if the necessary functionality is provided by the platform. Most useful to select "fifo" for real-time processing under Linux (and thus reduce chance of packet loss). <li><b>SchedulingPriority</b> <number><br> Scheduling priority to use. +<li><b>batchSize</b> <number><br> +This parameter is only meaningful if the system support recvmmsg() (newer Linux +OSs do this). The parameter is silently ignored if the system does not support +it. If supported, it sets the maximum number of UDP messages that can be obtained +with a single OS call. For systems with high UDP traffic, a relatively high batch +size can reduce system overhead and improve performance. However, this parameter +should not be overdone. For each buffer, max message size bytes are statically +required. Also, a too-high number leads to reduced efficiency, as some structures +need to be completely initialized before the OS call is done. We would suggest to not +set it above a value of 128, except if experimental results show that this is useful. +<li><b>threads</b> <number> (default 1), available since 7.5.5<br> +Number of worker threads to process incoming messages. These +threads are utilized to pull data off the network. On a busy system, additional +threads (but not more than there are CPUs/Cores) can help improving +performance and avoiding message loss. Note that with too many threads, performance +can suffer. +There is a hard upper limit on the number of threads that can be defined. +Currently, this limit is set to 32. It may increase in the future when massive +multicore processors become available. </ul> <p><b>Input Parameters</b>:</p> <ul> @@ -75,7 +104,36 @@ are defined for a single input and each of the inputnames shall be unique. Note that there currently is no differentiation between IPv4/v6 listeners on the same port. </li> +<li><b>defaultTZ</b> <timezone-info><br> +This is an <b>experimental</b> parameter; details may change at any time and it may +also be discoutinued without any early warning.<br> +Permits to set a default timezone for this listener. This is useful when working with +legacy syslog (RFC3164 et al) residing in different timezones. If set it will be used as +timezone for all messages <b>that do not contain timezone info</b>. +Currently, the format <b>must</b> be "+/-hh:mm", e.g. "-05:00", "+01:30". Other formats, +including TZ names (like EST) are NOT yet supported. Note that consequently no daylight +saving settings are evaluated when working with timezones. If an invalid format is used, +"interesting" things can happen, among them malformed timestamps and rsyslogd segfaults. +This will obviously be changed at the time this feature becomes non-experimental.</li> +<li><b>rcvbufSize</b> [size] - (available since 7.5.3) +This request a socket receive buffer of specific size from the operating system. +It is an expert parameter, which should only be changed for a good reason. Note that +setting this parameter disables Linux auto-tuning, which usually works pretty well. +The default value is 0, which means "keep the OS buffer size unchanged". This is a size +value. So in addition to pure integer values, sizes like "256k", "1m" and the like can +be specified. Note that setting very large sizes may require root or other special +privileges. Also note that the OS may slightly adjust the value or shrink it to a +system-set max value if the user is not sufficiently privileged. Technically, this +parameter will result in a setsockopt() call with SO_RCVBUF (and SO_RCVBUFFORCE if it +is available). </ul> +<p><b>See Also</b> +<ul> +<li>Description of +<a href="http://www.rsyslog.com/rsyslog-statistic-counter/">rsyslog statistic counters</a> +This also describes all imudp counters. +</ul> +<p> <b>Caveats/Known Bugs:</b> <ul> <li>Scheduling parameters are set <b>after</b> privileges have been dropped. @@ -91,6 +149,13 @@ user account. input(type="imudp" port="514") </textarea> +<p>The following sample is mostly equivalent to the first one, but request a +larger rcvuf size. Note that 1m most probably will not be honored by the OS +until the user is sufficiently privileged.</p> +<textarea rows="3" cols="60">module(load="imudp") # needs to be done just once +input(type="imudp" port="514" rcvbufSize="1m") +</textarea> + <p>In the next example, we set up three listeners at ports 10514, 10515 and 10516 and assign a listner name of "udp" to it, followed by the port number: </p> @@ -110,7 +175,6 @@ input(type="imudp" port=["10514","10515","10516"] inputname="" inputname.appendPort="on") </textarea> - <p><b>Legacy Configuration Directives</b>:</p> <p>Multiple receivers may be configured by specifying $UDPServerRun multiple times. @@ -129,7 +193,7 @@ equivalent to: SchedulingPolicy <li>$IMUDPSchedulingPriority <number> Available since 4.7.4+, 5.7.3+, 6.1.3+.<br> equivalent to: SchedulingPriority </ul> -<p><b>Sample:</b></p> +<p><b>Legacy Sample:</b></p> <p>This sets up an UPD server on port 514:<br> </p> <textarea rows="3" cols="60">$ModLoad imudp # needs to be done just once diff --git a/doc/imuxsock.html b/doc/imuxsock.html index 123771f..02b2f18 100644 --- a/doc/imuxsock.html +++ b/doc/imuxsock.html @@ -42,7 +42,7 @@ severity and configure things accordingly. To turn off rate limiting, set the interval to zero. <p><b>Unix log sockets can be flow-controlled.</b> That is, if processing queues fill up, the unix socket reader is blocked for a short while. This may be useful to prevent overruning -the queues (which may cause exessive disk-io where it actually would not be needed). However, +the queues (which may cause excessive disk-io where it actually would not be needed). However, flow-controlling a log socket (and especially the system log socket) can lead to a very unresponsive system. As such, flow control is disabled by default. That means any log records are places as quickly as possible into the processing queues. If you would like to have @@ -125,7 +125,7 @@ module documentation for a more in-depth description. to the next socket.</li> <li><b>RateLimit.Interval</b> [number] - specifies the rate-limiting interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number -of seconds (5 recommended) to activate rate-limiting. The default of 0 has been choosen +of seconds (5 recommended) to activate rate-limiting. The default of 0 has been chosen as people experienced problems with this feature activated by default. Now it needs an explicit opt-in by setting this parameter. </li> diff --git a/doc/licensing.html b/doc/licensing.html index 93a5093..0a4ab1f 100644 --- a/doc/licensing.html +++ b/doc/licensing.html @@ -22,7 +22,7 @@ real details, check source files and the files COPYING and COPYING.LESSER inside <p>Each of these components can be thought of as individual projects. In fact, some of the plugins have different main authors than the rest of the rsyslog package. All of these components are currently put together into a single "rsyslog" package (tarball) for -convinience: this makes it easier to distribute a consistent version where everything +convenience: this makes it easier to distribute a consistent version where everything is included (and in the right versions) to build a full system. Platform package maintainers in general take the overall package and split off the individual components, so that users can install only what they need. In source installations, this can be done via the diff --git a/doc/log_rotation_fix_size.html b/doc/log_rotation_fix_size.html index 51edf03..7cdecc6 100644 --- a/doc/log_rotation_fix_size.html +++ b/doc/log_rotation_fix_size.html @@ -31,7 +31,7 @@ Channels to achieve this. Putting the following directive</p> <p><pre> # start log rotation via outchannel -# outchannel definiation +# outchannel definition $outchannel log_rotation,/var/log/log_rotation.log, 52428800,/home/me/./log_rotation_script # activate the channel and log everything to it *.* :omfile:$log_rotation @@ -49,13 +49,13 @@ mv -f /var/log/log_rotation.log /var/log/log_rotation.log.1 <p>This moves the original log to a kind of backup log file. After the action was successfully performed rsyslog creates a new /var/log/log_rotation.log -file and fill it up with new logs. So the latest logs are always in log_roatation.log.</p> +file and fill it up with new logs. So the latest logs are always in log_rotation.log.</p> <h2>Conclusion</h2> <p>With this approach two files for logging are used, each with a maximum size of 50 MB. So we can say we have successfully configured a log rotation which satisfies our requirement. -We keep the logs at a fixed-size level of100 MB.</p> +We keep the logs at a fixed-size level of 100 MB.</p> <p>[<a href="manual.html">manual index</a>] [<a href="rsyslog_conf.html">rsyslog.conf</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> diff --git a/doc/lookup_tables.html b/doc/lookup_tables.html index 4ef5d59..e1fcd08 100644 --- a/doc/lookup_tables.html +++ b/doc/lookup_tables.html @@ -104,7 +104,7 @@ lookup_table(name="name" file="/path/to/file" reloadOnHUP="on|off") <h3>lookup() Function</h3> <p>This function is used to actually do the table lookup. Format: <pre> -lookup_table("name", indexvalue) +lookup("name", indexvalue) </pre> <h4>Parameters</h4> <ul> diff --git a/doc/manual.html b/doc/manual.html index a160ed3..c132775 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -19,7 +19,7 @@ professional services</a> available directly from the source!</p> <p><b>Please visit the <a href="http://www.rsyslog.com/sponsors">rsyslog sponsor's page</a> to honor the project sponsors or become one yourself!</b> We are very grateful for any help towards the project goals.</p> -<p><b>This documentation is for version 7.4.8 (v7.4-stable branch) of rsyslog.</b> +<p><b>This documentation is for version 7.6.1 (v7.6-stable branch) of rsyslog.</b> Visit the <i><a href="http://www.rsyslog.com/status">rsyslog status page</a></i></b> to obtain current version information and project status. </p><p><b>If you like rsyslog, you might diff --git a/doc/messageparser.html b/doc/messageparser.html index 370db59..d22021d 100644 --- a/doc/messageparser.html +++ b/doc/messageparser.html @@ -23,7 +23,7 @@ the rsyslog code). like input and output modules). That means that new message parsers can be added without modifying the rsyslog core, even without contributing something back to the project. -<p>But that doesn't answer what a message parser really is. What does ist mean to "parse a +<p>But that doesn't answer what a message parser really is. What does it mean to "parse a message" and, maybe more importantly, what is a message? To answer these questions correctly, we need to dig down into the relevant standards. <a href="http://tools.ietf.org/html/rfc5424">RFC5424</a> specifies a layered architecture @@ -49,7 +49,7 @@ reason) a single message into two and encapsulates these into two frames, there a message parser could undo that. <p>A typical example may be a multi-line message: let's assume some originator has generated a message for the format "A\nB" (where \n means LF). If that message is being transmitted -via plain tcp syslog, the frame delimiter is LF. So the sender will delimite the frame with +via plain tcp syslog, the frame delimiter is LF. So the sender will delimit the frame with LF, but otherwise send the message unmodified onto the wire (because that is how things are -unfortunately- done in plain tcp syslog...). So wire will see "A\nB\n". When this arrives at the receiver, the transport layer will undo the framing. When it sees the LF @@ -58,7 +58,7 @@ the receive will extract one complete message A and one complete message B, not that they once were both part of a large multi-line message. These two messages are then passed to the upper layers, where the message parsers receive them and extract information. However, the message parsers never know (or even have a chance to see) that A and B -belonged together. Even further, in rsyslog there is no guarnatee that A will be parsed +belonged together. Even further, in rsyslog there is no guarantee that A will be parsed before B - concurrent operations may cause the reverse order (and do so very validly). <p>The important lesson is: <b>message parsers can not be used to fix a broken framing</b>. You need a full protocol implementation to do that, what is the domain of input and @@ -73,10 +73,10 @@ the real-world evil that you can usually see. So I won't repeat that here. But i real problem is not the framing, but how to make malformed messages well-looking. <p><b>This is what message parsers permit you to do: take a (well-known) malformed message, parse it according to its semantics and generate perfectly valid internal message representations -from it.</b> So as long as messages are consistenly in the same wrong format (and they usually +from it.</b> So as long as messages are consistently in the same wrong format (and they usually are!), a message parser can look at that format, parse it, and make the message processable just -like it were wellformed in the first place. Plus, one can abuse the interface to do some other -"intersting" tricks, but that would take us to far. +like it were well formed in the first place. Plus, one can abuse the interface to do some other +"interesting" tricks, but that would take us to far. <p>While this functionality may not sound exciting, it actually solves a very big issue (that you only really understand if you have managed a system with various different syslog sources). Note that we were often able to process malformed messages in the past with the help of the @@ -113,15 +113,15 @@ interface probably extended, to support generic filter modules. These would need to the root of the parser chain. As mentioned, the current system already supports this. <p>The position inside the parser chain can be thought of as a priority: parser sitting earlier in the chain take precedence over those sitting later in it. So more specific -parser should go ealier in the chain. A good example of how this works is the default parser +parser should go earlier in the chain. A good example of how this works is the default parser set provided by rsyslog: rsyslog.rfc5424 and rsyslog.rfc3164, each one parses according to the rfc that has named it. RFC5424 was designed to be distinguishable from RFC3164 message by the sequence "1 " immediately after the so-called PRI-part (don't worry about these words, it is -sufficient if you understand there is a well-defined sequence used to indentify RFC5424 +sufficient if you understand there is a well-defined sequence used to identify RFC5424 messages). In contrary, RFC3164 actually permits everything as a valid message. Thus the RFC3164 parser will always parse a message, sometimes with quite unexpected outcome (there is a lot of guesswork involved in that parser, which unfortunately is unavoidable due to -existing techology limits). So the default parser chain is to try the RFC5424 parser first +existing technology limits). So the default parser chain is to try the RFC5424 parser first and after it the RFC3164 parser. If we have a 5424-formatted message, that parser will identify and parse it and the rsyslog engine will stop processing. But if we receive a legacy syslog message, the RFC5424 will detect that it can not parse it, return this status @@ -139,16 +139,16 @@ case, rsyslog has no other choice than to discard the message. If it does so, it a warning message, but only in the first 1,000 incidents. This limit is a safety measure against message-loops, which otherwise could quickly result from a parser chain misconfiguration. <b>If you do not tolerate loss of unparsable messages, you must ensure -that each message can be parsed.</b> You can easily achive this by always using the +that each message can be parsed.</b> You can easily achieve this by always using the "rsyslog-rfc3164" parser as the <i>last</i> parser inside parser chains. That may result in invalid parsing, but you will have a chance to see the invalid message (in debug mode, a warning message will be written to the debug log each time a message is dropped due to inability to parse it). <h3>Where are parser chains used?</h3> <p>We now know what parser chains are and how they operate. The question is now how many -parser chains can be active and how it is decicded which parser chain is used on which message. +parser chains can be active and how it is decided which parser chain is used on which message. This is controlled via <a href="multi_ruleset.html">rsyslog's rulesets</a>. In short, multiple -rulesets can be defined and there always exist at least one ruleset (for specifcs, follow +rulesets can be defined and there always exist at least one ruleset (for specifics, follow the <a href="multi_ruleset.html">link</a>). A parser chain is bound to a specific ruleset. This is done by virtue of defining parsers via the <a href="rsconf1_rulesetparser.html">$RulesetParser</a> configuration directive (for specifics, @@ -161,22 +161,22 @@ is added to the end of the (initially empty) ruleset's parser chain. <p>The correct answer is: generally yes, but it depends. First of all, remember that input modules (and specific listeners) may be bound to specific rulesets. As parser chains "reside" in rulesets, binding to a ruleset also binds to the parser chain that is bound to that ruleset. -As a number one prequisite, the input module must support binding to different rulesets. Not +As a number one prerequisite, the input module must support binding to different rulesets. Not all do, but their number is growing. For example, the important <a href="imudp.html">imudp</a> and <a href="imtcp.html">imtcp</a> input modules support that functionality. Those that do not (for example <a href="im3195">im3195</a>) can only utilize the default ruleset and thus the parser chain defined in that ruleset. <p>If you do not know if the input module in question supports ruleset binding, check -its documentation page. Those that support it have the requiered directives. +its documentation page. Those that support it have the required directives. <p>Note that it is currently under evaluation if rsyslog will support binding parser chains to specific inputs directly, without depending on the ruleset. There are some concerns that this may not be necessary but adds considerable complexity to the configuration. So this may or may not be possible in the future. In any case, if we decide to add it, input modules need to support it, so this functionality would require some time to implement. -<p>The coockbook recipe for using different parsers for different devices is given +<p>The cookbook recipe for using different parsers for different devices is given as an actual in-depth example in the <a href="rscon1_rulesetsparser.html">$RulesetParser</a> -configuration directive doc page. In short, it is acomplished by defining specific rulesets -for the required parser chains, definining different listener ports for each of the devices +configuration directive doc page. In short, it is accomplished by defining specific rulesets +for the required parser chains, defining different listener ports for each of the devices with different format and binding these listeners to the correct ruleset (and thus parser chains). Using that approach, a variety of different message formats can be supported via a single rsyslog instance. @@ -185,19 +185,19 @@ via a single rsyslog instance. <p>As of this writing, there exist only two message parsers, one for RFC5424 format and one for legacy syslog (loosely described in <a href="http://tools.ietf.org/html/rfc3164">RFC3164</a>). These parsers are built-in and -must not be explicitely loaded. However, message parsers can be added with relative ease +must not be explicitly loaded. However, message parsers can be added with relative ease by anyone knowing to code in C. Then, they can be loaded via $ModLoad just like any other loadable module. It is expected that the rsyslog project will be contributed additional message parsers over time, so that at some point there hopefully is a rich choice of them (I intend to add a browsable repository as soon as new parsers pop up). <h3>How to write a message parser?</h3> -<p>As a prequisite, you need to know the exact format that the device is sending. Then, you need +<p>As a prerequisite, you need to know the exact format that the device is sending. Then, you need moderate C coding skills, and a little bit of rsyslog internals. I guess the rsyslog specific part should not be that hard, as almost all information can be gained from the existing parsers. They are rather simple in structure and can be found under the "./tools" directory. They are named pmrfc3164.c and pmrfc5424.c. You need to follow the usual loadable module guidelines. It is my expectation that writing a parser should typically not take longer than a single -day, with maybe a day more to get aquainted with rsyslog. Of course, I am not sure if the number +day, with maybe a day more to get acquainted with rsyslog. Of course, I am not sure if the number is actually right. <p>If you can not program or have no time to do it, Adiscon can also write a message parser for you as @@ -209,7 +209,7 @@ provide a fast and efficient solution for this problem. Different parsers can be different devices, and they all convert message information into rsyslog's well-defined internal format. Message parsers were first introduced in rsyslog 5.3.4 and also offer some interesting ideas that may be explored in the future - up to full message normalization -capabilities. It is strongly recommended that anyone with a heterogenous environment take +capabilities. It is strongly recommended that anyone with a heterogeneous environment take a look at message parser capabilities. <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual diff --git a/doc/mmanon.html b/doc/mmanon.html index e14d75c..984872b 100644 --- a/doc/mmanon.html +++ b/doc/mmanon.html @@ -60,6 +60,12 @@ part is always zero-filled and replacementChar is of no use. If it is specified, an error message is emitted and the parameter ignored. </ul> +<p><b>See Also</b> +<ul> +<li><a href="http://www.rsyslog.com/howto-anonymize-messages-that-go-to-specific-files/">Howto anonymize messages that go to specific files</a> +</ul> + + <p><b>Caveats/Known Bugs:</b> <ul> <li><b>only IPv4</b> is supported diff --git a/doc/mmcount.html b/doc/mmcount.html new file mode 100644 index 0000000..1d06340 --- /dev/null +++ b/doc/mmcount.html @@ -0,0 +1,58 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>mmcount</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>mmcount</h1> +<p><b>Module Name: mmcount</b></p> +<p><b>Author: </b>Bala.FA <barumuga@redhat.com></p> +<p><b>Status: </b>Non project-supported module - contact author +or rsyslog mailing list for questions +<p><b>Available since</b>: 7.5.0</p> +<p><b>Description</b>:</p> +<p> +<pre> + mmcount: message modification plugin which counts messages + + This module provides the capability to count log messages by severity + or json property of given app-name. The count value is added into the + log message as json property named 'mmcount' + + Example usage of the module in the configuration file + + module(load="mmcount") + + # count each severity of appname gluster + action(type="mmcount" appname="gluster") + + # count each value of gf_code of appname gluster + action(type="mmcount" appname="glusterd" key="!gf_code") + + # count value 9999 of gf_code of appname gluster + action(type="mmcount" appname="glusterfsd" key="!gf_code" value="9999") + + # send email for every 50th mmcount + if $app-name == 'glusterfsd' and $!mmcount <> 0 and $!mmcount % 50 == 0 then { + $ActionMailSMTPServer smtp.example.com + $ActionMailFrom rsyslog@example.com + $ActionMailTo glusteradmin@example.com + $template mailSubject,"50th message of gf_code=9999 on %hostname%" + $template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'" + $ActionMailSubject mailSubject + $ActionExecOnlyOnceEveryInterval 30 + :ommail:;RSYSLOG_SyslogProtocol23Format + } +</pre> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/mmfields.html b/doc/mmfields.html new file mode 100644 index 0000000..885d6bc --- /dev/null +++ b/doc/mmfields.html @@ -0,0 +1,91 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>Field Extraction Module (mmfields)</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>Fields Extraction Module (mmfields)</h1> +<p><b>Module Name: mmfields</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Available since</b>: 7.5.1</p> +<p><b>Description</b>:</p> +<p>The mmfield module permits to extract fields. It is an alternate to using +the property replacer field extraction capabilities. In contrast to the +property replacer, all fields are extracted as once and stored inside the +structured data part (more precisely: they become Lumberjack [JSON] properties). +<p>Using this module is of special advantage if a field-based log format +is to be processed, like for example CEF <b>and</b> and either a large +number of fields is needed or a specific field is used multiple times +inside filters. In these scenarios, mmfields potentially offers better +performance than the property replacer of the RainerScript field extraction +method. The reason is that mmfields extracts all fields as one big sweep, +whereas the other methods extract fields individually, which requires +multiple passes through the same data. On the other hand, adding field +content to the rsyslog property dictionary also has some overhead, +so for high-performance use cases it is suggested to do some performance +testing before finally deciding which method to use. This is most important +if only a smaller subset of the fields is actually needed. +<p>In any case, mmfields provides a very handy and easy to use way to +parse structured data into a it's individual data items. Again, a primiary +use case was support for CEF (Common Event Format), which is made +extremely easy to do with this module. +<p>This module is implemented via the action interface. Thus it +can be conditionally used depending on some prequisites. +<p> </p> + +<p><b>Module Configuration Parameters</b>:</p> +<p>Currently none. +<p> </p> +<p><b>Action Confguration Parameters</b>:</p> +<ul> +<li><b>separator</b> - separatorChar (default ',')<br> +This is the character used to separate fields. Currently, only a single +character is permitted, while the RainerScript method permits to +specify multi-character separator strings. For CEF, this is not required. +If there is actual need to support multi-character separator strings, +support can relatively easy be added. It is suggested to request it on the +rsyslog mailing list, together with the use case - we intend to add +functionality only if there is a real use case behind the request +(in the past we too-often implemented things that actually never got used). +<br>The fields are named f<i>nbr</i>, where <i>nbr</i> is the field number +starting with one and being incremented for each field. +<li><b>jsonRoot</b> - path (default "!")<br> +This parameters specifies into which json path the extracted fields shall +be written. The default is to use the json root object itself. +</ul> + +<p><b>Caveats/Known Bugs:</b> +<ul> +<li>Currently none. +</ul> + +<p><b>Samples:</b></p> +<p>This is a very simple use case where each message is +parsed. The default separator character of comma is being used. +<p><textarea rows="5" cols="60">module(load="mmfields") +template(name="ftpl" type=string string="%$!%\n") +action(type="omfields") +action(type="omfile" file="/path/to/logfile" template="ftpl") +</textarea> + +<p>The following sample is similar to the previous one, but +this time the colon is used as separator and data is written +into the "$!mmfields" json path. +<p><textarea rows="5" cols="60">module(load="mmfields") +template(name="ftpl" type=string string="%$!%\n") +action(type="omfields" separator=":" jsonRoot="!mmfields") +action(type="omfile" file="/path/to/logfile" template="ftpl") +</textarea> + + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/mmjsonparse.html b/doc/mmjsonparse.html index c2c862d..20a44f7 100644 --- a/doc/mmjsonparse.html +++ b/doc/mmjsonparse.html @@ -15,6 +15,16 @@ that follow the CEE/lumberjack spec. The so-called "CEE cookie" is checked and, if present, the JSON-encoded structured message content is parsed. The properties are than available as original message properties. </p> +<p>The "CEE cookie" is the character squence "@cee:" which must prepend the +actual JSON. Note that the JSON must be valid and MUST NOT be followed by +any non-JSON message. If either of these conditions is not true, mmjsonparse +will <b>not</b> parse the associated JSON. This is based on the cookie +definition used in CEE/project lumberjack and is meant to aid against +an errornous detection of a message as being CEE where it is not. +<p>This also means that mmjsonparse currently is NOT a generic JSON +parser that picks up JSON from whereever it may occur in the message. This +is intentional, but future versions may support config parameters to +relax the format requirements. <p><b>Action specific Configuration Directives</b>:</p> <p>currently none <ul> diff --git a/doc/mmnormalize.html b/doc/mmnormalize.html index 8110023..fc6ec6d 100644 --- a/doc/mmnormalize.html +++ b/doc/mmnormalize.html @@ -17,7 +17,7 @@ a normal form. This is done so quickly, that it should be possible to normalize events in realtime. <p>This module is implemented via the output module interface. This means that mmnormalize should be called just like an action. After it has been called, -the normalized message properties are avaialable and can be accessed. These properties +the normalized message properties are available and can be accessed. These properties are called the "CEE/lumberjack" properties, because liblognorm creates a format that is inspired by the CEE/lumberjack approach. <p><b>Please note:</b> CEE/lumberjack properties are different from regular properties. diff --git a/doc/mmpstrucdata.html b/doc/mmpstrucdata.html new file mode 100644 index 0000000..8197d94 --- /dev/null +++ b/doc/mmpstrucdata.html @@ -0,0 +1,82 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>RFC5424 structured data parsing module (mmpstrucdata)</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>RFC5424 structured data parsing module (mmpstrucdata)</h1> +<p><b>Module Name: mmpstrucdata</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Available since</b>: 7.5.4</p> +<p><b>Description</b>:</p> +<p>The mmpstrucdata parses RFC5424 structured data into the message +json variable tree. +The data parsed, if available, is stored under "jsonRoot!rfc5424-sd!...". +<p> </p> + +<p><b>Module Configuration Parameters</b>:</p> +<p>Currently none. +<p> </p> +<p><b>Action Confguration Parameters</b>:</p> +<ul> +<li><b>jsonRoot</b> - default "!"<br> +Specifies into which json container the data shall be parsed to. +</ul> + +<p><b>See Also</b> +<ul> +<li><a href="http://www.rsyslog.com/howto-anonymize-messages-that-go-to-specific-files/">Howto anonymize messages that go to specific files</a> +</ul> + + +<p><b>Caveats/Known Bugs:</b> +<ul> +<li>this module is currently experimental; feedback is appreciated +<li>property names are treated case-insensitive in rsyslog. As such, +RFC5424 names are treated case-insensitive as well. If such names +only differ in case (what is not recommended anyways), problems will +occur. +<li>structured data with duplicate SD-IDs and SD-PARAMS is not + properly processed +</ul> + +<p><b>Samples:</b></p> +<p>In this snippet, we parse the message and emit all json variable to a file +with the message anonymized. Note that once mmpstrucdata has run, access to the +original message is no longer possible (execept if stored in user +variables before anonymization). +<p><textarea rows="5" cols="60">module(load="mmpstrucdata") +action(type="mmpstrucdata") +template(name="jsondump" type="string" string="%msg%: %$!%\n") +action(type="omfile" file="/path/to/log" template="jsondump") +</textarea> + +<p><b>A more practical one:</b> +<p>Take this example message (inspired by RFC5424 sample;)): +<p><code><34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"][id@2 test="tast"] BOM'su root' failed for lonvick on /dev/pts/8</code> +<p>We apply this configuration: +<p><textarea rows="6" cols="120">module(load="mmpstrucdata") +action(type="mmpstrucdata") +template(name="sample2" type="string" + string="ALL: %$!%\nSD: %$!RFC5424-SD%\nIUT:%$!rfc5424-sd!exampleSDID@32473!iut%\nRAWMSG: %rawmsg%\n\n") +action(type="omfile" file="/path/to/log" template="sample2") +</textarea> +<p>This will output: +<p><code>ALL: { "rfc5424-sd": { "examplesdid@32473": { "iut": "3", "eventsource": "Application", "eventid": "1011" }, "id@2": { "test": "tast" } } }</br> +SD: { "examplesdid@32473": { "iut": "3", "eventsource": "Application", "eventid": "1011" }, "id@2": { "test": "tast" } }</br> +IUT:3</br> +RAWMSG: <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"][id@2 test="tast"] BOM'su root' failed for lonvick on /dev/pts/8</code> +<p>As you can seem, you can address each of the individual items. Note that the +case of the RFC5424 parameter names has been converted to lower case. + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/mmrfc5424addhmac.html b/doc/mmrfc5424addhmac.html new file mode 100644 index 0000000..a54908c --- /dev/null +++ b/doc/mmrfc5424addhmac.html @@ -0,0 +1,88 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>mmrfc5424addhmac</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>mmrfc5424addhmac</h1> +<p><b>Module Name: mmrfc5424addhmac</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Available since</b>: 7.5.6</p> +<p><b>Description</b>:</p> +<p>This module adds a hmac to RFC5424 structured data if not already present. +This is a custom module and uses openssl as requested by the sponsor. +This works exclusively for RFC5424 formatted messages; all others are ignored. +<p>If both <a href="mmpstrucdata.html">mmpstrucdata</a> +and mmrfc5424addhmac are to be used, the recommended calling sequence is +<ol> +<li>mmrfc5424addhmac +<li>mmpstrucdata +</ol> +with that sequence, the generated hash will become available for mmpstrucdata. +<p> </p> + +<p><b>Module Configuration Parameters</b>:</p> +<p>Currently none. +<p> </p> +<p><b>Action Confguration Parameters</b>:</p> +<ul> +<li><b>key</b><br> +The "key" (string) to be used to generate the hmac. +<li><b>hashfunction</b><br> +An openssl hash function name for the function to be used. This is passed +on to openssl, so see the openssl list of supported function names. +<li><b>sd_id</b><br> +The RFC5424 structured data ID to be used by this module. This is the +SD-ID that will be added. Note that nothing is added if this SD-ID +is already present. +</ul> + +<p><b>Verification method</b> +<p>rsyslog does not contain any tools to verify a log file (this was not +part of the custom project). So you need to write your own verifier. +<p>When writing the verifier, keep in mind that the log file contains messages +with the hash SD-ID included. For obvious reasons, this SD-ID was not present when +the hash was created. So before the actual verification is done, this SD-ID must be +removed, and the remaining (original) message be verified. Also, it is important to +note that the output template must write the exact same message format that was +received. Otherwise, a verification failure will obviously occur - and must +so, because the message content actually was altered. +<p>So in a more formal description, verification of a message m can be done as follows: +<ol> +<li>let m' be m with the configured SD-ID removed (everything between []). Otherwise, +m' must be an exact duplicate of m. +<li>call openssl's HMAC function as follows:<br> +<code>HMAC(hashfunction, key, len(key), m', len(m'), hash, &hashlen);</code></br> +Where hashfunction and key are the configured values and hash is an output +buffer for the hash. +<li>let h be the extracted hash value obtained from m within the relevant SD-ID. Be sure to convert the hex string back to the actual byte values. +<li>now compare hash and h under consideration of the sizes. If these values match +the verification succeeds, otherwise the message was modified. +</ol> +<p>If you neeed help implementing a verifier function or want to sponsor development +of a verification tool, please simply email +<a href="sales@adiscon.com">sales@adiscon.com</a> for a quote. + +<p><b>See Also</b> +<ul> +<li><a href="http://www.rsyslog.com/how-to-add-a-hmac-to-rfc5424-structured-data-messages/">How to +add a HMAC to RFC5424 messages</a> +</ul> + +<p><b>Caveats/Known Bugs:</b> +<ul> +<li>none +</ul> + + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/mmsequence.html b/doc/mmsequence.html new file mode 100644 index 0000000..75ac57b --- /dev/null +++ b/doc/mmsequence.html @@ -0,0 +1,148 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>mmsequence</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>Number generator and counter module (mmsequence)</h1> +<p><b>Module Name: mmsequence</b></p> +<p><b>Author: </b>Pavel Levshin <pavel@levshin.spb.ru></p> +<p><b>Status: </b>Non project-supported module - contact author +or rsyslog mailing list for questions</p> +<p><b>Available since</b>: 7.5.6</p> +<p><b>Description</b>:</p> +<p>This module generates numeric sequences of different kinds. It can be used +to count messages up to a limit and to number them. It can generate random +numbers in a given range.</p> + +<p>This module is implemented via the output module interface, so it is +called just as an action. The number generated is stored in a variable.</p> +<p> </p> +<p><b>Action Parameters</b>:</p> +<ul> +<li><b>mode</b> "random" or "instance" or "key" + <p>Specifies mode of the action. In "random" mode, the module + generates uniformly distributed integer numbers in a range defined + by "from" and "to".</p> + + <p>In "instance" mode, which is default, the action produces a counter + in range [from, to). This counter is specific to this action instance.</p> + + <p>In "key" mode, the counter can be shared between multiple instances. + This counter is identified by a name, which is defined with "key" + parameter.</p> +</li> +<li><b>from</b> [non-negative integer], default "0" + <p>Starting value for counters and lower margin for random generator.</p> +</li> +<li><b>to</b> [positive integer], default "INT_MAX" + <p>Upper margin for all sequences. Note that this margin is not + inclusive. When next value for a counter is equal or greater than + this parameter, the counter resets to the starting value.</p> +</li> +<li><b>step</b> [non-negative integer], default "1" + <p>Increment for counters. If step is "0", it can be used to fetch + current value without modification. The latter not applies to + "random" mode. This is useful in "key" mode or to get constant + values in "instance" mode.</p> +</li> +<li><b>key</b> [word], default "" + <p>Name of the global counter which is used in this action.</p> +</li> +<li><b>var</b> [word], default "$!mmsequence" + <p>Name of the variable where the number will be stored. + Should start with "$".</p> +</li> +</ul> + + +<p><b>Sample</b>:</p> +<pre> +# load balance +Ruleset( + name="logd" + queue.workerthreads="5" + ){ + + Action( + type="mmsequence" + mode="instance" + from="0" + to="2" + var="$.seq" + ) + + if $.seq == "0" then { + Action( + type="mmnormalize" + userawmsg="on" + rulebase="/etc/rsyslog.d/rules.rb" + ) + } else { + Action( + type="mmnormalize" + userawmsg="on" + rulebase="/etc/rsyslog.d/rules.rb" + ) + } + + # output logic here +} + # generate random numbers + action( + type="mmsequence" + mode="random" + to="100" + var="$!rndz" + ) + # count from 0 to 99 + action( + type="mmsequence" + mode="instance" + to="100" + var="$!cnt1" + ) + # the same as before but the counter is global + action( + type="mmsequence" + mode="key" + key="key1" + to="100" + var="$!cnt2" + ) + # count specific messages but place the counter in every message + if $msg contains "txt" then + action( + type="mmsequence" + mode="key" + to="100" + var="$!cnt3" + ) + else + action( + type="mmsequence" + mode="key" + to="100" + step="0" + var="$!cnt3" + key="" + ) +</pre> + + +<p><b>Legacy Configuration Directives</b>:</p> + + <p>Not supported.</p> + + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/mmsnmptrapd.html b/doc/mmsnmptrapd.html index 699049d..fb50f6c 100644 --- a/doc/mmsnmptrapd.html +++ b/doc/mmsnmptrapd.html @@ -34,8 +34,8 @@ The following logic is applied to all message being processed: snmptrapd/severity/hostname. A configurable mapping table will be used to drive a new severity value from that severity string. If no mapping has been defined, the original severity is not changed. -<li>It replaces the "FromHost" value with the derived value from step2 -<li>It replaces the "Severity" value with the derived value from step 3 +<li>It replaces the "FromHost" value with the derived value from step 2 +<li>It replaces the "Severity" value with the derived value from step 3 </ol> <p>Note that the placement of this module inside the configuration is important. All actions before this modules is called will work on the unmodified message. All messages after it's call @@ -55,11 +55,11 @@ tells the module which start string inside the tag to look for. The default is matching incoming messages. It MUST not be given, except if two slashes are required for whatever reasons (so "tag/" results in a check for "tag//" at the start of the tag field). -<li><b>$mmsnmptrapdSeverityMapping</b> [severtiymap]<br> +<li><b>$mmsnmptrapdSeverityMapping</b> [severitymap]<br> This specifies the severity mapping table. It needs to be specified as a list. Note that due to the current config system <b>no whitespace</b> is supported inside the list, so be sure not to use any whitespace inside it.<br> -The list is constructed of Severtiy-Name/Severity-Value pairs, delimited by comma. +The list is constructed of Severity-Name/Severity-Value pairs, delimited by comma. Severity-Name is a case-sensitive string, e.g. "warning" and an associated numerical value (e.g. 4). Possible values are in the rage 0..7 and are defined in RFC5424, table 2. The diff --git a/doc/mmutf8fix.html b/doc/mmutf8fix.html new file mode 100644 index 0000000..6275c17 --- /dev/null +++ b/doc/mmutf8fix.html @@ -0,0 +1,105 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>Fix invalid UTF-8 Sequences (mmutf8fix)</title></head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>Fix invalid UTF-8 Sequences (mmutf8fix)</h1> +<p><b>Module Name: mmutf8fix</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Available since</b>: 7.5.4</p> +<p><b>Description</b>:</p> +<p>The mmutf8fix module permits to fix invalid UTF-8 sequences. +Most often, such invalid sequences result from syslog sources sending +in non-UTF character sets, e.g. ISO 8859. As syslog does not have a way +to convey the character set information, these sequences are not properly +handled. While they are typically uncritical with plain text files, they can +cause big headache with database sources as well as systems like ElasticSearch. +<p>The module supports different "fixing" modes and fixes. The current +implementation will always replace invalid bytes with a single US ASCII +character. Additional replacement modes will probably be added in the future, +depending on user demand. In the longer term +it could also be evolved into an any-charset-to-UTF8 converter. But +first let's see if it really gets into widespread enough use. + +<p><b>Proper Usage</b>:</p> +<p>Some notes are due for proper use of this module. This is a message modification +module utilizing the action interface, which means you call it like an action. +This gives great flexibility on the question on when and how to call this module. +Note that once it has been called, it actually modifies the message. The original +messsage is then no longer available. However, this does <b>not</b> change any +properties set, used or extracted before the modification is done. +<p>One potential use case is to normalize all messages. This is done by simply calling +mmutf8fix right in front of all other actions. +<p>If only a specific source (or set of sources) is known to cause problems, +mmutf8fix can be conditionally called only on messages from them. This also offers +performance benefits. If such multiple sources exists, it probably is a good idea +to define different listeners for their incoming traffic, bind them to specific +<a href="multi_ruleset.html">ruleset</a> and call mmutf8fix as first action in this +ruleset. + +<p><b>Module Configuration Parameters</b>:</p> +<p>Currently none. +<p> </p> +<p><b>Action Confguration Parameters</b>:</p> +<ul> +<li><b>mode</b> - <b>utf-8</b>/controlcharacters<br> +This sets the basic detection mode. +<br>In <b>utf-8</b> mode (the default), proper +UTF-8 encoding is checked and bytes which are not proper UTF-8 sequences +are acted on. If a proper multi-byte start sequence byte is detected but +any of the following bytes is invalid, the whole sequence is replaced by +the replacement method. This mode is most useful with non-US-ASCII character +sets, which validly includes multibyte sequences. Note that in this mode +control characters are NOT being replaced, because they are valid UTF-8. +<br>In <b>controlcharacters</b> mode, all bytes which do not represent a +printable US-ASCII character (codes 32 to 126) are replaced. Note that this +also mangles valid UTF-8 multi-byte sequences, as these are (deliberately) outside +of that character range. This mode is most useful if it is known that no +characters outside of the US-ASCII alphabet need to be processed. +<li><b>replacementChar</b> - default " " (space), a single character<br> +This is the character that invalid sequences are replaced by. Currently, it +MUST be a <b>printable</b> US-ASCII character. +</ul> + +<p><b>Caveats/Known Bugs:</b> +<ul> +<li>overlong UTF-8 encodings are currently not detected in utf-8 mode. +</ul> + +<p><b>Samples:</b></p> +<p>In this snippet, we write one file without fixing UTF-8 and another one +with the message fixed. Note that once mmutf8fix has run, access to the +original message is no longer possible. +<p><textarea rows="5" cols="60">module(load="mmutf8fix") +action(type="omfile" file="/path/to/non-fixed.log") +action(type="mmutf8fix") +action(type="omfile" file="/path/to/fixed.log") +</textarea> + +<p>In this sample, we fix only message originating from host 10.0.0.1. +<p><textarea rows="5" cols="60">module(load="mmutf8fix") +if $fromhost-ip == "10.0.0.1" then + action(type="mmutf8fix") +# all other actions here... +</textarea> + +<p>This is mostly the same as the previous sample, but uses "controlcharacters" +processing mode. +<p><textarea rows="5" cols="60">module(load="mmutf8fix") +if $fromhost-ip == "10.0.0.1" then + action(type="mmutf8fix" mode="controlcharacters") +# all other actions here... +</textarea> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/modules.html b/doc/modules.html index 4eae6db..0ed7d4f 100644 --- a/doc/modules.html +++ b/doc/modules.html @@ -70,7 +70,7 @@ other internal structures). Besides security, this also greatly simplifies the job of the output module developer.</p> <h2>Action Selectors</h2> <p>Modules (and rsyslog) need to know when they are called. For this, there must -a an action identification in selector lines. There are two syntaxes: the +be an action identification in selector lines. There are two syntaxes: the single-character syntax, where a single characters identifies a module (e.g. "*" for a wall message) and the modules designator syntax, where the module name is given between colons (e.g. ":ommysql:"). The single character syntax is diff --git a/doc/multi_ruleset.html b/doc/multi_ruleset.html index 83c495c..14a761c 100644 --- a/doc/multi_ruleset.html +++ b/doc/multi_ruleset.html @@ -5,7 +5,7 @@ <h1>Multiple Rulesets in rsyslog</h1> <p>Starting with version 4.5.0 and 5.1.1, <a href="http://www.rsyslog.com">rsyslog</a> supports multiple rulesets within a single configuration. -This is especially useful for routing the recpetion of remote messages to a set of specific rules. +This is especially useful for routing the reception of remote messages to a set of specific rules. Note that the input module must support binding to non-standard rulesets, so the functionality may not be available with all inputs. <p>In this document, I am using <a href="imtcp.html">imtcp</a>, an input module @@ -39,7 +39,7 @@ is the name space reserved for rsyslog use). If it finds this directive, it begi rule set (if the name was not yet know) or switches to an already-existing one (if the name was known). All rules defined between this $RuleSet directive and the next one are appended to the named ruleset. Note that the reserved name "RSYSLOG_DefaultRuleset" is used to -specify rsyslogd's default ruleset. You can use that name whereever you can use a ruleset name, +specify rsyslogd's default ruleset. You can use that name wherever you can use a ruleset name, including when binding an input to it. <p>Inside a ruleset, messages are processed as described above: they start with the first rule @@ -48,7 +48,7 @@ there are no more rules or the discard action is executed. Note that with multip no longer <b>all</b> rsyslog.conf rules are executed but <b>only</b> those that are contained within the specific ruleset. -<p>Inputs must explicitely bind to rulesets. If they don't do, the default ruleset is bound. +<p>Inputs must explicitly bind to rulesets. If they don't do, the default ruleset is bound. <p>This brings up the next question: @@ -58,10 +58,10 @@ it means that a specific input, or part of an input (like a tcp listener) will u ruleset to "pass its messages to". So when a new message arrives, it will be processed via the bound ruleset. Rule from all other rulesets are irrelevant and will never be processed. <p>This makes multiple rulesets very handy to process local and remote message via -seperate means: bind the respective receivers to different rule sets, and you do not need -to seperate the messages by any other method. +separate means: bind the respective receivers to different rule sets, and you do not need +to separate the messages by any other method. -<p>Binding to rulesets is input-specifc. For imtcp, this is done via the +<p>Binding to rulesets is input-specific. For imtcp, this is done via the <pre>input(type="imptcp" port="514" ruleset="rulesetname"); </pre> @@ -72,7 +72,7 @@ I personally think that it is best to define all rule sets at the top of rsyslog define the inputs at the bottom. This kind of reverses the traditional recommended ordering, but seems to be a really useful and straightforward way of doing things. <h2>Why are rulesets important for different parser configurations?</h2> -<p>Custom message parsers, used to handle differnet (and potentially otherwise-invalid) +<p>Custom message parsers, used to handle different (and potentially otherwise-invalid) message formats, can be bound to rulesets. So multiple rulesets can be a very useful way to handle devices sending messages in different malformed formats in a consistent way. Unfortunately, this is not uncommon in the syslog world. An in-depth explanation @@ -222,7 +222,7 @@ needs to insert messages into the main message queue. So if each of them wants t submit a newly arrived message into the queue at the same time, only one can do so while the others need to wait. With multiple rulesets, its own queue can be created for each ruleset. If now each listener is bound to its own ruleset, concurrent message submission is -possible. On a machine with a sufficiently large number of corse, this can result in +possible. On a machine with a sufficiently large number of cores, this can result in dramatic performance improvement. <p>It is highly advised that high-performance systems define a dedicated ruleset, with a dedicated queue for each of the inputs. diff --git a/doc/multi_ruleset_legacy_format.html b/doc/multi_ruleset_legacy_format.html index 273a4a0..03586ca 100644 --- a/doc/multi_ruleset_legacy_format.html +++ b/doc/multi_ruleset_legacy_format.html @@ -5,18 +5,18 @@ <h1>Multiple Rulesets in rsyslog</h1> <p>Starting with version 4.5.0 and 5.1.1, <a href="http://www.rsyslog.com">rsyslog</a> supports multiple rulesets within a single configuration. -This is especially useful for routing the recpetion of remote messages to a set of specific rules. +This is especially useful for routing the reception of remote messages to a set of specific rules. Note that the input module must support binding to non-standard rulesets, so the functionality -may not be available with all inputs.<p> +may not be available with all inputs.</p> <b>Attention: this guide is shortened and only contains the samples in legacy format.</b> -Please follow this link to the full guide in the new config format "list": <a href="http://www.rsyslog.com/doc/multi_ruleset.html">http://www.rsyslog.com/doc/multi_ruleset.html<a> +Please follow this link to the full guide in the new config format "list": <a href="http://www.rsyslog.com/doc/multi_ruleset.html">http://www.rsyslog.com/doc/multi_ruleset.html</a> <h2>Examples</h2> <h3>Split local and remote logging</h3> <p>Let's say you have a pretty standard system that logs its local messages to the usual bunch of files that are specified in the default rsyslog.conf. As an example, your rsyslog.conf -might look like this: +might look like this:</p> <pre> # ... module loading ... @@ -34,7 +34,7 @@ cron.* /var/log/cron <p>Now, you want to add receive messages from a remote system and log these to a special file, but you do not want to have these messages written to the files specified above. The traditional approach is to add a rule in front of all others that -filters on the message, processes it and then discards it: +filters on the message, processes it and then discards it:</p> <pre> # ... module loading ... @@ -55,7 +55,7 @@ cron.* /var/log/cron </pre> <p>Note the tilde character, which is the discard action!. Also note that we assume that -192.0.2.1 is the sole remote sender (to keep it simple). +192.0.2.1 is the sole remote sender (to keep it simple).</p> <p>With multiple rulesets, we can simply define a dedicated ruleset for the remote reception case and bind it to the receiver. This may be written as follows: @@ -88,7 +88,7 @@ cron.* /var/log/cron <p>Here, we need to switch back to the default ruleset after we have defined our custom one. This is why I recommend a different ordering, which I find more intuitive. The sample -below has it, and it leads to the same results: +below has it, and it leads to the same results:</p> <pre> # ... module loading ... @@ -116,27 +116,27 @@ $InputTCPServerRun 10514 </pre> <p>Here, we do not switch back to the default ruleset, because this is not needed as it is -completely defined when we begin the "remote" ruleset. +completely defined when we begin the "remote" ruleset.</p> <p>Now look at the examples and compare them to the single-ruleset solution. You will notice that we do <b>not</b> need a real filter in the multi-ruleset case: we can simply use "*.*" as all messages now means all messages that are being processed by this rule set and all of them come in via the TCP receiver! This is what makes using multiple -rulesets so much easier. +rulesets so much easier.</p> <h3>Split local and remote logging for three different ports</h3> <p>This example is almost like the first one, but it extends it a little bit. While it is very similar, I hope it is different enough to provide a useful example why you may want -to have more than two rulesets. +to have more than two rulesets.</p> <p>Again, we would like to use the "regular" log files for local logging, only. But this time we set up three syslog/tcp listeners, each one listening to a different port (in this example 10514, 10515, and 10516). Logs received from these receivers shall go into different files. Also, logs received from 10516 (and only from that port!) with "mail.*" priority, shall be written into a specif file and <b>not</b> be -written to 10516's general log file. +written to 10516's general log file.</p> -<p>This is the config: +<p>This is the config:</p> <pre> # ... module loading ... @@ -180,12 +180,12 @@ $InputTCPServerRun 10516 </pre> <p>Note that the "mail.*" rule inside the "remote10516" ruleset does -not affect processing inside any other rule set, including the default rule set. +not affect processing inside any other rule set, including the default rule set.</p> <p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> -project.<br> +project.<br/> Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/ns_gtls.html b/doc/ns_gtls.html index 0d02ad0..21a7f19 100644 --- a/doc/ns_gtls.html +++ b/doc/ns_gtls.html @@ -10,7 +10,7 @@ library</a>.</p> <p><b>Available since:</b> 3.19.0 (suggested minimum 3.19.8 and above)</p> <p style="font-weight: bold;">Supported Driver Modes</p> <ul> -<li>0 - unencrypted trasmission (just like <a href="ns_ptcp.html">ptcp</a> driver)</li> +<li>0 - unencrypted transmission (just like <a href="ns_ptcp.html">ptcp</a> driver)</li> <li>1 - TLS-protected operation</li> </ul> Note: mode 0 does not provide any benefit over the ptcp driver. This @@ -38,7 +38,7 @@ unauthorized access. It is recommended NOT to use this mode.</p> <p>x509/certvalid is a nonstandard mode. It validates the remote peers certificate, but does not check the subject name. This is weak authentication that may be useful in scenarios where multiple -devices are deployed and it is sufficient proof of authenticy when +devices are deployed and it is sufficient proof of authenticity when their certificates are signed by the CA the server trusts. This is better than anon authentication, but still not recommended. <b>Known Problems</b><br> diff --git a/doc/omelasticsearch.html b/doc/omelasticsearch.html new file mode 100644 index 0000000..618b706 --- /dev/null +++ b/doc/omelasticsearch.html @@ -0,0 +1,177 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html> + <head> + <meta content="en" http-equiv="Content-Language" /> + <title>Elasticsearch Output Module</title> + </head> + <body> + <p> + <a href="rsyslog_conf_modules.html">back</a></p> + <h1> + Elasticsearch Output Module</h1> + <p> + <b>Module Name: omelasticsearch</b></p> + <p> + <b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> + <p> + <b>Available since: </b>6.4.0+</p> + <p> + <b>Description</b>:</p> + <p> + This module provides native support for logging to <a href="http://www.elasticsearch.org/">Elasticsearch</a>.</p> + <p> + <b>Action Parameters</b>:</p> + <ul> + <li> + <b>server</b><br /> + Host name or IP address of the Elasticsearch server. Defaults to "localhost"</li> + <li> + <b>serverport</b><br /> + HTTP port to connect to Elasticsearch. Defaults to 9200</li> + <li> + <b>searchIndex</b><br /> + <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#index">Elasticsearch index</a> to send your logs to. Defaults to "system"</li> + <li> + <b>dynSearchIndex </b><on/<b>off</b>><br /> + Whether the string provided for <strong>searchIndex</strong> should be taken as a <a href="http://www.rsyslog.com/doc/rsyslog_conf_templates.html">template</a>. Defaults to "off", which means the index name will be taken literally. Otherwise, it will look for a template with that name, and the resulting string will be the index name. For example, let's assume you define a template named "date-days" containing "%timereported:1:10:date-rfc3339%". Then, with dynSearchIndex="on", if you say searchIndex="date-days", each log will be sent to and index named after the first 10 characters of the timestamp, like "2013-03-22".</li> + <li> + <b>searchType</b><br /> + <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#type">Elasticsearch type</a> to send your index to. Defaults to "events"</li> + <li> + <b>dynSearchType</b> <on/<strong>off</strong>><br /> + Like <strong>dynSearchIndex</strong>, it allows you to specify a <a href="http://www.rsyslog.com/doc/rsyslog_conf_templates.html">template</a> for <strong>searchType</strong>, instead of a static string.</li> + <li> + <strong>asyncrepl </strong><on/<strong>off</strong>><br /> + By default, an indexing operation returns after all <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#replica_shard">replica shards</a> have indexed the document. With asyncrepl="on" it will return after it was indexed on the <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#primary_shard">primary shard</a> only - thus trading some consistency for speed.</li> + <li> + <strong>timeout</strong><br /> + How long Elasticsearch will wait for a primary shard to be available for indexing your log before sending back an error. Defaults to "1m".</li> + <li> + <strong>template</strong><br /> + This is the JSON document that will be indexed in Elasticsearch. The resulting string needs to be a valid JSON, otherwise Elasticsearch will return an error. Defaults to:</li> + </ul> + <pre> +$template JSONDefault, "{\"message\":\"%msg:::json%\",\"fromhost\":\"%HOSTNAME:::json%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"timereported\":\"%timereported:::date-rfc3339%\",\"timegenerated\":\"%timegenerated:::date-rfc3339%\"}" +</pre> + <p> + Which will produce this sort of documents (pretty-printed here for readability):</p> + <ul> + </ul> + <pre> +{ + "message": " this is a test message", + "fromhost": "test-host", + "facility": "user", + "priority": "info", + "timereported": "2013-03-12T18:05:01.344864+02:00", + "timegenerated": "2013-03-12T18:05:01.344864+02:00" +}</pre> + <ul> + <li> + <strong>bulkmode </strong><on/<strong>off</strong>><br /> + The default "off" setting means logs are shipped one by one. Each in its own HTTP request, using the <a href="http://www.elasticsearch.org/guide/reference/api/index_.html">Index API</a>. Set it to "on" and it will use Elasticsearch's <a href="http://www.elasticsearch.org/guide/reference/api/bulk.html">Bulk API</a> to send multiple logs in the same request. The maximum number of logs sent in a single bulk request depends on your queue settings - usually limited by the <a href="http://www.rsyslog.com/doc/node35.html">dequeue batch size</a>. More information about queues can be found <a href="http://www.rsyslog.com/doc/node32.html">here</a>.</li> + <li> + <strong>parent</strong><br /> + Specifying a string here will index your logs with that string the parent ID of those logs. Please note that you need to define the <a href="http://www.elasticsearch.org/guide/reference/mapping/parent-field.html">parent field</a> in your <a href="http://www.elasticsearch.org/guide/reference/mapping/">mapping</a> for that to work. By default, logs are indexed without a parent.</li> + <li> + <strong>dynParent </strong><on/<strong>off</strong>><br /> + Using the same parent for all the logs sent in the same action is quite unlikely. So you'd probably want to turn this "on" and specify a <a href="http://www.rsyslog.com/doc/rsyslog_conf_templates.html">template</a> that will provide meaningful parent IDs for your logs.</li> + <li> + <strong>uid</strong><br /> + If you have basic HTTP authentication deployed (eg: through the <a href="https://github.com/Asquera/elasticsearch-http-basic">elasticsearch-basic plugin</a>), you can specify your user-name here.</li> + <li> + <strong>pwd</strong><br /> + Password for basic authentication.</li> + </ul> + <p> + <b>Samples:</b></p> + <p> + The following sample does the following:</p> + <ul> + <li> + loads the omelasticsearch module</li> + <li> + outputs all logs to Elasticsearch using the default settings</li> + </ul> + <pre> +module(load="omelasticsearch") +*.* action(type="omelasticsearch")</pre> + <p> + The following sample does the following:</p> + <ul> + <li> + loads the omelasticsearch module</li> + <li> + defines a template that will make the JSON contain the following properties (more info about what properties you can use <a href="http://www.rsyslog.com/doc/property_replacer.html">here</a>): + <ul> + <li> + RFC-3339 timestamp when the event was generated</li> + <li> + the message part of the event</li> + <li> + hostname of the system that generated the message</li> + <li> + severity of the event, as a string</li> + <li> + facility, as a string</li> + <li> + the tag of the event</li> + </ul> + </li> + <li> + outputs to Elasticsearch with the following settings + <ul> + <li> + host name of the server is myserver.local</li> + <li> + port is 9200</li> + <li> + JSON docs will look as defined in the template above</li> + <li> + index will be "test-index"</li> + <li> + type will be "test-type"</li> + <li> + activate bulk mode. For that to work effectively, we use an in-memory queue that can hold up to 5000 events. The maximum bulk size will be 300</li> + <li> + retry indefinitely if the HTTP request failed (eg: if the target server is down)</li> + </ul> + </li> + </ul> + <pre> +module(load="omelasticsearch") +template(name="testTemplate" + type="list" + option.json="on") { + constant(value="{") + constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339") + constant(value="\",\"message\":\"") property(name="msg") + constant(value="\",\"host\":\"") property(name="hostname") + constant(value="\",\"severity\":\"") property(name="syslogseverity-text") + constant(value="\",\"facility\":\"") property(name="syslogfacility-text") + constant(value="\",\"syslogtag\":\"") property(name="syslogtag") + constant(value="\"}") + } +*.* action(type="omelasticsearch" + server="myserver.local" + serverport="9200" + template="testTemplate" + searchIndex="test-index" + searchType="test-type" + bulkmode="on" + queue.type="linkedlist" + queue.size="5000" + queue.dequeuebatchsize="300" + action.resumeretrycount="-1")</pre> + <p> + </p> + <pre> +</pre> + <p> + [<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> + <p> + <font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br /> + Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the ASL 2.0.</font></p> + </body> +</html> + diff --git a/doc/omfile.html b/doc/omfile.html index 0f64f26..655e934 100644 --- a/doc/omfile.html +++ b/doc/omfile.html @@ -11,19 +11,37 @@ <p><b>Author: </b>Rainer Gerhards <rgergards@adiscon.com></p> <p><b>Description</b>:</p> <p>The omfile plug-in provides the core functionality of writing messages to files residing inside the local file system (which may actually be remote if methods like NFS are used). Both files named with static names as well files with names based on message content are supported by this module. It is a built-in module that does not need to be loaded. </p> -<p> </p> <p><b>Module Parameters</b>:</p> <ul> <li><strong>Template </strong>[templateName]<br> - sets a new default template for file actions.<br></li> + Set the default template to be used if an action is not + configured to use a specific template.<br></li> + + <li><strong>DirCreateMode </strong>[default 0700]<br> + Sets the default DirCreateMode to be used for an action + if no explicit one is specified.</br> + <li><strong>FileCreateMode </strong>[default 0644]<br> + Sets the default DirCreateMode to be used for an action + if no explicit one is specified.</br> </ul> <p> </p> <p><b>Action Parameters</b>:</p> <ul> - <li><strong>DynaFileCacheSize </strong>(not mandatory, default will be used)<br> - Defines a template to be used for the output. <br></li><br> + <li><strong>Template </strong>[templateName]<br> + Sets the template to be used for this action. If not specified, the + default template is applied.<br></li><br> + + <li><strong>DynaFileCacheSize </strong>(not mandatory, default 10)<br> + Applies only if dynamic filenames are used.<br> + Specifies the number of DynaFiles that will be kept open. The default is + 10. Note that this is a per-action value, so if you have multiple + dynafile actions, each of them have their individual caches (which means + the numbers sum up). Ideally, the cache size exactly matches the + need. You can use <a href="impstats.html">impstats</a> to tune this + value. Note that a too-low cache size can be a very considerable + performance bottleneck.<br></li><br> <li><strong>ZipLevel </strong>0..9 [default 0]<br> if greater 0, turns on gzip compression of the output file. The higher the number, the better the compression, but also the more CPU is required for zipping.<br></li><br> @@ -39,7 +57,8 @@ </li><br> <li><strong>FlushInterval </strong>(not mandatory, default will be used)<br> - Defines a template to be used for the output. <br></li><br> + Defines, in seconds, the interval after which unwritten data is + flushed.<br></li><br> <li><strong>ASyncWriting </strong>on/off [default off]<br> if turned on, the files will be written in asynchronous mode via a separate thread. In that case, double buffers will be used so that one buffer can be filled while the other buffer is being written. Note that in order to enable FlushInterval, AsyncWriting must be set to "on". Otherwise, the flush interval will be ignored. Also note that when FlushOnTXEnd is "on" but AsyncWriting is off, output will only be written when the buffer is full. This may take several hours, or even require a rsyslog shutdown. However, a buffer flush can be forced in that case by sending rsyslogd a HUP signal. <br></li><br> @@ -53,21 +72,33 @@ <li><strong>DirOwner </strong><br> Set the file owner for directories newly created. Please note that this setting does not affect the owner of directories already existing. The parameter is a user name, for which the userid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + <li><strong>DirOwnerNum </strong> available in 7.5.8+ and 8.1.4+<br> + Set the file owner for directories newly created. Please note that this setting does not affect the owner of directories already existing. The parameter is a numerical ID, which is used regardless of whether the user actually exists. This can be useful if the user mapping is not available to rsyslog during startup.<br></li><br> + <li><strong>DirGroup </strong><br> Set the group for directories newly created. Please note that this setting does not affect the group of directories already existing. The parameter is a group name, for which the groupid is obtained by rsyslogd on during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + <li><strong>DirGroupNum </strong available in 7.5.8+ and 8.1.4+><br> + Set the group for directories newly created. Please note that this setting does not affect the group of directories already existing. The parameter is a numerical ID, which is used regardless of whether the group actually exists. This can be useful if the group mapping is not available to rsyslog during startup.<br></li><br> + <li><strong>FileOwner </strong><br> Set the file owner for files newly created. Please note that this setting does not affect the owner of files already existing. The parameter is a user name, for which the userid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> + <li><strong>FileOwnerNum </strong> available in 7.5.8+ and 8.1.4+<br> + Set the file owner for files newly created. Please note that this setting does not affect the owner of files already existing. The parameter is a numerical ID, which which is used regardless of whether the user actually exists. This can be useful if the user mapping is not available to rsyslog during startup.<br></li><br> + <li><strong>FileGroup </strong><br> Set the group for files newly created. Please note that this setting does not affect the group of files already existing. The parameter is a group name, for which the groupid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.<br></li><br> - <li><strong>DirCreateMode </strong>[defaul 0700]<br> - This is the same as $FileCreateMode, but for directories automatically generated.<br></li><br> + <li><strong>FileGroupNum </strong> available in 7.5.8+ and 8.1.4+<br> + Set the group for files newly created. Please note that this setting does not affect the group of files already existing. The parameter is a numerical ID, which is used regardless of whether the group actually exists. This can be useful if the group mapping is not available to rsyslog during startup.<br></li><br> - <li><strong>FileCreateMode </strong>[default 0644]<br> + <li><strong>FileCreateMode </strong>[default equelly-named module parameter]<br> The FileCreateMode directive allows to specify the creation mode with which rsyslogd creates new files. If not specified, the value 0644 is used (which retains backward-compatibility with earlier releases). The value given must always be a 4-digit octal number, with the initial digit being zero. <br>Please note that the actual permission depend on rsyslogd's process umask. If in doubt, use "$umask 0000" right at the beginning of the configuration file to remove any restrictions. <br>FileCreateMode may be specified multiple times. If so, it specifies the creation mode for all selector lines that follow until the next $FileCreateMode directive. Order of lines is vitally important.<br></li><br> + <li><strong>DirCreateMode </strong>[default equelly-named module parameter]<br> + This is the same as FileCreateMode, but for directories automatically generated.<br></li><br> + <li><strong>FailOnCHOwnFailure </strong>on/off [default on]<br> This option modifies behaviour of file creation. If different owners or groups are specified for new files or directories and rsyslogd fails to set these new owners or groups, it will log an error and NOT write to the file in question if that option is set to "on". If it is set to "off", the error will be ignored and processing continues. Keep in mind, that the files in this case may be (in)accessible by people who should not have permission. The default is "on".<br></li><br> @@ -75,7 +106,8 @@ create directories on an as-needed basis<br></li><br> <li><strong>Sync </strong>on/off [default off]<br> - enables file syncing capability of omfile.<br></li><br> + enables file syncing capability of omfile. Note that this causes + an enormous performance hit if enabled.<br></li><br> <li><strong>File </strong><br> If the file already exists, new data is appended to it. Existing data is not truncated. If the file does not already exist, it is created. Files are kept open as long as rsyslogd is active. This conflicts with external log file rotation. In order to close a file after rotation, send rsyslogd a HUP signal after the file has been rotated away. <br></li><br> @@ -93,9 +125,6 @@ there only is one provider called "<a href="cryprov_gcry.html">gcry</a>".<br></li><br> - <li><strong>Template </strong>[templateName]<br> - sets a new default template for file actions.<br></li><br> - </ul> <p><b>See Also</b> <ul> @@ -124,11 +153,10 @@ unusable. </ul> <p><b>Sample:</b></p> <p>The following command writes all syslog messages into a file.</p> -<textarea rows="5" cols="60">Module (load="builtin:omfile") -*.* action(type="omfile" - DirCreateMode="0700" - FileCreateMode="0644" - File="/var/log/messages") +<textarea rows="5" cols="60">action(type="omfile" + DirCreateMode="0700" + FileCreateMode="0644" + File="/var/log/messages") </textarea> <br><br> diff --git a/doc/omfwd.html b/doc/omfwd.html index a541dd2..d535b29 100644 --- a/doc/omfwd.html +++ b/doc/omfwd.html @@ -13,14 +13,21 @@ <p>The omfwd plug-in provides the core functionality of traditional message forwarding via UDP and plain TCP. It is a built-in module that does not need to be loaded. </p> <p> </p> -<p><b>Global Configuration Directives</b>:</p> +<p><b><i>Note: this documentation describes features present in v7+ of +rsyslog. If you use an older version, scroll down to "legacy parameters".</i></b> +If you prefer, you can also +<a href="http://www.rsyslog.com/how-to-obtain-a-specific-doc-version/">obtain +a specific version of the rsyslog documentation</a>. +<p> </p> + +<p><b>Module Parameters</b>:</p> <ul> <li><strong>Template </strong>[templateName]<br> sets a non-standard default template for this module.<br></li> </ul> <p> </p> -<p><b>Action specific Configuration Directives</b>:</p> +<p><b>Action Parameters</b>:</p> <ul> <li><strong>Target </strong>string<br> Name or IP-Address of the system that shall receive messages. Any resolvable name is fine. <br></li><br> @@ -35,7 +42,67 @@ Framing-Mode to be for forwarding. This affects only TCP-based protocols. It is ignored for UDP. In protocol engineering, ``framing'' means how multiple messages over the same connection are separated. Usually, this is transparent to users. Unfortunately, the early syslog protocol evolved, and so there are cases where users need to specify the framing. The traditional framing is nontransparent. With it, messages are end when a LF (aka ``line break'', ``return'') is encountered, and the next message starts immediately after the LF. If multi-line messages are received, these are essentially broken up into multiple message, usually with all but the first message segment being incorrectly formatted. The octet-counting framing solves this issue. With it, each message is prefixed with the actual message length, so that a receivers knows exactly where the message ends. Multi-line messages cause no problem here. This mode is very close to the method described in RFC5425 for TLS-enabled syslog. Unfortunately, only few syslogd implementations support octet-counted framing. As such, the traditional framing is set as default, even though it has defects. If it is known that the receiver supports octet-counted framing, it is suggested to use that framing mode. <br></li><br> <li><strong>ZipLevel </strong>0..9 [default 0]<br> - Compression level for messages. Rsyslog implements a proprietary capability to zip transmitted messages. Note that compression happens on a message-per-message basis. As such, there is a performance gain only for larger messages. Before compressing a message, rsyslog checks if there is some gain by compression. If so, the message is sent compressed. If not, it is sent uncompressed. As such, it is totally valid that compressed and uncompressed messages are intermixed within a conversation. <br>The compression level is specified via the usual factor of 0 to 9, with 9 being the strongest compression (taking up most processing time) and 0 being no compression at all (taking up no extra processing time). <br></li><br> + Compression level for messages. + <br>Up until rsyslog 7.5.1, this was the only compression setting that + rsyslog understood. Starting with 7.5.1, we have different compression + modes. All of them are affected by the ziplevel. If, however, no mode + is explicitely set, setting ziplevel also turns on "single" + compression mode, so pre 7.5.1 configuration will continue to work + as expected. + <br>The compression level is specified via the usual factor of 0 to 9, with 9 being the strongest compression (taking up most processing time) and 0 being no compression at all (taking up no extra processing time). <br></li><br> + + <li><b>maxErrorMessages </b>integer [default 5], available since 7.5.4<br> + This sets the maximum number of error messages that omfwd + emits during regular operations. The reason for such an upper + limit is that error messages are conveyed back to rsyslog's + input message stream. So if there would be no limit, an endless + loop could be initiated if the failing action would need to + process its own error messages and the emit a new one. This is + also the reason why the default is very conservatively low. + Note that version prior to 7.5.4 did not report any error + messages for the same reason. Also note that with the initial + implementation only errors during UDP forwarding are logged.<br></li><br> + + <li><b>compression.mode</b> <i>mode</i><br> + <i>mode</i> is one of "none", "single", or "stream:always". The + default is "none", in which no compression happens at all. + <br>In "single" compression mode, Rsyslog implements a proprietary + capability to zip transmitted messages. That compression happens + on a message-per-message basis. As such, there is a performance gain + only for larger messages. Before compressing a message, rsyslog checks + if there is some gain by compression. If so, the message is sent + compressed. If not, it is sent uncompressed. As such, it is totally + valid that compressed and uncompressed messages are intermixed + within a conversation. + <br>In "stream:always" compression mode the full stream is being + compressed. This also uses non-standard protocol and is compatible + only with receives that have the same abilities. This mode offers + potentially very high compression ratios. With typical syslog + messages, it can be as high as 95+% compression (so only one twentieth + of data is actually transmitted!). Note that this mode introduces + extra latency, as data is only sent when the compressor emits new + compressed data. For typical syslog messages, this can mean that + some hundered messages may be held in local buffers before they are + actually sent. This mode has been introduced in 7.5.1. + <br><b>Note: currently only imptcp supports receiving stream-compressed + data.</b> + <br></li><br> + + <li><b>compression.stream.flushOnTXEnd</b> <i>[<b>on</b>/off</i>] (requires 7.5.3+)<br> + This setting affects stream compression mode, only. If enabled (the default), the compression + buffer will by emptied at the end of a rsyslog batch. If set to "off", + end of batch will not affect compression at all.<br> + While setting it to "off" can potentially greatly improve compression + ratio, it will also introduce severe delay between when a message is being processed + by rsyslog and actually sent out to the network. We have seen cases where for + several thousand message not a single byte was sent. This is good in the sense that + it can happen only if we have a great compression ratio. This is most probably + a very good mode for busy machines which will process several thousand messages + per second and te resulting short delay will not pose any problems. However, + the default is more conservative, while it works more "naturally" with even low + message traffic. Even in flush mode, notable compression should be achivable + (but we do not yet have practice reports on actual compression ratios). + <br></li><br> <li><strong>RebindInterval </strong>integer<br> Permits to specify an interval at which the current connection is broken and re-established. This setting is primarily an aid to load balancers. After the configured number of messages has been transmitted, the current connection is terminated and a new one started. Note that this setting applies to both TCP and UDP traffic. For UDP, the new ``connection'' uses a different source port (ports are cycled and not reused too frequently). This usually is perceived as a ``new connection'' by load balancers, which in turn forward messages to another physical target system. <br></li><br> @@ -64,12 +131,11 @@ <p><b>Caveats/Known Bugs:</b></p><ul><li>None.</li></ul> <p><b>Sample:</b></p> <p>The following command sends all syslog messages to a remote server via TCP port 10514.</p> -<textarea rows="5" cols="60">Module (load="builtin:omfwd") -*.* action(type="omfwd" -Target="192.168.2.11" -Port="10514" -Protocol="tcp" -) +<textarea rows="5" cols="60">action(type="omfwd" + Target="192.168.2.11" + Port="10514" + Protocol="tcp" + ) </textarea> <br><br> @@ -116,7 +182,7 @@ Protocol="tcp" index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/ommongodb.html b/doc/ommongodb.html new file mode 100644 index 0000000..a611264 --- /dev/null +++ b/doc/ommongodb.html @@ -0,0 +1,60 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>MongoDB Output Module</title> +</head> + +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>MongoDB Output Module</h1> +<p><b>Module Name: ommongodb</b></p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>This module provides native support for logging to MongoDB. +</p> +<p><b>Action Parameters</b>:</p> +<ul> +<li><b>server</b><br>Name or address of the MongoDB server +<li><b>serverport</b><br>Permits to select +a non-standard port for the MongoDB server. The default is 0, which means the +system default port is used. There is no need to specify this parameter unless +you know the server is running on a non-standard listen port. +<li><b>db</b><br>Database to use +<li><b>collection</b><br>Collection to use +<li><b>uid</b><br>logon userid used to connect to server. Must have proper permissions. +<li><b>pwd</b><br>the user's password +<li><b>template</b><br>Template to use when submitting messages. +</ul> +<p>Note rsyslog contains a canned default template to write to the MongoDB. It +will be used automatically if no other template is specified to be used. This template is: +<p> +<textarea rows="5" cols="80">template(name="BSON" type="string" string="\"sys\" : \"%hostname%\", \"time\" : \"%timereported:::rfc3339%\", \"time_rcvd\" : \"%timegenerated:::rfc3339%\", \"msg\" : \"%msg%\", \"syslog_fac\" : \"%syslogfacility%\", \"syslog_sever\" : \"%syslogseverity%\", \"syslog_tag\" : \"%syslogtag%\", \"procid\" : \"%programname%\", \"pid\" : \"%procid%\", \"level\" : \"%syslogpriority-text%\"") +</textarea> +<p>This creates the BSON document needed for MongoDB if no template is specified. The default +schema is aligned to CEE and project lumberjack. As such, the field names are standard +lumberjack field names, and <b>not</b> +<a href="property_replacer.html">rsyslog property names</a>. When specifying templates, be sure +to use rsyslog property names as given in the table. If you would like to use lumberjack-based +field names inside MongoDB (which probably is useful depending on the use case), you need to +select fields names based on the lumberjack schema. +If you just want to use a subset of the fields, but with lumberjack names, you can look up the +mapping in the default template. For example, the lumberjack field "level" contains the rsyslog +property "syslogpriority-text". +<p><b>Sample:</b></p> +<p>The following sample writes all syslog messages to the +database "syslog" and into the collection "log" on mongosever.example.com. The server is +being accessed under the account of "user" with password "pwd". +</p> +<textarea rows="5" cols="80">module(load="ommongodb") +*.* action(type="ommongodb" server="mongoserver.example.com" db="syslog" collection="log" uid="user" pwd="pwd") +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the ASL 2.0.</font></p> +</body></html> diff --git a/doc/omprog.html b/doc/omprog.html index 471ab22..4f36973 100644 --- a/doc/omprog.html +++ b/doc/omprog.html @@ -25,7 +25,34 @@ con re-using existing binaries. For the time being, it simply is not done. In th we may add an option for such pooling, provided that some demand for that is voiced. You can also mimic the same effect by defining multiple rulesets and including them (at the price of some slight performance loss). -<p><b>Configuration Directives</b>:</p> + +<p> </p> + +<p><b>Module Parameters</b>:</p> +<ul> + <li><strong>Template </strong>[templateName]<br> + sets a new default template for file actions.<br></li> + +</ul> +<p> </p> +<p><b>Action Parameters</b>:</p> +<ul> + <li><strong>binary </strong><br> + Mostly equivalent to the "binary" action parameter, but must contain the binary name + only. In legacy config, it is <b>not possible</b> to specify command line parameters. +</ul> +<p><b>Caveats/Known Bugs:</b></p><ul><li>None.</li></ul> +<p><b>Sample:</b></p> +<p>The following command writes all syslog messages into a file.</p> +<textarea rows="5" cols="85">Module (load="omprog") +*.* action(type="omprog" + binary="/pathto/omprog.py --parm1=\"value 1\" --parm2=value2" + template="RSYSLOG_TraditionalFileFormat") +</textarea> + +<br><br> + +<p><b>Legacy Configuration Directives</b>:</p> <ul> <li><b>$ActionOMProgBinary</b> <binary><br> The binary program to be executed. @@ -36,7 +63,7 @@ The binary program to be executed. [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008-2011 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2008-2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/omrelp.html b/doc/omrelp.html index 8858f88..8049eba 100644 --- a/doc/omrelp.html +++ b/doc/omrelp.html @@ -24,16 +24,89 @@ implementation).</p> rsyslog 7.3.10. For older versions, legacy configuration directives must be used. <ul> - <li><b>target </b>(mandatory)<br> + <li><b>target</b> (mandatory)<br> The target server to connect to. </li> - <li><b>template </b>(not mandatory, default "RSYSLOG_ForwardFormat")<br> + <li><b>template</b> (not mandatory, default "RSYSLOG_ForwardFormat")<br> Defines the template to be used for the output. </li> - <li><b>timeout </b>(not mandatory, default 90)<br> + <li><b>timeout</b> (not mandatory, default 90)<br> Timeout for relp sessions. If set too low, valid sessions may be considered dead and tried to recover. </li> + <li><b>windowSize</b> (not mandatory, default 0)<br> + This is an <b>expert parameter</b>. It permits to override the + RELP window size being used by the client. Changing the window + size has both an effect on performance as well as potential + message duplication in failure case. A larger window size means + more performance, but also potentially more duplicated + messages - and vice versa. The default 0 means that librelp's + default window size is being used, which is considered a + compromise between goals reached. For your information: + at the time of this writing, the librelp default window size + is 128 messages, but this may change at any time. + <br>Note that there is no equivalent server parameter, as the + client proposes and manages the window size in RELP protocol. + <li><b>tls</b> (not mandatory, values "on","off", default "off")<br> + If set to "on", the RELP connection will be encrypted by TLS, so that the data is protected against observers. Please note that both the client and the server must have set TLS to either "on" or "off". Other combinations lead to unpredictable results. + </li> + <li><b>tls.compression</b> (not mandatory, values "on","off", default "off")<br> + The controls if the TLS stream should be compressed (zipped). While this + increases CPU use, the network bandwidth should be reduced. Note that + typical text-based log records usually compress rather well. + </li> + <li><b>tls.permittedPeer</b> peer</br> + Places access restrictions on this forwarder. Only peers which + have been listed in this parameter may be connected to. + This guards against rouge servers and man-in-the-middle + attacks. The validation + bases on the certficate the remote peer presents.<br> + The <i>peer</i> parameter lists permitted certificate + fingerprints. Note that it is an array parameter, so either + a single or multiple fingerprints can be listed. When a + non-permitted peer is connected to, the refusal is logged together + with it's fingerprint. So if the administrator knows this was + a valid request, he can simple add the fingerprint by copy and + paste from the logfile to rsyslog.conf. It must be noted, though, + that this situation should usually not happen after initial + client setup and administrators should be alert in this case. + <br>Note that usually a single remote peer should be all that + is ever needed. Support for multiple peers is primarily included + in support of load balancing scenarios. If the connection + goes to a specific server, only one specific certificate is ever + expected (just like when connecting to a specific ssh server). + <br>To specify multiple fingerprints, just enclose them + in braces like this: + <br>tls.permittedPeer=["SHA1:...1", "SHA1:....2"] + <br>To specify just a single peer, you can either + specify the string directly or enclose it in braces. + </li> + <li><b>tls.authMode</b> mode</br> + Sets the mode used for mutual authentication. Supported values are + either "<i>fingerprint</i>" or "<i>name"</i>. + <br>Fingerprint mode basically is what SSH + does. It does not require a full PKI to be present, instead self-signed + certs can be used on all peers. Even if a CA certificate is given, the + validity of the peer cert is NOT verified against it. Only the + certificate fingerprint counts. + <br>In "name" mode, certificate validation happens. Here, the matching + is done against the certificate's subjectAltName and, as a fallback, + the subject common name. If the certificate contains multiple names, + a match on any one of these names is considered good and permits the + peer to talk to rsyslog. + <li><b>tls.prioritystring</b> (not mandatory, string)<br> + This parameter permits to specify the so-called "priority string" to + GnuTLS. This string gives complete control over all crypto parameters, + including compression setting. For this reason, when the prioritystring + is specified, the "tls.compression" parameter has no effect and is + ignored. + <br>Full information about how to construct a priority string can be + found in the GnuTLS manual. At the time of this writing, this + information was contained in + <a href="http://gnutls.org/manual/html_node/Priority-Strings.html">section 6.10 of the GnuTLS manual</a>. + <br><b>Note: this is an expert parameter.</b> Do not use if you do + not exactly know what you are doing. + </li> </ul> <p><b>Sample:</b></p> <p>The following sample sends all messages to the central server diff --git a/doc/omruleset.html b/doc/omruleset.html index f0d5f7b..3911929 100644 --- a/doc/omruleset.html +++ b/doc/omruleset.html @@ -10,6 +10,20 @@ <p><b>Module Name: omruleset</b></p> <p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> <p><b>Available Since</b>: 5.3.4</p> +<p><b><font color="red">Deprecated in</font></b>: 7.2.0+</p> +<p><b>Deprecation note</b></p> +<p><font color="red">This module exists only for backwards-compatibility +reasons.</font> +<b>Do no longer use it in new configurations.</b> It has been +replaced by the much more efficient +<a href="rainerscript_call.html">"call" RainerScript statement</a>. The +"call" statement supports everything omruleset does, but in an easier +to use way. +<br> +<br> +<br> +<br> +<br> <p><b>Description</b>:</p> <p>This is a very special "output" module. It permits to pass a message object to another rule set. While this is a very simple action, it enables very diff --git a/doc/queue_parameters.html b/doc/queue_parameters.html new file mode 100644 index 0000000..b89187b --- /dev/null +++ b/doc/queue_parameters.html @@ -0,0 +1,116 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>rsyslog queues parameters</title></head> +<body> +<a href="rsyslog_conf_global.html">back</a> +<p><h2>General Queue Parameters</h2></p> +<p>Queue parameters can be used together with the following statements: +<ul> +<li><a href="rsyslog_conf_actions.html">action()</a> +<li>ruleset()</a> +<li>main_queue()</a> +</ul> +<p> +Queues need to be configured in the action or ruleset it should affect. If nothing is configured, +default values will be used. Thus, the default ruleset has only the default main queue. Specific Action +queues are not set up by default.</p> +<ul> + <li><strong>queue.filename</strong> name<br> + File name to be used for the queue files. Please note that + this is actually just the file name. A directory can NOT be + specified in this paramter. If the files shall be + created in a specific directory, specify + queue.spoolDirectory for this. The filename is used to build + to complete path for queue files.</li> + <li><strong>queue.spoolDirectory</strong> name<br> + This is the directory into which queue files will be stored. + Note that the directory must exist, it is NOT automatically + created by rsyslog. If no spoolDirectory is specified, the + work directory is used. + </li> + <li><strong>queue.size</strong> number <br> + This is the maximum size of the queue in number of messages. + Note that setting the queue size to very small values (roughly + below 100 messages) is not supported and can lead to + unpredictable results.<br> + For more information on the current status of this restriction + see the <a href="http://www.rsyslog.com/lower-bound-for-queue-sizes/">rsyslog + FAQ: "lower bound for queue sizes"</a>.</li> + <li><strong>queue.dequeuebatchsize</strong> number + <br>default 16</li> + <li><strong>queue.maxdiskspace</strong> number + <br>The maximum size that all queue files together will use on disk. + Note that the actual size may be slightly larger than the configured max, as + rsyslog never writes partial queue records. + <li><strong>queue.highwatermark</strong> number + <br>This applies to disk-assisted queues, only. + When the queue fills up to this number of messages, the queue begins + to spool messages to disk. Please note that this should note happen as + part of usual processing, because disk queue mode is very considerably + slower than in-memory queue mode. Going to disk should be reserved for + cases where an output action destination is offline for some period. + <li><strong>queue.lowwatermark</strong> number + <br>default 2000</li> + <li><strong>queue.fulldelaymark</strong> number</br> + Number of messages when the queue should block delayable messages. Messages + are NO LONGER PROCESSED until the queue has sufficient space again. If + a message is delayable depends on the input. For example, messages + received via imtcp are delayable (because TCP can push back), but those + received via imudp are not (as UDP does not permit a push back). The + intent behind this setting is to leave some space in an almost-full + queue for non-delayable messages, which would be lost if the queue runs + out of space.</br></br> + Please note that if you use a DA queue, setting the fulldelaymark ABOVE the + highwatermark makes the queue never activate disk mode for delayable + inputs. So this is probably not what you want. + </li> + <li><strong>queue.lightdelaymark</strong> number</li> + <li><strong>queue.discardmark</strong> number + <br>default 9750]</li> + <li><strong>queue.discardseverity</strong> number + <br>*numerical* severity! default 8 (nothing discarded)</li> + <li><strong>queue.checkpointinterval</strong> number</li> + <li><strong>queue.syncqueuefiles</strong> on/off</li> + <li><strong>queue.type</strong> [FixedArray/LinkedList/<b>Direct</b>/Disk]</li> + <li><strong>queue.workerthreads</strong> number + <br>number of worker threads, default 1, recommended 1</li> + <li><strong>queue.timeoutshutdown</strong> number + <br>number is timeout in ms (1000ms is 1sec!), default 0 (indefinite)</li> + <li><strong>queue.timeoutactioncompletion</strong> number + <br>number is timeout in ms (1000ms is 1sec!), default 1000, 0 means immediate!</li> + <li><strong>queue.timeoutenqueue</strong> number + <br>number is timeout in ms (1000ms is 1sec!), default 2000, 0 means indefinite</li> + <li><strong>queue.timeoutworkerthreadshutdown</strong> number + <br>number is timeout in ms (1000ms is 1sec!), default 60000 (1 minute)</li> + <li><strong>queue.workerthreadminimummessages</strong> number + <br>default 100</li> + <li><strong>queue.maxfilesize</strong> size_nbr + <br> default 1m</li> + <li><strong>queue.saveonshutdown</strong> on/<b>off</b></li> + <li><strong>queue.dequeueslowdown</strong> number + <br>number is timeout in microseconds (1000000us is 1sec!), default 0 (no delay). Simple rate-limiting!</li> + <li><strong>queue.dequeuetimebegin</strong> number</li> + <li><strong>queue.dequeuetimeend</strong> number</li> +</ul> +<p><b>Sample:</b></p> +<p>The following is a sample of a TCP forwarding action with its own queue.</p> +<textarea rows="7" cols="60">action(type="omfwd" + target="192.168.2.11" + port="10514" + protocol="tcp" + queue.filename="forwarding" + queue.size="1000000" + queue.type="LinkedList" +) +</textarea> +<br><br> +[<a href="manual.html">manual index</a>] +[<a href="rsyslog_conf.html">rsyslog.conf</a>] +[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/rainerscript.html b/doc/rainerscript.html index b83184d..b42f79d 100644 --- a/doc/rainerscript.html +++ b/doc/rainerscript.html @@ -44,6 +44,45 @@ C programming language or PHP. <p>If in doubt how to properly escape, use the <a href="http://www.rsyslog.com/rainerscript-constant-string-escaper/">RainerScript String Escape Online Tool</a>. +<h2>Variable (Property) types</h2> +<p>All rsyslog properties (see the <a href="property_replacer.html">property +replacer</a> page for a list) can be used in RainerScript. In addition, it also +supports local variables. Local variables are local to the current message, but are +NOT message properties (e.g. the "$!" all JSON property does not contain +them). +<p>Only message json (CEE/Lumberjack) properties can be modified by +the "set" and "unset" statements, not any other message property. Obviously, +local variables are also modifieable. +<p>Message JSON property names start with "$!" where the bang character +represents the root. +<p>Local variables names start with "$.", where the dot denotes the root. +<p>Both JSON properties as well as local variables may contain an arbitrary +deep path before the final element. The bang character is always used as path +separator, no matter if it is a message property or a local variable. For example +"$!path1!path2!varname" is a three-level deep message property where as +the very similar looking "$.path1!path2!varname" specifies a three-level +deep local variable. +The bang or dot character immediately following the +dollar sign is used by rsyslog to separate the different types. +<h2>configuration objects</h2> +<h3>main_queue()</h3> +<p><i>This object is available since 7.5.3.</i> +This permits to specify parameters for the main message queue. Note that +only <a href="queue_parameters.html">queue-parameters</a> are permitted for this +config object. This permits to set the same options like in ruleset and action +queues. A special statement is needed for the main queue, because it is a +different object and cannot be configured via any other object. +<p>Note that when the main_queue() object is configured, the legacy +$MainMsgQ... statements are ignored. +<p>Example:</p> +<textarea rows="2" cols="60">main_queue(queue.size="100000" queue.type="LinkedList") +</textarea> +<h3>action()</h3> +The <a href="rsyslog_conf_actions.html">action</a> object is the primary +means of describing actions to be carried out. +<h3>global()</h3> +<p>This is used to set global configuration parameters. For details, please +see the <a href="global.html">rsyslog global configuration object</a>. <h2>Expressions</h2> The language supports arbitrary complex expressions. All usual operators are supported. The precedence of operations is as follows @@ -61,13 +100,6 @@ of a and b should be tested as "a <> b". The "not" operator should be reserved to cases where it actually is needed to form a complex boolean expression. In those cases, parenthesis are highly recommended. -<h2>configuration objects</h2> -<h3>action()</h3> -The <a href="rsyslog_conf_actions.html">action</a> object is the primary -means of describing actions to be carried out. -<h3>global()</h3> -<p>This is used to set global configuration parameters. For details, please -see the <a href="global.html">rsyslog global configuration object</a>. <h2>Lookup Tables</h2> <p><a href="lookup_tables.html">Lookup tables</a> are a powerful construct to obtain "class" information based on message content (e.g. to build @@ -110,6 +142,11 @@ equivalent single-character [field($msg, 44 ,3)] extraction. <li>prifilt(constant) - mimics a traditional PRI-based filter (like "*.*" or "mail.info"). The traditional filter string must be given as a <b>constant string</b>. Dynamic string evaluation is not permitted (for performance reasons). +<li>exec_template(templateName) -- executes a template and returns the string generated +by it. Note that templateName must be a constant string. This is primarily for performance +reasons. If there is a valid use case for dynamic template names, please let us know +and we will re-consider the decision. TemplateName can be any template name, including the +predefined ones. </ul> <p>The following example can be used to build a dynamic filter based on some environment variable: diff --git a/doc/rainerscript_call.html b/doc/rainerscript_call.html new file mode 100644 index 0000000..faab5c2 --- /dev/null +++ b/doc/rainerscript_call.html @@ -0,0 +1,51 @@ +<html> +<head> +<title>rsyslog "call" statement</title> +</head> +<body> +<a href="rsyslog_conf_global.html">back</a> + +<h1>The rsyslog "call" statement</h1> +<p>The rsyslog "call" statement is used to tie rulesets together. +It is modelled after the usual programming langauge "call" statement. Think +of a ruleset as a subroutine (what it really is!) and you get the picture. + +<p>The "call" statement can be used to call into any type of rulesets. +If a rule set has a queue assigned, the message will be posted to that queue +and processed asynchronously. Otherwise, the ruleset will be executed +synchronously and control returns to right after the call when the rule set +has finished execution. + +<p>Note that there is an important difference between asynchronous and +synchronous execution in regard to the "stop" statement. It will not affect +processing of the original message when run asynchronously. + +<p>The "call" statement replaces the deprecated omruleset module. It offers +all capabilities omruleset has, but works in a much more efficient way. +Note that omruleset was a hack that made calling rulesets possible +within the constraints of the pre-v7 engine. "call" is the clean solution for +the new engine. Especially for rulesets without associated queues (synchronous +operation), it has zero overhead (really!). omruleset always needs to +duplicate messages, which usually means at least ~250 bytes of memory writes, +some allocs and frees - and even more performance-intense operations. + +<h2>syntax</h2> +<p><code>call rulesetname</code></p> +<p>Where "rulesetname" is the name of a ruleset that is defined elsewhere +inside the configration. If the call is synchronous or asynchronous depends +on the ruleset parameters. This cannot be overriden by the "call" statement. + +<h2>related links</h2> +<ul> +<li><a href="http://blog.gerhards.net/2012/10/how-to-use-rsyslogs-ruleset-and-call.html">Blog posting announcing "call" statement (with sample)</a> +</ul> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2013 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 2 or higher.</font></p> +</body> +</html> diff --git a/doc/rsyslog_conf_actions.html b/doc/rsyslog_conf_actions.html index 50b13a0..f6f9cda 100644 --- a/doc/rsyslog_conf_actions.html +++ b/doc/rsyslog_conf_actions.html @@ -3,9 +3,14 @@ <body> <p>This is a part of the rsyslog.conf documentation.</p> <a href="rsyslog_conf.html">back</a> +<p><b><i>Note: this documentation describes features present in v7+ of +rsyslog. If you use an older version, scroll down to "legacy parameters".</i></b> +If you prefer, you can also +<a href="http://www.rsyslog.com/how-to-obtain-a-specific-doc-version/">obtain +a specific version of the rsyslog documentation</a>. <h2>Actions</h2> Action object describe what is to be done with a message. They are -implemented via <a href="rsyslog_conf_modules.html#om">outpout modules</a>. +implemented via <a href="rsyslog_conf_modules.html#om">output modules</a>. <p>The action object has different parameters: <ul> <li>those that apply to all actions and are action specific. These @@ -13,8 +18,11 @@ implemented via <a href="rsyslog_conf_modules.html#om">outpout modules</a>. <li>parameters for the action queue. While they also apply to all parameters, they are queue-specific, not action-specific (they are the same that are used in rulesets, for example). + The are documented separately under + <a href="queue_parameters.html">queue parameters</a>. <li>action-specific parameters. These are specific to a certain - type of actions. They are documented by the output module + type of actions. They are documented by the + <a href="rsyslog_conf_modules.html#om">output module</a> in question. </ul> <h3>General Action Parameters</h3> @@ -53,10 +61,35 @@ The question is if this is desired behavior? Or should the rule only be triggere <li><b>action.resumeInterval</b> integer <br>Sets the ActionResumeInterval for the action. The interval provided is always in seconds. Thus, multiply by 60 if you need minutes and 3,600 if you need hours (not recommended). When an action is suspended (e.g. destination can not be connected), the action is resumed for the configured interval. Thereafter, it is retried. If multiple retires fail, the interval is automatically extended. This is to prevent excessive ressource use for retires. After each 10 retries, the interval is extended by itself. To be precise, the actual interval is (numRetries / 10 + 1) * Action.ResumeInterval. so after the 10th try, it by default is 60 and after the 100th try it is 330.</li> + <li><b>action.reportSuspension</b> on/off + <br>Configures rsyslog to report suspension and reactivation + of the action. This is useful to note which actions have + problems (e.g. connecting to a remote system) and when. + The default for this setting is the equally-named global + parameter.</li> + <li><b>action.reportSuspensionContinuation</b> on/off + <br>Configures rsyslog to report continuation of action suspension. + This emits new messages whenever an action is to be retried, but + continues to fail. If set to "on", action.reportSuspension is + also automatically set to "on". + The default for this setting is the equally-named global + parameter.</li> </ul> -<h2>Legacy Format</h2> +<h2>Useful Links</h2> +<ul> +<li>Rainer's blog posting on the performance of + <a href="http://blog.gerhards.net/2013/06/rsyslog-performance-main-and-action.html">main + and action queue worker threads</a> +</ul> + +<br> +<br> +<br> + + +<h1>Legacy Format</h1> <p><b>Be warned that legacy action format is hard to get right. It is recommended to use RainerScript-Style action format whenever possible!</b> A key problem with legacy format is that a single action is defined via diff --git a/doc/rsyslog_conf_basic_structure.html b/doc/rsyslog_conf_basic_structure.html index f5d4891..54fb721 100644 --- a/doc/rsyslog_conf_basic_structure.html +++ b/doc/rsyslog_conf_basic_structure.html @@ -43,7 +43,7 @@ after the stop statement are never evaluated. <ul> <li><b>if expr then ... else ...</b> - conditional execution <li><b>stop</b> - stops processing the current message -<li><b>call</b> - calls a ruleset (just like a subroutine call) +<li><b><a href="rainerscript_call.html">call</a></b> - calls a ruleset (just like a subroutine call) <li><b>continue</b> - a NOP, useful e.g. inside the then part of an if </ul> diff --git a/doc/rsyslog_conf_global.html b/doc/rsyslog_conf_global.html index e48ed6d..b9a2ba2 100644 --- a/doc/rsyslog_conf_global.html +++ b/doc/rsyslog_conf_global.html @@ -136,7 +136,9 @@ our paper on <a href="multi_ruleset.html">using multiple rule sets in rsyslog</a <li><a href="omfile.html"><b>$CreateDirs</b></a> [<b>on</b>/off] - create directories on an as-needed basis</li> <li><a href="omfile.html"><b>$DirCreateMode</b></a></li> <li><a href="omfile.html"><b>$DirGroup</b></a></li> +<li><a href="omfile.html"><b>$DirGroupNum</b></a></li> <li><a href="omfile.html"><b>$DirOwner</b></a></li> +<li><a href="omfile.html"><b>$DirOwnerNum</b></a></li> <li><a href="rsconf1_dropmsgswithmaliciousdnsptrrecords.html"><b>$DropMsgsWithMaliciousDnsPTRRecords</b></a></li> <li><a href="rsconf1_droptrailinglfonreception.html"><b>$DropTrailingLFOnReception</b></a></li> <li><a href="omfile.html"><b>$DynaFileCacheSize</b></a></li> @@ -148,7 +150,9 @@ our paper on <a href="multi_ruleset.html">using multiple rule sets in rsyslog</a <li><a href="omfile.html"><b>$FailOnChownFailure</b></a></li> <li><a href="omfile.html"><b>$FileCreateMode</b></a></li> <li><a href="omfile.html"><b>$FileGroup</b></a></li> +<li><a href="omfile.html"><b>$FileGroupNum</b></a></li> <li><a href="omfile.html"><b>$FileOwner</b></a></li> +<li><a href="omfile.html"><b>$FileOwnerNum</b></a></li> <li><a href="rsconf1_generateconfiggraph.html"><b>$GenerateConfigGraph</b></a></li> <li><a href="rsconf1_gssforwardservicename.html"><b>$GssForwardServiceName</b></a></li> <li><a href="rsconf1_gsslistenservicename.html"><b>$GssListenServiceName</b></a></li> diff --git a/doc/rsyslog_conf_modules.html b/doc/rsyslog_conf_modules.html index 18d6b8a..8dc3ed5 100644 --- a/doc/rsyslog_conf_modules.html +++ b/doc/rsyslog_conf_modules.html @@ -111,13 +111,22 @@ probably an excellent starting base for writing a new module. Currently, the fol modules exist inside the source tree: <ul> <li><a href="mmanon.html">mmanon</a> - used to anonymize log messages. +<li><a href="mmcount.html">mmcount</a> - message modification plugin which counts messages +<li><a href="mmfields.html">mmfields</a> - used to extract fields from +specially formatted messages (e.g. CEF) <li><a href="mmnormalize.html">mmnormalize</a> - used to normalize log messages. Note that this actually is a <b>generic</b> module. <li><a href="mmjsonparse.html">mmjsonparse</a> - used to interpret CEE/lumberjack enabled structured log messages. +<li><a href="mmpstrucdata.html">mmpstrucdata</a> - used to parse RFC5424 structured data +into json message properties <li><a href="mmsnmptrapd.html">mmsnmptrapd</a> - uses information provided by snmptrapd inside the tag to correct the original sender system and priority of messages. Implemented via the output module interface. +<li><a href="mmutf8fix.html">mmutf8fix</a> - used to fix invalid UTF-8 character sequences +<li><a href="mmrfc5424addhmac.html">mmrfc5424addhmac</a> - custom module for adding HMACs to +rfc5424-formatted messages if not already present +<li><a href="mmsequence.html">mmsequence</a> - sequence generator and counter plugin </ul> <a name="lm"></a><h2>String Generator Modules</h2> diff --git a/doc/rsyslog_secure_tls.html b/doc/rsyslog_secure_tls.html new file mode 100644 index 0000000..0bef2b3 --- /dev/null +++ b/doc/rsyslog_secure_tls.html @@ -0,0 +1,114 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS-protected syslog: recommended scenario</title> +</head> +<body> + +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-06-17)</i></small></p> +<ul> +<li><a href="rsyslog_secure_tls.html">Overview</a> +<li><a href="tls_cert_scenario.html">Sample Scenario</a> +<li><a href="tls_cert_ca.html">Setting up the CA</a> +<li><a href="tls_cert_machine.html">Generating Machine Certificates</a> +<li><a href="tls_cert_server.html">Setting up the Central Server</a> +<li><a href="tls_cert_client.html">Setting up syslog Clients</a> +<li><a href="tls_cert_udp_relay.html">Setting up the UDP syslog relay</a> +<li><a href="tls_cert_summary.html">Wrapping it all up</a> +<li><a href="tls_cert_errmsgs.html">Frequently seen Error Messages</a> +</ul> + +<h2>Overview</h2> +<p>This document describes a secure way to set up rsyslog TLS. A secure logging +environment requires more than just encrypting the transmission channel. This document +provides one possible way to create such a secure system. +<p>Rsyslog's TLS authentication can be used very flexible and thus supports a +wide range of security policies. This section tries to give some advise on a +scenario that works well for many environments. However, it may not be suitable +for you - please assess you security needs before using the recommendations +below. Do not blame us if it doesn't provide what you need ;)</p> +<p>Our policy offers these security benefits:</p> +<ul> + <li>syslog messages are encrypted while traveling on the wire</li> + <li>the syslog sender authenticates to the syslog receiver; thus, the + receiver knows who is talking to it</li> + <li>the syslog receiver authenticates to the syslog sender; thus, the sender + can check if it indeed is sending to the expected receiver</li> + <li>the mutual authentication prevents man-in-the-middle attacks</li> +</ul> +<p>Our secrity goals are achived via public/private key security. As such, it is +vital that private keys are well protected and not accessible to third parties. +If private keys have become known to third parties, the system does not provide +any security at all. Also, our solution bases on X.509 certificates and a (very +limited) chain of trust. We have one instance (the CA) that issues all machine +certificates. The machine certificate indentifies a particular machine. hile in +theory (and practice), there could be several "sub-CA" that issues machine +certificates for a specific adminitrative domain, we do not include this in our +"simple yet secure" setup. If you intend to use this, rsyslog supports it, but +then you need to dig a bit more into the documentation (or use the forum to ask). +In general, if you depart from our simple model, you should have good reasons +for doing so and know quite well what you are doing - otherwise you may +compromise your system security.</p> +<p>Please note that security never comes without effort. In the scenario +described here, we have limited the effort as much as possible. What remains is +some setup work for the central CA, the certificate setup for each machine as +well as a few configuration commands that need to be applied to all of them. +Proably the most important limiting factor in our setup is that all senders and +receivers must support IETF's syslog-transport-tls standard (which is not +finalized yet). We use mandatory-to-implement technology, yet you may have +trouble finding all required features in some implementations. More often, +unfortunately, you will find that an implementation does not support the +upcoming IETF standard at all - especially in the "early days" (starting May +2008) when rsyslog is the only implementation of said standard.</p> +<p>Fortunately, rsyslog supports allmost every protocol that is out there in the +syslog world. So in cases where transport-tls is not available on a sender, we +recommend to use rsyslog as the initial relay. In that mode, the not-capabe +sender sends to rsyslog via another protocol, which then relays the message via +transport-tls to either another interim relay or the final destination (which, +of course, must by transport-tls capable). In such a scenario, it is best to try +see what the sender support. Maybe it is possible to use industry-standard plain +tcp syslog with it. Often you can even combine it with stunnel, which then, too, +enables a secure delivery to the first rsyslog relay. If all of that is not +possible, you can (and often must...) resort to UDP. Even though this is now +lossy and insecure, this is better than not having the ability to listen to that +device at all. It may even be reasonale secure if the uncapable sender and the +first rsyslog relay communicate via a private channel, e.g. a dedicated network +link.</p> +<p>One final word of caution: transport-tls protects the connection between the +sender and the receiver. It does not necessarily protect against attacks that +are present in the message itself. Especially in a relay environment, the +message may have been originated from a malicious system, which placed invalid +hostnames and/or other content into it. If there is no provisioning against such +things, these records may show up in the receivers' repository. -transport-tls +does not protect against this (but it may help, properly used). Keep in mind +that syslog-transport-tls provides hop-by-hop security. It does not provide +end-to-end security and it does not authenticate the message itself (just the +last sender).</p> +<h3>A very quick Intro</h3> +<p>If you'd like to get all information very rapidly, the graphic below contains +everything you need to know (from the certificate perspective) in a very condensed +manner. It is no surprise if the graphic puzzles you. In this case, <a href="tls_cert_scenario.html">simply read on</a> +for full instructions. +<p> +<img align="center" alt="TLS/SSL protected syslog" src="tls_cert.jpg"> +<h3>Feedback requested</h3> +<p>I would appreciate feedback on this tutorial. If you have +additional ideas, comments or find bugs (I *do* bugs - no way... ;)), +please +<a href="mailto:rgerhards@adiscon.com">let me know</a>.</p> +<h2>Revision History</h2> +<ul> +<li>2008-06-06 * <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> * Initial Version created</li> +<li>2008-06-18 * <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> * Greatly enhanced and modularized the doc</li> +</ul> +<h2>Copyright</h2> +<p>Copyright (c) 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> diff --git a/doc/tls_cert_ca.html b/doc/tls_cert_ca.html new file mode 100644 index 0000000..f366462 --- /dev/null +++ b/doc/tls_cert_ca.html @@ -0,0 +1,155 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS-protected syslog: scenario</title> +</head> +<body> + +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-06-17)</i></small></p> + +<ul> +<li><a href="rsyslog_secure_tls.html">Overview</a> +<li><a href="tls_cert_scenario.html">Sample Scenario</a> +<li><a href="tls_cert_ca.html">Setting up the CA</a> +<li><a href="tls_cert_machine.html">Generating Machine Certificates</a> +<li><a href="tls_cert_server.html">Setting up the Central Server</a> +<li><a href="tls_cert_client.html">Setting up syslog Clients</a> +<li><a href="tls_cert_udp_relay.html">Setting up the UDP syslog relay</a> +<li><a href="tls_cert_summary.html">Wrapping it all up</a> +</ul> + +<h3>Setting up the CA</h3> +<p>The first step is to set up a certificate authority (CA). It must be +maintained by a trustworthy person (or group) and approves the indentities of +all machines. It does so by issuing their certificates. In a small setup, the +administrator can provide the CA function. What is important is the the CA's +private key is well-protocted and machine certificates are only issued if it is +know they are valid (in a single-admin case that means the admin should not +issue certificates to anyone else except himself).</p> +<p>The CA creates a so-called self-signed certificate. That is, it approves its +own authenticy. This sounds useless, but the key point to understand is that +every machine will be provided a copy of the CA's certificate. Accepting this +certificate is a matter of trust. So by configuring the CA certificate, the +administrator tells <a href="http://www.rsyslog.com">rsyslog</a> which certificates to trust. This is the root of all +trust under this model. That is why the CA's private key is so important - +everyone getting hold of it is trusted by our rsyslog instances.</p> +<center><img src="tls_cert_ca.jpg"></center> +<p>To create a self-signed certificate, use the following commands with GnuTLS (which +is currently the only supported TLS library, what may change in the future). +Please note that GnuTLS' tools are not installed by default on many platforms. Also, +the tools do not necessarily come with the GnuTLS core package. If you do not +have certtool on your system, check if there is package for the GnuTLS tools available +(under Fedora, for example, this is named gnutls-utils-<version> and +it is NOT installed by default). </p> +<ol> +<li>generate the private key: +<pre>certtool --generate-privkey --outfile ca-key.pem</pre> +<br> +This takes a short while. Be sure to do some work on your workstation, +it waits for radom input. Switching between windows is sufficient ;) +</li> +<li>now create the (self-signed) CA certificate itself:<br> +<pre>certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem</pre> +This generates the CA certificate. This command queries you for a +number of things. Use appropriate responses. When it comes to +certificate validity, keep in mind that you need to recreate all +certificates when this one expires. So it may be a good idea to use a +long period, eg. 3650 days (roughly 10 years). You need to specify that +the certificates belongs to an authority. The certificate is used to +sign other certificates.<br> +</li> +</ol> +<h3>Sample Screen Session</h3> +<p>Text in red is user input. Please note that for some questions, there is no +user input given. This means the default was accepted by simply pressing the +enter key. +<code><pre> +[root@rgf9dev sample]# <font color="red">certtool --generate-privkey --outfile ca-key.pem --bits 2048</font> +Generating a 2048 bit RSA private key... +[root@rgf9dev sample]# <font color="red">certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem</font> +Generating a self signed certificate... +Please enter the details of the certificate's distinguished name. Just press enter to ignore a field. +Country name (2 chars): <font color="red">US</font> +Organization name: <font color="red">SomeOrg</font> +Organizational unit name: <font color="red">SomeOU</font> +Locality name: <font color="red">Somewhere</font> +State or province name: <font color="red">CA</font> +Common name: <font color="red">someName (not necessarily DNS!)</font> +UID: +This field should not be used in new certificates. +E-mail: +Enter the certificate's serial number (decimal): + + +Activation/Expiration time. +The certificate will expire in (days): <font color="red">3650</font> + + +Extensions. +Does the certificate belong to an authority? (Y/N): <font color="red">y</font> +Path length constraint (decimal, -1 for no constraint): +Is this a TLS web client certificate? (Y/N): +Is this also a TLS web server certificate? (Y/N): +Enter the e-mail of the subject of the certificate: <font color="red">someone@example.net</font> +Will the certificate be used to sign other certificates? (Y/N): <font color="red">y</font> +Will the certificate be used to sign CRLs? (Y/N): +Will the certificate be used to sign code? (Y/N): +Will the certificate be used to sign OCSP requests? (Y/N): +Will the certificate be used for time stamping? (Y/N): +Enter the URI of the CRL distribution point: +X.509 Certificate Information: + Version: 3 + Serial Number (hex): 485a365e + Validity: + Not Before: Thu Jun 19 10:35:12 UTC 2008 + Not After: Sun Jun 17 10:35:25 UTC 2018 + Subject: C=US,O=SomeOrg,OU=SomeOU,L=Somewhere,ST=CA,CN=someName (not necessarily DNS!) + Subject Public Key Algorithm: RSA + Modulus (bits 2048): + d9:9c:82:46:24:7f:34:8f:60:cf:05:77:71:82:61:66 + 05:13:28:06:7a:70:41:bf:32:85:12:5c:25:a7:1a:5a + 28:11:02:1a:78:c1:da:34:ee:b4:7e:12:9b:81:24:70 + ff:e4:89:88:ca:05:30:0a:3f:d7:58:0b:38:24:a9:b7 + 2e:a2:b6:8a:1d:60:53:2f:ec:e9:38:36:3b:9b:77:93 + 5d:64:76:31:07:30:a5:31:0c:e2:ec:e3:8d:5d:13:01 + 11:3d:0b:5e:3c:4a:32:d8:f3:b3:56:22:32:cb:de:7d + 64:9a:2b:91:d9:f0:0b:82:c1:29:d4:15:2c:41:0b:97 + Exponent: + 01:00:01 + Extensions: + Basic Constraints (critical): + Certificate Authority (CA): TRUE + Subject Alternative Name (not critical): + RFC822name: someone@example.net + Key Usage (critical): + Certificate signing. + Subject Key Identifier (not critical): + fbfe968d10a73ae5b70d7b434886c8f872997b89 +Other Information: + Public Key Id: + fbfe968d10a73ae5b70d7b434886c8f872997b89 + +Is the above information ok? (Y/N): <font color="red">y</font> + + +Signing certificate... +[root@rgf9dev sample]# <font color="red">chmod 400 ca-key.pem</font> +[root@rgf9dev sample]# <font color="red">ls -l</font> +total 8 +-r-------- 1 root root 887 2008-06-19 12:33 ca-key.pem +-rw-r--r-- 1 root root 1029 2008-06-19 12:36 ca.pem +[root@rgf9dev sample]# +</pre></code> +<p><font color="red"><b>Be sure to safeguard ca-key.pem!</b> Nobody except the CA itself +needs to have it. If some third party obtains it, you security is broken!</font> +<h2>Copyright</h2> +<p>Copyright (c) 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> diff --git a/doc/tls_cert_client.html b/doc/tls_cert_client.html new file mode 100644 index 0000000..53596fc --- /dev/null +++ b/doc/tls_cert_client.html @@ -0,0 +1,78 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS-protected syslog: client setup</title> +</head> +<body> + +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-07-03)</i></small></p> + +<ul> +<li><a href="rsyslog_secure_tls.html">Overview</a> +<li><a href="tls_cert_scenario.html">Sample Scenario</a> +<li><a href="tls_cert_ca.html">Setting up the CA</a> +<li><a href="tls_cert_machine.html">Generating Machine Certificates</a> +<li><a href="tls_cert_server.html">Setting up the Central Server</a> +<li><a href="tls_cert_client.html">Setting up syslog Clients</a> +<li><a href="tls_cert_udp_relay.html">Setting up the UDP syslog relay</a> +<li><a href="tls_cert_summary.html">Wrapping it all up</a> +</ul> + +<h3>Setting up a client</h3> +<p>In this step, we configure a client machine. We from our scenario, we use +zuse.example.net. You need to do the same steps for all other clients, too (in the +example, that meanst turng.example.net). The client check's the server's identity and +talks to it only if it is the expected server. This is a very important step. +Without it, you would not detect man-in-the-middle attacks or simple malicious servers +who try to get hold of your valuable log data. +<p><center><img src="tls_cert_100.jpg"></center> +<p>Steps to do: +<ul> +<li>make sure you have a functional CA (<a href="tls_cert_ca.html">Setting up the CA</a>) +<li>generate a machine certificate for zuse.example.net (follow instructions in + <a href="tls_cert_machine.html">Generating Machine Certificates</a>) +<li>make sure you copy over ca.pem, machine-key.pem ad machine-cert.pem to the client. +Ensure that no user except root can access them (<b>even read permissions are really bad</b>). +<li>configure the client so that it checks the server identity and sends messages only +if the server identity is known. Please note that you have the same options as when +configuring a server. However, we now use a single name only, because there is only one +central server. No using wildcards make sure that we will exclusively talk to that server +(otherwise, a compromised client may take over its role). If you load-balance to different +server identies, you obviously need to allow all of them. It still is suggested to use +explcit names. +</ul> +<p><b>At this point, please be reminded once again that your security needs may be quite different from +what we assume in this tutorial. Evaluate your options based on your security needs.</b> +<h3>Sample syslog.conf</h3> +<p>Keep in mind that this rsyslog.conf sends messages via TCP, only. Also, we do not +show any rules to write local files. Feel free to add them. +<code><pre> +# make gtls driver the default +$DefaultNetstreamDriver gtls + +# certificate files +$DefaultNetstreamDriverCAFile /rsyslog/protected/ca.pem +$DefaultNetstreamDriverCertFile /rsyslog/protected/machine-cert.pem +$DefaultNetstreamDriverKeyFile /rsyslog/protected/machine-key.pem + +$ActionSendStreamDriverAuthMode x509/name +$ActionSendStreamDriverPermittedPeer central.example.net +$ActionSendStreamDriverMode 1 # run driver in TLS-only mode +*.* @@central.example.net:10514 # forward everything to remote server +</pre></code> +<p>Note: the example above forwards every message to the remote server. Of course, +you can use the normal filters to restrict the set of information that is sent. +Depending on your message volume and needs, this may be a smart thing to do. +<p><font color="red"><b>Be sure to safeguard at least the private key (machine-key.pem)!</b> +If some third party obtains it, you security is broken!</font> +<h2>Copyright</h2> +<p>Copyright © 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> diff --git a/doc/tls_cert_errmsgs.html b/doc/tls_cert_errmsgs.html index d002174..2943efc 100644 --- a/doc/tls_cert_errmsgs.html +++ b/doc/tls_cert_errmsgs.html @@ -21,19 +21,6 @@ Gerhards</a> (2008-06-17)</i></small></p> <h3>Error Messages</h3> <p>This page covers error message you may see when setting up -<span style="float: left"> -<script type="text/javascript"><!-- -google_ad_client = "pub-3204610807458280"; -/* rsyslog doc inline */ -google_ad_slot = "5958614527"; -google_ad_width = 125; -google_ad_height = 125; -//--> -</script> -<script type="text/javascript" -src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> -</script> -</span> <a href="http://www.rsyslog.com">rsyslog</a> with TLS. Please note that many of the message stem back to the TLS library being used. In those cases, there is not always a good explanation available in rsyslog alone. diff --git a/doc/tls_cert_machine.html b/doc/tls_cert_machine.html new file mode 100644 index 0000000..fe2ee90 --- /dev/null +++ b/doc/tls_cert_machine.html @@ -0,0 +1,169 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS-protected syslog: generating the machine certificate</title> +</head> +<body> + +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-06-18)</i></small></p> + +<ul> +<li><a href="rsyslog_secure_tls.html">Overview</a> +<li><a href="tls_cert_scenario.html">Sample Scenario</a> +<li><a href="tls_cert_ca.html">Setting up the CA</a> +<li><a href="tls_cert_machine.html">Generating Machine Certificates</a> +<li><a href="tls_cert_server.html">Setting up the Central Server</a> +<li><a href="tls_cert_client.html">Setting up syslog Clients</a> +<li><a href="tls_cert_udp_relay.html">Setting up the UDP syslog relay</a> +<li><a href="tls_cert_summary.html">Wrapping it all up</a> +</ul> + +<h3>generating the machine certificate</h3> +<p>In this step, we generate certificates for each of the machines. Please note +that both clients and servers need certificates. The certificate identifies each +machine to the remote peer. The DNSName specified inside the certificate can +be specified inside the $<object>PermittedPeer config statements. +<p>For now, we assume that a single person (or group) is responsible for the whole +rsyslog system and thus it is OK if that single person is in posession of all +machine's private keys. This simplification permits us to use a somewhat less +complicated way of generating the machine certificates. So, we generate both the private +and public key on the CA (which is NOT a server!) and then copy them over to the +respective machines. +<p>If the roles of machine and CA administrators are split, the private key must +be generated by the machine administrator. This is done via a certificate request. +This request is then sent to the CA admin, which in turn generates the certificate +(containing the public key). The CA admin then sends back the certificate to the +machine admin, who installs it. That way, the CA admin never get's hold of the +machine's private key. Instructions for this mode will be given in a later revision +of this document. +<p><b>In any case, it is vital that the machine's private key is protected. Anybody +able to obtain that private key can imporsonate as the machine to which it belongs, thus +breaching your security.</b> +<h3>Sample Screen Session</h3> +<p>Text in red is user input. Please note that for some questions, there is no +user input given. This means the default was accepted by simply pressing the +enter key. +<p><b>Please note:</b> you need to substitute the names specified below with values +that match your environment. Most importantly, machine.example.net must be replaced +by the actual name of the machine that will be using this certificate. For example, +if you generate a certificate for a machine named "server.example.com", you need +to use that name. If you generate a certificate for "client.example.com", you need +to use this name. Make sure that each machine certificate has a unique name. If not, +you can not apply proper access control. +<code><pre> +[root@rgf9dev sample]# <font color="red">certtool --generate-privkey --outfile key.pem --bits 2048</font> +Generating a 2048 bit RSA private key... +[root@rgf9dev sample]# <font color="red">certtool --generate-request --load-privkey key.pem --outfile request.pem</font> +Generating a PKCS #10 certificate request... +Country name (2 chars): <font color="red">US</font> +Organization name: <font color="red">SomeOrg</font> +Organizational unit name: <font color="red">SomeOU</font> +Locality name: <font color="red">Somewhere</font> +State or province name: <font color="red">CA</font> +Common name: <font color="red">machine.example.net</font> +UID: +Enter a dnsName of the subject of the certificate: +Enter the IP address of the subject of the certificate: +Enter the e-mail of the subject of the certificate: +Enter a challange password: +Does the certificate belong to an authority? (y/N): <font color="red">n</font> +Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N): +Will the certificate be used for encryption (RSA ciphersuites)? (y/N): +Is this a TLS web client certificate? (y/N): <font color="red">y</font> +Is this also a TLS web server certificate? (y/N): <font color="red">y</font> +[root@rgf9dev sample]# <font color="red">certtool --generate-certificate --load-request request.pem --outfile cert.pem --load-ca-certificate ca.pem --load-ca-privkey ca-key.pem</font> +Generating a signed certificate... +Enter the certificate's serial number (decimal): + + +Activation/Expiration time. +The certificate will expire in (days): 1000 + + +Extensions. +Do you want to honour the extensions from the request? (y/N): +Does the certificate belong to an authority? (Y/N): <font color="red">n</font> +Is this a TLS web client certificate? (Y/N): <font color="red">y</font> +Is this also a TLS web server certificate? (Y/N): <font color="red">y</font> +Enter the dnsName of the subject of the certificate: <font color="red">machine.example.net</font> <i>{This is the name of the machine that will use the certificate}</i> +Enter the IP address of the subject of certificate: +Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (Y/N): +Will the certificate be used for encryption (RSA ciphersuites)? (Y/N): +X.509 Certificate Information: + Version: 3 + Serial Number (hex): 485a3819 + Validity: + Not Before: Thu Jun 19 10:42:54 UTC 2008 + Not After: Wed Mar 16 10:42:57 UTC 2011 + Subject: C=US,O=SomeOrg,OU=SomeOU,L=Somewhere,ST=CA,CN=machine.example.net + Subject Public Key Algorithm: RSA + Modulus (bits 2048): + b2:4e:5b:a9:48:1e:ff:2e:73:a1:33:ee:d8:a2:af:ae + 2f:23:76:91:b8:39:94:00:23:f2:6f:25:ad:c9:6a:ab + 2d:e6:f3:62:d8:3e:6e:8a:d6:1e:3f:72:e5:d8:b9:e0 + d0:79:c2:94:21:65:0b:10:53:66:b0:36:a6:a7:cd:46 + 1e:2c:6a:9b:79:c6:ee:c6:e2:ed:b0:a9:59:e2:49:da + c7:e3:f0:1c:e0:53:98:87:0d:d5:28:db:a4:82:36:ed + 3a:1e:d1:5c:07:13:95:5d:b3:28:05:17:2a:2b:b6:8e + 8e:78:d2:cf:ac:87:13:15:fc:17:43:6b:15:c3:7d:b9 + Exponent: + 01:00:01 + Extensions: + Basic Constraints (critical): + Certificate Authority (CA): FALSE + Key Purpose (not critical): + TLS WWW Client. + TLS WWW Server. + Subject Alternative Name (not critical): + DNSname: machine.example.net + Subject Key Identifier (not critical): + 0ce1c3dbd19d31fa035b07afe2e0ef22d90b28ac + Authority Key Identifier (not critical): + fbfe968d10a73ae5b70d7b434886c8f872997b89 +Other Information: + Public Key Id: + 0ce1c3dbd19d31fa035b07afe2e0ef22d90b28ac + +Is the above information ok? (Y/N): <font color="red">y</font> + + +Signing certificate... +[root@rgf9dev sample]# <font color="red">rm -f request.pem</font> +[root@rgf9dev sample]# <font color="red">ls -l</font> +total 16 +-r-------- 1 root root 887 2008-06-19 12:33 ca-key.pem +-rw-r--r-- 1 root root 1029 2008-06-19 12:36 ca.pem +-rw-r--r-- 1 root root 1074 2008-06-19 12:43 cert.pem +-rw-r--r-- 1 root root 887 2008-06-19 12:40 key.pem +[root@rgf9dev sample]# # it may be a good idea to rename the files to indicate where they belong to +[root@rgf9dev sample]# <font color="red">mv cert.pem machine-cert.pem</font> +[root@rgf9dev sample]# <font color="red">mv key.pem machine-key.pem</font> +[root@rgf9dev sample]# +</pre></code> +<h3>Distributing Files</h3> +<p>Provide the machine with: +<ul> +<li>a copy of ca.pem +<li>cert.pem +<li>key.pem +</ul> +<p>This is how the relevant part of rsyslog.conf looks on the target machine: +<p> +<code><pre> +$DefaultNetstreamDriverCAFile /home/rger/proj/rsyslog/sample/ca.pem +$DefaultNetstreamDriverCertFile /home/rger/proj/rsyslog/sample/machine-cert.pem +$DefaultNetstreamDriverKeyFile /home/rger/proj/rsyslog/sample/machine-key.pem +</pre></code> +<p><b><font color="red">Never</font> provide anyone with ca-key.pem!</b> Also, make sure +nobody but the machine in question gets hold of key.pem. +<h2>Copyright</h2> +<p>Copyright (c) 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> diff --git a/doc/tls_cert_scenario.html b/doc/tls_cert_scenario.html new file mode 100644 index 0000000..88c7657 --- /dev/null +++ b/doc/tls_cert_scenario.html @@ -0,0 +1,50 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS-protected syslog: scenario</title> +</head> +<body> + +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-06-17)</i></small></p> + +<ul> +<li><a href="rsyslog_secure_tls.html">Overview</a> +<li><a href="tls_cert_scenario.html">Sample Scenario</a> +<li><a href="tls_cert_ca.html">Setting up the CA</a> +<li><a href="tls_cert_machine.html">Generating Machine Certificates</a> +<li><a href="tls_cert_server.html">Setting up the Central Server</a> +<li><a href="tls_cert_client.html">Setting up syslog Clients</a> +<li><a href="tls_cert_udp_relay.html">Setting up the UDP syslog relay</a> +<li><a href="tls_cert_summary.html">Wrapping it all up</a> +<li><a href="tls_cert_errmsgs.html">Frequently seen Error Messages</a> +</ul> + +<h3>Sample Scenario</h3> +<p>We have a quite simple scenario. There is one central syslog server, +named central.example.net. These server is being reported to by two Linux +machines with name zuse.example.net and turing.example.net. Also, there is a +third client - ada.example.net - which send both its own messages to the central +server but also forwards messages receive from an UDP-only capable router. We +hav decided to use ada.example.net because it is in the same local network +segment as the router and so we enjoy TLS' security benefits for forwarding the +router messages inside the corporate network. All systems (except the router) use +<a href="http://www.rsyslog.com/">rsyslog</a> as the syslog software.</p> +<p><center><img src="tls_cert_100.jpg"></center> +<p>Please note that the CA must not necessarily be connected to the rest of the +network. Actually, it may be considered a security plus if it is not. If the CA +is reachable via the regular network, it should be sufficiently secured (firewal +rules et al). Keep in mind that if the CA's security is breached, your overall +system security is breached. +<p>In case the CA is compromised, you need to regenerate the CA's certificate as well +as all individual machines certificates. +<h2>Copyright</h2> +<p>Copyright (c) 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> diff --git a/doc/tls_cert_server.html b/doc/tls_cert_server.html new file mode 100644 index 0000000..b784be1 --- /dev/null +++ b/doc/tls_cert_server.html @@ -0,0 +1,114 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS-protected syslog: central server setup</title> +</head> +<body> + +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-06-18)</i></small></p> + +<ul> +<li><a href="rsyslog_secure_tls.html">Overview</a> +<li><a href="tls_cert_scenario.html">Sample Scenario</a> +<li><a href="tls_cert_ca.html">Setting up the CA</a> +<li><a href="tls_cert_machine.html">Generating Machine Certificates</a> +<li><a href="tls_cert_server.html">Setting up the Central Server</a> +<li><a href="tls_cert_client.html">Setting up syslog Clients</a> +<li><a href="tls_cert_udp_relay.html">Setting up the UDP syslog relay</a> +<li><a href="tls_cert_summary.html">Wrapping it all up</a> +</ul> + +<h3>Setting up the Central Server</h3> +<p>In this step, we configure the central server. We assume it accepts messages only +via TLS protected plain tcp based syslog from those peers that are explicitely permitted +to send to it. The picture below show our configuration. This step configures +the server central.example.net. +<p><center><img src="tls_cert_100.jpg"></center> +<p><i><font color="red"><b>Important:</b> Keep in mind that the order of configuration directives +is very important in rsyslog. As such, the samples given below do only work if the given +order is preserved.</font> Re-ordering the directives can break configurations and has broken them +in practice. If you intend to re-order them, please be sure that you fully understand how +the configuration language works and, most importantly, which statements form a block together. +Please also note that we understand the the current configuration file format is +ugly. However, there has been more important work in the way of enhancing it. If you would like +to contribute some time to improve the config file language, please let us know. Any help +is appreciated (be it doc or coding work!).</i> +<p>Steps to do: +<ul> +<li>make sure you have a functional CA (<a href="tls_cert_ca.html">Setting up the CA</a>) +<li>generate a machine certificate for central.example.net (follow instructions in + <a href="tls_cert_machine.html">Generating Machine Certificates</a>) +<li>make sure you copy over ca.pem, machine-key.pem ad machine-cert.pem to the central server. +Ensure that no user except root can access them (<b>even read permissions are really bad</b>). +<li>configure the server so that it accepts messages from all machines in the +example.net domain that have certificates from your CA. Alternatively, you may also +precisely define from which machine names messages are accepted. See sample rsyslog.conf +below. +</ul> +In this setup, we use wildcards to ease adding new systems. We permit the server to accept +messages from systems whos names match *.example.net. +<pre><code> +$InputTCPServerStreamDriverPermittedPeer *.example.net +</code></pre> +This will match zuse.example.net and +turing.example.net, but NOT pascal.otherdepartment.example.net. If the later would be desired, +you can (and need) to include additional permitted peer config statments: +<pre><code> +$InputTCPServerStreamDriverPermittedPeer *.example.net +$InputTCPServerStreamDriverPermittedPeer *.otherdepartment.example.net +$InputTCPServerStreamDriverPermittedPeer *.example.com +</code></pre> +<p>As can be seen with example.com, the different permitted peers need NOT to be in a single +domain tree. Also, individual machines can be configured. For example, if only zuse, turing +and ada should be able to talk to the server, you can achive this by: +<pre><code> +$InputTCPServerStreamDriverPermittedPeer zuse.example.net +$InputTCPServerStreamDriverPermittedPeer turing.example.net +$InputTCPServerStreamDriverPermittedPeer ada.example.net +</code></pre> +<p>As an extension to the (upcoming) IETF syslog/tls standard, you can specify some text +together with a domain component wildcard. So "*server.example.net", "server*.example.net" +are valid permitted peers. However "server*Fix.example.net" is NOT a valid wildcard. The +IETF standard permits no text along the wildcards. +<p>The reason we use wildcards in the default setup is that it makes it easy to add systems +without the need to change the central server's configuration. It is important to understand that +the central server will accept names <b>only</b> (no exception) if the client certificate was +signed by the CA we set up. So if someone tries to create a malicious certificate with +a name "zuse.example.net", the server will <b>not</b> accept it. So a wildcard is safe +as long as you ensure CA security is not breached. Actually, you authorize a client by issuing +the certificate to it. +<p><b>At this point, please be reminded once again that your security needs may be quite different from +what we assume in this tutorial. Evaluate your options based on your security needs.</b> +<h3>Sample syslog.conf</h3> +<p>Keep in mind that this rsyslog.conf accepts messages via TCP, only. The only other +source accepted is messages from the server itself. +<code><pre> +$ModLoad imuxsock # local messages +$ModLoad imtcp # TCP listener + +# make gtls driver the default +$DefaultNetstreamDriver gtls + +# certificate files +$DefaultNetstreamDriverCAFile /rsyslog/protected/ca.pem +$DefaultNetstreamDriverCertFile /rsyslog/protected/machine-cert.pem +$DefaultNetstreamDriverKeyFile /rsyslog/protected/machine-key.pem + +$InputTCPServerStreamDriverAuthMode x509/name +$InputTCPServerStreamDriverPermittedPeer *.example.net +$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode +$InputTCPServerRun 10514 # start up listener at port 10514 +</pre></code> +<p><font color="red"><b>Be sure to safeguard at least the private key (machine-key.pem)!</b> +If some third party obtains it, you security is broken!</font> +<h2>Copyright</h2> +<p>Copyright (c) 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> diff --git a/doc/tls_cert_summary.html b/doc/tls_cert_summary.html new file mode 100644 index 0000000..95844a8 --- /dev/null +++ b/doc/tls_cert_summary.html @@ -0,0 +1,53 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS-protected syslog: Summary</title> +</head> +<body> + +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-07-03)</i></small></p> + +<ul> +<li><a href="rsyslog_secure_tls.html">Overview</a> +<li><a href="tls_cert_scenario.html">Sample Scenario</a> +<li><a href="tls_cert_ca.html">Setting up the CA</a> +<li><a href="tls_cert_machine.html">Generating Machine Certificates</a> +<li><a href="tls_cert_server.html">Setting up the Central Server</a> +<li><a href="tls_cert_client.html">Setting up syslog Clients</a> +<li><a href="tls_cert_udp_relay.html">Setting up the UDP syslog relay</a> +<li><a href="tls_cert_summary.html">Wrapping it all up</a> +</ul> + +<h3>Summary</h3> +<p>If you followed the steps outlined in this documentation set, you now have +a reasonable (for most needs) secure setup for the following environment: +<center><img src="tls_cert_100.jpg"></center> +<p>You have learned about the security decisions involved and which we +made in this example. <b>Be once again reminded that you must make sure yourself +that whatever you do matches your security needs!</b> There is no guarantee that +what we generally find useful actually is. It may even be totally unsuitable for +your environment. +<p>In the example, we created a rsyslog certificate authority (CA). Guard the CA's +files. You need them whenever you need to create a new machine certificate. We also saw how +to generate the machine certificates themselfs and distribute them to the individual +machines. Also, you have found some configuration samples for a sever, a client and +a syslog relay. Hopefully, this will enable you to set up a similar system in many +environments. +<p>Please be warned that you defined some expiration dates for the certificates. +After they are reached, the certificates are no longer valid and rsyslog will NOT +accept them. At that point, syslog messages will no longer be transmitted (and rsyslogd +will heavily begin to complain). So it is a good idea to make sure that you renew the +certificates before they expire. Recording a reminder somewhere is probably a good +idea. +<p>If you have any more questions, please visit the <a href="http://kb.monitorware.com/rsyslog-f40.html">rsyslog forum</a> and simply ask ;) +<h2>Copyright</h2> +<p>Copyright (c) 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> diff --git a/doc/tls_cert_udp_relay.html b/doc/tls_cert_udp_relay.html new file mode 100644 index 0000000..d44689c --- /dev/null +++ b/doc/tls_cert_udp_relay.html @@ -0,0 +1,92 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS-protected syslog: UDP relay setup</title> +</head> +<body> + +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-07-03)</i></small></p> + +<ul> +<li><a href="rsyslog_secure_tls.html">Overview</a> +<li><a href="tls_cert_scenario.html">Sample Scenario</a> +<li><a href="tls_cert_ca.html">Setting up the CA</a> +<li><a href="tls_cert_machine.html">Generating Machine Certificates</a> +<li><a href="tls_cert_server.html">Setting up the Central Server</a> +<li><a href="tls_cert_client.html">Setting up syslog Clients</a> +<li><a href="tls_cert_udp_relay.html">Setting up the UDP syslog relay</a> +<li><a href="tls_cert_summary.html">Wrapping it all up</a> +</ul> + +<h3>Setting up the UDP syslog relay</h3> +<p>In this step, we configure the UDP relay ada.example.net. +As a reminder, that machine relays messages from a local router, which only +supports UDP syslog, to the central syslog server. The router does not talk +directly to it, because we would like to have TLS protection for its sensitve +logs. If the router and the syslog relay are on a sufficiently secure private +network, this setup can be considered reasonable secure. In any case, it is the +best alternative among the possible configuration scenarios. +<p><center><img src="tls_cert_100.jpg"></center> +<p>Steps to do: +<ul> +<li>make sure you have a functional CA (<a href="tls_cert_ca.html">Setting up the CA</a>) +<li>generate a machine certificate for ada.example.net (follow instructions in + <a href="tls_cert_machine.html">Generating Machine Certificates</a>) +<li>make sure you copy over ca.pem, machine-key.pem ad machine-cert.pem to the client. +Ensure that no user except root can access them (<b>even read permissions are really bad</b>). +<li>configure the client so that it checks the server identity and sends messages only +if the server identity is known. +</ul> +<p>These were essentially the same steps as for any +<a href="tls_cert_client.html">TLS syslog client</a>. We now need to add the +capability to forward the router logs: +<ul> +<li>make sure that the firewall rules permit message recpetion on UDP port 514 (if you use +a non-standard port for UDP syslog, make sure that port number is permitted). +<li>you may want to limit who can send syslog messages via UDP. A great place to do this +is inside the firewall, but you can also do it in rsyslog.conf via an $AllowedSender +directive. We have used one in the sample config below. Please be aware that this is +a kind of weak authentication, but definitely better than nothing... +<li>add the UDP input plugin to rsyslog's config and start a UDP listener +<li>make sure that your forwarding-filter permits to forward messages received +from the remote router to the server. In our sample scenario, we do not need to +add anything special, because all messages are forwarded. This includes messages +received from remote hosts. +</ul> +<p><b>At this point, please be reminded once again that your security needs may be quite different from +what we assume in this tutorial. Evaluate your options based on your security needs.</b> +<h3>Sample syslog.conf</h3> +<p>Keep in mind that this rsyslog.conf sends messages via TCP, only. Also, we do not +show any rules to write local files. Feel free to add them. +<code><pre> +# start a UDP listener for the remote router +$ModLoad imudp # load UDP server plugin +$AllowedSender UDP, 192.0.2.1 # permit only the router +$UDPServerRun 514 # listen on default syslog UDP port 514 + +# make gtls driver the default +$DefaultNetstreamDriver gtls + +# certificate files +$DefaultNetstreamDriverCAFile /rsyslog/protected/ca.pem +$DefaultNetstreamDriverCertFile /rsyslog/protected/machine-cert.pem +$DefaultNetstreamDriverKeyFile /rsyslog/protected/machine-key.pem + +$ActionSendStreamDriverAuthMode x509/name +$ActionSendStreamDriverPermittedPeer central.example.net +$ActionSendStreamDriverMode 1 # run driver in TLS-only mode +*.* @@central.example.net:10514 # forward everything to remote server +</pre></code> +<p><font color="red"><b>Be sure to safeguard at least the private key (machine-key.pem)!</b> +If some third party obtains it, you security is broken!</font> +<h2>Copyright</h2> +<p>Copyright © 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> |