diff options
Diffstat (limited to 'doc')
34 files changed, 4231 insertions, 1286 deletions
diff --git a/doc/Makefile.am b/doc/Makefile.am index 57e93a6..aa4e8a7 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -1,27 +1,44 @@ html_files = \ bugs.html \ + debug.html \ features.html \ generic_design.html \ + expression.html \ history.html \ how2help.html \ install.html \ ipv6.html \ + log_rotation_fix_size.html \ manual.html \ man_rsyslogd.html \ modules.html \ property_replacer.html \ + rsyslog_ng_comparison.html \ rsyslog_conf.html \ rsyslog-example.conf \ rsyslog_mysql.html \ rsyslog_packages.html \ + rsyslog_high_database_rate.html \ rsyslog_php_syslog_ng.html \ rsyslog_recording_pri.html \ rsyslog_stunnel.html \ - professional_support.html \ status.html \ syslog-protocol.html \ version_naming.html \ contributors.html \ + dev_queue.html \ + omsnmp.html \ + omlibdbi.html \ + imfile.html \ + imtcp.html \ + imgssapi.html \ + professional_support.html \ + queues.html \ + queueWorkerLogic.dia \ + queueWorkerLogic.jpg \ + queueWorkerLogic_small.jpg \ + rainerscript.html \ + rscript_abnf.html \ rsconf1_actionexeconlywhenpreviousissuspended.html \ rsconf1_actionresumeinterval.html \ rsconf1_allowedsender.html \ @@ -45,10 +62,13 @@ html_files = \ rsconf1_gssmode.html \ rsconf1_includeconfig.html \ rsconf1_mainmsgqueuesize.html \ + rsconf1_markmessageperiod.html \ rsconf1_modload.html \ rsconf1_moddir.html \ rsconf1_repeatedmsgreduction.html \ rsconf1_resetconfigvariables.html \ - rsconf1_umask.html + rsconf1_umask.html \ + v3compatibility.html \ + src/classes.dia EXTRA_DIST = $(html_files) diff --git a/doc/Makefile.in b/doc/Makefile.in index 2992376..ed21b20 100644 --- a/doc/Makefile.in +++ b/doc/Makefile.in @@ -135,6 +135,8 @@ htmldir = @htmldir@ includedir = @includedir@ infodir = @infodir@ install_sh = @install_sh@ +libdbi_cflags = @libdbi_cflags@ +libdbi_libs = @libdbi_libs@ libdir = @libdir@ libexecdir = @libexecdir@ localedir = @localedir@ @@ -152,9 +154,15 @@ program_transform_name = @program_transform_name@ psdir = @psdir@ pthreads_cflags = @pthreads_cflags@ pthreads_libs = @pthreads_libs@ +relp_cflags = @relp_cflags@ +relp_libs = @relp_libs@ +rfc3195_cflags = @rfc3195_cflags@ +rfc3195_libs = @rfc3195_libs@ rt_libs = @rt_libs@ sbindir = @sbindir@ sharedstatedir = @sharedstatedir@ +snmp_cflags = @snmp_cflags@ +snmp_libs = @snmp_libs@ srcdir = @srcdir@ sysconfdir = @sysconfdir@ target_alias = @target_alias@ @@ -163,28 +171,45 @@ top_srcdir = @top_srcdir@ zlib_libs = @zlib_libs@ html_files = \ bugs.html \ + debug.html \ features.html \ generic_design.html \ + expression.html \ history.html \ how2help.html \ install.html \ ipv6.html \ + log_rotation_fix_size.html \ manual.html \ man_rsyslogd.html \ modules.html \ property_replacer.html \ + rsyslog_ng_comparison.html \ rsyslog_conf.html \ rsyslog-example.conf \ rsyslog_mysql.html \ rsyslog_packages.html \ + rsyslog_high_database_rate.html \ rsyslog_php_syslog_ng.html \ rsyslog_recording_pri.html \ rsyslog_stunnel.html \ - professional_support.html \ status.html \ syslog-protocol.html \ version_naming.html \ contributors.html \ + dev_queue.html \ + omsnmp.html \ + omlibdbi.html \ + imfile.html \ + imtcp.html \ + imgssapi.html \ + professional_support.html \ + queues.html \ + queueWorkerLogic.dia \ + queueWorkerLogic.jpg \ + queueWorkerLogic_small.jpg \ + rainerscript.html \ + rscript_abnf.html \ rsconf1_actionexeconlywhenpreviousissuspended.html \ rsconf1_actionresumeinterval.html \ rsconf1_allowedsender.html \ @@ -208,11 +233,14 @@ html_files = \ rsconf1_gssmode.html \ rsconf1_includeconfig.html \ rsconf1_mainmsgqueuesize.html \ + rsconf1_markmessageperiod.html \ rsconf1_modload.html \ rsconf1_moddir.html \ rsconf1_repeatedmsgreduction.html \ rsconf1_resetconfigvariables.html \ - rsconf1_umask.html + rsconf1_umask.html \ + v3compatibility.html \ + src/classes.dia EXTRA_DIST = $(html_files) all: all-am diff --git a/doc/debug.html b/doc/debug.html new file mode 100644 index 0000000..de77f04 --- /dev/null +++ b/doc/debug.html @@ -0,0 +1,41 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>Debug Support</title></head> +<body> +<h1>Debug Support</h1> +<p> +Rsyslog provides a number of debug aids. Some of them are activated by +adding the --enable-rtinst ./configure option ("rtinst" means runtime +instrumentation). Turning debugging on obviously costs some performance +(in some cases considerable). +</p> +<p>This is document is just being created and thus terse.</p> +<p style="font-weight: bold;">Signals supported</p> +<p>SIGUSR1 - turns debug messages on and off (expect this signal +to go away over time)</p> +<p>SIGUSR2 - outputs debug information (including active threads +and a call stack) for the state when SIGUSR2 was received. This is a +one-time output. Can be sent as often as the user likes.</p> +<p style="font-weight: bold;">Environment Variables</p> +<p>There are two environment variables that set several debug settings. The "RSYSLOG_DEBUGLOG" (sample: RSYSLOG_DEBUGLOG="/path/to/debuglog/") +writes (allmost) +all debug message to the specified log file in addition to stdout. Some +system messages (e.g. segfault or abort message) are not written to the +file as we can not capture them. Runtime debug support is controlled by +"RSYSLOG_DEBUG". It contains an option string with the following +options possible (all are case insensitive):</p><ul><li><span style="font-weight: bold;">LogFuncFlow</span> - print out the logical flow of functions (entering and exiting them)</li><li><span style="font-weight: bold;">FileTrace</span> - specifies which files to trace LogFuncFlow. If <span style="font-weight: bold;">not</span> +set (the default), a LogFuncFlow trace is provided for all files. Set +to limit it to the files specified. FileTrace may be specified multiple +times, one file each (e.g. export RSYSLOG_DEBUG="LogFuncFlow +FileTrace=vm.c FileTrace=expr.c"</li><li><span style="font-weight: bold;">PrintFuncDB</span> - print the content of the debug function database whenever debug information is printed (e.g. abort case)!</li><li><span style="font-weight: bold;">PrintAllDebugInfoOnExit</span> - print all debug information immediately before rsyslogd exits (<span style="font-weight: bold; font-style: italic;">currently not implemented!</span>)</li><li><span style="font-weight: bold;">PrintMutexAction</span> - print mutex action as it happens. Useful for finding deadlocks and such.</li><li><span style="font-weight: bold;">NoLogTimeStamp</span> - do not prefix log lines with a timestamp (default is to do that).</li><li><span style="font-weight: bold;">NoStdOut</span> - do not emit debug messages to stdout. If RSYSLOG_DEBUGLOG is not set, this means no messages will be displayed at all.</li><li><span style="font-weight: bold;">help</span> - display a very short list of commands - hopefully a life saver if you can't access the documentation...</li></ul> +<ul> +</ul> +<p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html>
\ No newline at end of file diff --git a/doc/dev_queue.html b/doc/dev_queue.html new file mode 100644 index 0000000..bf2af7f --- /dev/null +++ b/doc/dev_queue.html @@ -0,0 +1,250 @@ +<html> +<head> +<title>rsyslog queue object</title> +</head> +<body> +<h1>The rsyslog queue object</h1> +<p>This page reflects the status as of 2008-01-17. The documentation is still incomplete. +Target audience is developers and users who would like to get an in-depth understanding of +queues as used in <a href="http://www.rsyslog.com/">rsyslog</a>.</p> +<p><b>Please note that this document is outdated and does not longer reflect the +specifics of the queue object. However, I have decided to leave it in the doc +set, as the overall picture provided still is quite OK. I intend to update this +document somewhat later when I have reached the "store-and-forward" milestone.</b></p> +<h1>Some definitions</h1> +<p>A queue is DA-enabled if it is configured to use disk-assisted mode when +there is need to. A queue is in DA mode (or DA run mode), when it actually runs +disk assisted.</p> +<h1>Implementation Details</h1> +<h2>Disk-Assisted Mode</h2> +<p>Memory-Type queues may utilize disk-assisted (DA) mode. DA mode is enabled +whenever a queue file name prefix is provided. This is called DA-enabled mode. +If DA-enabled, the queue operates as a regular memory queue until a high water +mark is reached. If that happens, the queue activates disk assistance (called +"runs disk assisted" or "runs DA" - you can find that often in source file +comments). To do so, it creates a helper queue instance (the DA queue). At that +point, there are two queues running - the primary queue's consumer changes to a +shuffle-to-DA-queue consumer and the original primary consumer is assigned to +the DA queue. Existing and new messages are spooled to the disk queue, where the +DA worker takes them from and passes them for execution to the actual consumer. +In essence, the primary queue has now become a memory buffer for the DA queue. +The primary queue will be drained until a low water mark is reached. At that +point, processing is held. New messages enqueued to the primary queue will not +be processed but kept in memory. Processing resumes when either the high water +mark is reached again or the DA queue indicates it is empty. If the DA queue is +empty, it is shut down and processing of the primary queue continues as a +regular in-memory queue (aka "DA mode is shut down"). The whole thing iterates +once the high water mark is hit again.</p> +<p>There is one special case: if the primary queue is shut down and could not +finish processing all messages within the configured timeout periods, the DA +queue is instantiated to take up the remaining messages. These will be preserved +and be processed during the next run. During that period, the DA queue runs in +"enqueue-only" mode and does not execute any consumer. Draining the primary +queue is typically very fast. If that behaviour is not desired, it can be turned +of via parameters. In that case, any remaining in-memory messages are lost.</p> +<p>Due to the fact that when running DA two queues work closely together and +worker threads (including the DA worker) may shut down at any time (due to +timeout), processing synchronization and startup and shutdown is somewhat +complex. I'll outline the exact conditions and steps down here. I also do this +so that I know clearly what to develop to, so please be patient if the +information is a bit too in-depth ;)</p> +<h2>DA Run Mode Initialization</h2> +<p>Three cases:</p> +<ol> + <li>any time during queueEnqObj() when the high water mark is hit</li> + <li>at queue startup if there is an on-disk queue present (presence of QI + file indicates presence of queue data)</li> + <li>at queue shutdown if remaining in-memory data needs to be persisted to + disk</li> +</ol> +<p>In <b>case 1</b>, the worker pool is running. When switching to DA mode, all +regular workers are sent termination commands. The DA worker is initiated. +Regular workers may run in parallel to the DA worker until they terminate. +Regular workers shall terminate as soon as their current consumer has completed. +They shall not execute the DA consumer.</p> +<p>In <b>case 2</b>, the worker pool is not yet running and is NOT started. The +DA worker is initiated.</p> +<p>In <b>case 3</b>, the worker pool is already shut down. The DA worker is +initiated. The DA queue runs in enqueue-only mode.</p> +<p>In all cases, the DA worker starts up and checks if DA mode is already fully +initialized. If not, it initializes it, what most importantly means construction +of the queue.</p> +<p>Then, regular worker processing is carried out. That is, the queue worker +will wait on empty queue and terminate after an timeout. However, If any message +is received, the DA consumer is executed. That consumer checks the low water +mark. If the low water mark is reached, it stops processing until either the +high water mark is reached again or the DA queue indicates it is empty (there is +a pthread_cond_t for this synchronization).</p> +<p>In theory, a <b>case-2</b> startup could lead to the worker becoming inactive +and terminating while waiting on the primary queue to fill. In practice, this is +highly unlikely (but only for the main message queue) because rsyslog issues a +startup message. HOWEVER, we can not rely on that, it would introduce a race. If +the primary rsyslog thread (the one that issues the message) is scheduled very +late and there is a low inactivty timeout for queue workers, the queue worker +may terminate before the startup message is issued. And if the on-disk queue +holds only a few messages, it may become empty before the DA worker is +re-initiated again. So it is possible that the DA run mode termination criteria +occurs while no DA worker is running on the primary queue.</p> +<p>In cases 1 and 3, the DA worker can never become inactive without hitting the +DA shutdown criteria. In <b>case 1</b>, it either shuffles messages from the +primary to the DA queue or it waits because it has the hit low water mark. </p> +<p>In <b>case 3</b>, it always shuffles messages between the queues (because, +that's the sole purpose of that run). In order for this to happen, the high +water mark has been set to the value of 1 when DA run mode has been initialized. +This ensures that the regular logic can be applied to drain the primary queue. +To prevent a hold due to reaching the low water mark, that mark must be changed +to 0 before the DA worker starts.</p> +<h2>DA Run Mode Shutdown</h2> +<p>In essence, DA run mode is terminated when the DA queue is empty and the +primary worker queue size is below the high water mark. It is also terminated +when the primary queue is shut down. The decision to switch back to regular +(non-DA) run mode is typically made by the DA worker. If it switches, the DA +queue is destructed and the regular worker pool is restarted. In some cases, the +queue shutdown process may initiate the "switch" (in this case more or less a +clean shutdown of the DA queue).</p> +<p>One might think that it would be more natural for the DA queue to detect +being idle and shut down itself. However, there are some issues associated with +that. Most importantly, all queue worker threads need to be shut down during +queue destruction. Only after that has happend, final destruction steps can +happen (else we would have a myriad of races). However, it is the DA queues +worker thread that detects it is empty (empty queue detection always happens at +the consumer side and must so). That would lead to the DA queue worker thread to +initiate DA queue destruction which in turn would lead to that very same thread +being canceled (because workers must shut down before the queue can be +destructed). Obviously, this does not work out (and I didn't even mention the +other issues - so let's forget about it). As such, the thread that enqueues +messages must destruct the queue - and that is the primary queue's DA worker +thread.</p> +<p>There are some subleties due to thread synchronization and the fact that the +DA consumer may not be running (in a <b>case-2 startup</b>). So it is not +trivial to reliably change the queue back from DA run mode to regular run mode. +The priority is a clean switch. We accept the fact that there may be situations +where we cleanly shut down DA run mode, just to re-enable it with the very next +message being enqueued. While unlikely, this will happen from time to time and +is considered perfectly legal. We can't predict the future and it would +introduce too great complexity to try to do something against that (that would +most probably even lead to worse performance under regular conditions).</p> +<p>The primary queue's DA worker thread may wait at two different places:</p> +<ol> + <li>after reaching the low water mark and waiting for either high water or + DA queue empty</li> + <li>at the regular pthread_cond_wait() on an empty primary queue</li> +</ol> +<p>Case 2 is unlikely, but may happen (see info above on a case 2 startup).</p> +<p><b>The DA worker may also not wait at all,</b> because it is actively +executing and shuffeling messages between the queues. In that case, however, the +program flow passes both of the two wait conditions but simply does not wait.</p> +<p><b>Finally, the DA worker may be inactive </b>(again, with a case-2 startup). +In that case no work(er) at all is executed. Most importantly, without the DA +worker being active, nobody will ever detect the need to change back to regular +mode. If we have this situation, the very next message enqueued will cause the +switch, because then the DA run mode shutdown criteria is met. However, it may +take close to eternal for this message to arrive. During that time, disk and +memory resources for the DA queue remain allocated. This also leaves processing +in a sub-optimal state and it may take longer than necessary to switch back to +regular queue mode when a message burst happens. In extreme cases, this could +even lead to shutdown of DA run mode, which takes so long that the high water +mark is passed and DA run mode is immediately re-initialized - while with an +immediate switch, the message burst may have been able to be processed by the +in-memory queue without DA support.</p> +<p>So in short, it is desirable switch to regular run mode as soon as possible. +To do this, we need an active DA worker. The easy solution is to initiate DA +worker startup from the DA queue's worker once it detects empty condition. To do +so, the DA queue's worker must call into a "<i>DA worker startup initiation</i>" +routine inside the main queue. As a reminder, the DA worker will most probably +not receive the "DA queue empty" signal in that case, because it will be long +sent (in most cases) before the DA worker even waits for it. So <b>it is vital +that DA run mode termination checks be done in the DA worker before it goes into +any wait condition</b>.</p> +<p>Please note that the "<i>DA worker startup initiation</i>" routine may be +called concurrently from multiple initiators. <b>To prevent a race, it must be +guarded by the queue mutex </b>and return without any action (and no error +code!) if the DA worker is already initiated.</p> +<p>All other cases can be handled by checking the termination criteria +immediately at the start of the worker and then once again for each run. The +logic follows this simplified flow diagram:</p> +<p align="center"><a href="queueWorkerLogic.jpg"> +<img border="0" src="queueWorkerLogic_small.jpg" width="431" height="605"></a></p> +<p>Some of the more subtle aspects of worker processing (e.g. enqueue thread +signaling and other fine things) have been left out in order to get the big +picture. What is called "check DA mode switchback..." right after "worker init" +is actually a check for the worker's termination criteria. Typically, <b>the +worker termination criteria is a shutdown request</b>. However, <b>for a DA +worker, termination is also requested if the queue size is below the high water +mark AND the DA queue is empty</b>. There is also a third termination criteria +and it is not even on the chart: that is the inactivity timeout, which exists in +all modes. Note that while the inactivity timeout shuts down a thread, it +logically does not terminate the worker pool (or DA worker): workers are +restarted on an as-needed basis. However, inactivity timeouts are very important +because they require us to restart workers in some situations where we may +expect a running one. So always keep them on your mind.</p> +<h2>Queue Destruction</h2> +<p>Now let's consider <b>the case of destruction of the primary queue. </b>During +destruction, our focus is on loosing as few messages as possible. If the +queue is not DA-enabled, there is nothing but the configured timeouts to handle +that situation. However, with a DA-enabled queue there are more options.</p> +<p>If the queue is DA-enabled, it may be <i>configured to persist messages to +disk before it is terminated</i>. In that case, loss of messages never occurs +(at the price of a potentially lengthy shutdown). Even if that setting is not +applied, the queue should drain as many messages as possible to the disk. For +that reason, it makes no sense to wait on a low water mark. Also, if the queue +is already in DA run mode, it does not make any sense to switch back to regular +run mode during termination and then try to process some messages via the +regular consumer. It is much more appropriate the try completely drain the queue +during the remaining timeout period. For the same reason, it is preferred that +no new consumers be activated (via the DA queue's worker), as they only cost +valuable CPU cycles and, more importantly, would potentially be long(er)-running +and possibly be needed to be cancelled. To prevent all of that, <b>queue +parameters are changed for DA-enabled queues:</b> the high water mark is to 1 +and the low water mark to 0 on the primary queue. The DA queue is commanded to +run in enqueue-only mode. If the primary queue is <i>configured to persist +messages to disk before it is terminated</i>, its SHUTDOWN timeout is changed to +to eternal. These parameters will cause the queue to drain as much as possible +to disk (and they may cause a case 3 DA run mode initiation). Please note that +once the primary queue has been drained, the DA queue's worker will +automatically switch back to regular (non-DA) run mode. <b>It must be ensured +that no worker cancellation occurs during that switchback</b>. Please note that +the queue may not switch back to regular run mode if it is not <i>configured to +persist messages to disk before it is terminated</i>. In order to apply the new +parameters, <b>worker threads must be awakened.</b> Remember we may not be in DA +run mode at this stage. In that case, the regular workers must be awakened, which +then will switch to DA run mode. No worker may be active, in that case one must +be initiated. If in DA run mode and the DA worker is inactive, the "<i>DA +worker startup initiation</i>" must be called to activate it. That routine +ensures only one DA worker is started even with multiple concurrent callers - +this may be the case here. The DA queue's worker may have requested DA worker +startup in order to terminate on empty queue (which will probably not be honored +as we have changed the low water mark).</p> +<p>After all this is done, the queue destructor requests termination of the +queue's worker threads. It will use the normal timeouts and potentially cancel +too-long running worker threads. <b>The shutdown process must ensure that all +workers reach running state before they are commanded to terminate</b>. +Otherwise it may run into a race condition that could lead to a false shutdown +with workers running asynchronously. As a few workers may have just been started +to initialize (to apply new parameter settings), the probability for this race +condition is extremely high, especially on single-CPU systems.</p> +<p>After all workers have been shut down (or cancelled), the queue may still be +in DA run mode. If so, this must be terminated, which now can simply be done by +destructing the DA queue object. This is not a real switchback to regular run +mode, but that doesn't matter because the queue object will soon be gone away.</p> +<p>Finally, the queue is mostly shut down and ready to be actually destructed. +As a last try, the queuePersists() entry point is called. It is used to persists +a non-DA-enabled queue in whatever way is possible for that queue. There may be +no implementation for the specific queue type. Please note that this is not just +a theoretical construct. This is an extremely important code path when the DA +queue itself is destructed. Remember that it is a queue object in its own right. +The DA queue is obviously not DA-enabled, so it calls into queuePersists() +during its destruction - this is what enables us to persist the disk queue!</p> +<p>After that point, left over queue resources (mutexes, dynamic memory, ...) +are freed and the queue object is actually destructed.</p> +<h2>Copyright</h2> +<p>Copyright (c) 2008 <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> +and <a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p>Permission is granted to copy, distribute and/or modify this document under +the terms of the GNU Free Documentation License, Version 1.2 or any later +version published by the Free Software Foundation; with no Invariant Sections, +no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be +viewed at <a href="http://www.gnu.org/copyleft/fdl.html"> +http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body> +</html>
\ No newline at end of file diff --git a/doc/expression.html b/doc/expression.html new file mode 100644 index 0000000..e7eb784 --- /dev/null +++ b/doc/expression.html @@ -0,0 +1,16 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>Expressions</title></head> +<body> +<h1>Expressions</h1> +<p>Rsyslog supports expressions at a growing number of places. So +far, they are supported for filtering messages.</p><p>Expression support is provided by RainerScript. For now, please see the formal expression definition in <a href="rainerscript.html">RainerScript ABNF</a>. It is the "expr" node.</p><p>C-like comments (/* some comment */) are supported <span style="font-weight: bold;">inside</span> the expression, but not yet in the rest of the configuration file.</p><p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html>
\ No newline at end of file diff --git a/doc/features.html b/doc/features.html index c71194d..9573030 100644 --- a/doc/features.html +++ b/doc/features.html @@ -1,66 +1,129 @@ -<html> -<head> -<title>rsyslog features</title> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>rsyslog features</title> + </head> <body> <h1>RSyslog - Features</h1> -<p><b>This page lists both current features as well as those being considered -for future versions of rsyslog.</b> If you think a feature is missing, drop -<a href="mailto:rgerhards@adiscon.com">Rainer</a> a note. Rsyslog is a vital -project. Features are added each few days. If you would like to keep up of what -is going on, you can also subscribe to the <a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">rsyslog mailing list</a>. +<p><b>This page lists both current features as well as +those being considered for future versions of rsyslog.</b> If you +think a feature is missing, drop +<a href="mailto:rgerhards@adiscon.com">Rainer</a> a +note. Rsyslog is a vital project. Features are added each few days. If +you would like to keep up of what is going on, you can also subscribe +to the <a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">rsyslog +mailing list</a>.</p> +<p><span style="font-weight: bold;">A better +structured feature list is now contained in our </span><a style="font-weight: bold;" href="rsyslog_ng_comparison.html">rsyslog +vs. syslog-ng comparison</a><span style="font-weight: bold;">. +</span>Probably that page will replace this one in the +future. </p> <h2>Current Features</h2> <ul> - - <li>native support for <a href="rsyslog_mysql.html">writing to MySQL databases</a><li> - native support for writing to Postgres databases<li>support for (plain) tcp - based syslog - much better reliability<li>support for sending and receiving - compressed syslog messages<li>ability to configure backup syslog/database - servers - if the primary fails, control is switched to a prioritized list of - backups<li>support for receiving messages via
reliable <a href="http://www.monitorware.com/Common/en/glossary/rfc3195.php">
RFC 3195</a> delivery<li>ability to generate file names and directories (log targets) - dynamically, based on many different properties<li>control of log output format, - including ability to present channel and priority as visible log data<li>good timestamp format control; at a minimum, ISO 8601/RFC 3339 - second-resolution UTC zone<li>ability to reformat message contents and work with substrings<li>support for - log files larger than 2gb<li>support for file size limitation and automatic - rollover command execution<li>support for running multiple rsyslogd - instances on a single machine<li>support for <a href="rsyslog_stunnel.html"> - ssl-protected syslog</a> (via stunnel)<li>ability to filter on any part of - the message, not just facility and severity<li>ability to use regular - expressions in filters<li>support for discarding - messages based on filters<li>ability to execute shell scripts on received - messages<li>control of whether the local hostname or the hostname of the - origin of the data is shown as the hostname in the output<li>ability to - preserve the original hostname in NAT environments and relay chains - <li>ability to limit the allowed network senders<li>powerful BSD-style
hostname and program name blocks for easy multi-host support<li>
multi-threaded (<a href="http://rgerhards.blogspot.com/2007/08/why-is-rsyslog-multi-threaded-and-is-it.html">is - this important? why?</a>)<li>very
experimental and volatile support for <a href="syslog-protocol.html">syslog-protocol</a>
compliant messages (it is volatile because standardization is currently
underway and this is a proof-of-concept implementation to aid this effort)<li> - experimental support for syslog-transport-tls based framing on syslog/tcp - connections<li> - a copy of klogd.c has been included under the name of rklogd for those Linux - systems that need one. So rsyslog is a full replacement for the sysklogd - package<li> - support for IPv6<li> - ability to control repeated line reduction ("last message repeated n times") - on a per selector-line basis<li> - supports sub-configuration files, which can be automatically read from - directories. Includes are specified in the main configuration file<li> - supports multiple actions per selector/filter condition<li> - MySQL and Postgres SQL functionality as a dynamically loadable plug-in<li> - support for GSS-API<li> - modular design for outputs - easily extensible</ul> +<li>native support for <a href="rsyslog_mysql.html">writing +to MySQL databases</a></li> +<li> native support for writing to Postgres databases</li> +<li>direct support for Firebird/Interbase, +OpenTDS (MS SQL, Sybase), SQLLite, Ingres, Oracle, and mSQL via libdbi, +a database abstraction layer (almost as good as native)</li> +<li>support for (plain) tcp based syslog - much better +reliability</li> +<li>support for sending and receiving compressed syslog messages</li> +<li>support for on-demand on-disk spooling of messages that can +not be processed fast enough (a great feature for <a href="rsyslog_high_database_rate.html">writing massive +amounts of syslog messages to a database</a>)</li> +<li>ability to monitor text files and convert their contents +into syslog messages (one per line)</li> +<li>ability to configure backup syslog/database servers - if +the primary fails, control is switched to a prioritized list of backups</li> +<li>support for receiving messages via reliable <a href="http://www.monitorware.com/Common/en/glossary/rfc3195.php"> +RFC 3195</a> delivery</li> +<li>ability to generate file names and directories (log +targets) dynamically, based on many different properties</li> +<li>control of log output format, including ability to present +channel and priority as visible log data</li> +<li>good timestamp format control; at a minimum, ISO 8601/RFC +3339 second-resolution UTC zone</li> +<li>ability to reformat message contents and work with +substrings</li> +<li>support for log files larger than 2gb</li> +<li>support for file size limitation and automatic rollover +command execution</li> +<li>support for running multiple rsyslogd instances on a single +machine</li> +<li>support for <a href="rsyslog_stunnel.html"> +ssl-protected syslog</a> (via stunnel)</li> +<li>ability to filter on any part of the message, not just +facility and severity</li> +<li>ability to use regular expressions in filters</li> +<li>support for discarding messages based on filters</li> +<li>ability to execute shell scripts on received messages</li> +<li>control of whether the local hostname or the hostname of +the origin of the data is shown as the hostname in the output</li> +<li>ability to preserve the original hostname in NAT +environments and relay chains </li> +<li>ability to limit the allowed network senders</li> +<li>powerful BSD-style hostname and program name blocks for +easy multi-host support</li> +<li> massively multi-threaded with dynamic work thread pools +that start up and shut themselves down on an as-needed basis (great for +high log volume on multicore machines)</li> +<li>very experimental and volatile support for <a href="syslog-protocol.html">syslog-protocol</a> +compliant messages (it is volatile because standardization is currently +underway and this is a proof-of-concept implementation to aid this +effort)</li> +<li> experimental support for syslog-transport-tls based +framing on syslog/tcp connections</li> +<li> the sysklogd's klogd functionality is implemented as the <i>imklog</i> +input plug-in. So rsyslog is a full replacement for the sysklogd package</li> +<li> support for IPv6</li> +<li> ability to control repeated line reduction ("last message +repeated n times") on a per selector-line basis</li> +<li> supports sub-configuration files, which can be +automatically read from directories. Includes are specified in the main +configuration file</li> +<li> supports multiple actions per selector/filter condition</li> +<li> MySQL and Postgres SQL functionality as a dynamically +loadable plug-in</li> +<li> modular design for inputs and outputs - easily extensible +via custom plugins</li> +<li> an easy-to-write to plugin interface</li> +<li> ability to send SNMP trap messages</li> +<li>support for arbitrary complex boolean, string and +arithmetic expressions in message filters</li> +</ul> <p> </p> <h2>Upcoming Features</h2> -<p>The list below is something like a repository of ideas we'd like to -implement. Features on this list are typically NOT scheduled for immediate -inclusion. We maintain a -<a href="http://bugzilla.adiscon.com/rsyslog-feature.html">feature -request tracker at our bugzilla</a>. This tracker has things typically within -reach of implementation. Users are encouraged to submit feature requests there -(or via our forums). <b>Please note that rsyslog v2 is feature-complete. New -features will be implemented in the v3 branch only. </b>Version 3 already has a -number of very exciting additional features.</p> +<p>The list below is something like a repository of ideas we'd +like to implement. Features on this list are typically NOT scheduled +for immediate inclusion. We maintain a +<a href="http://bugzilla.adiscon.com/rsyslog-feature.html">feature +request tracker at our bugzilla</a>. This tracker has things +typically within reach of implementation. Users are encouraged to +submit feature requests there (or via our forums). If we like them but +they look quite long-lived (aka "not soon to be implemented"), they +will possibly be migrated to this list here and at some time moved back +to the sourceforge tracker.</p> +<ul> +<li>implement native email-functionality in selector (probably +best done as a plug-in)</li> +<li>port it to more *nix variants (eg AIX and HP UX) - this +needs volunteers with access to those machines and knowledge </li> +<li>support for native SSL enryption of plain tcp syslog +sessions. This will most probably happen based on syslog-transport-tls.</li> +<li>pcre filtering - maybe (depending on feedback) - +simple regex already partly added. So far, this seems sufficient so +that there is no urgent need to do pcre</li> +<li>support for <a href="http://www.monitorware.com/Common/en/glossary/rfc3195.php">RFC +3195</a> as a sender - this is currently unlikely to happen, +because there is no real demand for it. Any work on RFC 3195 has been +suspend until we see some real interest in it. It is probably +much better to use TCP-based syslog, which is interoperable with a +large number of applications. You may also read my blog post on the +future of liblogging, which contains interesting information about the <a href="http://rgerhards.blogspot.com/2007/09/where-is-liblogging-heading-to.html"> +future of RFC 3195 in rsyslog</a>.</li> +</ul> <p>To see when each feature was added, see the -<a href="http://www.rsyslog.com/Topic4.phtml">rsyslog change log</a> (online -only).</p> -</body> -</html> +<a href="http://www.rsyslog.com/Topic4.phtml">rsyslog +change log</a> (online only).</p> +</body></html>
\ No newline at end of file diff --git a/doc/history.html b/doc/history.html index 0f9dbff..a06aaf5 100644 --- a/doc/history.html +++ b/doc/history.html @@ -1,7 +1,6 @@ -<html> -<head> -<title>rsyslog history</title> -</head> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>rsyslog history</title></head> <body> <h1>RSyslog - History</h1> @@ -10,28 +9,32 @@ reliable syslog over TCP, writing to MySQL databases and fully configurable output formats (including great timestamps).</b> Rsyslog was initiated by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a>. If you are interested to learn why Rainer initiated the project, you -may want to read his blog posting on "<a href="http://rgerhards.blogspot.com/2007/08/why-does-world-need-another-syslogd.html">why -the world needs another syslogd</a>".<p>Rsyslog has +may want to read his blog posting on "<a href="http://rgerhards.blogspot.com/2007/08/why-does-world-need-another-syslogd.html">why +the world needs another syslogd</a>".<p>Rsyslog has been forked in <b>2004</b> from the <a href="http://www.infodrom.org/projects/sysklogd/">sysklogd standard package</a>. The goal of the rsyslog project is to provide a feature-richer and reliable -syslog daemon while retaining drop-in replacement capabilities to stock syslogd. By "reliable", we mean support for reliable transmission +syslog daemon while retaining drop-in replacement capabilities to stock +syslogd. By "reliable", we mean support for reliable transmission modes like TCP or <a href="http://www.monitorware.com/Common/en/glossary/rfc3195.php">RFC 3195</a> (syslog-reliable). We do NOT imply that the sysklogd package is unreliable.</p> <p>The name "rsyslog" stems back to the planned support for syslog-reliable. Ironically, the initial release of rsyslog did NEITHER support syslog-reliable NOR tcp based syslog. Instead, it contained enhanced configurability and other enhancements -(like database support). The reason for this is that full support for RFC 3195 would require even more changes and especially fundamental architectural +(like database support). The reason for this is that full support for +RFC 3195 would require even more changes and especially fundamental +architectural changes. Also, questions asked on the loganalysis list and at other places indicated that RFC3195 is NOT a prime priority for users, but rather better control over the output format. So there we were, with a rsyslogd that covers a lot of enhancements, but not a single one -of these that made its name ;) Since version 0.9.2, receiving syslog messages -via plain tcp is finally supported, a bit later sending via TCP, too. Starting -with 1.11.0, RFC 3195 is finally support at the receiving side (a.k.a. "listener"). -Support for sending via RFC 3195 is still due. Anyhow, rsyslog has come much -closer to what it name promises.</p> +of these that made its name ;) Since version 0.9.2, receiving syslog +messages via plain tcp is finally supported, a bit later sending via +TCP, too. Starting with 1.11.0, RFC 3195 is finally supported at the +receiving side (a.k.a. "listener"). Support for sending via RFC 3195 is +still due. Anyhow, rsyslog has come much closer to what it name +promises.</p> <p> The database support was initially included so that our web-based syslog interface could be used. This is another open source project which can be found @@ -50,31 +53,31 @@ the syslogd binary with the one that comes with rsyslog. Of course, in order to use any of the new features, you must re-write your syslog.conf. To learn how to do this, please review our commented <a href="sample.conf.php">sample.conf</a> file. It outlines the enhancements over stock syslogd. Discussion has often -arisen of whether having an "old syslogd" logfile format is good or evil. So +arisen of whether having an "old syslogd" logfile format is good or evil. So far, this has not been solved (but Rainer likes the idea of a new format), so we need to live with it for the time being. It is planned to be reconsidered in the 3.x release time frame. -<p>If you are interested in the <a href="http://en.wikipedia.org/wiki/IHE">IHE</a> +</p><p>If you are interested in the <a href="http://en.wikipedia.org/wiki/IHE">IHE</a> environment, you might be interested to hear that rsyslog supports message with sizes of 32k and more. This feature has been tested, but by default is turned off (as it has some memory footprint that we didn't want to put on users not -actually requiring it). Search the file syslogd.c and search for "IHE" - you +actually requiring it). Search the file syslogd.c and search for "IHE" - you will find easy and precise instructions on what you need to change (it's just one line of code!). Please note that RFC 3195/COOKED supports 1K message sizes only. It'll probably support longer messages in the future, but it is our believe that using larger messages with current RFC 3195 is a violation of the -standard.<p>In <b>February 2007</b>, 1.13.1 was released and served for quite a +standard.</p><p>In <b>February 2007</b>, 1.13.1 was released and served for quite a while as a stable reference. Unfortunately, it was not later released as stable, -so the stable build became quite outdated.<p>In <b>June 2007</b>, Peter Vrabec from Red Hat helped us to create +so the stable build became quite outdated.</p><p>In <b>June 2007</b>, Peter Vrabec from Red Hat helped us to create RPM files for Fedora as well as supporting IPv6. There also seemed to be some interest from the Red Hat community. This interest and new ideas resulted in a -very busy time with many great additions.<p>In <b>July 2007</b>, Andrew +very busy time with many great additions.</p><p>In <b>July 2007</b>, Andrew Pantyukhin added BSD ports files for rsyslog and liblogging. We were strongly encouraged by this too. It looks like rsyslog is getting more and more momentum. -Let's see what comes next...<p>Also in <b>July 2007</b> (and beginning of +Let's see what comes next...</p><p>Also in <b>July 2007</b> (and beginning of August), Rainer remodeled the output part of rsyslog. It got a clean object model and is now prepared for a plug-in architecture. During that time, some base -ideas for the overall new object model appeared.<p>In <b>August 2007</b> +ideas for the overall new object model appeared.</p><p>In <b>August 2007</b> community involvement grew more and more. Also, more packages appeared. We were quite happy about that. To facilitate user contributions, we set up a <a href="http://wiki.rsyslog.com/">wiki</a> on August 10th, 2007. Also in August @@ -82,7 +85,7 @@ quite happy about that. To facilitate user contributions, we set up a 2.0.0 release. With its appearance, the pace of changes was deliberately reduced, in order to allow it to mature (see Rainers's <a href="http://rgerhards.blogspot.com/2007/07/pace-of-changes-in-rsyslog.html"> -blog post</a> on this topic, written a bit early, but covering the essence).<p> +blog post</a> on this topic, written a bit early, but covering the essence).</p><p> In <b>November 2007</b>, rsyslog became the default syslogd in Fedora 8. Obviously, that was something we *really* liked. Community involvement also is still growing. There is one sad thing to note: ever since summer, there is an @@ -90,7 +93,7 @@ extremely hard to find segfault bug. It happens on very rare occasions only and never in lab. We are hunting this bug for month now, but still could not get hold of it. Unfortunately, this also affects the new features schedule. It makes limited sense to implement new features if problems with existing ones are not -really understood.<p><b>December 2007</b> showed the appearance of a postgres +really understood.</p><p><b>December 2007</b> showed the appearance of a postgres output module, contributed by sur5r. With 1.20.0, December is also the first time since the bug hunt that we introduce other new features. It has been decided that we carefully will add features in order to not affect the overall project @@ -98,13 +101,24 @@ by these rare bugs. Still, the bug hunt is top priority, but we need to have mor data to analyze. At then end of December, it looked like the bug was found (a race condition), but further confirmation from the field is required before declaring victory. December also brings the initial development on <b>rsyslog v3</b>, -resulting in loadable input modules, now running on a separate thread each.<p>On +resulting in loadable input modules, now running on a separate thread each.</p><p>On <b>January, 2nd 2008</b>, rsyslog 1.21.2 is re-released as rsyslog v2.0.0 stable. This is a major milestone as far as the stable build is concerned. v3 is not yet officially announced. Other than the stable v2 build, v3 will not be backwards compatibile (including missing compatibility to stock sysklogd) for quite a while. Config file changes are required and some command line options do -no longer work due to the new design.<p>Be sure to visit Rainer's <a href="http://rgerhards.blogspot.com/">syslog blog</a> +no longer work due to the new design.</p><p>On <span style="font-weight: bold;">January, 31st 2008</span> +the new massively-multithreaded queue engine was released for the first +time. It was a major milestone, implementing a feature I dreamed of for +more than a year.</p><p>End of <span style="font-weight: bold;">February 2008</span> +saw the first note about RainerScript, a way to configure rsyslogd via +a script-language like configuration format. This effort evolved out of +the need to have complex expression support, which was also the first +use case. On February, 28th rsyslog 3.12.0 was released, the first +version to contain expression support. This also meant that rsyslog +from that date on supported all syslog-ng major features, but had a +number of major features exlusive to it. With 3.12.0, I consider +rsyslog fully superior to syslog-ng (except for platform support).</p><p>Be sure to visit Rainer's <a href="http://rgerhards.blogspot.com/">syslog blog</a> to get some more insight into the development and futures of rsyslog and syslog in general. Don't be shy to post to either the blog or the <a href="http://www.rsyslog.com/PNphpBB2.phtml">rsyslog forums</a>.</p> @@ -112,5 +126,4 @@ Don't be shy to post to either the blog or the <ul> <li><a href="http://www.rsyslog.com/Topic4.phtml">the rsyslog change log</a></li> </ul> -</body> -</html> +</body></html>
\ No newline at end of file diff --git a/doc/imfile.html b/doc/imfile.html new file mode 100644 index 0000000..d8a105b --- /dev/null +++ b/doc/imfile.html @@ -0,0 +1,134 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>Text File Input Monitor</title></head> +<body> +<h1>Text File Input Module</h1> +<p><b>Module Name: imfile</b></p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>Provides the ability to convert any standard text file into +a syslog message. A standard +text file is a file consisting of printable characters with lines +being delimited by LF.</p> +<p>The file is read line-by-line and any line read is passed to +rsyslog's rule engine. The rule engine applies filter conditons and +selects which actions needs to be carried out.</p> +<p>As new lines are written they are taken from the file and +processed. Please note that this happens based on a polling interval +and not immediately. The file monitor support file rotation. To fully +work, rsyslogd must run while the file is rotated. Then, any remaining +lines from the old file are read and processed and when done with that, +the new file is being processed from the beginning. If rsyslogd is +stopped during rotation, the new file is read, but any not-yet-reported +lines from the previous file can no longer be obtained.</p> +<p>When rsyslogd is stopped while monitoring a text file, it +records the last processed location and continues to work from there +upon restart. So no data is lost during a restart (except, as noted +above, if the file is rotated just in this very moment).</p> +<p>Currently, the file must have a fixed name and location +(directory). It is planned to add support for dynamically generating +file names in the future.</p> +<p>Multiple files may be monitored by specifying +$InputRunFileMonitor multiple times. +</p> +<p><b>Configuration Directives</b>:</p> +<ul> +<li><strong>$InputFileName /path/to/file</strong><br> +The file being monitored. So far, this must be an absolute name (no +macros or templates)</li> +<li><span style="font-weight: bold;">$InputFileTag +tag:</span><br> +The tag to be used for messages that originate from this file. If you +would like to see the colon after the tag, you need to specify it here +(as shown above).</li> +<li><span style="font-weight: bold;">$InputFileStateFile +<name-of-state-file></span><br> +Rsyslog must keep track of which parts of the to be monitored file it +already processed. This is done in the state file. This file always is +created in the rsyslog working directory (configurable via +$WorkDirectory). Be careful to use unique names for different files +being monitored. If there are duplicates, all sorts of "interesting" +things may happen. Rsyslog currently does not check if a name is +specified multiple times.</li> +<li><span style="font-weight: bold;">$InputFileFacility +facility</span><br> +The syslog facility to be assigned to lines read. Can be specified in +textual form (e.g. "local0", "local1", ...) or as numbers (e.g. 128 for +"local0"). Textual form is suggested. <span style="font-weight: bold;">Default</span> is +"local0".<span style="font-weight: bold;"></span></li> +<li><span style="font-weight: bold;">$InputFileSeverity</span><br> +The +syslog severity to be assigned to lines read. Can be specified in +textual form (e.g. "info", "warning", ...) or as numbers (e.g. 4 for +"info"). Textual form is suggested. <span style="font-weight: bold;">Default</span> +is "notice".</li> +<li><span style="font-weight: bold;">$InputRunFileMonitor</span><br> +This <span style="font-weight: bold;">activates</span> +the current monitor. It has no parameters. If you forget this +directive, no file monitoring will take place.</li> +<li><span style="font-weight: bold;">$InputFilePollInterval +seconds</span><br> +This is a global setting. It specifies how often files are to be polled +for new data. The time specified is in seconds. The <span style="font-weight: bold;">default value</span> is 10 +seconds. Please note that future +releases of imfile may support per-file polling intervals, but +currently this is not the case. If multiple $InputFilePollInterval +statements are present in rsyslog.conf, only the last one is used.<br> +A short poll interval provides more rapid message forwarding, but +requires more system ressources. While it is possible, we stongly +recommend not to set the polling interval to 0 seconds. That will make +rsyslogd become a CPU hog, taking up considerable ressources. It is +supported, however, for the few very unusual situations where this +level may be needed. Even if you need quick response, 1 seconds should +be well enough. Please note that imfile keeps reading files as long as +there is any data in them. So a "polling sleep" will only happen when +nothing is left to be processed.</li> +</ul> +<b>Caveats/Known Bugs:</b> +<p>So far, only 100 files can be monitored. If more are needed, +the source needs to be patched. See define MAX_INPUT_FILES in imfile.c</p><p>Powertop +users may want to notice that imfile utilizes polling. Thus, it is no +good citizen when it comes to conserving system power consumption. We +are currently evaluating to move to inotify(). However, there are a +number of subtle issues, which needs to be worked out first. We will +make the change as soon as we can. If you can afford it, we recommend +using a long polling interval in the mean time. +</p> +<p><b>Sample:</b></p> +<p>The following sample monitors two files. If you need just one, +remove the second one. If you need more, add them according to the +sample ;). This code must be placed in /etc/rsyslog.conf (or wherever +your distro puts rsyslog's config files). Note that only commands +actually needed need to be specified. The second file uses less +commands and uses defaults instead.<br> +</p> +<textarea rows="15" cols="60">$ModLoad imfile.so # +needs to be done just once +# File 1 +$InputFileName /path/to/file1 +$InputFileTag tag1: +$InputFileStateFile stat-file1 +$InputFileSeverity error +$InputFileFacility local7 +$InputRunFileMonitor +# File 2 +$InputFileName /path/to/file2 +$InputFileTag tag2: +$InputFileStateFile stat-file2 +$InputRunFileMonitor +# ... and so on ... +# +# check for new lines every 10 seconds +$InputFilePollingInterval 10 +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html>
\ No newline at end of file diff --git a/doc/imgssapi.html b/doc/imgssapi.html new file mode 100644 index 0000000..af15e59 --- /dev/null +++ b/doc/imgssapi.html @@ -0,0 +1,50 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>GSSAPI Syslog Input Module</title> + +</head> +<body> +<h1>GSSAPI Syslog Input Module</h1> +<p><b>Module Name: imgssapi</b></p> +<p><b>Author: </b>varmojfekoj</p> +<p><b>Description</b>:</p> +<p>Provides the ability to receive syslog messages from the +network protected via Kerberos 5 encryption and authentication. This +module also accept plain tcp syslog messages on the same port if configured to do so. If you need just plain tcp, use <a href="imtcp.html">imtcp</a> instead.</p> +<p><b>Configuration Directives</b>:</p> +<ul> +<li>InputGSSServerRun <port><br> +Starts a GSSAPI server on selected port - note that this runs +independently from the TCP server.</li> +<li>InputGSSServerServiceName <name><br> +The service name to use for the GSS server.</li> +<li>$InputGSSServerPermitPlainTCP on|<span style="font-weight: bold;">off</span><br> +Permits the server to receive plain tcp syslog (without GSS) on the +same port</li> +<li>$InputGSSServerMaxSessions <number><br> +Sets the maximum number of sessions supported</li> +</ul> +<b>Caveats/Known Bugs:</b> +<ul> +<li>module always binds to all interfaces</li> +<li>only a single listener can be bound</li> + +</ul> +<p><b>Sample:</b></p> +<p>This sets up a GSS server on port 1514 that also permits to +receive plain tcp syslog messages (on the same port):<br> +</p> +<textarea rows="15" cols="60">$ModLoad imtcp.so # needs to be done just once +$InputGSSServerRun 1514 +$InputGSSServerPermitPlainTCP on +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html>
\ No newline at end of file diff --git a/doc/imtcp.html b/doc/imtcp.html new file mode 100644 index 0000000..b2c6d21 --- /dev/null +++ b/doc/imtcp.html @@ -0,0 +1,51 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>TCP Syslog Input Module</title> + +</head> +<body> +<h1>TCP Syslog Input Module</h1> +<p><b>Module Name: imtcp</b></p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>Provides the ability to receive syslog messages via TCP. +Encryption can be provided by using <a href="rsyslog_stunnel.html">stunnel</a> +(an alternative is the use +the <a href="imgssapi.html">imgssapi</a> +modul).</p> +<p>In the future, multiple receivers may be configured by +specifying +$InputTCPServerRun multiple times. This is not currently supported. +</p> +<p><b>Configuration Directives</b>:</p> +<ul> +<li>$InputTCPServerRun <port><br> +Starts a TCP server on selected port</li> +<li>$InputTCPMaxSessions <number><br> +Sets the maximum number of sessions supported</li> +</ul> +<b>Caveats/Known Bugs:</b> +<ul> +<li>module always binds to all interfaces</li> +<li>only a single listener can be bound</li> +<li>can not be loaded together with <a href="imgssapi.html">imgssapi</a> +(which includes the functionality of imtcp)</li> +</ul> +<p><b>Sample:</b></p> +<p>This sets up a TCP server on port 514:<br> +</p> +<textarea rows="15" cols="60">$ModLoad imtcp.so # +needs to be done just once +$InputTCPServerRun 514 +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html>
\ No newline at end of file diff --git a/doc/install.html b/doc/install.html index bee136c..711552e 100644 --- a/doc/install.html +++ b/doc/install.html @@ -1,5 +1,5 @@ <html><head> -<title>SSL Encrypting syslog with stunnel</title> +<title>A guide on HOWTO install rsyslog</title> <meta name="KEYWORDS" content="syslog encryption, rsyslog, stunnel, secure syslog, tcp, reliable, howto, ssl"> </head> <body> @@ -53,7 +53,18 @@ the rsyslogd and the man pages to the relevant directories.</p> <p>In this step, you tell rsyslogd what to do with received messages. If you are upgrading from stock syslogd, /etc/syslog.conf is probably a good starting point. Rsyslogd understands stock syslogd syntax, so you can simply copy over -/etc/syslog.conf to /etc/rsyslog.conf. Then, edit rsyslog.conf for any +/etc/syslog.conf to /etc/rsyslog.conf. Note since version 3 rsyslog requires +to load plug-in modules to perform useful work (more about +<a href="v3compatibility.html">compatibilty notes v3</a>). To load the most common plug-ins, +add the following to the top of rsyslog.conf:</p> +<p> +$ModLoad immark.so # provides --MARK-- message capability <br /> +$ModLoad imudp.so # provides UDP syslog reception <br /> +$ModLoad imtcp.so # provides TCP syslog reception and GSS-API (if compiled to support it) <br /> +$ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) <br /> +$ModLoad imklog.so # provides kernel logging support (previously done by rklogd) <br /> +</p> +Change rsyslog.conf for any further enhancements you would like to see. For example, you can add database writing as outlined in the paper "<a href="rsyslog_mysql.html">Writing syslog Data to MySQL</a>" (remember you need to enable MySQL support during step 2 if you want to do diff --git a/doc/log_rotation_fix_size.html b/doc/log_rotation_fix_size.html new file mode 100644 index 0000000..0b9a3b2 --- /dev/null +++ b/doc/log_rotation_fix_size.html @@ -0,0 +1,59 @@ +<html><head> +<title>Keep the log file size accurate with log rotation</title> +<meta name="KEYWORDS" content="log rotation, howto, guide, fixed-size log"> +</head> +<body> +<h1>Log rotation with rsyslog</h1> + <P><small><i>Written by + Michael Meckelein</i></small></P> +<h2>Situation</h2> + +<p>Your environment does not allow you to store tons of logs? +You have limited disc space available for logging, for example +you want to log to a 124 MB RAM usb stick? Or you do not want to +keep all the logs for months, logs from the last days is sufficient? +Think about log rotation.</p> + +<h2>Log rotation based on a fixed log size</h2> + +<p>This small but hopefully useful article will show you the way +to keep your logs at a given size. The following sample is based on +rsyslog illustrating a simple but effective log rotation with a +maximum size condition.</p> + +<h2>Use Output Channels for fixed-length syslog files</h2> + +<p>Lets assume you do not want to spend more than 100 MB hard +disc space for you logs. With rsyslog you can configure Output +Channels to achieve this. Putting the following directive</p> + +<p><pre> +# start log rotation via outchannel +# outchannel definiation +$outchannel log_rotation,/var/log/log_rotation.log, 52428800,/home/me/./log_rotation_script +# activate the channel and log everything to it +*.* $log_rotation +# end log rotation via outchannel +</pre></p> + +<p>to ryslog.conf instruct rsyslog to log everything to the destination file +'/var/log/log_rotation.log' until the give file size of 50 MB is reached. If +the max file size is reached it will perform an action. In our case it executes +the script /home/me/log_rotation_script which contains a single command:</p> + +<p><pre> +mv -f /var/log/log_rotation.log /var/log/log_rotation.log.1 +</p></pre> + +<p>This moves the original log to a kind of backup log file. +After the action was successfully performed rsyslog creates a new /var/log/log_rotation.log +file and fill it up with new logs. So the latest logs are always in log_roatation.log.</p> + +<h2>Conclusion</h2> + +<p>With this approach two files for logging are used, each with a maximum size of 50 MB. So +we can say we have successfully configured a log rotation which satisfies our requirement. +We keep the logs at a fixed-size level of100 MB.</p> + +</body> +</html> diff --git a/doc/manual.html b/doc/manual.html index d4e0015..135d2fa 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -1,118 +1,103 @@ -<html> - -<head> - -<title>rsyslog documentation</title> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>rsyslog documentation</title> </head> - <body> - <h1>RSyslog - Documentation</h1> - -<p><b><a href="http://www.rsyslog.com/">Rsyslog</a> is an enhanced syslogd - -supporting, among others, <a href="rsyslog_mysql.html">MySQL</a>, -PostgreSQL, -<a href="http://wiki.rsyslog.com/index.php/FailoverSyslogServer">failover log -destinations</a>, syslog/tcp, - -fine grain output format control, and the ability to filter on any message part.</b> - -It is quite compatible to stock - -sysklogd and can be used as a drop-in replacement. Its <a href="features.html"> - -advanced features</a> make it suitable for enterprise-class, - -<a href="rsyslog_stunnel.html">encryption protected syslog</a> - -relay chains while at the same time being very easy to setup - -for the novice user. And as we know what enterprise users really need, there is also <a href="professional_support.html">professional rsyslog support</a> available directly from the source!</p> - -<p><b>Visit the <i> <a href="status.html">rsyslog status page</a></i></b> to obtain current - -version information and ports. <b>If you like rsyslog, you might want to lend us - -a helping hand. </b>It doesn't require a lot of time - even a single mouse click - -helps. Learn <a href="how2help.html">how to help the rsyslog project</a>.</p> - -<p><b>Follow the links below for the</b></p> - -<ul> - -<li><a href="man_rsyslogd.html">rsyslogd man page</a> - -<li><a href="rsyslog_conf.html">configuration file syntax (rsyslog.conf)</a><li> -<a href="property_replacer.html">property replacer, an important core component</a><li>a commented <a href="sample.conf.html">sample rsyslog.conf</a> - -<li><a href="bugs.html">rsyslog bug list</a><li><a href="rsyslog_packages.html"> -rsyslog packages</a><li><a href="generic_design.html">backgrounder on generic -syslog application design</a><!-- re-enable when updated <li><a href="contributors.html">contributor "Hall -of Fame"</a> --><li><a href="modules.html">description of rsyslog modules</a></ul> - +<p><b><a href="http://www.rsyslog.com/">Rsyslog</a> +is an enhanced syslogd +supporting, among others, <a href="rsyslog_mysql.html">MySQL</a>, +PostgreSQL, <a href="http://wiki.rsyslog.com/index.php/FailoverSyslogServer">failover +log destinations</a>, syslog/tcp, fine grain output format +control, high precision timestamps, queued operations and the ability to filter on any message +part.</b> +It is quite compatible to stock sysklogd and can be used as a drop-in +replacement. Its <a href="features.html"> +advanced features</a> make it suitable for enterprise-class, <a href="rsyslog_stunnel.html">encryption protected syslog</a> +relay chains while at the same time being very easy to setup for the +novice user. And as we know what enterprise users really need, there is +also <a href="professional_support.html">professional +rsyslog support</a> available directly from the source!</p> +<p><b>Visit the <i> <a href="status.html">rsyslog +status page</a></i></b> to obtain current +version information and ports. <b>If you like rsyslog, you might +want to lend us a helping hand. </b>It doesn't require a lot of +time - even a single mouse click helps. Learn <a href="how2help.html">how to help the rsyslog project</a>. +Due to popular demand, there is now a <a href="rsyslog_ng_comparison.html">side-by-side comparison +between rsyslog and syslog-ng</a>.</p> +<p>If you are upgrading from rsyslog v2 or stock sysklogd, +<a href="v3compatibility.html">be +sure to read the rsyslog v3 compatibility document!</a> It will work even +if you do not read the doc, but doing so will definitely improve your experience.</p> +<p><span style="font-weight: bold;"></span><b>Follow +the links below for the</b><br></p><ul> + +<li><a href="rsyslog_conf.html">configuration file +syntax (rsyslog.conf)</a></li> +<li> <a href="property_replacer.html">property +replacer, an important core component</a></li> +<li>a commented <a href="sample.conf.html">sample +rsyslog.conf</a> +</li> +<li><a href="bugs.html">rsyslog bug list</a></li> +<li><a href="rsyslog_packages.html"> rsyslog +packages</a></li> +<li><a href="generic_design.html">backgrounder on +generic syslog application design</a><!-- not good as it currently is ;) <li><a href="contributors.html">contributor "Hall of Fame"</a>--></li> +<li><a href="modules.html">description of rsyslog +modules</a></li><li><a href="man_rsyslogd.html">rsyslogd man page</a> +(heavily outdated)</li> +</ul> <p><b>We have some in-depth papers on</b></p> - <ul> - - <li><a href="install.html">installing rsyslog</a></li> - <li><a href="ipv6.html">rsyslog and IPv6</a> (which is fully supported)</li> - - <li><a href="rsyslog_stunnel.html">ssl-encrypting syslog with stunnel</a></li> - - <li><a href="rsyslog_mysql.html">writing syslog messages to MySQL</a></li> - - <li><a href="rsyslog_php_syslog_ng.html">using php-syslog-ng with rsyslog</a></li> - <li><a href="rsyslog_recording_pri.html">recording the syslog priority - (severity and facility) to the log file</a></li> - <li><a href="http://www.rsyslog.com/Article19.phtml">preserving syslog - sender over NAT</a> (online only)</li> - +<li><a href="install.html">installing rsyslog</a></li> +<li><a href="ipv6.html">rsyslog and IPv6</a> +(which is fully supported)</li> +<li><a href="rsyslog_stunnel.html">ssl-encrypting +syslog with stunnel</a></li> +<li><a href="rsyslog_mysql.html">writing syslog +messages to MySQL (and other databases as well)</a></li> +<li><a href="rsyslog_high_database_rate.html">writing +massive amounts of syslog messages to a database</a></li> +<li><a href="rsyslog_php_syslog_ng.html">using +php-syslog-ng with rsyslog</a></li> +<li><a href="rsyslog_recording_pri.html">recording +the syslog priority (severity and facility) to the log file</a></li> +<li><a href="http://www.rsyslog.com/Article19.phtml">preserving +syslog sender over NAT</a> (online only)</li> +<li><a href="debug.html">debug support in rsyslog</a></li> +<li><a href="dev_queue.html">the rsyslog message +queue object</a></li> </ul> - -<p>Our <a href="history.html">rsyslog history</a> page is for you if you would like to learn a little more - -on why there is an rsyslog at all. If you are interested why you should care -about rsyslog at all, you may want to read Rainer's essay on "<a href="http://rgerhards.blogspot.com/2007/08/why-does-world-need-another-syslogd.html">why -the world needs another syslogd</a>".</p> - -<p>Documentation is added continuously. Please note that the documentation here - -matches only the current version of rsyslog. If you use an older version, be sure - +<p>Our <a href="history.html">rsyslog history</a> +page is for you if you would like to learn a little more +on why there is an rsyslog at all. If you are interested why you should +care about rsyslog at all, you may want to read Rainer's essay on "<a href="http://rgerhards.blogspot.com/2007/08/why-does-world-need-another-syslogd.html">why +the world needs another syslogd</a>".</p> +<p>Documentation is added continuously. Please note that the +documentation here +matches only the current version of rsyslog. If you use an older +version, be sure to use the doc that came with it.</p> - <p><b>You can also browse the following online resources:</b></p> - <ul> - -<li>the <a href="http://wiki.rsyslog.com/">rsyslog wiki</a>, a community -resource</li> - -<li><a href="http://www.rsyslog.com/module-Static_Docs-view-f-manual.html.phtml">rsyslog online documentation</a></li> - -<li><a href="http://www.rsyslog.com/Topic3.phtml">rsyslog FAQ</a></li> - -<li><a href="http://www.rsyslog.com/PNphpBB2.phtml">rsyslog discussion forum</a></li> - -<li><a href="http://www.rsyslog.com/Topic4.phtml">rsyslog change log</a></li> - -<li><a href="http://www.monitorware.com/en/syslog-enabled-products/">syslog device configuration guide</a> (off-site)</li> - +<li>the <a href="http://wiki.rsyslog.com/">rsyslog +wiki</a>, a community resource which includes <a href="http://wiki.rsyslog.com/index.php/Configuration_Samples">rsyslog configuration examples</a></li> +<li><a href="http://www.rsyslog.com/module-Static_Docs-view-f-manual.html.phtml">rsyslog +online documentation (most current version only)</a></li> + +<li><a href="http://www.rsyslog.com/PNphpBB2.phtml">rsyslog +discussion forum - use this for technical support</a></li> +<li><a href="http://www.rsyslog.com/Topic4.phtml">rsyslog +change log</a></li> +<li><a href="http://www.rsyslog.com/Topic3.phtml">rsyslog +FAQ</a></li><li><a href="http://www.monitorware.com/en/syslog-enabled-products/">syslog +device configuration guide</a> (off-site)</li> </ul> - -<p>And don't forget about the <a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">rsyslog mailing list</a>. - -If you are interested in the "backstage", you may find - +<p>And don't forget about the <a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">rsyslog +mailing list</a>. If you are interested in the "backstage", you +may find <a href="http://www.gerhards.net/rainer">Rainer</a>'s - -<a href="http://rgerhards.blogspot.com/">blog</a> an interesting read (filter on -syslog and rsyslog tags).</p> - -</body> - -</html> - +<a href="http://rgerhards.blogspot.com/">blog</a> an +interesting read (filter on syslog and rsyslog tags).</p> +</body></html>
\ No newline at end of file diff --git a/doc/omlibdbi.html b/doc/omlibdbi.html new file mode 100644 index 0000000..c66dc06 --- /dev/null +++ b/doc/omlibdbi.html @@ -0,0 +1,126 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>Generic Database Output Module (omlibdbi)</title> + +</head> +<body> +<h1>Generic Database Output Module (omlibdbi)</h1> +<p><b>Module Name: omlibdbi</b></p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>This modules supports a large number of database systems via <a href="http://libdbi.sourceforge.net/">libdbi</a>. +Libdbi abstracts the database layer and provides drivers for many +systems. Drivers are available via the <a href="http://libdbi-drivers.sourceforge.net/">libdbi-drivers</a> +project. As of this writing, the following drivers are available:</p> +<ul> +<li><a href="http://www.firebird.sourceforge.net/">Firebird/Interbase</a></li> +<li><a href="http://www.freetds.org/">FreeTDS</a> +(provides access to <a href="http://www.microsoft.com/sql">MS +SQL Server</a> and <a href="http://www.sybase.com/products/informationmanagement/adaptiveserverenterprise">Sybase</a>)</li> +<li><a href="http://www.mysql.com/">MySQL</a> +(also +supported via the native ommysql plugin in rsyslog)</li> +<li><a href="http://www.postgresql.org/">PostgreSQL</a>(also +supported via the native +ommysql plugin in rsyslog)</li> +<li><a href="http://www.sqlite.org/">SQLite/SQLite3</a></li> +</ul> +<p>The following drivers are in various stages of completion:</p> +<ul> +<li><a href="http://ingres.com/">Ingres</a></li> +<li><a href="http://www.hughes.com.au/">mSQL</a></li> +<li><a href="http://www.oracle.com/">Oracle</a></li> +</ul> +<p>These drivers seem to be quite usable, at +least from an rsyslog point of view.</p> +<p>Libdbi provides a slim layer between rsyslog and the actual +database engine. We have not yet done any performance testing (e.g. +omlibdbi vs. ommysql) but honestly believe that the performance impact +should be irrelevant, if at all measurable. Part of that assumption is +that rsyslog just does the "insert" and most of the time is spent +either in the database engine or rsyslog itself. It's hard to think of +any considerable time spent in the libdbi abstraction layer.</p> +<p><span style="font-weight: bold;">Setup</span></p> +<p>In order for this plugin to work, you need to have libdbi, the +libdbi driver for your database backend and the client software for +your database backend installed. There are libdbi packages for many +distributions. Please note that rsyslogd requires a quite recent +version (0.8.3) of libdbi. It may work with older versions, but these +need some special ./configure options to support being called from a +dlopen()ed plugin (as omlibdbi is). So in short, you probably save you +a lot of headache if you make sure you have at least libdbi version +0.8.3 on your system. +</p> +<p><b>Configuration Directives</b>:</p> +<ul> +<li><span style="font-weight: bold;">$ActionLibdbiDriverDirectory /path/to/dbd/drivers</span><br>This +is a global setting. It points libdbi to its driver directory. Usually, +you do not need to set it. If you installed libdbi-driver's at a +non-standard location, you may need to specify the directory here. If +you are unsure, do <span style="font-weight: bold;">not</span> use this configuration directive. Usually, everything works just fine.<strong></strong></li><li><strong>$ActionLibdbiDriver drivername</strong><br> +Name of the dbidriver to use, see libdbi-drivers documentation. As a +quick excerpt, at least those were available at the time of this +writiting "mysql" (suggest to use ommysql instead), "firebird" (Firbird +and InterBase), "ingres", "msql", "Oracle", "sqlite", "sqlite3", +"freetds" (for Microsoft SQL and Sybase) and "pgsql" (suggest to use +ompgsql instead).</li> +<li><span style="font-weight: bold;">$ActionLibdbiHost +hostname</span><br> +The host to connect to.</li> +<li><span style="font-weight: bold;">$ActionLibdbiUserName +user</span><br> +The user used to connect to the database.</li> +<li><span style="font-weight: bold;">$ActionlibdbiPassword</span><br> +That user's password.</li> +<li><span style="font-weight: bold;">$ActionlibdbiDBName +db</span><br> +The database that shall be written to.</li> +<li><span style="font-weight: bold;">selector +line: :omlibdbi:<span style="font-style: italic;">;template</span></span><br> +executes the recently configured omlibdbi action. The ;template part is +optional. If no template is provided, a default template is used (which +is currently optimized for MySQL - sorry, folks...)</li> +</ul> +<b>Caveats/Known Bugs:</b> +<p>You must make sure that any templates used for omlibdbi +properly escape strings. This is usually done by supplying the SQL (or +STDSQL) option to the template. Omlibdbi rejects templates without this +option for security reasons. However, omlibdbi does not detect if you +used the right option for your backend. Future versions of rsyslog +(with full expression support) will provide advanced +ways of handling this situation. So far, you must be careful. The +default template provided by rsyslog is suitable for MySQL, but not +necessarily for your database backend. Be careful!</p> +<p>If you receive the rsyslog error message "libdbi or libdbi +drivers not present on this system" you may either not have libdbi and +its drivers installed or (very probably) the version is earlier than +0.8.3. In this case, you need to make sure you have at least 0.8.3 and +the libdbi driver for your database backend present on your system.</p><p>I +do not have most of the database supported by omlibdbi in my lab. So it +received limited cross-platform tests. If you run into troubles, be +sure the let us know at <a href="http://www.rsyslog.com">http://www.rsyslog.com</a>.</p> +<p><b>Sample:</b></p> +<p>The following sample writes all syslog messages to the +database "syslog_db" on mysqlsever.example.com. The server is MySQL and +being accessed under the account of "user" with password "pwd" (if you +have empty passwords, just remove the $ActionLibdbiPassword line).<br> +</p> +<textarea rows="15" cols="60">$ModLoad omlibdbi.so +$ActionLibdbiDriver mysql +$ActionLibdbiHost mysqlserver.example.com +$ActionLibdbiUserName user +$ActionLibdbiPassword pwd +$ActionLibdbiDBName syslog_db +*.* :omlibdbi: +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html>
\ No newline at end of file diff --git a/doc/omsnmp.html b/doc/omsnmp.html new file mode 100644 index 0000000..5b10bd9 --- /dev/null +++ b/doc/omsnmp.html @@ -0,0 +1,174 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>SNMP Output Module</title></head> + +<body> + +<h1>SNMP Output Module</h1> +<p><b>Module Name: omsnmp</b></p> +<p><b>Author: Andre Lorbach <alorbach@adiscon.com></b></p> +<p><b>Description</b>:</p> +<p>Provides the ability to send syslog messages as an SNMPv1 & v2c traps. By +default, SNMPv2c is preferred. The syslog message is wrapped into a OCTED +STRING variable. This module uses the <a target="_blank" href="http://net-snmp.sourceforge.net/"> +NET-SNMP</a> library. In order to compile this module, you will need to have the +<a target="_blank" href="http://net-snmp.sourceforge.net/">NET-SNMP</a> +developer (headers) package installed. </p> +<p> </p> +<p><b>Action Line:</b></p> +<p>%omsnmp% without any further parameters.</p> +<p> </p> +<p><b>Configuration Directives</b>:</p> +<ul> + <li><strong>$actionsnmptransport </strong>(This parameter is optional, the + default value is "udp")<br> + <br> + Defines the transport type you wish to use. Technically we can support all + transport types which are supported by NET-SNMP. <br> + To name a few possible values: <br> + <br> + udp, tcp, udp6, tcp6, icmp, icmp6 ...<br> + <br> + Example: <strong>$actionsnmptransport udp<br> + </strong></li> + <li><strong>$actionsnmptarget</strong><br> + <br> + This can be a hostname or ip address, and is our snmp target host. This + parameter is required, if the snmptarget is not defined, nothing will be + send. <br> + <br> + Example: <strong>$actionsnmptarget server.domain.xxx</strong><br> + </li> + <li><strong>$actionsnmptargetport </strong>(This parameter is optional, the + default value is "162")<br> + <br> + The port which will be used, common values are port 162 or 161. <br> + <br> + Example: <strong>$actionsnmptargetport 162</strong><br> + </li> + <li><strong>$actionsnmpversion </strong>(This parameter is optional, the + default value is "1")<br> + <br> + There can only be two choices for this parameter for now. <br> + 0 means SNMPv1 will be used.<br> + 1 means SNMPv2c will be used. <br> + Any other value will default to 1. <br> + <br> + Example: <strong>$actionsnmpversion 1</strong><br> + </li> + <li><strong>$actionsnmpcommunity </strong>(This parameter is optional, the + default value is "public")<br> + <br> + This sets the used SNMP Community.<br> + <br> + Example:<strong> $actionsnmpcommunity public<br> + </strong><br> + </li> + <li><strong>$actionsnmptrapoid </strong>(This parameter is + optional, the default value is "1.3.6.1.4.1.19406.1.2.1" which means + "ADISCON-MONITORWARE-MIB::syslogtrap")<br> + This configuration parameter is used for <strong>SNMPv2</strong> only.<br> + <br> + This is the OID which defines the trap-type, or notifcation-type rsyslog + uses to send the trap. <br> + In order to decode this OID, you will need to have the + ADISCON-MONITORWARE-MIB and ADISCON-MIB mibs installed on the receiver side. Downloads of these mib files + can be found here: <br> + <a href="http://www.adiscon.org/download/ADISCON-MIB.txt"> + http://www.adiscon.org/download/ADISCON-MIB.txt</a><br> + <a href="http://www.adiscon.org/download/ADISCON-MONITORWARE-MIB.txt"> + http://www.adiscon.org/download/ADISCON-MONITORWARE-MIB.txt</a><br> + <br> + Thanks to the net-snmp + mailinglist for the help and the recommendations ;).<br> + <br> + Example: <strong>$actionsnmptrapoid 1.3.6.1.4.1.19406.1.2.1<br> + </strong>If you have this MIBS installed, you can also configured with the + OID Name: <strong>$actionsnmptrapoid ADISCON-MONITORWARE-MIB::syslogtrap<br> + </strong> + </li> + <li><strong>$actionsnmpsyslogmessageoid </strong>(This parameter is + optional, the default value is "1.3.6.1.4.1.19406.1.1.2.1" which means + "ADISCON-MONITORWARE-MIB::syslogMsg")<br> + <br> + This OID will be used as a variable, type "OCTET STRING". This variable will + contain up to 255 characters of the original syslog message including syslog header. It is recommend to + use the default OID. <br> + In order to decode this OID, you will need to have the + ADISCON-MONITORWARE-MIB and ADISCON-MIB mibs installed on the receiver side. + To download these custom mibs, see the description of <strong>$actionsnmptrapoid. + </strong><br> + <br> + Example: <strong>$actionsnmpsyslogmessageoid 1.3.6.1.4.1.19406.1.1.2.1<br> + </strong>If you have this MIBS installed, you can also configured with the + OID Name: <strong>$actionsnmpsyslogmessageoid + ADISCON-MONITORWARE-MIB::syslogMsg<br> + </strong><br> + </li> + <li><strong>$actionsnmpenterpriseoid </strong>(This parameter is optional, + the default value is "1.3.6.1.4.1.3.1.1" which means "enterprises.cmu.1.1")<br> + <br> + Customize this value if needed. I recommend to use the default value unless + you require to use a different OID. <br> + This configuration parameter is used for <strong>SNMPv1</strong> only. It + has no effect if <strong>SNMPv2</strong> is used. <br> + <br> + Example: <strong>$actionsnmpenterpriseoid 1.3.6.1.4.1.3.1.1 <br> + </strong><br> + </li> + <li><strong>$actionsnmpspecifictype </strong>(This parameter is optional, + the default value is "0")<strong> </strong><br> + <br> + This is the specific trap number. This configuration parameter is used for + <strong>SNMPv1</strong> only. It has no effect if <strong>SNMPv2</strong> is + used. <br> + <br> + Example: <strong>$actionsnmpspecifictype 0<br> + </strong><br> + </li> + <li><strong>$actionsnmptraptype</strong> (This parameter is optional, the + default value is "6" which means SNMP_TRAP_ENTERPRISESPECIFIC) <br> + <br> + There are only 7 Possible trap types defined which can be used here. These + trap types are: <br> + 0 = SNMP_TRAP_COLDSTART<br> + 1 = SNMP_TRAP_WARMSTART<br> + 2 = SNMP_TRAP_LINKDOWN<br> + 3 = SNMP_TRAP_LINKUP<br> + 4 = SNMP_TRAP_AUTHFAIL<br> + 5 = SNMP_TRAP_EGPNEIGHBORLOSS<br> + 6 = SNMP_TRAP_ENTERPRISESPECIFIC<br> + <br> + Any other value will default to 6 automatically. This configuration + parameter is used for <strong>SNMPv1</strong> only. It has no effect if + <strong>SNMPv2</strong> is used. <br> + <br> + Example: <strong>$actionsnmptraptype 6</strong><br> + </li> +</ul> +<p> </p> +<p><b>Caveats/Known Bugs:</b></p><ul><li>In order to decode the custom OIDs, you + will need to have the adiscon mibs installed. </li></ul> +<p><b>Sample:</b></p> +<p>The following commands send every message as a snmp trap.</p> +<textarea rows="10" cols="60">$ModLoad omsnmp.so + +$actionsnmptransport udp +$actionsnmptarget localhost +$actionsnmptargetport 162 +$actionsnmpversion 1 +$actionsnmpcommunity public + +*.* :omsnmp: +</textarea> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html>
\ No newline at end of file diff --git a/doc/professional_support.html b/doc/professional_support.html index 7f5e837..1a9e652 100644 --- a/doc/professional_support.html +++ b/doc/professional_support.html @@ -53,5 +53,5 @@ project.<br> Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. -Released under the GNU GPL version 2 or higher.</font></p> +Released under the GNU GPL version 3 or higher.</font></p> </body></html> diff --git a/doc/property_replacer.html b/doc/property_replacer.html index 8b777a7..3484acf 100644 --- a/doc/property_replacer.html +++ b/doc/property_replacer.html @@ -1,76 +1,163 @@ -<html> -<head> -<title>The Rsyslogd Property Replacer</title> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>The Rsyslogd Property Replacer</title> + </head> <body> <h1>The Property Replacer</h1> -<p><b>The property replacer is a core component in rsyslogd's output system.</b> -A syslog message has a number of well-defined properties (see below). Each of -this properties can be accessed <b>and</b> manipulated by the property replacer. -With it, it is easy to use only part of a property value or manipulate the value, -e.g. by converting all characters to lower case.</p> +<p><b>The property replacer is a core component in +rsyslogd's output system.</b> A syslog message has a number of +well-defined properties (see below). Each of this properties can be +accessed <b>and</b> manipulated by the property replacer. +With it, it is easy to use only part of a property value or manipulate +the value, e.g. by converting all characters to lower case.</p> <h1>Accessing Properties</h1> -<p>Syslog message properties are used inside templates. They are accessed by putting them between percent signs. Properties can be modified by -the property replacer. The full syntax is as follows:</p> +<p>Syslog message properties are used inside templates. They are +accessed by putting them between percent signs. Properties can be +modified by the property replacer. The full syntax is as follows:</p> <blockquote><b><code>%propname:fromChar:toChar:options%</code></b></blockquote> <h2>Available Properties</h2> -<p><b><code>propname</code></b> is the name of the property to access. It is case-sensitive. +<p><b><code>propname</code></b> is the +name of the property to access. It is case-sensitive. Currently supported are:</p> <table> -<tr><td><b>msg</b></td><td>the MSG part of the message (aka "the message" ;))</td></tr> -<tr><td><b>rawmsg</b></td><td>the message excactly as it was received from the -socket. Should be useful for debugging.</td></tr> -<tr><td><b>UxTradMsg</b></td><td>will disappear soon - do NOT use!</td></tr> -<tr><td><b>HOSTNAME</b></td><td>hostname from the message</td></tr> -<tr><td><b>source</b></td><td>alias for HOSTNAME</td></tr> -<tr><td><b>FROMHOST</b></td><td>hostname of the system the message was received - from (in a relay chain, this is the system immediately in front of us and - not necessarily the original sender)</td></tr> -<tr><td><b>syslogtag</b></td><td>TAG from the message</td></tr> -<tr><td><b>programname</b></td><td>the "static" part of the tag, as defined by -BSD syslogd. For example, when TAG is "named[12345]", programname is "named".</td></tr> -<tr><td><b>PRI</b></td><td>PRI part of the message - undecoded (single value)</td></tr> -<tr><td><b>PRI-text</b></td><td>the PRI part of the message in a textual form - (e.g. "syslog.info")</td></tr> -<tr><td><b>IUT</b></td><td>the monitorware InfoUnitType - used when talking -to a <a href="http://www.monitorware.com">MonitorWare</a> backend (also for - <a href="http://www.phplogcon.org/">phpLogCon</a>)</td></tr> -<tr><td><b>syslogfacility</b></td><td>the facility from the message - in numerical form</td></tr> -<tr> - <td><b>syslogfacility-text</b></td><td>the facility from the message - in - text form</td> -</tr> -<tr> - <td><b>syslogseverity</b></td><td>severity from the - message - in numerical form</td> -</tr> -<tr><td><b>syslogseverity-text</b></td><td>severity from the - message - in text form</td></tr> -<tr><td><b>syslogpriority</b></td><td>an alias for syslogseverity - included for - historical reasons (be careful: it still is the severity, not PRI!)</td></tr> -<tr><td><b>syslogpriority-text</b></td><td>an alias for syslogseverity-text</td></tr> -<tr><td><b>timegenerated</b></td><td>timestamp when the message was RECEIVED. Always in - high resolution</td></tr> -<tr><td><b>timereported</b></td><td>timestamp from the message. Resolution depends on +<tbody> +<tr> +<td><b>msg</b></td> +<td>the MSG part of the message (aka "the message" ;))</td> +</tr> +<tr> +<td><b>rawmsg</b></td> +<td>the message excactly as it was received from the +socket. Should be useful for debugging.</td> +</tr> +<tr> +<td><b>UxTradMsg</b></td> +<td>will disappear soon - do NOT use!</td> +</tr> +<tr> +<td><b>HOSTNAME</b></td> +<td>hostname from the message</td> +</tr> +<tr> +<td><b>source</b></td> +<td>alias for HOSTNAME</td> +</tr> +<tr> +<td><b>FROMHOST</b></td> +<td>hostname of the system the message was received from +(in a relay chain, this is the system immediately in front of us and +not necessarily the original sender)</td> +</tr> +<tr> +<td><b>syslogtag</b></td> +<td>TAG from the message</td> +</tr> +<tr> +<td><b>programname</b></td> +<td>the "static" part of the tag, as defined by +BSD syslogd. For example, when TAG is "named[12345]", programname is +"named".</td> +</tr> +<tr> +<td><b>PRI</b></td> +<td>PRI part of the message - undecoded (single value)</td> +</tr> +<tr> +<td><b>PRI-text</b></td> +<td>the PRI part of the message in a textual form (e.g. +"syslog.info")</td> +</tr> +<tr> +<td><b>IUT</b></td> +<td>the monitorware InfoUnitType - used when talking +to a <a href="http://www.monitorware.com">MonitorWare</a> +backend (also for <a href="http://www.phplogcon.org/">phpLogCon</a>)</td> +</tr> +<tr> +<td><b>syslogfacility</b></td> +<td>the facility from the message - in numerical form</td> +</tr> +<tr> +<td><b>syslogfacility-text</b></td> +<td>the facility from the message - in text form</td> +</tr> +<tr> +<td><b>syslogseverity</b></td> +<td>severity from the message - in numerical form</td> +</tr> +<tr> +<td><b>syslogseverity-text</b></td> +<td>severity from the message - in text form</td> +</tr> +<tr> +<td><b>syslogpriority</b></td> +<td>an alias for syslogseverity - included for historical +reasons (be careful: it still is the severity, not PRI!)</td> +</tr> +<tr> +<td><b>syslogpriority-text</b></td> +<td>an alias for syslogseverity-text</td> +</tr> +<tr> +<td><b>timegenerated</b></td> +<td>timestamp when the message was RECEIVED. Always in high +resolution</td> +</tr> +<tr> +<td><b>timereported</b></td> +<td>timestamp from the message. Resolution depends on what was provided in the message (in most cases, -only seconds)</td></tr> -<tr><td><b>TIMESTAMP</b></td><td>alias for timereported</td></tr> -<tr><td><b>PROTOCOL-VERSION</b></td><td>The contents of the PROTCOL-VERSION - field from IETF draft draft-ietf-syslog-protcol</td></tr> -<tr><td><b>STRUCTURED-DATA</b></td><td>The contents of the STRUCTURED-DATA field - from IETF draft draft-ietf-syslog-protocol</td></tr> -<tr><td><b>APP-NAME</b></td><td>The contents of the APP-NAME field from IETF - draft draft-ietf-syslog-protocol</td></tr> -<tr><td><b>PROCID</b></td><td>The contents of the PROCID field from IETF draft - draft-ietf-syslog-protocol</td></tr> -<tr><td height="24"><b>MSGID</b></td><td height="24">The contents of the MSGID field from IETF draft - draft-ietf-syslog-protocol</td></tr> -<tr><td><b>$NOW</b></td><td>The current date stamp in the format YYYY-MM-DD</td></tr> -<tr><td><b>$YEAR</b></td><td>The current year (4-digit)</td></tr> -<tr><td><b>$MONTH</b></td><td>The current month (2-digit)</td></tr> -<tr><td><b>$DAY</b></td><td>The current day of the month (2-digit)</td></tr> -<tr><td><b>$HOUR</b></td><td>The current hour in military (24 hour) time - (2-digit)</td></tr> +only seconds)</td> +</tr> +<tr> +<td><b>TIMESTAMP</b></td> +<td>alias for timereported</td> +</tr> +<tr> +<td><b>PROTOCOL-VERSION</b></td> +<td>The contents of the PROTCOL-VERSION field from IETF +draft draft-ietf-syslog-protcol</td> +</tr> +<tr> +<td><b>STRUCTURED-DATA</b></td> +<td>The contents of the STRUCTURED-DATA field from IETF +draft draft-ietf-syslog-protocol</td> +</tr> +<tr> +<td><b>APP-NAME</b></td> +<td>The contents of the APP-NAME field from IETF draft +draft-ietf-syslog-protocol</td> +</tr> +<tr> +<td><b>PROCID</b></td> +<td>The contents of the PROCID field from IETF draft +draft-ietf-syslog-protocol</td> +</tr> +<tr> +<td height="24"><b>MSGID</b></td> +<td height="24">The contents of the MSGID field from +IETF draft draft-ietf-syslog-protocol</td> +</tr> +<tr> +<td><b>$NOW</b></td> +<td>The current date stamp in the format YYYY-MM-DD</td> +</tr> +<tr> +<td><b>$YEAR</b></td> +<td>The current year (4-digit)</td> +</tr> +<tr> +<td><b>$MONTH</b></td> +<td>The current month (2-digit)</td> +</tr> +<tr> +<td><b>$DAY</b></td> +<td>The current day of the month (2-digit)</td> +</tr> +<tr> +<td><b>$HOUR</b></td> +<td>The current hour in military (24 hour) time (2-digit)</td> +</tr> <tr> <td><b>$HHOUR</b></td> <td>The current half hour we are in. From minute 0 to 29, @@ -83,93 +170,122 @@ from 30 to 59 it is always 1.</td> range from 0 to 3 (for the four quater hours that are in each hour)</td> </tr> <tr> -<tr><td><b>$MINUTE</b></td><td>The current minute (2-digit)</td></tr> +<td><b>$MINUTE</b></td> +<td>The current minute (2-digit)</td> +</tr> +</tbody> </table> -<p>Properties starting with a $-sign are so-called system properties. These do -NOT stem from the message but are rather internally-generated.</p> +<p>Properties starting with a $-sign are so-called system +properties. These do NOT stem from the message but are rather +internally-generated.</p> <h2>Character Positions</h2> -<p><b><code>FromChar</code></b> and <b><code>toChar</code></b> are used to build substrings. They specify the offset within -the string that should be copied. Offset counting starts at 1, so if you need to -obtain the first 2 characters of the message text, you can use this syntax: -"%msg:1:2%". If you do not whish to specify from and to, but you want to specify -options, you still need to include the colons. For example, if you would like to -convert the full message text to lower case, use "%msg:::lowercase%". -If you would like to extract from a position until the end of the string, you -can place a dollar-sign ("$") in toChar (e.g. %msg:10:$%, which will extract -from position 10 to the end of the string).<p> -There is also support for <b>regular expressions</b>. To use them, you need to -place a "R" into FromChar. This tells rsyslog that a regular expression instead -of position-based extraction is desired. The actual regular expression must then -be provided in toChar. The regular expression <b>must</b> be followed by the -string "--end". It denotes the end of the regular expression and will not become -part of it. If you are using regular expressions, the property replacer will -return the part of the property text that matches the regular expression. An -example for a property replacer sequence with a regular expression is: "%msg:R:.*Sev:. -\(.*\) \[.*--end%"<br> -<p> -<b>Also, extraction can be done based on so-called "fields"</b>. To do so, place -a "F" into FromChar. A field in its current definition is anything -that is delimited by a delimiter character. The delimiter by default is TAB -(US-ASCII value 9). However, if can be changed to any other US-ASCII character -by specifying a comma and the <b>decimal</b> US-ASCII value of the delimiter immediately after the -"F". For example, to use comma (",") as a delimiter, use this field specifier: -"F,44". If your syslog data is delimited, -this is a quicker way to extract than via regular expressions (actually, a *much* -quicker way). Field counting starts at 1. Field zero is accepted, but will -always lead to a "field not found" error. The same happens if a field number -higher than the number of fields in the property is requested. The field number -must be placed in the "ToChar" parameter. An example where the 3rd field -(delimited by TAB) from -the msg property is extracted is as follows: "%msg:F:3%". The same -example with semicolon as delimiter is "%msg:F,59:3%".<p> -Please note that the special characters "F" and "R" are case-sensitive. Only -upper case works, lower case will return an error. There are no white spaces -permitted inside the sequence (that will lead to error messages and will NOT -provide the intended result).<br> +<p><b><code>FromChar</code></b> and <b><code>toChar</code></b> +are used to build substrings. They specify the offset within the string +that should be copied. Offset counting starts at 1, so if you need to +obtain the first 2 characters of the message text, you can use this +syntax: "%msg:1:2%". If you do not whish to specify from and to, but +you want to specify options, you still need to include the colons. For +example, if you would like to convert the full message text to lower +case, use "%msg:::lowercase%". If you would like to extract from a +position until the end of the string, you can place a dollar-sign ("$") +in toChar (e.g. %msg:10:$%, which will extract from position 10 to the +end of the string).</p> +<p>There is also support for <b>regular expressions</b>. +To use them, you need to place a "R" into FromChar. This tells rsyslog +that a regular expression instead of position-based extraction is +desired. The actual regular expression must then be provided in toChar. +The regular expression <b>must</b> be followed by the +string "--end". It denotes the end of the regular expression and will +not become part of it. If you are using regular expressions, the +property replacer will return the part of the property text that +matches the regular expression. An example for a property replacer +sequence with a regular expression is: "%msg:R:.*Sev:. \(.*\) +\[.*--end%"<br> +</p> +<p><b>Also, extraction can be done based on so-called +"fields"</b>. To do so, place a "F" into FromChar. A field in its +current definition is anything that is delimited by a delimiter +character. The delimiter by default is TAB (US-ASCII value 9). However, +if can be changed to any other US-ASCII character by specifying a comma +and the <b>decimal</b> US-ASCII value of the delimiter +immediately after the "F". For example, to use comma (",") as a +delimiter, use this field specifier: "F,44". If your syslog +data is delimited, this is a quicker way to extract than via regular +expressions (actually, a *much* quicker way). Field counting starts at +1. Field zero is accepted, but will always lead to a "field not found" +error. The same happens if a field number higher than the number of +fields in the property is requested. The field number must be placed in +the "ToChar" parameter. An example where the 3rd field (delimited by +TAB) from the msg property is extracted is as follows: "%msg:F:3%". The +same example with semicolon as delimiter is "%msg:F,59:3%".</p> +<p>Please note that the special characters "F" and "R" are +case-sensitive. Only upper case works, lower case will return an error. +There are no white spaces permitted inside the sequence (that will lead +to error messages and will NOT provide the intended result).<br> +</p> <h2>Property Options</h2> -<b><code>property options</code></b> are case-insensitive. Currently, the following options -are defined:</p> +<b><code>property options</code></b> are +case-insensitive. Currently, the following options are defined: +<p></p> <table> -<tr><td><b>uppercase</b></td><td>convert property to lowercase only</td></tr> -<tr><td><b>lowercase</b></td><td>convert property text to uppercase only</td></tr> -<tr><td><b>drop-last-lf</b></td><td>The last LF in the message (if any), is dropped. - Especially useful for PIX.</td></tr> -<tr><td><b>date-mysql</b></td><td>format as mysql date</td></tr> -<tr><td><b>date-rfc3164</b></td><td>format as RFC 3164 date</td></tr> -<tr><td><b>date-rfc3339</b></td><td>format as RFC 3339 date</td></tr> -<tr> - <td><b>escape-cc</b></td><td>replace control characters (ASCII value 127 and - values less then 32) with an escape sequence. The sequence is "#<charval>" - where charval is the 3-digit decimal value of the control character. For - example, a tabulator would be replaced by "#009".<br> - Note: using this option requires that - <a href="../../rsyslog/doc/rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> - is set to off.</td> -</tr> -<tr> - <td><b>space-cc</b></td><td>replace control characters by spaces<br> - Note: using this option requires that - <a href="../../rsyslog/doc/rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> - is set to off.</td> -</tr> -<tr> - <td><b>drop-cc</b></td><td>drop control characters - the resulting string - will neither contain control characters, escape sequences nor any other - replacement character like space.<br> - Note: using this option requires that - <a href="../../rsyslog/doc/rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> - is set to off.</td> +<tbody> +<tr> +<td><b>uppercase</b></td> +<td>convert property to lowercase only</td> +</tr> +<tr> +<td><b>lowercase</b></td> +<td>convert property text to uppercase only</td> +</tr> +<tr> +<td><b>drop-last-lf</b></td> +<td>The last LF in the message (if any), is dropped. +Especially useful for PIX.</td> +</tr> +<tr> +<td><b>date-mysql</b></td> +<td>format as mysql date</td> +</tr> +<tr> +<td><b>date-rfc3164</b></td> +<td>format as RFC 3164 date</td> </tr> +<tr> +<td><b>date-rfc3339</b></td> +<td>format as RFC 3339 date</td> +</tr> +<tr> +<td><b>escape-cc</b></td> +<td>replace control characters (ASCII value 127 and values +less then 32) with an escape sequence. The sequnce is +"#<charval>" where charval is the 3-digit decimal value +of the control character. For example, a tabulator would be replaced by +"#009".<br> +Note: using this option requires that <a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> +is set to off.</td> +</tr> +<tr> +<td><b>space-cc</b></td> +<td>replace control characters by spaces<br> +Note: using this option requires that <a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> +is set to off.</td> +</tr> +<tr> +<td><b>drop-cc</b></td> +<td>drop control characters - the resulting string will +neither contain control characters, escape sequences nor any other +replacement character like space.<br> +Note: using this option requires that <a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> +is set to off.</td> +</tr> +</tbody> </table> - <h2>Further Links</h2> <ul> - <li>Article on "<a href="rsyslog_recording_pri.html">Recording the Priority of - Syslog Messages</a>" (describes use of templates to record severity and - facility of a message)</li> - <li><a href="rsyslog_conf.html">Configuration file syntax</a>, this is where you - actually use the property replacer.</li> +<li>Article on "<a href="rsyslog_recording_pri.html">Recording +the Priority of Syslog Messages</a>" (describes use of templates +to record severity and facility of a message)</li> +<li><a href="rsyslog_conf.html">Configuration file +syntax</a>, this is where you actually use the property replacer.</li> </ul> - -</body> -</html> +</body></html>
\ No newline at end of file diff --git a/doc/queueWorkerLogic.dia b/doc/queueWorkerLogic.dia Binary files differnew file mode 100644 index 0000000..068ea50 --- /dev/null +++ b/doc/queueWorkerLogic.dia diff --git a/doc/queueWorkerLogic.jpg b/doc/queueWorkerLogic.jpg Binary files differnew file mode 100644 index 0000000..fb143c4 --- /dev/null +++ b/doc/queueWorkerLogic.jpg diff --git a/doc/queueWorkerLogic_small.jpg b/doc/queueWorkerLogic_small.jpg Binary files differnew file mode 100644 index 0000000..4fae6d2 --- /dev/null +++ b/doc/queueWorkerLogic_small.jpg diff --git a/doc/queues.html b/doc/queues.html new file mode 100644 index 0000000..80641d8 --- /dev/null +++ b/doc/queues.html @@ -0,0 +1,350 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="de"> +<title>Understanding rsyslog queues</title></head> +<body> + +<h1>Understanding rsyslog Queues</h1> +<p>Rsyslog uses queues whenever two activities need to be loosely coupled. With a +queue, one part of the system "produces" something while another part "consumes" +this something. The "something" is most often syslog messages, but queues may +also be used for other purposes.</p> +<p>The most prominent example is the main message queue. Whenever rsyslog +receives a message (e.g. locally, via UDP, TCP or in whatever else way), it +places these messages into the main message queue. Later, it is dequeued by the +rule processor, which then evaluates which actions are to be carried out. In +front of each action, there is also a queue, which potentially de-couples the +filter processing from the actual action (e.g. writing to file, database or +forwarding to another host).</p> +<h1>Queue Modes</h1> +<p>Rsyslog supports different queue modes, some with submodes. Each of them has +specific advantages and disadvantages. Selecting the right queue mode is quite +important when tuning rsyslogd. The queue mode (aka "type") is set via the "<i>$<object>QueueType</i>" +config directive.</p> +<h2>Direct Queues</h2> +<p>Direct queues are <b>non</b>-queuing queues. A queue in direct mode does +neither queue nor buffer any of the queue elements but rather passes the element +directly (and immediately) from the producer to the consumer. This sounds strange, +but there is a good reason for this queue type.</p> +<p>Direct mode queues allow to use queues generically, even in places where +queuing is not always desired. A good example is the queue in front of output +actions. While it makes perfect sense to buffer forwarding actions or database +writes, it makes only limited sense to build up a queue in front of simple local +file writes. Yet, rsyslog still has a queue in front of every action. So for +file writes, the queue mode can simply be set to "direct", in which case no +queuing happens.</p> +<p>Please note that a direct queue also is the only queue type that passes back +the execution return code (success/failure) from the consumer to the producer. +This, for example, is needed for the backup action logic. Consequently, backup +actions require the to-be-checked action to use a "direct" mode queue.</p> +<p>To create a direct queue, use the "<i>$<object>QueueType Direct</i>" config +directive.</p> +<h2>Disk Queues</h2> +<p>Disk queues use disk drives for buffering. The important fact is that the +always use the disk and do not buffer anything in memory. Thus, the queue is +ultra-reliable, but by far the slowest mode. For regular use cases, this queue +mode is not recommended. It is useful if log data is so important that it must +not be lost, even in extreme cases.</p> +<p>When a disk queue is written, it is done in chunks. Each chunk receives its +individual file. Files are named with a prefix (set via the "<i>$<object>QueueFilename</i>" +config directive) and followed by a 7-digit number (starting at one and +incremented for each file). Chunks are 10mb by default, a different size can be +set via the"<i>$<object>QueueMaxFileSize</i>" config directive. Note that +the size limit is not a sharp one: rsyslog always writes one complete queue +entry, even if it violates the size limit. So chunks are actually a little but +(usually less than 1k) larger then the configured size. Each chunk also has a +different size for the same reason. If you observe different chunk sizes, you +can relax: this is not a problem.</p> +<p>Writing in chunks is used so that processed data can quickly be deleted and +is free for other uses - while at the same time keeping no artificial upper +limit on disk space used. If a disk quota is set (instructions further below), +be sure that the quota/chunk size allows at least two chunks to be written. +Rsyslog currently does not check that and will fail miserably if a single chunk +is over the quota.</p> +<p>Creating new chunks costs performance but provides quicker ability to free +disk space. The 10mb default is considered a good compromise between these two. +However, it may make sense to adapt these settings to local policies. For +example, if a disk queue is written on a dedicated 200gb disk, it may make sense +to use a 2gb (or even larger) chunk size.</p> +<p>Please note, however, that the disk queue by default does not update its +housekeeping structures every time it writes to disk. This is for performance +reasons. In the event of failure, data will still be lost (except when manually +is mangled with the file structures). However, disk queues can be set to write +bookkeeping information on checkpoints (every n records), so that this can be +made ultra-reliable, too. If the checkpoint interval is set to one, no data can +be lost, but the queue is exceptionally slow.</p> +<p>Each queue can be placed on a different disk for best performance and/or +isolation. This is currently selected by specifying different <i>$WorkDirectory</i> +config directives before the queue creation statement.</p> +<p>To create a disk queue, use the "<i>$<object>QueueType Disk</i>" config +directive. Checkpoint intervals can be specified via "<i>$<object>QueueCheckpointInterval</i>", +with 0 meaning no checkpoints. </p> +<h2>In-Memory Queues</h2> +<p>In-memory queue mode is what most people have on their mind when they think +about computing queues. Here, the enqueued data elements are held in memory. +Consequently, in-memory queues are very fast. But of course, they do not survive +any program or operating system abort (what usually is tolerable and unlikely). +Be sure to use an UPS if you use in-memory mode and your log data is important +to you. Note that even in-memory queues may hold data for an infinite amount of +time when e.g. an output destination system is down and there is no reason to move +the data out of memory (lying around in memory for an extended period of time is +NOT a reason). Pure in-memory queues can't even store queue elements anywhere +else than in core memory. </p> +<p>There exist two different in-memory queue modes: LinkedList and FixedArray. +Both are quite similar from the user's point of view, but utilize different +algorithms. </p> +<p>A FixedArray queue uses a fixed, pre-allocated array that holds pointers to +queue elements. The majority of space is taken up by the actual user data +elements, to which the pointers in the array point. The pointer array itself is +comparatively small. However, it has a certain memory footprint even if the +queue is empty. As there is no need to dynamically allocate any housekeeping +structures, FixedArray offers the best run time performance (uses the least CPU +cycle). FixedArray is best if there is a relatively low number of queue elements +expected and performance is desired. It is the default mode for the main message +queue (with a limit of 10,000 elements).</p> +<p>A LinkedList queue is quite the opposite. All housekeeping structures are +dynamically allocated (in a linked list, as its name implies). This requires +somewhat more runtime processing overhead, but ensures that memory is only +allocated in cases where it is needed. LinkedList queues are especially +well-suited for queues where only occasionally a than-high number of elements +need to be queued. A use case may be occasional message burst. Memory +permitting, it could be limited to e.g. 200,000 elements which would take up +only memory if in use. A FixedArray queue may have a too large static memory +footprint in such cases.</p> +<p><b>In general, it is advised to use LinkedList mode if in doubt</b>. The +processing overhead compared to FixedArray is low and may be +<span style="font-size: 12pt; line-height: 115%; font-family: 'Times New Roman',serif;" lang="EN-US"> +outweigh </span>by the reduction in memory use. Paging in most-often-unused +pointer array pages can be much slower than dynamically allocating them.</p> +<p>To create an in-memory queue, use the "<i>$<object>QueueType LinkedList</i>" +or "<i>$<object>QueueType FixedArray</i>" config directive.</p> +<h3>Disk-Assisted Memory Queues</h3> +<p>If a disk queue name is defined for in-memory queues (via <i> +$<object>QueueFileName</i>), they automatically +become "disk-assisted" (DA). In that mode, data is written to disk (and read +back) on an as-needed basis.</p> +<p>Actually, the regular memory queue (called the +"primary queue") and a disk queue (called the "DA queue") work in tandem in this +mode. Most importantly, the disk queue is activated if the primary queue is full +or needs to be persisted on shutdown. Disk-assisted queues combine the +advantages of pure memory queues with those of pure disk queues. Under normal +operations, they are very fast and messages will never touch the disk. But if +there is need to, an unlimited amount of messages can be buffered (actually +limited by free disk space only) and data can be persisted between rsyslogd runs.</p> +<p>With a DA-queue, both disk-specific and in-memory specific configuration +parameters can be set. From the user's point of view, think of a DA queue like a +"super-queue" which does all within a single queue [from the code perspective, +there is some specific handling for this case, so it is actually much like a +single object].</p> +<p>DA queues are typically used to de-couple potentially long-running and +unreliable actions (to make them reliable). For example, it is recommended to +use a disk-assisted linked list in-memory queue in front of each database and +"send via tcp" action. Doing so makes these actions reliable and de-couples +their potential low execution speed from the rest of your rules (e.g. the local +file writes). There is a howto on <a href="rsyslog_high_database_rate.html"> +massive database inserts</a> which nicely describes this use case. It may even +be a good read if you do not intend to use databases.</p> +<p>With DA queues, we do not simply write out everything to disk and then run as +a disk queue once the in-memory queue is full. A much smarter algorithm is used, +which involves a "high watermark" and a "low watermark". Both specify numbers of +queued items. If the queue size reaches high watermark elements, the queue +begins to write data elements to disk. It does so until it reaches the low water +mark elements. At this point, it stops writing until either high water mark is +reached again or the on-disk queue becomes empty, in which case the queue +reverts back to in-memory mode, only. While holding at the low watermark, new +elements are actually enqueued in memory. They are eventually written to disk, +but only if the high water mark is ever reached again. If it isn't, these items +never touch the disk. So even when a queue runs disk-assisted, there is +in-memory data present (this is a big difference to pure disk queues!).</p> +<p>This algorithm prevents unnecessary disk writes, but also leaves some +additional buffer space for message bursts. Remember that creating disk files +and writing to them is a lengthy operation. It is too lengthy to e.g. block +receiving UDP messages. Doing so would result in message loss. Thus, the queue +initiates DA mode, but still is able to receive messages and enqueue them - as +long as the maximum queue size is not reached. The number of elements between +the high water mark and the maximum queue size serves as this "emergency +buffer". Size it according to your needs, if traffic is very bursty you will +probably need a large buffer here. Keep in mind, though, that under normal +operations these queue elements will probably never be used. Setting the high +water mark too low will cause disk-assistance to be turned on more often than +actually needed.</p> +<p>The water marks can be set via the "<i>$<object>QueueHighWatermark</i>" and +"<i>$<object>QueueHighWatermark</i>" configuration file directives. Note that +these are actual numbers, not precentages. Be sure they make sense (also in +respect to "<i>$<object>QueueSize</i>"), as rsyslodg does currently not perform +any checks on the numbers provided. It is easy to screw up the system here (yes, +a feature enhancement request is filed ;)).</p> +<h1>Limiting the Queue Size</h1> +<p>All queues, including disk queues, have a limit of the number of elements +they can enqueue. This is set via the "<i>$<object>QueueSize</i>" config +parameter. Note that the size is specified in number of enqueued elements, not +their actual memory size. Memory size limits can not be set. A conservative +assumption is that a single syslog messages takes up 512 bytes on average +(in-memory, NOT on the wire, this *is* a difference).</p> +<p>Disk assisted queues are special in that they do <b>not</b> have any size +limit. The enqueue an unlimited amount of elements. To prevent running out of +space, disk and disk-assisted queues can be size-limited via the "<i>$<object>QueueMaxDiskSpace</i>" +configuration parameter. If it is not set, the limit is only available free +space (and reaching this limit is currently not very gracefully handled, so +avoid running into it!). If a limit is set, the queue can not grow larger than +it. Note, however, that the limit is approximate. The engine always writes +complete records. As such, it is possible that slightly more than the set limit +is used (usually less than 1k, given the average message size). Keeping strictly +on the limit would be a performance hurt, and thus the design decision was to +favour performance. If you don't like that policy, simply specify a slightly +lower limit (e.g. 999,999K instead of 1G).</p> +<p>In general, it is a good idea to limit the pysical disk space even if you +dedicate a whole disk to rsyslog. That way, you prevent it from running out of +space (future version will have an auto-size-limit logic, that then kicks in in +such situations).</p> +<h1>Worker Thread Pools</h1> +<p>Each queue (except in "direct" mode) has an associated pool of worker +threads. Worker threads carry out the action to be performed on the data +elements enqueued. As an actual sample, the main message queue's worker task is +to apply filter logic to each incoming message and enqueue them to the relevant +output queues (actions).</p> +<p>Worker threads are started and stopped on an as-needed basis. On a system +without activity, there may be no worker at all running. One is automatically +started when a message comes in. Similarily, additional workers are started if +the queue grows above a specific size. The "<i>$<object>QueueWorkerThreadMinimumMessages</i>" +config parameter controls worker startup. If it is set to the minimum number of +elements that must be enqueued in order to justify a new worker startup. For +example, let's assume it is set to 100. As long as no more than 100 messages are +in the queue, a single worker will be used. When more than 100 messages arrive, +a new worker thread is automatically started. Similarily, a third worker will be +started when there are at least 300 messages, a forth when reaching 400 and so +on.</p> +<p>It, however, does not make sense to have too many worker threads running in +parall. Thus, the upper limit ca be set via "<i>$<object>QueueWorkerThreads</i>". +If it, for example, is set to four, no more than four workers will ever be +started, no matter how many elements are enqueued. </p> +<p>Worker threads that have been started are kept running until an inactivity +timeout happens. The timeout can be set via "<i>$<object>QueueWorkerTimeoutShutdown</i>" +and is specified in milliseconds. If you do not like to keep the workers +running, simply set it to 0, which means immediate timeout and thus immediate +shutdown. But consider that creating threads involves some overhead, and this is +why we keep them running.</p> +<h2>Discarding Messages</h2> +<p>If the queue reaches the so called "discard watermark" (a number of queued +elements), less important messages can automatically be discarded. This is in an +effort to save queue space for more important messages, which you even less like +to loose. Please note that whenever there are more than "discard watermark" +messages, both newly incoming as well as already enqueued low-priority messages +are discarded. The algorithm discards messages newly coming in and those at the +front of the queue.</p> +<p>The discard watermark is a last resort setting. It should be set sufficiently +high, but low enough to allow for large message burst. Please note that it take +effect immediately and thus shows effect promptly - but that doesn't help if the +burst mainly consist of high-priority messages...</p> +<p>The discard watermark is set via the "<i>$<object>QueueDiscardMark</i>" +directive. The priority of messages to be discarded is set via "<i>$<object>QueueDiscardSeverity</i>". +This directive accepts both the usual textual severity as well as a +numerical one. To understand it, you must be aware of the numerical +severity values. They are defined in RFC 3164:</p> +<pre> Numerical Severity<br> Code<br><br> 0 Emergency: system is unusable<br> 1 Alert: action must be taken immediately<br> 2 Critical: critical conditions<br> 3 Error: error conditions<br> 4 Warning: warning conditions<br> 5 Notice: normal but significant condition<br> 6 Informational: informational messages<br> 7 Debug: debug-level messages</pre> +<p>Anything of the specified severity and (numerically) above it is +discarded. To turn message discarding off, simply specify the discard +watermark to be higher than the queue size. An alternative is to +specify the numerical value 8 as DiscardSeverity. This is also the +default setting to prevent unintentional message loss. So if you would +like to use message discarding, you need to set" <i>$<object>QueueDiscardSeverity</i>" to an actual value.</p> +<p>An interesting application is with disk-assisted queues: if the discard +watermark is set lower than the high watermark, message discarding will start +before the queue becomes disk-assisted. This may be a good thing if you would +like to switch to disk-assisted mode only in cases where it is absolutely +unavoidable and you prefer to discard less important messages first.</p> +<h1>Filled-Up Queues</h1> +<p>If the queue has either reached its configured maximum number of entries or +disk space, it is finally full. If so, rsyslogd throttles the data element +submitter. If that, for example, is a reliable input (TCP, local log socket), +that will slow down the message originator which is a good +<span style="font-size: 12pt; line-height: 115%; font-family: 'Times New Roman',serif;" lang="EN-US"> +resolution </span>for this scenario.</p> +<p>During +<span style="font-size: 12pt; line-height: 115%; font-family: 'Times New Roman',serif;" lang="EN-US"> +throtteling</span>, a disk-assisted queue continues to write to disk and +messages are also discarded based on severity as well as regular dequeuing and +processing continues. So chances are good the situation will be resolved by +simply throttling. Note, though, that throtteling is highly undesirable for +unreliable sources, like UDP message reception. So it is not a good thing to run +into throtteling mode at all.</p> +<p>We can not hold processing +<span style="font-size: 12pt; line-height: 115%; font-family: 'Times New Roman',serif;" lang="EN-US"> +infinitely</span>, not even when throtteling. For example, throtteling the local +log socket too long would cause the system at whole come to a standstill. To +prevent this, rsyslogd times out after a configured period ("<i>$<object>QueueTimeoutEnqueue</i>", +specified in milliseconds) if no space becomes available. As a last resort, it +then discards the newly arrived message.</p> +<p>If you do not like throtteling, set the timeout to 0 - the message will then +immediately be discarded. If you use a high timeout, be sure you know what you +do. If a high main message queue enqueue timeout is set, it can lead to +something like a complete hang of the system. The same problem does not apply to +action queues.</p> +<h2>Rate Limiting</h2> +<p>Rate limiting provides a way to prevent rsyslogd from processing things too +fast. It can, for example, prevent overruning a receiver system.</p> +<p>Currently, there are only limited rate-limiting features available. The "<i>$<object>QueueDequeueSlowdown</i>" +directive allows to specify how long (in microseconds) dequeueing should be +delayed. While simple, it still is powerful. For example, using a +DequeueSlowdown delay of 1,000 microseconds on a UDP send action ensures that no +more than 1,000 messages can be sent within a second (actually less, as there is +also some time needed for the processing itself). </p> +<h2>Terminating Queues</h2> +<p>Terminating a process sounds easy, but can be complex. +<span style="font-size: 12pt; line-height: 115%; font-family: 'Times New Roman',serif;" lang="EN-US"> +Terminating </span>a running queue is in fact the most complex operation a queue +object can perform. You don't see that from a user's point of view, but its +quite hard work for the developer to do everything in the right order.</p> +<p>The complexity arises when the queue has still data enqueued when it +finishes. Rsyslog tries to preserve as much of it as possible. As a first +measure, there is a regular queue time out ("<i>$<object>QueueTimeoutShutdown</i>", +specified in milliseconds): the queue workers are given that time period to +finish processing the queue.</p> +<p>If after that period there is still data in the queue, workers are instructed +to finish the current data element and then terminate. This essentially means +any other data is lost. There is another timeout ("<i>$<object>QueueTimeoutActionCompletion</i>", +also specified in milliseconds) that specifies how long the workers have to +finish the current element. If that timeout expires, any remaining workers are +cancelled and the queue is brought down.</p> +<p>If you do not like to lose data on shutdown, the "<i>$<object>QueueSaveOnShutdown</i>" +parameter can be set to "on". This requires either a disk or disk-assisted +queue. If set, rsyslogd ensures that any queue elements are saved to disk before +it terminates. This includes data elements there were begun being processed by +workers that needed to be cancelled due to too-long processing. For a large +queue, this operation may be lengthy. No timeout applies to a required shutdown +save.</p> +<h1>Where are Queues Used?</h1> +<p> Currently, queues are used for the main message queue and for the +actions.</p> +<p>There is a single main message queue inside rsyslog. Each input module +delivers messages to it. The main message queue worker filters messages based on +rules specified in rsyslog.conf and dispatches them to the individual action +queues. Once a message is in an action queue, it is deleted from the main +message queue.</p> +<p>There are multiple action queues, one for each configured action. By default, +these queues operate in direct (non-queueing) mode. Action queues are fully +configurable and thus can be changed to whatever is best for the given use case.</p> +<p>Future versions of rsyslog will most probably utilize queues at other places, +too.</p> +<p> +<span style="font-size: 12pt; line-height: 115%; font-family: 'Times New Roman',serif;" lang="EN-US"> +Wherever </span>"<i><object></i>" was used above in the config file +statements, substitute "<i><object></i>" with either "MainMsg" or "Action". The +former will set main message queue +<span style="font-size: 12pt; line-height: 115%; font-family: 'Times New Roman',serif;" lang="EN-US"> +parameters</span>, the later parameters for the next action that will be +created. Action queue parameters can not be modified once the action has been +specified. For example, to tell the main message queue to save its content on +shutdown, use <i>$MainMsgQueueSaveOnShutdown on</i>".</p> +<p>If the same parameter is specified multiple times before a queue is created, +the last one specified takes precedence. The main message queue is created after +parsing the config file and all of its potential includes. An action queue is +created each time an action selector is specified. Action queue parameters are +reset to default after an action queue has been created (to provide a clean +environment for the next action).</p> +<p>Not all queues necessarily support the full set of queue configuration +parameters, because not all are applicable. For example, in current output +module design, actions do not support multi-threading. Consequently, the number +of worker threads is fixed to one for action queues and can not be changed.</p> + +</body></html>
\ No newline at end of file diff --git a/doc/rainerscript.html b/doc/rainerscript.html new file mode 100644 index 0000000..ef0e41c --- /dev/null +++ b/doc/rainerscript.html @@ -0,0 +1,63 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>RainerScript</title> + +</head> +<body> +<h1>RainerScript</h1> +<p><b>RainerScript is a scripting language specifically +designed and well-suited +for processing network events and configuring event processors</b> +(with the most prominent sample being syslog). While RainerScript is +theoritically usable with various softwares, it currently is being +used, and developed for, rsyslog. Please note that RainerScript may not +be abreviated as rscript, because that's somebody elses trademark.</p> +<p>RainerScript is currently under development. It has its first +appearance in rsyslog 3.12.0, where it provides complex expression +support. However, this is only a very partial implementatio of the +scripting language. Due to technical restrictions, the final +implementation will have a slightly different syntax. So while you are +invited to use the full power of expresssions, you unfortunatley need +to be prepared to change your configuration files at some later points. +Maintaining backwards-compatibility at this point would cause us to +make too much compromise. Defering the release until everything is +perfect is also not a good option. So use your own judgement.</p> +<p>A formal definition of the language can be found in <a href="rscript_abnf.html">RainerScript ABNF</a>. The +rest of this document describes the language from the user's point of +view. Please note that this doc is also currently under development and +can (and will) probably improve as time progresses. If you have +questions, use the rsyslog forum. Feedback is also always welcome.</p> +<h2>Data Types</h2> +RainerScript is a typeless language. That doesn't imply you don't need +to care about types. Of course, expressions like "A" + "B" will not +return a valid result, as you can't really add two letters (to +concatenate them, use the concatenation operator &). + However, all type conversions are automatically done by the +script interpreter when there is need to do so.<br> +<h2>Expressions</h2> +The language supports arbitrary complex expressions. All usual +operators are supported. The precedence of operations is as follows +(with operations being higher in the list being carried out before +those lower in the list, e.g. multiplications are done before additions.<br> +<ul> +<li>expressions in parenthesis</li><li>not, unary minus</li><li>*, /, % (modulus, as in C)</li><li>+, -, & (string concatenation)</li><li>==, !=, <>, <, >, <=, >=, contains (strings!), startswith (strings!)</li><li>and</li><li>or</li> +</ul>For example, "not a == b" probably returns not what you intended. +The script processor will first evaluate "not a" and then compare the +resulting boolean to the value of b. What you probably intended to do +is "not (a == b)". And if you just want to test for inequality, we +highly suggest to use "!=" or "<>". Both are exactly the same and +are provided so that you can pick whichever you like best. So inquality +of a and b should be tested as "a <> b". The "not" operator +should be reserved to cases where it actually is needed to form a +complex boolean expression. In those cases, parenthesis are highly +recommended. +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html>
\ No newline at end of file diff --git a/doc/rsconf1_filecreatemode.html b/doc/rsconf1_filecreatemode.html index 7c6f171..c844086 100644 --- a/doc/rsconf1_filecreatemode.html +++ b/doc/rsconf1_filecreatemode.html @@ -30,6 +30,6 @@ index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> Copyright © 2007 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL -version 2 or higher.</font></p> +version 3 or higher.</font></p> </body> </html> diff --git a/doc/rsconf1_markmessageperiod.html b/doc/rsconf1_markmessageperiod.html new file mode 100644 index 0000000..9b6590c --- /dev/null +++ b/doc/rsconf1_markmessageperiod.html @@ -0,0 +1,30 @@ +<html> +<head> +<title>rsyslog.conf file</title> +</head> +<body> +<h2>$MarkMessagePeriod</h2> +<p><b>Type:</b> specific to immark input module</p> +<p><b>Default:</b> 1800 (20 minutes)</p> +<p><b>Description:</b></p> +<p>This specifies when mark messages are to be written to output modules. The +time specified is in seconds. Specifying 0 is possible and disables mark +messages. In that case, however, it is more efficient to NOT load the immark +input module.</p> +<p>So far, there is only one mark message process and any subsequent +$MarkMessagePeriod overwrites the previous.</p> +<p><b>This directive is only available after the immark input module has been +loaded.</b></p> +<p><b>Sample:</b></p> +<p><code><b>$MarkMessagePeriod 600 # mark messages appear every 10 Minutes</b></code></p> +<p><b>Available since:</b> rsyslog 3.0.0</p> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2007 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 2 or higher.</font></p> +</body> +</html>
\ No newline at end of file diff --git a/doc/rscript_abnf.html b/doc/rscript_abnf.html new file mode 100644 index 0000000..97de728 --- /dev/null +++ b/doc/rscript_abnf.html @@ -0,0 +1,41 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>RainerScript ABNF</title></head> +<body> +<h1>RainerScript ABNF</h1> +<p>This is the formal definition of RainerScript, as supported by +rsyslog configuration. Please note that this currently is working +document and the actual implementation may be quite different.</p> +<p>The +first glimpse of RainerScript will be available as part of rsyslog +3.12.0 expression support. However, the 3.12. series of rsyslog will +have a partial script implementaiton, which will not necessariy be +compatible with the later full implementation. So if you use it, be +prepared for some config file changes as RainerScript evolves.</p> +<p>C-like comments (/* some comment */) are supported in all pure +RainerScript lines. However, legacy-mapped lines do not support them. +All lines support the hash mark "#" as a comment initiator. Everything +between the hash and the end of line is a comment (just like // in C++ +and many other languages).</p> +<h2>Formal Definition</h2> +<p>Below is the formal language definitionin ABNF (RFC 2234) +format: <br> +</p> +<pre>; <span style="font-weight: bold;">all of this is a working document and may change!</span> -- rgerhards, 2008-02-24<br><br>script := *stmt<br>stmt := (if_stmt / block / vardef / run_s / load_s)<br>vardef := "var" ["scope" = ("global" / "event")] <br>block := "begin" stmt "end"<br>load_s := "load" constraint ("module") modpath params ; load mod only if expr is true<br>run_s := "run" constraint ("input") name<br>constraint:= "if" expr ; constrains some one-time commands<br>modpath := expr<br>params := ["params" *1param *("," param) "endparams"]<br>param := paramname) "=" expr<br>paramname := [*(obqualifier ".") name]<br>modpath:= ; path to module<br>?line? := cfsysline / cfli<br>cfsysline:= BOL "$" *char EOL ; how to handle the first line? (no EOL in front!)<br>BOL := ; Begin of Line - implicitely set on file beginning and after each EOL<br>EOL := 0x0a ;LF<br>if_stmt := "if" expr "then"<br>old_filter:= BOL facility "." severity ; no whitespace allowed between BOL and facility!<br>facility := "*" / "auth" / "authpriv" / "cron" / "daemon" / "kern" / "lpr" / <br> "mail" / "mark" / "news" / "security" / "syslog" / "user" / "uucp" / <br> "local0" .. "local7" / "mark"<br> ; The keyword security should not be used anymore<br> ; mark is just internal<br>severity := TBD ; not really relevant in this context<br><br>; and now the actual expression<br>expr := e_and *("or" e_and)<br>e_and := e_cmp *("and" e_cmp)<br>e_cmp := val 0*1(cmp_op val)<br>val := term *(("+" / "-" / "&") term)<br>term := factor *(("*" / "/" / "%") factor)<br>factor := ["not"] ["-"] terminal<br>terminal := var / constant / function / ( "(" expr ")" )<br>function := name "(" *("," expr) ")"<br>var := "$" varname<br>varname := msgvar / sysvar<br>msgvar := name<br>sysvar := "$" name<br>name := alpha *(alnum)<br>constant := string / number<br>string := simpstr / tplstr ; tplstr will be implemented in next phase<br>simpstr := "'" *char "'" ; use your imagination for char ;)<br>tplstr := '"' template '"' ; not initially implemented<br>number := ["-"] 1*digit ; 0nn = octal, 0xnn = hex, nn = decimal<br>cmp_op := "==" / "!=" / "<>" / "<" / ">" / "<=" / ">=" / "contains" / "contains_i" / "startswith" / "startswith_i"<br>digit := %x30-39<br>alpha := "a" ... "z" # all letters<br>alnum :* alpha / digit / "_" /"-" # "-" necessary to cover currently-existing message properties<br></pre> +<h2>Samples</h2> +<p>Some samples of RainerScript:</p><p>define function IsLinux<br>begin<br> if $environ contains "linux" then return true else return false<br>end</p><p>load if IsLinux() 'imklog.so' params name='klog' endparams /* load klog under linux only */<br>run if IsLinux() input 'klog'<br>load 'ommysql.so'</p><p>if $message contains "error" then<br> action<br> type='ommysql.so', queue.mode='disk', queue.highwatermark = 300,<br> action.dbname='events', action.dbuser='uid',<br> + [?action.template='templatename'?] or [?action.sql='insert into +table... values('&$facility&','&$severity&...?]<br> endaction<br><br>... or ...</p><p>define action writeMySQL<br> type='ommysql.so', queue.mode='disk', queue.highwatermark = 300,<br> action.dbname='events', action.dbuser='uid',<br> [?action.template='templatename'?] or [?action.sql='insert into table... values('<span style="font-family: monospace;"> &</span> $facility & ',' & $severity &...?]<br> endaction</p><p>if $message contains "error" then action writeMySQL</p><p>ALTERNATE APPROACH</p><p>define function IsLinux(<br> if $environ contains "linux" then return true else return false<br>)</p><p>load if IsLinux() 'imklog.so' params name='klog' endparams /* load klog under linux only */<br>run if IsLinux() input 'klog'<br>load 'ommysql.so'</p><p>if $message contains "error" then<br> action(<br> type='ommysql.so', queue.mode='disk', queue.highwatermark = 300,<br> action.dbname='events', action.dbuser='uid',<br> + [?action.template='templatename'?] or [?action.sql='insert into +table... values('&$facility&','&$severity&...?]<br> )<br><br>... or ...</p><p>define action writeMySQL(<br> type='ommysql.so', queue.mode='disk', queue.highwatermark = 300,<br> action.dbname='events', action.dbuser='uid',<br> + [?action.template='templatename'?] or [?action.sql='insert into +table... values('&$facility&','&$severity&...?]<br> )</p><p>if $message contains "error" then action writeMySQL(action.dbname='differentDB')</p><p></p><p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html>
\ No newline at end of file diff --git a/doc/rsyslog_conf.html b/doc/rsyslog_conf.html index bf878e8..5931a24 100644 --- a/doc/rsyslog_conf.html +++ b/doc/rsyslog_conf.html @@ -1,701 +1,1007 @@ -<html> -<head> -<title>rsyslog.conf file</title> -</head> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>rsyslog.conf file</title></head> <body> <h1>rsyslog.conf configuration file</h1> -<p><b>This document is currently being enhanced. Please pardon its current -appearance.</b></p> -<p><b>Rsyslogd is configured via the rsyslog.conf file</b>, typically found in -/etc. By default, rsyslogd reads the file /etc/rsyslog.conf. This may be changed -by a command line option.</p> +<p><b>This document is currently being enhanced. Please +pardon its current appearance.</b></p> +<p><b>Rsyslogd is configured via the rsyslog.conf file</b>, +typically found in /etc. By default, rsyslogd reads the file +/etc/rsyslog.conf. This may be changed by a command line option.</p> <p><a href="http://wiki.rsyslog.com/index.php/Configuration_Samples"> Configuration file examples can be found in the rsyslog wiki</a>.</p> -<p>There is also one sample file provided together with the documentation set. -If you do not like to read, be sure to have at least a quick look at -<a href="rsyslog-example.conf">rsyslog-example.conf</a>. </p> -<p>While rsyslogd contains enhancements over standard syslogd, efforts have been -made to keep the configuration file as compatible as possible. While, for -obvious reasons, <a href="features.html">enhanced features</a> require a -different config file syntax, rsyslogd should be able to work with a standard -syslog.conf file. This is especially useful while you are migrating from syslogd -to rsyslogd.</p> +<p>There is also one sample file provided together with the +documentation set. If you do not like to read, be sure to have at least +a quick look at +<a href="rsyslog-example.conf">rsyslog-example.conf</a>. +</p> +<p>While rsyslogd contains enhancements over standard syslogd, +efforts have been made to keep the configuration file as compatible as +possible. While, for obvious reasons, <a href="features.html">enhanced +features</a> require a different config file syntax, rsyslogd +should be able to work with a standard syslog.conf file. This is +especially useful while you are migrating from syslogd to rsyslogd.</p> +<h2>Modules</h2> +<p>Rsyslog has a modular design. Consequently, there is a growing +number of modules. Here is the entry point to their documentation and +what they do (list is currently not complete)</p> +<ul> +<li><a href="omsnmp.html">omsnmp</a> - SNMP +trap output module</li><li>omrelp - RELP output module</li> +<li>omgss - output module for GSS-enabled syslog</li> +<li>ommysql - output module for MySQL</li> +<li>ompgsql - output module for PostgreSQL</li> +<li><a href="omlibdbi.html">omlibdbi</a> - +generic database output module (Firebird/Interbase, MS SQL, Sybase, +SQLLite, Ingres, Oracle, mSQL)</li> +<li><a href="imfile.html">imfile</a> +- input module for text files</li><li>imrelp - RELP input module</li> +<li>imudp - udp syslog message input</li> +<li><a href="imtcp.html">imtcp</a> - input +plugin for plain tcp syslog</li> +<li><a href="imgssapi.html">imgssapi</a> - +input plugin for plain tcp and GSS-enable syslog</li> +<li>immark - support for mark messages</li> +<li>imklog - kernel logging</li><li><a href="imuxsock.html">imuxsock</a> - unix sockets, including the system log socket</li> +</ul> +<p>Please note that each module provides configuration +directives, which are NOT necessarily being listed below. Also +remember, that a modules configuration directive (and functionality) is +only available if it has been loaded (using $ModLoad).</p> +<h2>Lines</h2> +Lines can be continued by specifying a backslash ("\") as the last +character of the line.<br> <h2>Global Directives</h2> -<p>All global directives need to be specified on a line by their own and must -start with a dollar-sign. Here is a list in alphabetical order. Follow links for -a description.</p> +<p>All global directives need to be specified on a line by their +own and must start with a dollar-sign. Here is a list in alphabetical +order. Follow links for a description.</p> +<p>Not all directives have an in-depth description right now. +Default values for them are in bold. A more in-depth description will +appear as implementation progresses. Directives may change during that +process, we will NOT try hard to maintain backwards compatibility +(after all, v3 is still very early in development and quite +unstable...). So you have been warned ;)</p> +<p><b>Be sure to read information about <a href="queues.html">queues in rsyslog</a></b> - +many parameter settings modify queue parameters. If in doubt, use the +default, it is usually well-chosen and applicable in most cases.</p> <ul> - <li><a href="rsconf1_actionexeconlywhenpreviousissuspended.html">$ActionExecOnlyWhenPreviousIsSuspended</a></li> - <li><a href="rsconf1_actionresumeinterval.html">$ActionResumeInterval</a></li> - <li><a href="rsconf1_allowedsender.html">$AllowedSender</a></li> - <li><a href="rsconf1_controlcharacterescapeprefix.html">$ControlCharacterEscapePrefix</a></li> - <li><a href="rsconf1_debugprintcfsyslinehandlerlist.html">$DebugPrintCFSyslineHandlerList</a></li> - <li><a href="rsconf1_debugprintmodulelist.html">$DebugPrintModuleList</a></li> - <li><a href="rsconf1_debugprinttemplatelist.html">$DebugPrintTemplateList</a></li> - <li><a href="rsconf1_dircreatemode.html">$DirCreateMode</a></li> - <li><a href="rsconf1_dirgroup.html">$DirGroup</a></li> - <li><a href="rsconf1_dirowner.html">$DirOwner</a></li> - <li><a href="rsconf1_dropmsgswithmaliciousdnsptrrecords.html">$DropMsgsWithMaliciousDnsPTRRecords</a></li> - <li><a href="rsconf1_droptrailinglfonreception.html">$DropTrailingLFOnReception</a></li> - <li><a href="rsconf1_dynafilecachesize.html">$DynaFileCacheSize</a></li> - <li><a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a></li> - <li><a href="rsconf1_failonchownfailure.html">$FailOnChownFailure</a></li> - <li><a href="rsconf1_filecreatemode.html">$FileCreateMode</a></li> - <li><a href="rsconf1_filegroup.html">$FileGroup</a></li> - <li><a href="rsconf1_fileowner.html">$FileOwner</a></li> - <li><a href="rsconf1_gssforwardservicename.html">$GssForwardServiceName</a></li> - <li><a href="rsconf1_gsslistenservicename.html">$GssListenServiceName</a></li> - <li><a href="rsconf1_gssmode.html">$GssMode</a></li> - <li><a href="rsconf1_includeconfig.html">$IncludeConfig</a></li> - <li><a href="rsconf1_mainmsgqueuesize.html">$MainMsgQueueSize</a></li> - <li><a href="rsconf1_moddir.html">$ModDir</a></li> - <li><a href="rsconf1_modload.html">$ModLoad</a></li> - <li><a href="rsconf1_repeatedmsgreduction.html">$RepeatedMsgReduction</a></li> - <li><a href="rsconf1_resetconfigvariables.html">$ResetConfigVariables</a></li> - <li><a href="rsconf1_umask.html">$UMASK</a></li> +<li><a href="rsconf1_actionexeconlywhenpreviousissuspended.html">$ActionExecOnlyWhenPreviousIsSuspended</a></li><li>$ActionFileDefaultTemplate [templateName] - sets a new default template for file actions</li><li>$ActionFileEnableSync [on/<span style="font-weight: bold;">off</span>] - enables file syncing capability of omfile</li><li>$ActionForwardDefaultTemplate [templateName] - sets a new default template for UDP and plain TCP forwarding action</li><li>$ActionGSSForwardDefaultTemplate [templateName] - sets a new default template for GSS-API forwarding action</li> +<li>$ActionQueueCheckpointInterval <number></li> +<li>$ActionQueueDequeueSlowdown <number> [number +is timeout in <i> micro</i>seconds (1000000us is 1sec!), +default 0 (no delay). Simple rate-limiting!]</li> +<li>$ActionQueueDiscardMark <number> [default +9750]</li> +<li>$ActionQueueDiscardSeverity <number> +[*numerical* severity! default 4 (warning)]</li> +<li>$ActionQueueFileName <name></li> +<li>$ActionQueueHighWaterMark <number> [default +8000]</li> +<li>$ActionQueueImmediateShutdown [on/<b>off</b>]</li> +<li>$ActionQueueSize <number></li> +<li>$ActionQueueLowWaterMark <number> [default +2000]</li> +<li>$ActionQueueMaxFileSize <size_nbr>, default 1m</li> +<li>$ActionQueueTimeoutActionCompletion <number> +[number is timeout in ms (1000ms is 1sec!), default 1000, 0 means +immediate!]</li> +<li>$ActionQueueTimeoutEnqueue <number> [number +is timeout in ms (1000ms is 1sec!), default 2000, 0 means indefinite]</li> +<li>$ActionQueueTimeoutShutdown <number> [number +is timeout in ms (1000ms is 1sec!), default 0 (indefinite)]</li> +<li>$ActionQueueWorkerTimeoutThreadShutdown +<number> [number is timeout in ms (1000ms is 1sec!), +default 60000 (1 minute)]</li> +<li>$ActionQueueType [FixedArray/LinkedList/<b>Direct</b>/Disk]</li> +<li>$ActionQueueSaveOnShutdown [on/<b>off</b>] +</li> +<li>$ActionQueueWorkerThreads <number>, num +worker threads, default 1, recommended 1</li> +<li>$ActionQueueWorkerThreadMinumumMessages +<number>, default 100</li> +<li><a href="rsconf1_actionresumeinterval.html">$ActionResumeInterval</a></li><li>$ActionResumeRetryCount <number> [default 0, +-1 means eternal]</li> +<li><a href="rsconf1_allowedsender.html">$AllowedSender</a></li> +<li><a href="rsconf1_controlcharacterescapeprefix.html">$ControlCharacterEscapePrefix</a></li> +<li><a href="rsconf1_debugprintcfsyslinehandlerlist.html">$DebugPrintCFSyslineHandlerList</a></li> +<li>$DebugPrintKernelSymbols (imklog) [on/<b>off</b>]</li> +<li><a href="rsconf1_debugprintmodulelist.html">$DebugPrintModuleList</a></li> +<li><a href="rsconf1_debugprinttemplatelist.html">$DebugPrintTemplateList</a></li> +<li><a href="rsconf1_dircreatemode.html">$DirCreateMode</a></li> +<li><a href="rsconf1_dirgroup.html">$DirGroup</a></li> +<li><a href="rsconf1_dirowner.html">$DirOwner</a></li> +<li><a href="rsconf1_dropmsgswithmaliciousdnsptrrecords.html">$DropMsgsWithMaliciousDnsPTRRecords</a></li> +<li><a href="rsconf1_droptrailinglfonreception.html">$DropTrailingLFOnReception</a></li> +<li><a href="rsconf1_dynafilecachesize.html">$DynaFileCacheSize</a></li> +<li><a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a></li> +<li><a href="rsconf1_failonchownfailure.html">$FailOnChownFailure</a></li> +<li><a href="rsconf1_filecreatemode.html">$FileCreateMode</a></li> +<li><a href="rsconf1_filegroup.html">$FileGroup</a></li> +<li><a href="rsconf1_fileowner.html">$FileOwner</a></li> +<li><a href="rsconf1_gssforwardservicename.html">$GssForwardServiceName</a></li> +<li><a href="rsconf1_gsslistenservicename.html">$GssListenServiceName</a></li> +<li><a href="rsconf1_gssmode.html">$GssMode</a></li> +<li><a href="rsconf1_includeconfig.html">$IncludeConfig</a></li> +<li>$klogSymbolLookup (imklog) [<b>on</b>/off] -- +former klogd -x option</li> +<li>$klogUseSyscallInterface (imklog) [on/<b>off</b>] +-- former klogd -2 option</li> +<li>$klogSymbolsTwice (imklog) [on/<b>off</b>] -- +former klogd -s option</li> +<li>$MainMsgQueueCheckpointInterval <number></li> +<li>$MainMsgQueueDequeueSlowdown <number> [number +is timeout in <i> micro</i>seconds (1000000us is 1sec!), +default 0 (no delay). Simple rate-limiting!]</li> +<li>$MainMsgQueueDiscardMark <number> [default +9750]</li> +<li>$MainMsgQueueDiscardSeverity <severity> +[either a textual or numerical severity! default 4 (warning)]</li> +<li>$MainMsgQueueFileName <name></li> +<li>$MainMsgQueueHighWaterMark <number> [default +8000]</li> +<li>$MainMsgQueueImmediateShutdown [on/<b>off</b>]</li> +<li><a href="rsconf1_mainmsgqueuesize.html">$MainMsgQueueSize</a></li> +<li>$MainMsgQueueLowWaterMark <number> [default +2000]</li> +<li>$MainMsgQueueMaxFileSize <size_nbr>, default +1m</li> +<li>$MainMsgQueueTimeoutActionCompletion +<number> [number is timeout in ms (1000ms is 1sec!), +default +1000, 0 means immediate!]</li> +<li>$MainMsgQueueTimeoutEnqueue <number> [number +is timeout in ms (1000ms is 1sec!), default 2000, 0 means indefinite]</li> +<li>$MainMsgQueueTimeoutShutdown <number> [number +is timeout in ms (1000ms is 1sec!), default 0 (indefinite)]</li> +<li>$MainMsgQueueWorkerTimeoutThreadShutdown +<number> [number is timeout in ms (1000ms is 1sec!), +default 60000 (1 minute)]</li> +<li>$MainMsgQueueType [<b>FixedArray</b>/LinkedList/Direct/Disk]</li> +<li>$MainMsgQueueSaveOnShutdown [on/<b>off</b>] +</li> +<li>$MainMsgQueueWorkerThreads <number>, num +worker threads, default 1, recommended 1</li> +<li>$MainMsgQueueWorkerThreadMinumumMessages +<number>, default 100</li> +<li><a href="rsconf1_markmessageperiod.html">$MarkMessagePeriod</a> +(immark)</li> +<li><a href="rsconf1_moddir.html">$ModDir</a></li> +<li><a href="rsconf1_modload.html">$ModLoad</a></li> + +<li><a href="rsconf1_repeatedmsgreduction.html">$RepeatedMsgReduction</a></li> +<li><a href="rsconf1_resetconfigvariables.html">$ResetConfigVariables</a></li> +<li>$WorkDirectory <name> (directory for spool +and other work files)</li><li>$UDPServerAddress <IP> (imudp) -- local IP +address (or name) the UDP listens should bind to</li> +<li>$UDPServerRun <port> (imudp) -- former +-r<port> option, default 514, start UDP server on this +port, "*" means all addresses</li> +<li><a href="rsconf1_umask.html">$UMASK</a></li> </ul> +<p><b>Where <size_nbr> is specified above,</b> +modifiers can be used after the number part. For example, 1k means +1024. Supported are k(ilo), m(ega), g(iga), t(era), p(eta) and e(xa). +Lower case letters refer to the traditional binary defintion (e.g. 1m +equals 1,048,576) whereas upper case letters refer to their new +1000-based definition (e.g 1M equals 1,000,000).</p> +<p>Numbers may include '.' and ',' for readability. So you can +for example specify either "1000" or "1,000" with the same result. +Please note that rsyslogd simply ignores the punctuation. Form it's +point of view, "1,,0.0.,.,0" also has the value 1000. </p> <h2>Basic Structure</h2> -<p>Rsyslog supports standard sysklogd's configuration file format and extends -it. So in general, you can take a "normal" syslog.conf and use it together with -rsyslogd. It will understand everything. However, to use most of rsyslogd's -unique features, you need to add extended configuration directives.<p>Rsyslogd -supports the classical, selector-based rule lines. They are still at the heart -of it and all actions are initiated via rule lines. A rule lines is any line not -starting with a $ or the comment sign (#). Lines starting with $ carry -rsyslog-specific directives.<p>Every rule line consists of two fields, a selector field and an action field. -These two fields are separated by one or more spaces or tabs. The selector field -specifies a pattern of facilities and priorities belonging to the specified -action.<br> -<br> -Lines starting with a hash mark ("#'') and empty lines are ignored. - +<p>Rsyslog supports standard sysklogd's configuration file format +and extends it. So in general, you can take a "normal" syslog.conf and +use it together with rsyslogd. It will understand everything. However, +to use most of rsyslogd's unique features, you need to add extended +configuration directives.</p> +<p>Rsyslogd supports the classical, selector-based rule lines. +They are still at the heart of it and all actions are initiated via +rule lines. A rule lines is any line not starting with a $ or the +comment sign (#). Lines starting with $ carry rsyslog-specific +directives.</p> +<p>Every rule line consists of two fields, a selector field and +an action field. These two fields are separated by one or more spaces +or tabs. The selector field specifies a pattern of facilities and +priorities belonging to the specified action.<br> +<br> +Lines starting with a hash mark ("#'') and empty lines are ignored. +</p> <h2>Templates</h2> -<p>Templates are a key feature of rsyslog. They allow to specify any format a user -might want. They are also used for dynamic file name generation. Every output in rsyslog uses templates - this holds true for files, -user messages and so on. The database writer expects its template to be a proper -SQL statement - so this is highly customizable too. You might ask how does all -of this work when no templates at all are specified. Good question ;) The answer -is simple, though. Templates compatible with the stock syslogd formats are -hardcoded into rsyslogd. So if no template is specified, we use one of these -hardcoded templates. Search for "template_" in syslogd.c and you will find the +<p>Templates are a key feature of rsyslog. They allow to specify +any +format a user might want. They are also used for dynamic file name +generation. Every output in rsyslog uses templates - this holds true +for files, user messages and so on. The database writer expects its +template to be a proper SQL statement - so this is highly customizable +too. You might ask how does all of this work when no templates at all +are specified. Good question ;) The answer is simple, though. Templates +compatible with the stock syslogd formats are hardcoded into rsyslogd. +So if no template is specified, we use one of these hardcoded +templates. Search for "template_" in syslogd.c and you will find the hardcoded ones.</p> -<p>A template consists of a template directive, a name, the actual template text -and optional options. A sample is:</p> -<blockquote><code>$template MyTemplateName,"\7Text %property% some more text\n",<options></code></blockquote> -<p>The "$template" is the template directive. It tells rsyslog that this line -contains a template. "MyTemplateName" is the template name. All -other config lines refer to this name. The text within quotes is the actual -template text. The backslash is an escape character, much as it is in C. It does -all these "cool" things. For example, \7 rings the bell (this is an ASCII -value), \n is a new line. C programmers and perl coders have the advantage of -knowing this, but the set in rsyslog is a bit restricted currently. -<p> -All text in the template is used literally, except for things within percent -signs. These are properties and allow you access to the contents of the syslog -message. Properties are accessed via the property replacer (nice name, huh) and -it can do cool things, too. For example, it can pick a substring or do -date-specific formatting. More on this is below, on some lines of the property -replacer.<br> -<br> -The <options> part is optional. It carries options influencing the template as -whole. See details below. Be sure NOT to mistake template options with property -options - the later ones are processed by the property replacer and apply to a -SINGLE property, only (and not the whole template).<br> +<p>A template consists of a template directive, a name, the +actual template text and optional options. A sample is:</p> +<blockquote><code>$template MyTemplateName,"\7Text +%property% some more text\n",<options></code></blockquote> +<p>The "$template" is the template directive. It tells rsyslog +that this line contains a template. "MyTemplateName" is the template +name. All +other config lines refer to this name. The text within quotes is the +actual template text. The backslash is an escape character, much as it +is in C. It does all these "cool" things. For example, \7 rings the +bell (this is an ASCII value), \n is a new line. C programmers and perl +coders have the advantage of knowing this, but the set in rsyslog is a +bit restricted currently. +</p> +<p>All text in the template is used literally, except for things +within percent signs. These are properties and allow you access to the +contents of the syslog message. Properties are accessed via the +property replacer (nice name, huh) and it can do cool things, too. For +example, it can pick a substring or do date-specific formatting. More +on this is below, on some lines of the property replacer.<br> +<br> +The <options> part is optional. It carries options +influencing the template as whole. See details below. Be sure NOT to +mistake template options with property options - the later ones are +processed by the property replacer and apply to a SINGLE property, only +(and not the whole template).<br> <br> Template options are case-insensitive. Currently defined are: </p> -<p><b>sql</b> - format the string suitable for a SQL statement in MySQL format. This will -replace single quotes ("'") and the backslash character by their -backslash-escaped counterpart ("\'" and "\\") inside each field. Please note -that in MySQL configuration, the <code class="literal">NO_BACKSLASH_ESCAPES</code> +<p><b>sql</b> - format the string suitable for a SQL +statement in MySQL format. This will replace single quotes ("'") and +the backslash character by their backslash-escaped counterpart ("\'" +and "\\") inside each field. Please note that in MySQL configuration, +the <code class="literal">NO_BACKSLASH_ESCAPES</code> mode must be turned off for this format to work (this is the default).</p> -<p><b>stdsql</b> - format the string suitable for a SQL statement that is to be -sent to a standards-compliant sql server. This will -replace single quotes ("'") by two single quotes ("''") inside each field. -You must use stdsql together with MySQL if in MySQL configuration the -<code class="literal">NO_BACKSLASH_ESCAPES</code> is turned on.</p> -<p>Either the <b>sql</b> or <b>stdsql</b> -option <b>must</b> be specified when a template is used for writing to a database, -otherwise injection might occur. Please note that due to the unfortunate fact -that several vendors have violated the sql standard and introduced their own -escape methods, it is impossible to have a single option doing all the work. -So you yourself must make sure you are using the right format. <b>If you choose +<p><b>stdsql</b> - format the string suitable for a +SQL statement that is to be sent to a standards-compliant sql server. +This will replace single quotes ("'") by two single quotes ("''") +inside each field. You must use stdsql together with MySQL if in MySQL +configuration the +<code class="literal">NO_BACKSLASH_ESCAPES</code> is +turned on.</p> +<p>Either the <b>sql</b> or <b>stdsql</b> +option <b>must</b> be specified when a template is used +for writing to a database, otherwise injection might occur. Please note +that due to the unfortunate fact that several vendors have violated the +sql standard and introduced their own escape methods, it is impossible +to have a single option doing all the work. So you yourself +must make sure you are using the right format. <b>If you choose the wrong one, you are still vulnerable to sql injection.</b><br> <br> -Please note that the database writer *checks* that the sql option is present in -the template. If it is not present, the write database action is disabled. This -is to guard you against accidental forgetting it and then becoming vulnerable -to SQL injection. The sql option can also be useful with files - especially if -you want to import them into a database on another machine for performance -reasons. However, do NOT use it if you do not have a real need for it - among -others, it takes some toll on the processing time. Not much, but on a really -busy system you might notice it ;)</p> -<p>The default template for the write to database action has the sql option set. -As we currently support only MySQL and the sql option matches the default MySQL -configuration, this is a good choice. However, if you have turned on -<code class="literal">NO_BACKSLASH_ESCAPES</code> in your MySQL config, you need -to supply a template with the stdsql option. Otherwise you will become -vulnerable to SQL injection. <br> +Please note that the database writer *checks* that the sql option is +present in the template. If it is not present, the write database +action is disabled. This is to guard you against accidental forgetting +it and then becoming vulnerable to SQL injection. The sql option can +also be useful with files - especially if you want to import them into +a database on another machine for performance reasons. However, do NOT +use it if you do not have a real need for it - among others, it takes +some toll on the processing time. Not much, but on a really busy system +you might notice it ;)</p> +<p>The default template for the write to database action has the +sql option set. As we currently support only MySQL and the sql option +matches the default MySQL configuration, this is a good choice. +However, if you have turned on +<code class="literal">NO_BACKSLASH_ESCAPES</code> in +your MySQL config, you need to supply a template with the stdsql +option. Otherwise you will become vulnerable to SQL injection. <br> <br> To escape:<br> % = \%<br> \ = \\ --> '\' is used to escape (as in C)<br> -$template TraditionalFormat,%timegenerated% %HOSTNAME% %syslogtag%%msg%\n"<br> -<br> -Properties can be accessed by the <a href="property_replacer.html">property replacer</a> -(see there for details).</p> -<p><b>Please note that as of 1.15.0, templates can also by used to generate -selector lines with dynamic file names.</b> For example, if you would like to -split syslog messages from different hosts to different files (one per host), -you can define the following template:</p> -<blockquote><code>$template DynFile,"/var/log/system-%HOSTNAME%.log"</code></blockquote> -<p>This template can then be used when defining an output selector line. It will -result in something like "/var/log/system-localhost.log"</p> +$template TraditionalFormat,%timegenerated% %HOSTNAME% +%syslogtag%%msg%\n"<br> +<br> +Properties can be accessed by the <a href="property_replacer.html">property +replacer</a> (see there for details).</p> +<p><b>Please note that templates can also by +used to generate selector lines with dynamic file names.</b> For +example, if you would like to split syslog messages from different +hosts to different files (one per host), you can define the following +template:</p> +<blockquote><code>$template +DynFile,"/var/log/system-%HOSTNAME%.log"</code></blockquote> +<p>This template can then be used when defining an output +selector line. It will result in something like +"/var/log/system-localhost.log"</p><p>Template +names beginning with "RSYSLOG_" are reserved for rsyslog use. Do NOT +use them if, otherwise you may receive a conflict in the future (and +quite unpredictable behaviour). There is a small set of pre-defined +templates that you can use without the need to define it:</p><ul><li><span style="font-weight: bold;">RSYSLOG_TraditionalFileFormat</span> - the "old style" default log file format with low-precision timestamps</li><li><span style="font-weight: bold;">RSYSLOG_FileFormat</span> - a modern-style logfile format similar to TraditionalFileFormat, buth with high-precision timestamps and timezone information</li><li><span style="font-weight: bold;">RSYSLOG_TraditionalForwardFormat</span> +- the traditional forwarding format with low-precision timestamps. Most +useful if you send messages to other syslogd's or rsyslogd below +version 3.12.5.</li><li><span style="font-weight: bold;">RSYSLOG_ForwardFormat</span> +- a new high-precision forwarding format very similar to the +traditional one, but with high-precision timestamps and timezone +information. Recommended to be used when sending messages to rsyslog +3.12.5 or above.</li><li><span style="font-weight: bold;">RSYSLOG_SyslogProtocol23Format</span> +- the format specified in IETF's internet-draft +ietf-syslog-protocol-23, which is assumed to be come the new syslog +standard RFC. This format includes several improvements. The rsyslog +message parser understands this format, so you can use it together with +all relatively recent versions of rsyslog. Other syslogd's may get +hopelessly confused if receiving that format, so check before you use +it. Note that the format is unlikely to change when the final RFC comes +out, but this may happen.</li></ul> <h2>Output Channels</h2> -<p>Output Channels are a new concept first introduced in rsyslog 0.9.0. <b>As of this -writing, it is most likely that they will be replaced by something different in -the future.</b> So if you -use them, be prepared to change you configuration file syntax when you upgrade -to a later release.<br> -<br> -The idea behind output channel definitions is that it shall provide an umbrella -for any type of output that the user might want. In essence,<br> -this is the "file" part of selector lines (and this is why we are not sure -output channel syntax will stay after the next review). There is a<br> -difference, though: selector channels both have filter conditions (currently -facility and severity) as well as the output destination. Output channels define -the output definition, only. As of this build, they can only be used to write to -files - not pipes, ttys or whatever else. If we stick with output channels, this -will change over time.</p> -<p>In concept, an output channel includes everything needed to know about an -output actions. In practice, the current implementation only carries<br> -a filename, a maximum file size and a command to be issued when this file size -is reached. More things might be present in future version, which might also -change the syntax of the directive.</p> -<p>Output channels are defined via an $outchannel directive. It's syntax is as -follows:<br> +<p>Output Channels are a new concept first introduced in rsyslog +0.9.0. <b>As of this writing, it is most likely that they will +be replaced by something different in the future.</b> So if you +use them, be prepared to change you configuration file syntax when you +upgrade to a later release.<br> +<br> +The idea behind output channel definitions is that it shall provide an +umbrella for any type of output that the user might want. In essence,<br> +this is the "file" part of selector lines (and this is why we are not +sure output channel syntax will stay after the next review). There is a<br> +difference, though: selector channels both have filter conditions +(currently facility and severity) as well as the output destination. +Output channels define the output definition, only. As of this build, +they can only be used to write to files - not pipes, ttys or whatever +else. If we stick with output channels, this will change over time.</p> +<p>In concept, an output channel includes everything needed to +know about an output actions. In practice, the current implementation +only carries<br> +a filename, a maximum file size and a command to be issued when this +file size is reached. More things might be present in future version, +which might also change the syntax of the directive.</p> +<p>Output channels are defined via an $outchannel directive. It's +syntax is as follows:<br> <br> $outchannel name,file-name,max-size,action-on-max-size<br> <br> -name is the name of the output channel (not the file), file-name is the file -name to be written to, max-size the maximum allowed size and action-on-max-size -a command to be issued when the max size is reached. This command always has -exactly one parameter. The binary is that part of action-on-max-size before the -first space, its parameter is everything behind that space.<br> -<br> -Please note that max-size is queried BEFORE writing the log message to the file. -So be sure to set this limit reasonably low so that any message might fit. For -the current release, setting it 1k lower than you expected is helpful. The -max-size must always be specified in bytes - there are no special symbols (like -1k, 1m,...) at this point of development.<br> -<br> -Keep in mind that $outchannel just defines a channel with "name". It does not -activate it. To do so, you must use a selector line (see below). That selector -line includes the channel name plus an $ sign in front of it. A sample might be:<br> +name is the name of the output channel (not the file), file-name is the +file name to be written to, max-size the maximum allowed size and +action-on-max-size a command to be issued when the max size is reached. +This command always has exactly one parameter. The binary is that part +of action-on-max-size before the first space, its parameter is +everything behind that space.<br> +<br> +Please note that max-size is queried BEFORE writing the log message to +the file. So be sure to set this limit reasonably low so that any +message might fit. For the current release, setting it 1k lower than +you expected is helpful. The max-size must always be specified in bytes +- there are no special symbols (like 1k, 1m,...) at this point of +development.<br> +<br> +Keep in mind that $outchannel just defines a channel with "name". It +does not activate it. To do so, you must use a selector line (see +below). That selector line includes the channel name plus an $ sign in +front of it. A sample might be:<br> <br> *.* $mychannel<br> <br> -In its current form, output channels primarily provide the ability to size-limit -an output file. To do so, specify a maximum size. When this size is reached, -rsyslogd will execute the action-on-max-size command and then reopen the file -and retry. The command should be something like a log rotation script or a -similar thing.</p> -<p>If there is no action-on-max-size command or the command did not resolve the -situation, the file is closed and never reopened by rsyslogd (except, of course, -by huping it). This logic was integrated when we first experienced severe issues -with files larger 2gb, which could lead to rsyslogd dumping core. In such cases, -it is more appropriate to stop writing to a single file. Meanwhile, rsyslogd has -been fixed to support files larger 2gb, but obviously only on file systems and -operating system versions that do so. So it can still make sense to enforce a -2gb file size limit.</p> +In its current form, output channels primarily provide the ability to +size-limit an output file. To do so, specify a maximum size. When this +size is reached, rsyslogd will execute the action-on-max-size command +and then reopen the file and retry. The command should be something +like a <a href="log_rotation_fix_size.html">log rotation +script</a> or a similar thing.</p> +<p>If there is no action-on-max-size command or the command did +not resolve the situation, the file is closed and never reopened by +rsyslogd (except, of course, by huping it). This logic was integrated +when we first experienced severe issues with files larger 2gb, which +could lead to rsyslogd dumping core. In such cases, it is more +appropriate to stop writing to a single file. Meanwhile, rsyslogd has +been fixed to support files larger 2gb, but obviously only on file +systems and operating system versions that do so. So it can still make +sense to enforce a 2gb file size limit.</p> <h2>Filter Conditions</h2> -<p>Rsyslog offers two different types "filter conditions":</p> +<p>Rsyslog offers four different types "filter conditions":</p> <ul> - <li>"traditional" severity and facility based selectors</li> - <li>property-based filters</li> +<li>BSD-style blocks</li> +<li>"traditional" severity and facility based selectors</li> +<li>property-based filters</li> +<li>expression-based filters</li> </ul> <h3>Blocks</h3> -<p>Rsyslogd supports BSD-style blocks inside rsyslog.conf. Each block of lines -is separated from the previous block by a program or hostname specification. A -block will only log messages corresponding to the most recent program and -hostname specifications given. Thus, a block which selects ‘ppp’ as the program, -directly followed by a block that selects messages from the hostname ‘dialhost’, -then the second block will only log messages from the ppp program on dialhost. +<p>Rsyslogd supports BSD-style blocks inside rsyslog.conf. Each +block of lines is separated from the previous block by a program or +hostname specification. A block will only log messages corresponding to +the most recent program and hostname specifications given. Thus, a +block which selects ‘ppp’ as the program, directly followed by a block +that selects messages from the hostname ‘dialhost’, then the second +block will only log messages from the ppp program on dialhost. </p> -<p>A program specification is a line beginning with ‘!prog’ and the following -blocks will be associated with calls to syslog from that specific program. A -program specification for ‘foo’ will also match any message logged by the kernel -with the prefix ‘foo: ’. Alternatively, a program specification ‘-foo’ causes the -following blocks to be applied to messages from any program but the one specified. - -A hostname specification of the form ‘+hostname’ and -the following blocks will be applied to messages received from the specified -hostname. Alternatively, a hostname specification ‘-hostname’ causes the -following blocks to be applied to messages from any host but the one specified. - -If the hostname is given as ‘@’, the local hostname will be used. (NOT YET -IMPLEMENTED) A program or hostname specification may be reset by giving the -program or hostname as ‘*’.</p> -<p>Please note that the "#!prog", "#+hostname" and "#-hostname" syntax available -in BSD syslogd is not supported by rsyslogd. By default, no hostname or program -is set.</p> +<p>A program specification is a line beginning with ‘!prog’ and +the following blocks will be associated with calls to syslog from that +specific program. A program specification for ‘foo’ will also match any +message logged by the kernel with the prefix ‘foo: ’. Alternatively, a +program specification ‘-foo’ causes the following blocks to be applied +to messages from any program but the one specified. A hostname +specification of the form ‘+hostname’ and the following blocks will be +applied to messages received from the specified hostname. +Alternatively, a hostname specification ‘-hostname’ causes the +following blocks to be applied to messages from any host but the one +specified. If the hostname is given as ‘@’, the local hostname will be +used. (NOT YET IMPLEMENTED) A program or hostname specification may be +reset by giving the program or hostname as ‘*’.</p> +<p>Please note that the "#!prog", "#+hostname" and "#-hostname" +syntax available in BSD syslogd is not supported by rsyslogd. By +default, no hostname or program is set.</p> <h3>Selectors</h3> -<p><b>Selectors are the traditional way of filtering syslog messages.</b> They -have been kept in rsyslog with their original syntax, because it is well-known, -highly effective and also needed for compatibility with stock syslogd -configuration files. If you just need to filter based on priority and facility, -you should do this with selector lines. They are <b>not</b> second-class -citizens in rsyslog and offer the best performance for this job.</p> -<p>The selector field itself again consists of two parts, a facility and a -priority, separated by a period (``.''). Both parts are case insensitive and can -also be specified as decimal numbers, but don't do that, you have been warned. -Both facilities and priorities are described in rsyslog(3). The names mentioned -below correspond to the similar LOG_-values in /usr/include/rsyslog.h.<br><br>The facility is one of the following keywords: auth, authpriv, cron, daemon, -kern, lpr, mail, mark, news, security (same as auth), syslog, user, uucp and -local0 through local7. The keyword security should not be used anymore and mark -is only for internal use and therefore should not be used in applications. -Anyway, you may want to specify and redirect these messages here. The facility -specifies the subsystem that produced the message, i.e. all mail programs log -with the mail facility (LOG_MAIL) if they log using syslog.<br><br>Please note that the upcoming next syslog-RFC specifies many more facilities. -Support for them will be added in a future version of rsyslog, which might -require changes to existing configuration files.<br><br>The priority is one of the following keywords, in ascending order: debug, info, -notice, warning, warn (same as warning), err, error (same as err), crit, alert, -emerg, panic (same as emerg). The keywords error, warn and panic are deprecated -and should not be used anymore. The priority defines the severity of the message<br> -<br>The behavior of the original BSD syslogd is that all messages of the specified -priority and higher are logged according to the given action. Rsyslogd behaves the same, but has some extensions.<br><br>In addition to the above mentioned names the rsyslogd(8) understands the -following extensions: An asterisk (``*'') stands for all facilities or all -priorities, depending on where it is used (before or after the period). The -keyword none stands for no priority of the given facility.<br><br>You can specify multiple facilities with the same priority pattern in one -statement using the comma (``,'') operator. You may specify as much facilities -as you want. Remember that only the facility part from such a statement is -taken, a priority part would be skipped.</p> -<p>Multiple selectors may be specified for a single action using the semicolon -(``;'') separator. Remember that each selector in the selector field is capable -to overwrite the preceding ones. Using this behavior you can exclude some -priorities from the pattern.</p> -<p>Rsyslogd has a syntax extension to the original BSD source, that makes its -use more intuitively. You may precede every priority with an equation sign -(``='') to specify only this single priority and not any of the above. You may -also (both is valid, too) precede the priority with an exclamation mark (``!'') -to ignore all that priorities, either exact this one or this and any higher -priority. If you use both extensions than the exclamation mark must occur before -the equation sign, just use it intuitively.</p> +<p><b>Selectors are the traditional way of filtering syslog +messages.</b> They have been kept in rsyslog with their original +syntax, because it is well-known, highly effective and also needed for +compatibility with stock syslogd configuration files. If you just need +to filter based on priority and facility, you should do this with +selector lines. They are <b>not</b> second-class citizens +in rsyslog and offer the best performance for this job.</p> +<p>The selector field itself again consists of two parts, a +facility and a priority, separated by a period (".''). Both parts are +case insensitive and can also be specified as decimal numbers, but +don't do that, you have been warned. Both facilities and priorities are +described in rsyslog(3). The names mentioned below correspond to the +similar LOG_-values in /usr/include/rsyslog.h.<br> +<br> +The facility is one of the following keywords: auth, authpriv, cron, +daemon, kern, lpr, mail, mark, news, security (same as auth), syslog, +user, uucp and local0 through local7. The keyword security should not +be used anymore and mark is only for internal use and therefore should +not be used in applications. Anyway, you may want to specify and +redirect these messages here. The facility specifies the subsystem that +produced the message, i.e. all mail programs log with the mail facility +(LOG_MAIL) if they log using syslog.<br> +<br> +The priority is one of the following keywords, in ascending order: +debug, info, notice, warning, warn (same as warning), err, error (same +as err), crit, alert, emerg, panic (same as emerg). The keywords error, +warn and panic are deprecated and should not be used anymore. The +priority defines the severity of the message.<br> +<br> +The behavior of the original BSD syslogd is that all messages of the +specified priority and higher are logged according to the given action. +Rsyslogd behaves the same, but has some extensions.<br> +<br> +In addition to the above mentioned names the rsyslogd(8) understands +the following extensions: An asterisk ("*'') stands for all facilities +or all priorities, depending on where it is used (before or after the +period). The keyword none stands for no priority of the given facility.<br> +<br> +You can specify multiple facilities with the same priority pattern in +one statement using the comma (",'') operator. You may specify as much +facilities as you want. Remember that only the facility part from such +a statement is taken, a priority part would be skipped.</p> +<p>Multiple selectors may be specified for a single action using +the semicolon (";'') separator. Remember that each selector in the +selector field is capable to overwrite the preceding ones. Using this +behavior you can exclude some priorities from the pattern.</p> +<p>Rsyslogd has a syntax extension to the original BSD source, +that makes its use more intuitively. You may precede every priority +with an equation sign ("='') to specify only this single priority and +not any of the above. You may also (both is valid, too) precede the +priority with an exclamation mark ("!'') to ignore all that +priorities, either exact this one or this and any higher priority. If +you use both extensions than the exclamation mark must occur before the +equation sign, just use it intuitively.</p> <h3>Property-Based Filters</h3> -<p>Property-based filters are unique to rsyslogd. They allow to filter on any -property, like HOSTNAME, syslogtag and msg. A list of all currently-supported -properties can be found in the <a href="property_replacer.html">property -replacer documentation</a> (but keep in mind that only the properties, not the -replacer is supported). With this filter, each properties can be checked against -a specified value, using a specified compare operation. Currently, there is only -a single compare operation (contains) available, but additional operations will be added in the -future.</p> -<p>A property-based filter must start with a colon in column 0. This tells -rsyslogd that it is the new filter type. The colon must be followed by the -property name, a comma, the name of the compare operation to carry out, another -comma and then the value to compare against. This value must be quoted. There -can be spaces and tabs between the commas. Property names and compare operations -are case-sensitive, so "msg" works, while "MSG" is an invalid property name. In -brief, the syntax is as follows:</p> -<p><code><b>:property, [!]compare-operation, "value"</b></code></p> -<p>The following <b>compare-operations</b> are currently supported:</p> -<table border="1" width="100%" id="table1"> - <tr> - <td>contains</td> - <td>Checks if the string provided in value is contained in the property. - There must be an exact match, wildcards are not supported.</td> - </tr> - <tr> - <td>isequal</td> - <td>Compares the "value" string provided and the property contents. - These two values must be exactly equal to match. The difference to - contains is that contains searches for the value anywhere inside the - property value, whereas all characters must be identical for isequal. As - such, isequal is most useful for fields like syslogtag or FROMHOST, - where you probably know the exact contents.</td> - </tr> - <tr> - <td>startswith</td> - <td>Checks if the value is found exactly at the beginning of the - property value. For example, if you search for "val" with<p><code><b>:msg, - startswith, "val"</b></code></p> - <p>it will be a match if msg contains "values are in this message" but - it won't match if the msg contains "There are values in this message" - (in the later case, contains would match). Please note that "startswith" - is by far faster than regular expressions. So even once they are - implemented, it can make very much sense (performance-wise) to use "startswith".</td> - </tr> - <tr> - <td>regex</td> - <td>Compares the property against the provided regular expression.</td> - </tr> +<p>Property-based filters are unique to rsyslogd. They allow to +filter on any property, like HOSTNAME, syslogtag and msg. A list of all +currently-supported properties can be found in the <a href="property_replacer.html">property replacer documentation</a> +(but keep in mind that only the properties, not the replacer is +supported). With this filter, each properties can be checked against a +specified value, using a specified compare operation. Currently, there +is only a single compare operation (contains) available, but additional +operations will be added in the future.</p> +<p>A property-based filter must start with a colon in column 0. +This tells rsyslogd that it is the new filter type. The colon must be +followed by the property name, a comma, the name of the compare +operation to carry out, another comma and then the value to compare +against. This value must be quoted. There can be spaces and tabs +between the commas. Property names and compare operations are +case-sensitive, so "msg" works, while "MSG" is an invalid property +name. In brief, the syntax is as follows:</p> +<p><code><b>:property, [!]compare-operation, "value"</b></code></p> +<p>The following <b>compare-operations</b> are +currently supported:</p> +<table id="table1" border="1" width="100%"> +<tbody> +<tr> +<td>contains</td> +<td>Checks if the string provided in value is contained in +the property. There must be an exact match, wildcards are not supported.</td> +</tr> +<tr> +<td>isequal</td> +<td>Compares the "value" string provided and the property +contents. These two values must be exactly equal to match. The +difference to contains is that contains searches for the value anywhere +inside the property value, whereas all characters must be identical for +isequal. As such, isequal is most useful for fields like syslogtag or +FROMHOST, where you probably know the exact contents.</td> +</tr> +<tr> +<td>startswith</td> +<td>Checks if the value is found exactly at the beginning +of the property value. For example, if you search for "val" with +<p><code><b>:msg, startswith, "val"</b></code></p> +<p>it will be a match if msg contains "values are in this +message" but it won't match if the msg contains "There are values in +this message" (in the later case, contains would match). Please note +that "startswith" is by far faster than regular expressions. So even +once they are implemented, it can make very much sense +(performance-wise) to use "startswith".</p> +</td> +</tr> +<tr> +<td>regex</td> +<td>Compares the property against the provided POSIX regular +expression.</td> +</tr> +</tbody> </table> -<p>You can use the bang-character (!) immediately in front of a -compare-operation, the outcome of this operation is negated. For example, if msg -contains "This is an informative message", the following sample would not match:</p> -<p><code><b>:msg, contains, "error"</b></code></p> +<p>You can use the bang-character (!) immediately in front of a +compare-operation, the outcome of this operation is negated. For +example, if msg contains "This is an informative message", the +following sample would not match:</p> +<p><code><b>:msg, contains, "error"</b></code></p> <p>but this one matches:</p> -<p><code><b>:msg, !contains, "error"</b></code></p> -<p>Using negation can be useful if you would like to do some generic processing -but exclude some specific events. You can use the discard action in conjunction -with that. A sample would be:</p> -<p><code><b>*.* /var/log/allmsgs-including-informational.log<br> -:msg, contains, "informational" <font color="#FF0000" size="4">~</font> -<br>*.* /var/log/allmsgs-but-informational.log</b></code></p> -<p>Do not overlook the red tilde in line 2! In this sample, all messages are -written to the file allmsgs-including-informational.log. Then, all messages -containing the string "informational" are discarded. That means the config file -lines below the "discard line" (number 2 in our sample) will not be applied to -this message. Then, all remaining lines will also be written to the file -allmsgs-but-informational.log.</p> -<p><b>Value</b> is a quoted string. It supports some escape sequences:</p> -<p>\" - the quote character (e.g. "String with \"Quotes\"")<br> -\\ - the backslash character (e.g. "C:\\tmp")</p> -<p>Escape sequences always start with a backslash. Additional escape sequences -might be added in the future. Backslash characters <b>must</b> be escaped. Any -other sequence then those outlined above is invalid and may lead to -unpredictable results.</p> -<p>Probably, "msg" is the most prominent use case of property based filters. It -is the actual message text. If you would like to filter based on some message -content (e.g. the presence of a specific code), this can be done easily by:</p> -<p><code><b>:msg, contains, "ID-4711"</b></code></p> -<p>This filter will match when the message contains the string "ID-4711". Please -note that the comparison is case-sensitive, so it would not match if "id-4711" -would be contained in the message.</p> -<p>Getting property-based filters right can sometimes be challenging. In order -to help you do it with as minimal effort as possible, rsyslogd spits out debug -information for all property-based filters during their evaluation. To enable -this, run rsyslogd in foreground and specify the "-d" option.</p> -<p>Boolean operations inside property based filters (like 'message contains -"ID17" or message contains "ID18"') are currently not supported -(except for "not" as outlined above). Please note -that while it is possible to query facility and severity via property-based filters, -it is far more advisable to use classic selectors (see above) for those -cases.</p> +<p><code><b>:msg, !contains, "error"</b></code></p> +<p>Using negation can be useful if you would like to do some +generic processing but exclude some specific events. You can use the +discard action in conjunction with that. A sample would be:</p> +<p><code><b>*.* +/var/log/allmsgs-including-informational.log<br> +:msg, contains, "informational" <font color="#ff0000" size="4">~</font> +<br> +*.* /var/log/allmsgs-but-informational.log</b></code></p> +<p>Do not overlook the red tilde in line 2! In this sample, all +messages are written to the file allmsgs-including-informational.log. +Then, all messages containing the string "informational" are discarded. +That means the config file lines below the "discard line" (number 2 in +our sample) will not be applied to this message. Then, all remaining +lines will also be written to the file allmsgs-but-informational.log.</p> +<p><b>Value</b> is a quoted string. It supports some +escape sequences:</p> +<p>\" - the quote character (e.g. "String with \"Quotes\"")<br> +\\ - the backslash character (e.g. "C:\\tmp")</p> +<p>Escape sequences always start with a backslash. Additional +escape sequences might be added in the future. Backslash characters <b>must</b> +be escaped. Any other sequence then those outlined above is invalid and +may lead to unpredictable results.</p> +<p>Probably, "msg" is the most prominent use case of property +based filters. It is the actual message text. If you would like to +filter based on some message content (e.g. the presence of a specific +code), this can be done easily by:</p> +<p><code><b>:msg, contains, "ID-4711"</b></code></p> +<p>This filter will match when the message contains the string +"ID-4711". Please note that the comparison is case-sensitive, so it +would not match if "id-4711" would be contained in the message.</p> +<p><code><b>:msg, regex, "fatal .* error"</b></code></p> +<p>This filter uses a POSIX regular expression. It matches when the +string contains the words "fatal" and "error" with anything in between +(e.g. "fatal net error" and "fatal lib error" but not "fatal error" as +two spaces are required by the regular expression!).</p><p>Getting property-based filters right can sometimes be +challenging. In order to help you do it with as minimal effort as +possible, rsyslogd spits out debug information for all property-based +filters during their evaluation. To enable this, run rsyslogd in +foreground and specify the "-d" option.</p> +<p>Boolean operations inside property based filters (like +'message contains "ID17" or message contains "ID18"') are currently not +supported (except for "not" as outlined above). Please note that while +it is possible to query facility and severity via property-based +filters, it is far more advisable to use classic selectors (see above) +for those cases.</p> +<h3>Expression-Based Filters</h3> +Expression based filters allow +filtering on arbitrary complex expressions, which can include boolean, +arithmetic and string operations. Expression filters will evolve into a +full configuration scripting language. Unfortunately, their syntax will +slightly change during that process. So if you use them now, you need +to be prepared to change your configuration files some time later. +However, we try to implement the scripting facility as soon as possible +(also in respect to stage work needed). So the window of exposure is +probably not too long.<br> +<br> +Expression based filters are indicated by the keyword "if" in column 1 +of a new line. They have this format:<br> +<br> +if expr then action-part-of-selector-line<br> +<br> +"If" and "then" are fixed keywords that mus be present. "expr" is a +(potentially quite complex) expression. So the <a href="expression.h">expression documentation</a> for +details. "action-part-of-selector-line" is an action, just as you know +it (e.g. "/var/log/logfile" to write to that file).<br> +<br> +A few quick samples:<br> +<br> +<code> +*.* /var/log/file1 # the traditional way<br> +if $msg contains 'error' /var/log/errlog # the expression-based way<br> +</code> +<br> +Right now, you need to specify numerical values if you would like to +check for facilities and severity. These can be found in <a href="http://www.ietf.org/rfc/rfc3164.txt">RFC 3164</a>. +If you don't like that, you can of course also use the textual property +- just be sure to use the right one. As expression support is enhanced, +this will change. For example, if you would like to filter on message +that have facility local0, start with "DEVNAME" and have either +"error1" or "error0" in their message content, you could use the +following filter:<br> +<br> +<code> +if $syslogfacility-text == 'local0' and $msg +startswith 'DEVNAME' and ($msg contains 'error1' or $msg contains +'error0') then /var/log/somelog<br> +</code> +<br> +Please note that the above <span style="font-weight: bold;">must +all be on one line</span>! And if you would like to store all +messages except those that contain "error1" or "error0", you just need +to add a "not":<br> +<br> +<code> +if $syslogfacility-text == 'local0' and $msg +startswith 'DEVNAME' and <span style="font-weight: bold;">not</span> +($msg contains 'error1' or $msg contains +'error0') then /var/log/somelog<br> +</code> +<br>If you would like to do case-insensitive comparisons, use +"contains_i" instead of "contains" and "startswith_i" instead of +"startswith".<br><br>Note that regular expressions are currently NOT +supported in expression-based filters. These will be added later when +function support is added to the expression engine (the reason is that +regular expressions will be a separate loadable module, which requires +some more prequisites before it can be implemented).<br> <h2>ACTIONS</h2> -<p>The action field of a rule describes what to do with the message. In general, -message content is written to a kind of "logfile". But also other actions might -be done, like writing to a database table or forwarding to another host.<br> -<br> -Templates can be used with all actions. If used, the specified template is used -to generate the message content (instead of the default template). To specify a -template, write a semicolon after the action value immediately followed by the -template name.<br> -<br> -Beware: templates MUST be defined BEFORE they are used. It is OK to define some -templates, then use them in selector lines, define more templates and use use -them in the following selector lines. But it is NOT permitted to use a template -in a selector line that is above its definition. If you do this, the action will be ignored.</p> -<p><b>You can have multiple actions for a single selector </b> (or more -precisely a single filter of such a selector line). Each action must be on its -own line and the line must start with an ampersand ('&') character and have no -filters. An example would be</p> +<p>The action field of a rule describes what to do with the +message. In general, message content is written to a kind of "logfile". +But also other actions might be done, like writing to a database table +or forwarding to another host.<br> +<br> +Templates can be used with all actions. If used, the specified template +is used to generate the message content (instead of the default +template). To specify a template, write a semicolon after the action +value immediately followed by the template name.<br> +<br> +Beware: templates MUST be defined BEFORE they are used. It is OK to +define some templates, then use them in selector lines, define more +templates and use use them in the following selector lines. But it is +NOT permitted to use a template in a selector line that is above its +definition. If you do this, the action will be ignored.</p> +<p><b>You can have multiple actions for a single selector </b> (or +more precisely a single filter of such a selector line). Each action +must be on its own line and the line must start with an ampersand +('&') character and have no filters. An example would be</p> <p><code><b>*.=crit rger<br> & root<br> & /var/log/critmsgs</b></code></p> -<p>These three lines send critical messages to the user rger and root and also -store them in /var/log/critmsgs. <b>Using multiple actions per selector is</b> -convenient and also <b>offers a performance benefit</b>. As the filter needs to -be evaluated only once, there is less computation required to process the -directive compared to the otherwise-equal config directives below:</p> +<p>These three lines send critical messages to the user rger and +root and also store them in /var/log/critmsgs. <b>Using multiple +actions per selector is</b> convenient and also <b>offers +a performance benefit</b>. As the filter needs to be evaluated +only once, there is less computation required to process the directive +compared to the otherwise-equal config directives below:</p> <p><code><b>*.=crit rger<br> *.=crit root<br> *.=crit /var/log/critmsgs</b></code></p> <p> </p> <h3>Regular File</h3> -<p>Typically messages are logged to real files. The file has to be specified with -full pathname, beginning with a slash "/''.<br> +<p>Typically messages are logged to real files. The file has to +be specified with full pathname, beginning with a slash "/''.<br> <br> -You may prefix each entry with the minus ``-'' sign to omit syncing the file -after every logging. Note that you might lose information if the system crashes -right behind a write attempt. Nevertheless this might give you back some -performance, especially if you run programs that use +You may prefix each entry with the minus "-'' sign to omit syncing the +file after every logging. Note that you might lose information if the +system crashes right behind a write attempt. Nevertheless this might +give you back some performance, especially if you run programs that use logging in a very verbose manner.</p> -<p>If your system is connected to a reliable UPS and you receive lots of log -data (e.g. firewall logs), it might be a very good idea to turn of -syncing by specifying the "-" in front of the file name. </p> -<p><b>The filename can be either static </b>(always the same) or <b>dynamic</b> -(different based on message received). The later is useful if you would -automatically split messages into different files based on some message -criteria. For example, dynamic file name selectors allow you to split messages -into different files based on the host that sent them. With dynamic file names, -everything is automatic and you do not need any filters. </p> -<p>It works via the template system. First, you define a template for the file -name. An example can be seen above in the description of template. We will use -the "DynFile" template defined there. Dynamic filenames are indicated by -specifying a questions mark "?" instead of a slash, followed by the template -name. Thus, the selector line for our dynamic file name would look as follows:</p> +<p>If your system is connected to a reliable UPS and you receive +lots of log data (e.g. firewall logs), it might be a very good idea to +turn of +syncing by specifying the "-" in front of the file name. </p> +<p><b>The filename can be either static </b>(always +the same) or <b>dynamic</b> (different based on message +received). The later is useful if you would automatically split +messages into different files based on some message criteria. For +example, dynamic file name selectors allow you to split messages into +different files based on the host that sent them. With dynamic file +names, everything is automatic and you do not need any filters. </p> +<p>It works via the template system. First, you define a template +for the file name. An example can be seen above in the description of +template. We will use the "DynFile" template defined there. Dynamic +filenames are indicated by specifying a questions mark "?" instead of a +slash, followed by the template name. Thus, the selector line for our +dynamic file name would look as follows:</p> <blockquote> <code>*.* ?DynFile</code> </blockquote> -<p>That's all you need to do. Rsyslog will now automatically generate file names -for you and store the right messages into the right files. Please note that the -minus sign also works with dynamic file name selectors. Thus, to avoid syncing, -you may use</p> +<p>That's all you need to do. Rsyslog will now automatically +generate file names for you and store the right messages into the right +files. Please note that the minus sign also works with dynamic file +name selectors. Thus, to avoid syncing, you may use</p> <blockquote> <code>*.* -?DynFile</code></blockquote> -<p>And of course you can use templates to specify the output format:</p> +<p>And of course you can use templates to specify the output +format:</p> <blockquote> <code>*.* ?DynFile;MyTemplate</code></blockquote> -<p><b>A word of caution:</b> rsyslog creates files as needed. So if a new host -is using your syslog server, rsyslog will automatically create a new file for -it.</p> - -<p><b>Creating directories is also supported</b>. For example you can use the hostname as directory -and the program name as file name:</p> +<p><b>A word of caution:</b> rsyslog creates files as +needed. So if a new host is using your syslog server, rsyslog will +automatically create a new file for it.</p> +<p><b>Creating directories is also supported</b>. For +example you can use the hostname as directory and the program name as +file name:</p> <blockquote> <code>$template DynFile,"/var/log/%HOSTNAME%/%programname%.log"</code></blockquote> - <h3>Named Pipes</h3> -<p>This version of rsyslogd(8) has support for logging output to named pipes (fifos). -A fifo or named pipe can be used as a destination for log messages by prepending -a pipe symbol (``|'') to the name of the file. This is handy for debugging. Note -that the fifo must be created with the mkfifo(1) command before rsyslogd(8) is -started.</p> +<p>This version of rsyslogd(8) has support for logging output to +named pipes (fifos). A fifo or named pipe can be used as a destination +for log messages by prepending a pipe symbol ("|'') to the name of the +file. This is handy for debugging. Note that the fifo must be created +with the mkfifo(1) command before rsyslogd(8) is started.</p> <h3>Terminal and Console</h3> -<p>If the file you specified is a tty, special tty-handling is done, same with -/dev/console.</p> +<p>If the file you specified is a tty, special tty-handling is +done, same with /dev/console.</p> <h3>Remote Machine</h3> -<p>Rsyslogd provides full remote logging, i.e. is able to send messages to a -remote host running rsyslogd(8) and to receive messages from remote hosts. -Using this feature you're able to control all syslog messages on one host, if -all other machines will log remotely to that. This tears down<br> +<p>Rsyslogd provides full remote logging, i.e. is able to send +messages to a remote host running rsyslogd(8) and to receive messages +from remote hosts. Using this feature you're able to control all syslog +messages on one host, if all other machines will log remotely to that. +This tears down<br> administration needs.<br> <br> -<b>Please note that this version of rsyslogd by default does NOT forward messages -it has received from the network to another host. Specify the "-h" option to enable this.</b></p> -<p>To forward messages to another host, prepend the hostname with the at sign ("@"). -A single at sign means that messages will be forwarded via UDP protocol (the -standard for syslog). If you prepend two at signs ("@@"), the messages will be -transmitted via TCP. Please note that plain TCP based syslog is not officially -standardized, but most major syslogds support it (e.g. syslog-ng or WinSyslog). -The forwarding action indicator (at-sign) can be followed by one or more options. -If they are given, they must be immediately (without a space) following the -final at sign and be enclosed in parenthesis. The individual options must be -separated by commas. The following options are right now defined:</p> -<table border="1" width="100%" id="table2"> - <tr> - <td> - <p align="center"><b>z<number></b></td> - <td>Enable zlib-compression for the message. The <number> is the - compression level. It can be 1 (lowest gain, lowest CPU overhead) to 9 (maximum - compression, highest CPU overhead). The level can also be 0, which means - "no compression". If given, the "z" option is ignored. So this does not - make an awful lot of sense. There is hardly a difference between level 1 - and 9 for typical syslog messages. You can expect a compression gain - between 0% and 30% for typical messages. Very chatty messages may - compress up to 50%, but this is seldom seen with typically traffic. - Please note that rsyslogd checks the compression gain. Messages with 60 - bytes or less will never be compressed. This is because compression gain - is pretty unlikely and we prefer to save CPU cycles. Messages over that - size are always compressed. However, it is checked if there is a gain in - compression and only if there is, the compressed message is transmitted. - Otherwise, the uncompressed messages is transmitted. This saves the - receiver CPU cycles for decompression. It also prevents small message to - actually become larger in compressed form.<p><b>Please note that when a - TCP transport is used, compression will also turn on - syslog-transport-tls framing. See the "o" option for important - information on the implications.</b></p> - <p>Compressed messages are automatically detected and decompressed by - the receiver. There is nothing that needs to be configured on the - receiver side.</td> - </tr> - <tr> - <td> - <p align="center"><b>o</b></td> - <td><b>This option is experimental. Use at your own risk and only if you - know why you need it! If in doubt, do NOT turn it on.</b><p>This option - is only valid for plain TCP based transports. It selects a different - framing based on IETF internet draft syslog-transport-tls-06. This - framing offers some benefits over traditional LF-based framing. However, - the standardization effort is not yet complete. There may be changes in - upcoming versions of this standard. Rsyslog will be kept in line with - the standard. There is some chance that upcoming changes will be - incompatible to the current specification. In this case, all systems - using -transport-tls framing must be upgraded. There will be no effort - made to retain compatibility between different versions of rsyslog. The - primary reason for that is that it seems technically impossible to - provide compatibility between some of those changes. So you should take - this note very serious. It is not something we do not *like* to do (and - may change our mind if enough people beg...), it is something we most - probably *can not* do for technical reasons (aka: you can beg as much as - you like, it won't change anything...).</p> - <p>The most important implication is that compressed syslog messages via - TCP must be considered with care. Unfortunately, it is technically - impossible to transfer compressed records over traditional syslog plain - tcp transports, so you are left with two evil choices...</td> - </tr> +<b>Please note that this version of rsyslogd by default does NOT +forward messages it has received from the network to another host. +Specify the "-h" option to enable this.</b></p> +<p>To forward messages to another host, prepend the hostname with +the at sign ("@"). A single at sign means that messages will +be forwarded via UDP protocol (the standard for syslog). If you prepend +two at signs ("@@"), the messages will be transmitted via TCP. Please +note that plain TCP based syslog is not officially standardized, but +most major syslogds support it (e.g. syslog-ng or WinSyslog). The +forwarding action indicator (at-sign) can be followed by one or more +options. If they are given, they must be immediately (without a space) +following the final at sign and be enclosed in parenthesis. The +individual options must be separated by commas. The following options +are right now defined:</p> +<table id="table2" border="1" width="100%"> +<tbody> +<tr> +<td> +<p align="center"><b>z<number></b></p> +</td> +<td>Enable zlib-compression for the message. The +<number> is the compression level. It can be 1 (lowest +gain, lowest CPU overhead) to 9 (maximum compression, highest CPU +overhead). The level can also be 0, which means "no compression". If +given, the "z" option is ignored. So this does not make an awful lot of +sense. There is hardly a difference between level 1 and 9 for typical +syslog messages. You can expect a compression gain between 0% and 30% +for typical messages. Very chatty messages may compress up to 50%, but +this is seldom seen with typically traffic. Please note that rsyslogd +checks the compression gain. Messages with 60 bytes or less will never +be compressed. This is because compression gain is pretty unlikely and +we prefer to save CPU cycles. Messages over that size are always +compressed. However, it is checked if there is a gain in compression +and only if there is, the compressed message is transmitted. Otherwise, +the uncompressed messages is transmitted. This saves the receiver CPU +cycles for decompression. It also prevents small message to actually +become larger in compressed form. +<p><b>Please note that when a TCP transport is used, +compression will also turn on syslog-transport-tls framing. See the "o" +option for important information on the implications.</b></p> +<p>Compressed messages are automatically detected and +decompressed by the receiver. There is nothing that needs to be +configured on the receiver side.</p> +</td> +</tr> +<tr> +<td> +<p align="center"><b>o</b></p> +</td> +<td><b>This option is experimental. Use at your own +risk and only if you know why you need it! If in doubt, do NOT turn it +on.</b> +<p>This option is only valid for plain TCP based +transports. It selects a different framing based on IETF internet draft +syslog-transport-tls-06. This framing offers some benefits over +traditional LF-based framing. However, the standardization effort is +not yet complete. There may be changes in upcoming versions of this +standard. Rsyslog will be kept in line with the standard. There is some +chance that upcoming changes will be incompatible to the current +specification. In this case, all systems using -transport-tls framing +must be upgraded. There will be no effort made to retain compatibility +between different versions of rsyslog. The primary reason for that is +that it seems technically impossible to provide compatibility between +some of those changes. So you should take this note very serious. It is +not something we do not *like* to do (and may change our mind if enough +people beg...), it is something we most probably *can not* do for +technical reasons (aka: you can beg as much as you like, it won't +change anything...).</p> +<p>The most important implication is that compressed syslog +messages via TCP must be considered with care. Unfortunately, it is +technically impossible to transfer compressed records over traditional +syslog plain tcp transports, so you are left with two evil choices...</p> +</td> +</tr> +</tbody> </table> <p><br> The hostname may be followed by a colon and the destination port.</p> <p>The following is an example selector line with forwarding:</p> <p>*.* @@(o,z9)192.168.0.1:1470</p> -<p>In this example, messages are forwarded via plain TCP with experimental -framing and maximum compression to the host 192.168.0.1 at port 1470.</p> +<p>In this example, messages are forwarded via plain TCP with +experimental framing and maximum compression to the host 192.168.0.1 at +port 1470.</p> <p>*.* @192.168.0.1</p> -<p>In the example above, messages are forwarded via UDP to the machine -192.168.0.1, the destination port defaults to 514. Messages will not be -compressed.</p> -<p><b>Note to sysklogd users:</b> sysklogd does <b>not</b> support RFC 3164 -format, which is the default forwarding template in rsyslog. As such, you will -experience duplicate hostnames if rsyslog is the sender and sysklogd is the -receiver. The fix is simple: you need to use a different template. Use that one:</p> -<p class="MsoPlainText">$template sysklogd,"<%PRI%>%TIMESTAMP% -%syslogtag%%msg%\""<br> +<p>In the example above, messages are forwarded via UDP to the +machine 192.168.0.1, the destination port defaults to 514. Messages +will not be compressed.</p> +<p><b>Note to sysklogd users:</b> sysklogd does <b>not</b> +support RFC 3164 format, which is the default forwarding template in +rsyslog. As such, you will experience duplicate hostnames if rsyslog is +the sender and sysklogd is the receiver. The fix is simple: you need to +use a different template. Use that one:</p> +<p class="MsoPlainText">$template +sysklogd,"<%PRI%>%TIMESTAMP% %syslogtag%%msg%\""<br> *.* @192.168.0.1;sysklogd</p> <h3>List of Users</h3> -<p>Usually critical messages are also directed to ``root'' on that machine. You can -specify a list of users that shall get the message by simply writing the login. -You may specify more than one user by separating them with commas (",''). If -they're logged in they get the message. Don't think a mail would be sent, that -might be too late.</p> +<p>Usually critical messages are also directed to "root'' on +that machine. You can specify a list of users that shall get the +message by simply writing the login. You may specify more than one user +by separating them with commas (",''). If they're logged in they get +the message. Don't think a mail would be sent, that might be too late.</p> <h3>Everyone logged on</h3> -<p>Emergency messages often go to all users currently online to notify them that -something strange is happening with the system. To specify this wall(1)-feature -use an asterisk ("*'').</p> +<p>Emergency messages often go to all users currently online to +notify them that something strange is happening with the system. To +specify this wall(1)-feature use an asterisk ("*'').</p> <h3>Call Plugin</h3> -<p>This is a generic way to call an output plugin. The plugin must support this -functionality. Actual parameters depend on the module, so see the module's doc -on what to supply. The general syntax is as follows:</p> +<p>This is a generic way to call an output plugin. The plugin +must support this functionality. Actual parameters depend on the +module, so see the module's doc on what to supply. The general syntax +is as follows:</p> <p>:modname:params;template</p> -<p>Currently, the ommysql database output module supports this syntax (in -addtion to the ">" syntax it traditionally supported). For ommysql, the module -name is "ommysql" and the params are the traditional ones. The ;template part is -not module specific, it is generic rsyslog functionality available to all -modules.</p> +<p>Currently, the ommysql database output module supports this +syntax (in addtion to the ">" syntax it traditionally +supported). For ommysql, the module name is "ommysql" and the params +are the traditional ones. The ;template part is not module specific, it +is generic rsyslog functionality available to all modules.</p> <p>As an example, the ommysql module may be called as follows:</p> <p>:ommysql:dbhost,dbname,dbuser,dbpassword;dbtemplate</p> -<p>For details, please see the "Database Table" section of this documentation.</p> -<p>Note: as of this writing, the ":modname:" part is hardcoded into the module. -So the name to use is not necessarily the name the module's plugin file is -called.</p> +<p>For details, please see the "Database Table" section of this +documentation.</p> +<p>Note: as of this writing, the ":modname:" part is hardcoded +into the module. So the name to use is not necessarily the name the +module's plugin file is called.</p> <h3>Database Table</h3> -<p>This allows logging of the message to a database table. Currently, only MySQL -databases are supported. However, other database drivers will most probably be -developed as plugins. By default, a <a href="http://www.monitorware.com/">MonitorWare</a>-compatible schema is required -for this to work. You can create that schema with the createDB.SQL file that -came with the rsyslog package. You can also<br> -use any other schema of your liking - you just need to define a proper template -and assign this template to the action.<br> -<br> -The database writer is called by specifying a greater-then sign (">") in front -of the database connect information. Immediately after that<br> -sign the database host name must be given, a comma, the database name, another -comma, the database user, a comma and then the user's password. If a specific -template is to be used, a semicolon followed by the template name can follow -the connect information. This is as follows:<br> +<p>This allows logging of the message to a database table. +Currently, only MySQL databases are supported. However, other database +drivers will most probably be developed as plugins. By default, a <a href="http://www.monitorware.com/">MonitorWare</a>-compatible +schema is required for this to work. You can create that schema with +the createDB.SQL file that came with the rsyslog package. You can also<br> +use any other schema of your liking - you just need to define a proper +template and assign this template to the action.<br> +<br> +The database writer is called by specifying a greater-then sign +(">") in front of the database connect information. Immediately +after that<br> +sign the database host name must be given, a comma, the database name, +another comma, the database user, a comma and then the user's password. +If a specific template is to be used, a semicolon followed by the +template name can follow the connect information. This is as follows:<br> <br> >dbhost,dbname,dbuser,dbpassword;dbtemplate</p> -<p><b>Important: to use the database functionality, the MySQL output module must be -loaded in the config file</b> BEFORE the first database table action is used. This is done by -placing the</p> +<p><b>Important: to use the database functionality, the +MySQL output module must be loaded in the config file</b> BEFORE +the first database table action is used. This is done by placing the</p> <p><code><b>$ModLoad MySQL</b></code></p> -<p>directive some place above the first use of the database write (we recommend -doing at the the beginning of the config file).</p> +<p>directive some place above the first use of the database write +(we recommend doing at the the beginning of the config file).</p> <h3>Discard</h3> -<p>If the discard action is carried out, the received message is immediately -discarded. No further processing of it occurs. Discard has primarily been added -to filter out messages before carrying on any further processing. For obvious -reasons, the results of "discard" are depending on where in the configuration -file it is being used. Please note that once a message has been discarded there -is no way to retrieve it in later configuration file lines.</p> -<p>Discard can be highly effective if you want to filter out some annoying -messages that otherwise would fill your log files. To do that, place the discard -actions early in your log files. This often plays well with property-based -filters, giving you great freedom in specifying what you do not want.</p> -<p>Discard is just the single tilde character with no further parameters:</p> +<p>If the discard action is carried out, the received message is +immediately discarded. No further processing of it occurs. Discard has +primarily been added to filter out messages before carrying on any +further processing. For obvious reasons, the results of "discard" are +depending on where in the configuration file it is being used. Please +note that once a message has been discarded there is no way to retrieve +it in later configuration file lines.</p> +<p>Discard can be highly effective if you want to filter out some +annoying messages that otherwise would fill your log files. To do that, +place the discard actions early in your log files. This often plays +well with property-based filters, giving you great freedom in +specifying what you do not want.</p> +<p>Discard is just the single tilde character with no further +parameters:</p> <p>~</p> <p>For example,</p> <p>*.* ~</p> -<p>discards everything (ok, you can achive the same by not running rsyslogd at -all...).</p> +<p>discards everything (ok, you can achive the same by not +running rsyslogd at all...).</p> <h3>Output Channel</h3> -<p>Binds an output channel definition (see there for details) to this action. -Output channel actions must start with a $-sign, e.g. if you would like to bind -your output channel definition "mychannel" to the action, use "$mychannel". -Output channels support template definitions like all all other actions.</p> +<p>Binds an output channel definition (see there for details) to +this action. Output channel actions must start with a $-sign, e.g. if +you would like to bind your output channel definition "mychannel" to +the action, use "$mychannel". Output channels support template +definitions like all all other actions.</p> <h3>Shell Execute</h3> -<p>This executes a program in a subshell. The program is passed the -template-generated message as the only command line parameter. Rsyslog waits -until the program terminates and only then continues to run.</p> +<p>This executes a program in a subshell. The program is passed +the template-generated message as the only command line parameter. +Rsyslog waits until the program terminates and only then continues to +run.</p> <p>^program-to-execute;template</p> -<p>The program-to-execute can be any valid executable. It receives the template -string as a single parameter (argv[1]).</p> -<p><b>WARNING:</b> The Shell Execute action was added to serve an urgent need. -While it is considered reasonable save when used with some thinking, its -implications must be considered. The current implementation uses a system() call -to execute the command. This is not the best way to do it (and will hopefully -changed in further releases). Also, proper escaping of special characters is -done to prevent command injection. However, attackers always find smart ways to -circumvent escaping, so we can not say if the escaping applied will really safe -you from all hassles. Lastly, rsyslog will wait until the shell command -terminates. Thus, a program error in it (e.g. an infinite loop) can actually -disable rsyslog. Even without that, during the programs run-time no messages are -processed by rsyslog. As the IP stacks buffers are quickly overflowed, this -bears an increased risk of message loss. You must be aware of these implications. -Even though they are severe, there are several cases where the "shell execute" -action is very useful. This is the reason why we have included it in its current -form. To mitigate its risks, always a) test your program thoroughly, b) make -sure its runtime is as short as possible (if it requires a longer run-time, you -might want to spawn your own sub-shell asynchronously), c) apply proper -firewalling so that only known senders can send syslog messages to rsyslog. -Point c) is especially important: if rsyslog is accepting message from any hosts, -chances are much higher that an attacker might try to exploit the "shell execute" -action.</p> +<p>The program-to-execute can be any valid executable. It +receives the template string as a single parameter (argv[1]).</p> +<p><b>WARNING:</b> The Shell Execute action was added +to serve an urgent need. While it is considered reasonable save when +used with some thinking, its implications must be considered. The +current implementation uses a system() call to execute the command. +This is not the best way to do it (and will hopefully changed in +further releases). Also, proper escaping of special characters is done +to prevent command injection. However, attackers always find smart ways +to circumvent escaping, so we can not say if the escaping applied will +really safe you from all hassles. Lastly, rsyslog will wait until the +shell command terminates. Thus, a program error in it (e.g. an infinite +loop) can actually disable rsyslog. Even without that, during the +programs run-time no messages are processed by rsyslog. As the IP +stacks buffers are quickly overflowed, this bears an increased risk of +message loss. You must be aware of these implications. Even though they +are severe, there are several cases where the "shell execute" action is +very useful. This is the reason why we have included it in its current +form. To mitigate its risks, always a) test your program thoroughly, b) +make sure its runtime is as short as possible (if it requires a longer +run-time, you might want to spawn your own sub-shell asynchronously), +c) apply proper firewalling so that only known senders can send syslog +messages to rsyslog. Point c) is especially important: if rsyslog is +accepting message from any hosts, chances are much higher that an +attacker might try to exploit the "shell execute" action.</p> <h2>TEMPLATE NAME</h2> -<p>Every ACTION can be followed by a template name. If so, that template is used -for message formatting. If no name is given, a hard-coded default template is -used for the action. There can only be one template name for each given action. -The default template is specific to each action. For a description of what a -template is and what you can do with it, see "TEMPLATES" at the top of this -document.</p> +<p>Every ACTION can be followed by a template name. If so, that +template is used for message formatting. If no name is given, a +hard-coded default template is used for the action. There can only be +one template name for each given action. The default template is +specific to each action. For a description of what a template is and +what you can do with it, see "TEMPLATES" at the top of this document.</p> <h2>EXAMPLES</h2> -<p>Below are example for templates and selector lines. I hope they are -self-explanatory. If not, please see www.monitorware.com/rsyslog/ for advise.</p> +<p>Below are example for templates and selector lines. I hope +they are self-explanatory. If not, please see +www.monitorware.com/rsyslog/ for advise.</p> <h3>TEMPLATES</h3> -<p>Please note that the samples are split across multiple lines. A template MUST -NOT actually be split across multiple lines.<br> +<p>Please note that the samples are split across multiple lines. +A template MUST NOT actually be split across multiple lines.<br> <br> A template that resembles traditional syslogd file output:<br> -$template TraditionalFormat,"%timegenerated% %HOSTNAME%<br> -%syslogtag%%msg:::drop-last-lf%\n"<br> +$template TraditionalFormat,"%timegenerated% %HOSTNAME%<br> +%syslogtag%%msg:::drop-last-lf%\n"<br> <br> A template that tells you a little more about the message:<br> -$template precise,"%syslogpriority%,%syslogfacility%,%timegenerated%,%HOSTNAME%,<br> -%syslogtag%,%msg%\n"<br> +$template +precise,"%syslogpriority%,%syslogfacility%,%timegenerated%,%HOSTNAME%,<br> +%syslogtag%,%msg%\n"<br> <br> A template for RFC 3164 format:<br> -$template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%"<br> +$template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% +%syslogtag%%msg%"<br> <br> A template for the format traditonally used for user messages:<br> -$template usermsg," XXXX%syslogtag%%msg%\n\r"<br> +$template usermsg," XXXX%syslogtag%%msg%\n\r"<br> <br> And a template with the traditonal wall-message format:<br> -$template wallmsg,"\r\n\7Message from syslogd@%HOSTNAME% at %timegenerated%<br> +$template wallmsg,"\r\n\7Message from syslogd@%HOSTNAME% at +%timegenerated%<br> <br> A template that can be used for the database write (please note the SQL<br> template option)<br> -$template MySQLInsert,"insert iut, message, receivedat values<br> +$template MySQLInsert,"insert iut, message, receivedat values<br> ('%iut%', '%msg:::UPPERCASE%', '%timegenerated:::date-mysql%')<br> -into systemevents\r\n", SQL<br> +into systemevents\r\n", SQL<br> <br> -The following template emulates <a href="http://www.winsyslog.com/en/">WinSyslog</a> -format (it's an <a href="http://www.adiscon.com/en/">Adiscon</a> format, you do -not feel bad if you don't know it ;)). It's interesting to see how it takes -different parts out of the date stamps. What happens is that the date stamp is -split into the actual date and time and the these two are combined with just a -comma in between them.<br> +The following template emulates <a href="http://www.winsyslog.com/en/">WinSyslog</a> +format (it's an <a href="http://www.adiscon.com/en/">Adiscon</a> +format, you do not feel bad if you don't know it ;)). It's interesting +to see how it takes different parts out of the date stamps. What +happens is that the date stamp is split into the actual date and time +and the these two are combined with just a comma in between them.<br> <br> -$template WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%,<br> +$template WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%,<br> %timegenerated:12:19:date-rfc3339%,%timegenerated:1:10:date-rfc3339%,<br> %timegenerated:12:19:date-rfc3339%,%syslogfacility%,%syslogpriority%,<br> -%syslogtag%%msg%\n"</p> +%syslogtag%%msg%\n"</p> <h3>SELECTOR LINES</h3> <p># Store critical stuff in critical<br> #<br> *.=crit;kern.none /var/adm/critical<br> <br> -This will store all messages with the priority crit in the file /var/adm/critical, -except for any kernel message.<br> +This will store all messages with the priority crit in the file +/var/adm/critical, except for any kernel message.<br> <br> <br> # Kernel messages are first, stored in the kernel<br> @@ -709,20 +1015,21 @@ kern.crit @finlandia;RFC3164fmt<br> kern.crit /dev/console<br> kern.info;kern.!err /var/adm/kernel-info<br> <br> -The first rule direct any message that has the kernel facility to the file /var/adm/kernel.<br> +The first rule direct any message that has the kernel facility to the +file /var/adm/kernel.<br> <br> -The second statement directs all kernel messages of the priority crit and higher -to the remote host finlandia. This is useful, because if the host crashes and -the disks get irreparable errors you might not be able to read the stored -messages. If they're on a remote host, too, you still can try to find out the -reason for the crash.<br> +The second statement directs all kernel messages of the priority crit +and higher to the remote host finlandia. This is useful, because if the +host crashes and the disks get irreparable errors you might not be able +to read the stored messages. If they're on a remote host, too, you +still can try to find out the reason for the crash.<br> <br> -The third rule directs these messages to the actual console, so the person who -works on the machine will get them, too.<br> +The third rule directs these messages to the actual console, so the +person who works on the machine will get them, too.<br> <br> -The fourth line tells rsyslogd to save all kernel messages that come with -priorities from info up to warning in the file /var/adm/kernel-info. Everything -from err and higher is excluded.<br> +The fourth line tells rsyslogd to save all kernel messages that come +with priorities from info up to warning in the file +/var/adm/kernel-info. Everything from err and higher is excluded.<br> <br> <br> # The tcp wrapper loggs with mail.info, we display<br> @@ -730,25 +1037,26 @@ from err and higher is excluded.<br> #<br> mail.=info /dev/tty12<br> <br> -This directs all messages that uses mail.info (in source LOG_MAIL | LOG_INFO) to -/dev/tty12, the 12th console. For example the tcpwrapper tcpd(8) uses this as -it's default.<br> +This directs all messages that uses mail.info (in source LOG_MAIL | +LOG_INFO) to /dev/tty12, the 12th console. For example the tcpwrapper +tcpd(8) uses this as it's default.<br> <br> <br> # Store all mail concerning stuff in a file<br> #<br> mail.*;mail.!=info /var/adm/mail<br> <br> -This pattern matches all messages that come with the mail facility, except for -the info priority. These will be stored in the file /var/adm/mail.<br> +This pattern matches all messages that come with the mail facility, +except for the info priority. These will be stored in the file +/var/adm/mail.<br> <br> <br> # Log all mail.info and news.info messages to info<br> #<br> mail,news.=info /var/adm/info<br> <br> -This will extract all messages that come either with mail.info or with news.info -and store them in the file /var/adm/info.<br> +This will extract all messages that come either with mail.info or with +news.info and store them in the file /var/adm/info.<br> <br> <br> # Log info and notice messages to messages file<br> @@ -756,8 +1064,8 @@ and store them in the file /var/adm/info.<br> *.=info;*.=notice;\<br> mail.none /var/log/messages<br> <br> -This lets rsyslogd log all messages that come with either the info or the notice -facility into the file /var/log/messages, except for all<br> +This lets rsyslogd log all messages that come with either the info or +the notice facility into the file /var/log/messages, except for all<br> messages that use the mail facility.<br> <br> <br> @@ -766,17 +1074,17 @@ messages that use the mail facility.<br> *.=info;\<br> mail,news.none /var/log/messages<br> <br> -This statement causes rsyslogd to log all messages that come with the info -priority to the file /var/log/messages. But any message coming either with the -mail or the news facility will not be stored.<br> +This statement causes rsyslogd to log all messages that come with the +info priority to the file /var/log/messages. But any message coming +either with the mail or the news facility will not be stored.<br> <br> <br> # Emergency messages will be displayed using wall<br> #<br> *.=emerg *<br> <br> -This rule tells rsyslogd to write all emergency messages to all currently logged -in users. This is the wall action.<br> +This rule tells rsyslogd to write all emergency messages to all +currently logged in users. This is the wall action.<br> <br> <br> # Messages of the priority alert will be directed<br> @@ -784,37 +1092,38 @@ in users. This is the wall action.<br> #<br> *.alert root,rgerhards<br> <br> -This rule directs all messages with a priority of alert or higher to the -terminals of the operator, i.e. of the users ``root'' and ``rgerhards'' if -they're logged in.<br> +This rule directs all messages with a priority of alert or higher to +the terminals of the operator, i.e. of the users "root'' and +"rgerhards'' if they're logged in.<br> <br> <br> *.* @finlandia<br> <br> -This rule would redirect all messages to a remote host called finlandia. This is -useful especially in a cluster of machines where all syslog messages will be -stored on only one machine.<br> +This rule would redirect all messages to a remote host called +finlandia. This is useful especially in a cluster of machines where all +syslog messages will be stored on only one machine.<br> <br> -In the format shown above, UDP is used for transmitting the message. The -destination port is set to the default auf 514. Rsyslog is also capable of using -much more secure and reliable TCP sessions for message forwarding. Also, the -destination port can be specified. To select TCP, simply add one additional @ in -front of the host name (that is, @host is UPD, @@host is TCP). For example:<br> +In the format shown above, UDP is used for transmitting the message. +The destination port is set to the default auf 514. Rsyslog is also +capable of using much more secure and reliable TCP sessions for message +forwarding. Also, the destination port can be specified. To select TCP, +simply add one additional @ in front of the host name (that is, @host +is UPD, @@host is TCP). For example:<br> <br> <br> *.* @@finlandia<br> <br> -To specify the destination port on the remote machine, use a colon followed by -the port number after the machine name. The following forwards to port 1514 on -finlandia:<br> +To specify the destination port on the remote machine, use a colon +followed by the port number after the machine name. The following +forwards to port 1514 on finlandia:<br> <br> <br> *.* @@finlandia:1514<br> <br> -This syntax works both with TCP and UDP based syslog. However, you will probably -primarily need it for TCP, as there is no well-accepted port for this transport -(it is non-standard). For UDP, you can usually stick with the default auf 514, -but might want to modify it for security rea-<br> +This syntax works both with TCP and UDP based syslog. However, you will +probably primarily need it for TCP, as there is no well-accepted port +for this transport (it is non-standard). For UDP, you can usually stick +with the default auf 514, but might want to modify it for security rea-<br> sons. If you would like to do that, it's quite easy:<br> <br> <br> @@ -824,27 +1133,29 @@ sons. If you would like to do that, it's quite easy:<br> <br> *.* >dbhost,dbname,dbuser,dbpassword;dbtemplate<br> <br> -This rule writes all message to the database "dbname" hosted on "dbhost". The -login is done with user "dbuser" and password "dbpassword". The actual table -that is updated is specified within the template (which contains the insert -statement). The template is called "dbtemplate" in this case.</p> -<p>:msg,contains,"error" @errorServer</p> -<p>This rule forwards all messages that contain the word "error" in the msg part -to the server "errorServer". Forwarding is via UDP. Please note the colon in -fron</p> +This rule writes all message to the database "dbname" hosted on +"dbhost". The login is done with user "dbuser" and password +"dbpassword". The actual table that is updated is specified within the +template (which contains the insert statement). The template is called +"dbtemplate" in this case.</p> +<p>:msg,contains,"error" @errorServer</p> +<p>This rule forwards all messages that contain the word "error" +in the msg part to the server "errorServer". Forwarding is via UDP. +Please note the colon in fron</p> <h2>CONFIGURATION FILE SYNTAX DIFFERENCES</h2> -<p>Rsyslogd uses a slightly different syntax for its configuration file than the -original BSD sources. Originally all messages of a specific priority and above -were forwarded to the log file. The modifiers ``='', ``!'' and ``-'' were added -to make rsyslogd more flexible and to use it in a more intuitive manner.<br> -<br> -The original BSD syslogd doesn't understand spaces as separators between the -selector and the action field.<br> -<br> -When compared to syslogd from sysklogd package, rsyslogd offers additional -<a href="features.html">features</a> (like template and database support). For obvious reasons, the syntax for -defining such features is available -in rsyslogd, only.<br> +<p>Rsyslogd uses a slightly different syntax for its +configuration file than the original BSD sources. Originally all +messages of a specific priority and above were forwarded to the log +file. The modifiers "='', "!'' and "!-'' were added to make rsyslogd +more flexible and to use it in a more intuitive manner.<br> +<br> +The original BSD syslogd doesn't understand spaces as separators +between the selector and the action field.<br> +<br> +When compared to syslogd from sysklogd package, rsyslogd offers +additional +<a href="features.html">features</a> (like template +and database support). For obvious reasons, the syntax for defining +such features is available in rsyslogd, only.<br> </p> -</body> -</html> +</body></html>
\ No newline at end of file diff --git a/doc/rsyslog_high_database_rate.html b/doc/rsyslog_high_database_rate.html new file mode 100644 index 0000000..5b0a29a --- /dev/null +++ b/doc/rsyslog_high_database_rate.html @@ -0,0 +1,175 @@ +<html><head> + +<title>Handling a massive syslog database insert rate with Rsyslog</title> + +<meta name="KEYWORDS" content="syslog, rsyslog, reliable, howto, database, postgresql, mysql, buffering, disk, queue"> + +</head> + +<body> + +<h1>Handling a massive syslog database insert rate with Rsyslog</h1> + + <P><small><i>Written by + + <a href="http://www.gerhards.net/rainer">Rainer + + Gerhards</a> (2008-01-31)</i></small></P> + +<h2>Abstract</h2> + +<p><i><b>In this paper, I describe how log massive amounts of +<a href="http://www.monitorware.com/en/topics/syslog/">syslog</a> + +messages to a database. </b>This HOWTO is currently under development and thus a +bit brief. Updates are promised ;).</i></p> + +<h2>The Intention</h2> + +<p>Database updates are inherently slow when it comes to storing syslog +messages. However, there are a number of applications where it is handy to have +the message inside a database. Rsyslog supports native database writing via +output plugins. As of this writing, there are plugins available for MySQL an +PostgreSQL. Maybe additional plugins have become available by the time you read +this. Be sure to check.</p> +<p>In order to successfully write messages to a database backend, the backend +must be capable to record messages at the expected average arrival rate. This is +the rate if you take all messages that can arrive within a day and divide it by +86400 (the number of seconds per day). Let's say you expect 43,200,000 messages +per day. That's an average rate of 500 messages per second (mps). Your database +server MUST be able to handle that amount of message per second on a sustained +rate. If it doesn't, you either need to add an additional server, lower the +number of message - or forget about it.</p> +<p>However, this is probably not your peak rate. Let's simply assume your +systems work only half a day, that's 12 hours (and, yes, I know this is +unrealistic, but you'll get the point soon). So your average rate is actually +1,000 mps during work hours and 0 mps during non-work hours. To make matters +worse, workload is not divided evenly during the day. So you may have peaks of +up to 10,000mps while at other times the load may go down to maybe just 100mps. +Peaks may stay well above 2,000mps for a few minutes.</p> +<p>So how the hack you will be able to handle all of this traffic (including the +peaks) with a database server that is just capable of inserting a maximum of +500mps?</p> +<p>The key here is buffering. Messages that the database server is not capable +to handle will be buffered until it is. Of course, that means database insert +are NOT real-time. If you need real-time inserts, you need to make sure your +database server can handle traffic at the actual peak rate. But lets assume you +are OK with some delay.</p> +<p>Buffering is fine. But how about these massive amounts of data? That can't be +hold in memory, so don't we run out of luck with buffering? The key here is that +rsyslog can not only buffer in memory but also buffer to disk (this may remind +you of "spooling" which gets you the right idea). There are several queuing +modes available, offering differnent throughput. In general, the idea is to +buffer in memory until the memory buffer is exhausted and switch to +disk-buffering when needed (and only as long as needed). All of this is handled +automatically and transparently by rsyslog.</p> +<p>With our above scenario, the disk buffer would build up during the day and +rsyslog would use the night to drain it. Obviously, this is an extreme example, +but it shows what can be done. Please note that queue content survies rsyslogd +restarts, so even a reboot of the system will not cause any message loss.</p> +<h2>How To Setup</h2> +<p>Frankly, it's quite easy. You just need to do is instruct rsyslog to use a +disk queue and then configure your action. There is nothing else to do. With the +following simple config file, you log anything you receive to a MySQL database +and have buffering applied automatically.</p> +<textarea rows="11" cols="80">$ModLoad ommysql.so # load the output driver (use ompgsql.so for PostgreSQL) +$ModLoad imudp.so # network reception +$UDPServerRun 514 # start a udp server at port 514 +$ModLoad imuxsock.so # local message reception + +$WorkDirectory /rsyslog/work # default location for work (spool) files +$MainMsgQueueFileName mainq # set file name, also enables disk mode + +$ActionResumeRetryCount -1 # infinite retries on insert failure +# for PostgreSQL replace :ommysql: by :ompgsql: below: +*.* :ommysql:hostname,dbname,userid,password; +</textarea> +<p>The simple setup above has one drawback: the write database action is +executed together with all other actions. Typically, local files are also +written. These local file writes are now bound to the speed of the database +action. So if the database is down, or threre is a large backlog, local files +are also not (or late) written.</p> +<p><b>There is an easy way to avoid this with rsyslog.</b> It involves a +slightly more complicated setup. In rsyslog, each action can utilize its own +queue. If so, messages are simply pulled over from the main queue and then the +action queue handles action processing on its own. This way, main processing and +the action are de-coupled. In the above example, this means that local file +writes will happen immediately while the database writes are queued. As a +side-note, each action can have its own queue, so if you would like to more than +a single database or send messages reliably to another host, you can do all of +this on their own queues, de-coupling their processing speeds.</p> +<p>The configuration for the de-coupled database write involves just a few more +commands:</p> +<textarea rows="11" cols="80">$ModLoad ommysql.so # load the output driver (use ompgsql.so for PostgreSQL) +$ModLoad imudp.so # network reception +$UDPServerRun 514 # start a udp server at port 514 +$ModLoad imuxsock.so # local message reception + +$WorkDirectory /rsyslog/work # default location for work (spool) files + +$ActionQueueType LinkedList # use asynchronous processing +$ActionQueueFileName dbq # set file name, also enables disk mode +$ActionResumeRetryCount -1 # infinite retries on insert failure +# for PostgreSQL replace :ommysql: by :ompgsql: below: +*.* :ommysql:hostname,dbname,userid,password; +</textarea> +<p><b>This is the recommended configuration for this use case.</b> It requires +rsyslog 3.11.0 or above.</p> +<p>In this example, the main message queue is NOT disk-assisted (there is no +$MainMsgQueueFileName directive). We still could do that, but have not done it +because there seems to be no need. The only slow running action is the database +writer and it has its own queue. So there is no real reason to use a large main +message queue (except, of course, if you expect *really* heavy traffic bursts).</p> +<p>Note that you can modify a lot of queue performance parameters, but the above +config will get you going with default values. If you consider using this on a real +busy server, it is strongly recommended to invest some time in setting the tuning +parameters to appropriate values.</p> + +<h3>Feedback requested</h3> + +<P>I would appreciate feedback on this tutorial. If you have additional ideas, + +comments or find bugs (I *do* bugs - no way... ;)), please + +<a href="mailto:rgerhards@adiscon.com">let me know</a>.</P> + +<h2>Revision History</h2> + +<ul> + + <li>2008-01-28 * + + <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> * Initial Version created</li> + <li>2008-01-28 * + + <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> + * Updated to new v3.11.0 capabilities</li> + +</ul> +<h2>Copyright</h2> + +<p>Copyright (c) 2008 + +<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> and + +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> + +<p> Permission is granted to copy, distribute and/or modify this document + + under the terms of the GNU Free Documentation License, Version 1.2 + + or any later version published by the Free Software Foundation; + + with no Invariant Sections, no Front-Cover Texts, and no Back-Cover + + Texts. A copy of the license can be viewed at + +<a href="http://www.gnu.org/copyleft/fdl.html"> + +http://www.gnu.org/copyleft/fdl.html</a>.</p> + + + +</body> + +</html> diff --git a/doc/rsyslog_mysql.html b/doc/rsyslog_mysql.html index 53ee30c..57a779d 100644 --- a/doc/rsyslog_mysql.html +++ b/doc/rsyslog_mysql.html @@ -1,249 +1,262 @@ -<html><head> -<title>Writing syslog Data to MySQL</title> -<meta name="KEYWORDS" content="syslog, mysql, syslog to mysql, howto"> -</head> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>Writing syslog Data to MySQL</title> + +<meta name="KEYWORDS" content="syslog, mysql, syslog to mysql, howto"></head> <body> <h1>Writing syslog messages to MySQL</h1> - <P><small><i>Written by - <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer - Gerhards</a> (2005-08-02)</i></small></P> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-02-28)</i></small></p> <h2>Abstract</h2> <p><i><b>In this paper, I describe how to write <a href="http://www.monitorware.com/en/topics/syslog/">syslog</a> -messages to a <a href="http://www.mysql.com">MySQL</a> database.</b> Having -syslog messages in a database is often handy, especially when you intend to set -up a front-end for viewing them. This paper describes an approach -with <a href="http://www.rsyslog.com/">rsyslogd</a>, an alternative enhanced -syslog daemon natively supporting MySQL. I describe the components needed -to be installed and how to configure them.</i></p> +messages to a <a href="http://www.mysql.com">MySQL</a> +database.</b> Having syslog messages in a database is often +handy, especially when you intend to set up a front-end for viewing +them. This paper describes an approach with <a href="http://www.rsyslog.com/">rsyslogd</a>, +an +alternative enhanced syslog daemon natively supporting MySQL. I +describe the components needed to be installed and how to configure +them. Please note that as of this writing, rsyslog supports a variety +of databases. While this guide is still MySQL-focussed, you +can probably use it together with other ones too. You just need to +modify a few settings.</i></p> <h2>Background</h2> -<p>In many cases, syslog data is simply written to text files. This approach has -some advantages, most notably it is very fast and efficient. However, data -stored in text files is not readily accessible for real-time viewing and analysis. -To do that, the messages need to be in a database. There are various -ways to store syslog messages in a database. For example, some have the syslogd -write text files which are later feed via a separate script into the database. -Others have written scripts taking the data (via a pipe) from a -non-database-aware syslogd and store them as they appear. Some others use -database-aware syslogds and make them write the data directly to the database. -In this paper, I use that "direct write" approach. I think it is superior, -because the syslogd itself knows the status of the database connection and thus -can handle it intelligently (well ... hopefully ;)). I use rsyslogd to acomplish -this, simply because I have initiated the rsyslog project with -database-awareness as one goal.</p> -<p><b>One word of caution:</b> while message storage in the database provides an -excellent foundation for interactive analysis, it comes at a cost. Database i/o -is considerably slower than text file i/o. As such, directly writing to -the database makes sense only if your message volume is low enough to allow a) -the syslogd, b) the network, and c) the database server to catch -up with it. Some time ago, I have written a paper on -<a href="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php"> -optimizing syslog server performance</a>. While this paper talks about -Window-based solutions, the ideas in it are generic enough to apply here, too. -So it might be worth reading if you anticipate medium high to high traffic. If you -anticipate really high traffic (or very large traffic spikes), you should -seriously consider forgetting about direct database writes - in my opinion, such -a situation needs either a very specialised system or a different approach (the -text-file-to-database approach might work better for you in this case). +<p>In many cases, syslog data is simply written to text files. +This approach has some advantages, most notably it is very fast and +efficient. However, data stored in text files is not readily accessible +for real-time viewing and analysis. To do that, the messages need to be +in a database. There are various ways to store syslog messages in a +database. For example, some have the syslogd write text files which are +later feed via a separate script into the database. Others have written +scripts taking the data (via a pipe) from a non-database-aware syslogd +and store them as they appear. Some others use database-aware syslogds +and make them write the data directly to the database. In this paper, I +use that "direct write" approach. I think it is superior, because the +syslogd itself knows the status of the database connection and thus can +handle it intelligently (well ... hopefully ;)). I use rsyslogd to +acomplish this, simply because I have initiated the rsyslog project +with database-awareness as one goal.</p> +<p><b>One word of caution:</b> while message storage +in the database provides an excellent foundation for interactive +analysis, it comes at a cost. Database i/o is considerably slower than +text file i/o. As such, directly writing to the database makes sense +only if your message volume is low enough to allow a) the syslogd, b) +the network, and c) the database server to catch up with it. Some time +ago, I have written a paper on +<a href="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php">optimizing +syslog server performance</a>. While this paper talks about +Window-based solutions, the ideas in it are generic enough to apply +here, too. So it might be worth reading if you anticipate medium high +to high traffic. If you anticipate really high traffic (or very large +traffic spikes), you should seriously consider forgetting about direct +database writes - in my opinion, such a situation needs either a very +specialised system or a different approach (the text-file-to-database +approach might work better for you in this case). </p> <h2>Overall System Setup</h2> -<P>In this paper, I concentrate on the server side. If you are thinking about -interactive syslog message review, you probably want to centralize syslog. In -such a scenario, you have multiple machines (the so-called clients) send their -data to a central machine (called server in this context). While I expect such a -setup to be typical when you are interested in storing messages in the database, -I do not describe how to set it up. This is beyond the scope of this paper. If -you search a little, you will probably find many good descriptions on how to -centralize syslog. If you do that, it might be a good idea to do it securely, so -you might also be interested in my paper on <a href="rsyslog_stunnel.html"> -ssl-encrypting syslog message transfer</a>.</P> -<P>No matter how the messages arrive at the server, their processing is always the -same. So you can use this paper in combination with any description for -centralized syslog reporting.</P> -<P>As I already said, I use rsyslogd on the server. It has intrinsic support for -talking to MySQL databases. For obvious reasons, we also need an instance of -MySQL running. To keep us focussed, the setup of MySQL itself is also beyond the -scope of this paper. I assume that you have successfully installed MySQL and -also have a front-end at hand to work with it (for example, -<a href="http://www.phpmyadmin.net/">phpMyAdmin</a>). Please make sure that this -is installed, actually working and you have a basic understanding of how to -handle it.</P> +<p>In this paper, I concentrate on the server side. If you are +thinking about interactive syslog message review, you probably want to +centralize syslog. In such a scenario, you have multiple machines (the +so-called clients) send their data to a central machine (called server +in this context). While I expect such a setup to be typical when you +are interested in storing messages in the database, I do not describe +how to set it up. This is beyond the scope of this paper. If you search +a little, you will probably find many good descriptions on how to +centralize syslog. If you do that, it might be a good idea to do it +securely, so you might also be interested in my paper on <a href="rsyslog_stunnel.html"> +ssl-encrypting syslog message transfer</a>.</p> +<p>No matter how the messages arrive at the server, their +processing is always the same. So you can use this paper in combination +with any description for centralized syslog reporting.</p> +<p>As I already said, I use rsyslogd on the server. It has +intrinsic support for talking to MySQL databases. For obvious reasons, +we also need an instance of MySQL running. To keep us focussed, the +setup of MySQL itself is also beyond the scope of this paper. I assume +that you have successfully installed MySQL and also have a front-end at +hand to work with it (for example, +<a href="http://www.phpmyadmin.net/">phpMyAdmin</a>). +Please make sure that this is installed, actually working and you have +a basic understanding of how to handle it.</p> <h2>Setting up the system</h2> -<p>You need to download and install rsyslogd first. Obtain it from the -<a href="http://www.rsyslog.com/">rsyslog site</a>. Make sure that you disable -stock syslogd, otherwise you will experience some difficulties.</p> -<p>It is important to understand how rsyslogd talks to the database. In rsyslogd, -there is the concept of "templates". Basically, a template is a string that -includes some replacement characters, which are called "properties" in rsyslog. -Properties are accessed via the "<a href="property_replacer.html">Property -Replacer</a>". Simply said, you access properties by including their name -between percent signs inside the template. For example, if the syslog message is -"Test", the template "%msg%" would be expanded to "Test". Rsyslogd supports -sending template text as a SQL statement to MySQL. As such, the template must be -a valid SQL statement. There is no limit in what the statement might be, but -there are some obvious and not so obvious choices. For example, a template "drop -table xxx" is possible, but does not make an awful lot of sense. In practice, -you will always use an "insert" statment inside the template.</p> -<p>An example: if you would just like to store the msg part of the full syslog -message, you have probably created a table "syslog" with a single column "message". -In such a case, a good template would be "insert into syslog(message) values ('%msg%')". -With the example above, that would be expanded to "insert into syslog(message) -values('Test')". This expanded string is then sent to the database. It's that -easy, no special magic. The only thing you must ensure is that your template -expands to a proper SQL statement and that this statement matches your database -design.</p> -<p>Does that mean you need to create database schema yourself and also must -fully understand rsyslogd's properties? No, that's not needed. Because we -anticipated that folks are probably more interested in getting things going instead -of designing them from scratch. So we have provided a default schema as well -as build-in support for it. This schema also offers an additional -benefit: rsyslog is part of <a href="http://www.adiscon.com/en/">Adiscon</a>'s -<a href="http://www.monitorware.com/en/">MonitorWare product line</a> (which -includes open source and closed source members). All of these tools share the -same default schema and know how to operate on it. For this reason, the default -schema is also called the "MonitorWare Schema". If you use it, you can simply -add <a href="http://www.phplogcon.org/">phpLogCon, a GPLed syslog web interface</a>, -to your system and have instant interactive access to your database. So there -are some benefits in using the provided schema.</p> -<p>The schema definition is contained in the file "createDB.sql". It comes with -the rsyslog package. Review it to check that the database name is acceptable for -you. Be sure to leave the table and field names unmodified, because -otherwise you need to customize rsyslogd's default sql template, which we do not -do -in this paper. Then, run the script with your favourite MySQL tool. Double-check -that the table was successfully created.</p> - -<p>MySQL support in rsyslog is integrated via a loadable plug-in module. To use the database -functionality, MySQL must be enabled in the config file BEFORE the first database table action is +<p>You need to download and install rsyslogd first. Obtain it +from the +<a href="http://www.rsyslog.com/">rsyslog site</a>. +Make sure that you disable stock syslogd, otherwise you will experience +some difficulties. On some distributions (Fedora 8 and above, for +example), rsyslog may already by the default syslogd, in which case you +obviously do not need to do anything specific. For many others, there +are prebuild packages available. If you use either, please make sure +that you have the required database plugins for your database +available. It usually is a separate package and typically <span style="font-weight: bold;">not</span> installed by default.</p> +<p>It is important to understand how rsyslogd talks to the +database. In rsyslogd, there is the concept of "templates". Basically, +a template is a string that includes some replacement characters, which +are called "properties" in rsyslog. Properties are accessed via the "<a href="property_replacer.html">Property Replacer</a>". +Simply said, you access properties by including their name between +percent signs inside the template. For example, if the syslog message +is "Test", the template "%msg%" would be expanded to "Test". Rsyslogd +supports sending template text as a SQL statement to MySQL. As such, +the template must be a valid SQL statement. There is no limit in what +the statement might be, but there are some obvious and not so obvious +choices. For example, a template "drop table xxx" is possible, but does +not make an awful lot of sense. In practice, you will always use an +"insert" statment inside the template.</p> +<p>An example: if you would just like to store the msg part of +the full syslog message, you have probably created a table "syslog" +with a single column "message". In such a case, a good template would +be "insert into syslog(message) values ('%msg%')". With the example +above, that would be expanded to "insert into syslog(message) +values('Test')". This expanded string is then sent to the database. +It's that easy, no special magic. The only thing you must ensure is +that your template expands to a proper SQL statement and that this +statement matches your database design.</p> +<p>Does that mean you need to create database schema yourself and +also must fully understand rsyslogd's properties? No, that's not +needed. Because we anticipated that folks are probably more interested +in getting things going instead of designing them from scratch. So we +have provided a default schema as well as build-in support for it. This +schema also offers an additional benefit: rsyslog is part of <a href="http://www.adiscon.com/en/">Adiscon</a>'s +<a href="http://www.monitorware.com/en/">MonitorWare +product line</a> (which includes open source and closed source +members). All of these tools share the same default schema and know how +to operate on it. For this reason, the default schema is also called +the "MonitorWare Schema". If you use it, you can simply add <a href="http://www.phplogcon.org/">phpLogCon, a GPLed syslog +web interface</a>, to your system and have instant interactive +access to your database. So there are some benefits in using the +provided schema.</p> +<p>The schema definition is contained in the file "createDB.sql". +It comes with the rsyslog package. Review it to check that the database +name is acceptable for you. Be sure to leave the table and field names +unmodified, because otherwise you need to customize rsyslogd's default +sql template, which we do not do in this paper. Then, run the script +with your favourite MySQL tool. Double-check that the table was +successfully created.</p> +<p>MySQL support in rsyslog is integrated via a loadable plug-in +module. To use the database +functionality, MySQL must be enabled in the config file BEFORE the +first database table action is used. This is done by placing the</p> <blockquote> - <p><code>$ModLoad MySQL</code></p> +<p><code>$ModLoad ommysql.so</code></p> </blockquote> -<p>directive at the begining of /etc/rsyslog.conf</p> - -<p>Next, we need to tell rsyslogd to write data to the database. As we use -the default schema, we do NOT need to define a template for this. We can use the -hardcoded one (rsyslogd handles the proper template linking). So all we need to -do is add a simple selector line to /etc/rsyslog.conf:</p> +<p>directive at the begining of /etc/rsyslog.conf. For other databases, use their plugin name (e.g. ompgsql.so).</p> +<p>Next, we need to tell rsyslogd to write data to the database. +As we use the default schema, we do NOT need to define a template for +this. We can use the hardcoded one (rsyslogd handles the proper +template linking). So all we need to do is add a simple selector line +to /etc/rsyslog.conf:</p> <blockquote> - <p><code>*.* - >database-server,database-name,database-userid,database-password</code></p> +<p><code>*.* :ommysql:database-server,database-name,database-userid,database-password</code></p> </blockquote> -<p>In many cases, MySQL will run on the local machine. In this case, you can -simply use "127.0.0.1" for <i>database-server</i>. This can be especially -advisable, if you do not need to expose MySQL to any process outside of the -local machine. In this case, you can simply bind it to 127.0.0.1, which provides -a quite secure setup. Of course, also supports remote MySQL instances. In that -case, use the remote server name (e.g. mysql.example.com) or IP-address. The <i> -database-name</i> by default is "syslog". If you have modified the default, use -your name here. <i>Database-userid</i> and <i>-password</i> are the credentials -used to connect to the database. As they are stored in clear text in -rsyslog.conf, that user should have only the least possible privileges. It is -sufficient to grant it INSERT privileges to the systemevents table, only. As a -side note, it is strongly advisable to make the rsyslog.conf file readable by -root only - if you make it world-readable, everybody could obtain the password -(and eventually other vital information from it). In our example, let's assume -you have created a MySQL user named "syslogwriter" with a password of -"topsecret" (just to say it bluntly: such a password is NOT a good idea...). If -your MySQL database is on the local machine, your rsyslog.conf line might look -like in this sample:</p> +<p>Again, other databases have other selector names, e.g. ":ompgsql:" +instead of ":ommysql:". See the output plugin's documentation for +details.</p><p>In many cases, MySQL will run on the local machine. In this +case, you can simply use "127.0.0.1" for <i>database-server</i>. +This can be especially advisable, if you do not need to expose MySQL to +any process outside of the local machine. In this case, you can simply +bind it to 127.0.0.1, which provides a quite secure setup. Of course, +also supports remote MySQL instances. In that case, use the remote +server name (e.g. mysql.example.com) or IP-address. The <i> +database-name</i> by default is "syslog". If you have modified +the default, use your name here. <i>Database-userid</i> +and <i>-password</i> are the credentials used to connect +to the database. As they are stored in clear text in rsyslog.conf, that +user should have only the least possible privileges. It is sufficient +to grant it INSERT privileges to the systemevents table, only. As a +side note, it is strongly advisable to make the rsyslog.conf file +readable by root only - if you make it world-readable, everybody could +obtain the password (and eventually other vital information from it). +In our example, let's assume you have created a MySQL user named +"syslogwriter" with a password of "topsecret" (just to say it bluntly: +such a password is NOT a good idea...). If your MySQL database is on +the local machine, your rsyslog.conf line might look like in this +sample:</p> <blockquote> - <p><code>*.* - >127.0.0.1,syslog,syslogwriter,topsecret</code></p> +<p><code>*.* :ommysql:127.0.0.1,syslog,syslogwriter,topsecret</code></p> </blockquote> -<p>Save rsyslog.conf, restart rsyslogd - and you should see syslog messages -being stored in the "systemevents" table!</p> -<p>The example line stores every message to the database. Especially if you have -a high traffic volume, you will probably limit the amount of messages being -logged. This is easy to acomplish: the "write database" action is just a regular -selector line. As such, you can apply normal selector-line filtering. If, for -example, you are only interested in messages from the mail subsystem, you can -use the following selector line:</p> +<p>Save rsyslog.conf, restart rsyslogd - and you should see +syslog messages being stored in the "systemevents" table!</p> +<p>The example line stores every message to the database. +Especially if you have a high traffic volume, you will probably limit +the amount of messages being logged. This is easy to acomplish: the +"write database" action is just a regular selector line. As such, you +can apply normal selector-line filtering. If, for example, you are only +interested in messages from the mail subsystem, you can use the +following selector line:</p> <blockquote> - <p><code>mail.* - >127.0.0.1,syslog,syslogwriter,topsecret</code></p> +<p><code>mail.* </code><code>:ommysql:</code><code>127.0.0.1,syslog,syslogwriter,topsecret</code></p> </blockquote> -<p>Review the <a href="rsyslog_conf.html">rsyslog.conf</a> documentation for -details on selector lines and their filtering.</p> -<p><b>You have now completed everything necessary to store syslog messages to -the MySQL database.</b> If you would like to try out a front-end, you might want -to look at <a href="http://www.phplogcon.org/">phpLogCon</a>, which displays -syslog data in a browser. As of this writing, phpLogCon is not yet a powerful -tool, but it's open source, so it might be a starting point for your own -solution.</p> +<p>Review the <a href="rsyslog_conf.html">rsyslog.conf</a> +documentation for details on selector lines and their filtering.</p> +<p><b>You have now completed everything necessary to store +syslog messages to the MySQL database.</b> If you would like to +try out a front-end, you might want to look at <a href="http://www.phplogcon.org/">phpLogCon</a>, which +displays syslog data in a browser. As of this writing, phpLogCon is not +yet a powerful tool, but it's open source, so it might be a starting +point for your own solution.</p> <h2>On Reliability...</h2> -<p><b>This section needs updating. You can now solve the issue with failover -database servers. Read the <a href="rsyslog_conf.html">rsyslog.conf </a>doc on -that</b>.</p> -<p>Rsyslogd writes syslog messages directly to the database. This implies that -the database must be available at the time of message arrival. If the database -is offline, no space is left or something else goes wrong - rsyslogd can not -write the database record. If rsyslogd is unable to store a message, it performs -one retry. This is helpful if the database server was restarted. In this case, -the previous connection was broken but a reconnect immediately succeeds. However, -if the database is down for an extended period of time, an immediate retry does -not help. While rsyslogd could retry until it finally succeeds, that would have -negative impact. Syslog messages keep coming in. If rsyslogd would be busy -retrying the database, it would not be able to process these messages. -Ultimately, this would lead to loss of newly arrived messages.</p> -<p>In most cases, rsyslogd is configured not only to write to the database but -to perform other actions as well. In the always-retry scenario, that would mean -no other actions would be carried out. As such, the design of rsyslogd is -limited to a single retry. If that does not succeed, the current message is will -not be written to the database and the MySQL database writer be suspended for a -short period of time. Obviously, this leads to the loss of the current message -as well as all messages received during the suspension period. But they are only -lost in regard to the database, all other actions are correctly carried out. -While not perfect, we consider this to be a better approach then the potential -loss of all messages in all actions.</p> -<p><b>In short: try to avoid database downtime if you do not want to experience -message loss.</b></p> -<p>Please note that this restriction is not rsyslogd specific. All approaches to -real-time database storage share this problem area.</p> +<p>Rsyslogd writes syslog messages directly to the database. This +implies that the database must be available at the time of message +arrival. If the database is offline, no space is left or something else +goes wrong - rsyslogd can not write the database record. If rsyslogd is +unable to store a message, it performs one retry. This is helpful if +the database server was restarted. In this case, the previous +connection was broken but a reconnect immediately succeeds. However, if +the database is down for an extended period of time, an immediate retry +does not help.</p> +<p>Message loss in this scenario can easily be prevented with +rsyslog. All you need to do is run the database writer in queued mode. +This is now described in a generic way and I do not intend to duplicate +it here. So please be sure to read "<a href="rsyslog_high_database_rate.html">Handling a massive +syslog database insert rate with Rsyslog</a>", which describes +the scenario and also includes configuration examples.</p> <h2>Conclusion</h2> -<P>With minimal effort, you can use rsyslogd to write syslog messages to a MySQL -database. Once the messages are arrived there, you can interactivley review and -analyse them. In practice, the messages are also stored in text files for -longer-term archival and the databases are cleared out after some time (to avoid -becoming too slow). If you expect an extremely high syslog message volume, -storing it in real-time to the database may outperform your database server. In -such cases, either filter out some messages or think about alternate approaches -involving non-real-time database writing (beyond the scope of this paper).</P> -<P>The method outlined in this paper provides an easy to setup and maintain -solution for most use cases, especially with low and medium syslog message -volume (or fast database servers).</P> +<p>With minimal effort, you can use rsyslogd to write syslog +messages to a MySQL database. You can even make it absolutely fail-safe +and protect it against database server downtime. Once the messages are +arrived there, you +can interactivley review and analyse them. In practice, the messages +are also stored in text files for longer-term archival and the +databases are cleared out after some time (to avoid becoming too slow). +If you expect an extremely high syslog message volume, storing it in +real-time to the database may outperform your database server. In such +cases, either filter out some messages or used queued mode (which in +general is recommended with databases).</p> +<p>The method outlined in this paper provides an easy to setup +and maintain solution for most use cases.</p> <h3>Feedback Requested</h3> -<P>I would appreciate feedback on this paper. If you have additional ideas, -comments or find bugs, please -<a href="mailto:rgerhards@adiscon.com">let me know</a>.</P> +<p>I would appreciate feedback on this paper. If you have +additional ideas, comments or find bugs, please +<a href="mailto:rgerhards@adiscon.com">let me know</a>.</p> <h2>References and Additional Material</h2> <ul> - <li><a href="http://www.rsyslog.com">www.rsyslog.com</a> - the rsyslog site</li> - <li> - <a href="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php"> - Paper on Syslog Server Optimization</a></li> +<li><a href="http://www.rsyslog.com">www.rsyslog.com</a> +- the rsyslog site</li> +<li> <a href="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php"> +Paper on Syslog Server Optimization</a></li> </ul> <h2>Revision History</h2> <ul> - <li>2005-08-02 * - <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> * - initial version created</li> - <li>2005-08-03 * - <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> - * added references to demo site</li> - <li>2007-06-13 * - <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> - * removed demo site - was torn down because too expensive for usage count</li> +<li>2005-08-02 * <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> * initial version created</li> +<li>2005-08-03 * <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> * added references to demo site</li> +<li>2007-06-13 * <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> * removed demo site - was torn down because too +expensive for usage count</li> +<li>2008-02-21 * <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> * updated reliability section, can now be done with +on-demand disk queues</li><li>2008-02-28 * <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> * added info on other databases, updated syntax to more recent one</li> </ul> <h2>Copyright</h2> -<p>Copyright (c) 2005-2007 -<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> -and <a href="http://www.adiscon.com/en/">Adiscon</a>.</p> -<p>Permission is granted to copy, distribute and/or modify this document under -the terms of the GNU Free Documentation License, Version 1.2 or any later -version published by the Free Software Foundation; with no Invariant Sections, -no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be -viewed at <a href="http://www.gnu.org/copyleft/fdl.html"> +<p>Copyright (c) 2005-2008 +<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and <a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p>Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at <a href="http://www.gnu.org/copyleft/fdl.html"> http://www.gnu.org/copyleft/fdl.html</a>.</p> -</body> -</html> +</body></html>
\ No newline at end of file diff --git a/doc/rsyslog_ng_comparison.html b/doc/rsyslog_ng_comparison.html new file mode 100644 index 0000000..547501a --- /dev/null +++ b/doc/rsyslog_ng_comparison.html @@ -0,0 +1,531 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta content="de" http-equiv="Content-Language"><title>rsyslog vs. syslog-ng - a comparison</title></head> +<body> +<h1>rsyslog vs. syslog-ng</h1> +<p><small><i>Written by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> +(2008-02-28)</i></small></p> +<p>We have often been asked about a comparison sheet between +rsyslog and syslog-ng. Unfortunately, I do not know much about +syslog-ng, I did not even use it once. Also, there seems to be no +comprehensive feature sheet available for syslog-ng (that recently +changed, see below). So I started this +comparison, but it probably is not complete. For sure, I miss some +syslog-ng features. This is not an attempt to let rsyslog shine more +than it should. I just used the <a href="features.html">rsyslog +feature sheet</a> as a starting point, simply because it was +available. If you would like to add anything to the chart, or correct +it, please simply <a href="mailto:rgerhards@adiscon.com">drop +me a line</a>. I would love to see a real honest and up-to-date +comparison sheet, so please don't be shy ;)</p> +<table border="1"> +<tbody> +<tr> +<td valign="top"><b>Feature</b></td> +<td valign="top"><b>rsyslog</b></td> +<td valign="top"><b>syslog-ng</b></td> +</tr> + + +<tr> +<td colspan="3" valign="top"><br><b>Input Sources</b><br></td> +</tr> +<tr><td valign="top">UNIX domain socket</td> +<td valign="top">yes</td> +<td valign="top">yes</td><td> +</td></tr> +<tr> +<td valign="top">UDP</td> +<td valign="top">yes</td> +<td valign="top">yes</td><td> +</td></tr> +<tr> +<td valign="top">TCP</td> +<td valign="top">yes</td> +<td valign="top">yes</td><td> +</td></tr> +<tr> +<td valign="top">RFC 3195/BEEP</td> +<td valign="top">yes (needs separate build process)</td> +<td valign="top">no</td><td> +</td></tr> +<tr> +<td valign="top">kernel log</td> +<td valign="top">yes</td> +<td valign="top">yes</td><td> +</td></tr> +<tr> +<td valign="top">file</td> +<td valign="top">yes</td> +<td valign="top">yes</td><td> +</td></tr> +<tr> +<td valign="top">mark message generator as an optional input</td> +<td valign="top">yes</td> +<td valign="top">no (?)</td><td> +</td></tr> +<tr> +<td valign="top">Windows Event Log</td> +<td valign="top">via <a href="http://www.eventreporter.com">EventReporter</a> +or <a href="http://www.mwagent.com">MonitorWare Agent</a> +(both commercial software)</td> +<td valign="top">via separate Windows agent, paid edition only</td> +</tr> + + +<tr> +<td colspan="3" valign="top"><b><br>Network (Protocol) Support</b><br></td> +</tr> +<tr> +<td valign="top">support for (plain) tcp based syslog</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">support for GSS-API</td> +<td valign="top">yes</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td valign="top">ability to limit the allowed +network senders (syslog ACLs)</td> +<td valign="top">yes</td> +<td valign="top">yes (?)</td> +</tr> +<tr> +<td valign="top">support for syslog-transport-tls +based framing on syslog/tcp connections</td> +<td valign="top">yes</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td valign="top">udp syslog</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">on the wire (zlib) message +compression</td> +<td valign="top">yes</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td valign="top">support for receiving messages via +reliable <a href="http://www.monitorware.com/Common/en/glossary/rfc3195.php">RFC +3195</a> delivery</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td valign="top">support for <a href="rsyslog_stunnel.html">ssl-protected +syslog</a> </td> +<td valign="top"><a href="rsyslog_stunnel.html">via +stunnel</a></td> +<td valign="top">via stunnel<br> +paid edition natively</td> +</tr> +<tr> +<td valign="top">support for IETF's new +syslog-protocol draft</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td valign="top">support for IPv6</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">native ability to send SNMP traps</td> +<td valign="top">yes</td> +<td valign="top">?</td> +</tr> +<tr> +<td valign="top">ability to preserve the original +hostname in NAT environments and relay chains</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> + + +<tr> +<td colspan="3" valign="top"><br><b>Message Filtering</b><br></td> +</tr> +<tr><td valign="top">Filtering for syslog facility and priority</td> +<td valign="top">yes</td> +<td valign="top">yes</td><td> +</td></tr> +<tr> +<td valign="top">Filtering for hostname</td> +<td valign="top">yes</td> +<td valign="top">yes</td><td> +</td></tr> +<tr> +<td valign="top">Filtering for application</td> +<td valign="top">yes</td> +<td valign="top">yes</td><td> +</td></tr> +<tr> +<td valign="top">Filtering for message contents</td> +<td valign="top">yes</td> +<td valign="top">yes</td><td> +</td></tr> +<tr> +<td valign="top">Filtering for sending IP address</td> +<td valign="top">yes</td> +<td valign="top">yes</td><td> +</td></tr> +<tr> +<td valign="top">ability to filter on any other message +field not mentioned above +(including substrings and the like)</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td>support for complex filters, using full boolean algebra +with and/or/not operators and parenthesis</td> +<td>yes</td> +<td>yes</td> +</tr> +<tr> +<td>Support for reusable filters: specify a filter once and +use it in multiple selector lines</td> +<td>no</td> +<td>yes</td> +</tr> +<tr> +<td>support for arbritrary complex arithmetic and string +expressions inside filters</td> +<td>yes</td> +<td>no</td> +</tr> +<tr> +<td valign="top">ability to use regular expressions +in filters</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">support for discarding messages +based on filters</td> +<td valign="top">yes</td> +<td valign="top">yes</td><td> +</td></tr> +<tr> +<td valign="top">powerful BSD-style hostname and +program name blocks for easy multi-host support</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td></td> +<td></td> +<td></td> +</tr> + + +<tr> +<td colspan="3" valign="top"><br><b>Supported Database Outputs</b><br></td> +</tr> +<tr> +<td valign="top">MySQL</td> +<td valign="top"><a href="rsyslog_mysql.html">yes</a> +(native ommysql, <a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">yes (via libdibi)</td> +</tr> +<tr> +<td valign="top">PostgreSQL</td> +<td valign="top">yes (native ompgsql, <a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">yes (via libdibi)</td> +</tr> +<tr> +<td valign="top">Oracle</td> +<td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">yes (via libdibi)</td> +</tr> +<tr> +<td valign="top">SQLite</td> +<td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">yes (via libdibi)</td> +</tr> +<tr> +<td valign="top">Microsoft SQL (Open TDS)</td> +<td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td valign="top">Sybase (Open TDS)</td> +<td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td valign="top">Firebird/Interbase</td> +<td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td valign="top">Ingres</td> +<td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td valign="top">mSQL</td> +<td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">no (?)</td> +</tr> + + +<tr> +<td colspan="3" valign="top"><br><b>Enterprise Features</b><br></td> +</tr> +<tr> +<td valign="top">support for on-demand on-disk +spooling of messages</td> +<td valign="top">yes</td> +<td valign="top">paid edition only</td> +</tr> +<tr> +<td valign="top">ability to limit disk space used +by spool files</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">each action can use its own, independant +set of spool files</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td valign="top">different sets of spool files can +be placed on different disk</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td valign="top">ability to configure backup +syslog/database servers </td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td>Professional Support</td> +<td><a href="professional_support.html">yes</a></td> +<td>yes</td> +</tr> + + +<tr> +<td colspan="3" valign="top"><br><b>Config File</b><br></td> +</tr> +<tr> +<td valign="top">config file format</td> +<td valign="top">compatible to legacy syslogd but +ugly</td> +<td valign="top">clean but not backwards compatible</td> +</tr> +<tr> +<td valign="top">ability to include config file from +within other config files</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td height="25" valign="top">ability to +include all config files +existing in a specific directory</td> +<td height="25" valign="top">yes</td> +<td height="25" valign="top">no</td> +</tr> + + + +<tr> +<td colspan="3" valign="top"><br><b>Extensibility</b><br></td> +</tr> +<tr> +<td valign="top">Functionality split in separately loadable +modules</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td valign="top">Support for third-party input plugins</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +</tr> +<tr><td valign="top">Support for third-party output plugins</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> + + + +<tr> +<td colspan="3" valign="top"><br><b>Other Features</b><br></td> +</tr> +<tr> +</tr><tr> +<td valign="top">ability to generate file names and +directories (log targets) dynamically</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">control of log output format, +including ability to present channel and priority as visible log data</td> +<td valign="top">yes</td> +<td valign="top">not sure...</td> +</tr> +<tr> +<td valign="top">good timestamp format control; at a +minimum, ISO 8601/RFC 3339 second-resolution UTC zone</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">ability to reformat message +contents and work with substrings</td> +<td valign="top">yes</td> +<td valign="top">I think yes</td> +</tr> +<tr> +<td valign="top">support for log files larger than +2gb</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">support for log file size +limitation +and automatic rollover command execution</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">support for running multiple +syslogd instances on a single machine</td> +<td valign="top">yes</td> +<td valign="top">? (but I think yes)</td> +</tr> +<tr> +<td valign="top">ability to execute shell scripts on +received messages</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">ability to pipe messages to a +continously running program</td> +<td valign="top">no</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">massively multi-threaded for +tomorrow's multi-core machines</td> +<td valign="top">yes</td> +<td valign="top">no (only multithreaded with +database destinations)</td> +</tr> +<tr> +<td valign="top">ability to control repeated line +reduction ("last message repeated n times") on a per selector-line basis</td> +<td valign="top">yes</td> +<td valign="top">yes (?)</td> +</tr> +<tr> +<td valign="top">supports multiple actions per +selector/filter condition</td> +<td valign="top">yes</td> +<td valign="top">yes</td><td> +</td></tr> +<tr> +<td valign="top">web interface</td> +<td valign="top"><a href="http://www.phplogcon.org">phpLogCon</a><br> +[also works with <a href="http://freshmeat.net/projects/php-syslog-ng/"> +php-syslog-ng</a>]</td> +<td valign="top"><a href="http://freshmeat.net/projects/php-syslog-ng/"> +php-syslog-ng</a></td> +</tr> +<tr> +<td valign="top">using text files as input source</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">rate-limiting output actions</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">discard low-priority messages under +system stress</td> +<td valign="top">yes</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td height="43" valign="top">flow control +(slow down message reception when system is busy)</td> +<td height="43" valign="top">yes (advanced, with multiple ways to slow down inputs depending on individual input capabilities, based on watermarks)</td> +<td height="43" valign="top">yes (limited? "stops accepting messages")</td> +</tr> +<tr> +<td valign="top">rewriting messages</td> +<td valign="top">yes</td> +<td valign="top">yes (at least I think so...)</td> +</tr> +<tr> +<td valign="top">output data into various formats</td> +<td valign="top">yes</td> +<td valign="top">yes (looks somewhat limited to me)</td> +</tr> +<tr> +<td valign="top">ability to control "message +repeated n times" generation</td> +<td valign="top">yes</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td valign="top">license</td> +<td valign="top">GPLv3 (GPLv2 for v2 branch)</td> +<td valign="top">GPL (paid edition is closed source)</td> +</tr> +<tr> +<td valign="top">supported platforms</td> +<td valign="top">Linux, BSD, anecdotical seen on +Solaris; compilation and basic testing done on HP UX</td> +<td valign="top">many popular *nixes</td> +</tr> +<tr> +<td valign="top">DNS cache</td> +<td valign="top">no</td> +<td valign="top">yes</td> +</tr> + + +</tbody> +</table> +<p>While the <span style="font-weight: bold;">rsyslog</span> +project was initiated in 2004, it <span style="font-weight: bold;">is +build on the main author's (Rainer Gerhards) 12+ years of +logging experience</span>. Rainer, for example, also +wrote the first <a href="http://www.winsyslog.com/Common/en/News/WinSyslog-1996-03-31.php">Windows +syslog server</a> in early 1996 and invented the <a href="http://www.eventreporter.com/Common/en/News/EvntSLog-1997-03-23.php">eventlog-to-syslog</a> +class of applications in early 1997. He did custom logging development +and consulting even before he wrote these products. Rsyslog draws on +that vast experience and sometimes even on the code.</p> +<p>Based on a discussion I had, I also wrote about the <b>political +argument why it is good to have another strong syslogd besides syslog-ng</b>. +You may want to read it at my blog at "<a href="http://rgerhards.blogspot.com/2007/08/why-does-world-need-another-syslogd.html">Why +does the world need another syslogd?</a>".</p> +<p>Balabit, the vendor of syslog-ng, has just recently done a +feature sheet. I have not yet been able to fully work through it. In +the mean time, you may want to read it in parallel. It is available at +<a href="http://www.balabit.com/network-security/syslog-ng/features/detailed/">Balabit's +site</a>.</p> +<p>This document is current as of 2008-02-28 and definitely +incomplete (I did not yet manage to complete it!).</p> +</body></html>
\ No newline at end of file diff --git a/doc/rsyslog_recording_pri.html b/doc/rsyslog_recording_pri.html index 48852ca..1dcf00c 100644 --- a/doc/rsyslog_recording_pri.html +++ b/doc/rsyslog_recording_pri.html @@ -1,5 +1,5 @@ <html><head> -<title>Writing syslog Data to MySQL</title> +<title>Recording the Priority of Syslog Messages</title> <meta name="KEYWORDS" content="syslog, mysql, syslog to mysql, howto"> </head> <body> diff --git a/doc/src/classes.dia b/doc/src/classes.dia Binary files differnew file mode 100644 index 0000000..70e9156 --- /dev/null +++ b/doc/src/classes.dia diff --git a/doc/status.html b/doc/status.html index cc19731..d7111a5 100644 --- a/doc/status.html +++ b/doc/status.html @@ -1,48 +1,49 @@ -<html> -<head> -<title>rsyslog status page</title> -</head> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>rsyslog status page</title></head> <body> <h2>rsyslog status page</h2> -<p>This page reflects the status as of 2008-03-27.</p> +<p>This page reflects the status as of 2008-04-04.</p> <h2>Current Releases</h2> -p><b>development:</b> 3.12.4 - -<a href="http://www.rsyslog.com/Article195.phtml">change -log</a> - -<a href="http://www.rsyslog.com/Downloads-index-req-getit-lid-89.phtml">download</a></p> -<p><b><font color="#ff0000"><a href="v3compatibility.html">If you used version 2, be sure to read the rsyslog v3 -compatibility document!</a></font></b><br> -Documentation for 3.x is currently partly sparse. If you need -assistance, please -<a href="http://www.rsyslog.com/PNphpBB2.phtml">post in -the rsyslog forums</a>!</p> -<p><b>stable:</b> 2.0.4 - <a href="http://www.rsyslog.com/Article197.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-index-req-getit-lid-90.phtml">download</a></p> -<p> (<a href="version_naming.html">How are versions named?</a>)</p> + +<p><b>development:</b> 3.15.0 - +<a href="http://www.rsyslog.com/Article203.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-index-req-getit-lid-93.phtml">download</a></p> + +<p><b>v3 stable:</b> 3.14.0 - <a href="http://www.rsyslog.com/Article205.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-index-req-getit-lid-94.phtml">download</a> + +<br><b>v2 stable:</b> 2.0.4 - <a href="http://www.rsyslog.com/Article197.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-index-req-getit-lid-90.phtml">download</a> +<br>v0 and v1 are depricated and no longer supported. If you absolutely do not like to +upgrade, you may consider purchasing a +<a href="professional_support.html">commercial rsyslog support package</a>. Just let us point +out that it is really not a good idea to still run a v0 version. + +<p><a href="v3compatibility.html">If you updgrade from version 2, be sure to read the rsyslog v3 +compatibility document.</a></p> +<p>(<a href="version_naming.html">How are versions named?</a>)</p> + <h2>Platforms</h2> -<p>Thankfully, a number of folks have begin to build packages and help port -rsyslog to other platforms. As such, -<a href="http://wiki.rsyslog.com/index.php/Platforms">the platform list is now -maintained inside the rsyslog wiki</a>. Platform maintainers perhaps have posted -extra information there. If you do platform-specific work, feel free to add -information to the wiki.</p> -<p>Rsyslog is the default syslogd in Fedora 8.</p> +<p>Thankfully, a number of folks have begin to build packages and +help port rsyslog to other platforms. As such, +<a href="http://wiki.rsyslog.com/index.php/Platforms">the +platform list is now maintained inside the rsyslog wiki</a>. +Platform maintainers perhaps have posted extra information there. If +you do platform-specific work, feel free to add information to the wiki.</p> +<p>Rsyslog is the default syslogd in Fedora 8 and above.</p> <h2>Additional information</h2> <p><b>Currently supported features are listed on the <a href="features.html">rsyslog features page</a>.</b></p> <ul> - <li>The rsyslog home page is <a href="http://www.rsyslog.com">www.rsyslog.com</a>.</li> - <li>Mailing list info can be found at - <a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">http://lists.adiscon.com/rsyslog</a>.</li> - <li>The change log can be found at - <a href="http://www.rsyslog.com/Topic4.phtml"> - http://www.rsyslog.com/Topic4.phtml</a>. </li> - <li>Online documentation is available at - <a href="http://www.rsyslog.com/doc">http://www.rsyslog.com/doc</a>.</li> - <li>You may also find <a href="http://rgerhards.blogspot.com/">Rainer's blog</a> an interesting read.</li> +<li>The rsyslog home page is <a href="http://www.rsyslog.com">www.rsyslog.com</a>.</li> +<li>Mailing list info can be found at <a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">http://lists.adiscon.com/rsyslog</a>.</li> +<li>The change log can be found at <a href="http://www.rsyslog.com/Topic4.phtml"> +http://www.rsyslog.com/Topic4.phtml</a>. </li> +<li>Online documentation is available at <a href="http://www.rsyslog.com/doc">http://www.rsyslog.com/doc</a>.</li> +<li>You may also find <a href="http://rgerhards.blogspot.com/">Rainer's blog</a> +an interesting read.</li> </ul> -<p>The project was initiated in 2004 by +<p>The project was initiated in 2003 and seriouosly begun in 2004 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> -and is currently being maintained by him. See the <a href="history.html">history -page</a> for more background information.</p> -</body> -</html> +and is currently being maintained by him. See the <a href="history.html">history page</a> for more +background information.</p> +</body></html> diff --git a/doc/v3compatibility.html b/doc/v3compatibility.html new file mode 100644 index 0000000..c6e4fec --- /dev/null +++ b/doc/v3compatibility.html @@ -0,0 +1,196 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>Compatibility notes for rsyslog v3</title> + +<meta name="KEYWORDS" content="syslog, mysql, syslog to mysql, howto"></head> +<body> +<h1>Compatibility Notes for rsyslog v4</h1> +<p><small><i>Written by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> +(2008-03-28)</i></small></p> +<p>Rsyslog aims to be a drop-in replacement for sysklogd. +However, version 3 has some considerable enhancements, which lead to +some backward compatibility issues both in regard to sysklogd and +rsyslog v1 and v2. Most of these issues are avoided by default by not +specifying the -c option on the rsyslog command line. That will enable +backwards-compatibility mode. However, please note that things may be +suboptimal in backward compatibility mode, so the advise is to work +through this document, update your rsyslog.conf, remove the no longer +supported startup options and then add -c3 as the first option to the +rsyslog command line. That will enable native mode.</p> +<p>Please note that rsyslogd helps you during that process by +logging appropriate messages about compatibility mode and +backwards-compatibility statemtents automatically generated. You may +want your syslogd log for those. They immediately follow rsyslogd's +startup message.</p> +<h2>Inputs</h2> +<p>With v2 and below, inputs were automatically started together +with rsyslog. In v3, inputs are optional! They come in the form of +plug-in modules. +<font color="#ff0000"><b>At least one input module +must be loaded to make rsyslog do any useful work.</b></font> +The config file directives doc briefly lists which config statements +are available by which modules.</p> +<p>It is suggested that input modules be loaded in the top part +of the config file. Here is an example, also highlighting the most +important modules:</p> +<p><b>$ModLoad immark # provides --MARK-- +message capability<br> +$ModLoad imudp # provides UDP syslog reception<br> +$ModLoad imtcp # provides TCP syslog reception<br> +</b><b>$ModLoad imgssapi # provides GSSAPI syslog +reception<br> +</b><b>$ModLoad imuxsock # provides support for local +system logging (e.g. +via logger command)<br> +$ModLoad imklog # provides kernel logging support (previously done +by rklogd)</b></p> +<h2>Command Line Options</h2> +<p>A number of command line options have been removed. New config +file directives have been added for them. The -h and -e option have +been removed even in compatibility mode. They are ignored but an +informative message is logged. Please note that -h was never supported +in v2, but was silently ignored. It disappeared some time ago in the +final v1 builds. It can be replaced by applying proper filtering inside +syslog.conf.</p> +<h2>-c option / Compatibility Mode</h2> +<p>The -c option is new and tells rsyslogd about the desired +backward compatibility mode. It must always be the first option on the +command line, as it influences processing of the other options. To use +the rsyslog v3 native +interface, specify -c3. To use compatibility mode , +either do not use -c at all or use -c<vers> where vers is +the +rsyslog version that it shall be compatible to. Use -c0 to be +command-line compatible to sysklogd.</p><p><span style="font-weight: bold;">Please note that rsyslogd issues warning messages if the -c3 command line option is not given.</span> +This is to alert you that your are running in compatibility mode. +Compatibility mode interfers with you rsyslog.conf commands and may +cause some undesired side-effects. It is meant to be used with a plain +old rsyslog.conf - if you use new features, things become messy. So the +best advise is to work through this document, convert your options and +config file and then use rsyslog in native mode. In order to aid you in +this process, rsyslog logs every compatibility-mode config file +directive it has generated. So you can simply copy them from your +logfile and paste them to the config.</p> +<h2>-e Option</h2> +This option is no longer supported, as the "last message repeated n +times" feature is now turned off by default. We changed this default +because this feature is causing a lot of trouble and we need to make it +either go away or change the way it works. For more information, please +see our dedicted <a href="http://www.rsyslog.com/PNphpBB2-viewtopic-p-1130.phtml">forum +thread on "last message repeated n times"</a>. This thread also +contains information on how to configure rsyslogd so that it continues +to support this feature (as long as it is not totally removed). +<h2>-m Option</h2> +<p>The -m command line option is emulated in compatibiltiy mode. +To replace it, use the following config directives (compatibility mode +auto-generates them):</p> +<p><b>$ModLoad immark<br> +$MarkMessageInterval 1800 # 30 minutes</b></p> +<h2>-r Option</h2> +<p>Is no longer available in native mode. However, it +is +understood in compatibility mode (if no -c option is given). Use the <b>$UDPSeverRun +<port></b> config file directives. You can now also +set the local address the server should listen to via <b>$UDPServerAddress +<ip></b> config directive.</p> +<p>The following example configures an UDP syslog server at the +local address 192.0.2.1 on port 514:</p> +<p><b>$ModLoad imudp.so<br> +$UDPSeverAddress 192.0.2.1 # this MUST be before the $UDPServerRun +directive!<br> +$UDPServerRun 514</b></p> +<p>"$UDPServerAddress *" means listen on all local interfaces. +This is the default if no directive is specified.</p> +<p>Please note that now multiple listeners are supported. For +example, you can do the following:</p> +<p><b>$ModLoad imudp.so<br> +$UDPSeverAddress 192.0.2.1 # this MUST be before the $UDPServerRun +directive!<br> +$UDPServerRun 514<br> +$UDPSeverAddress * # all local interfaces<br> +$UDPServerRun 1514</b></p> +<p>These config file settings run two listeners: one +at 192.0.2.1:514 and one on port 1514, which listens on all local +interfaces.</p> +<h2>Default port for UDP (and TCP) Servers</h2> +<p>Please note that with pre-v3 rsyslogd, a service database +lookup was made when a UDP server was started and no port was +configured. Only if that failed, the IANA default of 514 was used. For +TCP servers, this lookup was never done and 514 always used if no +specific port was configured. For consitency, both TCP and UDP now use +port 514 as default. If a lookup is desired, you need to specify it in +the "Run" directive, e.g. "<i>$UDPServerRun syslog</i>".</p> +<h2>klogd</h2> +<p>klogd has (finally) been replaced by a loadable input module. +To enable klogd functionality, do</p> +<p><b>$ModLoad imklog.so</b></p> +<p>Note that this can not be handled by the compatibility layer, +as klogd was a separate binary.A limited set of klogd command line +settings is now supported +via rsyslog.conf. That set of configuration directives is to be +expanded. </p> +<h2>Output File Syncing</h2> +Rsyslogd tries to keep as compatible to +stock syslogd as possible. As such, it retained stock syslogd's default +of syncing every file write if not specified otherwise (by placing a +dash in front of the output file name). While this was a useful feature +in past days where hardware was much less reliable and UPS seldom, this +no longer is useful in today's worl. Instead, the syncing is a high +performace hit. With it, rsyslogd writes files around 50 *times* slower +than without it. It also affects overall system performance due to the +high IO activity. In rsyslog v3, syncing has been turned off by +default. This is done via a specific configuration directive +"$ActionFileEnableSync on/off" which is off by default. So even if +rsyslogd finds sync selector lines, it ignores them by default. In +order to enable file syncing, the administrator must specify +"$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures +that syncing only happens in some installations where the administrator +actually wanted that (performance-intense) feature. In the fast +majority of cases (if not all), this dramatically increases rsyslogd +performance without any negative effects. +<h2>Output File Format</h2> +<p>Rsyslog supports high precision RFC 3339 timestamps and puts these into +local log files by default. This is a departure from previous syslogd +behaviour. We decided to sacrify some backward-compatibility in an +effort to provide a better logging solution. Rsyslog has been +supporting the high-precision timestamps for over three years as of +this writing, but nobody used them because they were not default (one +may also assume that most people didn't even know about them). Now, we +are writing the great high-precision time stamps, which greatly aid in +getting the right sequence of logging events. If you do not like that, +you can easily turn them off by placing +</p><p style="font-weight: bold;"><code>$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat</code> +</p><p>right at the start of your rsyslog.conf. This will use the +previous format. Please note that the name is case-sensitive and must +be specificed exactly as shown above. Please also note that you can of +course use any other format of your liking. To do so, simply specify +the template to use or set a new default template via the +$ActionFileDefaultTemplate directive. Keep in mind, though, that +templates must be defined before they are used.</p><p>Keep in mind that +when receiving messages from remote hosts, the timestamp is just as +precise as the remote host provided it. In most cases, this means you +will only a receive a standard timestamp with second precision. If +rsyslog is running at the remote end, you can configure it to provide +high-precision timestamps (see below).</p><h2>Forwarding Format</h2><p>When +forwarding messages to remote syslog servers, rsyslogd by default uses +the plain old syslog format with second-level resolution inside the +timestamps. We could have made it emit high precision timestamps. +However, that would have broken almost all receivers, including earlier +versions of rsyslog. To avoid this hassle, high-precision timestamps +need to be explicitely enabled. To make this as painless as possible, +rsyslog comes with a canned template that contains everything +necessary. To enable high-precision timestamps, just use:</p><p style="font-weight: bold;"><code>$ActionForwardDefaultTemplate RSYSLOG_ForwardFormat # for plain TCP and UDP</code></p><p style="font-weight: bold;"><code>$ActionGSSForwardDefaultTemplate RSYSLOG_ForwardFormat # for GSS-API</code></p><p>And, of course, you can always set different forwarding formats by just specifying the right template.</p><p>If +you are running in a system with only rsyslog 3.12.5 and above in the +receiver roles, it is suggested to add one (or both) of the above +statements to the top of your rsyslog.conf (but after the $ModLoad's!) +- that will enable you to use the best in timestamp support availble. +Please note that when you use this format with other receivers, they +will probably become pretty confused and not detect the timestamp at +all. In earlier rsyslog versions, for example, that leads to +duplication of timestamp and hostname fields and disables the detection +of the orignal hostname in a relayed/NATed environment. So use the new +format with care. </p><h2>Queue Modes for the Main Message Queue</h2> +<p>Either "FixedArray" or "LinkedList" is recommended. "Direct" +is available, but should not be used except for a very good reason +("Direct" disables queueing and will potentially lead to message loss +on the input side).</p> +</body></html>
\ No newline at end of file diff --git a/doc/version_naming.html b/doc/version_naming.html index a685f5f..8c1b918 100644 --- a/doc/version_naming.html +++ b/doc/version_naming.html @@ -1,33 +1,130 @@ -<html> -<head> -<title>rsyslog bugs and annoyances</title> -</head> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>rsyslog version naming</title></head> <body> <h1>Version Naming</h1> -<p>This document briefly outlines the strategy for naming versions. It applies -to versions 1.0.0 and above. Versions below that are all unstable and have a -different naming schema.</p> -<p><b>Please note that version naming is currently being changed. There is a -<a href="http://rgerhards.blogspot.com/2007/08/on-rsyslog-versions.html">blog +<p style="font-weight: bold;">This is the proposal on how versions should be named in the future:</p><p>Rsyslog version naming has undergone a number of changes in +the past. Our sincere hopes is that the scheme outlined here will serve +us well for the future. In general, a three-number versioning scheme +with a potential development state indication is used. It follows this +pattern:</p> +<p>major.minor.patchlevel[-devstate]</p> +<p>where devstate has some forther structure: +-<releaseReason><releaseNumber></p> +<p>All stable builds come without the devstate part. All unstable +development version come with it.</p> +<p>The <span style="font-weight: bold;">major</span> +version is incremented whenever something really important happens. A +single new feature, even if important, does not justify an increase in +the major version. There is no hard rule when the major version needs +an increment. It mostly is a soft factor, when the developers and/or +the community think there has been sufficient change to justify that. +Major version increments are expected to happen quite infrequently, +maybe around once a year. A major version increment has important +implications from the support side: without support contracts, the +current major version's last stable release and the last stable release +of the version immediately below it are supported (Adiscon, the rsyslog +sponsor, offers <a href="professional_support.html">support contracts</a> covering all other versions).</p> +<p>The <span style="font-weight: bold;">minor</span> version is +incremented whenever a non-trivial new feature is planned to be added. +Triviality of a feature is simply determined by time estimated to +implement a feature. If that's more than a few days, it is considered a +non-trivial feature. Whenever a new minor version is begun, the desired +feature is identified and will be the primary focus of that major.minor +version. Trivial features may justify a new minor version if they +either do not look trivial from the user's point of view or change +something quite considerable (so we need to alert users). A minor +version increment may also be done for some other good reasons that the +developers have.</p> +<p>The <span style="font-weight: bold;">patchlevel</span> is incremented whenever there is a bugfix or very minor feature added to a (stable or development) release.</p><p>The <span style="font-weight: bold;">devstate</span> +is important during development of a feature. It helps the developers +to release versions with new features to the general public and in the +hope that this will result in some testing. To understand how it works, +we need to look at the release cycle: As already said, at the start of +a new minor version, a new non-trivial feature to be implemented in +that version is selected. Development on this feature begins. At the +current pace of development, getting initial support for such a +non-trivial feature typically takes between two and four weeks. During +this time, new feature requests come in. Also, we may find out that it +may be just the right time to implement some not yet targeted feature +requests. A reason for this is that the minor release's feature focus +is easier to implement if the other feature is implemented first. This +is a quite common thing to happen. So development on the primary focus +may hold for a short period while we implement something else. Even +unrelated, but very trivial feature requests (maybe an hour's worth of +time to implement), may be done in between. Once we have implemented +these things, we would like to release as quickly as possible (even +more if someone has asked for the feature). So we do not like to wait +for the original focus feature to be ready (what could take maybe three +more weeks). As a result, we release the new features. But that version +will also include partial code of the focus feature. Typically this +doesn't hurt as long as noone tries to use it (what of course would +miserably fail). But still, part of the new code is already in it. When +we release such a "minor-feature enhanced" but "focus-feature not yet +completed" version, we need a way to flag it. In current thinking, that +is using a "<span style="font-weight: bold;">-mf<version></span>" <span style="font-weight: bold;">devstate</span> +in the version number ("mf" stands for "minor feature"). Version +numbers for -mf releases start at 0 for the first release and are +monotonically incremented. Once the focus feature has been fully +implemented, a new version now actually supporting that feature will be +released. Now, the release reason is changed to the well-know "<span style="font-weight: bold;">-rc<version></span>" +where "rc" stands for release candidate. For the first release +candidate, the version starts at 0 again and is incremented +monotonically for each subsequent release. Please note that a -rc0 may +only have bare functionality but later -rc's have a richer one. If new +minor features are implemented and released once we have reached rc +stage, still a new rc version is issued. The difference between "mf" +and "rc" is simply the presence of the desired feature. No support is +provided for -mf versions once the first -rc version has been released. +And only the most current -rc version is supported.</p><p>The -rc is +removed and the version declared stable when we think it has undergone +sufficient testing and look sufficiently well. Then, it'll turn into a +stable release. Stable minor releases never receive non-trivial new +features. There may be more than one -rc releases without a stable +release present at the same time. In fact, most often we will work on +the next minor development version while the previous minor version is +still a -rc because it is not yet considered sufficiently stable.</p><p>Note: <span style="font-weight: bold;">the +absence of the -devstate part indicates that a release is stable. +Following the same logic, any release with a -devstate part is unstable.</span></p><p>A quick sample: </p><p>4.0.0 +is the stable release. We begin to implement relp, moving to +major.minor to 4.1. While we develop it, someone requests a trivial +feature, which we implement. We need to release, so we will have +4.1.0-mf0. Another new feature is requested, move to 4.1.0-mf2. A first +version of RELP is implemented: 4.1.0-rc0. A new trivial feature is +implemented: 4.1.0-rc1. Relp is being enhanced: 4.1.0-rc2. We now feel +RELP is good enough for the time being and begin to implement TLS on +plain /Tcp syslog: logical increment to 4.2. Now another new feature in +that tree: 4.2.0-mf0. Note that we now have 4.0.0 (stable) and +4.1.0-rc2 and 4.1.0-mf0 (both devel). We find a big bug in RELP coding. +Two new releases: 4.1.0-rc3, 4.2.0-mf1 (the bug fix acts like a +non-focus feature change). We release TLS: 4.2.0-rc0. Another RELP bug +fix 4.1.0-rc4, 4.2.0-rc1. After a while, RELP is matured: 4.1.0 +(stable). Now support for 4.0.x stable ends. It, however, is still +provided for 3.x.x (in the actual case 2.x.x, because v3 was under the +old naming scheme and now stable v3 was ever released).</p><p style="font-weight: bold;">This is how it is done so far:</p><p>This document briefly outlines the strategy for naming +versions. It applies to versions 1.0.0 and above. Versions below that +are all unstable and have a different naming schema.</p> +<p><b>Please note that version naming is currently being +changed. There is a +<a href="http://rgerhards.blogspot.com/2007/08/on-rsyslog-versions.html">blog post about future rsyslog versions</a>.</b></p> -<p>The major version is incremented whenever a considerate, major features have -been added. This is expected to happen quite infrequently.</p> -<p>The minor version number is incremented whenever there is "sufficient need" -(at the discretion of the developers). There is a notable difference between -stable and unstable branches. The <b>stable branch</b> always has a minor -version number in the range from 0 to 9. It is expected that the stable branch -will receive bug and security fixes only. So the range of minor version numbers -should be quite sufficient.</p> -<p>For the <b>unstable branch</b>, minor version numbers always start at 10 and -are incremented as needed (again, at the discretion of the developers). Here, -new minor versions include both fixes as well as new features (hopefully most of -the time). They are expected to be released quite often.</p> -<p>The patch level (third number) is incremented whenever a really minor thing -must be added to an existing version. This is expected to happen quite -infrequently.</p> -<p>In general, the unstable branch carries all new development. Once it -concludes with a sufficiently-enhanced, quite stable version, a new major stable -version is assigned.</p> - -</body> -</html> +<p>The major version is incremented whenever a considerate, major +features have been added. This is expected to happen quite infrequently.</p> +<p>The minor version number is incremented whenever there is +"sufficient need" (at the discretion of the developers). There is a +notable difference between stable and unstable branches. The <b>stable +branch</b> always has a minor version number in the range from 0 +to 9. It is expected that the stable branch will receive bug and +security fixes only. So the range of minor version numbers should be +quite sufficient.</p> +<p>For the <b>unstable branch</b>, minor version +numbers always start at 10 and are incremented as needed (again, at the +discretion of the developers). Here, new minor versions include both +fixes as well as new features (hopefully most of the time). They are +expected to be released quite often.</p> +<p>The patch level (third number) is incremented whenever a +really minor thing must be added to an existing version. This is +expected to happen quite infrequently.</p> +<p>In general, the unstable branch carries all new development. +Once it concludes with a sufficiently-enhanced, quite stable version, a +new major stable version is assigned.</p> +</body></html>
\ No newline at end of file |
