diff options
Diffstat (limited to 'doc/omelasticsearch.html')
| -rw-r--r-- | doc/omelasticsearch.html | 177 |
1 files changed, 0 insertions, 177 deletions
diff --git a/doc/omelasticsearch.html b/doc/omelasticsearch.html deleted file mode 100644 index 618b706..0000000 --- a/doc/omelasticsearch.html +++ /dev/null @@ -1,177 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html> - <head> - <meta content="en" http-equiv="Content-Language" /> - <title>Elasticsearch Output Module</title> - </head> - <body> - <p> - <a href="rsyslog_conf_modules.html">back</a></p> - <h1> - Elasticsearch Output Module</h1> - <p> - <b>Module Name: omelasticsearch</b></p> - <p> - <b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> - <p> - <b>Available since: </b>6.4.0+</p> - <p> - <b>Description</b>:</p> - <p> - This module provides native support for logging to <a href="http://www.elasticsearch.org/">Elasticsearch</a>.</p> - <p> - <b>Action Parameters</b>:</p> - <ul> - <li> - <b>server</b><br /> - Host name or IP address of the Elasticsearch server. Defaults to "localhost"</li> - <li> - <b>serverport</b><br /> - HTTP port to connect to Elasticsearch. Defaults to 9200</li> - <li> - <b>searchIndex</b><br /> - <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#index">Elasticsearch index</a> to send your logs to. Defaults to "system"</li> - <li> - <b>dynSearchIndex </b><on/<b>off</b>><br /> - Whether the string provided for <strong>searchIndex</strong> should be taken as a <a href="http://www.rsyslog.com/doc/rsyslog_conf_templates.html">template</a>. Defaults to "off", which means the index name will be taken literally. Otherwise, it will look for a template with that name, and the resulting string will be the index name. For example, let's assume you define a template named "date-days" containing "%timereported:1:10:date-rfc3339%". Then, with dynSearchIndex="on", if you say searchIndex="date-days", each log will be sent to and index named after the first 10 characters of the timestamp, like "2013-03-22".</li> - <li> - <b>searchType</b><br /> - <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#type">Elasticsearch type</a> to send your index to. Defaults to "events"</li> - <li> - <b>dynSearchType</b> <on/<strong>off</strong>><br /> - Like <strong>dynSearchIndex</strong>, it allows you to specify a <a href="http://www.rsyslog.com/doc/rsyslog_conf_templates.html">template</a> for <strong>searchType</strong>, instead of a static string.</li> - <li> - <strong>asyncrepl </strong><on/<strong>off</strong>><br /> - By default, an indexing operation returns after all <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#replica_shard">replica shards</a> have indexed the document. With asyncrepl="on" it will return after it was indexed on the <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#primary_shard">primary shard</a> only - thus trading some consistency for speed.</li> - <li> - <strong>timeout</strong><br /> - How long Elasticsearch will wait for a primary shard to be available for indexing your log before sending back an error. Defaults to "1m".</li> - <li> - <strong>template</strong><br /> - This is the JSON document that will be indexed in Elasticsearch. The resulting string needs to be a valid JSON, otherwise Elasticsearch will return an error. Defaults to:</li> - </ul> - <pre> -$template JSONDefault, "{\"message\":\"%msg:::json%\",\"fromhost\":\"%HOSTNAME:::json%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"timereported\":\"%timereported:::date-rfc3339%\",\"timegenerated\":\"%timegenerated:::date-rfc3339%\"}" -</pre> - <p> - Which will produce this sort of documents (pretty-printed here for readability):</p> - <ul> - </ul> - <pre> -{ - "message": " this is a test message", - "fromhost": "test-host", - "facility": "user", - "priority": "info", - "timereported": "2013-03-12T18:05:01.344864+02:00", - "timegenerated": "2013-03-12T18:05:01.344864+02:00" -}</pre> - <ul> - <li> - <strong>bulkmode </strong><on/<strong>off</strong>><br /> - The default "off" setting means logs are shipped one by one. Each in its own HTTP request, using the <a href="http://www.elasticsearch.org/guide/reference/api/index_.html">Index API</a>. Set it to "on" and it will use Elasticsearch's <a href="http://www.elasticsearch.org/guide/reference/api/bulk.html">Bulk API</a> to send multiple logs in the same request. The maximum number of logs sent in a single bulk request depends on your queue settings - usually limited by the <a href="http://www.rsyslog.com/doc/node35.html">dequeue batch size</a>. More information about queues can be found <a href="http://www.rsyslog.com/doc/node32.html">here</a>.</li> - <li> - <strong>parent</strong><br /> - Specifying a string here will index your logs with that string the parent ID of those logs. Please note that you need to define the <a href="http://www.elasticsearch.org/guide/reference/mapping/parent-field.html">parent field</a> in your <a href="http://www.elasticsearch.org/guide/reference/mapping/">mapping</a> for that to work. By default, logs are indexed without a parent.</li> - <li> - <strong>dynParent </strong><on/<strong>off</strong>><br /> - Using the same parent for all the logs sent in the same action is quite unlikely. So you'd probably want to turn this "on" and specify a <a href="http://www.rsyslog.com/doc/rsyslog_conf_templates.html">template</a> that will provide meaningful parent IDs for your logs.</li> - <li> - <strong>uid</strong><br /> - If you have basic HTTP authentication deployed (eg: through the <a href="https://github.com/Asquera/elasticsearch-http-basic">elasticsearch-basic plugin</a>), you can specify your user-name here.</li> - <li> - <strong>pwd</strong><br /> - Password for basic authentication.</li> - </ul> - <p> - <b>Samples:</b></p> - <p> - The following sample does the following:</p> - <ul> - <li> - loads the omelasticsearch module</li> - <li> - outputs all logs to Elasticsearch using the default settings</li> - </ul> - <pre> -module(load="omelasticsearch") -*.* action(type="omelasticsearch")</pre> - <p> - The following sample does the following:</p> - <ul> - <li> - loads the omelasticsearch module</li> - <li> - defines a template that will make the JSON contain the following properties (more info about what properties you can use <a href="http://www.rsyslog.com/doc/property_replacer.html">here</a>): - <ul> - <li> - RFC-3339 timestamp when the event was generated</li> - <li> - the message part of the event</li> - <li> - hostname of the system that generated the message</li> - <li> - severity of the event, as a string</li> - <li> - facility, as a string</li> - <li> - the tag of the event</li> - </ul> - </li> - <li> - outputs to Elasticsearch with the following settings - <ul> - <li> - host name of the server is myserver.local</li> - <li> - port is 9200</li> - <li> - JSON docs will look as defined in the template above</li> - <li> - index will be "test-index"</li> - <li> - type will be "test-type"</li> - <li> - activate bulk mode. For that to work effectively, we use an in-memory queue that can hold up to 5000 events. The maximum bulk size will be 300</li> - <li> - retry indefinitely if the HTTP request failed (eg: if the target server is down)</li> - </ul> - </li> - </ul> - <pre> -module(load="omelasticsearch") -template(name="testTemplate" - type="list" - option.json="on") { - constant(value="{") - constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339") - constant(value="\",\"message\":\"") property(name="msg") - constant(value="\",\"host\":\"") property(name="hostname") - constant(value="\",\"severity\":\"") property(name="syslogseverity-text") - constant(value="\",\"facility\":\"") property(name="syslogfacility-text") - constant(value="\",\"syslogtag\":\"") property(name="syslogtag") - constant(value="\"}") - } -*.* action(type="omelasticsearch" - server="myserver.local" - serverport="9200" - template="testTemplate" - searchIndex="test-index" - searchType="test-type" - bulkmode="on" - queue.type="linkedlist" - queue.size="5000" - queue.dequeuebatchsize="300" - action.resumeretrycount="-1")</pre> - <p> - </p> - <pre> -</pre> - <p> - [<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> - <p> - <font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br /> - Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the ASL 2.0.</font></p> - </body> -</html> - |