diff options
Diffstat (limited to 'doc/omelasticsearch.html')
| -rw-r--r-- | doc/omelasticsearch.html | 177 |
1 files changed, 177 insertions, 0 deletions
diff --git a/doc/omelasticsearch.html b/doc/omelasticsearch.html new file mode 100644 index 0000000..618b706 --- /dev/null +++ b/doc/omelasticsearch.html @@ -0,0 +1,177 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html> + <head> + <meta content="en" http-equiv="Content-Language" /> + <title>Elasticsearch Output Module</title> + </head> + <body> + <p> + <a href="rsyslog_conf_modules.html">back</a></p> + <h1> + Elasticsearch Output Module</h1> + <p> + <b>Module Name: omelasticsearch</b></p> + <p> + <b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> + <p> + <b>Available since: </b>6.4.0+</p> + <p> + <b>Description</b>:</p> + <p> + This module provides native support for logging to <a href="http://www.elasticsearch.org/">Elasticsearch</a>.</p> + <p> + <b>Action Parameters</b>:</p> + <ul> + <li> + <b>server</b><br /> + Host name or IP address of the Elasticsearch server. Defaults to "localhost"</li> + <li> + <b>serverport</b><br /> + HTTP port to connect to Elasticsearch. Defaults to 9200</li> + <li> + <b>searchIndex</b><br /> + <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#index">Elasticsearch index</a> to send your logs to. Defaults to "system"</li> + <li> + <b>dynSearchIndex </b><on/<b>off</b>><br /> + Whether the string provided for <strong>searchIndex</strong> should be taken as a <a href="http://www.rsyslog.com/doc/rsyslog_conf_templates.html">template</a>. Defaults to "off", which means the index name will be taken literally. Otherwise, it will look for a template with that name, and the resulting string will be the index name. For example, let's assume you define a template named "date-days" containing "%timereported:1:10:date-rfc3339%". Then, with dynSearchIndex="on", if you say searchIndex="date-days", each log will be sent to and index named after the first 10 characters of the timestamp, like "2013-03-22".</li> + <li> + <b>searchType</b><br /> + <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#type">Elasticsearch type</a> to send your index to. Defaults to "events"</li> + <li> + <b>dynSearchType</b> <on/<strong>off</strong>><br /> + Like <strong>dynSearchIndex</strong>, it allows you to specify a <a href="http://www.rsyslog.com/doc/rsyslog_conf_templates.html">template</a> for <strong>searchType</strong>, instead of a static string.</li> + <li> + <strong>asyncrepl </strong><on/<strong>off</strong>><br /> + By default, an indexing operation returns after all <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#replica_shard">replica shards</a> have indexed the document. With asyncrepl="on" it will return after it was indexed on the <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#primary_shard">primary shard</a> only - thus trading some consistency for speed.</li> + <li> + <strong>timeout</strong><br /> + How long Elasticsearch will wait for a primary shard to be available for indexing your log before sending back an error. Defaults to "1m".</li> + <li> + <strong>template</strong><br /> + This is the JSON document that will be indexed in Elasticsearch. The resulting string needs to be a valid JSON, otherwise Elasticsearch will return an error. Defaults to:</li> + </ul> + <pre> +$template JSONDefault, "{\"message\":\"%msg:::json%\",\"fromhost\":\"%HOSTNAME:::json%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"timereported\":\"%timereported:::date-rfc3339%\",\"timegenerated\":\"%timegenerated:::date-rfc3339%\"}" +</pre> + <p> + Which will produce this sort of documents (pretty-printed here for readability):</p> + <ul> + </ul> + <pre> +{ + "message": " this is a test message", + "fromhost": "test-host", + "facility": "user", + "priority": "info", + "timereported": "2013-03-12T18:05:01.344864+02:00", + "timegenerated": "2013-03-12T18:05:01.344864+02:00" +}</pre> + <ul> + <li> + <strong>bulkmode </strong><on/<strong>off</strong>><br /> + The default "off" setting means logs are shipped one by one. Each in its own HTTP request, using the <a href="http://www.elasticsearch.org/guide/reference/api/index_.html">Index API</a>. Set it to "on" and it will use Elasticsearch's <a href="http://www.elasticsearch.org/guide/reference/api/bulk.html">Bulk API</a> to send multiple logs in the same request. The maximum number of logs sent in a single bulk request depends on your queue settings - usually limited by the <a href="http://www.rsyslog.com/doc/node35.html">dequeue batch size</a>. More information about queues can be found <a href="http://www.rsyslog.com/doc/node32.html">here</a>.</li> + <li> + <strong>parent</strong><br /> + Specifying a string here will index your logs with that string the parent ID of those logs. Please note that you need to define the <a href="http://www.elasticsearch.org/guide/reference/mapping/parent-field.html">parent field</a> in your <a href="http://www.elasticsearch.org/guide/reference/mapping/">mapping</a> for that to work. By default, logs are indexed without a parent.</li> + <li> + <strong>dynParent </strong><on/<strong>off</strong>><br /> + Using the same parent for all the logs sent in the same action is quite unlikely. So you'd probably want to turn this "on" and specify a <a href="http://www.rsyslog.com/doc/rsyslog_conf_templates.html">template</a> that will provide meaningful parent IDs for your logs.</li> + <li> + <strong>uid</strong><br /> + If you have basic HTTP authentication deployed (eg: through the <a href="https://github.com/Asquera/elasticsearch-http-basic">elasticsearch-basic plugin</a>), you can specify your user-name here.</li> + <li> + <strong>pwd</strong><br /> + Password for basic authentication.</li> + </ul> + <p> + <b>Samples:</b></p> + <p> + The following sample does the following:</p> + <ul> + <li> + loads the omelasticsearch module</li> + <li> + outputs all logs to Elasticsearch using the default settings</li> + </ul> + <pre> +module(load="omelasticsearch") +*.* action(type="omelasticsearch")</pre> + <p> + The following sample does the following:</p> + <ul> + <li> + loads the omelasticsearch module</li> + <li> + defines a template that will make the JSON contain the following properties (more info about what properties you can use <a href="http://www.rsyslog.com/doc/property_replacer.html">here</a>): + <ul> + <li> + RFC-3339 timestamp when the event was generated</li> + <li> + the message part of the event</li> + <li> + hostname of the system that generated the message</li> + <li> + severity of the event, as a string</li> + <li> + facility, as a string</li> + <li> + the tag of the event</li> + </ul> + </li> + <li> + outputs to Elasticsearch with the following settings + <ul> + <li> + host name of the server is myserver.local</li> + <li> + port is 9200</li> + <li> + JSON docs will look as defined in the template above</li> + <li> + index will be "test-index"</li> + <li> + type will be "test-type"</li> + <li> + activate bulk mode. For that to work effectively, we use an in-memory queue that can hold up to 5000 events. The maximum bulk size will be 300</li> + <li> + retry indefinitely if the HTTP request failed (eg: if the target server is down)</li> + </ul> + </li> + </ul> + <pre> +module(load="omelasticsearch") +template(name="testTemplate" + type="list" + option.json="on") { + constant(value="{") + constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339") + constant(value="\",\"message\":\"") property(name="msg") + constant(value="\",\"host\":\"") property(name="hostname") + constant(value="\",\"severity\":\"") property(name="syslogseverity-text") + constant(value="\",\"facility\":\"") property(name="syslogfacility-text") + constant(value="\",\"syslogtag\":\"") property(name="syslogtag") + constant(value="\"}") + } +*.* action(type="omelasticsearch" + server="myserver.local" + serverport="9200" + template="testTemplate" + searchIndex="test-index" + searchType="test-type" + bulkmode="on" + queue.type="linkedlist" + queue.size="5000" + queue.dequeuebatchsize="300" + action.resumeretrycount="-1")</pre> + <p> + </p> + <pre> +</pre> + <p> + [<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> + <p> + <font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br /> + Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the ASL 2.0.</font></p> + </body> +</html> + |