Imported Upstream version 1.0.3upstream/1.0.3

4 files changed, 183 insertions, 0 deletions

diff --git a/docs/reference/modules/gateway/fs.asciidoc b/docs/reference/modules/gateway/fs.asciidoc
new file mode 100644
index 0000000..8d765d3
--- /dev/null
+++ b/docs/reference/modules/gateway/fs.asciidoc
@@ -0,0 +1,39 @@
+[[modules-gateway-fs]]
+=== Shared FS Gateway
+
+*The shared FS gateway is deprecated and will be removed in a future
+version. Please use the
+<<modules-gateway-local,local gateway>>
+instead.*
+
+The file system based gateway stores the cluster meta data and indices
+in a *shared* file system. Note, since it is a distributed system, the
+file system should be shared between all different nodes. Here is an
+example config to enable it:
+
+[source,js]
+--------------------------------------------------
+gateway:
+    type: fs
+--------------------------------------------------
+
+[float]
+==== location
+
+The location where the gateway stores the cluster state can be set using
+the `gateway.fs.location` setting. By default, it will be stored under
+the `work` directory. Note, the `work` directory is considered a
+temporal directory with Elasticsearch (meaning it is safe to `rm -rf`
+it), the default location of the persistent gateway in work intentional,
+*it should be changed*.
+
+When explicitly specifying the `gateway.fs.location`, each node will
+append its `cluster.name` to the provided location. It means that the
+location provided can safely support several clusters.
+
+[float]
+==== concurrent_streams
+
+The `gateway.fs.concurrent_streams` allow to throttle the number of
+streams (per node) opened against the shared gateway performing the
+snapshot operation. It defaults to `5`.
diff --git a/docs/reference/modules/gateway/hadoop.asciidoc b/docs/reference/modules/gateway/hadoop.asciidoc
new file mode 100644
index 0000000..b55a4be
--- /dev/null
+++ b/docs/reference/modules/gateway/hadoop.asciidoc
@@ -0,0 +1,36 @@
+[[modules-gateway-hadoop]]
+=== Hadoop Gateway
+
+*The hadoop gateway is deprecated and will be removed in a future
+version. Please use the
+<<modules-gateway-local,local gateway>>
+instead.*
+
+The hadoop (HDFS) based gateway stores the cluster meta and indices data
+in hadoop. Hadoop support is provided as a plugin and installing is
+explained https://github.com/elasticsearch/elasticsearch-hadoop[here] or
+downloading the hadoop plugin and placing it under the `plugins`
+directory. Here is an example config to enable it:
+
+[source,js]
+--------------------------------------------------
+gateway:
+    type: hdfs
+    hdfs:
+        uri: hdfs://myhost:8022
+--------------------------------------------------
+
+[float]
+==== Settings
+
+The hadoop gateway requires two simple settings. The `gateway.hdfs.uri`
+controls the URI to connect to the hadoop cluster, for example:
+`hdfs://myhost:8022`. The `gateway.hdfs.path` controls the path under
+which the gateway will store the data.
+
+[float]
+==== concurrent_streams
+
+The `gateway.hdfs.concurrent_streams` allow to throttle the number of
+streams (per node) opened against the shared gateway performing the
+snapshot operation. It defaults to `5`.
diff --git a/docs/reference/modules/gateway/local.asciidoc b/docs/reference/modules/gateway/local.asciidoc
new file mode 100644
index 0000000..eb4c4f3
--- /dev/null
+++ b/docs/reference/modules/gateway/local.asciidoc
@@ -0,0 +1,57 @@
+[[modules-gateway-local]]
+=== Local Gateway
+
+The local gateway allows for recovery of the full cluster state and
+indices from the local storage of each node, and does not require a
+common node level shared storage.
+
+Note, different from shared gateway types, the persistency to the local
+gateway is *not* done in an async manner. Once an operation is
+performed, the data is there for the local gateway to recover it in case
+of full cluster failure.
+
+It is important to configure the `gateway.recover_after_nodes` setting
+to include most of the expected nodes to be started after a full cluster
+restart. This will insure that the latest cluster state is recovered.
+For example:
+
+[source,js]
+--------------------------------------------------
+gateway:
+    recover_after_nodes: 1
+    recover_after_time: 5m
+    expected_nodes: 2
+--------------------------------------------------
+
+[float]
+==== Dangling indices
+
+When a node joins the cluster, any shards/indices stored in its  local `data/`
+directory which do not already exist in the cluster will be imported into the
+cluster by default.  This functionality has two purposes:
+
+1. If a new master node is started which is unaware of the other indices in
+   the cluster, adding the old nodes will cause the old indices to be
+   imported, instead of being deleted.
+
+2. An old index can be added to an existing cluster by copying it to the
+   `data/` directory of a new node, starting the node and letting it join
+   the cluster. Once the index has been replicated to other nodes in the
+   cluster, the new node can be shut down and removed.
+
+The import of dangling indices can be controlled with the
+`gateway.local.auto_import_dangled` which accepts:
+
+[horizontal]
+`yes`::
+
+    Import dangling indices into the cluster (default).
+
+`close`::
+
+    Import dangling indices into the cluster state, but leave them closed.
+
+`no`::
+
+    Delete dangling indices after `gateway.local.dangling_timeout`, which
+    defaults to 2 hours.
diff --git a/docs/reference/modules/gateway/s3.asciidoc b/docs/reference/modules/gateway/s3.asciidoc
new file mode 100644
index 0000000..8f2f5d9
--- /dev/null
+++ b/docs/reference/modules/gateway/s3.asciidoc
@@ -0,0 +1,51 @@
+[[modules-gateway-s3]]
+=== S3 Gateway
+
+*The S3 gateway is deprecated and will be removed in a future version.
+Please use the <<modules-gateway-local,local
+gateway>> instead.*
+
+S3 based gateway allows to do long term reliable async persistency of
+the cluster state and indices directly to Amazon S3. Here is how it can
+be configured:
+
+[source,js]
+--------------------------------------------------
+cloud:
+    aws:
+        access_key: AKVAIQBF2RECL7FJWGJQ
+        secret_key: vExyMThREXeRMm/b/LRzEB8jWwvzQeXgjqMX+6br
+
+
+gateway:
+    type: s3
+    s3:
+        bucket: bucket_name
+--------------------------------------------------
+
+You’ll need to install the `cloud-aws` plugin, by running
+`bin/plugin install cloud-aws` before (re)starting elasticsearch.
+
+The following are a list of settings (prefixed with `gateway.s3`) that
+can further control the S3 gateway:
+
+[cols="<,<",options="header",]
+|=======================================================================
+|Setting |Description
+|`chunk_size` |Big files are broken down into chunks (to overcome AWS 5g
+limit and use concurrent snapshotting). Default set to `100m`.
+|=======================================================================
+
+[float]
+==== concurrent_streams
+
+The `gateway.s3.concurrent_streams` allow to throttle the number of
+streams (per node) opened against the shared gateway performing the
+snapshot operation. It defaults to `5`.
+
+[float]
+==== Region
+
+The `cloud.aws.region` can be set to a region and will automatically use
+the relevant settings for both `ec2` and `s3`. The available values are:
+`us-east-1`, `us-west-1`, `ap-southeast-1`, `eu-west-1`.