(For AWS) Configuring S3 Repository for Storing ElasticSearch Snapshots

Over time, the volume of data residing in the ES cluster may grow and impact the performance of search queries. A recommended mitigation for this is to provide the ability for ES to push its data to S3. The following shows how to configure both AWS infrastructure and ES to support this, using a dedicated service account:

1. Create S3 Bucket for storing ES snapshot

Perform this step using the AWS console or via API.

2. Create IAM policy restricting access to the bucket

Perform this step using the AWS console or via API. Create a policy as follows:

/* Create as 'Gestalt-Write-ElasticSearch-Snapshot-To-S3' allowing
R/W access to 'gestalt-logging-elasticsearch-snapshot' bucket for storing snapshots
*/

{
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::gestalt-logging-elasticsearch-snapshot"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::gestalt-logging-elasticsearch-snapshot/*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

3. Create IAM User and Attach Policy

Perform this step using the AWS console or via API. Gather the API Key and Secret

4. Configure ElasticSearch to push to S3 bucket

# Configure S3 repository in elasticsearch

s3_bucketname="gestalt-logging-elasticsearch-snapshot"
s3_endpoint="s3.us-east-1.amazonaws.com"
s3_access_key="..."
s3_secret_key="..."
es_url=http://<ES_HOST>:<ES_PORT>

echo "
{
  \"type\": \"s3\",
  \"settings\": {
    \"bucket\": \"$s3_bucketname\",
    \"endpoint\": \"$s3_endpoint\",
    \"access_key\": \"$s3_access_key\",
    \"secret_key\": \"$s3_secret_key\"
  }
}
" | http PUT $es_url/_snapshot/s3_repository