Libre Software People's Front

don't confuse it with People's Front of Open Source

Elasticsearch Snapshots in Digitalocean Spaces

leave a comment »

Elasticsearch snapshots are a really nice feature that you should master if you work with the Elastic stack. We at Bitergia use Elasticsearch clusters running on top of Digital Ocean droplets. This cloud provider offers an S3 compatible service named Spaces which can be set up to store those snapshots. Unluckily the documentation and examples available are very rare, so I thought this blog post may save you some time.

So let’s see an example about how to produce Elasticsearch 6.8.6 Snapshots in Digital Ocean spaces.

The first thing you need is to install in your cluster the plugin named ‘repository-s3’, which by the way is using an Apache 2 License. This plugin can be installed using the plugin manager:

$ sudo bin/elasticsearch-plugin install repository-s3

In case you are behind a proxy, you can also download the zip file and pass the file to the plugin manager. More info available at the documentation section of the S3 Repository Plugin.

Now, go to DigitalOcean and get the parameters from your Space, these are the ones we need:

  • ID and Key: the secrets to access to the space
  • endpoint: something like fra1.digitaloceanspaces.com
  • bucket name: I will use sanacl-testing in the examples below

In order to finish with the setup of Elasticsearch we have to store the secrets and the endpoint. Modify the file `elasticsearch.yml` to add a parameter with the endpoint:

# S3 compatible
s3.client.default.endpoint: fra1.digitaloceanspaces.com

Run the code below (with the correct credentials) in all your nodes:

AWS_ACCESS_KEY_ID='MY-EXAMPLE-ID'
AWS_SECRET_ACCESS_KEY='*****'
echo $AWS_ACCESS_KEY_ID | /usr/share/elasticsearch/bin/elasticsearch-keystore add --stdin s3.client.default.access_key
echo $AWS_SECRET_ACCESS_KEY | /usr/share/elasticsearch/bin/elasticsearch-keystore add --stdin s3.client.default.secret_key

Restart your Elasticsearch cluster and we are ready to start creating the snapshots. If your cluster does not start, have a look at the logs because any of the parameters above is incorrect.

Create the repository for the snapshots, the only mandatory setting is the bucket name:

$ cat query1.json
{
  "type": "s3",
  "settings": {
    "bucket": "sanacl-testing"
  }
}
$ curl -X PUT -k -H "Content-Type: application/json" -d @query1.json https://...:9200/_snapshot/es_backups

Now, create your first snapshot:


curl -X PUT -k -H "Content-Type: application/json" https://....:9200/_snapshot/es_backups/1

In case you are using Search Guard and want to restore some content, you will need both admin and key certificates. I’ve tested it restoring an index named ‘git’ like this:

$ cat query2.json
{
  “indices": "git,-searchguard",
  "include_global_state": false
}
$ curl -X POST -k -H "Content-Type: application/json" --cert admin_cert.pem --key admin_cert_key.pem -d @query2.json https://….:9200/_snapshot/es_backups/1/_restore

And that’s all folks, if you found a typo don’t hesitate to let me know. 

Have a nice day!

Written by sanacl

March 30, 2020 at 2:22 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: