Deploying ELK

Currently ELK is only available in a swarm deployment because of how the ELK stack is configured.

Most of the set up necessary for ELK - configuration files and/or data directories - is already done as part of filesystem initialization performed while executing:

./manage.sh init-filesystems

The current ELK data directories are created on /data/ directory so this directory must exist.

One manual step is to set the vm.max_map_count value required for running elasticsearch. This is done by adding vm.max_map_count=262144 line to /etc/sysctl.conf and then run sysctl -p (typically this must be done as root)

After that deploying the elk stack for monitoring the application only requires starting with:

./manage.sh start-elk

The command to stop the monitoring is:

./manage.sh stop-elk

Import data from an old stack

To import data from an old stack the old stack nodes must be whitelisted in the ELK_WHITELIST environment variable so that they can be accessible to the current cluster for importing indexes.

Some useful elasticsearch endpoints:

• List available indices

curl http://<escoordinator>:9200/_cat/indices

• Import an index from a remote cluster:

#!/bin/bash

remoteost=$1
indexName=$2

curl -H 'Content-Type: application/json' -X POST http://e03u08.int.janelia.org:9200/_reindex -d "{
    \"source\": {
        \"remote\": {
            \"host\": \"http://${remoteHost}:9200\"
        },
        \"index\": \"${indexName}\",
        \"query\": {
            \"match_all\": {}
        }
    },
    \"dest\": {
        \"index\": \"${indexName}\"
    }
}"

• Export kibana objects

Here’s an example to export kibana visualizations but the command is identical for any one of [config, index-pattern, visualization, search, dashboard, url] - just set the appropriate type

curl http://<oldkibanahost>:5601/api/saved_objects/_export -H 'kbn-xsrf: true' \
     -H 'Content-Type: application/json' \
     -d '{"type": "visualization" }' > local/kibana-visualizations.ndjson

• To import

Use the file generated by the above export command and run

curl -X POST http://e03u06.int.janelia.org:5601/api/saved_objects/_import \
    -H 'kbn-xsrf: true' \
    --form file=@local/kibana-visualizations.ndjson