Deploying ELK

How to deploy ELK for log monitoring

Currently ELK is only available in a swarm deployment because of how the ELK stack is configured.

Most of the set up necessary for ELK - configuration files and/or data directories - is already done as part of filesystem initialization performed while executing:

./ init-filesystems

The current ELK data directories are created on /data/ directory so this directory must exist.

One manual step is to set the vm.max_map_count value required for running elasticsearch. This is done by adding vm.max_map_count=262144 line to /etc/sysctl.conf and then run sysctl -p (typically this must be done as root)

After that deploying the elk stack for monitoring the application only requires starting with:

./ start-elk

The command to stop the monitoring is:

./ stop-elk

Import data from an old stack

To import data from an old stack the old stack nodes must be whitelisted in the ELK_WHITELIST environment variable so that they can be accessible to the current cluster for importing indexes.

Some useful elasticsearch endpoints:

  • List available indices
curl http://<escoordinator>:9200/_cat/indices
  • Import an index from a remote cluster:


curl -H 'Content-Type: application/json' -X POST -d "{
    \"source\": {
        \"remote\": {
            \"host\": \"http://${remoteHost}:9200\"
        \"index\": \"${indexName}\",
        \"query\": {
            \"match_all\": {}
    \"dest\": {
        \"index\": \"${indexName}\"
  • Export kibana objects

Here’s an example to export kibana visualizations but the command is identical for any one of [config, index-pattern, visualization, search, dashboard, url] - just set the appropriate type

curl http://<oldkibanahost>:5601/api/saved_objects/_export -H 'kbn-xsrf: true' \
     -H 'Content-Type: application/json' \
     -d '{"type": "visualization" }' > local/kibana-visualizations.ndjson
  • To import

Use the file generated by the above export command and run

curl -X POST \
    -H 'kbn-xsrf: true' \
    --form file=@local/kibana-visualizations.ndjson

Last modified May 10, 2022: imported jacs-cm documentation (b7ff1be)