Open Distro for Elasticsearch - Installation

Elasticsearch ecosystem has been in the popular choice for data and analytics purposes. In recent development, Elasticsearch is also available as license based deployments. Apache 2.0 Licensed Elasticsearch which is free of cost and provides very basic features compared to paid/licensed Elastic stack.

Open Distro for Elasticsearch is an alternative which is open source(apache 2.0 licensed) and maintained by Amazon. Apart from providing Elasticsearch and Kibana, this distribution also contains security, event monitoring & alerting, performance analysis, SQL query features, SQL JDBC. Open Distro for Elasticsearch and Kibana are available as RPM and Docker containers.

Open Distro for Elasticsearch features -

Component	Purpose
Elasticsearch	Data store and search engine
Kibana	Search frontend and visualizations
Security	Authentication and access control for your cluster
Alerting	Receive notifications when your data meets certain conditions
SQL	Use SQL or a piped processing language to query your data
Index State Management	Automate index operations
KNN	Find “nearest neighbors” in your vector data
Performance Analyzer	Monitor and optimize your cluster
Anomaly Detection	Identify atypical data and receive automatic notifications

Click here to know more for installation options.

Running a DEV setup --

I have created a docker compose file to install Elasticsearch, Kibana, Logstash and Filebeat. You will be able to quickly test a complete elastic search setup in your develeopment machine. You can refer the repository

https://gitlab.com/chittapriya/opendistroelk-logstash-filebeat

Note:- vm.max_map_count has to updated as mentioned in the below section.

Running a single instance -

If you want to run container with less RAM allocated to docker, you may try to run individual container.

Running Open Distro for Elastic search -

docker run --restart always -d -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -v opendistro-elk-vol:/usr/share/elasticsearch/data amazon/opendistro-for-elasticsearch:1.12.0

Get the IP address of Elasticsearch container - 
docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ContainerName or ID
72.17.0.2
Running Kibana
docker run --restart always -d -p 5601:5601 -e "ELASTICSEARCH_URL=https://172.17.0.2" -e "ELASTICSEARCH_HOSTS=https://172.17.0.2:9200" amazon/opendistro-for-elasticsearch-kibana:1.12.0
http://localhost:5601/
Default credentials - 
 Username: admin 
 Password: admin

Running cluster mode (single host) -
You may increase vm.max_map_count and allocate at least 8GB RAM 
On LINUX
The vm.max_map_count setting should be set permanently in /etc/sysctl.conf:
/etc/sysctl.conf -- modify in this file
grep vm.max_map_count /etc/sysctl.conf

vm.max_map_count = 262144 

Apply changes  temp

sudo sysctl -w vm.max_map_count = 262144
ON WINDOWS
windows with Docker Desktop WSL 2 
wsl -d docker-desktop
sysctl -w vm.max_map_count=262144

Use docker compose file -- refer this link for more information
version: '3'
services:
  odfe-node1:
    image: amazon/opendistro-for-elasticsearch:1.12.0
    container_name: odfe-node1
    environment:
      - cluster.name=odfe-cluster
      - node.name=odfe-node1
      - discovery.seed_hosts=odfe-node1,odfe-node2
      - cluster.initial_master_nodes=odfe-node1,odfe-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the Elasticsearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - odfe-data1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - odfe-net
  odfe-node2:
    image: amazon/opendistro-for-elasticsearch:1.12.0
    container_name: odfe-node2
    environment:
      - cluster.name=odfe-cluster
      - node.name=odfe-node2
      - discovery.seed_hosts=odfe-node1,odfe-node2
      - cluster.initial_master_nodes=odfe-node1,odfe-node2
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - odfe-data2:/usr/share/elasticsearch/data
    networks:
      - odfe-net
  kibana:
    image: amazon/opendistro-for-elasticsearch-kibana:1.12.0
    container_name: odfe-kibana
    ports:
      - 5601:5601
    expose:

      - "5601"
    environment:
      ELASTICSEARCH_URL: https://odfe-node1:9200
      ELASTICSEARCH_HOSTS: https://odfe-node1:9200
    networks:
      - odfe-net

volumes:
  odfe-data1:
  odfe-data2:

networks:
  odfe-net: 
To start -> docker-compose up -d
To stop -> docker-compose down
To stop and delete all volume -> docker-compose down -v
Elasticsearch Command
curl -XGET https://localhost:9200 -u 'admin:admin' --insecure
curl -XGET https://localhost:9200/_cat/nodes?v -u 'admin:admin' --insecure
curl -XGET https://localhost:9200/_cat/plugins?v -u 'admin:admin' --insecure

For WINDOWS machine use "

curl -XGET https://localhost:9200 -u "admin:admin" --insecure
curl -XGET https://localhost:9200/_cat/nodes?v -u "admin:admin" --insecure
curl -XGET https://localhost:9200/_cat/plugins?v -u "admin:admin" --insecure

Kibana -- 
http://localhost:5601/
Managed Open Distro Elasticsearch(AWS) - 
AWS managed Elasticsearch

References -
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_set_vm_max_map_count_to_at_least_262144

Search This Blog

KNOWLEDGE realm