Elasticsearch Ecosystem - Part1

-
Elasticsearch is distributed, search and analysis engine. There are many use cases which are supported by Elasticsearch- Log Analysis, Monitoring of Infrastructures, APM, Security analysis, Business analysis and applications. Data gets ingested into Elasticsearch from various sources. Mainly logs and metrices are ingested into Elasticsearch. We can design ingestion pipeline to process and transform data before it is placed inside Elasticsearch. To enable this, Elasticsearch eco system has various tools which we will discuss.

We have 4 main areas in which tools can be categorized but these are not a hard categories and has overlapping features -
1. Data exporter/shipper - configured at data source
2. Data collector - listening for data from various log sources
3. Data storage - storing/indexing data
4. Data visualization - visualization of data

Data shipper:-

Elastic-beats and Fluent-bit can be used for log shipper. Both are light weight product and be installed as agent and mainly used for transport purposes.
Elastic-beats - use oss distribution if you are not using Elasticsearch licensed product.



Fluent Bit is an open source Log Processor and Forwarder which allows you to collect any data like metrics and logs from different sources, enrich them with filters and send them to multiple destinations. 

Data collector :-

Logstash and Fluentd can be used for logs collector.

Logstash 

works on three components - INPUT plugins, FILTER plugins and OUTPUT plugins.
Use oss distribution if you are not using Elasticsearch licensed product.

Fluentd

List of data source (Input plugins)
List of data output (Output plugins)

Fluentd vs Fluent-bit

Both Logstash and Fluentd will coexist for various use cases for legacy or modern architecture. For kubernetes architecture fluentd is preferred because it has built-in Docker logging driver and parser and due to this, containers do not need an extra agent to push the logs to fluentd.

Data storage :- 

Open Distro for Elasticsearch is open source and maintained by AMAZON.
Elasticsearch has both opensource and licensed product.

Data visualization :-


Kibana used for visualization tool.
https://www.elastic.co/kibana
https://opendistro.github.io/for-elasticsearch-docs/docs/kibana/



Comments

Post a Comment

Popular posts from this blog

Python Development Environment : pyenv & VS Code

-
Image
While working on Python projects, we need to support projects with multiple versions of Python. Managing multiple versions in one machine is adifficult task. But if you use pyenv it becomes easy to manage. What is pyenv? pyenv is a tool to manage multiple Python versions. You may not need to install System Python if you use pyenv. Installing pyenv [Windows] Follow instructions as mentioned in this link  . If you are using WINDOWS operating system the I would prefer Chocolatey package manager. Chocolatey Installation Use powershell with admin rights > choco install pyenv-win To test if the installation is successful  > pyenv To see available python installation > pyenv install -l If you do not see you python version then, > pyenv update Installing pyenv [WSL2 - LINUX/Ubuntu] Install Home Brew package manager. follow the the instructions HomeBrew  and HomeBrew Installations > /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew...
Read more

Open Distro for Elasticsearch - Installation - Part 2

-
Image
Elasticsearch ecosystem has been in the popular choice for data and analytics purposes. In recent development, Elasticsearch is also available as license based  deployments. Apache 2.0 Licensed Elasticsearch which is free of cost and provides very basic features compared to paid/licensed Elastic stack.   Open Distro for Elasticsearch is an alternative which is open source(apache 2.0 licensed) and maintained by Amazon. Apart from providing Elasticsearch and Kibana, this distribution also contains security, event monitoring & alerting, performance analysis, SQL query features, SQL JDBC. Open Distro for Elasticsearch and Kibana are available as RPM and Docker containers. Open Distro for Elasticsearch features -  Component Purpose Elasticsearch Data store and search engine Kibana Search frontend and visualizations Security Authentication and access control for your cluster Alerting Receive notifications when your data meets certain conditions SQL Use SQL ...
Read more

WSL 2 on Windows - LINUX based development environment and Docker

-
Image
Windows operating system provides "Windows Subsystem for Linux" which can be utilized to develop using Linux operating system and have Linux based tools. Windows operating system is providing a great interoperability so that it is very easy to operate between Linux distribution and base operating system. I still remember the old days where we had to do so many things to have Linux OS with the base operating system. Popular choices were to have dedicated partitions to have different Operating systems or have Virtual Machine to host any OS of your choice. For both the cases, the development experience was not that great. Modern development is based on Containerization technologies and for Windows operating system, Docker Desktop is a popular choice to create docker environment in Windows operating system. Docker Desktop was using Hyper-v based virtualization technology and Linux OS to provide required environment. Later it started supporting WSL to provide containerization capa...
Read more