Introduction to Monitoring EC2 Instances with Grafana, Loki, and Promtail
In modern cloud-based environments like Amazon Web Services (AWS), monitoring the performance and health of your EC2 instances is crucial for maintaining system reliability and optimizing resource usage. Grafana, coupled with Loki and Promtail, offers a robust solution for visualizing metrics and logs from EC2 instances in a centralized and user-friendly manner.
Grafana
Grafana is an open-source platform used for monitoring, visualization, and analytics of time-series data. It allows users to create customizable dashboards and visualizations to observe and analyze metrics from various sources, such as databases, monitoring systems, and applications.
Loki
Loki is a highly-scalable log aggregation system designed to efficiently store and query logs from distributed systems. As a cost-effective and operationally simple solution, Loki seamlessly integrates with Grafana to provide powerful log monitoring capabilities for your EC2 instances.
Promtail
Promtail serves as an agent responsible for collecting logs from local files on your EC2 instances and forwarding them to Loki for storage and analysis. Its lightweight nature and straightforward deployment make it an ideal tool for monitoring logs in cloud environments.
Creating a dashboard using Grafana with the integration of Loki and Promtail
Steps involved:
Set up your EC2 instance. You can refer HERE.
Install Docker: We'll install Loki and Promtail using Docker, so Docker has to be installed first.
sudo apt-get update
sudo apt install docker.io
sudo usermod -aG docker $USER
sudo reboot
Download Loki Config file for Loki to run Loki in the Docker container
Create a folder and download the yaml file that will contain the required configuration for running Loki.
mkdir grafana_configs
cd grafana_configs
wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml
Run the docker container using the following command.
sudo docker run -d --name loki -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.8.0 --config.file=/mnt/config/loki-config.yaml
Open port 3100 in the security group to make the loki container running.
Navigate to the public url of the EC2 instance, "public URL:3100/ready"
You can also see the metrics which is the logs and the main purpose of Loki to collect and use "public URL:3100/metrics"
Download Promtail Config for Promtail to run in the Docker container
wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail
Run the Promtail docker container using the following command.
sudo docker run -d --name promtail -v $(pwd):/mnt/config -v /var/log:/var/log --link loki grafana/promtail:2.8.0 --config.file=/mnt/config/promtail-config.yaml
Add Loki as a Data Source in Grafana
In Grafana, click on "Data Sources".
Enter the URL of your Loki instance (e.g., http://localhost:3100).
Click "Save & test" to ensure Grafana can connect to Loki.
In the "Label filters" you can choose "job" and "varlogs" to show all the system logs. Then click "Run query".
Create a Dashboard
In "Label filters" choose "job" and "varlogs" and "line contains" to "error" to show all the lines with error.
Let's check the error lines in grafana log that is placed in /var/log/grafana/grafana.log
To fulfill the primary goal of displaying logs in Grafana, the path to the Grafana logs must be specified within the targets section of the Promtail configuration YAML file.
After editing the promtail_config.yaml file, restart your Promtail Docker container."
We can now choose the "Label filters" to set the "job" and "grafanalogs" with the line contains and visualization option to view in a graphical manner. We can add this to our dashboard.
Now, install nginx on the EC2 instance
To count how often 'nginx' appears during installation, use 'varlogs' as the label filter.
Here is the complete Grafana dashboard
Thank you for reading.