DevOps Observability FOSS Stack Tutorial

Prerequisites

For building this above we need some requirements

PS- These requirements can be adjusted to your preferences.

AWS Cloud- EC2

SCM- GitHub

OS- Ubuntu 20

CRI (Container runtime)- Docker

The FOSS Stack Consist of the following

1. Prometheus-

Prometheus is used to scrape the data metrics that we pull from exporters like node exporters. It used to push the metrics to Grafana also. It also has a TSDB (Time series database) for storing the metrics itself.

1. Grafana-

Grafana is a Data Visualization tool. Here we fetch the metrics from Prometheus and display them with the help of dashboards. We can integrate it with all available tools in the market.

1. Alert Manager-

Alert manager helps us to configure alert based on some rules using a config file. We will specify some alert rules in that, with the help of this we can identify the fired alert and will be notified using any medium like slack, email etc. So this is having a dashboard where all alerts can be seen and from Prometheus dashboard also we can see the configured rules for it.

1. Node Exporter-

Node exporter is the agent we are deploying to fetch the metrics and pushed to prometheus. So basically, node exporter can be deployed to collected the basic utilization of server metrics. While other agents are still available and can be used for pushing to prometheus

Source Code Repository Explained

├── LICENSE

├── README.md

├── SECURITY.md

├── alertmanager

│ └── alertmanager.yml

├── docker-compose.yaml

├── prometheus

│ ├── alertrules.yml

│ ├── prometheus.yml

│ └── targets.json

└── scripts

├── service-discovery.sh

└── userdata.sh

3 directories, 10 files

The alertmanager consist of alertmanager.yml file consists of the configuration for alertmanager. With the details consist of the email, slack notification details to be received.

The docker-compose.yaml includes our services prometheus, grafana, node-exporter, alertmanager. All services are mapped with a network called ‘monitor’ and restart flag as always.

The prometheus directory consist of alertrules.yml which is responsible for the alerts to be triggered from the prometheus UI to the alertmanager. The prometheus.yml config also we mapped the alertmanager endpoint to fetch, then we use file sd to scrape the metrics using targets.json file.

The scripts directory consist of service-discovery.sh that will automate the Public IP assignment to the prometheus.yml & targets.json. The userdata.sh updated with swap memory allocation & with all necessary packages installed.

Setting up FOSS Stack

Let’s SSH into our EC2 & let’s deploy our source code.

$ git clone https://github.com/Allu-Philip/DevOps-Observability.git

$ cd DevOps-Observability

After clone now we need to edit the configuration files prometheus.yml & targets.json. So we will run a script for automating this task.

$ sh scripts/servicediscovery.sh

After all changes in configs now lets spin up the containers

$ docker-compose -f monit.yml up -d

$ docker-compose -f monit.yml logs -f

For Example my public ip was 34.56.23.18

So let’s check the webpages as follows:

Prometheus- http://34.56.23.18:9090

Grafana- http://34.56.23.18:3000

Alertmanager- http://34.56.23.18:9093

Nodeexporter-http://34.56.23.18:9100

Let's now set up Grafana to work with our Prometheus endpoint. The default username and password for Grafana are admin. Now let's add the Prometheus data source to the Grafana dashboard settings by providing the Prometheus endpoint with the save &test option as seen below.

Let's import a dashboard with Linux metrics into Grafana now. We'll use ID 10180 for Linux metrics.

To sum it up, you now have all the tools you need to dive into this Observability stack. Enjoy your exploration.

References: -

https://prometheus.io/docs/guides/node-exporter/

https://grafana.com/grafana/dashboards/

https://prometheus.io/docs/alerting/latest/alertmanager/

https://grafana.com/docs/grafana/latest/

https://prometheus.io/docs/introduction/overview/