Monitoring is an essential part of DevOps. Every site you deploy, every app you deploy has to be continuously monitored since you never know when an error might arise. An error, even a small one can cause a catastrophic failure and cause the site or app to go down instantly.
Here comes the role of monitoring. Monitoring allows us to continuously track the state of the site or app and swiftly take action in case the infrastructure goes down.
Fortunately, a lot of monitoring tools are available in the market. One of these tools is Prometheus. It is a great open-source tool for real-time monitoring and alerting. It records real-time metrics in a time series database built using an HTTP pull model. These metrics tell us the current state of the site or app etc., in real-time.
However, keeping a track of these metrics is not very convenient due to their form. An average person always prefers an interactive and graphical representation over textual. Here the role of Grafana comes into play. It is multi-platform open-source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources. When grafana is configured with Prometheus as the data source, it can represent the metrics is an interactive GUI with graphs, charts, etc.
Pre-requisites:
- Minikube installed and running
- kubectl configured and added to the path
- A data source which is constantly exporting the metrics( For this task I am using a node exporter that is constantly exporting the metrics of a Linux VM at port number 9100 )
Manifest file
To deploy the setup I am going to use 2 main Kubernetes manifest file
Manifest file for Prometheus
apiVersion: v1 kind: Service metadata: name: prometheus labels: app: prometheus spec: ports: - port: 9090 nodePort: 30000 selector: app: prometheus tier: backend type: NodePort --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: prometheus-pv-claim labels: app: prometheus spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi --- apiVersion: v1 kind: ConfigMap metadata: name: prometheus-config labels: name: prometheus-config data: prometheus.yml: |- global: scrape_interval: 5s evaluation_interval: 5s alerting: alertmanagers: - static_configs: - targets: #rule_files: # - "first_rules.yml" # - "second_rules.yml"scrape_configs: - job_name: 'prometheus' #metrics default path is /metrics static_configs: - targets: ['localhost:9090']- job_name: 'node1' static_configs: - targets: ['192.168.99.101:9100'] --- apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 kind: Deployment metadata: name: prometheus labels: app: prometheus tier: backend spec: selector: matchLabels: app: prometheus tier: backend strategy: type: Recreate template: metadata: labels: app: prometheus tier: backend spec: containers: - image: prom/prometheus:latest name: prometheus args: - "--config.file=/etc/prometheus/prometheus.yml" ports: - containerPort: 9090 name: prometheus volumeMounts: - name: prometheus-persistent-storage mountPath: /prometheus - name: prometheus-config mountPath: /etc/prometheus/ volumes: - name: prometheus-persistent-storage persistentVolumeClaim: claimName: prometheus-pv-claim - name: prometheus-config configMap: name: prometheus-config
|
This file will deploy, configure and expose Prometheus at port 30000
Manifest file for Grafana
apiVersion: v1 kind: Service metadata: name: grafana labels: app: grafana spec: ports: - port: 3000 nodePort: 30001 selector: app: grafana tier: frontend type: NodePort --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: grafana-pv-claim labels: app: grafana spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi --- apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 kind: Deployment metadata: name: grafana labels: app: grafana tier: frontend spec: selector: matchLabels: app: grafana tier: frontend strategy: type: Recreate template: metadata: labels: app: grafana tier: frontend spec: containers: - image: grafana/grafana:latest name: grafana ports: - containerPort: 3000 name: grafana volumeMounts: - name: grafana-persistent-storage mountPath: /var/lib/grafana volumes: - name: grafana-persistent-storage persistentVolumeClaim: claimName: grafana-pv-claim
|
This file will deploy, configure and expose Grafana at port 30001
Process:
We will first create the Service for prometheus. This service will expose prometheus at port no. 30000. Due to this we are able to access the prometheus WebUI at http://<minikube ip>:30000
apiVersion: v1 kind: Service metadata: name: prometheus labels: app: prometheus spec: ports: - port: 9090 nodePort: 30000 selector: app: prometheus tier: backend type: NodePort
|
Creating a persistent volume claim. This claim allows us to create a volume that will store some data. In this case, we will configure it to store the metrics and allot it 20 GiB of space.
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: prometheus-pv-claim labels: app: prometheus spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi
|
Creating the config file for Prometheus. This is very important as in this file we configure Prometheus to import the metrics from node exporter
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
labels:
name: prometheus-config
data:
prometheus.yml: |-
global:
scrape_interval: 5s
evaluation_interval: 5s
alerting:
alertmanagers:
- static_configs:
- targets:
#rule_files:
# - "first_rules.yml"
# - "second_rules.yml"scrape_configs:
- job_name: 'prometheus'
#metrics default path is /metrics
static_configs:
- targets: ['localhost:9090']- job_name: 'node1'
static_configs:
- targets: ['192.168.99.101:9100']
|
Now we have to create the deployment for Prometheus. It will pull the Prometheus image from the docker hub, configure it, then create and mount the persistent volume at the destination folder. Prometheus usually stores the metrics at a folder called /metrics, so we mount the volume to this /metrics folder.
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 kind: Deployment metadata: name: prometheus labels: app: prometheus tier: backend spec: selector: matchLabels: app: prometheus tier: backend strategy: type: Recreate template: metadata: labels: app: prometheus tier: backend spec: containers: - image: prom/prometheus:latest name: prometheus args: - "--config.file=/etc/prometheus/prometheus.yml" ports: - containerPort: 9090 name: prometheus volumeMounts: - name: prometheus-persistent-storage mountPath: /prometheus - name: prometheus-config mountPath: /etc/prometheus/ volumes: - name: prometheus-persistent-storage persistentVolumeClaim: claimName: prometheus-pv-claim - name: prometheus-config configMap: name: prometheus-config
|
This deployment also mounts the Prometheus config file to the folder /etc/prometheus using a separate volume.
Now that manifest file for Prometheus is created we shall start with the one for Grafana. Similarly like Prometheus, we have to create a Service that exposes Grafana at a specified port( 30001 here )
apiVersion: v1 kind: Service metadata: name: grafana labels: app: grafana spec: ports: - port: 3000 nodePort: 30001 selector: app: grafana tier: frontend type: NodePort
|
We shall attach persistent storage for grafana also. For this, we need the persistent volume claim.
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: grafana-pv-claim labels: app: grafana spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi
|
Now we can create the deployment. This deployment will pull the grafana image from the docker hub, create the volume, and configure the container. We mount the volume at /var/lib/grafana
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 kind: Deployment metadata: name: grafana labels: app: grafana tier: frontend spec: selector: matchLabels: app: grafana tier: frontend strategy: type: Recreate template: metadata: labels: app: grafana tier: frontend spec: containers: - image: grafana/grafana:latest name: grafana ports: - containerPort: 3000 name: grafana volumeMounts: - name: grafana-persistent-storage mountPath: /var/lib/grafana volumes: - name: grafana-persistent-storage persistentVolumeClaim: claimName: grafana-pv-claim
|
So, now both the manifest files are ready and we can now deploy them. But, we shall create another file called kustomization.yaml. What this file does is bind the two resources (prometheus.yaml and grafana.yaml)
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - prometheus.yaml - grafana.yaml
|
Now we can deploy the whole setup using a single line of code. Using cmd head to the directory that houses the code and executes this
kubectl create -k .
Now you can see the whole setup is created and the pods are running


Head to the site http://<minikube ip>:30000 and you will see prometheus is deployed. You need to know a thing or two PromQL to navigate through prometheus TSDB

This query shows the no.of free bytes in the Linux VM memory where node exporter is running at the moment. This number keeps changing with time. To see the variation over time we now go to Grafana to see the graph. Go to http://<minikube ip>:30001 and you can see the Grafana login page. By default, the grafana login id and password are set to “admin”

From the grafana welcome screen select “Add data source”. There is a wide range of available options but, we shall choose Prometheus

This is the dashboard

In the URL section add the URL of Prometheus website

select “Save and Test”

Now we have to create a dashboard. Head over to the respective section and select “Add a new panel

In the metrics, field enter the query, make any other changes you want and then select apply

In the last, we can see the graphical representation of the metrics in real-time.
