Please enable Javascript for better experience...
 
Prometheus And Grafana Using Kubernetes
By Keshav Khatore | Jul 9, 2020 | In Articles | Total Views [ 297 ]
Taged In
(0 Like)
Rate

Monitoring is an essential part of DevOps. Every site you deploy, every app you deploy has to be continuously monitored since you never know when an error might arise. An error, even a small one can cause a catastrophic failure and cause the site or app to go down instantly.
Here comes the role of monitoring. Monitoring allows us to continuously track the state of the site or app and swiftly take action in case the infrastructure goes down.
Fortunately, a lot of monitoring tools are available in the market. One of these tools is Prometheus. It is a great open-source tool for real-time monitoring and alerting. It records real-time metrics in a time series database built using an HTTP pull model. These metrics tell us the current state of the site or app etc., in real-time.

However, keeping a track of these metrics is not very convenient due to their form. An average person always prefers an interactive and graphical representation over textual. Here the role of Grafana comes into play. It is multi-platform open-source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources. When grafana is configured with Prometheus as the data source, it can represent the metrics is an interactive GUI with graphs, charts, etc.

Pre-requisites:

  1. Minikube installed and running
  2. kubectl configured and added to the path
  3. A data source which is constantly exporting the metrics( For this task I am using a node exporter that is constantly exporting the metrics of a Linux VM at port number 9100 )

Manifest file

To deploy the setup I am going to use 2 main Kubernetes manifest file

Manifest file for Prometheus

apiVersion: v1
kind: Service
metadata:
  name: prometheus
  labels:
    app: prometheus
spec:
  ports:
    - port: 9090
      nodePort: 30000
  selector:
    app: prometheus
    tier: backend
  type: NodePort
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-pv-claim
  labels:
    app: prometheus
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  labels:
    name: prometheus-config
data:
 prometheus.yml: |-
  global:
    scrape_interval: 5s
    evaluation_interval: 5s
  alerting:
    alertmanagers:
    - static_configs:
      - targets:
   #rule_files:
   # - "first_rules.yml"
   # - "second_rules.yml"scrape_configs:
    - job_name: 'prometheus'
    #metrics default path is /metrics
      static_configs:
      - targets: ['localhost:9090']- job_name: 'node1'
      static_configs:
      - targets: ['192.168.99.101:9100']
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: prometheus
  labels:
    app: prometheus
    tier: backend
spec:
  selector:
    matchLabels:
      app: prometheus
      tier: backend
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: prometheus
        tier: backend
    spec:
      containers:
      - image: prom/prometheus:latest
        name: prometheus
        args:
        - "--config.file=/etc/prometheus/prometheus.yml"
        ports:
        - containerPort: 9090
          name: prometheus
        volumeMounts:
        - name: prometheus-persistent-storage
          mountPath: /prometheus
        - name: prometheus-config
          mountPath: /etc/prometheus/
      volumes:
      - name: prometheus-persistent-storage
        persistentVolumeClaim:
          claimName: prometheus-pv-claim
      - name: prometheus-config
        configMap:
          name: prometheus-config

 

 This file will deploy, configure and expose Prometheus at port 30000

Manifest file for Grafana

apiVersion: v1
kind: Service
metadata:
name: grafana
labels:
app: grafana
spec:
ports:
- port: 3000
nodePort: 30001
selector:
app: grafana
tier: frontend
type: NodePort
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grafana-pv-claim
labels:
app: grafana
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: grafana
labels:
app: grafana
tier: frontend
spec:
selector:
matchLabels:
app: grafana
tier: frontend
strategy:
type: Recreate
template:
metadata:
labels:
app: grafana
tier: frontend
spec:
containers:
- image: grafana/grafana:latest
name: grafana
ports:
- containerPort: 3000
name: grafana
volumeMounts:
- name: grafana-persistent-storage
mountPath: /var/lib/grafana
volumes:
- name: grafana-persistent-storage
persistentVolumeClaim:
claimName: grafana-pv-claim

 This file will deploy, configure and expose Grafana at port 30001

Process:
We will first create the Service for prometheus. This service will expose prometheus at port no. 30000. Due to this we are able to access the prometheus WebUI at http://<minikube ip>:30000

apiVersion: v1
kind: Service
metadata:
name: prometheus
labels:
app: prometheus
spec:
ports:
- port: 9090
nodePort: 30000
selector:
app: prometheus
tier: backend
type: NodePort

  Creating a persistent volume claim. This claim allows us to create a volume that will store some data. In this case, we will configure it to store the metrics and allot it 20 GiB of space.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: prometheus-pv-claim
labels:
app: prometheus
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi

 Creating the config file for Prometheus. This is very important as in this file we configure Prometheus to import the metrics from node exporter

apiVersion: v1

kind: ConfigMap

metadata:

  name: prometheus-config

  labels:

    name: prometheus-config

data:

 prometheus.yml: |-

  global:

    scrape_interval: 5s

    evaluation_interval: 5s

  alerting:

    alertmanagers:

    - static_configs:

      - targets:

   #rule_files:

   # - "first_rules.yml"

   # - "second_rules.yml"scrape_configs:

    - job_name: 'prometheus'

    #metrics default path is /metrics

      static_configs:

      - targets: ['localhost:9090']- job_name: 'node1'

      static_configs:

      - targets: ['192.168.99.101:9100']

 

Now we have to create the deployment for Prometheus. It will pull the Prometheus image from the docker hub, configure it, then create and mount the persistent volume at the destination folder. Prometheus usually stores the metrics at a folder called /metrics, so we mount the volume to this /metrics folder.

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: prometheus
labels:
app: prometheus
tier: backend
spec:
selector:
matchLabels:
app: prometheus
tier: backend
strategy:
type: Recreate
template:
metadata:
labels:
app: prometheus
tier: backend
spec:
containers:
- image: prom/prometheus:latest
name: prometheus
args:
- "--config.file=/etc/prometheus/prometheus.yml"
ports:
- containerPort: 9090
name: prometheus
volumeMounts:
- name: prometheus-persistent-storage
mountPath: /prometheus
- name: prometheus-config
mountPath: /etc/prometheus/
volumes:
- name: prometheus-persistent-storage
persistentVolumeClaim:
claimName: prometheus-pv-claim
- name: prometheus-config
configMap:
name: prometheus-config

 

This deployment also mounts the Prometheus config file to the folder /etc/prometheus using a separate volume.
Now that manifest file for Prometheus is created we shall start with the one for Grafana. Similarly like Prometheus, we have to create a Service that exposes Grafana at a specified port( 30001 here )

apiVersion: v1
kind: Service
metadata:
name: grafana
labels:
app: grafana
spec:
ports:
- port: 3000
nodePort: 30001
selector:
app: grafana
tier: frontend
type: NodePort

 We shall attach persistent storage for grafana also. For this, we need the persistent volume claim.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grafana-pv-claim
labels:
app: grafana
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi

 Now we can create the deployment. This deployment will pull the grafana image from the docker hub, create the volume, and configure the container. We mount the volume at /var/lib/grafana

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: grafana
labels:
app: grafana
tier: frontend
spec:
selector:
matchLabels:
app: grafana
tier: frontend
strategy:
type: Recreate
template:
metadata:
labels:
app: grafana
tier: frontend
spec:
containers:
- image: grafana/grafana:latest
name: grafana
ports:
- containerPort: 3000
name: grafana
volumeMounts:
- name: grafana-persistent-storage
mountPath: /var/lib/grafana
volumes:
- name: grafana-persistent-storage
persistentVolumeClaim:
claimName: grafana-pv-claim

 

So, now both the manifest files are ready and we can now deploy them. But, we shall create another file called kustomization.yaml. What this file does is bind the two resources (prometheus.yaml and grafana.yaml)

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- prometheus.yaml
- grafana.yaml

 Now we can deploy the whole setup using a single line of code. Using cmd head to the directory that houses the code and executes this

kubectl create -k .

Now you can see the whole setup is created and the pods are running

Head to the site http://<minikube ip>:30000 and you will see prometheus is deployed. You need to know a thing or two PromQL to navigate through prometheus TSDB


This query shows the no.of free bytes in the Linux VM memory where node exporter is running at the moment. This number keeps changing with time. To see the variation over time we now go to Grafana to see the graph. Go to http://<minikube ip>:30001 and you can see the Grafana login page. By default, the grafana login id and password are set to “admin”



From the grafana welcome screen select “Add data source”. There is a wide range of available options but, we shall choose Prometheus

This is the dashboard

In the URL section add the URL of Prometheus website

select “Save and Test”


Now we have to create a dashboard. Head over to the respective section and select “Add a new panel


In the metrics, field enter the query, make any other changes you want and then select apply


In the last, we can see the graphical representation of the metrics in real-time.





Thanks for visiting here. Share this article if you found it useful.
Like Facebook Page https://www.facebook.com/VMwareInsight/
Connect to twitter https://twitter.com/imPranayK
Subscribe my Channel https://www.youtube.com/vmwareinsight
Connect over Linkedin https://in.linkedin.com/in/impranayk
Share this on Social Media

About the Author

Keshav Khatore
Keshav Khatore
Member VMwareInsight.com

Public profile: user/profile/99905478


Follow me

facebook linkedin twitter G+ VMTN youtube

I am Keshav Khatore student of Lovely Professional University, Phagwara, Punjab, India, currently pursuing Masters In Computer Application.I am a Member of Redhat Linux, the biggest network on the planet and it's a glad thing for me to be a piece of such extraordinary and lofty community. I expert in Linux and have certification Red-hat Certified System Administration . For Contact- keshavmah27@gmail.com || LinkedIn- https://www.linkedin.com/in/keshav-khatore/

 
Please SignUp/Login to comment...

Or comment as anonymous...
* Name
* Email ID
Comment
 
Sponsors
 
 
 
 
 
Facebook Likes