Set up Grafana monitoring

Grafana and Prometheus are tools used to monitor services, servers and any other software using the Open metrics. Here is an example of implementation using Docker compose:

Installing Grafana and Prometheus

version: '3.7'

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alert.rules:/etc/prometheus/alert.rules
    ports:
      - 9090:9090
    networks:
      - "monitoring-network"
    expose:
      - 9090
  grafana:
    image: grafana/grafana
    container_name: grafana
    volumes:
      - ./grafana:/var/lib/grafana
    ports:
      - 3000:3000
    networks:
      - "monitoring-network"
    depends_on:
      - prometheus
  alertmanager:
    image: prom/alertmanager
    container_name: alertmanager
    volumes:
      - ./alertmanager:/etc/alertmanager
    ports:
      - 9093:9093
    networks:
      - "monitoring-network"

networks:
  monitoring-network:
    driver: bridge

Target: run Cadvisor

In the target side (the service that will be monitored), we must install the exporter, for example Cadvisor.

sudo docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=9418:8080 \
  --detach=true \
  --name=cadvisor \
  --restart=always \
  gcr.io/google-containers/cadvisor:latest

Prometheus: target configuration

To generate the Open Metrics data. Then, we need to include the target configs in the prometheus.yml the target config section.

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - "/etc/prometheus/alert.rules"
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: "Docker Job"
    static_configs:
    - targets: ['192.168.x.y:9323']