Sergey Chulanov's site
Posts Tags Categories Series About me
Sergey Chulanov's site

Prometheus grafana

Sergey Chulanov included in Linux Devops
   655 words   4 minutes 
Contents

https://github.com/docker/awesome-compose/blob/master/prometheus-grafana/compose.yaml

You need to install docker with compose.

Install linux node exporter on nodes:

Download node_exporter:

curl -LO https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar xvfz node_exporter-1.6.1.linux-amd64.tar.gz

Add user:

sudo groupadd node_exporter
sudo useradd -g node_exporter -m -s /sbin/nologin node_exporter

Put node_exporter file in PATH directory:

sudo install node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Now, create a service file to running node_exporter process using systemd:

sudo bash -c 'cat <<EOF > /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
# for https://grafana.com/grafana/dashboards/1860-node-exporter-full/ added --collector.systemd --collector.processes
ExecStart=/usr/local/bin/node_exporter --collector.systemd --collector.processes

[Install]
WantedBy=multi-user.target
EOF'

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

Check:

sudo systemctl status node_exporter
curl -s http://0.0.0.0:9100/metrics | head

Script to install node_exporter on linux

node_exporter_install.sh:

#!/usr/bin/env bash

set -euo pipefail

NODE_EXPORTER_VERSION=$(curl -sL https://api.github.com/repos/prometheus/node_exporter/releases/latest | grep "tag_name"   | sed -E 's/.*"([^"]+)".*/\1/'|sed 's/v//')

cd /tmp/
wget https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXPORTER_VERSION}/node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz
tar -xzvf node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz
cd node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64
cp node_exporter /usr/local/bin

# create user
useradd --no-create-home --shell /bin/false node_exporter

chown node_exporter:node_exporter /usr/local/bin/node_exporter

echo '[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --collector.systemd --collector.processes --collector.textfile.directory=/var/lib/node_exporter/

[Install]
WantedBy=multi-user.target' > /etc/systemd/system/node_exporter.service

# enable node_exporter in systemctl
systemctl daemon-reload
systemctl start node_exporter
systemctl enable node_exporter

echo "Setup complete.
Add the following lines to /etc/prometheus/prometheus.yml:

  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']
"

Dashboards:

If you see the following error, it could be due to SELinux being enabled.

Failed to execute command: Permission denied

Check if SELinux is enabled

sestatus

To fix the permission issue, run the following to update the context and restart the service:

sudo restorecon -rv /usr/local/bin/node_exporter
sudo systemctl restart node_exporter

Ok, let’s run prometheus and grafana:

Put files into directory:

mkdir -p prometheus_grafana/prometheus
cd prometheus_grafana

cat <<EOF> compose.yml
services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    ports:
      - 9090:9090
    restart: unless-stopped
    volumes:
      - ./prometheus:/etc/prometheus
      - prom_data:/prometheus
  grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - 3000:3000
    restart: unless-stopped
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=grafana
    volumes:
      - ./grafana:/etc/grafana/provisioning/datasources
volumes:
  prom_data:
EOF

cat <<EOF> prometheus/prometheus.yml
global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
alerting:
  alertmanagers:
  - static_configs:
    - targets: []
    scheme: http
    timeout: 10s
    api_version: v1
scrape_configs:
- job_name: prometheus
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets: ['192.168.1.21:9100']
    labels:
      instance: monit
  - targets: ['192.168.1.50:9100']
    labels:
      instance: balmora
  - targets: ['192.168.1.72:9100']
    labels:
      instance: vvardenfel
EOF

Up docker compose:

docker compose up -d
docker ps
  CONTAINER ID   IMAGE             COMMAND                  CREATED          STATUS          PORTS                                       NAMES
  cb811359cd82   prom/prometheus   "/bin/prometheus --c…"   29 seconds ago   Up 23 seconds   0.0.0.0:9090->9090/tcp, :::9090->9090/tcp   prometheus
  a08f74f3fc2e   grafana/grafana   "/run.sh"                2 minutes ago    Up 2 minutes    0.0.0.0:3000->3000/tcp, :::3000->3000/tcp   grafana

Let’s check!:

curl -D - -s  localhost:3000/login -o /dev/null
HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: text/html; charset=UTF-8
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-Xss-Protection: 1; mode=block
Date: Sun, 19 Nov 2023 16:07:15 GMT
Transfer-Encoding: chunked

Dashboard for SMART

smartctl should work

sudo apt install smartmontools
sudo smartctl --version

Download the script:

curl -O https://raw.githubusercontent.com/olegeech-me/S.M.A.R.T-disk-monitoring-for-Prometheus/master/smartmon.sh
sudo install smartmon.sh /usr/local/bin/smartmon.sh

Create directory to store our metrics:

sudo mkdir -pv /var/lib/node_exporter/textfile_collector

We ca add into crontab:

*/5 * * * * sudo bash -c "/usr/local/bin/smartmon.sh > /var/lib/node_exporter/textfile_collector/smart_metrics.prom"

Or you can use systemd timer:

# /etc/systemd/system/smart_metrics.service
[Unit]
Description=Service to gen smart metrics
After=network.target

[Service]
Type=oneshot
ExecStart=bash -c "/usr/local/bin/smartmon.sh > /var/lib/node_exporter/textfile_collector/smart_metrics.prom"
User=root
Group=root

[Install]
WantedBy=multi-user.target
# /etc/systemd/system/smart_metrics.timer
[Unit]
Description=Wait some minutes

[Timer]
OnActiveSec=5min
AccuracySec=1m

[Install]
WantedBy=timers.target

And start it:

sudo systemctl daemon-reload
sudo systemctl enable --now smart_metrics.timer
systemctl list-timers

Add --collector.textfile.directory=/var/lib/node_exporter/textfile_collector into /etc/systemd/system/node_exporter.service:

...
ExecStart=/usr/local/bin/node_exporter --collector.systemd --collector.processes --collector.textfile.directory=/var/lib/node_exporter/textfile_collector
...

sudo systemctl restart node_exporter.service
sudo systemctl daemon-reload

Windows exporter

Download the latest version, put it in C:\Program Files\windows_exporter

New-Item -Path "C:\Program Files\" -Name "windows_exporter" -ItemType "directory"
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
Invoke-WebRequest "https://github.com/prometheus-community/windows_exporter/releases/download/v0.24.0/windows_exporter-0.24.0-amd64.exe" -OutFile "C:\Program Files\windows_exporter\windows_exporter-amd64.exe"

Run for test puprposes:

& "C:\Program Files\windows_exporter\windows_exporter-amd64.exe" --collectors.enabled  "[defaults],process,terminal_services"

Check the windows_exporter on links:

  • http://localhost:9182/health
  • http://localhost:9182/version
  • http://localhost:9182/metrics

And create service using powershell:

sc.exe create windows_exporter binPath= "C:\Program Files\windows_exporter\windows_exporter-amd64.exe"
# sc.exe delete windows_exporter

Get-Service windows_exporter
Start-Service windows_exporter
Set-Service windows_exporter -startuptype automatic

Open port:

netsh advfirewall firewall add rule name="windows_exporter" dir=in action=allow protocol=TCP localport=9182
Updated on 2026-09-16
Read Markdown
 Prometheus, Grafana, Monitoring, Docker
Back | Home