https://github.com/docker/awesome-compose/blob/master/prometheus-grafana/compose.yaml
You need to install docker with compose.
Install linux node exporter on nodes:
Download node_exporter:
1
2
|
curl -LO https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar xvfz node_exporter-1.6.1.linux-amd64.tar.gz
|
Add user:
1
2
|
sudo groupadd node_exporter
sudo useradd -g node_exporter -m -s /sbin/nologin node_exporter
|
Put node_exporter file in PATH directory:
1
2
|
sudo install node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
|
Now, create a service file to running node_exporter process using systemd:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
sudo bash -c 'cat <<EOF > /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
# for https://grafana.com/grafana/dashboards/1860-node-exporter-full/ added --collector.systemd --collector.processes
ExecStart=/usr/local/bin/node_exporter --collector.systemd --collector.processes
[Install]
WantedBy=multi-user.target
EOF'
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
|
Check:
1
2
|
sudo systemctl status node_exporter
curl -s http://0.0.0.0:9100/metrics | head
|
Script to install node_exporter on linux
node_exporter_install.sh:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
|
#!/usr/bin/env bash
set -euo pipefail
NODE_EXPORTER_VERSION=$(curl -sL https://api.github.com/repos/prometheus/node_exporter/releases/latest | grep "tag_name" | sed -E 's/.*"([^"]+)".*/\1/'|sed 's/v//')
cd /tmp/
wget https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXPORTER_VERSION}/node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz
tar -xzvf node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz
cd node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64
cp node_exporter /usr/local/bin
# create user
useradd --no-create-home --shell /bin/false node_exporter
chown node_exporter:node_exporter /usr/local/bin/node_exporter
echo '[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --collector.systemd --collector.processes --collector.textfile.directory=/var/lib/node_exporter/
[Install]
WantedBy=multi-user.target' > /etc/systemd/system/node_exporter.service
# enable node_exporter in systemctl
systemctl daemon-reload
systemctl start node_exporter
systemctl enable node_exporter
echo "Setup complete.
Add the following lines to /etc/prometheus/prometheus.yml:
- job_name: 'node_exporter'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9100']
"
|
Dashboards:
If you see the following error, it could be due to SELinux being enabled.
1
|
Failed to execute command: Permission denied
|
Check if SELinux is enabled
To fix the permission issue, run the following to update the context and restart the service:
1
2
|
sudo restorecon -rv /usr/local/bin/node_exporter
sudo systemctl restart node_exporter
|
Ok, let’s run prometheus and grafana:
Put files into directory:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
|
mkdir -p prometheus_grafana/prometheus
cd prometheus_grafana
cat <<EOF> compose.yml
services:
prometheus:
image: prom/prometheus
container_name: prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
ports:
- 9090:9090
restart: unless-stopped
volumes:
- ./prometheus:/etc/prometheus
- prom_data:/prometheus
grafana:
image: grafana/grafana
container_name: grafana
ports:
- 3000:3000
restart: unless-stopped
environment:
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=grafana
volumes:
- ./grafana:/etc/grafana/provisioning/datasources
volumes:
prom_data:
EOF
cat <<EOF> prometheus/prometheus.yml
global:
scrape_interval: 15s
scrape_timeout: 10s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets: []
scheme: http
timeout: 10s
api_version: v1
scrape_configs:
- job_name: prometheus
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
static_configs:
- targets: ['192.168.1.21:9100']
labels:
instance: monit
- targets: ['192.168.1.50:9100']
labels:
instance: balmora
- targets: ['192.168.1.72:9100']
labels:
instance: vvardenfel
EOF
|
Up docker compose:
1
2
3
4
5
|
docker compose up -d
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cb811359cd82 prom/prometheus "/bin/prometheus --c…" 29 seconds ago Up 23 seconds 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp prometheus
a08f74f3fc2e grafana/grafana "/run.sh" 2 minutes ago Up 2 minutes 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp grafana
|
Let’s check!:
1
2
3
4
5
6
7
8
9
|
curl -D - -s localhost:3000/login -o /dev/null
HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: text/html; charset=UTF-8
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-Xss-Protection: 1; mode=block
Date: Sun, 19 Nov 2023 16:07:15 GMT
Transfer-Encoding: chunked
|
Dashboard for SMART
smartctl should work
1
2
|
sudo apt install smartmontools
sudo smartctl --version
|
Download the script:
1
2
|
curl -O https://raw.githubusercontent.com/olegeech-me/S.M.A.R.T-disk-monitoring-for-Prometheus/master/smartmon.sh
sudo install smartmon.sh /usr/local/bin/smartmon.sh
|
Create directory to store our metrics:
1
|
sudo mkdir -pv /var/lib/node_exporter/textfile_collector
|
We ca add into crontab:
1
|
*/5 * * * * sudo bash -c "/usr/local/bin/smartmon.sh > /var/lib/node_exporter/textfile_collector/smart_metrics.prom"
|
Or you can use systemd timer:
1
2
3
4
5
6
7
8
9
10
11
12
13
|
# /etc/systemd/system/smart_metrics.service
[Unit]
Description=Service to gen smart metrics
After=network.target
[Service]
Type=oneshot
ExecStart=bash -c "/usr/local/bin/smartmon.sh > /var/lib/node_exporter/textfile_collector/smart_metrics.prom"
User=root
Group=root
[Install]
WantedBy=multi-user.target
|
1
2
3
4
5
6
7
8
9
10
|
# /etc/systemd/system/smart_metrics.timer
[Unit]
Description=Wait some minutes
[Timer]
OnActiveSec=5min
AccuracySec=1m
[Install]
WantedBy=timers.target
|
And start it:
1
2
3
|
sudo systemctl daemon-reload
sudo systemctl enable --now smart_metrics.timer
systemctl list-timers
|
Add --collector.textfile.directory=/var/lib/node_exporter/textfile_collector into /etc/systemd/system/node_exporter.service:
1
2
3
4
5
6
|
...
ExecStart=/usr/local/bin/node_exporter --collector.systemd --collector.processes --collector.textfile.directory=/var/lib/node_exporter/textfile_collector
...
sudo systemctl restart node_exporter.service
sudo systemctl daemon-reload
|
Windows exporter
Download the latest version, put it in C:\Program Files\windows_exporter
1
2
3
|
New-Item -Path "C:\Program Files\" -Name "windows_exporter" -ItemType "directory"
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
Invoke-WebRequest "https://github.com/prometheus-community/windows_exporter/releases/download/v0.24.0/windows_exporter-0.24.0-amd64.exe" -OutFile "C:\Program Files\windows_exporter\windows_exporter-amd64.exe"
|
Run for test puprposes:
1
|
& "C:\Program Files\windows_exporter\windows_exporter-amd64.exe" --collectors.enabled "[defaults],process,terminal_services"
|
Check the windows_exporter on links:
- http://localhost:9182/health
- http://localhost:9182/version
- http://localhost:9182/metrics
And create service using powershell:
1
2
3
4
5
6
|
sc.exe create windows_exporter binPath= "C:\Program Files\windows_exporter\windows_exporter-amd64.exe"
# sc.exe delete windows_exporter
Get-Service windows_exporter
Start-Service windows_exporter
Set-Service windows_exporter -startuptype automatic
|
Open port:
1
|
netsh advfirewall firewall add rule name="windows_exporter" dir=in action=allow protocol=TCP localport=9182
|