Monitor Server with Prometheus and Grafana
Prometheus is used an open source software, that can collect metrics and alerting.
Steps:
1. We install Prometheus & Grafana on only 1 server which we want to make a monitoring server. (Master server)
2. We install collecting data metrics setup on all server which we want to get monitor. (Edge servers)
3. You can also install collecting data metrics setup on grafana/prometheus monitoring server as well to monitor grafana server itself. (Master as well as edge)
You can download latest version of Prometheus from
https://prometheus.io/download/
Create a user
| useradd --no-create-home --system --shell /bin/false prometheus |
Download and Install prometheus
| cd /usr/local/src wget https://github.com/prometheus/prometheus/releases/download/v2.31.0-rc.1/prometheus-2.31.0-rc.1.linux-amd64.tar.gz tar xvf prometheus-2.31.0-rc.1.linux-amd64.tar.gz cd prometheus-2.31.0-rc.1.linux-amd64 mv prometheus /usr/local/bin/ mv promtool /usr/local/bin/ mkdir /etc/prometheus mkdir /var/lib/prometheus mv consoles /etc/prometheus mv console_libraries /etc/prometheus mv prometheus.yml /etc/prometheus chown prometheus:prometheus /etc/prometheus chown prometheus:prometheus /var/lib/prometheus |
Create a service file
| vi /etc/systemd/system/prometheus.service |
Add following content
|
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target
|
Enable prometheus to start on boot
| systemctl enable prometheus |
Start prometheus
|
systemctl start prometheus
systemctl status prometheus
|
Prometheus runs on port 9090, you can access prometheus at
| http://YOUR_SERVER_IP:9090/graph |
Prometheus have some basic graphing features, but you can’t use it for monitoring. To create dash board and monitor, you need to use grafana.
Collecting Data
Node Exporter is used to collect data from servers. All monitored servers need Node Exporter installed. You can download latest version of NodeExporter from
https://github.com/prometheus/node_exporter/releases
Lets create a user for Node Exporter to run
| useradd --no-create-home --system --shell /bin/false node_exporter |
Install Node Exporter
|
cd /usr/local/src
wget https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-amd64.tar.gz
tar xvf node_exporter-1.2.2.linux-amd64.tar.gz
cd /usr/local/src/node_exporter-1.2.2.linux-amd64/
mv node_exporter /usr/local/bin/
|
Create a systemd service file for node exporter
| vi /etc/systemd/system/node_exporter.service |
Add
|
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
|
Enable and start Node Exporter
|
systemctl enable node_exporter
systemctl start node_exporter
systemctl status node_exporter
|
Node Exporter run on port 9100 and expose system metrics on url
| http://SERVER_IP:9100/metrics |
Adding Servers to Prometheus
Once Node Exporter installed on a server, you need to tell Prometheus to get data from the Node Exporter you just installed. To do this, edit Prometheus configuration file.
| vi /etc/prometheus/prometheus.yml |
Add following
|
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: "prometheus"
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ["localhost:9090"]
- job_name: 'Server1'
scrape_interval: 5s
static_configs:
- targets: ['server1.com:9100']
- job_name: 'Server2'
scrape_interval: 5s
static_configs:
- targets: ['server2.com:9100']
- job_name: 'Server3'
scrape_interval: 5s
static_configs:
- targets: ['server3.com:9100']
|
To monitor multiple servers, you can add more servers in targets line.
Restart prometheus
| systemctl restart prometheus |
Grafana
Grafana is used to visualise data collected by Prometheus. You can download Grafana from
https://grafana.com/grafana/download
Grafana offers free cloud hosted version with some limitation (1 user, 5 dashboards). Free version is suitable if you are getting started and don’t want to install your own. You can signup for cloud hosted version at
If you decide to install your own Grafana, you can run
|
cd /usr/local/src
wget https://dl.grafana.com/oss/release/grafana_7.3.7_amd64.deb
dpkg -i grafana_7.3.7_amd64.deb
|
Enable and start grafana
|
systemctl enable grafana-server
systemctl start grafana-server
systemctl status grafana-server
|
If you did your own install, grafana runs on port 3000. To access, use url
| http://SERVER_IP:3000/login |
Default username and passwords are “admin”. Once logged in you will be asked to set password for grafana admin user.
Before you can use Grafana, you need to set a data source and create dash board. In our case, data source is prometheus. To connect Grafana to your Prometheus installation, go to Settings > Data Sources
On next page, select Prometheus
On next page, for URL, enter http://PROMETHUS_SERVER_IP:9090, scroll down, click on “Save & Test” button. If grafana can connect to your prometheus installation, you should see success message with “Data source is working”. If not, you need to check your firewall rules.
Creating Grafana Dashboards
Grafana displays data in dashboards. You can create your own or use pre existing dash boards. You can find pre-made dash boards at
https://grafana.com/grafana/dashboards
On my grafana installation, i used dashboard
https://grafana.com/grafana/dashboards/11074
To add this dashboard to your Grafana, click on the + button, then select Import. On next screen, you can enter ID for the dash board you need to import. In this case 11074. Click “Load” button to import the dashboard.
You can edit Panels in grafana dashboard to see how it is created. You can create a new dash board with panel you need. This way your dashboards only show required information.