Prometheus: Configuring Prometheus alert rules

This documentation is part of the Alerts guide. You can view the complete guide here: How Prometheus alerts work and how to configure them.

👋 Welcome to the Stackhero documentation!

Stackhero offers a ready-to-use Prometheus cloud solution that provides numerous benefits, including:

• Alert Manager included to send alerts to Slack, Mattermost, PagerDuty, etc.
• Dedicated email server to send unlimited email alerts.
• Blackbox to probe HTTP, ICMP, TCP, and more.
• Easy configuration with an online configuration file editor.
• Effortless updates with just a click.
• Optimal performance and robust security powered by a private and dedicated VM.

Save time and simplify your life: it only takes 5 minutes to try Stackhero's Prometheus cloud hosting solution!

You can adjust Prometheus alert rules by editing the rules-alert.yml file. To do this, access your Stackhero dashboard, select your Prometheus service, and click on "Prometheus alert rules configuration".

We have already added some default alert rules to your Stackhero for Prometheus instance, so you typically won't need to modify the rules-alert.yml file unless customisation is required.

Below is an example of an alert that triggers if disk usage exceeds 90%:

- alert: "HostOutOfDiskSpace"
  expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 10 and ON (instance, device, mountpoint) node_filesystem_readonly == 0
  for: 2m
  labels:
    severity: "warning"
  annotations:
    summary: "Host out of disk space (instance {{ $labels.instance }})"
    description: "Disk is almost full (< 10% left)"
    value: "{{ $value }}"

Here is another example that predicts potential disk space saturation within the next 24 hours:

- alert: "HostDiskWillFillIn24Hours"
  expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 10 and ON (instance, device, mountpoint) predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs"}[1h], 24 * 3600) < 0 and ON (instance, device, mountpoint) node_filesystem_readonly == 0
  for: 2m
  labels:
    severity: "warning"
  annotations:
    summary: "Host disk will fill in 24 hours (instance {{ $labels.instance }})"
    description: "Filesystem is predicted to run out of space within the next 24 hours at the current write rate"
    value: "{{ $value }}"

You can find many additional alert rule examples on the Awesome Prometheus Alerts website.