Prometheus: Configuring Prometheus alert rules

This documentation is part of the Alerts guide. You can view the complete guide here: How Prometheus alerts work and how to configure them.

👋 Welcome to the Stackhero documentation!

Stackhero offers a ready-to-use Prometheus cloud solution designed for reliability and simplicity:

• Alert Manager is included, allowing you to send alerts directly to Slack, Mattermost, PagerDuty, and other popular platforms.
• A dedicated email server enables you to send unlimited email alerts with no additional configuration required.
• Blackbox is provided, so you can probe HTTP, ICMP, TCP, and more for comprehensive monitoring.
• Simple configuration with the online configuration file editor—no need to manage YAML manually.
• Easy updates with just one click.
• Optimal performance and enhanced security thanks to your own private, dedicated VM.

Save time and make your life easier: it takes just 5 minutes to try Stackhero's Prometheus cloud hosting solution!

You can adjust Prometheus alert rules by editing the rules-alert.yml file. To do this, access your Stackhero dashboard, select your Prometheus service, and click on "Prometheus alert rules configuration".

We have already added some default alert rules to your Stackhero for Prometheus instance, so you typically won't need to modify the rules-alert.yml file unless customisation is required.

Below is an example of an alert that triggers if disk usage exceeds 90%:

- alert: "HostOutOfDiskSpace"
  expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 10 and ON (instance, device, mountpoint) node_filesystem_readonly == 0
  for: 2m
  labels:
    severity: "warning"
  annotations:
    summary: "Host out of disk space (instance {{ $labels.instance }})"
    description: "Disk is almost full (< 10% left)"
    value: "{{ $value }}"

Here is another example that predicts potential disk space saturation within the next 24 hours:

- alert: "HostDiskWillFillIn24Hours"
  expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 10 and ON (instance, device, mountpoint) predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs"}[1h], 24 * 3600) < 0 and ON (instance, device, mountpoint) node_filesystem_readonly == 0
  for: 2m
  labels:
    severity: "warning"
  annotations:
    summary: "Host disk will fill in 24 hours (instance {{ $labels.instance }})"
    description: "Filesystem is predicted to run out of space within the next 24 hours at the current write rate"
    value: "{{ $value }}"

You can find many additional alert rule examples on the Awesome Prometheus Alerts website.