Prometheus: Introduction to Prometheus alerts

This documentation is part of the Alerts guide. You can view the complete guide here: How Prometheus alerts work and how to configure them.

👋 Welcome to the Stackhero documentation!

Stackhero offers a ready-to-use Prometheus cloud solution that provides numerous benefits, including:

• Alert Manager included to send alerts to Slack, Mattermost, PagerDuty, etc.
• Dedicated email server to send unlimited email alerts.
• Blackbox to probe HTTP, ICMP, TCP, and more.
• Easy configuration with an online configuration file editor.
• Effortless updates with just a click.
• Optimal performance and robust security powered by a private and dedicated VM.

Save time and simplify your life: it only takes 5 minutes to try Stackhero's Prometheus cloud hosting solution!

Prometheus can analyse your metrics and trigger alerts based on rules that you define. With Stackhero for Prometheus, alerts are processed in two stages. First, Prometheus alert rules are evaluated and then Alert Manager takes over.

Everything is pre-installed and configured with Stackhero for Prometheus so you only need to perform minimal setup, such as adding your email address, to start receiving alerts.

Big picture of Stackhero for Prometheus

Introduction to Prometheus alert rules

When Prometheus retrieves metrics, it evaluates them against rules specified in the rules-alert.yml file. These alert rules define thresholds and time windows for triggering alerts based on the metrics collected.

For example, an alert can be triggered if disk usage exceeds 80%. Additionally, rules can be set up to predict future conditions and send an alert if it estimates that disk space will be completely filled within the next 24 hours.

Another common use case involves detecting unusual behaviour. For instance, if there is a sudden spike in network bandwidth usage, an alert can be triggered to help detect potential distributed denial-of-service (DDoS) attacks or data exfiltration attempts.

Prometheus alert rules are included directly in the Prometheus server.

Introduction to Alert Manager

Alert Manager receives alerts that Prometheus alert rules have triggered. It deduplicates alerts, groups them, and then forwards them via various notification channels such as email, Slack, Mattermost, PagerDuty, and more. Its configuration file is alert-manager.yml.

For example, if a server slowdown occurs, Prometheus alert rules might trigger separate alerts for increased load and CPU usage. Alert Manager receives these alerts, groups them as they relate to the same server, and sends a single consolidated notification to the appropriate recipient or team based on your configuration.

If the slowdown continues, Prometheus will keep sending alerts but Alert Manager will suppress duplicate messages for a specified period to prevent flooding your team with redundant alerts.

You can also silence or completely inhibit alerts if needed. Once the underlying issue is resolved, a recovery message is sent to notify your teams.

This example illustrates a common scenario, but you can fully customise the setup to meet your specific requirements.

Alert Manager is not included by default with Prometheus. To save you time and simplify the process, we have integrated and configured Alert Manager in Stackhero for Prometheus so you can send alerts in just a few minutes, with minimal effort.