Prometheus: Troubleshooting

Errors you can encounter with Prometheus

👋 Welcome to the Stackhero documentation!

Stackhero delivers a fully managed Prometheus cloud platform designed for reliability and simplicity:

• Alert Manager is built in, so you can route alerts directly to Slack, Mattermost, PagerDuty, and other popular destinations.
• A dedicated email server lets you send unlimited email alerts without extra setup.
• Blackbox is included, allowing you to probe HTTP, ICMP, TCP, and other protocols for comprehensive monitoring.
• Configure your instance quickly using the online configuration file editor. There is no need to manage YAML by hand.
• Apply updates with a single click. Stackhero handles the upgrade process for you, minimizing downtime and manual intervention.
• High performance and strong security are built in, thanks to your own private, dedicated infrastructure.

Get up and running in about 5 minutes. Stackhero takes care of the setup so you can focus on monitoring, not maintenance. Try Prometheus cloud hosting on Stackhero to streamline your monitoring and alerting workflows.

Solving the error "received unsupported Content-Type "..." and no fallback_scrape_protocol specified for target"

With the release of Prometheus v3, it is now necessary for target servers to include Content-Type header in their responses to inform Prometheus about the protocol of the metrics contained in the reply. This affects tools like Node Exporter, which should now respond to Prometheus HTTP requests with the appropriate Content-Type. For more details, you might want to check the Prometheus documentation.

If this requirement is not met, you might encounter an error message like this:

received unsupported Content-Type "application/octet-stream" and no fallback_scrape_protocol specified for target

Here are some ways you can resolve this issue:

1. Update your target servers

If you are working with a target server like Node Exporter, consider updating it to the latest version. Recent updates typically ensure that the Content-Type header is correctly defined, which should resolve the error in Prometheus.

2. Define the Content-Type header for your target servers

For custom target servers, such as an API route you have developed to return Prometheus metrics, you can directly set the Content-Type header in the response.

For example, if you are using HapiJS, instead of returning your metrics like this:

return metrics

You can set the Content-Type header like this:

return h.response(metrics).type('text/plain;version=0.0.4');

Here are the supported Content-Type headers based on the protocol of the metrics you are returning:

• PrometheusProto: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited
• PrometheusText 0.0.4: text/plain;version=0.0.4
• PrometheusText 1.0.0: text/plain;version=1.0.0;escaping=allow-utf-8
• OpenMetricsText 0.0.1: application/openmetrics-text;version=0.0.1
• OpenMetricsText 1.0.0: application/openmetrics-text;version=1.0.0

3. Define a fallback protocol

You can also define a fallback protocol in your prometheus.yml configuration file. This protocol will be used if the target server does not specify a Content-Type header.

Here is an example:

  - job_name: "my-job"
    # [...]
    fallback_scrape_protocol: PrometheusText0.0.4

Supported values are PrometheusProto, PrometheusText0.0.4, PrometheusText1.0.0, OpenMetricsText0.0.1, and OpenMetricsText1.0.0.