Graylog: Handle errors about OpenSearch's read-only indices

This documentation is part of the Handle retention guide. View the full guide here: How to configure log retention.

👋 Welcome to the Stackhero documentation!

Stackhero offers a ready-to-use Graylog cloud solution that provides a host of benefits, including:

• Unlimited and dedicated SMTP email server included.
• Effortless updates with just a click.
• Customizable domain name secured with HTTPS (for example, https://logs.your-company.com).
• Optimal performance and robust security powered by a private and dedicated VM.

Save time and simplify your life: it only takes 5 minutes to try Stackhero's Graylog cloud hosting solution!

Occasionally, OpenSearch may switch to read-only mode and you might encounter errors such as:

1. "Flood stage disk watermark exceeded, all indices on this node will be marked read-only"
2. "FORBIDDEN/12/index read-only / allow delete (api)"

These errors occur as part of OpenSearch's protection mechanism when disk space is critically low. When available disk space drops below 7 GB, OpenSearch sets indices to read-only as a precautionary measure to prevent data corruption.

If you encounter these errors, you have two options:

1. Reconfigure your retention policy to keep fewer logs. After adjusting the policy, delete the oldest index to free up disk space and allow OpenSearch to switch back to read-write mode. Please note that deleting an index means that all data in that index will be lost.
2. Upgrade your instance to one with a larger disk. With a single click in your Stackhero dashboard, the instance will restart with additional disk space and OpenSearch will automatically return to read-write mode.