Graylog: Handling errors related to OpenSearch read-only indices

This documentation is part of the Manage retention guide. You can view the complete guide here: How to configure log retention.

👋 Welcome to the Stackhero documentation!

Stackhero offers a ready-to-use Graylog cloud solution that provides a host of benefits, including:

• Unlimited and dedicated SMTP email server included.
• Effortless updates with just a click.
• Customizable domain name secured with HTTPS (for example, https://logs.your-company.com).
• Optimal performance and robust security powered by a private and dedicated VM.

Save time and simplify your life: it only takes 5 minutes to try Stackhero's Graylog cloud hosting solution!

Sometimes, OpenSearch may switch to read-only mode and you may encounter errors such as:

1. "Flood stage disk watermark exceeded, all indices on this node will be marked read-only"
2. "FORBIDDEN/12/index read-only / allow delete (api)"

These errors occur as part of OpenSearch's protection mechanism when disk space becomes critically low. When available disk space drops below 7 GB, OpenSearch sets indices to read-only as a precaution to prevent data corruption.

If you encounter these errors, you have two options:

1. Reconfigure your retention policy to keep fewer logs. After adjusting the policy, delete the oldest index to free up disk space and allow OpenSearch to return to read-write mode. Please note that deleting an index will permanently erase all data it contains.
2. Upgrade to an instance with a larger disk. With a single click from your Stackhero dashboard, the instance will restart with more disk space and OpenSearch will automatically return to read-write mode.