Graylog: Handling the error "failed to parse field [XXXX] of type [YYYY]"

This documentation is part of the Getting started guide. You can view the complete guide here: How to get started with Graylog.

👋 Welcome to Stackhero documentation

Stackhero offers a fully managed Graylog cloud solution designed for speed and simplicity. You can:

• Rely on an unlimited, dedicated SMTP email server included with your service.
• Apply updates effortlessly with a single click, with no manual intervention required.
• Use a custom domain name secured by HTTPS (for example, https://logs.your-company.com), providing your team with secure, direct access.
• Benefit from strong performance and security on a private, dedicated infrastructure, with no shared resources or noisy neighbours.

Focus on your data, not your tools: you can get started with Stackhero's Graylog cloud hosting solution in just a few minutes.

You may encounter an error such as:

org.opensearch.index.mapper.MapperParsingException: failed to parse field [time] of type [long] in document with id 'xxxx'

You can view this error in the logs available in the Stackhero dashboard or in the Graylog admin panel under System > Overview > Indexer failures.

This error means that a log was sent with a value for the time field that does not match the expected type (in this case, a numeric value of type "long"). Graylog relies on OpenSearch's dynamic mapping feature. When a log is sent for the first time, OpenSearch tries to infer the field types. For example, if a log contains the field time with the numeric value 1234, OpenSearch defines it as a numeric field. If another log is sent with the field time set to "abcd", a string, OpenSearch will reject it because it expects a numeric value.

Please note that the field name time is used for illustration purposes only. This can apply to any field name and type.

To resolve this issue, you need to redefine the type expected by OpenSearch. For more details, refer to the official Graylog documentation.

Updating the OpenSearch mapping

Before proceeding, enable OpenSearch access in the Stackhero dashboard. Go to your Graylog service and click the "Configure" button to activate OpenSearch access.

Please be careful with these changes, as incorrect configuration can block your OpenSearch cluster and may result in data loss. If you are unsure, do not proceed.

1. Define your new mapping. In this example, we redefine the time field as a string type. You can find the available types in the OpenSearch field datatypes documentation.

2. Save the following content to a file named graylog-custom-mapping.json:

   {
     "template": "graylog_*",
     "mappings": {
       "message": {
         "properties": {
           "time": {
             "type": "string",
             "index": "not_analyzed"
           }
         }
       }
     }
   }
   

3. Upload this file using the following curl command (replace <XXXXXX>.stackhero-network.com with your instance domain name):

   curl -u 'admin' -X PUT -d @'graylog-custom-mapping.json' -H 'Content-Type: application/json' 'https://<XXXXXX>.stackhero-network.com/opensearch/_template/graylog-custom-mapping?pretty'
   

   You should receive a response like:

   {
     "acknowledged": true
   }
   

4. Finally, check that the mapping has been updated with this command (replace the domain as needed):

   curl -u 'admin' -X GET 'https://<XXXXXX>.stackhero-network.com/opensearch/graylog_deflector/_mapping?pretty'