Graylog: Choose input types
How to choose the right Graylog input type
👋 Welcome to the Stackhero documentation!
Stackhero offers a ready-to-use Graylog cloud solution that provides a host of benefits, including:
- Unlimited and dedicated SMTP email server included.
- Effortless updates with just a click.
- Customizable domain name secured with HTTPS (for example, https://logs.your-company.com).
- Optimal performance and robust security powered by a private and dedicated VM.
Save time and simplify your life: it only takes 5 minutes to try Stackhero's Graylog cloud hosting solution!
Graylog provides several types of inputs. These inputs let you send logs to the Graylog server or allow Graylog to retrieve logs from another source.
In the vast majority of cases, you will want to send data from an application or a device to the Graylog server. We will not cover advanced scenarios where Graylog needs to fetch logs from elsewhere. However, if this is what you want to do, rest assured—it is very straightforward to set up.
To select the most suitable input type, you first need to determine the log format you want to send to Graylog. We recommend using the GELF format, as explained below.
Next, you will need to choose the protocol: UDP or TCP.
Choosing the input format
The first step is to choose the format of your log messages. There are three main options: RAW/Plaintext, Syslog, or GELF.
RAW/Plaintext format
This is the simplest format. It works with any system because it sends a plain text message. It is very easy to use and ideal for quick tests, such as sending a message using the netcat utility.
To test it, you can create a "Raw/Plaintext UDP" input and then send a test log using one of the following commands:
- From macOS:
echo "Hello Graylog from UDP" | nc -u -w1 -c <XXXXXX>.stackhero-network.com 5555 - From Linux:
echo "Hello Graylog from UDP" | nc -u -w1 <XXXXXX>.stackhero-network.com 5555
It's very simple, isn't it? However, since this is a basic format, you will quickly run into limitations. Therefore, we do not recommend using it beyond simple tests.
Syslog format
This is a widely used format for storing and transmitting logs in server environments. It is found on Linux, Windows, *BSD, and other operating systems.
If you are using Syslog, Rsyslog, or another compatible service and want to send logs from a server to Graylog, the syslog input is the simplest option.
However, since GELF is more powerful, if you can send messages in GELF format, we recommend doing so, as explained below.
Note that the syslog format is limited to 1024 characters.
GELF format (recommended)
GELF stands for "Graylog Extended Log Format". This format was created by Graylog and is now widely adopted by many products, becoming a standard.
The GELF format is powerful because it allows you to send logs in a structured way. Structured logging lets you transmit multiple fields and values, instead of a single unstructured text message. This makes it easier for Graylog to analyze and search for information.
For example, if you want to send a log containing an IP address and a request type, you can send a message with two fields named ip and request_type. On the Graylog side, you can then search directly on these fields, such as searching for all requests with the query: request_type: POST.
GELF is based on JSON and must include at least the version, host, and short_message fields. You can then add other fields, which must start with an underscore (_).
For example, a simple message with an additional field named device_id looks like this:
{
"version": "1.1",
"host": "myIotDevice",
"short_message": "Something is happening",
"_device_id": "abcd"
}
You can then search for all messages coming from the device "abcd" on Graylog with the query: device_id: abcd.
For more details about GELF, you can refer to the official specification here: https://docs.graylog.org/en/4.0/pages/gelf.html.
If you want to send logs from servers or applications, you can use one of the many available libraries that automatically create and send your messages. Examples are available for Node.js, .NET, and Python.
Choosing the right protocol
Once you have chosen the format (we hope you chose GELF!), you need to select the communication protocol between TCP and UDP.
UDP protocol
The UDP protocol is very simple. Your application sends the message to the server without waiting for a receipt confirmation. There is no dialogue between the client and the server, which means the server does not confirm receipt of the message.
The advantage of this approach is that if your Graylog server is slow or unavailable, it will not impact your application's performance. The downside is that you cannot guarantee that every log sent by your application will actually reach the server. Also, encryption is not available with UDP.
Pros
- Does not impact your application if Graylog is slow or unavailable.
Cons
- Some messages may be lost.
- Encryption is not available.
- Message size is limited to 8192 bytes.
TCP protocol
The TCP protocol, unlike UDP, allows for an exchange of information between your application and Graylog. This means you receive confirmation that the message has been received by Graylog.
The benefit of TCP is that you can resend a message if it is not received by Graylog (for example, if the server is unavailable). The drawback is that if Graylog is slow or unresponsive, your application must wait for a response or a timeout, which can slow it down.
Another advantage is that you can encrypt your communications with TLS (also known as SSL).
Pros
- Ensures that the message has been received.
- Encryption is available (TLS).
- No message size limitation
Cons
- Can slow down the sender if Graylog is slow to respond or unavailable.
Conclusion
We hope this guide has provided you with all the information you need to choose the best format and protocol for sending your logs to Graylog.
Remember, you can run multiple inputs at the same time by using different ports for each. For example, you can have a "syslog UDP" input on port 514 and a "GELF TCP" input on port 12201.