Sending logs to Humio using the elasticsearch-http() destination of syslog-ng
One of the most popular syslog-ng destinations is Elasticsearch. Humio, a log management provider, supports a broad range of ingest options and interfaces, including an Elasticsearch-compatible API. Last week, Humio announced Humio Community Edition, which provides the full Humio experience for free, with some limitations on daily ingestion and retention time. I tested the Community Edition, and it works perfectly well with syslog-ng.
If you come from the Humio side, you might wonder what syslog-ng is. It is an application for high performance central log collection. Traditionally, syslog messages were collected centrally and saved to text files. Nowadays, syslog-ng acts more like a log management layer: collects log messages from hosts, saves them for long term storage, but also forwards them to multiple destinations, like SIEMs and other log analysis solutions. This way, it is enough to collect log messages only once, and syslog-ng delivers the right log messages to the right destinations in the right format, after some initial processing.
Humio is available as a cloud service or self-hosted, where you can send all your logs for storage and analysis. It has an easy-to-use interface to query log messages which can be extended with further analytics possibilities from the Humio marketplace.
From this blog, you can learn how to get started with Humio and syslog-ng. While Humio provides many other APIs for log ingestion, I focus on the elasticsearch-http() destination of syslog-ng, demonstrating that there is no vendor lock-in: the same driver works equally well for Elastic’s Elasticsearch, AWS’s OpenSearch and for Humio.