We are happy to announce a new major release of the Logging operator.
It’s been a while since we wrote about the Logging operator. In the background, we were actively working on awesome new features to make Kubernetes logging as convenient and effortless as possible. Among its new features, one stands out as it brings fundamentally new functionality to the Logging operator: the option to use syslog-ng as the central processing element of the Logging operator.
The initial plan for the Logging operator was always to be a collector-independent solution for Kubernetes. We chose Fluent Bit and Fluentd as our first approach because at that time these were the most widely used tools and the team had a lot of experience with them. We are happy that the Logging operator is thriving and reaching more and more users. Naturally, this brings new challenges and we need to re-evaluate the way how the operator works.
We got an interesting case from a Logging operator user. They have relatively big clusters with an application set-up that requires around 100-150 flows to handle the different kinds of applications. Our initial routing implementation for Fluentd has its limitations:
We have worked intensively with this customer to optimize their configuration and we managed to triple the performance of the log processing, but it became clear that big clusters produce a lot of log messages, and our optimizations around Fluentd won’t be able to keep up with large-scale production traffic. So, we had to think outside the box of Logging operator, and add a new tool that can route and process log traffic at scale.
There is an overwhelming number of different tools out there, so we gathered a list of requirements that we expected from the new tool:
Syslog-ng is a mature open-source log management tool that has been used by large enterprises for over twenty years, with a broad feature set and excellent performance. It is actively maintained and developed, has a large, worldwide user base, and is known to work on practically everything, from e-readers to cars and airplanes.
We worked closely with the syslog-ng developers and decided to roll together. Extending syslog-ng capabilities to run smoothly on top of Kubernetes is a much smaller scope than evaluating brand-new tools and doing thorough testing of the basic capabilities. We are not saying that other tools like Vector, Tremor, or OpenTelemetry Collector won’t do the job but we have trust in syslog-ng.
However, syslog-ng didn’t support cloud-native Kubernetes use cases out of the box and showed some weird behavior if its configuration was changed frequently. Also, the format of its configuration needs some getting used to. To overcome these problems, we have:
We have worked with our customers and tested the syslog-ng support in Logging operator in their environment to make sure it meets their requirements. Right now, they are already using it in production, and the solution is stable and has solved their performance issues.
Fluentd remains fully supported in Logging operator, and Logging operator 4.0 is fully compatible with the older 3.x releases.
Logging operator can use Fluentd and syslog-ng as log forwarders to receive, filter, and transform the incoming logs, and to transport them to one or more destination outputs. Fluentd and syslog-ng have their separate capabilities and they support their own destination outputs (of course, there are overlaps). Currently, only a limited number of syslog-ng outputs are available in Logging operator, but we are actively working on extending the list. However, if you’re sending logs to Sumo Logic, or a generic HTTP output, you might want to try syslog-ng.
Routing with syslog-ng is much more flexible than with Fluentd. You can match strings or regular expressions to any part of the message, for example, to route an exact type of message from a container. Like match only the 4XX codes from an Nginx access log.
match: and: - regexp: value: json.kubernetes.labels.app.kubernetes.io/instance pattern: one-eye-log-generator type: string - regexp: value: json.code Pattern: 4* type: glob
We are planning to make more syslog-ng features available from Logging operator in the future, and also to add new features to Logging operator based on syslog-ng. For details, see the Logging operator documentation and out quick start guides.
A list with all the changes since 3.0.0 would be a bit overwhelming: we released 3.0.0 almost three years ago, even the last minor release (3.17.0) is over a year old now. So to keep things short and manageable, let’s see the notable changes and bugfixes since the last patch release (3.17.10):
We’d like to thank the syslog-ng developers for their time and help in making syslog-ng ready for the Kubernetes world. (For example, making ARM builds of syslog-ng available.)
Of course, apart from the bigger features, there have been many smaller additions and fine-tunings in the various Logging operator plugins and outputs, and we’d like to express our sincere and utmost gratitude to all our contributors for actively supporting the project.
If you are watching the Logging operator project, you might have noticed some major changes. The reason for this is that the team behind the Logging operator wanted to spend more time on the project. As a result, Logging operator got its own organization, and the former team (now Axoflow) and Cisco will work together on the success of the project.
What has changed: