Gyula Fora and Matyas Orhidi have started a series on log aggregation with Apache Flink:
There are several off-the-shelf solutions available on the market for log aggregation, which come with their own stack of components and operational difficulties. For example, notable logging frameworks that are widely used in the industry are ELK stack and Graylog.
Unfortunately, there is no clear cut solution that works for every application, and different logging solutions might be more suitable for certain use cases. The log processing of real-time applications should for instance also happen in real-time, otherwise, we lose timely information that may be required to successfully operate the system.
In this blog post, we dive deep into logging for real-time applications.
This post is mostly understanding and setup, but it leads into processing and visualization.