Building Observability Tools At Scale

Kevin Feasel

2018-06-05

Tools

Kevin Lew and Sangeeta Narayanan give us some lessons learned from building logging and monitoring solutions at Netflix:

We started our tooling efforts with providing visibility into device and server logs, so that our users can go to one tool instead of having to use separate data-specific tools or logging into servers. Providing visibility into logs is valuable because log messages include important contextual information, especially when errors occur.

However, at some point in our business growth, storing device and server logs didn’t scale because the increasing volume of log data caused our storage cost to balloon and query times to increase. Besides reducing our storage retention time period, we addressed scalability by implementing a real-time stream processing platform called Mantis. Instead of saving all logs to persistent storage, Mantis enables our users to stream logs into memory, and keep only those logs that match SQL-like query criteria. Users also have the choice to transform and save matching logs to persistent storage. A query that retrieves a sample of playback start events for the Apple iPad is shown in the following screenshot:

It’s an interesting post.

Related Posts

Database Health Monitor 2.8.2

Steve Stedman has a new version of the Database Health Monitor: Version 2.8.2 is the July 2019 Release of Database Health Monitor. There was a 2.8 release on July 17, 2019, followed shortly by a 2.8.1 and 2.8.2 release after a report of a bug in the blocking queries report that was fixed. Read on to […]

Read More

SentryOne Plan Explorer in Azure Data Studio

Aaron Bertrand gives us the behind-the-scenes look at SentryOne’s Plan Explorer extension for Azure Data Studio: Like the desktop application, the Plan Explorer extension is designed to provide you with richer graphical execution plans for your real-time queries against SQL Server. It is based on a modest subset of functionality; we’ve started with just the […]

Read More

Categories

June 2018
MTWTFSS
« May Jul »
 123
45678910
11121314151617
18192021222324
252627282930