Building Observability Tools At Scale

Kevin Feasel

2018-06-05

Tools

Kevin Lew and Sangeeta Narayanan give us some lessons learned from building logging and monitoring solutions at Netflix:

We started our tooling efforts with providing visibility into device and server logs, so that our users can go to one tool instead of having to use separate data-specific tools or logging into servers. Providing visibility into logs is valuable because log messages include important contextual information, especially when errors occur.

However, at some point in our business growth, storing device and server logs didn’t scale because the increasing volume of log data caused our storage cost to balloon and query times to increase. Besides reducing our storage retention time period, we addressed scalability by implementing a real-time stream processing platform called Mantis. Instead of saving all logs to persistent storage, Mantis enables our users to stream logs into memory, and keep only those logs that match SQL-like query criteria. Users also have the choice to transform and save matching logs to persistent storage. A query that retrieves a sample of playback start events for the Apple iPad is shown in the following screenshot:

It’s an interesting post.

Related Posts

Workload Capture with WorkloadTools

Gianluca Sartori continues a series on WorkloadTools: Last week I showed you how to use WorkloadTools to analyze a workload. As you have seen, using SqlWorkload to extract performance data from your workload is extremely easy and it just takes a few keystrokes in your favorite text editor to craft the perfect .json configuration file. […]

Read More

Using WorkloadTools to Analyze a Workload

Gianluca Sartori shares an example of configuring and using the WorkloadTools SQL workload analyzer: Now that the analysis database contains the performance data, you can use WorkloadViewer to visualize it and draw your conclusions. WorkloadViewer is a GUI tool that reads performance data from the analysis database and gives a graphical representation using charts and […]

Read More

Categories

June 2018
MTWTFSS
« May Jul »
 123
45678910
11121314151617
18192021222324
252627282930