Approaches to Deleting Data in Batches

Andy Mallon shares a couple approaches to deleting data in batches:

In this scenario, we’re going to keep the data for X days after it’s created. Then we delete it. That’s it. X could be 3 days or 3 years–it doesn’t matter, we’ll follow the same design pattern.

In today’s world, we generate loads of log data, sensor data, telemetry data, etc. All that data is super duper valuable. But only for a while. Eventually, all that granular data becomes less useful, and isn’t worth keeping around. Maybe it gets aggregated, summarized, or maybe it just gets thrown out.

You’ll have a lot of data with more complex requirements, but I think you’ll also be surprised at how much data has simple date-based retention based on it’s creation.

Also read the comments, as they include additional techniques.

Related Posts

When tempdb Spills Attack

Josh Darnell ran into a problem with a SQL Agent job: One of my colleagues reached out to me recently about a production issue where a SQL Server Agent job had failed with this error message: Msg 1105, Level 17, State 2, Line 15Could not allocate space for object ‘dbo.SORT temporary run storage: 140737513062400’ in […]

Read More

Simple Query Zen

Erik Darling wants you to simplify your life queries: See, when a query is big and complicated to you, there’s a pretty good chance you’re gonna get a big and complicated query plan, because it’s big and complicated to the optimizer, too. This isn’t to say the optimizer is dumb or bad or ugly; it’s […]

Read More

Categories

April 2019
MTWTFSS
« Mar May »
1234567
891011121314
15161718192021
22232425262728
2930