Press "Enter" to skip to content

Category: Integration Services

Choosing the Right Logging Level in SSIS

Andy Brownsword does a bit of logging:

When creating SQL Agent jobs to execute SSIS packages we can choose the level of logging to be captured. Different settings are more beneficial under the right circumstances so it’s important to understand the differences to make the right decision.

These settings control the internal logging done by SSIS. This is out of the box and freely available, so why not use it effectively.

The real trick is that if you swallow all of the exceptions and errors, everybody will just assume your code is working perfectly and boom, problem solved. Or you could read Andy’s post and get actual information. Whatever works for you, I suppose.

Leave a Comment

SSIS Slowdowns in Paging to Disk

Andy Brownsword notes a major performance risk in Integration Services:

One particular performance issue with SSIS data flows can fly under the radar – spilling to disk. This isn’t clearly visible through regular debugging or execution so can go unnoticed. And it hurts.

Paging to disk is bad for performance. Disks are much slower to access than memory, so we want to keep our data away when possible.

Andy calls out two reasons why we might find spilling to disk, as well as how to track if this is happening.

Leave a Comment

Selective Caching in SSIS

Andy Brownsword takes us through a pattern:

We’ve recently looked at how caching can improve performance and I wanted to show how we can eek even more performance out of caches by using a custom approach I’ll term Selective Caching.

I’ll note here that there’s a potential gotcha with this approach which we’ll get to before the end of the post!

Click through for a description of the pattern and when it starts to break down.

Comments closed

Using a File Cache in SSIS

Andy Brownsword makes those SSIS jobs run faster:

Last week we looked at using a cache to improve lookup performance. We saw how a cache improves performance by being able to reuse reference data repeatedly. That used a regular cache but it’s not the only option available to us.

In this post we’re going to look at the File Cache option which can achieve the same results – plus a little more.

My experience with file caches is that they’re a bit finicky but when you get them running, they can provide a significant speed-up to data enrichment tasks.

Comments closed

Optimizing Multiple Lookup Transformations in SSIS

Andy Brownsword doesn’t want to keep hitting the database:

Lookup transformations provide us a way to access related values from another source, such as retrieving surrogate keys in data warehousing. When we need multiple lookups to the same reference data we can improve performance through the use of a Cache.

If we consider data warehousing, a prime example of this would be an order table which has values for Order Date, Dispatch Date, Delivery Date, etc. All of these would require a lookup to a calendar dimension.

This is a perfect use case for a cache.

Read on to see how the cache connector works.

Comments closed

Tuning SSIS Data Flow Buffers

Andy Brownsword speeds things up:

When using data flows in SSIS packages we want the flow to be fast and fluid. Improving performance of the flows will vary in different packages, but one element can consistently help – tuning memory utilisation.

In this post we’ll look at tuning the memory utilisation by altering the number of rows passing through the flow concurrently. Specifically we’re looking at the following properties:

  • DefaultBufferSize
  • DefaultBufferMaxRows

This is a pretty big deal, finding the right size that allows you to keep data flowing smoothly without having to wait for buffers to fill. The defaults are aggressively low in most cases. And good on Andy for providing a series of tests to give practical numbers.

Comments closed

Handling a Sort Operation in SQL Server Integration Services

Andy Brownsword knows that sometimes, the only winning move is not to play:

Last time out we discussed blocking transformations, what they are, the impact of them, and touched on how to deal with them. In this post we’re going a step further to tackle one of them head on.

Here we’ll demonstrate the impact of blocking caused by the Sort transformation, and look at two options for solving this and slashing execution time.

Sorts aren’t the only blocking transformation that you should push back down to your source (if possible), but it is the most common example.

Comments closed

Blocking Transformations in SSIS Data Flows

Andy Brownsword doesn’t want any blockers:

We have a platter of transformations to use when constructing data flows in SSIS packages. Not all transformations are equal though, and some can catch us out with performance impact as data volumes scale.

In this post we’ll look at blocking transformations which can trip us up if we’re not careful with them (or avoid them).

Click through for some examples of blocking transformations, as well as some tips on how to manage them.

Comments closed

Dealing with Optional Carriage Returns in SSIS

Andy Brownsword has fun with file formats:

When ingesting files in SSIS via Flat File Connections, a consistent format is key. Sometimes that isn’t the case. Here we’ll look at an example where the carriage return (CR\r) may or may not be included in the file.

Pepperidge Farms remembers back in the day when Windows, MacOS, and Linux (or any flavor of UNIX for that matter) each had a different way of ending a line: line feed, carriage return, or both. And of course most tools weren’t smart enough to figure out which your particular text file followed and display it correctly.

Comments closed