Press "Enter" to skip to content

Rounding Errors by Data Type in DAX

Marco Russo and Alberto Ferrari shave of fractions of a cent:

The first reason to choose a data type is the range of numbers supported and the precision. However, the result of a mathematical operation may produce a number that cannot be represented in the chosen data type, which requires a rounding operation. Therefore, the result of one same sequence of operations can produce different results depending on the data type and the order of execution. In this article, we discuss the typical rounding behavior for each data type and how to avoid possible issues in your DAX formulas because of any differences from the results you may have expected.

Read on to learn what granularity limits exist for integers, fixed decimal numbers, and floating point operations.