Bryan Shalloway has an interesting article for us:
In classification problems, under and over sampling techniques shift the distribution of predicted probabilities towards the minority class. If your problem requires accurate probabilities you will need to adjust your predictions in some way during post-processing (or at another step) to account for this.
Bryan has a clear example showing this problem in action.