Hui Li has an explanation of what regularization is and how it works to reduce the likelihood of overfitting training data:
Assume that the red line is the regression model we learn from the training data set. It can be seen that the learned model fits the training data set perfectly, while it cannot generalize well to the data not included in the training set. There are several ways to avoid the problem of overfitting.
To remedy this problem, we could:
- Get more training examples.
- Use a simple predictor.
- Select a subsample of features.
In this blog post, we focus on the second and third ways to avoid overfitting by introducing regularization on the parameters βi of the model.
Read the whole thing.