Assume that the red line is the regression model we learn from the training data set. It can be seen that the learned model fits the training data set perfectly, while it cannot generalize well to the data not included in the training set. There are several ways to avoid the problem of overfitting.
To remedy this problem, we could:
- Get more training examples.
- Use a simple predictor.
- Select a subsample of features.
In this blog post, we focus on the second and third ways to avoid overfitting by introducing regularization on the parametersof the model.
Read the whole thing.