Machine Learning- Bias and variance Trade-off

In Machine learning world, there is hardly any person who has not heard about Bias and variance trade-off. In this post, I will briefly reiterate the term for the ones who are new to machine learning.

Let’s start with Bias, what is Bias in terms of Machine Learning? Well, Bias is nothing but the model error .In other words, Bias refers to ability of the model to fit the data or to approximate the data. Higher the bias, lower is the ability of the model to approximate the data or higher is the error. So what do you think? How should a perfect model behave? To answer the question, Model should certainly have low bias i.e the error should be low. Sounds simple so far, hmm? Ok let me ask you another question. How low should the bias be for a model so that model is considered an ideal model. Well, there is no answer to this question (I will explain why) and that is where I will introduce the second term, Variance.

Variance refers to consistency of accuracy of a model from data set to data set. In other words, the model should have consistent accuracy across different but similar data sets. The lower the variance more effective is the model. So we can say that in an ideal situation, the model would have low bias and low variance but this is where Bias and Variance Trade-off comes in.

Unfortunately, there is always a trade-off between bias and variance. If you try to achieve low bias on training data then you may suffer from high variance on test data; If you try to achieve low variance then it comes at the cost of higher bias. Let’s try to understand it in more detail with the help of an example:

Consider the linear regression model in below example. This model will have low variance because it is smoother predictor, which means that this model should behave consistently across different but similar data sets because this model is not trying to fit each and every training point. However, this model has bias because it has higher error rate.

 Now, consider another example, with low bias. In this example, the model is trying to fit each and every training data point(over fitting) so this model will have low bias but will have higher variance .In other words, this model will behave perfectly on training data but would not predict well on test data.

On test data, it would have errors as shown below.

So question comes, what is the best state? Objective of any machine learning algorithm is to handle this trade-off in a way that there is neither too much bias and nor too much of variance. The objective is to attain that sweet point where your model fit the data enough that it describes it well but does not over fit to increase variance.

Reference:

Applied predictive Analysis- Dean Abbott-Wiley