Why do we have to normalize the input for an artificial neural network?

machine-learning neural-network normalization
karla picture karla · Jan 12, 2011 · Viewed 111.5k times · Source

It is a principal question, regarding the theory of neural networks:

Why do we have to normalize the input for a neural network?

I understand that sometimes, when for example the input values are non-numerical a certain transformation must be performed, but when we have a numerical input? Why the numbers must be in a certain interval?

What will happen if the data is not normalized?

Answer

finnw picture finnw · Jan 12, 2011

It's explained well here.

If the input variables are combined linearly, as in an MLP [multilayer perceptron], then it is rarely strictly necessary to standardize the inputs, at least in theory. The reason is that any rescaling of an input vector can be effectively undone by changing the corresponding weights and biases, leaving you with the exact same outputs as you had before. However, there are a variety of practical reasons why standardizing the inputs can make training faster and reduce the chances of getting stuck in local optima. Also, weight decay and Bayesian estimation can be done more conveniently with standardized inputs.

Related questions

Batch normalization during testing

For batch normalization during testing, how does one calculate the mean and variance of each activation input (in each layer and input dimension)? Does one record the means and variances from training, calculate the means and variances of the entire …

Read More
Is it right to normalize data and/or weight vectors in a SOM?

So I am being stumped by something that (should) be simple: I have written a SOM for a simple 'play' two-dimensional data set. Here is the data: You can make out 3 clusters by yourself. Now, there are two things that …

Read More
What is the role of the bias in neural networks?

I'm aware of the gradient descent and the back-propagation algorithm. What I don't get is: when is using a bias important and how do you use it? For example, when mapping the AND function, when I use 2 inputs and 1 output, …

Read More