In which cases is the cross-entropy preferred over the mean squared error?

machine-learning neural-network backpropagation mean-square-error cross-entropy
Amogh Mishra picture Amogh Mishra · Apr 9, 2016 · Viewed 37k times · Source

Although both of the above methods provide a better score for the better closeness of prediction, still cross-entropy is preferred. Is it in every case or there are some peculiar scenarios where we prefer cross-entropy over MSE?

Answer

lejlot picture lejlot · Apr 9, 2016

Cross-entropy is prefered for classification, while mean squared error is one of the best choices for regression. This comes directly from the statement of the problems itself - in classification you work with very particular set of possible output values thus MSE is badly defined (as it does not have this kind of knowledge thus penalizes errors in incompatible way). To better understand the phenomena it is good to follow and understand the relations between

  1. cross entropy
  2. logistic regression (binary cross entropy)
  3. linear regression (MSE)

You will notice that both can be seen as a maximum likelihood estimators, simply with different assumptions about the dependent variable.

Related questions

How to use k-fold cross validation in a neural network

We are writing a small ANN which is supposed to categorize 7000 products into 7 classes based on 10 input variables. In order to do this we have to use k-fold cross validation but we are kind of confused. We have this excerpt …

Read More
What is the difference between back-propagation and feed-forward Neural Network?

What is the difference between back-propagation and feed-forward neural networks? By googling and reading, I found that in feed-forward there is only forward direction, but in back-propagation once we need to do a forward-propagation and then back-propagation. I referred to …

Read More
Understanding Neural Network Backpropagation

Update: a better formulation of the issue. I'm trying to understand the backpropagation algorithm with an XOR neural network as an example. For this case there are 2 input neurons + 1 bias, 2 neurons in the hidden layer + 1 bias, and 1 output neuron. A …

Read More