Sudden drop in accuracy while training a deep neural net

neural-network deep-learning mxnet
sau picture sau · May 5, 2016 · Viewed 13.8k times · Source

I am using mxnet to train a 11-class image classifier. I am observing a weird behavior training accuracy was increasing slowly and went upto 39% and in next epoch it went down to 9% and then it stays close to 9% for rest of the training. I restarted the training with saved model (with 39% training accuracy) keeping all other parameter same . Now training accuracy is increasing again. What can be the reason here ? I am not able to understand it . And its getting difficult to train the model this way as it requires me to see training accuracy values constantly.

learning rate is constant at 0.01

Answer

mrphoenix13 picture mrphoenix13 · May 5, 2016

as you can see your late accuracy is near random one. there is 2 common issue in this kind of cases.

  • your learning rate is high. try to lower it
  • The error (or entropy) you are trying to use is giving you NaN value. if you are trying to use entropies with log functions you must use them precisely.

Related questions

Epoch vs Iteration when training neural networks

What is the difference between epoch and iteration when training a multi-layer perceptron?

Read More
Why binary_crossentropy and categorical_crossentropy give different performances for the same problem?

I'm trying to train a CNN to categorize text by topic. When I use binary cross-entropy I get ~80% accuracy, with categorical cross-entropy I get ~50% accuracy. I don't understand why this is. It's a multiclass problem, doesn't that mean that I …

Read More
How to interpret "loss" and "accuracy" for a machine learning model

When I trained my neural network with Theano or Tensorflow, they will report a variable called "loss" per epoch. How should I interpret this variable? Higher loss is better or worse, or what does it mean for the final performance (…

Read More