Buffered data was truncated after reaching the output size limit

python keras deep-learning google-colaboratory
lycutter picture lycutter · Jul 22, 2018 · Viewed 18.2k times · Source

When I use Colaboratory to run my NIN model, it occurs an error in the output of training process which tells "Buffered data was truncated after reaching the output size limit." in the 61th epoch. I have no idea about this crash.Is my code still running ? How can I solve this problem?

Here is some information about the output of my training process:

Epoch 57/200
391/391 [==============================] - 53s 135ms/step - loss: 0.8365 - acc: 0.7784 - val_loss: 0.9250 - val_acc: 0.7625
Epoch 58/200
 28/391 [=>............................] - ETA: 46s - loss: 0.8356 - acc: 0.7835391/391 [==============================] - 53s 136ms/step - loss: 0.8288 - acc: 0.7811 - val_loss: 0.8977 - val_acc: 0.7608
Epoch 59/200
326/391 [========================>.....] - ETA: 8s - loss: 0.8309 - acc: 0.7789391/391 [==============================] - 53s 136ms/step - loss: 0.8297 - acc: 0.7798 - val_loss: 0.9030 - val_acc: 0.7628
Epoch 60/200
391/391 [==============================] - 53s 134ms/step - loss: 0.8245 - acc: 0.7825 - val_loss: 0.8378 - val_acc: 0.7767
Epoch 61/200
 28/391 [=>............................] - ETA: 46s - loss: 0.8281 - acc: 0.7879390/391 [============================>.] - ETA: 0s - loss: 0.8177 - acc: 0.7851Buffered data was truncated after reaching the output size limit.

Answer

Praveen kumar picture Praveen kumar · Apr 4, 2019

Even if RAM | GPU | DISK on colab is free, this error still comes because there is a limited memory for displaying output of a cell on colab. Assuming the memory limit is around 2Mb to 5Mb when we run many epochs(148+) during training, it tends to fill that memory and hence the output is truncated because there is no more memory left free to display the buffered epochs. However, the machine keeps running in the background and the output is processed but it is not displayed because of the buffered limit. You will still get your desired output.

One solution is not to use verbose=1 (use 0 instead).

Related questions

Keras, How to get the output of each layer?

I have trained a binary classification model with CNN, and here is my code model = Sequential() model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1], border_mode='valid', input_shape=input_shape)) model.add(Activation('relu')) model.add(Convolution2…

Read More
Keras, how do I predict after I trained a model?

I'm playing with the reuters-example dataset and it runs fine (my model is trained). I read about how to save a model, so I could load it later to use again. But how do I use this saved model to …

Read More
What is the use of verbose in Keras while validating the model?

I'm running the LSTM model for the first time. Here is my model: opt = Adam(0.002) inp = Input(...) print(inp) x = Embedding(....)(inp) x = LSTM(...)(x) x = BatchNormalization()(x) pred = Dense(5,activation='softmax')(x) model = Model(inp,pred) model.compile(....) idx = …

Read More