Bidirectional LSTM with Batch Normalization in Keras

python keras lstm recurrent-neural-network keras-layer
abolfazl picture abolfazl · Jun 21, 2017 · Viewed 8k times · Source

I was wondering how to implement biLSTM with Batch Normalization (BN) in Keras. I know that BN layer should be between linearity and nonlinearity, i.e., activation. This is easy to implement with CNN or Dense layers. But, how to do this with biLSTM?

Thanks in advance.

Answer

albarji picture albarji · Aug 31, 2017

If you want to apply BatchNormalization over the linear outputs of an LSTM you can do it as

from keras.models import Sequential
from keras.layers.recurrent import LSTM
from keras.layers.wrappers import Bidirectional
from keras.layers.normalization import BatchNormalization

model = Sequential()
model.add(Bidirectional(LSTM(128, activation=None), input_shape=(256,10)))
model.add(BatchNormalization())

Essentially, you are removing the non-linear activations of the LSTM (but not the gate activations), and then applying BatchNormalization to the outpus.

If what you want is to apply BatchNormalization into one of the inside flows of the LSTM, such as recurrent flows, I'm afraid that feature has not been implemented in Keras.

Related questions

Error when checking model input: expected lstm_1_input to have 3 dimensions, but got array with shape (339732, 29)

My input is simply a csv file with 339732 rows and two columns : the first being 29 feature values, i.e. X the second being a binary label value, i.e. Y I am trying to train my data on a stacked …

Read More
ValueError: Input 0 is incompatible with layer lstm_13: expected ndim=3, found ndim=4

I am trying for multi-class classification and here are the details of my training input and output: train_input.shape= (1, 95000, 360) (95000 length input array with each element being an array of 360 length) train_output.shape = (1, 95000, 22) (22 Classes are there) model = Sequential() model.…

Read More
How to get the output shape of a layer in Keras?

I have the following code in Keras (Basically I am modifying this code for my use) and I get this error: 'ValueError: Error when checking target: expected conv3d_3 to have 5 dimensions, but got array with shape (10, 4096)' Code: from …

Read More