What's the difference between "hidden" and "output" in PyTorch LSTM?

deep-learning pytorch lstm recurrent-neural-network tensor
Nathaniel picture Nathaniel · Jan 17, 2018 · Viewed 23k times · Source

I'm having trouble understanding the documentation for PyTorch's LSTM module (and also RNN and GRU, which are similar). Regarding the outputs, it says:

Outputs: output, (h_n, c_n)

  • output (seq_len, batch, hidden_size * num_directions): tensor containing the output features (h_t) from the last layer of the RNN, for each t. If a torch.nn.utils.rnn.PackedSequence has been given as the input, the output will also be a packed sequence.
  • h_n (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t=seq_len
  • c_n (num_layers * num_directions, batch, hidden_size): tensor containing the cell state for t=seq_len

It seems that the variables output and h_n both give the values of the hidden state. Does h_n just redundantly provide the last time step that's already included in output, or is there something more to it than that?

Answer

nnnmmm picture nnnmmm · Jan 17, 2018

I made a diagram. The names follow the PyTorch docs, although I renamed num_layers to w.

output comprises all the hidden states in the last layer ("last" depth-wise, not time-wise). (h_n, c_n) comprises the hidden states after the last timestep, t = n, so you could potentially feed them into another LSTM.

LSTM diagram

The batch dimension is not included.

Related questions

PyTorch - contiguous()

I was going through this example of a LSTM language model on github (link). What it does in general is pretty clear to me. But I'm still struggling to understand what calling contiguous() does, which occurs several times in the …

Read More
PyTorch - How to deactivate dropout in evaluation mode

This is the model I defined it is a simple lstm with 2 fully connect layers. import copy import torch import torch.nn as nn import torch.nn.functional as F import torch.optim as optim class mylstm(nn.Module): def __…

Read More
How to initialize weights in PyTorch?

How to initialize the weights and biases (for example, with He or Xavier initialization) in a network in PyTorch?

Read More