How to do gradient clipping in pytorch?

python machine-learning deep-learning pytorch gradient-descent
Gulzar picture Gulzar · Feb 15, 2019 · Viewed 40.7k times · Source

What is the correct way to perform gradient clipping in pytorch?

I have an exploding gradients problem, and I need to program my way around it.

Answer

Rahul picture Rahul · May 10, 2019

A more complete example

optimizer.zero_grad()        
loss, hidden = model(data, hidden, targets)
loss.backward()

torch.nn.utils.clip_grad_norm_(model.parameters(), args.clip)
optimizer.step()

Source: https://github.com/pytorch/pytorch/issues/309

Related questions

How to get mini-batches in pytorch in a clean and efficient way?

I was trying to do a simple thing which was train a linear model with Stochastic Gradient Descent (SGD) using torch: import numpy as np import torch from torch.autograd import Variable import pdb def get_batch2(X,Y,M,…

Read More
Concatenate Two Tensors in Pytorch

RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 2. Got 32 and 71 in dimension 0 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:87 I have a tensor of shape [71 32 1]. I want to make it of shape [100 32 1] by padding zero vectors. …

Read More
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

I am trying to train the following CNN as follows, but I keep getting the same error regarding .cuda() and I am not sure how to fix it. Here is a chunk of my code so far. import matplotlib.pyplot …

Read More