Caffe | Check failed: error == cudaSuccess (2 vs. 0) out of memory

amazon-web-services neural-network deep-learning caffe
Abhilash Panigrahi picture Abhilash Panigrahi · Nov 18, 2015 · Viewed 23.7k times · Source

I am trying to train a network on Caffe. I have image size of 512x640. Batch size is 1. I'm trying to implement FCN-8s.

I am currently running this on a Amazon EC2 instance (g2.2xlarge) with 4GB of GPU memory. But when I run the solver, it immediately throws out an error

Check failed: error == cudaSuccess (2 vs. 0)  out of memory
*** Check failure stack trace: ***
Aborted (core dumped)

Can someone help me proceed from here?

Answer

Shai picture Shai · Nov 19, 2015

The error you get is indeed out of memory, but it's not the RAM, but rather GPU memory (note that the error comes from CUDA).
Usually, when caffe is out of memory - the first thing to do is reduce the batch size (at the cost of gradient accuracy), but since you are already at batch size = 1...
Are you sure batch size is 1 for both TRAIN and TEST phases?

Related questions

Permission denied (publickey) when SSH Access to Amazon EC2 instance

I want to use my Amazon ec2 instance but faced the following error: Permission denied (publickey). I have created my key pair and downloaded .pem file. Given: chmod 600 pem file. Then, this command ssh -i /home/kashif/serverkey.pem ubuntu@…

Read More
Trying to SSH into an Amazon Ec2 instance - permission error

This is probably a stupidly simple question to some :) I've created a new linux instance on Amazon EC2, and as part of that downloaded the .pem file to allow me to SSH in. When I tried to ssh with: ssh …

Read More
Downloading an entire S3 bucket?

I noticed that there doesn't seem to be an option to download an entire S3 bucket from the AWS Management Console. Is there an easy way to grab everything in one of my buckets? I was thinking about making the …

Read More