Running out of disk space in Amazon EC2, can't find what I am using my storage for

Derek Corcoran picture Derek Corcoran · Jun 14, 2018 · Viewed 8.1k times · Source

I am Running an AWS ami using a T2.large instance using the US East. I was trying to upload some data and I ran in the terminal:

df -h

and I got this result:

Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           799M  8.6M  790M   2% /run
/dev/xvda1      9.7G  9.6G   32M 100% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           799M     0  799M   0% /run/user/1000

I know I have not uploaded 9.7 GB of data to the instance, but I don't know what /dev/xvda1 is or how to access it.

I also assume that all the tmpfs are temporal files, how can I erase those?

Answering some of the questions in the coments, I runned

sudo du -sh /*

And I got:

16M /bin
124M    /boot
0   /dev
6.5M    /etc
2.7G    /home
0   /initrd.img
0   /initrd.img.old
4.0K    /jupyterhub_cookie_secret
16K /jupyterhub.sqlite
268M    /lib
4.0K    /lib64
16K /lost+found
4.0K    /media
4.0K    /mnt 
562M    /opt
du: cannot access '/proc/15616/task/15616/fd/4': No such file or directory
du: cannot access '/proc/15616/task/15616/fdinfo/4': No such file or directory
du: cannot access '/proc/15616/fd/4': No such file or directory
du: cannot access '/proc/15616/fdinfo/4': No such file or directory
0   /proc
28K /root
8.6M    /run
14M /sbin
8.0K    /snap 
8.0K    /srv
0   /sys
64K /tmp
4.7G    /usr
1.5G    /var
0   /vmlinuz
0   /vmlinuz.old

Answer

guest picture guest · Jun 14, 2018

When you run out of root filesystem space, and aren't doing anything that you know consumes space, then 99% of the time (+/- 98%) it's a logfile. Run this:

sudo du -s /var/log/* | sort -n

You'll see a listing of all of the sub-directories in /var/log (which is the standard logging destination for Linux systems), and at the end you'll probably see an entry with a very large number next to it. If you don't see anything there, then the next place to try is /tmp (which I'd do with du -sh /tmp since it prints a single number with "human" scaling). And if that doesn't work, then you need to run the original command on the root of the filesystem, /* (that may take some time).

Assuming that it is a logfile, then you should take a look at it to see if there's an error in the related application. If not, you may just need to learn about logrotate.