python del not freeing all the memory

Question

In my python program, I use pandas to read a csv file and store in memory:

data = pandas.read_csv('data.csv')

Before running the above command I check the free memory with free -m and the output is 1704. After running the above command the output is 729. I run

del(data)

to free the memory used by data. Now when I check the free memory the output is 1093 which is much less than the original 1704. Where did the rest go? How can I free it? I'm running all these in ipython and even exiting ipython doesn't free up that memory.

Thanks.

Answer 1

"Exiting ipython doesn't free up that memory" means that you're seeing the effect at the OS level. You're probably seeing the effect of memory caching. That data is saved in memory until the space is needed or the file is read again - instead of reading the file from disk it will be read from the 'inactive' section of memory.

Assuming you're on some unix/linux variant, for details about ram usage

cat /proc/meminfo

Your available memory is Free + Inactive. The Free is available now, and the OS will reclaim space from Inactive as necessary. I keep the following alias (in csh) around to check

alias 'freemem' cat /proc/meminfo | grep -i 'inactive:\|memfree:'

Python's del will remove your variable from the namespace. So if that memory isn't referenced anywhere else in your code, python's garbage collection will kick in and clear data from memory. Or if you want to force it:

import gc
foo = range(10000000)
del foo
gc.collect()

Regardless of all this, the OS will keep the file contents cached so it doesn't have to do redundant disk reads when you try to read that same file again.