Django memory leak: possible causes?

python django memory-leaks uwsgi
trinchet picture trinchet · Aug 2, 2014 · Viewed 11.3k times · Source

I've a Django application that every so often is getting into memory leak.

I am not using large data that could overload the memory, in fact the application 'eats' memory incrementally (in a week the memory goes from ~ 70 MB to 4GB), that is why I suspect the garbage collector is missing something, I am not sure though. Also, it seems as this increment is not dependant of the number of requests.

Obvious things like DEBUG=True, leaving open files, etc... no apply here.

I'm using uWSGI 2.0.3 (+ nginx) and Django 1.4.5

I could set up wsgi so that restart the server when the memory exceeds certain limit, but I wouldn't like to do that since that is not a solution really.

Are there any well know situations where the garbage collector "doesn't do its work properly"? Could it provide some code examples?

Is there any configuration of uWSGI + Django that could cause this?

Answer

trinchet picture trinchet · Aug 7, 2014

I haven't found the exact stuff I'm looking for (each project is a world!), but following some clues and advices I managed to solve the issue. I share with you a few links that could help if you are facing a similar problem.

django memory leaks, part I, django memory leaks, part II, Finding and fixing memory leaks in Python and Debugging memory leaks in a WSGI application.

Some useful SO answers/questions: Which Python memory profiler is recommended?, Is there any working memory profiler for Python3, Python memory leaks and Python: Memory leak debugging

Update

pyuwsgimemhog is a new tool that helps to find out where the leak is.

Related questions

Django memory usage going up with every request

I moved my first Django project from DjangoEurope to Webfaction, and that started an issue looking like a memory leak. With every single request memory usage of the server process goes up about 500kb. It never goes down. This goes …

Read More
Python/Django polling of database has memory leak

I've got a Python script running Django for database and memcache, but it's notably runnning as a standalone daemon (i.e. not responding to webserver requests). The daemon checks a Django model Requisition for objects with a status=STATUS_NEW, …

Read More
'pip' is not recognized as an internal or external command

I'm running into a weird error when trying to install Django on my computer. This is the sequence that I typed into my command line: C:\Python34> python get-pip.py Requirement already up-to-date: pip in c:\python34\lib\site-packages …

Read More