Sharing static global data among processes in a Gunicorn / Flask app

python flask multiprocessing wsgi gunicorn
Doctor J picture Doctor J · Nov 10, 2014 · Viewed 8.5k times · Source

I have a Flask app running under Gunicorn, using the sync worker type with 20 worker processes. The app reads a lot of data on startup, which takes time and uses memory. Worse, each process loads its own copy, which causes it to take even longer and take 20X the memory. The data is static and doesn't change. I'd like to load it once and have all 20 workers share it.

If I use the preload_app setting, it only loads in one thread, and initially only takes 1X memory, but then seems to baloon to 20X once requests start coming in. I need fast random access to the data, so I'd rather not do IPC.

Is there any way to share static data among Gunicorn processes?

Answer

aaa90210 picture aaa90210 · Nov 11, 2014

Memory mapped files will allow you to share pages between processes.

https://docs.python.org/2/library/mmap.html

Note that memory consumption statistics are usually misleading and unhelpful. It is usually better to consider the output of vmstat and see if you are swapping a lot.

Related questions

Using python multiprocessing Pool in the terminal and in code modules for Django or Flask

When using multiprocessing.Pool in python with the following code, there is some bizarre behavior. from multiprocessing import Pool p = Pool(3) def f(x): return x threads = [p.apply_async(f, [i]) for i in range(20)] for t in threads: …

Read More
Multiprocess within flask app spinning up 2 processes

I am building a flask app and need some background processes to run. I decided to go with multiprocess, but it's producing two processes when running within Flask. Does anyone know why this would happen? I've tested it on OS …

Read More
Sharing a lock between gunicorn workers

Is there a good way to share a multiprocessing Lock between gunicorn workers? I am trying to write a json API with Flask. Some of the API calls will interact a python class that manages a running process (like ffmpeg …

Read More