Gevent multicore usage

python multiprocessing gevent
thesonix picture thesonix · Mar 25, 2013 · Viewed 10.2k times · Source

I'm just started with python gevent and I was wondering about the cpu / mulitcore usage of the library.

Trying some examples doing many requests via the monkeypatched urllib I noticed, that they were running just on one core using 99% load.

How can I use all cores with gevent using python? Is there best practice? Or are there any side-effects using multiple processes and gevent?

BR dan

Answer

Calvin Cheng picture Calvin Cheng · Mar 25, 2013

Gevent gives you the ability to deal with blocking requests. It does not give you the ability to run on multi-core.

There's only one greenlet (gevent's coroutine) running in a python process at any time. The real benefit of gevent is that it is very powerful when it deals with I/O bottlenecks (which is usually the case for general web apps, web apps serving API endpoints, web-based chat apps or backend and, in general, networked apps). When we do some CPU-heavy computations, there will be no performance-gain from using gevent. When an app is I/O bound, gevent is pure magic.

There is one simple rule: Greenlets get switched away whenever an I/O-operation would block or when you do the switch explicitly (e.g. with gevent.sleep() )

The built-in python threads actually behave in the same (pseudo) "concurrent" way as gevent's greenlets.

The key difference is this - greenlets use cooperative multitasking, where threads use preemptive multitasking. What this means is that a greenlet will never stop executing and "yield" to another greenlet unless it uses certain "yielding" functions (like gevent.socket.socket.recv or gevent.sleep).

Threads, on the other hand, will yield to other threads (sometimes unpredictably) based on when the operating system decides to swap them out.

And finally, to utilize multi-core in Python - if that's what you want - we have to depend on the multiprocessing module (which is a built-in module in Python). This "gets around GIL". Other alternatives include using Jython or executing tasks in parallel (on different CPUs) using a task queue, e.g. Zeromq.

I wrote a very long explanation here - http://learn-gevent-socketio.readthedocs.org/en/latest/. If you care to dive into the details. :-D

Related questions

Using Celery on processes and gevent in tasks at the same time

I'd like to use Celery as a queue for my tasks so my web app could enqueue a task, return a response and the task will be processed meanwhile / someday / ... I build a kind of API, so I don't know …

Read More
Gevent monkeypatching breaking multiprocessing

I am attempting to use multiprocessing's pool to run a group of processes, each of which will run a gevent pool of greenlets. The reason for this is that there is a lot of network activity, but also a lot …

Read More
How to use multiprocessing pool.map with multiple arguments?

In the Python multiprocessing library, is there a variant of pool.map which supports multiple arguments? text = "test" def harvester(text, case): X = case[0] text+ str(X) if __name__ == '__main__': pool = multiprocessing.Pool(processes=6) case = RAW_DATASET pool.…

Read More