Efficient Python to Python IPC

python ipc inter-process-communicat
Jonathan picture Jonathan · Oct 20, 2011 · Viewed 31.8k times · Source

What would be an inter-process communication (IPC) framework\technique with the following requirements:

  • Transfer native Python objects between two Python processes
  • Efficient in time and CPU (RAM efficiency irrelevant)
  • Cross-platform Win\Linux
  • Nice to have: works with PyPy

UPDATE 1: the processes are on the same host and use the same versions of Python and other modules

UPDATE 2: the processes are run independently by the user, no one of them spawns the others

Answer

Raymond Hettinger picture Raymond Hettinger · Oct 20, 2011

Native objects don't get shared between processes (due to reference counting).

Instead, you can pickle them and share them using unix domain sockets, mmap, zeromq, or an intermediary such a sqlite3 that is designed for concurrent accesses.

Related questions

wait process until all subprocess finish?

I have a main process which creates two or more sub processes, I want main process to wait until all sub processes finish their operations and exits? # main_script.py p1 = subprocess.Popen(['python script1.py']) p2 = subprocess.Popen(['python …

Read More
On localhost, how do I pick a free port number?

I'm trying to play with inter-process communication and since I could not figure out how to use named pipes under Windows I thought I'll use network sockets. Everything happens locally. The server is able to launch slaves in a separate …

Read More
Combining node.js and Python

Node.js is a perfect match for our web project, but there are few computational tasks for which we would prefer Python. We also already have a Python code for them. We are highly concerned about speed, what is the …

Read More