How do you share data between a parent and forked child process in Python?

python fork share
user12163 picture user12163 · Oct 20, 2008 · Viewed 11.5k times · Source

I'm pretty sure one would do this using the os.plock(op) function, but I have no idea how. Also, if there's a better way, I'd be grateful to find out. Code snippets are very welcome.

Answer

Florian Bösch picture Florian Bösch · Oct 20, 2008

Subprocess replaces os.popen, os.system, os.spawn, popen2 and commands. A simple example for piping would be:

p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]

You could also use a memory mapped file with the flag=MAP_SHARED for shared memory between processes.

multiprocessing abstracts both pipes and shared memory and provides a higher level interface. Taken from the Processing documentation:

from multiprocessing import Process, Pipe

def f(conn):
    conn.send([42, None, 'hello'])
    conn.close()

if __name__ == '__main__':
    parent_conn, child_conn = Pipe()
    p = Process(target=f, args=(child_conn,))
    p.start()
    print parent_conn.recv()   # prints "[42, None, 'hello']"
    p.join()

Related questions

Run child processes as different user from a long running Python process

I've got a long running, daemonized Python process that uses subprocess to spawn new child processes when certain events occur. The long running process is started by a user with super user privileges. I need the child processes it spawns …

Read More
How to spawn a new independent process in Python

I have a some Python code that occasionally needs to span a new process to run a shell script in a "fire and forget" manner, i.e. without blocking. The shell script will not communicate with the original Python code …

Read More
Python:How os.fork() works?

I am learning multiprocessing in python. I tried multiprocessing and after I read the source code of multiprocessing module, I found it use os.fork(), so I wrote some code to test os.fork(), but I am stuck. My code …

Read More