Pipe vs. Temporary File

pipe temporary-files
Matt Fichman picture Matt Fichman · Aug 8, 2011 · Viewed 13.7k times · Source

Is there a big performance difference between:

  • Process A writing to a temp file, and process B reading that file
  • Process A writing to a pipe, and process B reading from that pipe

I'm curious to know what the answer is for both Windows and *nix.

EDIT: I should have asked: Does the buffer cache eliminate the difference between a temp file and a pipe?

Answer

Jonathan Leffler picture Jonathan Leffler · Aug 8, 2011

One big difference is that with the pipe, processes A and B can be running concurrently, so that B gets to work on the output from A before A has finished producing it. Further, the size of the pipe is limited, so A won't be able to produce vastly more data than B has consumed; it will be made to wait for B to catch up.

If the volume of data is big, then writing to the temporary file involves disk activity, even if only for creating and then destroying the file. The data might well stay in the in-memory buffer pools - so no disk I/O there - even for surprisingly large files. Writing to the pipe 'never' involves writing to disk.

Related questions

Redirect Windows cmd stdout and stderr to a single file

I'm trying to redirect all output (stdout + stderr) of a DOS command to a single file: C:\>dir 1> a.txt 2> a.txt The process cannot access the file because it is being used by another process. Is …

Read More
How can I pipe stderr, and not stdout?

I have a program that writes information to stdout and stderr, and I need to process the stderr with grep, leaving stdout aside. Using a temporary file, one could do it in two steps: command > /dev/null 2> temp.…

Read More
Retrieving the output of subprocess.call()

How can I get the output of a process run using subprocess.call()? Passing a StringIO.StringIO object to stdout gives this error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Library/Frameworks/…

Read More