reading from stdin, while consuming no more memory than needed

python pipe stdin line-by-line
xtofl picture xtofl · Jul 16, 2010 · Viewed 9.6k times · Source

I am trying to create a line-by-line filter in python. However, stdin.readlines() reads all lines in before starting to process, and python runs out of memory (MemoryError).

How can I have just one line in memory at a time?

The kind of code I have:

for line in sys.stdin.readlines():
    if( filter.apply( line ) ):
        print( line )

(note: I'm on 2.6)

Answer

rkhayrov picture rkhayrov · Jul 16, 2010 
for line in sys.stdin:
    ...

Or call .readline() in a loop.

Related questions

How to open every file in a folder?

I have a python script parse.py, which in the script open a file, say file1, and then do something maybe print out the total number of characters. filename = 'file1' f = open(filename, 'r') content = f.read() print filename, …

Read More
How do you read from stdin in python from a pipe which has no ending

I've problem to read from Standard input or pipe in python when the pipe is from a "open" (do not know right name) file. I have as example pipetest.py: import sys import time k = 0 try: for line in sys.…

Read More
Retrieving the output of subprocess.call()

How can I get the output of a process run using subprocess.call()? Passing a StringIO.StringIO object to stdout gives this error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Library/Frameworks/…

Read More