python: read lines from compressed text files

python large-files compression
delete_this_account picture delete_this_account · May 12, 2012 · Viewed 73.1k times · Source

Is it easy to read a line from a gz-compressed text file using python without extracting the file completely? I have a text.gz file which is aroud 200mb. When I extract it, it becomes 7.4gb. And this is not the only file I have to read. For the total process, I have to read 10 files. Although this will be a sequential job, I think it will a smart thing to do it without extarcting the whole information. I do not even know that it is possible. How can it be done using python? I need to read a text file line-by-line.

Answer

fferri picture fferri · Jun 16, 2015

Using gzip.GzipFile:

import gzip

with gzip.open('input.gz','rt') as f:
    for line in f:
        print('got line', line)

Note: gzip.open(filename, mode) is an alias for gzip.GzipFile(filename, mode). I prefer the former, as it looks similar to with open(...) as f: used for opening uncompressed files.

Related questions

How to read line-delimited JSON from large file (line by line)

I'm trying to load a large file (2GB in size) filled with JSON strings, delimited by newlines. Ex: { "key11": value11, "key12": value12, } { "key21": value21, "key22": value22, } … The way I'm importing it now is: content = open(file_path, "r").read() j_…

Read More
Is there a memory efficient and fast way to load big json files in python?

I have some json files with 500MB. If I use the "trivial" json.load to load its content all at once, it will consume a lot of memory. Is there a way to read partially the file? If it was …

Read More
Python: How to read huge text file into memory

I'm using Python 2.6 on a Mac Mini with 1GB RAM. I want to read in a huge text file $ ls -l links.csv; file links.csv; tail links.csv -rw-r--r-- 1 user user 469904280 30 Nov 22:42 links.csv links.csv: ASCII text, with …

Read More