Top "Large-files" questions

Large files, whether binary or text, can sometimes be problematic even for an experienced programmer.

Reading and processing big text file of 25GB

I have to read a big text file of, say, 25 GB and need to process this file within 15-20 minutes. …

java file file-io large-files
Using Python Iterparse For Large XML Files

I need to write a parser in Python that can process some extremely large files ( > 2 GB ) on a computer …

python xml lxml large-files elementtree
Seeking and reading large files in a Linux C++ application

I am running into integer overflow using the standard ftell and fseek options inside of G++, but I guess I …

c++ c linux gnu large-files
Error tokenizing data. C error: out of memory pandas python, large file csv

I have a large csv file of 3.5 go and I want to read it using pandas. This is my code: …

python csv pandas memory large-files
Processing large JSON files in PHP

I am trying to process somewhat large (possibly up to 200M) JSON files. The structure of the file is basically …

php json large-files
Are there any good workarounds to the GitHub 100MB file size limit for text files?

I have a 190 MB plain text file that I want to track on github. The text file is a pronounciation …

git github large-files pre-commit-hook post-commit-hook
Read lines by number from a large file

I have a file with 15 million lines (will not fit in memory). I also have a small vector of line …

r large-files line-numbers
Java: InputStream too slow to read huge files

I have to read a 53 MB file character by character. When I do it in C++ using ifstream, it is …

java inputstream large-files java-native-interface
Searching for a string in a large text file - profiling various methods in python

This question has been asked many times. After spending some time reading the answers, I did some quick profiling to …

python performance search profiling large-files
How do I read a large CSV file with Scala Stream class?

How do I read a large CSV file (> 1 Gb) with a Scala Stream? Do you have a code example? …

scala csv streaming large-files