Working with huge files in VIM

hoju picture hoju · Oct 20, 2009 · Viewed 129.7k times · Source

I tried opening a huge (~2GB) file in VIM but it choked. I don't actually need to edit the file, just jump around efficiently.

How can I go about working with very large files in VIM?

Answer

Florian picture Florian · Jul 29, 2011

I had a 12GB file to edit today. The vim LargeFile plugin did not work for me. It still used up all my memory and then printed an error message :-(. I could not use hexedit for either, as it cannot insert anything, just overwrite. Here is an alternative approach:

You split the file, edit the parts and then recombine it. You still need twice the disk space though.

  • Grep for something surrounding the line you would like to edit:

    grep -n 'something' HUGEFILE | head -n 1
    
  • Extract that range of the file. Say the lines you want to edit are at line 4 and 5. Then do:

    sed -n -e '4,5p' -e '5q' HUGEFILE > SMALLPART
    
    • The -n option is required to suppress the default behaviour of sed to print everything
    • 4,5p prints lines 4 and 5
    • 5q aborts sed after processing line 5
  • Edit SMALLPART using your favourite editor.

  • Combine the file:

    (head -n 3 HUGEFILE; cat SMALLPART; sed -e '1,5d' HUGEFILE) > HUGEFILE.new 
    
    • i.e: pick all the lines before the edited lines from the HUGEFILE (which in this case is the top 3 lines), combine it with the edited lines (in this case lines 4 and 5) and use this combined set of lines to replace the equivalent (in this case the top 5 lines) in the HUGEFILE and write it all to a new file.

    HUGEFILE.new will now be your edited file, you can delete the original HUGEFILE.