Compress directory to tar.gz with Commons Compress

awfulHack picture awfulHack · Nov 19, 2012 · Viewed 29.6k times · Source

I'm running into a problem using the commons compress library to create a tar.gz of a directory. I have a directory structure that is as follows.

parent/
    child/
        file1.raw
        fileN.raw

I'm using the following code to do the compression. It runs fine without exceptions. However, when I try to decompress that tar.gz, I get a single file with the name "childDirToCompress". Its the correct size so the files have clearly been appended to each other in the tarring process. The desired output would be a directory. I can't figure out what I'm doing wrong. Can any wise commons compresser set me upon the correct path?

CreateTarGZ() throws CompressorException, FileNotFoundException, ArchiveException, IOException {
            File f = new File("parent");
            File f2 = new File("parent/childDirToCompress");

            File outFile = new File(f2.getAbsolutePath() + ".tar.gz");
            if(!outFile.exists()){
                outFile.createNewFile();
            }
            FileOutputStream fos = new FileOutputStream(outFile);

            TarArchiveOutputStream taos = new TarArchiveOutputStream(new GZIPOutputStream(new BufferedOutputStream(fos)));
            taos.setBigNumberMode(TarArchiveOutputStream.BIGNUMBER_STAR); 
            taos.setLongFileMode(TarArchiveOutputStream.LONGFILE_GNU);
            addFilesToCompression(taos, f2, ".");
            taos.close();

        }

        private static void addFilesToCompression(TarArchiveOutputStream taos, File file, String dir) throws IOException{
            taos.putArchiveEntry(new TarArchiveEntry(file, dir));

            if (file.isFile()) {
                BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file));
                IOUtils.copy(bis, taos);
                taos.closeArchiveEntry();
                bis.close();
            }

            else if(file.isDirectory()) {
                taos.closeArchiveEntry();
                for (File childFile : file.listFiles()) {
                    addFilesToCompression(taos, childFile, file.getName());

                }
            }
        }

Answer

user3613365 picture user3613365 · May 7, 2014

I followed this solution and it worked until I was processing a larger set of files and it randomly crashes after processing 15000 - 16000 files. the following line is leaking file handlers:

IOUtils.copy(new FileInputStream(f), tOut);

and the code crashed with a "Too many open files" error at the OS level The following minor change fix the problem:

FileInputStream in = new FileInputStream(f);
IOUtils.copy(in, tOut);
in.close();