How to create a file of size more than 2GB in Linux/Unix?

linux tar filesize
Ram picture Ram · Feb 18, 2009 · Viewed 9.5k times · Source

I have this home work where I have to transfer a very big file from one source to multiple machines using bittorrent kinda of algorithm. Initially I am cutting the files in to chunks and I transfer chunks to all the targets. Targets have the intelligence to share the chunks they have with other targets. It works fine. I wanted to transfer a 4GB file so I tarred four 1GB files. It didn't error out when I created the 4GB tar file but at the other end while assembling all the chunks back to the original file it errors out saying file size limit exceeded. How can I go about solving this 2GB limitation problem?

Answer

codelogic picture codelogic · Feb 18, 2009

I can think of two possible reasons:

  • You don't have Large File Support enabled in your Linux kernel
  • Your application isn't compiled with large file support (you might need to pass gcc extra flags to tell it to use 64-bit versions of certain file I/O functions. e.g. gcc -D_FILE_OFFSET_BITS=64)

Related questions

Shell command to tar directory excluding certain files/folders

Is there a simple shell command/script that supports excluding certain files/folders from being archived? I have a directory that need to be archived with a sub directory that has a number of very large files I do not …

Read More
How to extract filename.tar.gz file

I want to extract an archive named filename.tar.gz. Using tar -xzvf filename.tar.gz doesn't extract the file. it is gives this error: gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error exit delayed from …

Read More
tar: add all files and directories in current directory INCLUDING .svn and so on

I try to tar.gz a directory and use tar -czf workspace.tar.gz * The resulting tar includes .svn directories in subdirs but NOT in the current directory (as * gets expanded to only 'visible' files before it is passed to …

Read More