Google Answers Logo
View Question
 
Q: UNIX Question! tar size constraint. ( Answered 5 out of 5 stars,   6 Comments )
Question  
Subject: UNIX Question! tar size constraint.
Category: Computers > Operating Systems
Asked by: pwharff-ga
List Price: $4.00
Posted: 12 Jun 2002 23:47 PDT
Expires: 19 Jun 2002 23:47 PDT
Question ID: 25116
What is the size constraint to "tar" in a UNIX or Linux environment?
Answer  
Subject: Re: UNIX Question! tar size constraint.
Answered By: philip_lynx-ga on 13 Jun 2002 01:05 PDT
Rated:5 out of 5 stars
 
Hi pwharff,

The quick answer is: 2^63-1 for the archive, 68'719'476'735 (8^12-1)
bytes for each file, if your environment permits that.
 
as I understand your question, you want to know if you can produce tar
files that are biger than 2 GBytes (and how big you can really make
them). The answer to this question depends on a few simple parameters:

1) Does your operating system support large files?
2) What version of tar are you using?
3) What is the underlying file system?

You can answer question 1) for yourself by verifying that your kernel
supports 64bit file descriptors. For Linux this is the case for
several years now. A quick look in /usr/include/sys/features.h will
tell you, if there is any line containing 'FILE_OFFSET_BITS'. If there
is, your OS very very probably has support for large files.

For Solaris, just check whether 'man largefile' works, or try 'getconf
-a|grep LARGEFILE'. If it works, then you have support for large files
in the operating system. Again, support for large files has been there
for several years.

For other operating systems, try "man -k large file', and see what you
get -- I'll be gladly providing help if you need to ask for
clarification to this answer. Something like "cd /usr/include; grep
'FILE_OFFSET_BITS' * */*" should tell you quickly if there is standard
large file support.

2) What version of tar are you using? This is important. Obviously,
older tar programs won't be able to handle files or archives that are
larger than 2^31-1 bytes (2.1 Gigabytes). Try running 'tar --version'.
If the first line indicates you are using gnu tar, then any version
newer than 1.12.64 will in principle be able to provide you with large
files. Try to run this command: "strings `which tar`|grep 64", and you
should see some lines saying lseek64, creat64, fopen64. If yes, your
tar contains support for large files.

If your tar program does not contain support for large files (most
really do, but maybe you are working on a machine older than 1998?),
you can download the newest gnu tar from ftp://ftp.gnu.org/pub/gnu/tar
and compile it for yourself.

The size of files you put into a tar archive (not the archive itself)
is limited to 11 octal digits, the max. size of a single file is thus
ca. 68 GBytes.

3) Given that both your operating system (and C library), and tar
application support large files, the only really limiting factor is
the file system that you try to create the file in. The theoretical
limit for the tar archive size is 2^63-1 (9'223'372 Terabytes), but
you will reach more practical limits (disk or tape size) much quicker.
Also take into consideration what the file system is. DOS FAT 12
filesystems don't allow files as big as the Linux EXT2, or Sun UFS
file systems.

If you need more precise data (for a specific file system type, or for
the OS, etc.) please do not hesitate to ask for clarification.

I hope my answer is helpful to you,

--philip

Clarification of Answer by philip_lynx-ga on 13 Jun 2002 06:27 PDT
Well, I am glad you, pwharff, and all you commenters, like my answer.
Thank you for the nice rating ;-)

You asked me whether I find answers in the man pages. Yes and no. If
you know what  you are looking for, man pages are great. If you don't,
thinks get a little harder, and you have to dig around. Documentation
for software packages, HOW-TOs, FAQs etc. abound on the Internet. And
there is always the source, if you are inclined to that.
For your particular question, I have looked at man pages on Linux and
Solaris, examined the relevant files in /usr/include (because I have
not been doing large-file stuff for a while things were a little hazy
for me). Finally, I downloaded the latest tar sources, looked at the
ChangeLog in there to see when large-file support was added, and
examined tar.h et al. to see what size limitations were inherent in
the data format.

Uh. This may sound like a lot, but trust me, it was quick, and I
enjoyed it.
pwharff-ga rated this answer:5 out of 5 stars
Excellent answer!!!  Thanks for all the clafication and steps. Do you
find most of your answers in the man pages?

Comments  
Subject: Re: UNIX Question! tar size constraint.
From: axlrose0484-ga on 13 Jun 2002 04:28 PDT
 
This is a very good answer and very simple to understand, gr8
Subject: Re: UNIX Question! tar size constraint.
From: ashwin42-ga on 13 Jun 2002 05:20 PDT
 
Very comprehensive answer. I would like to commend the writer for
covering all the factors.
Subject: Re: UNIX Question! tar size constraint.
From: linux_learner-ga on 13 Jun 2002 05:20 PDT
 
Tis is very informative. Especially for me who is learning the ropes
in Linix internals and system adminstration.
Subject: Re: UNIX Question! tar size constraint.
From: dreams-ga on 13 Jun 2002 05:30 PDT
 
Very Good.. and Concise...
Subject: Re: UNIX Question! tar size constraint.
From: eaglej-ga on 13 Jun 2002 10:38 PDT
 
My version of GNU tar docs also mention that the official POSIX tar
spec limits files to 8GB, but that gnu tar will generate non-posix
(therefore possible non-portable files) with sizes up to somethe like
2^88 bytes. So tars that you want to use on any other POSIX computer
are limited to 8GB files.
Subject: Re: UNIX Question! tar size constraint.
From: weird-ga on 17 Jun 2002 09:26 PDT
 
Please consider also the fact that, since a lot of tar files are
gzip'ed, the maximum filesize may be limited to 2 GB, as gzip (version
1.2.4) supports filesizes upto 2GB. The new version (that's due RSN,
according to http://www.gzip.org/) should support files larger then
2GB. There already is a beta version of gzip (1.3.3) with support for
large files. Check your particular Linux distribution's gzip version
with `gzip --version`.

Important Disclaimer: Answers and comments provided on Google Answers are general information, and are not intended to substitute for informed professional medical, psychiatric, psychological, tax, legal, investment, accounting, or other professional advice. Google does not endorse, and expressly disclaims liability for any product, manufacturer, distributor, service or service provider mentioned or any opinion expressed in answers or comments. Please read carefully the Google Answers Terms of Service.

If you feel that you have found inappropriate content, please let us know by emailing us at answers-support@google.com with the question ID listed above. Thank you.
Search Google Answers for
Google Answers  


Google Home - Answers FAQ - Terms of Service - Privacy Policy