1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
|
<!-- Creator : groff version 1.22.4 -->
<!-- CreationDate: Wed Jun 12 21:10:18 2019 -->
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta name="generator" content="groff -Thtml, see www.gnu.org">
<meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
<meta name="Content-Style" content="text/css">
<style type="text/css">
p { margin-top: 0; margin-bottom: 0; vertical-align: top }
pre { margin-top: 0; margin-bottom: 0; vertical-align: top }
table { margin-top: 0; margin-bottom: 0; vertical-align: top }
h1 { text-align: center }
</style>
<title></title>
</head>
<body>
<hr>
<p>LIBARCHIVE-FORMATS(5) BSD File Formats Manual
LIBARCHIVE-FORMATS(5)</p>
<p style="margin-top: 1em"><b>NAME</b></p>
<p style="margin-left:6%;"><b>libarchive-formats</b>
— archive formats supported by the libarchive
library</p>
<p style="margin-top: 1em"><b>DESCRIPTION</b></p>
<p style="margin-left:6%;">The libarchive(3) library reads
and writes a variety of streaming archive formats. Generally
speaking, all of these archive formats consist of a series
of “entries”. Each entry stores a single file
system object, such as a file, directory, or symbolic
link.</p>
<p style="margin-left:6%; margin-top: 1em">The following
provides a brief description of each format supported by
libarchive, with some information about recognized
extensions or limitations of the current library support.
Note that just because a format is supported by libarchive
does not imply that a program that uses libarchive will
support that format. Applications that use libarchive
specify which formats they wish to support, though many
programs do use libarchive convenience functions to enable
all supported formats.</p>
<p style="margin-left:6%; margin-top: 1em"><b>Tar
Formats</b> <br>
The libarchive(3) library can read most tar archives. It can
write POSIX-standard “ustar” and “pax
interchange” formats as well as v7 tar format and a
subset of the legacy GNU tar format.</p>
<p style="margin-left:6%; margin-top: 1em">All tar formats
store each entry in one or more 512-byte records. The first
record is used for file metadata, including filename,
timestamp, and mode information, and the file data is stored
in subsequent records. Later variants have extended this by
either appropriating undefined areas of the header record,
extending the header to multiple records, or by storing
special entries that modify the interpretation of subsequent
entries.</p>
<p style="margin-top: 1em"><b>gnutar</b></p>
<p style="margin-left:17%; margin-top: 1em">The
libarchive(3) library can read most GNU-format tar archives.
It currently supports the most popular GNU extensions,
including modern long filename and linkname support, as well
as atime and ctime data. The libarchive library does not
support multi-volume archives, nor the old GNU long filename
format. It can read GNU sparse file entries, including the
new POSIX-based formats.</p>
<p style="margin-left:17%; margin-top: 1em">The
libarchive(3) library can write GNU tar format, including
long filename and linkname support, as well as atime and
ctime data.</p>
<p style="margin-top: 1em"><b>pax</b></p>
<p style="margin-left:17%; margin-top: 1em">The
libarchive(3) library can read and write POSIX-compliant pax
interchange format archives. Pax interchange format archives
are an extension of the older ustar format that adds a
separate entry with additional attributes stored as
key/value pairs immediately before each regular entry. The
presence of these additional entries is the only difference
between pax interchange format and the older ustar format.
The extended attributes are of unlimited length and are
stored as UTF-8 Unicode strings. Keywords defined in the
standard are in all lowercase; vendors are allowed to define
custom keys by preceding them with the vendor name in all
uppercase. When writing pax archives, libarchive uses many
of the SCHILY keys defined by Joerg Schilling’s
“star” archiver and a few LIBARCHIVE keys. The
libarchive library can read most of the SCHILY keys and most
of the GNU keys introduced by GNU tar. It silently ignores
any keywords that it does not understand.</p>
<p style="margin-left:17%; margin-top: 1em">The pax
interchange format converts filenames to Unicode and stores
them using the UTF-8 encoding. Prior to libarchive 3.0,
libarchive erroneously assumed that the system
wide-character routines natively supported Unicode. This
caused it to mis-handle non-ASCII filenames on systems that
did not satisfy this assumption.</p>
<p style="margin-top: 1em"><b>restricted pax</b></p>
<p style="margin-left:17%;">The libarchive library can also
write pax archives in which it attempts to suppress the
extended attributes entry whenever possible. The result will
be identical to a ustar archive unless the extended
attributes entry is required to store a long file name, long
linkname, extended ACL, file flags, or if any of the
standard ustar data (user name, group name, UID, GID, etc)
cannot be fully represented in the ustar header. In all
cases, the result can be dearchived by any program that can
read POSIX-compliant pax interchange format archives.
Programs that correctly read ustar format (see below) will
also be able to read this format; any extended attributes
will be extracted as separate files stored in
<i>PaxHeader</i> directories.</p>
<p style="margin-top: 1em"><b>ustar</b></p>
<p style="margin-left:17%; margin-top: 1em">The libarchive
library can both read and write this format. This format has
the following limitations:</p>
<p><b>•</b></p>
<p style="margin-left:22%;">Device major and minor numbers
are limited to 21 bits. Nodes with larger numbers will not
be added to the archive.</p>
<p><b>•</b></p>
<p style="margin-left:22%;">Path names in the archive are
limited to 255 bytes. (Shorter if there is no / character in
exactly the right place.)</p>
<p><b>•</b></p>
<p style="margin-left:22%;">Symbolic links and hard links
are stored in the archive with the name of the referenced
file. This name is limited to 100 bytes.</p>
<p><b>•</b></p>
<p style="margin-left:22%;">Extended attributes, file
flags, and other extended security information cannot be
stored.</p>
<p><b>•</b></p>
<p style="margin-left:22%;">Archive entries are limited to
8 gigabytes in size.</p>
<p style="margin-left:17%;">Note that the pax interchange
format has none of these restrictions. The ustar format is
old and widely supported. It is recommended when
compatibility is the primary concern.</p>
<p style="margin-top: 1em"><b>v7</b></p>
<p style="margin-left:17%; margin-top: 1em">The libarchive
library can read and write the legacy v7 tar format. This
format has the following limitations:</p>
<p><b>•</b></p>
<p style="margin-left:22%;">Only regular files,
directories, and symbolic links can be archived. Block and
character device nodes, FIFOs, and sockets cannot be
archived.</p>
<p><b>•</b></p>
<p style="margin-left:22%;">Path names in the archive are
limited to 100 bytes.</p>
<p><b>•</b></p>
<p style="margin-left:22%;">Symbolic links and hard links
are stored in the archive with the name of the referenced
file. This name is limited to 100 bytes.</p>
<p><b>•</b></p>
<p style="margin-left:22%;">User and group information are
stored as numeric IDs; there is no provision for storing
user or group names.</p>
<p><b>•</b></p>
<p style="margin-left:22%;">Extended attributes, file
flags, and other extended security information cannot be
stored.</p>
<p><b>•</b></p>
<p style="margin-left:22%;">Archive entries are limited to
8 gigabytes in size.</p>
<p style="margin-left:17%;">Generally, users should prefer
the ustar format for portability as the v7 tar format is
both less useful and less portable.</p>
<p style="margin-left:6%; margin-top: 1em">The libarchive
library also reads a variety of commonly-used extensions to
the basic tar format. These extensions are recognized
automatically whenever they appear.</p>
<p style="margin-top: 1em">Numeric extensions.</p>
<p style="margin-left:17%;">The POSIX standards require
fixed-length numeric fields to be written with some
character position reserved for terminators. Libarchive
allows these fields to be written without terminator
characters. This extends the allowable range; in particular,
ustar archives with this extension can support entries up to
64 gigabytes in size. Libarchive also recognizes base-256
values in most numeric fields. This essentially removes all
limitations on file size, modification time, and device
numbers.</p>
<p style="margin-top: 1em">Solaris extensions</p>
<p style="margin-left:17%;">Libarchive recognizes ACL and
extended attribute records written by Solaris tar.</p>
<p style="margin-left:6%; margin-top: 1em">The first tar
program appeared in Seventh Edition Unix in 1979. The first
official standard for the tar file format was the
“ustar” (Unix Standard Tar) format defined by
POSIX in 1988. POSIX.1-2001 extended the ustar format to
create the “pax interchange” format.</p>
<p style="margin-left:6%; margin-top: 1em"><b>Cpio
Formats</b> <br>
The libarchive library can read a number of common cpio
variants and can write “odc” and
“newc” format archives. A cpio archive stores
each entry as a fixed-size header followed by a
variable-length filename and variable-length data. Unlike
the tar format, the cpio format does only minimal padding of
the header or file data. There are several cpio variants,
which differ primarily in how they store the initial header:
some store the values as octal or hexadecimal numbers in
ASCII, others as binary values of varying byte order and
length.</p>
<p style="margin-top: 1em"><b>binary</b></p>
<p style="margin-left:17%; margin-top: 1em">The libarchive
library transparently reads both big-endian and
little-endian variants of the original binary cpio format.
This format used 32-bit binary values for file size and
mtime, and 16-bit binary values for the other fields.</p>
<p style="margin-top: 1em"><b>odc</b></p>
<p style="margin-left:17%; margin-top: 1em">The libarchive
library can both read and write this POSIX-standard format,
which is officially known as the “cpio interchange
format” or the “octet-oriented cpio archive
format” and sometimes unofficially referred to as the
“old character format”. This format stores the
header contents as octal values in ASCII. It is standard,
portable, and immune from byte-order confusion. File sizes
and mtime are limited to 33 bits (8GB file size), other
fields are limited to 18 bits.</p>
<p style="margin-top: 1em"><b>SVR4/newc</b></p>
<p style="margin-left:17%;">The libarchive library can read
both CRC and non-CRC variants of this format. The SVR4
format uses eight-digit hexadecimal values for all header
fields. This limits file size to 4GB, and also limits the
mtime and other fields to 32 bits. The SVR4 format can
optionally include a CRC of the file contents, although
libarchive does not currently verify this CRC.</p>
<p style="margin-left:6%; margin-top: 1em">Cpio first
appeared in PWB/UNIX 1.0, which was released within AT&T
in 1977. PWB/UNIX 1.0 formed the basis of System III Unix,
released outside of AT&T in 1981. This makes cpio older
than tar, although cpio was not included in Version 7
AT&T Unix. As a result, the tar command became much
better known in universities and research groups that used
Version 7. The combination of the <b>find</b> and
<b>cpio</b> utilities provided very precise control over
file selection. Unfortunately, the format has many
limitations that make it unsuitable for widespread use. Only
the POSIX format permits files over 4GB, and its 18-bit
limit for most other fields makes it unsuitable for modern
systems. In addition, cpio formats only store numeric
UID/GID values (not usernames and group names), which can
make it very difficult to correctly transfer archives across
systems with dissimilar user numbering.</p>
<p style="margin-left:6%; margin-top: 1em"><b>Shar
Formats</b> <br>
A “shell archive” is a shell script that, when
executed on a POSIX-compliant system, will recreate a
collection of file system objects. The libarchive library
can write two different kinds of shar archives:</p>
<p style="margin-top: 1em"><b>shar</b></p>
<p style="margin-left:17%; margin-top: 1em">The traditional
shar format uses a limited set of POSIX commands, including
echo(1), mkdir(1), and sed(1). It is suitable for portably
archiving small collections of plain text files. However, it
is not generally well-suited for large archives (many
implementations of sh(1) have limits on the size of a
script) nor should it be used with non-text files.</p>
<p style="margin-top: 1em"><b>shardump</b></p>
<p style="margin-left:17%;">This format is similar to shar
but encodes files using uuencode(1) so that the result will
be a plain text file regardless of the file contents. It
also includes additional shell commands that attempt to
reproduce as many file attributes as possible, including
owner, mode, and flags. The additional commands used to
restore file attributes make shardump archives less portable
than plain shar archives.</p>
<p style="margin-left:6%; margin-top: 1em"><b>ISO9660
format</b> <br>
Libarchive can read and extract from files containing
ISO9660-compliant CDROM images. In many cases, this can
remove the need to burn a physical CDROM just in order to
read the files contained in an ISO9660 image. It also avoids
security and complexity issues that come with virtual mounts
and loopback devices. Libarchive supports the most common
Rockridge extensions and has partial support for Joliet
extensions. If both extensions are present, the Joliet
extensions will be used and the Rockridge extensions will be
ignored. In particular, this can create problems with
hardlinks and symlinks, which are supported by Rockridge but
not by Joliet.</p>
<p style="margin-left:6%; margin-top: 1em">Libarchive reads
ISO9660 images using a streaming strategy. This allows it to
read compressed images directly (decompressing on the fly)
and allows it to read images directly from network sockets,
pipes, and other non-seekable data sources. This strategy
works well for optimized ISO9660 images created by many
popular programs. Such programs collect all directory
information at the beginning of the ISO9660 image so it can
be read from a physical disk with a minimum of seeking.
However, not all ISO9660 images can be read in this
fashion.</p>
<p style="margin-left:6%; margin-top: 1em">Libarchive can
also write ISO9660 images. Such images are fully optimized
with the directory information preceding all file data. This
is done by storing all file data to a temporary file while
collecting directory information in memory. When the image
is finished, libarchive writes out the directory structure
followed by the file data. The location used for the
temporary file can be changed by the usual environment
variables.</p>
<p style="margin-left:6%; margin-top: 1em"><b>Zip
format</b> <br>
Libarchive can read and write zip format archives that have
uncompressed entries and entries compressed with the
“deflate” algorithm. Other zip compression
algorithms are not supported. It can extract jar archives,
archives that use Zip64 extensions and self-extracting zip
archives. Libarchive can use either of two different
strategies for reading Zip archives: a streaming strategy
which is fast and can handle extremely large archives, and a
seeking strategy which can correctly process self-extracting
Zip archives and archives with deleted members or other
in-place modifications.</p>
<p style="margin-left:6%; margin-top: 1em">The streaming
reader processes Zip archives as they are read. It can read
archives of arbitrary size from tape or network sockets, and
can decode Zip archives that have been separately compressed
or encoded. However, self-extracting Zip archives and
archives with certain types of modifications cannot be
correctly handled. Such archives require that the reader
first process the Central Directory, which is ordinarily
located at the end of a Zip archive and is thus inaccessible
to the streaming reader. If the program using libarchive has
enabled seek support, then libarchive will use this to
processes the central directory first.</p>
<p style="margin-left:6%; margin-top: 1em">In particular,
the seeking reader must be used to correctly handle
self-extracting archives. Such archives consist of a program
followed by a regular Zip archive. The streaming reader
cannot parse the initial program portion, but the seeking
reader starts by reading the Central Directory from the end
of the archive. Similarly, Zip archives that have been
modified in-place can have deleted entries or other garbage
data that can only be accurately detected by first reading
the Central Directory.</p>
<p style="margin-left:6%; margin-top: 1em"><b>Archive
(library) file format</b> <br>
The Unix archive format (commonly created by the ar(1)
archiver) is a general-purpose format which is used almost
exclusively for object files to be read by the link editor
ld(1). The ar format has never been standardised. There are
two common variants: the GNU format derived from SVR4, and
the BSD format, which first appeared in 4.4BSD. The two
differ primarily in their handling of filenames longer than
15 characters: the GNU/SVR4 variant writes a filename table
at the beginning of the archive; the BSD format stores each
long filename in an extension area adjacent to the entry.
Libarchive can read both extensions, including archives that
may include both types of long filenames. Programs using
libarchive can write GNU/SVR4 format if they provide an
entry called <i>//</i> containing a filename table to be
written into the archive before any of the entries. Any
entries whose names are not in the filename table will be
written using BSD-style long filenames. This can cause
problems for programs such as GNU ld that do not support the
BSD-style long filenames.</p>
<p style="margin-left:6%; margin-top: 1em"><b>mtree</b>
<br>
Libarchive can read and write files in mtree(5) format. This
format is not a true archive format, but rather a textual
description of a file hierarchy in which each line specifies
the name of a file and provides specific metadata about that
file. Libarchive can read all of the keywords supported by
both the NetBSD and FreeBSD versions of mtree(8), although
many of the keywords cannot currently be stored in an
archive_entry object. When writing, libarchive supports use
of the archive_write_set_options(3) interface to specify
which keywords should be included in the output. If
libarchive was compiled with access to suitable
cryptographic libraries (such as the OpenSSL libraries), it
can compute hash entries such as <b>sha512</b> or <b>md5</b>
from file data being written to the mtree writer.</p>
<p style="margin-left:6%; margin-top: 1em">When reading an
mtree file, libarchive will locate the corresponding files
on disk using the <b>contents</b> keyword if present or the
regular filename. If it can locate and open the file on
disk, it will use that to fill in any metadata that is
missing from the mtree file and will read the file contents
and return those to the program using libarchive. If it
cannot locate and open the file on disk, libarchive will
return an error for any attempt to read the entry body.</p>
<p style="margin-left:6%; margin-top: 1em"><b>7-Zip</b>
<br>
Libarchive can read and write 7-Zip format archives. TODO:
Need more information</p>
<p style="margin-left:6%; margin-top: 1em"><b>CAB</b> <br>
Libarchive can read Microsoft Cabinet ( “CAB”)
format archives. TODO: Need more information.</p>
<p style="margin-left:6%; margin-top: 1em"><b>LHA</b> <br>
TODO: Information about libarchive’s LHA support</p>
<p style="margin-left:6%; margin-top: 1em"><b>RAR</b> <br>
Libarchive has limited support for reading RAR format
archives. Currently, libarchive can read RARv3 format
archives which have been either created uncompressed, or
compressed using any of the compression methods supported by
the RARv3 format. Libarchive can also read self-extracting
RAR archives.</p>
<p style="margin-left:6%; margin-top: 1em"><b>Warc</b> <br>
Libarchive can read and write “web archives”.
TODO: Need more information</p>
<p style="margin-left:6%; margin-top: 1em"><b>XAR</b> <br>
Libarchive can read and write the XAR format used by many
Apple tools. TODO: Need more information</p>
<p style="margin-top: 1em"><b>SEE ALSO</b></p>
<p style="margin-left:6%;">ar(1), cpio(1), mkisofs(1),
shar(1), tar(1), zip(1), zlib(3), cpio(5), mtree(5),
tar(5)</p>
<p style="margin-left:6%; margin-top: 1em">BSD
December 27, 2016 BSD</p>
<hr>
</body>
</html>
|
