About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / xz.txt




Custom Search

Based on kernel version 3.16. Page generated on 2014-08-06 21:41 EST.

1	
2	XZ data compression in Linux
3	============================
4	
5	Introduction
6	
7	    XZ is a general purpose data compression format with high compression
8	    ratio and relatively fast decompression. The primary compression
9	    algorithm (filter) is LZMA2. Additional filters can be used to improve
10	    compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters
11	    improve compression ratio of executable data.
12	
13	    The XZ decompressor in Linux is called XZ Embedded. It supports
14	    the LZMA2 filter and optionally also BCJ filters. CRC32 is supported
15	    for integrity checking. The home page of XZ Embedded is at
16	    <http://tukaani.org/xz/embedded.html>, where you can find the
17	    latest version and also information about using the code outside
18	    the Linux kernel.
19	
20	    For userspace, XZ Utils provide a zlib-like compression library
21	    and a gzip-like command line tool. XZ Utils can be downloaded from
22	    <http://tukaani.org/xz/>.
23	
24	XZ related components in the kernel
25	
26	    The xz_dec module provides XZ decompressor with single-call (buffer
27	    to buffer) and multi-call (stateful) APIs. The usage of the xz_dec
28	    module is documented in include/linux/xz.h.
29	
30	    The xz_dec_test module is for testing xz_dec. xz_dec_test is not
31	    useful unless you are hacking the XZ decompressor. xz_dec_test
32	    allocates a char device major dynamically to which one can write
33	    .xz files from userspace. The decompressed output is thrown away.
34	    Keep an eye on dmesg to see diagnostics printed by xz_dec_test.
35	    See the xz_dec_test source code for the details.
36	
37	    For decompressing the kernel image, initramfs, and initrd, there
38	    is a wrapper function in lib/decompress_unxz.c. Its API is the
39	    same as in other decompress_*.c files, which is defined in
40	    include/linux/decompress/generic.h.
41	
42	    scripts/xz_wrap.sh is a wrapper for the xz command line tool found
43	    from XZ Utils. The wrapper sets compression options to values suitable
44	    for compressing the kernel image.
45	
46	    For kernel makefiles, two commands are provided for use with
47	    $(call if_needed). The kernel image should be compressed with
48	    $(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2
49	    dictionary. It will also append a four-byte trailer containing the
50	    uncompressed size of the file, which is needed by the boot code.
51	    Other things should be compressed with $(call if_needed,xzmisc)
52	    which will use no BCJ filter and 1 MiB LZMA2 dictionary.
53	
54	Notes on compression options
55	
56	    Since the XZ Embedded supports only streams with no integrity check or
57	    CRC32, make sure that you don't use some other integrity check type
58	    when encoding files that are supposed to be decoded by the kernel. With
59	    liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32
60	    when encoding. With the xz command line tool, use --check=none or
61	    --check=crc32.
62	
63	    Using CRC32 is strongly recommended unless there is some other layer
64	    which will verify the integrity of the uncompressed data anyway.
65	    Double checking the integrity would probably be waste of CPU cycles.
66	    Note that the headers will always have a CRC32 which will be validated
67	    by the decoder; you can only change the integrity check type (or
68	    disable it) for the actual uncompressed data.
69	
70	    In userspace, LZMA2 is typically used with dictionary sizes of several
71	    megabytes. The decoder needs to have the dictionary in RAM, thus big
72	    dictionaries cannot be used for files that are intended to be decoded
73	    by the kernel. 1 MiB is probably the maximum reasonable dictionary
74	    size for in-kernel use (maybe more is OK for initramfs). The presets
75	    in XZ Utils may not be optimal when creating files for the kernel,
76	    so don't hesitate to use custom settings. Example:
77	
78	        xz --check=crc32 --lzma2=dict=512KiB inputfile
79	
80	    An exception to above dictionary size limitation is when the decoder
81	    is used in single-call mode. Decompressing the kernel itself is an
82	    example of this situation. In single-call mode, the memory usage
83	    doesn't depend on the dictionary size, and it is perfectly fine to
84	    use a big dictionary: for maximum compression, the dictionary should
85	    be at least as big as the uncompressed data itself.
86	
87	Future plans
88	
89	    Creating a limited XZ encoder may be considered if people think it is
90	    useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at
91	    the fastest settings, so it isn't clear if LZMA2 encoder is wanted
92	    into the kernel.
93	
94	    Support for limited random-access reading is planned for the
95	    decompression code. I don't know if it could have any use in the
96	    kernel, but I know that it would be useful in some embedded projects
97	    outside the Linux kernel.
98	
99	Conformance to the .xz file format specification
100	
101	    There are a couple of corner cases where things have been simplified
102	    at expense of detecting errors as early as possible. These should not
103	    matter in practice all, since they don't cause security issues. But
104	    it is good to know this if testing the code e.g. with the test files
105	    from XZ Utils.
106	
107	Reporting bugs
108	
109	    Before reporting a bug, please check that it's not fixed already
110	    at upstream. See <http://tukaani.org/xz/embedded.html> to get the
111	    latest code.
112	
113	    Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on
114	    Freenode and talk to Larhzu. I don't actively read LKML or other
115	    kernel-related mailing lists, so if there's something I should know,
116	    you should email to me personally or use IRC.
117	
118	    Don't bother Igor Pavlov with questions about the XZ implementation
119	    in the kernel or about XZ Utils. While these two implementations
120	    include essential code that is directly based on Igor Pavlov's code,
121	    these implementations aren't maintained nor supported by him.
Hide Line Numbers
About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Information is copyright its respective author. All material is available from the Linux Kernel Source distributed under a GPL License. This page is provided as a free service by mjmwired.net.