About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / filesystems / overlayfs.txt


Based on kernel version 4.16.1. Page generated on 2018-04-09 11:53 EST.

1	Written by: Neil Brown
2	Please see MAINTAINERS file for where to send questions.
3	
4	Overlay Filesystem
5	==================
6	
7	This document describes a prototype for a new approach to providing
8	overlay-filesystem functionality in Linux (sometimes referred to as
9	union-filesystems).  An overlay-filesystem tries to present a
10	filesystem which is the result over overlaying one filesystem on top
11	of the other.
12	
13	The result will inevitably fail to look exactly like a normal
14	filesystem for various technical reasons.  The expectation is that
15	many use cases will be able to ignore these differences.
16	
17	This approach is 'hybrid' because the objects that appear in the
18	filesystem do not all appear to belong to that filesystem.  In many
19	cases an object accessed in the union will be indistinguishable
20	from accessing the corresponding object from the original filesystem.
21	This is most obvious from the 'st_dev' field returned by stat(2).
22	
23	While directories will report an st_dev from the overlay-filesystem,
24	non-directory objects may report an st_dev from the lower filesystem or
25	upper filesystem that is providing the object.  Similarly st_ino will
26	only be unique when combined with st_dev, and both of these can change
27	over the lifetime of a non-directory object.  Many applications and
28	tools ignore these values and will not be affected.
29	
30	In the special case of all overlay layers on the same underlying
31	filesystem, all objects will report an st_dev from the overlay
32	filesystem and st_ino from the underlying filesystem.  This will
33	make the overlay mount more compliant with filesystem scanners and
34	overlay objects will be distinguishable from the corresponding
35	objects in the original filesystem.
36	
37	Upper and Lower
38	---------------
39	
40	An overlay filesystem combines two filesystems - an 'upper' filesystem
41	and a 'lower' filesystem.  When a name exists in both filesystems, the
42	object in the 'upper' filesystem is visible while the object in the
43	'lower' filesystem is either hidden or, in the case of directories,
44	merged with the 'upper' object.
45	
46	It would be more correct to refer to an upper and lower 'directory
47	tree' rather than 'filesystem' as it is quite possible for both
48	directory trees to be in the same filesystem and there is no
49	requirement that the root of a filesystem be given for either upper or
50	lower.
51	
52	The lower filesystem can be any filesystem supported by Linux and does
53	not need to be writable.  The lower filesystem can even be another
54	overlayfs.  The upper filesystem will normally be writable and if it
55	is it must support the creation of trusted.* extended attributes, and
56	must provide valid d_type in readdir responses, so NFS is not suitable.
57	
58	A read-only overlay of two read-only filesystems may use any
59	filesystem type.
60	
61	Directories
62	-----------
63	
64	Overlaying mainly involves directories.  If a given name appears in both
65	upper and lower filesystems and refers to a non-directory in either,
66	then the lower object is hidden - the name refers only to the upper
67	object.
68	
69	Where both upper and lower objects are directories, a merged directory
70	is formed.
71	
72	At mount time, the two directories given as mount options "lowerdir" and
73	"upperdir" are combined into a merged directory:
74	
75	  mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,\
76	  workdir=/work /merged
77	
78	The "workdir" needs to be an empty directory on the same filesystem
79	as upperdir.
80	
81	Then whenever a lookup is requested in such a merged directory, the
82	lookup is performed in each actual directory and the combined result
83	is cached in the dentry belonging to the overlay filesystem.  If both
84	actual lookups find directories, both are stored and a merged
85	directory is created, otherwise only one is stored: the upper if it
86	exists, else the lower.
87	
88	Only the lists of names from directories are merged.  Other content
89	such as metadata and extended attributes are reported for the upper
90	directory only.  These attributes of the lower directory are hidden.
91	
92	whiteouts and opaque directories
93	--------------------------------
94	
95	In order to support rm and rmdir without changing the lower
96	filesystem, an overlay filesystem needs to record in the upper filesystem
97	that files have been removed.  This is done using whiteouts and opaque
98	directories (non-directories are always opaque).
99	
100	A whiteout is created as a character device with 0/0 device number.
101	When a whiteout is found in the upper level of a merged directory, any
102	matching name in the lower level is ignored, and the whiteout itself
103	is also hidden.
104	
105	A directory is made opaque by setting the xattr "trusted.overlay.opaque"
106	to "y".  Where the upper filesystem contains an opaque directory, any
107	directory in the lower filesystem with the same name is ignored.
108	
109	readdir
110	-------
111	
112	When a 'readdir' request is made on a merged directory, the upper and
113	lower directories are each read and the name lists merged in the
114	obvious way (upper is read first, then lower - entries that already
115	exist are not re-added).  This merged name list is cached in the
116	'struct file' and so remains as long as the file is kept open.  If the
117	directory is opened and read by two processes at the same time, they
118	will each have separate caches.  A seekdir to the start of the
119	directory (offset 0) followed by a readdir will cause the cache to be
120	discarded and rebuilt.
121	
122	This means that changes to the merged directory do not appear while a
123	directory is being read.  This is unlikely to be noticed by many
124	programs.
125	
126	seek offsets are assigned sequentially when the directories are read.
127	Thus if
128	
129	  - read part of a directory
130	  - remember an offset, and close the directory
131	  - re-open the directory some time later
132	  - seek to the remembered offset
133	
134	there may be little correlation between the old and new locations in
135	the list of filenames, particularly if anything has changed in the
136	directory.
137	
138	Readdir on directories that are not merged is simply handled by the
139	underlying directory (upper or lower).
140	
141	renaming directories
142	--------------------
143	
144	When renaming a directory that is on the lower layer or merged (i.e. the
145	directory was not created on the upper layer to start with) overlayfs can
146	handle it in two different ways:
147	
148	1. return EXDEV error: this error is returned by rename(2) when trying to
149	   move a file or directory across filesystem boundaries.  Hence
150	   applications are usually prepared to hande this error (mv(1) for example
151	   recursively copies the directory tree).  This is the default behavior.
152	
153	2. If the "redirect_dir" feature is enabled, then the directory will be
154	   copied up (but not the contents).  Then the "trusted.overlay.redirect"
155	   extended attribute is set to the path of the original location from the
156	   root of the overlay.  Finally the directory is moved to the new
157	   location.
158	
159	There are several ways to tune the "redirect_dir" feature.
160	
161	Kernel config options:
162	
163	- OVERLAY_FS_REDIRECT_DIR:
164	    If this is enabled, then redirect_dir is turned on by  default.
165	- OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW:
166	    If this is enabled, then redirects are always followed by default. Enabling
167	    this results in a less secure configuration.  Enable this option only when
168	    worried about backward compatibility with kernels that have the redirect_dir
169	    feature and follow redirects even if turned off.
170	
171	Module options (can also be changed through /sys/module/overlay/parameters/*):
172	
173	- "redirect_dir=BOOL":
174	    See OVERLAY_FS_REDIRECT_DIR kernel config option above.
175	- "redirect_always_follow=BOOL":
176	    See OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW kernel config option above.
177	- "redirect_max=NUM":
178	    The maximum number of bytes in an absolute redirect (default is 256).
179	
180	Mount options:
181	
182	- "redirect_dir=on":
183	    Redirects are enabled.
184	- "redirect_dir=follow":
185	    Redirects are not created, but followed.
186	- "redirect_dir=off":
187	    Redirects are not created and only followed if "redirect_always_follow"
188	    feature is enabled in the kernel/module config.
189	- "redirect_dir=nofollow":
190	    Redirects are not created and not followed (equivalent to "redirect_dir=off"
191	    if "redirect_always_follow" feature is not enabled).
192	
193	When the NFS export feature is enabled, every copied up directory is
194	indexed by the file handle of the lower inode and a file handle of the
195	upper directory is stored in a "trusted.overlay.upper" extended attribute
196	on the index entry.  On lookup of a merged directory, if the upper
197	directory does not match the file handle stores in the index, that is an
198	indication that multiple upper directories may be redirected to the same
199	lower directory.  In that case, lookup returns an error and warns about
200	a possible inconsistency.
201	
202	Because lower layer redirects cannot be verified with the index, enabling
203	NFS export support on an overlay filesystem with no upper layer requires
204	turning off redirect follow (e.g. "redirect_dir=nofollow").
205	
206	
207	Non-directories
208	---------------
209	
210	Objects that are not directories (files, symlinks, device-special
211	files etc.) are presented either from the upper or lower filesystem as
212	appropriate.  When a file in the lower filesystem is accessed in a way
213	the requires write-access, such as opening for write access, changing
214	some metadata etc., the file is first copied from the lower filesystem
215	to the upper filesystem (copy_up).  Note that creating a hard-link
216	also requires copy_up, though of course creation of a symlink does
217	not.
218	
219	The copy_up may turn out to be unnecessary, for example if the file is
220	opened for read-write but the data is not modified.
221	
222	The copy_up process first makes sure that the containing directory
223	exists in the upper filesystem - creating it and any parents as
224	necessary.  It then creates the object with the same metadata (owner,
225	mode, mtime, symlink-target etc.) and then if the object is a file, the
226	data is copied from the lower to the upper filesystem.  Finally any
227	extended attributes are copied up.
228	
229	Once the copy_up is complete, the overlay filesystem simply
230	provides direct access to the newly created file in the upper
231	filesystem - future operations on the file are barely noticed by the
232	overlay filesystem (though an operation on the name of the file such as
233	rename or unlink will of course be noticed and handled).
234	
235	
236	Multiple lower layers
237	---------------------
238	
239	Multiple lower layers can now be given using the the colon (":") as a
240	separator character between the directory names.  For example:
241	
242	  mount -t overlay overlay -olowerdir=/lower1:/lower2:/lower3 /merged
243	
244	As the example shows, "upperdir=" and "workdir=" may be omitted.  In
245	that case the overlay will be read-only.
246	
247	The specified lower directories will be stacked beginning from the
248	rightmost one and going left.  In the above example lower1 will be the
249	top, lower2 the middle and lower3 the bottom layer.
250	
251	
252	Sharing and copying layers
253	--------------------------
254	
255	Lower layers may be shared among several overlay mounts and that is indeed
256	a very common practice.  An overlay mount may use the same lower layer
257	path as another overlay mount and it may use a lower layer path that is
258	beneath or above the path of another overlay lower layer path.
259	
260	Using an upper layer path and/or a workdir path that are already used by
261	another overlay mount is not allowed and may fail with EBUSY.  Using
262	partially overlapping paths is not allowed but will not fail with EBUSY.
263	If files are accessed from two overlayfs mounts which share or overlap the
264	upper layer and/or workdir path the behavior of the overlay is undefined,
265	though it will not result in a crash or deadlock.
266	
267	Mounting an overlay using an upper layer path, where the upper layer path
268	was previously used by another mounted overlay in combination with a
269	different lower layer path, is allowed, unless the "inodes index" feature
270	is enabled.
271	
272	With the "inodes index" feature, on the first time mount, an NFS file
273	handle of the lower layer root directory, along with the UUID of the lower
274	filesystem, are encoded and stored in the "trusted.overlay.origin" extended
275	attribute on the upper layer root directory.  On subsequent mount attempts,
276	the lower root directory file handle and lower filesystem UUID are compared
277	to the stored origin in upper root directory.  On failure to verify the
278	lower root origin, mount will fail with ESTALE.  An overlayfs mount with
279	"inodes index" enabled will fail with EOPNOTSUPP if the lower filesystem
280	does not support NFS export, lower filesystem does not have a valid UUID or
281	if the upper filesystem does not support extended attributes.
282	
283	It is quite a common practice to copy overlay layers to a different
284	directory tree on the same or different underlying filesystem, and even
285	to a different machine.  With the "inodes index" feature, trying to mount
286	the copied layers will fail the verification of the lower root file handle.
287	
288	
289	Non-standard behavior
290	---------------------
291	
292	The copy_up operation essentially creates a new, identical file and
293	moves it over to the old name.  The new file may be on a different
294	filesystem, so both st_dev and st_ino of the file may change.
295	
296	Any open files referring to this inode will access the old data.
297	
298	Unless "inode index" feature is enabled, if a file with multiple hard
299	links is copied up, then this will "break" the link.  Changes will not be
300	propagated to other names referring to the same inode.
301	
302	Unless "redirect_dir" feature is enabled, rename(2) on a lower or merged
303	directory will fail with EXDEV.
304	
305	Changes to underlying filesystems
306	---------------------------------
307	
308	Offline changes, when the overlay is not mounted, are allowed to either
309	the upper or the lower trees.
310	
311	Changes to the underlying filesystems while part of a mounted overlay
312	filesystem are not allowed.  If the underlying filesystem is changed,
313	the behavior of the overlay is undefined, though it will not result in
314	a crash or deadlock.
315	
316	When the overlay NFS export feature is enabled, overlay filesystems
317	behavior on offline changes of the underlying lower layer is different
318	than the behavior when NFS export is disabled.
319	
320	On every copy_up, an NFS file handle of the lower inode, along with the
321	UUID of the lower filesystem, are encoded and stored in an extended
322	attribute "trusted.overlay.origin" on the upper inode.
323	
324	When the NFS export feature is enabled, a lookup of a merged directory,
325	that found a lower directory at the lookup path or at the path pointed
326	to by the "trusted.overlay.redirect" extended attribute, will verify
327	that the found lower directory file handle and lower filesystem UUID
328	match the origin file handle that was stored at copy_up time.  If a
329	found lower directory does not match the stored origin, that directory
330	will not be merged with the upper directory.
331	
332	
333	
334	NFS export
335	----------
336	
337	When the underlying filesystems supports NFS export and the "nfs_export"
338	feature is enabled, an overlay filesystem may be exported to NFS.
339	
340	With the "nfs_export" feature, on copy_up of any lower object, an index
341	entry is created under the index directory.  The index entry name is the
342	hexadecimal representation of the copy up origin file handle.  For a
343	non-directory object, the index entry is a hard link to the upper inode.
344	For a directory object, the index entry has an extended attribute
345	"trusted.overlay.upper" with an encoded file handle of the upper
346	directory inode.
347	
348	When encoding a file handle from an overlay filesystem object, the
349	following rules apply:
350	
351	1. For a non-upper object, encode a lower file handle from lower inode
352	2. For an indexed object, encode a lower file handle from copy_up origin
353	3. For a pure-upper object and for an existing non-indexed upper object,
354	   encode an upper file handle from upper inode
355	
356	The encoded overlay file handle includes:
357	 - Header including path type information (e.g. lower/upper)
358	 - UUID of the underlying filesystem
359	 - Underlying filesystem encoding of underlying inode
360	
361	This encoding format is identical to the encoding format file handles that
362	are stored in extended attribute "trusted.overlay.origin".
363	
364	When decoding an overlay file handle, the following steps are followed:
365	
366	1. Find underlying layer by UUID and path type information.
367	2. Decode the underlying filesystem file handle to underlying dentry.
368	3. For a lower file handle, lookup the handle in index directory by name.
369	4. If a whiteout is found in index, return ESTALE. This represents an
370	   overlay object that was deleted after its file handle was encoded.
371	5. For a non-directory, instantiate a disconnected overlay dentry from the
372	   decoded underlying dentry, the path type and index inode, if found.
373	6. For a directory, use the connected underlying decoded dentry, path type
374	   and index, to lookup a connected overlay dentry.
375	
376	Decoding a non-directory file handle may return a disconnected dentry.
377	copy_up of that disconnected dentry will create an upper index entry with
378	no upper alias.
379	
380	When overlay filesystem has multiple lower layers, a middle layer
381	directory may have a "redirect" to lower directory.  Because middle layer
382	"redirects" are not indexed, a lower file handle that was encoded from the
383	"redirect" origin directory, cannot be used to find the middle or upper
384	layer directory.  Similarly, a lower file handle that was encoded from a
385	descendant of the "redirect" origin directory, cannot be used to
386	reconstruct a connected overlay path.  To mitigate the cases of
387	directories that cannot be decoded from a lower file handle, these
388	directories are copied up on encode and encoded as an upper file handle.
389	On an overlay filesystem with no upper layer this mitigation cannot be
390	used NFS export in this setup requires turning off redirect follow (e.g.
391	"redirect_dir=nofollow").
392	
393	The overlay filesystem does not support non-directory connectable file
394	handles, so exporting with the 'subtree_check' exportfs configuration will
395	cause failures to lookup files over NFS.
396	
397	When the NFS export feature is enabled, all directory index entries are
398	verified on mount time to check that upper file handles are not stale.
399	This verification may cause significant overhead in some cases.
400	
401	
402	Testsuite
403	---------
404	
405	There's testsuite developed by David Howells at:
406	
407	  git://git.infradead.org/users/dhowells/unionmount-testsuite.git
408	
409	Run as root:
410	
411	  # cd unionmount-testsuite
412	  # ./run --ov
Hide Line Numbers


About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog