About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / DocBook / filesystems.tmpl

Custom Search

Based on kernel version 4.10.8. Page generated on 2017-04-01 14:43 EST.

1	<?xml version="1.0" encoding="UTF-8"?>
2	<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
3		"http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
5	<book id="Linux-filesystems-API">
6	 <bookinfo>
7	  <title>Linux Filesystems API</title>
9	  <legalnotice>
10	   <para>
11	     This documentation is free software; you can redistribute
12	     it and/or modify it under the terms of the GNU General Public
13	     License as published by the Free Software Foundation; either
14	     version 2 of the License, or (at your option) any later
15	     version.
16	   </para>
18	   <para>
19	     This program is distributed in the hope that it will be
20	     useful, but WITHOUT ANY WARRANTY; without even the implied
22	     See the GNU General Public License for more details.
23	   </para>
25	   <para>
26	     You should have received a copy of the GNU General Public
27	     License along with this program; if not, write to the Free
28	     Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
29	     MA 02111-1307 USA
30	   </para>
32	   <para>
33	     For more details see the file COPYING in the source
34	     distribution of Linux.
35	   </para>
36	  </legalnotice>
37	 </bookinfo>
39	<toc></toc>
41	  <chapter id="vfs">
42	     <title>The Linux VFS</title>
43	     <sect1 id="the_filesystem_types"><title>The Filesystem types</title>
44	!Iinclude/linux/fs.h
45	     </sect1>
46	     <sect1 id="the_directory_cache"><title>The Directory Cache</title>
47	!Efs/dcache.c
48	!Iinclude/linux/dcache.h
49	     </sect1>
50	     <sect1 id="inode_handling"><title>Inode Handling</title>
51	!Efs/inode.c
52	!Efs/bad_inode.c
53	     </sect1>
54	     <sect1 id="registration_and_superblocks"><title>Registration and Superblocks</title>
55	!Efs/super.c
56	     </sect1>
57	     <sect1 id="file_locks"><title>File Locks</title>
58	!Efs/locks.c
59	!Ifs/locks.c
60	     </sect1>
61	     <sect1 id="other_functions"><title>Other Functions</title>
62	!Efs/mpage.c
63	!Efs/namei.c
64	!Efs/buffer.c
65	!Eblock/bio.c
66	!Efs/seq_file.c
67	!Efs/filesystems.c
68	!Efs/fs-writeback.c
69	!Efs/block_dev.c
70	     </sect1>
71	  </chapter>
73	  <chapter id="proc">
74	     <title>The proc filesystem</title>
76	     <sect1 id="sysctl_interface"><title>sysctl interface</title>
77	!Ekernel/sysctl.c
78	     </sect1>
80	     <sect1 id="proc_filesystem_interface"><title>proc filesystem interface</title>
81	!Ifs/proc/base.c
82	     </sect1>
83	  </chapter>
85	  <chapter id="fs_events">
86	     <title>Events based on file descriptors</title>
87	!Efs/eventfd.c
88	  </chapter>
90	  <chapter id="sysfs">
91	     <title>The Filesystem for Exporting Kernel Objects</title>
92	!Efs/sysfs/file.c
93	!Efs/sysfs/symlink.c
94	  </chapter>
96	  <chapter id="debugfs">
97	     <title>The debugfs filesystem</title>
99	     <sect1 id="debugfs_interface"><title>debugfs interface</title>
100	!Efs/debugfs/inode.c
101	!Efs/debugfs/file.c
102	     </sect1>
103	  </chapter>
105	  <chapter id="LinuxJDBAPI">
106	  <chapterinfo>
107	  <title>The Linux Journalling API</title>
109	  <authorgroup>
110	  <author>
111	     <firstname>Roger</firstname>
112	     <surname>Gammans</surname>
113	     <affiliation>
114	     <address>
115	      <email>rgammans@computer-surgery.co.uk</email>
116	     </address>
117	    </affiliation>
118	     </author>
119	  </authorgroup>
121	  <authorgroup>
122	   <author>
123	    <firstname>Stephen</firstname>
124	    <surname>Tweedie</surname>
125	    <affiliation>
126	     <address>
127	      <email>sct@redhat.com</email>
128	     </address>
129	    </affiliation>
130	   </author>
131	  </authorgroup>
133	  <copyright>
134	   <year>2002</year>
135	   <holder>Roger Gammans</holder>
136	  </copyright>
137	  </chapterinfo>
139	  <title>The Linux Journalling API</title>
141	    <sect1 id="journaling_overview">
142	     <title>Overview</title>
143	    <sect2 id="journaling_details">
144	     <title>Details</title>
145	<para>
146	The journalling layer is  easy to use. You need to
147	first of all create a journal_t data structure. There are
148	two calls to do this dependent on how you decide to allocate the physical
149	media on which the journal resides. The jbd2_journal_init_inode() call
150	is for journals stored in filesystem inodes, or the jbd2_journal_init_dev()
151	call can be used for journal stored on a raw device (in a continuous range
152	of blocks). A journal_t is a typedef for a struct pointer, so when
153	you are finally finished make sure you call jbd2_journal_destroy() on it
154	to free up any used kernel memory.
155	</para>
157	<para>
158	Once you have got your journal_t object you need to 'mount' or load the journal
159	file. The journalling layer expects the space for the journal was already
160	allocated and initialized properly by the userspace tools.  When loading the
161	journal you must call jbd2_journal_load() to process journal contents.  If the
162	client file system detects the journal contents does not need to be processed
163	(or even need not have valid contents), it may call jbd2_journal_wipe() to
164	clear the journal contents before calling jbd2_journal_load().
165	</para>
167	<para>
168	Note that jbd2_journal_wipe(..,0) calls jbd2_journal_skip_recovery() for you if
169	it detects any outstanding transactions in the journal and similarly
170	jbd2_journal_load() will call jbd2_journal_recover() if necessary.  I would
171	advise reading ext4_load_journal() in fs/ext4/super.c for examples on this
172	stage.
173	</para>
175	<para>
176	Now you can go ahead and start modifying the underlying
177	filesystem. Almost.
178	</para>
180	<para>
182	You still need to actually journal your filesystem changes, this
183	is done by wrapping them into transactions. Additionally you
184	also need to wrap the modification of each of the buffers
185	with calls to the journal layer, so it knows what the modifications
186	you are actually making are. To do this use jbd2_journal_start() which
187	returns a transaction handle.
188	</para>
190	<para>
191	jbd2_journal_start()
192	and its counterpart jbd2_journal_stop(), which indicates the end of a
193	transaction are nestable calls, so you can reenter a transaction if necessary,
194	but remember you must call jbd2_journal_stop() the same number of times as
195	jbd2_journal_start() before the transaction is completed (or more accurately
196	leaves the update phase). Ext4/VFS makes use of this feature to simplify
197	handling of inode dirtying, quota support, etc.
198	</para>
200	<para>
201	Inside each transaction you need to wrap the modifications to the
202	individual buffers (blocks). Before you start to modify a buffer you
203	need to call jbd2_journal_get_{create,write,undo}_access() as appropriate,
204	this allows the journalling layer to copy the unmodified data if it
205	needs to. After all the buffer may be part of a previously uncommitted
206	transaction.
207	At this point you are at last ready to modify a buffer, and once
208	you are have done so you need to call jbd2_journal_dirty_{meta,}data().
209	Or if you've asked for access to a buffer you now know is now longer
210	required to be pushed back on the device you can call jbd2_journal_forget()
211	in much the same way as you might have used bforget() in the past.
212	</para>
214	<para>
215	A jbd2_journal_flush() may be called at any time to commit and checkpoint
216	all your transactions.
217	</para>
219	<para>
220	Then at umount time , in your put_super() you can then call jbd2_journal_destroy()
221	to clean up your in-core journal object.
222	</para>
224	<para>
225	Unfortunately there a couple of ways the journal layer can cause a deadlock.
226	The first thing to note is that each task can only have
227	a single outstanding transaction at any one time, remember nothing
228	commits until the outermost jbd2_journal_stop(). This means
229	you must complete the transaction at the end of each file/inode/address
230	etc. operation you perform, so that the journalling system isn't re-entered
231	on another journal. Since transactions can't be nested/batched
232	across differing journals, and another filesystem other than
233	yours (say ext4) may be modified in a later syscall.
234	</para>
236	<para>
237	The second case to bear in mind is that jbd2_journal_start() can
238	block if there isn't enough space in the journal for your transaction
239	(based on the passed nblocks param) - when it blocks it merely(!) needs to
240	wait for transactions to complete and be committed from other tasks,
241	so essentially we are waiting for jbd2_journal_stop(). So to avoid
242	deadlocks you must treat jbd2_journal_start/stop() as if they
243	were semaphores and include them in your semaphore ordering rules to prevent
244	deadlocks. Note that jbd2_journal_extend() has similar blocking behaviour to
245	jbd2_journal_start() so you can deadlock here just as easily as on
246	jbd2_journal_start().
247	</para>
249	<para>
250	Try to reserve the right number of blocks the first time. ;-). This will
251	be the maximum number of blocks you are going to touch in this transaction.
252	I advise having a look at at least ext4_jbd.h to see the basis on which
253	ext4 uses to make these decisions.
254	</para>
256	<para>
257	Another wriggle to watch out for is your on-disk block allocation strategy.
258	Why? Because, if you do a delete, you need to ensure you haven't reused any
259	of the freed blocks until the transaction freeing these blocks commits. If you
260	reused these blocks and crash happens, there is no way to restore the contents
261	of the reallocated blocks at the end of the last fully committed transaction.
263	One simple way of doing this is to mark blocks as free in internal in-memory
264	block allocation structures only after the transaction freeing them commits.
265	Ext4 uses journal commit callback for this purpose.
266	</para>
268	<para>
269	With journal commit callbacks you can ask the journalling layer to call a
270	callback function when the transaction is finally committed to disk, so that
271	you can do some of your own management. You ask the journalling layer for
272	calling the callback by simply setting journal->j_commit_callback function
273	pointer and that function is called after each transaction commit. You can also
274	use transaction->t_private_list for attaching entries to a transaction that
275	need processing when the transaction commits.
276	</para>
278	<para>
279	JBD2 also provides a way to block all transaction updates via
280	jbd2_journal_{un,}lock_updates(). Ext4 uses this when it wants a window with a
281	clean and stable fs for a moment.  E.g.
282	</para>
284	<programlisting>
286		jbd2_journal_lock_updates() //stop new stuff happening..
287		jbd2_journal_flush()        // checkpoint everything.
288		..do stuff on stable fs
289		jbd2_journal_unlock_updates() // carry on with filesystem use.
290	</programlisting>
292	<para>
293	The opportunities for abuse and DOS attacks with this should be obvious,
294	if you allow unprivileged userspace to trigger codepaths containing these
295	calls.
296	</para>
298	    </sect2>
300	    <sect2 id="jbd_summary">
301	     <title>Summary</title>
302	<para>
303	Using the journal is a matter of wrapping the different context changes,
304	being each mount, each modification (transaction) and each changed buffer
305	to tell the journalling layer about them.
306	</para>
308	    </sect2>
310	    </sect1>
312	    <sect1 id="data_types">
313	     <title>Data Types</title>
314	     <para>
315		The journalling layer uses typedefs to 'hide' the concrete definitions
316		of the structures used. As a client of the JBD2 layer you can
317		just rely on the using the pointer as a magic cookie  of some sort.
319		Obviously the hiding is not enforced as this is 'C'.
320	     </para>
321		<sect2 id="structures"><title>Structures</title>
322	!Iinclude/linux/jbd2.h
323		</sect2>
324	    </sect1>
326	    <sect1 id="functions">
327	     <title>Functions</title>
328	     <para>
329		The functions here are split into two groups those that
330		affect a journal as a whole, and those which are used to
331		manage transactions
332	     </para>
333		<sect2 id="journal_level"><title>Journal Level</title>
334	!Efs/jbd2/journal.c
335	!Ifs/jbd2/recovery.c
336		</sect2>
337		<sect2 id="transaction_level"><title>Transasction Level</title>
338	!Efs/jbd2/transaction.c
339		</sect2>
340	    </sect1>
341	    <sect1 id="see_also">
342	     <title>See also</title>
343		<para>
344		  <citation>
345		   <ulink url="http://kernel.org/pub/linux/kernel/people/sct/ext3/journal-design.ps.gz">
346		   	Journaling the Linux ext2fs Filesystem, LinuxExpo 98, Stephen Tweedie
347		   </ulink>
348		  </citation>
349		</para>
350		<para>
351		   <citation>
352		   <ulink url="http://olstrans.sourceforge.net/release/OLS2000-ext3/OLS2000-ext3.html">
353		   	Ext3 Journalling FileSystem, OLS 2000, Dr. Stephen Tweedie
354		   </ulink>
355		   </citation>
356		</para>
357	    </sect1>
359	  </chapter>
361	  <chapter id="splice">
362	      <title>splice API</title>
363	  <para>
364		splice is a method for moving blocks of data around inside the
365		kernel, without continually transferring them between the kernel
366		and user space.
367	  </para>
368	!Ffs/splice.c
369	  </chapter>
371	  <chapter id="pipes">
372	      <title>pipes API</title>
373	  <para>
374		Pipe interfaces are all for in-kernel (builtin image) use.
375		They are not exported for use by modules.
376	  </para>
377	!Iinclude/linux/pipe_fs_i.h
378	!Ffs/pipe.c
379	  </chapter>
381	</book>
Hide Line Numbers
About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Information is copyright its respective author. All material is available from the Linux Kernel Source distributed under a GPL License. This page is provided as a free service by mjmwired.net.