About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / fault-injection




Custom Search

Based on kernel version 3.13. Page generated on 2014-01-20 22:02 EST.

1	Fault injection capabilities infrastructure
2	===========================================
3	
4	See also drivers/md/faulty.c and "every_nth" module option for scsi_debug.
5	
6	
7	Available fault injection capabilities
8	--------------------------------------
9	
10	o failslab
11	
12	  injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
13	
14	o fail_page_alloc
15	
16	  injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
17	
18	o fail_make_request
19	
20	  injects disk IO errors on devices permitted by setting
21	  /sys/block/<device>/make-it-fail or
22	  /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
23	
24	o fail_mmc_request
25	
26	  injects MMC data errors on devices permitted by setting
27	  debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
28	
29	Configure fault-injection capabilities behavior
30	-----------------------------------------------
31	
32	o debugfs entries
33	
34	fault-inject-debugfs kernel module provides some debugfs entries for runtime
35	configuration of fault-injection capabilities.
36	
37	- /sys/kernel/debug/fail*/probability:
38	
39		likelihood of failure injection, in percent.
40		Format: <percent>
41	
42		Note that one-failure-per-hundred is a very high error rate
43		for some testcases.  Consider setting probability=100 and configure
44		/sys/kernel/debug/fail*/interval for such testcases.
45	
46	- /sys/kernel/debug/fail*/interval:
47	
48		specifies the interval between failures, for calls to
49		should_fail() that pass all the other tests.
50	
51		Note that if you enable this, by setting interval>1, you will
52		probably want to set probability=100.
53	
54	- /sys/kernel/debug/fail*/times:
55	
56		specifies how many times failures may happen at most.
57		A value of -1 means "no limit".
58	
59	- /sys/kernel/debug/fail*/space:
60	
61		specifies an initial resource "budget", decremented by "size"
62		on each call to should_fail(,size).  Failure injection is
63		suppressed until "space" reaches zero.
64	
65	- /sys/kernel/debug/fail*/verbose
66	
67		Format: { 0 | 1 | 2 }
68		specifies the verbosity of the messages when failure is
69		injected.  '0' means no messages; '1' will print only a single
70		log line per failure; '2' will print a call trace too -- useful
71		to debug the problems revealed by fault injection.
72	
73	- /sys/kernel/debug/fail*/task-filter:
74	
75		Format: { 'Y' | 'N' }
76		A value of 'N' disables filtering by process (default).
77		Any positive value limits failures to only processes indicated by
78		/proc/<pid>/make-it-fail==1.
79	
80	- /sys/kernel/debug/fail*/require-start:
81	- /sys/kernel/debug/fail*/require-end:
82	- /sys/kernel/debug/fail*/reject-start:
83	- /sys/kernel/debug/fail*/reject-end:
84	
85		specifies the range of virtual addresses tested during
86		stacktrace walking.  Failure is injected only if some caller
87		in the walked stacktrace lies within the required range, and
88		none lies within the rejected range.
89		Default required range is [0,ULONG_MAX) (whole of virtual address space).
90		Default rejected range is [0,0).
91	
92	- /sys/kernel/debug/fail*/stacktrace-depth:
93	
94		specifies the maximum stacktrace depth walked during search
95		for a caller within [require-start,require-end) OR
96		[reject-start,reject-end).
97	
98	- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:
99	
100		Format: { 'Y' | 'N' }
101		default is 'N', setting it to 'Y' won't inject failures into
102		highmem/user allocations.
103	
104	- /sys/kernel/debug/failslab/ignore-gfp-wait:
105	- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:
106	
107		Format: { 'Y' | 'N' }
108		default is 'N', setting it to 'Y' will inject failures
109		only into non-sleep allocations (GFP_ATOMIC allocations).
110	
111	- /sys/kernel/debug/fail_page_alloc/min-order:
112	
113		specifies the minimum page allocation order to be injected
114		failures.
115	
116	o Boot option
117	
118	In order to inject faults while debugfs is not available (early boot time),
119	use the boot option:
120	
121		failslab=
122		fail_page_alloc=
123		fail_make_request=
124		mmc_core.fail_request=<interval>,<probability>,<space>,<times>
125	
126	How to add new fault injection capability
127	-----------------------------------------
128	
129	o #include <linux/fault-inject.h>
130	
131	o define the fault attributes
132	
133	  DECLARE_FAULT_INJECTION(name);
134	
135	  Please see the definition of struct fault_attr in fault-inject.h
136	  for details.
137	
138	o provide a way to configure fault attributes
139	
140	- boot option
141	
142	  If you need to enable the fault injection capability from boot time, you can
143	  provide boot option to configure it. There is a helper function for it:
144	
145		setup_fault_attr(attr, str);
146	
147	- debugfs entries
148	
149	  failslab, fail_page_alloc, and fail_make_request use this way.
150	  Helper functions:
151	
152		fault_create_debugfs_attr(name, parent, attr);
153	
154	- module parameters
155	
156	  If the scope of the fault injection capability is limited to a
157	  single kernel module, it is better to provide module parameters to
158	  configure the fault attributes.
159	
160	o add a hook to insert failures
161	
162	  Upon should_fail() returning true, client code should inject a failure.
163	
164		should_fail(attr, size);
165	
166	Application Examples
167	--------------------
168	
169	o Inject slab allocation failures into module init/exit code
170	
171	#!/bin/bash
172	
173	FAILTYPE=failslab
174	echo Y > /sys/kernel/debug/$FAILTYPE/task-filter
175	echo 10 > /sys/kernel/debug/$FAILTYPE/probability
176	echo 100 > /sys/kernel/debug/$FAILTYPE/interval
177	echo -1 > /sys/kernel/debug/$FAILTYPE/times
178	echo 0 > /sys/kernel/debug/$FAILTYPE/space
179	echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
180	echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
181	
182	faulty_system()
183	{
184		bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
185	}
186	
187	if [ $# -eq 0 ]
188	then
189		echo "Usage: $0 modulename [ modulename ... ]"
190		exit 1
191	fi
192	
193	for m in $*
194	do
195		echo inserting $m...
196		faulty_system modprobe $m
197	
198		echo removing $m...
199		faulty_system modprobe -r $m
200	done
201	
202	------------------------------------------------------------------------------
203	
204	o Inject page allocation failures only for a specific module
205	
206	#!/bin/bash
207	
208	FAILTYPE=fail_page_alloc
209	module=$1
210	
211	if [ -z $module ]
212	then
213		echo "Usage: $0 <modulename>"
214		exit 1
215	fi
216	
217	modprobe $module
218	
219	if [ ! -d /sys/module/$module/sections ]
220	then
221		echo Module $module is not loaded
222		exit 1
223	fi
224	
225	cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start
226	cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end
227	
228	echo N > /sys/kernel/debug/$FAILTYPE/task-filter
229	echo 10 > /sys/kernel/debug/$FAILTYPE/probability
230	echo 100 > /sys/kernel/debug/$FAILTYPE/interval
231	echo -1 > /sys/kernel/debug/$FAILTYPE/times
232	echo 0 > /sys/kernel/debug/$FAILTYPE/space
233	echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
234	echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
235	echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem
236	echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth
237	
238	trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT
239	
240	echo "Injecting errors into the module $module... (interrupt to stop)"
241	sleep 1000000
242	
243	Tool to run command with failslab or fail_page_alloc
244	----------------------------------------------------
245	In order to make it easier to accomplish the tasks mentioned above, we can use
246	tools/testing/fault-injection/failcmd.sh.  Please run a command
247	"./tools/testing/fault-injection/failcmd.sh --help" for more information and
248	see the following examples.
249	
250	Examples:
251	
252	Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab
253	allocation failure.
254	
255		# ./tools/testing/fault-injection/failcmd.sh \
256			-- make -C tools/testing/selftests/ run_tests
257	
258	Same as above except to specify 100 times failures at most instead of one time
259	at most by default.
260	
261		# ./tools/testing/fault-injection/failcmd.sh --times=100 \
262			-- make -C tools/testing/selftests/ run_tests
263	
264	Same as above except to inject page allocation failure instead of slab
265	allocation failure.
266	
267		# env FAILCMD_TYPE=fail_page_alloc \
268			./tools/testing/fault-injection/failcmd.sh --times=100 \
269	                -- make -C tools/testing/selftests/ run_tests
Hide Line Numbers
About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Information is copyright its respective author. All material is available from the Linux Kernel Source distributed under a GPL License. This page is provided as a free service by mjmwired.net.