About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / fault-injection




Custom Search

Based on kernel version 4.7.2. Page generated on 2016-08-22 22:45 EST.

1	Fault injection capabilities infrastructure
2	===========================================
3	
4	See also drivers/md/faulty.c and "every_nth" module option for scsi_debug.
5	
6	
7	Available fault injection capabilities
8	--------------------------------------
9	
10	o failslab
11	
12	  injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
13	
14	o fail_page_alloc
15	
16	  injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
17	
18	o fail_futex
19	
20	  injects futex deadlock and uaddr fault errors.
21	
22	o fail_make_request
23	
24	  injects disk IO errors on devices permitted by setting
25	  /sys/block/<device>/make-it-fail or
26	  /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
27	
28	o fail_mmc_request
29	
30	  injects MMC data errors on devices permitted by setting
31	  debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
32	
33	Configure fault-injection capabilities behavior
34	-----------------------------------------------
35	
36	o debugfs entries
37	
38	fault-inject-debugfs kernel module provides some debugfs entries for runtime
39	configuration of fault-injection capabilities.
40	
41	- /sys/kernel/debug/fail*/probability:
42	
43		likelihood of failure injection, in percent.
44		Format: <percent>
45	
46		Note that one-failure-per-hundred is a very high error rate
47		for some testcases.  Consider setting probability=100 and configure
48		/sys/kernel/debug/fail*/interval for such testcases.
49	
50	- /sys/kernel/debug/fail*/interval:
51	
52		specifies the interval between failures, for calls to
53		should_fail() that pass all the other tests.
54	
55		Note that if you enable this, by setting interval>1, you will
56		probably want to set probability=100.
57	
58	- /sys/kernel/debug/fail*/times:
59	
60		specifies how many times failures may happen at most.
61		A value of -1 means "no limit".
62	
63	- /sys/kernel/debug/fail*/space:
64	
65		specifies an initial resource "budget", decremented by "size"
66		on each call to should_fail(,size).  Failure injection is
67		suppressed until "space" reaches zero.
68	
69	- /sys/kernel/debug/fail*/verbose
70	
71		Format: { 0 | 1 | 2 }
72		specifies the verbosity of the messages when failure is
73		injected.  '0' means no messages; '1' will print only a single
74		log line per failure; '2' will print a call trace too -- useful
75		to debug the problems revealed by fault injection.
76	
77	- /sys/kernel/debug/fail*/task-filter:
78	
79		Format: { 'Y' | 'N' }
80		A value of 'N' disables filtering by process (default).
81		Any positive value limits failures to only processes indicated by
82		/proc/<pid>/make-it-fail==1.
83	
84	- /sys/kernel/debug/fail*/require-start:
85	- /sys/kernel/debug/fail*/require-end:
86	- /sys/kernel/debug/fail*/reject-start:
87	- /sys/kernel/debug/fail*/reject-end:
88	
89		specifies the range of virtual addresses tested during
90		stacktrace walking.  Failure is injected only if some caller
91		in the walked stacktrace lies within the required range, and
92		none lies within the rejected range.
93		Default required range is [0,ULONG_MAX) (whole of virtual address space).
94		Default rejected range is [0,0).
95	
96	- /sys/kernel/debug/fail*/stacktrace-depth:
97	
98		specifies the maximum stacktrace depth walked during search
99		for a caller within [require-start,require-end) OR
100		[reject-start,reject-end).
101	
102	- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:
103	
104		Format: { 'Y' | 'N' }
105		default is 'N', setting it to 'Y' won't inject failures into
106		highmem/user allocations.
107	
108	- /sys/kernel/debug/failslab/ignore-gfp-wait:
109	- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:
110	
111		Format: { 'Y' | 'N' }
112		default is 'N', setting it to 'Y' will inject failures
113		only into non-sleep allocations (GFP_ATOMIC allocations).
114	
115	- /sys/kernel/debug/fail_page_alloc/min-order:
116	
117		specifies the minimum page allocation order to be injected
118		failures.
119	
120	- /sys/kernel/debug/fail_futex/ignore-private:
121	
122		Format: { 'Y' | 'N' }
123		default is 'N', setting it to 'Y' will disable failure injections
124		when dealing with private (address space) futexes.
125	
126	o Boot option
127	
128	In order to inject faults while debugfs is not available (early boot time),
129	use the boot option:
130	
131		failslab=
132		fail_page_alloc=
133		fail_make_request=
134		fail_futex=
135		mmc_core.fail_request=<interval>,<probability>,<space>,<times>
136	
137	How to add new fault injection capability
138	-----------------------------------------
139	
140	o #include <linux/fault-inject.h>
141	
142	o define the fault attributes
143	
144	  DECLARE_FAULT_INJECTION(name);
145	
146	  Please see the definition of struct fault_attr in fault-inject.h
147	  for details.
148	
149	o provide a way to configure fault attributes
150	
151	- boot option
152	
153	  If you need to enable the fault injection capability from boot time, you can
154	  provide boot option to configure it. There is a helper function for it:
155	
156		setup_fault_attr(attr, str);
157	
158	- debugfs entries
159	
160	  failslab, fail_page_alloc, and fail_make_request use this way.
161	  Helper functions:
162	
163		fault_create_debugfs_attr(name, parent, attr);
164	
165	- module parameters
166	
167	  If the scope of the fault injection capability is limited to a
168	  single kernel module, it is better to provide module parameters to
169	  configure the fault attributes.
170	
171	o add a hook to insert failures
172	
173	  Upon should_fail() returning true, client code should inject a failure.
174	
175		should_fail(attr, size);
176	
177	Application Examples
178	--------------------
179	
180	o Inject slab allocation failures into module init/exit code
181	
182	#!/bin/bash
183	
184	FAILTYPE=failslab
185	echo Y > /sys/kernel/debug/$FAILTYPE/task-filter
186	echo 10 > /sys/kernel/debug/$FAILTYPE/probability
187	echo 100 > /sys/kernel/debug/$FAILTYPE/interval
188	echo -1 > /sys/kernel/debug/$FAILTYPE/times
189	echo 0 > /sys/kernel/debug/$FAILTYPE/space
190	echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
191	echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
192	
193	faulty_system()
194	{
195		bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
196	}
197	
198	if [ $# -eq 0 ]
199	then
200		echo "Usage: $0 modulename [ modulename ... ]"
201		exit 1
202	fi
203	
204	for m in $*
205	do
206		echo inserting $m...
207		faulty_system modprobe $m
208	
209		echo removing $m...
210		faulty_system modprobe -r $m
211	done
212	
213	------------------------------------------------------------------------------
214	
215	o Inject page allocation failures only for a specific module
216	
217	#!/bin/bash
218	
219	FAILTYPE=fail_page_alloc
220	module=$1
221	
222	if [ -z $module ]
223	then
224		echo "Usage: $0 <modulename>"
225		exit 1
226	fi
227	
228	modprobe $module
229	
230	if [ ! -d /sys/module/$module/sections ]
231	then
232		echo Module $module is not loaded
233		exit 1
234	fi
235	
236	cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start
237	cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end
238	
239	echo N > /sys/kernel/debug/$FAILTYPE/task-filter
240	echo 10 > /sys/kernel/debug/$FAILTYPE/probability
241	echo 100 > /sys/kernel/debug/$FAILTYPE/interval
242	echo -1 > /sys/kernel/debug/$FAILTYPE/times
243	echo 0 > /sys/kernel/debug/$FAILTYPE/space
244	echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
245	echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
246	echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem
247	echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth
248	
249	trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT
250	
251	echo "Injecting errors into the module $module... (interrupt to stop)"
252	sleep 1000000
253	
254	Tool to run command with failslab or fail_page_alloc
255	----------------------------------------------------
256	In order to make it easier to accomplish the tasks mentioned above, we can use
257	tools/testing/fault-injection/failcmd.sh.  Please run a command
258	"./tools/testing/fault-injection/failcmd.sh --help" for more information and
259	see the following examples.
260	
261	Examples:
262	
263	Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab
264	allocation failure.
265	
266		# ./tools/testing/fault-injection/failcmd.sh \
267			-- make -C tools/testing/selftests/ run_tests
268	
269	Same as above except to specify 100 times failures at most instead of one time
270	at most by default.
271	
272		# ./tools/testing/fault-injection/failcmd.sh --times=100 \
273			-- make -C tools/testing/selftests/ run_tests
274	
275	Same as above except to inject page allocation failure instead of slab
276	allocation failure.
277	
278		# env FAILCMD_TYPE=fail_page_alloc \
279			./tools/testing/fault-injection/failcmd.sh --times=100 \
280	                -- make -C tools/testing/selftests/ run_tests
Hide Line Numbers
About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Information is copyright its respective author. All material is available from the Linux Kernel Source distributed under a GPL License. This page is provided as a free service by mjmwired.net.