Based on kernel version 2.6.26. Page generated on 2008-07-16 21:12 EST.
1 Fault injection capabilities infrastructure 2 =========================================== 3 4 See also drivers/md/faulty.c and "every_nth" module option for scsi_debug. 5 6 7 Available fault injection capabilities 8 -------------------------------------- 9 10 o failslab 11 12 injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...) 13 14 o fail_page_alloc 15 16 injects page allocation failures. (alloc_pages(), get_free_pages(), ...) 17 18 o fail_make_request 19 20 injects disk IO errors on devices permitted by setting 21 /sys/block/<device>/make-it-fail or 22 /sys/block/<device>/<partition>/make-it-fail. (generic_make_request()) 23 24 Configure fault-injection capabilities behavior 25 ----------------------------------------------- 26 27 o debugfs entries 28 29 fault-inject-debugfs kernel module provides some debugfs entries for runtime 30 configuration of fault-injection capabilities. 31 32 - /debug/fail*/probability: 33 34 likelihood of failure injection, in percent. 35 Format: <percent> 36 37 Note that one-failure-per-hundred is a very high error rate 38 for some testcases. Consider setting probability=100 and configure 39 /debug/fail*/interval for such testcases. 40 41 - /debug/fail*/interval: 42 43 specifies the interval between failures, for calls to 44 should_fail() that pass all the other tests. 45 46 Note that if you enable this, by setting interval>1, you will 47 probably want to set probability=100. 48 49 - /debug/fail*/times: 50 51 specifies how many times failures may happen at most. 52 A value of -1 means "no limit". 53 54 - /debug/fail*/space: 55 56 specifies an initial resource "budget", decremented by "size" 57 on each call to should_fail(,size). Failure injection is 58 suppressed until "space" reaches zero. 59 60 - /debug/fail*/verbose 61 62 Format: { 0 | 1 | 2 } 63 specifies the verbosity of the messages when failure is 64 injected. '0' means no messages; '1' will print only a single 65 log line per failure; '2' will print a call trace too -- useful 66 to debug the problems revealed by fault injection. 67 68 - /debug/fail*/task-filter: 69 70 Format: { 'Y' | 'N' } 71 A value of 'N' disables filtering by process (default). 72 Any positive value limits failures to only processes indicated by 73 /proc/<pid>/make-it-fail==1. 74 75 - /debug/fail*/require-start: 76 - /debug/fail*/require-end: 77 - /debug/fail*/reject-start: 78 - /debug/fail*/reject-end: 79 80 specifies the range of virtual addresses tested during 81 stacktrace walking. Failure is injected only if some caller 82 in the walked stacktrace lies within the required range, and 83 none lies within the rejected range. 84 Default required range is [0,ULONG_MAX) (whole of virtual address space). 85 Default rejected range is [0,0). 86 87 - /debug/fail*/stacktrace-depth: 88 89 specifies the maximum stacktrace depth walked during search 90 for a caller within [require-start,require-end) OR 91 [reject-start,reject-end). 92 93 - /debug/fail_page_alloc/ignore-gfp-highmem: 94 95 Format: { 'Y' | 'N' } 96 default is 'N', setting it to 'Y' won't inject failures into 97 highmem/user allocations. 98 99 - /debug/failslab/ignore-gfp-wait: 100 - /debug/fail_page_alloc/ignore-gfp-wait: 101 102 Format: { 'Y' | 'N' } 103 default is 'N', setting it to 'Y' will inject failures 104 only into non-sleep allocations (GFP_ATOMIC allocations). 105 106 - /debug/fail_page_alloc/min-order: 107 108 specifies the minimum page allocation order to be injected 109 failures. 110 111 o Boot option 112 113 In order to inject faults while debugfs is not available (early boot time), 114 use the boot option: 115 116 failslab= 117 fail_page_alloc= 118 fail_make_request=<interval>,<probability>,<space>,<times> 119 120 How to add new fault injection capability 121 ----------------------------------------- 122 123 o #include <linux/fault-inject.h> 124 125 o define the fault attributes 126 127 DECLARE_FAULT_INJECTION(name); 128 129 Please see the definition of struct fault_attr in fault-inject.h 130 for details. 131 132 o provide a way to configure fault attributes 133 134 - boot option 135 136 If you need to enable the fault injection capability from boot time, you can 137 provide boot option to configure it. There is a helper function for it: 138 139 setup_fault_attr(attr, str); 140 141 - debugfs entries 142 143 failslab, fail_page_alloc, and fail_make_request use this way. 144 Helper functions: 145 146 init_fault_attr_entries(entries, attr, name); 147 void cleanup_fault_attr_entries(entries); 148 149 - module parameters 150 151 If the scope of the fault injection capability is limited to a 152 single kernel module, it is better to provide module parameters to 153 configure the fault attributes. 154 155 o add a hook to insert failures 156 157 Upon should_fail() returning true, client code should inject a failure. 158 159 should_fail(attr, size); 160 161 Application Examples 162 -------------------- 163 164 o Inject slab allocation failures into module init/exit code 165 166 #!/bin/bash 167 168 FAILTYPE=failslab 169 echo Y > /debug/$FAILTYPE/task-filter 170 echo 10 > /debug/$FAILTYPE/probability 171 echo 100 > /debug/$FAILTYPE/interval 172 echo -1 > /debug/$FAILTYPE/times 173 echo 0 > /debug/$FAILTYPE/space 174 echo 2 > /debug/$FAILTYPE/verbose 175 echo 1 > /debug/$FAILTYPE/ignore-gfp-wait 176 177 faulty_system() 178 { 179 bash -c "echo 1 > /proc/self/make-it-fail && exec $*" 180 } 181 182 if [ $# -eq 0 ] 183 then 184 echo "Usage: $0 modulename [ modulename ... ]" 185 exit 1 186 fi 187 188 for m in $* 189 do 190 echo inserting $m... 191 faulty_system modprobe $m 192 193 echo removing $m... 194 faulty_system modprobe -r $m 195 done 196 197 ------------------------------------------------------------------------------ 198 199 o Inject page allocation failures only for a specific module 200 201 #!/bin/bash 202 203 FAILTYPE=fail_page_alloc 204 module=$1 205 206 if [ -z $module ] 207 then 208 echo "Usage: $0 <modulename>" 209 exit 1 210 fi 211 212 modprobe $module 213 214 if [ ! -d /sys/module/$module/sections ] 215 then 216 echo Module $module is not loaded 217 exit 1 218 fi 219 220 cat /sys/module/$module/sections/.text > /debug/$FAILTYPE/require-start 221 cat /sys/module/$module/sections/.data > /debug/$FAILTYPE/require-end 222 223 echo N > /debug/$FAILTYPE/task-filter 224 echo 10 > /debug/$FAILTYPE/probability 225 echo 100 > /debug/$FAILTYPE/interval 226 echo -1 > /debug/$FAILTYPE/times 227 echo 0 > /debug/$FAILTYPE/space 228 echo 2 > /debug/$FAILTYPE/verbose 229 echo 1 > /debug/$FAILTYPE/ignore-gfp-wait 230 echo 1 > /debug/$FAILTYPE/ignore-gfp-highmem 231 echo 10 > /debug/$FAILTYPE/stacktrace-depth 232 233 trap "echo 0 > /debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT 234 235 echo "Injecting errors into the module $module... (interrupt to stop)" 236 sleep 1000000