Based on kernel version 4.9. Page generated on 2016-12-21 14:37 EST.
1 2 SCSI EH 3 ====================================== 4 5 This document describes SCSI midlayer error handling infrastructure. 6 Please refer to Documentation/scsi/scsi_mid_low_api.txt for more 7 information regarding SCSI midlayer. 8 9 TABLE OF CONTENTS 10 11  How SCSI commands travel through the midlayer and to EH 12 [1-1] struct scsi_cmnd 13 [1-2] How do scmd's get completed? 14 [1-2-1] Completing a scmd w/ scsi_done 15 [1-2-2] Completing a scmd w/ timeout 16 [1-3] How EH takes over 17  How SCSI EH works 18 [2-1] EH through fine-grained callbacks 19 [2-1-1] Overview 20 [2-1-2] Flow of scmds through EH 21 [2-1-3] Flow of control 22 [2-2] EH through transportt->eh_strategy_handler() 23 [2-2-1] Pre transportt->eh_strategy_handler() SCSI midlayer conditions 24 [2-2-2] Post transportt->eh_strategy_handler() SCSI midlayer conditions 25 [2-2-3] Things to consider 26 27 28  How SCSI commands travel through the midlayer and to EH 29 30 [1-1] struct scsi_cmnd 31 32 Each SCSI command is represented with struct scsi_cmnd (== scmd). A 33 scmd has two list_head's to link itself into lists. The two are 34 scmd->list and scmd->eh_entry. The former is used for free list or 35 per-device allocated scmd list and not of much interest to this EH 36 discussion. The latter is used for completion and EH lists and unless 37 otherwise stated scmds are always linked using scmd->eh_entry in this 38 discussion. 39 40 41 [1-2] How do scmd's get completed? 42 43 Once LLDD gets hold of a scmd, either the LLDD will complete the 44 command by calling scsi_done callback passed from midlayer when 45 invoking hostt->queuecommand() or the block layer will time it out. 46 47 48 [1-2-1] Completing a scmd w/ scsi_done 49 50 For all non-EH commands, scsi_done() is the completion callback. It 51 just calls blk_complete_request() to delete the block layer timer and 52 raise SCSI_SOFTIRQ 53 54 SCSI_SOFTIRQ handler scsi_softirq calls scsi_decide_disposition() to 55 determine what to do with the command. scsi_decide_disposition() 56 looks at the scmd->result value and sense data to determine what to do 57 with the command. 58 59 - SUCCESS 60 scsi_finish_command() is invoked for the command. The 61 function does some maintenance chores and then calls 62 scsi_io_completion() to finish the I/O. 63 scsi_io_completion() then notifies the block layer on 64 the completed request by calling blk_end_request and 65 friends or figures out what to do with the remainder 66 of the data in case of an error. 67 68 - NEEDS_RETRY 69 - ADD_TO_MLQUEUE 70 scmd is requeued to blk queue. 71 72 - otherwise 73 scsi_eh_scmd_add(scmd, 0) is invoked for the command. See 74 [1-3] for details of this function. 75 76 77 [1-2-2] Completing a scmd w/ timeout 78 79 The timeout handler is scsi_times_out(). When a timeout occurs, this 80 function 81 82 1. invokes optional hostt->eh_timed_out() callback. Return value can 83 be one of 84 85 - BLK_EH_HANDLED 86 This indicates that eh_timed_out() dealt with the timeout. 87 The command is passed back to the block layer and completed 88 via __blk_complete_requests(). 89 90 *NOTE* After returning BLK_EH_HANDLED the SCSI layer is 91 assumed to be finished with the command, and no other 92 functions from the SCSI layer will be called. So this 93 should typically only be returned if the eh_timed_out() 94 handler raced with normal completion. 95 96 - BLK_EH_RESET_TIMER 97 This indicates that more time is required to finish the 98 command. Timer is restarted. This action is counted as a 99 retry and only allowed scmd->allowed + 1(!) times. Once the 100 limit is reached, action for BLK_EH_NOT_HANDLED is taken instead. 101 102 - BLK_EH_NOT_HANDLED 103 eh_timed_out() callback did not handle the command. 104 Step #2 is taken. 105 106 2. If the host supports asynchronous completion (as indicated by the 107 no_async_abort setting in the host template) scsi_abort_command() 108 is invoked to schedule an asynchrous abort. If that fails 109 Step #3 is taken. 110 111 2. scsi_eh_scmd_add(scmd, SCSI_EH_CANCEL_CMD) is invoked for the 112 command. See [1-3] for more information. 113 114 [1-3] Asynchronous command aborts 115 116 After a timeout occurs a command abort is scheduled from 117 scsi_abort_command(). If the abort is successful the command 118 will either be retried (if the number of retries is not exhausted) 119 or terminated with DID_TIME_OUT. 120 Otherwise scsi_eh_scmd_add() is invoked for the command. 121 See [1-4] for more information. 122 123 [1-4] How EH takes over 124 125 scmds enter EH via scsi_eh_scmd_add(), which does the following. 126 127 1. Turns on scmd->eh_eflags as requested. It's 0 for error 128 completions and SCSI_EH_CANCEL_CMD for timeouts. 129 130 2. Links scmd->eh_entry to shost->eh_cmd_q 131 132 3. Sets SHOST_RECOVERY bit in shost->shost_state 133 134 4. Increments shost->host_failed 135 136 5. Wakes up SCSI EH thread if shost->host_busy == shost->host_failed 137 138 As can be seen above, once any scmd is added to shost->eh_cmd_q, 139 SHOST_RECOVERY shost_state bit is turned on. This prevents any new 140 scmd to be issued from blk queue to the host; eventually, all scmds on 141 the host either complete normally, fail and get added to eh_cmd_q, or 142 time out and get added to shost->eh_cmd_q. 143 144 If all scmds either complete or fail, the number of in-flight scmds 145 becomes equal to the number of failed scmds - i.e. shost->host_busy == 146 shost->host_failed. This wakes up SCSI EH thread. So, once woken up, 147 SCSI EH thread can expect that all in-flight commands have failed and 148 are linked on shost->eh_cmd_q. 149 150 Note that this does not mean lower layers are quiescent. If a LLDD 151 completed a scmd with error status, the LLDD and lower layers are 152 assumed to forget about the scmd at that point. However, if a scmd 153 has timed out, unless hostt->eh_timed_out() made lower layers forget 154 about the scmd, which currently no LLDD does, the command is still 155 active as long as lower layers are concerned and completion could 156 occur at any time. Of course, all such completions are ignored as the 157 timer has already expired. 158 159 We'll talk about how SCSI EH takes actions to abort - make LLDD 160 forget about - timed out scmds later. 161 162 163  How SCSI EH works 164 165 LLDD's can implement SCSI EH actions in one of the following two 166 ways. 167 168 - Fine-grained EH callbacks 169 LLDD can implement fine-grained EH callbacks and let SCSI 170 midlayer drive error handling and call appropriate callbacks. 171 This will be discussed further in [2-1]. 172 173 - eh_strategy_handler() callback 174 This is one big callback which should perform whole error 175 handling. As such, it should do all chores the SCSI midlayer 176 performs during recovery. This will be discussed in [2-2]. 177 178 Once recovery is complete, SCSI EH resumes normal operation by 179 calling scsi_restart_operations(), which 180 181 1. Checks if door locking is needed and locks door. 182 183 2. Clears SHOST_RECOVERY shost_state bit 184 185 3. Wakes up waiters on shost->host_wait. This occurs if someone 186 calls scsi_block_when_processing_errors() on the host. 187 (*QUESTION* why is it needed? All operations will be blocked 188 anyway after it reaches blk queue.) 189 190 4. Kicks queues in all devices on the host in the asses 191 192 193 [2-1] EH through fine-grained callbacks 194 195 [2-1-1] Overview 196 197 If eh_strategy_handler() is not present, SCSI midlayer takes charge 198 of driving error handling. EH's goals are two - make LLDD, host and 199 device forget about timed out scmds and make them ready for new 200 commands. A scmd is said to be recovered if the scmd is forgotten by 201 lower layers and lower layers are ready to process or fail the scmd 202 again. 203 204 To achieve these goals, EH performs recovery actions with increasing 205 severity. Some actions are performed by issuing SCSI commands and 206 others are performed by invoking one of the following fine-grained 207 hostt EH callbacks. Callbacks may be omitted and omitted ones are 208 considered to fail always. 209 210 int (* eh_abort_handler)(struct scsi_cmnd *); 211 int (* eh_device_reset_handler)(struct scsi_cmnd *); 212 int (* eh_bus_reset_handler)(struct scsi_cmnd *); 213 int (* eh_host_reset_handler)(struct scsi_cmnd *); 214 215 Higher-severity actions are taken only when lower-severity actions 216 cannot recover some of failed scmds. Also, note that failure of the 217 highest-severity action means EH failure and results in offlining of 218 all unrecovered devices. 219 220 During recovery, the following rules are followed 221 222 - Recovery actions are performed on failed scmds on the to do list, 223 eh_work_q. If a recovery action succeeds for a scmd, recovered 224 scmds are removed from eh_work_q. 225 226 Note that single recovery action on a scmd can recover multiple 227 scmds. e.g. resetting a device recovers all failed scmds on the 228 device. 229 230 - Higher severity actions are taken iff eh_work_q is not empty after 231 lower severity actions are complete. 232 233 - EH reuses failed scmds to issue commands for recovery. For 234 timed-out scmds, SCSI EH ensures that LLDD forgets about a scmd 235 before reusing it for EH commands. 236 237 When a scmd is recovered, the scmd is moved from eh_work_q to EH 238 local eh_done_q using scsi_eh_finish_cmd(). After all scmds are 239 recovered (eh_work_q is empty), scsi_eh_flush_done_q() is invoked to 240 either retry or error-finish (notify upper layer of failure) recovered 241 scmds. 242 243 scmds are retried iff its sdev is still online (not offlined during 244 EH), REQ_FAILFAST is not set and ++scmd->retries is less than 245 scmd->allowed. 246 247 248 [2-1-2] Flow of scmds through EH 249 250 1. Error completion / time out 251 ACTION: scsi_eh_scmd_add() is invoked for scmd 252 - set scmd->eh_eflags 253 - add scmd to shost->eh_cmd_q 254 - set SHOST_RECOVERY 255 - shost->host_failed++ 256 LOCKING: shost->host_lock 257 258 2. EH starts 259 ACTION: move all scmds to EH's local eh_work_q. shost->eh_cmd_q 260 is cleared. 261 LOCKING: shost->host_lock (not strictly necessary, just for 262 consistency) 263 264 3. scmd recovered 265 ACTION: scsi_eh_finish_cmd() is invoked to EH-finish scmd 266 - clear scmd->eh_eflags 267 - scsi_setup_cmd_retry() 268 - move from local eh_work_q to local eh_done_q 269 LOCKING: none 270 CONCURRENCY: at most one thread per separate eh_work_q to 271 keep queue manipulation lockless 272 273 4. EH completes 274 ACTION: scsi_eh_flush_done_q() retries scmds or notifies upper 275 layer of failure. May be called concurrently but must have 276 a no more than one thread per separate eh_work_q to 277 manipulate the queue locklessly 278 - scmd is removed from eh_done_q and scmd->eh_entry is cleared 279 - if retry is necessary, scmd is requeued using 280 scsi_queue_insert() 281 - otherwise, scsi_finish_command() is invoked for scmd 282 - zero shost->host_failed 283 LOCKING: queue or finish function performs appropriate locking 284 285 286 [2-1-3] Flow of control 287 288 EH through fine-grained callbacks start from scsi_unjam_host(). 289 290 <<scsi_unjam_host>> 291 292 1. Lock shost->host_lock, splice_init shost->eh_cmd_q into local 293 eh_work_q and unlock host_lock. Note that shost->eh_cmd_q is 294 cleared by this action. 295 296 2. Invoke scsi_eh_get_sense. 297 298 <<scsi_eh_get_sense>> 299 300 This action is taken for each error-completed 301 (!SCSI_EH_CANCEL_CMD) commands without valid sense data. Most 302 SCSI transports/LLDDs automatically acquire sense data on 303 command failures (autosense). Autosense is recommended for 304 performance reasons and as sense information could get out of 305 sync between occurrence of CHECK CONDITION and this action. 306 307 Note that if autosense is not supported, scmd->sense_buffer 308 contains invalid sense data when error-completing the scmd 309 with scsi_done(). scsi_decide_disposition() always returns 310 FAILED in such cases thus invoking SCSI EH. When the scmd 311 reaches here, sense data is acquired and 312 scsi_decide_disposition() is called again. 313 314 1. Invoke scsi_request_sense() which issues REQUEST_SENSE 315 command. If fails, no action. Note that taking no action 316 causes higher-severity recovery to be taken for the scmd. 317 318 2. Invoke scsi_decide_disposition() on the scmd 319 320 - SUCCESS 321 scmd->retries is set to scmd->allowed preventing 322 scsi_eh_flush_done_q() from retrying the scmd and 323 scsi_eh_finish_cmd() is invoked. 324 325 - NEEDS_RETRY 326 scsi_eh_finish_cmd() invoked 327 328 - otherwise 329 No action. 330 331 3. If !list_empty(&eh_work_q), invoke scsi_eh_abort_cmds(). 332 333 <<scsi_eh_abort_cmds>> 334 335 This action is taken for each timed out command when 336 no_async_abort is enabled in the host template. 337 hostt->eh_abort_handler() is invoked for each scmd. The 338 handler returns SUCCESS if it has succeeded to make LLDD and 339 all related hardware forget about the scmd. 340 341 If a timedout scmd is successfully aborted and the sdev is 342 either offline or ready, scsi_eh_finish_cmd() is invoked for 343 the scmd. Otherwise, the scmd is left in eh_work_q for 344 higher-severity actions. 345 346 Note that both offline and ready status mean that the sdev is 347 ready to process new scmds, where processing also implies 348 immediate failing; thus, if a sdev is in one of the two 349 states, no further recovery action is needed. 350 351 Device readiness is tested using scsi_eh_tur() which issues 352 TEST_UNIT_READY command. Note that the scmd must have been 353 aborted successfully before reusing it for TEST_UNIT_READY. 354 355 4. If !list_empty(&eh_work_q), invoke scsi_eh_ready_devs() 356 357 <<scsi_eh_ready_devs>> 358 359 This function takes four increasingly more severe measures to 360 make failed sdevs ready for new commands. 361 362 1. Invoke scsi_eh_stu() 363 364 <<scsi_eh_stu>> 365 366 For each sdev which has failed scmds with valid sense data 367 of which scsi_check_sense()'s verdict is FAILED, 368 START_STOP_UNIT command is issued w/ start=1. Note that 369 as we explicitly choose error-completed scmds, it is known 370 that lower layers have forgotten about the scmd and we can 371 reuse it for STU. 372 373 If STU succeeds and the sdev is either offline or ready, 374 all failed scmds on the sdev are EH-finished with 375 scsi_eh_finish_cmd(). 376 377 *NOTE* If hostt->eh_abort_handler() isn't implemented or 378 failed, we may still have timed out scmds at this point 379 and STU doesn't make lower layers forget about those 380 scmds. Yet, this function EH-finish all scmds on the sdev 381 if STU succeeds leaving lower layers in an inconsistent 382 state. It seems that STU action should be taken only when 383 a sdev has no timed out scmd. 384 385 2. If !list_empty(&eh_work_q), invoke scsi_eh_bus_device_reset(). 386 387 <<scsi_eh_bus_device_reset>> 388 389 This action is very similar to scsi_eh_stu() except that, 390 instead of issuing STU, hostt->eh_device_reset_handler() 391 is used. Also, as we're not issuing SCSI commands and 392 resetting clears all scmds on the sdev, there is no need 393 to choose error-completed scmds. 394 395 3. If !list_empty(&eh_work_q), invoke scsi_eh_bus_reset() 396 397 <<scsi_eh_bus_reset>> 398 399 hostt->eh_bus_reset_handler() is invoked for each channel 400 with failed scmds. If bus reset succeeds, all failed 401 scmds on all ready or offline sdevs on the channel are 402 EH-finished. 403 404 4. If !list_empty(&eh_work_q), invoke scsi_eh_host_reset() 405 406 <<scsi_eh_host_reset>> 407 408 This is the last resort. hostt->eh_host_reset_handler() 409 is invoked. If host reset succeeds, all failed scmds on 410 all ready or offline sdevs on the host are EH-finished. 411 412 5. If !list_empty(&eh_work_q), invoke scsi_eh_offline_sdevs() 413 414 <<scsi_eh_offline_sdevs>> 415 416 Take all sdevs which still have unrecovered scmds offline 417 and EH-finish the scmds. 418 419 5. Invoke scsi_eh_flush_done_q(). 420 421 <<scsi_eh_flush_done_q>> 422 423 At this point all scmds are recovered (or given up) and 424 put on eh_done_q by scsi_eh_finish_cmd(). This function 425 flushes eh_done_q by either retrying or notifying upper 426 layer of failure of the scmds. 427 428 429 [2-2] EH through transportt->eh_strategy_handler() 430 431 transportt->eh_strategy_handler() is invoked in the place of 432 scsi_unjam_host() and it is responsible for whole recovery process. 433 On completion, the handler should have made lower layers forget about 434 all failed scmds and either ready for new commands or offline. Also, 435 it should perform SCSI EH maintenance chores to maintain integrity of 436 SCSI midlayer. IOW, of the steps described in [2-1-2], all steps 437 except for #1 must be implemented by eh_strategy_handler(). 438 439 440 [2-2-1] Pre transportt->eh_strategy_handler() SCSI midlayer conditions 441 442 The following conditions are true on entry to the handler. 443 444 - Each failed scmd's eh_flags field is set appropriately. 445 446 - Each failed scmd is linked on scmd->eh_cmd_q by scmd->eh_entry. 447 448 - SHOST_RECOVERY is set. 449 450 - shost->host_failed == shost->host_busy 451 452 453 [2-2-2] Post transportt->eh_strategy_handler() SCSI midlayer conditions 454 455 The following conditions must be true on exit from the handler. 456 457 - shost->host_failed is zero. 458 459 - Each scmd's eh_eflags field is cleared. 460 461 - Each scmd is in such a state that scsi_setup_cmd_retry() on the 462 scmd doesn't make any difference. 463 464 - shost->eh_cmd_q is cleared. 465 466 - Each scmd->eh_entry is cleared. 467 468 - Either scsi_queue_insert() or scsi_finish_command() is called on 469 each scmd. Note that the handler is free to use scmd->retries and 470 ->allowed to limit the number of retries. 471 472 473 [2-2-3] Things to consider 474 475 - Know that timed out scmds are still active on lower layers. Make 476 lower layers forget about them before doing anything else with 477 those scmds. 478 479 - For consistency, when accessing/modifying shost data structure, 480 grab shost->host_lock. 481 482 - On completion, each failed sdev must have forgotten about all 483 active scmds. 484 485 - On completion, each failed sdev must be ready for new commands or 486 offline. 487 488 489 -- 490 Tejun Heo 491 firstname.lastname@example.org 492 11th September 2005