Based on kernel version 2.6.25. Page generated on 2008-04-18 21:22 EST.
1 2 3 EDAC - Error Detection And Correction 4 5 Written by Doug Thompson <dougthompson[AT]xmission[DOT]com> 6 7 Dec 2005 7 17 Jul 2007 Updated 8 9 10 EDAC is maintained and written by: 11 12 Doug Thompson, Dave Jiang, Dave Peterson et al, 13 original author: Thayne Harbaugh, 14 15 Contact: 16 website: bluesmoke.sourceforge.net 17 mailing list: bluesmoke-devel[AT]lists.sourceforge[DOT]net 18 19 "bluesmoke" was the name for this device driver when it was "out-of-tree" 20 and maintained at sourceforge.net. When it was pushed into 2.6.16 for the 21 first time, it was renamed to 'EDAC'. 22 23 The bluesmoke project at sourceforge.net is now utilized as a 'staging area' 24 for EDAC development, before it is sent upstream to kernel.org 25 26 At the bluesmoke/EDAC project site, is a series of quilt patches against 27 recent kernels, stored in a SVN respository. For easier downloading, there 28 is also a tarball snapshot available. 29 30 ============================================================================ 31 EDAC PURPOSE 32 33 The 'edac' kernel module goal is to detect and report errors that occur 34 within the computer system running under linux. 35 36 MEMORY 37 38 In the initial release, memory Correctable Errors (CE) and Uncorrectable 39 Errors (UE) are the primary errors being harvested. These types of errors 40 are harvested by the 'edac_mc' class of device. 41 42 Detecting CE events, then harvesting those events and reporting them, 43 CAN be a predictor of future UE events. With CE events, the system can 44 continue to operate, but with less safety. Preventive maintenance and 45 proactive part replacement of memory DIMMs exhibiting CEs can reduce 46 the likelihood of the dreaded UE events and system 'panics'. 47 48 NON-MEMORY 49 50 A new feature for EDAC, the edac_device class of device, was added in 51 the 2.6.23 version of the kernel. 52 53 This new device type allows for non-memory type of ECC hardware detectors 54 to have their states harvested and presented to userspace via the sysfs 55 interface. 56 57 Some architectures have ECC detectors for L1, L2 and L3 caches, along with DMA 58 engines, fabric switches, main data path switches, interconnections, 59 and various other hardware data paths. If the hardware reports it, then 60 a edac_device device probably can be constructed to harvest and present 61 that to userspace. 62 63 64 PCI BUS SCANNING 65 66 In addition, PCI Bus Parity and SERR Errors are scanned for on PCI devices 67 in order to determine if errors are occurring on data transfers. 68 69 The presence of PCI Parity errors must be examined with a grain of salt. 70 There are several add-in adapters that do NOT follow the PCI specification 71 with regards to Parity generation and reporting. The specification says 72 the vendor should tie the parity status bits to 0 if they do not intend 73 to generate parity. Some vendors do not do this, and thus the parity bit 74 can "float" giving false positives. 75 76 In the kernel there is a pci device attribute located in sysfs that is 77 checked by the EDAC PCI scanning code. If that attribute is set, 78 PCI parity/error scannining is skipped for that device. The attribute 79 is: 80 81 broken_parity_status 82 83 as is located in /sys/devices/pci<XXX>/0000:XX:YY.Z directorys for 84 PCI devices. 85 86 FUTURE HARDWARE SCANNING 87 88 EDAC will have future error detectors that will be integrated with 89 EDAC or added to it, in the following list: 90 91 MCE Machine Check Exception 92 MCA Machine Check Architecture 93 NMI NMI notification of ECC errors 94 MSRs Machine Specific Register error cases 95 and other mechanisms. 96 97 These errors are usually bus errors, ECC errors, thermal throttling 98 and the like. 99 100 101 ============================================================================ 102 EDAC VERSIONING 103 104 EDAC is composed of a "core" module (edac_core.ko) and several Memory 105 Controller (MC) driver modules. On a given system, the CORE 106 is loaded and one MC driver will be loaded. Both the CORE and 107 the MC driver (or edac_device driver) have individual versions that reflect 108 current release level of their respective modules. 109 110 Thus, to "report" on what version a system is running, one must report both 111 the CORE's and the MC driver's versions. 112 113 114 LOADING 115 116 If 'edac' was statically linked with the kernel then no loading is 117 necessary. If 'edac' was built as modules then simply modprobe the 118 'edac' pieces that you need. You should be able to modprobe 119 hardware-specific modules and have the dependencies load the necessary core 120 modules. 121 122 Example: 123 124 $> modprobe amd76x_edac 125 126 loads both the amd76x_edac.ko memory controller module and the edac_mc.ko 127 core module. 128 129 130 ============================================================================ 131 EDAC sysfs INTERFACE 132 133 EDAC presents a 'sysfs' interface for control, reporting and attribute 134 reporting purposes. 135 136 EDAC lives in the /sys/devices/system/edac directory. 137 138 Within this directory there currently reside 2 'edac' components: 139 140 mc memory controller(s) system 141 pci PCI control and status system 142 143 144 ============================================================================ 145 Memory Controller (mc) Model 146 147 First a background on the memory controller's model abstracted in EDAC. 148 Each 'mc' device controls a set of DIMM memory modules. These modules are 149 laid out in a Chip-Select Row (csrowX) and Channel table (chX). There can 150 be multiple csrows and multiple channels. 151 152 Memory controllers allow for several csrows, with 8 csrows being a typical value. 153 Yet, the actual number of csrows depends on the electrical "loading" 154 of a given motherboard, memory controller and DIMM characteristics. 155 156 Dual channels allows for 128 bit data transfers to the CPU from memory. 157 Some newer chipsets allow for more than 2 channels, like Fully Buffered DIMMs 158 (FB-DIMMs). The following example will assume 2 channels: 159 160 161 Channel 0 Channel 1 162 =================================== 163 csrow0 | DIMM_A0 | DIMM_B0 | 164 csrow1 | DIMM_A0 | DIMM_B0 | 165 =================================== 166 167 =================================== 168 csrow2 | DIMM_A1 | DIMM_B1 | 169 csrow3 | DIMM_A1 | DIMM_B1 | 170 =================================== 171 172 In the above example table there are 4 physical slots on the motherboard 173 for memory DIMMs: 174 175 DIMM_A0 176 DIMM_B0 177 DIMM_A1 178 DIMM_B1 179 180 Labels for these slots are usually silk screened on the motherboard. Slots 181 labeled 'A' are channel 0 in this example. Slots labeled 'B' 182 are channel 1. Notice that there are two csrows possible on a 183 physical DIMM. These csrows are allocated their csrow assignment 184 based on the slot into which the memory DIMM is placed. Thus, when 1 DIMM 185 is placed in each Channel, the csrows cross both DIMMs. 186 187 Memory DIMMs come single or dual "ranked". A rank is a populated csrow. 188 Thus, 2 single ranked DIMMs, placed in slots DIMM_A0 and DIMM_B0 above 189 will have 1 csrow, csrow0. csrow1 will be empty. On the other hand, 190 when 2 dual ranked DIMMs are similarly placed, then both csrow0 and 191 csrow1 will be populated. The pattern repeats itself for csrow2 and 192 csrow3. 193 194 The representation of the above is reflected in the directory tree 195 in EDAC's sysfs interface. Starting in directory 196 /sys/devices/system/edac/mc each memory controller will be represented 197 by its own 'mcX' directory, where 'X" is the index of the MC. 198 199 200 ..../edac/mc/ 201 | 202 |->mc0 203 |->mc1 204 |->mc2 205 .... 206 207 Under each 'mcX' directory each 'csrowX' is again represented by a 208 'csrowX', where 'X" is the csrow index: 209 210 211 .../mc/mc0/ 212 | 213 |->csrow0 214 |->csrow2 215 |->csrow3 216 .... 217 218 Notice that there is no csrow1, which indicates that csrow0 is 219 composed of a single ranked DIMMs. This should also apply in both 220 Channels, in order to have dual-channel mode be operational. Since 221 both csrow2 and csrow3 are populated, this indicates a dual ranked 222 set of DIMMs for channels 0 and 1. 223 224 225 Within each of the 'mc','mcX' and 'csrowX' directories are several 226 EDAC control and attribute files. 227 228 229 ============================================================================ 230 DIRECTORY 'mc' 231 232 In directory 'mc' are EDAC system overall control and attribute files: 233 234 235 Panic on UE control file: 236 237 'edac_mc_panic_on_ue' 238 239 An uncorrectable error will cause a machine panic. This is usually 240 desirable. It is a bad idea to continue when an uncorrectable error 241 occurs - it is indeterminate what was uncorrected and the operating 242 system context might be so mangled that continuing will lead to further 243 corruption. If the kernel has MCE configured, then EDAC will never 244 notice the UE. 245 246 LOAD TIME: module/kernel parameter: panic_on_ue=[0|1] 247 248 RUN TIME: echo "1" >/sys/devices/system/edac/mc/edac_mc_panic_on_ue 249 250 251 Log UE control file: 252 253 'edac_mc_log_ue' 254 255 Generate kernel messages describing uncorrectable errors. These errors 256 are reported through the system message log system. UE statistics 257 will be accumulated even when UE logging is disabled. 258 259 LOAD TIME: module/kernel parameter: log_ue=[0|1] 260 261 RUN TIME: echo "1" >/sys/devices/system/edac/mc/edac_mc_log_ue 262 263 264 Log CE control file: 265 266 'edac_mc_log_ce' 267 268 Generate kernel messages describing correctable errors. These 269 errors are reported through the system message log system. 270 CE statistics will be accumulated even when CE logging is disabled. 271 272 LOAD TIME: module/kernel parameter: log_ce=[0|1] 273 274 RUN TIME: echo "1" >/sys/devices/system/edac/mc/edac_mc_log_ce 275 276 277 Polling period control file: 278 279 'edac_mc_poll_msec' 280 281 The time period, in milliseconds, for polling for error information. 282 Too small a value wastes resources. Too large a value might delay 283 necessary handling of errors and might loose valuable information for 284 locating the error. 1000 milliseconds (once each second) is the current 285 default. Systems which require all the bandwidth they can get, may 286 increase this. 287 288 LOAD TIME: module/kernel parameter: poll_msec=[0|1] 289 290 RUN TIME: echo "1000" >/sys/devices/system/edac/mc/edac_mc_poll_msec 291 292 293 ============================================================================ 294 'mcX' DIRECTORIES 295 296 297 In 'mcX' directories are EDAC control and attribute files for 298 this 'X" instance of the memory controllers: 299 300 301 Counter reset control file: 302 303 'reset_counters' 304 305 This write-only control file will zero all the statistical counters 306 for UE and CE errors. Zeroing the counters will also reset the timer 307 indicating how long since the last counter zero. This is useful 308 for computing errors/time. Since the counters are always reset at 309 driver initialization time, no module/kernel parameter is available. 310 311 RUN TIME: echo "anything" >/sys/devices/system/edac/mc/mc0/counter_reset 312 313 This resets the counters on memory controller 0 314 315 316 Seconds since last counter reset control file: 317 318 'seconds_since_reset' 319 320 This attribute file displays how many seconds have elapsed since the 321 last counter reset. This can be used with the error counters to 322 measure error rates. 323 324 325 326 Memory Controller name attribute file: 327 328 'mc_name' 329 330 This attribute file displays the type of memory controller 331 that is being utilized. 332 333 334 Total memory managed by this memory controller attribute file: 335 336 'size_mb' 337 338 This attribute file displays, in count of megabytes, of memory 339 that this instance of memory controller manages. 340 341 342 Total Uncorrectable Errors count attribute file: 343 344 'ue_count' 345 346 This attribute file displays the total count of uncorrectable 347 errors that have occurred on this memory controller. If panic_on_ue 348 is set this counter will not have a chance to increment, 349 since EDAC will panic the system. 350 351 352 Total UE count that had no information attribute fileY: 353 354 'ue_noinfo_count' 355 356 This attribute file displays the number of UEs that 357 have occurred have occurred with no informations as to which DIMM 358 slot is having errors. 359 360 361 Total Correctable Errors count attribute file: 362 363 'ce_count' 364 365 This attribute file displays the total count of correctable 366 errors that have occurred on this memory controller. This 367 count is very important to examine. CEs provide early 368 indications that a DIMM is beginning to fail. This count 369 field should be monitored for non-zero values and report 370 such information to the system administrator. 371 372 373 Total Correctable Errors count attribute file: 374 375 'ce_noinfo_count' 376 377 This attribute file displays the number of CEs that 378 have occurred wherewith no informations as to which DIMM slot 379 is having errors. Memory is handicapped, but operational, 380 yet no information is available to indicate which slot 381 the failing memory is in. This count field should be also 382 be monitored for non-zero values. 383 384 Device Symlink: 385 386 'device' 387 388 Symlink to the memory controller device. 389 390 Sdram memory scrubbing rate: 391 392 'sdram_scrub_rate' 393 394 Read/Write attribute file that controls memory scrubbing. The scrubbing 395 rate is set by writing a minimum bandwith in bytes/sec to the attribute 396 file. The rate will be translated to an internal value that gives at 397 least the specified rate. 398 399 Reading the file will return the actual scrubbing rate employed. 400 401 If configuration fails or memory scrubbing is not implemented, the value 402 of the attribute file will be -1. 403 404 405 406 ============================================================================ 407 'csrowX' DIRECTORIES 408 409 In the 'csrowX' directories are EDAC control and attribute files for 410 this 'X" instance of csrow: 411 412 413 Total Uncorrectable Errors count attribute file: 414 415 'ue_count' 416 417 This attribute file displays the total count of uncorrectable 418 errors that have occurred on this csrow. If panic_on_ue is set 419 this counter will not have a chance to increment, since EDAC 420 will panic the system. 421 422 423 Total Correctable Errors count attribute file: 424 425 'ce_count' 426 427 This attribute file displays the total count of correctable 428 errors that have occurred on this csrow. This 429 count is very important to examine. CEs provide early 430 indications that a DIMM is beginning to fail. This count 431 field should be monitored for non-zero values and report 432 such information to the system administrator. 433 434 435 Total memory managed by this csrow attribute file: 436 437 'size_mb' 438 439 This attribute file displays, in count of megabytes, of memory 440 that this csrow contains. 441 442 443 Memory Type attribute file: 444 445 'mem_type' 446 447 This attribute file will display what type of memory is currently 448 on this csrow. Normally, either buffered or unbuffered memory. 449 Examples: 450 Registered-DDR 451 Unbuffered-DDR 452 453 454 EDAC Mode of operation attribute file: 455 456 'edac_mode' 457 458 This attribute file will display what type of Error detection 459 and correction is being utilized. 460 461 462 Device type attribute file: 463 464 'dev_type' 465 466 This attribute file will display what type of DRAM device is 467 being utilized on this DIMM. 468 Examples: 469 x1 470 x2 471 x4 472 x8 473 474 475 Channel 0 CE Count attribute file: 476 477 'ch0_ce_count' 478 479 This attribute file will display the count of CEs on this 480 DIMM located in channel 0. 481 482 483 Channel 0 UE Count attribute file: 484 485 'ch0_ue_count' 486 487 This attribute file will display the count of UEs on this 488 DIMM located in channel 0. 489 490 491 Channel 0 DIMM Label control file: 492 493 'ch0_dimm_label' 494 495 This control file allows this DIMM to have a label assigned 496 to it. With this label in the module, when errors occur 497 the output can provide the DIMM label in the system log. 498 This becomes vital for panic events to isolate the 499 cause of the UE event. 500 501 DIMM Labels must be assigned after booting, with information 502 that correctly identifies the physical slot with its 503 silk screen label. This information is currently very 504 motherboard specific and determination of this information 505 must occur in userland at this time. 506 507 508 Channel 1 CE Count attribute file: 509 510 'ch1_ce_count' 511 512 This attribute file will display the count of CEs on this 513 DIMM located in channel 1. 514 515 516 Channel 1 UE Count attribute file: 517 518 'ch1_ue_count' 519 520 This attribute file will display the count of UEs on this 521 DIMM located in channel 0. 522 523 524 Channel 1 DIMM Label control file: 525 526 'ch1_dimm_label' 527 528 This control file allows this DIMM to have a label assigned 529 to it. With this label in the module, when errors occur 530 the output can provide the DIMM label in the system log. 531 This becomes vital for panic events to isolate the 532 cause of the UE event. 533 534 DIMM Labels must be assigned after booting, with information 535 that correctly identifies the physical slot with its 536 silk screen label. This information is currently very 537 motherboard specific and determination of this information 538 must occur in userland at this time. 539 540 541 ============================================================================ 542 SYSTEM LOGGING 543 544 If logging for UEs and CEs are enabled then system logs will have 545 error notices indicating errors that have been detected: 546 547 EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0, 548 channel 1 "DIMM_B1": amd76x_edac 549 550 EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0, 551 channel 1 "DIMM_B1": amd76x_edac 552 553 554 The structure of the message is: 555 the memory controller (MC0) 556 Error type (CE) 557 memory page (0x283) 558 offset in the page (0xce0) 559 the byte granularity (grain 8) 560 or resolution of the error 561 the error syndrome (0xb741) 562 memory row (row 0) 563 memory channel (channel 1) 564 DIMM label, if set prior (DIMM B1 565 and then an optional, driver-specific message that may 566 have additional information. 567 568 Both UEs and CEs with no info will lack all but memory controller, 569 error type, a notice of "no info" and then an optional, 570 driver-specific error message. 571 572 573 574 ============================================================================ 575 PCI Bus Parity Detection 576 577 578 On Header Type 00 devices the primary status is looked at 579 for any parity error regardless of whether Parity is enabled on the 580 device. (The spec indicates parity is generated in some cases). 581 On Header Type 01 bridges, the secondary status register is also 582 looked at to see if parity occurred on the bus on the other side of 583 the bridge. 584 585 586 SYSFS CONFIGURATION 587 588 Under /sys/devices/system/edac/pci are control and attribute files as follows: 589 590 591 Enable/Disable PCI Parity checking control file: 592 593 'check_pci_parity' 594 595 596 This control file enables or disables the PCI Bus Parity scanning 597 operation. Writing a 1 to this file enables the scanning. Writing 598 a 0 to this file disables the scanning. 599 600 Enable: 601 echo "1" >/sys/devices/system/edac/pci/check_pci_parity 602 603 Disable: 604 echo "0" >/sys/devices/system/edac/pci/check_pci_parity 605 606 607 608 Panic on PCI PARITY Error: 609 610 'panic_on_pci_parity' 611 612 613 This control files enables or disables panicking when a parity 614 error has been detected. 615 616 617 module/kernel parameter: panic_on_pci_parity=[0|1] 618 619 Enable: 620 echo "1" >/sys/devices/system/edac/pci/panic_on_pci_parity 621 622 Disable: 623 echo "0" >/sys/devices/system/edac/pci/panic_on_pci_parity 624 625 626 Parity Count: 627 628 'pci_parity_count' 629 630 This attribute file will display the number of parity errors that 631 have been detected. 632 633 634 635 ======================================================================= 636 637 638 EDAC_DEVICE type of device 639 640 In the header file, edac_core.h, there is a series of edac_device structures 641 and APIs for the EDAC_DEVICE. 642 643 User space access to an edac_device is through the sysfs interface. 644 645 At the location /sys/devices/system/edac (sysfs) new edac_device devices will 646 appear. 647 648 There is a three level tree beneath the above 'edac' directory. For example, 649 the 'test_device_edac' device (found at the bluesmoke.sourceforget.net website) 650 installs itself as: 651 652 /sys/devices/systm/edac/test-instance 653 654 in this directory are various controls, a symlink and one or more 'instance' 655 directorys. 656 657 The standard default controls are: 658 659 log_ce boolean to log CE events 660 log_ue boolean to log UE events 661 panic_on_ue boolean to 'panic' the system if an UE is encountered 662 (default off, can be set true via startup script) 663 poll_msec time period between POLL cycles for events 664 665 The test_device_edac device adds at least one of its own custom control: 666 667 test_bits which in the current test driver does nothing but 668 show how it is installed. A ported driver can 669 add one or more such controls and/or attributes 670 for specific uses. 671 One out-of-tree driver uses controls here to allow 672 for ERROR INJECTION operations to hardware 673 injection registers 674 675 The symlink points to the 'struct dev' that is registered for this edac_device. 676 677 INSTANCES 678 679 One or more instance directories are present. For the 'test_device_edac' case: 680 681 test-instance0 682 683 684 In this directory there are two default counter attributes, which are totals of 685 counter in deeper subdirectories. 686 687 ce_count total of CE events of subdirectories 688 ue_count total of UE events of subdirectories 689 690 BLOCKS 691 692 At the lowest directory level is the 'block' directory. There can be 0, 1 693 or more blocks specified in each instance. 694 695 test-block0 696 697 698 In this directory the default attributes are: 699 700 ce_count which is counter of CE events for this 'block' 701 of hardware being monitored 702 ue_count which is counter of UE events for this 'block' 703 of hardware being monitored 704 705 706 The 'test_device_edac' device adds 4 attributes and 1 control: 707 708 test-block-bits-0 for every POLL cycle this counter 709 is incremented 710 test-block-bits-1 every 10 cycles, this counter is bumped once, 711 and test-block-bits-0 is set to 0 712 test-block-bits-2 every 100 cycles, this counter is bumped once, 713 and test-block-bits-1 is set to 0 714 test-block-bits-3 every 1000 cycles, this counter is bumped once, 715 and test-block-bits-2 is set to 0 716 717 718 reset-counters writing ANY thing to this control will 719 reset all the above counters. 720 721 722 Use of the 'test_device_edac' driver should any others to create their own 723 unique drivers for their hardware systems. 724 725 The 'test_device_edac' sample driver is located at the 726 bluesmoke.sourceforge.net project site for EDAC.
