Based on kernel version 4.16.1. Page generated on 2018-04-09 11:53 EST.
1 LIBNVDIMM: Non-Volatile Devices 2 libnvdimm - kernel / libndctl - userspace helper library 3 linux-nvdimm@lists.01.org 4 v13 5 6 7 Glossary 8 Overview 9 Supporting Documents 10 Git Trees 11 LIBNVDIMM PMEM and BLK 12 Why BLK? 13 PMEM vs BLK 14 BLK-REGIONs, PMEM-REGIONs, Atomic Sectors, and DAX 15 Example NVDIMM Platform 16 LIBNVDIMM Kernel Device Model and LIBNDCTL Userspace API 17 LIBNDCTL: Context 18 libndctl: instantiate a new library context example 19 LIBNVDIMM/LIBNDCTL: Bus 20 libnvdimm: control class device in /sys/class 21 libnvdimm: bus 22 libndctl: bus enumeration example 23 LIBNVDIMM/LIBNDCTL: DIMM (NMEM) 24 libnvdimm: DIMM (NMEM) 25 libndctl: DIMM enumeration example 26 LIBNVDIMM/LIBNDCTL: Region 27 libnvdimm: region 28 libndctl: region enumeration example 29 Why Not Encode the Region Type into the Region Name? 30 How Do I Determine the Major Type of a Region? 31 LIBNVDIMM/LIBNDCTL: Namespace 32 libnvdimm: namespace 33 libndctl: namespace enumeration example 34 libndctl: namespace creation example 35 Why the Term "namespace"? 36 LIBNVDIMM/LIBNDCTL: Block Translation Table "btt" 37 libnvdimm: btt layout 38 libndctl: btt creation example 39 Summary LIBNDCTL Diagram 40 41 42 Glossary 43 -------- 44 45 PMEM: A system-physical-address range where writes are persistent. A 46 block device composed of PMEM is capable of DAX. A PMEM address range 47 may span an interleave of several DIMMs. 48 49 BLK: A set of one or more programmable memory mapped apertures provided 50 by a DIMM to access its media. This indirection precludes the 51 performance benefit of interleaving, but enables DIMM-bounded failure 52 modes. 53 54 DPA: DIMM Physical Address, is a DIMM-relative offset. With one DIMM in 55 the system there would be a 1:1 system-physical-address:DPA association. 56 Once more DIMMs are added a memory controller interleave must be 57 decoded to determine the DPA associated with a given 58 system-physical-address. BLK capacity always has a 1:1 relationship 59 with a single-DIMM's DPA range. 60 61 DAX: File system extensions to bypass the page cache and block layer to 62 mmap persistent memory, from a PMEM block device, directly into a 63 process address space. 64 65 DSM: Device Specific Method: ACPI method to to control specific 66 device - in this case the firmware. 67 68 DCR: NVDIMM Control Region Structure defined in ACPI 6 Section 5.2.25.5. 69 It defines a vendor-id, device-id, and interface format for a given DIMM. 70 71 BTT: Block Translation Table: Persistent memory is byte addressable. 72 Existing software may have an expectation that the power-fail-atomicity 73 of writes is at least one sector, 512 bytes. The BTT is an indirection 74 table with atomic update semantics to front a PMEM/BLK block device 75 driver and present arbitrary atomic sector sizes. 76 77 LABEL: Metadata stored on a DIMM device that partitions and identifies 78 (persistently names) storage between PMEM and BLK. It also partitions 79 BLK storage to host BTTs with different parameters per BLK-partition. 80 Note that traditional partition tables, GPT/MBR, are layered on top of a 81 BLK or PMEM device. 82 83 84 Overview 85 -------- 86 87 The LIBNVDIMM subsystem provides support for three types of NVDIMMs, namely, 88 PMEM, BLK, and NVDIMM devices that can simultaneously support both PMEM 89 and BLK mode access. These three modes of operation are described by 90 the "NVDIMM Firmware Interface Table" (NFIT) in ACPI 6. While the LIBNVDIMM 91 implementation is generic and supports pre-NFIT platforms, it was guided 92 by the superset of capabilities need to support this ACPI 6 definition 93 for NVDIMM resources. The bulk of the kernel implementation is in place 94 to handle the case where DPA accessible via PMEM is aliased with DPA 95 accessible via BLK. When that occurs a LABEL is needed to reserve DPA 96 for exclusive access via one mode a time. 97 98 Supporting Documents 99 ACPI 6: http://www.uefi.org/sites/default/files/resources/ACPI_6.0.pdf 100 NVDIMM Namespace: http://pmem.io/documents/NVDIMM_Namespace_Spec.pdf 101 DSM Interface Example: http://pmem.io/documents/NVDIMM_DSM_Interface_Example.pdf 102 Driver Writer's Guide: http://pmem.io/documents/NVDIMM_Driver_Writers_Guide.pdf 103 104 Git Trees 105 LIBNVDIMM: https://git.kernel.org/cgit/linux/kernel/git/djbw/nvdimm.git 106 LIBNDCTL: https://github.com/pmem/ndctl.git 107 PMEM: https://github.com/01org/prd 108 109 110 LIBNVDIMM PMEM and BLK 111 ------------------ 112 113 Prior to the arrival of the NFIT, non-volatile memory was described to a 114 system in various ad-hoc ways. Usually only the bare minimum was 115 provided, namely, a single system-physical-address range where writes 116 are expected to be durable after a system power loss. Now, the NFIT 117 specification standardizes not only the description of PMEM, but also 118 BLK and platform message-passing entry points for control and 119 configuration. 120 121 For each NVDIMM access method (PMEM, BLK), LIBNVDIMM provides a block 122 device driver: 123 124 1. PMEM (nd_pmem.ko): Drives a system-physical-address range. This 125 range is contiguous in system memory and may be interleaved (hardware 126 memory controller striped) across multiple DIMMs. When interleaved the 127 platform may optionally provide details of which DIMMs are participating 128 in the interleave. 129 130 Note that while LIBNVDIMM describes system-physical-address ranges that may 131 alias with BLK access as ND_NAMESPACE_PMEM ranges and those without 132 alias as ND_NAMESPACE_IO ranges, to the nd_pmem driver there is no 133 distinction. The different device-types are an implementation detail 134 that userspace can exploit to implement policies like "only interface 135 with address ranges from certain DIMMs". It is worth noting that when 136 aliasing is present and a DIMM lacks a label, then no block device can 137 be created by default as userspace needs to do at least one allocation 138 of DPA to the PMEM range. In contrast ND_NAMESPACE_IO ranges, once 139 registered, can be immediately attached to nd_pmem. 140 141 2. BLK (nd_blk.ko): This driver performs I/O using a set of platform 142 defined apertures. A set of apertures will access just one DIMM. 143 Multiple windows (apertures) allow multiple concurrent accesses, much like 144 tagged-command-queuing, and would likely be used by different threads or 145 different CPUs. 146 147 The NFIT specification defines a standard format for a BLK-aperture, but 148 the spec also allows for vendor specific layouts, and non-NFIT BLK 149 implementations may have other designs for BLK I/O. For this reason 150 "nd_blk" calls back into platform-specific code to perform the I/O. 151 One such implementation is defined in the "Driver Writer's Guide" and "DSM 152 Interface Example". 153 154 155 Why BLK? 156 -------- 157 158 While PMEM provides direct byte-addressable CPU-load/store access to 159 NVDIMM storage, it does not provide the best system RAS (recovery, 160 availability, and serviceability) model. An access to a corrupted 161 system-physical-address address causes a CPU exception while an access 162 to a corrupted address through an BLK-aperture causes that block window 163 to raise an error status in a register. The latter is more aligned with 164 the standard error model that host-bus-adapter attached disks present. 165 Also, if an administrator ever wants to replace a memory it is easier to 166 service a system at DIMM module boundaries. Compare this to PMEM where 167 data could be interleaved in an opaque hardware specific manner across 168 several DIMMs. 169 170 PMEM vs BLK 171 BLK-apertures solve these RAS problems, but their presence is also the 172 major contributing factor to the complexity of the ND subsystem. They 173 complicate the implementation because PMEM and BLK alias in DPA space. 174 Any given DIMM's DPA-range may contribute to one or more 175 system-physical-address sets of interleaved DIMMs, *and* may also be 176 accessed in its entirety through its BLK-aperture. Accessing a DPA 177 through a system-physical-address while simultaneously accessing the 178 same DPA through a BLK-aperture has undefined results. For this reason, 179 DIMMs with this dual interface configuration include a DSM function to 180 store/retrieve a LABEL. The LABEL effectively partitions the DPA-space 181 into exclusive system-physical-address and BLK-aperture accessible 182 regions. For simplicity a DIMM is allowed a PMEM "region" per each 183 interleave set in which it is a member. The remaining DPA space can be 184 carved into an arbitrary number of BLK devices with discontiguous 185 extents. 186 187 BLK-REGIONs, PMEM-REGIONs, Atomic Sectors, and DAX 188 -------------------------------------------------- 189 190 One of the few 191 reasons to allow multiple BLK namespaces per REGION is so that each 192 BLK-namespace can be configured with a BTT with unique atomic sector 193 sizes. While a PMEM device can host a BTT the LABEL specification does 194 not provide for a sector size to be specified for a PMEM namespace. 195 This is due to the expectation that the primary usage model for PMEM is 196 via DAX, and the BTT is incompatible with DAX. However, for the cases 197 where an application or filesystem still needs atomic sector update 198 guarantees it can register a BTT on a PMEM device or partition. See 199 LIBNVDIMM/NDCTL: Block Translation Table "btt" 200 201 202 Example NVDIMM Platform 203 ----------------------- 204 205 For the remainder of this document the following diagram will be 206 referenced for any example sysfs layouts. 207 208 209 (a) (b) DIMM BLK-REGION 210 +-------------------+--------+--------+--------+ 211 +------+ | pm0.0 | blk2.0 | pm1.0 | blk2.1 | 0 region2 212 | imc0 +--+- - - region0- - - +--------+ +--------+ 213 +--+---+ | pm0.0 | blk3.0 | pm1.0 | blk3.1 | 1 region3 214 | +-------------------+--------v v--------+ 215 +--+---+ | | 216 | cpu0 | region1 217 +--+---+ | | 218 | +----------------------------^ ^--------+ 219 +--+---+ | blk4.0 | pm1.0 | blk4.0 | 2 region4 220 | imc1 +--+----------------------------| +--------+ 221 +------+ | blk5.0 | pm1.0 | blk5.0 | 3 region5 222 +----------------------------+--------+--------+ 223 224 In this platform we have four DIMMs and two memory controllers in one 225 socket. Each unique interface (BLK or PMEM) to DPA space is identified 226 by a region device with a dynamically assigned id (REGION0 - REGION5). 227 228 1. The first portion of DIMM0 and DIMM1 are interleaved as REGION0. A 229 single PMEM namespace is created in the REGION0-SPA-range that spans most 230 of DIMM0 and DIMM1 with a user-specified name of "pm0.0". Some of that 231 interleaved system-physical-address range is reclaimed as BLK-aperture 232 accessed space starting at DPA-offset (a) into each DIMM. In that 233 reclaimed space we create two BLK-aperture "namespaces" from REGION2 and 234 REGION3 where "blk2.0" and "blk3.0" are just human readable names that 235 could be set to any user-desired name in the LABEL. 236 237 2. In the last portion of DIMM0 and DIMM1 we have an interleaved 238 system-physical-address range, REGION1, that spans those two DIMMs as 239 well as DIMM2 and DIMM3. Some of REGION1 is allocated to a PMEM namespace 240 named "pm1.0", the rest is reclaimed in 4 BLK-aperture namespaces (for 241 each DIMM in the interleave set), "blk2.1", "blk3.1", "blk4.0", and 242 "blk5.0". 243 244 3. The portion of DIMM2 and DIMM3 that do not participate in the REGION1 245 interleaved system-physical-address range (i.e. the DPA address past 246 offset (b) are also included in the "blk4.0" and "blk5.0" namespaces. 247 Note, that this example shows that BLK-aperture namespaces don't need to 248 be contiguous in DPA-space. 249 250 This bus is provided by the kernel under the device 251 /sys/devices/platform/nfit_test.0 when CONFIG_NFIT_TEST is enabled and 252 the nfit_test.ko module is loaded. This not only test LIBNVDIMM but the 253 acpi_nfit.ko driver as well. 254 255 256 LIBNVDIMM Kernel Device Model and LIBNDCTL Userspace API 257 ---------------------------------------------------- 258 259 What follows is a description of the LIBNVDIMM sysfs layout and a 260 corresponding object hierarchy diagram as viewed through the LIBNDCTL 261 API. The example sysfs paths and diagrams are relative to the Example 262 NVDIMM Platform which is also the LIBNVDIMM bus used in the LIBNDCTL unit 263 test. 264 265 LIBNDCTL: Context 266 Every API call in the LIBNDCTL library requires a context that holds the 267 logging parameters and other library instance state. The library is 268 based on the libabc template: 269 https://git.kernel.org/cgit/linux/kernel/git/kay/libabc.git 270 271 LIBNDCTL: instantiate a new library context example 272 273 struct ndctl_ctx *ctx; 274 275 if (ndctl_new(&ctx) == 0) 276 return ctx; 277 else 278 return NULL; 279 280 LIBNVDIMM/LIBNDCTL: Bus 281 ------------------- 282 283 A bus has a 1:1 relationship with an NFIT. The current expectation for 284 ACPI based systems is that there is only ever one platform-global NFIT. 285 That said, it is trivial to register multiple NFITs, the specification 286 does not preclude it. The infrastructure supports multiple busses and 287 we we use this capability to test multiple NFIT configurations in the 288 unit test. 289 290 LIBNVDIMM: control class device in /sys/class 291 292 This character device accepts DSM messages to be passed to DIMM 293 identified by its NFIT handle. 294 295 /sys/class/nd/ndctl0 296 |-- dev 297 |-- device -> ../../../ndbus0 298 |-- subsystem -> ../../../../../../../class/nd 299 300 301 302 LIBNVDIMM: bus 303 304 struct nvdimm_bus *nvdimm_bus_register(struct device *parent, 305 struct nvdimm_bus_descriptor *nfit_desc); 306 307 /sys/devices/platform/nfit_test.0/ndbus0 308 |-- commands 309 |-- nd 310 |-- nfit 311 |-- nmem0 312 |-- nmem1 313 |-- nmem2 314 |-- nmem3 315 |-- power 316 |-- provider 317 |-- region0 318 |-- region1 319 |-- region2 320 |-- region3 321 |-- region4 322 |-- region5 323 |-- uevent 324 `-- wait_probe 325 326 LIBNDCTL: bus enumeration example 327 Find the bus handle that describes the bus from Example NVDIMM Platform 328 329 static struct ndctl_bus *get_bus_by_provider(struct ndctl_ctx *ctx, 330 const char *provider) 331 { 332 struct ndctl_bus *bus; 333 334 ndctl_bus_foreach(ctx, bus) 335 if (strcmp(provider, ndctl_bus_get_provider(bus)) == 0) 336 return bus; 337 338 return NULL; 339 } 340 341 bus = get_bus_by_provider(ctx, "nfit_test.0"); 342 343 344 LIBNVDIMM/LIBNDCTL: DIMM (NMEM) 345 --------------------------- 346 347 The DIMM device provides a character device for sending commands to 348 hardware, and it is a container for LABELs. If the DIMM is defined by 349 NFIT then an optional 'nfit' attribute sub-directory is available to add 350 NFIT-specifics. 351 352 Note that the kernel device name for "DIMMs" is "nmemX". The NFIT 353 describes these devices via "Memory Device to System Physical Address 354 Range Mapping Structure", and there is no requirement that they actually 355 be physical DIMMs, so we use a more generic name. 356 357 LIBNVDIMM: DIMM (NMEM) 358 359 struct nvdimm *nvdimm_create(struct nvdimm_bus *nvdimm_bus, void *provider_data, 360 const struct attribute_group **groups, unsigned long flags, 361 unsigned long *dsm_mask); 362 363 /sys/devices/platform/nfit_test.0/ndbus0 364 |-- nmem0 365 | |-- available_slots 366 | |-- commands 367 | |-- dev 368 | |-- devtype 369 | |-- driver -> ../../../../../bus/nd/drivers/nvdimm 370 | |-- modalias 371 | |-- nfit 372 | | |-- device 373 | | |-- format 374 | | |-- handle 375 | | |-- phys_id 376 | | |-- rev_id 377 | | |-- serial 378 | | `-- vendor 379 | |-- state 380 | |-- subsystem -> ../../../../../bus/nd 381 | `-- uevent 382 |-- nmem1 383 [..] 384 385 386 LIBNDCTL: DIMM enumeration example 387 388 Note, in this example we are assuming NFIT-defined DIMMs which are 389 identified by an "nfit_handle" a 32-bit value where: 390 Bit 3:0 DIMM number within the memory channel 391 Bit 7:4 memory channel number 392 Bit 11:8 memory controller ID 393 Bit 15:12 socket ID (within scope of a Node controller if node controller is present) 394 Bit 27:16 Node Controller ID 395 Bit 31:28 Reserved 396 397 static struct ndctl_dimm *get_dimm_by_handle(struct ndctl_bus *bus, 398 unsigned int handle) 399 { 400 struct ndctl_dimm *dimm; 401 402 ndctl_dimm_foreach(bus, dimm) 403 if (ndctl_dimm_get_handle(dimm) == handle) 404 return dimm; 405 406 return NULL; 407 } 408 409 #define DIMM_HANDLE(n, s, i, c, d) \ 410 (((n & 0xfff) << 16) | ((s & 0xf) << 12) | ((i & 0xf) << 8) \ 411 | ((c & 0xf) << 4) | (d & 0xf)) 412 413 dimm = get_dimm_by_handle(bus, DIMM_HANDLE(0, 0, 0, 0, 0)); 414 415 LIBNVDIMM/LIBNDCTL: Region 416 ---------------------- 417 418 A generic REGION device is registered for each PMEM range or BLK-aperture 419 set. Per the example there are 6 regions: 2 PMEM and 4 BLK-aperture 420 sets on the "nfit_test.0" bus. The primary role of regions are to be a 421 container of "mappings". A mapping is a tuple of <DIMM, 422 DPA-start-offset, length>. 423 424 LIBNVDIMM provides a built-in driver for these REGION devices. This driver 425 is responsible for reconciling the aliased DPA mappings across all 426 regions, parsing the LABEL, if present, and then emitting NAMESPACE 427 devices with the resolved/exclusive DPA-boundaries for the nd_pmem or 428 nd_blk device driver to consume. 429 430 In addition to the generic attributes of "mapping"s, "interleave_ways" 431 and "size" the REGION device also exports some convenience attributes. 432 "nstype" indicates the integer type of namespace-device this region 433 emits, "devtype" duplicates the DEVTYPE variable stored by udev at the 434 'add' event, "modalias" duplicates the MODALIAS variable stored by udev 435 at the 'add' event, and finally, the optional "spa_index" is provided in 436 the case where the region is defined by a SPA. 437 438 LIBNVDIMM: region 439 440 struct nd_region *nvdimm_pmem_region_create(struct nvdimm_bus *nvdimm_bus, 441 struct nd_region_desc *ndr_desc); 442 struct nd_region *nvdimm_blk_region_create(struct nvdimm_bus *nvdimm_bus, 443 struct nd_region_desc *ndr_desc); 444 445 /sys/devices/platform/nfit_test.0/ndbus0 446 |-- region0 447 | |-- available_size 448 | |-- btt0 449 | |-- btt_seed 450 | |-- devtype 451 | |-- driver -> ../../../../../bus/nd/drivers/nd_region 452 | |-- init_namespaces 453 | |-- mapping0 454 | |-- mapping1 455 | |-- mappings 456 | |-- modalias 457 | |-- namespace0.0 458 | |-- namespace_seed 459 | |-- numa_node 460 | |-- nfit 461 | | `-- spa_index 462 | |-- nstype 463 | |-- set_cookie 464 | |-- size 465 | |-- subsystem -> ../../../../../bus/nd 466 | `-- uevent 467 |-- region1 468 [..] 469 470 LIBNDCTL: region enumeration example 471 472 Sample region retrieval routines based on NFIT-unique data like 473 "spa_index" (interleave set id) for PMEM and "nfit_handle" (dimm id) for 474 BLK. 475 476 static struct ndctl_region *get_pmem_region_by_spa_index(struct ndctl_bus *bus, 477 unsigned int spa_index) 478 { 479 struct ndctl_region *region; 480 481 ndctl_region_foreach(bus, region) { 482 if (ndctl_region_get_type(region) != ND_DEVICE_REGION_PMEM) 483 continue; 484 if (ndctl_region_get_spa_index(region) == spa_index) 485 return region; 486 } 487 return NULL; 488 } 489 490 static struct ndctl_region *get_blk_region_by_dimm_handle(struct ndctl_bus *bus, 491 unsigned int handle) 492 { 493 struct ndctl_region *region; 494 495 ndctl_region_foreach(bus, region) { 496 struct ndctl_mapping *map; 497 498 if (ndctl_region_get_type(region) != ND_DEVICE_REGION_BLOCK) 499 continue; 500 ndctl_mapping_foreach(region, map) { 501 struct ndctl_dimm *dimm = ndctl_mapping_get_dimm(map); 502 503 if (ndctl_dimm_get_handle(dimm) == handle) 504 return region; 505 } 506 } 507 return NULL; 508 } 509 510 511 Why Not Encode the Region Type into the Region Name? 512 ---------------------------------------------------- 513 514 At first glance it seems since NFIT defines just PMEM and BLK interface 515 types that we should simply name REGION devices with something derived 516 from those type names. However, the ND subsystem explicitly keeps the 517 REGION name generic and expects userspace to always consider the 518 region-attributes for four reasons: 519 520 1. There are already more than two REGION and "namespace" types. For 521 PMEM there are two subtypes. As mentioned previously we have PMEM where 522 the constituent DIMM devices are known and anonymous PMEM. For BLK 523 regions the NFIT specification already anticipates vendor specific 524 implementations. The exact distinction of what a region contains is in 525 the region-attributes not the region-name or the region-devtype. 526 527 2. A region with zero child-namespaces is a possible configuration. For 528 example, the NFIT allows for a DCR to be published without a 529 corresponding BLK-aperture. This equates to a DIMM that can only accept 530 control/configuration messages, but no i/o through a descendant block 531 device. Again, this "type" is advertised in the attributes ('mappings' 532 == 0) and the name does not tell you much. 533 534 3. What if a third major interface type arises in the future? Outside 535 of vendor specific implementations, it's not difficult to envision a 536 third class of interface type beyond BLK and PMEM. With a generic name 537 for the REGION level of the device-hierarchy old userspace 538 implementations can still make sense of new kernel advertised 539 region-types. Userspace can always rely on the generic region 540 attributes like "mappings", "size", etc and the expected child devices 541 named "namespace". This generic format of the device-model hierarchy 542 allows the LIBNVDIMM and LIBNDCTL implementations to be more uniform and 543 future-proof. 544 545 4. There are more robust mechanisms for determining the major type of a 546 region than a device name. See the next section, How Do I Determine the 547 Major Type of a Region? 548 549 How Do I Determine the Major Type of a Region? 550 ---------------------------------------------- 551 552 Outside of the blanket recommendation of "use libndctl", or simply 553 looking at the kernel header (/usr/include/linux/ndctl.h) to decode the 554 "nstype" integer attribute, here are some other options. 555 556 1. module alias lookup: 557 558 The whole point of region/namespace device type differentiation is to 559 decide which block-device driver will attach to a given LIBNVDIMM namespace. 560 One can simply use the modalias to lookup the resulting module. It's 561 important to note that this method is robust in the presence of a 562 vendor-specific driver down the road. If a vendor-specific 563 implementation wants to supplant the standard nd_blk driver it can with 564 minimal impact to the rest of LIBNVDIMM. 565 566 In fact, a vendor may also want to have a vendor-specific region-driver 567 (outside of nd_region). For example, if a vendor defined its own LABEL 568 format it would need its own region driver to parse that LABEL and emit 569 the resulting namespaces. The output from module resolution is more 570 accurate than a region-name or region-devtype. 571 572 2. udev: 573 574 The kernel "devtype" is registered in the udev database 575 # udevadm info --path=/devices/platform/nfit_test.0/ndbus0/region0 576 P: /devices/platform/nfit_test.0/ndbus0/region0 577 E: DEVPATH=/devices/platform/nfit_test.0/ndbus0/region0 578 E: DEVTYPE=nd_pmem 579 E: MODALIAS=nd:t2 580 E: SUBSYSTEM=nd 581 582 # udevadm info --path=/devices/platform/nfit_test.0/ndbus0/region4 583 P: /devices/platform/nfit_test.0/ndbus0/region4 584 E: DEVPATH=/devices/platform/nfit_test.0/ndbus0/region4 585 E: DEVTYPE=nd_blk 586 E: MODALIAS=nd:t3 587 E: SUBSYSTEM=nd 588 589 ...and is available as a region attribute, but keep in mind that the 590 "devtype" does not indicate sub-type variations and scripts should 591 really be understanding the other attributes. 592 593 3. type specific attributes: 594 595 As it currently stands a BLK-aperture region will never have a 596 "nfit/spa_index" attribute, but neither will a non-NFIT PMEM region. A 597 BLK region with a "mappings" value of 0 is, as mentioned above, a DIMM 598 that does not allow I/O. A PMEM region with a "mappings" value of zero 599 is a simple system-physical-address range. 600 601 602 LIBNVDIMM/LIBNDCTL: Namespace 603 ------------------------- 604 605 A REGION, after resolving DPA aliasing and LABEL specified boundaries, 606 surfaces one or more "namespace" devices. The arrival of a "namespace" 607 device currently triggers either the nd_blk or nd_pmem driver to load 608 and register a disk/block device. 609 610 LIBNVDIMM: namespace 611 Here is a sample layout from the three major types of NAMESPACE where 612 namespace0.0 represents DIMM-info-backed PMEM (note that it has a 'uuid' 613 attribute), namespace2.0 represents a BLK namespace (note it has a 614 'sector_size' attribute) that, and namespace6.0 represents an anonymous 615 PMEM namespace (note that has no 'uuid' attribute due to not support a 616 LABEL). 617 618 /sys/devices/platform/nfit_test.0/ndbus0/region0/namespace0.0 619 |-- alt_name 620 |-- devtype 621 |-- dpa_extents 622 |-- force_raw 623 |-- modalias 624 |-- numa_node 625 |-- resource 626 |-- size 627 |-- subsystem -> ../../../../../../bus/nd 628 |-- type 629 |-- uevent 630 `-- uuid 631 /sys/devices/platform/nfit_test.0/ndbus0/region2/namespace2.0 632 |-- alt_name 633 |-- devtype 634 |-- dpa_extents 635 |-- force_raw 636 |-- modalias 637 |-- numa_node 638 |-- sector_size 639 |-- size 640 |-- subsystem -> ../../../../../../bus/nd 641 |-- type 642 |-- uevent 643 `-- uuid 644 /sys/devices/platform/nfit_test.1/ndbus1/region6/namespace6.0 645 |-- block 646 | `-- pmem0 647 |-- devtype 648 |-- driver -> ../../../../../../bus/nd/drivers/pmem 649 |-- force_raw 650 |-- modalias 651 |-- numa_node 652 |-- resource 653 |-- size 654 |-- subsystem -> ../../../../../../bus/nd 655 |-- type 656 `-- uevent 657 658 LIBNDCTL: namespace enumeration example 659 Namespaces are indexed relative to their parent region, example below. 660 These indexes are mostly static from boot to boot, but subsystem makes 661 no guarantees in this regard. For a static namespace identifier use its 662 'uuid' attribute. 663 664 static struct ndctl_namespace *get_namespace_by_id(struct ndctl_region *region, 665 unsigned int id) 666 { 667 struct ndctl_namespace *ndns; 668 669 ndctl_namespace_foreach(region, ndns) 670 if (ndctl_namespace_get_id(ndns) == id) 671 return ndns; 672 673 return NULL; 674 } 675 676 LIBNDCTL: namespace creation example 677 Idle namespaces are automatically created by the kernel if a given 678 region has enough available capacity to create a new namespace. 679 Namespace instantiation involves finding an idle namespace and 680 configuring it. For the most part the setting of namespace attributes 681 can occur in any order, the only constraint is that 'uuid' must be set 682 before 'size'. This enables the kernel to track DPA allocations 683 internally with a static identifier. 684 685 static int configure_namespace(struct ndctl_region *region, 686 struct ndctl_namespace *ndns, 687 struct namespace_parameters *parameters) 688 { 689 char devname[50]; 690 691 snprintf(devname, sizeof(devname), "namespace%d.%d", 692 ndctl_region_get_id(region), paramaters->id); 693 694 ndctl_namespace_set_alt_name(ndns, devname); 695 /* 'uuid' must be set prior to setting size! */ 696 ndctl_namespace_set_uuid(ndns, paramaters->uuid); 697 ndctl_namespace_set_size(ndns, paramaters->size); 698 /* unlike pmem namespaces, blk namespaces have a sector size */ 699 if (parameters->lbasize) 700 ndctl_namespace_set_sector_size(ndns, parameters->lbasize); 701 ndctl_namespace_enable(ndns); 702 } 703 704 705 Why the Term "namespace"? 706 707 1. Why not "volume" for instance? "volume" ran the risk of confusing 708 ND (libnvdimm subsystem) to a volume manager like device-mapper. 709 710 2. The term originated to describe the sub-devices that can be created 711 within a NVME controller (see the nvme specification: 712 http://www.nvmexpress.org/specifications/), and NFIT namespaces are 713 meant to parallel the capabilities and configurability of 714 NVME-namespaces. 715 716 717 LIBNVDIMM/LIBNDCTL: Block Translation Table "btt" 718 --------------------------------------------- 719 720 A BTT (design document: http://pmem.io/2014/09/23/btt.html) is a stacked 721 block device driver that fronts either the whole block device or a 722 partition of a block device emitted by either a PMEM or BLK NAMESPACE. 723 724 LIBNVDIMM: btt layout 725 Every region will start out with at least one BTT device which is the 726 seed device. To activate it set the "namespace", "uuid", and 727 "sector_size" attributes and then bind the device to the nd_pmem or 728 nd_blk driver depending on the region type. 729 730 /sys/devices/platform/nfit_test.1/ndbus0/region0/btt0/ 731 |-- namespace 732 |-- delete 733 |-- devtype 734 |-- modalias 735 |-- numa_node 736 |-- sector_size 737 |-- subsystem -> ../../../../../bus/nd 738 |-- uevent 739 `-- uuid 740 741 LIBNDCTL: btt creation example 742 Similar to namespaces an idle BTT device is automatically created per 743 region. Each time this "seed" btt device is configured and enabled a new 744 seed is created. Creating a BTT configuration involves two steps of 745 finding and idle BTT and assigning it to consume a PMEM or BLK namespace. 746 747 static struct ndctl_btt *get_idle_btt(struct ndctl_region *region) 748 { 749 struct ndctl_btt *btt; 750 751 ndctl_btt_foreach(region, btt) 752 if (!ndctl_btt_is_enabled(btt) 753 && !ndctl_btt_is_configured(btt)) 754 return btt; 755 756 return NULL; 757 } 758 759 static int configure_btt(struct ndctl_region *region, 760 struct btt_parameters *parameters) 761 { 762 btt = get_idle_btt(region); 763 764 ndctl_btt_set_uuid(btt, parameters->uuid); 765 ndctl_btt_set_sector_size(btt, parameters->sector_size); 766 ndctl_btt_set_namespace(btt, parameters->ndns); 767 /* turn off raw mode device */ 768 ndctl_namespace_disable(parameters->ndns); 769 /* turn on btt access */ 770 ndctl_btt_enable(btt); 771 } 772 773 Once instantiated a new inactive btt seed device will appear underneath 774 the region. 775 776 Once a "namespace" is removed from a BTT that instance of the BTT device 777 will be deleted or otherwise reset to default values. This deletion is 778 only at the device model level. In order to destroy a BTT the "info 779 block" needs to be destroyed. Note, that to destroy a BTT the media 780 needs to be written in raw mode. By default, the kernel will autodetect 781 the presence of a BTT and disable raw mode. This autodetect behavior 782 can be suppressed by enabling raw mode for the namespace via the 783 ndctl_namespace_set_raw_mode() API. 784 785 786 Summary LIBNDCTL Diagram 787 ------------------------ 788 789 For the given example above, here is the view of the objects as seen by the 790 LIBNDCTL API: 791 +---+ 792 |CTX| +---------+ +--------------+ +---------------+ 793 +-+-+ +-> REGION0 +---> NAMESPACE0.0 +--> PMEM8 "pm0.0" | 794 | | +---------+ +--------------+ +---------------+ 795 +-------+ | | +---------+ +--------------+ +---------------+ 796 | DIMM0 <-+ | +-> REGION1 +---> NAMESPACE1.0 +--> PMEM6 "pm1.0" | 797 +-------+ | | | +---------+ +--------------+ +---------------+ 798 | DIMM1 <-+ +-v--+ | +---------+ +--------------+ +---------------+ 799 +-------+ +-+BUS0+---> REGION2 +-+-> NAMESPACE2.0 +--> ND6 "blk2.0" | 800 | DIMM2 <-+ +----+ | +---------+ | +--------------+ +----------------------+ 801 +-------+ | | +-> NAMESPACE2.1 +--> ND5 "blk2.1" | BTT2 | 802 | DIMM3 <-+ | +--------------+ +----------------------+ 803 +-------+ | +---------+ +--------------+ +---------------+ 804 +-> REGION3 +-+-> NAMESPACE3.0 +--> ND4 "blk3.0" | 805 | +---------+ | +--------------+ +----------------------+ 806 | +-> NAMESPACE3.1 +--> ND3 "blk3.1" | BTT1 | 807 | +--------------+ +----------------------+ 808 | +---------+ +--------------+ +---------------+ 809 +-> REGION4 +---> NAMESPACE4.0 +--> ND2 "blk4.0" | 810 | +---------+ +--------------+ +---------------+ 811 | +---------+ +--------------+ +----------------------+ 812 +-> REGION5 +---> NAMESPACE5.0 +--> ND1 "blk5.0" | BTT0 | 813 +---------+ +--------------+ +---------------+------+ 814