About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / nvdimm




Custom Search

Based on kernel version 4.9. Page generated on 2016-12-21 14:36 EST.

1				  LIBNVDIMM: Non-Volatile Devices
2		      libnvdimm - kernel / libndctl - userspace helper library
3				   linux-nvdimm@lists.01.org
4					      v13
5	
6	
7		Glossary
8		Overview
9		    Supporting Documents
10		    Git Trees
11		LIBNVDIMM PMEM and BLK
12		Why BLK?
13		    PMEM vs BLK
14		        BLK-REGIONs, PMEM-REGIONs, Atomic Sectors, and DAX
15		Example NVDIMM Platform
16		LIBNVDIMM Kernel Device Model and LIBNDCTL Userspace API
17		    LIBNDCTL: Context
18		        libndctl: instantiate a new library context example
19		    LIBNVDIMM/LIBNDCTL: Bus
20		        libnvdimm: control class device in /sys/class
21		        libnvdimm: bus
22		        libndctl: bus enumeration example
23		    LIBNVDIMM/LIBNDCTL: DIMM (NMEM)
24		        libnvdimm: DIMM (NMEM)
25		        libndctl: DIMM enumeration example
26		    LIBNVDIMM/LIBNDCTL: Region
27		        libnvdimm: region
28		        libndctl: region enumeration example
29		        Why Not Encode the Region Type into the Region Name?
30		        How Do I Determine the Major Type of a Region?
31		    LIBNVDIMM/LIBNDCTL: Namespace
32		        libnvdimm: namespace
33		        libndctl: namespace enumeration example
34		        libndctl: namespace creation example
35		        Why the Term "namespace"?
36		    LIBNVDIMM/LIBNDCTL: Block Translation Table "btt"
37		        libnvdimm: btt layout
38		        libndctl: btt creation example
39		Summary LIBNDCTL Diagram
40	
41	
42	Glossary
43	--------
44	
45	PMEM: A system-physical-address range where writes are persistent.  A
46	block device composed of PMEM is capable of DAX.  A PMEM address range
47	may span an interleave of several DIMMs.
48	
49	BLK: A set of one or more programmable memory mapped apertures provided
50	by a DIMM to access its media.  This indirection precludes the
51	performance benefit of interleaving, but enables DIMM-bounded failure
52	modes.
53	
54	DPA: DIMM Physical Address, is a DIMM-relative offset.  With one DIMM in
55	the system there would be a 1:1 system-physical-address:DPA association.
56	Once more DIMMs are added a memory controller interleave must be
57	decoded to determine the DPA associated with a given
58	system-physical-address.  BLK capacity always has a 1:1 relationship
59	with a single-DIMM's DPA range.
60	
61	DAX: File system extensions to bypass the page cache and block layer to
62	mmap persistent memory, from a PMEM block device, directly into a
63	process address space.
64	
65	DSM: Device Specific Method: ACPI method to to control specific
66	device - in this case the firmware.
67	
68	DCR: NVDIMM Control Region Structure defined in ACPI 6 Section 5.2.25.5.
69	It defines a vendor-id, device-id, and interface format for a given DIMM.
70	
71	BTT: Block Translation Table: Persistent memory is byte addressable.
72	Existing software may have an expectation that the power-fail-atomicity
73	of writes is at least one sector, 512 bytes.  The BTT is an indirection
74	table with atomic update semantics to front a PMEM/BLK block device
75	driver and present arbitrary atomic sector sizes.
76	
77	LABEL: Metadata stored on a DIMM device that partitions and identifies
78	(persistently names) storage between PMEM and BLK.  It also partitions
79	BLK storage to host BTTs with different parameters per BLK-partition.
80	Note that traditional partition tables, GPT/MBR, are layered on top of a
81	BLK or PMEM device.
82	
83	
84	Overview
85	--------
86	
87	The LIBNVDIMM subsystem provides support for three types of NVDIMMs, namely,
88	PMEM, BLK, and NVDIMM devices that can simultaneously support both PMEM
89	and BLK mode access.  These three modes of operation are described by
90	the "NVDIMM Firmware Interface Table" (NFIT) in ACPI 6.  While the LIBNVDIMM
91	implementation is generic and supports pre-NFIT platforms, it was guided
92	by the superset of capabilities need to support this ACPI 6 definition
93	for NVDIMM resources.  The bulk of the kernel implementation is in place
94	to handle the case where DPA accessible via PMEM is aliased with DPA
95	accessible via BLK.  When that occurs a LABEL is needed to reserve DPA
96	for exclusive access via one mode a time.
97	
98	Supporting Documents
99	ACPI 6: http://www.uefi.org/sites/default/files/resources/ACPI_6.0.pdf
100	NVDIMM Namespace: http://pmem.io/documents/NVDIMM_Namespace_Spec.pdf
101	DSM Interface Example: http://pmem.io/documents/NVDIMM_DSM_Interface_Example.pdf
102	Driver Writer's Guide: http://pmem.io/documents/NVDIMM_Driver_Writers_Guide.pdf
103	
104	Git Trees
105	LIBNVDIMM: https://git.kernel.org/cgit/linux/kernel/git/djbw/nvdimm.git
106	LIBNDCTL: https://github.com/pmem/ndctl.git
107	PMEM: https://github.com/01org/prd
108	
109	
110	LIBNVDIMM PMEM and BLK
111	------------------
112	
113	Prior to the arrival of the NFIT, non-volatile memory was described to a
114	system in various ad-hoc ways.  Usually only the bare minimum was
115	provided, namely, a single system-physical-address range where writes
116	are expected to be durable after a system power loss.  Now, the NFIT
117	specification standardizes not only the description of PMEM, but also
118	BLK and platform message-passing entry points for control and
119	configuration.
120	
121	For each NVDIMM access method (PMEM, BLK), LIBNVDIMM provides a block
122	device driver:
123	
124	    1. PMEM (nd_pmem.ko): Drives a system-physical-address range.  This
125	    range is contiguous in system memory and may be interleaved (hardware
126	    memory controller striped) across multiple DIMMs.  When interleaved the
127	    platform may optionally provide details of which DIMMs are participating
128	    in the interleave.
129	
130	    Note that while LIBNVDIMM describes system-physical-address ranges that may
131	    alias with BLK access as ND_NAMESPACE_PMEM ranges and those without
132	    alias as ND_NAMESPACE_IO ranges, to the nd_pmem driver there is no
133	    distinction.  The different device-types are an implementation detail
134	    that userspace can exploit to implement policies like "only interface
135	    with address ranges from certain DIMMs".  It is worth noting that when
136	    aliasing is present and a DIMM lacks a label, then no block device can
137	    be created by default as userspace needs to do at least one allocation
138	    of DPA to the PMEM range.  In contrast ND_NAMESPACE_IO ranges, once
139	    registered, can be immediately attached to nd_pmem.
140	
141	    2. BLK (nd_blk.ko): This driver performs I/O using a set of platform
142	    defined apertures.  A set of apertures will access just one DIMM.
143	    Multiple windows (apertures) allow multiple concurrent accesses, much like
144	    tagged-command-queuing, and would likely be used by different threads or
145	    different CPUs.
146	
147	    The NFIT specification defines a standard format for a BLK-aperture, but
148	    the spec also allows for vendor specific layouts, and non-NFIT BLK
149	    implementations may have other designs for BLK I/O.  For this reason
150	    "nd_blk" calls back into platform-specific code to perform the I/O.
151	    One such implementation is defined in the "Driver Writer's Guide" and "DSM
152	    Interface Example".
153	
154	
155	Why BLK?
156	--------
157	
158	While PMEM provides direct byte-addressable CPU-load/store access to
159	NVDIMM storage, it does not provide the best system RAS (recovery,
160	availability, and serviceability) model.  An access to a corrupted
161	system-physical-address address causes a CPU exception while an access
162	to a corrupted address through an BLK-aperture causes that block window
163	to raise an error status in a register.  The latter is more aligned with
164	the standard error model that host-bus-adapter attached disks present.
165	Also, if an administrator ever wants to replace a memory it is easier to
166	service a system at DIMM module boundaries.  Compare this to PMEM where
167	data could be interleaved in an opaque hardware specific manner across
168	several DIMMs.
169	
170	PMEM vs BLK
171	BLK-apertures solve these RAS problems, but their presence is also the
172	major contributing factor to the complexity of the ND subsystem.  They
173	complicate the implementation because PMEM and BLK alias in DPA space.
174	Any given DIMM's DPA-range may contribute to one or more
175	system-physical-address sets of interleaved DIMMs, *and* may also be
176	accessed in its entirety through its BLK-aperture.  Accessing a DPA
177	through a system-physical-address while simultaneously accessing the
178	same DPA through a BLK-aperture has undefined results.  For this reason,
179	DIMMs with this dual interface configuration include a DSM function to
180	store/retrieve a LABEL.  The LABEL effectively partitions the DPA-space
181	into exclusive system-physical-address and BLK-aperture accessible
182	regions.  For simplicity a DIMM is allowed a PMEM "region" per each
183	interleave set in which it is a member.  The remaining DPA space can be
184	carved into an arbitrary number of BLK devices with discontiguous
185	extents.
186	
187	BLK-REGIONs, PMEM-REGIONs, Atomic Sectors, and DAX
188	--------------------------------------------------
189	
190	One of the few
191	reasons to allow multiple BLK namespaces per REGION is so that each
192	BLK-namespace can be configured with a BTT with unique atomic sector
193	sizes.  While a PMEM device can host a BTT the LABEL specification does
194	not provide for a sector size to be specified for a PMEM namespace.
195	This is due to the expectation that the primary usage model for PMEM is
196	via DAX, and the BTT is incompatible with DAX.  However, for the cases
197	where an application or filesystem still needs atomic sector update
198	guarantees it can register a BTT on a PMEM device or partition.  See
199	LIBNVDIMM/NDCTL: Block Translation Table "btt"
200	
201	
202	Example NVDIMM Platform
203	-----------------------
204	
205	For the remainder of this document the following diagram will be
206	referenced for any example sysfs layouts.
207	
208	
209	                             (a)               (b)           DIMM   BLK-REGION
210	          +-------------------+--------+--------+--------+
211	+------+  |       pm0.0       | blk2.0 | pm1.0  | blk2.1 |    0      region2
212	| imc0 +--+- - - region0- - - +--------+        +--------+
213	+--+---+  |       pm0.0       | blk3.0 | pm1.0  | blk3.1 |    1      region3
214	   |      +-------------------+--------v        v--------+
215	+--+---+                               |                 |
216	| cpu0 |                                     region1
217	+--+---+                               |                 |
218	   |      +----------------------------^        ^--------+
219	+--+---+  |           blk4.0           | pm1.0  | blk4.0 |    2      region4
220	| imc1 +--+----------------------------|        +--------+
221	+------+  |           blk5.0           | pm1.0  | blk5.0 |    3      region5
222	          +----------------------------+--------+--------+
223	
224	In this platform we have four DIMMs and two memory controllers in one
225	socket.  Each unique interface (BLK or PMEM) to DPA space is identified
226	by a region device with a dynamically assigned id (REGION0 - REGION5).
227	
228	    1. The first portion of DIMM0 and DIMM1 are interleaved as REGION0. A
229	    single PMEM namespace is created in the REGION0-SPA-range that spans most
230	    of DIMM0 and DIMM1 with a user-specified name of "pm0.0". Some of that
231	    interleaved system-physical-address range is reclaimed as BLK-aperture
232	    accessed space starting at DPA-offset (a) into each DIMM.  In that
233	    reclaimed space we create two BLK-aperture "namespaces" from REGION2 and
234	    REGION3 where "blk2.0" and "blk3.0" are just human readable names that
235	    could be set to any user-desired name in the LABEL.
236	
237	    2. In the last portion of DIMM0 and DIMM1 we have an interleaved
238	    system-physical-address range, REGION1, that spans those two DIMMs as
239	    well as DIMM2 and DIMM3.  Some of REGION1 is allocated to a PMEM namespace
240	    named "pm1.0", the rest is reclaimed in 4 BLK-aperture namespaces (for
241	    each DIMM in the interleave set), "blk2.1", "blk3.1", "blk4.0", and
242	    "blk5.0".
243	
244	    3. The portion of DIMM2 and DIMM3 that do not participate in the REGION1
245	    interleaved system-physical-address range (i.e. the DPA address past
246	    offset (b) are also included in the "blk4.0" and "blk5.0" namespaces.
247	    Note, that this example shows that BLK-aperture namespaces don't need to
248	    be contiguous in DPA-space.
249	
250	    This bus is provided by the kernel under the device
251	    /sys/devices/platform/nfit_test.0 when CONFIG_NFIT_TEST is enabled and
252	    the nfit_test.ko module is loaded.  This not only test LIBNVDIMM but the
253	    acpi_nfit.ko driver as well.
254	
255	
256	LIBNVDIMM Kernel Device Model and LIBNDCTL Userspace API
257	----------------------------------------------------
258	
259	What follows is a description of the LIBNVDIMM sysfs layout and a
260	corresponding object hierarchy diagram as viewed through the LIBNDCTL
261	API.  The example sysfs paths and diagrams are relative to the Example
262	NVDIMM Platform which is also the LIBNVDIMM bus used in the LIBNDCTL unit
263	test.
264	
265	LIBNDCTL: Context
266	Every API call in the LIBNDCTL library requires a context that holds the
267	logging parameters and other library instance state.  The library is
268	based on the libabc template:
269	https://git.kernel.org/cgit/linux/kernel/git/kay/libabc.git
270	
271	LIBNDCTL: instantiate a new library context example
272	
273		struct ndctl_ctx *ctx;
274	
275		if (ndctl_new(&ctx) == 0)
276			return ctx;
277		else
278			return NULL;
279	
280	LIBNVDIMM/LIBNDCTL: Bus
281	-------------------
282	
283	A bus has a 1:1 relationship with an NFIT.  The current expectation for
284	ACPI based systems is that there is only ever one platform-global NFIT.
285	That said, it is trivial to register multiple NFITs, the specification
286	does not preclude it.  The infrastructure supports multiple busses and
287	we we use this capability to test multiple NFIT configurations in the
288	unit test.
289	
290	LIBNVDIMM: control class device in /sys/class
291	
292	This character device accepts DSM messages to be passed to DIMM
293	identified by its NFIT handle.
294	
295		/sys/class/nd/ndctl0
296		|-- dev
297		|-- device -> ../../../ndbus0
298		|-- subsystem -> ../../../../../../../class/nd
299	
300	
301	
302	LIBNVDIMM: bus
303	
304		struct nvdimm_bus *nvdimm_bus_register(struct device *parent,
305		       struct nvdimm_bus_descriptor *nfit_desc);
306	
307		/sys/devices/platform/nfit_test.0/ndbus0
308		|-- commands
309		|-- nd
310		|-- nfit
311		|-- nmem0
312		|-- nmem1
313		|-- nmem2
314		|-- nmem3
315		|-- power
316		|-- provider
317		|-- region0
318		|-- region1
319		|-- region2
320		|-- region3
321		|-- region4
322		|-- region5
323		|-- uevent
324		`-- wait_probe
325	
326	LIBNDCTL: bus enumeration example
327	Find the bus handle that describes the bus from Example NVDIMM Platform
328	
329		static struct ndctl_bus *get_bus_by_provider(struct ndctl_ctx *ctx,
330				const char *provider)
331		{
332			struct ndctl_bus *bus;
333	
334			ndctl_bus_foreach(ctx, bus)
335				if (strcmp(provider, ndctl_bus_get_provider(bus)) == 0)
336					return bus;
337	
338			return NULL;
339		}
340	
341		bus = get_bus_by_provider(ctx, "nfit_test.0");
342	
343	
344	LIBNVDIMM/LIBNDCTL: DIMM (NMEM)
345	---------------------------
346	
347	The DIMM device provides a character device for sending commands to
348	hardware, and it is a container for LABELs.  If the DIMM is defined by
349	NFIT then an optional 'nfit' attribute sub-directory is available to add
350	NFIT-specifics.
351	
352	Note that the kernel device name for "DIMMs" is "nmemX".  The NFIT
353	describes these devices via "Memory Device to System Physical Address
354	Range Mapping Structure", and there is no requirement that they actually
355	be physical DIMMs, so we use a more generic name.
356	
357	LIBNVDIMM: DIMM (NMEM)
358	
359		struct nvdimm *nvdimm_create(struct nvdimm_bus *nvdimm_bus, void *provider_data,
360				const struct attribute_group **groups, unsigned long flags,
361				unsigned long *dsm_mask);
362	
363		/sys/devices/platform/nfit_test.0/ndbus0
364		|-- nmem0
365		|   |-- available_slots
366		|   |-- commands
367		|   |-- dev
368		|   |-- devtype
369		|   |-- driver -> ../../../../../bus/nd/drivers/nvdimm
370		|   |-- modalias
371		|   |-- nfit
372		|   |   |-- device
373		|   |   |-- format
374		|   |   |-- handle
375		|   |   |-- phys_id
376		|   |   |-- rev_id
377		|   |   |-- serial
378		|   |   `-- vendor
379		|   |-- state
380		|   |-- subsystem -> ../../../../../bus/nd
381		|   `-- uevent
382		|-- nmem1
383		[..]
384	
385	
386	LIBNDCTL: DIMM enumeration example
387	
388	Note, in this example we are assuming NFIT-defined DIMMs which are
389	identified by an "nfit_handle" a 32-bit value where:
390	Bit 3:0 DIMM number within the memory channel
391	Bit 7:4 memory channel number
392	Bit 11:8 memory controller ID
393	Bit 15:12 socket ID (within scope of a Node controller if node controller is present)
394	Bit 27:16 Node Controller ID
395	Bit 31:28 Reserved
396	
397		static struct ndctl_dimm *get_dimm_by_handle(struct ndctl_bus *bus,
398		       unsigned int handle)
399		{
400			struct ndctl_dimm *dimm;
401	
402			ndctl_dimm_foreach(bus, dimm)
403				if (ndctl_dimm_get_handle(dimm) == handle)
404					return dimm;
405	
406			return NULL;
407		}
408	
409		#define DIMM_HANDLE(n, s, i, c, d) \
410			(((n & 0xfff) << 16) | ((s & 0xf) << 12) | ((i & 0xf) << 8) \
411			 | ((c & 0xf) << 4) | (d & 0xf))
412	
413		dimm = get_dimm_by_handle(bus, DIMM_HANDLE(0, 0, 0, 0, 0));
414	
415	LIBNVDIMM/LIBNDCTL: Region
416	----------------------
417	
418	A generic REGION device is registered for each PMEM range or BLK-aperture
419	set.  Per the example there are 6 regions: 2 PMEM and 4 BLK-aperture
420	sets on the "nfit_test.0" bus.  The primary role of regions are to be a
421	container of "mappings".  A mapping is a tuple of <DIMM,
422	DPA-start-offset, length>.
423	
424	LIBNVDIMM provides a built-in driver for these REGION devices.  This driver
425	is responsible for reconciling the aliased DPA mappings across all
426	regions, parsing the LABEL, if present, and then emitting NAMESPACE
427	devices with the resolved/exclusive DPA-boundaries for the nd_pmem or
428	nd_blk device driver to consume.
429	
430	In addition to the generic attributes of "mapping"s, "interleave_ways"
431	and "size" the REGION device also exports some convenience attributes.
432	"nstype" indicates the integer type of namespace-device this region
433	emits, "devtype" duplicates the DEVTYPE variable stored by udev at the
434	'add' event, "modalias" duplicates the MODALIAS variable stored by udev
435	at the 'add' event, and finally, the optional "spa_index" is provided in
436	the case where the region is defined by a SPA.
437	
438	LIBNVDIMM: region
439	
440		struct nd_region *nvdimm_pmem_region_create(struct nvdimm_bus *nvdimm_bus,
441				struct nd_region_desc *ndr_desc);
442		struct nd_region *nvdimm_blk_region_create(struct nvdimm_bus *nvdimm_bus,
443				struct nd_region_desc *ndr_desc);
444	
445		/sys/devices/platform/nfit_test.0/ndbus0
446		|-- region0
447		|   |-- available_size
448		|   |-- btt0
449		|   |-- btt_seed
450		|   |-- devtype
451		|   |-- driver -> ../../../../../bus/nd/drivers/nd_region
452		|   |-- init_namespaces
453		|   |-- mapping0
454		|   |-- mapping1
455		|   |-- mappings
456		|   |-- modalias
457		|   |-- namespace0.0
458		|   |-- namespace_seed
459		|   |-- numa_node
460		|   |-- nfit
461		|   |   `-- spa_index
462		|   |-- nstype
463		|   |-- set_cookie
464		|   |-- size
465		|   |-- subsystem -> ../../../../../bus/nd
466		|   `-- uevent
467		|-- region1
468		[..]
469	
470	LIBNDCTL: region enumeration example
471	
472	Sample region retrieval routines based on NFIT-unique data like
473	"spa_index" (interleave set id) for PMEM and "nfit_handle" (dimm id) for
474	BLK.
475	
476		static struct ndctl_region *get_pmem_region_by_spa_index(struct ndctl_bus *bus,
477				unsigned int spa_index)
478		{
479			struct ndctl_region *region;
480	
481			ndctl_region_foreach(bus, region) {
482				if (ndctl_region_get_type(region) != ND_DEVICE_REGION_PMEM)
483					continue;
484				if (ndctl_region_get_spa_index(region) == spa_index)
485					return region;
486			}
487			return NULL;
488		}
489	
490		static struct ndctl_region *get_blk_region_by_dimm_handle(struct ndctl_bus *bus,
491				unsigned int handle)
492		{
493			struct ndctl_region *region;
494	
495			ndctl_region_foreach(bus, region) {
496				struct ndctl_mapping *map;
497	
498				if (ndctl_region_get_type(region) != ND_DEVICE_REGION_BLOCK)
499					continue;
500				ndctl_mapping_foreach(region, map) {
501					struct ndctl_dimm *dimm = ndctl_mapping_get_dimm(map);
502	
503					if (ndctl_dimm_get_handle(dimm) == handle)
504						return region;
505				}
506			}
507			return NULL;
508		}
509	
510	
511	Why Not Encode the Region Type into the Region Name?
512	----------------------------------------------------
513	
514	At first glance it seems since NFIT defines just PMEM and BLK interface
515	types that we should simply name REGION devices with something derived
516	from those type names.  However, the ND subsystem explicitly keeps the
517	REGION name generic and expects userspace to always consider the
518	region-attributes for four reasons:
519	
520	    1. There are already more than two REGION and "namespace" types.  For
521	    PMEM there are two subtypes.  As mentioned previously we have PMEM where
522	    the constituent DIMM devices are known and anonymous PMEM.  For BLK
523	    regions the NFIT specification already anticipates vendor specific
524	    implementations.  The exact distinction of what a region contains is in
525	    the region-attributes not the region-name or the region-devtype.
526	
527	    2. A region with zero child-namespaces is a possible configuration.  For
528	    example, the NFIT allows for a DCR to be published without a
529	    corresponding BLK-aperture.  This equates to a DIMM that can only accept
530	    control/configuration messages, but no i/o through a descendant block
531	    device.  Again, this "type" is advertised in the attributes ('mappings'
532	    == 0) and the name does not tell you much.
533	
534	    3. What if a third major interface type arises in the future?  Outside
535	    of vendor specific implementations, it's not difficult to envision a
536	    third class of interface type beyond BLK and PMEM.  With a generic name
537	    for the REGION level of the device-hierarchy old userspace
538	    implementations can still make sense of new kernel advertised
539	    region-types.  Userspace can always rely on the generic region
540	    attributes like "mappings", "size", etc and the expected child devices
541	    named "namespace".  This generic format of the device-model hierarchy
542	    allows the LIBNVDIMM and LIBNDCTL implementations to be more uniform and
543	    future-proof.
544	
545	    4. There are more robust mechanisms for determining the major type of a
546	    region than a device name.  See the next section, How Do I Determine the
547	    Major Type of a Region?
548	
549	How Do I Determine the Major Type of a Region?
550	----------------------------------------------
551	
552	Outside of the blanket recommendation of "use libndctl", or simply
553	looking at the kernel header (/usr/include/linux/ndctl.h) to decode the
554	"nstype" integer attribute, here are some other options.
555	
556	    1. module alias lookup:
557	
558	    The whole point of region/namespace device type differentiation is to
559	    decide which block-device driver will attach to a given LIBNVDIMM namespace.
560	    One can simply use the modalias to lookup the resulting module.  It's
561	    important to note that this method is robust in the presence of a
562	    vendor-specific driver down the road.  If a vendor-specific
563	    implementation wants to supplant the standard nd_blk driver it can with
564	    minimal impact to the rest of LIBNVDIMM.
565	
566	    In fact, a vendor may also want to have a vendor-specific region-driver
567	    (outside of nd_region).  For example, if a vendor defined its own LABEL
568	    format it would need its own region driver to parse that LABEL and emit
569	    the resulting namespaces.  The output from module resolution is more
570	    accurate than a region-name or region-devtype.
571	
572	    2. udev:
573	
574	    The kernel "devtype" is registered in the udev database
575	    # udevadm info --path=/devices/platform/nfit_test.0/ndbus0/region0
576	    P: /devices/platform/nfit_test.0/ndbus0/region0
577	    E: DEVPATH=/devices/platform/nfit_test.0/ndbus0/region0
578	    E: DEVTYPE=nd_pmem
579	    E: MODALIAS=nd:t2
580	    E: SUBSYSTEM=nd
581	
582	    # udevadm info --path=/devices/platform/nfit_test.0/ndbus0/region4
583	    P: /devices/platform/nfit_test.0/ndbus0/region4
584	    E: DEVPATH=/devices/platform/nfit_test.0/ndbus0/region4
585	    E: DEVTYPE=nd_blk
586	    E: MODALIAS=nd:t3
587	    E: SUBSYSTEM=nd
588	
589	    ...and is available as a region attribute, but keep in mind that the
590	    "devtype" does not indicate sub-type variations and scripts should
591	    really be understanding the other attributes.
592	
593	    3. type specific attributes:
594	
595	    As it currently stands a BLK-aperture region will never have a
596	    "nfit/spa_index" attribute, but neither will a non-NFIT PMEM region.  A
597	    BLK region with a "mappings" value of 0 is, as mentioned above, a DIMM
598	    that does not allow I/O.  A PMEM region with a "mappings" value of zero
599	    is a simple system-physical-address range.
600	
601	
602	LIBNVDIMM/LIBNDCTL: Namespace
603	-------------------------
604	
605	A REGION, after resolving DPA aliasing and LABEL specified boundaries,
606	surfaces one or more "namespace" devices.  The arrival of a "namespace"
607	device currently triggers either the nd_blk or nd_pmem driver to load
608	and register a disk/block device.
609	
610	LIBNVDIMM: namespace
611	Here is a sample layout from the three major types of NAMESPACE where
612	namespace0.0 represents DIMM-info-backed PMEM (note that it has a 'uuid'
613	attribute), namespace2.0 represents a BLK namespace (note it has a
614	'sector_size' attribute) that, and namespace6.0 represents an anonymous
615	PMEM namespace (note that has no 'uuid' attribute due to not support a
616	LABEL).
617	
618		/sys/devices/platform/nfit_test.0/ndbus0/region0/namespace0.0
619		|-- alt_name
620		|-- devtype
621		|-- dpa_extents
622		|-- force_raw
623		|-- modalias
624		|-- numa_node
625		|-- resource
626		|-- size
627		|-- subsystem -> ../../../../../../bus/nd
628		|-- type
629		|-- uevent
630		`-- uuid
631		/sys/devices/platform/nfit_test.0/ndbus0/region2/namespace2.0
632		|-- alt_name
633		|-- devtype
634		|-- dpa_extents
635		|-- force_raw
636		|-- modalias
637		|-- numa_node
638		|-- sector_size
639		|-- size
640		|-- subsystem -> ../../../../../../bus/nd
641		|-- type
642		|-- uevent
643		`-- uuid
644		/sys/devices/platform/nfit_test.1/ndbus1/region6/namespace6.0
645		|-- block
646		|   `-- pmem0
647		|-- devtype
648		|-- driver -> ../../../../../../bus/nd/drivers/pmem
649		|-- force_raw
650		|-- modalias
651		|-- numa_node
652		|-- resource
653		|-- size
654		|-- subsystem -> ../../../../../../bus/nd
655		|-- type
656		`-- uevent
657	
658	LIBNDCTL: namespace enumeration example
659	Namespaces are indexed relative to their parent region, example below.
660	These indexes are mostly static from boot to boot, but subsystem makes
661	no guarantees in this regard.  For a static namespace identifier use its
662	'uuid' attribute.
663	
664	static struct ndctl_namespace *get_namespace_by_id(struct ndctl_region *region,
665	                unsigned int id)
666	{
667	        struct ndctl_namespace *ndns;
668	
669	        ndctl_namespace_foreach(region, ndns)
670	                if (ndctl_namespace_get_id(ndns) == id)
671	                        return ndns;
672	
673	        return NULL;
674	}
675	
676	LIBNDCTL: namespace creation example
677	Idle namespaces are automatically created by the kernel if a given
678	region has enough available capacity to create a new namespace.
679	Namespace instantiation involves finding an idle namespace and
680	configuring it.  For the most part the setting of namespace attributes
681	can occur in any order, the only constraint is that 'uuid' must be set
682	before 'size'.  This enables the kernel to track DPA allocations
683	internally with a static identifier.
684	
685	static int configure_namespace(struct ndctl_region *region,
686	                struct ndctl_namespace *ndns,
687	                struct namespace_parameters *parameters)
688	{
689	        char devname[50];
690	
691	        snprintf(devname, sizeof(devname), "namespace%d.%d",
692	                        ndctl_region_get_id(region), paramaters->id);
693	
694	        ndctl_namespace_set_alt_name(ndns, devname);
695	        /* 'uuid' must be set prior to setting size! */
696	        ndctl_namespace_set_uuid(ndns, paramaters->uuid);
697	        ndctl_namespace_set_size(ndns, paramaters->size);
698	        /* unlike pmem namespaces, blk namespaces have a sector size */
699	        if (parameters->lbasize)
700	                ndctl_namespace_set_sector_size(ndns, parameters->lbasize);
701	        ndctl_namespace_enable(ndns);
702	}
703	
704	
705	Why the Term "namespace"?
706	
707	    1. Why not "volume" for instance?  "volume" ran the risk of confusing
708	    ND (libnvdimm subsystem) to a volume manager like device-mapper.
709	
710	    2. The term originated to describe the sub-devices that can be created
711	    within a NVME controller (see the nvme specification:
712	    http://www.nvmexpress.org/specifications/), and NFIT namespaces are
713	    meant to parallel the capabilities and configurability of
714	    NVME-namespaces.
715	
716	
717	LIBNVDIMM/LIBNDCTL: Block Translation Table "btt"
718	---------------------------------------------
719	
720	A BTT (design document: http://pmem.io/2014/09/23/btt.html) is a stacked
721	block device driver that fronts either the whole block device or a
722	partition of a block device emitted by either a PMEM or BLK NAMESPACE.
723	
724	LIBNVDIMM: btt layout
725	Every region will start out with at least one BTT device which is the
726	seed device.  To activate it set the "namespace", "uuid", and
727	"sector_size" attributes and then bind the device to the nd_pmem or
728	nd_blk driver depending on the region type.
729	
730		/sys/devices/platform/nfit_test.1/ndbus0/region0/btt0/
731		|-- namespace
732		|-- delete
733		|-- devtype
734		|-- modalias
735		|-- numa_node
736		|-- sector_size
737		|-- subsystem -> ../../../../../bus/nd
738		|-- uevent
739		`-- uuid
740	
741	LIBNDCTL: btt creation example
742	Similar to namespaces an idle BTT device is automatically created per
743	region.  Each time this "seed" btt device is configured and enabled a new
744	seed is created.  Creating a BTT configuration involves two steps of
745	finding and idle BTT and assigning it to consume a PMEM or BLK namespace.
746	
747		static struct ndctl_btt *get_idle_btt(struct ndctl_region *region)
748		{
749			struct ndctl_btt *btt;
750	
751			ndctl_btt_foreach(region, btt)
752				if (!ndctl_btt_is_enabled(btt)
753						&& !ndctl_btt_is_configured(btt))
754					return btt;
755	
756			return NULL;
757		}
758	
759		static int configure_btt(struct ndctl_region *region,
760				struct btt_parameters *parameters)
761		{
762			btt = get_idle_btt(region);
763	
764			ndctl_btt_set_uuid(btt, parameters->uuid);
765			ndctl_btt_set_sector_size(btt, parameters->sector_size);
766			ndctl_btt_set_namespace(btt, parameters->ndns);
767			/* turn off raw mode device */
768			ndctl_namespace_disable(parameters->ndns);
769			/* turn on btt access */
770			ndctl_btt_enable(btt);
771		}
772	
773	Once instantiated a new inactive btt seed device will appear underneath
774	the region.
775	
776	Once a "namespace" is removed from a BTT that instance of the BTT device
777	will be deleted or otherwise reset to default values.  This deletion is
778	only at the device model level.  In order to destroy a BTT the "info
779	block" needs to be destroyed.  Note, that to destroy a BTT the media
780	needs to be written in raw mode.  By default, the kernel will autodetect
781	the presence of a BTT and disable raw mode.  This autodetect behavior
782	can be suppressed by enabling raw mode for the namespace via the
783	ndctl_namespace_set_raw_mode() API.
784	
785	
786	Summary LIBNDCTL Diagram
787	------------------------
788	
789	For the given example above, here is the view of the objects as seen by the
790	LIBNDCTL API:
791	            +---+
792	            |CTX|    +---------+   +--------------+  +---------------+
793	            +-+-+  +-> REGION0 +---> NAMESPACE0.0 +--> PMEM8 "pm0.0" |
794	              |    | +---------+   +--------------+  +---------------+
795	+-------+     |    | +---------+   +--------------+  +---------------+
796	| DIMM0 <-+   |    +-> REGION1 +---> NAMESPACE1.0 +--> PMEM6 "pm1.0" |
797	+-------+ |   |    | +---------+   +--------------+  +---------------+
798	| DIMM1 <-+ +-v--+ | +---------+   +--------------+  +---------------+
799	+-------+ +-+BUS0+---> REGION2 +-+-> NAMESPACE2.0 +--> ND6  "blk2.0" |
800	| DIMM2 <-+ +----+ | +---------+ | +--------------+  +----------------------+
801	+-------+ |        |             +-> NAMESPACE2.1 +--> ND5  "blk2.1" | BTT2 |
802	| DIMM3 <-+        |               +--------------+  +----------------------+
803	+-------+          | +---------+   +--------------+  +---------------+
804	                   +-> REGION3 +-+-> NAMESPACE3.0 +--> ND4  "blk3.0" |
805	                   | +---------+ | +--------------+  +----------------------+
806	                   |             +-> NAMESPACE3.1 +--> ND3  "blk3.1" | BTT1 |
807	                   |               +--------------+  +----------------------+
808	                   | +---------+   +--------------+  +---------------+
809	                   +-> REGION4 +---> NAMESPACE4.0 +--> ND2  "blk4.0" |
810	                   | +---------+   +--------------+  +---------------+
811	                   | +---------+   +--------------+  +----------------------+
812	                   +-> REGION5 +---> NAMESPACE5.0 +--> ND1  "blk5.0" | BTT0 |
813	                     +---------+   +--------------+  +---------------+------+
814	
Hide Line Numbers
About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Information is copyright its respective author. All material is available from the Linux Kernel Source distributed under a GPL License. This page is provided as a free service by mjmwired.net.