About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / device-mapper / thin-provisioning.txt


Based on kernel version 4.16.1. Page generated on 2018-04-09 11:52 EST.

1	Introduction
2	============
3	
4	This document describes a collection of device-mapper targets that
5	between them implement thin-provisioning and snapshots.
6	
7	The main highlight of this implementation, compared to the previous
8	implementation of snapshots, is that it allows many virtual devices to
9	be stored on the same data volume.  This simplifies administration and
10	allows the sharing of data between volumes, thus reducing disk usage.
11	
12	Another significant feature is support for an arbitrary depth of
13	recursive snapshots (snapshots of snapshots of snapshots ...).  The
14	previous implementation of snapshots did this by chaining together
15	lookup tables, and so performance was O(depth).  This new
16	implementation uses a single data structure to avoid this degradation
17	with depth.  Fragmentation may still be an issue, however, in some
18	scenarios.
19	
20	Metadata is stored on a separate device from data, giving the
21	administrator some freedom, for example to:
22	
23	- Improve metadata resilience by storing metadata on a mirrored volume
24	  but data on a non-mirrored one.
25	
26	- Improve performance by storing the metadata on SSD.
27	
28	Status
29	======
30	
31	These targets are very much still in the EXPERIMENTAL state.  Please
32	do not yet rely on them in production.  But do experiment and offer us
33	feedback.  Different use cases will have different performance
34	characteristics, for example due to fragmentation of the data volume.
35	
36	If you find this software is not performing as expected please mail
37	dm-devel@redhat.com with details and we'll try our best to improve
38	things for you.
39	
40	Userspace tools for checking and repairing the metadata are under
41	development.
42	
43	Cookbook
44	========
45	
46	This section describes some quick recipes for using thin provisioning.
47	They use the dmsetup program to control the device-mapper driver
48	directly.  End users will be advised to use a higher-level volume
49	manager such as LVM2 once support has been added.
50	
51	Pool device
52	-----------
53	
54	The pool device ties together the metadata volume and the data volume.
55	It maps I/O linearly to the data volume and updates the metadata via
56	two mechanisms:
57	
58	- Function calls from the thin targets
59	
60	- Device-mapper 'messages' from userspace which control the creation of new
61	  virtual devices amongst other things.
62	
63	Setting up a fresh pool device
64	------------------------------
65	
66	Setting up a pool device requires a valid metadata device, and a
67	data device.  If you do not have an existing metadata device you can
68	make one by zeroing the first 4k to indicate empty metadata.
69	
70	    dd if=/dev/zero of=$metadata_dev bs=4096 count=1
71	
72	The amount of metadata you need will vary according to how many blocks
73	are shared between thin devices (i.e. through snapshots).  If you have
74	less sharing than average you'll need a larger-than-average metadata device.
75	
76	As a guide, we suggest you calculate the number of bytes to use in the
77	metadata device as 48 * $data_dev_size / $data_block_size but round it up
78	to 2MB if the answer is smaller.  If you're creating large numbers of
79	snapshots which are recording large amounts of change, you may find you
80	need to increase this.
81	
82	The largest size supported is 16GB: If the device is larger,
83	a warning will be issued and the excess space will not be used.
84	
85	Reloading a pool table
86	----------------------
87	
88	You may reload a pool's table, indeed this is how the pool is resized
89	if it runs out of space.  (N.B. While specifying a different metadata
90	device when reloading is not forbidden at the moment, things will go
91	wrong if it does not route I/O to exactly the same on-disk location as
92	previously.)
93	
94	Using an existing pool device
95	-----------------------------
96	
97	    dmsetup create pool \
98		--table "0 20971520 thin-pool $metadata_dev $data_dev \
99			 $data_block_size $low_water_mark"
100	
101	$data_block_size gives the smallest unit of disk space that can be
102	allocated at a time expressed in units of 512-byte sectors.
103	$data_block_size must be between 128 (64KB) and 2097152 (1GB) and a
104	multiple of 128 (64KB).  $data_block_size cannot be changed after the
105	thin-pool is created.  People primarily interested in thin provisioning
106	may want to use a value such as 1024 (512KB).  People doing lots of
107	snapshotting may want a smaller value such as 128 (64KB).  If you are
108	not zeroing newly-allocated data, a larger $data_block_size in the
109	region of 256000 (128MB) is suggested.
110	
111	$low_water_mark is expressed in blocks of size $data_block_size.  If
112	free space on the data device drops below this level then a dm event
113	will be triggered which a userspace daemon should catch allowing it to
114	extend the pool device.  Only one such event will be sent.
115	
116	No special event is triggered if a just resumed device's free space is below
117	the low water mark. However, resuming a device always triggers an
118	event; a userspace daemon should verify that free space exceeds the low
119	water mark when handling this event.
120	
121	A low water mark for the metadata device is maintained in the kernel and
122	will trigger a dm event if free space on the metadata device drops below
123	it.
124	
125	Updating on-disk metadata
126	-------------------------
127	
128	On-disk metadata is committed every time a FLUSH or FUA bio is written.
129	If no such requests are made then commits will occur every second.  This
130	means the thin-provisioning target behaves like a physical disk that has
131	a volatile write cache.  If power is lost you may lose some recent
132	writes.  The metadata should always be consistent in spite of any crash.
133	
134	If data space is exhausted the pool will either error or queue IO
135	according to the configuration (see: error_if_no_space).  If metadata
136	space is exhausted or a metadata operation fails: the pool will error IO
137	until the pool is taken offline and repair is performed to 1) fix any
138	potential inconsistencies and 2) clear the flag that imposes repair.
139	Once the pool's metadata device is repaired it may be resized, which
140	will allow the pool to return to normal operation.  Note that if a pool
141	is flagged as needing repair, the pool's data and metadata devices
142	cannot be resized until repair is performed.  It should also be noted
143	that when the pool's metadata space is exhausted the current metadata
144	transaction is aborted.  Given that the pool will cache IO whose
145	completion may have already been acknowledged to upper IO layers
146	(e.g. filesystem) it is strongly suggested that consistency checks
147	(e.g. fsck) be performed on those layers when repair of the pool is
148	required.
149	
150	Thin provisioning
151	-----------------
152	
153	i) Creating a new thinly-provisioned volume.
154	
155	  To create a new thinly- provisioned volume you must send a message to an
156	  active pool device, /dev/mapper/pool in this example.
157	
158	    dmsetup message /dev/mapper/pool 0 "create_thin 0"
159	
160	  Here '0' is an identifier for the volume, a 24-bit number.  It's up
161	  to the caller to allocate and manage these identifiers.  If the
162	  identifier is already in use, the message will fail with -EEXIST.
163	
164	ii) Using a thinly-provisioned volume.
165	
166	  Thinly-provisioned volumes are activated using the 'thin' target:
167	
168	    dmsetup create thin --table "0 2097152 thin /dev/mapper/pool 0"
169	
170	  The last parameter is the identifier for the thinp device.
171	
172	Internal snapshots
173	------------------
174	
175	i) Creating an internal snapshot.
176	
177	  Snapshots are created with another message to the pool.
178	
179	  N.B.  If the origin device that you wish to snapshot is active, you
180	  must suspend it before creating the snapshot to avoid corruption.
181	  This is NOT enforced at the moment, so please be careful!
182	
183	    dmsetup suspend /dev/mapper/thin
184	    dmsetup message /dev/mapper/pool 0 "create_snap 1 0"
185	    dmsetup resume /dev/mapper/thin
186	
187	  Here '1' is the identifier for the volume, a 24-bit number.  '0' is the
188	  identifier for the origin device.
189	
190	ii) Using an internal snapshot.
191	
192	  Once created, the user doesn't have to worry about any connection
193	  between the origin and the snapshot.  Indeed the snapshot is no
194	  different from any other thinly-provisioned device and can be
195	  snapshotted itself via the same method.  It's perfectly legal to
196	  have only one of them active, and there's no ordering requirement on
197	  activating or removing them both.  (This differs from conventional
198	  device-mapper snapshots.)
199	
200	  Activate it exactly the same way as any other thinly-provisioned volume:
201	
202	    dmsetup create snap --table "0 2097152 thin /dev/mapper/pool 1"
203	
204	External snapshots
205	------------------
206	
207	You can use an external _read only_ device as an origin for a
208	thinly-provisioned volume.  Any read to an unprovisioned area of the
209	thin device will be passed through to the origin.  Writes trigger
210	the allocation of new blocks as usual.
211	
212	One use case for this is VM hosts that want to run guests on
213	thinly-provisioned volumes but have the base image on another device
214	(possibly shared between many VMs).
215	
216	You must not write to the origin device if you use this technique!
217	Of course, you may write to the thin device and take internal snapshots
218	of the thin volume.
219	
220	i) Creating a snapshot of an external device
221	
222	  This is the same as creating a thin device.
223	  You don't mention the origin at this stage.
224	
225	    dmsetup message /dev/mapper/pool 0 "create_thin 0"
226	
227	ii) Using a snapshot of an external device.
228	
229	  Append an extra parameter to the thin target specifying the origin:
230	
231	    dmsetup create snap --table "0 2097152 thin /dev/mapper/pool 0 /dev/image"
232	
233	  N.B. All descendants (internal snapshots) of this snapshot require the
234	  same extra origin parameter.
235	
236	Deactivation
237	------------
238	
239	All devices using a pool must be deactivated before the pool itself
240	can be.
241	
242	    dmsetup remove thin
243	    dmsetup remove snap
244	    dmsetup remove pool
245	
246	Reference
247	=========
248	
249	'thin-pool' target
250	------------------
251	
252	i) Constructor
253	
254	    thin-pool <metadata dev> <data dev> <data block size (sectors)> \
255		      <low water mark (blocks)> [<number of feature args> [<arg>]*]
256	
257	    Optional feature arguments:
258	
259	      skip_block_zeroing: Skip the zeroing of newly-provisioned blocks.
260	
261	      ignore_discard: Disable discard support.
262	
263	      no_discard_passdown: Don't pass discards down to the underlying
264				   data device, but just remove the mapping.
265	
266	      read_only: Don't allow any changes to be made to the pool
267			 metadata.
268	
269	      error_if_no_space: Error IOs, instead of queueing, if no space.
270	
271	    Data block size must be between 64KB (128 sectors) and 1GB
272	    (2097152 sectors) inclusive.
273	
274	
275	ii) Status
276	
277	    <transaction id> <used metadata blocks>/<total metadata blocks>
278	    <used data blocks>/<total data blocks> <held metadata root>
279	    ro|rw|out_of_data_space [no_]discard_passdown [error|queue]_if_no_space
280	    needs_check|-
281	
282	    transaction id:
283		A 64-bit number used by userspace to help synchronise with metadata
284		from volume managers.
285	
286	    used data blocks / total data blocks
287		If the number of free blocks drops below the pool's low water mark a
288		dm event will be sent to userspace.  This event is edge-triggered and
289		it will occur only once after each resume so volume manager writers
290		should register for the event and then check the target's status.
291	
292	    held metadata root:
293		The location, in blocks, of the metadata root that has been
294		'held' for userspace read access.  '-' indicates there is no
295		held root.
296	
297	    discard_passdown|no_discard_passdown
298		Whether or not discards are actually being passed down to the
299		underlying device.  When this is enabled when loading the table,
300		it can get disabled if the underlying device doesn't support it.
301	
302	    ro|rw|out_of_data_space
303		If the pool encounters certain types of device failures it will
304		drop into a read-only metadata mode in which no changes to
305		the pool metadata (like allocating new blocks) are permitted.
306	
307		In serious cases where even a read-only mode is deemed unsafe
308		no further I/O will be permitted and the status will just
309		contain the string 'Fail'.  The userspace recovery tools
310		should then be used.
311	
312	    error_if_no_space|queue_if_no_space
313		If the pool runs out of data or metadata space, the pool will
314		either queue or error the IO destined to the data device.  The
315		default is to queue the IO until more space is added or the
316		'no_space_timeout' expires.  The 'no_space_timeout' dm-thin-pool
317		module parameter can be used to change this timeout -- it
318		defaults to 60 seconds but may be disabled using a value of 0.
319	
320	    needs_check
321		A metadata operation has failed, resulting in the needs_check
322		flag being set in the metadata's superblock.  The metadata
323		device must be deactivated and checked/repaired before the
324		thin-pool can be made fully operational again.  '-' indicates
325		needs_check is not set.
326	
327	iii) Messages
328	
329	    create_thin <dev id>
330	
331		Create a new thinly-provisioned device.
332		<dev id> is an arbitrary unique 24-bit identifier chosen by
333		the caller.
334	
335	    create_snap <dev id> <origin id>
336	
337		Create a new snapshot of another thinly-provisioned device.
338		<dev id> is an arbitrary unique 24-bit identifier chosen by
339		the caller.
340		<origin id> is the identifier of the thinly-provisioned device
341		of which the new device will be a snapshot.
342	
343	    delete <dev id>
344	
345		Deletes a thin device.  Irreversible.
346	
347	    set_transaction_id <current id> <new id>
348	
349		Userland volume managers, such as LVM, need a way to
350		synchronise their external metadata with the internal metadata of the
351		pool target.  The thin-pool target offers to store an
352		arbitrary 64-bit transaction id and return it on the target's
353		status line.  To avoid races you must provide what you think
354		the current transaction id is when you change it with this
355		compare-and-swap message.
356	
357	    reserve_metadata_snap
358	
359	        Reserve a copy of the data mapping btree for use by userland.
360	        This allows userland to inspect the mappings as they were when
361	        this message was executed.  Use the pool's status command to
362	        get the root block associated with the metadata snapshot.
363	
364	    release_metadata_snap
365	
366	        Release a previously reserved copy of the data mapping btree.
367	
368	'thin' target
369	-------------
370	
371	i) Constructor
372	
373	    thin <pool dev> <dev id> [<external origin dev>]
374	
375	    pool dev:
376		the thin-pool device, e.g. /dev/mapper/my_pool or 253:0
377	
378	    dev id:
379		the internal device identifier of the device to be
380		activated.
381	
382	    external origin dev:
383		an optional block device outside the pool to be treated as a
384		read-only snapshot origin: reads to unprovisioned areas of the
385		thin target will be mapped to this device.
386	
387	The pool doesn't store any size against the thin devices.  If you
388	load a thin target that is smaller than you've been using previously,
389	then you'll have no access to blocks mapped beyond the end.  If you
390	load a target that is bigger than before, then extra blocks will be
391	provisioned as and when needed.
392	
393	ii) Status
394	
395	     <nr mapped sectors> <highest mapped sector>
396	
397		If the pool has encountered device errors and failed, the status
398		will just contain the string 'Fail'.  The userspace recovery
399		tools should then be used.
400	
401	    In the case where <nr mapped sectors> is 0, there is no highest
402	    mapped sector and the value of <highest mapped sector> is unspecified.
Hide Line Numbers


About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog