About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / block / data-integrity.txt


Based on kernel version 4.16.1. Page generated on 2018-04-09 11:52 EST.

1	----------------------------------------------------------------------
2	1. INTRODUCTION
3	
4	Modern filesystems feature checksumming of data and metadata to
5	protect against data corruption.  However, the detection of the
6	corruption is done at read time which could potentially be months
7	after the data was written.  At that point the original data that the
8	application tried to write is most likely lost.
9	
10	The solution is to ensure that the disk is actually storing what the
11	application meant it to.  Recent additions to both the SCSI family
12	protocols (SBC Data Integrity Field, SCC protection proposal) as well
13	as SATA/T13 (External Path Protection) try to remedy this by adding
14	support for appending integrity metadata to an I/O.  The integrity
15	metadata (or protection information in SCSI terminology) includes a
16	checksum for each sector as well as an incrementing counter that
17	ensures the individual sectors are written in the right order.  And
18	for some protection schemes also that the I/O is written to the right
19	place on disk.
20	
21	Current storage controllers and devices implement various protective
22	measures, for instance checksumming and scrubbing.  But these
23	technologies are working in their own isolated domains or at best
24	between adjacent nodes in the I/O path.  The interesting thing about
25	DIF and the other integrity extensions is that the protection format
26	is well defined and every node in the I/O path can verify the
27	integrity of the I/O and reject it if corruption is detected.  This
28	allows not only corruption prevention but also isolation of the point
29	of failure.
30	
31	----------------------------------------------------------------------
32	2. THE DATA INTEGRITY EXTENSIONS
33	
34	As written, the protocol extensions only protect the path between
35	controller and storage device.  However, many controllers actually
36	allow the operating system to interact with the integrity metadata
37	(IMD).  We have been working with several FC/SAS HBA vendors to enable
38	the protection information to be transferred to and from their
39	controllers.
40	
41	The SCSI Data Integrity Field works by appending 8 bytes of protection
42	information to each sector.  The data + integrity metadata is stored
43	in 520 byte sectors on disk.  Data + IMD are interleaved when
44	transferred between the controller and target.  The T13 proposal is
45	similar.
46	
47	Because it is highly inconvenient for operating systems to deal with
48	520 (and 4104) byte sectors, we approached several HBA vendors and
49	encouraged them to allow separation of the data and integrity metadata
50	scatter-gather lists.
51	
52	The controller will interleave the buffers on write and split them on
53	read.  This means that Linux can DMA the data buffers to and from
54	host memory without changes to the page cache.
55	
56	Also, the 16-bit CRC checksum mandated by both the SCSI and SATA specs
57	is somewhat heavy to compute in software.  Benchmarks found that
58	calculating this checksum had a significant impact on system
59	performance for a number of workloads.  Some controllers allow a
60	lighter-weight checksum to be used when interfacing with the operating
61	system.  Emulex, for instance, supports the TCP/IP checksum instead.
62	The IP checksum received from the OS is converted to the 16-bit CRC
63	when writing and vice versa.  This allows the integrity metadata to be
64	generated by Linux or the application at very low cost (comparable to
65	software RAID5).
66	
67	The IP checksum is weaker than the CRC in terms of detecting bit
68	errors.  However, the strength is really in the separation of the data
69	buffers and the integrity metadata.  These two distinct buffers must
70	match up for an I/O to complete.
71	
72	The separation of the data and integrity metadata buffers as well as
73	the choice in checksums is referred to as the Data Integrity
74	Extensions.  As these extensions are outside the scope of the protocol
75	bodies (T10, T13), Oracle and its partners are trying to standardize
76	them within the Storage Networking Industry Association.
77	
78	----------------------------------------------------------------------
79	3. KERNEL CHANGES
80	
81	The data integrity framework in Linux enables protection information
82	to be pinned to I/Os and sent to/received from controllers that
83	support it.
84	
85	The advantage to the integrity extensions in SCSI and SATA is that
86	they enable us to protect the entire path from application to storage
87	device.  However, at the same time this is also the biggest
88	disadvantage. It means that the protection information must be in a
89	format that can be understood by the disk.
90	
91	Generally Linux/POSIX applications are agnostic to the intricacies of
92	the storage devices they are accessing.  The virtual filesystem switch
93	and the block layer make things like hardware sector size and
94	transport protocols completely transparent to the application.
95	
96	However, this level of detail is required when preparing the
97	protection information to send to a disk.  Consequently, the very
98	concept of an end-to-end protection scheme is a layering violation.
99	It is completely unreasonable for an application to be aware whether
100	it is accessing a SCSI or SATA disk.
101	
102	The data integrity support implemented in Linux attempts to hide this
103	from the application.  As far as the application (and to some extent
104	the kernel) is concerned, the integrity metadata is opaque information
105	that's attached to the I/O.
106	
107	The current implementation allows the block layer to automatically
108	generate the protection information for any I/O.  Eventually the
109	intent is to move the integrity metadata calculation to userspace for
110	user data.  Metadata and other I/O that originates within the kernel
111	will still use the automatic generation interface.
112	
113	Some storage devices allow each hardware sector to be tagged with a
114	16-bit value.  The owner of this tag space is the owner of the block
115	device.  I.e. the filesystem in most cases.  The filesystem can use
116	this extra space to tag sectors as they see fit.  Because the tag
117	space is limited, the block interface allows tagging bigger chunks by
118	way of interleaving.  This way, 8*16 bits of information can be
119	attached to a typical 4KB filesystem block.
120	
121	This also means that applications such as fsck and mkfs will need
122	access to manipulate the tags from user space.  A passthrough
123	interface for this is being worked on.
124	
125	
126	----------------------------------------------------------------------
127	4. BLOCK LAYER IMPLEMENTATION DETAILS
128	
129	4.1 BIO
130	
131	The data integrity patches add a new field to struct bio when
132	CONFIG_BLK_DEV_INTEGRITY is enabled.  bio_integrity(bio) returns a
133	pointer to a struct bip which contains the bio integrity payload.
134	Essentially a bip is a trimmed down struct bio which holds a bio_vec
135	containing the integrity metadata and the required housekeeping
136	information (bvec pool, vector count, etc.)
137	
138	A kernel subsystem can enable data integrity protection on a bio by
139	calling bio_integrity_alloc(bio).  This will allocate and attach the
140	bip to the bio.
141	
142	Individual pages containing integrity metadata can subsequently be
143	attached using bio_integrity_add_page().
144	
145	bio_free() will automatically free the bip.
146	
147	
148	4.2 BLOCK DEVICE
149	
150	Because the format of the protection data is tied to the physical
151	disk, each block device has been extended with a block integrity
152	profile (struct blk_integrity).  This optional profile is registered
153	with the block layer using blk_integrity_register().
154	
155	The profile contains callback functions for generating and verifying
156	the protection data, as well as getting and setting application tags.
157	The profile also contains a few constants to aid in completing,
158	merging and splitting the integrity metadata.
159	
160	Layered block devices will need to pick a profile that's appropriate
161	for all subdevices.  blk_integrity_compare() can help with that.  DM
162	and MD linear, RAID0 and RAID1 are currently supported.  RAID4/5/6
163	will require extra work due to the application tag.
164	
165	
166	----------------------------------------------------------------------
167	5.0 BLOCK LAYER INTEGRITY API
168	
169	5.1 NORMAL FILESYSTEM
170	
171	    The normal filesystem is unaware that the underlying block device
172	    is capable of sending/receiving integrity metadata.  The IMD will
173	    be automatically generated by the block layer at submit_bio() time
174	    in case of a WRITE.  A READ request will cause the I/O integrity
175	    to be verified upon completion.
176	
177	    IMD generation and verification can be toggled using the
178	
179	      /sys/block/<bdev>/integrity/write_generate
180	
181	    and
182	
183	      /sys/block/<bdev>/integrity/read_verify
184	
185	    flags.
186	
187	
188	5.2 INTEGRITY-AWARE FILESYSTEM
189	
190	    A filesystem that is integrity-aware can prepare I/Os with IMD
191	    attached.  It can also use the application tag space if this is
192	    supported by the block device.
193	
194	
195	    bool bio_integrity_prep(bio);
196	
197	      To generate IMD for WRITE and to set up buffers for READ, the
198	      filesystem must call bio_integrity_prep(bio).
199	
200	      Prior to calling this function, the bio data direction and start
201	      sector must be set, and the bio should have all data pages
202	      added.  It is up to the caller to ensure that the bio does not
203	      change while I/O is in progress.
204	      Complete bio with error if prepare failed for some reson.
205	
206	
207	5.3 PASSING EXISTING INTEGRITY METADATA
208	
209	    Filesystems that either generate their own integrity metadata or
210	    are capable of transferring IMD from user space can use the
211	    following calls:
212	
213	
214	    struct bip * bio_integrity_alloc(bio, gfp_mask, nr_pages);
215	
216	      Allocates the bio integrity payload and hangs it off of the bio.
217	      nr_pages indicate how many pages of protection data need to be
218	      stored in the integrity bio_vec list (similar to bio_alloc()).
219	
220	      The integrity payload will be freed at bio_free() time.
221	
222	
223	    int bio_integrity_add_page(bio, page, len, offset);
224	
225	      Attaches a page containing integrity metadata to an existing
226	      bio.  The bio must have an existing bip,
227	      i.e. bio_integrity_alloc() must have been called.  For a WRITE,
228	      the integrity metadata in the pages must be in a format
229	      understood by the target device with the notable exception that
230	      the sector numbers will be remapped as the request traverses the
231	      I/O stack.  This implies that the pages added using this call
232	      will be modified during I/O!  The first reference tag in the
233	      integrity metadata must have a value of bip->bip_sector.
234	
235	      Pages can be added using bio_integrity_add_page() as long as
236	      there is room in the bip bio_vec array (nr_pages).
237	
238	      Upon completion of a READ operation, the attached pages will
239	      contain the integrity metadata received from the storage device.
240	      It is up to the receiver to process them and verify data
241	      integrity upon completion.
242	
243	
244	5.4 REGISTERING A BLOCK DEVICE AS CAPABLE OF EXCHANGING INTEGRITY
245	    METADATA
246	
247	    To enable integrity exchange on a block device the gendisk must be
248	    registered as capable:
249	
250	    int blk_integrity_register(gendisk, blk_integrity);
251	
252	      The blk_integrity struct is a template and should contain the
253	      following:
254	
255	        static struct blk_integrity my_profile = {
256	            .name                   = "STANDARDSBODY-TYPE-VARIANT-CSUM",
257	            .generate_fn            = my_generate_fn,
258	       	    .verify_fn              = my_verify_fn,
259		    .tuple_size             = sizeof(struct my_tuple_size),
260		    .tag_size               = <tag bytes per hw sector>,
261	        };
262	
263	      'name' is a text string which will be visible in sysfs.  This is
264	      part of the userland API so chose it carefully and never change
265	      it.  The format is standards body-type-variant.
266	      E.g. T10-DIF-TYPE1-IP or T13-EPP-0-CRC.
267	
268	      'generate_fn' generates appropriate integrity metadata (for WRITE).
269	
270	      'verify_fn' verifies that the data buffer matches the integrity
271	      metadata.
272	
273	      'tuple_size' must be set to match the size of the integrity
274	      metadata per sector.  I.e. 8 for DIF and EPP.
275	
276	      'tag_size' must be set to identify how many bytes of tag space
277	      are available per hardware sector.  For DIF this is either 2 or
278	      0 depending on the value of the Control Mode Page ATO bit.
279	
280	----------------------------------------------------------------------
281	2007-12-24 Martin K. Petersen <martin.petersen@oracle.com>
Hide Line Numbers


About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog