About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / vfio-mediated-device.txt




Custom Search

Based on kernel version 4.16.1. Page generated on 2018-04-09 11:53 EST.

1	.. include:: <isonum.txt>
2	
3	=====================
4	VFIO Mediated devices
5	=====================
6	
7	:Copyright: |copy| 2016, NVIDIA CORPORATION. All rights reserved.
8	:Author: Neo Jia <cjia@nvidia.com>
9	:Author: Kirti Wankhede <kwankhede@nvidia.com>
10	
11	This program is free software; you can redistribute it and/or modify
12	it under the terms of the GNU General Public License version 2 as
13	published by the Free Software Foundation.
14	
15	
16	Virtual Function I/O (VFIO) Mediated devices[1]
17	===============================================
18	
19	The number of use cases for virtualizing DMA devices that do not have built-in
20	SR_IOV capability is increasing. Previously, to virtualize such devices,
21	developers had to create their own management interfaces and APIs, and then
22	integrate them with user space software. To simplify integration with user space
23	software, we have identified common requirements and a unified management
24	interface for such devices.
25	
26	The VFIO driver framework provides unified APIs for direct device access. It is
27	an IOMMU/device-agnostic framework for exposing direct device access to user
28	space in a secure, IOMMU-protected environment. This framework is used for
29	multiple devices, such as GPUs, network adapters, and compute accelerators. With
30	direct device access, virtual machines or user space applications have direct
31	access to the physical device. This framework is reused for mediated devices.
32	
33	The mediated core driver provides a common interface for mediated device
34	management that can be used by drivers of different devices. This module
35	provides a generic interface to perform these operations:
36	
37	* Create and destroy a mediated device
38	* Add a mediated device to and remove it from a mediated bus driver
39	* Add a mediated device to and remove it from an IOMMU group
40	
41	The mediated core driver also provides an interface to register a bus driver.
42	For example, the mediated VFIO mdev driver is designed for mediated devices and
43	supports VFIO APIs. The mediated bus driver adds a mediated device to and
44	removes it from a VFIO group.
45	
46	The following high-level block diagram shows the main components and interfaces
47	in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM
48	devices as examples, as these devices are the first devices to use this module::
49	
50	     +---------------+
51	     |               |
52	     | +-----------+ |  mdev_register_driver() +--------------+
53	     | |           | +<------------------------+              |
54	     | |  mdev     | |                         |              |
55	     | |  bus      | +------------------------>+ vfio_mdev.ko |<-> VFIO user
56	     | |  driver   | |     probe()/remove()    |              |    APIs
57	     | |           | |                         +--------------+
58	     | +-----------+ |
59	     |               |
60	     |  MDEV CORE    |
61	     |   MODULE      |
62	     |   mdev.ko     |
63	     | +-----------+ |  mdev_register_device() +--------------+
64	     | |           | +<------------------------+              |
65	     | |           | |                         |  nvidia.ko   |<-> physical
66	     | |           | +------------------------>+              |    device
67	     | |           | |        callbacks        +--------------+
68	     | | Physical  | |
69	     | |  device   | |  mdev_register_device() +--------------+
70	     | | interface | |<------------------------+              |
71	     | |           | |                         |  i915.ko     |<-> physical
72	     | |           | +------------------------>+              |    device
73	     | |           | |        callbacks        +--------------+
74	     | |           | |
75	     | |           | |  mdev_register_device() +--------------+
76	     | |           | +<------------------------+              |
77	     | |           | |                         | ccw_device.ko|<-> physical
78	     | |           | +------------------------>+              |    device
79	     | |           | |        callbacks        +--------------+
80	     | +-----------+ |
81	     +---------------+
82	
83	
84	Registration Interfaces
85	=======================
86	
87	The mediated core driver provides the following types of registration
88	interfaces:
89	
90	* Registration interface for a mediated bus driver
91	* Physical device driver interface
92	
93	Registration Interface for a Mediated Bus Driver
94	------------------------------------------------
95	
96	The registration interface for a mediated bus driver provides the following
97	structure to represent a mediated device's driver::
98	
99	     /*
100	      * struct mdev_driver [2] - Mediated device's driver
101	      * @name: driver name
102	      * @probe: called when new device created
103	      * @remove: called when device removed
104	      * @driver: device driver structure
105	      */
106	     struct mdev_driver {
107		     const char *name;
108		     int  (*probe)  (struct device *dev);
109		     void (*remove) (struct device *dev);
110		     struct device_driver    driver;
111	     };
112	
113	A mediated bus driver for mdev should use this structure in the function calls
114	to register and unregister itself with the core driver:
115	
116	* Register::
117	
118	    extern int  mdev_register_driver(struct mdev_driver *drv,
119					   struct module *owner);
120	
121	* Unregister::
122	
123	    extern void mdev_unregister_driver(struct mdev_driver *drv);
124	
125	The mediated bus driver is responsible for adding mediated devices to the VFIO
126	group when devices are bound to the driver and removing mediated devices from
127	the VFIO when devices are unbound from the driver.
128	
129	
130	Physical Device Driver Interface
131	--------------------------------
132	
133	The physical device driver interface provides the mdev_parent_ops[3] structure
134	to define the APIs to manage work in the mediated core driver that is related
135	to the physical device.
136	
137	The structures in the mdev_parent_ops structure are as follows:
138	
139	* dev_attr_groups: attributes of the parent device
140	* mdev_attr_groups: attributes of the mediated device
141	* supported_config: attributes to define supported configurations
142	
143	The functions in the mdev_parent_ops structure are as follows:
144	
145	* create: allocate basic resources in a driver for a mediated device
146	* remove: free resources in a driver when a mediated device is destroyed
147	
148	The callbacks in the mdev_parent_ops structure are as follows:
149	
150	* open: open callback of mediated device
151	* close: close callback of mediated device
152	* ioctl: ioctl callback of mediated device
153	* read : read emulation callback
154	* write: write emulation callback
155	* mmap: mmap emulation callback
156	
157	A driver should use the mdev_parent_ops structure in the function call to
158	register itself with the mdev core driver::
159	
160		extern int  mdev_register_device(struct device *dev,
161		                                 const struct mdev_parent_ops *ops);
162	
163	However, the mdev_parent_ops structure is not required in the function call
164	that a driver should use to unregister itself with the mdev core driver::
165	
166		extern void mdev_unregister_device(struct device *dev);
167	
168	
169	Mediated Device Management Interface Through sysfs
170	==================================================
171	
172	The management interface through sysfs enables user space software, such as
173	libvirt, to query and configure mediated devices in a hardware-agnostic fashion.
174	This management interface provides flexibility to the underlying physical
175	device's driver to support features such as:
176	
177	* Mediated device hot plug
178	* Multiple mediated devices in a single virtual machine
179	* Multiple mediated devices from different physical devices
180	
181	Links in the mdev_bus Class Directory
182	-------------------------------------
183	The /sys/class/mdev_bus/ directory contains links to devices that are registered
184	with the mdev core driver.
185	
186	Directories and files under the sysfs for Each Physical Device
187	--------------------------------------------------------------
188	
189	::
190	
191	  |- [parent physical device]
192	  |--- Vendor-specific-attributes [optional]
193	  |--- [mdev_supported_types]
194	  |     |--- [<type-id>]
195	  |     |   |--- create
196	  |     |   |--- name
197	  |     |   |--- available_instances
198	  |     |   |--- device_api
199	  |     |   |--- description
200	  |     |   |--- [devices]
201	  |     |--- [<type-id>]
202	  |     |   |--- create
203	  |     |   |--- name
204	  |     |   |--- available_instances
205	  |     |   |--- device_api
206	  |     |   |--- description
207	  |     |   |--- [devices]
208	  |     |--- [<type-id>]
209	  |          |--- create
210	  |          |--- name
211	  |          |--- available_instances
212	  |          |--- device_api
213	  |          |--- description
214	  |          |--- [devices]
215	
216	* [mdev_supported_types]
217	
218	  The list of currently supported mediated device types and their details.
219	
220	  [<type-id>], device_api, and available_instances are mandatory attributes
221	  that should be provided by vendor driver.
222	
223	* [<type-id>]
224	
225	  The [<type-id>] name is created by adding the device driver string as a prefix
226	  to the string provided by the vendor driver. This format of this name is as
227	  follows::
228	
229		sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name);
230	
231	  (or using mdev_parent_dev(mdev) to arrive at the parent device outside
232	  of the core mdev code)
233	
234	* device_api
235	
236	  This attribute should show which device API is being created, for example,
237	  "vfio-pci" for a PCI device.
238	
239	* available_instances
240	
241	  This attribute should show the number of devices of type <type-id> that can be
242	  created.
243	
244	* [device]
245	
246	  This directory contains links to the devices of type <type-id> that have been
247	  created.
248	
249	* name
250	
251	  This attribute should show human readable name. This is optional attribute.
252	
253	* description
254	
255	  This attribute should show brief features/description of the type. This is
256	  optional attribute.
257	
258	Directories and Files Under the sysfs for Each mdev Device
259	----------------------------------------------------------
260	
261	::
262	
263	  |- [parent phy device]
264	  |--- [$MDEV_UUID]
265	         |--- remove
266	         |--- mdev_type {link to its type}
267	         |--- vendor-specific-attributes [optional]
268	
269	* remove (write only)
270	
271	Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can
272	fail the remove() callback if that device is active and the vendor driver
273	doesn't support hot unplug.
274	
275	Example::
276	
277		# echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove
278	
279	Mediated device Hot plug
280	------------------------
281	
282	Mediated devices can be created and assigned at runtime. The procedure to hot
283	plug a mediated device is the same as the procedure to hot plug a PCI device.
284	
285	Translation APIs for Mediated Devices
286	=====================================
287	
288	The following APIs are provided for translating user pfn to host pfn in a VFIO
289	driver::
290	
291		extern int vfio_pin_pages(struct device *dev, unsigned long *user_pfn,
292					  int npage, int prot, unsigned long *phys_pfn);
293	
294		extern int vfio_unpin_pages(struct device *dev, unsigned long *user_pfn,
295					    int npage);
296	
297	These functions call back into the back-end IOMMU module by using the pin_pages
298	and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently
299	these callbacks are supported in the TYPE1 IOMMU module. To enable them for
300	other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide
301	these two callback functions.
302	
303	Using the Sample Code
304	=====================
305	
306	mtty.c in samples/vfio-mdev/ directory is a sample driver program to
307	demonstrate how to use the mediated device framework.
308	
309	The sample driver creates an mdev device that simulates a serial port over a PCI
310	card.
311	
312	1. Build and load the mtty.ko module.
313	
314	   This step creates a dummy device, /sys/devices/virtual/mtty/mtty/
315	
316	   Files in this device directory in sysfs are similar to the following::
317	
318	     # tree /sys/devices/virtual/mtty/mtty/
319	        /sys/devices/virtual/mtty/mtty/
320	        |-- mdev_supported_types
321	        |   |-- mtty-1
322	        |   |   |-- available_instances
323	        |   |   |-- create
324	        |   |   |-- device_api
325	        |   |   |-- devices
326	        |   |   `-- name
327	        |   `-- mtty-2
328	        |       |-- available_instances
329	        |       |-- create
330	        |       |-- device_api
331	        |       |-- devices
332	        |       `-- name
333	        |-- mtty_dev
334	        |   `-- sample_mtty_dev
335	        |-- power
336	        |   |-- autosuspend_delay_ms
337	        |   |-- control
338	        |   |-- runtime_active_time
339	        |   |-- runtime_status
340	        |   `-- runtime_suspended_time
341	        |-- subsystem -> ../../../../class/mtty
342	        `-- uevent
343	
344	2. Create a mediated device by using the dummy device that you created in the
345	   previous step::
346	
347	     # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" >	\
348	              /sys/devices/virtual/mtty/mtty/mdev_supported_types/mtty-2/create
349	
350	3. Add parameters to qemu-kvm::
351	
352	     -device vfio-pci,\
353	      sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001
354	
355	4. Boot the VM.
356	
357	   In the Linux guest VM, with no hardware on the host, the device appears
358	   as  follows::
359	
360	     # lspci -s 00:05.0 -xxvv
361	     00:05.0 Serial controller: Device 4348:3253 (rev 10) (prog-if 02 [16550])
362	             Subsystem: Device 4348:3253
363	             Physical Slot: 5
364	             Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
365	     Stepping- SERR- FastB2B- DisINTx-
366	             Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
367	     <TAbort- <MAbort- >SERR- <PERR- INTx-
368	             Interrupt: pin A routed to IRQ 10
369	             Region 0: I/O ports at c150 [size=8]
370	             Region 1: I/O ports at c158 [size=8]
371	             Kernel driver in use: serial
372	     00: 48 43 53 32 01 00 00 02 10 02 00 07 00 00 00 00
373	     10: 51 c1 00 00 59 c1 00 00 00 00 00 00 00 00 00 00
374	     20: 00 00 00 00 00 00 00 00 00 00 00 00 48 43 53 32
375	     30: 00 00 00 00 00 00 00 00 00 00 00 00 0a 01 00 00
376	
377	     In the Linux guest VM, dmesg output for the device is as follows:
378	
379	     serial 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10
380	     0000:00:05.0: ttyS1 at I/O 0xc150 (irq = 10) is a 16550A
381	     0000:00:05.0: ttyS2 at I/O 0xc158 (irq = 10) is a 16550A
382	
383	
384	5. In the Linux guest VM, check the serial ports::
385	
386	     # setserial -g /dev/ttyS*
387	     /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4
388	     /dev/ttyS1, UART: 16550A, Port: 0xc150, IRQ: 10
389	     /dev/ttyS2, UART: 16550A, Port: 0xc158, IRQ: 10
390	
391	6. Using minicom or any terminal emulation program, open port /dev/ttyS1 or
392	   /dev/ttyS2 with hardware flow control disabled.
393	
394	7. Type data on the minicom terminal or send data to the terminal emulation
395	   program and read the data.
396	
397	   Data is loop backed from hosts mtty driver.
398	
399	8. Destroy the mediated device that you created::
400	
401	     # echo 1 > /sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001/remove
402	
403	References
404	==========
405	
406	1. See Documentation/vfio.txt for more information on VFIO.
407	2. struct mdev_driver in include/linux/mdev.h
408	3. struct mdev_parent_ops in include/linux/mdev.h
409	4. struct vfio_iommu_driver_ops in include/linux/vfio.h
Hide Line Numbers
About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Information is copyright its respective author. All material is available from the Linux Kernel Source distributed under a GPL License. This page is provided as a free service by mjmwired.net.