About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / MSI-HOWTO.txt


Based on kernel version 2.6.27. Page generated on 2008-10-13 09:53 EST.

1			The MSI Driver Guide HOWTO
2		Tom L Nguyen tom.l.nguyen[AT]intel[DOT]com
3				10/03/2003
4		Revised Feb 12, 2004 by Martine Silbermann
5			email: Martine.Silbermann[AT]hp[DOT]com
6		Revised Jun 25, 2004 by Tom L Nguyen
7	
8	1. About this guide
9	
10	This guide describes the basics of Message Signaled Interrupts (MSI),
11	the advantages of using MSI over traditional interrupt mechanisms,
12	and how to enable your driver to use MSI or MSI-X. Also included is
13	a Frequently Asked Questions (FAQ) section.
14	
15	1.1 Terminology
16	
17	PCI devices can be single-function or multi-function.  In either case,
18	when this text talks about enabling or disabling MSI on a "device
19	function," it is referring to one specific PCI device and function and
20	not to all functions on a PCI device (unless the PCI device has only
21	one function).
22	
23	2. Copyright 2003 Intel Corporation
24	
25	3. What is MSI/MSI-X?
26	
27	Message Signaled Interrupt (MSI), as described in the PCI Local Bus
28	Specification Revision 2.3 or later, is an optional feature, and a
29	required feature for PCI Express devices. MSI enables a device function
30	to request service by sending an Inbound Memory Write on its PCI bus to
31	the FSB as a Message Signal Interrupt transaction. Because MSI is
32	generated in the form of a Memory Write, all transaction conditions,
33	such as a Retry, Master-Abort, Target-Abort or normal completion, are
34	supported.
35	
36	A PCI device that supports MSI must also support pin IRQ assertion
37	interrupt mechanism to provide backward compatibility for systems that
38	do not support MSI. In systems which support MSI, the bus driver is
39	responsible for initializing the message address and message data of
40	the device function's MSI/MSI-X capability structure during device
41	initial configuration.
42	
43	An MSI capable device function indicates MSI support by implementing
44	the MSI/MSI-X capability structure in its PCI capability list. The
45	device function may implement both the MSI capability structure and
46	the MSI-X capability structure; however, the bus driver should not
47	enable both.
48	
49	The MSI capability structure contains Message Control register,
50	Message Address register and Message Data register. These registers
51	provide the bus driver control over MSI. The Message Control register
52	indicates the MSI capability supported by the device. The Message
53	Address register specifies the target address and the Message Data
54	register specifies the characteristics of the message. To request
55	service, the device function writes the content of the Message Data
56	register to the target address. The device and its software driver
57	are prohibited from writing to these registers.
58	
59	The MSI-X capability structure is an optional extension to MSI. It
60	uses an independent and separate capability structure. There are
61	some key advantages to implementing the MSI-X capability structure
62	over the MSI capability structure as described below.
63	
64		- Support a larger maximum number of vectors per function.
65	
66		- Provide the ability for system software to configure
67		each vector with an independent message address and message
68		data, specified by a table that resides in Memory Space.
69	
70	        - MSI and MSI-X both support per-vector masking. Per-vector
71		masking is an optional extension of MSI but a required
72		feature for MSI-X. Per-vector masking provides the kernel the
73		ability to mask/unmask a single MSI while running its
74		interrupt service routine. If per-vector masking is
75		not supported, then the device driver should provide the
76		hardware/software synchronization to ensure that the device
77		generates MSI when the driver wants it to do so.
78	
79	4. Why use MSI?
80	
81	As a benefit to the simplification of board design, MSI allows board
82	designers to remove out-of-band interrupt routing. MSI is another
83	step towards a legacy-free environment.
84	
85	Due to increasing pressure on chipset and processor packages to
86	reduce pin count, the need for interrupt pins is expected to
87	diminish over time. Devices, due to pin constraints, may implement
88	messages to increase performance.
89	
90	PCI Express endpoints uses INTx emulation (in-band messages) instead
91	of IRQ pin assertion. Using INTx emulation requires interrupt
92	sharing among devices connected to the same node (PCI bridge) while
93	MSI is unique (non-shared) and does not require BIOS configuration
94	support. As a result, the PCI Express technology requires MSI
95	support for better interrupt performance.
96	
97	Using MSI enables the device functions to support two or more
98	vectors, which can be configured to target different CPUs to
99	increase scalability.
100	
101	5. Configuring a driver to use MSI/MSI-X
102	
103	By default, the kernel will not enable MSI/MSI-X on all devices that
104	support this capability. The CONFIG_PCI_MSI kernel option
105	must be selected to enable MSI/MSI-X support.
106	
107	5.1 Including MSI/MSI-X support into the kernel
108	
109	To allow MSI/MSI-X capable device drivers to selectively enable
110	MSI/MSI-X (using pci_enable_msi()/pci_enable_msix() as described
111	below), the VECTOR based scheme needs to be enabled by setting
112	CONFIG_PCI_MSI during kernel config.
113	
114	Since the target of the inbound message is the local APIC, providing
115	CONFIG_X86_LOCAL_APIC must be enabled as well as CONFIG_PCI_MSI.
116	
117	5.2 Configuring for MSI support
118	
119	Due to the non-contiguous fashion in vector assignment of the
120	existing Linux kernel, this version does not support multiple
121	messages regardless of a device function is capable of supporting
122	more than one vector. To enable MSI on a device function's MSI
123	capability structure requires a device driver to call the function
124	pci_enable_msi() explicitly.
125	
126	5.2.1 API pci_enable_msi
127	
128	int pci_enable_msi(struct pci_dev *dev)
129	
130	With this new API, a device driver that wants to have MSI
131	enabled on its device function must call this API to enable MSI.
132	A successful call will initialize the MSI capability structure
133	with ONE vector, regardless of whether a device function is
134	capable of supporting multiple messages. This vector replaces the
135	pre-assigned dev->irq with a new MSI vector. To avoid a conflict
136	of the new assigned vector with existing pre-assigned vector requires
137	a device driver to call this API before calling request_irq().
138	
139	5.2.2 API pci_disable_msi
140	
141	void pci_disable_msi(struct pci_dev *dev)
142	
143	This API should always be used to undo the effect of pci_enable_msi()
144	when a device driver is unloading. This API restores dev->irq with
145	the pre-assigned IOAPIC vector and switches a device's interrupt
146	mode to PCI pin-irq assertion/INTx emulation mode.
147	
148	Note that a device driver should always call free_irq() on the MSI vector
149	that it has done request_irq() on before calling this API. Failure to do
150	so results in a BUG_ON() and a device will be left with MSI enabled and
151	leaks its vector.
152	
153	5.2.3 MSI mode vs. legacy mode diagram
154	
155	The below diagram shows the events which switch the interrupt
156	mode on the MSI-capable device function between MSI mode and
157	PIN-IRQ assertion mode.
158	
159		 ------------   pci_enable_msi 	 ------------------------
160		|	     | <===============	| 			 |
161		| MSI MODE   |	  	     	| PIN-IRQ ASSERTION MODE |
162		| 	     | ===============>	|			 |
163	 	 ------------	pci_disable_msi  ------------------------
164	
165	
166	Figure 1. MSI Mode vs. Legacy Mode
167	
168	In Figure 1, a device operates by default in legacy mode. Legacy
169	in this context means PCI pin-irq assertion or PCI-Express INTx
170	emulation. A successful MSI request (using pci_enable_msi()) switches
171	a device's interrupt mode to MSI mode. A pre-assigned IOAPIC vector
172	stored in dev->irq will be saved by the PCI subsystem and a new
173	assigned MSI vector will replace dev->irq.
174	
175	To return back to its default mode, a device driver should always call
176	pci_disable_msi() to undo the effect of pci_enable_msi(). Note that a
177	device driver should always call free_irq() on the MSI vector it has
178	done request_irq() on before calling pci_disable_msi(). Failure to do
179	so results in a BUG_ON() and a device will be left with MSI enabled and
180	leaks its vector. Otherwise, the PCI subsystem restores a device's
181	dev->irq with a pre-assigned IOAPIC vector and marks the released
182	MSI vector as unused.
183	
184	Once being marked as unused, there is no guarantee that the PCI
185	subsystem will reserve this MSI vector for a device. Depending on
186	the availability of current PCI vector resources and the number of
187	MSI/MSI-X requests from other drivers, this MSI may be re-assigned.
188	
189	For the case where the PCI subsystem re-assigns this MSI vector to
190	another driver, a request to switch back to MSI mode may result
191	in being assigned a different MSI vector or a failure if no more
192	vectors are available.
193	
194	5.3 Configuring for MSI-X support
195	
196	Due to the ability of the system software to configure each vector of
197	the MSI-X capability structure with an independent message address
198	and message data, the non-contiguous fashion in vector assignment of
199	the existing Linux kernel has no impact on supporting multiple
200	messages on an MSI-X capable device functions. To enable MSI-X on
201	a device function's MSI-X capability structure requires its device
202	driver to call the function pci_enable_msix() explicitly.
203	
204	The function pci_enable_msix(), once invoked, enables either
205	all or nothing, depending on the current availability of PCI vector
206	resources. If the PCI vector resources are available for the number
207	of vectors requested by a device driver, this function will configure
208	the MSI-X table of the MSI-X capability structure of a device with
209	requested messages. To emphasize this reason, for example, a device
210	may be capable for supporting the maximum of 32 vectors while its
211	software driver usually may request 4 vectors. It is recommended
212	that the device driver should call this function once during the
213	initialization phase of the device driver.
214	
215	Unlike the function pci_enable_msi(), the function pci_enable_msix()
216	does not replace the pre-assigned IOAPIC dev->irq with a new MSI
217	vector because the PCI subsystem writes the 1:1 vector-to-entry mapping
218	into the field vector of each element contained in a second argument.
219	Note that the pre-assigned IOAPIC dev->irq is valid only if the device
220	operates in PIN-IRQ assertion mode. In MSI-X mode, any attempt at
221	using dev->irq by the device driver to request for interrupt service
222	may result in unpredictable behavior.
223	
224	For each MSI-X vector granted, a device driver is responsible for calling
225	other functions like request_irq(), enable_irq(), etc. to enable
226	this vector with its corresponding interrupt service handler. It is
227	a device driver's choice to assign all vectors with the same
228	interrupt service handler or each vector with a unique interrupt
229	service handler.
230	
231	5.3.1 Handling MMIO address space of MSI-X Table
232	
233	The PCI 3.0 specification has implementation notes that MMIO address
234	space for a device's MSI-X structure should be isolated so that the
235	software system can set different pages for controlling accesses to the
236	MSI-X structure. The implementation of MSI support requires the PCI
237	subsystem, not a device driver, to maintain full control of the MSI-X
238	table/MSI-X PBA (Pending Bit Array) and MMIO address space of the MSI-X
239	table/MSI-X PBA.  A device driver is prohibited from requesting the MMIO
240	address space of the MSI-X table/MSI-X PBA. Otherwise, the PCI subsystem
241	will fail enabling MSI-X on its hardware device when it calls the function
242	pci_enable_msix().
243	
244	5.3.2 API pci_enable_msix
245	
246	int pci_enable_msix(struct pci_dev *dev, struct msix_entry *entries, int nvec)
247	
248	This API enables a device driver to request the PCI subsystem
249	to enable MSI-X messages on its hardware device. Depending on
250	the availability of PCI vectors resources, the PCI subsystem enables
251	either all or none of the requested vectors.
252	
253	Argument 'dev' points to the device (pci_dev) structure.
254	
255	Argument 'entries' is a pointer to an array of msix_entry structs.
256	The number of entries is indicated in argument 'nvec'.
257	struct msix_entry is defined in /driver/pci/msi.h:
258	
259	struct msix_entry {
260		u16 	vector; /* kernel uses to write alloc vector */
261		u16	entry; /* driver uses to specify entry */
262	};
263	
264	A device driver is responsible for initializing the field 'entry' of
265	each element with a unique entry supported by MSI-X table. Otherwise,
266	-EINVAL will be returned as a result. A successful return of zero
267	indicates the PCI subsystem completed initializing each of the requested
268	entries of the MSI-X table with message address and message data.
269	Last but not least, the PCI subsystem will write the 1:1
270	vector-to-entry mapping into the field 'vector' of each element. A
271	device driver is responsible for keeping track of allocated MSI-X
272	vectors in its internal data structure.
273	
274	A return of zero indicates that the number of MSI-X vectors was
275	successfully allocated. A return of greater than zero indicates
276	MSI-X vector shortage. Or a return of less than zero indicates
277	a failure. This failure may be a result of duplicate entries
278	specified in second argument, or a result of no available vector,
279	or a result of failing to initialize MSI-X table entries.
280	
281	5.3.3 API pci_disable_msix
282	
283	void pci_disable_msix(struct pci_dev *dev)
284	
285	This API should always be used to undo the effect of pci_enable_msix()
286	when a device driver is unloading. Note that a device driver should
287	always call free_irq() on all MSI-X vectors it has done request_irq()
288	on before calling this API. Failure to do so results in a BUG_ON() and
289	a device will be left with MSI-X enabled and leaks its vectors.
290	
291	5.3.4 MSI-X mode vs. legacy mode diagram
292	
293	The below diagram shows the events which switch the interrupt
294	mode on the MSI-X capable device function between MSI-X mode and
295	PIN-IRQ assertion mode (legacy).
296	
297		 ------------   pci_enable_msix(,,n) ------------------------
298		|	     | <===============	    | 			     |
299		| MSI-X MODE |	  	     	    | PIN-IRQ ASSERTION MODE |
300		| 	     | ===============>	    |			     |
301	 	 ------------	pci_disable_msix     ------------------------
302	
303	Figure 2. MSI-X Mode vs. Legacy Mode
304	
305	In Figure 2, a device operates by default in legacy mode. A
306	successful MSI-X request (using pci_enable_msix()) switches a
307	device's interrupt mode to MSI-X mode. A pre-assigned IOAPIC vector
308	stored in dev->irq will be saved by the PCI subsystem; however,
309	unlike MSI mode, the PCI subsystem will not replace dev->irq with
310	assigned MSI-X vector because the PCI subsystem already writes the 1:1
311	vector-to-entry mapping into the field 'vector' of each element
312	specified in second argument.
313	
314	To return back to its default mode, a device driver should always call
315	pci_disable_msix() to undo the effect of pci_enable_msix(). Note that
316	a device driver should always call free_irq() on all MSI-X vectors it
317	has done request_irq() on before calling pci_disable_msix(). Failure
318	to do so results in a BUG_ON() and a device will be left with MSI-X
319	enabled and leaks its vectors. Otherwise, the PCI subsystem switches a
320	device function's interrupt mode from MSI-X mode to legacy mode and
321	marks all allocated MSI-X vectors as unused.
322	
323	Once being marked as unused, there is no guarantee that the PCI
324	subsystem will reserve these MSI-X vectors for a device. Depending on
325	the availability of current PCI vector resources and the number of
326	MSI/MSI-X requests from other drivers, these MSI-X vectors may be
327	re-assigned.
328	
329	For the case where the PCI subsystem re-assigned these MSI-X vectors
330	to other drivers, a request to switch back to MSI-X mode may result
331	being assigned with another set of MSI-X vectors or a failure if no
332	more vectors are available.
333	
334	5.4 Handling function implementing both MSI and MSI-X capabilities
335	
336	For the case where a function implements both MSI and MSI-X
337	capabilities, the PCI subsystem enables a device to run either in MSI
338	mode or MSI-X mode but not both. A device driver determines whether it
339	wants MSI or MSI-X enabled on its hardware device. Once a device
340	driver requests for MSI, for example, it is prohibited from requesting
341	MSI-X; in other words, a device driver is not permitted to ping-pong
342	between MSI mod MSI-X mode during a run-time.
343	
344	5.5 Hardware requirements for MSI/MSI-X support
345	
346	MSI/MSI-X support requires support from both system hardware and
347	individual hardware device functions.
348	
349	5.5.1 Required x86 hardware support
350	
351	Since the target of MSI address is the local APIC CPU, enabling
352	MSI/MSI-X support in the Linux kernel is dependent on whether existing
353	system hardware supports local APIC. Users should verify that their
354	system supports local APIC operation by testing that it runs when
355	CONFIG_X86_LOCAL_APIC=y.
356	
357	In SMP environment, CONFIG_X86_LOCAL_APIC is automatically set;
358	however, in UP environment, users must manually set
359	CONFIG_X86_LOCAL_APIC. Once CONFIG_X86_LOCAL_APIC=y, setting
360	CONFIG_PCI_MSI enables the VECTOR based scheme and the option for
361	MSI-capable device drivers to selectively enable MSI/MSI-X.
362	
363	Note that CONFIG_X86_IO_APIC setting is irrelevant because MSI/MSI-X
364	vector is allocated new during runtime and MSI/MSI-X support does not
365	depend on BIOS support. This key independency enables MSI/MSI-X
366	support on future IOxAPIC free platforms.
367	
368	5.5.2 Device hardware support
369	
370	The hardware device function supports MSI by indicating the
371	MSI/MSI-X capability structure on its PCI capability list. By
372	default, this capability structure will not be initialized by
373	the kernel to enable MSI during the system boot. In other words,
374	the device function is running on its default pin assertion mode.
375	Note that in many cases the hardware supporting MSI have bugs,
376	which may result in system hangs. The software driver of specific
377	MSI-capable hardware is responsible for deciding whether to call
378	pci_enable_msi or not. A return of zero indicates the kernel
379	successfully initialized the MSI/MSI-X capability structure of the
380	device function. The device function is now running on MSI/MSI-X mode.
381	
382	5.6 How to tell whether MSI/MSI-X is enabled on device function
383	
384	At the driver level, a return of zero from the function call of
385	pci_enable_msi()/pci_enable_msix() indicates to a device driver that
386	its device function is initialized successfully and ready to run in
387	MSI/MSI-X mode.
388	
389	At the user level, users can use the command 'cat /proc/interrupts'
390	to display the vectors allocated for devices and their interrupt
391	MSI/MSI-X modes ("PCI-MSI"/"PCI-MSI-X"). Below shows MSI mode is
392	enabled on a SCSI Adaptec 39320D Ultra320 controller.
393	
394	           CPU0       CPU1
395	  0:     324639          0    IO-APIC-edge  timer
396	  1:       1186          0    IO-APIC-edge  i8042
397	  2:          0          0          XT-PIC  cascade
398	 12:       2797          0    IO-APIC-edge  i8042
399	 14:       6543          0    IO-APIC-edge  ide0
400	 15:          1          0    IO-APIC-edge  ide1
401	169:          0          0   IO-APIC-level  uhci-hcd
402	185:          0          0   IO-APIC-level  uhci-hcd
403	193:        138         10         PCI-MSI  aic79xx
404	201:         30          0         PCI-MSI  aic79xx
405	225:         30          0   IO-APIC-level  aic7xxx
406	233:         30          0   IO-APIC-level  aic7xxx
407	NMI:          0          0
408	LOC:     324553     325068
409	ERR:          0
410	MIS:          0
411	
412	6. MSI quirks
413	
414	Several PCI chipsets or devices are known to not support MSI.
415	The PCI stack provides 3 possible levels of MSI disabling:
416	* on a single device
417	* on all devices behind a specific bridge
418	* globally
419	
420	6.1. Disabling MSI on a single device
421	
422	Under some circumstances it might be required to disable MSI on a
423	single device.  This may be achieved by either not calling pci_enable_msi()
424	or all, or setting the pci_dev->no_msi flag before (most of the time
425	in a quirk).
426	
427	6.2. Disabling MSI below a bridge
428	
429	The vast majority of MSI quirks are required by PCI bridges not
430	being able to route MSI between busses. In this case, MSI have to be
431	disabled on all devices behind this bridge. It is achieves by setting
432	the PCI_BUS_FLAGS_NO_MSI flag in the pci_bus->bus_flags of the bridge
433	subordinate bus. There is no need to set the same flag on bridges that
434	are below the broken bridge. When pci_enable_msi() is called to enable
435	MSI on a device, pci_msi_supported() takes care of checking the NO_MSI
436	flag in all parent busses of the device.
437	
438	Some bridges actually support dynamic MSI support enabling/disabling
439	by changing some bits in their PCI configuration space (especially
440	the Hypertransport chipsets such as the nVidia nForce and Serverworks
441	HT2000). It may then be required to update the NO_MSI flag on the
442	corresponding devices in the sysfs hierarchy. To enable MSI support
443	on device "0000:00:0e", do:
444	
445		echo 1 > /sys/bus/pci/devices/0000:00:0e/msi_bus
446	
447	To disable MSI support, echo 0 instead of 1. Note that it should be
448	used with caution since changing this value might break interrupts.
449	
450	6.3. Disabling MSI globally
451	
452	Some extreme cases may require to disable MSI globally on the system.
453	For now, the only known case is a Serverworks PCI-X chipsets (MSI are
454	not supported on several busses that are not all connected to the
455	chipset in the Linux PCI hierarchy). In the vast majority of other
456	cases, disabling only behind a specific bridge is enough.
457	
458	For debugging purpose, the user may also pass pci=nomsi on the kernel
459	command-line to explicitly disable MSI globally. But, once the appro-
460	priate quirks are added to the kernel, this option should not be
461	required anymore.
462	
463	6.4. Finding why MSI cannot be enabled on a device
464	
465	Assuming that MSI are not enabled on a device, you should look at
466	dmesg to find messages that quirks may output when disabling MSI
467	on some devices, some bridges or even globally.
468	Then, lspci -t gives the list of bridges above a device. Reading
469	/sys/bus/pci/devices/0000:00:0e/msi_bus will tell you whether MSI
470	are enabled (1) or disabled (0). In 0 is found in a single bridge
471	msi_bus file above the device, MSI cannot be enabled.
472	
473	7. FAQ
474	
475	Q1. Are there any limitations on using the MSI?
476	
477	A1. If the PCI device supports MSI and conforms to the
478	specification and the platform supports the APIC local bus,
479	then using MSI should work.
480	
481	Q2. Will it work on all the Pentium processors (P3, P4, Xeon,
482	AMD processors)? In P3 IPI's are transmitted on the APIC local
483	bus and in P4 and Xeon they are transmitted on the system
484	bus. Are there any implications with this?
485	
486	A2. MSI support enables a PCI device sending an inbound
487	memory write (0xfeexxxxx as target address) on its PCI bus
488	directly to the FSB. Since the message address has a
489	redirection hint bit cleared, it should work.
490	
491	Q3. The target address 0xfeexxxxx will be translated by the
492	Host Bridge into an interrupt message. Are there any
493	limitations on the chipsets such as Intel 8xx, Intel e7xxx,
494	or VIA?
495	
496	A3. If these chipsets support an inbound memory write with
497	target address set as 0xfeexxxxx, as conformed to PCI
498	specification 2.3 or latest, then it should work.
499	
500	Q4. From the driver point of view, if the MSI is lost because
501	of errors occurring during inbound memory write, then it may
502	wait forever. Is there a mechanism for it to recover?
503	
504	A4. Since the target of the transaction is an inbound memory
505	write, all transaction termination conditions (Retry,
506	Master-Abort, Target-Abort, or normal completion) are
507	supported. A device sending an MSI must abide by all the PCI
508	rules and conditions regarding that inbound memory write. So,
509	if a retry is signaled it must retry, etc... We believe that
510	the recommendation for Abort is also a retry (refer to PCI
511	specification 2.3 or latest).
Hide Line Numbers


About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog