About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / rapidio


Based on kernel version 4.16.1. Page generated on 2018-04-09 11:53 EST.

1	                          The Linux RapidIO Subsystem
2	
3	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4	
5	The RapidIO standard is a packet-based fabric interconnect standard designed for
6	use in embedded systems. Development of the RapidIO standard is directed by the
7	RapidIO Trade Association (RTA). The current version of the RapidIO specification
8	is publicly available for download from the RTA web-site [1].
9	
10	This document describes the basics of the Linux RapidIO subsystem and provides
11	information on its major components.
12	
13	1 Overview
14	----------
15	
16	Because the RapidIO subsystem follows the Linux device model it is integrated
17	into the kernel similarly to other buses by defining RapidIO-specific device and
18	bus types and registering them within the device model.
19	
20	The Linux RapidIO subsystem is architecture independent and therefore defines
21	architecture-specific interfaces that provide support for common RapidIO
22	subsystem operations.
23	
24	2. Core Components
25	------------------
26	
27	A typical RapidIO network is a combination of endpoints and switches.
28	Each of these components is represented in the subsystem by an associated data
29	structure. The core logical components of the RapidIO subsystem are defined
30	in include/linux/rio.h file.
31	
32	2.1 Master Port
33	
34	A master port (or mport) is a RapidIO interface controller that is local to the
35	processor executing the Linux code. A master port generates and receives RapidIO
36	packets (transactions). In the RapidIO subsystem each master port is represented
37	by a rio_mport data structure. This structure contains master port specific
38	resources such as mailboxes and doorbells. The rio_mport also includes a unique
39	host device ID that is valid when a master port is configured as an enumerating
40	host.
41	
42	RapidIO master ports are serviced by subsystem specific mport device drivers
43	that provide functionality defined for this subsystem. To provide a hardware
44	independent interface for RapidIO subsystem operations, rio_mport structure
45	includes rio_ops data structure which contains pointers to hardware specific
46	implementations of RapidIO functions.
47	
48	2.2 Device
49	
50	A RapidIO device is any endpoint (other than mport) or switch in the network.
51	All devices are presented in the RapidIO subsystem by corresponding rio_dev data
52	structure. Devices form one global device list and per-network device lists
53	(depending on number of available mports and networks).
54	
55	2.3 Switch
56	
57	A RapidIO switch is a special class of device that routes packets between its
58	ports towards their final destination. The packet destination port within a
59	switch is defined by an internal routing table. A switch is presented in the
60	RapidIO subsystem by rio_dev data structure expanded by additional rio_switch
61	data structure, which contains switch specific information such as copy of the
62	routing table and pointers to switch specific functions.
63	
64	The RapidIO subsystem defines the format and initialization method for subsystem
65	specific switch drivers that are designed to provide hardware-specific
66	implementation of common switch management routines.
67	
68	2.4 Network
69	
70	A RapidIO network is a combination of interconnected endpoint and switch devices.
71	Each RapidIO network known to the system is represented by corresponding rio_net
72	data structure. This structure includes lists of all devices and local master
73	ports that form the same network. It also contains a pointer to the default
74	master port that is used to communicate with devices within the network.
75	
76	2.5 Device Drivers
77	
78	RapidIO device-specific drivers follow Linux Kernel Driver Model and are
79	intended to support specific RapidIO devices attached to the RapidIO network.
80	
81	2.6 Subsystem Interfaces
82	
83	RapidIO interconnect specification defines features that may be used to provide
84	one or more common service layers for all participating RapidIO devices. These
85	common services may act separately from device-specific drivers or be used by
86	device-specific drivers. Example of such service provider is the RIONET driver
87	which implements Ethernet-over-RapidIO interface. Because only one driver can be
88	registered for a device, all common RapidIO services have to be registered as
89	subsystem interfaces. This allows to have multiple common services attached to
90	the same device without blocking attachment of a device-specific driver.
91	
92	3. Subsystem Initialization
93	---------------------------
94	
95	In order to initialize the RapidIO subsystem, a platform must initialize and
96	register at least one master port within the RapidIO network. To register mport
97	within the subsystem controller driver's initialization code calls function
98	rio_register_mport() for each available master port.
99	
100	After all active master ports are registered with a RapidIO subsystem,
101	an enumeration and/or discovery routine may be called automatically or
102	by user-space command.
103	
104	RapidIO subsystem can be configured to be built as a statically linked or
105	modular component of the kernel (see details below).
106	
107	4. Enumeration and Discovery
108	----------------------------
109	
110	4.1 Overview
111	------------
112	
113	RapidIO subsystem configuration options allow users to build enumeration and
114	discovery methods as statically linked components or loadable modules.
115	An enumeration/discovery method implementation and available input parameters
116	define how any given method can be attached to available RapidIO mports:
117	simply to all available mports OR individually to the specified mport device.
118	
119	Depending on selected enumeration/discovery build configuration, there are
120	several methods to initiate an enumeration and/or discovery process:
121	
122	  (a) Statically linked enumeration and discovery process can be started
123	  automatically during kernel initialization time using corresponding module
124	  parameters. This was the original method used since introduction of RapidIO
125	  subsystem. Now this method relies on enumerator module parameter which is
126	  'rio-scan.scan' for existing basic enumeration/discovery method.
127	  When automatic start of enumeration/discovery is used a user has to ensure
128	  that all discovering endpoints are started before the enumerating endpoint
129	  and are waiting for enumeration to be completed.
130	  Configuration option CONFIG_RAPIDIO_DISC_TIMEOUT defines time that discovering
131	  endpoint waits for enumeration to be completed. If the specified timeout
132	  expires the discovery process is terminated without obtaining RapidIO network
133	  information. NOTE: a timed out discovery process may be restarted later using
134	  a user-space command as it is described below (if the given endpoint was
135	  enumerated successfully).
136	
137	  (b) Statically linked enumeration and discovery process can be started by
138	  a command from user space. This initiation method provides more flexibility
139	  for a system startup compared to the option (a) above. After all participating
140	  endpoints have been successfully booted, an enumeration process shall be
141	  started first by issuing a user-space command, after an enumeration is
142	  completed a discovery process can be started on all remaining endpoints.
143	
144	  (c) Modular enumeration and discovery process can be started by a command from
145	  user space. After an enumeration/discovery module is loaded, a network scan
146	  process can be started by issuing a user-space command.
147	  Similar to the option (b) above, an enumerator has to be started first.
148	
149	  (d) Modular enumeration and discovery process can be started by a module
150	  initialization routine. In this case an enumerating module shall be loaded
151	  first.
152	
153	When a network scan process is started it calls an enumeration or discovery
154	routine depending on the configured role of a master port: host or agent.
155	
156	Enumeration is performed by a master port if it is configured as a host port by
157	assigning a host destination ID greater than or equal to zero. The host
158	destination ID can be assigned to a master port using various methods depending
159	on RapidIO subsystem build configuration:
160	
161	  (a) For a statically linked RapidIO subsystem core use command line parameter
162	  "rapidio.hdid=" with a list of destination ID assignments in order of mport
163	  device registration. For example, in a system with two RapidIO controllers
164	  the command line parameter "rapidio.hdid=-1,7" will result in assignment of
165	  the host destination ID=7 to the second RapidIO controller, while the first
166	  one will be assigned destination ID=-1.
167	
168	  (b) If the RapidIO subsystem core is built as a loadable module, in addition
169	  to the method shown above, the host destination ID(s) can be specified using
170	  traditional methods of passing module parameter "hdid=" during its loading:
171	  - from command line: "modprobe rapidio hdid=-1,7", or
172	  - from modprobe configuration file using configuration command "options",
173	    like in this example: "options rapidio hdid=-1,7". An example of modprobe
174	    configuration file is provided in the section below.
175	
176	  NOTES:
177	  (i) if "hdid=" parameter is omitted all available mport will be assigned
178	  destination ID = -1;
179	  (ii) the "hdid=" parameter in systems with multiple mports can have
180	  destination ID assignments omitted from the end of list (default = -1).
181	
182	If the host device ID for a specific master port is set to -1, the discovery
183	process will be performed for it.
184	
185	The enumeration and discovery routines use RapidIO maintenance transactions
186	to access the configuration space of devices.
187	
188	NOTE: If RapidIO switch-specific device drivers are built as loadable modules
189	they must be loaded before enumeration/discovery process starts.
190	This requirement is cased by the fact that enumeration/discovery methods invoke
191	vendor-specific callbacks on early stages.
192	
193	4.2 Automatic Start of Enumeration and Discovery
194	------------------------------------------------
195	
196	Automatic enumeration/discovery start method is applicable only to built-in
197	enumeration/discovery RapidIO configuration selection. To enable automatic
198	enumeration/discovery start by existing basic enumerator method set use boot
199	command line parameter "rio-scan.scan=1".
200	
201	This configuration requires synchronized start of all RapidIO endpoints that
202	form a network which will be enumerated/discovered. Discovering endpoints have
203	to be started before an enumeration starts to ensure that all RapidIO
204	controllers have been initialized and are ready to be discovered. Configuration
205	parameter CONFIG_RAPIDIO_DISC_TIMEOUT defines time (in seconds) which
206	a discovering endpoint will wait for enumeration to be completed.
207	
208	When automatic enumeration/discovery start is selected, basic method's
209	initialization routine calls rio_init_mports() to perform enumeration or
210	discovery for all known mport devices.
211	
212	Depending on RapidIO network size and configuration this automatic
213	enumeration/discovery start method may be difficult to use due to the
214	requirement for synchronized start of all endpoints.
215	
216	4.3 User-space Start of Enumeration and Discovery
217	-------------------------------------------------
218	
219	User-space start of enumeration and discovery can be used with built-in and
220	modular build configurations. For user-space controlled start RapidIO subsystem
221	creates the sysfs write-only attribute file '/sys/bus/rapidio/scan'. To initiate
222	an enumeration or discovery process on specific mport device, a user needs to
223	write mport_ID (not RapidIO destination ID) into that file. The mport_ID is a
224	sequential number (0 ... RIO_MAX_MPORTS) assigned during mport device
225	registration. For example for machine with single RapidIO controller, mport_ID
226	for that controller always will be 0.
227	
228	To initiate RapidIO enumeration/discovery on all available mports a user may
229	write '-1' (or RIO_MPORT_ANY) into the scan attribute file.
230	
231	4.4 Basic Enumeration Method
232	----------------------------
233	
234	This is an original enumeration/discovery method which is available since
235	first release of RapidIO subsystem code. The enumeration process is
236	implemented according to the enumeration algorithm outlined in the RapidIO
237	Interconnect Specification: Annex I [1].
238	
239	This method can be configured as statically linked or loadable module.
240	The method's single parameter "scan" allows to trigger the enumeration/discovery
241	process from module initialization routine.
242	
243	This enumeration/discovery method can be started only once and does not support
244	unloading if it is built as a module.
245	
246	The enumeration process traverses the network using a recursive depth-first
247	algorithm. When a new device is found, the enumerator takes ownership of that
248	device by writing into the Host Device ID Lock CSR. It does this to ensure that
249	the enumerator has exclusive right to enumerate the device. If device ownership
250	is successfully acquired, the enumerator allocates a new rio_dev structure and
251	initializes it according to device capabilities.
252	
253	If the device is an endpoint, a unique device ID is assigned to it and its value
254	is written into the device's Base Device ID CSR.
255	
256	If the device is a switch, the enumerator allocates an additional rio_switch
257	structure to store switch specific information. Then the switch's vendor ID and
258	device ID are queried against a table of known RapidIO switches. Each switch
259	table entry contains a pointer to a switch-specific initialization routine that
260	initializes pointers to the rest of switch specific operations, and performs
261	hardware initialization if necessary. A RapidIO switch does not have a unique
262	device ID; it relies on hopcount and routing for device ID of an attached
263	endpoint if access to its configuration registers is required. If a switch (or
264	chain of switches) does not have any endpoint (except enumerator) attached to
265	it, a fake device ID will be assigned to configure a route to that switch.
266	In the case of a chain of switches without endpoint, one fake device ID is used
267	to configure a route through the entire chain and switches are differentiated by
268	their hopcount value.
269	
270	For both endpoints and switches the enumerator writes a unique component tag
271	into device's Component Tag CSR. That unique value is used by the error
272	management notification mechanism to identify a device that is reporting an
273	error management event.
274	
275	Enumeration beyond a switch is completed by iterating over each active egress
276	port of that switch. For each active link, a route to a default device ID
277	(0xFF for 8-bit systems and 0xFFFF for 16-bit systems) is temporarily written
278	into the routing table. The algorithm recurs by calling itself with hopcount + 1
279	and the default device ID in order to access the device on the active port.
280	
281	After the host has completed enumeration of the entire network it releases
282	devices by clearing device ID locks (calls rio_clear_locks()). For each endpoint
283	in the system, it sets the Discovered bit in the Port General Control CSR
284	to indicate that enumeration is completed and agents are allowed to execute
285	passive discovery of the network.
286	
287	The discovery process is performed by agents and is similar to the enumeration
288	process that is described above. However, the discovery process is performed
289	without changes to the existing routing because agents only gather information
290	about RapidIO network structure and are building an internal map of discovered
291	devices. This way each Linux-based component of the RapidIO subsystem has
292	a complete view of the network. The discovery process can be performed
293	simultaneously by several agents. After initializing its RapidIO master port
294	each agent waits for enumeration completion by the host for the configured wait
295	time period. If this wait time period expires before enumeration is completed,
296	an agent skips RapidIO discovery and continues with remaining kernel
297	initialization.
298	
299	4.5 Adding New Enumeration/Discovery Method
300	-------------------------------------------
301	
302	RapidIO subsystem code organization allows addition of new enumeration/discovery
303	methods as new configuration options without significant impact to the core
304	RapidIO code.
305	
306	A new enumeration/discovery method has to be attached to one or more mport
307	devices before an enumeration/discovery process can be started. Normally,
308	method's module initialization routine calls rio_register_scan() to attach
309	an enumerator to a specified mport device (or devices). The basic enumerator
310	implementation demonstrates this process.
311	
312	4.6 Using Loadable RapidIO Switch Drivers
313	-----------------------------------------
314	
315	In the case when RapidIO switch drivers are built as loadable modules a user
316	must ensure that they are loaded before the enumeration/discovery starts.
317	This process can be automated by specifying pre- or post- dependencies in the
318	RapidIO-specific modprobe configuration file as shown in the example below.
319	
320	  File /etc/modprobe.d/rapidio.conf:
321	  ----------------------------------
322	
323	  # Configure RapidIO subsystem modules
324	
325	  # Set enumerator host destination ID (overrides kernel command line option)
326	  options rapidio hdid=-1,2
327	
328	  # Load RapidIO switch drivers immediately after rapidio core module was loaded
329	  softdep rapidio post: idt_gen2 idtcps tsi57x
330	
331	  # OR :
332	
333	  # Load RapidIO switch drivers just before rio-scan enumerator module is loaded
334	  softdep rio-scan pre: idt_gen2 idtcps tsi57x
335	
336	  --------------------------
337	
338	NOTE: In the example above, one of "softdep" commands must be removed or
339	commented out to keep required module loading sequence.
340	
341	A. References
342	-------------
343	
344	[1] RapidIO Trade Association. RapidIO Interconnect Specifications.
345	    http://www.rapidio.org.
346	[2] Rapidio TA. Technology Comparisons.
347	    http://www.rapidio.org/education/technology_comparisons/
348	[3] RapidIO support for Linux.
349	    http://lwn.net/Articles/139118/
350	[4] Matt Porter. RapidIO for Linux. Ottawa Linux Symposium, 2005
351	    http://www.kernel.org/doc/ols/2005/ols2005v2-pages-43-56.pdf
Hide Line Numbers


About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog