About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / scsi / libsas.txt




Custom Search

Based on kernel version 3.16. Page generated on 2014-08-06 21:41 EST.

1	SAS Layer
2	---------
3	
4	The SAS Layer is a management infrastructure which manages
5	SAS LLDDs.  It sits between SCSI Core and SAS LLDDs.  The
6	layout is as follows: while SCSI Core is concerned with
7	SAM/SPC issues, and a SAS LLDD+sequencer is concerned with
8	phy/OOB/link management, the SAS layer is concerned with:
9	
10	      * SAS Phy/Port/HA event management (LLDD generates,
11	        SAS Layer processes),
12	      * SAS Port management (creation/destruction),
13	      * SAS Domain discovery and revalidation,
14	      * SAS Domain device management,
15	      * SCSI Host registration/unregistration,
16	      * Device registration with SCSI Core (SAS) or libata
17	        (SATA), and
18	      * Expander management and exporting expander control
19	        to user space.
20	
21	A SAS LLDD is a PCI device driver.  It is concerned with
22	phy/OOB management, and vendor specific tasks and generates
23	events to the SAS layer.
24	
25	The SAS Layer does most SAS tasks as outlined in the SAS 1.1
26	spec.
27	
28	The sas_ha_struct describes the SAS LLDD to the SAS layer.
29	Most of it is used by the SAS Layer but a few fields need to
30	be initialized by the LLDDs.
31	
32	After initializing your hardware, from the probe() function
33	you call sas_register_ha(). It will register your LLDD with
34	the SCSI subsystem, creating a SCSI host and it will
35	register your SAS driver with the sysfs SAS tree it creates.
36	It will then return.  Then you enable your phys to actually
37	start OOB (at which point your driver will start calling the
38	notify_* event callbacks).
39	
40	Structure descriptions:
41	
42	struct sas_phy --------------------
43	Normally this is statically embedded to your driver's
44	phy structure:
45		struct my_phy {
46		       blah;
47		       struct sas_phy sas_phy;
48		       bleh;
49		};
50	And then all the phys are an array of my_phy in your HA
51	struct (shown below).
52	
53	Then as you go along and initialize your phys you also
54	initialize the sas_phy struct, along with your own
55	phy structure.
56	
57	In general, the phys are managed by the LLDD and the ports
58	are managed by the SAS layer.  So the phys are initialized
59	and updated by the LLDD and the ports are initialized and
60	updated by the SAS layer.
61	
62	There is a scheme where the LLDD can RW certain fields,
63	and the SAS layer can only read such ones, and vice versa.
64	The idea is to avoid unnecessary locking.
65	
66	enabled -- must be set (0/1)
67	id -- must be set [0,MAX_PHYS)
68	class, proto, type, role, oob_mode, linkrate -- must be set
69	oob_mode --  you set this when OOB has finished and then notify
70	the SAS Layer.
71	
72	sas_addr -- this normally points to an array holding the sas
73	address of the phy, possibly somewhere in your my_phy
74	struct.
75	
76	attached_sas_addr -- set this when you (LLDD) receive an
77	IDENTIFY frame or a FIS frame, _before_ notifying the SAS
78	layer.  The idea is that sometimes the LLDD may want to fake
79	or provide a different SAS address on that phy/port and this
80	allows it to do this.  At best you should copy the sas
81	address from the IDENTIFY frame or maybe generate a SAS
82	address for SATA directly attached devices.  The Discover
83	process may later change this.
84	
85	frame_rcvd -- this is where you copy the IDENTIFY/FIS frame
86	when you get it; you lock, copy, set frame_rcvd_size and
87	unlock the lock, and then call the event.  It is a pointer
88	since there's no way to know your hw frame size _exactly_,
89	so you define the actual array in your phy struct and let
90	this pointer point to it.  You copy the frame from your
91	DMAable memory to that area holding the lock.
92	
93	sas_prim -- this is where primitives go when they're
94	received.  See sas.h. Grab the lock, set the primitive,
95	release the lock, notify.
96	
97	port -- this points to the sas_port if the phy belongs
98	to a port -- the LLDD only reads this. It points to the
99	sas_port this phy is part of.  Set by the SAS Layer.
100	
101	ha -- may be set; the SAS layer sets it anyway.
102	
103	lldd_phy -- you should set this to point to your phy so you
104	can find your way around faster when the SAS layer calls one
105	of your callbacks and passes you a phy.  If the sas_phy is
106	embedded you can also use container_of -- whatever you
107	prefer.
108	
109	
110	struct sas_port --------------------
111	The LLDD doesn't set any fields of this struct -- it only
112	reads them.  They should be self explanatory.
113	
114	phy_mask is 32 bit, this should be enough for now, as I
115	haven't heard of a HA having more than 8 phys.
116	
117	lldd_port -- I haven't found use for that -- maybe other
118	LLDD who wish to have internal port representation can make
119	use of this.
120	
121	
122	struct sas_ha_struct --------------------
123	It normally is statically declared in your own LLDD
124	structure describing your adapter:
125	struct my_sas_ha {
126	       blah;
127	       struct sas_ha_struct sas_ha;
128	       struct my_phy phys[MAX_PHYS];
129	       struct sas_port sas_ports[MAX_PHYS]; /* (1) */
130	       bleh;
131	};
132	
133	(1) If your LLDD doesn't have its own port representation.
134	
135	What needs to be initialized (sample function given below).
136	
137	pcidev
138	sas_addr -- since the SAS layer doesn't want to mess with
139		 memory allocation, etc, this points to statically
140		 allocated array somewhere (say in your host adapter
141		 structure) and holds the SAS address of the host
142		 adapter as given by you or the manufacturer, etc.
143	sas_port
144	sas_phy -- an array of pointers to structures. (see
145		note above on sas_addr).
146		These must be set.  See more notes below.
147	num_phys -- the number of phys present in the sas_phy array,
148		 and the number of ports present in the sas_port
149		 array.  There can be a maximum num_phys ports (one per
150		 port) so we drop the num_ports, and only use
151		 num_phys.
152	
153	The event interface:
154	
155		/* LLDD calls these to notify the class of an event. */
156		void (*notify_ha_event)(struct sas_ha_struct *, enum ha_event);
157		void (*notify_port_event)(struct sas_phy *, enum port_event);
158		void (*notify_phy_event)(struct sas_phy *, enum phy_event);
159	
160	When sas_register_ha() returns, those are set and can be
161	called by the LLDD to notify the SAS layer of such events
162	the SAS layer.
163	
164	The port notification:
165	
166		/* The class calls these to notify the LLDD of an event. */
167		void (*lldd_port_formed)(struct sas_phy *);
168		void (*lldd_port_deformed)(struct sas_phy *);
169	
170	If the LLDD wants notification when a port has been formed
171	or deformed it sets those to a function satisfying the type.
172	
173	A SAS LLDD should also implement at least one of the Task
174	Management Functions (TMFs) described in SAM:
175	
176		/* Task Management Functions. Must be called from process context. */
177		int (*lldd_abort_task)(struct sas_task *);
178		int (*lldd_abort_task_set)(struct domain_device *, u8 *lun);
179		int (*lldd_clear_aca)(struct domain_device *, u8 *lun);
180		int (*lldd_clear_task_set)(struct domain_device *, u8 *lun);
181		int (*lldd_I_T_nexus_reset)(struct domain_device *);
182		int (*lldd_lu_reset)(struct domain_device *, u8 *lun);
183		int (*lldd_query_task)(struct sas_task *);
184	
185	For more information please read SAM from T10.org.
186	
187	Port and Adapter management:
188	
189		/* Port and Adapter management */
190		int (*lldd_clear_nexus_port)(struct sas_port *);
191		int (*lldd_clear_nexus_ha)(struct sas_ha_struct *);
192	
193	A SAS LLDD should implement at least one of those.
194	
195	Phy management:
196	
197		/* Phy management */
198		int (*lldd_control_phy)(struct sas_phy *, enum phy_func);
199	
200	lldd_ha -- set this to point to your HA struct. You can also
201	use container_of if you embedded it as shown above.
202	
203	A sample initialization and registration function
204	can look like this (called last thing from probe())
205	*but* before you enable the phys to do OOB:
206	
207	static int register_sas_ha(struct my_sas_ha *my_ha)
208	{
209		int i;
210		static struct sas_phy   *sas_phys[MAX_PHYS];
211		static struct sas_port  *sas_ports[MAX_PHYS];
212	
213		my_ha->sas_ha.sas_addr = &my_ha->sas_addr[0];
214	
215		for (i = 0; i < MAX_PHYS; i++) {
216			sas_phys[i] = &my_ha->phys[i].sas_phy;
217			sas_ports[i] = &my_ha->sas_ports[i];
218		}
219	
220		my_ha->sas_ha.sas_phy  = sas_phys;
221		my_ha->sas_ha.sas_port = sas_ports;
222		my_ha->sas_ha.num_phys = MAX_PHYS;
223	
224		my_ha->sas_ha.lldd_port_formed = my_port_formed;
225	
226		my_ha->sas_ha.lldd_dev_found = my_dev_found;
227		my_ha->sas_ha.lldd_dev_gone = my_dev_gone;
228	
229		my_ha->sas_ha.lldd_max_execute_num = lldd_max_execute_num; (1)
230	
231		my_ha->sas_ha.lldd_queue_size = ha_can_queue;
232		my_ha->sas_ha.lldd_execute_task = my_execute_task;
233	
234		my_ha->sas_ha.lldd_abort_task     = my_abort_task;
235		my_ha->sas_ha.lldd_abort_task_set = my_abort_task_set;
236		my_ha->sas_ha.lldd_clear_aca      = my_clear_aca;
237		my_ha->sas_ha.lldd_clear_task_set = my_clear_task_set;
238		my_ha->sas_ha.lldd_I_T_nexus_reset= NULL; (2)
239		my_ha->sas_ha.lldd_lu_reset       = my_lu_reset;
240		my_ha->sas_ha.lldd_query_task     = my_query_task;
241	
242		my_ha->sas_ha.lldd_clear_nexus_port = my_clear_nexus_port;
243		my_ha->sas_ha.lldd_clear_nexus_ha = my_clear_nexus_ha;
244	
245		my_ha->sas_ha.lldd_control_phy = my_control_phy;
246	
247		return sas_register_ha(&my_ha->sas_ha);
248	}
249	
250	(1) This is normally a LLDD parameter, something of the
251	lines of a task collector.  What it tells the SAS Layer is
252	whether the SAS layer should run in Direct Mode (default:
253	value 0 or 1) or Task Collector Mode (value greater than 1).
254	
255	In Direct Mode, the SAS Layer calls Execute Task as soon as
256	it has a command to send to the SDS, _and_ this is a single
257	command, i.e. not linked.
258	
259	Some hardware (e.g. aic94xx) has the capability to DMA more
260	than one task at a time (interrupt) from host memory.  Task
261	Collector Mode is an optional feature for HAs which support
262	this in their hardware.  (Again, it is completely optional
263	even if your hardware supports it.)
264	
265	In Task Collector Mode, the SAS Layer would do _natural_
266	coalescing of tasks and at the appropriate moment it would
267	call your driver to DMA more than one task in a single HA
268	interrupt. DMBS may want to use this by insmod/modprobe
269	setting the lldd_max_execute_num to something greater than
270	1.
271	
272	(2) SAS 1.1 does not define I_T Nexus Reset TMF.
273	
274	Events
275	------
276	
277	Events are _the only way_ a SAS LLDD notifies the SAS layer
278	of anything.  There is no other method or way a LLDD to tell
279	the SAS layer of anything happening internally or in the SAS
280	domain.
281	
282	Phy events:
283		PHYE_LOSS_OF_SIGNAL, (C)
284		PHYE_OOB_DONE,
285		PHYE_OOB_ERROR,      (C)
286		PHYE_SPINUP_HOLD.
287	
288	Port events, passed on a _phy_:
289		PORTE_BYTES_DMAED,      (M)
290		PORTE_BROADCAST_RCVD,   (E)
291		PORTE_LINK_RESET_ERR,   (C)
292		PORTE_TIMER_EVENT,      (C)
293		PORTE_HARD_RESET.
294	
295	Host Adapter event:
296		HAE_RESET
297	
298	A SAS LLDD should be able to generate
299		- at least one event from group C (choice),
300		- events marked M (mandatory) are mandatory (only one),
301		- events marked E (expander) if it wants the SAS layer
302		  to handle domain revalidation (only one such).
303		- Unmarked events are optional.
304	
305	Meaning:
306	
307	HAE_RESET -- when your HA got internal error and was reset.
308	
309	PORTE_BYTES_DMAED -- on receiving an IDENTIFY/FIS frame
310	PORTE_BROADCAST_RCVD -- on receiving a primitive
311	PORTE_LINK_RESET_ERR -- timer expired, loss of signal, loss
312	of DWS, etc. (*)
313	PORTE_TIMER_EVENT -- DWS reset timeout timer expired (*)
314	PORTE_HARD_RESET -- Hard Reset primitive received.
315	
316	PHYE_LOSS_OF_SIGNAL -- the device is gone (*)
317	PHYE_OOB_DONE -- OOB went fine and oob_mode is valid
318	PHYE_OOB_ERROR -- Error while doing OOB, the device probably
319	got disconnected. (*)
320	PHYE_SPINUP_HOLD -- SATA is present, COMWAKE not sent.
321	
322	(*) should set/clear the appropriate fields in the phy,
323	    or alternatively call the inlined sas_phy_disconnected()
324	    which is just a helper, from their tasklet.
325	
326	The Execute Command SCSI RPC:
327	
328		int (*lldd_execute_task)(struct sas_task *, int num,
329					 unsigned long gfp_flags);
330	
331	Used to queue a task to the SAS LLDD.  @task is the tasks to
332	be executed.  @num should be the number of tasks being
333	queued at this function call (they are linked listed via
334	task::list), @gfp_mask should be the gfp_mask defining the
335	context of the caller.
336	
337	This function should implement the Execute Command SCSI RPC,
338	or if you're sending a SCSI Task as linked commands, you
339	should also use this function.
340	
341	That is, when lldd_execute_task() is called, the command(s)
342	go out on the transport *immediately*.  There is *no*
343	queuing of any sort and at any level in a SAS LLDD.
344	
345	The use of task::list is two-fold, one for linked commands,
346	the other discussed below.
347	
348	It is possible to queue up more than one task at a time, by
349	initializing the list element of struct sas_task, and
350	passing the number of tasks enlisted in this manner in num.
351	
352	Returns: -SAS_QUEUE_FULL, -ENOMEM, nothing was queued;
353		 0, the task(s) were queued.
354	
355	If you want to pass num > 1, then either
356	A) you're the only caller of this function and keep track
357	   of what you've queued to the LLDD, or
358	B) you know what you're doing and have a strategy of
359	   retrying.
360	
361	As opposed to queuing one task at a time (function call),
362	batch queuing of tasks, by having num > 1, greatly
363	simplifies LLDD code, sequencer code, and _hardware design_,
364	and has some performance advantages in certain situations
365	(DBMS).
366	
367	The LLDD advertises if it can take more than one command at
368	a time at lldd_execute_task(), by setting the
369	lldd_max_execute_num parameter (controlled by "collector"
370	module parameter in aic94xx SAS LLDD).
371	
372	You should leave this to the default 1, unless you know what
373	you're doing.
374	
375	This is a function of the LLDD, to which the SAS layer can
376	cater to.
377	
378	int lldd_queue_size
379		The host adapter's queue size.  This is the maximum
380	number of commands the lldd can have pending to domain
381	devices on behalf of all upper layers submitting through
382	lldd_execute_task().
383	
384	You really want to set this to something (much) larger than
385	1.
386	
387	This _really_ has absolutely nothing to do with queuing.
388	There is no queuing in SAS LLDDs.
389	
390	struct sas_task {
391		dev -- the device this task is destined to
392		list -- must be initialized (INIT_LIST_HEAD)
393		task_proto -- _one_ of enum sas_proto
394		scatter -- pointer to scatter gather list array
395		num_scatter -- number of elements in scatter
396		total_xfer_len -- total number of bytes expected to be transferred
397		data_dir -- PCI_DMA_...
398		task_done -- callback when the task has finished execution
399	};
400	
401	DISCOVERY
402	---------
403	
404	The sysfs tree has the following purposes:
405	    a) It shows you the physical layout of the SAS domain at
406	       the current time, i.e. how the domain looks in the
407	       physical world right now.
408	    b) Shows some device parameters _at_discovery_time_.
409	
410	This is a link to the tree(1) program, very useful in
411	viewing the SAS domain:
412	ftp://mama.indstate.edu/linux/tree/
413	I expect user space applications to actually create a
414	graphical interface of this.
415	
416	That is, the sysfs domain tree doesn't show or keep state if
417	you e.g., change the meaning of the READY LED MEANING
418	setting, but it does show you the current connection status
419	of the domain device.
420	
421	Keeping internal device state changes is responsibility of
422	upper layers (Command set drivers) and user space.
423	
424	When a device or devices are unplugged from the domain, this
425	is reflected in the sysfs tree immediately, and the device(s)
426	removed from the system.
427	
428	The structure domain_device describes any device in the SAS
429	domain.  It is completely managed by the SAS layer.  A task
430	points to a domain device, this is how the SAS LLDD knows
431	where to send the task(s) to.  A SAS LLDD only reads the
432	contents of the domain_device structure, but it never creates
433	or destroys one.
434	
435	Expander management from User Space
436	-----------------------------------
437	
438	In each expander directory in sysfs, there is a file called
439	"smp_portal".  It is a binary sysfs attribute file, which
440	implements an SMP portal (Note: this is *NOT* an SMP port),
441	to which user space applications can send SMP requests and
442	receive SMP responses.
443	
444	Functionality is deceptively simple:
445	
446	1. Build the SMP frame you want to send. The format and layout
447	   is described in the SAS spec.  Leave the CRC field equal 0.
448	open(2)
449	2. Open the expander's SMP portal sysfs file in RW mode.
450	write(2)
451	3. Write the frame you built in 1.
452	read(2)
453	4. Read the amount of data you expect to receive for the frame you built.
454	   If you receive different amount of data you expected to receive,
455	   then there was some kind of error.
456	close(2)
457	All this process is shown in detail in the function do_smp_func()
458	and its callers, in the file "expander_conf.c".
459	
460	The kernel functionality is implemented in the file
461	"sas_expander.c".
462	
463	The program "expander_conf.c" implements this. It takes one
464	argument, the sysfs file name of the SMP portal to the
465	expander, and gives expander information, including routing
466	tables.
467	
468	The SMP portal gives you complete control of the expander,
469	so please be careful.
Hide Line Numbers
About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Information is copyright its respective author. All material is available from the Linux Kernel Source distributed under a GPL License. This page is provided as a free service by mjmwired.net.