About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / media-framework.txt


Based on kernel version 4.3. Page generated on 2015-11-02 12:50 EST.

1	Linux kernel media framework
2	============================
3	
4	This document describes the Linux kernel media framework, its data structures,
5	functions and their usage.
6	
7	
8	Introduction
9	------------
10	
11	The media controller API is documented in DocBook format in
12	Documentation/DocBook/media/v4l/media-controller.xml. This document will focus
13	on the kernel-side implementation of the media framework.
14	
15	
16	Abstract media device model
17	---------------------------
18	
19	Discovering a device internal topology, and configuring it at runtime, is one
20	of the goals of the media framework. To achieve this, hardware devices are
21	modelled as an oriented graph of building blocks called entities connected
22	through pads.
23	
24	An entity is a basic media hardware building block. It can correspond to
25	a large variety of logical blocks such as physical hardware devices
26	(CMOS sensor for instance), logical hardware devices (a building block
27	in a System-on-Chip image processing pipeline), DMA channels or physical
28	connectors.
29	
30	A pad is a connection endpoint through which an entity can interact with
31	other entities. Data (not restricted to video) produced by an entity
32	flows from the entity's output to one or more entity inputs. Pads should
33	not be confused with physical pins at chip boundaries.
34	
35	A link is a point-to-point oriented connection between two pads, either
36	on the same entity or on different entities. Data flows from a source
37	pad to a sink pad.
38	
39	
40	Media device
41	------------
42	
43	A media device is represented by a struct media_device instance, defined in
44	include/media/media-device.h. Allocation of the structure is handled by the
45	media device driver, usually by embedding the media_device instance in a
46	larger driver-specific structure.
47	
48	Drivers register media device instances by calling
49	
50		media_device_register(struct media_device *mdev);
51	
52	The caller is responsible for initializing the media_device structure before
53	registration. The following fields must be set:
54	
55	 - dev must point to the parent device (usually a pci_dev, usb_interface or
56	   platform_device instance).
57	
58	 - model must be filled with the device model name as a NUL-terminated UTF-8
59	   string. The device/model revision must not be stored in this field.
60	
61	The following fields are optional:
62	
63	 - serial is a unique serial number stored as a NUL-terminated ASCII string.
64	   The field is big enough to store a GUID in text form. If the hardware
65	   doesn't provide a unique serial number this field must be left empty.
66	
67	 - bus_info represents the location of the device in the system as a
68	   NUL-terminated ASCII string. For PCI/PCIe devices bus_info must be set to
69	   "PCI:" (or "PCIe:") followed by the value of pci_name(). For USB devices,
70	   the usb_make_path() function must be used. This field is used by
71	   applications to distinguish between otherwise identical devices that don't
72	   provide a serial number.
73	
74	 - hw_revision is the hardware device revision in a driver-specific format.
75	   When possible the revision should be formatted with the KERNEL_VERSION
76	   macro.
77	
78	 - driver_version is formatted with the KERNEL_VERSION macro. The version
79	   minor must be incremented when new features are added to the userspace API
80	   without breaking binary compatibility. The version major must be
81	   incremented when binary compatibility is broken.
82	
83	Upon successful registration a character device named media[0-9]+ is created.
84	The device major and minor numbers are dynamic. The model name is exported as
85	a sysfs attribute.
86	
87	Drivers unregister media device instances by calling
88	
89		media_device_unregister(struct media_device *mdev);
90	
91	Unregistering a media device that hasn't been registered is *NOT* safe.
92	
93	
94	Entities, pads and links
95	------------------------
96	
97	- Entities
98	
99	Entities are represented by a struct media_entity instance, defined in
100	include/media/media-entity.h. The structure is usually embedded into a
101	higher-level structure, such as a v4l2_subdev or video_device instance,
102	although drivers can allocate entities directly.
103	
104	Drivers initialize entities by calling
105	
106		media_entity_init(struct media_entity *entity, u16 num_pads,
107				  struct media_pad *pads, u16 extra_links);
108	
109	The media_entity name, type, flags, revision and group_id fields can be
110	initialized before or after calling media_entity_init. Entities embedded in
111	higher-level standard structures can have some of those fields set by the
112	higher-level framework.
113	
114	As the number of pads is known in advance, the pads array is not allocated
115	dynamically but is managed by the entity driver. Most drivers will embed the
116	pads array in a driver-specific structure, avoiding dynamic allocation.
117	
118	Drivers must set the direction of every pad in the pads array before calling
119	media_entity_init. The function will initialize the other pads fields.
120	
121	Unlike the number of pads, the total number of links isn't always known in
122	advance by the entity driver. As an initial estimate, media_entity_init
123	pre-allocates a number of links equal to the number of pads plus an optional
124	number of extra links. The links array will be reallocated if it grows beyond
125	the initial estimate.
126	
127	Drivers register entities with a media device by calling
128	
129		media_device_register_entity(struct media_device *mdev,
130					     struct media_entity *entity);
131	
132	Entities are identified by a unique positive integer ID. Drivers can provide an
133	ID by filling the media_entity id field prior to registration, or request the
134	media controller framework to assign an ID automatically. Drivers that provide
135	IDs manually must ensure that all IDs are unique. IDs are not guaranteed to be
136	contiguous even when they are all assigned automatically by the framework.
137	
138	Drivers unregister entities by calling
139	
140		media_device_unregister_entity(struct media_entity *entity);
141	
142	Unregistering an entity will not change the IDs of the other entities, and the
143	ID will never be reused for a newly registered entity.
144	
145	When a media device is unregistered, all its entities are unregistered
146	automatically. No manual entities unregistration is then required.
147	
148	Drivers free resources associated with an entity by calling
149	
150		media_entity_cleanup(struct media_entity *entity);
151	
152	This function must be called during the cleanup phase after unregistering the
153	entity. Note that the media_entity instance itself must be freed explicitly by
154	the driver if required.
155	
156	Entities have flags that describe the entity capabilities and state.
157	
158		MEDIA_ENT_FL_DEFAULT indicates the default entity for a given type.
159		This can be used to report the default audio and video devices or the
160		default camera sensor.
161	
162	Logical entity groups can be defined by setting the group ID of all member
163	entities to the same non-zero value. An entity group serves no purpose in the
164	kernel, but is reported to userspace during entities enumeration. The group_id
165	field belongs to the media device driver and must not by touched by entity
166	drivers.
167	
168	Media device drivers should define groups if several entities are logically
169	bound together. Example usages include reporting
170	
171		- ALSA, VBI and video nodes that carry the same media stream
172		- lens and flash controllers associated with a sensor
173	
174	- Pads
175	
176	Pads are represented by a struct media_pad instance, defined in
177	include/media/media-entity.h. Each entity stores its pads in a pads array
178	managed by the entity driver. Drivers usually embed the array in a
179	driver-specific structure.
180	
181	Pads are identified by their entity and their 0-based index in the pads array.
182	Both information are stored in the media_pad structure, making the media_pad
183	pointer the canonical way to store and pass link references.
184	
185	Pads have flags that describe the pad capabilities and state.
186	
187		MEDIA_PAD_FL_SINK indicates that the pad supports sinking data.
188		MEDIA_PAD_FL_SOURCE indicates that the pad supports sourcing data.
189	
190	One and only one of MEDIA_PAD_FL_SINK and MEDIA_PAD_FL_SOURCE must be set for
191	each pad.
192	
193	- Links
194	
195	Links are represented by a struct media_link instance, defined in
196	include/media/media-entity.h. Each entity stores all links originating at or
197	targeting any of its pads in a links array. A given link is thus stored
198	twice, once in the source entity and once in the target entity. The array is
199	pre-allocated and grows dynamically as needed.
200	
201	Drivers create links by calling
202	
203		media_entity_create_link(struct media_entity *source, u16 source_pad,
204					 struct media_entity *sink,   u16 sink_pad,
205					 u32 flags);
206	
207	An entry in the link array of each entity is allocated and stores pointers
208	to source and sink pads.
209	
210	Links have flags that describe the link capabilities and state.
211	
212		MEDIA_LNK_FL_ENABLED indicates that the link is enabled and can be used
213		to transfer media data. When two or more links target a sink pad, only
214		one of them can be enabled at a time.
215		MEDIA_LNK_FL_IMMUTABLE indicates that the link enabled state can't be
216		modified at runtime. If MEDIA_LNK_FL_IMMUTABLE is set, then
217		MEDIA_LNK_FL_ENABLED must also be set since an immutable link is always
218		enabled.
219	
220	
221	Graph traversal
222	---------------
223	
224	The media framework provides APIs to iterate over entities in a graph.
225	
226	To iterate over all entities belonging to a media device, drivers can use the
227	media_device_for_each_entity macro, defined in include/media/media-device.h.
228	
229		struct media_entity *entity;
230	
231		media_device_for_each_entity(entity, mdev) {
232			/* entity will point to each entity in turn */
233			...
234		}
235	
236	Drivers might also need to iterate over all entities in a graph that can be
237	reached only through enabled links starting at a given entity. The media
238	framework provides a depth-first graph traversal API for that purpose.
239	
240	Note that graphs with cycles (whether directed or undirected) are *NOT*
241	supported by the graph traversal API. To prevent infinite loops, the graph
242	traversal code limits the maximum depth to MEDIA_ENTITY_ENUM_MAX_DEPTH,
243	currently defined as 16.
244	
245	Drivers initiate a graph traversal by calling
246	
247		media_entity_graph_walk_start(struct media_entity_graph *graph,
248					      struct media_entity *entity);
249	
250	The graph structure, provided by the caller, is initialized to start graph
251	traversal at the given entity.
252	
253	Drivers can then retrieve the next entity by calling
254	
255		media_entity_graph_walk_next(struct media_entity_graph *graph);
256	
257	When the graph traversal is complete the function will return NULL.
258	
259	Graph traversal can be interrupted at any moment. No cleanup function call is
260	required and the graph structure can be freed normally.
261	
262	Helper functions can be used to find a link between two given pads, or a pad
263	connected to another pad through an enabled link
264	
265		media_entity_find_link(struct media_pad *source,
266				       struct media_pad *sink);
267	
268		media_entity_remote_pad(struct media_pad *pad);
269	
270	Refer to the kerneldoc documentation for more information.
271	
272	
273	Use count and power handling
274	----------------------------
275	
276	Due to the wide differences between drivers regarding power management needs,
277	the media controller does not implement power management. However, the
278	media_entity structure includes a use_count field that media drivers can use to
279	track the number of users of every entity for power management needs.
280	
281	The use_count field is owned by media drivers and must not be touched by entity
282	drivers. Access to the field must be protected by the media device graph_mutex
283	lock.
284	
285	
286	Links setup
287	-----------
288	
289	Link properties can be modified at runtime by calling
290	
291		media_entity_setup_link(struct media_link *link, u32 flags);
292	
293	The flags argument contains the requested new link flags.
294	
295	The only configurable property is the ENABLED link flag to enable/disable a
296	link. Links marked with the IMMUTABLE link flag can not be enabled or disabled.
297	
298	When a link is enabled or disabled, the media framework calls the
299	link_setup operation for the two entities at the source and sink of the link,
300	in that order. If the second link_setup call fails, another link_setup call is
301	made on the first entity to restore the original link flags.
302	
303	Media device drivers can be notified of link setup operations by setting the
304	media_device::link_notify pointer to a callback function. If provided, the
305	notification callback will be called before enabling and after disabling
306	links.
307	
308	Entity drivers must implement the link_setup operation if any of their links
309	is non-immutable. The operation must either configure the hardware or store
310	the configuration information to be applied later.
311	
312	Link configuration must not have any side effect on other links. If an enabled
313	link at a sink pad prevents another link at the same pad from being enabled,
314	the link_setup operation must return -EBUSY and can't implicitly disable the
315	first enabled link.
316	
317	
318	Pipelines and media streams
319	---------------------------
320	
321	When starting streaming, drivers must notify all entities in the pipeline to
322	prevent link states from being modified during streaming by calling
323	
324		media_entity_pipeline_start(struct media_entity *entity,
325					    struct media_pipeline *pipe);
326	
327	The function will mark all entities connected to the given entity through
328	enabled links, either directly or indirectly, as streaming.
329	
330	The media_pipeline instance pointed to by the pipe argument will be stored in
331	every entity in the pipeline. Drivers should embed the media_pipeline structure
332	in higher-level pipeline structures and can then access the pipeline through
333	the media_entity pipe field.
334	
335	Calls to media_entity_pipeline_start() can be nested. The pipeline pointer must
336	be identical for all nested calls to the function.
337	
338	media_entity_pipeline_start() may return an error. In that case, it will
339	clean up any of the changes it did by itself.
340	
341	When stopping the stream, drivers must notify the entities with
342	
343		media_entity_pipeline_stop(struct media_entity *entity);
344	
345	If multiple calls to media_entity_pipeline_start() have been made the same
346	number of media_entity_pipeline_stop() calls are required to stop streaming. The
347	media_entity pipe field is reset to NULL on the last nested stop call.
348	
349	Link configuration will fail with -EBUSY by default if either end of the link is
350	a streaming entity. Links that can be modified while streaming must be marked
351	with the MEDIA_LNK_FL_DYNAMIC flag.
352	
353	If other operations need to be disallowed on streaming entities (such as
354	changing entities configuration parameters) drivers can explicitly check the
355	media_entity stream_count field to find out if an entity is streaming. This
356	operation must be done with the media_device graph_mutex held.
357	
358	
359	Link validation
360	---------------
361	
362	Link validation is performed by media_entity_pipeline_start() for any
363	entity which has sink pads in the pipeline. The
364	media_entity::link_validate() callback is used for that purpose. In
365	link_validate() callback, entity driver should check that the properties of
366	the source pad of the connected entity and its own sink pad match. It is up
367	to the type of the entity (and in the end, the properties of the hardware)
368	what matching actually means.
369	
370	Subsystems should facilitate link validation by providing subsystem specific
371	helper functions to provide easy access for commonly needed information, and
372	in the end provide a way to use driver-specific callbacks.
Hide Line Numbers


About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog