About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / networking / timestamping.txt




Custom Search

Based on kernel version 4.1. Page generated on 2015-06-28 12:13 EST.

1	
2	1. Control Interfaces
3	
4	The interfaces for receiving network packages timestamps are:
5	
6	* SO_TIMESTAMP
7	  Generates a timestamp for each incoming packet in (not necessarily
8	  monotonic) system time. Reports the timestamp via recvmsg() in a
9	  control message as struct timeval (usec resolution).
10	
11	* SO_TIMESTAMPNS
12	  Same timestamping mechanism as SO_TIMESTAMP, but reports the
13	  timestamp as struct timespec (nsec resolution).
14	
15	* IP_MULTICAST_LOOP + SO_TIMESTAMP[NS]
16	  Only for multicast:approximate transmit timestamp obtained by
17	  reading the looped packet receive timestamp.
18	
19	* SO_TIMESTAMPING
20	  Generates timestamps on reception, transmission or both. Supports
21	  multiple timestamp sources, including hardware. Supports generating
22	  timestamps for stream sockets.
23	
24	
25	1.1 SO_TIMESTAMP:
26	
27	This socket option enables timestamping of datagrams on the reception
28	path. Because the destination socket, if any, is not known early in
29	the network stack, the feature has to be enabled for all packets. The
30	same is true for all early receive timestamp options.
31	
32	For interface details, see `man 7 socket`.
33	
34	
35	1.2 SO_TIMESTAMPNS:
36	
37	This option is identical to SO_TIMESTAMP except for the returned data type.
38	Its struct timespec allows for higher resolution (ns) timestamps than the
39	timeval of SO_TIMESTAMP (ms).
40	
41	
42	1.3 SO_TIMESTAMPING:
43	
44	Supports multiple types of timestamp requests. As a result, this
45	socket option takes a bitmap of flags, not a boolean. In
46	
47	  err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, (void *) val, &val);
48	
49	val is an integer with any of the following bits set. Setting other
50	bit returns EINVAL and does not change the current state.
51	
52	
53	1.3.1 Timestamp Generation
54	
55	Some bits are requests to the stack to try to generate timestamps. Any
56	combination of them is valid. Changes to these bits apply to newly
57	created packets, not to packets already in the stack. As a result, it
58	is possible to selectively request timestamps for a subset of packets
59	(e.g., for sampling) by embedding an send() call within two setsockopt
60	calls, one to enable timestamp generation and one to disable it.
61	Timestamps may also be generated for reasons other than being
62	requested by a particular socket, such as when receive timestamping is
63	enabled system wide, as explained earlier.
64	
65	SOF_TIMESTAMPING_RX_HARDWARE:
66	  Request rx timestamps generated by the network adapter.
67	
68	SOF_TIMESTAMPING_RX_SOFTWARE:
69	  Request rx timestamps when data enters the kernel. These timestamps
70	  are generated just after a device driver hands a packet to the
71	  kernel receive stack.
72	
73	SOF_TIMESTAMPING_TX_HARDWARE:
74	  Request tx timestamps generated by the network adapter.
75	
76	SOF_TIMESTAMPING_TX_SOFTWARE:
77	  Request tx timestamps when data leaves the kernel. These timestamps
78	  are generated in the device driver as close as possible, but always
79	  prior to, passing the packet to the network interface. Hence, they
80	  require driver support and may not be available for all devices.
81	
82	SOF_TIMESTAMPING_TX_SCHED:
83	  Request tx timestamps prior to entering the packet scheduler. Kernel
84	  transmit latency is, if long, often dominated by queuing delay. The
85	  difference between this timestamp and one taken at
86	  SOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independent
87	  of protocol processing. The latency incurred in protocol
88	  processing, if any, can be computed by subtracting a userspace
89	  timestamp taken immediately before send() from this timestamp. On
90	  machines with virtual devices where a transmitted packet travels
91	  through multiple devices and, hence, multiple packet schedulers,
92	  a timestamp is generated at each layer. This allows for fine
93	  grained measurement of queuing delay.
94	
95	SOF_TIMESTAMPING_TX_ACK:
96	  Request tx timestamps when all data in the send buffer has been
97	  acknowledged. This only makes sense for reliable protocols. It is
98	  currently only implemented for TCP. For that protocol, it may
99	  over-report measurement, because the timestamp is generated when all
100	  data up to and including the buffer at send() was acknowledged: the
101	  cumulative acknowledgment. The mechanism ignores SACK and FACK.
102	
103	
104	1.3.2 Timestamp Reporting
105	
106	The other three bits control which timestamps will be reported in a
107	generated control message. Changes to the bits take immediate
108	effect at the timestamp reporting locations in the stack. Timestamps
109	are only reported for packets that also have the relevant timestamp
110	generation request set.
111	
112	SOF_TIMESTAMPING_SOFTWARE:
113	  Report any software timestamps when available.
114	
115	SOF_TIMESTAMPING_SYS_HARDWARE:
116	  This option is deprecated and ignored.
117	
118	SOF_TIMESTAMPING_RAW_HARDWARE:
119	  Report hardware timestamps as generated by
120	  SOF_TIMESTAMPING_TX_HARDWARE when available.
121	
122	
123	1.3.3 Timestamp Options
124	
125	The interface supports the options
126	
127	SOF_TIMESTAMPING_OPT_ID:
128	
129	  Generate a unique identifier along with each packet. A process can
130	  have multiple concurrent timestamping requests outstanding. Packets
131	  can be reordered in the transmit path, for instance in the packet
132	  scheduler. In that case timestamps will be queued onto the error
133	  queue out of order from the original send() calls. It is not always
134	  possible to uniquely match timestamps to the original send() calls
135	  based on timestamp order or payload inspection alone, then.
136	
137	  This option associates each packet at send() with a unique
138	  identifier and returns that along with the timestamp. The identifier
139	  is derived from a per-socket u32 counter (that wraps). For datagram
140	  sockets, the counter increments with each sent packet. For stream
141	  sockets, it increments with every byte.
142	
143	  The counter starts at zero. It is initialized the first time that
144	  the socket option is enabled. It is reset each time the option is
145	  enabled after having been disabled. Resetting the counter does not
146	  change the identifiers of existing packets in the system.
147	
148	  This option is implemented only for transmit timestamps. There, the
149	  timestamp is always looped along with a struct sock_extended_err.
150	  The option modifies field ee_data to pass an id that is unique
151	  among all possibly concurrently outstanding timestamp requests for
152	  that socket.
153	
154	
155	SOF_TIMESTAMPING_OPT_CMSG:
156	
157	  Support recv() cmsg for all timestamped packets. Control messages
158	  are already supported unconditionally on all packets with receive
159	  timestamps and on IPv6 packets with transmit timestamp. This option
160	  extends them to IPv4 packets with transmit timestamp. One use case
161	  is to correlate packets with their egress device, by enabling socket
162	  option IP_PKTINFO simultaneously.
163	
164	
165	SOF_TIMESTAMPING_OPT_TSONLY:
166	
167	  Applies to transmit timestamps only. Makes the kernel return the
168	  timestamp as a cmsg alongside an empty packet, as opposed to
169	  alongside the original packet. This reduces the amount of memory
170	  charged to the socket's receive budget (SO_RCVBUF) and delivers
171	  the timestamp even if sysctl net.core.tstamp_allow_data is 0.
172	  This option disables SOF_TIMESTAMPING_OPT_CMSG.
173	
174	
175	New applications are encouraged to pass SOF_TIMESTAMPING_OPT_ID to
176	disambiguate timestamps and SOF_TIMESTAMPING_OPT_TSONLY to operate
177	regardless of the setting of sysctl net.core.tstamp_allow_data.
178	
179	An exception is when a process needs additional cmsg data, for
180	instance SOL_IP/IP_PKTINFO to detect the egress network interface.
181	Then pass option SOF_TIMESTAMPING_OPT_CMSG. This option depends on
182	having access to the contents of the original packet, so cannot be
183	combined with SOF_TIMESTAMPING_OPT_TSONLY.
184	
185	
186	1.4 Bytestream Timestamps
187	
188	The SO_TIMESTAMPING interface supports timestamping of bytes in a
189	bytestream. Each request is interpreted as a request for when the
190	entire contents of the buffer has passed a timestamping point. That
191	is, for streams option SOF_TIMESTAMPING_TX_SOFTWARE will record
192	when all bytes have reached the device driver, regardless of how
193	many packets the data has been converted into.
194	
195	In general, bytestreams have no natural delimiters and therefore
196	correlating a timestamp with data is non-trivial. A range of bytes
197	may be split across segments, any segments may be merged (possibly
198	coalescing sections of previously segmented buffers associated with
199	independent send() calls). Segments can be reordered and the same
200	byte range can coexist in multiple segments for protocols that
201	implement retransmissions.
202	
203	It is essential that all timestamps implement the same semantics,
204	regardless of these possible transformations, as otherwise they are
205	incomparable. Handling "rare" corner cases differently from the
206	simple case (a 1:1 mapping from buffer to skb) is insufficient
207	because performance debugging often needs to focus on such outliers.
208	
209	In practice, timestamps can be correlated with segments of a
210	bytestream consistently, if both semantics of the timestamp and the
211	timing of measurement are chosen correctly. This challenge is no
212	different from deciding on a strategy for IP fragmentation. There, the
213	definition is that only the first fragment is timestamped. For
214	bytestreams, we chose that a timestamp is generated only when all
215	bytes have passed a point. SOF_TIMESTAMPING_TX_ACK as defined is easy to
216	implement and reason about. An implementation that has to take into
217	account SACK would be more complex due to possible transmission holes
218	and out of order arrival.
219	
220	On the host, TCP can also break the simple 1:1 mapping from buffer to
221	skbuff as a result of Nagle, cork, autocork, segmentation and GSO. The
222	implementation ensures correctness in all cases by tracking the
223	individual last byte passed to send(), even if it is no longer the
224	last byte after an skbuff extend or merge operation. It stores the
225	relevant sequence number in skb_shinfo(skb)->tskey. Because an skbuff
226	has only one such field, only one timestamp can be generated.
227	
228	In rare cases, a timestamp request can be missed if two requests are
229	collapsed onto the same skb. A process can detect this situation by
230	enabling SOF_TIMESTAMPING_OPT_ID and comparing the byte offset at
231	send time with the value returned for each timestamp. It can prevent
232	the situation by always flushing the TCP stack in between requests,
233	for instance by enabling TCP_NODELAY and disabling TCP_CORK and
234	autocork.
235	
236	These precautions ensure that the timestamp is generated only when all
237	bytes have passed a timestamp point, assuming that the network stack
238	itself does not reorder the segments. The stack indeed tries to avoid
239	reordering. The one exception is under administrator control: it is
240	possible to construct a packet scheduler configuration that delays
241	segments from the same stream differently. Such a setup would be
242	unusual.
243	
244	
245	2 Data Interfaces
246	
247	Timestamps are read using the ancillary data feature of recvmsg().
248	See `man 3 cmsg` for details of this interface. The socket manual
249	page (`man 7 socket`) describes how timestamps generated with
250	SO_TIMESTAMP and SO_TIMESTAMPNS records can be retrieved.
251	
252	
253	2.1 SCM_TIMESTAMPING records
254	
255	These timestamps are returned in a control message with cmsg_level
256	SOL_SOCKET, cmsg_type SCM_TIMESTAMPING, and payload of type
257	
258	struct scm_timestamping {
259		struct timespec ts[3];
260	};
261	
262	The structure can return up to three timestamps. This is a legacy
263	feature. Only one field is non-zero at any time. Most timestamps
264	are passed in ts[0]. Hardware timestamps are passed in ts[2].
265	
266	ts[1] used to hold hardware timestamps converted to system time.
267	Instead, expose the hardware clock device on the NIC directly as
268	a HW PTP clock source, to allow time conversion in userspace and
269	optionally synchronize system time with a userspace PTP stack such
270	as linuxptp. For the PTP clock API, see Documentation/ptp/ptp.txt.
271	
272	2.1.1 Transmit timestamps with MSG_ERRQUEUE
273	
274	For transmit timestamps the outgoing packet is looped back to the
275	socket's error queue with the send timestamp(s) attached. A process
276	receives the timestamps by calling recvmsg() with flag MSG_ERRQUEUE
277	set and with a msg_control buffer sufficiently large to receive the
278	relevant metadata structures. The recvmsg call returns the original
279	outgoing data packet with two ancillary messages attached.
280	
281	A message of cm_level SOL_IP(V6) and cm_type IP(V6)_RECVERR
282	embeds a struct sock_extended_err. This defines the error type. For
283	timestamps, the ee_errno field is ENOMSG. The other ancillary message
284	will have cm_level SOL_SOCKET and cm_type SCM_TIMESTAMPING. This
285	embeds the struct scm_timestamping.
286	
287	
288	2.1.1.2 Timestamp types
289	
290	The semantics of the three struct timespec are defined by field
291	ee_info in the extended error structure. It contains a value of
292	type SCM_TSTAMP_* to define the actual timestamp passed in
293	scm_timestamping.
294	
295	The SCM_TSTAMP_* types are 1:1 matches to the SOF_TIMESTAMPING_*
296	control fields discussed previously, with one exception. For legacy
297	reasons, SCM_TSTAMP_SND is equal to zero and can be set for both
298	SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE. It
299	is the first if ts[2] is non-zero, the second otherwise, in which
300	case the timestamp is stored in ts[0].
301	
302	
303	2.1.1.3 Fragmentation
304	
305	Fragmentation of outgoing datagrams is rare, but is possible, e.g., by
306	explicitly disabling PMTU discovery. If an outgoing packet is fragmented,
307	then only the first fragment is timestamped and returned to the sending
308	socket.
309	
310	
311	2.1.1.4 Packet Payload
312	
313	The calling application is often not interested in receiving the whole
314	packet payload that it passed to the stack originally: the socket
315	error queue mechanism is just a method to piggyback the timestamp on.
316	In this case, the application can choose to read datagrams with a
317	smaller buffer, possibly even of length 0. The payload is truncated
318	accordingly. Until the process calls recvmsg() on the error queue,
319	however, the full packet is queued, taking up budget from SO_RCVBUF.
320	
321	
322	2.1.1.5 Blocking Read
323	
324	Reading from the error queue is always a non-blocking operation. To
325	block waiting on a timestamp, use poll or select. poll() will return
326	POLLERR in pollfd.revents if any data is ready on the error queue.
327	There is no need to pass this flag in pollfd.events. This flag is
328	ignored on request. See also `man 2 poll`.
329	
330	
331	2.1.2 Receive timestamps
332	
333	On reception, there is no reason to read from the socket error queue.
334	The SCM_TIMESTAMPING ancillary data is sent along with the packet data
335	on a normal recvmsg(). Since this is not a socket error, it is not
336	accompanied by a message SOL_IP(V6)/IP(V6)_RECVERROR. In this case,
337	the meaning of the three fields in struct scm_timestamping is
338	implicitly defined. ts[0] holds a software timestamp if set, ts[1]
339	is again deprecated and ts[2] holds a hardware timestamp if set.
340	
341	
342	3. Hardware Timestamping configuration: SIOCSHWTSTAMP and SIOCGHWTSTAMP
343	
344	Hardware time stamping must also be initialized for each device driver
345	that is expected to do hardware time stamping. The parameter is defined in
346	/include/linux/net_tstamp.h as:
347	
348	struct hwtstamp_config {
349		int flags;	/* no flags defined right now, must be zero */
350		int tx_type;	/* HWTSTAMP_TX_* */
351		int rx_filter;	/* HWTSTAMP_FILTER_* */
352	};
353	
354	Desired behavior is passed into the kernel and to a specific device by
355	calling ioctl(SIOCSHWTSTAMP) with a pointer to a struct ifreq whose
356	ifr_data points to a struct hwtstamp_config. The tx_type and
357	rx_filter are hints to the driver what it is expected to do. If
358	the requested fine-grained filtering for incoming packets is not
359	supported, the driver may time stamp more than just the requested types
360	of packets.
361	
362	A driver which supports hardware time stamping shall update the struct
363	with the actual, possibly more permissive configuration. If the
364	requested packets cannot be time stamped, then nothing should be
365	changed and ERANGE shall be returned (in contrast to EINVAL, which
366	indicates that SIOCSHWTSTAMP is not supported at all).
367	
368	Only a processes with admin rights may change the configuration. User
369	space is responsible to ensure that multiple processes don't interfere
370	with each other and that the settings are reset.
371	
372	Any process can read the actual configuration by passing this
373	structure to ioctl(SIOCGHWTSTAMP) in the same way.  However, this has
374	not been implemented in all drivers.
375	
376	/* possible values for hwtstamp_config->tx_type */
377	enum {
378		/*
379		 * no outgoing packet will need hardware time stamping;
380		 * should a packet arrive which asks for it, no hardware
381		 * time stamping will be done
382		 */
383		HWTSTAMP_TX_OFF,
384	
385		/*
386		 * enables hardware time stamping for outgoing packets;
387		 * the sender of the packet decides which are to be
388		 * time stamped by setting SOF_TIMESTAMPING_TX_SOFTWARE
389		 * before sending the packet
390		 */
391		HWTSTAMP_TX_ON,
392	};
393	
394	/* possible values for hwtstamp_config->rx_filter */
395	enum {
396		/* time stamp no incoming packet at all */
397		HWTSTAMP_FILTER_NONE,
398	
399		/* time stamp any incoming packet */
400		HWTSTAMP_FILTER_ALL,
401	
402		/* return value: time stamp all packets requested plus some others */
403		HWTSTAMP_FILTER_SOME,
404	
405		/* PTP v1, UDP, any kind of event packet */
406		HWTSTAMP_FILTER_PTP_V1_L4_EVENT,
407	
408		/* for the complete list of values, please check
409		 * the include file /include/linux/net_tstamp.h
410		 */
411	};
412	
413	3.1 Hardware Timestamping Implementation: Device Drivers
414	
415	A driver which supports hardware time stamping must support the
416	SIOCSHWTSTAMP ioctl and update the supplied struct hwtstamp_config with
417	the actual values as described in the section on SIOCSHWTSTAMP.  It
418	should also support SIOCGHWTSTAMP.
419	
420	Time stamps for received packets must be stored in the skb. To get a pointer
421	to the shared time stamp structure of the skb call skb_hwtstamps(). Then
422	set the time stamps in the structure:
423	
424	struct skb_shared_hwtstamps {
425		/* hardware time stamp transformed into duration
426		 * since arbitrary point in time
427		 */
428		ktime_t	hwtstamp;
429	};
430	
431	Time stamps for outgoing packets are to be generated as follows:
432	- In hard_start_xmit(), check if (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)
433	  is set no-zero. If yes, then the driver is expected to do hardware time
434	  stamping.
435	- If this is possible for the skb and requested, then declare
436	  that the driver is doing the time stamping by setting the flag
437	  SKBTX_IN_PROGRESS in skb_shinfo(skb)->tx_flags , e.g. with
438	
439	      skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
440	
441	  You might want to keep a pointer to the associated skb for the next step
442	  and not free the skb. A driver not supporting hardware time stamping doesn't
443	  do that. A driver must never touch sk_buff::tstamp! It is used to store
444	  software generated time stamps by the network subsystem.
445	- Driver should call skb_tx_timestamp() as close to passing sk_buff to hardware
446	  as possible. skb_tx_timestamp() provides a software time stamp if requested
447	  and hardware timestamping is not possible (SKBTX_IN_PROGRESS not set).
448	- As soon as the driver has sent the packet and/or obtained a
449	  hardware time stamp for it, it passes the time stamp back by
450	  calling skb_hwtstamp_tx() with the original skb, the raw
451	  hardware time stamp. skb_hwtstamp_tx() clones the original skb and
452	  adds the timestamps, therefore the original skb has to be freed now.
453	  If obtaining the hardware time stamp somehow fails, then the driver
454	  should not fall back to software time stamping. The rationale is that
455	  this would occur at a later time in the processing pipeline than other
456	  software time stamping and therefore could lead to unexpected deltas
457	  between time stamps.
Hide Line Numbers
About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Information is copyright its respective author. All material is available from the Linux Kernel Source distributed under a GPL License. This page is provided as a free service by mjmwired.net.