About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / networking / timestamping.txt




Custom Search

Based on kernel version 4.7.2. Page generated on 2016-08-22 22:47 EST.

1	
2	1. Control Interfaces
3	
4	The interfaces for receiving network packages timestamps are:
5	
6	* SO_TIMESTAMP
7	  Generates a timestamp for each incoming packet in (not necessarily
8	  monotonic) system time. Reports the timestamp via recvmsg() in a
9	  control message as struct timeval (usec resolution).
10	
11	* SO_TIMESTAMPNS
12	  Same timestamping mechanism as SO_TIMESTAMP, but reports the
13	  timestamp as struct timespec (nsec resolution).
14	
15	* IP_MULTICAST_LOOP + SO_TIMESTAMP[NS]
16	  Only for multicast:approximate transmit timestamp obtained by
17	  reading the looped packet receive timestamp.
18	
19	* SO_TIMESTAMPING
20	  Generates timestamps on reception, transmission or both. Supports
21	  multiple timestamp sources, including hardware. Supports generating
22	  timestamps for stream sockets.
23	
24	
25	1.1 SO_TIMESTAMP:
26	
27	This socket option enables timestamping of datagrams on the reception
28	path. Because the destination socket, if any, is not known early in
29	the network stack, the feature has to be enabled for all packets. The
30	same is true for all early receive timestamp options.
31	
32	For interface details, see `man 7 socket`.
33	
34	
35	1.2 SO_TIMESTAMPNS:
36	
37	This option is identical to SO_TIMESTAMP except for the returned data type.
38	Its struct timespec allows for higher resolution (ns) timestamps than the
39	timeval of SO_TIMESTAMP (ms).
40	
41	
42	1.3 SO_TIMESTAMPING:
43	
44	Supports multiple types of timestamp requests. As a result, this
45	socket option takes a bitmap of flags, not a boolean. In
46	
47	  err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, (void *) val,
48	                   sizeof(val));
49	
50	val is an integer with any of the following bits set. Setting other
51	bit returns EINVAL and does not change the current state.
52	
53	The socket option configures timestamp generation for individual
54	sk_buffs (1.3.1), timestamp reporting to the socket's error
55	queue (1.3.2) and options (1.3.3). Timestamp generation can also
56	be enabled for individual sendmsg calls using cmsg (1.3.4).
57	
58	
59	1.3.1 Timestamp Generation
60	
61	Some bits are requests to the stack to try to generate timestamps. Any
62	combination of them is valid. Changes to these bits apply to newly
63	created packets, not to packets already in the stack. As a result, it
64	is possible to selectively request timestamps for a subset of packets
65	(e.g., for sampling) by embedding an send() call within two setsockopt
66	calls, one to enable timestamp generation and one to disable it.
67	Timestamps may also be generated for reasons other than being
68	requested by a particular socket, such as when receive timestamping is
69	enabled system wide, as explained earlier.
70	
71	SOF_TIMESTAMPING_RX_HARDWARE:
72	  Request rx timestamps generated by the network adapter.
73	
74	SOF_TIMESTAMPING_RX_SOFTWARE:
75	  Request rx timestamps when data enters the kernel. These timestamps
76	  are generated just after a device driver hands a packet to the
77	  kernel receive stack.
78	
79	SOF_TIMESTAMPING_TX_HARDWARE:
80	  Request tx timestamps generated by the network adapter. This flag
81	  can be enabled via both socket options and control messages.
82	
83	SOF_TIMESTAMPING_TX_SOFTWARE:
84	  Request tx timestamps when data leaves the kernel. These timestamps
85	  are generated in the device driver as close as possible, but always
86	  prior to, passing the packet to the network interface. Hence, they
87	  require driver support and may not be available for all devices.
88	  This flag can be enabled via both socket options and control messages.
89	
90	
91	SOF_TIMESTAMPING_TX_SCHED:
92	  Request tx timestamps prior to entering the packet scheduler. Kernel
93	  transmit latency is, if long, often dominated by queuing delay. The
94	  difference between this timestamp and one taken at
95	  SOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independent
96	  of protocol processing. The latency incurred in protocol
97	  processing, if any, can be computed by subtracting a userspace
98	  timestamp taken immediately before send() from this timestamp. On
99	  machines with virtual devices where a transmitted packet travels
100	  through multiple devices and, hence, multiple packet schedulers,
101	  a timestamp is generated at each layer. This allows for fine
102	  grained measurement of queuing delay. This flag can be enabled
103	  via both socket options and control messages.
104	
105	SOF_TIMESTAMPING_TX_ACK:
106	  Request tx timestamps when all data in the send buffer has been
107	  acknowledged. This only makes sense for reliable protocols. It is
108	  currently only implemented for TCP. For that protocol, it may
109	  over-report measurement, because the timestamp is generated when all
110	  data up to and including the buffer at send() was acknowledged: the
111	  cumulative acknowledgment. The mechanism ignores SACK and FACK.
112	  This flag can be enabled via both socket options and control messages.
113	
114	
115	1.3.2 Timestamp Reporting
116	
117	The other three bits control which timestamps will be reported in a
118	generated control message. Changes to the bits take immediate
119	effect at the timestamp reporting locations in the stack. Timestamps
120	are only reported for packets that also have the relevant timestamp
121	generation request set.
122	
123	SOF_TIMESTAMPING_SOFTWARE:
124	  Report any software timestamps when available.
125	
126	SOF_TIMESTAMPING_SYS_HARDWARE:
127	  This option is deprecated and ignored.
128	
129	SOF_TIMESTAMPING_RAW_HARDWARE:
130	  Report hardware timestamps as generated by
131	  SOF_TIMESTAMPING_TX_HARDWARE when available.
132	
133	
134	1.3.3 Timestamp Options
135	
136	The interface supports the options
137	
138	SOF_TIMESTAMPING_OPT_ID:
139	
140	  Generate a unique identifier along with each packet. A process can
141	  have multiple concurrent timestamping requests outstanding. Packets
142	  can be reordered in the transmit path, for instance in the packet
143	  scheduler. In that case timestamps will be queued onto the error
144	  queue out of order from the original send() calls. It is not always
145	  possible to uniquely match timestamps to the original send() calls
146	  based on timestamp order or payload inspection alone, then.
147	
148	  This option associates each packet at send() with a unique
149	  identifier and returns that along with the timestamp. The identifier
150	  is derived from a per-socket u32 counter (that wraps). For datagram
151	  sockets, the counter increments with each sent packet. For stream
152	  sockets, it increments with every byte.
153	
154	  The counter starts at zero. It is initialized the first time that
155	  the socket option is enabled. It is reset each time the option is
156	  enabled after having been disabled. Resetting the counter does not
157	  change the identifiers of existing packets in the system.
158	
159	  This option is implemented only for transmit timestamps. There, the
160	  timestamp is always looped along with a struct sock_extended_err.
161	  The option modifies field ee_data to pass an id that is unique
162	  among all possibly concurrently outstanding timestamp requests for
163	  that socket.
164	
165	
166	SOF_TIMESTAMPING_OPT_CMSG:
167	
168	  Support recv() cmsg for all timestamped packets. Control messages
169	  are already supported unconditionally on all packets with receive
170	  timestamps and on IPv6 packets with transmit timestamp. This option
171	  extends them to IPv4 packets with transmit timestamp. One use case
172	  is to correlate packets with their egress device, by enabling socket
173	  option IP_PKTINFO simultaneously.
174	
175	
176	SOF_TIMESTAMPING_OPT_TSONLY:
177	
178	  Applies to transmit timestamps only. Makes the kernel return the
179	  timestamp as a cmsg alongside an empty packet, as opposed to
180	  alongside the original packet. This reduces the amount of memory
181	  charged to the socket's receive budget (SO_RCVBUF) and delivers
182	  the timestamp even if sysctl net.core.tstamp_allow_data is 0.
183	  This option disables SOF_TIMESTAMPING_OPT_CMSG.
184	
185	
186	New applications are encouraged to pass SOF_TIMESTAMPING_OPT_ID to
187	disambiguate timestamps and SOF_TIMESTAMPING_OPT_TSONLY to operate
188	regardless of the setting of sysctl net.core.tstamp_allow_data.
189	
190	An exception is when a process needs additional cmsg data, for
191	instance SOL_IP/IP_PKTINFO to detect the egress network interface.
192	Then pass option SOF_TIMESTAMPING_OPT_CMSG. This option depends on
193	having access to the contents of the original packet, so cannot be
194	combined with SOF_TIMESTAMPING_OPT_TSONLY.
195	
196	
197	1.3.4. Enabling timestamps via control messages
198	
199	In addition to socket options, timestamp generation can be requested
200	per write via cmsg, only for SOF_TIMESTAMPING_TX_* (see Section 1.3.1).
201	Using this feature, applications can sample timestamps per sendmsg()
202	without paying the overhead of enabling and disabling timestamps via
203	setsockopt:
204	
205	  struct msghdr *msg;
206	  ...
207	  cmsg			       = CMSG_FIRSTHDR(msg);
208	  cmsg->cmsg_level	       = SOL_SOCKET;
209	  cmsg->cmsg_type	       = SO_TIMESTAMPING;
210	  cmsg->cmsg_len	       = CMSG_LEN(sizeof(__u32));
211	  *((__u32 *) CMSG_DATA(cmsg)) = SOF_TIMESTAMPING_TX_SCHED |
212					 SOF_TIMESTAMPING_TX_SOFTWARE |
213					 SOF_TIMESTAMPING_TX_ACK;
214	  err = sendmsg(fd, msg, 0);
215	
216	The SOF_TIMESTAMPING_TX_* flags set via cmsg will override
217	the SOF_TIMESTAMPING_TX_* flags set via setsockopt.
218	
219	Moreover, applications must still enable timestamp reporting via
220	setsockopt to receive timestamps:
221	
222	  __u32 val = SOF_TIMESTAMPING_SOFTWARE |
223		      SOF_TIMESTAMPING_OPT_ID /* or any other flag */;
224	  err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, (void *) val,
225	                   sizeof(val));
226	
227	
228	1.4 Bytestream Timestamps
229	
230	The SO_TIMESTAMPING interface supports timestamping of bytes in a
231	bytestream. Each request is interpreted as a request for when the
232	entire contents of the buffer has passed a timestamping point. That
233	is, for streams option SOF_TIMESTAMPING_TX_SOFTWARE will record
234	when all bytes have reached the device driver, regardless of how
235	many packets the data has been converted into.
236	
237	In general, bytestreams have no natural delimiters and therefore
238	correlating a timestamp with data is non-trivial. A range of bytes
239	may be split across segments, any segments may be merged (possibly
240	coalescing sections of previously segmented buffers associated with
241	independent send() calls). Segments can be reordered and the same
242	byte range can coexist in multiple segments for protocols that
243	implement retransmissions.
244	
245	It is essential that all timestamps implement the same semantics,
246	regardless of these possible transformations, as otherwise they are
247	incomparable. Handling "rare" corner cases differently from the
248	simple case (a 1:1 mapping from buffer to skb) is insufficient
249	because performance debugging often needs to focus on such outliers.
250	
251	In practice, timestamps can be correlated with segments of a
252	bytestream consistently, if both semantics of the timestamp and the
253	timing of measurement are chosen correctly. This challenge is no
254	different from deciding on a strategy for IP fragmentation. There, the
255	definition is that only the first fragment is timestamped. For
256	bytestreams, we chose that a timestamp is generated only when all
257	bytes have passed a point. SOF_TIMESTAMPING_TX_ACK as defined is easy to
258	implement and reason about. An implementation that has to take into
259	account SACK would be more complex due to possible transmission holes
260	and out of order arrival.
261	
262	On the host, TCP can also break the simple 1:1 mapping from buffer to
263	skbuff as a result of Nagle, cork, autocork, segmentation and GSO. The
264	implementation ensures correctness in all cases by tracking the
265	individual last byte passed to send(), even if it is no longer the
266	last byte after an skbuff extend or merge operation. It stores the
267	relevant sequence number in skb_shinfo(skb)->tskey. Because an skbuff
268	has only one such field, only one timestamp can be generated.
269	
270	In rare cases, a timestamp request can be missed if two requests are
271	collapsed onto the same skb. A process can detect this situation by
272	enabling SOF_TIMESTAMPING_OPT_ID and comparing the byte offset at
273	send time with the value returned for each timestamp. It can prevent
274	the situation by always flushing the TCP stack in between requests,
275	for instance by enabling TCP_NODELAY and disabling TCP_CORK and
276	autocork.
277	
278	These precautions ensure that the timestamp is generated only when all
279	bytes have passed a timestamp point, assuming that the network stack
280	itself does not reorder the segments. The stack indeed tries to avoid
281	reordering. The one exception is under administrator control: it is
282	possible to construct a packet scheduler configuration that delays
283	segments from the same stream differently. Such a setup would be
284	unusual.
285	
286	
287	2 Data Interfaces
288	
289	Timestamps are read using the ancillary data feature of recvmsg().
290	See `man 3 cmsg` for details of this interface. The socket manual
291	page (`man 7 socket`) describes how timestamps generated with
292	SO_TIMESTAMP and SO_TIMESTAMPNS records can be retrieved.
293	
294	
295	2.1 SCM_TIMESTAMPING records
296	
297	These timestamps are returned in a control message with cmsg_level
298	SOL_SOCKET, cmsg_type SCM_TIMESTAMPING, and payload of type
299	
300	struct scm_timestamping {
301		struct timespec ts[3];
302	};
303	
304	The structure can return up to three timestamps. This is a legacy
305	feature. Only one field is non-zero at any time. Most timestamps
306	are passed in ts[0]. Hardware timestamps are passed in ts[2].
307	
308	ts[1] used to hold hardware timestamps converted to system time.
309	Instead, expose the hardware clock device on the NIC directly as
310	a HW PTP clock source, to allow time conversion in userspace and
311	optionally synchronize system time with a userspace PTP stack such
312	as linuxptp. For the PTP clock API, see Documentation/ptp/ptp.txt.
313	
314	2.1.1 Transmit timestamps with MSG_ERRQUEUE
315	
316	For transmit timestamps the outgoing packet is looped back to the
317	socket's error queue with the send timestamp(s) attached. A process
318	receives the timestamps by calling recvmsg() with flag MSG_ERRQUEUE
319	set and with a msg_control buffer sufficiently large to receive the
320	relevant metadata structures. The recvmsg call returns the original
321	outgoing data packet with two ancillary messages attached.
322	
323	A message of cm_level SOL_IP(V6) and cm_type IP(V6)_RECVERR
324	embeds a struct sock_extended_err. This defines the error type. For
325	timestamps, the ee_errno field is ENOMSG. The other ancillary message
326	will have cm_level SOL_SOCKET and cm_type SCM_TIMESTAMPING. This
327	embeds the struct scm_timestamping.
328	
329	
330	2.1.1.2 Timestamp types
331	
332	The semantics of the three struct timespec are defined by field
333	ee_info in the extended error structure. It contains a value of
334	type SCM_TSTAMP_* to define the actual timestamp passed in
335	scm_timestamping.
336	
337	The SCM_TSTAMP_* types are 1:1 matches to the SOF_TIMESTAMPING_*
338	control fields discussed previously, with one exception. For legacy
339	reasons, SCM_TSTAMP_SND is equal to zero and can be set for both
340	SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE. It
341	is the first if ts[2] is non-zero, the second otherwise, in which
342	case the timestamp is stored in ts[0].
343	
344	
345	2.1.1.3 Fragmentation
346	
347	Fragmentation of outgoing datagrams is rare, but is possible, e.g., by
348	explicitly disabling PMTU discovery. If an outgoing packet is fragmented,
349	then only the first fragment is timestamped and returned to the sending
350	socket.
351	
352	
353	2.1.1.4 Packet Payload
354	
355	The calling application is often not interested in receiving the whole
356	packet payload that it passed to the stack originally: the socket
357	error queue mechanism is just a method to piggyback the timestamp on.
358	In this case, the application can choose to read datagrams with a
359	smaller buffer, possibly even of length 0. The payload is truncated
360	accordingly. Until the process calls recvmsg() on the error queue,
361	however, the full packet is queued, taking up budget from SO_RCVBUF.
362	
363	
364	2.1.1.5 Blocking Read
365	
366	Reading from the error queue is always a non-blocking operation. To
367	block waiting on a timestamp, use poll or select. poll() will return
368	POLLERR in pollfd.revents if any data is ready on the error queue.
369	There is no need to pass this flag in pollfd.events. This flag is
370	ignored on request. See also `man 2 poll`.
371	
372	
373	2.1.2 Receive timestamps
374	
375	On reception, there is no reason to read from the socket error queue.
376	The SCM_TIMESTAMPING ancillary data is sent along with the packet data
377	on a normal recvmsg(). Since this is not a socket error, it is not
378	accompanied by a message SOL_IP(V6)/IP(V6)_RECVERROR. In this case,
379	the meaning of the three fields in struct scm_timestamping is
380	implicitly defined. ts[0] holds a software timestamp if set, ts[1]
381	is again deprecated and ts[2] holds a hardware timestamp if set.
382	
383	
384	3. Hardware Timestamping configuration: SIOCSHWTSTAMP and SIOCGHWTSTAMP
385	
386	Hardware time stamping must also be initialized for each device driver
387	that is expected to do hardware time stamping. The parameter is defined in
388	/include/linux/net_tstamp.h as:
389	
390	struct hwtstamp_config {
391		int flags;	/* no flags defined right now, must be zero */
392		int tx_type;	/* HWTSTAMP_TX_* */
393		int rx_filter;	/* HWTSTAMP_FILTER_* */
394	};
395	
396	Desired behavior is passed into the kernel and to a specific device by
397	calling ioctl(SIOCSHWTSTAMP) with a pointer to a struct ifreq whose
398	ifr_data points to a struct hwtstamp_config. The tx_type and
399	rx_filter are hints to the driver what it is expected to do. If
400	the requested fine-grained filtering for incoming packets is not
401	supported, the driver may time stamp more than just the requested types
402	of packets.
403	
404	Drivers are free to use a more permissive configuration than the requested
405	configuration. It is expected that drivers should only implement directly the
406	most generic mode that can be supported. For example if the hardware can
407	support HWTSTAMP_FILTER_V2_EVENT, then it should generally always upscale
408	HWTSTAMP_FILTER_V2_L2_SYNC_MESSAGE, and so forth, as HWTSTAMP_FILTER_V2_EVENT
409	is more generic (and more useful to applications).
410	
411	A driver which supports hardware time stamping shall update the struct
412	with the actual, possibly more permissive configuration. If the
413	requested packets cannot be time stamped, then nothing should be
414	changed and ERANGE shall be returned (in contrast to EINVAL, which
415	indicates that SIOCSHWTSTAMP is not supported at all).
416	
417	Only a processes with admin rights may change the configuration. User
418	space is responsible to ensure that multiple processes don't interfere
419	with each other and that the settings are reset.
420	
421	Any process can read the actual configuration by passing this
422	structure to ioctl(SIOCGHWTSTAMP) in the same way.  However, this has
423	not been implemented in all drivers.
424	
425	/* possible values for hwtstamp_config->tx_type */
426	enum {
427		/*
428		 * no outgoing packet will need hardware time stamping;
429		 * should a packet arrive which asks for it, no hardware
430		 * time stamping will be done
431		 */
432		HWTSTAMP_TX_OFF,
433	
434		/*
435		 * enables hardware time stamping for outgoing packets;
436		 * the sender of the packet decides which are to be
437		 * time stamped by setting SOF_TIMESTAMPING_TX_SOFTWARE
438		 * before sending the packet
439		 */
440		HWTSTAMP_TX_ON,
441	};
442	
443	/* possible values for hwtstamp_config->rx_filter */
444	enum {
445		/* time stamp no incoming packet at all */
446		HWTSTAMP_FILTER_NONE,
447	
448		/* time stamp any incoming packet */
449		HWTSTAMP_FILTER_ALL,
450	
451		/* return value: time stamp all packets requested plus some others */
452		HWTSTAMP_FILTER_SOME,
453	
454		/* PTP v1, UDP, any kind of event packet */
455		HWTSTAMP_FILTER_PTP_V1_L4_EVENT,
456	
457		/* for the complete list of values, please check
458		 * the include file /include/linux/net_tstamp.h
459		 */
460	};
461	
462	3.1 Hardware Timestamping Implementation: Device Drivers
463	
464	A driver which supports hardware time stamping must support the
465	SIOCSHWTSTAMP ioctl and update the supplied struct hwtstamp_config with
466	the actual values as described in the section on SIOCSHWTSTAMP.  It
467	should also support SIOCGHWTSTAMP.
468	
469	Time stamps for received packets must be stored in the skb. To get a pointer
470	to the shared time stamp structure of the skb call skb_hwtstamps(). Then
471	set the time stamps in the structure:
472	
473	struct skb_shared_hwtstamps {
474		/* hardware time stamp transformed into duration
475		 * since arbitrary point in time
476		 */
477		ktime_t	hwtstamp;
478	};
479	
480	Time stamps for outgoing packets are to be generated as follows:
481	- In hard_start_xmit(), check if (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)
482	  is set no-zero. If yes, then the driver is expected to do hardware time
483	  stamping.
484	- If this is possible for the skb and requested, then declare
485	  that the driver is doing the time stamping by setting the flag
486	  SKBTX_IN_PROGRESS in skb_shinfo(skb)->tx_flags , e.g. with
487	
488	      skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
489	
490	  You might want to keep a pointer to the associated skb for the next step
491	  and not free the skb. A driver not supporting hardware time stamping doesn't
492	  do that. A driver must never touch sk_buff::tstamp! It is used to store
493	  software generated time stamps by the network subsystem.
494	- Driver should call skb_tx_timestamp() as close to passing sk_buff to hardware
495	  as possible. skb_tx_timestamp() provides a software time stamp if requested
496	  and hardware timestamping is not possible (SKBTX_IN_PROGRESS not set).
497	- As soon as the driver has sent the packet and/or obtained a
498	  hardware time stamp for it, it passes the time stamp back by
499	  calling skb_hwtstamp_tx() with the original skb, the raw
500	  hardware time stamp. skb_hwtstamp_tx() clones the original skb and
501	  adds the timestamps, therefore the original skb has to be freed now.
502	  If obtaining the hardware time stamp somehow fails, then the driver
503	  should not fall back to software time stamping. The rationale is that
504	  this would occur at a later time in the processing pipeline than other
505	  software time stamping and therefore could lead to unexpected deltas
506	  between time stamps.
Hide Line Numbers
About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Information is copyright its respective author. All material is available from the Linux Kernel Source distributed under a GPL License. This page is provided as a free service by mjmwired.net.