About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / networking / timestamping.txt




Custom Search

Based on kernel version 3.19. Page generated on 2015-02-13 21:22 EST.

1	
2	1. Control Interfaces
3	
4	The interfaces for receiving network packages timestamps are:
5	
6	* SO_TIMESTAMP
7	  Generates a timestamp for each incoming packet in (not necessarily
8	  monotonic) system time. Reports the timestamp via recvmsg() in a
9	  control message as struct timeval (usec resolution).
10	
11	* SO_TIMESTAMPNS
12	  Same timestamping mechanism as SO_TIMESTAMP, but reports the
13	  timestamp as struct timespec (nsec resolution).
14	
15	* IP_MULTICAST_LOOP + SO_TIMESTAMP[NS]
16	  Only for multicast:approximate transmit timestamp obtained by
17	  reading the looped packet receive timestamp.
18	
19	* SO_TIMESTAMPING
20	  Generates timestamps on reception, transmission or both. Supports
21	  multiple timestamp sources, including hardware. Supports generating
22	  timestamps for stream sockets.
23	
24	
25	1.1 SO_TIMESTAMP:
26	
27	This socket option enables timestamping of datagrams on the reception
28	path. Because the destination socket, if any, is not known early in
29	the network stack, the feature has to be enabled for all packets. The
30	same is true for all early receive timestamp options.
31	
32	For interface details, see `man 7 socket`.
33	
34	
35	1.2 SO_TIMESTAMPNS:
36	
37	This option is identical to SO_TIMESTAMP except for the returned data type.
38	Its struct timespec allows for higher resolution (ns) timestamps than the
39	timeval of SO_TIMESTAMP (ms).
40	
41	
42	1.3 SO_TIMESTAMPING:
43	
44	Supports multiple types of timestamp requests. As a result, this
45	socket option takes a bitmap of flags, not a boolean. In
46	
47	  err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, (void *) val, &val);
48	
49	val is an integer with any of the following bits set. Setting other
50	bit returns EINVAL and does not change the current state.
51	
52	
53	1.3.1 Timestamp Generation
54	
55	Some bits are requests to the stack to try to generate timestamps. Any
56	combination of them is valid. Changes to these bits apply to newly
57	created packets, not to packets already in the stack. As a result, it
58	is possible to selectively request timestamps for a subset of packets
59	(e.g., for sampling) by embedding an send() call within two setsockopt
60	calls, one to enable timestamp generation and one to disable it.
61	Timestamps may also be generated for reasons other than being
62	requested by a particular socket, such as when receive timestamping is
63	enabled system wide, as explained earlier.
64	
65	SOF_TIMESTAMPING_RX_HARDWARE:
66	  Request rx timestamps generated by the network adapter.
67	
68	SOF_TIMESTAMPING_RX_SOFTWARE:
69	  Request rx timestamps when data enters the kernel. These timestamps
70	  are generated just after a device driver hands a packet to the
71	  kernel receive stack.
72	
73	SOF_TIMESTAMPING_TX_HARDWARE:
74	  Request tx timestamps generated by the network adapter.
75	
76	SOF_TIMESTAMPING_TX_SOFTWARE:
77	  Request tx timestamps when data leaves the kernel. These timestamps
78	  are generated in the device driver as close as possible, but always
79	  prior to, passing the packet to the network interface. Hence, they
80	  require driver support and may not be available for all devices.
81	
82	SOF_TIMESTAMPING_TX_SCHED:
83	  Request tx timestamps prior to entering the packet scheduler. Kernel
84	  transmit latency is, if long, often dominated by queuing delay. The
85	  difference between this timestamp and one taken at
86	  SOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independent
87	  of protocol processing. The latency incurred in protocol
88	  processing, if any, can be computed by subtracting a userspace
89	  timestamp taken immediately before send() from this timestamp. On
90	  machines with virtual devices where a transmitted packet travels
91	  through multiple devices and, hence, multiple packet schedulers,
92	  a timestamp is generated at each layer. This allows for fine
93	  grained measurement of queuing delay.
94	
95	SOF_TIMESTAMPING_TX_ACK:
96	  Request tx timestamps when all data in the send buffer has been
97	  acknowledged. This only makes sense for reliable protocols. It is
98	  currently only implemented for TCP. For that protocol, it may
99	  over-report measurement, because the timestamp is generated when all
100	  data up to and including the buffer at send() was acknowledged: the
101	  cumulative acknowledgment. The mechanism ignores SACK and FACK.
102	
103	
104	1.3.2 Timestamp Reporting
105	
106	The other three bits control which timestamps will be reported in a
107	generated control message. Changes to the bits take immediate
108	effect at the timestamp reporting locations in the stack. Timestamps
109	are only reported for packets that also have the relevant timestamp
110	generation request set.
111	
112	SOF_TIMESTAMPING_SOFTWARE:
113	  Report any software timestamps when available.
114	
115	SOF_TIMESTAMPING_SYS_HARDWARE:
116	  This option is deprecated and ignored.
117	
118	SOF_TIMESTAMPING_RAW_HARDWARE:
119	  Report hardware timestamps as generated by
120	  SOF_TIMESTAMPING_TX_HARDWARE when available.
121	
122	
123	1.3.3 Timestamp Options
124	
125	The interface supports the options
126	
127	SOF_TIMESTAMPING_OPT_ID:
128	
129	  Generate a unique identifier along with each packet. A process can
130	  have multiple concurrent timestamping requests outstanding. Packets
131	  can be reordered in the transmit path, for instance in the packet
132	  scheduler. In that case timestamps will be queued onto the error
133	  queue out of order from the original send() calls. It is not always
134	  possible to uniquely match timestamps to the original send() calls
135	  based on timestamp order or payload inspection alone, then.
136	
137	  This option associates each packet at send() with a unique
138	  identifier and returns that along with the timestamp. The identifier
139	  is derived from a per-socket u32 counter (that wraps). For datagram
140	  sockets, the counter increments with each sent packet. For stream
141	  sockets, it increments with every byte.
142	
143	  The counter starts at zero. It is initialized the first time that
144	  the socket option is enabled. It is reset each time the option is
145	  enabled after having been disabled. Resetting the counter does not
146	  change the identifiers of existing packets in the system.
147	
148	  This option is implemented only for transmit timestamps. There, the
149	  timestamp is always looped along with a struct sock_extended_err.
150	  The option modifies field ee_data to pass an id that is unique
151	  among all possibly concurrently outstanding timestamp requests for
152	  that socket.
153	
154	
155	SOF_TIMESTAMPING_OPT_CMSG:
156	
157	  Support recv() cmsg for all timestamped packets. Control messages
158	  are already supported unconditionally on all packets with receive
159	  timestamps and on IPv6 packets with transmit timestamp. This option
160	  extends them to IPv4 packets with transmit timestamp. One use case
161	  is to correlate packets with their egress device, by enabling socket
162	  option IP_PKTINFO simultaneously.
163	
164	
165	1.4 Bytestream Timestamps
166	
167	The SO_TIMESTAMPING interface supports timestamping of bytes in a
168	bytestream. Each request is interpreted as a request for when the
169	entire contents of the buffer has passed a timestamping point. That
170	is, for streams option SOF_TIMESTAMPING_TX_SOFTWARE will record
171	when all bytes have reached the device driver, regardless of how
172	many packets the data has been converted into.
173	
174	In general, bytestreams have no natural delimiters and therefore
175	correlating a timestamp with data is non-trivial. A range of bytes
176	may be split across segments, any segments may be merged (possibly
177	coalescing sections of previously segmented buffers associated with
178	independent send() calls). Segments can be reordered and the same
179	byte range can coexist in multiple segments for protocols that
180	implement retransmissions.
181	
182	It is essential that all timestamps implement the same semantics,
183	regardless of these possible transformations, as otherwise they are
184	incomparable. Handling "rare" corner cases differently from the
185	simple case (a 1:1 mapping from buffer to skb) is insufficient
186	because performance debugging often needs to focus on such outliers.
187	
188	In practice, timestamps can be correlated with segments of a
189	bytestream consistently, if both semantics of the timestamp and the
190	timing of measurement are chosen correctly. This challenge is no
191	different from deciding on a strategy for IP fragmentation. There, the
192	definition is that only the first fragment is timestamped. For
193	bytestreams, we chose that a timestamp is generated only when all
194	bytes have passed a point. SOF_TIMESTAMPING_TX_ACK as defined is easy to
195	implement and reason about. An implementation that has to take into
196	account SACK would be more complex due to possible transmission holes
197	and out of order arrival.
198	
199	On the host, TCP can also break the simple 1:1 mapping from buffer to
200	skbuff as a result of Nagle, cork, autocork, segmentation and GSO. The
201	implementation ensures correctness in all cases by tracking the
202	individual last byte passed to send(), even if it is no longer the
203	last byte after an skbuff extend or merge operation. It stores the
204	relevant sequence number in skb_shinfo(skb)->tskey. Because an skbuff
205	has only one such field, only one timestamp can be generated.
206	
207	In rare cases, a timestamp request can be missed if two requests are
208	collapsed onto the same skb. A process can detect this situation by
209	enabling SOF_TIMESTAMPING_OPT_ID and comparing the byte offset at
210	send time with the value returned for each timestamp. It can prevent
211	the situation by always flushing the TCP stack in between requests,
212	for instance by enabling TCP_NODELAY and disabling TCP_CORK and
213	autocork.
214	
215	These precautions ensure that the timestamp is generated only when all
216	bytes have passed a timestamp point, assuming that the network stack
217	itself does not reorder the segments. The stack indeed tries to avoid
218	reordering. The one exception is under administrator control: it is
219	possible to construct a packet scheduler configuration that delays
220	segments from the same stream differently. Such a setup would be
221	unusual.
222	
223	
224	2 Data Interfaces
225	
226	Timestamps are read using the ancillary data feature of recvmsg().
227	See `man 3 cmsg` for details of this interface. The socket manual
228	page (`man 7 socket`) describes how timestamps generated with
229	SO_TIMESTAMP and SO_TIMESTAMPNS records can be retrieved.
230	
231	
232	2.1 SCM_TIMESTAMPING records
233	
234	These timestamps are returned in a control message with cmsg_level
235	SOL_SOCKET, cmsg_type SCM_TIMESTAMPING, and payload of type
236	
237	struct scm_timestamping {
238		struct timespec ts[3];
239	};
240	
241	The structure can return up to three timestamps. This is a legacy
242	feature. Only one field is non-zero at any time. Most timestamps
243	are passed in ts[0]. Hardware timestamps are passed in ts[2].
244	
245	ts[1] used to hold hardware timestamps converted to system time.
246	Instead, expose the hardware clock device on the NIC directly as
247	a HW PTP clock source, to allow time conversion in userspace and
248	optionally synchronize system time with a userspace PTP stack such
249	as linuxptp. For the PTP clock API, see Documentation/ptp/ptp.txt.
250	
251	2.1.1 Transmit timestamps with MSG_ERRQUEUE
252	
253	For transmit timestamps the outgoing packet is looped back to the
254	socket's error queue with the send timestamp(s) attached. A process
255	receives the timestamps by calling recvmsg() with flag MSG_ERRQUEUE
256	set and with a msg_control buffer sufficiently large to receive the
257	relevant metadata structures. The recvmsg call returns the original
258	outgoing data packet with two ancillary messages attached.
259	
260	A message of cm_level SOL_IP(V6) and cm_type IP(V6)_RECVERR
261	embeds a struct sock_extended_err. This defines the error type. For
262	timestamps, the ee_errno field is ENOMSG. The other ancillary message
263	will have cm_level SOL_SOCKET and cm_type SCM_TIMESTAMPING. This
264	embeds the struct scm_timestamping.
265	
266	
267	2.1.1.2 Timestamp types
268	
269	The semantics of the three struct timespec are defined by field
270	ee_info in the extended error structure. It contains a value of
271	type SCM_TSTAMP_* to define the actual timestamp passed in
272	scm_timestamping.
273	
274	The SCM_TSTAMP_* types are 1:1 matches to the SOF_TIMESTAMPING_*
275	control fields discussed previously, with one exception. For legacy
276	reasons, SCM_TSTAMP_SND is equal to zero and can be set for both
277	SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE. It
278	is the first if ts[2] is non-zero, the second otherwise, in which
279	case the timestamp is stored in ts[0].
280	
281	
282	2.1.1.3 Fragmentation
283	
284	Fragmentation of outgoing datagrams is rare, but is possible, e.g., by
285	explicitly disabling PMTU discovery. If an outgoing packet is fragmented,
286	then only the first fragment is timestamped and returned to the sending
287	socket.
288	
289	
290	2.1.1.4 Packet Payload
291	
292	The calling application is often not interested in receiving the whole
293	packet payload that it passed to the stack originally: the socket
294	error queue mechanism is just a method to piggyback the timestamp on.
295	In this case, the application can choose to read datagrams with a
296	smaller buffer, possibly even of length 0. The payload is truncated
297	accordingly. Until the process calls recvmsg() on the error queue,
298	however, the full packet is queued, taking up budget from SO_RCVBUF.
299	
300	
301	2.1.1.5 Blocking Read
302	
303	Reading from the error queue is always a non-blocking operation. To
304	block waiting on a timestamp, use poll or select. poll() will return
305	POLLERR in pollfd.revents if any data is ready on the error queue.
306	There is no need to pass this flag in pollfd.events. This flag is
307	ignored on request. See also `man 2 poll`.
308	
309	
310	2.1.2 Receive timestamps
311	
312	On reception, there is no reason to read from the socket error queue.
313	The SCM_TIMESTAMPING ancillary data is sent along with the packet data
314	on a normal recvmsg(). Since this is not a socket error, it is not
315	accompanied by a message SOL_IP(V6)/IP(V6)_RECVERROR. In this case,
316	the meaning of the three fields in struct scm_timestamping is
317	implicitly defined. ts[0] holds a software timestamp if set, ts[1]
318	is again deprecated and ts[2] holds a hardware timestamp if set.
319	
320	
321	3. Hardware Timestamping configuration: SIOCSHWTSTAMP and SIOCGHWTSTAMP
322	
323	Hardware time stamping must also be initialized for each device driver
324	that is expected to do hardware time stamping. The parameter is defined in
325	/include/linux/net_tstamp.h as:
326	
327	struct hwtstamp_config {
328		int flags;	/* no flags defined right now, must be zero */
329		int tx_type;	/* HWTSTAMP_TX_* */
330		int rx_filter;	/* HWTSTAMP_FILTER_* */
331	};
332	
333	Desired behavior is passed into the kernel and to a specific device by
334	calling ioctl(SIOCSHWTSTAMP) with a pointer to a struct ifreq whose
335	ifr_data points to a struct hwtstamp_config. The tx_type and
336	rx_filter are hints to the driver what it is expected to do. If
337	the requested fine-grained filtering for incoming packets is not
338	supported, the driver may time stamp more than just the requested types
339	of packets.
340	
341	A driver which supports hardware time stamping shall update the struct
342	with the actual, possibly more permissive configuration. If the
343	requested packets cannot be time stamped, then nothing should be
344	changed and ERANGE shall be returned (in contrast to EINVAL, which
345	indicates that SIOCSHWTSTAMP is not supported at all).
346	
347	Only a processes with admin rights may change the configuration. User
348	space is responsible to ensure that multiple processes don't interfere
349	with each other and that the settings are reset.
350	
351	Any process can read the actual configuration by passing this
352	structure to ioctl(SIOCGHWTSTAMP) in the same way.  However, this has
353	not been implemented in all drivers.
354	
355	/* possible values for hwtstamp_config->tx_type */
356	enum {
357		/*
358		 * no outgoing packet will need hardware time stamping;
359		 * should a packet arrive which asks for it, no hardware
360		 * time stamping will be done
361		 */
362		HWTSTAMP_TX_OFF,
363	
364		/*
365		 * enables hardware time stamping for outgoing packets;
366		 * the sender of the packet decides which are to be
367		 * time stamped by setting SOF_TIMESTAMPING_TX_SOFTWARE
368		 * before sending the packet
369		 */
370		HWTSTAMP_TX_ON,
371	};
372	
373	/* possible values for hwtstamp_config->rx_filter */
374	enum {
375		/* time stamp no incoming packet at all */
376		HWTSTAMP_FILTER_NONE,
377	
378		/* time stamp any incoming packet */
379		HWTSTAMP_FILTER_ALL,
380	
381		/* return value: time stamp all packets requested plus some others */
382		HWTSTAMP_FILTER_SOME,
383	
384		/* PTP v1, UDP, any kind of event packet */
385		HWTSTAMP_FILTER_PTP_V1_L4_EVENT,
386	
387		/* for the complete list of values, please check
388		 * the include file /include/linux/net_tstamp.h
389		 */
390	};
391	
392	3.1 Hardware Timestamping Implementation: Device Drivers
393	
394	A driver which supports hardware time stamping must support the
395	SIOCSHWTSTAMP ioctl and update the supplied struct hwtstamp_config with
396	the actual values as described in the section on SIOCSHWTSTAMP.  It
397	should also support SIOCGHWTSTAMP.
398	
399	Time stamps for received packets must be stored in the skb. To get a pointer
400	to the shared time stamp structure of the skb call skb_hwtstamps(). Then
401	set the time stamps in the structure:
402	
403	struct skb_shared_hwtstamps {
404		/* hardware time stamp transformed into duration
405		 * since arbitrary point in time
406		 */
407		ktime_t	hwtstamp;
408	};
409	
410	Time stamps for outgoing packets are to be generated as follows:
411	- In hard_start_xmit(), check if (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)
412	  is set no-zero. If yes, then the driver is expected to do hardware time
413	  stamping.
414	- If this is possible for the skb and requested, then declare
415	  that the driver is doing the time stamping by setting the flag
416	  SKBTX_IN_PROGRESS in skb_shinfo(skb)->tx_flags , e.g. with
417	
418	      skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
419	
420	  You might want to keep a pointer to the associated skb for the next step
421	  and not free the skb. A driver not supporting hardware time stamping doesn't
422	  do that. A driver must never touch sk_buff::tstamp! It is used to store
423	  software generated time stamps by the network subsystem.
424	- Driver should call skb_tx_timestamp() as close to passing sk_buff to hardware
425	  as possible. skb_tx_timestamp() provides a software time stamp if requested
426	  and hardware timestamping is not possible (SKBTX_IN_PROGRESS not set).
427	- As soon as the driver has sent the packet and/or obtained a
428	  hardware time stamp for it, it passes the time stamp back by
429	  calling skb_hwtstamp_tx() with the original skb, the raw
430	  hardware time stamp. skb_hwtstamp_tx() clones the original skb and
431	  adds the timestamps, therefore the original skb has to be freed now.
432	  If obtaining the hardware time stamp somehow fails, then the driver
433	  should not fall back to software time stamping. The rationale is that
434	  this would occur at a later time in the processing pipeline than other
435	  software time stamping and therefore could lead to unexpected deltas
436	  between time stamps.
Hide Line Numbers
About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Information is copyright its respective author. All material is available from the Linux Kernel Source distributed under a GPL License. This page is provided as a free service by mjmwired.net.