About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / networking / rxrpc.txt




Custom Search

Based on kernel version 4.16.1. Page generated on 2018-04-09 11:53 EST.

1				    ======================
2				    RxRPC NETWORK PROTOCOL
3				    ======================
4	
5	The RxRPC protocol driver provides a reliable two-phase transport on top of UDP
6	that can be used to perform RxRPC remote operations.  This is done over sockets
7	of AF_RXRPC family, using sendmsg() and recvmsg() with control data to send and
8	receive data, aborts and errors.
9	
10	Contents of this document:
11	
12	 (*) Overview.
13	
14	 (*) RxRPC protocol summary.
15	
16	 (*) AF_RXRPC driver model.
17	
18	 (*) Control messages.
19	
20	 (*) Socket options.
21	
22	 (*) Security.
23	
24	 (*) Example client usage.
25	
26	 (*) Example server usage.
27	
28	 (*) AF_RXRPC kernel interface.
29	
30	 (*) Configurable parameters.
31	
32	
33	========
34	OVERVIEW
35	========
36	
37	RxRPC is a two-layer protocol.  There is a session layer which provides
38	reliable virtual connections using UDP over IPv4 (or IPv6) as the transport
39	layer, but implements a real network protocol; and there's the presentation
40	layer which renders structured data to binary blobs and back again using XDR
41	(as does SunRPC):
42	
43			+-------------+
44			| Application |
45			+-------------+
46			|     XDR     |		Presentation
47			+-------------+
48			|    RxRPC    |		Session
49			+-------------+
50			|     UDP     |		Transport
51			+-------------+
52	
53	
54	AF_RXRPC provides:
55	
56	 (1) Part of an RxRPC facility for both kernel and userspace applications by
57	     making the session part of it a Linux network protocol (AF_RXRPC).
58	
59	 (2) A two-phase protocol.  The client transmits a blob (the request) and then
60	     receives a blob (the reply), and the server receives the request and then
61	     transmits the reply.
62	
63	 (3) Retention of the reusable bits of the transport system set up for one call
64	     to speed up subsequent calls.
65	
66	 (4) A secure protocol, using the Linux kernel's key retention facility to
67	     manage security on the client end.  The server end must of necessity be
68	     more active in security negotiations.
69	
70	AF_RXRPC does not provide XDR marshalling/presentation facilities.  That is
71	left to the application.  AF_RXRPC only deals in blobs.  Even the operation ID
72	is just the first four bytes of the request blob, and as such is beyond the
73	kernel's interest.
74	
75	
76	Sockets of AF_RXRPC family are:
77	
78	 (1) created as type SOCK_DGRAM;
79	
80	 (2) provided with a protocol of the type of underlying transport they're going
81	     to use - currently only PF_INET is supported.
82	
83	
84	The Andrew File System (AFS) is an example of an application that uses this and
85	that has both kernel (filesystem) and userspace (utility) components.
86	
87	
88	======================
89	RXRPC PROTOCOL SUMMARY
90	======================
91	
92	An overview of the RxRPC protocol:
93	
94	 (*) RxRPC sits on top of another networking protocol (UDP is the only option
95	     currently), and uses this to provide network transport.  UDP ports, for
96	     example, provide transport endpoints.
97	
98	 (*) RxRPC supports multiple virtual "connections" from any given transport
99	     endpoint, thus allowing the endpoints to be shared, even to the same
100	     remote endpoint.
101	
102	 (*) Each connection goes to a particular "service".  A connection may not go
103	     to multiple services.  A service may be considered the RxRPC equivalent of
104	     a port number.  AF_RXRPC permits multiple services to share an endpoint.
105	
106	 (*) Client-originating packets are marked, thus a transport endpoint can be
107	     shared between client and server connections (connections have a
108	     direction).
109	
110	 (*) Up to a billion connections may be supported concurrently between one
111	     local transport endpoint and one service on one remote endpoint.  An RxRPC
112	     connection is described by seven numbers:
113	
114		Local address	}
115		Local port	} Transport (UDP) address
116		Remote address	}
117		Remote port	}
118		Direction
119		Connection ID
120		Service ID
121	
122	 (*) Each RxRPC operation is a "call".  A connection may make up to four
123	     billion calls, but only up to four calls may be in progress on a
124	     connection at any one time.
125	
126	 (*) Calls are two-phase and asymmetric: the client sends its request data,
127	     which the service receives; then the service transmits the reply data
128	     which the client receives.
129	
130	 (*) The data blobs are of indefinite size, the end of a phase is marked with a
131	     flag in the packet.  The number of packets of data making up one blob may
132	     not exceed 4 billion, however, as this would cause the sequence number to
133	     wrap.
134	
135	 (*) The first four bytes of the request data are the service operation ID.
136	
137	 (*) Security is negotiated on a per-connection basis.  The connection is
138	     initiated by the first data packet on it arriving.  If security is
139	     requested, the server then issues a "challenge" and then the client
140	     replies with a "response".  If the response is successful, the security is
141	     set for the lifetime of that connection, and all subsequent calls made
142	     upon it use that same security.  In the event that the server lets a
143	     connection lapse before the client, the security will be renegotiated if
144	     the client uses the connection again.
145	
146	 (*) Calls use ACK packets to handle reliability.  Data packets are also
147	     explicitly sequenced per call.
148	
149	 (*) There are two types of positive acknowledgment: hard-ACKs and soft-ACKs.
150	     A hard-ACK indicates to the far side that all the data received to a point
151	     has been received and processed; a soft-ACK indicates that the data has
152	     been received but may yet be discarded and re-requested.  The sender may
153	     not discard any transmittable packets until they've been hard-ACK'd.
154	
155	 (*) Reception of a reply data packet implicitly hard-ACK's all the data
156	     packets that make up the request.
157	
158	 (*) An call is complete when the request has been sent, the reply has been
159	     received and the final hard-ACK on the last packet of the reply has
160	     reached the server.
161	
162	 (*) An call may be aborted by either end at any time up to its completion.
163	
164	
165	=====================
166	AF_RXRPC DRIVER MODEL
167	=====================
168	
169	About the AF_RXRPC driver:
170	
171	 (*) The AF_RXRPC protocol transparently uses internal sockets of the transport
172	     protocol to represent transport endpoints.
173	
174	 (*) AF_RXRPC sockets map onto RxRPC connection bundles.  Actual RxRPC
175	     connections are handled transparently.  One client socket may be used to
176	     make multiple simultaneous calls to the same service.  One server socket
177	     may handle calls from many clients.
178	
179	 (*) Additional parallel client connections will be initiated to support extra
180	     concurrent calls, up to a tunable limit.
181	
182	 (*) Each connection is retained for a certain amount of time [tunable] after
183	     the last call currently using it has completed in case a new call is made
184	     that could reuse it.
185	
186	 (*) Each internal UDP socket is retained [tunable] for a certain amount of
187	     time [tunable] after the last connection using it discarded, in case a new
188	     connection is made that could use it.
189	
190	 (*) A client-side connection is only shared between calls if they have have
191	     the same key struct describing their security (and assuming the calls
192	     would otherwise share the connection).  Non-secured calls would also be
193	     able to share connections with each other.
194	
195	 (*) A server-side connection is shared if the client says it is.
196	
197	 (*) ACK'ing is handled by the protocol driver automatically, including ping
198	     replying.
199	
200	 (*) SO_KEEPALIVE automatically pings the other side to keep the connection
201	     alive [TODO].
202	
203	 (*) If an ICMP error is received, all calls affected by that error will be
204	     aborted with an appropriate network error passed through recvmsg().
205	
206	
207	Interaction with the user of the RxRPC socket:
208	
209	 (*) A socket is made into a server socket by binding an address with a
210	     non-zero service ID.
211	
212	 (*) In the client, sending a request is achieved with one or more sendmsgs,
213	     followed by the reply being received with one or more recvmsgs.
214	
215	 (*) The first sendmsg for a request to be sent from a client contains a tag to
216	     be used in all other sendmsgs or recvmsgs associated with that call.  The
217	     tag is carried in the control data.
218	
219	 (*) connect() is used to supply a default destination address for a client
220	     socket.  This may be overridden by supplying an alternate address to the
221	     first sendmsg() of a call (struct msghdr::msg_name).
222	
223	 (*) If connect() is called on an unbound client, a random local port will
224	     bound before the operation takes place.
225	
226	 (*) A server socket may also be used to make client calls.  To do this, the
227	     first sendmsg() of the call must specify the target address.  The server's
228	     transport endpoint is used to send the packets.
229	
230	 (*) Once the application has received the last message associated with a call,
231	     the tag is guaranteed not to be seen again, and so it can be used to pin
232	     client resources.  A new call can then be initiated with the same tag
233	     without fear of interference.
234	
235	 (*) In the server, a request is received with one or more recvmsgs, then the
236	     the reply is transmitted with one or more sendmsgs, and then the final ACK
237	     is received with a last recvmsg.
238	
239	 (*) When sending data for a call, sendmsg is given MSG_MORE if there's more
240	     data to come on that call.
241	
242	 (*) When receiving data for a call, recvmsg flags MSG_MORE if there's more
243	     data to come for that call.
244	
245	 (*) When receiving data or messages for a call, MSG_EOR is flagged by recvmsg
246	     to indicate the terminal message for that call.
247	
248	 (*) A call may be aborted by adding an abort control message to the control
249	     data.  Issuing an abort terminates the kernel's use of that call's tag.
250	     Any messages waiting in the receive queue for that call will be discarded.
251	
252	 (*) Aborts, busy notifications and challenge packets are delivered by recvmsg,
253	     and control data messages will be set to indicate the context.  Receiving
254	     an abort or a busy message terminates the kernel's use of that call's tag.
255	
256	 (*) The control data part of the msghdr struct is used for a number of things:
257	
258	     (*) The tag of the intended or affected call.
259	
260	     (*) Sending or receiving errors, aborts and busy notifications.
261	
262	     (*) Notifications of incoming calls.
263	
264	     (*) Sending debug requests and receiving debug replies [TODO].
265	
266	 (*) When the kernel has received and set up an incoming call, it sends a
267	     message to server application to let it know there's a new call awaiting
268	     its acceptance [recvmsg reports a special control message].  The server
269	     application then uses sendmsg to assign a tag to the new call.  Once that
270	     is done, the first part of the request data will be delivered by recvmsg.
271	
272	 (*) The server application has to provide the server socket with a keyring of
273	     secret keys corresponding to the security types it permits.  When a secure
274	     connection is being set up, the kernel looks up the appropriate secret key
275	     in the keyring and then sends a challenge packet to the client and
276	     receives a response packet.  The kernel then checks the authorisation of
277	     the packet and either aborts the connection or sets up the security.
278	
279	 (*) The name of the key a client will use to secure its communications is
280	     nominated by a socket option.
281	
282	
283	Notes on sendmsg:
284	
285	 (*) MSG_WAITALL can be set to tell sendmsg to ignore signals if the peer is
286	     making progress at accepting packets within a reasonable time such that we
287	     manage to queue up all the data for transmission.  This requires the
288	     client to accept at least one packet per 2*RTT time period.
289	
290	     If this isn't set, sendmsg() will return immediately, either returning
291	     EINTR/ERESTARTSYS if nothing was consumed or returning the amount of data
292	     consumed.
293	
294	
295	Notes on recvmsg:
296	
297	 (*) If there's a sequence of data messages belonging to a particular call on
298	     the receive queue, then recvmsg will keep working through them until:
299	
300	     (a) it meets the end of that call's received data,
301	
302	     (b) it meets a non-data message,
303	
304	     (c) it meets a message belonging to a different call, or
305	
306	     (d) it fills the user buffer.
307	
308	     If recvmsg is called in blocking mode, it will keep sleeping, awaiting the
309	     reception of further data, until one of the above four conditions is met.
310	
311	 (2) MSG_PEEK operates similarly, but will return immediately if it has put any
312	     data in the buffer rather than sleeping until it can fill the buffer.
313	
314	 (3) If a data message is only partially consumed in filling a user buffer,
315	     then the remainder of that message will be left on the front of the queue
316	     for the next taker.  MSG_TRUNC will never be flagged.
317	
318	 (4) If there is more data to be had on a call (it hasn't copied the last byte
319	     of the last data message in that phase yet), then MSG_MORE will be
320	     flagged.
321	
322	
323	================
324	CONTROL MESSAGES
325	================
326	
327	AF_RXRPC makes use of control messages in sendmsg() and recvmsg() to multiplex
328	calls, to invoke certain actions and to report certain conditions.  These are:
329	
330		MESSAGE ID		SRT DATA	MEANING
331		=======================	=== ===========	===============================
332		RXRPC_USER_CALL_ID	sr- User ID	App's call specifier
333		RXRPC_ABORT		srt Abort code	Abort code to issue/received
334		RXRPC_ACK		-rt n/a		Final ACK received
335		RXRPC_NET_ERROR		-rt error num	Network error on call
336		RXRPC_BUSY		-rt n/a		Call rejected (server busy)
337		RXRPC_LOCAL_ERROR	-rt error num	Local error encountered
338		RXRPC_NEW_CALL		-r- n/a		New call received
339		RXRPC_ACCEPT		s-- n/a		Accept new call
340		RXRPC_EXCLUSIVE_CALL	s-- n/a		Make an exclusive client call
341		RXRPC_UPGRADE_SERVICE	s-- n/a		Client call can be upgraded
342		RXRPC_TX_LENGTH		s-- data len	Total length of Tx data
343	
344		(SRT = usable in Sendmsg / delivered by Recvmsg / Terminal message)
345	
346	 (*) RXRPC_USER_CALL_ID
347	
348	     This is used to indicate the application's call ID.  It's an unsigned long
349	     that the app specifies in the client by attaching it to the first data
350	     message or in the server by passing it in association with an RXRPC_ACCEPT
351	     message.  recvmsg() passes it in conjunction with all messages except
352	     those of the RXRPC_NEW_CALL message.
353	
354	 (*) RXRPC_ABORT
355	
356	     This is can be used by an application to abort a call by passing it to
357	     sendmsg, or it can be delivered by recvmsg to indicate a remote abort was
358	     received.  Either way, it must be associated with an RXRPC_USER_CALL_ID to
359	     specify the call affected.  If an abort is being sent, then error EBADSLT
360	     will be returned if there is no call with that user ID.
361	
362	 (*) RXRPC_ACK
363	
364	     This is delivered to a server application to indicate that the final ACK
365	     of a call was received from the client.  It will be associated with an
366	     RXRPC_USER_CALL_ID to indicate the call that's now complete.
367	
368	 (*) RXRPC_NET_ERROR
369	
370	     This is delivered to an application to indicate that an ICMP error message
371	     was encountered in the process of trying to talk to the peer.  An
372	     errno-class integer value will be included in the control message data
373	     indicating the problem, and an RXRPC_USER_CALL_ID will indicate the call
374	     affected.
375	
376	 (*) RXRPC_BUSY
377	
378	     This is delivered to a client application to indicate that a call was
379	     rejected by the server due to the server being busy.  It will be
380	     associated with an RXRPC_USER_CALL_ID to indicate the rejected call.
381	
382	 (*) RXRPC_LOCAL_ERROR
383	
384	     This is delivered to an application to indicate that a local error was
385	     encountered and that a call has been aborted because of it.  An
386	     errno-class integer value will be included in the control message data
387	     indicating the problem, and an RXRPC_USER_CALL_ID will indicate the call
388	     affected.
389	
390	 (*) RXRPC_NEW_CALL
391	
392	     This is delivered to indicate to a server application that a new call has
393	     arrived and is awaiting acceptance.  No user ID is associated with this,
394	     as a user ID must subsequently be assigned by doing an RXRPC_ACCEPT.
395	
396	 (*) RXRPC_ACCEPT
397	
398	     This is used by a server application to attempt to accept a call and
399	     assign it a user ID.  It should be associated with an RXRPC_USER_CALL_ID
400	     to indicate the user ID to be assigned.  If there is no call to be
401	     accepted (it may have timed out, been aborted, etc.), then sendmsg will
402	     return error ENODATA.  If the user ID is already in use by another call,
403	     then error EBADSLT will be returned.
404	
405	 (*) RXRPC_EXCLUSIVE_CALL
406	
407	     This is used to indicate that a client call should be made on a one-off
408	     connection.  The connection is discarded once the call has terminated.
409	
410	 (*) RXRPC_UPGRADE_SERVICE
411	
412	     This is used to make a client call to probe if the specified service ID
413	     may be upgraded by the server.  The caller must check msg_name returned to
414	     recvmsg() for the service ID actually in use.  The operation probed must
415	     be one that takes the same arguments in both services.
416	
417	     Once this has been used to establish the upgrade capability (or lack
418	     thereof) of the server, the service ID returned should be used for all
419	     future communication to that server and RXRPC_UPGRADE_SERVICE should no
420	     longer be set.
421	
422	 (*) RXRPC_TX_LENGTH
423	
424	     This is used to inform the kernel of the total amount of data that is
425	     going to be transmitted by a call (whether in a client request or a
426	     service response).  If given, it allows the kernel to encrypt from the
427	     userspace buffer directly to the packet buffers, rather than copying into
428	     the buffer and then encrypting in place.  This may only be given with the
429	     first sendmsg() providing data for a call.  EMSGSIZE will be generated if
430	     the amount of data actually given is different.
431	
432	     This takes a parameter of __s64 type that indicates how much will be
433	     transmitted.  This may not be less than zero.
434	
435	The symbol RXRPC__SUPPORTED is defined as one more than the highest control
436	message type supported.  At run time this can be queried by means of the
437	RXRPC_SUPPORTED_CMSG socket option (see below).
438	
439	
440	==============
441	SOCKET OPTIONS
442	==============
443	
444	AF_RXRPC sockets support a few socket options at the SOL_RXRPC level:
445	
446	 (*) RXRPC_SECURITY_KEY
447	
448	     This is used to specify the description of the key to be used.  The key is
449	     extracted from the calling process's keyrings with request_key() and
450	     should be of "rxrpc" type.
451	
452	     The optval pointer points to the description string, and optlen indicates
453	     how long the string is, without the NUL terminator.
454	
455	 (*) RXRPC_SECURITY_KEYRING
456	
457	     Similar to above but specifies a keyring of server secret keys to use (key
458	     type "keyring").  See the "Security" section.
459	
460	 (*) RXRPC_EXCLUSIVE_CONNECTION
461	
462	     This is used to request that new connections should be used for each call
463	     made subsequently on this socket.  optval should be NULL and optlen 0.
464	
465	 (*) RXRPC_MIN_SECURITY_LEVEL
466	
467	     This is used to specify the minimum security level required for calls on
468	     this socket.  optval must point to an int containing one of the following
469	     values:
470	
471	     (a) RXRPC_SECURITY_PLAIN
472	
473		 Encrypted checksum only.
474	
475	     (b) RXRPC_SECURITY_AUTH
476	
477		 Encrypted checksum plus packet padded and first eight bytes of packet
478		 encrypted - which includes the actual packet length.
479	
480	     (c) RXRPC_SECURITY_ENCRYPTED
481	
482		 Encrypted checksum plus entire packet padded and encrypted, including
483		 actual packet length.
484	
485	 (*) RXRPC_UPGRADEABLE_SERVICE
486	
487	     This is used to indicate that a service socket with two bindings may
488	     upgrade one bound service to the other if requested by the client.  optval
489	     must point to an array of two unsigned short ints.  The first is the
490	     service ID to upgrade from and the second the service ID to upgrade to.
491	
492	 (*) RXRPC_SUPPORTED_CMSG
493	
494	     This is a read-only option that writes an int into the buffer indicating
495	     the highest control message type supported.
496	
497	
498	========
499	SECURITY
500	========
501	
502	Currently, only the kerberos 4 equivalent protocol has been implemented
503	(security index 2 - rxkad).  This requires the rxkad module to be loaded and,
504	on the client, tickets of the appropriate type to be obtained from the AFS
505	kaserver or the kerberos server and installed as "rxrpc" type keys.  This is
506	normally done using the klog program.  An example simple klog program can be
507	found at:
508	
509		http://people.redhat.com/~dhowells/rxrpc/klog.c
510	
511	The payload provided to add_key() on the client should be of the following
512	form:
513	
514		struct rxrpc_key_sec2_v1 {
515			uint16_t	security_index;	/* 2 */
516			uint16_t	ticket_length;	/* length of ticket[] */
517			uint32_t	expiry;		/* time at which expires */
518			uint8_t		kvno;		/* key version number */
519			uint8_t		__pad[3];
520			uint8_t		session_key[8];	/* DES session key */
521			uint8_t		ticket[0];	/* the encrypted ticket */
522		};
523	
524	Where the ticket blob is just appended to the above structure.
525	
526	
527	For the server, keys of type "rxrpc_s" must be made available to the server.
528	They have a description of "<serviceID>:<securityIndex>" (eg: "52:2" for an
529	rxkad key for the AFS VL service).  When such a key is created, it should be
530	given the server's secret key as the instantiation data (see the example
531	below).
532	
533		add_key("rxrpc_s", "52:2", secret_key, 8, keyring);
534	
535	A keyring is passed to the server socket by naming it in a sockopt.  The server
536	socket then looks the server secret keys up in this keyring when secure
537	incoming connections are made.  This can be seen in an example program that can
538	be found at:
539	
540		http://people.redhat.com/~dhowells/rxrpc/listen.c
541	
542	
543	====================
544	EXAMPLE CLIENT USAGE
545	====================
546	
547	A client would issue an operation by:
548	
549	 (1) An RxRPC socket is set up by:
550	
551		client = socket(AF_RXRPC, SOCK_DGRAM, PF_INET);
552	
553	     Where the third parameter indicates the protocol family of the transport
554	     socket used - usually IPv4 but it can also be IPv6 [TODO].
555	
556	 (2) A local address can optionally be bound:
557	
558		struct sockaddr_rxrpc srx = {
559			.srx_family	= AF_RXRPC,
560			.srx_service	= 0,  /* we're a client */
561			.transport_type	= SOCK_DGRAM,	/* type of transport socket */
562			.transport.sin_family	= AF_INET,
563			.transport.sin_port	= htons(7000), /* AFS callback */
564			.transport.sin_address	= 0,  /* all local interfaces */
565		};
566		bind(client, &srx, sizeof(srx));
567	
568	     This specifies the local UDP port to be used.  If not given, a random
569	     non-privileged port will be used.  A UDP port may be shared between
570	     several unrelated RxRPC sockets.  Security is handled on a basis of
571	     per-RxRPC virtual connection.
572	
573	 (3) The security is set:
574	
575		const char *key = "AFS:cambridge.redhat.com";
576		setsockopt(client, SOL_RXRPC, RXRPC_SECURITY_KEY, key, strlen(key));
577	
578	     This issues a request_key() to get the key representing the security
579	     context.  The minimum security level can be set:
580	
581		unsigned int sec = RXRPC_SECURITY_ENCRYPTED;
582		setsockopt(client, SOL_RXRPC, RXRPC_MIN_SECURITY_LEVEL,
583			   &sec, sizeof(sec));
584	
585	 (4) The server to be contacted can then be specified (alternatively this can
586	     be done through sendmsg):
587	
588		struct sockaddr_rxrpc srx = {
589			.srx_family	= AF_RXRPC,
590			.srx_service	= VL_SERVICE_ID,
591			.transport_type	= SOCK_DGRAM,	/* type of transport socket */
592			.transport.sin_family	= AF_INET,
593			.transport.sin_port	= htons(7005), /* AFS volume manager */
594			.transport.sin_address	= ...,
595		};
596		connect(client, &srx, sizeof(srx));
597	
598	 (5) The request data should then be posted to the server socket using a series
599	     of sendmsg() calls, each with the following control message attached:
600	
601		RXRPC_USER_CALL_ID	- specifies the user ID for this call
602	
603	     MSG_MORE should be set in msghdr::msg_flags on all but the last part of
604	     the request.  Multiple requests may be made simultaneously.
605	
606	     An RXRPC_TX_LENGTH control message can also be specified on the first
607	     sendmsg() call.
608	
609	     If a call is intended to go to a destination other than the default
610	     specified through connect(), then msghdr::msg_name should be set on the
611	     first request message of that call.
612	
613	 (6) The reply data will then be posted to the server socket for recvmsg() to
614	     pick up.  MSG_MORE will be flagged by recvmsg() if there's more reply data
615	     for a particular call to be read.  MSG_EOR will be set on the terminal
616	     read for a call.
617	
618	     All data will be delivered with the following control message attached:
619	
620		RXRPC_USER_CALL_ID	- specifies the user ID for this call
621	
622	     If an abort or error occurred, this will be returned in the control data
623	     buffer instead, and MSG_EOR will be flagged to indicate the end of that
624	     call.
625	
626	A client may ask for a service ID it knows and ask that this be upgraded to a
627	better service if one is available by supplying RXRPC_UPGRADE_SERVICE on the
628	first sendmsg() of a call.  The client should then check srx_service in the
629	msg_name filled in by recvmsg() when collecting the result.  srx_service will
630	hold the same value as given to sendmsg() if the upgrade request was ignored by
631	the service - otherwise it will be altered to indicate the service ID the
632	server upgraded to.  Note that the upgraded service ID is chosen by the server.
633	The caller has to wait until it sees the service ID in the reply before sending
634	any more calls (further calls to the same destination will be blocked until the
635	probe is concluded).
636	
637	
638	====================
639	EXAMPLE SERVER USAGE
640	====================
641	
642	A server would be set up to accept operations in the following manner:
643	
644	 (1) An RxRPC socket is created by:
645	
646		server = socket(AF_RXRPC, SOCK_DGRAM, PF_INET);
647	
648	     Where the third parameter indicates the address type of the transport
649	     socket used - usually IPv4.
650	
651	 (2) Security is set up if desired by giving the socket a keyring with server
652	     secret keys in it:
653	
654		keyring = add_key("keyring", "AFSkeys", NULL, 0,
655				  KEY_SPEC_PROCESS_KEYRING);
656	
657		const char secret_key[8] = {
658			0xa7, 0x83, 0x8a, 0xcb, 0xc7, 0x83, 0xec, 0x94 };
659		add_key("rxrpc_s", "52:2", secret_key, 8, keyring);
660	
661		setsockopt(server, SOL_RXRPC, RXRPC_SECURITY_KEYRING, "AFSkeys", 7);
662	
663	     The keyring can be manipulated after it has been given to the socket. This
664	     permits the server to add more keys, replace keys, etc. whilst it is live.
665	
666	 (3) A local address must then be bound:
667	
668		struct sockaddr_rxrpc srx = {
669			.srx_family	= AF_RXRPC,
670			.srx_service	= VL_SERVICE_ID, /* RxRPC service ID */
671			.transport_type	= SOCK_DGRAM,	/* type of transport socket */
672			.transport.sin_family	= AF_INET,
673			.transport.sin_port	= htons(7000), /* AFS callback */
674			.transport.sin_address	= 0,  /* all local interfaces */
675		};
676		bind(server, &srx, sizeof(srx));
677	
678	     More than one service ID may be bound to a socket, provided the transport
679	     parameters are the same.  The limit is currently two.  To do this, bind()
680	     should be called twice.
681	
682	 (4) If service upgrading is required, first two service IDs must have been
683	     bound and then the following option must be set:
684	
685		unsigned short service_ids[2] = { from_ID, to_ID };
686		setsockopt(server, SOL_RXRPC, RXRPC_UPGRADEABLE_SERVICE,
687			   service_ids, sizeof(service_ids));
688	
689	     This will automatically upgrade connections on service from_ID to service
690	     to_ID if they request it.  This will be reflected in msg_name obtained
691	     through recvmsg() when the request data is delivered to userspace.
692	
693	 (5) The server is then set to listen out for incoming calls:
694	
695		listen(server, 100);
696	
697	 (6) The kernel notifies the server of pending incoming connections by sending
698	     it a message for each.  This is received with recvmsg() on the server
699	     socket.  It has no data, and has a single dataless control message
700	     attached:
701	
702		RXRPC_NEW_CALL
703	
704	     The address that can be passed back by recvmsg() at this point should be
705	     ignored since the call for which the message was posted may have gone by
706	     the time it is accepted - in which case the first call still on the queue
707	     will be accepted.
708	
709	 (7) The server then accepts the new call by issuing a sendmsg() with two
710	     pieces of control data and no actual data:
711	
712		RXRPC_ACCEPT		- indicate connection acceptance
713		RXRPC_USER_CALL_ID	- specify user ID for this call
714	
715	 (8) The first request data packet will then be posted to the server socket for
716	     recvmsg() to pick up.  At that point, the RxRPC address for the call can
717	     be read from the address fields in the msghdr struct.
718	
719	     Subsequent request data will be posted to the server socket for recvmsg()
720	     to collect as it arrives.  All but the last piece of the request data will
721	     be delivered with MSG_MORE flagged.
722	
723	     All data will be delivered with the following control message attached:
724	
725		RXRPC_USER_CALL_ID	- specifies the user ID for this call
726	
727	 (9) The reply data should then be posted to the server socket using a series
728	     of sendmsg() calls, each with the following control messages attached:
729	
730		RXRPC_USER_CALL_ID	- specifies the user ID for this call
731	
732	     MSG_MORE should be set in msghdr::msg_flags on all but the last message
733	     for a particular call.
734	
735	(10) The final ACK from the client will be posted for retrieval by recvmsg()
736	     when it is received.  It will take the form of a dataless message with two
737	     control messages attached:
738	
739		RXRPC_USER_CALL_ID	- specifies the user ID for this call
740		RXRPC_ACK		- indicates final ACK (no data)
741	
742	     MSG_EOR will be flagged to indicate that this is the final message for
743	     this call.
744	
745	(11) Up to the point the final packet of reply data is sent, the call can be
746	     aborted by calling sendmsg() with a dataless message with the following
747	     control messages attached:
748	
749		RXRPC_USER_CALL_ID	- specifies the user ID for this call
750		RXRPC_ABORT		- indicates abort code (4 byte data)
751	
752	     Any packets waiting in the socket's receive queue will be discarded if
753	     this is issued.
754	
755	Note that all the communications for a particular service take place through
756	the one server socket, using control messages on sendmsg() and recvmsg() to
757	determine the call affected.
758	
759	
760	=========================
761	AF_RXRPC KERNEL INTERFACE
762	=========================
763	
764	The AF_RXRPC module also provides an interface for use by in-kernel utilities
765	such as the AFS filesystem.  This permits such a utility to:
766	
767	 (1) Use different keys directly on individual client calls on one socket
768	     rather than having to open a whole slew of sockets, one for each key it
769	     might want to use.
770	
771	 (2) Avoid having RxRPC call request_key() at the point of issue of a call or
772	     opening of a socket.  Instead the utility is responsible for requesting a
773	     key at the appropriate point.  AFS, for instance, would do this during VFS
774	     operations such as open() or unlink().  The key is then handed through
775	     when the call is initiated.
776	
777	 (3) Request the use of something other than GFP_KERNEL to allocate memory.
778	
779	 (4) Avoid the overhead of using the recvmsg() call.  RxRPC messages can be
780	     intercepted before they get put into the socket Rx queue and the socket
781	     buffers manipulated directly.
782	
783	To use the RxRPC facility, a kernel utility must still open an AF_RXRPC socket,
784	bind an address as appropriate and listen if it's to be a server socket, but
785	then it passes this to the kernel interface functions.
786	
787	The kernel interface functions are as follows:
788	
789	 (*) Begin a new client call.
790	
791		struct rxrpc_call *
792		rxrpc_kernel_begin_call(struct socket *sock,
793					struct sockaddr_rxrpc *srx,
794					struct key *key,
795					unsigned long user_call_ID,
796					s64 tx_total_len,
797					gfp_t gfp,
798					rxrpc_notify_rx_t notify_rx,
799					bool upgrade);
800	
801	     This allocates the infrastructure to make a new RxRPC call and assigns
802	     call and connection numbers.  The call will be made on the UDP port that
803	     the socket is bound to.  The call will go to the destination address of a
804	     connected client socket unless an alternative is supplied (srx is
805	     non-NULL).
806	
807	     If a key is supplied then this will be used to secure the call instead of
808	     the key bound to the socket with the RXRPC_SECURITY_KEY sockopt.  Calls
809	     secured in this way will still share connections if at all possible.
810	
811	     The user_call_ID is equivalent to that supplied to sendmsg() in the
812	     control data buffer.  It is entirely feasible to use this to point to a
813	     kernel data structure.
814	
815	     tx_total_len is the amount of data the caller is intending to transmit
816	     with this call (or -1 if unknown at this point).  Setting the data size
817	     allows the kernel to encrypt directly to the packet buffers, thereby
818	     saving a copy.  The value may not be less than -1.
819	
820	     notify_rx is a pointer to a function to be called when events such as
821	     incoming data packets or remote aborts happen.
822	
823	     upgrade should be set to true if a client operation should request that
824	     the server upgrade the service to a better one.  The resultant service ID
825	     is returned by rxrpc_kernel_recv_data().
826	
827	     If this function is successful, an opaque reference to the RxRPC call is
828	     returned.  The caller now holds a reference on this and it must be
829	     properly ended.
830	
831	 (*) End a client call.
832	
833		void rxrpc_kernel_end_call(struct socket *sock,
834					   struct rxrpc_call *call);
835	
836	     This is used to end a previously begun call.  The user_call_ID is expunged
837	     from AF_RXRPC's knowledge and will not be seen again in association with
838	     the specified call.
839	
840	 (*) Send data through a call.
841	
842		typedef void (*rxrpc_notify_end_tx_t)(struct sock *sk,
843						      unsigned long user_call_ID,
844						      struct sk_buff *skb);
845	
846		int rxrpc_kernel_send_data(struct socket *sock,
847					   struct rxrpc_call *call,
848					   struct msghdr *msg,
849					   size_t len,
850					   rxrpc_notify_end_tx_t notify_end_rx);
851	
852	     This is used to supply either the request part of a client call or the
853	     reply part of a server call.  msg.msg_iovlen and msg.msg_iov specify the
854	     data buffers to be used.  msg_iov may not be NULL and must point
855	     exclusively to in-kernel virtual addresses.  msg.msg_flags may be given
856	     MSG_MORE if there will be subsequent data sends for this call.
857	
858	     The msg must not specify a destination address, control data or any flags
859	     other than MSG_MORE.  len is the total amount of data to transmit.
860	
861	     notify_end_rx can be NULL or it can be used to specify a function to be
862	     called when the call changes state to end the Tx phase.  This function is
863	     called with the call-state spinlock held to prevent any reply or final ACK
864	     from being delivered first.
865	
866	 (*) Receive data from a call.
867	
868		int rxrpc_kernel_recv_data(struct socket *sock,
869					   struct rxrpc_call *call,
870					   void *buf,
871					   size_t size,
872					   size_t *_offset,
873					   bool want_more,
874					   u32 *_abort,
875					   u16 *_service)
876	
877	      This is used to receive data from either the reply part of a client call
878	      or the request part of a service call.  buf and size specify how much
879	      data is desired and where to store it.  *_offset is added on to buf and
880	      subtracted from size internally; the amount copied into the buffer is
881	      added to *_offset before returning.
882	
883	      want_more should be true if further data will be required after this is
884	      satisfied and false if this is the last item of the receive phase.
885	
886	      There are three normal returns: 0 if the buffer was filled and want_more
887	      was true; 1 if the buffer was filled, the last DATA packet has been
888	      emptied and want_more was false; and -EAGAIN if the function needs to be
889	      called again.
890	
891	      If the last DATA packet is processed but the buffer contains less than
892	      the amount requested, EBADMSG is returned.  If want_more wasn't set, but
893	      more data was available, EMSGSIZE is returned.
894	
895	      If a remote ABORT is detected, the abort code received will be stored in
896	      *_abort and ECONNABORTED will be returned.
897	
898	      The service ID that the call ended up with is returned into *_service.
899	      This can be used to see if a call got a service upgrade.
900	
901	 (*) Abort a call.
902	
903		void rxrpc_kernel_abort_call(struct socket *sock,
904					     struct rxrpc_call *call,
905					     u32 abort_code);
906	
907	     This is used to abort a call if it's still in an abortable state.  The
908	     abort code specified will be placed in the ABORT message sent.
909	
910	 (*) Intercept received RxRPC messages.
911	
912		typedef void (*rxrpc_interceptor_t)(struct sock *sk,
913						    unsigned long user_call_ID,
914						    struct sk_buff *skb);
915	
916		void
917		rxrpc_kernel_intercept_rx_messages(struct socket *sock,
918						   rxrpc_interceptor_t interceptor);
919	
920	     This installs an interceptor function on the specified AF_RXRPC socket.
921	     All messages that would otherwise wind up in the socket's Rx queue are
922	     then diverted to this function.  Note that care must be taken to process
923	     the messages in the right order to maintain DATA message sequentiality.
924	
925	     The interceptor function itself is provided with the address of the socket
926	     and handling the incoming message, the ID assigned by the kernel utility
927	     to the call and the socket buffer containing the message.
928	
929	     The skb->mark field indicates the type of message:
930	
931		MARK				MEANING
932		===============================	=======================================
933		RXRPC_SKB_MARK_DATA		Data message
934		RXRPC_SKB_MARK_FINAL_ACK	Final ACK received for an incoming call
935		RXRPC_SKB_MARK_BUSY		Client call rejected as server busy
936		RXRPC_SKB_MARK_REMOTE_ABORT	Call aborted by peer
937		RXRPC_SKB_MARK_NET_ERROR	Network error detected
938		RXRPC_SKB_MARK_LOCAL_ERROR	Local error encountered
939		RXRPC_SKB_MARK_NEW_CALL		New incoming call awaiting acceptance
940	
941	     The remote abort message can be probed with rxrpc_kernel_get_abort_code().
942	     The two error messages can be probed with rxrpc_kernel_get_error_number().
943	     A new call can be accepted with rxrpc_kernel_accept_call().
944	
945	     Data messages can have their contents extracted with the usual bunch of
946	     socket buffer manipulation functions.  A data message can be determined to
947	     be the last one in a sequence with rxrpc_kernel_is_data_last().  When a
948	     data message has been used up, rxrpc_kernel_data_consumed() should be
949	     called on it.
950	
951	     Messages should be handled to rxrpc_kernel_free_skb() to dispose of.  It
952	     is possible to get extra refs on all types of message for later freeing,
953	     but this may pin the state of a call until the message is finally freed.
954	
955	 (*) Accept an incoming call.
956	
957		struct rxrpc_call *
958		rxrpc_kernel_accept_call(struct socket *sock,
959					 unsigned long user_call_ID);
960	
961	     This is used to accept an incoming call and to assign it a call ID.  This
962	     function is similar to rxrpc_kernel_begin_call() and calls accepted must
963	     be ended in the same way.
964	
965	     If this function is successful, an opaque reference to the RxRPC call is
966	     returned.  The caller now holds a reference on this and it must be
967	     properly ended.
968	
969	 (*) Reject an incoming call.
970	
971		int rxrpc_kernel_reject_call(struct socket *sock);
972	
973	     This is used to reject the first incoming call on the socket's queue with
974	     a BUSY message.  -ENODATA is returned if there were no incoming calls.
975	     Other errors may be returned if the call had been aborted (-ECONNABORTED)
976	     or had timed out (-ETIME).
977	
978	 (*) Allocate a null key for doing anonymous security.
979	
980		struct key *rxrpc_get_null_key(const char *keyname);
981	
982	     This is used to allocate a null RxRPC key that can be used to indicate
983	     anonymous security for a particular domain.
984	
985	 (*) Get the peer address of a call.
986	
987		void rxrpc_kernel_get_peer(struct socket *sock, struct rxrpc_call *call,
988					   struct sockaddr_rxrpc *_srx);
989	
990	     This is used to find the remote peer address of a call.
991	
992	 (*) Set the total transmit data size on a call.
993	
994		void rxrpc_kernel_set_tx_length(struct socket *sock,
995						struct rxrpc_call *call,
996						s64 tx_total_len);
997	
998	     This sets the amount of data that the caller is intending to transmit on a
999	     call.  It's intended to be used for setting the reply size as the request
1000	     size should be set when the call is begun.  tx_total_len may not be less
1001	     than zero.
1002	
1003	 (*) Check to see the completion state of a call so that the caller can assess
1004	     whether it needs to be retried.
1005	
1006		enum rxrpc_call_completion {
1007			RXRPC_CALL_SUCCEEDED,
1008			RXRPC_CALL_REMOTELY_ABORTED,
1009			RXRPC_CALL_LOCALLY_ABORTED,
1010			RXRPC_CALL_LOCAL_ERROR,
1011			RXRPC_CALL_NETWORK_ERROR,
1012		};
1013	
1014		int rxrpc_kernel_check_call(struct socket *sock, struct rxrpc_call *call,
1015					    enum rxrpc_call_completion *_compl,
1016					    u32 *_abort_code);
1017	
1018	     On return, -EINPROGRESS will be returned if the call is still ongoing; if
1019	     it is finished, *_compl will be set to indicate the manner of completion,
1020	     *_abort_code will be set to any abort code that occurred.  0 will be
1021	     returned on a successful completion, -ECONNABORTED will be returned if the
1022	     client failed due to a remote abort and anything else will return an
1023	     appropriate error code.
1024	
1025	     The caller should look at this information to decide if it's worth
1026	     retrying the call.
1027	
1028	 (*) Retry a client call.
1029	
1030		int rxrpc_kernel_retry_call(struct socket *sock,
1031					    struct rxrpc_call *call,
1032					    struct sockaddr_rxrpc *srx,
1033					    struct key *key);
1034	
1035	     This attempts to partially reinitialise a call and submit it again whilst
1036	     reusing the original call's Tx queue to avoid the need to repackage and
1037	     re-encrypt the data to be sent.  call indicates the call to retry, srx the
1038	     new address to send it to and key the encryption key to use for signing or
1039	     encrypting the packets.
1040	
1041	     For this to work, the first Tx data packet must still be in the transmit
1042	     queue, and currently this is only permitted for local and network errors
1043	     and the call must not have been aborted.  Any partially constructed Tx
1044	     packet is left as is and can continue being filled afterwards.
1045	
1046	     It returns 0 if the call was requeued and an error otherwise.
1047	
1048	 (*) Get call RTT.
1049	
1050		u64 rxrpc_kernel_get_rtt(struct socket *sock, struct rxrpc_call *call);
1051	
1052	     Get the RTT time to the peer in use by a call.  The value returned is in
1053	     nanoseconds.
1054	
1055	 (*) Check call still alive.
1056	
1057		u32 rxrpc_kernel_check_life(struct socket *sock,
1058					    struct rxrpc_call *call);
1059	
1060	     This returns a number that is updated when ACKs are received from the peer
1061	     (notably including PING RESPONSE ACKs which we can elicit by sending PING
1062	     ACKs to see if the call still exists on the server).  The caller should
1063	     compare the numbers of two calls to see if the call is still alive after
1064	     waiting for a suitable interval.
1065	
1066	     This allows the caller to work out if the server is still contactable and
1067	     if the call is still alive on the server whilst waiting for the server to
1068	     process a client operation.
1069	
1070	     This function may transmit a PING ACK.
1071	
1072	
1073	=======================
1074	CONFIGURABLE PARAMETERS
1075	=======================
1076	
1077	The RxRPC protocol driver has a number of configurable parameters that can be
1078	adjusted through sysctls in /proc/net/rxrpc/:
1079	
1080	 (*) req_ack_delay
1081	
1082	     The amount of time in milliseconds after receiving a packet with the
1083	     request-ack flag set before we honour the flag and actually send the
1084	     requested ack.
1085	
1086	     Usually the other side won't stop sending packets until the advertised
1087	     reception window is full (to a maximum of 255 packets), so delaying the
1088	     ACK permits several packets to be ACK'd in one go.
1089	
1090	 (*) soft_ack_delay
1091	
1092	     The amount of time in milliseconds after receiving a new packet before we
1093	     generate a soft-ACK to tell the sender that it doesn't need to resend.
1094	
1095	 (*) idle_ack_delay
1096	
1097	     The amount of time in milliseconds after all the packets currently in the
1098	     received queue have been consumed before we generate a hard-ACK to tell
1099	     the sender it can free its buffers, assuming no other reason occurs that
1100	     we would send an ACK.
1101	
1102	 (*) resend_timeout
1103	
1104	     The amount of time in milliseconds after transmitting a packet before we
1105	     transmit it again, assuming no ACK is received from the receiver telling
1106	     us they got it.
1107	
1108	 (*) max_call_lifetime
1109	
1110	     The maximum amount of time in seconds that a call may be in progress
1111	     before we preemptively kill it.
1112	
1113	 (*) dead_call_expiry
1114	
1115	     The amount of time in seconds before we remove a dead call from the call
1116	     list.  Dead calls are kept around for a little while for the purpose of
1117	     repeating ACK and ABORT packets.
1118	
1119	 (*) connection_expiry
1120	
1121	     The amount of time in seconds after a connection was last used before we
1122	     remove it from the connection list.  Whilst a connection is in existence,
1123	     it serves as a placeholder for negotiated security; when it is deleted,
1124	     the security must be renegotiated.
1125	
1126	 (*) transport_expiry
1127	
1128	     The amount of time in seconds after a transport was last used before we
1129	     remove it from the transport list.  Whilst a transport is in existence, it
1130	     serves to anchor the peer data and keeps the connection ID counter.
1131	
1132	 (*) rxrpc_rx_window_size
1133	
1134	     The size of the receive window in packets.  This is the maximum number of
1135	     unconsumed received packets we're willing to hold in memory for any
1136	     particular call.
1137	
1138	 (*) rxrpc_rx_mtu
1139	
1140	     The maximum packet MTU size that we're willing to receive in bytes.  This
1141	     indicates to the peer whether we're willing to accept jumbo packets.
1142	
1143	 (*) rxrpc_rx_jumbo_max
1144	
1145	     The maximum number of packets that we're willing to accept in a jumbo
1146	     packet.  Non-terminal packets in a jumbo packet must contain a four byte
1147	     header plus exactly 1412 bytes of data.  The terminal packet must contain
1148	     a four byte header plus any amount of data.  In any event, a jumbo packet
1149	     may not exceed rxrpc_rx_mtu in size.
Hide Line Numbers
About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Information is copyright its respective author. All material is available from the Linux Kernel Source distributed under a GPL License. This page is provided as a free service by mjmwired.net.