About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / adding-syscalls.txt


Based on kernel version 4.9. Page generated on 2016-12-21 14:28 EST.

1	Adding a New System Call
2	========================
3	
4	This document describes what's involved in adding a new system call to the
5	Linux kernel, over and above the normal submission advice in
6	Documentation/SubmittingPatches.
7	
8	
9	System Call Alternatives
10	------------------------
11	
12	The first thing to consider when adding a new system call is whether one of
13	the alternatives might be suitable instead.  Although system calls are the
14	most traditional and most obvious interaction points between userspace and the
15	kernel, there are other possibilities -- choose what fits best for your
16	interface.
17	
18	 - If the operations involved can be made to look like a filesystem-like
19	   object, it may make more sense to create a new filesystem or device.  This
20	   also makes it easier to encapsulate the new functionality in a kernel module
21	   rather than requiring it to be built into the main kernel.
22	     - If the new functionality involves operations where the kernel notifies
23	       userspace that something has happened, then returning a new file
24	       descriptor for the relevant object allows userspace to use
25	       poll/select/epoll to receive that notification.
26	     - However, operations that don't map to read(2)/write(2)-like operations
27	       have to be implemented as ioctl(2) requests, which can lead to a
28	       somewhat opaque API.
29	 - If you're just exposing runtime system information, a new node in sysfs
30	   (see Documentation/filesystems/sysfs.txt) or the /proc filesystem may be
31	   more appropriate.  However, access to these mechanisms requires that the
32	   relevant filesystem is mounted, which might not always be the case (e.g.
33	   in a namespaced/sandboxed/chrooted environment).  Avoid adding any API to
34	   debugfs, as this is not considered a 'production' interface to userspace.
35	 - If the operation is specific to a particular file or file descriptor, then
36	   an additional fcntl(2) command option may be more appropriate.  However,
37	   fcntl(2) is a multiplexing system call that hides a lot of complexity, so
38	   this option is best for when the new function is closely analogous to
39	   existing fcntl(2) functionality, or the new functionality is very simple
40	   (for example, getting/setting a simple flag related to a file descriptor).
41	 - If the operation is specific to a particular task or process, then an
42	   additional prctl(2) command option may be more appropriate.  As with
43	   fcntl(2), this system call is a complicated multiplexor so is best reserved
44	   for near-analogs of existing prctl() commands or getting/setting a simple
45	   flag related to a process.
46	
47	
48	Designing the API: Planning for Extension
49	-----------------------------------------
50	
51	A new system call forms part of the API of the kernel, and has to be supported
52	indefinitely.  As such, it's a very good idea to explicitly discuss the
53	interface on the kernel mailing list, and it's important to plan for future
54	extensions of the interface.
55	
56	(The syscall table is littered with historical examples where this wasn't done,
57	together with the corresponding follow-up system calls -- eventfd/eventfd2,
58	dup2/dup3, inotify_init/inotify_init1,  pipe/pipe2, renameat/renameat2 -- so
59	learn from the history of the kernel and plan for extensions from the start.)
60	
61	For simpler system calls that only take a couple of arguments, the preferred
62	way to allow for future extensibility is to include a flags argument to the
63	system call.  To make sure that userspace programs can safely use flags
64	between kernel versions, check whether the flags value holds any unknown
65	flags, and reject the system call (with EINVAL) if it does:
66	
67	    if (flags & ~(THING_FLAG1 | THING_FLAG2 | THING_FLAG3))
68	        return -EINVAL;
69	
70	(If no flags values are used yet, check that the flags argument is zero.)
71	
72	For more sophisticated system calls that involve a larger number of arguments,
73	it's preferred to encapsulate the majority of the arguments into a structure
74	that is passed in by pointer.  Such a structure can cope with future extension
75	by including a size argument in the structure:
76	
77	    struct xyzzy_params {
78	        u32 size; /* userspace sets p->size = sizeof(struct xyzzy_params) */
79	        u32 param_1;
80	        u64 param_2;
81	        u64 param_3;
82	    };
83	
84	As long as any subsequently added field, say param_4, is designed so that a
85	zero value gives the previous behaviour, then this allows both directions of
86	version mismatch:
87	
88	 - To cope with a later userspace program calling an older kernel, the kernel
89	   code should check that any memory beyond the size of the structure that it
90	   expects is zero (effectively checking that param_4 == 0).
91	 - To cope with an older userspace program calling a newer kernel, the kernel
92	   code can zero-extend a smaller instance of the structure (effectively
93	   setting param_4 = 0).
94	
95	See perf_event_open(2) and the perf_copy_attr() function (in
96	kernel/events/core.c) for an example of this approach.
97	
98	
99	Designing the API: Other Considerations
100	---------------------------------------
101	
102	If your new system call allows userspace to refer to a kernel object, it
103	should use a file descriptor as the handle for that object -- don't invent a
104	new type of userspace object handle when the kernel already has mechanisms and
105	well-defined semantics for using file descriptors.
106	
107	If your new xyzzy(2) system call does return a new file descriptor, then the
108	flags argument should include a value that is equivalent to setting O_CLOEXEC
109	on the new FD.  This makes it possible for userspace to close the timing
110	window between xyzzy() and calling fcntl(fd, F_SETFD, FD_CLOEXEC), where an
111	unexpected fork() and execve() in another thread could leak a descriptor to
112	the exec'ed program. (However, resist the temptation to re-use the actual value
113	of the O_CLOEXEC constant, as it is architecture-specific and is part of a
114	numbering space of O_* flags that is fairly full.)
115	
116	If your system call returns a new file descriptor, you should also consider
117	what it means to use the poll(2) family of system calls on that file
118	descriptor. Making a file descriptor ready for reading or writing is the
119	normal way for the kernel to indicate to userspace that an event has
120	occurred on the corresponding kernel object.
121	
122	If your new xyzzy(2) system call involves a filename argument:
123	
124	    int sys_xyzzy(const char __user *path, ..., unsigned int flags);
125	
126	you should also consider whether an xyzzyat(2) version is more appropriate:
127	
128	    int sys_xyzzyat(int dfd, const char __user *path, ..., unsigned int flags);
129	
130	This allows more flexibility for how userspace specifies the file in question;
131	in particular it allows userspace to request the functionality for an
132	already-opened file descriptor using the AT_EMPTY_PATH flag, effectively giving
133	an fxyzzy(3) operation for free:
134	
135	 - xyzzyat(AT_FDCWD, path, ..., 0) is equivalent to xyzzy(path,...)
136	 - xyzzyat(fd, "", ..., AT_EMPTY_PATH) is equivalent to fxyzzy(fd, ...)
137	
138	(For more details on the rationale of the *at() calls, see the openat(2) man
139	page; for an example of AT_EMPTY_PATH, see the fstatat(2) man page.)
140	
141	If your new xyzzy(2) system call involves a parameter describing an offset
142	within a file, make its type loff_t so that 64-bit offsets can be supported
143	even on 32-bit architectures.
144	
145	If your new xyzzy(2) system call involves privileged functionality, it needs
146	to be governed by the appropriate Linux capability bit (checked with a call to
147	capable()), as described in the capabilities(7) man page.  Choose an existing
148	capability bit that governs related functionality, but try to avoid combining
149	lots of only vaguely related functions together under the same bit, as this
150	goes against capabilities' purpose of splitting the power of root.  In
151	particular, avoid adding new uses of the already overly-general CAP_SYS_ADMIN
152	capability.
153	
154	If your new xyzzy(2) system call manipulates a process other than the calling
155	process, it should be restricted (using a call to ptrace_may_access()) so that
156	only a calling process with the same permissions as the target process, or
157	with the necessary capabilities, can manipulate the target process.
158	
159	Finally, be aware that some non-x86 architectures have an easier time if
160	system call parameters that are explicitly 64-bit fall on odd-numbered
161	arguments (i.e. parameter 1, 3, 5), to allow use of contiguous pairs of 32-bit
162	registers.  (This concern does not apply if the arguments are part of a
163	structure that's passed in by pointer.)
164	
165	
166	Proposing the API
167	-----------------
168	
169	To make new system calls easy to review, it's best to divide up the patchset
170	into separate chunks.  These should include at least the following items as
171	distinct commits (each of which is described further below):
172	
173	 - The core implementation of the system call, together with prototypes,
174	   generic numbering, Kconfig changes and fallback stub implementation.
175	 - Wiring up of the new system call for one particular architecture, usually
176	   x86 (including all of x86_64, x86_32 and x32).
177	 - A demonstration of the use of the new system call in userspace via a
178	   selftest in tools/testing/selftests/.
179	 - A draft man-page for the new system call, either as plain text in the
180	   cover letter, or as a patch to the (separate) man-pages repository.
181	
182	New system call proposals, like any change to the kernel's API, should always
183	be cc'ed to linux-api@vger.kernel.org.
184	
185	
186	Generic System Call Implementation
187	----------------------------------
188	
189	The main entry point for your new xyzzy(2) system call will be called
190	sys_xyzzy(), but you add this entry point with the appropriate
191	SYSCALL_DEFINEn() macro rather than explicitly.  The 'n' indicates the number
192	of arguments to the system call, and the macro takes the system call name
193	followed by the (type, name) pairs for the parameters as arguments.  Using
194	this macro allows metadata about the new system call to be made available for
195	other tools.
196	
197	The new entry point also needs a corresponding function prototype, in
198	include/linux/syscalls.h, marked as asmlinkage to match the way that system
199	calls are invoked:
200	
201	    asmlinkage long sys_xyzzy(...);
202	
203	Some architectures (e.g. x86) have their own architecture-specific syscall
204	tables, but several other architectures share a generic syscall table. Add your
205	new system call to the generic list by adding an entry to the list in
206	include/uapi/asm-generic/unistd.h:
207	
208	    #define __NR_xyzzy 292
209	    __SYSCALL(__NR_xyzzy, sys_xyzzy)
210	
211	Also update the __NR_syscalls count to reflect the additional system call, and
212	note that if multiple new system calls are added in the same merge window,
213	your new syscall number may get adjusted to resolve conflicts.
214	
215	The file kernel/sys_ni.c provides a fallback stub implementation of each system
216	call, returning -ENOSYS.  Add your new system call here too:
217	
218	    cond_syscall(sys_xyzzy);
219	
220	Your new kernel functionality, and the system call that controls it, should
221	normally be optional, so add a CONFIG option (typically to init/Kconfig) for
222	it. As usual for new CONFIG options:
223	
224	 - Include a description of the new functionality and system call controlled
225	   by the option.
226	 - Make the option depend on EXPERT if it should be hidden from normal users.
227	 - Make any new source files implementing the function dependent on the CONFIG
228	   option in the Makefile (e.g. "obj-$(CONFIG_XYZZY_SYSCALL) += xyzzy.c").
229	 - Double check that the kernel still builds with the new CONFIG option turned
230	   off.
231	
232	To summarize, you need a commit that includes:
233	
234	 - CONFIG option for the new function, normally in init/Kconfig
235	 - SYSCALL_DEFINEn(xyzzy, ...) for the entry point
236	 - corresponding prototype in include/linux/syscalls.h
237	 - generic table entry in include/uapi/asm-generic/unistd.h
238	 - fallback stub in kernel/sys_ni.c
239	
240	
241	x86 System Call Implementation
242	------------------------------
243	
244	To wire up your new system call for x86 platforms, you need to update the
245	master syscall tables.  Assuming your new system call isn't special in some
246	way (see below), this involves a "common" entry (for x86_64 and x32) in
247	arch/x86/entry/syscalls/syscall_64.tbl:
248	
249	    333   common   xyzzy     sys_xyzzy
250	
251	and an "i386" entry in arch/x86/entry/syscalls/syscall_32.tbl:
252	
253	    380   i386     xyzzy     sys_xyzzy
254	
255	Again, these numbers are liable to be changed if there are conflicts in the
256	relevant merge window.
257	
258	
259	Compatibility System Calls (Generic)
260	------------------------------------
261	
262	For most system calls the same 64-bit implementation can be invoked even when
263	the userspace program is itself 32-bit; even if the system call's parameters
264	include an explicit pointer, this is handled transparently.
265	
266	However, there are a couple of situations where a compatibility layer is
267	needed to cope with size differences between 32-bit and 64-bit.
268	
269	The first is if the 64-bit kernel also supports 32-bit userspace programs, and
270	so needs to parse areas of (__user) memory that could hold either 32-bit or
271	64-bit values.  In particular, this is needed whenever a system call argument
272	is:
273	
274	 - a pointer to a pointer
275	 - a pointer to a struct containing a pointer (e.g. struct iovec __user *)
276	 - a pointer to a varying sized integral type (time_t, off_t, long, ...)
277	 - a pointer to a struct containing a varying sized integral type.
278	
279	The second situation that requires a compatibility layer is if one of the
280	system call's arguments has a type that is explicitly 64-bit even on a 32-bit
281	architecture, for example loff_t or __u64.  In this case, a value that arrives
282	at a 64-bit kernel from a 32-bit application will be split into two 32-bit
283	values, which then need to be re-assembled in the compatibility layer.
284	
285	(Note that a system call argument that's a pointer to an explicit 64-bit type
286	does *not* need a compatibility layer; for example, splice(2)'s arguments of
287	type loff_t __user * do not trigger the need for a compat_ system call.)
288	
289	The compatibility version of the system call is called compat_sys_xyzzy(), and
290	is added with the COMPAT_SYSCALL_DEFINEn() macro, analogously to
291	SYSCALL_DEFINEn.  This version of the implementation runs as part of a 64-bit
292	kernel, but expects to receive 32-bit parameter values and does whatever is
293	needed to deal with them.  (Typically, the compat_sys_ version converts the
294	values to 64-bit versions and either calls on to the sys_ version, or both of
295	them call a common inner implementation function.)
296	
297	The compat entry point also needs a corresponding function prototype, in
298	include/linux/compat.h, marked as asmlinkage to match the way that system
299	calls are invoked:
300	
301	    asmlinkage long compat_sys_xyzzy(...);
302	
303	If the system call involves a structure that is laid out differently on 32-bit
304	and 64-bit systems, say struct xyzzy_args, then the include/linux/compat.h
305	header file should also include a compat version of the structure (struct
306	compat_xyzzy_args) where each variable-size field has the appropriate compat_
307	type that corresponds to the type in struct xyzzy_args.  The
308	compat_sys_xyzzy() routine can then use this compat_ structure to parse the
309	arguments from a 32-bit invocation.
310	
311	For example, if there are fields:
312	
313	    struct xyzzy_args {
314	        const char __user *ptr;
315	        __kernel_long_t varying_val;
316	        u64 fixed_val;
317	        /* ... */
318	    };
319	
320	in struct xyzzy_args, then struct compat_xyzzy_args would have:
321	
322	    struct compat_xyzzy_args {
323	        compat_uptr_t ptr;
324	        compat_long_t varying_val;
325	        u64 fixed_val;
326	        /* ... */
327	    };
328	
329	The generic system call list also needs adjusting to allow for the compat
330	version; the entry in include/uapi/asm-generic/unistd.h should use
331	__SC_COMP rather than __SYSCALL:
332	
333	    #define __NR_xyzzy 292
334	    __SC_COMP(__NR_xyzzy, sys_xyzzy, compat_sys_xyzzy)
335	
336	To summarize, you need:
337	
338	 - a COMPAT_SYSCALL_DEFINEn(xyzzy, ...) for the compat entry point
339	 - corresponding prototype in include/linux/compat.h
340	 - (if needed) 32-bit mapping struct in include/linux/compat.h
341	 - instance of __SC_COMP not __SYSCALL in include/uapi/asm-generic/unistd.h
342	
343	
344	Compatibility System Calls (x86)
345	--------------------------------
346	
347	To wire up the x86 architecture of a system call with a compatibility version,
348	the entries in the syscall tables need to be adjusted.
349	
350	First, the entry in arch/x86/entry/syscalls/syscall_32.tbl gets an extra
351	column to indicate that a 32-bit userspace program running on a 64-bit kernel
352	should hit the compat entry point:
353	
354	    380   i386     xyzzy     sys_xyzzy    compat_sys_xyzzy
355	
356	Second, you need to figure out what should happen for the x32 ABI version of
357	the new system call.  There's a choice here: the layout of the arguments
358	should either match the 64-bit version or the 32-bit version.
359	
360	If there's a pointer-to-a-pointer involved, the decision is easy: x32 is
361	ILP32, so the layout should match the 32-bit version, and the entry in
362	arch/x86/entry/syscalls/syscall_64.tbl is split so that x32 programs hit the
363	compatibility wrapper:
364	
365	    333   64       xyzzy     sys_xyzzy
366	    ...
367	    555   x32      xyzzy     compat_sys_xyzzy
368	
369	If no pointers are involved, then it is preferable to re-use the 64-bit system
370	call for the x32 ABI (and consequently the entry in
371	arch/x86/entry/syscalls/syscall_64.tbl is unchanged).
372	
373	In either case, you should check that the types involved in your argument
374	layout do indeed map exactly from x32 (-mx32) to either the 32-bit (-m32) or
375	64-bit (-m64) equivalents.
376	
377	
378	System Calls Returning Elsewhere
379	--------------------------------
380	
381	For most system calls, once the system call is complete the user program
382	continues exactly where it left off -- at the next instruction, with the
383	stack the same and most of the registers the same as before the system call,
384	and with the same virtual memory space.
385	
386	However, a few system calls do things differently.  They might return to a
387	different location (rt_sigreturn) or change the memory space (fork/vfork/clone)
388	or even architecture (execve/execveat) of the program.
389	
390	To allow for this, the kernel implementation of the system call may need to
391	save and restore additional registers to the kernel stack, allowing complete
392	control of where and how execution continues after the system call.
393	
394	This is arch-specific, but typically involves defining assembly entry points
395	that save/restore additional registers and invoke the real system call entry
396	point.
397	
398	For x86_64, this is implemented as a stub_xyzzy entry point in
399	arch/x86/entry/entry_64.S, and the entry in the syscall table
400	(arch/x86/entry/syscalls/syscall_64.tbl) is adjusted to match:
401	
402	    333   common   xyzzy     stub_xyzzy
403	
404	The equivalent for 32-bit programs running on a 64-bit kernel is normally
405	called stub32_xyzzy and implemented in arch/x86/entry/entry_64_compat.S,
406	with the corresponding syscall table adjustment in
407	arch/x86/entry/syscalls/syscall_32.tbl:
408	
409	    380   i386     xyzzy     sys_xyzzy    stub32_xyzzy
410	
411	If the system call needs a compatibility layer (as in the previous section)
412	then the stub32_ version needs to call on to the compat_sys_ version of the
413	system call rather than the native 64-bit version.  Also, if the x32 ABI
414	implementation is not common with the x86_64 version, then its syscall
415	table will also need to invoke a stub that calls on to the compat_sys_
416	version.
417	
418	For completeness, it's also nice to set up a mapping so that user-mode Linux
419	still works -- its syscall table will reference stub_xyzzy, but the UML build
420	doesn't include arch/x86/entry/entry_64.S implementation (because UML
421	simulates registers etc).  Fixing this is as simple as adding a #define to
422	arch/x86/um/sys_call_table_64.c:
423	
424	    #define stub_xyzzy sys_xyzzy
425	
426	
427	Other Details
428	-------------
429	
430	Most of the kernel treats system calls in a generic way, but there is the
431	occasional exception that may need updating for your particular system call.
432	
433	The audit subsystem is one such special case; it includes (arch-specific)
434	functions that classify some special types of system call -- specifically
435	file open (open/openat), program execution (execve/exeveat) or socket
436	multiplexor (socketcall) operations. If your new system call is analogous to
437	one of these, then the audit system should be updated.
438	
439	More generally, if there is an existing system call that is analogous to your
440	new system call, it's worth doing a kernel-wide grep for the existing system
441	call to check there are no other special cases.
442	
443	
444	Testing
445	-------
446	
447	A new system call should obviously be tested; it is also useful to provide
448	reviewers with a demonstration of how user space programs will use the system
449	call.  A good way to combine these aims is to include a simple self-test
450	program in a new directory under tools/testing/selftests/.
451	
452	For a new system call, there will obviously be no libc wrapper function and so
453	the test will need to invoke it using syscall(); also, if the system call
454	involves a new userspace-visible structure, the corresponding header will need
455	to be installed to compile the test.
456	
457	Make sure the selftest runs successfully on all supported architectures.  For
458	example, check that it works when compiled as an x86_64 (-m64), x86_32 (-m32)
459	and x32 (-mx32) ABI program.
460	
461	For more extensive and thorough testing of new functionality, you should also
462	consider adding tests to the Linux Test Project, or to the xfstests project
463	for filesystem-related changes.
464	 - https://linux-test-project.github.io/
465	 - git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git
466	
467	
468	Man Page
469	--------
470	
471	All new system calls should come with a complete man page, ideally using groff
472	markup, but plain text will do.  If groff is used, it's helpful to include a
473	pre-rendered ASCII version of the man page in the cover email for the
474	patchset, for the convenience of reviewers.
475	
476	The man page should be cc'ed to linux-man@vger.kernel.org
477	For more details, see https://www.kernel.org/doc/man-pages/patches.html
478	
479	References and Sources
480	----------------------
481	
482	 - LWN article from Michael Kerrisk on use of flags argument in system calls:
483	   https://lwn.net/Articles/585415/
484	 - LWN article from Michael Kerrisk on how to handle unknown flags in a system
485	   call: https://lwn.net/Articles/588444/
486	 - LWN article from Jake Edge describing constraints on 64-bit system call
487	   arguments: https://lwn.net/Articles/311630/
488	 - Pair of LWN articles from David Drysdale that describe the system call
489	   implementation paths in detail for v3.14:
490	    - https://lwn.net/Articles/604287/
491	    - https://lwn.net/Articles/604515/
492	 - Architecture-specific requirements for system calls are discussed in the
493	   syscall(2) man-page:
494	   http://man7.org/linux/man-pages/man2/syscall.2.html#NOTES
495	 - Collated emails from Linus Torvalds discussing the problems with ioctl():
496	   http://yarchive.net/comp/linux/ioctl.html
497	 - "How to not invent kernel interfaces", Arnd Bergmann,
498	   http://www.ukuug.org/events/linux2007/2007/papers/Bergmann.pdf
499	 - LWN article from Michael Kerrisk on avoiding new uses of CAP_SYS_ADMIN:
500	   https://lwn.net/Articles/486306/
501	 - Recommendation from Andrew Morton that all related information for a new
502	   system call should come in the same email thread:
503	   https://lkml.org/lkml/2014/7/24/641
504	 - Recommendation from Michael Kerrisk that a new system call should come with
505	   a man page: https://lkml.org/lkml/2014/6/13/309
506	 - Suggestion from Thomas Gleixner that x86 wire-up should be in a separate
507	   commit: https://lkml.org/lkml/2014/11/19/254
508	 - Suggestion from Greg Kroah-Hartman that it's good for new system calls to
509	   come with a man-page & selftest: https://lkml.org/lkml/2014/3/19/710
510	 - Discussion from Michael Kerrisk of new system call vs. prctl(2) extension:
511	   https://lkml.org/lkml/2014/6/3/411
512	 - Suggestion from Ingo Molnar that system calls that involve multiple
513	   arguments should encapsulate those arguments in a struct, which includes a
514	   size field for future extensibility: https://lkml.org/lkml/2015/7/30/117
515	 - Numbering oddities arising from (re-)use of O_* numbering space flags:
516	    - commit 75069f2b5bfb ("vfs: renumber FMODE_NONOTIFY and add to uniqueness
517	      check")
518	    - commit 12ed2e36c98a ("fanotify: FMODE_NONOTIFY and __O_SYNC in sparc
519	      conflict")
520	    - commit bb458c644a59 ("Safer ABI for O_TMPFILE")
521	 - Discussion from Matthew Wilcox about restrictions on 64-bit arguments:
522	   https://lkml.org/lkml/2008/12/12/187
523	 - Recommendation from Greg Kroah-Hartman that unknown flags should be
524	   policed: https://lkml.org/lkml/2014/7/17/577
525	 - Recommendation from Linus Torvalds that x32 system calls should prefer
526	   compatibility with 64-bit versions rather than 32-bit versions:
527	   https://lkml.org/lkml/2011/8/31/244
Hide Line Numbers


About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog