About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / development-process / 2.Process


Based on kernel version 4.8. Page generated on 2016-10-06 23:10 EST.

1	2: HOW THE DEVELOPMENT PROCESS WORKS
2	
3	Linux kernel development in the early 1990's was a pretty loose affair,
4	with relatively small numbers of users and developers involved.  With a
5	user base in the millions and with some 2,000 developers involved over the
6	course of one year, the kernel has since had to evolve a number of
7	processes to keep development happening smoothly.  A solid understanding of
8	how the process works is required in order to be an effective part of it.
9	
10	
11	2.1: THE BIG PICTURE
12	
13	The kernel developers use a loosely time-based release process, with a new
14	major kernel release happening every two or three months.  The recent
15	release history looks like this:
16	
17		2.6.38	March 14, 2011
18		2.6.37	January 4, 2011
19		2.6.36	October 20, 2010
20		2.6.35	August 1, 2010
21		2.6.34	May 15, 2010
22		2.6.33	February 24, 2010
23	
24	Every 2.6.x release is a major kernel release with new features, internal
25	API changes, and more.  A typical 2.6 release can contain nearly 10,000
26	changesets with changes to several hundred thousand lines of code.  2.6 is
27	thus the leading edge of Linux kernel development; the kernel uses a
28	rolling development model which is continually integrating major changes.
29	
30	A relatively straightforward discipline is followed with regard to the
31	merging of patches for each release.  At the beginning of each development
32	cycle, the "merge window" is said to be open.  At that time, code which is
33	deemed to be sufficiently stable (and which is accepted by the development
34	community) is merged into the mainline kernel.  The bulk of changes for a
35	new development cycle (and all of the major changes) will be merged during
36	this time, at a rate approaching 1,000 changes ("patches," or "changesets")
37	per day.
38	
39	(As an aside, it is worth noting that the changes integrated during the
40	merge window do not come out of thin air; they have been collected, tested,
41	and staged ahead of time.  How that process works will be described in
42	detail later on).
43	
44	The merge window lasts for approximately two weeks.  At the end of this
45	time, Linus Torvalds will declare that the window is closed and release the
46	first of the "rc" kernels.  For the kernel which is destined to be 2.6.40,
47	for example, the release which happens at the end of the merge window will
48	be called 2.6.40-rc1.  The -rc1 release is the signal that the time to
49	merge new features has passed, and that the time to stabilize the next
50	kernel has begun.
51	
52	Over the next six to ten weeks, only patches which fix problems should be
53	submitted to the mainline.  On occasion a more significant change will be
54	allowed, but such occasions are rare; developers who try to merge new
55	features outside of the merge window tend to get an unfriendly reception.
56	As a general rule, if you miss the merge window for a given feature, the
57	best thing to do is to wait for the next development cycle.  (An occasional
58	exception is made for drivers for previously-unsupported hardware; if they
59	touch no in-tree code, they cannot cause regressions and should be safe to
60	add at any time).
61	
62	As fixes make their way into the mainline, the patch rate will slow over
63	time.  Linus releases new -rc kernels about once a week; a normal series
64	will get up to somewhere between -rc6 and -rc9 before the kernel is
65	considered to be sufficiently stable and the final 2.6.x release is made.
66	At that point the whole process starts over again.
67	
68	As an example, here is how the 2.6.38 development cycle went (all dates in
69	2011):
70	
71		January 4	2.6.37 stable release
72		January 18	2.6.38-rc1, merge window closes
73		January 21	2.6.38-rc2
74		February 1	2.6.38-rc3
75		February 7	2.6.38-rc4
76		February 15	2.6.38-rc5
77		February 21	2.6.38-rc6
78		March 1		2.6.38-rc7
79		March 7		2.6.38-rc8
80		March 14	2.6.38 stable release
81	
82	How do the developers decide when to close the development cycle and create
83	the stable release?  The most significant metric used is the list of
84	regressions from previous releases.  No bugs are welcome, but those which
85	break systems which worked in the past are considered to be especially
86	serious.  For this reason, patches which cause regressions are looked upon
87	unfavorably and are quite likely to be reverted during the stabilization
88	period.
89	
90	The developers' goal is to fix all known regressions before the stable
91	release is made.  In the real world, this kind of perfection is hard to
92	achieve; there are just too many variables in a project of this size.
93	There comes a point where delaying the final release just makes the problem
94	worse; the pile of changes waiting for the next merge window will grow
95	larger, creating even more regressions the next time around.  So most 2.6.x
96	kernels go out with a handful of known regressions though, hopefully, none
97	of them are serious.
98	
99	Once a stable release is made, its ongoing maintenance is passed off to the
100	"stable team," currently consisting of Greg Kroah-Hartman.  The stable team
101	will release occasional updates to the stable release using the 2.6.x.y
102	numbering scheme.  To be considered for an update release, a patch must (1)
103	fix a significant bug, and (2) already be merged into the mainline for the
104	next development kernel.  Kernels will typically receive stable updates for
105	a little more than one development cycle past their initial release.  So,
106	for example, the 2.6.36 kernel's history looked like:
107	
108		October 10	2.6.36 stable release
109		November 22	2.6.36.1
110		December 9	2.6.36.2
111		January 7	2.6.36.3
112		February 17	2.6.36.4
113	
114	2.6.36.4 was the final stable update for the 2.6.36 release.
115	
116	Some kernels are designated "long term" kernels; they will receive support
117	for a longer period.  As of this writing, the current long term kernels
118	and their maintainers are:
119	
120		2.6.27	Willy Tarreau		(Deep-frozen stable kernel)
121		2.6.32	Greg Kroah-Hartman
122		2.6.35	Andi Kleen		(Embedded flag kernel)
123	
124	The selection of a kernel for long-term support is purely a matter of a
125	maintainer having the need and the time to maintain that release.  There
126	are no known plans for long-term support for any specific upcoming
127	release.
128	
129	
130	2.2: THE LIFECYCLE OF A PATCH
131	
132	Patches do not go directly from the developer's keyboard into the mainline
133	kernel.  There is, instead, a somewhat involved (if somewhat informal)
134	process designed to ensure that each patch is reviewed for quality and that
135	each patch implements a change which is desirable to have in the mainline.
136	This process can happen quickly for minor fixes, or, in the case of large
137	and controversial changes, go on for years.  Much developer frustration
138	comes from a lack of understanding of this process or from attempts to
139	circumvent it.
140	
141	In the hopes of reducing that frustration, this document will describe how
142	a patch gets into the kernel.  What follows below is an introduction which
143	describes the process in a somewhat idealized way.  A much more detailed
144	treatment will come in later sections.
145	
146	The stages that a patch goes through are, generally:
147	
148	 - Design.  This is where the real requirements for the patch - and the way
149	   those requirements will be met - are laid out.  Design work is often
150	   done without involving the community, but it is better to do this work
151	   in the open if at all possible; it can save a lot of time redesigning
152	   things later.
153	
154	 - Early review.  Patches are posted to the relevant mailing list, and
155	   developers on that list reply with any comments they may have.  This
156	   process should turn up any major problems with a patch if all goes
157	   well.
158	
159	 - Wider review.  When the patch is getting close to ready for mainline
160	   inclusion, it should be accepted by a relevant subsystem maintainer -
161	   though this acceptance is not a guarantee that the patch will make it
162	   all the way to the mainline.  The patch will show up in the maintainer's
163	   subsystem tree and into the -next trees (described below).  When the
164	   process works, this step leads to more extensive review of the patch and
165	   the discovery of any problems resulting from the integration of this
166	   patch with work being done by others.
167	
168	-  Please note that most maintainers also have day jobs, so merging
169	   your patch may not be their highest priority.  If your patch is
170	   getting feedback about changes that are needed, you should either
171	   make those changes or justify why they should not be made.  If your
172	   patch has no review complaints but is not being merged by its
173	   appropriate subsystem or driver maintainer, you should be persistent
174	   in updating the patch to the current kernel so that it applies cleanly
175	   and keep sending it for review and merging.
176	
177	 - Merging into the mainline.  Eventually, a successful patch will be
178	   merged into the mainline repository managed by Linus Torvalds.  More
179	   comments and/or problems may surface at this time; it is important that
180	   the developer be responsive to these and fix any issues which arise.
181	
182	 - Stable release.  The number of users potentially affected by the patch
183	   is now large, so, once again, new problems may arise.
184	
185	 - Long-term maintenance.  While it is certainly possible for a developer
186	   to forget about code after merging it, that sort of behavior tends to
187	   leave a poor impression in the development community.  Merging code
188	   eliminates some of the maintenance burden, in that others will fix
189	   problems caused by API changes.  But the original developer should
190	   continue to take responsibility for the code if it is to remain useful
191	   in the longer term.
192	
193	One of the largest mistakes made by kernel developers (or their employers)
194	is to try to cut the process down to a single "merging into the mainline"
195	step.  This approach invariably leads to frustration for everybody
196	involved.
197	
198	
199	2.3: HOW PATCHES GET INTO THE KERNEL
200	
201	There is exactly one person who can merge patches into the mainline kernel
202	repository: Linus Torvalds.  But, of the over 9,500 patches which went
203	into the 2.6.38 kernel, only 112 (around 1.3%) were directly chosen by Linus
204	himself.  The kernel project has long since grown to a size where no single
205	developer could possibly inspect and select every patch unassisted.  The
206	way the kernel developers have addressed this growth is through the use of
207	a lieutenant system built around a chain of trust.
208	
209	The kernel code base is logically broken down into a set of subsystems:
210	networking, specific architecture support, memory management, video
211	devices, etc.  Most subsystems have a designated maintainer, a developer
212	who has overall responsibility for the code within that subsystem.  These
213	subsystem maintainers are the gatekeepers (in a loose way) for the portion
214	of the kernel they manage; they are the ones who will (usually) accept a
215	patch for inclusion into the mainline kernel.
216	
217	Subsystem maintainers each manage their own version of the kernel source
218	tree, usually (but certainly not always) using the git source management
219	tool.  Tools like git (and related tools like quilt or mercurial) allow
220	maintainers to track a list of patches, including authorship information
221	and other metadata.  At any given time, the maintainer can identify which
222	patches in his or her repository are not found in the mainline.
223	
224	When the merge window opens, top-level maintainers will ask Linus to "pull"
225	the patches they have selected for merging from their repositories.  If
226	Linus agrees, the stream of patches will flow up into his repository,
227	becoming part of the mainline kernel.  The amount of attention that Linus
228	pays to specific patches received in a pull operation varies.  It is clear
229	that, sometimes, he looks quite closely.  But, as a general rule, Linus
230	trusts the subsystem maintainers to not send bad patches upstream.
231	
232	Subsystem maintainers, in turn, can pull patches from other maintainers.
233	For example, the networking tree is built from patches which accumulated
234	first in trees dedicated to network device drivers, wireless networking,
235	etc.  This chain of repositories can be arbitrarily long, though it rarely
236	exceeds two or three links.  Since each maintainer in the chain trusts
237	those managing lower-level trees, this process is known as the "chain of
238	trust."
239	
240	Clearly, in a system like this, getting patches into the kernel depends on
241	finding the right maintainer.  Sending patches directly to Linus is not
242	normally the right way to go.
243	
244	
245	2.4: NEXT TREES
246	
247	The chain of subsystem trees guides the flow of patches into the kernel,
248	but it also raises an interesting question: what if somebody wants to look
249	at all of the patches which are being prepared for the next merge window?
250	Developers will be interested in what other changes are pending to see
251	whether there are any conflicts to worry about; a patch which changes a
252	core kernel function prototype, for example, will conflict with any other
253	patches which use the older form of that function.  Reviewers and testers
254	want access to the changes in their integrated form before all of those
255	changes land in the mainline kernel.  One could pull changes from all of
256	the interesting subsystem trees, but that would be a big and error-prone
257	job.
258	
259	The answer comes in the form of -next trees, where subsystem trees are
260	collected for testing and review.  The older of these trees, maintained by
261	Andrew Morton, is called "-mm" (for memory management, which is how it got
262	started).  The -mm tree integrates patches from a long list of subsystem
263	trees; it also has some patches aimed at helping with debugging.
264	
265	Beyond that, -mm contains a significant collection of patches which have
266	been selected by Andrew directly.  These patches may have been posted on a
267	mailing list, or they may apply to a part of the kernel for which there is
268	no designated subsystem tree.  As a result, -mm operates as a sort of
269	subsystem tree of last resort; if there is no other obvious path for a
270	patch into the mainline, it is likely to end up in -mm.  Miscellaneous
271	patches which accumulate in -mm will eventually either be forwarded on to
272	an appropriate subsystem tree or be sent directly to Linus.  In a typical
273	development cycle, approximately 5-10% of the patches going into the
274	mainline get there via -mm.
275	
276	The current -mm patch is available in the "mmotm" (-mm of the moment)
277	directory at:
278	
279		http://www.ozlabs.org/~akpm/mmotm/
280	
281	Use of the MMOTM tree is likely to be a frustrating experience, though;
282	there is a definite chance that it will not even compile.
283	
284	The primary tree for next-cycle patch merging is linux-next, maintained by
285	Stephen Rothwell.  The linux-next tree is, by design, a snapshot of what
286	the mainline is expected to look like after the next merge window closes.
287	Linux-next trees are announced on the linux-kernel and linux-next mailing
288	lists when they are assembled; they can be downloaded from:
289	
290		http://www.kernel.org/pub/linux/kernel/next/
291	
292	Linux-next has become an integral part of the kernel development process;
293	all patches merged during a given merge window should really have found
294	their way into linux-next some time before the merge window opens.
295	
296	
297	2.4.1: STAGING TREES
298	
299	The kernel source tree contains the drivers/staging/ directory, where
300	many sub-directories for drivers or filesystems that are on their way to
301	being added to the kernel tree live.  They remain in drivers/staging while
302	they still need more work; once complete, they can be moved into the
303	kernel proper.  This is a way to keep track of drivers that aren't
304	up to Linux kernel coding or quality standards, but people may want to use
305	them and track development.
306	
307	Greg Kroah-Hartman currently maintains the staging tree.  Drivers that
308	still need work are sent to him, with each driver having its own
309	subdirectory in drivers/staging/.  Along with the driver source files, a
310	TODO file should be present in the directory as well.  The TODO file lists
311	the pending work that the driver needs for acceptance into the kernel
312	proper, as well as a list of people that should be Cc'd for any patches to
313	the driver.  Current rules require that drivers contributed to staging
314	must, at a minimum, compile properly.
315	
316	Staging can be a relatively easy way to get new drivers into the mainline
317	where, with luck, they will come to the attention of other developers and
318	improve quickly.  Entry into staging is not the end of the story, though;
319	code in staging which is not seeing regular progress will eventually be
320	removed.  Distributors also tend to be relatively reluctant to enable
321	staging drivers.  So staging is, at best, a stop on the way toward becoming
322	a proper mainline driver.
323	
324	
325	2.5: TOOLS
326	
327	As can be seen from the above text, the kernel development process depends
328	heavily on the ability to herd collections of patches in various
329	directions.  The whole thing would not work anywhere near as well as it
330	does without suitably powerful tools.  Tutorials on how to use these tools
331	are well beyond the scope of this document, but there is space for a few
332	pointers.
333	
334	By far the dominant source code management system used by the kernel
335	community is git.  Git is one of a number of distributed version control
336	systems being developed in the free software community.  It is well tuned
337	for kernel development, in that it performs quite well when dealing with
338	large repositories and large numbers of patches.  It also has a reputation
339	for being difficult to learn and use, though it has gotten better over
340	time.  Some sort of familiarity with git is almost a requirement for kernel
341	developers; even if they do not use it for their own work, they'll need git
342	to keep up with what other developers (and the mainline) are doing.
343	
344	Git is now packaged by almost all Linux distributions.  There is a home
345	page at:
346	
347		http://git-scm.com/
348	
349	That page has pointers to documentation and tutorials.
350	
351	Among the kernel developers who do not use git, the most popular choice is
352	almost certainly Mercurial:
353	
354		http://www.selenic.com/mercurial/
355	
356	Mercurial shares many features with git, but it provides an interface which
357	many find easier to use.
358	
359	The other tool worth knowing about is Quilt:
360	
361		http://savannah.nongnu.org/projects/quilt/
362	
363	Quilt is a patch management system, rather than a source code management
364	system.  It does not track history over time; it is, instead, oriented
365	toward tracking a specific set of changes against an evolving code base.
366	Some major subsystem maintainers use quilt to manage patches intended to go
367	upstream.  For the management of certain kinds of trees (-mm, for example),
368	quilt is the best tool for the job.
369	
370	
371	2.6: MAILING LISTS
372	
373	A great deal of Linux kernel development work is done by way of mailing
374	lists.  It is hard to be a fully-functioning member of the community
375	without joining at least one list somewhere.  But Linux mailing lists also
376	represent a potential hazard to developers, who risk getting buried under a
377	load of electronic mail, running afoul of the conventions used on the Linux
378	lists, or both.
379	
380	Most kernel mailing lists are run on vger.kernel.org; the master list can
381	be found at:
382	
383		http://vger.kernel.org/vger-lists.html
384	
385	There are lists hosted elsewhere, though; a number of them are at
386	lists.redhat.com.
387	
388	The core mailing list for kernel development is, of course, linux-kernel.
389	This list is an intimidating place to be; volume can reach 500 messages per
390	day, the amount of noise is high, the conversation can be severely
391	technical, and participants are not always concerned with showing a high
392	degree of politeness.  But there is no other place where the kernel
393	development community comes together as a whole; developers who avoid this
394	list will miss important information.
395	
396	There are a few hints which can help with linux-kernel survival:
397	
398	- Have the list delivered to a separate folder, rather than your main
399	  mailbox.  One must be able to ignore the stream for sustained periods of
400	  time.
401	
402	- Do not try to follow every conversation - nobody else does.  It is
403	  important to filter on both the topic of interest (though note that
404	  long-running conversations can drift away from the original subject
405	  without changing the email subject line) and the people who are
406	  participating.
407	
408	- Do not feed the trolls.  If somebody is trying to stir up an angry
409	  response, ignore them.
410	
411	- When responding to linux-kernel email (or that on other lists) preserve
412	  the Cc: header for all involved.  In the absence of a strong reason (such
413	  as an explicit request), you should never remove recipients.  Always make
414	  sure that the person you are responding to is in the Cc: list.  This
415	  convention also makes it unnecessary to explicitly ask to be copied on
416	  replies to your postings.
417	
418	- Search the list archives (and the net as a whole) before asking
419	  questions.  Some developers can get impatient with people who clearly
420	  have not done their homework.
421	
422	- Avoid top-posting (the practice of putting your answer above the quoted
423	  text you are responding to).  It makes your response harder to read and
424	  makes a poor impression.
425	
426	- Ask on the correct mailing list.  Linux-kernel may be the general meeting
427	  point, but it is not the best place to find developers from all
428	  subsystems.
429	
430	The last point - finding the correct mailing list - is a common place for
431	beginning developers to go wrong.  Somebody who asks a networking-related
432	question on linux-kernel will almost certainly receive a polite suggestion
433	to ask on the netdev list instead, as that is the list frequented by most
434	networking developers.  Other lists exist for the SCSI, video4linux, IDE,
435	filesystem, etc. subsystems.  The best place to look for mailing lists is
436	in the MAINTAINERS file packaged with the kernel source.
437	
438	
439	2.7: GETTING STARTED WITH KERNEL DEVELOPMENT
440	
441	Questions about how to get started with the kernel development process are
442	common - from both individuals and companies.  Equally common are missteps
443	which make the beginning of the relationship harder than it has to be.
444	
445	Companies often look to hire well-known developers to get a development
446	group started.  This can, in fact, be an effective technique.  But it also
447	tends to be expensive and does not do much to grow the pool of experienced
448	kernel developers.  It is possible to bring in-house developers up to speed
449	on Linux kernel development, given the investment of a bit of time.  Taking
450	this time can endow an employer with a group of developers who understand
451	the kernel and the company both, and who can help to train others as well.
452	Over the medium term, this is often the more profitable approach.
453	
454	Individual developers are often, understandably, at a loss for a place to
455	start.  Beginning with a large project can be intimidating; one often wants
456	to test the waters with something smaller first.  This is the point where
457	some developers jump into the creation of patches fixing spelling errors or
458	minor coding style issues.  Unfortunately, such patches create a level of
459	noise which is distracting for the development community as a whole, so,
460	increasingly, they are looked down upon.  New developers wishing to
461	introduce themselves to the community will not get the sort of reception
462	they wish for by these means.
463	
464	Andrew Morton gives this advice for aspiring kernel developers
465	
466		The #1 project for all kernel beginners should surely be "make sure
467		that the kernel runs perfectly at all times on all machines which
468		you can lay your hands on".  Usually the way to do this is to work
469		with others on getting things fixed up (this can require
470		persistence!) but that's fine - it's a part of kernel development.
471	
472	(http://lwn.net/Articles/283982/).
473	
474	In the absence of obvious problems to fix, developers are advised to look
475	at the current lists of regressions and open bugs in general.  There is
476	never any shortage of issues in need of fixing; by addressing these issues,
477	developers will gain experience with the process while, at the same time,
478	building respect with the rest of the development community.
Hide Line Numbers


About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog