Based on kernel version 4.9. Page generated on 2016-12-21 14:28 EST.
1 .. _development_process: 2 3 How the development process works 4 ================================= 5 6 Linux kernel development in the early 1990's was a pretty loose affair, 7 with relatively small numbers of users and developers involved. With a 8 user base in the millions and with some 2,000 developers involved over the 9 course of one year, the kernel has since had to evolve a number of 10 processes to keep development happening smoothly. A solid understanding of 11 how the process works is required in order to be an effective part of it. 12 13 The big picture 14 --------------- 15 16 The kernel developers use a loosely time-based release process, with a new 17 major kernel release happening every two or three months. The recent 18 release history looks like this: 19 20 ====== ================= 21 2.6.38 March 14, 2011 22 2.6.37 January 4, 2011 23 2.6.36 October 20, 2010 24 2.6.35 August 1, 2010 25 2.6.34 May 15, 2010 26 2.6.33 February 24, 2010 27 ====== ================= 28 29 Every 2.6.x release is a major kernel release with new features, internal 30 API changes, and more. A typical 2.6 release can contain nearly 10,000 31 changesets with changes to several hundred thousand lines of code. 2.6 is 32 thus the leading edge of Linux kernel development; the kernel uses a 33 rolling development model which is continually integrating major changes. 34 35 A relatively straightforward discipline is followed with regard to the 36 merging of patches for each release. At the beginning of each development 37 cycle, the "merge window" is said to be open. At that time, code which is 38 deemed to be sufficiently stable (and which is accepted by the development 39 community) is merged into the mainline kernel. The bulk of changes for a 40 new development cycle (and all of the major changes) will be merged during 41 this time, at a rate approaching 1,000 changes ("patches," or "changesets") 42 per day. 43 44 (As an aside, it is worth noting that the changes integrated during the 45 merge window do not come out of thin air; they have been collected, tested, 46 and staged ahead of time. How that process works will be described in 47 detail later on). 48 49 The merge window lasts for approximately two weeks. At the end of this 50 time, Linus Torvalds will declare that the window is closed and release the 51 first of the "rc" kernels. For the kernel which is destined to be 2.6.40, 52 for example, the release which happens at the end of the merge window will 53 be called 2.6.40-rc1. The -rc1 release is the signal that the time to 54 merge new features has passed, and that the time to stabilize the next 55 kernel has begun. 56 57 Over the next six to ten weeks, only patches which fix problems should be 58 submitted to the mainline. On occasion a more significant change will be 59 allowed, but such occasions are rare; developers who try to merge new 60 features outside of the merge window tend to get an unfriendly reception. 61 As a general rule, if you miss the merge window for a given feature, the 62 best thing to do is to wait for the next development cycle. (An occasional 63 exception is made for drivers for previously-unsupported hardware; if they 64 touch no in-tree code, they cannot cause regressions and should be safe to 65 add at any time). 66 67 As fixes make their way into the mainline, the patch rate will slow over 68 time. Linus releases new -rc kernels about once a week; a normal series 69 will get up to somewhere between -rc6 and -rc9 before the kernel is 70 considered to be sufficiently stable and the final 2.6.x release is made. 71 At that point the whole process starts over again. 72 73 As an example, here is how the 2.6.38 development cycle went (all dates in 74 2011): 75 76 ============== =============================== 77 January 4 2.6.37 stable release 78 January 18 2.6.38-rc1, merge window closes 79 January 21 2.6.38-rc2 80 February 1 2.6.38-rc3 81 February 7 2.6.38-rc4 82 February 15 2.6.38-rc5 83 February 21 2.6.38-rc6 84 March 1 2.6.38-rc7 85 March 7 2.6.38-rc8 86 March 14 2.6.38 stable release 87 ============== =============================== 88 89 How do the developers decide when to close the development cycle and create 90 the stable release? The most significant metric used is the list of 91 regressions from previous releases. No bugs are welcome, but those which 92 break systems which worked in the past are considered to be especially 93 serious. For this reason, patches which cause regressions are looked upon 94 unfavorably and are quite likely to be reverted during the stabilization 95 period. 96 97 The developers' goal is to fix all known regressions before the stable 98 release is made. In the real world, this kind of perfection is hard to 99 achieve; there are just too many variables in a project of this size. 100 There comes a point where delaying the final release just makes the problem 101 worse; the pile of changes waiting for the next merge window will grow 102 larger, creating even more regressions the next time around. So most 2.6.x 103 kernels go out with a handful of known regressions though, hopefully, none 104 of them are serious. 105 106 Once a stable release is made, its ongoing maintenance is passed off to the 107 "stable team," currently consisting of Greg Kroah-Hartman. The stable team 108 will release occasional updates to the stable release using the 2.6.x.y 109 numbering scheme. To be considered for an update release, a patch must (1) 110 fix a significant bug, and (2) already be merged into the mainline for the 111 next development kernel. Kernels will typically receive stable updates for 112 a little more than one development cycle past their initial release. So, 113 for example, the 2.6.36 kernel's history looked like: 114 115 ============== =============================== 116 October 10 2.6.36 stable release 117 November 22 2.6.36.1 118 December 9 2.6.36.2 119 January 7 2.6.36.3 120 February 17 2.6.36.4 121 ============== =============================== 122 123 2.6.36.4 was the final stable update for the 2.6.36 release. 124 125 Some kernels are designated "long term" kernels; they will receive support 126 for a longer period. As of this writing, the current long term kernels 127 and their maintainers are: 128 129 ====== ====================== =========================== 130 2.6.27 Willy Tarreau (Deep-frozen stable kernel) 131 2.6.32 Greg Kroah-Hartman 132 2.6.35 Andi Kleen (Embedded flag kernel) 133 ====== ====================== =========================== 134 135 The selection of a kernel for long-term support is purely a matter of a 136 maintainer having the need and the time to maintain that release. There 137 are no known plans for long-term support for any specific upcoming 138 release. 139 140 141 The lifecycle of a patch 142 ------------------------ 143 144 Patches do not go directly from the developer's keyboard into the mainline 145 kernel. There is, instead, a somewhat involved (if somewhat informal) 146 process designed to ensure that each patch is reviewed for quality and that 147 each patch implements a change which is desirable to have in the mainline. 148 This process can happen quickly for minor fixes, or, in the case of large 149 and controversial changes, go on for years. Much developer frustration 150 comes from a lack of understanding of this process or from attempts to 151 circumvent it. 152 153 In the hopes of reducing that frustration, this document will describe how 154 a patch gets into the kernel. What follows below is an introduction which 155 describes the process in a somewhat idealized way. A much more detailed 156 treatment will come in later sections. 157 158 The stages that a patch goes through are, generally: 159 160 - Design. This is where the real requirements for the patch - and the way 161 those requirements will be met - are laid out. Design work is often 162 done without involving the community, but it is better to do this work 163 in the open if at all possible; it can save a lot of time redesigning 164 things later. 165 166 - Early review. Patches are posted to the relevant mailing list, and 167 developers on that list reply with any comments they may have. This 168 process should turn up any major problems with a patch if all goes 169 well. 170 171 - Wider review. When the patch is getting close to ready for mainline 172 inclusion, it should be accepted by a relevant subsystem maintainer - 173 though this acceptance is not a guarantee that the patch will make it 174 all the way to the mainline. The patch will show up in the maintainer's 175 subsystem tree and into the -next trees (described below). When the 176 process works, this step leads to more extensive review of the patch and 177 the discovery of any problems resulting from the integration of this 178 patch with work being done by others. 179 180 - Please note that most maintainers also have day jobs, so merging 181 your patch may not be their highest priority. If your patch is 182 getting feedback about changes that are needed, you should either 183 make those changes or justify why they should not be made. If your 184 patch has no review complaints but is not being merged by its 185 appropriate subsystem or driver maintainer, you should be persistent 186 in updating the patch to the current kernel so that it applies cleanly 187 and keep sending it for review and merging. 188 189 - Merging into the mainline. Eventually, a successful patch will be 190 merged into the mainline repository managed by Linus Torvalds. More 191 comments and/or problems may surface at this time; it is important that 192 the developer be responsive to these and fix any issues which arise. 193 194 - Stable release. The number of users potentially affected by the patch 195 is now large, so, once again, new problems may arise. 196 197 - Long-term maintenance. While it is certainly possible for a developer 198 to forget about code after merging it, that sort of behavior tends to 199 leave a poor impression in the development community. Merging code 200 eliminates some of the maintenance burden, in that others will fix 201 problems caused by API changes. But the original developer should 202 continue to take responsibility for the code if it is to remain useful 203 in the longer term. 204 205 One of the largest mistakes made by kernel developers (or their employers) 206 is to try to cut the process down to a single "merging into the mainline" 207 step. This approach invariably leads to frustration for everybody 208 involved. 209 210 How patches get into the Kernel 211 ------------------------------- 212 213 There is exactly one person who can merge patches into the mainline kernel 214 repository: Linus Torvalds. But, of the over 9,500 patches which went 215 into the 2.6.38 kernel, only 112 (around 1.3%) were directly chosen by Linus 216 himself. The kernel project has long since grown to a size where no single 217 developer could possibly inspect and select every patch unassisted. The 218 way the kernel developers have addressed this growth is through the use of 219 a lieutenant system built around a chain of trust. 220 221 The kernel code base is logically broken down into a set of subsystems: 222 networking, specific architecture support, memory management, video 223 devices, etc. Most subsystems have a designated maintainer, a developer 224 who has overall responsibility for the code within that subsystem. These 225 subsystem maintainers are the gatekeepers (in a loose way) for the portion 226 of the kernel they manage; they are the ones who will (usually) accept a 227 patch for inclusion into the mainline kernel. 228 229 Subsystem maintainers each manage their own version of the kernel source 230 tree, usually (but certainly not always) using the git source management 231 tool. Tools like git (and related tools like quilt or mercurial) allow 232 maintainers to track a list of patches, including authorship information 233 and other metadata. At any given time, the maintainer can identify which 234 patches in his or her repository are not found in the mainline. 235 236 When the merge window opens, top-level maintainers will ask Linus to "pull" 237 the patches they have selected for merging from their repositories. If 238 Linus agrees, the stream of patches will flow up into his repository, 239 becoming part of the mainline kernel. The amount of attention that Linus 240 pays to specific patches received in a pull operation varies. It is clear 241 that, sometimes, he looks quite closely. But, as a general rule, Linus 242 trusts the subsystem maintainers to not send bad patches upstream. 243 244 Subsystem maintainers, in turn, can pull patches from other maintainers. 245 For example, the networking tree is built from patches which accumulated 246 first in trees dedicated to network device drivers, wireless networking, 247 etc. This chain of repositories can be arbitrarily long, though it rarely 248 exceeds two or three links. Since each maintainer in the chain trusts 249 those managing lower-level trees, this process is known as the "chain of 250 trust." 251 252 Clearly, in a system like this, getting patches into the kernel depends on 253 finding the right maintainer. Sending patches directly to Linus is not 254 normally the right way to go. 255 256 257 Next trees 258 ---------- 259 260 The chain of subsystem trees guides the flow of patches into the kernel, 261 but it also raises an interesting question: what if somebody wants to look 262 at all of the patches which are being prepared for the next merge window? 263 Developers will be interested in what other changes are pending to see 264 whether there are any conflicts to worry about; a patch which changes a 265 core kernel function prototype, for example, will conflict with any other 266 patches which use the older form of that function. Reviewers and testers 267 want access to the changes in their integrated form before all of those 268 changes land in the mainline kernel. One could pull changes from all of 269 the interesting subsystem trees, but that would be a big and error-prone 270 job. 271 272 The answer comes in the form of -next trees, where subsystem trees are 273 collected for testing and review. The older of these trees, maintained by 274 Andrew Morton, is called "-mm" (for memory management, which is how it got 275 started). The -mm tree integrates patches from a long list of subsystem 276 trees; it also has some patches aimed at helping with debugging. 277 278 Beyond that, -mm contains a significant collection of patches which have 279 been selected by Andrew directly. These patches may have been posted on a 280 mailing list, or they may apply to a part of the kernel for which there is 281 no designated subsystem tree. As a result, -mm operates as a sort of 282 subsystem tree of last resort; if there is no other obvious path for a 283 patch into the mainline, it is likely to end up in -mm. Miscellaneous 284 patches which accumulate in -mm will eventually either be forwarded on to 285 an appropriate subsystem tree or be sent directly to Linus. In a typical 286 development cycle, approximately 5-10% of the patches going into the 287 mainline get there via -mm. 288 289 The current -mm patch is available in the "mmotm" (-mm of the moment) 290 directory at: 291 292 http://www.ozlabs.org/~akpm/mmotm/ 293 294 Use of the MMOTM tree is likely to be a frustrating experience, though; 295 there is a definite chance that it will not even compile. 296 297 The primary tree for next-cycle patch merging is linux-next, maintained by 298 Stephen Rothwell. The linux-next tree is, by design, a snapshot of what 299 the mainline is expected to look like after the next merge window closes. 300 Linux-next trees are announced on the linux-kernel and linux-next mailing 301 lists when they are assembled; they can be downloaded from: 302 303 http://www.kernel.org/pub/linux/kernel/next/ 304 305 Linux-next has become an integral part of the kernel development process; 306 all patches merged during a given merge window should really have found 307 their way into linux-next some time before the merge window opens. 308 309 310 Staging trees 311 ------------- 312 313 The kernel source tree contains the drivers/staging/ directory, where 314 many sub-directories for drivers or filesystems that are on their way to 315 being added to the kernel tree live. They remain in drivers/staging while 316 they still need more work; once complete, they can be moved into the 317 kernel proper. This is a way to keep track of drivers that aren't 318 up to Linux kernel coding or quality standards, but people may want to use 319 them and track development. 320 321 Greg Kroah-Hartman currently maintains the staging tree. Drivers that 322 still need work are sent to him, with each driver having its own 323 subdirectory in drivers/staging/. Along with the driver source files, a 324 TODO file should be present in the directory as well. The TODO file lists 325 the pending work that the driver needs for acceptance into the kernel 326 proper, as well as a list of people that should be Cc'd for any patches to 327 the driver. Current rules require that drivers contributed to staging 328 must, at a minimum, compile properly. 329 330 Staging can be a relatively easy way to get new drivers into the mainline 331 where, with luck, they will come to the attention of other developers and 332 improve quickly. Entry into staging is not the end of the story, though; 333 code in staging which is not seeing regular progress will eventually be 334 removed. Distributors also tend to be relatively reluctant to enable 335 staging drivers. So staging is, at best, a stop on the way toward becoming 336 a proper mainline driver. 337 338 339 Tools 340 ----- 341 342 As can be seen from the above text, the kernel development process depends 343 heavily on the ability to herd collections of patches in various 344 directions. The whole thing would not work anywhere near as well as it 345 does without suitably powerful tools. Tutorials on how to use these tools 346 are well beyond the scope of this document, but there is space for a few 347 pointers. 348 349 By far the dominant source code management system used by the kernel 350 community is git. Git is one of a number of distributed version control 351 systems being developed in the free software community. It is well tuned 352 for kernel development, in that it performs quite well when dealing with 353 large repositories and large numbers of patches. It also has a reputation 354 for being difficult to learn and use, though it has gotten better over 355 time. Some sort of familiarity with git is almost a requirement for kernel 356 developers; even if they do not use it for their own work, they'll need git 357 to keep up with what other developers (and the mainline) are doing. 358 359 Git is now packaged by almost all Linux distributions. There is a home 360 page at: 361 362 http://git-scm.com/ 363 364 That page has pointers to documentation and tutorials. 365 366 Among the kernel developers who do not use git, the most popular choice is 367 almost certainly Mercurial: 368 369 http://www.selenic.com/mercurial/ 370 371 Mercurial shares many features with git, but it provides an interface which 372 many find easier to use. 373 374 The other tool worth knowing about is Quilt: 375 376 http://savannah.nongnu.org/projects/quilt/ 377 378 Quilt is a patch management system, rather than a source code management 379 system. It does not track history over time; it is, instead, oriented 380 toward tracking a specific set of changes against an evolving code base. 381 Some major subsystem maintainers use quilt to manage patches intended to go 382 upstream. For the management of certain kinds of trees (-mm, for example), 383 quilt is the best tool for the job. 384 385 386 Mailing lists 387 ------------- 388 389 A great deal of Linux kernel development work is done by way of mailing 390 lists. It is hard to be a fully-functioning member of the community 391 without joining at least one list somewhere. But Linux mailing lists also 392 represent a potential hazard to developers, who risk getting buried under a 393 load of electronic mail, running afoul of the conventions used on the Linux 394 lists, or both. 395 396 Most kernel mailing lists are run on vger.kernel.org; the master list can 397 be found at: 398 399 http://vger.kernel.org/vger-lists.html 400 401 There are lists hosted elsewhere, though; a number of them are at 402 lists.redhat.com. 403 404 The core mailing list for kernel development is, of course, linux-kernel. 405 This list is an intimidating place to be; volume can reach 500 messages per 406 day, the amount of noise is high, the conversation can be severely 407 technical, and participants are not always concerned with showing a high 408 degree of politeness. But there is no other place where the kernel 409 development community comes together as a whole; developers who avoid this 410 list will miss important information. 411 412 There are a few hints which can help with linux-kernel survival: 413 414 - Have the list delivered to a separate folder, rather than your main 415 mailbox. One must be able to ignore the stream for sustained periods of 416 time. 417 418 - Do not try to follow every conversation - nobody else does. It is 419 important to filter on both the topic of interest (though note that 420 long-running conversations can drift away from the original subject 421 without changing the email subject line) and the people who are 422 participating. 423 424 - Do not feed the trolls. If somebody is trying to stir up an angry 425 response, ignore them. 426 427 - When responding to linux-kernel email (or that on other lists) preserve 428 the Cc: header for all involved. In the absence of a strong reason (such 429 as an explicit request), you should never remove recipients. Always make 430 sure that the person you are responding to is in the Cc: list. This 431 convention also makes it unnecessary to explicitly ask to be copied on 432 replies to your postings. 433 434 - Search the list archives (and the net as a whole) before asking 435 questions. Some developers can get impatient with people who clearly 436 have not done their homework. 437 438 - Avoid top-posting (the practice of putting your answer above the quoted 439 text you are responding to). It makes your response harder to read and 440 makes a poor impression. 441 442 - Ask on the correct mailing list. Linux-kernel may be the general meeting 443 point, but it is not the best place to find developers from all 444 subsystems. 445 446 The last point - finding the correct mailing list - is a common place for 447 beginning developers to go wrong. Somebody who asks a networking-related 448 question on linux-kernel will almost certainly receive a polite suggestion 449 to ask on the netdev list instead, as that is the list frequented by most 450 networking developers. Other lists exist for the SCSI, video4linux, IDE, 451 filesystem, etc. subsystems. The best place to look for mailing lists is 452 in the MAINTAINERS file packaged with the kernel source. 453 454 455 Getting started with Kernel development 456 --------------------------------------- 457 458 Questions about how to get started with the kernel development process are 459 common - from both individuals and companies. Equally common are missteps 460 which make the beginning of the relationship harder than it has to be. 461 462 Companies often look to hire well-known developers to get a development 463 group started. This can, in fact, be an effective technique. But it also 464 tends to be expensive and does not do much to grow the pool of experienced 465 kernel developers. It is possible to bring in-house developers up to speed 466 on Linux kernel development, given the investment of a bit of time. Taking 467 this time can endow an employer with a group of developers who understand 468 the kernel and the company both, and who can help to train others as well. 469 Over the medium term, this is often the more profitable approach. 470 471 Individual developers are often, understandably, at a loss for a place to 472 start. Beginning with a large project can be intimidating; one often wants 473 to test the waters with something smaller first. This is the point where 474 some developers jump into the creation of patches fixing spelling errors or 475 minor coding style issues. Unfortunately, such patches create a level of 476 noise which is distracting for the development community as a whole, so, 477 increasingly, they are looked down upon. New developers wishing to 478 introduce themselves to the community will not get the sort of reception 479 they wish for by these means. 480 481 Andrew Morton gives this advice for aspiring kernel developers 482 483 :: 484 485 The #1 project for all kernel beginners should surely be "make sure 486 that the kernel runs perfectly at all times on all machines which 487 you can lay your hands on". Usually the way to do this is to work 488 with others on getting things fixed up (this can require 489 persistence!) but that's fine - it's a part of kernel development. 490 491 (http://lwn.net/Articles/283982/). 492 493 In the absence of obvious problems to fix, developers are advised to look 494 at the current lists of regressions and open bugs in general. There is 495 never any shortage of issues in need of fixing; by addressing these issues, 496 developers will gain experience with the process while, at the same time, 497 building respect with the rest of the development community.