Based on kernel version 2.6.33. Page generated on 2010-02-24 15:35 EST.
1 CGROUPS 2 ------- 3 4 Written by Paul Menage <menage[AT]google[DOT]com> based on 5 Documentation/cgroups/cpusets.txt 6 7 Original copyright statements from cpusets.txt: 8 Portions Copyright (C) 2004 BULL SA. 9 Portions Copyright (c) 2004-2006 Silicon Graphics, Inc. 10 Modified by Paul Jackson <pj[AT]sgi[DOT]com> 11 Modified by Christoph Lameter <clameter[AT]sgi[DOT]com> 12 13 CONTENTS: 14 ========= 15 16 1. Control Groups 17 1.1 What are cgroups ? 18 1.2 Why are cgroups needed ? 19 1.3 How are cgroups implemented ? 20 1.4 What does notify_on_release do ? 21 1.5 How do I use cgroups ? 22 2. Usage Examples and Syntax 23 2.1 Basic Usage 24 2.2 Attaching processes 25 3. Kernel API 26 3.1 Overview 27 3.2 Synchronization 28 3.3 Subsystem API 29 4. Questions 30 31 1. Control Groups 32 ================= 33 34 1.1 What are cgroups ? 35 ---------------------- 36 37 Control Groups provide a mechanism for aggregating/partitioning sets of 38 tasks, and all their future children, into hierarchical groups with 39 specialized behaviour. 40 41 Definitions: 42 43 A *cgroup* associates a set of tasks with a set of parameters for one 44 or more subsystems. 45 46 A *subsystem* is a module that makes use of the task grouping 47 facilities provided by cgroups to treat groups of tasks in 48 particular ways. A subsystem is typically a "resource controller" that 49 schedules a resource or applies per-cgroup limits, but it may be 50 anything that wants to act on a group of processes, e.g. a 51 virtualization subsystem. 52 53 A *hierarchy* is a set of cgroups arranged in a tree, such that 54 every task in the system is in exactly one of the cgroups in the 55 hierarchy, and a set of subsystems; each subsystem has system-specific 56 state attached to each cgroup in the hierarchy. Each hierarchy has 57 an instance of the cgroup virtual filesystem associated with it. 58 59 At any one time there may be multiple active hierarchies of task 60 cgroups. Each hierarchy is a partition of all tasks in the system. 61 62 User level code may create and destroy cgroups by name in an 63 instance of the cgroup virtual file system, specify and query to 64 which cgroup a task is assigned, and list the task pids assigned to 65 a cgroup. Those creations and assignments only affect the hierarchy 66 associated with that instance of the cgroup file system. 67 68 On their own, the only use for cgroups is for simple job 69 tracking. The intention is that other subsystems hook into the generic 70 cgroup support to provide new attributes for cgroups, such as 71 accounting/limiting the resources which processes in a cgroup can 72 access. For example, cpusets (see Documentation/cgroups/cpusets.txt) allows 73 you to associate a set of CPUs and a set of memory nodes with the 74 tasks in each cgroup. 75 76 1.2 Why are cgroups needed ? 77 ---------------------------- 78 79 There are multiple efforts to provide process aggregations in the 80 Linux kernel, mainly for resource tracking purposes. Such efforts 81 include cpusets, CKRM/ResGroups, UserBeanCounters, and virtual server 82 namespaces. These all require the basic notion of a 83 grouping/partitioning of processes, with newly forked processes ending 84 in the same group (cgroup) as their parent process. 85 86 The kernel cgroup patch provides the minimum essential kernel 87 mechanisms required to efficiently implement such groups. It has 88 minimal impact on the system fast paths, and provides hooks for 89 specific subsystems such as cpusets to provide additional behaviour as 90 desired. 91 92 Multiple hierarchy support is provided to allow for situations where 93 the division of tasks into cgroups is distinctly different for 94 different subsystems - having parallel hierarchies allows each 95 hierarchy to be a natural division of tasks, without having to handle 96 complex combinations of tasks that would be present if several 97 unrelated subsystems needed to be forced into the same tree of 98 cgroups. 99 100 At one extreme, each resource controller or subsystem could be in a 101 separate hierarchy; at the other extreme, all subsystems 102 would be attached to the same hierarchy. 103 104 As an example of a scenario (originally proposed by vatsa[AT]in.ibm[DOT]com) 105 that can benefit from multiple hierarchies, consider a large 106 university server with various users - students, professors, system 107 tasks etc. The resource planning for this server could be along the 108 following lines: 109 110 CPU : Top cpuset 111 / \ 112 CPUSet1 CPUSet2 113 | | 114 (Profs) (Students) 115 116 In addition (system tasks) are attached to topcpuset (so 117 that they can run anywhere) with a limit of 20% 118 119 Memory : Professors (50%), students (30%), system (20%) 120 121 Disk : Prof (50%), students (30%), system (20%) 122 123 Network : WWW browsing (20%), Network File System (60%), others (20%) 124 / \ 125 Prof (15%) students (5%) 126 127 Browsers like Firefox/Lynx go into the WWW network class, while (k)nfsd go 128 into NFS network class. 129 130 At the same time Firefox/Lynx will share an appropriate CPU/Memory class 131 depending on who launched it (prof/student). 132 133 With the ability to classify tasks differently for different resources 134 (by putting those resource subsystems in different hierarchies) then 135 the admin can easily set up a script which receives exec notifications 136 and depending on who is launching the browser he can 137 138 # echo browser_pid > /mnt/<restype>/<userclass>/tasks 139 140 With only a single hierarchy, he now would potentially have to create 141 a separate cgroup for every browser launched and associate it with 142 approp network and other resource class. This may lead to 143 proliferation of such cgroups. 144 145 Also lets say that the administrator would like to give enhanced network 146 access temporarily to a student's browser (since it is night and the user 147 wants to do online gaming :)) OR give one of the students simulation 148 apps enhanced CPU power, 149 150 With ability to write pids directly to resource classes, it's just a 151 matter of : 152 153 # echo pid > /mnt/network/<new_class>/tasks 154 (after some time) 155 # echo pid > /mnt/network/<orig_class>/tasks 156 157 Without this ability, he would have to split the cgroup into 158 multiple separate ones and then associate the new cgroups with the 159 new resource classes. 160 161 162 163 1.3 How are cgroups implemented ? 164 --------------------------------- 165 166 Control Groups extends the kernel as follows: 167 168 - Each task in the system has a reference-counted pointer to a 169 css_set. 170 171 - A css_set contains a set of reference-counted pointers to 172 cgroup_subsys_state objects, one for each cgroup subsystem 173 registered in the system. There is no direct link from a task to 174 the cgroup of which it's a member in each hierarchy, but this 175 can be determined by following pointers through the 176 cgroup_subsys_state objects. This is because accessing the 177 subsystem state is something that's expected to happen frequently 178 and in performance-critical code, whereas operations that require a 179 task's actual cgroup assignments (in particular, moving between 180 cgroups) are less common. A linked list runs through the cg_list 181 field of each task_struct using the css_set, anchored at 182 css_set->tasks. 183 184 - A cgroup hierarchy filesystem can be mounted for browsing and 185 manipulation from user space. 186 187 - You can list all the tasks (by pid) attached to any cgroup. 188 189 The implementation of cgroups requires a few, simple hooks 190 into the rest of the kernel, none in performance critical paths: 191 192 - in init/main.c, to initialize the root cgroups and initial 193 css_set at system boot. 194 195 - in fork and exit, to attach and detach a task from its css_set. 196 197 In addition a new file system, of type "cgroup" may be mounted, to 198 enable browsing and modifying the cgroups presently known to the 199 kernel. When mounting a cgroup hierarchy, you may specify a 200 comma-separated list of subsystems to mount as the filesystem mount 201 options. By default, mounting the cgroup filesystem attempts to 202 mount a hierarchy containing all registered subsystems. 203 204 If an active hierarchy with exactly the same set of subsystems already 205 exists, it will be reused for the new mount. If no existing hierarchy 206 matches, and any of the requested subsystems are in use in an existing 207 hierarchy, the mount will fail with -EBUSY. Otherwise, a new hierarchy 208 is activated, associated with the requested subsystems. 209 210 It's not currently possible to bind a new subsystem to an active 211 cgroup hierarchy, or to unbind a subsystem from an active cgroup 212 hierarchy. This may be possible in future, but is fraught with nasty 213 error-recovery issues. 214 215 When a cgroup filesystem is unmounted, if there are any 216 child cgroups created below the top-level cgroup, that hierarchy 217 will remain active even though unmounted; if there are no 218 child cgroups then the hierarchy will be deactivated. 219 220 No new system calls are added for cgroups - all support for 221 querying and modifying cgroups is via this cgroup file system. 222 223 Each task under /proc has an added file named 'cgroup' displaying, 224 for each active hierarchy, the subsystem names and the cgroup name 225 as the path relative to the root of the cgroup file system. 226 227 Each cgroup is represented by a directory in the cgroup file system 228 containing the following files describing that cgroup: 229 230 - tasks: list of tasks (by pid) attached to that cgroup. This list 231 is not guaranteed to be sorted. Writing a thread id into this file 232 moves the thread into this cgroup. 233 - cgroup.procs: list of tgids in the cgroup. This list is not 234 guaranteed to be sorted or free of duplicate tgids, and userspace 235 should sort/uniquify the list if this property is required. 236 Writing a tgid into this file moves all threads with that tgid into 237 this cgroup. 238 - notify_on_release flag: run the release agent on exit? 239 - release_agent: the path to use for release notifications (this file 240 exists in the top cgroup only) 241 242 Other subsystems such as cpusets may add additional files in each 243 cgroup dir. 244 245 New cgroups are created using the mkdir system call or shell 246 command. The properties of a cgroup, such as its flags, are 247 modified by writing to the appropriate file in that cgroups 248 directory, as listed above. 249 250 The named hierarchical structure of nested cgroups allows partitioning 251 a large system into nested, dynamically changeable, "soft-partitions". 252 253 The attachment of each task, automatically inherited at fork by any 254 children of that task, to a cgroup allows organizing the work load 255 on a system into related sets of tasks. A task may be re-attached to 256 any other cgroup, if allowed by the permissions on the necessary 257 cgroup file system directories. 258 259 When a task is moved from one cgroup to another, it gets a new 260 css_set pointer - if there's an already existing css_set with the 261 desired collection of cgroups then that group is reused, else a new 262 css_set is allocated. The appropriate existing css_set is located by 263 looking into a hash table. 264 265 To allow access from a cgroup to the css_sets (and hence tasks) 266 that comprise it, a set of cg_cgroup_link objects form a lattice; 267 each cg_cgroup_link is linked into a list of cg_cgroup_links for 268 a single cgroup on its cgrp_link_list field, and a list of 269 cg_cgroup_links for a single css_set on its cg_link_list. 270 271 Thus the set of tasks in a cgroup can be listed by iterating over 272 each css_set that references the cgroup, and sub-iterating over 273 each css_set's task set. 274 275 The use of a Linux virtual file system (vfs) to represent the 276 cgroup hierarchy provides for a familiar permission and name space 277 for cgroups, with a minimum of additional kernel code. 278 279 1.4 What does notify_on_release do ? 280 ------------------------------------ 281 282 If the notify_on_release flag is enabled (1) in a cgroup, then 283 whenever the last task in the cgroup leaves (exits or attaches to 284 some other cgroup) and the last child cgroup of that cgroup 285 is removed, then the kernel runs the command specified by the contents 286 of the "release_agent" file in that hierarchy's root directory, 287 supplying the pathname (relative to the mount point of the cgroup 288 file system) of the abandoned cgroup. This enables automatic 289 removal of abandoned cgroups. The default value of 290 notify_on_release in the root cgroup at system boot is disabled 291 (0). The default value of other cgroups at creation is the current 292 value of their parents notify_on_release setting. The default value of 293 a cgroup hierarchy's release_agent path is empty. 294 295 1.5 How do I use cgroups ? 296 -------------------------- 297 298 To start a new job that is to be contained within a cgroup, using 299 the "cpuset" cgroup subsystem, the steps are something like: 300 301 1) mkdir /dev/cgroup 302 2) mount -t cgroup -ocpuset cpuset /dev/cgroup 303 3) Create the new cgroup by doing mkdir's and write's (or echo's) in 304 the /dev/cgroup virtual file system. 305 4) Start a task that will be the "founding father" of the new job. 306 5) Attach that task to the new cgroup by writing its pid to the 307 /dev/cgroup tasks file for that cgroup. 308 6) fork, exec or clone the job tasks from this founding father task. 309 310 For example, the following sequence of commands will setup a cgroup 311 named "Charlie", containing just CPUs 2 and 3, and Memory Node 1, 312 and then start a subshell 'sh' in that cgroup: 313 314 mount -t cgroup cpuset -ocpuset /dev/cgroup 315 cd /dev/cgroup 316 mkdir Charlie 317 cd Charlie 318 /bin/echo 2-3 > cpuset.cpus 319 /bin/echo 1 > cpuset.mems 320 /bin/echo $$ > tasks 321 sh 322 # The subshell 'sh' is now running in cgroup Charlie 323 # The next line should display '/Charlie' 324 cat /proc/self/cgroup 325 326 2. Usage Examples and Syntax 327 ============================ 328 329 2.1 Basic Usage 330 --------------- 331 332 Creating, modifying, using the cgroups can be done through the cgroup 333 virtual filesystem. 334 335 To mount a cgroup hierarchy with all available subsystems, type: 336 # mount -t cgroup xxx /dev/cgroup 337 338 The "xxx" is not interpreted by the cgroup code, but will appear in 339 /proc/mounts so may be any useful identifying string that you like. 340 341 To mount a cgroup hierarchy with just the cpuset and numtasks 342 subsystems, type: 343 # mount -t cgroup -o cpuset,memory hier1 /dev/cgroup 344 345 To change the set of subsystems bound to a mounted hierarchy, just 346 remount with different options: 347 # mount -o remount,cpuset,ns hier1 /dev/cgroup 348 349 Now memory is removed from the hierarchy and ns is added. 350 351 Note this will add ns to the hierarchy but won't remove memory or 352 cpuset, because the new options are appended to the old ones: 353 # mount -o remount,ns /dev/cgroup 354 355 To Specify a hierarchy's release_agent: 356 # mount -t cgroup -o cpuset,release_agent="/sbin/cpuset_release_agent" \ 357 xxx /dev/cgroup 358 359 Note that specifying 'release_agent' more than once will return failure. 360 361 Note that changing the set of subsystems is currently only supported 362 when the hierarchy consists of a single (root) cgroup. Supporting 363 the ability to arbitrarily bind/unbind subsystems from an existing 364 cgroup hierarchy is intended to be implemented in the future. 365 366 Then under /dev/cgroup you can find a tree that corresponds to the 367 tree of the cgroups in the system. For instance, /dev/cgroup 368 is the cgroup that holds the whole system. 369 370 If you want to change the value of release_agent: 371 # echo "/sbin/new_release_agent" > /dev/cgroup/release_agent 372 373 It can also be changed via remount. 374 375 If you want to create a new cgroup under /dev/cgroup: 376 # cd /dev/cgroup 377 # mkdir my_cgroup 378 379 Now you want to do something with this cgroup. 380 # cd my_cgroup 381 382 In this directory you can find several files: 383 # ls 384 cgroup.procs notify_on_release tasks 385 (plus whatever files added by the attached subsystems) 386 387 Now attach your shell to this cgroup: 388 # /bin/echo $$ > tasks 389 390 You can also create cgroups inside your cgroup by using mkdir in this 391 directory. 392 # mkdir my_sub_cs 393 394 To remove a cgroup, just use rmdir: 395 # rmdir my_sub_cs 396 397 This will fail if the cgroup is in use (has cgroups inside, or 398 has processes attached, or is held alive by other subsystem-specific 399 reference). 400 401 2.2 Attaching processes 402 ----------------------- 403 404 # /bin/echo PID > tasks 405 406 Note that it is PID, not PIDs. You can only attach ONE task at a time. 407 If you have several tasks to attach, you have to do it one after another: 408 409 # /bin/echo PID1 > tasks 410 # /bin/echo PID2 > tasks 411 ... 412 # /bin/echo PIDn > tasks 413 414 You can attach the current shell task by echoing 0: 415 416 # echo 0 > tasks 417 418 2.3 Mounting hierarchies by name 419 -------------------------------- 420 421 Passing the name=<x> option when mounting a cgroups hierarchy 422 associates the given name with the hierarchy. This can be used when 423 mounting a pre-existing hierarchy, in order to refer to it by name 424 rather than by its set of active subsystems. Each hierarchy is either 425 nameless, or has a unique name. 426 427 The name should match [\w.-]+ 428 429 When passing a name=<x> option for a new hierarchy, you need to 430 specify subsystems manually; the legacy behaviour of mounting all 431 subsystems when none are explicitly specified is not supported when 432 you give a subsystem a name. 433 434 The name of the subsystem appears as part of the hierarchy description 435 in /proc/mounts and /proc/<pid>/cgroups. 436 437 438 3. Kernel API 439 ============= 440 441 3.1 Overview 442 ------------ 443 444 Each kernel subsystem that wants to hook into the generic cgroup 445 system needs to create a cgroup_subsys object. This contains 446 various methods, which are callbacks from the cgroup system, along 447 with a subsystem id which will be assigned by the cgroup system. 448 449 Other fields in the cgroup_subsys object include: 450 451 - subsys_id: a unique array index for the subsystem, indicating which 452 entry in cgroup->subsys[] this subsystem should be managing. 453 454 - name: should be initialized to a unique subsystem name. Should be 455 no longer than MAX_CGROUP_TYPE_NAMELEN. 456 457 - early_init: indicate if the subsystem needs early initialization 458 at system boot. 459 460 Each cgroup object created by the system has an array of pointers, 461 indexed by subsystem id; this pointer is entirely managed by the 462 subsystem; the generic cgroup code will never touch this pointer. 463 464 3.2 Synchronization 465 ------------------- 466 467 There is a global mutex, cgroup_mutex, used by the cgroup 468 system. This should be taken by anything that wants to modify a 469 cgroup. It may also be taken to prevent cgroups from being 470 modified, but more specific locks may be more appropriate in that 471 situation. 472 473 See kernel/cgroup.c for more details. 474 475 Subsystems can take/release the cgroup_mutex via the functions 476 cgroup_lock()/cgroup_unlock(). 477 478 Accessing a task's cgroup pointer may be done in the following ways: 479 - while holding cgroup_mutex 480 - while holding the task's alloc_lock (via task_lock()) 481 - inside an rcu_read_lock() section via rcu_dereference() 482 483 3.3 Subsystem API 484 ----------------- 485 486 Each subsystem should: 487 488 - add an entry in linux/cgroup_subsys.h 489 - define a cgroup_subsys object called <name>_subsys 490 491 Each subsystem may export the following methods. The only mandatory 492 methods are create/destroy. Any others that are null are presumed to 493 be successful no-ops. 494 495 struct cgroup_subsys_state *create(struct cgroup_subsys *ss, 496 struct cgroup *cgrp) 497 (cgroup_mutex held by caller) 498 499 Called to create a subsystem state object for a cgroup. The 500 subsystem should allocate its subsystem state object for the passed 501 cgroup, returning a pointer to the new object on success or a 502 negative error code. On success, the subsystem pointer should point to 503 a structure of type cgroup_subsys_state (typically embedded in a 504 larger subsystem-specific object), which will be initialized by the 505 cgroup system. Note that this will be called at initialization to 506 create the root subsystem state for this subsystem; this case can be 507 identified by the passed cgroup object having a NULL parent (since 508 it's the root of the hierarchy) and may be an appropriate place for 509 initialization code. 510 511 void destroy(struct cgroup_subsys *ss, struct cgroup *cgrp) 512 (cgroup_mutex held by caller) 513 514 The cgroup system is about to destroy the passed cgroup; the subsystem 515 should do any necessary cleanup and free its subsystem state 516 object. By the time this method is called, the cgroup has already been 517 unlinked from the file system and from the child list of its parent; 518 cgroup->parent is still valid. (Note - can also be called for a 519 newly-created cgroup if an error occurs after this subsystem's 520 create() method has been called for the new cgroup). 521 522 int pre_destroy(struct cgroup_subsys *ss, struct cgroup *cgrp); 523 524 Called before checking the reference count on each subsystem. This may 525 be useful for subsystems which have some extra references even if 526 there are not tasks in the cgroup. If pre_destroy() returns error code, 527 rmdir() will fail with it. From this behavior, pre_destroy() can be 528 called multiple times against a cgroup. 529 530 int can_attach(struct cgroup_subsys *ss, struct cgroup *cgrp, 531 struct task_struct *task, bool threadgroup) 532 (cgroup_mutex held by caller) 533 534 Called prior to moving a task into a cgroup; if the subsystem 535 returns an error, this will abort the attach operation. If a NULL 536 task is passed, then a successful result indicates that *any* 537 unspecified task can be moved into the cgroup. Note that this isn't 538 called on a fork. If this method returns 0 (success) then this should 539 remain valid while the caller holds cgroup_mutex. If threadgroup is 540 true, then a successful result indicates that all threads in the given 541 thread's threadgroup can be moved together. 542 543 void attach(struct cgroup_subsys *ss, struct cgroup *cgrp, 544 struct cgroup *old_cgrp, struct task_struct *task, 545 bool threadgroup) 546 (cgroup_mutex held by caller) 547 548 Called after the task has been attached to the cgroup, to allow any 549 post-attachment activity that requires memory allocations or blocking. 550 If threadgroup is true, the subsystem should take care of all threads 551 in the specified thread's threadgroup. Currently does not support any 552 subsystem that might need the old_cgrp for every thread in the group. 553 554 void fork(struct cgroup_subsy *ss, struct task_struct *task) 555 556 Called when a task is forked into a cgroup. 557 558 void exit(struct cgroup_subsys *ss, struct task_struct *task) 559 560 Called during task exit. 561 562 int populate(struct cgroup_subsys *ss, struct cgroup *cgrp) 563 (cgroup_mutex held by caller) 564 565 Called after creation of a cgroup to allow a subsystem to populate 566 the cgroup directory with file entries. The subsystem should make 567 calls to cgroup_add_file() with objects of type cftype (see 568 include/linux/cgroup.h for details). Note that although this 569 method can return an error code, the error code is currently not 570 always handled well. 571 572 void post_clone(struct cgroup_subsys *ss, struct cgroup *cgrp) 573 (cgroup_mutex held by caller) 574 575 Called at the end of cgroup_clone() to do any parameter 576 initialization which might be required before a task could attach. For 577 example in cpusets, no task may attach before 'cpus' and 'mems' are set 578 up. 579 580 void bind(struct cgroup_subsys *ss, struct cgroup *root) 581 (cgroup_mutex and ss->hierarchy_mutex held by caller) 582 583 Called when a cgroup subsystem is rebound to a different hierarchy 584 and root cgroup. Currently this will only involve movement between 585 the default hierarchy (which never has sub-cgroups) and a hierarchy 586 that is being created/destroyed (and hence has no sub-cgroups). 587 588 4. Questions 589 ============ 590 591 Q: what's up with this '/bin/echo' ? 592 A: bash's builtin 'echo' command does not check calls to write() against 593 errors. If you use it in the cgroup file system, you won't be 594 able to tell whether a command succeeded or failed. 595 596 Q: When I attach processes, only the first of the line gets really attached ! 597 A: We can only return one error code per call to write(). So you should also 598 put only ONE pid.