About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / power / freezing-of-tasks.txt


Based on kernel version 4.16.1. Page generated on 2018-04-09 11:53 EST.

1	Freezing of tasks
2		(C) 2007 Rafael J. Wysocki <rjw@sisk.pl>, GPL
3	
4	I. What is the freezing of tasks?
5	
6	The freezing of tasks is a mechanism by which user space processes and some
7	kernel threads are controlled during hibernation or system-wide suspend (on some
8	architectures).
9	
10	II. How does it work?
11	
12	There are three per-task flags used for that, PF_NOFREEZE, PF_FROZEN
13	and PF_FREEZER_SKIP (the last one is auxiliary).  The tasks that have
14	PF_NOFREEZE unset (all user space processes and some kernel threads) are
15	regarded as 'freezable' and treated in a special way before the system enters a
16	suspend state as well as before a hibernation image is created (in what follows
17	we only consider hibernation, but the description also applies to suspend).
18	
19	Namely, as the first step of the hibernation procedure the function
20	freeze_processes() (defined in kernel/power/process.c) is called.  A system-wide
21	variable system_freezing_cnt (as opposed to a per-task flag) is used to indicate
22	whether the system is to undergo a freezing operation. And freeze_processes()
23	sets this variable.  After this, it executes try_to_freeze_tasks() that sends a
24	fake signal to all user space processes, and wakes up all the kernel threads.
25	All freezable tasks must react to that by calling try_to_freeze(), which
26	results in a call to __refrigerator() (defined in kernel/freezer.c), which sets
27	the task's PF_FROZEN flag, changes its state to TASK_UNINTERRUPTIBLE and makes
28	it loop until PF_FROZEN is cleared for it. Then, we say that the task is
29	'frozen' and therefore the set of functions handling this mechanism is referred
30	to as 'the freezer' (these functions are defined in kernel/power/process.c,
31	kernel/freezer.c & include/linux/freezer.h). User space processes are generally
32	frozen before kernel threads.
33	
34	__refrigerator() must not be called directly.  Instead, use the
35	try_to_freeze() function (defined in include/linux/freezer.h), that checks
36	if the task is to be frozen and makes the task enter __refrigerator().
37	
38	For user space processes try_to_freeze() is called automatically from the
39	signal-handling code, but the freezable kernel threads need to call it
40	explicitly in suitable places or use the wait_event_freezable() or
41	wait_event_freezable_timeout() macros (defined in include/linux/freezer.h)
42	that combine interruptible sleep with checking if the task is to be frozen and
43	calling try_to_freeze().  The main loop of a freezable kernel thread may look
44	like the following one:
45	
46		set_freezable();
47		do {
48			hub_events();
49			wait_event_freezable(khubd_wait,
50					!list_empty(&hub_event_list) ||
51					kthread_should_stop());
52		} while (!kthread_should_stop() || !list_empty(&hub_event_list));
53	
54	(from drivers/usb/core/hub.c::hub_thread()).
55	
56	If a freezable kernel thread fails to call try_to_freeze() after the freezer has
57	initiated a freezing operation, the freezing of tasks will fail and the entire
58	hibernation operation will be cancelled.  For this reason, freezable kernel
59	threads must call try_to_freeze() somewhere or use one of the
60	wait_event_freezable() and wait_event_freezable_timeout() macros.
61	
62	After the system memory state has been restored from a hibernation image and
63	devices have been reinitialized, the function thaw_processes() is called in
64	order to clear the PF_FROZEN flag for each frozen task.  Then, the tasks that
65	have been frozen leave __refrigerator() and continue running.
66	
67	
68	Rationale behind the functions dealing with freezing and thawing of tasks:
69	-------------------------------------------------------------------------
70	
71	freeze_processes():
72	  - freezes only userspace tasks
73	
74	freeze_kernel_threads():
75	  - freezes all tasks (including kernel threads) because we can't freeze
76	    kernel threads without freezing userspace tasks
77	
78	thaw_kernel_threads():
79	  - thaws only kernel threads; this is particularly useful if we need to do
80	    anything special in between thawing of kernel threads and thawing of
81	    userspace tasks, or if we want to postpone the thawing of userspace tasks
82	
83	thaw_processes():
84	  - thaws all tasks (including kernel threads) because we can't thaw userspace
85	    tasks without thawing kernel threads
86	
87	
88	III. Which kernel threads are freezable?
89	
90	Kernel threads are not freezable by default.  However, a kernel thread may clear
91	PF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZE
92	directly is not allowed).  From this point it is regarded as freezable
93	and must call try_to_freeze() in a suitable place.
94	
95	IV. Why do we do that?
96	
97	Generally speaking, there is a couple of reasons to use the freezing of tasks:
98	
99	1. The principal reason is to prevent filesystems from being damaged after
100	hibernation.  At the moment we have no simple means of checkpointing
101	filesystems, so if there are any modifications made to filesystem data and/or
102	metadata on disks, we cannot bring them back to the state from before the
103	modifications.  At the same time each hibernation image contains some
104	filesystem-related information that must be consistent with the state of the
105	on-disk data and metadata after the system memory state has been restored from
106	the image (otherwise the filesystems will be damaged in a nasty way, usually
107	making them almost impossible to repair).  We therefore freeze tasks that might
108	cause the on-disk filesystems' data and metadata to be modified after the
109	hibernation image has been created and before the system is finally powered off.
110	The majority of these are user space processes, but if any of the kernel threads
111	may cause something like this to happen, they have to be freezable.
112	
113	2. Next, to create the hibernation image we need to free a sufficient amount of
114	memory (approximately 50% of available RAM) and we need to do that before
115	devices are deactivated, because we generally need them for swapping out.  Then,
116	after the memory for the image has been freed, we don't want tasks to allocate
117	additional memory and we prevent them from doing that by freezing them earlier.
118	[Of course, this also means that device drivers should not allocate substantial
119	amounts of memory from their .suspend() callbacks before hibernation, but this
120	is a separate issue.]
121	
122	3. The third reason is to prevent user space processes and some kernel threads
123	from interfering with the suspending and resuming of devices.  A user space
124	process running on a second CPU while we are suspending devices may, for
125	example, be troublesome and without the freezing of tasks we would need some
126	safeguards against race conditions that might occur in such a case.
127	
128	Although Linus Torvalds doesn't like the freezing of tasks, he said this in one
129	of the discussions on LKML (http://lkml.org/lkml/2007/4/27/608):
130	
131	"RJW:> Why we freeze tasks at all or why we freeze kernel threads?
132	
133	Linus: In many ways, 'at all'.
134	
135	I _do_ realize the IO request queue issues, and that we cannot actually do
136	s2ram with some devices in the middle of a DMA.  So we want to be able to
137	avoid *that*, there's no question about that.  And I suspect that stopping
138	user threads and then waiting for a sync is practically one of the easier
139	ways to do so.
140	
141	So in practice, the 'at all' may become a 'why freeze kernel threads?' and
142	freezing user threads I don't find really objectionable."
143	
144	Still, there are kernel threads that may want to be freezable.  For example, if
145	a kernel thread that belongs to a device driver accesses the device directly, it
146	in principle needs to know when the device is suspended, so that it doesn't try
147	to access it at that time.  However, if the kernel thread is freezable, it will
148	be frozen before the driver's .suspend() callback is executed and it will be
149	thawed after the driver's .resume() callback has run, so it won't be accessing
150	the device while it's suspended.
151	
152	4. Another reason for freezing tasks is to prevent user space processes from
153	realizing that hibernation (or suspend) operation takes place.  Ideally, user
154	space processes should not notice that such a system-wide operation has occurred
155	and should continue running without any problems after the restore (or resume
156	from suspend).  Unfortunately, in the most general case this is quite difficult
157	to achieve without the freezing of tasks.  Consider, for example, a process
158	that depends on all CPUs being online while it's running.  Since we need to
159	disable nonboot CPUs during the hibernation, if this process is not frozen, it
160	may notice that the number of CPUs has changed and may start to work incorrectly
161	because of that.
162	
163	V. Are there any problems related to the freezing of tasks?
164	
165	Yes, there are.
166	
167	First of all, the freezing of kernel threads may be tricky if they depend one
168	on another.  For example, if kernel thread A waits for a completion (in the
169	TASK_UNINTERRUPTIBLE state) that needs to be done by freezable kernel thread B
170	and B is frozen in the meantime, then A will be blocked until B is thawed, which
171	may be undesirable.  That's why kernel threads are not freezable by default.
172	
173	Second, there are the following two problems related to the freezing of user
174	space processes:
175	1. Putting processes into an uninterruptible sleep distorts the load average.
176	2. Now that we have FUSE, plus the framework for doing device drivers in
177	userspace, it gets even more complicated because some userspace processes are
178	now doing the sorts of things that kernel threads do
179	(https://lists.linux-foundation.org/pipermail/linux-pm/2007-May/012309.html).
180	
181	The problem 1. seems to be fixable, although it hasn't been fixed so far.  The
182	other one is more serious, but it seems that we can work around it by using
183	hibernation (and suspend) notifiers (in that case, though, we won't be able to
184	avoid the realization by the user space processes that the hibernation is taking
185	place).
186	
187	There are also problems that the freezing of tasks tends to expose, although
188	they are not directly related to it.  For example, if request_firmware() is
189	called from a device driver's .resume() routine, it will timeout and eventually
190	fail, because the user land process that should respond to the request is frozen
191	at this point.  So, seemingly, the failure is due to the freezing of tasks.
192	Suppose, however, that the firmware file is located on a filesystem accessible
193	only through another device that hasn't been resumed yet.  In that case,
194	request_firmware() will fail regardless of whether or not the freezing of tasks
195	is used.  Consequently, the problem is not really related to the freezing of
196	tasks, since it generally exists anyway.
197	
198	A driver must have all firmwares it may need in RAM before suspend() is called.
199	If keeping them is not practical, for example due to their size, they must be
200	requested early enough using the suspend notifier API described in
201	Documentation/driver-api/pm/notifiers.rst.
202	
203	VI. Are there any precautions to be taken to prevent freezing failures?
204	
205	Yes, there are.
206	
207	First of all, grabbing the 'pm_mutex' lock to mutually exclude a piece of code
208	from system-wide sleep such as suspend/hibernation is not encouraged.
209	If possible, that piece of code must instead hook onto the suspend/hibernation
210	notifiers to achieve mutual exclusion. Look at the CPU-Hotplug code
211	(kernel/cpu.c) for an example.
212	
213	However, if that is not feasible, and grabbing 'pm_mutex' is deemed necessary,
214	it is strongly discouraged to directly call mutex_[un]lock(&pm_mutex) since
215	that could lead to freezing failures, because if the suspend/hibernate code
216	successfully acquired the 'pm_mutex' lock, and hence that other entity failed
217	to acquire the lock, then that task would get blocked in TASK_UNINTERRUPTIBLE
218	state. As a consequence, the freezer would not be able to freeze that task,
219	leading to freezing failure.
220	
221	However, the [un]lock_system_sleep() APIs are safe to use in this scenario,
222	since they ask the freezer to skip freezing this task, since it is anyway
223	"frozen enough" as it is blocked on 'pm_mutex', which will be released
224	only after the entire suspend/hibernation sequence is complete.
225	So, to summarize, use [un]lock_system_sleep() instead of directly using
226	mutex_[un]lock(&pm_mutex). That would prevent freezing failures.
227	
228	V. Miscellaneous
229	/sys/power/pm_freeze_timeout controls how long it will cost at most to freeze
230	all user space processes or all freezable kernel threads, in unit of millisecond.
231	The default value is 20000, with range of unsigned integer.
Hide Line Numbers


About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog