About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / virtual / kvm / mmu.txt

Custom Search

Based on kernel version 4.13.3. Page generated on 2017-09-23 13:56 EST.

1	The x86 kvm shadow mmu
2	======================
4	The mmu (in arch/x86/kvm, files mmu.[ch] and paging_tmpl.h) is responsible
5	for presenting a standard x86 mmu to the guest, while translating guest
6	physical addresses to host physical addresses.
8	The mmu code attempts to satisfy the following requirements:
10	- correctness: the guest should not be able to determine that it is running
11	               on an emulated mmu except for timing (we attempt to comply
12	               with the specification, not emulate the characteristics of
13	               a particular implementation such as tlb size)
14	- security:    the guest must not be able to touch host memory not assigned
15	               to it
16	- performance: minimize the performance penalty imposed by the mmu
17	- scaling:     need to scale to large memory and large vcpu guests
18	- hardware:    support the full range of x86 virtualization hardware
19	- integration: Linux memory management code must be in control of guest memory
20	               so that swapping, page migration, page merging, transparent
21	               hugepages, and similar features work without change
22	- dirty tracking: report writes to guest memory to enable live migration
23	               and framebuffer-based displays
24	- footprint:   keep the amount of pinned kernel memory low (most memory
25	               should be shrinkable)
26	- reliability:  avoid multipage or GFP_ATOMIC allocations
28	Acronyms
29	========
31	pfn   host page frame number
32	hpa   host physical address
33	hva   host virtual address
34	gfn   guest frame number
35	gpa   guest physical address
36	gva   guest virtual address
37	ngpa  nested guest physical address
38	ngva  nested guest virtual address
39	pte   page table entry (used also to refer generically to paging structure
40	      entries)
41	gpte  guest pte (referring to gfns)
42	spte  shadow pte (referring to pfns)
43	tdp   two dimensional paging (vendor neutral term for NPT and EPT)
45	Virtual and real hardware supported
46	===================================
48	The mmu supports first-generation mmu hardware, which allows an atomic switch
49	of the current paging mode and cr3 during guest entry, as well as
50	two-dimensional paging (AMD's NPT and Intel's EPT).  The emulated hardware
51	it exposes is the traditional 2/3/4 level x86 mmu, with support for global
52	pages, pae, pse, pse36, cr0.wp, and 1GB pages.  Work is in progress to support
53	exposing NPT capable hardware on NPT capable hosts.
55	Translation
56	===========
58	The primary job of the mmu is to program the processor's mmu to translate
59	addresses for the guest.  Different translations are required at different
60	times:
62	- when guest paging is disabled, we translate guest physical addresses to
63	  host physical addresses (gpa->hpa)
64	- when guest paging is enabled, we translate guest virtual addresses, to
65	  guest physical addresses, to host physical addresses (gva->gpa->hpa)
66	- when the guest launches a guest of its own, we translate nested guest
67	  virtual addresses, to nested guest physical addresses, to guest physical
68	  addresses, to host physical addresses (ngva->ngpa->gpa->hpa)
70	The primary challenge is to encode between 1 and 3 translations into hardware
71	that support only 1 (traditional) and 2 (tdp) translations.  When the
72	number of required translations matches the hardware, the mmu operates in
73	direct mode; otherwise it operates in shadow mode (see below).
75	Memory
76	======
78	Guest memory (gpa) is part of the user address space of the process that is
79	using kvm.  Userspace defines the translation between guest addresses and user
80	addresses (gpa->hva); note that two gpas may alias to the same hva, but not
81	vice versa.
83	These hvas may be backed using any method available to the host: anonymous
84	memory, file backed memory, and device memory.  Memory might be paged by the
85	host at any time.
87	Events
88	======
90	The mmu is driven by events, some from the guest, some from the host.
92	Guest generated events:
93	- writes to control registers (especially cr3)
94	- invlpg/invlpga instruction execution
95	- access to missing or protected translations
97	Host generated events:
98	- changes in the gpa->hpa translation (either through gpa->hva changes or
99	  through hva->hpa changes)
100	- memory pressure (the shrinker)
102	Shadow pages
103	============
105	The principal data structure is the shadow page, 'struct kvm_mmu_page'.  A
106	shadow page contains 512 sptes, which can be either leaf or nonleaf sptes.  A
107	shadow page may contain a mix of leaf and nonleaf sptes.
109	A nonleaf spte allows the hardware mmu to reach the leaf pages and
110	is not related to a translation directly.  It points to other shadow pages.
112	A leaf spte corresponds to either one or two translations encoded into
113	one paging structure entry.  These are always the lowest level of the
114	translation stack, with optional higher level translations left to NPT/EPT.
115	Leaf ptes point at guest pages.
117	The following table shows translations encoded by leaf ptes, with higher-level
118	translations in parentheses:
120	 Non-nested guests:
121	  nonpaging:     gpa->hpa
122	  paging:        gva->gpa->hpa
123	  paging, tdp:   (gva->)gpa->hpa
124	 Nested guests:
125	  non-tdp:       ngva->gpa->hpa  (*)
126	  tdp:           (ngva->)ngpa->gpa->hpa
128	(*) the guest hypervisor will encode the ngva->gpa translation into its page
129	    tables if npt is not present
131	Shadow pages contain the following information:
132	  role.level:
133	    The level in the shadow paging hierarchy that this shadow page belongs to.
134	    1=4k sptes, 2=2M sptes, 3=1G sptes, etc.
135	  role.direct:
136	    If set, leaf sptes reachable from this page are for a linear range.
137	    Examples include real mode translation, large guest pages backed by small
138	    host pages, and gpa->hpa translations when NPT or EPT is active.
139	    The linear range starts at (gfn << PAGE_SHIFT) and its size is determined
140	    by role.level (2MB for first level, 1GB for second level, 0.5TB for third
141	    level, 256TB for fourth level)
142	    If clear, this page corresponds to a guest page table denoted by the gfn
143	    field.
144	  role.quadrant:
145	    When role.cr4_pae=0, the guest uses 32-bit gptes while the host uses 64-bit
146	    sptes.  That means a guest page table contains more ptes than the host,
147	    so multiple shadow pages are needed to shadow one guest page.
148	    For first-level shadow pages, role.quadrant can be 0 or 1 and denotes the
149	    first or second 512-gpte block in the guest page table.  For second-level
150	    page tables, each 32-bit gpte is converted to two 64-bit sptes
151	    (since each first-level guest page is shadowed by two first-level
152	    shadow pages) so role.quadrant takes values in the range 0..3.  Each
153	    quadrant maps 1GB virtual address space.
154	  role.access:
155	    Inherited guest access permissions in the form uwx.  Note execute
156	    permission is positive, not negative.
157	  role.invalid:
158	    The page is invalid and should not be used.  It is a root page that is
159	    currently pinned (by a cpu hardware register pointing to it); once it is
160	    unpinned it will be destroyed.
161	  role.cr4_pae:
162	    Contains the value of cr4.pae for which the page is valid (e.g. whether
163	    32-bit or 64-bit gptes are in use).
164	  role.nxe:
165	    Contains the value of efer.nxe for which the page is valid.
166	  role.cr0_wp:
167	    Contains the value of cr0.wp for which the page is valid.
168	  role.smep_andnot_wp:
169	    Contains the value of cr4.smep && !cr0.wp for which the page is valid
170	    (pages for which this is true are different from other pages; see the
171	    treatment of cr0.wp=0 below).
172	  role.smap_andnot_wp:
173	    Contains the value of cr4.smap && !cr0.wp for which the page is valid
174	    (pages for which this is true are different from other pages; see the
175	    treatment of cr0.wp=0 below).
176	  role.smm:
177	    Is 1 if the page is valid in system management mode.  This field
178	    determines which of the kvm_memslots array was used to build this
179	    shadow page; it is also used to go back from a struct kvm_mmu_page
180	    to a memslot, through the kvm_memslots_for_spte_role macro and
181	    __gfn_to_memslot.
182	  role.ad_disabled:
183	    Is 1 if the MMU instance cannot use A/D bits.  EPT did not have A/D
184	    bits before Haswell; shadow EPT page tables also cannot use A/D bits
185	    if the L1 hypervisor does not enable them.
186	  gfn:
187	    Either the guest page table containing the translations shadowed by this
188	    page, or the base page frame for linear translations.  See role.direct.
189	  spt:
190	    A pageful of 64-bit sptes containing the translations for this page.
191	    Accessed by both kvm and hardware.
192	    The page pointed to by spt will have its page->private pointing back
193	    at the shadow page structure.
194	    sptes in spt point either at guest pages, or at lower-level shadow pages.
195	    Specifically, if sp1 and sp2 are shadow pages, then sp1->spt[n] may point
196	    at __pa(sp2->spt).  sp2 will point back at sp1 through parent_pte.
197	    The spt array forms a DAG structure with the shadow page as a node, and
198	    guest pages as leaves.
199	  gfns:
200	    An array of 512 guest frame numbers, one for each present pte.  Used to
201	    perform a reverse map from a pte to a gfn. When role.direct is set, any
202	    element of this array can be calculated from the gfn field when used, in
203	    this case, the array of gfns is not allocated. See role.direct and gfn.
204	  root_count:
205	    A counter keeping track of how many hardware registers (guest cr3 or
206	    pdptrs) are now pointing at the page.  While this counter is nonzero, the
207	    page cannot be destroyed.  See role.invalid.
208	  parent_ptes:
209	    The reverse mapping for the pte/ptes pointing at this page's spt. If
210	    parent_ptes bit 0 is zero, only one spte points at this page and
211	    parent_ptes points at this single spte, otherwise, there exists multiple
212	    sptes pointing at this page and (parent_ptes & ~0x1) points at a data
213	    structure with a list of parent sptes.
214	  unsync:
215	    If true, then the translations in this page may not match the guest's
216	    translation.  This is equivalent to the state of the tlb when a pte is
217	    changed but before the tlb entry is flushed.  Accordingly, unsync ptes
218	    are synchronized when the guest executes invlpg or flushes its tlb by
219	    other means.  Valid for leaf pages.
220	  unsync_children:
221	    How many sptes in the page point at pages that are unsync (or have
222	    unsynchronized children).
223	  unsync_child_bitmap:
224	    A bitmap indicating which sptes in spt point (directly or indirectly) at
225	    pages that may be unsynchronized.  Used to quickly locate all unsychronized
226	    pages reachable from a given page.
227	  mmu_valid_gen:
228	    Generation number of the page.  It is compared with kvm->arch.mmu_valid_gen
229	    during hash table lookup, and used to skip invalidated shadow pages (see
230	    "Zapping all pages" below.)
231	  clear_spte_count:
232	    Only present on 32-bit hosts, where a 64-bit spte cannot be written
233	    atomically.  The reader uses this while running out of the MMU lock
234	    to detect in-progress updates and retry them until the writer has
235	    finished the write.
236	  write_flooding_count:
237	    A guest may write to a page table many times, causing a lot of
238	    emulations if the page needs to be write-protected (see "Synchronized
239	    and unsynchronized pages" below).  Leaf pages can be unsynchronized
240	    so that they do not trigger frequent emulation, but this is not
241	    possible for non-leafs.  This field counts the number of emulations
242	    since the last time the page table was actually used; if emulation
243	    is triggered too frequently on this page, KVM will unmap the page
244	    to avoid emulation in the future.
246	Reverse map
247	===========
249	The mmu maintains a reverse mapping whereby all ptes mapping a page can be
250	reached given its gfn.  This is used, for example, when swapping out a page.
252	Synchronized and unsynchronized pages
253	=====================================
255	The guest uses two events to synchronize its tlb and page tables: tlb flushes
256	and page invalidations (invlpg).
258	A tlb flush means that we need to synchronize all sptes reachable from the
259	guest's cr3.  This is expensive, so we keep all guest page tables write
260	protected, and synchronize sptes to gptes when a gpte is written.
262	A special case is when a guest page table is reachable from the current
263	guest cr3.  In this case, the guest is obliged to issue an invlpg instruction
264	before using the translation.  We take advantage of that by removing write
265	protection from the guest page, and allowing the guest to modify it freely.
266	We synchronize modified gptes when the guest invokes invlpg.  This reduces
267	the amount of emulation we have to do when the guest modifies multiple gptes,
268	or when the a guest page is no longer used as a page table and is used for
269	random guest data.
271	As a side effect we have to resynchronize all reachable unsynchronized shadow
272	pages on a tlb flush.
275	Reaction to events
276	==================
278	- guest page fault (or npt page fault, or ept violation)
280	This is the most complicated event.  The cause of a page fault can be:
282	  - a true guest fault (the guest translation won't allow the access) (*)
283	  - access to a missing translation
284	  - access to a protected translation
285	    - when logging dirty pages, memory is write protected
286	    - synchronized shadow pages are write protected (*)
287	  - access to untranslatable memory (mmio)
289	  (*) not applicable in direct mode
291	Handling a page fault is performed as follows:
293	 - if the RSV bit of the error code is set, the page fault is caused by guest
294	   accessing MMIO and cached MMIO information is available.
295	   - walk shadow page table
296	   - check for valid generation number in the spte (see "Fast invalidation of
297	     MMIO sptes" below)
298	   - cache the information to vcpu->arch.mmio_gva, vcpu->arch.access and
299	     vcpu->arch.mmio_gfn, and call the emulator
300	 - If both P bit and R/W bit of error code are set, this could possibly
301	   be handled as a "fast page fault" (fixed without taking the MMU lock).  See
302	   the description in Documentation/virtual/kvm/locking.txt.
303	 - if needed, walk the guest page tables to determine the guest translation
304	   (gva->gpa or ngpa->gpa)
305	   - if permissions are insufficient, reflect the fault back to the guest
306	 - determine the host page
307	   - if this is an mmio request, there is no host page; cache the info to
308	     vcpu->arch.mmio_gva, vcpu->arch.access and vcpu->arch.mmio_gfn
309	 - walk the shadow page table to find the spte for the translation,
310	   instantiating missing intermediate page tables as necessary
311	   - If this is an mmio request, cache the mmio info to the spte and set some
312	     reserved bit on the spte (see callers of kvm_mmu_set_mmio_spte_mask)
313	 - try to unsynchronize the page
314	   - if successful, we can let the guest continue and modify the gpte
315	 - emulate the instruction
316	   - if failed, unshadow the page and let the guest continue
317	 - update any translations that were modified by the instruction
319	invlpg handling:
321	  - walk the shadow page hierarchy and drop affected translations
322	  - try to reinstantiate the indicated translation in the hope that the
323	    guest will use it in the near future
325	Guest control register updates:
327	- mov to cr3
328	  - look up new shadow roots
329	  - synchronize newly reachable shadow pages
331	- mov to cr0/cr4/efer
332	  - set up mmu context for new paging mode
333	  - look up new shadow roots
334	  - synchronize newly reachable shadow pages
336	Host translation updates:
338	  - mmu notifier called with updated hva
339	  - look up affected sptes through reverse map
340	  - drop (or update) translations
342	Emulating cr0.wp
343	================
345	If tdp is not enabled, the host must keep cr0.wp=1 so page write protection
346	works for the guest kernel, not guest guest userspace.  When the guest
347	cr0.wp=1, this does not present a problem.  However when the guest cr0.wp=0,
348	we cannot map the permissions for gpte.u=1, gpte.w=0 to any spte (the
349	semantics require allowing any guest kernel access plus user read access).
351	We handle this by mapping the permissions to two possible sptes, depending
352	on fault type:
354	- kernel write fault: spte.u=0, spte.w=1 (allows full kernel access,
355	  disallows user access)
356	- read fault: spte.u=1, spte.w=0 (allows full read access, disallows kernel
357	  write access)
359	(user write faults generate a #PF)
361	In the first case there are two additional complications:
362	- if CR4.SMEP is enabled: since we've turned the page into a kernel page,
363	  the kernel may now execute it.  We handle this by also setting spte.nx.
364	  If we get a user fetch or read fault, we'll change spte.u=1 and
365	  spte.nx=gpte.nx back.  For this to work, KVM forces EFER.NX to 1 when
366	  shadow paging is in use.
367	- if CR4.SMAP is disabled: since the page has been changed to a kernel
368	  page, it can not be reused when CR4.SMAP is enabled. We set
369	  CR4.SMAP && !CR0.WP into shadow page's role to avoid this case. Note,
370	  here we do not care the case that CR4.SMAP is enabled since KVM will
371	  directly inject #PF to guest due to failed permission check.
373	To prevent an spte that was converted into a kernel page with cr0.wp=0
374	from being written by the kernel after cr0.wp has changed to 1, we make
375	the value of cr0.wp part of the page role.  This means that an spte created
376	with one value of cr0.wp cannot be used when cr0.wp has a different value -
377	it will simply be missed by the shadow page lookup code.  A similar issue
378	exists when an spte created with cr0.wp=0 and cr4.smep=0 is used after
379	changing cr4.smep to 1.  To avoid this, the value of !cr0.wp && cr4.smep
380	is also made a part of the page role.
382	Large pages
383	===========
385	The mmu supports all combinations of large and small guest and host pages.
386	Supported page sizes include 4k, 2M, 4M, and 1G.  4M pages are treated as
387	two separate 2M pages, on both guest and host, since the mmu always uses PAE
388	paging.
390	To instantiate a large spte, four constraints must be satisfied:
392	- the spte must point to a large host page
393	- the guest pte must be a large pte of at least equivalent size (if tdp is
394	  enabled, there is no guest pte and this condition is satisfied)
395	- if the spte will be writeable, the large page frame may not overlap any
396	  write-protected pages
397	- the guest page must be wholly contained by a single memory slot
399	To check the last two conditions, the mmu maintains a ->disallow_lpage set of
400	arrays for each memory slot and large page size.  Every write protected page
401	causes its disallow_lpage to be incremented, thus preventing instantiation of
402	a large spte.  The frames at the end of an unaligned memory slot have
403	artificially inflated ->disallow_lpages so they can never be instantiated.
405	Zapping all pages (page generation count)
406	=========================================
408	For the large memory guests, walking and zapping all pages is really slow
409	(because there are a lot of pages), and also blocks memory accesses of
410	all VCPUs because it needs to hold the MMU lock.
412	To make it be more scalable, kvm maintains a global generation number
413	which is stored in kvm->arch.mmu_valid_gen.  Every shadow page stores
414	the current global generation-number into sp->mmu_valid_gen when it
415	is created.  Pages with a mismatching generation number are "obsolete".
417	When KVM need zap all shadow pages sptes, it just simply increases the global
418	generation-number then reload root shadow pages on all vcpus.  As the VCPUs
419	create new shadow page tables, the old pages are not used because of the
420	mismatching generation number.
422	KVM then walks through all pages and zaps obsolete pages.  While the zap
423	operation needs to take the MMU lock, the lock can be released periodically
424	so that the VCPUs can make progress.
426	Fast invalidation of MMIO sptes
427	===============================
429	As mentioned in "Reaction to events" above, kvm will cache MMIO
430	information in leaf sptes.  When a new memslot is added or an existing
431	memslot is changed, this information may become stale and needs to be
432	invalidated.  This also needs to hold the MMU lock while walking all
433	shadow pages, and is made more scalable with a similar technique.
435	MMIO sptes have a few spare bits, which are used to store a
436	generation number.  The global generation number is stored in
437	kvm_memslots(kvm)->generation, and increased whenever guest memory info
438	changes.  This generation number is distinct from the one described in
439	the previous section.
441	When KVM finds an MMIO spte, it checks the generation number of the spte.
442	If the generation number of the spte does not equal the global generation
443	number, it will ignore the cached MMIO information and handle the page
444	fault through the slow path.
446	Since only 19 bits are used to store generation-number on mmio spte, all
447	pages are zapped when there is an overflow.
449	Unfortunately, a single memory access might access kvm_memslots(kvm) multiple
450	times, the last one happening when the generation number is retrieved and
451	stored into the MMIO spte.  Thus, the MMIO spte might be created based on
452	out-of-date information, but with an up-to-date generation number.
454	To avoid this, the generation number is incremented again after synchronize_srcu
455	returns; thus, the low bit of kvm_memslots(kvm)->generation is only 1 during a
456	memslot update, while some SRCU readers might be using the old copy.  We do not
457	want to use an MMIO sptes created with an odd generation number, and we can do
458	this without losing a bit in the MMIO spte.  The low bit of the generation
459	is not stored in MMIO spte, and presumed zero when it is extracted out of the
460	spte.  If KVM is unlucky and creates an MMIO spte while the low bit is 1,
461	the next access to the spte will always be a cache miss.
464	Further reading
465	===============
467	- NPT presentation from KVM Forum 2008
468	  http://www.linux-kvm.org/wiki/images/c/c8/KvmForum2008%24kdf2008_21.pdf
Hide Line Numbers
About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Information is copyright its respective author. All material is available from the Linux Kernel Source distributed under a GPL License. This page is provided as a free service by mjmwired.net.