Intel Virtualization Technology

Roadmap and
VT-d Support in Xen

Jun Nakajima
Intel Open Source Technology Center
 Legal Disclaimer
y INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO
LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL
PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS
AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER,
AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF
INTEL® PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A
PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR
OTHER INTELLECTUAL PROPERTY RIGHT. INTEL PRODUCTS ARE NOT INTENDED FOR USE IN
MEDICAL, LIFE SAVING, OR LIFE SUSTAINING APPLICATIONS.
y Intel may make changes to specifications and product descriptions at any time, without notice.
y All products, dates, and figures specified are preliminary based on current expectations, and are subject to
change without notice.
y Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which
may cause the product to deviate from published specifications. Current characterized errata are available on
request.
y Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the
United States and other countries.
y *Other names and brands may be claimed as the property of others.
y Copyright © 2007 Intel Corporation.
Throughout this presentation:
VT-x refers to Intel® VT for IA-32 and Intel® 64
VT-i refers to the Intel® VT for IA-64, and
VT-d refers to Intel® VT for Directed I/O
2
 Intel® VT Roadmap: Overview
Standards for I/O-device sharing:
Vector 3: PCI-SIG
• Natively sharable I/O devices
• Endpoint DMA-translation caching
I/O Focus

Infrastructure for I/O-device virtualization:

Vector 2: • DMA protection and remapping
VT-d • Interrupt filtering and remapping
Platform Focus

Establish foundation … followed by on going evolution of support:

Vector 1: VT-x for virtualization in the • Microarchitectural (e.g., lower VM entry/exit costs)
Intel® 64 and Itanium® • Architectural (e.g., extended page tables – EPT)
Processor Focus VT-i architectures…

Software-only VMMs Simpler and more Improved CPU and I/O virtualization Performance
VMM • Binary translation Secure VMMs through and Functionality as VMMs exploit infrastructure
Software • Paravirtualization foundation of provided by VT-x, VT-i, VT-d
• Device Emulation virtualizable ISAs
Evolution
Past Today
No Hardware VMM software evolution over
Support
time with hardware support
3
 New Feature Highlights

y APIC TPR Virtualization

– Significantly reduce VM exits caused by access to local
APIC TPR (not CR8)
– Submitted a patch (last month, not in yet)

y Virtual-processor Identifiers (VPIDs)

– Supports retention of TLB entries across VM switches

y Extended page tables (EPT)

y NMI-window Exiting
– Enables timely delivery of NMIs to guest OS

4
 New Feature Highlights (cont.)

y Preemption Timer
– Allows VMM to bound guest-OS execution time

y Descriptor-table Exiting
– Enables VMM to protect IDT, GDT, etc. from attack in
guest OS

y Interrupt remapping (VT-d2)

5
Future Extensions: VPIDs

VPIDs: General Idea

y TLBs cache for multiple address spaces

y Address spaces distinguished by VPIDs

– Host software runs with VPID zero
– Each virtual CPU has its own non-zero VPID

y CPU uses VPIDs to prevent TLB sharing

6
Future Extensions: VPIDs

VPIDs: Details

y New VM-execution controls:

– Use VPID (single-bit control)
– VPID value

y If use VPID is set:

– Guest’s VPID used while guest is executing
– No TLB flushes on entry to or exit from guest

y If use VPID is clear:

– Guest execution uses VPID zero
– TLB flushes on entry and exit

y New instruction for VMM to flush per VPID

7
Future Extensions: EPT

EPT: Overview
CR3 EPT Base Pointer

Guest Intel® 64 Guest EPT Host

Linear Page Physical Page Physical
Address Tables Address Tables Address

y Intel® 64 page tables

– Map guest-linear to guest-physical (translated again)
– Can be read and written by guest

y New EPT page tables under VMM control

– Map guest-physical to host-physical (accesses memory)
– Referenced by new EPT base pointer

y No VM exits due to page faults, INVLPG, or CR3 accesses

8
Future Extensions: EPT

EPT Page Tables

y Page-table details similar to Intel® 64:

– Each table has 512 8-byte entries (4KB)
– 4 levels of page tables
– Permission bits for read, write, execute

y Disallowed accesses
– Called EPT violations
– Cause VM exits

9
 VT-d Overview
y VT-d provides infrastructure for I/O virtualization
– Defines architecture for DMA and interrupt remapping
– Common architecture across IA platforms
– Will be supported broadly across Intel® chipsets

CPU CPU
System Bus

North Bridge
DRAM
VT-d
Integrated PCIe* Root Ports
Devices

PCI Express
South PCI, LPC,
Bridge Legacy devices, …

10 *Other names and brands may be claimed as the property of others

VT-d Applied to Pass-through Model
Pass-through Model
Direct Device Assignment to Guest OS
VM0 VMn
– Guest OS directly programs physical device
Guest OS Guest OS
– For legacy guests, hypervisor sets up guest- to and Apps and Apps
host-physical DMA mapping
Device Device
– For remapping aware guests, hypervisor Drivers Drivers
involved in map/unmap of DMA buffers

PCI-SIG I/O Virtualization Working Group Hypervisor

– Activity towards standardizing natively
sharable I/O devices Assigned
– IOV devices provide virtual interfaces, each Devices
independently assignable to VMs
Pro: Highest Performance
Pro: Smaller Hypervisor
Pro: Device-assisted sharing
Con: VM Migration Limits

11
 DMA Remapping: Features

y Translates DMA requests from all devices

– DMA requests specify DMA Virtual Address
– Hardware translates to Host Physical Address

y Flexible DMA virtual address space management

– DMA address space per device or sharable across devices
– Page granular memory management

y Other Features
– H/W caching of frequently used remapping structures
– Support for PCIe* Address Translation Services (ATS)
– Improved RAS by reporting DMA faults to software

12 *Other names and brands may be claimed as the property of others

 DMA Remapping: Hardware
Overview
DMA Requests
Dev 31, Func 7
Device ID Virtual Address Length … Dev P, Func 2
Bus 255
4KB
Bus N Page
Frame
Fault Generation Bus 0
Dev P, Func 1
4KB Page
Dev 0, Func 0 Tables

DMA Remapping Address Translation

Device Device D1 Structures
Engine Assignment
Translation Cache Structures
Device D2
Address Translation
Context Cache Structures

Memory Access with Host Physical Memory-resident Partitioning &

Address Translation Structures

13
 DMA Remapping: Page Walk
Requestor ID DMA Virtual Address
15 8 7 3 2 0 63 57 56 48 47 39 38 30 29 21 20 12 11 0

000000b 000000000b
Level-4 Level-3 Level-2 Level-1
Bus Device Func Page Offset
table offset table offset table offset table offset

Base
4KB Page

Level-4
Level-3
Example entry specifying Page Table
Page Table
Level-2
4-level page table Page Table
Level-1
Page Table

14
 Interrupt Virtualization

y Drivers for direct assigned devices run within VM

– Driver only aware of virtual CPU of the VM
– Device interrupts needs to be delivered to virtual CPU
– VT-x provides support for virtual CPU interrupt delivery

y Support lacking to isolate & route device interrupts

– Any direct assigned MSI capable device can generate any
physical interrupt (no interrupt isolation)
– No support to drain in-flight interrupts destined to a CPU
– No easy way to re-direct device interrupts (require IPIs)

Interrupt remapping enables

interrupt isolation and routing

15
 Interrupt Remapping
y Interrupt request specify request & originator IDs
– Remap hardware transforms request to physical interrupt

y Interrupt remapping hardware

– Enforces isolation through use of originator ID
– Generated interrupts with attributes in remap structure
– Caches frequently used remap structures
– S/W may modify remap for efficient interrupt re-direction

y Applicable to all interrupt sources

– Legacy interrupts delivered through I/O APICs
– Message signaled interrupts (MSI, MSI-X)
– Works with existing device hardware

16
 VT-d Support in Xen

y Device assignment by hypercalls

– Device assignment
– Give the ownership of the device
– I/O port access
– Unblock or remapping
– IRQ mapping
– Remap interrupts
– MMIO handling
– Set up translation in the shadow page table so that the
guest can directly access the device memory
y PCI config space virtualization
– BAR virtualization

y VT-d table for the device assigned

– Detect VT-d via ACPI tables
– Build (static) page tables for the device (BDF) using the P2M routines

17
 Current Status

y Sanity Checks
– Assigned PCIe E1000 add-on card to 32-bit FC5 on 64-bit
Xen.
– "scp" test shows near-native performance on the test
machine (e.g 200+Mbps).

y Submitted the patches to xen-devl mailing

list this month

y Testing on other guests

18
19