PCI Express® Basics

Richard Solomon
LSI Corporation

Copyright © 2012, PCI-SIG, All Rights Reserved 1

 Acknowledgements
 I would like to acknowledge the contributions of
Ravi Budruk, Mindshare, Inc.

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 2

 PCI Express® Introduction
 PCI Express architecture is a high performance, IO
interconnect for peripherals in
computing/communication platforms
 Evolved from PCITM and PCI-XTM architectures
 Yet PCI Express architecture is significantly different from its
predecessors PCI and PCI-X
 PCI Express is a serial point-to-point interconnect
between two devices
 Implements packet based protocol for information
transfer
 Scalable performance based on number of signal
Lanes implemented on the PCI Express interconnect

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 3

 PCI Express Terminology

PCI Express Device A

Signal Link

Wire Lane

PCI Express Device B

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 4

 PCI Express Throughput
Link Width
x1 x2 x4 x8 x12 x16 x32
PCIe 1.x BW (GB/s) 0.5 1 2 4 6 8 16

PCIe 2.x BW (GB/s) 1 2 4 8 12 16 32

PCIe 3.0 BW (GB/s) 2 4 8 16 24 32 64

Derivation of these numbers:

 2.5 GT/s (PCIe 1.x), 5.0 GT/s (PCIe 2.x), or 8GT/s (PCIe
3.0) signaling in each direction
 20% overhead due to 8b/10b encoding in 1.x and 2.x
 “Aggregate bandwidth”, implying traffic in both directions
PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 5
 PCI Express Features
 Point-to-point connection
 Serial bus means fewer pins
 Scalable: x1, x2, x4, x8, x12, x16, x32
 Dual Simplex connection
 2.5, 5.0 and 8.0 GT/s transfer/direction/s
 Packet based transaction protocol

Packet
PCIe PCIe
Device Link (x1, x2, x4, x8, x12, x16 or x32) Device
A B
Packet

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 6

 Differential Signaling
 Electrical characteristics of PCI Express signal
 Differential signaling
– Transmitter Differential Peak voltage = 0.4 - 0.6 V
– Transmitter Common mode voltage = 0 - 3.6 V

D-

Vcm
V Diffp
D+

 Two devices at opposite ends of a Link may support

different DC common mode voltages

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 7

 Additional Features
 Switches used to interconnect multiple devices
 Packet based protocol
 Bandwidth and clocking
 Same memory, IO and configuration address space as
PCI
 Similar transaction types as PCI with additional message
transaction
 PCI Express Transactions include:
 memory read/write, memory read lock, IO read/write,
configuration read/write, message requests
 Split transaction model for non-posted

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 8

 Additional Features
 Data Integrity and Error Handling
 RAS capable (Reliable, Available, Serviceable)
 Data integrity at: 1) Link level, 2) end-to-end
 Virtual channels (VCs) and traffic classes (TCs)
to support differentiated traffic or Quality of
Service (QoS)
 The ability to define levels of performance for packets
of different TCs
 8 TC’s and 8 VC’s available

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 9

 Additional Features
 Flow Control
 No retry as in PCI
 MSI style interrupt handling
 Also supports legacy PCI interrupt handling in-band
 Advanced power management
 Active State PM
 PCI compatible PM

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 10

 Additional Features
 Hot Plug and Hot Swap support
 Native
 No sideband signals
 PCI compatible software model
 PCI configuration and enumeration software can
be used to enumerate PCI Express hardware
 PCI Express system will boot existing OS
 PCI Express supports existing device drivers
 New additional configuration address space
requires OS and driver update

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 11

 PCI Express Topology
CPU

Root Complex
Bus 0 (Internal) Memory

PCIe 1 PCIe 6 PCIe 7

Switch
PCIe PCIe 3 Switch PCIe Virtual
PCI
Endpoint Endpoint PCIe Bridge
Bus 2
Bridge To
Virtual Virtual Virtual
PCIe 4 PCIe 5 PCI/PCI-X PCI PCI PCI
Bridge Bridge Bridge

PCIe Legacy
Endpoint Endpoint PCI/PCI-X
Bus 8

Legend
PCI Express Device Downstream Port
PCI Express Device Upstream Port
PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 12
 PCI Express System
Processor

FSB
PCI Express
GFX
GFX Root Complex
DDR
SDRAM
PCI Express Slots
Serial ATA PCI
HDD
USB 2.0 IO Controller Hub
(ICH) IEEE
LPC 1394

S Slot
IO
COM1 GB
COM2 Add-In Add-In Add-In
Ethernet

PCI Express PCI Express

Link

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 13

 Transaction Types,
Address Spaces
 Request are translated to one of four transaction types by
the Transaction Layer:
1. Memory Read or Memory Write. Used to transfer data from or to a
memory mapped location
– The protocol also supports a locked memory read transaction variant.
2. I/O Read or I/O Write. Used to transfer data from or to an I/O location
– These transactions are restricted to supporting legacy endpoint
devices.
3. Configuration Read or Configuration Write. Used to discover device
capabilities, program features, and check status in the 4KB PCI Express
configuration space.
4. Messages. Handled like posted writes. Used for event signaling and
general purpose messaging.

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 14

 PCI Express TLP Types
Description Abbreviated
Name
Memory Read Request MRd

Memory Read Request – Locked Access MRdLk

Memory Write Request MWr

IO Read Request IORd
IO Write Request IOWr
Configuration Read Request Type 0 and Type 1 CfgRd0, CfgRd1

Configuration Write Request Type 0 and Type 1 CfgWr0, CfgWr1

Message Request without Data Payload Msg

Message Request with Data Payload MsgD
Completion without Data (used for IO, configuration write completions and read Cpl
completion with error completion status)

Completion with Data (used for memory, IO and configuration read completions) CplD

Completion for Locked Memory Read without Data (used for error status) CplLk

Completion for Locked Memory Read with Data CplDLk

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 15

 Three Methods For Packet
Routing
 Each request or completion header is tagged as to its type, and
each of the packet types is routed based on one of three
schemes:
 Address Routing
 ID Routing
 Implicit Routing
 Memory and IO requests use address routing.
 Completions and Configuration cycles use ID routing.
 Message requests have selectable routing based on a 3-bit
code in the message routing sub-field of the header type field.

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 16

 Programmed I/O Transaction
Processor Processor

MRd FSB
Requester:
-Step 1: Root Complex (requester)
initiates Memory Read Request (MRd) Root Complex
-Step 4: Root Complex receives CplD DDR
SDRAM
MRd CplD

Switch A Switch C
MRd

CplD

Switch B Endpoint Endpoint Endpoint

MRd CplD
Completer:
Endpoint Endpoint -Step 2: Endpoint (completer)
receives MRd
-Step 3: Endpoint returns
Completion with data (CplD)
PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 17
 DMA Transaction
Processor Processor

FSB
Completer:
-Step 2: Root Complex (completer)
receives MRd Root Complex
-Step 3: Root Complex returns DDR
Completion with data (CplD) SDRAM
CplD MRd

Switch A Switch C
CplD

MRd

Switch B Endpoint Endpoint Endpoint

CplD MRd
Requester:
Endpoint Endpoint -Step 1: Endpoint (requester)
initiates Memory Read Request (MRd)
-Step 4: Endpoint receives CplD
PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 18
 Peer-to-Peer Transaction
Processor Processor

FSB

Root Complex
DDR
SDRAM
CplD MRd MRd CplD

Switch A Switch C
CplD

MRd MRd CplD

Switch B Endpoint Endpoint Endpoint Completer:

-Step 2: Endpoint (completer)
receives MRd
CplD MRd -Step 3: Endpoint returns
Completion with data (CplD)
Endpoint Endpoint Requester:
-Step 1: Endpoint (requester)
initiates Memory Read Request (MRd)
-Step 4: Endpoint receives CplD
PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 19
 PCI Express Device Layers
PCI Express Device A PCI Express Device B

Device Core Device Core

PCI Express Core PCI Express Core

Logic Interface Logic Interface
TX RX TX RX
Transaction Layer Transaction Layer

Data Link Layer Data Link Layer

Physical Layer Physical Layer

Link

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 20

 TLP Origin and Destination
PCI Express Device A PCI Express Device B

Device Core Device Core

PCI Express Core PCI Express Core

Logic Interface Logic Interface
TX RX TX RX

TLP Transaction Layer Transaction Layer TLP

Transmitted Received
Data Link Layer Data Link Layer

Physical Layer Physical Layer

Link

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 21

 TLP Structure
Information in core section of TLP comes
from Software Layer / Device Core

Bit transmit direction

Start Sequence Header Data Payload ECRC LCRC End

1B 2B 3-4 DW 0-1024 DW 1DW 1DW 1B

Created by Transaction Layer

Appended by Data Link Layer

Appended by Physical Layer

*Slightly different at 8GT/s*
PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 22
 DLLP Origin and Destination
PCI Express Device A PCI Express Device B

Device Core Device Core

PCI Express Core PCI Express Core

Logic Interface Logic Interface
TX RX TX RX
Transaction Layer Transaction Layer

DLLP Data Link Layer Data Link Layer DLLP

Transmitted Received
Physical Layer Physical Layer

Link

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 23

 DLLP Structure
Bit transmit direction

Start DLLP CRC End

1B 4B 2B 1B

Data Link Layer

 ACK / NAK Packets

Appended by Physical Layer  Flow Control Packets
*Slightly different at 8GT/s*
 Power Management Packets
 Vendor Defined Packets

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 24

 Ordered-Set Origin and
Destination
PCI Express Device A PCI Express Device B

Device Core Device Core

PCI Express Core PCI Express Core

Logic Interface Logic Interface
TX RX TX RX
Transaction Layer Transaction Layer

Data Link Layer Data Link Layer

Ordered-Set Ordered-Set
Physical Layer Physical Layer
Transmitted Received

Link

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 25

 Ordered-Set Structure in
2.5GT/s and 5GT/s modes
COM Identifier Identifier Identifier

 Training Sequence One (TS1)

 16 character set: 1 COM, 15 TS1 data characters
 Training Sequence Two (TS2)
 16 character set: 1 COM, 15 TS2 data characters
 SKIP
 4 character set: 1 COM followed by 3 SKP identifiers
 Fast Training Sequence (FTS)
 4 characters: 1 COM followed by 3 FTS identifiers
 Electrical Idle (IDLE)
 4 characters: 1 COM followed by 3 IDL identifiers
 Electrical Idle Exit (EIEOS) (new to 2.0 spec)
 16 characters
PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 26
 Quality of Service
Processor Processor

PCI Express
GFX
GFX Root Complex
DDR
Endpoint SDRAM

10Gb
InfiniBand
InfiniBand Switch Ethernet Switch Fiber
Switch
Endpoint Channel
“Out-of-Box” Endpoint
RAID Disk array
10Gb PCI Express
Add-In Switch SCSI
Ethernet to-PCI
Endpoint Endpoint Endpoint
Slot PCI
Video Slots
SCSI S IEEE
PCI Express Camera IO 1394
Link Endpoint COM1
COM2

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 27

 PCI Express Flow Control
 Credit-based flow control is point-to-point
based, not end-to-end
Buffer space
available
TLP
VC Buffer

Transmitter Receiver

Flow Control DLLP (FCx)

Receiver sends Flow Control Packets (FCP) which are a type of DLLP (Data Link Layer Packet)
to provide the transmitter with credits so that it can transmit packets to the receiver

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 28

 ACK/NAK Protocol Overview
Transmit Receiver
Device A Device B
From To
Transaction Layer Transaction Layer
Tx Rx
Data Link Layer Data Link Layer
TLP DLLP DLLP TLP
ACK / ACK /
Sequence TLP LCRC NAK NAK
Sequence TLP LCRC

Replay
Buffer De-mux De-mux

Error
Mux Mux Check

Tx Rx Tx Rx
DLLP
ACK /
NAK

Link

TLP
Sequence TLP LCRC

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 29

 ACK/NAK Protocol:
Point-to-Point

1a. Request 2a. Request

4b. ACK 3b. ACK

Requester Switch Completer

1b. ACK 2b. ACK

4a. Completion 3a. Completion

ACK returned for good reception of Request or Completion

NAK returned for error reception of Request or Completion

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 30

 Interrupt Model:
Three Methods
 PCI Express supports three interrupt reporting
mechanisms:
1. Message Signaled Interrupts (MSI)
– Legacy endpoints are required to support MSI (or MSI-X) with 32- or
64-bit MSI capability register implementation
– Native PCI Express endpoints are required to support MSI with 64-bit MSI
capability register implementation
2. Message Signaled Interrupts - X (MSI-X)
– Legacy and native endpoints are required to support MSI-X (or MSI)
and implement the associated MSI-X capability register
3. INTx Emulation.
– Native and Legacy endpoints are required to support Legacy INTx
Emulation
– PCI Express defines in-band messages which emulate the four
physical interrupt signals (INTA-INTD) routed between PCI devices
and the system interrupt controller
– Forwarding support required by switches

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 31

 Native and Legacy
Interrupts

PCIe -

PCIe PCIe

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 32

 PCI Express
Configuration Space

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 33

 Thank you for attending the
PCI-SIG Developers Conference 2012

For more information please go to

[Link]

PCI-SIG Developers Conference Copyright © 2012, PCI-SIG, All Rights Reserved 34