IPSec VPN

NGFW: TAC Advanced Training 9.0

Joshua Yendrapati
ETAC

Confidential and Proprietary.

For WW Professional Services & Customer Support Organization
DO NOT SHARE outside of the Organization
WW Professional Services & Customer Support Organization Securing Intellectual Property Guideline

• WW Professional Services & Customer Support Organization prohibits any

practice of posting and uploading assets even on internal sites without
sufficient access control to prevent materials accessible outside of the WW
Professional Services & Customer Support Organization or Palo Alto
Networks.
• All product related presentations are to be considered highly confidential and
NOT to be shared outside of the WW Professional Services & Customer
Support Organization.
Resource: https://intranet.paloaltonetworks.com/docs/DOC-26009

It's the responsibility of everyone in the WW Professional Services &

Customer Support Organization to protect Palo Alto Networks
Intellectual Properties!
2 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Agenda

• Overview

• VPN Concepts

• Verifying Tunnels

• Troubleshooting

3 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IPsec Fundamentals Review

• IPsec is a network protocol suite used to authenticate and encrypt traffic sent
over IP networks
• Provides:
• Mutual authentication
• Data-origin authentication
• Data integrity
• Data confidentiality
• Replay Protection

• Can be used to securely tunnel traffic between

• Two hosts
• Two sites (site-to-site)
• Host to site (remote access)

4 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
What Is A Tunnel?

10.1.10.5 A B
10.1.20.3
10.1.10.6 Trust IP: 10.1.10.1 Trust IP: 10.1.20.1
Untrust IP: 1.1.1.1 10.1.20.4
Untrust IP: 2.2.2.1

Original Packet

SRC IP DST IP Proto SRC Port DST Port Data

10.1.10.5 10.1.20.3 6 34390 80 get

• The original packet is sent using private addresses (not publicly routable)

• The firewall will perform a route lookup for the destination, and identify the
appropriate next hop

5 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
What Is A Tunnel?

10.1.10.5 A B 10.1.20.3
10.1.10.6 Trust IP: 10.1.10.1 Trust IP: 10.1.20.1
Untrust IP: 1.1.1.1
10.1.20.4
Untrust IP: 2.2.2.1

Public Addressing
Private Addressing
SRC IP DST IP Proto XXX SRC IP DST IP Proto SRC Port DST Port Data
XXX10.1.10.5 10.1.20.3 6 34390 80 get
2.2.2.1 1.1.1.1 XX
Tunnel Header Original Packet
Tunnel Trailer

• Here a tunnel is created when the firewall encapsulates the original packet
and sends it across the Internet

6 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
What Is A Tunnel?

10.1.10.5 A B 10.1.20.3
10.1.10.6 Trust IP: 10.1.10.1 Trust IP: 10.1.20.1
Untrust IP: 1.1.1.1
10.1.20.4
Untrust IP: 2.2.2.1

Original Packet
SRC IP DST IP Proto SRC Port DST Port Data
10.1.10.5 10.1.20.3 6 34390 80 get

• The VPN peer will decapsulate the encrypted packet once received

• The original packet remains intact and is sent to the destination

7 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Agenda

• Overview

• VPN concepts

• Verifying tunnels

• Troubleshooting

8 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
VPN Concepts

In this section we will discuss:

• IPsec
• Encapsulating Security Payload (ESP)
• Authentication Header (AH)
• Security Associations (SA)
• ESP Packet Flow

9 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IPsec Components
Uses two protocols to provide traffic security:

• Encapsulating Security Payload (ESP)

• confidentiality
• traffic flow confidentiality
• connectionless integrity
• data origin authentication
• anti-replay service

• Authentication Header (AH)

• connectionless integrity
• data origin authentication
• anti-replay service

10 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Encapsulation Security Payload (ESP)
• Encapsulating Security Payload (ESP)
• IP Protocol 50
• Tunnel Mode
• Data Encryption
▪ uses encryption algorithms like DES, 3DES, AES
▪ encrypts original datagram
▪ does not encrypt tunnel header
• Data Authentication
▪ uses authentication algorithms like HMAC_MD5
and HMAC_SHA
▪ for original datagram and ESP header
▪ does not authenticate tunnel header

11 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Authentication Header (AH)

• Authentication Header (AH)

• IP Protocol 51
• Tunnel Mode
• Data Authentication
▪ uses authentication algorithms like HMAC_MD5
and HMAC_SHA
▪ authenticates entire datagram including tunnel header

12 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Security Association (SA)
• An association between two communicating entities that consists of:
• Security Parameter Index (SPI, Used to Identify this SA)
• Security protocol (AH or ESP)
• Encryption and/or Authentication Algorithms
• Encryption and/or Authentication Keys
• Key Lifetime

• Identified by the Security Index (SPI) number, destination IP address and IP

protocol (AH or ESP)
• Bi-directional traffic requires two SAs (most common)
• SPIs must be unique for each tunnel between devices in an IPsec domain

13 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
ESP Concepts

10.1.10.5 A B
10.1.20.
3
10.1.10.6 Trust IP: 10.1.10.1 Trust IP: 10.1.20.1 10.1.20.
Untrust IP: 2.2.2.1 Untrust IP: 1.1.1.1 4
Security Database Security Database
Name: Manual-to-1.1.1.1 Name: Manual-to-2.2.2.1
Gateway IP: 1.1.1.1 Gateway IP: 2.2.2.1
Security Index (SPI) Security Index (SPI)
local: 3001, remote: 3002 local: 3002, remote: 3001
Security Protocol: ESP Security Protocol: ESP
Encryption Alg: DES Encryption Alg: DES
Encryption Key: 123abc… Encryption Key: 123abc…
Authentication Alg: MD5 Authentication Alg: MD5
Authentication Key: 345dea… Authentication Key: 345dea…

• Encryption using manual keys

• Security Index, encryption algorithm, authentication algorithm, encryption keys and authentication
keys must be manually exchanged before an IPsec tunnel can be established

14 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IPsec Example - Packet Flow

10.1.10.5 A B
10.1.20.
3
10.1.10.6 Trust IP: 10.1.10.1 Trust IP: 10.1.20.1 10.1.20.
Untrust IP: 2.2.2.1 Untrust IP: 1.1.1.1 4
Original packet
SRC IP DST IP Proto SRC Port DST Port Data
10.1.10.5 10.1.20.3 6 34390 80 get

• When the packet comes in the trust interface

1. A session lookup is performed, if no match
2. A route lookup is performed on the original destination
• This is performed to find the correct policy set
3. Route exists for 10.1.20.3 out the tunnel interface
4. The PAN will know that this datagram initiated from the trust to VPN so the outgoing policy is searched
for a match

15 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IPsec Example - Packet Flow

10.1.10.5 A B
10.1.20.
3
10.1.10.6 Trust IP: 10.1.10.1 Trust IP: 10.1.20.1 10.1.20.
Untrust IP: 2.2.2.1 Untrust IP: 1.1.1.1 4

Original Datagram

3DES { SRC IP
10.1.10.5
DST IP Proto SRC Port
10.1.20.3 6 34390
DST Port
80
Data
get , Encrypt Keys }
SRC IP DST IP Proto SRC Port DST Port Data
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

• The entire original datagram is encrypted with the encryption algorithm and keys specified in
the outgoing policy

16 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IPsec Example - Packet Flow

HMAC_SHA { ESP SRC IP DST IP Proto SRC Port DST Port Data
Info XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ,Auth_Key }
= 98cd890c 28df9893 120a0b32 3e45fa9c 789290e4

• With IPsec using ESP, HMAC is performed on the encrypted original datagram
and the ESP header before the tunnel header is added

• The result of the HMAC hash is added to the ESP Hash field after the ESP
tunnel trailer

17 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IPsec Example - Packet Flow

10.1.10.5 A B
10.1.20.
3
10.1.10.6 Trust IP: 10.1.10.1 Trust IP: 10.1.20.1 10.1.20.
Untrust IP: 2.2.2.1 Untrust IP: 1.1.1.1 4
|----------------------------- Authenticated
---------------------------|
|------------- Clear ----------------|------------------- Encrypted -----------------------------|----Clear ----|
SRC IP DST IP Proto ESP SRC IP DST IP Proto SRC Port DST Port Data
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
ESP Hash
2.2.2.1 1.1.1.1 50 SPI 3002
Tunnel Header 98cd8… Original Datagram
Tunnel Trailer

• The ESP tunnel header and trailer are added

• The ESP packet is then sent across the Internet

18 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IPsec Example - Packet Flow

10.1.10.5 A B
10.1.20.
3
10.1.10.6 Trust IP: 10.1.10.1 Trust IP: 10.1.20.1 10.1.20.
Untrust IP: 2.2.2.1 Untrust IP: 1.1.1.1 4
SRC IP DST IP Proto ESP SRC IP DST IP Proto SRC Port DST Port Data
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
ESP Hash
2.2.2.1 1.1.1.1 50 SPI 3002
Tunnel Header 98cd8… Original Datagram
Tunnel Trailer
• When the datagram is received by 1.1.1.1, no policy can be checked because the original
datagram is encrypted

• Reads protocol 50 and calls the ESP process which then reads the SPI number 3002

• The local SPI’s are searched for a match for the SPI in the ESP header in the local SAD
(Security Association Database)

19 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Security Association Database - (SAD)

• The Security Association Database Security Database

Name: Manual-to-2.2.2.1
is searched and a match for this Gateway IP: 2.2.2.1
example is found Security Index (SPI)
local: 3002, remote: 3001
• Once the match is found the Security Protocol: ESP
datagram is first authenticated using Encryption Alg: DES
HMAC and the Authentication Key Encryption Key: 123abc…
Authentication Alg: MD5
Authentication Key: 345dea…

20 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IPsec Example - Packet Flow

HMAC_SHA { ESP SRC IP DST IP Proto SRC Port DST Port Data
Info XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ,Auth_Key }
= 98cd890c 28df9893 120a0b32 3e45fa9c 789290e4

• Verifies the hash sent in the ESP packet by performing its own hash using
HMAC on the encrypted datagram and the local authentication key

• If the hash matches, the datagram is then decrypted

• If the hash does not match, the datagram is dropped and you can see auth
error

21 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IPsec Example - Packet Flow

10.1.10.5 A B
10.1.20.
3
10.1.10.6 Trust IP: 10.1.10.1 Trust IP: 10.1.20.1 10.1.20.
Untrust IP: 2.2.2.1 Untrust IP: 1.1.1.1 4

Original Datagram

3DES { SRC IP DST IP Proto SRC Port DST Port Data

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , Encrypt Keys }
SRC IP DST IP Proto SRC Port DST Port Data
10.1.10.5 10.1.20.3 6 34390 80 get

• The entire original datagram is decrypted with the encryption algorithm and keys
specified in the Security Association Database

22 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IPsec Example - Packet Flow

10.1.10.5 A B
10.1.20.
3
10.1.10.6 Trust IP: 10.1.10.1 Trust IP: 10.1.20.1 10.1.20.
Untrust IP: 2.2.2.1 Untrust IP: 1.1.1.1 4

SRC IP DST IP Proto SRC Port DST Port Data

Original Datagram
10.1.10.5 10.1.20.3 6 34390 80 get

• Finally, the original datagram is checked against a matching outgoing policy for
the SPI number used
• If the datagram is a reverse match with all the fields, it is sent out the trust
interface

23 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Nat-Traversal (NAT-T)

• ESP is IP protocol 50

• Some NAT devices cannot handle non-TCP/UDP traffic

• Nat-Traversal (NAT-T) will encapsulate ESP with UDP 4500 to accommodate

NAT devices

• This is also commonly used for VPN clients although GlobalProtect uses UDP
port 4501

24 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
TCP MSS

• The Maximum Transmission Unit (MTU) is 1500 bytes for Ethernet

• The maximum frame size for Ethernet is 1514 bytes untagged or 1518
bytes with vlan tag
• ESP adds 36 bytes or more to a packet
• If your IP packet that is to be tunneled is >1464 the resulting packet will
exceed 1500 bytes
• This will cause the ESP packet to be fragmented which can drastically
decrease speed of IP communication

25 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
TCP MSS Adjustment

• Adjusting the maximum segment size (MSS)

• Based on MTU, will lower MSS by 40 bytes to account for IP and TCP header and/or IPSec
encapsulation
• If a packet cannot fit within the egress interface MTU without fragmenting, this setting
allows an adjustment to be made
• During 3-way handshake, MSS will be rewritten before transmitting packet out
• This setting should be enabled on the inside interface.
26 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Internet Key Exchange (IKE)

• Establishes Security Associations for creating IPsec VPN tunnels over the
Internet
• Proposals containing encryption and authentication algorithms are negotiated
• Encryption and Authentication Keys are created automatically and have the
ability to be re-keyed frequently
• Easier to scale than IPsec with Manual Keys
• Uses UDP port 500

• Nat-Traversal, switches to UDP port 4500

27 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Types of IKE protocol
• There are two versions of IKE protocol. IKEv1 and IKEv2, with the later being the most recent
version. Here are the key differences:

IKEv1 IKEv2
Phase 1 Exchange Modes: Only one exchange procedure is
Main mode: 6 messages defined and it uses 4 messages.
Aggressive mode: 3 messages
Both sides must use the same Each side can use a different
authentication method authentication method

No reliability as the messages Each message is ACK’d and

are not ACK’d sequenced
Lifetime for SA’s need to be Lifetime for SA’s is Not
agreed between the peers negotiated. Each peer can
delete SA’s at any time
Doesn’t support EAP – Supports EAP
Extensible Authentication
Protocol.

28 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Initial Phases: IKEv1 VS IKEv2

• IKEv1 has the following phases:

• Phase 1: Sets up mutual authentication of peers, cryptographic parameters and creates
session keys
• Phase 2: Creates IPSec tunnel by using the session keys created in Phase 1

• IKEv2 has the following initial phases of negotiation:

• IKE_SA_INIT Exchange
• IKE_AUTH Exchange

• IKEv2 combines the Phase 2 information in IKEv1 into the

IKE_AUTH exchange
• PAN-OS supports IKEv2 from 7.0 and above

29 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IKEv1

• Phase 1
• Sets up secure channel from security gateway to security gateway
• The Diffie-Hellman key exchange algorithm is used to generate a symmetric key
common to the communicating gateways

• Phase 2
• Sets up a secure tunnel for network to network communication from security gateway to
security gateway
• Phase 2 security associations are created using a Phase 1 secure channel
• The Diffie-Hellman key exchange algorithm may be used to create perfect forward
secrecy

30 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IKEv1 Phase 1 - Main Mode: 6 Messages

10.1.10.5 A B
10.1.20.
3
10.1.10.6 Trust IP: 10.1.10.1 Trust IP: 10.1.20.1 10.1.20.
Untrust IP: 2.2.2.1 Untrust IP: 1.1.1.1 4

Message 1:
Cookie I, SA Proposal List
Message 2:
Cookie I, Cookie R, SA Proposal Accept
Message 3:
DH Public Key A, Nonce I (Random #)
Message 4:
DH Public Key B, Nonce R (Random #)
Message 5:
Identification I, Identification Hash I
Message 6:
Identification R, Identification Hash R

31 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IKEv1 Phase 1: Main vs. Aggressive Mode

• Main mode has a six packet exchange

• More secure
• Used most often for site-to-site VPN
• Peer ID for both sides are static IP

• Aggressive mode has a three packet exchange

• Less secure
• Used for dynamic peer (VPN peer has dynamic IP)
• Aggressive mode initiator local-id as FQDN for peer ID
• Aggressive mode responder peer-id as FQDN

32 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Aggressive Mode Example

Static peer side Dynamic peer side

Peer IP Type Dynamic is selected when peer IP or FQDN value is unknown. In such cases, it is up to the peer to
initiate IKE gateway negotiation
33 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Aggressive Mode Example
• In the advanced tab for both side select the aggressive mode as shown below

34 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IKEv1 Phase 2 - Quick Mode: 3 Messages

10.1.10.5 A B
10.1.20.
3
10.1.10.6 Trust IP: 10.1.10.1 Trust IP: 10.1.20.1 10.1.20.
Untrust IP: 2.2.2.1 Untrust IP: 1.1.1.1 4
Message 1:
Hash using Phase 1 information,
Message ID, SA Proposal List, Nonce I, Message 2:
[DH Public Key I ], Proxy ID Hash using Phase 1 information,
Message ID, SA Proposal List Accept,
Nonce R, [DH Public Key I ], Proxy ID
Message 3:
Hash using Phase 1 information,
Message ID, Nonce I, Nonce R

35 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IKEv2

• IKE_SA_INIT Exchange
• This is the initial exchange in which the peers establish a secure channel. After this
exchange, all further exchanges are encrypted

• The exchanges contain only two packets. This exchange combines all the information
usually exchanged in messages 1 to 4 in IKEv1

• IKE_AUTH Exchange
• This exchange is used to authenticate the remote peer and create the first IPsec SA

• The exchange contains the ISAKMP ID along with an authentication payload. Additionally,
the exchange also includes the SA and Traffic Selector payloads that describe the IPsec
SA to be created

36 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IKEv2
• After IKE_SA_INIT exchange and the IKE_AUTH exchange, both peers will have one SA built
and will be ready to encrypt traffic for the proxy ID's matching the trigger packet

• CREATE_CHILD_SA Exchange
• This exchange serves the same function that the Quick mode exchange does in IKEv1. All
the subsequent traffic that is matching other proxy id’s will trigger the CREATE_CHILD_SA
exchange

37 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IKEv2 - Messages

10.1.10.5 A B
10.1.20.
3
10.1.10.6 Trust IP: 10.1.10.1 Trust IP: 10.1.20.1 10.1.20.
Untrust IP: 2.2.2.1 Untrust IP: 1.1.1.1 4

Message 1:
Cookie I, SA Proposal List, DH Public Key A, Nonce
: Message 2
Cookie I, R, SA Proposal List, DH Public Key B, Nonce

Message 3:
Identification I, Identification Hash I, SPI
: Message 4
Identification R, Identification Hash R, SPI

Message 1 and Message 2 are part of IKE_SA_INIT Exchange

Message 3 and 4 are part of IKE_AUTH Exchange

38 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Dead Peer Detection

• DPD is a method used by devices to verify the current existence and

availability of IPsec peer devices

• DPD peer sends encrypted payloads (R-U-THERE) to peer on UDP 500

• Sender waits for DPD acknowledgements (R-U-THERE-ACK)

• Most IPsec peers (PAN-OS included) can respond with R-U-THERE-ACK

messages
• R-U-THERE messages will not be sent unless DPD is enabled

• Can cause issues if one side has DPD enabled and other side does not as
side which does not may not be aware of connectivity loss (Cisco for example
has enabled by default)

39 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Tunnel Monitoring
• Can perform similar function as DPD to determine if peer connectivity is lost
• Sends pings across tunnel
• DPD just confirms gateway-to-gateway connectivity and not actual tunnel traffic
connectivity

• Can bring tunnel up immediately and rekey automatically. Not dependent on tunnel
traffic to bring tunnel up
• Also can mark tunnel interfaces as down if pings are lost, useful for affecting
dynamic routing protocols
• Must specify IP on tunnel interface and specify a ping target on other side of VPN
• Tunnel monitoring is proprietary protocol (not part of IPSec/ISAKMP protocol).
Similar to VPN monitoring on Juniper devices, though officially interop is not
supported

40 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
PAN-OS IPsec Enhancements

• PAN-OS 8.0
• IKE Peer and IPsec Tunnel Capacity Enhancements

• PAN-OS 8.1
• FQDN Support for IKE Gateway Peer IP Address

41 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Agenda

• Overview

• VPN Concepts

• Verifying Tunnels

• Troubleshooting

42 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Testing the VPN

• Ping from a device on the far network, through the VPN, and to a target PC on
the local network protected by the PAN firewall. The first ping will fail, but the
rest should be successful. Examine the system log on the PAN firewall, either
via:

• Monitor > Logs > System, or

• CLI:
> show log system subtype equal vpn direction equal backward

• Can manually initiate VPN:

> test vpn ipsec-sa tunnel <name>

43 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Testing the VPN
• Check system log with (subtype eq vpn)

• You want to see messages that look like the following—this is a successful VPN startup:

• In CLI can run below command

> show log system direction equal backward subtype equal vpn

44 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Confirmation of VPN

Confirmation

• When the tunnel is up, Network > IPSec Tunnels page should show the
phase 1 and 2 status in green:

IKE Phase 2 IKE Phase 1

Green=up Green=up
Red=down Red=down

45 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Confirmation of VPN

Can use following CLI commands to verify that tunnel is active:

46 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Verify Tunnel Flow

Can use following command to verify that tunnel is active:

47 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Verify Tunnel Flow

View by tunnel-id (enter id from the step above)

If tunnel monitoring is enabled, can check

here for any lost monitor pings

SPI in/out for this tunnel

Auth errors, could be corrupted packet or

fragmented ESP with some lost fragments

Replay packets could be ESP packets out-of-

order

Look for encap/decap counters movement

48 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Verify Tunnel Session

• To confirm via tunnel session, do the following:

> show session all filter protocol 50

Note: The c2s “ports” shown in the session when converted from decimal to HEX should match the SPI-out for that tunnel. The
s2c ports should match SPI-in. For example:

10.46.198.49[25307] -> 10.46.41.192[52487]

Converting as follows:
52487 = 0xCD07
25307 = 0x62DB

SPI-out would be CD0762DB.

49 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Verify Tunnel Session

• View by session ID#

50 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Verify Tunnel Creation Reason

• "debug keymgr list-sa" command displays additional information

Initiator

admin@PA> debug keymgr list-sa

active keys: 1
gateway ID tunnel ID local SPI peer SPI tx seq type remain from show recent
---------- --------- --------- -------- ------ ---- ------ ---- -----------
2 2 8DD40CE7 CD0762DB 0 2 1886 IPSec-SA rekey True

Responder

admin@PA> debug keymgr list-sa

active keys: 1
gateway ID tunnel ID local SPI peer SPI tx seq type remain from show recent
---------- --------- --------- -------- ------ ---- ------ ---- -----------
2 2 CD0762DB 8DD40CE7 0 2 1738 IPSec responder True

51 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
keymgr.log

• Easy to follow SPI creation/deletion, logged with INFO level

2018-09-26 11:47:23.789 +0200 [INFO]: { 2: 2}: SPI 98452296 inserted by IKE responder, return 0 0.
2018-09-26 11:48:56.948 +0200 [INFO]: { 2: 2}: SPI 98452296 removed by IKE config change, return 0 0.
2018-09-26 11:49:20.891 +0200 [INFO]: { 2: 2}: SPI DDC42F64 inserted by IKE responder, return 0 0.
2018-09-26 12:41:36.011 +0200 [INFO]: { 2: 2}: SPI CE73EAAE inserted by IPSec responder, return 0 0.
2018-09-26 12:41:36.020 +0200 [INFO]: { 2: 2}: SPI DDC42F64 removed by keymodify, return 0 0.
2018-09-26 13:32:55.981 +0200 [INFO]: { 2: 2}: SPI E3823644 inserted by IPSec responder, return 0 0.
2018-09-26 13:32:55.990 +0200 [INFO]: { 2: 2}: SPI CE73EAAE removed by keymodify, return 0 0.

52 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Agenda

• Overview

• VPN Concepts

• Verifying Tunnels

• Troubleshooting

53 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Troubleshooting Methodology
Basic troubleshooting begins by first isolating the issue and then focusing where the problem is occurring.
One common approach is to start with Layer1 and work your way up

• First confirm physical connectivity of Internet link at physical and data link level. Can use ping and
confirming connectivity to remote peer IP

• Assuming pings are successful, confirm IKE phase 1

• Once phase 1 is confirmed, then confirm phase 2

• Finally, confirm that traffic is flowing across the VPN

Note that when troubleshooting VPNs it is often necessary to look at both VPN peers as issue may be on
either side. You can only get so far in troubleshooting if you can access one peer only. Also often the
responder side will be more useful for troubleshooting as most relevant error messages will be on
responder. So if remote peer is not admin accessible, then should suggest to request remote peer to
initiate the VPN.

54 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Troubleshooting
• Make sure over-all layer3 configuration is valid
• No overlapping IP/subnet
• SSL-VPN IP pool should not overlap with any existing subnets
• Routing table populated everywhere
• Router, PC, firewall, L3 switch etc.

• Make sure there are no layer3 loops

• Example1: default route point to tunnel I/F
• Example2: OSPF over tunnel I/F created specific route for tunnel end-point

• Make sure there are no layer2 loop

• Example: layer3 forwarding path go through same switch twice

55 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Troubleshooting – IKE (1 of 2)
• Daemon health check
• ikemgr, keymgr & other system software

• Configuration check
• show vpn gateway/tunnel
• User can search for gateway(s), tunnel(s), ike-sa(s) or ipsec-sa(s) by matching for a string. The string is case
sensitive.
> show vpn gateway match <string>
> show vpn tunnel match <string>

• To check config mismatch, try to make PAN side as responder

> test vpn ike-sa/ipsec-sa makes PA as initiator

• Check system log

> show log system subtype equal vpn direction equal backward

56 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Troubleshooting – IKE (2 of 2)
• Debug negotiation process
• Check IKE’s corresponding flow session
> show session all filter destination <peerIP> destination-port 500
> show vpn ike-sa

• Check debug log

> less mp-log ikemgr.log

• Packet capture
> debug ike pcap on/off/delete
> debug ike global on debug
> scp export debug-pcap
> view-pcap <options> debug-pcap ikemgr.pcap

57 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Troubleshooting – IPSec
• Check data-plane health
• Look for errors in tund log
> less dp-log tund.log

• Check routes/FIB
> show routing route
> show routing fib

• Check tunnel status

> show vpn flow tunnel-id <id>
▪ Shows tunnel config and counters
▪ Shows tunnel monitor status

• Check encap session status

• Check UDP session for NAT-T

• Check tunnel lookup tables (especially for multiple binding)

> show vpn tunnel
> show vpn gateway

58 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Per Tunnel Debugs
• Per IKE gateway debug filters

• Per IPSEC-tunnel debug filters

• Tailing ikemgr logs will show the logs only for the specific gateway and tunnel for which the debugs have been enabled

• Similarly, filters can be applied specific to gateway(s) and/or tunnel(s) on keymgr

59 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
PCAP | Type – IKE

• Used mainly when there are proposal mismatches

• Can be used to see what proposals the peer is sending

• All debugging must be turned off when troubleshooting is finished

• IKE debugging writes to the “ikemgr.pcap” file

60 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IKE | VPN Error messages
Issue Initiator error Responder error
Wrong IP / no connection P1 - Timeout P1 - Timeout
No matching P1 Proposal P1 - Timeout No suitable proposal (P1)
Mismatched Peer ID P1 - Timeout Peer identifier does not match

No matching P2 Proposal No proposal chosen No suitable proposal (P2)

PFS Group mismatch P2 - Timeout Pfs group mismatch
Mismatched Proxy ID P2 - Timeout cannot find matching phase-2
tunnel

61 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IKE | CLI Syntax
> debug ike global on debug
• Activates debug level for ikemgr.log
> debug ike pcap on
• Activates a PCAP of all IKE traffic
> view-pcap <options> debug-pcap ikemgr.pcap
• Displays the PCAP in the CLI.
• Can be used to view it in real time with the follow option
> debug ike pcap off
• Turns off packet capture
> debug ike global on normal
• Sets ikemgr debug level back to normal
> scp export debug-pcap from ikemgr.pcap to [user@SCP-IP:path]
• Copies PCAP off of the firewall
> debug ike pcap delete
• Removes the ikemgr.pcap file
> less mp-log ikemgr.log
• View ikemgr.log debug log

62 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Reading VPN Error messages from System Log

peer identifier (type fqdn [bad.peer]) does not match remote

Remote2.

Name of local Phase 1 IKE Gateway object Remote sides Phase 1 peer configuration

IKE phase-2 negotiation failed when processing proxy ID.

cannot find matching phase-2 tunnel for received proxy ID.
received local id: 192.168.41.1/24 type IPv4_subnet protocol
0 port 0, received remote id: 192.168.42.1/24 type
IPv4_subnet protocol 0 port 0.
The “remote proxy ID” from the other side
The “Local Proxy ID” from the other side

63 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IKE | Sample – IKE Phase 1
Phase 1 Gateway

id cookie ->: phase 1 I ident: SA Lifetime of 8 hours

(sa: doi=ipsec situation=identity 7080 is Hex for 28800
(p: #1 protoid=isakmp transform=2 28800 seconds = 8 hours
(t: #1 id=ike (type=lifetype value=sec) (type=lifeduration value=7080)
(type=enc value=aes) (type=keylen value=0080) (type=auth value=preshared)
(type=hash value=sha1) (type=group desc value=modp1024))

Pre-shared key for

authentication
AES 128 (Hex 80 = 128) Modp DH Group
768 1
SHA1 for Hash DH Group 2
1024 2
1536 5
2048 14

64 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
IKE | Sample – IKE Phase 2

Phase 2 Tunnel
id cookie ->: phase 2/others I oakley-quick: Lifetime of 60 minutes
(hash: len=20) SPI Hex e10 = 3600
(sa: doi=ipsec situation=identity 3600 seconds = 1 hour
(p: #1 protoid=ipsec-esp transform=3 spi=a960c9a3
(t: #1 id=aes (type=lifetype value=sec)(type=life value=0e10)(type=enc
mode value=tunnel) (type=keylen value=0080) (type=auth value=hmac-sha1)
(type=group desc value=modp1024))

SHA 1
AES 128
IPSec ESP
If it was AH the value PFS enabled
would be “Transport” DH Group 2

65 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Tunnel Resources

• Tunnel resources in use

66 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Tunnel Resources

• Tunnel resources in use

67 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Viewing Tunnel ID or Context

68 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Tunnel Resources

• Total and In-Use Contexts and Next hops

69 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Tunnel Resources

• Total and In-Use Contexts and Next hops

70 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
 Questions?

71 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
THANK YOU

Email: [email protected]
 Appendix 1:
Configurations
Capacity Limits

7k- 7k- 5260 5250 5220 PA- PA- VM- VM- VM- VM-
XM- NPC Large Medium Small 5k 3k 700 500 300 50
NPC
IKE 4000 2000 5000 4000 3000 2000 200 200 1000 1000 250
Peers 0 0
IPSec 12000 8000 15000 12000 10000 8000 300 800 4000 2000 250
Tunnel 0 0
s

• The number of tunnels will be limited to the max number of routes supported, if each tunnel has its own IP address (each
IP address is counted as an entry in the routing table).

74 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Configure Tunnel Interface

Part 1: Configure tunnel endpoint on the PAN device

Network > Interfaces screen. Create a new tunnel interface. Assign the following
parameters:
• Name: tunnel.1
• Virtual router: (select the existing virtual router)
• Zone: (select the layer 3 internal zone from which the traffic will originate)

75 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Configure IKE Gateway

Network > Network Profiles > IKE Gateways screen. Click New, and enter the
following parameters:
• IKE gateway: gw-to-siteX (or any name of your choosing)

• Local IP address: (select the firewall interface that is closest to the other vpn endpoint. This
is called the “public” interface of the firewall.)

• Peer IP address: (enter the IP address of the “public” interface on the other vpn endpoint)

• Pre-shared key: (enter a key of your choosing, and remember it so you can enter it in the
other firewall’s VPN configuration)

76 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Configure IKE Phase 2

Network > IPSec Tunnels screen. Create a new VPN with the following parameters:

• Name: vpn-to-siteX (or any name of your choosing)

• Tunnel interface: (pull down to select tunnel.1)

• IKE gateway: (pull down to select the IKE gateway you created in the previous step)

• If the other side of the tunnel is configured as a policy-based VPN, then:

• Click “Show advanced options”

• Enter the local proxy ID and remote proxy ID to match the other side:
Note: Cisco VPN peers are policy-based. Proxy-IDs need to be configured for each access-list entry
(local-remote IP pair) from Cisco.

77 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Configure IKE Phase 2

• Once you click OK, the IPSec tunnel will appear in the list, with the status
circles colored red to indicate the tunnel is down.

78 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Configure Route for Tunnel Interface

Network > Virtual Routers screen. Edit your existing virtual router.

• Add a new route for the network that is behind the other VPN endpoint. For
interface, select “tunnel.1”. There is no need to enter a value for next hop.

• Click Add to add the static route.

79 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Configure Remote Peer

Part 2: Configure the tunnel on the other firewall

Configure the other end of the tunnel for a route based VPN.

• By default, the Palo Alto device uses 3des/aes128 with sha1, PFS with DH
group 2.

• Note: If you want to change the PAN settings for IKE phase1 or phase2, go to
Network > Network Profiles
and edit either IKE Crypto > default for phase 1 proposals or IPSec Crypto >
default for phase 2 proposals.)

80 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
 Appendix 2:
Tunnel creation and
deletion causes in logs
Tunnel creation - possible causes (1 of 2)

Tunnel creation cause Comment

unspecified Unspecified reason
HA peer IPSec synced from HA peer
rasmgr IPSec tunnel installed by rasmgr
satd IPSec tunnel installed by satd (LSVPN)
test IKE-SA "test vpn ike-sa" CLI command
test IPSec-SA "test vpn ipsec-sa" CLI command
IKE-SA rekey local started IKE rekey

82 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Tunnel creation - possible causes (2 of 2)

Tunnel creation cause Comment

IPSec-SA rekey locally started IPSec rekey (IKE drived, IKEv2)
IKE responder IKE request initiated by peer
IPSec responder IPSec renegotiation initiated by peer
DPD timeout IKE renegotiation due to IKE sending failure
keyacquire outgoing user traffic (traffic-driven IKE negotiation)
keyexpire DP driven IPsec rekey (IKEv1, IKEv2)
path monitor outgoing monitoring traffic

83 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Tunnel deletion - possible causes (1 of 3)

Tunnel deletion cause Comment

unspecified unspecified reason
HA peer from HA peer
keymgr expire keymgr triggered key expiration
keymgr shutdown keymgr process exit
IKE cli clear manually cleared by ikemgr CLI
IKE config change config change detected by ikemgr
address change IKE gateway local address change
FDQN update IKE FQDN peer address change

84 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Tunnel deletion - possible causes (2 of 3)

Tunnel deletion cause Comment

HA ageout Ageout on passive when HA2 sync disabled
IKE SA delete IKE delete message received
IKE expire IKE expire
IPSec lifetime IPSec SA lifetime expired
IKE abort Default (catch-all) reason, IKEv2 only
IKE shutdown ikemgr exit
rasmgr rasgmr request
satd satd request

85 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.
Tunnel deletion - possible causes (3 of 3)

Tunnel deletion cause Comment

config update IPSEC tunnel config change
keymodify new IPSEC key installed, old key is removed
keyexpire DP triggered IPSEC key expiration
SSL start GP reconnect, clear previous tunnel
SSL stop GP disconnect
SSL keyexpire GP tunnel expires
tunnel cli clear IPSEC key manually cleared by CLI

86 | © 2019 Palo Alto Networks, Inc. All Rights Reserved. Confidential and Proprietary
DO NOT SHARE outside of WW Professional Services & Customer Support Organization.