AX Troubleshooting
Section 9
1
Section objectives
Learn troubleshooting techniques and tools
Fix broken configuration
2
Troubleshooting philosophy
Bottom-up approach
Basic Network – L1 / L2 / L3
Server Status
TCP / UDP - L4
HTTP ( HTTPS ) - L7
Persistence L4/7
Performance
3
Problem identification
Description of the issue
Type of service
Identify the VIP / V-Port that has the issue
Identify affected features
Frequency
Determine frequency of problem occurrence
Scale
Check whether it happens to every client or only certain clients
Find the possible layer of issue & start there
4
Logs
Before starting the investigation look at the logs
AX logs many informational, warning, and error messages. The log is the first place to check
when experiencing any issues.
Port/Interface up/down messages
L2 loop detection warnings
Unicast/Multicast/Broadcast packet limit warnings
MAC address movement warnings
Duplicate IP warnings
Server & service port up/down messages
Application-specific error messages: SLB, PBSLB, HTTP, HA, and so on
5
Physical layer
Typical Symptoms
Slow Network ( High Latency )
Collisions
CRC Errors
Half Duplex
Speed
Interface Up & Down
Interface flapping
Any indication from show log
Configuration / Status on next hop switch / router
Any specific configuration on next hop’s interface
6
L2/3
Is VIP reachable via ping?
Look up the configured routes, learned MAC & ARP
Is IP checksum correct on every packet?
Check the access list
Is Ping always working?
Any flapping in MAC – interface
Any flapping in ARP – MAC
What’s the MAC / ARP learned on next hop?
Is it outbound traffic initiated by backend server?
Check the access list
Check the IP NAT configuration & usage
7
Server status
Health Check Results?
Health-check type, health-check up / down reason
Show log (log entries on up / down events)
8
TCP
Test TCP access
Use telnet [VIP] [v-port] to try establishing connection
Packet trace from client side
After sending SYN to VIP, any SYN-ACK / RST from VIP?
Is the connection terminated prematurely?
Responses from server & responses from VIP?
Do all the packets have proper IPs?
Is the TCP checksum correct?
Packet trace from server side (or AX)
Is AX able to establish connection to server?
Use axdebug
9
UDP
Packet trace from both client side & server side
What are the source and destination protocol ports?
Is server able to receive client’s packet?
Is client able to receive server’s packet?
Do all the packets have proper IPs?
Is the UDP checksum correct?
Use axdebug
10
HTTP
What are the L7 features enabled
Try without the advanced features first (compression, connection reuse, and so on)
Packet trace from client side & server side
Is server receiving the request sent by the AX?
Any standard HTTP header missing? (host, method, … and so on)
Do all of the HTTP headers have desired values?
Response Code from server’s response?
Size of request / response payload?
Is it taking a long time to process the request?
What are the cookies?
Use axdebug
11
HTTPS
Enabled Features?
Client SSL?
Server SSL?
Packet trace from client side & server side
Is client able to finish SSL Handshake with VIP?
Is AX able to finish SSL Handshake with server?
Any issues pertaining to redirect?
Decrypted trace
Are there any absolute links in Javascripts / Links / Images (http://xxx)?
Use axdebug
12
Performance
When?
When most of the traffic is working fine and you are trying to grab a quick picture of
everything
Performance Issue
Latency
13
Useful commands
Network
show int eth 1
show arp
show mac
show access-list
show ip route
show health stat
SLB
show slb […]
Performance
show memory system
show cpu
14
AXdebug
AXdebug
Captured files are in pcap format (Wireshark / tcpdump)
Able to see every detail of the packets the AX receives & sends
AXdebug is session based
If one pkt matches filter, dump all the following pkts in the same session
15
AXdebug: filters
Prepare capture
Capture file name
Filters
Multiple filters are “OR”-ed
AX# axdebug
AX(axdebug)# filter 1
AX(axdebug-filter:1)# ip 1.2.3.4 /32
AX(axdebug)# capture save file_name
Stop AXdebug trace
AX# no axdebug
Export AXdebug trace
16
Lab
Use session-control and packet-level CLI tools
17
Course map
Section 0: Course Introduction
Section 1: Load Balancing Concepts
Section 2: AX Management
Section 3: FTP
Section 4: HTTP
Section 5: HTTPS
Section 6: AX Acceleration
Section 7: AX Security
Section 8: High Availability
Section 9: AX Troubleshooting
Section 10: aFleX
18
