01.11.

2017

DNS Troubleshooting

PRESENTED BY:
Introduction to BIGIP DNS
DNS Hierarchy

Sample of a Zone File

$TTL 86400 ; 24 hours could have been written as 24h or 1d
; $TTL used for all RRs without explicit TTL value
$ORIGIN example.com.
@ 1D IN SOA ns1.example.com. hostmaster.example.com. (
2002022401 ; serial
3H ; refresh
15 ; retry
1w ; expire
3h ; nxdomain ttl
)

IN NS ns1.example.com. ; in the domain

IN NS ns2.smokeyjoe.com. ; external to domain
IN MX 10 mail.another.com. ; external mail provider

; server host definitions

ns1 IN A 192.168.0.1 ;name server definition
www IN A 192.168.0.2 ;web server definition
ftp IN CNAME www.example.com. ;ftp server definition

; non server domain hosts

bill IN A 192.168.0.3
fred IN A 192.168.0.4
BIG-IP DNS (formerly Global Traffic Manager or GTM)

It’s a load balancer for DNS queries (caching, traffic management), decision making to load
balance between datacenters.

Terms
• Wide IP - Maps FQDN to one or more pools of virtual servers that host content of the domain.
It responds to listener requests. This will auto create a zone that matches the Wide IP

• Server object - Server defined in BIG-IP DNS is either a BIG-IP or other 3rd party system
responsible for owning one or more virtual server service.
i. BIGIP devices (LTM/ASM/APM/etc) – Standalone/Redudant-Pair
ii. Generic LB/Host (3rd party system) – Citrix LB, Cisco CSS, Centos machine

• Listener - BIG-IP uses TCP/UDP listeners to respond to DNS queries.

• Pool - In BIG-IP DNS a pool contains one or more virtual servers.

BIGIP Resolution
Hierarchy
Listener
Wide IP
- Maps FQDN to set of Virtual Server that host the domain content
- Uses Pool to organize Virtual Server
iQuery communication and troubleshooting
iQuery

Purpose
Establishing communication between GTM and other system to be in a same Sync Group

Requirement
1) DNS members must be running on same version (source: K13703)
i. BIG-IP DNS synchronization group communication
ii. Monitored BIG-IP systems must run the same or newer big3d version as the DNS / GTM
that are monitoring them
2) Sync parameter must defined properly
3) NTP in sync
4) Port lockdown allowing 4353 and 443
5) Compatible big3d version
Virtual Server/Link Autodiscovery (K13994)

1) Virtual server and link auto-discovery feature allows:

2) BIG-IP DNS and BIG-IP Link Controller systems to automatically discover virtual servers and
links that are associated with defined BIG-IP systems.
3) Uses iQuery protocol to automatically discover objects on the remote BIG-IP system if enabled.

• The BIG-IP DNS configuration contains one or more BIG-IP server objects
• TCP port 4353 is allowed between the BIG-IP DNS system and target BIG-IP systems
• The target BIG-IP system's virtual server addresses must not employ network address
translation

Important Note:
K9138: The BIG-IP GTM system disables virtual server auto-discovery for BIG-IP systems
that use translated virtual server addresses

K14106: Troubleshooting virtual server and link auto-discovery (11.x - 13.x)

- telnet <remote_bigip_selfip> 4353
- iqdump
K13312: Overview of the BIG-IP GTM big3d_install, bigip_add, and
gtm_add utilities (11.x - 13.x)
Requirement of iQuery
- TCP port 4353
- SSH port 22 (for initial certificate transfer/copy)

bigip_add
- Exchanges iQuery SSL certificate with remove BIGIP
- Append Local GTM system’s certificate to remote BIGIP authorized cert (stored in /config/big3d/client.crt)
- Append remote iQuery cert to local GTM list of authenticate iQuery (stored in .config/gtm/server.crt)

bigip_install (K13703)
- Similar to bigip_add but install the big3d version if its older than the local GTM F5 system
- To check: run:
# /usr/sbin/big3d -v (default big3d agent)
#/shared/bin/big3d –v (executable file)

gtm_add
- Integrate new GTM system into existing sync group
- Replace current config (bigip_gtm.conf, named.conf and the name zone files)
Troubleshoot iQuery
1. Config Utility
- Check the status of the server object (Global Traffic -> Server -> Server List
- iQuery Stat (Statistic -> Global Traffic -> Statistic Type -> iQuery)
- Summary Stat (Statistic -> Global Traffic)

2. TMSH
- Server (tmsh show /gtm server all)
- iQuery (tmsh show /gtm iquery all)
- GTM (tmsh show /gtm)

3. /var/log/gtm

4. Verify the big3d version

# /usr/sbin/big3d -v (default big3d agent)
#/shared/bin/big3d –v (executable file)

5. Check the iQuery processes

# netstat –nap | grep 4353
Cont*
6. Iqdump utility (run from the GTM)

iqdump 10.10.10.20 <sync_group_name>

• If the iQuery channel is not established, it will prompt error

46947856243768:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:1168:

• If the iQuery channel is established, iqdump returns XML similar to the following example:

<!-- Local hostname: lc1.example.com -->

<!-- Connected to big3d at: ::ffff:10.10.10.10:4353 -->
<!-- Subscribing to syncgroup: default -->
<!-- Tue May 6 09:55:43 2014 -->
<xml_connection>
<version>11.5.1</version>
<big3d>big3d Version 11.5.1.0.0.110</big3d>

7. Verify device Certificate

openssl x509 -noout -text -in /config/httpd/conf/ssl.crt/server.crt
- Verify the certificate validity date and confirm whether the certificate is expired.
- If necessary, renew the certificate. To do so, refer to K6353: Updating an SSL device certificate on a BIG-IP system.
Prober pool
Collection of device that perform monitor probes of servers to gather data about the health and performance
of the resources on the servers

By default, the members of a GTM sync group dynamically determine the best BIG-IP device within the sync
group configuration to use as the prober for the non-BIG-IP device server objects. Devices defined within the
same data center as the server object to be probed are preferred. If no local prober is available, a remote
prober is used.
Debugging
To enable debugging

tmsh modify /sys db log.gtm.level value debug

tmsh modify /sys db log.big3d.level value debug
tmsh modify /sys db gtm.debugprobelogging value enable

To disable debugging

tmsh modify /sys db log.gtm.level value debug

tmsh modify /sys db log.big3d.level value debug
tmsh modify /sys db gtm.debugprobelogging value enable

Collect qkview and full tar ball for review.

#qkview –s0

#tar -czvpf /var/tmp/$HOSTNAME-logs.tgz /var/log

Configure Decision Logging
https://devcentral.f5.com/articles/configuring-decision-logging-for-the-f5-big-ip-global-traffic-
manager

• Modify WIP(s) to enable LB Decision Logging

• Log Publisher
• DNS Logging Profile
• Custom DNS Profile and attach the logging profile
• Apply the DNS Profile to the Listener
DNS Express
DNS Express
Alows the BIG-IP to perform zone transfers from multiple primary DNS servers that are responsible
for different zones, perform a zone transfer from the local BIND server on the BIG-IP, and serve
DNS records faster than the primary DNS servers and the local BIND server.

• Perform zone transfers from multiple primary DNS servers that are responsible for different
zones.
• Perform a zone transfer from the local BIND server on the BIG-IP system.
• Serve DNS records faster than the primary DNS servers
K15298: Overview of the dnsxdump utility
You can use the dnsxdump utility to view the DNS Express database information, which includes zone
information and statistics.

• The DB Dump section of the dnsxdump utility output displays the zone information for all configured DNS
Express zones.
• The DB Stats section of the dnsxdump utility output displays a cumulative count of records for all configured
DNS Express zones.

dnsxdump > /var/tmp/my_zones.txt

Zonerunner
Managing the BIG-IP BIND configuration file

ZoneRunner utility is used to manage both DNS zone files and the BIND configuration file on the BIG-IP
GTM system

• Import and transfer DNS zone files

• Manage zone resource records
• Manage views
• Manage a local nameserver and the associated configuration file, named.conf
• Transfer zone files to a nameserver
• Import only primary zone files from a nameserver
Cont*
By default, BIG-IP GTM is configured to secure BIND to not allow zone transfers except from the
localhost. Modify the allow-transfer statement to include the IP address of the GTM. You can modify the
following allow-transfer statement to use the IP address of the GTM.

DNS > Zones > ZoneRunner > named Configuration.

allow-transfer { localhost; 192.168.10.105; }

To verify zone transfers are working properly

# dig @<IP address> es.net. axfr

Directory of the zone file stored

# cd /var/named/config/namedb/

Check the named configuration

K7032: Freezing zone files to allow manual update to ZoneRunner-
managed zone files
All changes made to a zone using dynamic update are written to the zone's journal file.

When the BIG-IP DNS system restarts after a shutdown, the system replays the journal file to incorporate any
updates that took place after the last zone file update into the zone.

Dynamic update periodically flushes the complete contents of the updated zone to its zone file and
automatically deletes the journal file.

i. cd /var/named/config/namedb
ii. cp <zone_filename> <zone_filename>.original
iii. bigstart stop zrd
iv. rndc freeze <zone name> <class> <view>
v. Manually edit the zone for any changes
vi. rndc sync -clean
vii. Run the named-checkzone command to check the file for any syntax errors
• named-checkzone askf5.net db.external.askf5.net
• named-checkconf -t /var/named -z -j /config/named.conf
viii. rndc thaw <zone name> <class> <view>
ix. bigstart start zrd

.
Behaviour of zrd
- When a new dns express zone is added , it writes the data to zxfrd.bin
- It then copies from zxfrd.bin to zxfrd-tmp.bin (15sec timer)
- Rename the zxfrd-tmp.bin to tmmdns.bin (database)

TMM then reload the database from tmmdns.bin

- For VIPRION, csyncd monitors tmmdns.bin for any changes
- Csyncd trigger tmm to reload on primary blades then populate to other blades

# bigstart stop
# rm -rf /var/db/{tmmdns.bin,zxfrd.bin}
# bigstart start

tmsh modify sys db log.zxfrd.level value debug

Collect qkview and full tar ball for review.

#qkview –s0

#tar -czvpf /var/tmp/$HOSTNAME-logs.tgz /var/log