How to properly and safely collect debugs on

an IOS router

How to properly and safely collect debugs on an IOS router

Prepared By Steve Holl, CCIE#22739

Purpose on page 2

Is running debugs safe to do on production routers? on page 2

I don't think I need this; I debug to a syslog server. on page 2

Why can't I just log to the monitor with 'terminal monitor'? on page 3

Recommended configuration on page

What do these commands do? on page 4

How do I run these debugs? on page 6

What debugs should I be running? on page 6

How do I collect the debugs? on page 7

How do I disable debugs? on page 8

What do I do if the router is too busy, and I can't pull debugs for an issue before the log
overwrites itself? on page 8

Postings may contain unverified user-created content and change frequently. The content is provided as-is and
is not warrantied by Cisco.
1
How to properly and safely collect debugs on an IOS router

Purpose:

It is quite often that someone is concerned about running debugs on a production gateway,
thinking that it may cause performance impact. The purpose of this document is to de-
mystify and clarify these conceptions.

If you don't want to understand the concepts and want to just know what commands to
enter, just use the 'Recommended Configuration' on page section below.

Is running debugs safe to do on production routers?

Debugs can be run safely in almost all environments where voice runs on an IOS router.
That being said, due diligence is recommended when turning debugs on in production, so
keep an eye on 'show processor cpu history' when enabling debugs one-at-a-time to ensure
there is minimal impact.

To prove the point that it is safe to run IOS debugs in production with this recommended
configuration, I am able to run 'debug all' (yes, that's every debug IOS is capable of running)
in my lab on a CME with 300 registered phones and only bring the CPU impact up by 40%,
when default rate limiting and queuing are enabled. I'd never recommend enabling 'debug
all' in any real-life circumstance since the majority of the messages will be rate limited and
dropped before being logged, but it demonstrates how effective disabling the console/
monitor and rate/queue limiting allow for verbose debugs to run stable on a busy router.
Typically only 10-20% CPU overhead is needed for even the most verbose debugs.

I don't think I need this; I debug to a syslog server.

In my experience, 80% of the time I receive debugs from a syslog server, they contain
dropped messages and render an accurate analysis impossible; one can observe this
behavior when sequence numbers are enabled on the IOS side. This is because syslog
is UDP by default, and the messages usually end up being rate-limited heavily. There are
ways to debug to the syslog more reliably (use TCP for syslog, or Reliable Delivery and
Filtering via BEEP) but those concepts are outside the scope of this document.

Postings may contain unverified user-created content and change frequently. The content is provided as-is and
is not warrantied by Cisco.
2
How to properly and safely collect debugs on an IOS router

Why can't I just log to the monitor with 'terminal monitor'?

We want to debug to the log to prevent any debug messages which are sent in bursts from
being dropped. For example, an H.245 debug may have 40-50 lines of debugs all arriving
at the same millisecond, and is usually too fast for the monitor to print out and large chuncks
of the debug output will end up missing from the screen output. By logging to the buffer, we
eliminate scenarios where the terminal monitor would have dropped messges.

Recommended Configuration:

Router(config)# service sequence-numbersRouter(config)# service timestamps debug

Router(config)# default logging rate-limit

Router(config)# default logging queue-limit

Router(config)# voice iec syslog

<Enable debugs, then wait for issue to occur.>

...

<Enable session capture to txt file in terminal program.>

Router# terminal length 0

Router# show logging

Postings may contain unverified user-created content and change frequently. The content is provided as-is and
is not warrantied by Cisco.
3
How to properly and safely collect debugs on an IOS router

What do these commands do?

Now we will run down through each of these commands and explain how they behave, since
scenarios may dictate some deviation from this template.

service timestamps debug datetime localtime msec- Ensures that local router time is written
to all debugs, with millisecond accuracy. This is useful for finding calls based on time.
Generally speaking, millisecond times allow for you to group debug lines into logical related
events when two lines occur within the same millisecond.

logging buffered 10000000 debug- Tells the router to send debugs to its local buffer log in
system memory. The buffer size is set in bytes, and is 10 MB here. The size of the buffer
which you may need depends upon call volume, duration of time the buffer needs to store,
system memory still available (leverage 'show memory statistic history' and 'show memory
summary' for this).

no logging console- By default, the router sends debugs to the console. In IOS, the console
has the highest priority out of any process. It also runs at very slow speeds (commonly
9600bps). Due to this, if debugs are sent to the router faster than the console speed, it can
starve console input, and/or cause the CPU to go to 100%.

To alleviate this behavior, when running any debugs in IOS, it is imperative that sending
debugs to the console is disabled by entering this command.

no logging monitor- This command prevents the router from sending debugs in real-time
to the router's VTY (telnet/SSH) session. Since we will be pulling debugs reactively, we
don't want anything to scroll in real-time. Also, the terminal monitor has a habit of dropping
messages if they arrive in bursts, like most voice debugs do.

default logging rate-limit- By default, the router does rate limit messages. This is usually
recommended to be left on to ensure router stability. If a TAC engineer suspects that the

Postings may contain unverified user-created content and change frequently. The content is provided as-is and
is not warrantied by Cisco.
4
How to properly and safely collect debugs on an IOS router

router is dropping debugs before they make it to the router's logging buffer, they may ask
this to be increased to a larger value or disabled. Note that changing this in environments
with high traffic volume may cause CPU instability, since it will ensure every debug message
makes it to the logging buffer.

default logging queue-limit- By default, the router does queue messages, as well. There
is a finite amount of memory the router will store in queue while waiting to be written to the
logging buffer. This is usually recommended to be left on to ensure router stability. If a
TAC engineer suspects that the router is dropping debugs before they make it to the router's
logging buffer, they may ask this to be increased to a larger value or disabled. Note that
changing this in busy environments may cause CPU instability for the same reasons as
mentioned previously.

service sequence-numbers- This command writes the sequence number of the debug in the
line. This is useful (essentially required) when sending to a syslog server, to identify if any
debug messages to the syslog server have been dropped in the network. The sequence
number will be the first item in the debug, before the timestamp and actual message. Note
that this is different from the timestamp/sequence number may write on the syslog log files,
if applicable.

001033: *Apr 27 14:29:25.867: %IPPHONE-6-REG_ALARM: NAME=SEP000A10000075 Load=P00308000500 Parms=Status/IPaddr

Note that sequence numbers are written after

rate-limiting, so they won't be useful to identify if IOS is dropping debugs before writing them
to the buffer log.

voice iec syslog- This command prints out extra information in scenarios where the router is
the origin of a disconnect. This is specific to voice, and is useful to TAC engineers.

%VOICE_IEC-3-GW: C SCRIPTS: Internal Error (Incompatible protocols): IEC=1.1.47.11.23.0 on callID 31102

The ouput can be decoded with:

Postings may contain unverified user-created content and change frequently. The content is provided as-is and
is not warrantied by Cisco.
5
How to properly and safely collect debugs on an IOS router

Router# show voice iec description 1.1.47.11.23.0

IEC Version: 1 Entity: 1 (Gateway) Category: 47 (no resource (47)) Subsystem: 11 (C SCRIPTS) Error: 23 (Incompatible
protocols) Diagnostic Code: 0

How do I run these debugs?

Once you have the debugs setup, enable the relevant debugs which have been requested to
you by the TAC engineer, CSC member, or via the Multiservice Voice Debug Lookup tool.

What debugs should I be running?

While we are talking about how to collect debugs, you may wonder what debugs you should
collect to troubleshoot your call issues. We have a very useful tool to assist you with this,
called the Multiservice Voice Debug Lookup.

Personally, here is what I like to collect as a starting point for the common call failures
across IOS gateways:

H.323

debug voip ccapi inout

debug h225 asn1

debug h245 asn1

debug cch323 all

debug ip tcp transaction

Postings may contain unverified user-created content and change frequently. The content is provided as-is and
is not warrantied by Cisco.
6
How to properly and safely collect debugs on an IOS router

SIP

debug voip ccapi inout

debug ccsip messages

debug voip rtp session named-event

MGCP

debug voip ccapi inout

debug mgcp packet

debug ip tcp transaction

<Be sure to enable appropriate POTS debugs, too.>

ISDN

debug voip ccapi inout

debug isdn q931

Analog or Non-ISDN POTS

debug voip ccapi inout

debug vpm signal

How do I collect the debugs?

Once you have the buffer configured and debugs enabled, it will write all output to the buffer.
The buffer is a rolling buffer, so when you reach the limit of the defined buffer size, then the

Postings may contain unverified user-created content and change frequently. The content is provided as-is and
is not warrantied by Cisco.
7
How to properly and safely collect debugs on an IOS router

oldest information at the top of the buffer is removed to make way for the newest information
to be added at the bottom of the log.

After the issue occurs, issue 'terminal length 0' to prevent having to hit <enter> or <space>
every page. Then, issue 'show logging' and the current buffer content will dump to the
screen. Telnet is preferred over console connections due to faster transmission capability.
Ideally, have your terminal application that runs on your desktop configured to log all of this
session's terminal output to a local .txt file. My favorite applications for this are PuTTy or
SecureCRT. Mac's Terminal application handles large buffers very well, and select all/copy/
paste into notepad after the log is dumped to the screen work best for me there.

How do I disable debugs?

After debugs have been collected for an issue, you can disable them all with:

Router# undebug all

This isn't necessary, but if you want to restore synchronous logging to the console and VTY
sessions, enable them again with:

Router(config)# logging consoleRouter(config)# logging monitor

What do I do if the router is too busy, and I can't pull debugs for an issue before the log overwrites
itself?

There are some advanced techniques we have in TAC by using TCL and EEM to leverage
this issue, if necessary. Consult TAC if you feel this step is necessary.

Postings may contain unverified user-created content and change frequently. The content is provided as-is and
is not warrantied by Cisco.
8