BRKTRS-2034 Smart Licensing Troubleshooting and Case Studies
BRKTRS-2034 Smart Licensing Troubleshooting and Case Studies
Smart Licensing
Troubleshooting and Case
Studies
Shashank Singh
Principal Engineer
Cisco Customer Experience
@shashankcisco
BRKTRS-2034
#CiscoLive
Cisco Webex App
Questions?
Use Cisco Webex App to chat
with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install the Webex App or go directly to the Webex space Enter your personal notes here
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
• Introduction
• Troubleshooting Smart Licensing
Agenda • Case Studies
• Appendix
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Introduction
Smart Licensing Using Policy
• Smart Licensing using Policy first
Cisco
implemented in IOSXE version 17.3.2. Factory-
reporting Smart
• What is different from old smart Account
licensing approach?
• How are new commands in Smart Direct-
Online/Offline
reporting Supported
Licensing using Policy interpreted? reporting
Reporting via
CSLU/SSM Cisco Smart
On Prem Licensing Utility
(CSLU), SSM on
prem or other
RUM Reports
reporting methods
RUM Reports
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Difference between SL and SLP
Key Differences
Legacy Smart License (SL) Smart Licensing using Policy (SLP)
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Smart Licensing using Policy – Design Options
Agenda DIY?
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Troubleshooting
Smart Licensing
Smart Licensing Using Policy
Smart Licensing using Policy on Catalyst Switching Migrate Catalyst License to Smart Licensing Using
Platforms Policy
https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9300-series-
switches/216545-smart-licensing-using-policy-on-catalyst.html https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9300-series-
switches/216520-migrate-catalyst-license-to-smart-licens.html
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
SLP Connected directly to CSSM – 1/3
Direct Connect - Smart Transport
Direct Connect - Call-home Transport
Step 1. When using direct connection to Cisco Smart Software Manager (CSSM) on a Switch supporting
Smart Licensing using Policy (IOSXE 17.3.2 or newer) these steps help establish a connection with the portal and get the
licenses:
Switch#ping smartreceiver.cisco.com
Switch#telnet smartreceiver.cisco.com 443
• If you can’t reach the server by the domain name you can configure a DNS static entry, for example:
• In order to obtain the IP address of smartreceiver.cisco.com, use nslookup or similar utility. Currently, it’s being load
balanced between these IP addresses:
The call-home transport can still be used for Smart Licensing Using Policy. However it’s a legacy method and it is
recommended to use smart transport instead.
Switch(config)# license smart transport smart
Switch(config)# license smart url default
If communication with CSSM should occur in a particular VRF, assign specific source interface as needed:
In case you are using a PROXY for internet reachability please configure the following commands:
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Troubleshoot trust
SLP Connected directly to CSSM – 3/3 installation/sync
Switch will establish trust with CSSM using the token obtained from the
Virtual Account on CSSM.
Switch# license smart trust idtoken <TOKEN> all force
After a few minutes at the very bottom of the ‘show license status’
output you should see the trust code was installed
Switch#show license status
<...>
Trust Code Installed: Feb 25 18:37:51 2021 UTC
After a few minutes in the ‘show license status’ output you should see the the “Last report push” and “Last ACK
received” timestamps
Switch#show license status All licenses used on the Switch will
<...> be included in the license usage
Usage Reporting: report. These licenses are displayed
Last ACK received: Mar 27 22:33:28 2021 UTC in ‘show license summary’.
Next ACK deadline: Jun 25 22:33:28 2021 UTC
Reporting push interval: 30 days
Next ACK push check: <none> In order to push the report to CSSM,
Next report push: Apr 26 22:29:28 2021 UTC issue “license smart sync all”
Last report push: Mar 27 22:29:28 2021 UTC command.
Last report file write: <none>
BRKTRS-2034 13
Troubleshooting Scenarios – Product Instance
1. Check current license status 5. Check for communication issues 8.Check report frequency
Check the output of “show license status”. Syslog will appear if there are In the output of ‘show license tech
communication issues explaining further support’, under section of ‘Usage
2.Check policy installation status details. Reporting:”,
From the output of ‘show license tech support’, next timestamp of reporting is displayed.
under “Policy” section, data should be 6. Check if authcode is installed
displayed. In the output of ‘show license tech Next report push: Oct 19
support’, under section “Authorizations:”, 21:52:13 2021 UTC
Policy: info of auth code installation is displayed.
Policy in use: Installed On 9.Check ACK deadline.
Oct 08 19:07:28 2021 UTC Authorization type: In the output of ‘show license tech
Policy name: SLE Policy SMART AUTHORIZATION INSTALLED support’, under section of ‘Usage
Reporting:”, time of ACK deadline is
7.Check if reporting is required displayed.
3.Check trust is installed/established In the output of ‘show license tech
From the output of ‘show license tech support’, support’, under section “Usage Next ACK deadline:
data should be displayed at the bottom. Reporting:” data is displayed reflecting Nov 18 21:45:17 2021 UTC
reporting status.
Local Product-instance: 10.Check if ACK was received.
P:CSR1000V,S:9R0LNVXST8K, If no reporting required: In the output of ‘show license tech
state[2], Trust Data INSTALLED Reporting push interval: 0 support’, under section of ‘Usage
Overall Trust: INSTALLED (2) Reporting:”, info about ACK is displayed
If any reporting required:
4.Check Purchase info Reporting push interval: 30 Last ACK received:
In the output of ‘show license tech support’, days State(4) InPolicy(30) Oct 19 21:45:17 2021 UTC
purchase info if available is displayed under
#CiscoLive 14
section “Purchased Licenses:”. BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public
Smart Licensing – Cisco SL Utility
Agenda
CSLU using PUSH Mode
CSLU Auto Discovery
CSLU using Disconnected Mode
C:\Users\<user-name>\AppData\Roaming\CSLU\var\logs
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Troubleshooting Scenarios – CSLU
CSLU can’t pull report from product-instance CSLU has intermittent connection to CSSM
• Can CSLU ping the product-instance? • Check log files (Both product-instance and
CSLU)
• Does product-instance have HTTPS
configured? • On Product-instance check ‘show license
history message’ output
• Is product-instance’s password properly
set in CSLU? • “Download for Cisco” can be used to
manually download reports when
• Put product-instance’s IP address in connection is down.
browser. If HTTPS is properly configured,
its management page will load. If this fails, • If CSLU has proper connectivity to CSSM,
CSLU won’t be able to pull reports. “Download for Cisco” should return the
following error:
• Check alerts and logs files from the CSLU ‘there won’t be any files to download,
and the product-instance . they were all set to CSSM’
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Additional Tips
Scenario: Running into SLP issues.
Scenario: Policy failure due to time mismatch. Switch What data to collect from the switch ?
may not register because a policy was installed, but an
error was detected while parsing the policy code, and
installation failed. • show license all
• show license tech support
A signature mismatch: This means that the system clock • show license eventlog
is not accurate. • show license history message
A timestamp mismatch: This means the system clock on • request platform software trace rotate all
the product instance is not synchronized with CSSM. • show logging process iosrp internal start last boot to-
file bootflash:<file-name>
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Case Studies
Switch Registration is Failing
Server Reachability Issues - 1/2
Switch License Authorization "Failure reason: Fail to send out Call Home HTTP message."
Behavior
Switch will not register and the DLC process can not be done
Logs
*May 27 19:45:49.721: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart
Software Manager or satellite : Fail to send out Call Home HTTP message.
*May 27 19:45:49.722: %SMART_LIC-3-AUTH_RENEW_FAILED: Authorization renewal with the Cisco
Smart Software Manager or satellite : Communication message send error for udi PID:WS-C3850-24P,SN:xxxxxxxxxx
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Switch Registration is Failing
Server Reachability Issues - 2/2
Switch License Authorization "Failure reason: Fail to send out Call Home HTTP message."
What to look into?
• Verify HTTP configuration on the switch is correct. show run | section http
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
License Authorization Renewal Failure
Logs
%SMART_LIC-3-AUTH_RENEW_FAILED: Authorization renewal with the Cisco Smart Software Manager (CSSM) :
Error received from Smart Software Manager: Data and signature do not match for udi PID:C9300-
24P,SN:XXXXXXXXX
%SMART_LIC-3-AUTH_RENEW_FAILED: Authorization renewal with the Cisco Smart Software Manager (CSSM) : NULL
for udi 'nullPtr' *Nov 30 22:38:24.137: %SMART_LIC-5-END_POINT_RESET: End Point list reset
%SMART_LIC-3-AUTH_RENEW_FAILED: Authorization renewal with the Cisco Smart Software Manager (CSSM) :
Agent received a failure status in a response message. Please check the Agent log file for the detailed
message. for udi PID:C9300-24P,SN: XXXXXXXXX
Remedy
• The error message indicates an issue with license authorization and the UDI for this switch on the
CSSM side.
• Re-authorize the license by generating a new token and registering the Switch again manually.
#license smart deregister
#license smart register idtoken <insert token> force
#license smart renew auth
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
QuoVadis Certificate Expiration FN – 72323
Field Notice
https://www.cisco.com/c/en/us/support/docs/field-notices/723/fn72323.html
• For all versions of IOS-XE Products, some Secure Sockets Link (SSL) certificates issued from the
QuoVadis root certificate authority (CA) trust chain before 2021-03-31 cannot be renewed from
this CA.
• As a result, certain functions such as Smart Licensing and Smart Call Home will switch to a new
trust certificate (IdenTrust) before QuoVadis certification expires.
Logs
%SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart Software Manager (CSSM) : Fail to send out Call Home HTTP
message.
%SMART_LIC-3-AUTH_RENEW_FAILED: Authorization renewal with the Cisco Smart Software Manager (CSSM) : Communication message send error
for udi <>
%SMART_LIC-3-AGENT_REG_FAILED: Smart Agent for Licensing Registration with the Cisco Smart Software Manager (CSSM) failed: Fail to
send out Call Home HTTP message.
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
QuoVadis Certificate Expiration FN – 72323
Remedy
• For Cisco IOS XE-based products, upgrade the software to any of the fixed releases - 16.12.6,
17.3.4, 17.4.2, 17.5.1, 17.6.1, 17.7.1 or later to resolve the root CA certificate issue for affected
platforms.
• Manual import of the IdenTrust Commercial Root CA 1 from the product's trust store.
Use a simple text editor and copy the certificate (without the BEGIN CERTIFICATE
and END CERTIFICATE lines). CA Certificate
Status: Available
Switch#config t Certificate Serial Number (hex): 0A01800000002
Certificate Usage: Signature
Enter the crypto pki trustpool import terminal command.
Issuer:
cn=IdenTrust Commercial Root CA 1
Switch(config)#crypto pki trustpool import terminal o=IdenTrust
% Enter PEM-formatted CA certificate. c=US
% End with a blank line or "quit" on a line by itself. Subject:
MIIFYDCCA0igAwIBAgIQCgFCgAAAAUUjyES1AAAAAjANBgkqhkiG9w0BAQsFADBK cn=IdenTrust Commercial Root CA 1
MQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MScwJQYDVQQDEx5JZGVu o=IdenTrust
VHJ1c3QgQ29tbWVyY2lhbCBSb290IENBIDEwHhcNMTQwMTE2MTgxMjIzWhcNMzQw c=US
Validity Date:
<snip>
start date: 02:12:23 CST Jan 17 2014
"% PEM files import succeeded". end date: 02:12:23 CST Jan 17 2034
Enter exit. Associated Trustpoints: Trustpool
Trustpool: Downloaded
Switch(config)#exit
Conditions
Smart Licensing and RUM reports are used in the Switch. The CPU usage triggers when the RUM report list is used.
High CPU utilization due to RUM reports - 2/2
Verification
C9300#show license tech support | include RUM reports
Send Utility RUM reports: Feb 22 16:04:53 2021 MET (34 minutes, 45 seconds remaining)
Save unreported RUM Reports: Feb 22 16:21:26 2021 MET (51 minutes, 18 seconds remaining)
Process Utility RUM reports: Feb 22 17:04:38 2021 MET (1 hours, 34 minutes, 30 seconds remaining)
Workaround
Delete the RUM reports list.
C9300#license smart factory reset
%Warning: reload required after "license smart factory reset" command
Alternatively, using the offline/reservation method can mitigate this as usage reports are
manually handled. Details for this can be found here:
https://www.cisco.com/c/en/us/td/docs/routers/sl_using_policy/b-sl-using-
policy/how_to_configure_workflows.html#Cisco_Concept.dita_7057e18c-3c69-4d91-841b-
0b5beb7a2d88
Scenario: “Operation not supported because the agent is running in Permanent License
Reservation mode”
Behavior:
Running “license smart register idtoken <x>” on the Switch returns a message stating it is in
permanent license reservation mode causing the token not to loaded
Solution:
Disable license reservation mode:
switch#configure terminal
switch(config)#no license smart reservation
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
For Your
Scenario: Switch was running IPBase when it was converted instead of desired IPServices
level
Behavior:
Switch goes back to “Out-Of-Compliance” once the license is changed to IPServices
Solution:
Reach out to Cisco Global Licensing Operations for the desired license to be manually
deposited in the corresponding Smart Account
• Contact by Phone – 1(800)553-2447 / 1(408)526-7209
• Open a Case Online - https://www.cisco.com/c/en/us/support/index.html
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
For Your
Scenario: Switch was registered with a token generated from the incorrect Virtual Account.
Behavior:
Switch shows “Out-Of-Compliance” even though the license was converted
Typically seen in the following scenarios:
• When switch is deregistered and registered later
• When staging and/or replacing a failing switch
Solution:
<next slide>
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
License Deposited in Wrong Virtual Account – For Your
2/2
Reference
Solution:
Navigate to CSSM > Inventory > Licenses > Select License > Actions > Transfer…
Example Account
Enterprise Access
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Related Sessions@Cisco Live 2022
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Pay for Learning with
Cisco Learning Credits
Cisco Learning and Certifications (CLCs) are prepaid training
vouchers redeemed directly
From technology training and team development to Cisco certifications and learning with Cisco.
plans, let us help you empower your business and career. www.cisco.com/go/certs
Here at the event? Visit us at The Learning and Certifications lounge at the World of Solutions
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
• Visit the Cisco Showcase
for related demos
BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Smart Licensing using Policy - Closing
Comments
Customer!
#CiscoLive BRKTRS-2034 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Thank you
#CiscoLive
#CiscoLive