Nokia Cloud Signaling Director

Release 20.5

Monitoring and Troubleshooting Guide

id9YZ-09148-MT11-PCZZA
Issue: 2.0
The information in this document applies solely to the hardware/software product (“Product”) specified
herein, and only as specified herein.

This document is intended for use by Nokia's customers (“You”) only, and it may not be used except for
the purposes defined in the agreement between You and Nokia (“Agreement”) under which this document
is distributed. No part of this document may be used, copied, reproduced, modified or transmitted in any
form or means without the prior written permission of Nokia. If you have not entered into an Agreement
applicable to the Product, or if that Agreement has expired or has been terminated, You may not use
this document in any manner and You are obliged to return it to Nokia and destroy or delete any copies
thereof.

The document has been prepared to be used by professional and properly trained personnel, and
You assume full responsibility when using it. Nokia welcome Your comments as part of the process of
continuous development and improvement of the documentation.

This document and its contents are provided as a convenience to You. Any information or statements
concerning the suitability, capacity, fitness for purpose or performance of the Product are given solely
on an “as is” and “as available” basis in this document, and Nokia reserves the right to change any such
information and statements without notice. Nokia has made all reasonable efforts to ensure that the
content of this document is adequate and free of material errors and omissions, and Nokia will correct
errors that You identify in this document. But, Nokia' total liability for any errors in the document is strictly
limited to the correction of such error(s). Nokia does not warrant that the use of the software in the Product
will be uninterrupted or error-free.

NO WA RR AN TY O F AN Y K I ND , EI T H ER EX PR E S S O R I M P L I E D , I N C L U D I N G B U T
NOT LI M I T ED T O AN Y W AR RA NTY O F AV AI L A B I L I T Y , A C C U R A C Y , R E L I A B I L I T Y ,
TITL E, NO N- I NFR I NG E M EN T, M ER CH AN T AB I L I T Y O R F I T N E S S F O R A P A R T I C U L A R
PURPO S E, I S M AD E I N RE LATI O N T O T HE CO N T E N T O F T H I S D O C U M E N T . I N N O
EVENT W I LL NO KI A BE LI AB LE FO R AN Y D AM A G E S , I N C L U D I N G B U T N O T L I M I T E D
TO SPECI AL, D I RE CT, I N DI R EC T, I NC I DE NT A L O R C O N S E Q U E N T I A L O R A N Y
L OSSES, SUCH AS BU T NO T LI M I T E D T O L O S S O F P R O F I T , R E V E N U E , B U S I N E S S
INTERRUPT I O N, B US I NE SS O PP O RT U NI T Y OR D A T A T H A T M A Y A R I S E F R O M T H E
USE O F TH I S D O CU M EN T O R TH E I N F O RM A T I O N I N I T , E V E N I N T H E C A S E O F
ERRO RS I N O R O M I SS I O NS FRO M T H I S D O CU M E N T O R I T S C O N T E N T .

This document is Nokia’ proprietary and confidential information, which may not be distributed or disclosed
to any third parties without the prior written consent of Nokia.

Nokia is a registered trademark of Nokia Corporation. Other product names mentioned in this document
may be trademarks of their respective owners, and they are mentioned for identification purposes only.

Copyright © 2020 Nokia. All rights reserved.

Important Notice on Product Safety

This product may present safety risks due to laser, electricity, heat, and other sources of danger.

Only trained and qualified personnel may install, operate, maintain or otherwise handle this product and
only after having carefully read the safety information applicable to this product.

The safety information is provided in the Safety Information section in the “Legal, Safety and
Environmental Information” part of this document or documentation set.

Nokia is continually striving to reduce the adverse environmental effects of its products and services. We
would like to encourage you as our customers and users to join us in working towards a cleaner, safer
environment. Please recycle product packaging and follow the recommendations for power use and proper
disposal of our products and their components.

If you should have questions regarding our Environmental Policy or any of the environmental services we
offer, please contact us at Nokia for any additional information.
Monitoring and Troubleshooting Guide

Contents
1 About this document...................................................................................................................................... 7
1.1 Reason for new issue.............................................................................................................................. 7
1.2 Intended Audience....................................................................................................................................8
1.3 Conventions used..................................................................................................................................... 8
1.4 Related documentation............................................................................................................................. 8
1.5 Document support.................................................................................................................................... 9
1.6 Technical support..................................................................................................................................... 9
1.7 How to order.............................................................................................................................................9
1.8 How to comment...................................................................................................................................... 9

2 Troubleshooting............................................................................................................................................ 10
2.1 Troubleshooting CBAM...........................................................................................................................10
2.2 Troubleshooting during Installation.........................................................................................................10
2.3 Troubleshooting during Upgrade............................................................................................................ 12
2.4 Troubleshooting Post Upgrade...............................................................................................................13
2.4.1 Troubleshooting during LCM operations........................................................................................ 13
2.5 Troubleshoot Service Manager.............................................................................................................. 15
2.5.1 SM GUI failure scenarios...............................................................................................................16
2.5.2 ME import failure scenarios........................................................................................................... 16
2.6 Troubleshooting CSD............................................................................................................................. 18
2.6.1 Application issues and errors.........................................................................................................18
2.6.2 To troubleshoot healthMachines or vnfcMapList........................................................................... 19
2.6.3 To troubleshoot issues related to CSD application processes.......................................................20
2.6.4 To troubleshoot an interface connection........................................................................................20
2.6.5 To troubleshoot Diameter peer and route issues.......................................................................... 21
2.6.6 To manually disconnect a peer......................................................................................................24
2.6.7 Routing and Peering...................................................................................................................... 24
2.6.8 Rules Engine.................................................................................................................................. 25
2.6.9 Errors seen during CBAM operations (scale-in, scale-out, rebuild)............................................... 25
2.6.10 Compute-Host Restart..................................................................................................................33
2.6.10.1 To stop or start VMs during maintenance window.............................................................. 33
2.6.11 Error: VNF package upload failed, Repository for create exists already: repository: ddebvnf..... 34
2.6.12 Graceful reboot on redundant nodes........................................................................................... 34
2.6.13 Redundant nodes become MASTER at the same time............................................................... 34
2.6.14 To troubleshoot ETCD Cluster Service Degradation alarm......................................................... 35
2.6.15 Clearing prometheus data............................................................................................................37
2.6.16 Troubleshooting call failures........................................................................................................ 38
2.6.17 Checking for memory leaks in ASR process............................................................................... 39
2.7 Troubleshooting Grafana........................................................................................................................ 40
2.8 Troubleshooting CSD Analytics..............................................................................................................41
2.9 CSD error messages.............................................................................................................................. 41
2.10 Tracing and Debugging...................................................................................................................... 144
2.11 CSD remote logging with rsyslog....................................................................................................... 146

3
Monitoring and Troubleshooting Guide

2.12 Troubleshooting System..................................................................................................................... 147

2.12.1 Host node or Compute node unavailability................................................................................ 148
2.13 Troubleshooting and monitoring Aerospike........................................................................................ 148
2.13.1 To collect Aerospike database configuration and log information for offline analysis.................150
2.13.2 Monitoring database status........................................................................................................ 150
2.13.3 Debugging the connectivity between geo-redundant sites.........................................................153
2.14 Managing CSD processes..................................................................................................................154
2.14.1 About CSD processes................................................................................................................154
2.14.2 Start CSD application process................................................................................................... 156
2.14.3 Stop CSD application process................................................................................................... 156
2.14.4 AsControl command................................................................................................................... 156
2.14.5 Selectively start all the application process............................................................................... 156
2.14.6 Selectively stop all the application process............................................................................... 157
2.14.7 Start the process........................................................................................................................ 157
2.14.8 Stop the process........................................................................................................................ 157
2.14.9 Check the process status.......................................................................................................... 157
2.14.10 To check the health of the node..............................................................................................157
2.14.11 Check the CSD node status.................................................................................................... 159
2.15 Defense mechanism........................................................................................................................... 160
2.16 Log collection and VM recovery.........................................................................................................160
2.16.1 Procedure to recover a VM that is not accessible through SSH................................................162
2.17 Collecting information for support.......................................................................................................162
2.18 Known issues......................................................................................................................................165
2.18.1 Known issue............................................................................................................................... 165
2.18.2 Error: StaleStateException......................................................................................................... 165
2.18.3 Error: There is an Unknown Error from Server..........................................................................165
2.18.4 Error: Could not find result context... .. <attributes>.................................................................. 166
2.18.5 Error: Record doesn't exist........................................................................................................ 166
2.18.6 Error: Could not add user ddeadmin......................................................................................... 166
2.18.7 Error: Result=0001 Package NOKIA config has reported an error. Please choose one of the
following: * 1) Retry 2) Skip Package and Continue Choice: [1-2,?,n,p,q] retry returning 9............ 166
2.18.8 Error: Rule System Version (RSV) provisioning error................................................................167
2.18.9 Error: Prometheus service is shown DOWN in Grafana dashboard.......................................... 167
2.18.10 Error: Opening storage failed unexpected end of JSON input.................................................167
2.18.11 Error: RabbitMQ Error On Write Thread.................................................................................. 169
2.18.12 Error: Could not fetch from host.............................................................................................. 170
2.18.13 Error: Error configuring sctp kernel paramaters.......................................................................171
2.18.14 Error: "failed to get the VM UUID for node:Active-IO"........................................................ 172
2.18.15 Toggling in OAM nodes - zabbix-server.................................................................................. 172
2.18.16 Call failure during data migration............................................................................................. 173

3 Bare-metal CSD troubleshooting.............................................................................................................. 174

3.1 Managing CSD processes and log functions....................................................................................... 174
3.2 Switchover issues due to Netmon........................................................................................................176
3.3 Troubleshooting RHEL load modules process..................................................................................... 177
3.3.1 RHEL load modules process is inactive...................................................................................... 177

4
Monitoring and Troubleshooting Guide

3.4 Troubleshooting CALM CALM ETCD is Unavailable alarm................................................................. 178

4 Alarms.......................................................................................................................................................... 179
4.1 CSD alarms.......................................................................................................................................... 179
4.2 SM Application alarms..........................................................................................................................185
4.3 Platform alarm details for CSD and SM...............................................................................................188

5 Logging functions....................................................................................................................................... 193

5.1 Service Manager log files.....................................................................................................................193
5.1.1 Log files........................................................................................................................................ 193
5.1.2 SM logs........................................................................................................................................ 194
5.2 CSD logs...............................................................................................................................................196
5.2.1 CSD logging functions..................................................................................................................196
5.2.1.1 About logging functions....................................................................................................... 196
5.2.1.2 Changing CSD logging levels..............................................................................................197
5.2.1.3 To Change the logging levels of CSD functions..................................................................198
5.2.1.4 Enabling logs from OAM VM............................................................................................... 198
5.2.1.5 To change logging levels on individual VM......................................................................... 199
5.2.1.6 To enable OSSI log collection............................................................................................. 199
5.2.1.7 To enable logs on APP nodes............................................................................................ 200
5.2.1.8 To view CSD log files.......................................................................................................... 200
5.2.2 Log rotation on CSD and SM...................................................................................................... 200
5.2.3 Deletion of User or a Group........................................................................................................ 203
5.2.3.1 Performing Deletion of User or a Group............................................................................. 203

6 Network Troubleshooting...........................................................................................................................204

7 Changes in CSD release............................................................................................................................ 205

7.1 Scripts................................................................................................................................................... 205
7.2 Changes from previous release to this release................................................................................... 205
7.2.1 Software name of Bare-metal variant.......................................................................................... 205
7.2.2 Output of DDE_status.................................................................................................................. 205
7.2.3 Changes in Provisioning log........................................................................................................ 206
7.2.4 Changes in message filter criteria............................................................................................... 206
7.2.5 Auth-Application-Id AVP changes in Diameter error responses.................................................. 207
7.2.6 Change in DDE_control status.....................................................................................................207
7.2.7 Active-Active IO load balancer changes...................................................................................... 208
7.2.8 Route-Record AVP changes........................................................................................................ 208
7.2.9 Changes in XinetD service...........................................................................................................208
7.2.10 Location of 3gpp.xml file for application counters......................................................................209
7.2.11 healthMachines...........................................................................................................................209
7.2.12 In-Service Upgrade Changes..................................................................................................... 209
7.2.13 Metrics Configuration form......................................................................................................... 210
7.2.14 Change of Metrics in CSV format - Transaction Data Record (TDR)........................................ 210
7.2.15 Dashboard is now Analytics.......................................................................................................211
7.2.16 Support for Ganglia GUI............................................................................................................ 211

5
Monitoring and Troubleshooting Guide

7.2.17 Improper disk utilization of all the VMs during OAM node reboot..............................................211
7.2.18 External SLF timeout value........................................................................................................211
7.2.19 Changes in SCTP Association Profile....................................................................................... 211
7.2.20 Changes in Peer connection status dashboard......................................................................... 212
7.2.21 Changes in CSD and SM internal communication matrices...................................................... 213
7.2.22 Database internal IP address.....................................................................................................216
7.2.23 VMware Changes....................................................................................................................... 216
7.2.24 Changes in Diameter Peer form................................................................................................ 216
7.2.25 Changes in Peer Management Dashboard................................................................................217
7.2.26 Changes in SLF Identity to server pool form............................................................................. 217
7.2.27 Changes in SLF Lookup Table Form.........................................................................................217
7.2.28 SCTP link overload handling mechanism is introduced.............................................................218
7.2.29 Changes in Grafana................................................................................................................... 218
7.2.30 Changes in output of listDiameterPeers.sh................................................................................218
7.2.31 Change in alarm log storage path............................................................................................. 219
7.2.32 Change in alarm structure..........................................................................................................219
7.2.33 Changes in Peer disconnect detected alarm............................................................................. 220
7.2.34 Changes in Brevity control for Peer disconnect detected alarm................................................ 221
7.2.35 MariaDB and Zabbix alarms are deprecated............................................................................. 221
7.3 Troubleshooting and debugging commands........................................................................................ 221

6
Monitoring and Troubleshooting Guide About this document

1 About this document

This document describes the architecture, logging functions, managing Cloud Signaling Director
(CSD) processes, and troubleshooting aspects of the Cloud Signaling Director.

1.1 Reason for new issue

This is the Issue 02 of the CSD Monitoring and Troubleshooting Guide for release 20.5.

Issue Date Comments

Issue 02 June 2020 The following change impacts

this release:

• Minor updates on
Troubleshooting and
debugging commands.

Issue 01 May 2020 The following changes impacts

this release:

• Changes in message
filter criteria: lists all the
unsupported criteria
removed from the message
filters in the SM GUI.

• Alarm logs are now stored

in /var/log/calm/alma.
log

• alarm type is now event

type

• Severity of Peer Disconnect

Deteced alarm is now
Major.

• In the SLF Lookup Table

Form, the Status criteria
now supports the following
actions with the OR criteria.

- Provisioned

- Partial Provision

id9YZ-09148-MT11-PCZZA © 2020 Nokia 7

2.0
Monitoring and Troubleshooting Guide About this document

Issue Date Comments

- Failed

- New

- Delete Attempted

Table 1: Document history

1.2 Intended Audience

The intended audience of this document includes:

• Service Provisioners - people who provision the CSD data using the Service Manager.
• Service Administrators - people who monitor and troubleshoot the CSD.
• Field Support Personnel - people who will install the CSD.

1.3 Conventions used

Appearance Description

emphasis Text that is emphasized

graphical user interface text Text that is displayed in a graphical user interface

system input Text that the user types as input to a system

system output Text that a system displays or prints

variable A value or command-line parameter that the user

provides

[] Text or a value that is optional

{ value1 | value2 } A choice of values or variables from which one

value or variable is used
{ variable1 | 8.7 variable2 }

1.4 Related documentation

For CSD and Service Manager:

• CSD - Release 20.5 Installation and Upgrade Operations Guide for Cloud Deployments
• CSD - Release 20.5 User Guide

id9YZ-09148-MT11-PCZZA © 2020 Nokia 8

2.0
Monitoring and Troubleshooting Guide About this document

1.5 Document support

For support in using this or any other document, contact Nokia at the following URL:

• Customer Documentation Welcome Page.

1.6 Technical support

For technical support, contact your local customer support team. See the Support web site (https://
networks.nokia.com/support/) for contact information.

1.7 How to order

To order Nokia documents contact your local sales representative or use Support Portal. Accessing
the documentation and software in the portal requires authentication.

1.8 How to comment

Note to reviewers: The following "How to comment" text will appear in the final document when it is
published. However, the feedback method described below is for use only on final documents. Please
send your review comments to the author using the process you were given when you received this
draft document.

To comment on this document, go to the Online Comment Form (https://documentation.nokia.com/

comments/) or e-mail your comments to the Comments Hotline (mailto: [email protected]).

id9YZ-09148-MT11-PCZZA © 2020 Nokia 9

2.0
Monitoring and Troubleshooting Guide Troubleshooting

2 Troubleshooting
This chapter describes how to manage logging functions and troubleshoot common problems that
arises in the CSD.

2.1 Troubleshooting CBAM

CBAM troubleshooting

For details on CBAM troubleshooting, refer the CloudBand Application Manager CBAM 19.5 SP1
(v19.5.1), CloudBand Application Manager Troubleshooting Guide, DN09247262 from Discovery
Center.

2.2 Troubleshooting during Installation

This topic provides information on common errors or issues encountered while installing the CSD
product.

Ensure that the NTP servers are configured and reachable in respective CSAR packages.

Problem and Solution

Problem Solution

Service unavailable alarm is notified after instantiation. Verify that the service is already up on the relevant
VM. If the status is yes, then clear the alarm manually.

VNF instantiation fails Check the VIM or OpenStack password.

Unauthorized invalid user or password (Disable debug

mode to suppress these details)

Upon stack-creation failure, if the graceful termination This occurs only when all stack creation fails. Only
fails with the following error: forceful termination must be made upon stack creation
failure.
• Status: 500
• Detail: 'NoneType' object is not iterable

If scaling of the VNF fails with the following error: Add the resources on to the cloud.

WorkflowHeatOperationError:
Stack operation error! Stack id:
33909b8e-4ddd-4070-b67b-2976fa6cf8ca,
expected status: COMPLETE, actual
id9YZ-09148-MT11-PCZZA © 2020 Nokia 10
2.0
Monitoring and Troubleshooting Guide Troubleshooting

Problem Solution

status: FAILED, reason: resources.

DB_aspect_group: OverLimit:
resources[13].resources.datastore_
volume: VolumeSizeExceedsAvailableQuota:
Requested volume or snapshot exceeds
allowed gigabytes quota. Requested
10G, quota is 400G and 396G has been
consumed. (HTTP 413) (Request-ID: req-
bae95eb7-accc-4571-8462-e3046ffea35b)

The HA MASTER RECOVERING alarm is seen during Ignore the same.

the CSD-OAM and SM-OAM installation, until the
system stabilizes.

If CBAM throws the following error during instantiation Follow the steps listed here to troubleshoot the issue:
of VNF:
1. Log in to CBAM as root
ERROR: AuthorizationFailure: 2. Copy the certificate of the vlab to /etc/pki/ca-trust/
Authorization Failed: SSL exception source/anchors/.
connecting to https://10.193.134. 3. Run the command update-ca-trust.
100:13000/v2.0/tokens: ("bad handshake:
Error([('SSL routines', 'ssl3_get_
server_certificate', 'certificate verify
failed')],)",)

If installation or upgrade fails with the following error: Configure the retries=3 in the CBAMs ansible.
cfg configuration file and perform the operation again.
*UNREACHABLE! => {\"changed\": false,
This enables the CBAM to try for a maximum of three
\"msg\": \"Failed to connect to the
times, before declaring the ansible operation as failed.
host via ssh.\", \"unreachable\": true}
\nfatal: [41dff769-3f6c-4dcd-aac4- For example, following is the sample output.
d5e4f61ce 587]: *
[defaults]

#remote_tmp = /tmp/ansible_tmp

local_tmp = /tmp/ansible_tmp nocows=1

# gathering = explicit

# debug = True [ssh_connection] control_

path=/tmp/ssh_control_path_%%h-%%p-%%r
scp_if_ssh=True

retries=3

Things to be noted before installation

id9YZ-09148-MT11-PCZZA © 2020 Nokia 11

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Before installation, ensure that the following is taken care of:

1. In the file /opt/nokia/cbam-venv/lib/python2.7/site-packages/cbam_workflows/actions/ansible_cfg/

ansible.cfg ensure that defaults section is commented as shown here:

[defaults]#remote_tmp = /tmp/ansible_tmplocal_tmp = /tmp/

ansible_tmpnocows=1
2. In the CSAR package under ansible\files\initdata ensure that sdc_conf.xlsx or
sdc_conf.json is configured appropriately for all tabs.
3. Configure appropriate open stack resources such as network id, subnets, ips, flavors, images are
configured in the json file which is given as input for instantiation.

2.3 Troubleshooting during Upgrade

This topic provides information on common errors or issues encountered during upgrade.

The Imagesu operation in CBAM fails due to the following underlying infrastructure and connectivity
errors with CBAM:

• Slowness in internal network within a VNF.

• Higher external network latency between CBAM and VIM to VNF (OAM).
• Slowness in VIP reachability on OAM node within VNF.
• Slowness in Disk or Cinder.

Problem Solution Applicability

SSH connectivity error: Verify the SSH OpenStack and VMware.

connectivity of VM from
fatal: [d31b2d8f-1caf-41d9-
OAM node identified by
a1da-9db4e9948a38]: UNREACHABLE! =>
d31b2d8f-1caf-41d9-
{"changed": false, "msg": "Failed
a1da-9db4e9948a38.
to connect to the host via ssh.",
"unreachable": true} If the connectivity is good,
then retry the upgrade or
During upgrade on VMware or OpenStack, this error
backout procedure.
indicates issues with internal SSH connectivity.

Rebuild error: Retry the upgrade or OpenStack

rollback procedure.
NovaAction.servers.rebuild
failed: Cannot 'rebuild' instance
d8aae443-28f2-4785-8bbf-fed08a78e528
while it is in task_state rebuild_
spawning (HTTP 409) (Request-ID:
req-9524a26c-365e-43c9-80d6-
b57f97eaf93a)", u'result': u"Failure
caused by error in tasks: rebuild_vm

This error indicates that the OpenStack rebuild is

taking longer time than expected.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 12

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Problem Solution Applicability

JavaScriptAction failed error: Ensure to provide the OpenStack and VMware.

valid instance names.
JavaScriptAction failed: Invalid node
CBAM-f6fb7212bf4e4ed2a1382b92ffdbf2e0-
db-0 not part of VNF!!!!

This error indicates that the incorrect instance name is

provided in the input.

JavaScriptAction failed error: Ensure that the set of OpenStack and VMware.
instances are separated
JavaScriptAction failed: Invalid node
by a single space.
not part of VNF!!!!

This error indicates that an incorrect syntax is provided

in the input parameter.

For example, more than one space between input

parameters.

Table 2: Problem and Solution

2.4 Troubleshooting Post Upgrade

This topic provides information on common errors or issues encountered post upgrade.

2.4.1 Troubleshooting during LCM operations

Problem Solution Applicability

LCM operation fails due to etcd cluster unavailability Execute as described

during scale operation of DB with a message no in the following Order
leader. of command execution
during failure of LCM
etcd cluster is unavailable due to misconfiguration. operation due to cluster
unavailability on page
13

Table 3: Problem and Solution

Order of command execution during failure of LCM operation due to cluster unavailability

Execute the following command in the path.

source /etc/etcd/etcd.conf

To verify the member list:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 13

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Execute the following command.

etcdctl --endpoints ${ETCDCTL_ENDPOINT} member list

To verify the cluster status:

Execute the following command.

etcdctl --endpoints ${ETCDCTL_ENDPOINT} cluster-health

To remove a member:

Execute the following command.

etcdctl --endpoints ${ETCDCTL_ENDPOINT} member remove ${memberId}

To add a member:

Execute the following command.

etcdctl --endpoints ${ETCDCTL_ENDPOINT} member add ${ETCD_NAME} ${ETCD_

LISTEN_PEER_URLS}

Procedure from Scale-in to Scale-out operation:

1. Perform a ForceScaleIn operation.

2. Verify the cluster status to find the unreachable member by executing the following command.

source /etc/etcd/etcd.client.conf
etcdctl --endpoints ${ETCDCTL_ENDPOINT} cluster-health

For example, following is a sample.

failed to check the health of member 9076f5eb9fdcce57 on http://<IP_

Address>:2379:
Get http://<IP_Address>:2379/health: dial tcp <IP_Address>:2379:
connect: connection refused
member 9076f5eb9fdcce57 is unreachable: http://<IP_Address>:2379 are
all unreachable
member a2da499cbf9d381b is healthy: got healthy result from http://
<IP_Address>:2379
member f944e6e62d39e09c is healthy: got healthy result from http://
<IP_Address>:2379
cluster is degraded

9076f5eb9fdcce57 is the unreachable member ID.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 14

2.0
Monitoring and Troubleshooting Guide Troubleshooting

3. Remove the unreachable member based on the identified unreachable member ID identified
previously by running the following command.

etcdctl --endpoints ${ETCDCTL_ENDPOINT} member remove 9076f5eb9fdcce57

4. Verify the cluster status:

If the cluster is healthy, then check the member list.

etcdctl --endpoints ${ETCDCTL_ENDPOINT} member list

a2da499cbf9d381b: name=sdc-4 peerURLs=http://<IP_Address>:2380
clientURLs=http://<IP_Address>:2379 isLeader=false
f944e6e62d39e09c: name=sdc-2 peerURLs=http://<IP_Address>:2380
clientURLs=http://<IP_Address>:2379 isLeader=false

If all the members have no leader, then wait until one member is a leader.

etcdctl --endpoints ${ETCDCTL_ENDPOINT} member list

a2da499cbf9d381b: name=sdc-4 peerURLs=http://<IP_Address>:2380
clientURLs=http://<IP_Address>:2379 isLeader=true
f944e6e62d39e09c: name=sdc-2 peerURLs=http://<IP_Address>:2380
clientURLs=http://<IP_Address>:2379 isLeader=false

5. Scale-out operation.
If a cluster is healthy and one member is in a leader state, then execute the scale-out operation.

To perform a manual scale-out operation:

1. Log in to the unreachable member node and add the member to the cluster.

source /etc/etcd/etcd.client.conf
source /etc/etcd/etcd.conf
etcdctl --endpoints ${ETCDCTL_ENDPOINT}
member add ${ETCD_NAME} ${ETCD_LISTEN_PEER_URLS}

2. Remove the

etcd

data.

rm -rf ${ETCD_DATA_DIR}/*

3. In the /etc/etc/etcd.conf file modify the cluster state from new to existing.

For example, ETCD_INITIAL_CLUSTER_STATE="new" to

ETCD_INITIAL_CLUSTER_STATE="existing"

id9YZ-09148-MT11-PCZZA © 2020 Nokia 15

2.0
Monitoring and Troubleshooting Guide Troubleshooting

2.5 Troubleshoot Service Manager

This section describes the failure scenarios for Managed Element (ME) export and import feature,
managing logging functions, and troubleshooting common issues that arises in Service Manager (SM).
It also lists down the various alarms and alerts commonly found in SM.

2.5.1 SM GUI failure scenarios

SM GUI Error

• When the Error! Couldn't fetch from SM error is thrown in the SM GUI.

Log in to SM OAM and execute the following script.

[root@smbvnf-oame-1 ~]# healthMachines

smbvnf-oame-0(STANDBY) OK
smbvnf-oame-1(ACTIVE) OK
smbvnf-db-1(NA) NOK
smbvnf-db-0(NA) NOK
[root@smbvnf-oame-1 ~]#

Solution: Identify the reason for outage of database and execute the following command on OAM
node as a root user.

AsControl start db

Note: If fall back operation is performed after upgrading only database nodes, you need to
stop or start the standby OAM node in the VNF.

2.5.2 ME import failure scenarios

Following are the scenarios when the ME import operation fails:

1. Import operation fails, if any primary field information is missing or an incorrect field information
described in the ME import request.

The import operation fails displaying the following error message:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 16

2.0
Monitoring and Troubleshooting Guide Troubleshooting

2. Import operation fails, if a duplicate provisioning ME record exists and the SM user forgets to
select the Overwrite option during the ME import.

The import operation fails displaying the following error message:

3. Import operation fails, if the associated ME profile does not exist in SM.

The import operation fails displaying the following error message:

Note:

In case, if the ME import fails,

• collect and examine the logs at the following file path.

/opt/tpa/logs/ImportExport.log

• If distribution on the fly fails, collect and examine the logs at the following file path.

/opt/tpa/logs/Provisioning.log

• to refer CSD and SM application logs, refer the following path.

/opt/tpa/logs/DESMApplication.log

Note:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 17

2.0
Monitoring and Troubleshooting Guide Troubleshooting

The features in the SM GUI Configuration > Diameter Peering > Request Timeout Profile
and KPI Statistics are applicable from 18.8 release onwards and not intended for previous
version of MEs.

Ensure that you refrain provisioning them to the previous version MEs:

• In case, it is provisioned to the previous version MEs, user can recover the profile by
cloning it and delete the erroneous profile.
• If the profile deletion fails, try to unprovision from previous version MEs and delete it.

2.6 Troubleshooting CSD

CAUTION! Terminating the active services or processes on CSD could lead to service
disruption.

2.6.1 Application issues and errors

Problem: Connection is not establishing between the peers:

Solution: Perform the following steps:

1. Problem: Connection is not establishing between the peers:

1. Ping the floating IP address or the IP address of the IO node which listens for diameter from
the peer. If the ping is not successful, then check the neutron port status in the VIM and rectify
the same. Additionally, check the external routes on the respective node.
2. Try to ping the IP address of the IO node from peer again, when it is successful, establish the
Diameter connection over TCP/SCTP.

2. Problem: SCTP multihoming peer is not connected.

Solution: Ping both the primary and secondary IP addresses. If the ping is successful for
secondary IP address and the ping fails for primary IP address then update the secondary IP
address in the Remote Primary IP / Hostname * field of the Diameter Peer Form and then re-
distribute the diameter peer form.

Note: If an outbound SCTP multi-homed peer is connected and the remote primary and
secondary interfaces goes down and only the secondary interface comes up, then the
SCTP connection will not be established.

This will not affect the connected peers.

3. Problem: Getting Unable_to_Deliver error for diameter request:

Solution: Perform the following steps:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 18

2.0
Monitoring and Troubleshooting Guide Troubleshooting

a) Check the error message sent in the response, if it is unable to figure out how to
route…, then it is an issue with peers. Take the DDESMApplication.log from APP node, pcap,
and then verify the connectivity and AVPs in the message.
b) Log in to the Service Manager GUI and check the Dashboard tab for Peer connectivity, verify
whether all the peers are in connected state. If not, check the connectivity between the peer(s)
and ensure that all the required peers are connected for the request to process.
c) If the peer to which the request to be sent is not connected and request has the Destination-
Host, then the error is received.
d) If the Destination-Host does not exist, then check the configurations on SM.

1. Check the message filter configuration, whether the criteria is proper and action is triggered
in routing profile and so on.
2. If routing is done with SLF and the lookup failed, check for SLF lookup failures in the
countable events of SM and check for flow counters file in CSD active OAM in path

/opt/tpa/logs/.

Following may cause SLF Failures, SLF_LOOKUP_FAILURE:

a. CSD sends Sh-UDR to external server and Sh-UDA is not received from the External
HSS server because of public-identity value mismatch. Then Countable Event,
SLF_LOOKUP_ FAILURE is triggered for SLF external and the lookup fails.
b. If there is a mismatch between the configured Identity value in Identity To Serverpool
form and Identity value sent by the client , Countable Event SLF_LOOKUP_FAILURE is
triggered for SLF internal and the lookup fails.
SLF_UNREACHABLE: CSD sends UDR to external server and
UDA is received from external HSS server with the Result-Code:
DIAMETER_UNABLE_TO_DELIVER(3002), Countable Event SLF_UNREACHABLE is
triggered.SLF_INDIRECT_MAPPING_FAILURE:If there is mismatch between extracted
mapping key for indirect mapping from returned UDA and configured key in Simple Map
and Set, SLF_INDIRECT_MAPPING_FAILURE is triggered.

In flow counters file, Failure Reason Column is updated as SLF_LOOKUP_FAILURE,

SLF_ UNREACHABLE, SLF_INDIRECT_MAPPING_FAILURE for the respective
failures and Request Type column is updated as SLF_LOOKUP_FAILURE_COUNT,
SLF_ UNREACHABLE_COUNT, SLF_INDIRECT_MAPPING_FAILURE_COUNT for
respective failures.
3. If SLF lookup is successful, and server pool name is received, then check whether
the configuration of server pool is proper and all the servers in the pool are active and
connected.
4. If RSV is added, check the RSV rules.
e) Verify other routing-related configurations.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 19

2.0
Monitoring and Troubleshooting Guide Troubleshooting

2.6.2 To troubleshoot healthMachines or vnfcMapList

When the commands healthMachines or vnfcMapList hangs.

Perform the following steps:

1. Check if all the three oam-0, oam-1, db-0 instances of etcd are running fine by running
thecommand systemctl status etcd.

2. Start all the instances using the command systemctl start etcd.

2.6.3 To troubleshoot issues related to CSD application processes

healthMachines provides information about the following processes, other information related to the
processes like PID, start time and so on can be fetched using the following commands:

For OAM, IO, and APP nodes:

systemctl status NOKIAasr

For DB node:

systemctl status NOKIAasdb

2.6.4 To troubleshoot an interface connection

Note: The diameteragent.cfg file is not automatically replicated. Update the file on both
the active and standby OAM.

1. Open a console window and log in to the CSD as a rootuser.

2. Perform a ping command from the CSD to the other system.

Note: If there is more than one network interface on the CSD, specify the required
source interface for the ping command using the -i parameter.

3. Perform one of the following:

• If the ping command is successful, go to Step 4

• If the ping command fails, go to Step 5

4. Check the local Diameter protocol configuration.

1. Open the /opt/tpa/osgi/instance/diameteragent.cfg file using a text editor.

2. Locate the line that starts with:

### Diameter ###

id9YZ-09148-MT11-PCZZA © 2020 Nokia 20

2.0
Monitoring and Troubleshooting Guide Troubleshooting

3. Verify that the realm and hostname values on the other system matches with CSD's realm and
host name values in the following lines:

diameteragent.originRealm = realm

diameteragent.originHost =

<AddressGroup>-.diameteragent.client.src = IP_address (or comma-

separated list of local IP_ addresses)

IP_address (or comma-separated list of local IP_addresses

4. Verify that the IP_address is the IP address of the local network interface used by the CSD to
reach the other system.

By default, CSD uses the primary IP address of the primary network interface to communicate
with other systems. The peer IP address can return a single IP address or a list of IP
addresses that are configured for Diameter to listen to and return in CEA messages. The client
source IP address can return a single IP address or a list of IP addresses that are configured
for Diameter to listen to and returns the CEA messages. The list of IP addresses that can
be returned is limited to the configured addresses for the diameteragent.client.src
properties in the diameteragent.cfg file.

5. Verify that the local network and firewall configurations are correct.

6. Repeat steps 1 on page 20 – 5 on page 21 to check connectivity again.

7. If the other system is upstream, verify that the configuration on the other system is correctly
configured to communicate with the CSD.

To troubleshoot connections and diameter packets

Take the tcpdump on IO on specific ports to troubleshoot further. Also, ensure that the port is added in
the pcap.

2.6.5 To troubleshoot Diameter peer and route issues

Note: A route is displayed in the listDiameterRoutes.sh script, if the route

configuration is active and the peer that the route references is connected. If the route
configuration is inactive or the peer is not connected, then route is not displayed in
listDiameterRoutes.sh script.

1. Log in to the CSD Active OAM node as root.

2. Switch to the /opt/tpa/bin directory.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 21

2.0
Monitoring and Troubleshooting Guide Troubleshooting

3. Create a list by typing:

$ list-diameter-peers

The system displays information of the local system, followed by any connected peers. For
example:

Peers for server '0-0-4': Local Peer:

Origin-Host: c1t14dde1host.realm2 Origin-Realm: realm2
Connected: true Connected Address: null Quarantined: false
Origin-State-Id: 1472565553
Hostname: BBTPNJ33HRPvSGSx-L-NK-01-c1t14dde100000000-IO-Node- 002-
DDE178-00
Port: 3868 Protocol: TCP IP addresses:
Firmware rev: 2000210000
Product name: Smart Plan Suite (DDE) Vendor ID: IETF_IANA
Supported Vendor IDs: Inband Security IDs: Application: Relay Name:
Tcp1
Remote Peers: 2
Origin-Host: clientHost1633.clientRealm1633 Origin-Realm:
clientRealm1633
Connected: true
Connected Address: 135.121.114.188 Quarantined: false
Origin-State-Id: 2
Hostname: 135.121.114.188
Port: 4003 Protocol: TCP IP addresses:
IPV6: fe80:0:0:0:5054:ff:fe41:8d60 IPV6: 0:0:0:0:0:0:0:1
IPV4: 135.121.114.188
Firmware rev: 1
Product name: Alcatel-Lucent 5780 DSC Test Client Vendor ID: ALU
Supported Vendor IDs: ALU,3GPP2,CISCOCSG2,VerizonWireless, 3GPP,ETSI
Inband Security IDs: Accounting applications: Authorization
applications:
Gxx (16777266)
NASREQ (1)
S9 (16777267)
Sy (16777302)
BaseAccounting (3)
SWx (16777265)
Sd (16777303)
Rx (16777236)
Ro / Gy / CreditControl (4) Gx (16777238)
ICCSy (111)
Sh (16777217)
S6a/S6d (16777251)
Cx/Dx (16777216)
Rq (16777222)
S13/S13prime (16777252)

id9YZ-09148-MT11-PCZZA © 2020 Nokia 22

2.0
Monitoring and Troubleshooting Guide Troubleshooting

E2E4 (16777231)
Vendor-specific applications: Gxx (16777266)
S9 (16777267)
Sy (16777302)
SWx (16777265)
Sd (16777303)
Rx (16777236)
Ro / Gy / CreditControl (4) Gx (16777238)
ICCSy (111)
Sh (16777217)
S6a/S6d (16777251)
Cx/Dx (16777216)
Rq (16777222)
S13/S13prime (16777252)
E2E4 (16777231)
Origin-Host: clientHost163.clientRealm163
Origin-Realm: clientRealm163 Connected: true
Connected Address: 135.121.114.188 Quarantined: false
Origin-State-Id: 2
Hostname: 135.121.114.188
Port: 56788 Protocol: TCP IP addresses:
IPV4: 135.121.114.188
IPV6: 0:0:0:0:0:0:0:1
IPV6: fe80:0:0:0:5054:ff:fe41:8d60
Firmware rev: 1
Product name: Alcatel-Lucent 5780 DSC Test Client Vendor ID: ALU
Supported Vendor IDs: ALU,3GPP2,CISCOCSG2,VerizonWireless, 3GPP,ETSI
Inband Security IDs: Accounting applications: Authorization
applications:
Gxx (16777266)
NASREQ (1)
S9 (16777267)
Sy (16777302)
BaseAccounting (3)
SWx (16777265)
Sd (16777303)
Rx (16777236)
Ro / Gy / CreditControl (4) Gx (16777238)
ICCSy (111)
Sh (16777217)
S6a/S6d (16777251)
Cx/Dx (16777216)
Rq (16777222)
S13/S13prime (16777252)
E2E4 (16777231)
Vendor-specific applications: Gxx (16777266)
S9 (16777267)
Sy (16777302)
SWx (16777265)
Sd (16777303)
id9YZ-09148-MT11-PCZZA © 2020 Nokia 23
2.0
Monitoring and Troubleshooting Guide Troubleshooting

Rx (16777236)
Ro / Gy / CreditControl (4) Gx (16777238)
ICCSy (111)
Sh (16777217)
Routes for server 'APP-Node-002': Diameter routes.
Peer: pcrf01-l-nk-lb04c-gx-vzr.vzimspcrf.com
Destination 'vzimspcrf.com', applications {Gx}, priority 2
Host routes.
No routes are applicable.
Routes for server 'APP-Node-003': Diameter routes.
Peer: pcrf01-l-nk-lb04c-gx-vzr.vzimspcrf.com
Destination 'vzimspcrf.com', applications {Gx}, priority 2
Host routes.
No routes are applicable.

To view the diameter messages that are not processed by CSD

1. Open a console window and log in to the CSD as a rootuser.

2. From the OAM server, collect the logs from all OAM using SaveLogFilesscript.
3. Review the log entries of DDESMApplication.log to identify any failed message processing.
4. Close the DDESMApplication.log file.

2.6.6 To manually disconnect a peer

To manually disconnect a peer:

1. Log in to SM GUI using URL, <https://<ip_address>/servicemanager/.

2. Click Analytics tab >Peer Management Dashboard.

3. Choose the peer to be disconnected from the list.

4. Click disconnect and click OK.

5. Select the disconnect cause in the Select Disconnect Cause pop-up window.

6. Click Disconnect.

2.6.7 Routing and Peering

If routing is not as per the expectation, then check the realm and host based routing configurations on
CSD. Also take the pcap and verify the contents being sent from the inbound peer in the request and
verify whether they are matching with the routing configuration.

Verify the following:

• If the routing profile is configured correctly.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 24
2.0
Monitoring and Troubleshooting Guide Troubleshooting

• If the criteria to match is correct and whether the action is triggered (Server pool, SLF).
• If the configuration for routing decision is proper and all the peers are connected.

2.6.8 Rules Engine

For RSV rules, validate the rules and ensure that no errors are displayed before saving it.

To provision an RSV, it has to be first changed from DRAFT to RELEASE state and then it has to
be provisioned on CSD. Provisioning the RSV in DRAFT state will fail. After the RSV is distributed to
CSD, the user must activate it by clicking Provision on SM GUI and selecting the ME for activation.
Unless the RSV is in ACTIVE state, the rules are not effective on the CSD, they are not applicable,
and the calls are not routed as expected.

Rules Engine overwrites the previous routing-decision, without checking whether the routing-decision
is already taken.

Whereas Routing-profile checks whether routing-decision is already taken, and does not overwrite the
previous routing-decision.

Following is an example of RSV configuration with DiameterAnswer trigger:

if
DiameterAnswer.Origin-Host = "peer2.nsn.com" and
DiameterAnswer.Result-Code = 3004 and
PeerTableContext.Choose-Peer-By-Origin-Host ( "retry-peer.nsn.com" ) = 1
then
DiameterRequest.Route-To-Chosen-Peer
DiameterRequest trigger is configured as,
LoadBalancerContext.Select-Destination-With-Pool-Name = "Server-Pool1"

In this case, load-balancer action overwrites Route-To-Chosen-Peer. Load load-balancer can be

configured in routing-profile if it is required to route with Route-To-Chosen-Peer

Note:

To check which Rule table is matching, execute the command

DDE_logLevelPOLICYDECISION DEBUG.

2.6.9 Errors seen during CBAM operations (scale-in, scale-out, rebuild)

During VNF scale-in, scale-out, and rebuild you might encounter the following error scenarios:

Case 1: VNF scale-in operation - scale in failed ( UNREACHABLE to SHUTDOWN VM)

Error message:

AnsibleExecutionException: Ansible action execution failed with exit

code: 3, details can be found in the logs (possible cause: fatal:
[4e9a67e4-408d-4271-8f5d-e57dd8fab691]: UNREACHABLE! => {"changed":
id9YZ-09148-MT11-PCZZA © 2020 Nokia 25
2.0
Monitoring and Troubleshooting Guide Troubleshooting

false, "msg": "Failed to connect to the host via ssh.", "unreachable":

true} [DEPRECATION WARNING]: Specifying include variables at the top
level of the task is deprecated. Refer the: http://docs.ansible.
com/ansible/playbooks_roles.html#task-include-files-and-
encouraging-reuse for currently supported syntax regarding included files
and variables. This feature will be removed in a future release.
Deprecation warnings can be disabled by setting deprecation_warnings=
False in ansible.cfg.)

Troubleshooting steps:

1. Get the VM instance ID from Heat VM by using the command:

[root@heattoolkitvm ~(v1226)]# nova list | grep

"501130a1e37a43d281e4264c5e23bc13" --color | grep

"4e9a67e4-408d-4271-8f5d-e57dd8fab691"

Result: The VM instance ID is known from the output.

| 4e9a67e4-408d-4271-8f5d-e57dd8fab691 |
CBAM-501130a1e37a43d281e4264c5e23bc13-db-1 | SHUTOFF |
- | Shutdown |
CBAM-501130a1e37a43d281e4264c5e23bc13-internal_network=192.168.3.
41

2. Boot the VM which is in a shutdown state using the action type hardreboot for HealOne operation,
as shown in the following figure:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 26

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Case 2: VNF scale-out operation - scale out failed( UNREACHABLE to SHUTDOWN VM)

Error message:

AnsibleExecutionException: Ansible action execution failed with exit

code: 2, details can be found in the logs (possible cause: fatal: [
4e9a67e4-408d-4271-8f5d-e57dd8fab691 ]: UNREACHABLE! = {"changed":
false, "msg": "All items completed", "results": [{"_ansible_item_result":
true, "item": {"dest": "/root/.ssh/id_rsa", "mode": "0600", "src":
"~/.ssh/id_rsa"}, "msg": "Failed to connect to the host via ssh.",
"unreachable": true}, {"_ansible_item_result": true, "item": {"dest":
"/root/.ssh/id_rsa.pub", "mode": "0644", "src": "~/.ssh/id_rsa.pub"},
"msg": "Failed to connect to the host via ssh.", "unreachable": true}]}
fatal: [334d4a1b-c9ee-4840-8fd0-a770c0e2e543]: FAILED! = {"changed":
true, "cmd": ["/usr/bin/gangliaSetup"], "delta": "0:00:00.588738", "end":
"2018-03-08 13:18:00.812260", "failed": true, "invocation": {"module_
args": {"_raw_params": "/usr/bin/gangliaSetup", "_uses_shell": false,
"chdir": null, "creates": null, "executable": null, "removes": null,

id9YZ-09148-MT11-PCZZA © 2020 Nokia 27

2.0
Monitoring and Troubleshooting Guide Troubleshooting

"warn": true}, "module_name": "command"}, "rc": 1, "start": "2018-03-08

13:18:00.223522", "stderr": "/usr/bin/gangliaSetupManual: option requires
an argument -- i", "stdout": "operation: start\ndatasource: 192.168.3.10,
192.168.3.11\ncluster: CBAM-501130a1e37a43d281e4264c5e23bc13\nlocal ip:
\ngetLocalVNFCType:app\nnode type: app\nHELP: gangliaSetupManual\n\nTool
to create gmetad configuration file.\n\nUSAGE: gangliaSetupManual [-h]
[-o operation] [-n nodetype] [-c cluster] [-d datasource] [-i
authHost]\n\nOPTIONS:\n o operation : start|restart|stop. Default is
start\n n nodetype : app|oam. Default is app\n c cluster : Cluster
name to use for gridname. Default is myCluster\n d datasource : Hosts
where gmond datasources are running used by\n gmetad to gather
statistics. Multiple hosts\n are added with comma separated list (i.e.
host1,host2).\n Default is hostname\n i authHost : IP to use for
ganglia authorization. Most likely this is\n the IP used to ssh into this
VM. Default is hostname -i\n h: Show this message", "stdout_lines":
["operation: start", "datasource: 192.168.3.10,192.168.3.11", "cluster:
CBAM-501130a1e37a43d281e4264c5e23bc13", "local ip: ", "getLocalVNFCType:
app", "node type: app", "HELP: gangliaSetupManual", "", "Tool to create
gmetad configuration file.", "", "USAGE: gangliaSetupManual [-h] [-o
operation] [-n nodetype] [-c cluster] [-d datasource] [-i authHost]", "",
"OPTIONS:", " o operation : start|restart|stop. Default is start", "
n nodetype : app|oam. Default is app", " c cluster : Cluster name
to use for gridname. Default is myCluster", " d datasource : Hosts
where gmond datasources are running used by", " gmetad to gather
statistics. Multiple hosts", " are added with comma separated list (i.e.
host1,host2).", " Default is hostname", " i authHost : IP to use for
ganglia authorization. Most likely this is", " the IP used to ssh into
this VM. Default is hostname -i", " h: Show this message"], "warnings":
[]} [DEPRECATION WARNING]: Specifying include variables at the top level
of the task is deprecated. Please see: http://docs.ansible.
com/ansible/playbooks_roles.html#task-include-files-and-
encouraging-reuse for currently supported syntax regarding included files
and variables. This feature will be removed in a future release.
Deprecation warnings can be disabled by setting deprecation_warnings=
False in ansible.cfg. [DEPRECATION WARNING]: Skipping task due to
undefined Error, in the future this will be a fatal error.: 'dict object'
has no attribute 'stdout'. This feature will be removed in a future
release. Deprecation warnings can be disabled by setting deprecation_
warnings=False in ansible.cfg. [DEPRECATION WARNING]: Skipping task due
to undefined Error, in the future this will be a fatal error.: 'dict
object' has no attribute 'stdout'. This feature will be removed in a
future release. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg. [DEPRECATION WARNING]:
Skipping task due to undefined Error, in the future this will be a fatal
error.: 'dict object' has no attribute 'stdout'. This feature will be
removed in a future release. Deprecation warnings can be disabled by
setting deprecation_warnings=False in ansible.cfg. [DEPRECATION WARNING]:
Skipping task due to undefined Error, in the future this will be a fatal
error.: 'dict object' has no attribute 'stdout'. This feature will be
removed in a future release. Deprecation warnings can be disabled by
id9YZ-09148-MT11-PCZZA © 2020 Nokia 28
2.0
Monitoring and Troubleshooting Guide Troubleshooting

setting deprecation_warnings=False in ansible.cfg. [DEPRECATION WARNING]:

Skipping task due to undefined Error, in the future this will be a fatal
error.: 'dict object' has no attribute 'stdout'. This feature will be
removed in a future release. Deprecation warnings can be disabled by
setting deprecation_warnings=False in ansible.cfg. [DEPRECATION WARNING]:
Skipping task due to undefined Error, in the future this will be a fatal
error.: 'dict object' has no attribute 'stdout'. This feature will be
removed in a future release. Deprecation warnings can be disabled by
setting deprecation_warnings=False in ansible.cfg. [DEPRECATION WARNING]:
Skipping task due to undefined Error, in the future this will be a fatal
error.: 'dict object' has no attribute 'stdout'. This feature will be
removed in a future release. Deprecation warnings can be disabled by
setting deprecation_warnings=False in ansible.cfg. [DEPRECATION WARNING]:
Skipping task due to undefined Error, in the future this will be a fatal
error.: 'dict object' has no attribute 'stdout'. This feature will be
removed in a future release. Deprecation warnings can be disabled by
setting deprecation_warnings=False in ansible.cfg. [DEPRECATION WARNING]:
Skipping task due to undefined Error, in the future this will be a fatal
error.: 'dict object' has no attribute 'stdout'. This feature will be
removed in a future release. Deprecation warnings can be disabled by
setting deprecation_warnings=False in ansible.cfg.)

Troubleshooting steps:

Follow the steps to troubleshoot:

1. Get the VM instance ID from Heat VM by using the command:

[root@heattoolkitvm ~(v1226)]# nova list | grep

"501130a1e37a43d281e4264c5e23bc13" --color | grep

"4e9a67e4-408d-4271-8f5d-e57dd8fab691"|

Result: The VM instance ID is known from the output.

| 4e9a67e4-408d-4271-8f5d-e57dd8fab691 |
CBAM-501130a1e37a43d281e4264c5e23bc13-db-1 | SHUTOFF |
- | Shutdown |
CBAM-501130a1e37a43d281e4264c5e23bc13-internal_network=192.168.3.
41

2. Boot the VM which is in a shutdown state using the action type hardreboot for HealOne operation,
as shown in the following figure:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 29

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Case 3: When you rebuild a VM (A) when VM (B is in SHUTDOWN state) the VM fails

Healone (custom) fails.

Name Healone (custom)

State failed

Started 08-03-2018 13:25

Finished 08-03-2018 13:31

Operation ID CBAM-3578d0a7b1014951b5eaaddaded92931

Table 4:

Troubleshooting steps:

Follow the steps to troubleshoot:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 30

2.0
Monitoring and Troubleshooting Guide Troubleshooting

1. Get the operation ID from workflows.log by running the command:

[cbam@cbam cbam]$ grep "3578d0a7b1014951b5eaaddaded92931"

workflows.log | grep UNREACH

Result: The operation ID is known from the output:

Mar 8 08:57:54 localhost cbam:

[WORKFLOWS|CBAM-501130a1e37a43d281e4264c5e23bc13|CBAM-
3578d0a7b1014951b5eaaddaded92931|MainThread|INFO|cbam_workflows.
docker.dofatal: [4e9a67e4-408d-4271-8f5d-e57dd8fab691]: UNREACHABLE!
=>
{"changed": false, "msg": "All items completed", "results":
[{"_ansible_item_result": true, "item": {"dest": "/root/.ssh/id_rsa",
"mode": "0600", "src": "~/.ssh/id_rsa"}, "msg": "Failed to connect to
the host via ssh.", "unreachable": true}, {"_ansible_item_result":
true, "item": {"dest": "/root/.ssh/id_rsa.pub", "mode": "0644", "src":
"~/.ssh/id_rsa.pub"}, "msg": "Failed to connect to the host via ssh.",
"unreachable": true}]}
Mar 8 09:01:08 localhost cbam:
[WORKFLOWS|CBAM-501130a1e37a43d281e4264c5e23bc13|CBAM-
3578d0a7b1014951b5eaaddaded92931|GreenThread-6703|INFO|workflow_trace]
Task 'heal_deploy_hook' (43c654c5-859c-44b6-965d-4c2ae9f17b98)
[RUNNING -> ERROR, msg=AnsibleExecutionException: Ansible action
execution failed with exit code: 3, details can be found in the logs
(possible cause: fatal: [4e9a67e4-408d-4271-8f5d-e57dd8fab691]:
UNREACHABLE! => {"changed": false, "msg": "All items completed",
"results": [{"_ansible_item_result": true, "item": {"dest":
"/root/.ssh/id_rsa", "mode": "0600", "src": "~/.ssh/id_rsa"}, "msg":
"Failed to connect to the host via ssh.", "unreachable": true},
{"_ansible_item_result": true, "item": {"dest": "/root/.ssh/id_rsa.
pub", "mode": "0644", "src": "~/.ssh/id_rsa.pub"}, "msg": "Failed to
connect to the host via ssh.", "unreachable": true}]}

2. Get the VM instance ID from Heat VM by using the command:

[root@heattoolkitvm ~(v1226)]# nova list | grep

"501130a1e37a43d281e4264c5e23bc13" --color | grep

"4e9a67e4-408d-4271-8f5d-e57dd8fab691"

| 4e9a67e4-408d-4271-8f5d-e57dd8fab691 |

CBAM-501130a1e37a43d281e4264c5e23bc13-db-1 | SHUTOFF |

- | Shutdown |

CBAM-501130a1e37a43d281e4264c5e23bc13-internal_network=192.168.3.

41

id9YZ-09148-MT11-PCZZA © 2020 Nokia 31

2.0
Monitoring and Troubleshooting Guide Troubleshooting

3. Boot the VM which is in a shutdown state using the action type hardreboot for HealOne operation,
as shown in the following figure.

Case 4: VM heal failed for action_type - rebuild

Error message:

Mar 21 10:25:13 localhost cbam:

[WORKFLOWS|CBAM-b23f5ea9aeb344afb36e25a16bc0c094|CBAM-
30c02953156d44078d66c30be21e148f|GreenThread-1959|INFO|workflow_trace]
Task 'healing' (6ec49362-f182-43d0-8a13-c779cda366a4) [RUNNING -> ERROR,
msg=Failure caused by error in tasks: heal_deploy_hook
heal_deploy_hook [task_ex_id=9a661c9e-feff-4d83-972e-76f1fb304796] ->
AnsibleExecutionException: Ansible action execution failed with exit
code: 2, details can be found in the logs (possible cause: fatal:
[777f85c9-061d-4434-b768-5e3c647bcdf4 -> 10.15.166.111]: FAILED! =>
{"changed": false, "elapsed": 120, "failed": true, "invocation":
{"module_args":

id9YZ-09148-MT11-PCZZA © 2020 Nokia 32

2.0
Monitoring and Troubleshooting Guide Troubleshooting

{"connect_timeout": 5, "delay": 0, "exclude_hosts": null, "host":

"192.168.3.31", "path": null, "port": 22, "search_regex": null, "state":
"started", "timeout": 120}
, "module_name": "wait_for"}, "msg": "Timeout when waiting for
192.168.3.31:22"}
[DEPRECATION WARNING]: Specifying include variables at the top level of
the
task is deprecated. Please see:
http://docs.ansible.com/ansible/playbooks_roles.
html#task-include-files-and-encouraging-reuse

Troubleshooting method:

In case of VM rebuild failure, check the logs in workflows.log of CloudBand Application Manager
(CBAM). If you observe the connectivity or time-out error, then recover the VM using the following
method.

1. Heal the failed VM again with the action_type as hardreboot and wait for the node to be
accessible.
2. Heal again with action_type as rebuild.

Case 5: Scale-out operation fails due to insufficient resources

Cause: Scale-out of a VM fails due to insufficient system resources including memory, CPU, and so
on.

Solution: Perform the scale-in operation by specifying ForceScaleIn field as yes. This deletes the VM
where you perform the scale-out operation.

2.6.10 Compute-Host Restart

Reason:

If compute-hosts on which CSD VMs have been hosted re-starts or reboots, and if the compute-host
comes up successfully, then the VMs reaches a SHUTOFF state.

Solution or Workaround:

Restart the VMs using openstack-dashboard (or uisng the nova start command).

2.6.10.1 To stop or start VMs during maintenance window

During maintenance window start or stop the VMs in the following order:

SHUTOFF the SM VMs in the following sequence:

• All OAM VMs

• All DB VMs

SHUTOFF the CSD VMs in the following sequence:

• All IO VMs
id9YZ-09148-MT11-PCZZA © 2020 Nokia 33
2.0
Monitoring and Troubleshooting Guide Troubleshooting

• All App VMs

• All OAM VMs
• All DB VMs

Power on the SM VMs in the following sequence:

• All DB VMs
• All OAM VMs

Power on the CSD VMs in the following sequence:

• All DB VMs
• All OAM VMs
• All App VMs
• All IO VMs

2.6.11 Error: VNF package upload failed, Repository for create exists already:
repository: ddebvnf

Reason:

When you use the same CBAM to install two CSD or SM stack, you encounter the error VNF package
upload failed, Repository to create already exist: repository: ddebvnf.

Solution or Workaround:

The descriptor_id in VNFC requires changes or it has to be different for both the stacks.

2.6.12 Graceful reboot on redundant nodes

Follow the given steps to perform a graceful reboot:

1. Check the DDE_status on the node to be rebooted.

2. If the node is a master node, then follow the steps:

• Run the DDE_control become-slave command for the switchover to happen.

If the switch over is successful, then proceed to reboot the node with reboot -f command.
3. If the node is in the SLAVE state, then proceed to reboot the node with reboot -f command.

Note: The following command is applicable only for OAM nodes.

DDE_control become-slave

2.6.13 Redundant nodes become MASTER at the same time

In scenarios where the redundant nodes cannot communicate between each other due to network
issues or abrupt failure of keepalived service, then both the nodes may become HA Master. This can
be verified by executing the following command as a root user:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 34

2.0
Monitoring and Troubleshooting Guide Troubleshooting

ha role

Recovery action:

Log in to both redundant nodes and check the status of keepalived service using the following
command.

systemctl status keepalived.service

If the keepalived service is inactive or stopped on the node, then reboot that node using the command.

reboot -f

2.6.14 To troubleshoot ETCD Cluster Service Degradation alarm

The following are the scenarios where you encounter an ETCD Cluster Service Degradation alarm.

Scenarios Solution or Workaround Result

If one out of three etcd Check the cluster healthy status One out of three instances are
instances are with down, the

down, a MAJOR alarm is seen etcdctl command. output of the etcdctl

and there is command, prompts
etcdctl --
no impact on the system's endpointshttp:// the information to the user.
functionality, as <ip_address>:<port
no>cluster-health
remaining two instances does
the etcd

related tasks properly.

If two out of three etcd instances Check the cluster health status Two out of three instances are
are with down, the

down, a CRITICAL alarm is etcdctl command. output of the etcdctl

seen and this command prompts
etcdctl --endpoints
is the indication to the user that the information to the user.
http://<ip_
etcd
address>:<port no>
related tasks are not functioning
cluster-health
properly.

If one etcd instance which was Check the cluster health status One out of three instances are
down with down, the

earlier comes up, and now two etcdctl command. output of the etcdctl
etcd command prompts
etcdctl --endpoints
instances are up and one the information to the user.
http://<ip_
instance is
address>:<port no>

id9YZ-09148-MT11-PCZZA © 2020 Nokia 35

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Scenarios Solution or Workaround Result

down, a CLEAR alarm is raised cluster-health
to clear the

CRITICAL alarm and a MAJOR

alarm is

raised. There is no impact on

the system's

functionality, as remaining two

instances

does the etcd related tasks

properly.

If one etcd instance comes up, Check the cluster health status All three instances are up, the
and now all with output of

the three etcd instances are up, etcdctl command. the etcdctl command
a CLEAR prompts the
etcdctl --endpoints
alarm is raised to clear the information to the user.
http://<ip_
MAJOR alarm.
address>:<port no>
This indicates that all the three
cluster-health
instances

are performing the etcd related

tasks

properly.

NOTICE:

If all the etcd instances are up and running and the cluster is healthy, then no alarm is seen.

But when the ETCD Cluster Service Degradation alarm is raised with CRITICAL severity and

when it returns to normal state, then a MAJOR alarm is seen at NetAct which the operator

needs to clear manually.

Things to be noted while configuring the etcd

The following are the points to be noted while configuring the etcd:

• Always recommended to use odd number of cluster members.

An etcd cluster requires a majority of nodes (quorum), to agree on updates to the cluster state.

For a cluster with n members, quorum is (n/2)+1.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 36

2.0
Monitoring and Troubleshooting Guide Troubleshooting

For any odd-sized cluster, adding one node always increases the number of nodes necessary for
quorum.

• Fast disks are the most critical factor for etcd deployment which affects performance and stability.

A slow disk increases the etcd request latency and potentially degrade cluster stability. Since

etcd’s consensus protocol depends on persistent storage of metadata to a log.

The majority of etcd cluster members must write down every request to disk.

2.6.15 Clearing prometheus data

This is required when there is not enough cinder volume on the system for prometheus.

In the following path delete the folders, that are created with older time stamp.

/prometheus/data

For example, as described in the following section.

drwxr-xr-x. 3 ddeadmin dde 4096 Sep 21 17:15 01CQXZ52VJSTRH909CPCJB7DDQ

drwxr-xr-x. 3 ddeadmin dde 4096 Sep 21 18:15 01CQY2JX9SXW92BS28GBR1893E
drwxr-xr-x. 3 ddeadmin dde 4096 Sep 21 18:15 01CQY2JXZRNVFWFSK1R26CX810
drwxr-xr-x. 3 ddeadmin dde 4096 Sep 21 18:45 01CQY49V3ADF3EQKPF4BZA29YG

Modifying Scrape interval and retention time post installation

The scrape interval and retention time after installation can be changed in the prometheus.yml in
the following path.

/opt/tpa/statistics/prometheus/

To modify the scrape_interval for the intended metrics:

1. Obtain the current scrape_interval data.

/149_shabana_CSD_Destro/config/kpiconfigs/aerospike_scrape_interval
{"value": "5"}
/149_shabana_CSD_Destro/config/kpiconfigs/flow_scrape_interval
{"value": "10"}
/149_shabana_CSD_Destro/config/kpiconfigs/io_scrape_interval
{"value": "5"}
/149_shabana_CSD_Destro/config/kpiconfigs/jmx_scrape_interval
{"value": "5"}
/149_shabana_CSD_Destro/config/kpiconfigs/latency_scrape_interval
{"value": "5"}
/149_shabana_CSD_Destro/config/kpiconfigs/legacy_scrape_interval
{"value": "5"}

id9YZ-09148-MT11-PCZZA © 2020 Nokia 37

2.0
Monitoring and Troubleshooting Guide Troubleshooting

/149_shabana_CSD_Destro/config/kpiconfigs/system_scrape_interval
{"value": "5"}

2. Modify the scrape_interval duration of the intended metrics from the cuttent value to intended
value using the following command:

sdcctl set system config/kpiconfigs/flow_scrape_interval "{\"value\":

\"<scrape_interval_value>\"}"

For example, if the current scrape_interval value is 5s and the intended scrape_interval
is 10s:

sdcctl set system config/kpiconfigs/flow_scrape_interval "{\"value\":

\"10\"}"
3. Obtain the updated scrape_interval data.

/149_shabana_CSD_Destro/config/kpiconfigs/aerospike_scrape_interval
{"value": "5"}
/149_shabana_CSD_Destro/config/kpiconfigs/flow_scrape_interval
{"value": "10"}
/149_shabana_CSD_Destro/config/kpiconfigs/io_scrape_interval
{"value": "5"}
/149_shabana_CSD_Destro/config/kpiconfigs/jmx_scrape_interval
{"value": "5"}
/149_shabana_CSD_Destro/config/kpiconfigs/latency_scrape_interval
{"value": "5"}
/149_shabana_CSD_Destro/config/kpiconfigs/legacy_scrape_interval
{"value": "5"}
/149_shabana_CSD_Destro/config/kpiconfigs/system_scrape_interval
{"value": "5"}

4. Execute the following command on both the OAM nodes.

/opt/tpa/sbin/reconfigurePrometheus --reconfigure

Result: The /opt/tpa/statistics/prometheus/prometheus.yml is updated with the

modified scrape_interval duration.
5. Restart the following services on both the OAM nodes one after the other.

• prometheus.service
• grafana.service
• alertmanager.service

In the /opt/tpa/statistics/prometheus/prometheus-server path, change RETENTION

parameter for the retention time.

Note: If alarms related to TDR failure and Prometheus scrape interval are observed during
heal operation, they can be ignored. Since the heal node is not up with all the prometheus
related process and due to this the scraping can fail and this is an expected behavior.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 38

2.0
Monitoring and Troubleshooting Guide Troubleshooting

2.6.16 Troubleshooting call failures

If any call failures are observed with the error code 3002 (UNABLE_TO_Deliver) originating from

CSD, then it indicates that Diameter connection information may be out of sync between IO and the

Routing nodes. If any of the Routing node is out of sync, then the traffic processed by that node

responds with the error code 3002 (UNABLE_TO_Deliver). Following script is used to identify the

faulty node and take the node out of service for troubleshooting.

Help usage for DiameterAppIOAuditor.py tool:

usage: DiameterAppIOAuditor.py [-h] [--enable RoutingNode]

[--disable RoutingNode] [--check]
[--raise]
This script is used to check the overall diameter connection and traffic
status across all the Routing/Application nodes.
optional arguments:
-h, --help show this help message and exit
--enable RoutingNode Enables the diameter traffic on the specified
node
--disable RoutingNode Disables the diameter traffic on the specified
node
--check Checks the connection status across all routing
nodes
--raise Raises an alarm if the connections are out of
sync. To
be used along with --check

2.6.17 Checking for memory leaks in ASR process

The following script is used to check if there are any probable memory leaks in any of the ASR
process running in the following nodes:

1. SM and CSD OAM

2. IO
3. Routing

Help usage for ASRMemUsageMonitor.py tool:

usage: ASRMemUsageMonitor.py [-h] [--threshold THRESHOLD] [--ouFull

OUFULL]
[--gcAllowedDuration GCALLOWEDDURATION]
[--gcSince GCSINCE] [-v] [-s]
[--interval INTERVAL] [-l LOGLEVEL] [-a]
[{jstat,gc,all}]
This script attempts to make an intelligent guess about any application
servers running towards memory leak

id9YZ-09148-MT11-PCZZA © 2020 Nokia 39

2.0
Monitoring and Troubleshooting Guide Troubleshooting

positional arguments:
{jstat,gc,all}
optional arguments:
-h, --help show this help message and exit
--threshold THRESHOLD
How Many Consequtive times OU growth is OK
--ouFull OUFULL At what percent OU increase is considered
Threatening!
--gcAllowedDuration GCALLOWEDDURATION
GC Duration more than this value will be flagged
as
RED
--gcSince GCSINCE GC is Monitored for last gcSince minutes
-v, --verbosity
-s, --scheduled Scheduled, script is scheduled for running every
interval
--interval INTERVAL Interval at which a scheduler will run the thread
-l LOGLEVEL, --loglevel LOGLEVEL
Log level
-a, --alarm Alarms enabled if flag set

2.7 Troubleshooting Grafana

Process verification on installation

1. Ensure that after installation, the following are up and running on both OAM nodes.

• systemctl status prometheus

• systemctl status grafana
• systemctl status node_exporter (on all nodes)
• systemctl status asprom (on database nodes only)
2. For Cloud, ensure to create /prometheus/data directory after the installation.

/prometheus is the new mount created on installation. That is, the size of the cinder volume
specified during installation to store the Prometheus data.

For Bare-metal, ensure to create /opt/prometheus/data directory after the installation.

Diameter Routing Status Dashboard behavior

When the advanced metrics profile is attached for longer duration and if the flow metrics generated
exceeds the configured value of Metrics Cardinality limit, then during this time based on the generated
metrics, the following panels may not display any data.

• Reprocessed Request Count

• Diameter Application Distribution

id9YZ-09148-MT11-PCZZA © 2020 Nokia 40

2.0
Monitoring and Troubleshooting Guide Troubleshooting

• Diameter Result Code

The RC is displayed as zero when the Diameter Result Code is empty in the collected scrapes as the
sample limit is exceeded.

2.8 Troubleshooting CSD Analytics

This section describes the typical behavior or the troubleshooting involved on CSD analytics.

Jaeger Sampling Rate Behavior

When the jaeger sampling rate at the APP node is more than 100.

The <trace-without-root-span> is displayed, since the root of the trace is usually the last span
to arrive at the collection tier.

The query service fetches the trace and does not return the root span, since the root trace is yet to
arrive.

Figure 1: Behavior of traces

By default, Jaeger tracing is auto-enabled for Diameter Unable to deliver (3002) and
Diameter too busy (3004) at DEBUG level. Jaeger tracing gets enabled when there are failures
at APP only. It is recommended to enable Jaeger tracing at DEBUG level for troubleshooting purpose.

Note: If the you intend to remove the auto-enabling of Jaeger, then remove the alert from
alerts.default.csd.20.0.yml.j2 and alerts.default.csd.20.0.yml files
persent in /appdata/prometheus/alerts and then restart the prometheus service.

2.9 CSD error messages

Overload Protection Messages

The following section describes the Overload protection messages:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 41

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

OVERLOAD_PROTECTION_ Invalid profile name. An overload shedding profile by

00001 the name {0} does not exist.

OVERLOAD_PROTECTION_ Cannot delete overload The overload shedding profile

00002 shedding profile. {0} cannot be deleted as it is still
in use.

Routing Messages

The following section describes the Routing messages:

Number Message Description

ROUTING_00001 Invalid routing profile name. The routing profile {0} does not
exist.

ROUTING_00002 Cannot delete routing profile. The routing profile {0} cannot
be deleted as it is being used in
routing plugin configuration.

SLF Errors

Number Message Description

SLF_00000 Missing Identity to Destination The Identity to Destination

Configuration configuration is missing.

SLF_00001 Missing or empty Id. The Identity to Destination has a

missing or empty Id.

SLF_00002 Invalid Id format. The IdentityToDestination

configuration with id {0} is of
incorrect format.It must contain
Subscriber Group id and identity
id separated by semi-colon.

SLF_00003 Subscriber Group Id is not a The IdentityToDestination

number configuration with id {0} contains
an entry for the Subscriber
Group Id in the identity pattern
that is not a valid number.

SLF_00004 Subscriber Group Id mismatch The IdentityToDestination

configuration with id {0} and
Subscriber Group id{1} do not

id9YZ-09148-MT11-PCZZA © 2020 Nokia 42

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

contain matching Subscriber

Group IDs.

SLF_00005 Subscriber Group for Id does The IdentityToDestination

not exist configuration with id {0} and
Subscriber Group id{1} has
a Subscriber Group id that
does not match an existing
Subscriber Group.

SLF_00006 Missing or empty Destination One or more destination

string string(s) in the Identity to
Destination configuration with Id
{0} is/are missing or empty.

SLF_00007 Invalid Destination Strings One or more destination

string(s) in the Identity to
Destination configuration with Id
{0} has/have too many commas.

SLF_00008 Invalid DestinationIds DestinationIds list contains

provisioning a DestinationId value of {0},
which does not exist as a
provisioned Destination record
in the database.

SLF_00009 Invalid Destination and Only one of Destinations or

DestinationIds provisioning DestinationIds lists for Id {0}
may be provisioned. It is invalid
to provision both lists.

SLF_00010 Duplicate IdentityToDestination There is already an

record in database IdentityToDestination record
with Id {0} in the database.

SLF_00011 No IdentityToDestination record There is no

in database IdentityToDestination record
with Id {0} in the database.

SLF_00100 Missing SLF Plugin The SLF Plugin Configuration is

Configuration missing

SLF_00201 Missing Per-Server The SLF Per-Server

Configuration Subscriber Group configuration {0} is missing the
Name. Subscriber Group name.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 43

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

SLF_00202 Missing Per-Server The SLF Per-Server

Configuration Subscriber Group configuration Subscriber Group
name of {0} does not match an
existing Subscriber Group.

SLF_00203 Missing or empty identity The SLF Per-Server

attribute-value pairs configuration {0} has a missing
or empty identity attribute-value
pairs list.

SLF_00204 Missing Identity Type The SLF Per-Server

configuration {0} is missing the
identity type.

SLF_00205 Invalid Redirect Host Usage The SLF Per-Server

configuration {0} must have
Redirect Host Usage value of
between {1} and {2}.

SLF_00206 Invalid Redirect Max Cache The SLF Per-Server

Time configuration {0} must have
Redirect Max Cache Time value
1 or greater.

SLF_00207 Missing Per-Server The SLF Per-Server

Configuration SLF Lookup Table configuration {0} is missing the
SLF Lookup Table.

SLF_00208 Per-Server Configuration SLF The SLF Per-Server

Lookup Table does not exists configuration SLF Lookup Table
does not exist.

SLF_00300 Missing server list. The SLF server configuration list

is missing.

SLF_00301 Missing Per-server The SLF per-server

Configuration. configuration is missing.

SLF_00302 Missing or empty configuration. The SLF per-server

configuration has a missing or
empty configuration.

SLF_00400 Missing Destination The Destination configuration is

Configuration. missing.

SLF_00401 Missing or empty Destination Id. The Destination has a missing

or empty Id.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 44

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

SLF_00402 Destination Id cannot be a Destination Id must be a

negative value. positive integer value.

SLF_00403 Missing or empty Destination The Destination string in the

string. Destination with Id {0} is missing
or empty.

SLF_00404 Invalid Destination string. The Destination string in the

Destination configuration with Id
{0} has too many commas.

SLF_00405 Duplicate Destination record in There is already a Destination

database. with Id {0} in the database.

SLF_00406 No Destination record in There is no Destination with Id

database. {0} in the database.

SLF_00500 Missing Subscriber Group The Subscriber Group

Configuration. configuration is missing.

SLF_00501 Missing or empty Subscriber The Subscriber Group has a

Group Id. missing or empty Id.

SLF_00502 Subscriber Group Id cannot be Subscriber Group Id must be a

a negative value positive integer value.

SLF_00503 Duplicate Subscriber Group There is already a Subscriber

record in database Group with Id {0} in the
database

SLF_00504 No Subscriber Group record in There is no Subscriber Group

database. with Id {0} in the database.

SLF_00600 SLF Lookup Table Name must SLF Lookup Table with name
be unique {0} already exists.

SLF_00601 Missing Lookup Type. Lookup Type is missing.

SLF_00602 SLF Lookup Table for Id does The IdentityToServer Pool

not exist configuration with pattern {0}
has an SLF Lookup Table Id
that does not exists in SLF
Lookup Table.

SLF_00603 Missing SLF Lookup Table SLF Lookup Table Configuration

Configuration. is Missing.

SLF_00604 Missing Id ID is missing in SLF Lookup

Table Configuration.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 45

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

SLF_00605 Unknown Lookup Type Unknown Lookup Type {0}.

Valid Lookup Types are {1}.

SLF_00606 Missing Name Name is missing in SLF Lookup

Table.

SLF_00700 Missing IdentityToServerPool The IdentityToServerPool

Configuration configuration is missing.

SLF_00701 Missing Id The IdentityToServerPool has a

missing Id.

SLF_00702 Missing Pattern Pattern is missing or empty.

SLF_00703 Invalid Pattern Invalid regular expression {0}.

SLF_00704 Invalid Numeric Range. Invalid Numeric Range {0}.

Should be numeric ranges from
{1} and {2} and separated by
{3}.

SLF_00705 Server Pool Name is Missing. Server Pool Name is missing

in the IdentityToServerPool
configuration.

SLF_00706 Invalid Server Pool Server Pool Name {0} in

the IdentityToServerPool
configuration does not exists.

SLF_00707 Invalid IP Subnet Invalid IP Subnet {0}.

SLF_00708 Overlapping Numeric Range Overlapping Numeric Range.

Numeric Range Should not be
Overlapping.

Diameter Messages

The following section describes the Diameter messages:

Number Message Description

Server pool and pool host errors

DDM_00001 Duplicate server pool name The name {0} is not unique.

DDM_00006 Duplicate pool host FQDN Two pool hosts in pool {0} have
the same Diameter FQDN (host:
{1}, realm: {2}).

id9YZ-09148-MT11-PCZZA © 2020 Nokia 46

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

DDM_00012 Invalid administrative state An administrative state must be

specified.

DDM_00013 Invalid balancing algorithm A load balancing algorithm must

be specified.

DDM_00014 Invalid Diameter application Invalid Diameter application

Destination Mapping Errors

DDM_00401 Null or empty label Null or empty label.

DDM_00402 Null or empty value The destination mapping with

label {0} has a null or empty
value.

DDM_00403 Null or empty Destination-Realm The destination mapping with

label {0}, value {1} has a null or
empty Destination-Realm.

DDM_00404 Null or empty type The destination mapping with

label {0}, value {1} has a null
type.

DDM_00405 Null or empty Related Mapping The destination mapping with

value label {0}, value {1} has a null or
empty Related Mapping value.

DDM_00406 Circular reference in Related The destination mapping with

Mappings Label {0}, Value {1} has a
reference to its own identity in
Related Mappings.

Message Filter Errors

DDM_00500 Missing filter group name Filter group is missing name for
application plugin {0}.

DDM_00501 Duplicate filter subgroup Multiple filter subgroups with

name {0} for application plugin
{1}.

DDM_00502 Filter criterion with no values Null or empty value has been
set for attribute {0} of context {1}
for filter {2} within filter group {3}
for application plugin {4}.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 47

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

DDM_00503 Invalid filter subgroup name The referenced filter subgroup

with name {0} does not exist for
application plugin {1}.

DDM_00504 Duplicate filter name One or more filters have the

name {0} within filter group {1}
for application plugin {2}.

DDM_00506 Missing message filter name One or more filters for filter
group {0} are missing a name
for application plugin {1}.

DDM_00507 Non-existent filter criterion Filter criterion context {0} does

context name not exist for filter {1} within filter
group {2} for application plugin
{3}.

DDM_00508 Unsupported filter criterion Filter criterion context {0} is

context name unsupported for filter {1} within
filter group {2} for application
plugin {3}.

DDM_00509 Missing criterion context name One or more criterion are

missing a context name for filter
{0} within filter group {1} for
application plugin {2}.

DDM_00510 Missing attribute name One or more criterion are

missing an attribute name for
filter {0} within filter group {1} for
application plugin {2}.

DDM_00511 Missing action context name One or more actions are

missing a context name for filter
{0} within filter group {1} for
application plugin {2}.

DDM_00512 Missing action name One or more actions are

missing an action name for filter
{0} within filter group {1} for
application plugin {2}.

DDM_00513 Non-existent filter criterion Filter criterion attribute {0} does

attribute name not exist within context {1} for
filter {2} within filter group {3} for
application plugin {4}.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 48

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

DDM_00514 Unsupported filter criterion Filter criterion attribute {0} is

attribute name unsupported for context {1} for
filter {2} within filter group {3} for
application plugin {4}.

DDM_00515 Invalid number of filter criterion Invalid number of parameters

attribute parameters for attribute {0} of context {1} for
filter {2} within filter group {3} for
application plugin {4}. Expected
{5}, Actual {6}.

DDM_00516 Invalid filter criterion attribute Invalid parameter name {0} of

parameter value attribute {1} of context {2} for
filter {3} within filter group {4} for
application plugin {5}.

DDM_00517 Failed to convert filter criterion Failed to convert value {0} for
attribute parameter value parameter {1} of attribute {2} of
context {3} for filter {4} within
filter group {5} for application
plugin {6}.

DDM_00518 Unsupported filter criterion Operator {0} is not supported

operator for attribute {1} of context {2} for
filter {3} within filter group {4} for
application plugin {5}.

DDM_00519 Unsupported number of values Operator {0} does not support

filter criterion operator expression with {1} on LHS and
{2} on RHS for attribute {3} of
context {4} for filter {5} within
filter group {6} for application
plugin {7}.

DDM_00520 Filter criterion with no values No values have been set for
attribute {0} of context {1} for
filter {2} within filter group {3} for
application plugin {4}.

DDM_00521 Filter criterion unary operator Values not permitted for unary
with values operator {0} for attribute {1} of
context {2} for filter {3} within
filter group {4} for application
plugin {5}.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 49

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

DDM_00522 Failed to convert filter criterion Failed to convert value {0}

attribute value to type {1} for attribute {2} of
context {3} for filter {4} within
filter group {5} for application
plugin {6}.

DDM_00523 Non-existent filter criterion Filter action context {0} does

context name not exist for filter {1} within filter
group {2} for application plugin
{3}.

DDM_00524 Unsupported filter criterion Filter action context {0} is

context name unsupported for filter {1} within
filter group {2} for application
plugin {3}.

DDM_00525 Non-existent filter action name Filter action {0} does not exist
within context {1} for filter
{2} within filter group {3} for
application plugin {4}.

DDM_00526 Unsupported filter action name Filter action {0} is unsupported

for context {1} for filter {2} within
filter group {3} for application
plugin {4}.

DDM_00527 Invalid number of filter action Invalid number of parameters

parameters for action {0} of context {1} for
filter {2} within filter group {3} for
application plugin {4}. Expected
{5}, Actual {6}.

DDM_00528 Missing filter action parameter Missing expected parameter

name name {0} of action {1} of context
{2} for filter {3} within filter group
{4} for application plugin {5}.

DDM_00529 Failed to convert filter action Failed to convert value {0} to

parameter value type {1} for parameter {2} of
action {3} of context {4} for filter
{5} within filter group {6} for
application plugin {7}.

DDM_00530 Invalid regular expression for Value {0} is not a regular

filter criterion value expression for attribute {1} of
context {2} for filter {3} within

id9YZ-09148-MT11-PCZZA © 2020 Nokia 50

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

filter group {4} for application

plugin {5}.

DDM_00531 Invalid filter attribute parameter The value {0} is less than lower
value bound {1} for parameter {2} of
attribute {3} of context {4} for
filter {5} within filter group {6} for
application plugin{7}.

DDM_00532 Invalid filter attribute parameter The value {0} is greater than
value upper bound {1} for parameter
{2} of attribute {3} of context {4}
for filter {5} within filter group {6}
for application plugin{7}.

DDM_00533 Invalid filter action parameter The value {0} is less than lower
value bound {1} for parameter {2}
of action {3} of context {4} for
filter {5} within filter group {6} for
application plugin {7}.

DDM_00534 Invalid filter action parameter The value {0} is greater than
value upper bound {1} for parameter
{2} of action {3} of context {4} for
filter {5} within filter group {6} for
application plugin {7}.

DDM_00535 Failed to convert filter attribute Failed to convert value {0} to

range value type {1} for parameter {2} of
attribute {3} of context {4} for
filter {5} within filter group {6} for
application plugin {7}.

DDM_00536 Failed to convert filter action Failed to convert value {0} to

range value type {1} for parameter {2} of
action {3} of context {4} for filter
{5} within filter group {6} for
application plugin {7}.

DDM_00537 Missing-dependent attributes as The attribute {0} must be

criteria. accompanied by attributes {1}
in filter group {2}, filter {3} for
plugin {4}.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 51

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Application Plugin Configuration Errors

The following section describes the Application plugin configuration errors:

Number Message Description

DDM_00201 Request sequence is null. The request cannot be null.

DDM_00202 Response sequence is null. The response cannot be null.

DDM_00203 Invalid plugin name length in Plugin name {0} in request

request sequence sequence is not between {1}
and {2} characters in length.

DDM_00204 Invalid plugin parameter length Plugin parameter {0} in request

in request sequence sequence is not between {1}
and {2} characters in length.

DDM_00205 Plugin name/parameter Plugin name/parameter pair of

combination in request {0}, {1} has been duplicated in
sequence not unique. the request sequence.

DDM_00206 Invalid plugin name length in Plugin name {0} in response

response sequence sequence is not between {1}
and {2} characters in length.

DDM_00207 Invalid plugin parameter length Plugin parameter {0} in

in response sequence response sequence is not
between {1} and {2} characters
in length.

DDM_00208 Plugin name/parameter Plugin name/parameter pair of

combination in response {0}, {1} has been duplicated in
sequence not unique. the response sequence.

DDM_00209 Conflicting plugins Plugin configurations cannot

configurations exist for the Integrated system
type and also exist for the I/O or
Processor system types.

DDM_00210 Duplicate plugins configurations Only one plugins configuration

can exist per system type.

DDM_00211 Plugin does not exist. No plugin named {0} exists in

the system.

DDM_00212 Invalid plugin in request Plugin {0} does not support

sequence. requests so it cannot be in a
request sequence.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 52

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

DDM_00213 Invalid plugin in response Plugin {0} does not support

sequence responses so it cannot be in a
response sequence.

DDM_00214 Unsupported system type Plugin {0} does not support the
system type {1}.

DDM_00215 System type not set The system type is not set.

DDM_00216 Request and response The request and response

sequences are both empty. sequences cannot both be
empty.

Generic Binding Errors

The following section describes the Generic binding errors:

Number Message Description

DDM_00300 Generic Binding Key Label is a Generic Binding Key Label is a

mandatory field. mandatory field.

DDM_00301 Invalid Generic Binding Key Generic Binding Key Label

Label length. must be between {0} and {1}
characters in length.

DDM_00302 Generic Binding Key Type Generic Binding Key Type

cannot be null. cannot be null.

DDM_00303 Generic Binding Key Data Generic Binding Key Data

cannot be null. cannot be null.

DDM_00304 Generic Binding Key Data must Generic Binding Key Data must
be a numeric value when key be a numeric value when key
type is Long. type is Long; {0} is not correct.

DDM_00305 Invalid Generic Binding Key Generic Binding Key Data

Data length must be between {0} and {1}
characters in length.

DDM_00306 Generic Binding Value Label is Generic Binding Value Label is

a mandatory field. a mandatory field.

DDM_00307 Invalid Generic Binding Value Generic Binding Value Label

Label length. must be between {0} and {1}
characters in length.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 53

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

DDM_00308 Generic Binding Value Data is a Generic Binding Value Data is a

mandatory field. mandatory field.

DDM_00309 Generic Binding Value Data Generic Binding Value Data

must be a numeric value when must be a numeric value when
value type is Long. value type is Long; {0} is invalid.

DDM_00310 Invalid Generic Binding Value Generic Binding Value Data

Data length must be between {0} and {1}
characters in length.

DDM_00311 Key already exists, must be Key label and key data pair
unique. must be unique; key with label
[{0}] and data [{1}] already
exists.

DDM_00312 The Generic Binding to be The Generic Binding to be

erased no longer exists. erased is no longer present in
the system. Please refresh and
check if the Generic Binding is
present.

DDM_00313 Too many results returned Too many entries found under
during search. the key criteria provided.
Returned {0} and maximum
allowed during a search is {1}.
Please narrow down your
search criteria.

Common Error Messages

The following section describes the Common error messages:

Number Message Description

Countable Event Configuration Errors.

COMMON_00200 Interval threshold is greater than The interval threshold value

max. {0} cannot be greater than the
maximum number of events per
threshold {1}.

COMMON_00201 Interval threshold is less than 1. The interval threshold value {0}
cannot be less than 1.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 54

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

COMMON_00202 Max per interval is less than 1. The maximum per interval value
{0} cannot be less than 1.

COMMON_00203 Duplicate configuration found. Found a duplicate countable

event configuration with the
name <Countable event name>.

You must ignore this message,

as the trailing error is shown in
the logs at the time of bring up.

CSD displays the logs because

APP node which is coming up
first is writing the countable
events to database. Similarly,
other APP nodes also try to
write into database and this
leads to the duplicate countable
events found message being
displayed.

COMMON_00204 Invalid interval size. The countable event interval

size cannot be set less than the
minimum of {0} seconds.

Profile Errors

COMMON_00300 Invalid profile name. A CPU usage thresholds profile

with the name {0} does not
exist.

COMMON_00301 Missing name for profile. The profile must have a name.

COMMON_00302 Profile name too long. The profile name is greater than
the maximum length of {0}.

CPU Usage Threshold Profile Errors

COMMON_00400 Invalid lower threshold value. The CPU usage lower threshold
{0} is less than {1}.

COMMON_00401 Invalid upper threshold value. The CPU usage upper threshold
{0} is less than {1}.

COMMON_00402 Invalid resource threshold value. The CPU usage resource

threshold {0} is less than {1}.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 55

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

COMMON_00403 Invalid lower threshold value The CPU usage lower threshold
{0} is greater than {1}.

COMMON_00404 Invalid upper threshold value The CPU usage upper threshold
{0} is greater than {1}.

COMMON_00405 Invalid resource threshold value. The CPU usage resource

threshold {0} is greater than {1}.

COMMON_00406 Lower threshold exceeds upper The CPU usage lower threshold
threshold. {0} is greater than or equal to
the upper threshold {1}.

COMMON_00407 Resource threshold is less than The CPU usage upper threshold
or equal to upper threshold. {0} is greater than or equal to
the resource threshold {1}.

COMMON_00408 Cannot delete CPU Usage The CPU usage upper

Thresholds Profile. thresholds profile {0} cannot be
deleted as it is still in use.

SCTP Error Messages

Error Number Brief Description

DIAMETER_00250 Invalid RTO initial value The RTO initial value {0} is not
in the valid range of {1} to {2}.

DIAMETER_00251 Invalid RTO min value The RTO min value {0} is not in
the valid range of {1} to {2}.

DIAMETER_00252 Invalid RTO max value The RTO max value {0} is not in
the valid range of {1} to {2}.

DIAMETER_00253 Invalid max burst value The max burst value {0} is not in
the valid range of {1} to {2}.

DIAMETER_00254 Invalid cookie life value The valid cookie life {0} is not in
the valid range of {1} to {2}.

DIAMETER_00255 Invalid Association Max The Association Max

Retransmission value Retransmission {0} is not in the
valid range of {1} to {2}.

DIAMETER_00256 Invalid path Max The Path Max Retransmission

Retransmission value {0} is not in valid range of {1} to
{2}.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 56

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Error Number Brief Description

DIAMETER_00257 Invalid Max Init Retransmission The Max Init Retransmission {0}
value is not in valid range of {1} to {2}.

DIAMETER_00258 Invalid Heartbeat Interval value The Heartbeat Interval {0} is not
in valid range of {1} to {2}.

SM error component: Profile Entities

The following section describes SM errors for Profile Entities:

Sub-component: Throttling Profile

The following section describes the error messages for sub-component Throttling profile:

Number Message Description

Not applicable User is not Authorized to access Not applicable

the requested resource

MSAPI_00066 Delete operation is not allowed Not applicable

for record with provisioning state
{0}.

MSAPI_00005 Invalid request : Invalid value Not applicable

provided for parameter: {filter
type}

MSAPI_00009 Record not found in SM Record not found {0}

MSAPI_00066 Only delete operation can be Not applicable

performed on a DELETE_
ATTEMPTED record

MSAPI_00066 Save operation is not allowed Not applicable

for record with provisioning
state {0}. Only provisioning or
unprovisioning is allowed.

MSAPI_00216 Unprovisioning/Deleting is not Not applicable

allowed for this throttling profile
as it is referred by one or more
throttling config.

MSAPI_00217 Unprovisioning/Deleting is not Not applicable

allowed for this throttling profile
as it is referred by one or more
diameter peer.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 57

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00005 Invalid MEs in delete distribution Not applicable

list. Data is not distributed on
these MEs.

MSAPI_00050 Invalid MEs in the request Not applicable

DIAMETER_00243 Lower threshold exceeds upper Error Reference Number:

threshold. DIAMETER_00243. The
diameter latency lower threshold
{0} is greater than or equal to
the upper threshold {1}.

Table 5: Sub-component: Throttling profile

Sub-component: DownstreamOverload Profile

The following section describes error messages for sub-component DownstreamOverload Profile:

Number Message Description

Not applicable User is not Authorized to access Not applicable

the requested resource

MSAPI_00066 Only delete operation can be Not applicable

performed on a DELETE_
ATTEMPTED record

MSAPI_00066 Save operation is not allowed Not applicable

for record with provisioning
state {0}. Only provisioning or
unprovisioning is allowed.

MSAPI_00005 Invalid MEs in delete distribution Not applicable

list. Data is not distributed on
these MEs.

MSAPI_00105 Unprovisioning/Deleting is not Not applicable

allowed for this downstream
overload profile as it is referred
by one or more downstream
overload config or diameter
peer.

MSAPI_00066 Delete operation is not allowed Not applicable

for record with provisioning state
{0}.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 58

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00009 Record not found in SM Record not found {0}

MSAPI_00050 Invalid MEs in the request Not applicable

MSAPI_00075 Incompatible profile state Attached profile {0} should be in

RELEASE or ACTIVE state.

Table 6: Sub-component: DownstreamOverload Profile

Sub-component: Peerconnection Profile

The following section describes the error messages for sub-component Peerconnection Profile:

Number Message Description

MSAPI_00023 Invalid Value of (appending Not applicable

Enum Class Type)

NIL User is not Authorized to access Not applicable

the requested resource

MSAPI_00013 PeerConnection Profile must Not applicable

have at least one set configured
out of authapplicationids,
acctapplicationids,
vendorspecificauthapplication
ids, and
vendorspecificacctapplicationids

MSAPI_00005 Invalid MEs in delete distribution Not applicable

list. Data is not distributed on
these MEs.

MSAPI_00066 Only delete operation can be Not applicable

performed on a DELETE_
ATTEMPTED record

MSAPI_00066 Save operation is not allowed Not applicable

for record with provisioning
state {0}. Only provisioning or
unprovisioning is allowed.

MSAPI_00066 Delete operation is not allowed Not applicable

for record with provisioning state
{0}.

MSAPI_00009 Record not found in SM Record not found {0}

id9YZ-09148-MT11-PCZZA © 2020 Nokia 59

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00050 Invalid MEs in the request Not applicable

MSAPI_00102 Peer Connection profile Not applicable

cannot be unprovisioned or
deleted as there are diameter
peers associated to this peer
connection.

MSAPI_00096 Either authApplicationIds or For PeerConnectionProfile

acctApplicationIds contains {0}:Either authApplicationIds
invalid Value. or acctApplicationIds contains
invalid Value.

MSAPI_00162 Peer Connection Profile Error Reference Number:

contains an invalid AVPEntity MSAPI_00162.The Peer
AVP name. Connection Profile {0} has an
AVP Entity name of {1} that is
invalid. AVP of {1} is already
specified as an individual
attribute of the.

MSAPI_00013 PeerConnection Profile must Not applicable

have at least one set configured
out of authapplicationids,
acctapplicationids,
vendorspecificauthapplication
ids, and
vendorspecificacctapplicationids.

Table 7: Sub-component: Peerconnection Profile

Sub-component: SCTP Association Profile

The following section describes the error messages for sub-component SCTP Association Profile

Number Message Description

Not applicable User is not Authorized to access Not applicable

the requested resource

MSAPI_00023 Invalid Value of (appending Not applicable

Enum Class Type)

MSAPI_00066 Delete operation is not allowed Not applicable

for record with provisioning state
{0}.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 60

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00009 Record not found in SM Record not found {0}

MSAPI_00050 Invalid MEs in the request Not applicable

MSAPI_00066 Only delete operation can be Not applicable

performed on a DELETE_
ATTEMPTED record

MSAPI_00066 Save operation is not allowed Not applicable

for record with provisioning
state {0}. Only provisioning or
unprovisioning is allowed.

MSAPI_00102 Sctp Association profile Not applicable

cannot be unprovisioned or
deleted as there are diameter
peers associated to this peer
connection.

MSAPI_00005 Invalid MEs in delete distribution Not applicable

list. Data is not distributed on
these MEs.

Table 8: Sub-component: SCTP Association Profile

Sub-component: RequestTimeout Profile

The following section describes the error messages for sub-component RequestTimeout Profile:

Number Message Description

Not applicable User is not Authorized to access Not applicable

the requested resource

MSAPI_00023 Invalid Value of (appending Not applicable

Enum Class Type)

MSAPI_00066 Delete operation is not allowed Not applicable

for record with provisioning state
{0}.

MSAPI_00141 Request Timeout Profile cannot Not applicable

be unprovisioned or deleted
as it is referred by one or more
diameter peer .

id9YZ-09148-MT11-PCZZA © 2020 Nokia 61

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00142 Request timeout profile contains Not applicable

duplicate application code in
timeout configurations : {0}.

MSAPI_00143 The following application or Not applicable

command code is not supported
in diameter dictionary {0} .

MSAPI_00144 Application code not supported. Not applicable

MSAPI_00066 Only delete operation can be Not applicable

performed on a DELETE_
ATTEMPTED record

MSAPI_00066 Save operation is not allowed Not applicable

for record with provisioning
state {0}. Only provisioning or
unprovisioning is allowed.

MSAPI_00023 Invalid Value of (appending Not applicable

Enum Class Type)

MSAPI_00005 Invalid MEs in delete distribution Not applicable

list. Data is not distributed on
these MEs.

MSAPI_00009 Record not found in SM Record not found {0}

Table 9: Subcomponent: RequestTimeout Profile

Sub-component: KPI Statistics Profile

The following section describes the error messages for sub-component KPI Statistics Profile:

Number Message Description

MSAPI_00303 Statistics Profile cannot be Not applicable

unprovisioned or deleted as
there are Diameter peers
associated to this peer
connection.

MSAPI_00066 Delete operation is not allowed Not applicable

for record with provisioning state
{0}.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 62

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00066 Only delete operation can be Not applicable

performed on a DELETE_
ATTEMPTED record

MSAPI_00066 Save operation is not allowed Not applicable

for record with provisioning
state {0}. Only provisioning or
unprovisioning is allowed.

MSAPI_00009 Record not found in SM Record not found {0}

Not applicable User is not Authorized to access Not applicable

the requested resource

MSAPI_00005 Invalid MEs in delete distribution Not applicable

list. Data is not distributed on
these MEs.

Table 10: Sub-component: KPI Statistics Profile

Sub-component: Routing Profile

The following section describes the error messages for sub-component Routing Profile:

Number Message Description

Not applicable User is not Authorized to access Not applicable

the requested resource

MSAPI_00066 Delete operation is not allowed Not applicable

for record with provisioning state
{0}.

MSAPI_00066 Only delete operation can be Not applicable

performed on a DELETE_
ATTEMPTED record

MSAPI_00066 Save operation is not allowed Not applicable

for record with provisioning
state {0}. Only provisioning or
unprovisioning is allowed.

MSAPI_00009 Record not found in SM Record not found {0}

MSAPI_00219 Unprovisioning/Deleting is not Not applicable

allowed for this routing plugin

id9YZ-09148-MT11-PCZZA © 2020 Nokia 63

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

profile as it is referred by one or

more routing plugin config

ROUTING_00002 destinationId not distributed Not applicable

MSAPI_00005 Invalid MEs in delete distribution Not applicable

list. Data is not distributed on
these MEs.

Table 11: Sub-component: Routing Profile

Sub-component: Realm Based Routes

The following section describes the error messages for sub-component Realm Based Routes:

Number Message Description

MSAPI_00023 smProvState parameter must Not applicable

not be part of Request Body

MSAPI_00041 % not allowed. Not applicable

MSAPI_00174 Invalid Pattern for Realm. Not applicable

Not applicable User is not Authorized to access Not applicable

the requested resource

MSAPI_00005 Application parameter is invalid Not applicable

MSAPI_00009 Record not found in SM Record not found {0}

MSAPI_00012 Record does not exist Not applicable

MSAPI_00066 Delete operation is not allowed Not applicable

for record with provisioning state
{0}.

MSAPI_00066 Only delete operation can be Not applicable

performed on a DELETE_
ATTEMPTED record

MSAPI_00066 Save operation is not allowed Not applicable

for record with provisioning
state {0}. Only provisioning or
unprovisioning is allowed.

MSAPI_00012 Record does not exist Not applicable

MSAPI_00019 An active configuration cannot Not applicable

be deleted

id9YZ-09148-MT11-PCZZA © 2020 Nokia 64

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00025 Already Provisioned Not applicable

BULK_UNDISTRIBUTION_ Bulk UnDistribution Status : Not applicable

00001
PARTIAL \

Please refer :

/opt/tpa/logs/Provisioning.log

file for details.

BULK_UNDISTRIBUTION_ Bulk UnDistribution Status : Not applicable

00002 FAILED \

Please refer :

/opt/tpa/logs/Provisioning.log

file for details.

BULK_UNDISTRIBUTION_ Managed Element not Not applicable

00003 provisioned to any record.

MSAPI_00223 Bulk Delete Status : FAILED. Bulk Delete Status : FAILED \

Successfully deleted : {0}

records

Please refer /opt/tpa/logs/

DDESMApplication.log for

more details.

MSAPI_00224 Bulk Delete Status : PARTIAL. Bulk Delete Status : PARTIAL

\ Successfully deleted : {0}

records

Failed to delete : {1} records

Please refer /opt/tpa/logs/

DDESMApplication.log for

more details.

Table 12: Sub-component: Realm Based Routes

Sub-component: Host Based Routes

The following section describes the error messages for sub-component Host Based Routes:
id9YZ-09148-MT11-PCZZA © 2020 Nokia 65
2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00041 % not allowed. Not applicable

MSAPI_00174 Invalid Pattern for Realm. Not applicable

Not applicable User is not Authorized to access Not applicable

the requested resource

MSAPI_00005 Application parameter is invalid Not applicable

MSAPI_00009 Record not found in SM Record not found{0}

MSAPI_00012 Record does not exist Not applicable

MSAPI_00066 Delete operation is not allowed Not applicable

for record with provisioning state
{0}.

MSAPI_00066 Only delete operation can be Not applicable

performed on a DELETE_
ATTEMPTED record

MSAPI_00066 Save operation is not allowed Not applicable

for record with provisioning
state {0}. Only provisioning or
unprovisioning is allowed.

MSAPI_00012 Record does not exist Not applicable

MSAPI_00019 An active configuration cannot Not applicable

be deleted.

MSAPI_00025 Already Provisioned Not applicable

BULK_UNDISTRIBUTION_ Bulk UnDistribution Status : Not applicable

00001
PARTIAL \

Please refer :

/opt/tpa/logs/Provisioning.log

file for details.

BULK_UNDISTRIBUTION_ Bulk UnDistribution Status : Not applicable

00002
FAILED \

Please refer :

/opt/tpa/logs/Provisioning.log

file for details.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 66

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

BULK_UNDISTRIBUTION_ Managed Element not None of the selected records

00003
provisioned to any record. are \ provisioned to the

selected ME {0}.

MSAPI_00223 Bulk Delete Status : FAILED. Bulk Delete Status : FAILED \

Successfully deleted : {0}

records \

Failed to delete : {1} records \

Please refer /opt/tpa/logs/

DDESMApplication.log for

more details.

MSAPI_00224 Bulk Delete Status : PARTIAL. Bulk Delete Status : PARTIAL

Successfully deleted : {0}

records

Failed to delete : {1} records

Please refer /opt/tpa/logs/

DDESMApplication.log for

more details.

MSAPI_00147 Application codes {0} are Not applicable

invalid.

Table 13: Sub-component: Host Based Routes

SM Error component: SS7 SLF

The following section describes the SS7 SLF errors:

Sub-component: SS7 Destination

The following section describes the error messages for sub-component SS7 Destination:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 67

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00066 Delete operation is not allowed Not applicable

for record with provisioning state
{0}.

MSAPI_00066 Only delete operation can be Not applicable

performed on a DELETE_
ATTEMPTED record

MSAPI_00066 Save operation is not allowed Not applicable

for record with provisioning
state {0}. Only provisioning or
unprovisioning is allowed.

Not applicable User is not Authorized to access Not applicable

the requested resource

MSAPI_00115 SS7SLFDestination Network- Not applicable

Reference Id is in Use and
Cannot be Deleted

MSAPI_00023 Invalid Value. Invalid Value for {Entity name}

MSAPI_00114 SS7 SLF IdentityToDestination SS7 SLF IdentityToDestination

Network Id is Invlaid OR does Network Id: {0} does not Exist.
not Exist.

MSAPI_00119 Subscriber Id should not be Not applicable

more than 15 digits.

MSAPI_00009 Record not found in SM Record not found {0}

Table 14: Sub-component: SS7 Destination

Number Message Description

Not applicable User is not Authorized to access Not applicable

the requested resource

MSAPI_00066 Delete operation is not allowed Not applicable

for record with provisioning state
{0}.

MSAPI_00066 Only delete operation can be Not applicable

performed on a DELETE_
ATTEMPTED record

id9YZ-09148-MT11-PCZZA © 2020 Nokia 68

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00066 Save operation is not allowed Not applicable

for record with provisioning
state {0}. Only provisioning or
unprovisioning is allowed.

MSAPI_00009 Record not found in SM Record not found {0}

MSAPI_00005 Invalid MEs in delete distribution Not applicable

list. Data is not distributed on
these MEs.

Not applicable There are more than {0} records There are more than 10000
configured for SS7 Identity To records configured for SS7
Destination List. Please filter Identity To Destination
based on Subscriber Id only. List. Please filter based on
Subscriber Id only.

SLF_00001 Missing or empty Id. Error Reference Number:

SLF_00001.The Identity to
Destination has a missing or
empty Id.

SLF_00002 Invalid Id format. Error Reference Number:

SLF_00002.The
IdentityToDestination
configuration with id {0} is of
incorrect format. It must contain
Subscriber Group id and identity
id separated by semi-colon.

SLF_00003 Subscriber Group Id is not a Error Reference Number:

number. SLF_00003.The
IdentityToDestination
configuration with id {0} contains
an entry for the Subscriber
Group Id in the identity pattern
that is not a valid number.

SLF_00004 Subscriber Group Id mismatch. Error Reference Number:

SLF_00004.The
IdentityToDestination
configuration with id {0} and
Subscriber Group id {1} do not
contain matching Subscriber
Group ids.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 69

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

SLF_00005 Subscriber Group for Id does Error Reference Number:

not exist. SLF_00005.The
IdentityToDestination
configuration with id {0} and
Subscriber Group id {1} has
a Subscriber Group id that
does not match an existing
Subscriber Group.

SLF_00006 Missing or empty Destination Error Reference Number: SLF_

string. 00006.One or more destination
string(s) in the Identity to
Destination configuration with Id
{0} is/are missing or empty.

SLF_00007 Invalid Destination Strings. Error Reference Number: SLF_

00007.One or more destination
string(s) in the Identity to
Destination configuration with Id
{0} has/have too many commas.

Sub-component: SS7 Peer Configuration

The following section describes the error messages for sub-component SS7 Peer Configuration:

Number Message Description

Not applicable User is not Authorized to access Not applicable

the requested resource

MSAPI_00066 Delete operation is not allowed Not applicable

for record with provisioning state
{0}.

MSAPI_00066 Only delete operation can be Not applicable

performed on a DELETE_
ATTEMPTED record

MSAPI_00066 Save operation is not allowed Not applicable

for record with provisioning
state {0}. Only provisioning or
unprovisioning is allowed.

MSAPI_00009 Record not found in SM Record not found {0}

id9YZ-09148-MT11-PCZZA © 2020 Nokia 70

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00005 Invalid MEs in delete distribution Not applicable

list. Data is not distributed on
these MEs.

MSAPI_00023 Invalid Value. Invalid Value for {Entity Name}

Not applicable Failed Not applicable

MSAPI_00302 SS7 Peer Configuration Not applicable

cannot be deleted as it is
associated with one or more
SS7 IWFConfig Forms.

MSAPI_00110 Routing Context out of range. The Routing Context must be

between {0} and {1} or null.

MSAPI_00121 Invalid Value of Primary Point Invalid Value of Primary Point

Code,Only allow numeric Value Code,Only allow numeric Value
and special character (-) and special character (-)

MSAPI_00122 Invalid Value of Secondary Invalid Value of Secondary

Point Code,Only allow non- Point Code. Allowed value is
zero numeric Value and special either single numeric Value
character (-) OR \ three numeric values,
were value ranges between 0
to 255 that are separated by -
(hyphens)\

Example: X-Y-Z where

0<=X<=255, 0<=Y<255,
0<=Z<=255.

MSAPI_00111 SS7PeerConfig Endpoint The SS7PeerConfig Endpoint

Address is a mandatory field. Address field is mandatory.

MSAPI_00112 SS7PeerConfig Endpoint The SS7PeerConfig Endpoint

Address has an out of range Address has an out of range
port number. port number: {0}. A valid port
number is 1025 to 65535.

MSAPI_00116 EndPointAddress hostNames EndPointAddress hostNames

cannot be Null. cannot be Null.

MSAPI_00113 SS7PeerConfig Endpoint The SS7PeerConfig Endpoint

Address has an invalid host Address contains an invalid host
name or IP address. name or IP address:{0}.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 71

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00128 Duplicate HostName. HostName {0} already in Use.

MSAPI_00129 HostName and PortNumber HostName and PortNumber

Combination Should be Unique. Combination Should be Unique

Table 15: Sub-component: SS7 Peer Configuration

Sub-component: SS7 System Profile

The following section describes the error messages for subcomponent SS7 System Profile:

Number Message Description

Not applicable User is not Authorized to access Not applicable

the requested resource

MSAPI_00118 Only One SS7SystemProfile Not applicable

could be active at SM side.
sm.ss7slfconfig.active.error.
desc=SS7 System Profile is
already Active, Undistribute
or Delete operations are not
allowed.

MSAPI_00023 Invalid Value. Invalid Value for {Entity Name}

NIL User is not Authorized to access Not applicable

the requested resource

MSAPI_00118 Active SS7SystemProfile cannot Not applicable

be deleted.

MSAPI_00066 Delete operation is not allowed Not applicable

for record with provisioning state
{0}.

MSAPI_00066 Only delete operation can be Not applicable

performed on a DELETE_
ATTEMPTED record

MSAPI_00066 Save operation is not allowed Not applicable

for record with provisioning
state {0}. Only provisioning or
unprovisioning is allowed.

MSAPI_00009 Record not found in SM Record not found {0}

id9YZ-09148-MT11-PCZZA © 2020 Nokia 72

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00005 Invalid MEs in delete distribution Not applicable

list. Data is not distributed on
these MEs.

MSAPI_00118 Only One SS7SystemProfile Not applicable

could be active at SM side.

MSAPI_00118 Active SS7SystemProfile Not applicable

cannot be updated.Only active
Parameter allowed to get
Change while profile is Active.

MSAPI_00118 SS7 System Profile is already Not applicable

Active,Undistribute or Delete
operations are not allowed.

MSAPI_00106 Null or Empty Record not Null or Empty Record for SS7
allowed. System Profile : {1} is not
allowed for {0}.

MSAPI_00126 Each record of SIGNALING Record : {0} of SIGNALING

POINTCODES of SS7 System POINTCODES of SS7 System
Profile must not contain only 0 Profile must not contain only 0
as it's Value. as it's Value.

MSAPI_00127 Invalid Value of SIGNALING Invalid Value of SIGNALING

POINTCODES. POINTCODES : {0}.Max Value
for tingLabelFormat:ANSI_SLS_
8_BIT is 2^24-1 and Max Value
for RoutinglabelFormat:ITU is
2^14-1.

MSAPI_00123 Invalid Value of SIGNALING Invalid Value of SIGNALING

POINTCODES. POINTCODES : {0}.
Max hyphens(-) for
RoutingLabelFormat:ANSI_
SLS_8_BIT is 2 and
Max hyphens for
RoutinglabelFormat:"ITU" is 3.

MSAPI_00107 Network Appearance out of The Network Appearance must

range. be between {0} and {1} or null.

MSAPI_00125 Invalid Value of SIGNALING Invalid Value of SIGNALING

POINT CODE in POINT CODE in

id9YZ-09148-MT11-PCZZA © 2020 Nokia 73

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

SCCPADDRESS Of SS7 SCCPADDRESS Of SS7

System Profile. System Profile.

MSAPI_00108 Digits is a mandatory field The Digits field is mandatory

when Numbering Plan is set to
{0}.

MSAPI_00109 Signaling Point Code is a The Signaling Point Code field

mandatory field. is mandatory when Routing
Indicator is set to {0}.

Table 16: Sub-component: SS7 System Profile

Subcomponent: SS7 IWF Configuration

The following section describes the error messages for subcomponent SS7 IWF Configuration:

Number Message Description

Not applicable User is not Authorized to access Not applicable

the requested resource

MSAPI_00066 Delete operation is not allowed Not applicable

for record with provisioning state
{0}.

MSAPI_00066 Only delete operation can be Not applicable

performed on a DELETE_
ATTEMPTED record

MSAPI_00066 Save operation is not allowed Not applicable

for record with provisioning
state {0}. Only provisioning or
unprovisioning is allowed.

MSAPI_00009 Record not found in SM Record not found {0}

MSAPI_00005 Invalid MEs in delete distribution Not applicable

list. Data is not distributed on
these MEs.

MSAPI_00106 Null or Empty Record not Null or Empty Record for SS7
allowed. System Profile : {1} is not
allowed for {0}.

MSAPI_00210 Invalid Value of SIGNALING Not applicable

POINT CODE in Peer

id9YZ-09148-MT11-PCZZA © 2020 Nokia 74

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

SCCPADDRESS Of
SS7IWFConfig.

MSAPI_00108 Digits is a mandatory field. The Digits field is mandatory

when Numbering Plan is set to
{0}.The Digits field is mandatory
when Numbering Plan is set to
{0}.

MSAPI_00109 Signaling Point Code is a The Signaling Point Code field

mandatory field. is mandatory when Routing
Indicator is set to {0}.

Table 17: Subcomponent: SS7 IWF Config

SM Error component: Topology Hiding Configuration

The following section describes the error messages for Topology hiding configuration:

Sub-component: Topology Hiding Configuration

The following section describes the error messages for sub-component Topology Hiding
Configuration:

Number Message Description

Not applicable User is not Authorized to access Not applicable

the requested resource

MSAPI_00066 Delete operation is not allowed Not applicable

for record with provisioning state
{0}.

MSAPI_00066 Only delete operation can be Not applicable

performed on a DELETE_
ATTEMPTED record

MSAPI_00066 Save operation is not allowed Not applicable

for record with provisioning
state {0}. Only provisioning or
unprovisioning is allowed.

MSAPI_00009 Record not found in SM Record not found {0}

id9YZ-09148-MT11-PCZZA © 2020 Nokia 75

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

IMPORT_00003 Import Request contents are not Not applicable

correct.

Table 18: Sub-component: Topology Hiding Configuration

Sub-component: Home Network Identity

The following section describes the error messages for sub-component Home Network Identity:

Number Message Description

Not applicable User is not Authorized to access Not applicable

the requested resource

Not applicable Generic exception is thrown Not applicable

while fetching list of object
of type"com.nokia.dde.
ddmapps.topologyhiding.par.
HomeNetworkInfo"

MSAPI_00081 Generic Exception is thrown Not applicable

Table 19: Sub-component: Home Network Identity

SM Error component: Diameter Peer

The following section describes the error messages for Diameter Peer:

Number Message Description

MSAPI_00037 TCP, SCTP and TLS are only Not applicable

supported protocols.

MSAPI_00018 Primary address should not be Not applicable

repeated in secondary address
set

MSAPI_00164 Peer Type can be of either Fqdn Not applicable

or Regex.

MSAPI_00041 % not allowed. Not applicable

MSAPI_00163 Outbound Origin host should not Not applicable

be of type Reg Ex.

MSAPI_00029 Request Timeout should either Not applicable

be 0 or between 50 and 60000

id9YZ-09148-MT11-PCZZA © 2020 Nokia 76

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

DIAMETER_00020 Invalid port number. The value {0} is not in the valid
port number range of \ {1} to {2}.

MSAPI_00098 Invalid peer origin-host The origin-host {0} is not a valid

Diameter FQDN \ (fully qualified
domain name).

MSAPI_00139 Invalid Secondary Address The Secondary address: {0} is

not a valid FQDN.

MSAPI_00201 Invalid address group. Error Reference Number:

MSAPI_00201.\n\ Address
group can not be empty.

MSAPI_00202 Invalid address group. Error Reference Number:

MSAPI_00202.\n\ Address
group {0} for protocol {1} does
not exists.

MSAPI_00132 Invalid Primary Address Error Reference Number:

MSAPI_00132.Invalid \ Primary
Address : {0}.

MSAPI_00009 Record not found in SM Record not found {0}

MSAPI_00069 Invalid INGRESS peer profile Not a valid ingress peer profile
type {0}

MSAPI_00075 Incompatible profile state Attached profile {0} should be in

RELEASE or ACTIVE state

MSAPI_00068 Invalid EGRESS peer profile Not a valid egress peer profile
type {0}

MSAPI_00098 Invalid peer origin-host The origin-host {0} is not a valid

Diameter FQDN \ (fully qualified
domain name).

MSAPI_00165 Invalid IP Subnet IP subnet: {0} is invalid.

Connection type Inbound can \
have only valid IP subnet.

MSAPI_00140 Invalid IP address IP address: {0} is invalid.

Connection type Inbound can \
have only valid IP address.

BULK_UNDISTRIBUTION_ Bulk UnDistribution Status : Not applicable

00001 PARTIAL \ Please refer : /opt/

id9YZ-09148-MT11-PCZZA © 2020 Nokia 77

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

tpa/logs/Provisioning.log file for

details.

BULK_UNDISTRIBUTION_ Bulk UnDistribution Status : Not applicable

00002 FAILED \ Please refer : /opt/
tpa/logs/Provisioning.log file for
details

BULK_UNDISTRIBUTION_ Managed Element not None of the selected records

00003 provisioned to any record. are \ provisioned to the selected
ME {0}.

MSAPI_00223 Bulk Delete Status : FAILED Bulk Delete Status : FAILED

Successfully deleted : {0}
records Failed to delete : {1}
records Please refer /opt/tpa/
logs/DDESMApplication.log for
more details.

MSAPI_00224 Bulk Delete Status : PARTIAL Bulk Delete Status : PARTIAL

Successfully deleted : {0}
records Failed to delete : {1}
records Please refer /opt/tpa/
logs/DDESMApplication.log for
more details.

SM Error component: Configuration Entities

The following section describes the error messages for Configuration Entities:

Sub-component: Throttling Configuration

The following section describes the error messages for sub-component Throttling Configuration:

Number Message Description

MSAPI_00077 The value for consecutive The consecutive throttling

throttled intervals before raising before raising alarm must be set
alarm is invalid. to a value within the range of {0}
and {1}

MSAPI_00009 Record not found in SM Record not found {0}

MSAPI_00059 Invalid profile type {0} is not a valid {1} profile type

MSAPI_00075 Incompatible profile state Attached profile {0} should be in

RELEASE or ACTIVE state

id9YZ-09148-MT11-PCZZA © 2020 Nokia 78

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00070 profile not distributed {0} profile not distributed on

MEs {1}

Not applicable User is not Authorized to access Not applicable

the requested resource

Table 20: Sub-component: Throttling Configuration

Sub-component: Downstream Overload Configuration

The following section describes the error messages for sub-component Downstream Overload
Configuration:

Number Message Description

Not applicable DownStreamOverLoadConfig Not applicable

Name is missing.

Not applicable DownStreamOverLoadConfig Not applicable

Name's character range is 1 to
50 characters

Not applicable maximum size for profile Not applicable

description is 255 chars

MSAPI_00009 Record not found in SM Record not found {0}

MSAPI_00075 Incompatible profile state Attached profile {0} should be in

RELEASE or ACTIVE state

Not applicable User is not Authorized to access Not applicable

the requested resource

Table 21: Sub-component: Downstream Overload Configuration

SM Error component: Import/Export

The following section desccribes the error messages for Import/Export

Sub-component: Import Functionality

The following section describes the error messages for sub-component Import functionality:

Number Message Description

IMPORT_00003 Request Content for Entity does Not applicable

not exist in Import Request.

IMPORT_00005 CSV file is missing or incorrect Not applicable

id9YZ-09148-MT11-PCZZA © 2020 Nokia 79

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

IMPORT_00006 Import Partially Failed, Please Not applicable

refer /opt/tpa/logs/ImportExport.
log file for details.

IMPORT_00007 The version information in the Not applicable

imported data did not describe a
supported version.

MSAPI_00134 ME doesn't exist Not applicable

MSAPI_00118 Active SS7SystemProfile Not applicable

cannot be updated. Only active
Parameter allowed to get
Change while profile is Active.

Not applicable Duplicate Keys or key already Not applicable

exist in DB

Not applicable Error while creating a record Not applicable

locally in SM

Not applicable Error while provisioning the Not applicable

imported data:{ENTITY_ID}

Not applicable Import request has invalid/ Not applicable

missing required attributes
{ENTITY_ID}

Not applicable Error while Not applicable

Distribution{ENTITY_ID}

Not applicable Provisioned Data cannot be Not applicable

Overrided{ENTITY_ID}

Not applicable No Json Request for Entity: Not applicable

{ENTITY_NAME}

Not applicable Invalid Data{ENTITY_ID} Not applicable

Not applicable Error while creating record Not applicable

locally as ID is NULL

Not applicable Only One Active profile can Not applicable

exist on SM

Not applicable Invalid SLF Lookup Table Id in Not applicable

import request.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 80

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

Not applicable Internal error due to invalid data. Not applicable

"Please refer /opt/tpa/logs/
ImportExport.log file for details"

Not applicable Request Content for Entity does Not applicable

not exist in Import Request.

Not applicable Import file contains {0} SLF Not applicable

Lookup Table object in json file
which is not supported.

Not applicable Invalid lookup type found in Not applicable

imported file. Valid lookup types
are : {EXACT, IP_SUBNET,
NUMERIC_RANGE, REGEX}

MSAPI_00134 ME doesn't exist Not applicable

MSAPI_00118 Active SS7SystemProfile Not applicable

cannot be updated. Only active
Parameter allowed to get
Change while profile is Active.

IMPORT_DISTRIBUTION_ Import is successful, but Not applicable

00001 distribution failed, Please refer /
opt/tpa/logs/Provisioning.log file
for details.

Table 22: Sub-component: Import Functionality

Sub-component : Export Functionality

The following section describes the error messages for sub-component Export functionality:

Number Message Description

MSAPI_00220 Exporting through CSV is Not applicable

restricted to 10000 records.
Please export the records in
JSON.

MSAPI_00031 Invalid Query parameters. Not applicable

Please specify correct
parameters.

EXPORT_00002 Specified Queryparam Not applicable

'entityname' is not correct.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 81

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

EXPORT_00001 Unsupported/ Invalid Data Type Not applicable

EXPORT_00001 Invalid Request. Not applicable

EXPORT_00001 Request empty or Null. Not applicable

Table 23: Sub-component: Export Functionality

SM Error component: RSV

The following section describes the error messages for RSV:

Rule Errors

The following section describes the Rule errors:

Number Message Description

RULE_00001 Name is a mandatory field. Error Reference Number:

RULE_00001. The mandatory
Name field provides a unique
name for this object and must
be 1 to 255 characters.

RULE_00002 Name exceeds the maximum Error Reference Number:

length of 255 characters. RULE_00002. The Name field
must be 1 to 255 characters.

RULE_00003 Empty condition Criteria value. Error Reference Number:

RULE_00003. The Value field
of the condition criteria must be
assigned a value. The behavior
of the Value field depends on
the option that you chose for the
Value Type field.

RULE_00004 Empty Operator field for Criteria. Error Reference Number:

RULE_00004. The Operator
field must contain an Operator
value; for example, Equal.

RULE_00005 Empty Criteria value. Error Reference Number:

RULE_00005. You must assign
a condition in the Criteria drop-
down list.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 82

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

RULE_00007 Empty Action Attribute Error Reference Number:

Information. RULE_00007. An Action needs
an Action attribute.

RULE_00008 Name data is invalid. Error Reference Number:

RULE_00008. The name data
field should only contain letters,
numbers, underscores (_),
dashes (-), dots (.), and spaces.

RULE_00009 A rule needs at least one Action. Error Reference Number:

RULE_00009. Create at least
one Action for a rule.

RULE_00400 Attribute argument "{0}" is not Error Reference Number:

set. RULE_00400. If an Attribute
[Condition Criteria, Action, or
AttributeValueType] of a source/
result context is defined with
one or more arguments, then
all the Attribute arguments are
required in order to successfully
resolve the Attribute in Rule
Engine.

RULE_00401 Attribute argument "{0}" is not Error Reference Number:

defined in the dictionary. RULE_00401. The arguments of
an Attribute [Condition Criteria,
Action, or AttributeValueType]
must match the arguments in
the dictionary.

RULE_00402 Types are not compatible. Error Reference Number:

RULE_00402. The type
[type="{0}", class="{1}",
occurrence="{2}"] is not
compatible with [type="{3}",
class="{4}", occurrence="{5}"].

RULE_00404 Unable to find class "{0}". Error Reference Number:

RULE_00404. The system was
unable to find the class with
name "{0}".

id9YZ-09148-MT11-PCZZA © 2020 Nokia 83

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

RULE_00104 Argument value does not allow Error Reference Number:

arguments. RULE_00104. The value of
argument "{0}" does not allow
arguments

RULE_00106 Rule System Version is null Error Reference Number:

RULE_00106. Provided Rule
System Version object is null.

RULE_00107 Unable to find given Rule Error Reference Number:

System Version RULE_00107. Unable to find
the Rule System Version in the
database with name="{0}".

RULE_00108 Rule System Version in Active Error Reference Number:

state cannot be changed. RULE_00108. State of currently
Active Rule System Version
cannot be changed.

RULE_00109 Current Rule System Version Error Reference Number:

cannot be activated RULE_00109. Rule System
Version "{0}" needs to be set to
Active state in order to activate.

RULE_00110 Timed out waiting for Rule Error Reference Number:

System Version activation lock RULE_00110. Timed out
waiting for Rule System Version
activation lock.

RULE_00111 Exception acquiring activation Error Reference Number:

lock RULE_00111. Exception
acquiring activation lock.

RULE_00112 Rule System Version activation Error Reference Number:

failed RULE_00112.Rule System
Version activation failed.

RULE_00114 Could not switch Rule System Error Reference Number:

Version state RULE_00114. Could not switch
Rule System Version state.

RULE_00115 Could not fetch Rule System Error Reference Number:

Version RULE_00115. Could not fetch
Rule System Version.

RULE_00116 Could not clone Rule System Error Reference Number:

Version RULE_00116. Could not clone

id9YZ-09148-MT11-PCZZA © 2020 Nokia 84

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

Rule System Version with

specified name "{0}".

RULE_00118 Could not initialize Rule System Error Reference Number:

Version export. RULE_00118.=Could not
initialize Rule System Version
export.

RULE_00119 Name not provided for imported Error Reference Number:

Rule System Version RULE_00119. Name not
provided for imported Rule
System Version.

RULE_00120 Could not initialize Rule System Error Reference Number:

Version import/export RULE_00120. Could not
initialize Rule System Version
import/export.

RULE_00011 Default action "{0}.{1}" is not Error Reference Number:

valid for rule set "{2}" RULE_00011. The specified
default action or one of the
attribute values used in it are
not applicable for the rule set.

RULE_00017 Error while processing Action Error Reference Number:

"{0}" RULE_00017. The action data
may be corrupt. Try re-creating
and saving the action.

RULE_00018 A rule table that is being used Error Reference Number:

as an Action by a rule cannot be RULE_00018. The rule table
a top-level rule table must be re-created and not
flagged as a top-level rule table,
or the rule must use another
rule table that is not flagged
as a top-level rule table as an
action.

RULE_00019 Reference to rule table "{0}" Error Reference Number:

from rule table "{1}" causes a RULE_00019. The rule table
loop assigned to the rule table action
is already used in this rule table
action link. The rule table should
not be used again in this link to
avoid a rule table loop.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 85

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

RULE_00083 Rule names within a rule table Error Reference Number:

must be unique RULE_00083. The Name field
of each rule must be unique.
Modify the rule names of the
rule table and try saving again.

RULE_00085 Rule set is associated to an Error Reference Number:

invalid rule table RULE_00085. Rule set "{0}" is
associated to invalid rule table
"{1}".

RULE_00086 Action "{0}" is used too many Error Reference Number:

times in rule set "{1}" RULE_00086. The action "{0}"
in rule set "{1}" may only be
used "{2}" time(s).

RULE_00087 Rule Description exceeds 255 characters. Error Reference

the maximum length of 255 Number: RULE_00087. Rule
characters. "{0}" has a description "{1}"
characters long. The Rule
Description field must be 0 to
255 characters.

RULE_00089 Rule table "{0}" does not belong Error Reference Number:
to correct rule sets RULE_00089. The rule table
"{0}" is referenced by rule table
"{1}", so rule table "{0}" must
include at least the rule sets of
rule table "{1}".

RULE_00020 Attribute name "{0}" and result Error Reference Number:

context "{1}" does not match RULE_00020. The specified
any Attributes in Rule Attribute action Name and result context
Dictionary. does not match any attributes
in the rule dictionaries. The rule
dictionary files may be corrupt
or out of date.

RULE_00021 Result type "{0}" is outside the Error Reference Number:

valid range of: "{1}" to "{2}" RULE_00021. The rule
dictionary files may be corrupt
or out of date. Reinstall the rule
dictionary files on the system.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 86

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

RULE_00022 Attribute name "{0}" and source Error Reference Number:

context "{1}" does not match RULE_00022. The specified
any Attributes in Rule Attribute attribute Name and source
Dictionary context does not match any
attributes in the rule dictionaries.
The rule dictionary files may be
corrupt or out of date.

RULE_00023 Attribute "{0}" used as Attribute Error Reference Number:

Value Type is not found in the RULE_00023.The attribute
dictionary value type was not found in the
rule dictionaries.

RULE_00025 Error while getting Object from Error Reference Number:

the ValueType "{0}" RULE_00025. An error occurred
when trying to retrieve the
object from the value type.
Close and open the form, or
refresh and save the form.

RULE_00026 Type mismatch between Error Reference Number:

Attribute Value Type "{0}", RULE_00026. See the system
source context "{1}", attribute attribute rule dictionary file
"{2}" defined in rule dictionary for valid attribute value type
matches.

RULE_00027 A QoS Information entity needs Error Reference Number:

a QCI assigned to it. RULE_00027. A QoS Class
Identifier (QCI) must be
assigned to a QoS information
entity. A QoS information
entity can be assigned within
several other entities such as
QoS information action. If any
attribute of QoS information is
set within the main entity, a QCI
must also be present. However,
the main entity can be saved
without any QoS information.

RULE_00029 AttributeValueType properties Error Reference Number:

name or sourceContext cannot RULE_00029. The mandatory
be null. properties name and

id9YZ-09148-MT11-PCZZA © 2020 Nokia 87

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

sourceContext must be set for

the AttributeValueType.

RULE_00032 Error resolving rule attribute Error Reference Number:

dictionary. RULE_00032. The system
cannot resolve the rule attribute
dictionary cache.

RULE_00033 Action "{0}" is used too many Error Reference Number:

times in rule "{1}" RULE_00033. The action "{0}"
in rule "{1}" may only be used
"{2}" time(s) in a single rule.

RULE_00034 Action value "{0}" is too large for Error Reference Number:
attribute "{1}" RULE_00034.The value for
the action is too large for the
attribute.

RULE_00035 Adjustment value "{0}" is too Error Reference Number:

large for Integer attribute RULE_00035. The adjustment
value for the action is too large
for the attribute.

RULE_00037 Multiple occurrences of the Error Reference Number:

Charging/QoS rule definition RULE_00037. You cannot use
in Action "{0}" rule "{1}" is not the same Charging/QoS rule
allowed definition name multiple times
with action "{0}" within a single
rule.

RULE_00406 Action "{0}.{1}" does not have Error Reference Number:

parameter named "{2}" RULE_00406. All provided
parameters must be defined in
the attribute dictionary for the
action attribute.

RULE_00407 Action parameter "{2}" in action Error Reference Number:

"{0}.{1}" is not set RULE_00407. All mandatory
parameters defined in the
attribute dictionary for the action
must have a value.

RULE_00408 Action parameter "{0}" does not Error Reference Number:

have a value RULE_00408. All action
parameters must have a value.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 88

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

RULE_00412 QCI value is out of range. Error Reference Number:

RULE_00412. QCI value "{0}"
for attribute "{1}" must be in the
range of 1-9, 65, 66, 69, 70 and
128-254, all inclusive.

RULE_00413 Invalid custom data name Error Reference Number:

length. RULE_00413. The name "{0}"
for attribute "{1}" can contain 1
to {2} characters only.

RULE_00100 Condition value does not allow Error Reference Number:

arguments. RULE_00100. The value of
condition "{0}" does not allow
arguments.

RULE_00101 Condition operator is not Error Reference Number:

compatible with criteria attribute. RULE_00101. The operator
"{0}" for condition "{1}" is not
compatible with the criteria
attribute type.

RULE_00102 Condition operator is not Error Reference Number:

compatible with condition value. RULE_00102. The operator
"{0}" for condition "{1}" is not
compatible with the condition
value type.

RULE_00103 Condition value "{0}" is too large Error Reference Number:

for attribute "{1}" RULE_00103.The value for the
condition is too large for the
attribute.

RULE_00410 Condition adjustment value Error Reference Number:

does not allow arguments. RULE_00410. One or more
of the adjustment values of
condition [criteria name "{0}"]
does not allow arguments.

RULE_00411 Present/Not Present are not Error Reference Number:

applicable to non-nullable RULE_00411. Non-nullable
attributes. attribute "{0}.{1}" cannot be
used with the Present and Not
Present comparison operators.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 89

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

RULE_00150 Condition adjustment value Error Reference Number:

is prohibited if no adjustment RULE_00150. If a condition
operator is provided. does not have an adjustment
operator selected, no
adjustment values can be
provided.

RULE_00151 Condition adjustment value Error Reference Number:

is mandatory if an adjustment RULE_00151. If a condition
operator is provided. has an adjustment operator
selected, at least one
adjustment value must be
provided.

RULE_00152 Criteria adjustment value is Error Reference Number:

prohibited if no adjustment RULE_00152. If a criteria does
operator is provided. not have an adjustment operator
selected, no adjustment values
can be provided.

RULE_00153 Criteria adjustment value is Error Reference Number:

mandatory if an adjustment RULE_00153.If a criteria has an
operator is provided. adjustment operator selected, at
least one adjustment value must
be provided.

RULE_00040 The Active Rule System Version Error Reference Number:

cannot be deleted. RULE_00040. Rule System
Version "{0}" is in an Active
state and cannot be deleted.

RULE_00042 Modifying the description of a Error Reference Number:

read-only Rule System Version RULE_00042. Rule System
is not allowed. Version "{0}" is flagged as
read-only. You cannot remove,
add or modify the description.
Current description is "{1}",
changed description is "{2}".

RULE_00043 Active or Release Rule System Error Reference Number:

Versions cannot be modified. RULE_00043. Active or release
rule system versions cannot be
modified.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 90

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

RULE_00044 Cannot activate a Rule System Error Reference Number:

Version from a Draft state. RULE_00044.The state of rule
system version "{0}" cannot be
changed from Draft to Active.

RULE_00045 Read-only flag of Rule System Error Reference Number:

Version cannot be modified. RULE_00045.The
ReadOnlyAccess property of
rule system version "{0}" cannot
be modified during an update.

RULE_00046 Failed to deactivate Rule Error Reference Number:

System Version. RULE_00046. Failed to
deactivate the previously active
rule system version during
activation of rule system version
"{0}".

RULE_00047 Rule engine reload failed. Error Reference Number:

RULE_00047. Failed to reload
rule engine after activation of
rule system version "{0}".

RULE_00048 Invalid Rule System Version Error Reference Number:

state change - Active to Draft. RULE_00048.The state of rule
system version "{0}" cannot be
changed from Active to Draft.

RULE_00050 Rule table name must be unique Error Reference Number:

within the rule system version. RULE_00050.Rule tables must
have a name which is unique
within the associated rule
system version.

RULE_00051 Rule set name must be unique Error Reference Number:

within the rule system version. RULE_00051 Rule sets must
have a name which is unique
within the associated rule
system version.

RULE_00052 Active, release or read-only Error Reference Number:

rule system version cannot be RULE_00052. "{0}" in rule
modified. system version "{1}" cannot
be modified because the rule

id9YZ-09148-MT11-PCZZA © 2020 Nokia 91

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

system version is in the active

or release state or is read-only.

RULE_00053 Rule sets cannot be created or Error Reference Number:

removed. RULE_00053.Rule sets in
rule system version "{0}" are
predefined by the system and
are required for the system to
function correctly.

RULE_00054 Changing the state of a read- Error Reference Number:

only Rule System Version to RULE_00054. Rule System
DRAFT is not allowed. Version "{0}" is flagged as read-
only. Changing the state from
"{1}" to DRAFT is not allowed.

RULE_00055 A read-only Rule System Error Reference Number:

Version cannot be deleted. RULE_00055. Rule System
Version "{0}" is marked as read-
only and cannot be deleted.

RULE_00056 Rule System Version was not Error Reference Number:

found. RULE_00056. Rule System
Version "{0}" was not found in
the database.

RULE_00084 Name of the rule system version Error Reference Number:

is not unique. RULE_00084. Name of rule
system version "{0}" must be
unique. Modify the name and try
saving again.

RULE_00075 Duplicate Rule System Version Error Reference Number:

name. RULE_00075 Rule System
Version with specified name
"{0}" already exists.

RULE_00060 Rule table is associated to an Error Reference Number:

invalid rule table schedule RULE_00060.Rule table "{0}" is
associated to invalid rule table
schedule "{1}".

RULE_00061 Schedule frequency type can Error Reference Number:

only be set to Weekly Recurring. RULE_00061. Set the schedule
frequency type to Weekly
Recurring.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 92

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

RULE_00062 Description exceeds the Error Reference Number:

maximum length of 255 RULE_00062. The description
characters. is "{0}" characters long. The
Description field must be 0 to
255 characters.

RULE_00063 Number of schedule intervals Error Reference Number:

out of range. RULE_00063. The number
of schedule intervals must
be at least 1, but no more
than required to fit into one
week while complying with the
other interval restrictions (for
example, non-overlapping,
minimum length, and non-
adjacent).

RULE_00064 Schedule interval duration out of Error Reference Number:

range. RULE_00064. The intervals
can be recurring but cannot be
less than 30 min long after the
start time. The intervals cannot
overlap.

RULE_00065 Overlapping schedule intervals. Error Reference Number:

RULE_00065. One or more of
the scheduled intervals has a
stop time that is after the start
time of another interval, or has
a start time that is before the
stop time of another interval.
Review the scheduled intervals
to ensure that none of the
intervals overlap.

RULE_00066 Invalid interval time for Rule Error Reference Number:

Table Schedule. RULE_00066. The scheduled
interval start and stop time
must be on a valid 5 min
boundary. Time is displayed in
the format hh:mm:ss. Please
enter a numeric value between
00:00:00 (midnight) to 23:55:00.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 93

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

RULE_00068 Select at least one day of the Error Reference Number:

week option. RULE_00068. The schedule
frequency type is Weekly
Recurring; therefore, select at
least one day of the week, or
select all days of the week.

RULE_00070 Schedule interval start or stop Error Reference Number:

time out of range. RULE_00070. The scheduled
interval start and stop time
can only be on the hour. The
format displayed is hh:mm:ss.
The value entered can be any
number from 0 (midnight) to 23
(23:00:00).

RULE_00074 Rule table schedule name must Error Reference Number:

be unique within the rule system RULE_00074 Rule table
version. schedules must have a name
which is unique within the
associated rule system version.

RULE_00080 There is a problem with the Error Reference Number:

regular expression. RULE_00080. The pattern in
question is: {0} Error message:
{1}

RULE_00081 Invalid value type for regular Error Reference Number:

expression operator. RULE_00081. Condition values
used with the regular expression
operators must use a value type
of String.

RULE_00082 Invalid value for a condition. Error Reference Number:

RULE_00082. Condition value
must not be set when the
operator is Present or Not
Present.

RULE_00180 The value type "{0}" is not Error Reference Number:

valid for a numeric adjustment RULE_00180. For numeric
operator adjustment operators, all
attributes must be of a numeric

id9YZ-09148-MT11-PCZZA © 2020 Nokia 94

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

type: for example, Long, Integer,

or Date/Time.

RULE_00181 The value type "{0}" is not Error Reference Number:

valid for the Union adjustment RULE_00181. For the Union
operator adjustment operator, the
attribute must be a type
supported by lists, for example
String or IP address.

RULE_00182 All lists used in a Union Error Reference Number:

operation must be the same RULE_00182.Lists of different
type. types cannot be used in the
same Union operation.

RULE_00184 The adjustment data type is Error Reference Number:

invalid. RULE_00184. For numeric
adjustment operators, the
Adjustment Data Type field
used must be either a numeric
type (for example, Long) or
Attribute type.

RULE_00185 In Action formula, the data type Error Reference Number:

of the action is invalid. RULE_00185. When used in an
Action formula, the Data Type
field of the action must select
the Attribute type.

RULE_00186 The adjustment attribute value Error Reference Number:

must not be unbounded. RULE_00186. For the numeric
adjustment operators, the
adjustment attribute must be a
single occurrence attribute.

RULE_00187 In Action formula, the "{0}" Error Reference Number:

cannot be bounded RULE_00187. When used in
an Action formula, the attribute
used in the Action Data field
or in the Adjustment Data
field cannot be bounded in its
selected SourceContextType.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 95

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

RULE_00188 The adjustment value type Error Reference Number:

"{0}" is not valid for this Union RULE_00188. For the Union
operation adjustment operator, the
adjustment attribute must be of
a type supported by lists. For
example, String or IP address. It
must also match the type of the
attribute being adjusted.

RULE_00189 The adjustment attribute value Error Reference Number:

must be unbounded. RULE_00189. All attributes
used in a Union operation must
be unbounded.

RULE_00190 The value type "{0}" is not valid Error Reference Number:
for the Concatenate adjustment RULE_00190. For the
operator Concatenate adjustment
operator, the attribute must be
a type supported by lists, for
example String.

RULE_00191 All lists used in a Concatenate Error Reference Number:

operation must of be the same RULE_00191. Lists of different
type. types cannot be used in the
same Concatenate operation.

RULE_00192 The adjustment value type "{0}" Error Reference Number:

is not valid for this Concatenate RULE_00192. For the
operation Concatenate adjustment
operator, the adjustment
attribute must be of a type
supported by lists, for example
String. It must also match the
type of the attribute being
adjusted.

RULE_00193 The adjustment attribute value Error Reference Number:

must be unbounded. RULE_00193. All attributes
used in a Concatenate
operation must be unbounded.

RULE_00200 Rule table has invalid condition Error Reference Number:

for its associated RuleSets. RULE_00200. Rule table "{0}":
Rule "{1}" with Condition "{2}"
is not applicable for RuleSets
id9YZ-09148-MT11-PCZZA © 2020 Nokia 96
2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

"{3}". A rule contained in a rule

table has a condition that is not
applicable to the associated rule
sets on this rule table, all the
conflicted rule sets are listed.
Modify the conditions in this rule
table or the selected rule sets to
make the rule table applicable.

RULE_00201 Rule table has invalid action for Error Reference Number:
its associated RuleSets. RULE_00201. Rule table "{0}":
Rule "{1}" with action "{2}" is not
applicable for RuleSets "{3}". A
rule contained in rule table has
an action that is not applicable
to the associated rule sets on
this rule table. All the conflicted
rule sets are listed. Modify the
actions in this rule table or the
selected rule sets to make the
rule table applicable.

RULE_00301 A flow description cannot be Error Reference Number:

empty. RULE_00301.The Description
parameter is used to identify
the flow description and cannot
be empty. Please assign a flow
description to the Charging/QoS
rule definition.

RULE_00302 Invalid flow description string. Error Reference Number:

RULE_00302. Flow description
string validation error: "{0}".
A flow description string must
follow the format described in
RFC 3588.

RULE_00303 Invalid flow description string. Error Reference Number:

RULE_00303. You must set the
Action to PERMIT.

RULE_00304 Invalid flow description string. Error Reference Number:

RULE_00304. Invert modifier

id9YZ-09148-MT11-PCZZA © 2020 Nokia 97

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

(!) is not allowed in a flow

description string.

RULE_00305 Invalid rule type selected. Error Reference Number:

RULE_00305. Rule type
must be one of the following
types: provisioned template,
provisioned radius template,
predefined template, predefined
base template, predefined adc
template, predefined base adc
template or provisioned adc
template.

RULE_00306 Invalid values for predefined Error Reference Number:

template, predefined base RULE_00306. Only name,
template, predefined adc activation time, and deactivation
template or predefined base adc time may be provided for
template. predefined templates and
predefined base templates.

RULE_00311 Invalid guaranteed bitrate value. Error Reference Number:

RULE_00311. The guaranteed
bitrate value in a direction must
be less than or equal to the
maximum requested bandwidth
in that direction.

RULE_00312 Priority level is mandatory. Error Reference Number:

RULE_00312. The priority level
field is mandatory.

RULE_00317 Type cannot be changed. Error Reference Number:

RULE_00317. The type of a
Charging/QoS rule definition
cannot be changed.

RULE_00318 Service identifier and rating Error Reference Number:

group cannot be empty for RULE_00318. Service identifier
service identifier reporting level. level is not applicable when
the service identifier and
rating group have not been
provisioned in the charging rule.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 98

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

RULE_00319 Rating group cannot be empty Error Reference Number:

for rating group reporting level. RULE_00319. Rating group
level is not applicable when
rating group has not been
provisioned in the charging rule.

RULE_00320 Max Requested BW fields are Error Reference Number:

mandatory. RULE_00320. The Max
Requested Bandwidth fields are
mandatory.

RULE_00321 Guaranteed Bitrate fields are Error Reference Number:

mandatory for GBR QCIs. RULE_00321. The Guaranteed
Bitrate fields are mandatory for
GBR QCIs.

RULE_00322 Guaranteed Bitrate fields must Error Reference Number:

be empty for non-GBR QCIs. RULE_00322. Guaranteed
Bitrate fields must be empty for
non-GBR QCIs.

RULE_00323 Charging rule name must be Error Reference Number:

unique within the rule system RULE_00323 Charging rules
version. must have a name which is
unique within the associated
rule system version.

RULE_00324 Charging rule name AVP value Error Reference Number:

is a required field. RULE_00324Charging rules
must have a charging rule name
AVP value which must be 1 to
255 characters.

RULE_00325 Charging rule name AVP value Error Reference Number:

exceeds the maximum length of RULE_00325.The Charging rule
255 characters. name AVP value field must be 1
to 255 characters.

RULE_00326 Precedence field is mandatory Error Reference Number:

for provisioned template. RULE_00326. The Precedence
field is mandatory for
provisioned template.

RULE_00327 Priority Level is mandatory field Error Reference Number:

RULE_00327.A Priority Level
must be chosen if either

id9YZ-09148-MT11-PCZZA © 2020 Nokia 99

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

Pre-emption Capability or
Pre-emption Vulnerability is
specified.

RULE_00330 Monitoring Key must not be Error Reference Number:

provisioned. RULE_00330.The Predefined
Base ADC Rule Template and
Provisioned ADC Rule Template
must not have Monitoring Key
provisioned.

RULE_00331 Sponsored data connectivity Error Reference Number:

parameters and rating group are RULE_00331.Sponsored
mandatory when provisioning a connectivity level is not
sponsored reporting level. applicable when the sponsored
identity, application service
provider identity and rating
group have not been
provisioned in the charging rule.

RULE_00332 Flow configuration name must Error Reference Number:

be unique. RULE_00332.Flow configuration
name "{0}" is duplicated within
the pcc rule definition.

RULE_00333 Time configuration name must Error Reference Number:

be unique RULE_00333.Time
configuration name "{0}" is
duplicated within the pcc rule
definition.

RULE_00334 Time configuration name is not Error Reference Number:

valid RULE_00334.The name given
to the time configuration is not
valid.

RULE_00335 Flow configuration name is not Error Reference Number:

valid RULE_00335.The name given
to the flow configuration is not
valid.

RULE_00336 Invalid QCI value Error Reference Number:

RULE_00336.The QCI value is
invalid. Valid QCI range is from

id9YZ-09148-MT11-PCZZA © 2020 Nokia 100

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

1 to 9, 65, 66, 69, 70 and from

128 to 254, all inclusive.

RULE_00337 Redirect address type is Error Reference Number:

required RULE_00337.The redirect
address type is required when
the redirect server address is
present.

RULE_00338 Redirect server address is Error Reference Number:

required RULE_00338.The redirect
server address is required when
the redirect address type is
selected.

RULE_00339 Invalid IP address Error Reference Number:

RULE_00339.The value
"{0}" is not a valid. Enter a
valid IP address in one of
the 5780 DSC acceptable
formats; for example, 1.2.3.4, or
1:2:3:4:5:7:8, or 1.2.3.0/24.

RULE_00340 The address and address type Error Reference Number:

doesn't match RULE_00340.The redirect
server address doesn't match
the redirect address type.

RULE_00341 VSA configuration name "{0}" is Error Reference Number:

not valid RULE_00341.The name given
to the VSA configuration "{0}" is
not valid.

RULE_00350 Failed to load Rule Attribute Error Reference Number:

Dictionary "{0}" RULE_00350. Validation error
occurred loading the dictionary
file. See the system log (/opt/
tpa/logs/DDESMApplication.log)
for more information.

RULE_00351 Cannot resolve vendor "{0}" Error Reference Number:

qualified name for an Attribute RULE_00351. Could not resolve
"{1}" fully qualified domain name from
Diameter attribute information.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 101

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

RULE_00352 Failed to load rule attribute Error Reference Number:

dictionary schema file "{0}" RULE_00352. Validation error
occurred while loading or
parsing dictionary schema file
"{0}" See the system log (/opt/
tpa/logs/DDESMApplication.log)
for more information.

RULE_00353 Applicable Rule Set(s) not Error Reference Number:

specified for Source Context RULE_00353. Source Context
"{1}" of Attribute "{0}" of an Attribute must specify at
least one applicable Rule Set.

RULE_00354 Applicable Rule Set(s) not Error Reference Number:

specified for Result Context RULE_00354. Result Context
"{1}" of Attribute "{0}" of an Attribute must specify at
least one applicable Rule Set.

RULE_00355 Failed to load Diameter/VSA to Error Reference Number:

rule context map "{0}" RULE_00355. Error occurred
loading the dictionary file "{0}".
See the system log (/opt/tpa/
logs/DDESMApplication.log) for
more information.

RULE_00357 Failed to load diameter/VSA to Error Reference Number:

rule context map schema file RULE_00357. Validation error
"{0}" occurred while loading or
parsing dictionary schema file
"{0}"See the system log (/opt/
tpa/logs/DDESMApplication.log)
for more information.

RULE_00358 Failed to initialize XML parsing Error Reference Number:

context. RULE_00358. Failed to initialize
XML parsing context. See
the system log /opt/tpa/logs/
DDESMApplication.log for more
information.

RULE_00359 Failed to export Rule Attribute Error Reference Number:

Dictionary. RULE_00359. Error occurred
exporting the rule attribute
dictionary. See the system log (/

id9YZ-09148-MT11-PCZZA © 2020 Nokia 102

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

opt/tpa/logs/DDESMApplication.
log) for more information.

RULE_00390 Failed to load TAC data file "{0}" Error Reference Number:
RULE_00390. Validation error
occurred loading the TAC data
file. See the system log (/opt/
tpa/logs/DDESMApplication.log)
for more information.

RULE_00555 Queue IDs must be unique. Error Reference Number:

RULE_00555. The Queue ID
field of each queue must be
unique.

RULE_00556 QoS override mapping name Error Reference Number:

must be unique within the rule RULE_00556 QoS override
system version. mappings must have a name
which is unique within the
associated rule system version.

RULE_00557 Defined Rate is a mandatory Error Reference Number:

field. RULE_00557The Defined
Rate field is mandatory for an
operator defined aggregate rate.

RULE_00601 Rule variable name must be Error Reference Number:

unique within the rule system RULE_00601 Rule variable
version. must have a name which is
unique within the associated
rule system version.

RULE_00602 Rule variable name is a Error Reference Number:

mandatory field. RULE_00602.Rule variable
Name field is a mandatory field
and must be 1 to 80 characters.

RULE_00603 Rule variable name exceeds Error Reference Number:

the maximum length of 80 RULE_00603.Rule variable
characters. Name field must be 1 to 80
characters.

RULE_00604 Evaluation name must be Error Reference Number:

unique within the rule variable RULE_00604Evaluation must
definition. have a name which is unique

id9YZ-09148-MT11-PCZZA © 2020 Nokia 103

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

within the associated rule

variable definition.

RULE_00605 Rule variable has an invalid Error Reference Number:

value for its associated rule RULE_00605. Rule variable
sets. "{0}": Evaluation Rule "{1}"
with Condition "{2}.{3}" is not
applicable for rule sets "{4}".
An evaluation rule contained in
a rule variable has a condition
that is not applicable to the
associated rule sets on this
rule variable, all the conflicted
rule sets are listed. Modify the
conditions in this rule variable or
the selected rule sets to make
the rule variable valid.

RULE_00606 Rule variable has an invalid Error Reference Number:

value for its associated rule RULE_00606. Rule variable
sets. "{0}": Evaluation Rule "{1}" has
a value is not applicable for rule
sets "{2}". An evaluation rule
contained in rule variable has
a value that is not applicable to
the associated rule sets. All the
conflicted rule sets are listed.
Modify the value in this rule
variable or the selected rule sets
to make the rule variable valid.

RULE_00607 Rule variable has an invalid Error Reference Number:

default value for its associated RULE_00607. Rule variable
rule sets. "{0}" has default value that
is not applicable for rule sets
"{1}". The default value of rule
variable is not applicable to the
associated rule sets. All the
conflicted rule sets are listed.
Modify the default value in this
rule variable or the selected rule
sets to make the rule variable
valid.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 104

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

RULE_00608 Invalid rule set types for Error Reference Number:

application associated with rule RULE_00608. Rule variable
variable. "{0}" has extra rule set types
"{1}" that are not applicable
for application "{2}". The rule
set types assigned to the rule
variable are not all applicable
to the associated application.
All the extra rule set types are
listed. Modify the rule set types
of this rule variable to make it
valid.

RULE_00609 Condition of rule variable Error Reference Number:

references an undefined rule RULE_00609. Rule variable
variable. "{0}" has an evaluation "{1}"
whose condition references an
undefined rule variable "{2}".
Make sure the referenced rule
variable is defined.

RULE_00610 Condition of rule variable Error Reference Number:

references another rule variable RULE_00610. Rule variable
with incompatible rule set types. "{0}" with rule set types "{1}"
has an evaluation "{2}" whose
condition references a rule
variable "{3}" with rule set types
"{4}" that are not compatible.
The referenced rule variable
must have the same or a super
set of rule set types.

RULE_00611 Rule variable circular Rule variable circular

dependency detected. dependency detected.
RULE_00611_DETAIL=Error
Reference Number: RULE_
00611. Definition of rule variable
"{0}" has a dependency loop
detected in its referenced
variable path: "{1}". Modify the
rule variable definition to avoid
the loop.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 105

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

RULE_00612 Rule variable description Error Reference Number:

exceeds the maximum length of RULE_00612.Rule variable
255 characters. "{0}" has a description field
with "{1}" characters long. The
description length must be 0 to
255 characters.

RULE_00613 Rule variable has invalid value Error Reference Number:

type. RULE_00613.Rule variable "{0}"
has invalid value type "{1}". The
provided rule variable value type
must be from the valid value
type list "{2}".

RULE_00614 Condition of rule table Error Reference Number:

references an undefined rule RULE_00614. Rule table "{0}"
variable. has a rule "{1}" whose condition
references an undefined rule
variable "{2}". Make sure the
referenced rule variable is
defined

RULE_00615 Condition of rule table Error Reference Number:

references a rule variable with RULE_00615. Rule table "{0}"
incompatible rule set types. with rule set types "{1}" has
a rule "{2}" whose condition
references a rule variable "{3}"
with rule set types "{4}" that are
not compatible. The referenced
rule variable must have the
same or a super set of rule set
types.

RULE_00616 Rule variable does not have a Error Reference Number:

value type field. RULE_00616. The value type
field is missing in rule variable
"{0}". The value type field is
mandatory and cannot be null in
rule variable definition.

RULE_00617 Value of rule variable references Error Reference Number:

an undefined rule variable. RULE_00617. Rule variable
"{0}" has an evaluation "{1}"
whose value references an

id9YZ-09148-MT11-PCZZA © 2020 Nokia 106

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

undefined rule variable "{2}".

Make sure the referenced rule
variable is defined.

RULE_00618 Value of rule variable references Error Reference Number:

another rule variable with RULE_00618. Rule variable
incompatible rule set types. "{0}" with rule set types "{1}"
has an evaluation "{2}" whose
value references a rule variable
"{3}" with rule set types "{4}"
that are not compatible. The
referenced rule variable must
have the same or a super set of
rule set types.

RULE_00619 Default value of rule variable Error Reference Number:

references an undefined rule RULE_00619. Rule variable
variable. "{0}" has an default value that
references an undefined rule
variable "{1}". Make sure the
referenced rule variable is
defined.

RULE_00620 Default value of rule variable Error Reference Number:

references another rule variable RULE_00620. Rule variable
with incompatible rule set types. "{0}" with rule set types "{1}" has
an default value that references
a rule variable "{2}" with rule
set types "{3}" that are not
compatible. The referenced rule
variable must have the same or
a super set of rule set types.

RULE_00621 Action of rule table references Error Reference Number:

an undefined rule variable. RULE_00621. Rule table
"{0}" has a rule "{1}" with an
action "{2}" that references an
undefined rule variable "{3}".
Make sure the referenced rule
variable is defined.

RULE_00622 Action of rule table references a Error Reference Number:

rule variable with incompatible RULE_00622. Rule table "{0}"
rule set types. with rule set types "{1}" has a

id9YZ-09148-MT11-PCZZA © 2020 Nokia 107

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

rule "{2}" with an action "{3}"

that references a rule variable
"{4}" with rule set types "{5}"
that are not compatible. The
referenced rule variable must
have the same or a super set of
rule set types.

RULE_00623 Default action of rule set Error Reference Number:

references an undefined rule RULE_00623. Rule set "{0}"
variable. has a default action "{1}" that
references an undefined rule
variable "{2}". Make sure the
referenced rule variable is
defined.

RULE_00624 Default action of rule set Error Reference Number:

references a rule variable with RULE_00624. Rule set "{0}"
incompatible rule set types. has a default action "{1}" that
references a rule variable "{2}"
with rule set types "{3}" that are
not compatible. The referenced
rule variable must have the
same or a super set of rule set
types.

RULE_00625 Value is too large for the Error Reference Number:

variable. RULE_00625.Value "{0}" is too
large for variable "{1}".

RULE_00650 Rule variable evaluation name Error Reference Number:

is a mandatory field. RULE_00650.The rule variable
evaluation name field is a
mandatory field and must be 1
to 80 characters.

RULE_00651 Rule variable evaluation name Error Reference Number:

exceeds the maximum length of RULE_00651.The rule variable
80 characters. evaluation name field must be 1
to 80 characters.

RULE_00652 Rule variable evaluation Error Reference Number:

description exceeds the RULE_00652.Rule variable
maximum length of 255 evaluation "{0}" has a
characters. description "{1}" characters

id9YZ-09148-MT11-PCZZA © 2020 Nokia 108

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

long. The rule variable

evaluation description field must
be 0 to 255 characters.

RULE_00653 Rule variable evaluation value is Error Reference Number:

a mandatory field. RULE_00653.Rule variable
evaluation "{0}" has no value.
Value is a mandatory field.

RULE_00700 Missing value data. Error Reference Number:

RULE_00700.Data is a
mandatory field for values.

RULE_00701 Non-attribute value does not Error Reference Number:

allow arguments. RULE_00701.Values of type
"{0}" do not allow arguments.

RULE_00702 Invalid adjustment operator or Error Reference Number:

adjustment data. RULE_00702.When a formula
is used, the adjustment operator
field and adjustment data field
must both be assigned to a valid
value. If the formula is not used,
both fields must be empty.

RULE_00703 Non-attribute adjustment value Error Reference Number:

does not allow arguments. RULE_00703.Adjustment
values of type "{0}" do not allow
arguments.

RULE_00704 Rule plugin sequence are not Rule plugin sequence

same. configured {0} does not match
with plugin sequence from
incoming request {1}.

RULEAPI_DEFAULT Failed to persist object Not applicable

Table 24: Rule errors

Generic Value Errors

The following section describes the Generic Value errors:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 109

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

GV_00001 Type is a mandatory field. Error Reference Number: GV_

00001. Custom Data Type is a
mandatory field.

GV_00002 Value is a mandatory field. Error Reference Number: GV_

00002. Value is a mandatory
field, and must be between 1
and 256000 characters.

GV_00003 Value is too long. Error Reference Number: GV_

00003. Value must be between
1 and 256000 characters.

GV_00004 Invalid value. Error Reference Number: GV_

00004. The value [type="{0}",
value="{1}"] is not valid.

GV_00005 Invalid primary OCS URI. Error Reference Number: GV_

00005. The URI "{0}" is not
valid.

GV_00006 Invalid secondary OCS URI. Error Reference Number: GV_

00006. The URI "{0}" is not
valid.

GV_00007 Invalid primary OFCS URI. Error Reference Number: GV_

00007. The URI "{0}" is not
valid.

GV_00008 Invalid secondary OFCS value. Error Reference Number: GV_

00008. The URI "{0}" is not
valid.

GV_00009 Lists must have at least one Error Reference Number: GV_
value. 00009. Lists must have at least
one value.

GV_00010 Unsupported type for list. Error Reference Number: GV_

00010. The type "{0}" is not
supported for lists.

GV_00011 Format is a mandatory field. Error Reference Number: GV_

00011. Format is a mandatory
field.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 110

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

GV_00012 Value is a mandatory field. Error Reference Number: GV_

00012. Value is a mandatory
field when the format is Custom.

GV_00013 Direction is a mandatory field. Error Reference Number: GV_

00013. Direction is a mandatory
field when the format is Queue
or Rate.

GV_00014 Queue ID is a mandatory field. Error Reference Number:

GV_00014. Queue ID is a
mandatory field when the format
is Queue.

GV_00015 Unsupported format. Error Reference Number: GV_

00015. The format "{0}" is not
supported.

GV_00016 Mismatched enum types in list. Error Reference Number: GV_

00016.A list of enum values
may only contain values from a
single enum type.

GV_00017 Referenced object was not Error Reference Number: GV_

found. 00017.Unable to find referenced
object "{0}" of class "{1}".

Table 25: Generic Value errors

Rule API Errors

The following section describes the Rule API errors:

Number Message Description

RULEAPI_ERR_0001 No entry found for Rule System No entry found for Rule System
Version Version. Please check the
requested name ({0})

RULEAPI_ERR_0002 Bad Request Check the request parameter

RULEAPI_ERR_0003 Invalid Rule System Version Invalid Rule System Version

state state ({0})

RULEAPI_ERR_0004 Invalid Header application/json is the only

acceptable Header

id9YZ-09148-MT11-PCZZA © 2020 Nokia 111

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

RULEAPI_ERR_0005 No entry found for Rule System No entry found for Rule System
Version Version.

RULEAPI_ERR_0006 Invalid page size Page size not within a correct

range.

RULEAPI_ERR_0007 No entry for Rule Set No entry found for RuleSet.

Please check the requested
name ({0})

RULEAPI_ERR_0008 Rule System Version name is a Rule System Version name is a

mandatory parameter mandatory parameter

RULEAPI_ERR_0009 Import data not provided or Import data not provided or

properly formatted properly formatted

RULEAPI_ERR_0010 Requested URL is incorrect, Requested URL is incorrect,

please check the Requested please check the Requested
URL URL

RULEAPI_ERR_0011 Please check the media type of Please check the media type of
the request, supported media the request, supported media
type is JSON type is JSON

RULEAPI_ERR_0012 Not Acceptable, The requested Not Acceptable, The requested

resource is only capable resource is only capable
of generating content not of generating content not
acceptable according to the acceptable according to the
Accept headers sent in the Accept headers sent in the
request request

RULEAPI_ERR_0013 No Rule Set found. No entry found for Rule Set.

Please check the requested
name ({0})

RULEAPI_ERR_0021 No entry found for No entry found for

RuleTableSchedule RuleTableSchedule ({0}) in Rule
System Version ({1})

RULEAPI_ERR_0022 No entry found for Rule Group No entry found for Rule Group
({0}) in Rule System Version
({1})

RULEAPI_ERR_0023 Rule Group with the same name exists Rule Group ({0}) with the
already exists same name already exists Rule
System Version ({1})

id9YZ-09148-MT11-PCZZA © 2020 Nokia 112

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

RULEAPI_ERR_0024 Mismatched enum types in list. A list of enum values may only
contain values from a single
enum type.

RULEAPI_ERR_0025 Rule System Version state Rule System Version state

should be either in RELEASE or should be either in RELEASE or
ACTIVE state ACTIVE state

RULEAPI_ERR_0026 Rule System Version cannot Rule System Version cannot

be move from ACTIVE to be move from ACTIVE to
RELEASE, for making changes RELEASE, for making changes
do the clone of the Rule System do the clone of the Rule System
Version Version and make change

RULEAPI_ERR_0027 Only distributed Rule System Only distributed Rule System

Version can be activated or Version can be activated or
undistributed undistributed

RULEAPI_ERR_0028 No Rule System Version is No Rule System Version is

distributed the ME distributed the ME

RULEAPI_ERR_0029 RSV cannot be deleted from {0} RSV cannot be deleted from {0}
because {1} is/are offline because {1} is/are offline

RULEAPI_ERR_0030 RSV cannot be activated on {0}, RSV cannot be activated on {0},

since {1} is/are offline since {1} is/are offline

RULEAPI_ERR_0031 RSV cannot be deactivated on RSV cannot be deactivated on

{0}, since {1} is/are offline {0}, since {1} is/are offline

RULEAPI_ERR_0032 RSV cannot be distributed on RSV cannot be distributed on

{0}, since {1} is/are offline {0}, since {1} is/are offline

RULEAPI_ERR_0033 Rule Table configured is not Rule Table configured is not

mapped to any Rule Trigger. mapped to any Rule Trigger.

Table 26: Rule API errors

Import or Export errors

The following section describes the error messages for Import or Export:

Number Message Description

IMPORT_00001 Error in imported file Error Reference Number:

IMPORT_00001.The file

id9YZ-09148-MT11-PCZZA © 2020 Nokia 113

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

contains one or more syntax

errors.

IMPORT_00002 Error parsing Charging/QoS rule Error Reference Number:

definition list. IMPORT_00002.Cannot parse
the list of Charging/QoS rule
definitions.

IMPORT_00003 Error parsing rule table Error Reference Number:

schedule list. IMPORT_00003.Cannot parse
the list of rule table schedules.

IMPORT_00004 Error parsing rule table list. Error Reference Number:

IMPORT_00004.Cannot parse
the list of rule tables.

IMPORT_00005 Error parsing rule set list. Error Reference Number:

IMPORT_00005.Cannot parse
the list of rule sets.

IMPORT_00006 Error parsing rule set name. Error Reference Number:

IMPORT_00006.Cannot parse
name for a rule set.

IMPORT_00007 Error parsing default actions list Error Reference Number:

for rule set {0}. IMPORT_00007.Cannot parse
the list of default actions for the
rule set.

IMPORT_00008 Error parsing rule table Error Reference Number:

precedence list for rule set {0}. IMPORT_00008.Cannot parse
the rule table precedence list for
the rule set.

IMPORT_00009 Error parsing rule table name. Error Reference Number:

IMPORT_00009.Cannot parse
name for a rule table.

IMPORT_00010 Error parsing rule table Error Reference Number:

schedule name for rule table {0}. IMPORT_00010.Cannot parse
rule table schedule name for the
rule table.

IMPORT_00011 Error parsing rule list for rule Error Reference Number:
table {0}. IMPORT_00011.Cannot parse
list of rules for the rule table.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 114

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

IMPORT_00012 Error parsing rule name. Error Reference Number:

IMPORT_00012.Cannot parse
name for a rule.

IMPORT_00013 Missing rule name. Error Reference Number:

IMPORT_00013.Name is
missing for a rule. Name is a
mandatory value.

IMPORT_00015 Error parsing condition list for Error Reference Number:

rule {0}. IMPORT_00015.Cannot parse
conditions for the rule.

IMPORT_00016 Error parsing action list for rule Error Reference Number:
{0}. IMPORT_00016.Cannot parse
list of actions for the rule.

IMPORT_00018 Error parsing Action Name. Error Reference Number:

IMPORT_00018.Cannot parse
name for an action.

IMPORT_00019 Invalid Action Name {0}. Error Reference Number:

IMPORT_00019.Name entry
for the action is invalid because
it does not correspond to
an action in the current rule
dictionary.

IMPORT_00020 Error parsing action value for Error Reference Number:

action {0}.{1} IMPORT_00020.Cannot parse
action value for the action.

IMPORT_00022 Error parsing Charging/QoS rule Error Reference Number:

definition name. IMPORT_00022.Cannot parse
name for Charging/QoS rule
definition.

IMPORT_00023 Error parsing type for Charging/ Error Reference Number:

QoS rule definition {0}. IMPORT_00023.Cannot parse
type for this Charging/QoS rule
definition.

IMPORT_00024 Error parsing precedence for Error Reference Number:

Charging/QoS rule definition {0}. IMPORT_00024.Cannot parse
precedence for this Charging/
QoS rule definition.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 115

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

IMPORT_00025 Error parsing metering method Error Reference Number:

for Charging/QoS rule definition IMPORT_00025.Cannot parse
{0}. metering method for this
Charging/QoS rule definition.

IMPORT_00026 Error parsing rating group for Error Reference Number:

Charging/QoS rule definition {0}. IMPORT_00026.Cannot parse
rating group for this Charging/
QoS rule definition.

IMPORT_00048 Error persisting rule system Error Reference Number:

version (name={0}). IMPORT_00048.Persist failed
with the following error: {1}.

IMPORT_00050 Unknown rule table [name={0}] Error Reference Number:

used in rule [name={1}]. IMPORT_00050.The rule table
was not part of the imported
data.

IMPORT_00076 Unknown rule table [name={0}] Error Reference Number:

used in rule set [type={1}]. IMPORT_00076.The rule table
was not part of the imported
data.

IMPORT_00051 Error parsing Rule Table Error Reference Number:

Schedule name. IMPORT_00051.Cannot parse
name of a rule table schedule.

IMPORT_00052 Error parsing description for Error Reference Number:

Rule Table Schedule {0}. IMPORT_00052.Cannot parse
description for this rule table
schedule.

IMPORT_00053 Error parsing interval for Rule Error Reference Number:

Table Schedule {0}. IMPORT_00053.Cannot parse
interval for this rule table
schedule.

IMPORT_00054 Error parsing interval for Rule Error Reference Number:

Table Schedule {0}. IMPORT_00054.Parsing of
the interval field failed with the
following error: {1}.

IMPORT_00055 Error parsing time zone offset Error Reference Number:

for Rule Table Schedule {0}. IMPORT_00055.Cannot parse

id9YZ-09148-MT11-PCZZA © 2020 Nokia 116

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

time zone offset for this rule

table schedule.

IMPORT_00056 Error parsing daylight saving Error Reference Number:

offset for Rule Table Schedule IMPORT_00056.Cannot parse
{0}. daylight saving offset for this
rule table schedule.

IMPORT_00057 Error parsing frequency for Rule Error Reference Number:

Table Schedule {0}. IMPORT_00057.Cannot parse
frequency for this rule table
schedule.

IMPORT_00058 Error parsing period for Rule Error Reference Number:

Table Schedule {0}. IMPORT_00058.Cannot
parse period for this rule table
schedule.

IMPORT_00059 Error parsing period for Rule Error Reference Number:

Table Schedule {0}. IMPORT_00059.Parsing of
the period field failed with the
following error: {1}.

IMPORT_00060 Error parsing starting month of Error Reference Number:

interval. IMPORT_00060.Cannot parse
starting month for this interval.

IMPORT_00061 Error parsing starting day of Error Reference Number:

month of interval. IMPORT_00061.\Cannot parse
starting day of month for this
interval.

IMPORT_00062 Error parsing starting day of Error Reference Number:

week of interval. IMPORT_00062.Cannot parse
starting day of week for this
interval.

IMPORT_00063 Error parsing starting time of Error Reference Number:

interval. IMPORT_00063.Cannot parse
starting time for this interval.

IMPORT_00064 Error parsing stopping month of Error Reference Number:

interval. IMPORT_00064.Cannot parse
stopping month for this interval.

IMPORT_00065 Error parsing stopping day of Error Reference Number:

month of interval. IMPORT_00065.Cannot parse

id9YZ-09148-MT11-PCZZA © 2020 Nokia 117

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

stopping day of month for this

interval.

IMPORT_00066 Error parsing stopping day of Error Reference Number:

week of interval. IMPORT_00066.Cannot parse
stopping day of week for this
interval.

IMPORT_00067 Error parsing stopping time of Error Reference Number:

interval. IMPORT_00067.Cannot parse
stopping time for this interval.

IMPORT_00068 Error parsing period start time. Error Reference Number:

IMPORT_00068.Cannot parse
start time for this period.

IMPORT_00069 Error parsing period stop time. Error Reference Number:

IMPORT_00069.Cannot parse
stop time for this period.

IMPORT_00071 Error parsing rule for rule table Error Reference Number:
{0}. IMPORT_00071.Parsing of a
rule failed with the following
error: {1}.

IMPORT_00072 Error parsing conditions for rule Error Reference Number:

{0}. IMPORT_00072.Parsing of
a condition failed with the
following error: {1}.

IMPORT_00073 Error parsing action for rule {0}. Error Reference Number:
IMPORT_00073.Parsing of an
action failed with the following
error: {1}.

IMPORT_00082 Rule set name missing. Error Reference Number:

IMPORT_00082Name is a
mandatory field.

IMPORT_00083 Rule table name missing. Error Reference Number:

IMPORT_00083Name is a
mandatory field.

IMPORT_00165 Error parsing enabled flag for Error Reference Number:

rule [name={0}]. IMPORT_00165Cannot parse
enabled flag for the rule.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 118

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

IMPORT_00166 Name missing for action. Error Reference Number:

IMPORT_00166Name is a
mandatory field.

IMPORT_00169 Version information missing Error Reference Number:

from imported data. IMPORT_00169The imported
data did not contain version
information.

IMPORT_00172 Error parsing generic data group Error Reference Number:

list. IMPORT_00172Cannot parse
the list of generic data groups.

IMPORT_00173 Generic Data Group [name={0}] Error Reference Number:

is not unique. IMPORT_00173Another generic
data group with that name is
present on the system or in the
imported data.

IMPORT_00174 Generic Data Group name Error Reference Number:

missing. IMPORT_00174Name is a
mandatory field.

IMPORT_00175 Error parsing Generic Data Error Reference Number:

Group name. IMPORT_00175Cannot parse
the name of the Generic Data
Group.

IMPORT_00176 Error parsing description for Error Reference Number:

Generic Data Group [name={0}]. IMPORT_00176Cannot parse
the description of the Generic
Data Group.

IMPORT_00177 Generic Data Group priority Error Reference Number:

missing. IMPORT_00177Priority is a
mandatory field.

IMPORT_00178 Error parsing Generic Data Error Reference Number:

Group priority. IMPORT_00178Cannot parse
the priority of the Generic Data
Group.

IMPORT_00179 Error parsing AVP list for Error Reference Number:

Generic Data Group [name={0}]. IMPORT_00179Cannot parse
the list of AVPs for the Generic
Data Group.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 119

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

IMPORT_00180 Generic Data Group AVP name Error Reference Number:

missing. IMPORT_001807\AVP Name is
a mandatory field.

IMPORT_00181 Error parsing Generic Data Error Reference Number:

Group AVP's name. IMPORT_00181Cannot parse
the name of the Generic Data
Group AVP.

IMPORT_00182 Generic Data Group AVP type Error Reference Number:

missing. IMPORT_00182AVP type is a
mandatory field.

IMPORT_00183 Error parsing Generic Data Error Reference Number:

Group AVP type. IMPORT_00183Cannot parse
the type of the Generic Data
Group AVP.

IMPORT_00184 Generic Data Group AVP value Error Reference Number:

missing. IMPORT_00184AVP Value is a
mandatory field.

IMPORT_00185 Error parsing Generic Data Error Reference Number:

Group AVP value. IMPORT_00185Cannot parse
the value of the Generic Data
Group AVP.

IMPORT_00313 Unsupported managed object Error Reference Number:

reference. IMPORT_00313The reference
to managed object type {2} in
action {0}.{1} is not supported.

IMPORT_00314 Missing error message for Error Reference Number:

managed object reference IMPORT_00314The error code
verification. for a reference to managed
object type {0} from parent entity
{1} is not defined.

IMPORT_00317 Unknown attribute {0}.{1} used Error Reference Number:

in action. IMPORT_00317The referenced
attribute does not exist in the
attribute dictionary.

IMPORT_00330 Error parsing parameter list for Error Reference Number:

action {0}.{1}. IMPORT_00330.Cannot parse
list of parameters for the action.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 120

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

IMPORT_00331 Action parameter name missing. Error Reference Number:

IMPORT_00331Name is a
mandatory field.

IMPORT_00332 Error parsing name for action Error Reference Number:

parameter. IMPORT_00332Cannot parse
name for the action parameter.

IMPORT_00333 Top-level table flag missing. Error Reference Number:

IMPORT_00333Top-level
table flag is a mandatory field,
but is missing for rule table
[name={0}].

IMPORT_00334 Error parsing top-level table Error Reference Number:

flag. IMPORT_00334Cannot parse
top-level table flag for rule table
[name={0}].

IMPORT_00339 Condition container operator Error Reference Number:

field missing. IMPORT_00339Operator is a
mandatory field.

IMPORT_00340 Error parsing condition Error Reference Number:

container operator field. IMPORT_00340Cannot parse
condition container operator
field.

IMPORT_00341 Error parsing condition Error Reference Number:

container condition list. IMPORT_00341Cannot parse
condition container condition
list.

IMPORT_00342 Error parsing condition Error Reference Number:

container sub-container list. IMPORT_00342Cannot parse
condition container sub-
container list.

IMPORT_00343 Criteria source context type field Error Reference Number:

missing. IMPORT_00343Criteria source
context type is a mandatory
field.

IMPORT_00344 Error parsing criteria source Error Reference Number:

context type. IMPORT_00344Cannot parse
criteria source context type.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 121

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

IMPORT_00345 Criteria name field missing. Error Reference Number:

IMPORT_00345Criteria name is
a mandatory field.

IMPORT_00346 Error parsing criteria name. Error Reference Number:

IMPORT_00346Cannot parse
criteria name.

IMPORT_00347 Error parsing criteria argument Error Reference Number:

list. IMPORT_00347Cannot parse
criteria argument list.

IMPORT_00348 Condition comparison operator Error Reference Number:

field missing. IMPORT_00348Condition
comparison operator is a
mandatory field.

IMPORT_00349 Error parsing condition Error Reference Number:

comparison operator. IMPORT_00349Cannot parse
condition comparison operator.

IMPORT_00350 Error parsing condition value. Error Reference Number:

IMPORT_00350Cannot parse
condition value.

IMPORT_00351 Error parsing condition value Error Reference Number:

argument list. IMPORT_00351Cannot parse
condition value argument list.

IMPORT_00352 Error parsing condition Error Reference Number:

adjustment operator. IMPORT_00352Cannot parse
condition adjustment operator.

IMPORT_00353 Error parsing condition Error Reference Number:

adjustment value list. IMPORT_00353Cannot parse
condition adjustment value list.

IMPORT_00354 Adjustment value missing. Error Reference Number:

IMPORT_00354Adjustment
value is a mandatory field.

IMPORT_00355 Error parsing adjustment value. Error Reference Number:

IMPORT_00355Cannot parse
adjustment value.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 122

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

IMPORT_00356 Error parsing adjustment value Error Reference Number:

argument list. IMPORT_00356Cannot parse
adjustment value argument list.

IMPORT_00357 Attribute argument name field Error Reference Number:

missing. IMPORT_00357Argument name
is a mandatory field.

IMPORT_00358 Error parsing attribute argument Error Reference Number:

name. IMPORT_00358Cannot parse
argument name.

IMPORT_00359 Attribute argument value field Error Reference Number:

missing. IMPORT_00359Argument value
is a mandatory field.

IMPORT_00360 Error parsing attribute argument Error Reference Number:

value. IMPORT_00360Cannot parse
argument value.

IMPORT_00361 Value type field missing. Error Reference Number:

IMPORT_00361Value type is a
mandatory field.

IMPORT_00362 Error parsing value type. Error Reference Number:

IMPORT_00362Cannot parse
value type.

IMPORT_00363 Value content field missing. Error Reference Number:

IMPORT_00363Value content is
a mandatory field.

IMPORT_00364 Error parsing value content. Error Reference Number:

IMPORT_00364Cannot parse
value content.

IMPORT_00365 Action parameter data field Error Reference Number:

missing. IMPORT_00365Action
parameter data is a mandatory
field.

IMPORT_00366 Error parsing action parameter Error Reference Number:

data. IMPORT_00366Cannot parse
action parameter data.

IMPORT_00367 Error parsing action parameter Error Reference Number:

data argument list. IMPORT_00367Cannot parse

id9YZ-09148-MT11-PCZZA © 2020 Nokia 123

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

action parameter data argument

list.

IMPORT_00368 Error parsing action parameter Error Reference Number:

adjustment operator. IMPORT_00368Cannot parse
action parameter adjustment
operator.

IMPORT_00369 Error parsing action parameter Error Reference Number:

adjustment value list. IMPORT_00369Cannot parse
action parameter adjustment
value list.

IMPORT_00373 Error parsing argument value Error Reference Number:

argument list. IMPORT_00373Cannot parse
argument value argument list.

SYS_IMPORT_RSV_00001 Unable to resolve rule system Error Reference Number: SYS_

version service. IMPORT_RSV_00001The
system was unable to resolve
the rule system version service

SYS_IMPORT_RSV_00002 Failed to parse JSON. Error Reference Number: SYS_

IMPORT_RSV_00002.Parsing
of the JSON string failed with
the following error: {0}

SYS_IMPORT_RSV_00003 Failed to parse RSV Error Reference Number: SYS_

configuration group. IMPORT_RSV_00003.Parsing
of the RSV configuration group
failed with the following error:
{0}

SYS_IMPORT_RSV_00004 RSV name field is missing. Error Reference Number: SYS_

IMPORT_RSV_00004.The RSV
name field is missing.

SYS_IMPORT_RSV_00005 Failed to parse RSV name. Error Reference Number: SYS_

IMPORT_RSV_00005.Parsing
of the RSV name field failed
with the following error: {0}

SYS_IMPORT_RSV_00006 Failed to parse RSV list. Error Reference Number: SYS_

IMPORT_RSV_00005.Parsing
of the RSV list failed with the
following error: {0}

id9YZ-09148-MT11-PCZZA © 2020 Nokia 124

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

SYS_IMPORT_RSV_00007 Failed to parse RSV description. Error Reference Number: SYS_

IMPORT_RSV_00007.Parsing
of the RSV description field
failed with the following error:
{0}

SYS_IMPORT_RSV_00008 RSV content missing. Error Reference Number: SYS_

IMPORT_RSV_00008.The RSV
content field is missing.

SYS_IMPORT_RSV_00009 Failed to parse RSV content. Error Reference Number: SYS_

IMPORT_RSV_00009.Parsing
of the RSV content field failed
with the following error: {0}

SYS_IMPORT_RSV_00010 Invalid parameters for RSV Error Reference Number: SYS_

export. IMPORT_RSV_00010.The
parameters provided for RSV
export were invalid.

SYS_IMPORT_RSV_00011 Unknown RSV name {0} Error Reference Number: SYS_

provided to export. IMPORT_RSV_00011.The
provided RSV name is not
present on the system.

Table 27: Import or Export errors

SM Error component: Other Functionality

The following section describes the error messages for Other functionalities:

Sub-component: User Management

The following section describes the error messages for sub-component User management:

MSAPI_00054 Role provided to the user does Not applicable

not exist.

Not applicable User is not Authorized to access Not applicable

the requested resource

MSAPI_00173 The Account expiry date should Not applicable

be not be before the current
date.

MSAPI_00042 Operation successful, but email Not applicable

notification not sent as Email
Server Configuration is missing ,
id9YZ-09148-MT11-PCZZA © 2020 Nokia 125
2.0
Monitoring and Troubleshooting Guide Troubleshooting

Admin need to set the password

for user manually.

MSAPI_00044 Inactive/Locked User cannot Not applicable

perform any operation. Please
contact Administrator.

MSAPI_00045 New Password and Confirm Not applicable

Password do not Match.

MSAPI_00046 User Password is reset but Not applicable

email notification could not be
sent.

MSAPI_00047 Not Authorized. Credentials Not applicable

provided does not match with
requested resource

MSAPI_00048 User cannot delete itself or Not applicable

update itself as 'inactive'

MSAPI_00166 User's Account Password Not applicable

is expired. Please contact
Administrator.

MSAPI_00167 User's Account is expired. Not applicable

Please contact Administrator.

MSAPI_00169 Your account has been locked Not applicable

as you have \ exceeded
maximum wrong password
attempts. Please contact your
administrator \ to unlock it.

MSAPI_00170 User has to set all the three \ Not applicable

security answers.

MSAPI_00172 You have reached the maximum Not applicable

number of \ sessions. Please
logout from other sessions to
login again.

MSAPI_00173 The Account expiry date should Not applicable

be not be before the current
date.

MSAPI_00017 Your Session has become Not applicable

Stale. Please logout and login to
Continue.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 126

2.0
Monitoring and Troubleshooting Guide Troubleshooting

MSAPI_00017 The username or password you Not applicable

entered is incorrect.

MSAPI_00017 Invalid user credentials Not applicable

MSAPI_00017 Invalid user security token Not applicable

MSAPI_00017 Security Credentials or Not applicable

Password not set for this user

MSAPI_00003 Couldn't persist into SM Not applicable

MSAPI_00071 New password cannot be same Not applicable

as old password.

Table 28: Sub-component: User Management

Sub-component: Role Management

The following section describes the error messages for sub-component Role Management:

Number Message Description

MSAPI_00053 system' security domain is Not applicable

specific to default settings of
SM.

Not applicable Invalid value for mandatory Not applicable

field Role and \ character range
should be 1 to 50 characters in
lower case.

Table 29: Sub-component: Role Management

Sub-component: SM System Configuration

The following section describes the error messages for sub-component SM system configuration:

Number Messages Description

Not applicable Session Lock Timeout should Not applicable

be between 1 (min) and 1000
(mins)

Not applicable Provisioning Thread pool size Not applicable

should be between 1 and 400.

Not applicable ME communication Timeout Not applicable

should be between 500 (ms)
and 60000 (ms).

id9YZ-09148-MT11-PCZZA © 2020 Nokia 127

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Messages Description

Not applicable Number of provisioning Not applicable

attemmpts should be between 1
and 10.

Not applicable ME Health Check Interval Not applicable

should be between 5000 (ms)
and 3600000 (ms).

Not applicable Token expiry time should be Not applicable

between 1 (hr) and 10000 (hrs).

Not applicable User concurrency level should Not applicable

be between 0 and 10.

Not applicable Max Wrong Password Allow Not applicable

should be \ between 1 and 5.

Not applicable Password Expiry should be Not applicable

between 1 (day) \ and 365
(days).

Table 30: Sub-component: SM system configuration

Sub-component: ME System Configuration

The following section describes the error messages for subcomponent ME System Configuration

Number Message Description

MSAPI_00057 No System Parameters found Not applicable

for selected ME.

Table 31: Sub-component: ME System Configuration

Sub-component: Diameter Routing Quality

The following section describes the error messages for sub-component Diameter Routing Quality:

Number Message Description

Not applicable Route Quality name is missing Not applicable

or empty.

Not applicable Name should be 1 to 50 Not applicable

characters long.

Not applicable alarmThreshold value should be Not applicable

in range of 1 to 1000.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 128

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

Not applicable deprioritizationThreshold value Not applicable

should be in range of 1 to 1000.

Not applicable deprioritizationDuration value Not applicable

should be in range of 1 to 300.

Not applicable priorityReductionAmount value Not applicable

should be in range of 0 to 9.

Not applicable measurementInterval value Not applicable

should be in range of 1 to
155520.

Table 32: Sub-component: Diameter Routing Quality

Sub-component: SLF Lookup Table

The following section describes the error messages for sub-component SLF Lookup Table:

Number Message Description

Not applicable SLF Lookup Table name can Not applicable

not be empty.

Not applicable SLF Lookup Type can not be Not applicable

empty.

MSAPI_00604 Missing Id. Error Reference Number:

MSAPI_00604.\n\ Id is
missing in SLF Lookup Table
Configuration.

MSAPI_00606 Missing Name. Error Reference Number:

MSAPI_00606.\n\ Name is
missing in SLF Lookup Table.

MSAPI_00605 Unknown Lookup Type. Error Reference Number:

MSAPI_00605.\n\ Unknown
Lookup Type {0}. Valid Lookup
Types are {1};

MSAPI_00607 SLF Lookup Table Name can Error Reference Number:

not editable. MSAPI_00607.\n\ SLF Lookup
Table name can not editable
when provision status is {0}.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 129

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00600 SLF Lookup Table Name must Error Reference Number:

be unique. MSAPI_00600.\n\ SLF Lookup
Table with name {0} already
exists.

Table 33: Sub-component: SLF Lookup Table

SM Error Component: Identity To Server Pool

The following section describes the error messages for Identity To Server Pool:

Number Message Description

Not applicable Pattern can not be empty. Not applicable

Not applicable Minimum length of pattern is 1. Not applicable

Not applicable Server Pool Name can not be Not applicable

empty

Not applicable Minimum length of server pool Not applicable

name is 1.

Not applicable Server Pool Name contains Not applicable

invalid characters. Accept
only alphanumeric without
white space and the special
characters (-) and (_) .

Not applicable SLF Lookup Table does not Not applicable

exists.

MSAPI_00700 Missing IdentityToServerPool Error Reference Number:

Configuration. MSAPI_00700.\n\ The
IdentityToServerPool
configuration is missing.

MSAPI_00701 Missing Id. Error Reference Number:

MSAPI_00701.\n\ The
IdentityToServerPool has a
missing Id.

MSAPI_00702 Missing Pattern. Error Reference Number:

MSAPI_00702.\n\ Pattern is
missing or empty.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 130

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00602 SLF Lookup Table for Id does Error Reference Number:

not exist. MSAPI_00602. The
IdentityToServer Pool
configuration with pattern {0}
has a SLF Lookup Table Id that
does not exists in SLF Lookup
Table.

MSAPI_00601 Missing Lookup Type. Error Reference Number:

MSAPI_00601.\n\ Lookup Type
is either missing or not matching
with SLF Lookup Table.

MSAPI_00705 Server Pool Name is Missing. Error Reference Number:

MSAPI_00705.\n\ Server
Pool Name is missing in
the IdentityToServerPool
configuration.

MSAPI_00706 Invalid Server Pool. Error Reference Number:

MSAPI_00706.\n\ Server
Pool Name {0} in the
IdentityToServerPool
configuration does not exists.

MSAPI_00703 Invalid Pattern. Error Reference Number:

MSAPI_00703.\n\ Invalid regular
expression {0}.

MSAPI_00704 Invalid Numeric Range. Error Reference Number:

MSAPI_00704.\n\ Invalid
Numeric Range {0}. Should be
numeric ranges from {1} and {2}
and \n\ separated by {3}.

MSAPI_00707 Invalid IP Subnet. Error Reference Number:

MSAPI_00707.\n\ Invalid IP
Subnet {0}.

MSAPI_00221 Not applicable Failed to update the records, \

Please refer /opt/tpa/logs/

DDESMApplication.log for more
details.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 131

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00222 Not applicable Update has failed for partial

records, Please refer /opt/tpa/
logs/DDESMApplication.log for
more details.

MSAPI_00225 Not applicable Failed to delete the records, \

Please refer /opt/tpa/logs/

DDESMApplication.log for more
details.

MSAPI_00226 Not applicable Delete has failed for partial

records, Please refer /opt/tpa/
logs/DDESMApplication.log for
more details.

Table 34: SM Error Component: Identity To Server Pool

SLF Configuration

The following section describes the error messages for SLF configuration:

Number Message Description

Not applicable SLF name is missing or blank Not applicable

Not applicable SLF servername is missing or Not applicable

blank

Not applicable Either identityAVPs is missing or Not applicable

contains invalid value.

Not applicable Either destination is missing or Not applicable

contains invalid value.

Not applicable Either redirect is missing or Not applicable

contains invalid value.

Not applicable maximum size for description is Not applicable

255 chars

ENUM_00001 Identity Type is Missing. Error Reference Number:

ENUM_00001.\n\ Identity Type
is Missing.

ENUM_00002 Invalid Identity Type. Error Reference Number:

ENUM_00002.\n\ Identity type
{0} is invalid.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 132

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00074 SimpleValueMap not distributed {0} not distributed on MEs {1}

MSAPI_00009 Record not found in SM Record not found {0}

MSAPI_00304 Simple Map is not in provisioned Error Reference Number:

state. MSAPI_00304. \n\ Simple Map
{0} is not in provisioned state.

MSAPI_00300 SLF Lookup Name is Missing. Error Reference Number:

MSAPI_00300.\n\ SLF Lookup
Name is empty or missing in the
SLF Plugin configuration.

MSAPI_00301 Invalid SLF Lookup Table. Error Reference Number:

MSAPI_00301.\n\ SLF Lookup
Table ID {0} in the SLF Plugin
configuration does not \n\ exists.

Table 35: SLF Configuration

Sub-component: Server Pool

The following section describes the error messages for sub-component Server pool:

Number Message Description

Not applicable Not applicable SLF Lookup Table Id can not

be null. its.mename.reqd.on.
impicit_prov_true=Managed
Element Name is required when
\ provisioning Identity To Server
Pool.

Not applicable Not applicable SLF Lookup Table does not

exists.

Not applicable Not applicable Identity To Server Pool created

partially. Reason:\n\ a) Provision
failed as managed element not
found .\n\ b) Provision failed as
some of managed element are
offline.

Not applicable Not applicable Identity To Server Pool does not

exists.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 133

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00700 Missing IdentityToServerPool Error Reference Number:

Configuration. MSAPI_00700.The
IdentityToServerPool
configuration is missing.

MSAPI_00701 Missing Id. Error Reference Number:

MSAPI_00701.The
IdentityToServerPool has a
missing Id.

MSAPI_00702 Missing Pattern. Error Reference Number:

MSAPI_00702.Pattern is
missing or empty.

MSAPI_00703 Invalid Pattern. Error Reference Number:

MSAPI_00703.Invalid regular
expression {0}.

MSAPI_00704 Invalid Numeric Range. Error Reference Number:

MSAPI_00704.Invalid Numeric
Range {0}. Should be numeric
ranges from {1} and {2} and
separated by {3}.

MSAPI_00705 Server Pool Name is Missing. Error Reference Number:

MSAPI_00705.Server
Pool Name is missing in
the IdentityToServerPool
configuration.

MSAPI_00706 Invalid Server Pool. Error Reference Number:

MSAPI_00706.Server
Pool Name {0} in the
IdentityToServerPool
configuration does not exists.

MSAPI_00707 Invalid IP Subnet. Error Reference Number:

MSAPI_00707.Invalid IP Subnet
{0}.

MSAPI_00099 Active pool cannot be updated An active pool cannot be

updated

id9YZ-09148-MT11-PCZZA © 2020 Nokia 134

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00708 Missing Server Pool. Error Reference Number:

MSAPI_00708. Server pool is
either missing or empty.

Table 36: Sub-component: Server Pool

Sub-component: Map and Set

The following section describes the error messages for sub-component Map and Set:

Number Message Description

Not applicable Key Label is missing or blank. Not applicable

Not applicable Key Label should be between 1 Not applicable

and 50 characters long.

Not applicable invalid. Alphanumeric without Not applicable

white space and \ with special
characters (-) and (_) are only
accepted.

Not applicable Name is missing or blank. Not applicable

Not applicable Key Label is missing or blank. Not applicable

Not applicable Key Value should be between 1 Not applicable

and 50 characters long.

Not applicable Flag indicating Map/Set is Not applicable

missing.

Not applicable Duplicate value in list Value {0} is duplicated in the list

Table 37: Sub-component: Map and Set

Sub-component: Local Diameter Configuration

The following section describes the error messages for sub-component Local diameter configuration:

Number Message Description

MSAPI_00027 Not applicable Threshold values should be in

following order.Lower Threshold
< Upper Threshold < Resource
Threshold

id9YZ-09148-MT11-PCZZA © 2020 Nokia 135

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

Not applicable Not applicable Diameter Request Reconnect

Interval(reconnectInterval) must
between 1 and 60 seconds.

Table 38: Sub-component: Local diameter configuration

Countable Event Configuration

The following section describes the error messages for Countable event configuration

Number Message Description

MSAPI_00079 Not applicable Update of the Counatble Event

Config for the given Event name
doesn't exist at the chosen ME.

MSAPI_00080 Not applicable The interval threshold value

cannot be greater than the
maximum number of events per
threshold.

Table 39:

Common Error Codes

The following section describes the common error codes:

Filter Group Errors

The following section describes the Filter group errors:

Number Message Description

DDM_00500 Missing Filter Group Name Error Reference Number: DDM_

00500. Filter group is missing
name for application plugin {0}.

DDM_00501 Duplicate Filter Subgroup Error Reference Number: DDM_

00501.Multiple filter subgroups
with name {0} for application
plugin {1}.

DDM_00502 Filter Criterion With No Values Error Reference Number: DDM_

00502.Null or empty value has
been set for attribute {0} of
context {1} for filter {2} within

id9YZ-09148-MT11-PCZZA © 2020 Nokia 136

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

filter group {3} for application

plugin {4}.

DDM_00503 Invalid Filter Subgroup Name Error Reference Number: DDM_

00503.The referenced filter
subgroup with name {0} does
not exist for application E96
plugin {1}.

DDM_00504 Duplicate Filter Name Error Reference Number: DDM_

00504.One or more filters have
the name {0} within filter group
{1} for application plugin {2}.

DDM_00506 Missing Message Filter Name Error Reference Number: DDM_

00506.One or more filters for
filter group {0} are missing a
name for application plugin {1}.

DDM_00507 Non-Existent Filter Criterion Error Reference Number: DDM_

Context Name 00507.Filter criterion context {0}
does not exist for filter {1} within
filter group {2} for application
plugin {3}.

DDM_00508 Unsupported Filter Criterion Error Reference Number: DDM_

Context Name 00508.Filter criterion context
{0} is unsupported for filter
{1} within filter group {2} for
application plugin {3}.

DDM_00509 Missing Criterion Context Name Error Reference Number: DDM_

00509.One or more criterion
are missing a context name for
filter {0} within filter group {1} for
application plugin {2}.

DDM_00510 Missing Attribute Name Error Reference Number: DDM_

00510.One or more criterion are
missing an attribute name for
filter {0} within filter group {1} for
application plugin {2}.

DDM_00511 Missing Action Context Name Error Reference Number: DDM_

00511.One or more actions
are missing a context name for

id9YZ-09148-MT11-PCZZA © 2020 Nokia 137

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

filter {0} within filter group {1} for

application plugin {2}.

DDM_00512 Missing Action Name Error Reference Number: DDM_

00512.One or more actions
are missing an action name for
filter {0} within filter group {1} for
application plugin {2}.

DDM_00513 Non-Existent Filter Criterion Error Reference Number: DDM_

Attribute Name 00513.Filter criterion attribute
{0} does not exist within context
{1} for filter {2} within filter group
{3} for application plugin {4}.

DDM_00514 Unsupported Filter Criterion Error Reference Number: DDM_

Attribute Name 00514.Filter criterion attribute
{0} is unsupported for context
{1} for filter {2} within filter group
{3} for application plugin {4}.

DDM_00515 Invalid Number Of Filter Error Reference Number:

Criterion Attribute Parameters DDM_00515.Invalid number of
parameters for attribute {0} of
context {1} for filter {2} within
filter group {3} for application
plugin {4}. Expected {5}, Actual
{6}.

DDM_00516 Invalid Filter Criterion Attribute Error Reference Number: DDM_

Parameter Value 00516.Invalid parameter name
{0} of attribute {1} of context {2}
for filter {3} within filter group {4}
for application plugin {5}.

DDM_00517 Failed To Convert Filter Error Reference Number:

Criterion Attribute Parameter DDM_00517.Failed to convert
Value value {0} for parameter {1} of
attribute {2} of context {3} for
filter {4} within filter group {5}
for application plugin {6}. Valid
values are {7}

DDM_00518 Unsupported Filter Criterion Error Reference Number:

Operator DDM_00518.Operator {0} is

id9YZ-09148-MT11-PCZZA © 2020 Nokia 138

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

not supported for attribute {1}

of context {2} for filter {3} within
filter group {4} for application
plugin {5}.

DDM_00519 Unsupported Number of Values Error Reference Number: DDM_

Filter Criterion Operator 00519.Operator {0} does not
support expression with {1}
on LHS and {2} on RHS for
attribute {3} of context {4} for
filter {5} within filter group {6} for
application plugin {7}.

DDM_00520 Filter Criterion With No Values Error Reference Number: DDM_

00520.No values have been set
for attribute {0} of context {1} for
filter {2} within filter group {3} for
application plugin {4}.

DDM_00521 Filter Criterion Unary Operator Error Reference Number: DDM_

With Values 00521.Values not permitted for
unary operator {0} for attribute
{1} of context {2} for filter
{3} within filter group {4} for
application plugin {5}.

DDM_00522 Failed To Convert Filter Error Reference Number: DDM_

Criterion Attribute Value 00522.Failed to convert value
{0} to type {1} for attribute {2}
of context {3} for filter {4} within
filter group {5} for application
plugin {6}. Valid values are {7}

DDM_00523 Non-Existent Filter Criterion Error Reference Number: DDM_

Context Name 00523.Filter action context {0}
does not exist for filter {1} within
filter group {2} for application
plugin {3}.

DDM_00524 Unsupported Filter Criterion Error Reference Number: DDM_

Context Name 00524.Filter action context {0} is
unsupported for filter {1} within
filter group {2} for application
plugin {3}.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 139

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

DDM_00525 Non-Existent Filter Action Name Error Reference Number: DDM_

00525.Filter action {0} does
not exist within context {1} for
filter {2} within filter group {3} for
application plugin {4}.

DDM_00526 Unsupported Filter Action Name Error Reference Number:

DDM_00526.Filter action {0} is
unsupported for context {1} for
filter {2} within filter group {3} for
application plugin {4}.

DDM_00527 Invalid Number Of Filter Action Error Reference Number:

Parameters DDM_00527.Invalid number
of parameters for action {0} of
context {1} for filter {2} within
filter group {3} for application
plugin {4}. Expected {5}, Actual
{6}.

DDM_00528 Missing Filter Action Parameter Error Reference Number:

Name DDM_00528.Missing expected
parameter name {0} of action {1}
of context {2} for filter {3} within
filter group {4} for application
plugin {5}.

DDM_00529 Failed To Convert Filter Action Error Reference Number: DDM_

Parameter Value 00529.Failed to convert value
{0} to type {1} for parameter {2}
of action {3} of context {4} for
filter {5} within filter group {6}
for application plugin {7} Valid
values are {8}

DDM_00530 Invalid Regular Expression for Error Reference Number: DDM_

Filter Criterion Value 00530.Value {0} is not a regular
expression for attribute {1} of
context {2} for filter {3} within
filter group {4} for application
plugin {5}.

DDM_00531 Invalid Filter Attribute Parameter Error Reference Number: DDM_

Value 00531.The value {0} is less than
lower bound {1} for parameter
id9YZ-09148-MT11-PCZZA © 2020 Nokia 140
2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

{2} of attribute{3} of context {4}

for filter {5} within filter group {6}
for application plugin {7}.

DDM_00532 Invalid Filter Attribute Parameter Error Reference Number:

Value DDM_00532.The value {0} is
greater than upper bound {1}
for parameter {2} of attribute {3}
of context {4} for filter {5} within
filter group {6} for application
plugin {7}.

DDM_00533 Invalid Filter Action Parameter Error Reference Number: DDM_

Value 00533.The value {0} is less than
lower bound {1} for parameter
{2} of action {3} of context {4} for
filter {5} within filter group {6} for
application plugin {7}.

DDM_00534 Invalid Filter Action Parameter Error Reference Number:

Value DDM_00534.The value {0} is
greater than upper bound {1}
for parameter {2} of action {3}
of context {4} for filter {5} within
filter group {6} for application
plugin {7}.

DDM_00535 Failed To Convert Filter Error Reference Number: DDM_

Attribute Range Value 00535.Failed to convert value
{0} to type {1} for parameter {2}
of attribute {3}of context {4} for
filter {5} within filter group {6} for
application plugin {7}.

DDM_00536 Failed To Convert Filter Action Error Reference Number: DDM_

Range Value 00536.Failed to convert value
{0} to type {1} for parameter {2}
of action {3} of context {4} for
filter {5} within filter group {6} for
application plugin {7}.

DDM_00537 Missing dependent attributes as Error Reference Number: DDM_

criteria. 00537.The attribute {0} must be
accompanied by attributes {1}

id9YZ-09148-MT11-PCZZA © 2020 Nokia 141

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

in filter group {2}, filter {3} for

plugin {4}.

DDM_00538 Failed To Convert Filter Error Reference Number: DDM_

Attribute Range Value 00538.Failed to convert value
{0} to type {1} for attribute {2}
of context {3} for filter {4} within
filter group {5} for application
plugin {6}.

DDM_00539 Invalid Filter Criterion Attribute Error Reference Number: DDM_

Value Less Than Lower Bound 00539.The value {0} is less than
lower bound {1} for attribute {2}
of context {3} for filter {4} within
filter group {5} for application
plugin {6}.

DDM_00540 Filter Criterion Attribute Value Error Reference Number:

Greater Then Upper Bound DDM_00540.The value {0} is
greater than upper bound {1}
for attribute {2} of context {3} for
filter {4} within filter group {5} for
application plugin {6}.

DDM_00541 Invalid Filter Criterion Attribute Error Reference Number: DDM_

Value 00541.Invalid value {0} for
attribute {1} of context {2} for
filter {3} within filter group {4}
for application plugin {5}. Valid
values are {6}.

DDM_00542 Invalid Filter Criterion Attribute Error Reference Number: DDM_

Parameter Value 00542.Invalid value {0} for
parameter {1} of attribute {2} of
context {3} for filter {4} within
filter group {5} for application
plugin {6}. Valid values are {7}.

DDM_00543 Invalid Filter Action Parameter Error Reference Number: DDM_

Value 00543.Invalid value {0} for
parameter {1} of action {2} of
context {3} for filter {4} within

id9YZ-09148-MT11-PCZZA © 2020 Nokia 142

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

filter group {5} for application

plugin {6}. Valid values are {7}.

Table 40: Filter group errors

Database Errors

The following section describes the Database errors:

Number Message Description

MSAPI_00086 Record doesn't exist Not applicable

MSAPI_00009 Record not found in SM Not applicable

(appending Entity name)

MSAPI_00010 Failed to retrieve record/records Not applicable

from SM

MSAPI_00015 Couldn't fetch from SM Not applicable

MSAPI_00002 Invalid entity persistence Not applicable

MSAPI_00011 Invalid entity Not applicable

Table 41: Database errors

Provisioning Errors

The following section describes the provisioning errors:

Number Message Description

MSAPI_00049 Target MEs are missing for the Not applicable

record

MSAPI_00058 One or more records found Not applicable

which are already linked with
the MEs available in this request

MSAPI_00067 An ACTIVE profile can not be Not applicable

deleted from SM

MSAPI_00092 Active profile can not be Not applicable

updated. Only distribution to
new MEs and Retry provisioning
is allowed.

MSAPI_00030 ME reported error(s) Not applicable

id9YZ-09148-MT11-PCZZA © 2020 Nokia 143

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Number Message Description

MSAPI_00049 Target MEs are missing for the Not applicable

record

MSAPI_00024 Provisioning Not Allowed Not applicable

MSAPI_00009 Record not found in SM Record not found {0}

Not applicable ProvStatus [ aInProvStatus ] not Not applicable

supported

MSAPI_DEFAULT SM is not responding, Please Not applicable

contact the administrator

Not applicable ME Service not available Not applicable

Table 42: Provision errors

System Parameters

The following section describes the System parameters:

Number Message Description

MSAPI_00057 No System Parameters found Not applicable

for selected ME.

Table 43: System parameter

2.10 Tracing and Debugging

Enable log-level debug for troubleshooting

Trace or Debug should not be enabled on the live node where the traffic is more.

The following details should be shared with the services or support team offline.

• Details on VM sizing at which load debug is enabled.

For troubleshooting, the following default packages are enabled.

/opt/tpa/bin/DDE_logLevel
com.nokia.dde.ddm.common.plugins.PluginMessageServicesAndState DEBUG

/opt/tpa/bin/DDE_logLevel
com.nokia.dde.diameter.common.facade.jdiameter.RouteManagerBase DEBUG

The following table lists the packages for identifying more information on the logs according to
scenarios.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 144

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Scenario Package VNF Type

Routing com.nokia.dde.ddmapps. CSD

routing

Downstream Overload com.nokia.dde.ddmapps. CSD

downstreamoverload

Message Validation com.nokia.dde.ddmapps. CSD

msgvalidation

Overload Protection com.nokia.dde.ddmapps. CSD

overloadprotection

SLF com.nokia.dde.ddmapps. CSD

slf

Throttling com.nokia.dde.ddmapps. CSD

throttling

Topology Hiding com.nokia.dde.ddmapps. CSD

topologyhiding

Load Balancer com.nokia.dde.ddm. CSD

common.loadbalancing

RSV com.nokia.dde.ddm.server CSD

Provisioning PROVLOG SM

Import and export IMPORTEXPORT SM

Table 44: Debugging Scenarios

For example, while performing import or export operation following is the sample format for command
execution.

/opt/tpa/bin/DDE_logLevel IMPORTEXPORT $LogLevel

Note: The $LogLevel can have any of the following values:

• INFO
• DEBUG
• TRACE

Routing failure scenario:

The following packages are enabled to test the routing failure scenarios.

/opt/tpa/bin/DDE_logLevel com.nokia.dde.ddmapps.routing DEBUG

/opt/tpa/bin/DDE_logLevel
com.nokia.dde.ddm.common.plugins.PluginMessageServicesAndState DEBUG

id9YZ-09148-MT11-PCZZA © 2020 Nokia 145

2.0
Monitoring and Troubleshooting Guide Troubleshooting

/opt/tpa/bin/DDE_logLevel
com.nokia.dde.diameter.common.facade.jdiameter.RouteManagerBase DEBUG

Note: During trace collection, use valid package or class names specified for logLevel set.
As system does not display any errors in case of invalid package or class names specified.

2.11 CSD remote logging with rsyslog

This section describes the procedure to trace the CSD rsyslogs.

Perform the following steps on the central logging server:

1. Update the /etc/rsyslog.conf file by uncommenting the TCP port line and add the following
template to store the logs.

# Provides TCP syslog reception

$ModLoad imtcp
$InputTCPServerRun 514
$template RemoteLogsTesting,"/var/log/remotehosts/%HOSTNAME%/%$now%.
log"
if $fromhost-ip != '127.0.0.1' then -?RemoteLogsTesting
& stop

2. Restart rsyslog process using the following command.

systemctl restart rsyslog

3. Verify if the rsyslog service is started on the configured port using the following command.

netstat -anp | grep 514

Perform the following steps on both the OAM VMs of CSD or Service Manager:

4. Add the remote server details in /etc/rsyslog.conf file.

The following is the sample output after adding the remote server details.

# ### begin forwarding rule ###

# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create
multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as
possible)

id9YZ-09148-MT11-PCZZA © 2020 Nokia 146

2.0
Monitoring and Troubleshooting Guide Troubleshooting

*.* @@10.75.105.123:514

5. Restart the rsyslog process using the following command.

systemctl restart rsyslog

6. Verify if the central logging server is receiving the syslog.

2.12 Troubleshooting System

Monitoring

After the VNF is instantiated, monitor the stack creation status through orchestrator (heat stack-list)
and monitor the VM’s status on either dashboard or orchestrator (nova list). To monitor the installation
status or progress, get the vnc console of the VM and open in browser: “nova list; nova get-vnc-
console <ID of VM> novnc”.

[root@heattoolkitvm ~(keystone_vz_dde)]# nova get-vnc-console <ID of VM>

novnc
+-------+----------------------------------------------------------------
-----------------------+
| Type | Url
|
+-------+----------------------------------------------------------------
-----------------------+
| novnc | https://<IP of VM>:13080/vnc_auto.html?token=fef26ded-830d-
4aae-be71-383a6a015b00 |
+-------+----------------------------------------------------------------
-----------------------+
[root@heattoolkitvm ~(keystone_vz_dde)]#

Paste the URL in browser to open VM’s console and login as a root user to access logs. All the
installation logs are stored in path /opt/config/log.

Check the following log to know the installation status:

1. /var/log/cbam/<logs> (all.log, workflow.log and other logs on CBAM node) - to check the
workflow and installation traces.
2. /opt/tpa/logs/ - on individual nodes to check the system status.

Also check the /var/log/messages and verify the configuration status to check whether the internal
IP assigned by DHCP is configured or not (eth0 interface, 192.x.x.x IP range). If the internal IP is
not configured, then check whether the DHCP discover being sent and offers being received or not.
Installation could fail, if the node does not receive the DHCP offers within the specified time.

If the DHCP offers are received, then the internal IP is being plumbed and localization completes, then
monitor the OAM IP plumb on eth1. Log in to the VM using its OAM IP, after the IP is plumbed.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 147

2.0
Monitoring and Troubleshooting Guide Troubleshooting

One of the reason for GrowNode to fail is due to resource shortage, further each log file needs to
check for corresponding errors.

Troubleshooting

To... Do...

Add the appropriate route on diameter nodes. Ensure that the node-specific IP route information
is added in the Route tab of sdfc.conf.xls.

Check SNMP process status Execute the following command:

systemctl status calm-alma

Get the process status

healthMachines

health

Fetch active alarm status

aladmin list

Clear alarm using index number aladmin delete <Alarm_Index>

Create a user in CSD createUser <User>

For alarms, counters, and notifications Check the logs under directory /var/log/alarms in
active OAM node.

• /var/log/alarms -- to check alarms

2.12.1 Host node or Compute node unavailability

Unexpected unavailability of Compute node or Host node may result in call failures.

• If the Host node hosting the Active IO fails, call failures are expected till the calls are switched over
to Standby IO. Default switchover time is 8s.
• If the Host node hosting both the Active IO and Standby IO fails, call failures are expected till the
Host and VM becomes available.
• If the Host node hosting Application VMs fails, then the calls that are in process are expected to
fail. There is no impact on new calls or sessions handled by other available Application VMs.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 148

2.0
Monitoring and Troubleshooting Guide Troubleshooting

2.13 Troubleshooting and monitoring Aerospike

Collecting Aerospike database configuration and log information for offline analysis

On the CSD and SM, when Aerospike database issues require offline analysis, it is often necessary
to collect configuration and log information. This information is spread across the DB nodes in
the system and can be tedious and error prone to collect all the right information from all the right
nodes. Therefore, the CSD and SM provide a tool that collects all the required data and creates a
compressed .tar file on the active OAM. This single file can then be forwarded to Nokia’s technical
support for offline analysis.

Usage:

db-collect-logs -d <directory> [-p] [-v]

Where:

• no arguments or -h displays the help information.

• -d <directory> specifies the output directory of the .tar file. This file requires a large amount of
free space.
• -p specifies that platform logs, including OMlog and debuglog.
• -v enables verbose mode to list the .tar file contents.

For more information on log collection, see Troubleshooting on page 10.

Note:

The data or configuration written to a database node is replicated on another database node
in a local site. Thus, if two database nodes have master and replicated copies goes out of
service state, before re-syncing within the cluster then this may result in data or configuration
loss.

Aerospike credentials

This section lists all the credentials related to the aerospike database.

Tool Name User Name Description

aql dbUser Specifies the name of the user.

asbackup dbUser Specifies the name of the user.

asrestore dbUser Specifies the name of the user.

asinfo dbUser Specifies the name of the user.

asadm dbUser Specifies the name of the user.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 149

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Note: You need to enter the database password while executing the preceding tools. To get
the password, contact the Nokia support team.

2.13.1 To collect Aerospike database configuration and log information for offline
analysis

1. Log in to the active OAM as a root user.

2. The /BACKUP and the //BACKUP/aerospikeconflogs directory ( ) on the Active OAM enables
storage of the file. Use the following df command to view the available space. For example:

df [-h | -k] /BACKUP

3. Use the db-collect-logs tool to create a compressed tar file on the active OAM, in the
directory you specify.

To see the usage information, type the command name with no arguments:

db-collect-logs

An example command for collecting the needed logs, including the platform logs, with a verbose
option so that a list of collected information is displayed:

db-collect-logs -d /BACKUP/aerospikeconflogs -v -p

4. Find the output file and send it to Nokia technical support. When db-collect-logs tool
completes, the location of the resulting file is displayed:

Completed. Tar file created in [/BACKUP/DDELogs-2016-12-09.tar.gz]

2.13.2 Monitoring database status

The following are the namespaces in the database:

• dsclocal: All data under this namespace is available only to local site.
• dsc: This namespace is replicated on geo mate, when XDR is configured.
• dscglobal: All data under this namespace is replicated to geo mate and other configured remote
geo sites (when XDR is enabled).

The parameters to be considered while debugging XDR in the file /opt/tpa/aerospike/bin/

asinfo -U<username> -P<password> -v statistics -l:

• dlog_free_pct: must not be on the increase.

• xdr_ship_latency_avg: must not be on the increase. If the value is increasing it indicates that the
XDR data is being shared or synced.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 150

2.0
Monitoring and Troubleshooting Guide Troubleshooting

• xdr_ship_outstanding_objects: must not be on the increase. This parameter indicates the

remaining amount of data supposed to be synced between the sites.

To.. Do..

Check the DB content Use the aql tool provided by Aerospike. Run the aql
as ddeadmin user with the following credentials.

aql -U<username> -P<password>

Where:

-U<username> specifies the user name of the

aerospike database.

-P<password> specifies the password of the

database.

Note: To get the password, contact the Nokia support

team.

Show namespaces - displays all namespaces

available

Show sets - displays all sets across namespaces.

select * from <name space>.<set name> - displays

records in a particular set under a namespace.

Check the number of records in any set select * from <name space>.<set name>. This
displays all records and also at the end, displays the
total number of records.

Check the DB status /opt/tpa/aerospike/bin/asinfo -

U<username> -P<password> -v namespace/
dscglobal –l

Where:

-U<username> specifies the user name of the

aerospike database.

-P<password> specifies the password of the

database.

Note: To get the password, contact the Nokia support

team.

A good status of the DB is that no records must get

evicted with this command.

Example Output:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 151

2.0
Monitoring and Troubleshooting Guide Troubleshooting

To.. Do..

-sh-4.2$ /opt/tpa/aerospike/bin/
asinfo -v namespace/dscglobal -l |
grep -i evic type=device evicted-
objects=1103 evict-tenths-pct=5 evict-
hist-buckets=10000 cold-start-evict-
ttl=4294967295 -sh-4.2$

1103 records are evicted in this example, this

indicates that the DB is above 60% full, it is a feature
to randomly evict records if the DB is above 60% full.

Check sychronisation status The TIMELAG alarm is raised when the geo-
redundancy sites did not synchronize any data with
each other.

Example:

<AlarmCode>53005</AlarmCode>
<AdditionalText>TIMELAG=83</
AdditionalText>

Indicates that the sites have not synced for the past 83
seconds.

Check the DB version Run asinfo command from OAM with -h <DB
Host IP> as option (asinfo -h <DB IP>) and read the
version

If you are logged in as ddeadmin on OAM, then run

the asinfo command and read the version path of
asinfo is /opt/tpa/aerospike/bin.

Check Logs necessary to diagnose DB behavior The tar files are generated by running the command.

-U<username> -P<password> -e collectinfo

on all the DB nodes.

Where:

-U<username> specifies the user name of the

aerospike database.

-P<password> specifies the password of the

database.

Note: To get the password, contact the Nokia support

team.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 152

2.0
Monitoring and Troubleshooting Guide Troubleshooting

To.. Do..

Note: The following error No such file or

directory is thrown while executing asadm
command. This can be ignored. As, asadm is a third-
party command, it assumes that after installation
all the files are present in a location specified in the
script. Whereas, file storage path is customized.

Following is an example for No such file or

directory error:

[ERROR] cat: /etc/aerospike/aerospike.

conf: No such file or directory

Alternately, this command can also be executed on

OAM with -h <DB node>, -p <Port> option.

Check the following:

• aerospike.conf in /opt/tpa/aerospike/etc on all DB

nodes
• aerospike_warning.log in the patch /op/tpa/logs on
all DB nodes.
• aerospike.log in the patch /op/tpa/logs on all DB
nodes

Check the initial DB default configuration Contents of /opt/tpa/aerospike/etc/aerospike/

aerospike.conf file indicates with what default
configuration DB has started.

Check the DB running status The systemctl status NOKIAasdb tells us whether DB
is up and running or not.

Note:

If select * from <name space>.<set name> query times out in AQL, while retrieving records,
increase the query time out by setting set timeout <milli seconds>. For example, to set the
timeout to 10 seconds, execute

set timeout 10000

in AQL.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 153

2.0
Monitoring and Troubleshooting Guide Troubleshooting

2.13.3 Debugging the connectivity between geo-redundant sites

Problem Solution

When the network link goes down between The DB connectivity between local and remote
the two geo-redundant sites, CSD raises sites might be not be reachable. Use telnet,
XdrClusterUnreachableAlarm. ping or any other network troubleshooting tools
to verify if the DB nodes between the sites are
reachable.

Example: telnet <DB_IP> <Port_Number>

Table 45: Debugging the connectivity between Geo-redundant sites

2.14 Managing CSD processes

2.14.1 About CSD processes

Application processes running on OAM and APP (Non-OAM) nodes on both CSD and SM are started
automatically after installation. However, in troubleshooting situations, these processes can be
managed manually by the ddeadmin user on the respective nodes.

Application processes
VNF Type Applicable node
running

CSD OAM
• ASR

CSDIO
• ASR

CSDAPP
• ASR

DB
• Aerospike Daemon

SS7IO
• ASR

SM OAM
• ASR

DB

id9YZ-09148-MT11-PCZZA © 2020 Nokia 154

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Application processes
VNF Type Applicable node
running

• Aerospike Daemon

Table 46: Node details

To find the diameter’s (CSD application) process ID running on IO:

[root@NepalDDE-ddeio-1 ~]# ps -eaf | grep ddeadmin | grep "osgi/instance/

felix.properties"
ddeadmin 4544 4517 7 11:19 ? 00:05:53 java
-Xbootclasspath/p:bundles/com.nokia.as.dtls.provider-1.0.2.jar -
Dgroup.name=runtime -Dcomponent.name=component -Dinstance.name=instance
-Dinstance.pid=4544 -Dhost.name=localhost -
Dplatform.name=csf -Dconfig.dir=/opt/DDE_18_2_I108/osgi/instance
-DINSTALL_DIR=/opt/DDE_18_2_I108/osgi -
Dbundle.installer=bundles/com.alcatel.as.service.bundleinstaller.impl-1.
0.2.jar -Dfelix.config.properties=
file:/opt/DDE_18_2_I108/osgi/instance/felix.properties
-DFastCacheImpl.retryConnect=true -cp bundles/
org.apache.felix.main-5.6.10.jar:bundles/com.alcatel.as.felix.launcher-1.
0.3.jar:/opt/DDE_18_2_I108/
osgi/instance:/opt/DDE_18_2_I108/osgi/resource:/opt/DDE_18_2_I108/osgi/
resource -d64 -server -Xms3072m
-Xmx3072m -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.
jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=3333 -Djava.security.egd=file:/dev/./
urandom -Djavax.net.ssl.
keyStore=/opt/tpa/conf/sac/server.keystore -Djavax.net.ssl.
keyStorePassword=123456 -
Djavax.net.ssl.trustStore=/opt/tpa/conf/sac/server.keystore -
Djavax.net.ssl.trustStorePassword=123456 -XX:NewRatio=2
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -
XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses -XX:
+CMSClassUnloadingEnabled
-XX:+UseCompressedOops -XX:+CMSScavengeBeforeRemark -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -Xloggc:/opt/tpa/logs/gc.log -XX:
+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=8 -XX:GCLogFileSize=8m -XX:
+UnlockDiagnosticVMOptions
-XX:+UnsyncloadClass -Djava.awt.headless=true -Djava.library.path=lib:/
opt/tpa/lib
com.alcatel.as.felix.FelixLauncher /opt/DDE_18_2_I108/osgi/instance/
felix.properties
component runtime__instance <modid>NepalDDE-ddeio-1 4544
<fchost> <sudo-flag>

id9YZ-09148-MT11-PCZZA © 2020 Nokia 155

2.0
Monitoring and Troubleshooting Guide Troubleshooting

2.14.2 Start CSD application process

To start all the application processes running on a node:

1. Log in to the node as ddeadmin user.

2. Execute the following command.

DDE_control start

CAUTION! If any DB VM or DB process is stopped (effectively removing it from the DB

cluster) for a minimum of 6 hours continuously, then the disks present in /datastore/dsc
on that DB VM should be deleted before switching it on (effectively bringing it back to the DB
cluster).

2.14.3 Stop CSD application process

To stop all the application processes running on a node:

1. Log in to the node as ddeadmin user.

2. Execute the following command.

DDE_control stop

CAUTION! If any DB VM or DB process is stopped (effectively removing it from the DB

cluster) for a minimum of 6 hours continuously, then the disks present in /datastore/dsc
on that DB VM should be deleted before switching it on (effectively bringing it back to the DB
cluster).

2.14.4 AsControl command

The path of AsControl script is /opt/tpa/bin/AsControl.

[root@nepalDDE-oame-0 nepalDDE-ddeapp-0]# AsControl Usage: AsControl

<command> <NodeType> where 'command' is one of: start - start both the
database and asr server. stop - stop both the database and asr server.
where 'NodeType' is one of: oam - To perform above mentioned operation on
oam nodes. ddeapp - To perform above mentioned operation on application
nodes. ddeio - To perform above mentioned operation on IO nodes. db - To
perform above mentioned operation on database nodes. By default NodeType
will be all nodes.

2.14.5 Selectively start all the application process

To selectively start application processes of a node type:

1. Log in to the active OAM as ddeadmin user.

2. Execute the following command:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 156

2.0
Monitoring and Troubleshooting Guide Troubleshooting

AsControl start <NodeType>

Where <NodeType> can be OAM, DDEAPP, DDEIO, DB, and SS7IO.

2.14.6 Selectively stop all the application process

To selectively stop application processes of a node type:

1. Log in to the active OAM as ddeadmin user.

2. Execute the following command:

AsControl stop <NodeType>

Where <NodeType> can be OAM, DDEAPP, DDEIO, DB, and SS7IO.

2.14.7 Start the process

Execute the AsControl start command to start the process.

The usage of the AsControl script is in the following format.

AsControl <command> <NodeType>

where <command> is start.

<NodeType> is all.

2.14.8 Stop the process

Execute the AsControl stop command to stop the process.

The usage of the AsControl script is in the following format.

AsControl <command> <NodeType>

where <command> is stop.

<NodeType> is all.

2.14.9 Check the process status

On active node, log in as ddeadmin by executing the command su - ddeadmin

To check the process, execute the following command:

DDE_status

2.14.10 To check the health of the node

This tool is used to run diagnostic modules on the node and also to check the health of the system.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 157

2.0
Monitoring and Troubleshooting Guide Troubleshooting

For command usage and sample examples, please run /opt/healthcheck/bin/health -H on

OAM nodes, it lists general help information and verbose help for all modules.

Attention:

On running proc_netstat health command:

• number_of_tcp_connections and count of java process is expected to be more than

the default configured value 100 on IO node. The count is expected to be in the range
100 or more (based on the number of diameter peers configured).

• number_of_tcp_connections and count of asd process depends on number of APP

or IO nodes.

With two Application VMs, two IO, and two OAM nodes, the number of TCP connection
for asd process for each database VM is around 40. So, the count is expected to
increase when the number of Application VMs, IO, and OAM nodes gets increased.

[root@nepalDDE-oame-0 ~]# /opt/healthcheck/bin/health

SYNOPSIS:
health [global options(hHioldp)] module/suite [command options]
[parameters]
DESCRIPTION:
This tool is used to run diagnostic modules to check the
health of the system. Multiple modules may be grouped into
a suite and run with a single keyword. This is version
3.1.5.56 of the tool.
Global options are as follows
-h - General help information.
-H - General help information plus verbose help for all modules.
-i - List routines in common library.
-o - Additional output (logOnly stdout xml).
-l - Override the default log output directory with some other
directory. The default is /opt/healthcheck/log/healthcheck.
-d - Override the default xml file path with a new absolute file
path. The default is /opt/healthcheck/log/healthcheck/output.xml
-p - Plugin package where the specified module/suite is located. If
the module/suite is not found, other module/suite directories
will still be searched.
Modules are defined in:
/opt/healthcheck/modules or /opt/healthcheck/plugins/*/modules
Suites are defined in:
/opt/healthcheck/suites or /opt/healthcheck/plugins/*/suites
Logs and output are located in this directory (default):
/opt/healthcheck/log
Configuration files are located in this directory:
/opt/healthcheck/config

EXAMPLES:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 158

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Print general help for health check (this output) :

health -h
Run a particular module or suite:
health <module or suite>
Print general help information or detailed descriptive
information for a particular module or suite:
health <module or suite> -h
COMMON MODULE LIST:
health aide_audit [-c <config_file>] [i] [-n <node_id>] health cpu_
usage [-i interval]
health cron_check
health disk_usage
health dns
health equipage_info
health ethernet
health file_check -d <directories_list> -s <file_size> [-p <depth>] [-
n <node_id>]
health jarsign_check [-n <node_id>] [-f <jar_file>]
health last_restart [-t hours]
health ping
health proc_cpu_usage [-p <process_names>] [-] <number_of_top_
processes>] [-n <node_id>]
health proc_cpu_usage [-p <process_names>] [-] <number_of_top_
processes>] [-n <node_id>]
health proc_netstat [-n <node_id>]
health ram_usage
health time [-t sec]
health version [-n <node_id>]
COMMON SUITE LIST:
NONE

2.14.11 Check the CSD node status

On active OAM node, to list all the VMs and its status:

[root@NepalDDE-oame-1 ~]# healthMachines -h

Usage: healthMachines
This script shows the status of each nodes as OK/NOK based on services
running in each node. Following are the list of service check, made on
each node.
OAM, APP node, IO node: Will check for NOKIAasr service status.
Active IO node : Along with the service, extra check is made to
examine whether it's listening on 3868 port.
DB node : Will check for NOKIAasdb service status.
[root@NepalDDE-oame-1 ~]# healthMachines
NepalDDE-oame-0(STANDBY) OK
NepalDDE-oame-1(ACTIVE) OK
NepalDDE-ddeio-0(STANDBY) OK

id9YZ-09148-MT11-PCZZA © 2020 Nokia 159

2.0
Monitoring and Troubleshooting Guide Troubleshooting

NepalDDE-ddeio-1(ACTIVE) OK
NepalDDE-ddeapp-0(NA) OK
NepalDDE-ddeapp-1(NA) OK
NepalDDE-db-0(NA) OK
NepalDDE-db-1(NA) OK

2.15 Defense mechanism

Defense and filtering mechanism

The following restrictions are imposed on CSD for its defense:

• FTP and TELNET is blocked on all nodes.

The DB and Applications nodes are not reachable from outside of the Virtual Network Functions
(VNFs), however are accessible only from OAM and IO nodes.

ICMP is enabled on OAM (physical, floating IP) nodes and on IO (physical, floating IP) nodes.

• The access to the following are password protected:

– Log in to CBAM VM.

– Users are authenticated based on their user-name or password AUTH_URL and certificates
on CBIS lab.
– OpenStack resources cannot be accessed without proper credentials. All OpenStack
commands are validated against the keystones whenever executed.
– Access to OAM nodes is password protected, only a rmtadm user can log in to OAM nodes.

• IO node accepts the Diameter requests from all the peers when Whitelist is not enabled.

When Whitelist is enabled for the peers, the CSD starts accepting the Diameter traffic only from
the White listed peers and the rest are rejected.

Note: For inbound peers, if multiple un-configured or unexpected peer connection or

peer disconnections are observed, whitelist must be enabled.

• To troubleshoot issues like when peers are not able to connect to CSD, then you need to check
whether the appropriate ports are enabled in the firewall.

For example, to receive diameter messages from peers on port 3868, this port has to be enabled
in the firewall.

Refer to the Diameter routing Chapter in the CSD User Guide for complete details of the Whitelist
feature and of the plugin sequence followed in call processing.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 160

2.0
Monitoring and Troubleshooting Guide Troubleshooting

2.16 Log collection and VM recovery

In case of outages, collect the following inputs before recovering the VM:

1. Collect the heapdump from the IO and App VMs as a ddeadmin user using the following
command:

/opt/tpa/java/bin/jmap -dump:live,file=/opt/tpa/logs/xyz.bin `cat /opt/

tpa/osgi/var/tmp/pids/instance.pid`

Where xyz.bin is the file name which gets generated post successful execution of the command.

Note: The heapdump command triggers a Full GC (FGC) on the JVM and has traffic
impact. It is recommended to use only under R&D request or supervision. Also the cinder
(or path where the file gets stored) must have enough space to collect the heapdump file
(the available free space must be approximately equal to the RAM size of the VM).

2. Verify the connectivity between DB VMs and all other VMs, collect the output of the following
command from the OAM, IO and App VMs.

netstat -na | grep 3000

3. Verify the connectivity between IO and App VMs, collect the output of the following command from
the IO VM.

netstat -na | grep 1210

4. Collect the output of the following command from the App and IO VMs as a ddeadmin user.

listDiameterPeers.sh -s
5. Collect the aladmin list from the OAM VM.
6. Collect the tcpdump from the App VM for a very small duration.

Note: Do not collect the tcpdump, if the VM is handling high load.

7. Collect the output of the following command from all the VMs (These can be collected post
recovery of VM also).

df -k
8. Collect the output of the following command (These can be collected post recovery of VM also).

SaveLogFiles

Monitor the alarms, DDESMApplication logs, sar and top output. If App or IO VM is not responding or
responding with errors due to memory leak or overload, then the VM can be recovered by restarting
the application on the respective VMs.

The following command can be used to find the memory leaks on IO or App VMs. In case of memory
leak, the FGC value grows gradually and OU value gradually reaches to OC value.

/opt/tpa/java/bin/jstat -gcold `cat /opt/tpa/osgi/var/tmp/pids/

instance.pid` 2s

id9YZ-09148-MT11-PCZZA © 2020 Nokia 161

2.0
Monitoring and Troubleshooting Guide Troubleshooting

For example,

[root@ddebvnf-106-ddeio-0 ~]# /opt/tpa/java/bin/jstat -gcold `cat /opt/tpa/osgi/var/tmp/pids/instance.pid` 2s

MC MU CCSC CCSU OC OU YGC FGC FGCT GCT
65512.0 58617.8 9048.0 7161.0 2097152.0 42420.7 8455 4 0.426 426.522
65512.0 58617.8 9048.0 7161.0 2097152.0 42420.7 8455 4 0.426 426.522

2.16.1 Procedure to recover a VM that is not accessible through SSH

Prerequisite

1. Log in to the console of non-reachable VM through a dashboard.

Edit the /etc/ssh/sshd_config

PermitRootLogin yes and then restart the SSH service by executing the following command.

systemctl restart sshd

2. Copy the .ssh directory from any healthy VM (preferably OAM) to the non-reachable VM.

For example, as described in the following section.

scp -r .ssh root@<OAM VMs IP>:/root

2.17 Collecting information for support

Required information for support

When you require technical assistance to resolve a problem on the CSD, you can expedite the
assistance process by collecting the platform and software information that is listed in the table below.

Information type Action

Software and platform specifications Record the following:

• CSD release ID
• Platform information, including model, CPU
type, disk configuration and partitioning, and
the amount of installed RAM

System and software logs Collect the required log files in a compressed
archive. View the log files in the /opt/tpa/logs
directory (see To view CSD log files on page
200).

Execute the command SaveLogFiles on active

OAM to get all the logs.

Actions performed before and after the problem Collect the following:
occurred

id9YZ-09148-MT11-PCZZA © 2020 Nokia 162

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Information type Action

• screen captures or a text version of the error

or exception message
• a summary of the actions that were
performed before the problem occurred; for
example, a configuration change
• a description of the troubleshooting actions
performed and the results

Table 47: Information required by technical support

Usage of SaveLogFiles (Distributed CSD)

The logs can be found accessing the following path.

/opt/tpa/bin/SaveLogFiles

[root@ddebvnf-oame-1 BACKUP]# SaveLogFiles -h

OAM is active
/opt/tpa/bin/SaveLogFiles: illegal option -- h
Usage: SaveLogFiles [days] [small] [-n <nodes>] [-n <nodes> <days>]
[-p <days to collect prometheus data>]
where,
SaveLogFiles : To collect logs and prometheus
data for default 5
days and configuration from all the
nodes.
SaveLogFiles <days> : To collect logs for specified
days and prometheus
data for default 5 days
and configuration
from all the nodes.
SaveLogFiles <small> : To collect logs and prometheus
data from oam nodes
for a day and config
from all the nodes.
SaveLogFiles -n <nodes> : To collect logs and prometheus
data from specified
nodes and oam nodes for
default 5 days and
config from all nodes.
SaveLogFiles -n <nodes> <days> : To collect logs from specified
nodes and oam nodes
for specified days and config
from all nodes
and prometheus data for
default 5 days.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 163

2.0
Monitoring and Troubleshooting Guide Troubleshooting

SaveLogFiles -p <days> : User will be prompted to take

input, whether to
collect only the prometheus
data or the
logs, configs and prometheus data.
If yes, only the prometheus
data will be
collected for specified days else
logs will
also gets collected for
default 5 days.
SaveLogFiles -p<day1> <day2> : To collect logs for the
specified days day2 and
prometheus data for day1 days.
[root@ddebvnf-oame-1 BACKUP]#

Note: If space in /BACKUP directory is insufficient to collect prometheus data, free some
space in /BACKUP directory and re-run the script.

The following figure describes the SaveLogFiles tool output in /BACKUP directory of active OAM.

All the nodes configuration gets collected in a separate tar file and all the logs gets collected in
separate tar file as shown in the preceding figure.

Example:

Note: The oam_backup_volume size defined in the vnfd.scalable.tosca.yaml files

are used for storage of the SaveLogFiles (/BACKUP directory on Active OAM node). So
during instantiation, allocate this space based on the log storage requirement.

Example:

[root@ddebvnf-oame-1 BACKUP]# ls
AppServerConfigFiles.tar.gz LogFiles.tar.gz

Usage of SaveLogFiles (Bare-metal)

The following figure describes the SaveLogFiles tool output in /BACKUP directory of active node.

In Bare-metal, logs of both active and standby node gets collected in a single tar file and configuration
of each node gets collected in a separate tar file as shown in the preceding figure.

Example:

[root@dumpty BACKUP]# ls

id9YZ-09148-MT11-PCZZA © 2020 Nokia 164

2.0
Monitoring and Troubleshooting Guide Troubleshooting

DDE_configBackup.dumpty.CSD_18_8_I114.DDESM.20180907065003.tar
DDE_configBackup.humpty.CSD_18_8_I114.DDESM.20180907065003.tar
LogFiles.tar.gz

2.18 Known issues

2.18.1 Known issue

Error: NTP Sync issue

For Aerospike geo-sites DB should be in sync with NTP.

Reason:

Aerospike geo-sites DB is not in sync with NTP.

Solution/Workaround

Command to check NTP synchronization: ntpstat

If NTP is not in sync, then restart NTP Daemon using the following command on the VM:

/lib/app/appservice restart ntp

2.18.2 Error: StaleStateException

The alarms that are raised or cleared when the OAM node is down are not cleared and they must be
cleared manually.

When a database is down the following exception seen in IO node can be ignored.

Reason: The DiameterPeerStatus table is being updated by both the IO nodes. For an outbound
connection on a floating AddressGroup, both the IO nodes try to connect and update the same
record.

To resolve this condition, the StaleStateException is introduced.

During fetch and update of a record, if the other node has updated the same record, then the DB
throws the following error.

StaleStateException
Failed to add DiameterPeerStatus object to DB
com.nokia.dde.common.db.StaleStateException: Stale version for object
com.nokia.dde.diameter.par.DiameterPeerStatus

2.18.3 Error: There is an Unknown Error from Server

If SM Pilot switch-over occurs, either because of Active OAM crash/reboot or process crash, then
the SM GUI may not be accessible as it throws the error Error! There is an Unknown Error
from Server
id9YZ-09148-MT11-PCZZA © 2020 Nokia 165
2.0
Monitoring and Troubleshooting Guide Troubleshooting

Reason:

1. Scenario 1: When you click on any object, this error is thrown as tcp ports still try to connect to old
connection because of browser cache.
2. Scenario 2: The preceding error can also be displayed on SM GUI during bulk provisioning when
there is a network delay between SM and CSD OAM.

Solution/Workaround

1. Scenario 1: Refresh the browser or clear the browser cache and then log in to SM GUI.
2. Scenario 2: If the error is due to network delay, then re-provision the entries that are in
unprovisioned state.

2.18.4 Error: Could not find result context... .. <attributes>

After framing the rules, although the rules are correct, the error Error: Could not find result
context... .. <attributes> is displayed.

Reason:

The editor does not know that rule framing is completed.

Workaround:

Give “;” or press Enter after last statement to resolve the issue. And if error continues to exist, then the
rule has to be corrected.

2.18.5 Error: Record doesn't exist

In case of geo-redundant SM, for any GUI operations performed on ME at Site A, you receive the error
Record doesn't exist on ME tab of Service Manager GUI on Site B.

Reason:

ME status update does not happen as still the heart beat is not exchanged between SM and CSD.

Workaround:

Ensure that you wait for minimum 2 minutes before performing any SM GUI operations at Site B.

2.18.6 Error: Could not add user ddeadmin

During the installation of CSD, you see an error ERROR: Could not add user ddeadmin and the
installation is shown as successful in CBAM.

Reason:

Due to selinux permission, ddeadmin user is not allowed.

Workaround:

You can ignore the error as this user is added during the installation of CSD.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 166

2.0
Monitoring and Troubleshooting Guide Troubleshooting

2.18.7 Error: Result=0001 Package NOKIA config has reported an error. Please
choose one of the following: * 1) Retry 2) Skip Package and Continue Choice:
[1-2,?,n,p,q] retry returning 9
During installation of CSD, you see an error Error: Result=0001 Package NOKIA config
has reported an error. Please choose one of the following: * 1) Retry 2)
Skip Package and Continue Choice: [1-2,?,n,p,q] retry returning 9 and the
installation is shown as successful in CBAM.

Reason:

Due to selinux permission, ddeadmin user is not allowed.

Workaround:

You can ignore the error as this user is added during the installation of CSD.

2.18.8 Error: Rule System Version (RSV) provisioning error

When the standby node is down and you provision the RSV, then the SM throws the following error:

Provisioning Failed

Reason:

This is an expected behavior and there is no impact on functionality.

Workaround:

You can ignore this error as it is an expected behavior.

2.18.9 Error: Prometheus service is shown DOWN in Grafana dashboard

The prometheus service is shown DOWN in Grafana dashboard but the execution of the following
command on OAM node shows the service as ACTIVE (running).

systemctl status prometheus

Reason: On restarting prometheus service, it fails to start thereafter as the data is corrupted.

Workaround: From the following path, select and remove the latest corrupted data folder or file.

rm -rf /prometheus/data/<Corrupted_Data_folder_or_file>;systemctl start

prometheus;systemctl status prometheus;

2.18.10 Error: Opening storage failed unexpected end of JSON input

If the prometheus service terminates abruptly during storage of prometheus data, empty JSON files are
created in the appdata/prometheus/data folder and the following error message is thrown.

Opening storage failed unexpected end of JSON input

The following is the sample error format.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 167

2.0
Monitoring and Troubleshooting Guide Troubleshooting

Reason: The prometheus service becomes inactive as the prometheus service is disabled and the
preceding error is displayed in the following directory.

/var/log/messages

The following section describes the typical format of empty JSON files.

Workaround: Remove the prometheus data folders containing empty JSON files using the following
command:

rm -rf

Reload the daemon service using the following command:

systemctl daemon-reload

Post successful reload of daemon service, restart the prometheus.service.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 168

2.0
Monitoring and Troubleshooting Guide Troubleshooting

2.18.11 Error: RabbitMQ Error On Write Thread

The following error is thrown when all the DB nodes attempt to connect to the rabbitmq clusters
running on both the OAM nodes periodically. On successful establishment of connection, the attempts
are stopped and the recovered trace is returned.

<2019.11.15 21:34:04 902 +0530><E><ddebvnf-ddeapp-1>

<RabbitMQ Error On Write Thread><com.rabbitmq.client.impl.
ForgivingExceptionHandler:119>
An unexpected connection driver error occured
java.net.SocketException: Connection reset by peer (Write failed)
at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.
java:110)
at java.base/java.net.SocketOutputStream.write(SocketOutputStream.
java:150)
at java.base/java.io.BufferedOutputStream.
flushBuffer(BufferedOutputStream.java:81)
at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.
java:142)
at java.base/java.io.DataOutputStream.flush(DataOutputStream.java:123)
at com.rabbitmq.client.impl.SocketFrameHandler.flush(SocketFrameHandler.
java:177)
at com.rabbitmq.client.impl.AMQConnection.flush(AMQConnection.java:573)
at com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:134)
at com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.
java:452)
at com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.
java:433)
at com.rabbitmq.client.impl.AMQConnection.
handleConnectionClose(AMQConnection.java:848)
at com.rabbitmq.client.impl.AMQConnection.
processControlCommand(AMQConnection.java:801)
at com.rabbitmq.client.impl.AMQConnection$1.processAsync(AMQConnection.
java:244)

id9YZ-09148-MT11-PCZZA © 2020 Nokia 169

2.0
Monitoring and Troubleshooting Guide Troubleshooting

at com.rabbitmq.client.impl.AMQChannel.
handleCompleteInboundCommand(AMQChannel.java:182)
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:114)
at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.
java:652)
at com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.
java:48)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.
java:599)
at java.base/java.lang.Thread.run(Thread.java:834)
<2019.11.15 21:34:09 912 +0530><D><ddebvnf-ddeapp-1><AMQP Connection 192.
168.3.10:5672>
<com.rabbitmq.client.impl.recovery.AutorecoveringConnection:570>
Connection amqp://[email protected]:5672//csd has recovered
<2019.11.15 21:34:09 917 +0530><D><ddebvnf-ddeapp-1>
<AMQP Connection 192.168.3.10:5672><com.rabbitmq.client.impl.recovery.
AutorecoveringConnection:627>
Channel AMQChannel(amqp://[email protected]:5672//csd,1) has
recovered

Note: These exceptions can be ignored if they are not repeating after bringup.

2.18.12 Error: Could not fetch from host

Active Redundant node behavior in /tmp

CAUTION! Do not store anything in /tmp folder as it can cause switchover of redundant
VMs. In case if the /tmp of both the redundant nodes reaches full state, then both nodes will
switchover continuously.

If the /tmp of Active IO or Active OAM reaches a full state, then the switchover is triggered (thereby
restarting the keepalived.service) and written into the ha.log in the following path.

vi /var/log/ha/ha.log

The following is the sample format.

2019-11-08 19:51:38,193 CRITICAL ddesmha ['check', 'ACTIVE'] - FAIL

output=
ERROR: Could not fetch from host '192.168.3.21:8088' (result: 3).

HA mate Application Server is running hence switching over.

2019-11-08 19:51:38,314 CRITICAL ddesmha returned 1, restarting

2019-11-08 19:51:38,400 WARNING restart start
2019-11-08 19:51:38,406 INFO Exiting plugind
2019-11-08 19:51:39,445 INFO stopped keepalived

id9YZ-09148-MT11-PCZZA © 2020 Nokia 170

2.0
Monitoring and Troubleshooting Guide Troubleshooting

2019-11-08 19:51:40,748 WARNING restart start

2019-11-08 19:51:40,768 INFO stopped keepalived
2019-11-08 19:51:41,450 WARNING setting role to OOS
2019-11-08 19:51:41,452 INFO running notify ...
2019-11-08 19:51:41,978 INFO ha-role ['notify', 'OOS'] - OK
2019-11-08 19:51:41,979 INFO ha_sdc ['notify', 'OOS'] - OK
2019-11-08 19:51:41,979 INFO ha-vip ['notify', 'OOS'] - OK
2019-11-08 19:51:41,979 INFO ddesmha ['notify', 'OOS'] - OK
2019-11-08 19:51:42,770 WARNING setting role to OOS
2019-11-08 19:51:42,771 INFO running notify ...
2019-11-08 19:51:43,982 INFO ha-role ['notify', 'OOS'] - OK
2019-11-08 19:51:43,983 INFO ha_sdc ['notify', 'OOS'] - OK
2019-11-08 19:51:43,983 INFO ha-vip ['notify', 'OOS'] – OK
2019-11-08 19:51:43,983 INFO ddesmha ['notify', 'OOS'] - OK
2019-11-08 19:51:54,084 INFO start keepalived
2019-11-08 19:51:54,187 WARNING fail to acquire lock, ignore
check.
2019-11-08 19:51:54,198 WARNING setting role to STANDBY
2019-11-08 19:51:54,198 INFO calling plugins
2019-11-08 19:51:54,273 INFO ha-role ['notify', 'STANDBY'] - OK
2019-11-08 19:51:54,274 INFO ha_sdc ['notify', 'STANDBY'] - OK
2019-11-08 19:51:54,274 INFO ha-vip ['notify', 'STANDBY'] - OK
2019-11-08 19:51:54,274 INFO ddesmha ['notify', 'STANDBY'] - OK
2019-11-08 19:51:56,084 INFO start keepalived
2019-11-08 19:52:04,224 INFO Starting plugind
2019-11-08 19:52:04,540 ERROR Failed to generate plist
2019-11-08 19:52:14,210 INFO Starting plugind
2019-11-08 19:52:14,330 ERROR Failed to generate plist
2019-11-08 19:52:24,215 INFO Starting plugind
2019-11-08 19:52:24,342 ERROR Failed to generate plist
2019-11-08 19:52:34,213 INFO Starting plugind
2019-11-08 19:52:34,336 ERROR Failed to generate plist
2019-11-08 19:52:44,212 INFO Starting plugind
2019-11-08 19:52:46,356 ERROR Failed to generate plist
2019-11-08 19:52:54,219 INFO Starting plugind
2019-11-08 19:52:54,342 ERROR Failed to generate plist
2019-11-08 19:53:04,222 INFO Starting plugind

2.18.13 Error: Error configuring sctp kernel paramaters

If the /tmp of APP VM reaches a full state, and if the NOKIAasr service restarts due to any reason,
then the sctp kernel paramaters will not be set.

The execution of the following command displays the following error message as shown in the sample
format:

systemctl status NOKIAasr

id9YZ-09148-MT11-PCZZA © 2020 Nokia 171

2.0
Monitoring and Troubleshooting Guide Troubleshooting

The following is the sample format.

cannot create temp file for here-document: No space left on device

ERROR: Error configuring sctp kernel paramaters
Dec 12 15:08:31 DDEbvnf82-ddeapp-0 ddecontrol.sh[14721]: /opt/CSD_20_
0_I149/sbin/DDE_sctpsysctl: line 36: cannot create temp file for here-
document: No space left on device
Dec 12 15:08:31 DDEbvnf82-ddeapp-0 ddecontrol.sh[14721]: ERROR: Error
configuring sctp kernel paramaters.
Dec 12 15:08:31 DDEbvnf82-ddeapp-0 ddecontrol.sh[14721]: Starting DDE
App Server Thu Dec 12 15:08:31 CST 2019
Dec 12 15:08:31 DDEbvnf82-ddeapp-0 ddecontrol.sh[14786]: DDE Control Thu
Dec 12 15:08:31 CST 2019
Dec 12 15:08:31 DDEbvnf82-ddeapp-0 ddecontrol.sh[14786]: Waiting for app
server to start...
Dec 12 15:08:31 DDEbvnf82-ddeapp-0 ddecontrol.sh[14786]: App server is
started.

2.18.14 Error: "failed to get the VM UUID for node:Active-IO"

Whenever there is a peer disconnection event, the preceding errror message displayed in the
following path can be ignored.

/var/log/messages

Reason:

The objectName is defined as Active-IO for the Peer connection or disconnection alarm. If the
Peer connection, disconnection alarm or the SCTP Address Unavailable Alarm is raised with the
objectName as active IO VM name (For example, ddebvnf-ddeio-[0-1]) then post switchover
of IO VM, the auto clear of alarm fails due to mismatch in the objectName and the alarm must be
cleared manually. To prevent manual clearing of huge number of alarms, the objectName of the IO
VM is hard-coded as Active-IO. This enables the alarm to get auto cleared post switchover of IO
VMs.

Workaround: There is no functional impact and this error can be ignored.

2.18.15 Toggling in OAM nodes - zabbix-server

HA is not stable post heal rebuild of OAM in the fallback node

If the HA is not stable after heal rebuild of the OAM (in fallback node).

Check the ha.log in the following path.

/var/log/ha/ha.log

id9YZ-09148-MT11-PCZZA © 2020 Nokia 172

2.0
Monitoring and Troubleshooting Guide Troubleshooting

If HA is toggling due to zabbix-server, then remove the zabbix-server from the ha list for
recovering the setup using the following command.

ha rm zabbix-server

2.18.16 Call failure during data migration

The database failure may result in call failures as the datamigration takes place to distribute the
records to the available databases. Calls fail until the data migration is complete. This is an expected
behavior. The migration time depends on the amount of data and availability of databases.

Solution: Verify if the Datamigration is complete using /var/log/alarms in the CSD OAM node.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 173

2.0
Monitoring and Troubleshooting Guide Bare-metal CSD troubleshooting

3 Bare-metal CSD troubleshooting

This chapter describes the architecture of Bare-metal CSD system and how to manage logging
functions.

3.1 Managing CSD processes and log functions

Description

To Do

Check the process status

1. On active node, log in as ddeadmin by
executing the command su -ddeadmin.
2. Execute the command DDE_status to check
the process status.

Start processes Execute the command DDE_control start.

Stop processes Execute the command DDE_control stop.

Change the active to standby Execute the command DDE_control become-

slave.

Logging functions

Check diameter applications logs Execute the command vi /opt/tpa/logs/

DDESMApplication.log.

Take the tcpdump on Active node ew on specific

ports to troubleshoot further. Also, ensure that
the port is added in the pcap.

Change the log level Refer to Changing CSD logging levels section in
Logging functions on page 193.

Note: For information related to other logs, for

example, Provisioning.log, SubscriberTrace.log,
refer to Logging functions on page 193.

Check Aerospike logs Execute the command vi /opt/tpa/logs/

aero*.log.

Check installation logs Execute the commands vi /opt/tpa/logs/

install_DDE*.log and SPI.*log.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 174

2.0
Monitoring and Troubleshooting Guide Bare-metal CSD troubleshooting

To Do

Check for HA related logs Execute the command

vi /var/log/ha/ha.log

Service Manager GUI

To access the SM GUI Login URL.

For Cloud - https://<IP address of the SM>/

servicemanager/.

For Bare-Metal - https://<IP address of the

SM>:8143/servicemanager

List Diameter peers and routes

To list diameter peers and routes:

Execute the command listDiameterPeer.sh

Execute the command listDiameterRoutes.sh

Backup the DB

Execute the command [root@ddecompact1 bin]# ./DDE_dbBackup to backup the database.

The output files can be subsequently used by the DB restore tool.

Usage:

DDE_dbBackup -d <directory> [-r]

Where:

Without arguments, displays the help information

-d <directory> - Output directory for the backup files.

-h - Display the help information

-r - Remove any existing backup files in this directory before starting.

Restore the DB

Execute the command [root@sps181 bin]# ./DDE_dbRestore to restore the database from the
given backup files.

Usage:

DDE_dbRestore -d <directory>

id9YZ-09148-MT11-PCZZA © 2020 Nokia 175

2.0
Monitoring and Troubleshooting Guide Bare-metal CSD troubleshooting

Where:

Without arguments, displays the help information

-d <directory> - The directory name of the database backup

-h - Display the help information

Graceful switch over on redundant nodes

Follow the given steps to perform a graceful switch over.

1. Check the DDE_status on the node.

2. If the node is a master node, then gracefully perform switch over using following command:

DDE_control become-slave .

Configure snmpd target IP address

To change the snmpd target IP address or any other snmpd configuration change, edit the /etc/
snmp/snmpd.conf configuration on OAM VMs, in a following sequence.

1. Update the snmpd.conf configuration on the standby OAM.

2. Restart the snmpd on the standby OAM.
3. Perform a ha switch operation.
4. Repeat the steps 1 on page 176 - 2 on page 176 on other OAM.

Modifying Scrape interval and retention time post installation

The scrape interval and retention time after installation can be changed in the prometheus.yml in
the following path.

/opt/tpa/statistics/prometheus/

Change scrape_interval: for the required metrics.

In the /opt/tpa/statistics/prometheus/prometheus-server path, change RETENTION

parameter for the retention time.

3.2 Switchover issues due to Netmon

Description

Netmon utility is integrated to check the health of the bonding interfaces. Netmon performs PING
or ARP on the NETMON_MONITOR_IPS configured in the network interface files, and waits for
response. If netmon fails to receive response within NETMON_TIMEOUT time (2seconds), it will
trigger failover.

The netmon is configured in all the bonding interfaces mentioned in the sdc_conf's IPM tab. Netmon
configures the ifcfg files of bonding interfaces with the following options:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 176

2.0
Monitoring and Troubleshooting Guide Bare-metal CSD troubleshooting

NETMON_MONITOR_IPS: External IP Addresses are used for connectivity detection. Multiple IPs
can be provided as a comma-separated list. If multiple IPs are provided, only one needs to pass
connectivity testing to consider network as healthy network.

NETMON_CHECK_INTERVAL: NETMON_CHECK_INTERVAL is the number of seconds between

connectivity tests.

NETMON_CHECK_TIMEOUT: NETMON_CHECK_TIMEOUT is the number of seconds to allowed for

a reply from MONITOR IP.

NETMON_FAILURE_COUNT: NETMON_FAILURE_COUNT is the number of sequential failures

occurred before considering the network as unhealthy network.

For more resilience, users can add to monitor static IP of the mate node on the respective bonding
interface with a coma separated value. Along with this NETMON_FAILURE_COUNT can change such
that it is double the number of NETMON_MONITOR_IPS.

Following is an example of netmon configuration in ifcfg files after the preceding changes.

Where, 10.53.134.51 is the static IP on the bond0 of the mate node.

3.3 Troubleshooting RHEL load modules process

id9YZ-09148-MT11-PCZZA © 2020 Nokia 177

2.0
Monitoring and Troubleshooting Guide Bare-metal CSD troubleshooting

3.3.1 RHEL load modules process is inactive

Problem:

rhel-loadmodules process is inactive.

Solution:

To start the rhel-loadmodules process, perform the following steps:

1. Navigate to rhel-loadmodules.service from the following path:

vi /usr/lib/systemd/system/
2. Edit the following line:
Type=oneshot to Type=simple.

For example, as depicted in the following location:

3. Save the file.
4. Execute the following commands:
systemctl daemon-reload

systemctl restart rhel-loadmodules

3.4 Troubleshooting CALM CALM ETCD is Unavailable alarm

Problem

If the Network Eelement object is already added in NetAct and CSD is instantiated later then the
alarms are not visible in NetAct moinitor.

Solution

To overcome this issue, execute the following steps on the NetAct monitor:

1. Navigate to the respective CSD Network Element object:

Element management>Mediation Registration>De-Activate SNMPFM

2. Navigate to the respective CSD Network Element object:

Element management>Mediation Registration>Activate SNMPFM

3. Execute the following command on both the nodes.

systemctl restart calm-alma

Note: CALM ETCD is Unavailable alarm does not get cleared automatically. It must be
cleared manually from the etcd and NetAct.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 178

2.0
Monitoring and Troubleshooting Guide Alarms

4 Alarms

4.1 CSD alarms

Overview

Alarms are required for fault management of the system. Alarms are the way to indicate the operator
when something goes wrong in the system. For example, overload due to message queue, incoming
requests exceeding the throttling threshold for inbound, XDR link goes down between the geo-
redundant site. These are application-specific alarms, platform generates alarms when any node goes
down, unreachable or network error.

MIB files are seen in /opt/alarm

Example

local3.alert: Mar 6 12:02:18 NepalDDE-oame-1 calm-alma:{"severity":

"minor",
"objectName": "NepalDDE-ddeapp-0#null", "probableCause":
"communicationsProtocolError",
"additionalText": "ORIGINHOST=diahost.diarealm\
nPROTOCOL=unknown;51002;DDE-DIAMETER#;Peer Connection Detected",
"ackStatus": false, "time": "2018-03-06 12:02:18",
"VMUUID": "e4fc1f7e-1c49-4d2f-b434-59cbf81fa984 LeaseId
2015468559486589523",
"specificProblem": "PeerConnectAlarm", "VMName": "NepalDDE-ddeapp-0",
"alarmType": "communicationAlarm"}

The following table lists the CSD application alarm details that are raised by CSD application when a
specific event occurs and notifies to user.

x733 specific x733 x733 probable x733 3gg Auto

x733 event type Severity Action to be taken Notes Additional text
problem for trap objectname cause code clear

SystemOverload $HOSTNAME processingAlarm performance NA Minor, No It is a notification This alarm is raised $OVERLOAD
Major, used to notify when the host
Degraded COMPONENETS;
customer when that triggers this
Critical
the system gets alarm becomes $USAGESTATE;
overloaded. No overloaded. The $ALARMCODE
action is needed alarm describes
(53001);
if expected, in its payload
otherwise check the the system $ALARMGROUP;
message counts and components that $SPECIFIC
components which is are currently
PROBLEMDETAIL
getting overloaded. overloaded but
does not explicitly
state which of
those components
are critically
overloaded.

SystemOverload $HOSTNAME processingAlarm performance NA Critical No It is a notification This alarm is raised $OVERLOAD
used to notify when the host that
COMPONENETS;

id9YZ-09148-MT11-PCZZA © 2020 Nokia 179

2.0
Monitoring and Troubleshooting Guide Alarms

x733 specific x733 x733 probable x733 3gg Auto

x733 event type Severity Action to be taken Notes Additional text
problem for trap objectname cause code clear

ResourceCritical Degraded customer when the triggers this alarm $USAGESTATE;

system crossed the becomes critically
$ALARM
threshold level. No overloaded. The
CODE(53002);
action is needed if it alarm describes
expected, otherwise in its payload $ALARMGROUP;

need to reduce the the system $SPECIFIC

message count components that
PROBLEMDETAIL
and check the are currently
components which is overloaded but
getting overloaded. does not explicitly
state which of
those components
are critically
overloaded.

SystemOut $HOSTNAME processingAlarm outOfService NA Major No It is a notification This alarm is raised $ALARM CODE
used to notify when the system
ofService (53003);
customer when that triggers it is
the system is not brought out of $ALARMGROUP;
reachable and Out of service. $SPECIFIC
Service. As an action
PROBLEMDETAIL
user need to restart
the node.

XdrClusterUn $HOSTNAME processingAlarm application NA Major No It is a notification This alarm is raised $SPECIFICCLUSTER;
used to notify when the host
reachableAlarm Subsystem $ALARMCODE
customer when other that triggers this
Failure site went down or alarm detects that (53004);
not reachable. As an XDR cluster is $ALARMGROUP;
an action, customer unreachable.
$SPECIFIC
need to login to
respective node and PROBLEMDETAIL
restart the node.

XdrTimelagAlarm $HOSTNAME processingAlarm application NA Major No It is a notification This alarm is raised $ALARM CODE
used to notify the when the host that
Subsystem (53005);
user that the host triggers this alarm
Failure detects that XDR detects that an $ALARMGROUP;
cluster has too much XDR cluster has $SPECIFIC
lag time. As an too much time lag.
PROBLEMDETAIL
action user need to
check the logs for
node failure.

XdrOutstanding $HOSTNAME processingAlarm application NA Minor No It is a notification This alarm is raised $ALARM CODE
used to notify the when the host
RecordsAlarm Subsystem (53006);
user that the host that triggers this
Failure detects that XDR alarm detects $ALARMGROUP;
cluster has too many that an XDR $SPECIFIC
outstanding records. cluster has too
PROBLEMDETAIL
many outstanding
records.

DataMigration $HOSTNAME processingAlarm unspecifiedReason NA Minor No It is a notification This alarm is raised $ALARM CODE
used to notify when the host that
InProgressAlarm (53007);
customer when the triggers it detects
data migration starts. that data migration $ALARMGROUP;
No action needed if is in progress. $SPECIFIC
it is expected.
PROBLEMDETAIL

XdrDigestlog $HOSTNAME processingAlarm application NA Major No It is a notification This alarm is raised $ALARM CODE
used to notify that when Digestlog
ThresholdAlarm Subsystem (53008);
XDR digestlog has reached
Failure has reached the threshold $ALARMGROUP;
the thrashold percentage $SPECIFIC
percentage. As a
PROBLEMDETAIL
action user needs to
clear the XDR digest
log.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 180

2.0
Monitoring and Troubleshooting Guide Alarms

x733 specific x733 x733 probable x733 3gg Auto

x733 event type Severity Action to be taken Notes Additional text
problem for trap objectname cause code clear

PeerWhiteList $HOSTNAME communication communications NA Minor Yes It is a notification This alarm is raised $ALARM CODE
used to notify when there are
ThresholdAlarm Alarm ProtocolError (51001);
that there is too too many connect
many connection attempts from $ALARMGROUP;
attaempts from unauthorized peers $SPECIFIC
unauthorized peers.
PROBLEMDETAIL
As an action check
contents of the
countable event
"Rejected Peer
Connections" in the
database.

Peer Disconnect $HOSTNAME communication communications NA Major Yes It is a notification This alarm is raised $ORIGINHOST;
used to notify when the diameter
detected Alarm ProtocolError $PROTOCOL;
operator when peer disconnect
the peer gets event is detected $ALARM CODE
disconnected. See and cleared when (51003);
debug logs for more the diameter peer
$ALARMGROUP;
information. connection event
detected with the $SPECIFIC
same diameter PROBLEMDETAIL
peer. Note: If the
diameter peer is
disconnected and
does not reconnect,
then the alarm
must be cleared
manually.

SCTPAddress $HOSTNAME communication communications NA Minor Yes It is a notification This alarm is raised $ALARM
used to notify when the SCTP CODE(51004);
AvailableAlarm Alarm ProtocolError
whenever SCTP address available
$ALARMGROUP;
address available event is detected.
event detected. See $SPECIFIC
debug logs for more PROBLEMDETAIL
information.

SCTPAddress $HOSTNAME communication communications NA Minor Yes It is a notification This alarm is $ALARM CODE
used to notify raised when the
UnavailableAlarm Alarm ProtocolError (51005);
whenever SCTP SCTP address
address unavailable unavailable event is $ALARMGROUP;
event detected. See detected. $SPECIFIC
debug logs for more
PROBLEMDETAIL
information.

RouteFailure $HOSTNAME communication communications NA Minor Yes It is a notification This alarm is raised $ALARM CODE
used to notify the when there are
ThresholdAlarm Alarm ProtocolError (51006);
user that there are Too many routing
too many routing failures for this $ALARMGROUP;
failure for route route config. $SPECIFIC
config. As an action
PROBLEMDETAIL
user need to check
the contents of the
countable event
"Failed Routing
Attempts" in the
database.

SS7Failure $HOSTNAME communication communications NA Major No It is a notification This alarm is raised $ALARM
used to notify the when a SS7 SCTP CODE(55001);
Alarm Alarm ProtocolError
user that the SS7 connection failure
$ALARMGROUP;
sctp connection occurs.
failure occur. $SPECIFIC

PROBLEMDETAIL

MsgValidation $HOSTNAME processingAlarm authentication NA Minor Yes It is a notification This alarm is raised $ALARM CODE
used to notify the when a diameter
Alarm Failure (52001);
user that a diameter message validation
message validation error occurs. $ALARMGROUP;

id9YZ-09148-MT11-PCZZA © 2020 Nokia 181

2.0
Monitoring and Troubleshooting Guide Alarms

x733 specific x733 x733 probable x733 3gg Auto

x733 event type Severity Action to be taken Notes Additional text
problem for trap objectname cause code clear

error occur. As an $SPECIFIC

action user needs to
PROBLEMDETAIL
check the message
format and AVP's.

SLFFailureAlarm $HOSTNAME processingAlarm authentication NA Minor Yes It is a notification This alarm is raised $ALARM CODE
used to notify the when a diameter
Failure (52002);
user that there is SLF lookup failure
a diameter SLF occurs. $ALARMGROUP;
lookup failure occur. $SPECIFIC
As an action check
PROBLEMDETAIL
contents of the
countable event log
in the database.

ThrottlingAlarm $HOSTNAME processingAlarm congestion NA Minor No It is a notification This alarm is $ALARM CODE
used to notify raised when rate-
(52003);
customer when a limiting has been
rate-limiting has applied to diameter $ALARMGROUP;
been applied to the messages. $SPECIFIC
Diameter message.
PROBLEMDETAIL
As an action
evaluate configured
rate-limits and the
number of deployed
processing server
instances.

Both the OAM $HOSTNAME QualityOfService application NA Critical Yes Verify the HA This alarm is raised SPECIFIC
status on both the when both the
VMs are HA Subsystem PROBLEM
redundant VMs using OAM VMs are in
Active Or Standby Failure the command ha same HA states DETAIL=
status. If the ha (Active, Standby, Both the OAM VMs are
status is ACTIVE OOS). HA Active or Stand-by:
on both the VMs, 51999
then verify if the
DDE-APPLICATION#;
VRRP connectivity is
Both the OAM VMs are
broken between both
HA Active or Stand-by.
the VMs. This could
be a communication
failure between the
VMs.

If ha status is
OOS or STANDBY on
both VMs, verify the
logs in /var/log/
ha on the VMs for
possible indications.

Execute the
command ha
enable verbose
on a VM, to view the
detailed logging in /
var/log/ha.

Execute the
command ha
disable verbose
on the VM to disable
the ha traces.

To recover VMs from

split-brain, after the
network or external
factors which lead to
issue are corrected,
perform HA restart
on both the VMs
using the command
ha restart on
both the VMs.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 182

2.0
Monitoring and Troubleshooting Guide Alarms

x733 specific x733 x733 probable x733 3gg Auto

x733 event type Severity Action to be taken Notes Additional text
problem for trap objectname cause code clear

Note: This alarm is

raised and cleared
during instantiation.
This is an expected
behavior.

Both the IO $HOSTNAME QualityOfService application NA Critical Yes Verify the HA This alarm is raised SPECIFIC
status on both the when both the IO
VMs are HA Subsystem PROBLEM
redundant VMs using VMs are in same
Active Or Standby Failure the command ha HA states (Active, DETAIL=
status. If the ha Standby, OOS). Both the Diameter IO
status is ACTIVE VMs are HA Active or
on both the VMs, Stand-by: 51999
then verify if the
DDE-APPLICATION#;
VRRP connectivity is
Both the Diameter IO
broken between both
VMs are HA Active or
the VMs. This could
Stand-by.
be a communication
failure between the
VMs.

If ha status is
OOS or STANDBY on
both VMs, verify the
logs in /var/log/
ha on the VMs for
possible indications.

Execute the
command ha
enable verbose
on a VM, to view the
detailed logging in /
var/log/ha.

Execute the
command ha
disable verbose
on the VM to disable
the ha traces.

To recover VMs from

split-brain, after the
network or external
factors which lead to
issue are corrected,
perform HA restart
on both the VMs
using the command
ha restart on
both the VMs.

Note: This alarm is

raised and cleared
during instantiation.
This is an expected
behavior.

Both the SS7 IO $HOSTNAME QualityOfService application NA Critical Yes Verify the HA This alarm is raised SPECIFIC
status on both the when both the
VMs are HA Subsystem PROBLEM
redundant VMs using SS7 IO VMs are
Active Or Standby Failure the command ha in same HA states DETAIL=
status. If the ha (Active, Standby, Both the SS7 IO VMs
status is ACTIVE OOS). are HA Active or
on both the VMs, Stand-by: 51999
then verify if the
DDE-APPLICATION#;
VRRP connectivity is
Both the SS7 IO VMs
broken between both
are HA Active or
the VMs. This could
Stand-by.
be a communication
failure between the
VMs.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 183

2.0
Monitoring and Troubleshooting Guide Alarms

x733 specific x733 x733 probable x733 3gg Auto

x733 event type Severity Action to be taken Notes Additional text
problem for trap objectname cause code clear

If ha status is
OOS or STANDBY on
both VMs, verify the
logs in /var/log/
ha on the VMs for
possible indications.

Execute the
command ha
enable verbose
on a VM, to view the
detailed logging in /
var/log/ha.

Execute the
command ha
disable verbose
on the VM to disable
the ha traces.

To recover VMs from

split-brain, after the
network or external
factors which lead to
issue are corrected,
perform HA restart
on both the VMs
using the command
ha restart on
both the VMs.

Note: This alarm is

raised and cleared
during instantiation.
This is an expected
behavior.

RabbitMQ $HOSTNAME communication communications Not Major No Check in the This alarm is raised 53011: DDE_
DDESMApplication. when the APP SYSTEM#
Consume Alarm Protocol Appli
log if the or IO nodes are
RabitMQ Event
Failed Error cable rabbitmq unable to consume
Consumption
notifications about the RabbitMQ
add, update, delete notifications. Failed

of diameter peer, For example,

KPI profile and this behavior
Jaeger tracing are can be observed
consumed by IO or when there is
APP nodes and then no connectivity
manually clear the between APP to
alarm. databases.

The
DDESMApplication.
log indicates
whether the
notification is
consumed or not.
Possible failures
should be evident in
the log.

For example,
no database
connectivity.

correct the problem

Example: restore
the database
connectivity.

After correcting the

problem, retry the
action from the SM.
Upon successful

id9YZ-09148-MT11-PCZZA © 2020 Nokia 184

2.0
Monitoring and Troubleshooting Guide Alarms

x733 specific x733 x733 probable x733 3gg Auto

x733 event type Severity Action to be taken Notes Additional text
problem for trap objectname cause code clear

reception of the
change in App or
IO nodes, clear the
alarm manually.

Table 48: CSD application alarm details

Note: The Peer Disconnect detected alarm is not raised during any of the abrupt switchover
scenarios (such as reboot, power off, not responding, and so on) on the diameter IO
node. This alarm can be observed only during the graceful switchover of CSD IO such as
DDE_control become slave.

Note: If the Mux connection is not stable due to internal network issues or due to repeated
Block or Unblock of Mux communication ports, then the Peer Connections Mismatch with IO
alarm may be raised. This alarm is automatically cleared within 30 seconds.

Note: Post upgrade to CSD 20.2 Release, the raised alarms which were introduced after
CSD 19.2 Release (for VMware) or the alarms introduced after CSD 19.5 Release (for
OpenStack), are not cleared after falling back to the base release. These alarms must be
cleared manually (Forexample: DBClusterFailure, TDR Collection failed and so on).

Attention: For successful TDR collection, passwordless authentication must be enabled for
ddeadmin user.

4.2 SM Application alarms

SM alarms generated if ME goes offline

Following are the alarms that SM application generates if the Managed Element (ME) is unavailable
and cleared when the ME becomes available again:

local3.alert: Mar 10 11:48:19 smbvnf-oame-1 calm-alma:

{"severity": "major", "objectName": "smbvnf-oame-1#DDE",
"probableCause": "communicationsSubsystemFailure",
"additionalText": "MANAGEDELEMENTNAME=DDE-5952;
50001;DDE-SERVICEMANAGER#;Managed Element Communication Alarm: DDE-5952
is unavailable",
"ackStatus": false, "time": "2018-03-10 11:48:19",
"VMUUID": "5bccd951-38c8-442e-9c5f-a1da4eeb1bc8",
"specificProblem": "MEUnavailableAlarm",
"VMName": "smbvnf-oame-1",
"alarmType": "communicationAlarm"}

The following table lists the SM application alarm details that are raised by the SM application.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 185

2.0
Monitoring and Troubleshooting Guide Alarms

x733 specific
x733 x733 event x733 probable Action to be
problem for x733 3gg code Severity Auto clear Notes Additional text
objectname type cause taken
trap

SystemOut $HOSTNAME processingAlarm outOfService Not applicable. Major No It is a This alarm is $ALARM
notification raised when CODE
ofService
which is used to the system that
(53003);
notify customer triggers it is
when the brought out of $ALARM
system is not service. GROUP;
reachable and
$SPECIFIC
Out of Service.
As an action PROBLEM
user need to DETAIL
restart the
node.

XdrCluster $HOSTNAME processingAlarm application Not applicable. Major No It is a This alarm is $SPECIFIC
notification raised when
Unreachable Subsystem CLUSTER;
which is used to the host that
Alarm Failure notify customer triggers this $ALARM
when other site alarm detects CODE
goes down or is that an XDR (53004);
not reachable. cluster is
$ALARM
As an action, unreachable.
customer needs GROUP;
to login to $SPECIFIC
respective node
PROBLEM
and restart the
node. DETAIL

XdrTimelag $HOSTNAME processingAlarm application Not applicable. Major No It is a This alarm is $ALARM
notification raised when CODE(53005);
Alarm Subsystem
which is used the host that
$ALARM
Failure to notify the triggers this
user that the alarm detects GROUP;
host detects that an XDR $SPECIFIC
that the XDR cluster has too
PROBLEM
cluster has too much time lag.
much lag time. DETAIL
As an action,
user needs to
check the logs
for node failure.

Xdr $HOSTNAME processingAlarm application Not applicable. Minor No It is a This alarm is $ALARM
notification raised when CODE(53006);
Outstanding Subsystem
which is used to the host that
$ALARM
Records Failure notify the user triggers this
that the host alarm detects GROUP;
Alarm
detects that that an XDR $SPECIFIC
XDR cluster cluster has
PROBLEM
has too many too many
outstanding outstanding DETAIL
records. records.

Data $HOSTNAME processingAlarm unspecified Not applicable. Minor No It is a This alarm is $ALARM
notification raised when CODE(53007);
Migration Reason
which is used the host that
$ALARM
InProgress to notify the triggers it
customer detects that GROUP;
Alarm
when the data data migration $SPECIFIC
migration starts. is in progress.
PROBLEM
No action is
needed if it is DETAIL
expected.

Xdr $HOSTNAME processingAlarm application Not applicable. Major No It is a This alarm is $ALARM
notification raised when CODE
Digestlog Subsystem
which is used Digestlog
(53008);
Threshold Failure to notify that the has reached
XDR digestlog the threshold $ALARM
Alarm
has reached percentage GROUP;

id9YZ-09148-MT11-PCZZA © 2020 Nokia 186

2.0
Monitoring and Troubleshooting Guide Alarms

x733 specific
x733 x733 event x733 probable Action to be
problem for x733 3gg code Severity Auto clear Notes Additional text
objectname type cause taken
trap

the threshold $SPECIFIC

percentage. As
PROBLEM
an action user
needs to clear DETAIL

the XDR digest

log.

ME $HOSTNAME communication communications Not applicable. Major No It is a This alarm $MANAGED

notification is related to
Unavailable Alarm Subsystem ELEMENTNAME;
which is used managed
Alarm Failure to notify the element $ALARM
customer when communication CODE(50001);
the ME is alarm. $ALARM
unavailable or
GROUP;
not reachable.
As an action, $SPECIFIC
user needs PROBLEM
to check the
DETAIL
configuration
and CSD node
status.

Both the OAM $HOSTNAME QualityOfService application Not applicable. Critical Yes Verify the HA This alarm is SPECIFIC
status on both raised when
VMs are HA Subsystem PROBLEM
the redundant both the OAM
Active Or Failure VMs using the VMs are in DETAIL=
Standby command ha same HA Both the OAM
status. If the states (Active, VMs are HA
ha status Standby, OOS). Active or Stand-
is ACTIVE on by: 51999
both the VMs,
DDE-
then verify
APPLICATION#;
if the VRRP
Both the OAM
connectivity is
VMs are HA
broken between
Active or Stand-
both the VMs.
by.
This could be a
communication
failure between
the VMs.

If ha status
is OOS or
STANDBY on
both VMs,
verify the logs
in /var/log/
ha on the VMs
for possible
indications.

Execute the
command
ha enable
verbose on a
VM, to view the
detailed logging
in /var/log/
ha.

Execute the
command
ha disable
verbose on
the VM to
disable the ha
traces.

To recover
VMs from split-
brain, after

id9YZ-09148-MT11-PCZZA © 2020 Nokia 187

2.0
Monitoring and Troubleshooting Guide Alarms

x733 specific
x733 x733 event x733 probable Action to be
problem for x733 3gg code Severity Auto clear Notes Additional text
objectname type cause taken
trap

the network
or external
factors which
lead to issue
are corrected,
perform HA
restart on both
the VMs using
the command
ha restart
on both the
VMs.

Note: This
alarm is
raised and
cleared during
instantiation.
This is an
expected
behavior.

Table 49: SM application alarm details

4.3 Platform alarm details for CSD and SM

Platform alarm details for CSD and SM

x733 specific x733 probable Threshold Default Threshold Actions to be

x733 objectname x733 event type Severity Auto clear Additional text
problem for trap cause (reason) Value taken

HA Master HA master processing application Major Not applicable. Not applicable. Indicates the Yes Node transitioning
Recovering keepalive to ACTIV.
<hostname> ErrorAlarm Subsystem
component that
#HA Failure has experienced
a failover. This
is triggered once
the new Master
becomes available.
No further action is
necessary.

Excessive <hostname> QualityOfService authentication Minor Exceeded the EXCESSIVE It is automatically Yes Success/Failure
authentication threshold for AUTHENTICATION unblocked after indication,
#SECURITY Failure
failures the number of FAILURES LOG- the time period Login ID, Event
consecutive IN ACCESS IS specified by the Description,
login failures TEMPORARILY LOCKTIMEOUT Source IP Address
(MAX_LOGIN_ DISABLED parameter.
FAILURES) FOR ACCOUNT
tester[OS] default
value for MAX_
LOGIN_FAILURES
is 6 default
value for the
LOCKTIMEOUT
parameter is 5
minutes

DORMANT UNIX <hostname> QualityOfService Theshold Minor Warning that an ACCOUNT Not applicable. No Success/Failure
ACCOUNT account is about to DORMANT FOR indication,
#SECURITY Crossed
be locked due to AT LEAST b Login ID, Event
lack of use DAYS NO ACTION Description,
TAKEN. Source IP Address

Default b = 45

DORMANT <hostname> QualityOfService Theshold Minor An account is ACCOUNT Not applicable. No Success/Failure
UNIX ACCOUNT locked due to lack DORMANT FOR indication,
#SECURITY Crossed
LOCKED of use AT LEAST b DAYS Login ID, Event

id9YZ-09148-MT11-PCZZA © 2020 Nokia 188

2.0
Monitoring and Troubleshooting Guide Alarms

x733 specific x733 probable Threshold Default Threshold Actions to be

x733 objectname x733 event type Severity Auto clear Additional text
problem for trap cause (reason) Value taken

ACCOUNT HAS Description,

BEEN LOCKED. Source IP Address

Default b = 60

SECURITY <hostname> QualityOfService Theshold Minor Not applicable. Not applicable. No Success/Failure
AUDIT LOGGING indication,
#SECURITY Crossed
STARTED Login ID, Event
Description,
Source IP Address

SECURITY <hostname> QualityOfService Theshold Minor Not applicable. Not applicable. No Success/Failure
AUDIT LOGGING indication,
#SECURITY Crossed
STOPPED Login ID, Event
Description,
Source IP Address

SECURITY LOG <hostname> Processing Error Configuration Minor Not applicable. Not applicable. No Success/Failure
MODIFICATION indication,
#SECURITY or
Login ID, Event
Customization Description,
Error Source IP Address

DATE OR TIME <hostname> Processing Error Configuration Minor Not applicable. Not applicable. No Success/Failure
MODIFICATION indication,
#SECURITY or
Login ID, Event
Customization Description,
Error Source IP Address

FILE <hostname> Processing Error Configuration Minor Not applicable. Not applicable. No Success/Failure
MODIFICATION indication,
#SECURITY or
Login ID, Event
Customization Description,
Error Source IP Address

USER ACCOUNT <hostname> Processing Error Configuration Minor Not applicable. Not applicable. No Success/Failure
MODIFICATION indication,
#SECURITY or
Login ID, Event
Customization Description,
Error Source IP Address

ETCD Cluster oame qualityOf outofService Clear No action needed. Refer to the text
Service in ETCD cluster
#ETCD Service Service
Degradation health check.
Alarm

ETCD Cluster oame qualityOf outofService Major Manually check the No Refer to the text
Service ETCD status. in ETCD cluster
#ETCD Service Service
Degradation health check.
Alarm

ETCD Cluster oame qualityOf outofService Critical Manually check the No Refer to the text
Service ETCD status. in ETCD cluster
#ETCD Service Service
Degradation health check.
Alarm

NTP Server HOSTNAME Communications remoteNode Major Yes Clock Not

Connection Down Synchronized
of Node reporting Transmission
the error
Error

NTP Server HOSTNAME Communications remoteNode Major Yes Clock state

Connection Down indeterminate
of Node reporting Transmission
the error
Error

NTP Server HOSTNAME Communications remoteNode Clear Yes <CHRONY | NTP>

Connection Down
of Node reporting Transmission service starting
the error
Error

NTP Server HOSTNAME Communications remoteNode Clear Yes <CHRONY | NTP>

Connection Down
of Node reporting Transmission service stopped
the error
Error

NTP Server HOSTNAME of Communications remoteNode Clear Yes Clock

Connection Down Node reporting the Synchronized
Transmission
error
Error

Process is down: <hostname> <hostname> threshold Critical Notification to user Yes On

NOKIAasdb. that process gets
Crossed $SOURCEIP,
service stop. As an action
Process is down:
user needs to
restart the process. $PROCESSIP

id9YZ-09148-MT11-PCZZA © 2020 Nokia 189

2.0
Monitoring and Troubleshooting Guide Alarms

x733 specific x733 probable Threshold Default Threshold Actions to be

x733 objectname x733 event type Severity Auto clear Additional text
problem for trap cause (reason) Value taken

Process is down: <hostname> <hostname> threshold Critical Notification to user Yes On

NOKIAasr.service that process gets
Crossed $SOURCEIP,
stop. As an action
Process is down:
user needs to
restart the process. $PROCESSIP

Process is down: <hostname> <hostname> threshold Critical Notification to user Yes On

httpd.service that process gets
Crossed $SOURCEIP,
stop. As an action
Process is down:
user needs to
restart the process. $PROCESSIP

Process is down: <hostname> <hostname> threshold Critical Notification to user Yes On

gmetad.service that process gets
Crossed $SOURCEIP,
stop. As an action
Process is down:
user needs to
restart the process. $PROCESSIP

Process is down: <hostname> <hostname> threshold Critical Notification to user Yes On

gmond.service that process gets
Crossed $SOURCEIP,
stop. As an action
Process is down:
user needs to
restart the process. $PROCESSIP

Table 50: Platform alarm details for CSD

x733 specific x733 probable Threshold Default Threshold Actions to be

x733 objectname x733 alarm type Severity Auto clear Additional text
problem for trap cause (reason) Value taken

HA Master HA master processingErrorAlarm application Major Not applicable. Not applicable. Indicates the Yes Node transitioning
Recovering keepalive to ACTIV.
<hostname> Subsystem
component that
#HA Failure has experienced
a failover. This
is triggered once
the new Master
becomes available.
No further action is
necessary.

Excessive <hostname> QualityOfService authenticationFailure Minor Exceeded the EXCESSIVE It is automatically Yes Success/Failure
authentication threshold for AUTHENTICATION unblocked after indication,
#SECURITY
failures the number of FAILURES LOG- the time period Login ID, Event
consecutive IN ACCESS IS specified by the Description,
login failures TEMPORARILY LOCKTIMEOUT Source IP Address
(MAX_LOGIN_ DISABLED parameter.
FAILURES) FOR ACCOUNT
tester[OS] default
value for MAX_
LOGIN_FAILURES
is 6 default
value for the
LOCKTIMEOUT
parameter is 5
minutes

DORMANT UNIX <hostname> QualityOfService ThesholdCrossed Minor Warning that an ACCOUNT Not applicable. No Success/Failure
ACCOUNT account is about to DORMANT FOR indication,
#SECURITY
be locked due to AT LEAST b Login ID, Event
lack of use DAYS NO ACTION Description,
TAKEN. Default b Source IP Address
= 45

DORMANT <hostname> QualityOfService ThesholdCrossed Minor An account is ACCOUNT Not applicable. No Success/Failure
UNIX ACCOUNT locked due to lack DORMANT FOR indication,
#SECURITY
LOCKED of use AT LEAST b DAYS Login ID, Event
ACCOUNT HAS Description,
BEEN LOCKED. Source IP Address
Default b = 60

SECURITY <hostname> QualityOfService ThesholdCrossed Minor Not applicable. Not applicable. No Success/Failure
AUDIT LOGGING indication,
#SECURITY
STARTED Login ID, Event
Description,
Source IP Address

SECURITY <hostname> QualityOfService ThesholdCrossed Minor Not applicable. Not applicable. No Success/Failure
AUDIT LOGGING indication,
#SECURITY
STOPPED Login ID, Event
Description,
Source IP Address

id9YZ-09148-MT11-PCZZA © 2020 Nokia 190

2.0
Monitoring and Troubleshooting Guide Alarms

x733 specific x733 probable Threshold Default Threshold Actions to be

x733 objectname x733 alarm type Severity Auto clear Additional text
problem for trap cause (reason) Value taken

SECURITY LOG <hostname> Processing Error Configuration Minor Not applicable. Not applicable. No Success/Failure
MODIFICATION indication,
#SECURITY or
Login ID, Event
Customization Description,
Error Source IP Address

DATE OR TIME <hostname> Processing Error Configuration Minor Not applicable. Not applicable. No Success/Failure
MODIFICATION indication,
#SECURITY or
Login ID, Event
Customization Description,
Error Source IP Address

FILE <hostname> Processing Error Configuration Minor Not applicable. Not applicable. No Success/Failure
MODIFICATION indication,
#SECURITY or
Login ID, Event
Customization Description,
Error Source IP Address

USER ACCOUNT <hostname> Processing Error Configuration Minor Not applicable. Not applicable. No Success/Failure
MODIFICATION indication,
#SECURITY or
Login ID, Event
Customization Description,
Error Source IP Address

ETCD Cluster oame qualityOfService outofService Clear No action needed. Refer to the text
Service in ETCD cluster
#ETCD Alarm
Degradation health check.
Service

ETCD Cluster oame qualityOfService outofService Major Manually check the Refer to the text
Service ETCD status. in ETCD cluster
#ETCD Alarm
Degradation health check.
Service

ETCD Cluster oame qualityOfService outofService Critical Manually check the Refer to the text
Service ETCD status. in ETCD cluster
#ETCD Alarm
Degradation health check.
Service

NTP Server HOSTNAME of Communications remote Major Clock Not

Connection Down Node reporting the Synchronized
Node
error
Transmission

Error

NTP Server HOSTNAME of Communications remote Major Clock state

Connection Down Node reporting the indeterminate
Node
error
Transmission

Error

NTP Server HOSTNAME of Communications remote Clear <CHRONY | NTP>

Connection Down Node reporting the
Node service starting
error
Transmission

Error

NTP Server HOSTNAME of Communications remote Clear <CHRONY | NTP>

Connection Down Node reporting the
Node service stopped
error
Transmission

Error

NTP Server HOSTNAME of Communications remote Clear Clock

Connection Down Node reporting the Synchronized
Node
error
Transmission

Error

Process is down: <host <host thresholdCrossed Critical Notification to user Yes On

NOKIAasdb. that process gets
name> name> $SOURCEIP,
service stop. As an action
Process is down:
user needs to
restart the process. $PROCESSIP

Process is down: <host <host thresholdCrossed Critical Notification to user Yes On

NOKIAasr.service that process gets
name> name> $SOURCEIP,
stop. As an action
Process is down:
user needs to
restart the process. $PROCESSIP

Process is down: <host <host thresholdCrossed Critical Notification to user Yes On

httpd.service that process gets
name> name> $SOURCEIP,
stop. As an action
Process is down:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 191

2.0
Monitoring and Troubleshooting Guide Alarms

x733 specific x733 probable Threshold Default Threshold Actions to be

x733 objectname x733 alarm type Severity Auto clear Additional text
problem for trap cause (reason) Value taken

user needs to $PROCESSIP

restart the process.

Process is down: <host <host thresholdCrossed Critical Notification to user Yes On

gmetad.service that process gets
name> name> $SOURCEIP,
stop. As an action
Process is down:
user needs to
restart the process. $PROCESSIP

Process is down: <host <host thresholdCrossed Critical Notification to user Yes On

gmond.service that process gets
name> name> $SOURCEIP,
stop. As an action
Process is down:
user needs to
restart the process. $PROCESSIP

Table 51: Platform alarm details for SM

Note: During switchover of OAM nodes, the alarms (raise or clear) may not be sent from
CSD to NetAct. This depends on the duration for which Virtual IP address is unplumbed and
plumbed on the VM by OS network utilities.

In this case, alarms must be cleared manually post successful execution of the following
operations on active OAM VM:

• Rebuild
• Reboot
• Switchover

Clearing the alarm at SNMP Manager

lObjectName, lSpecificProblem, lAlarmProbableCause are mapped to alAlarmIndex.

snmpset at SNMP manager uses alAlarmIndex to clear an alarm.

SNMP manager uses any one variable between alNotificationIdentifier or alAlarmIndex

for clearing the alarm as it contains same values.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 192

2.0
Monitoring and Troubleshooting Guide Logging functions

5 Logging functions

5.1 Service Manager log files

This section describes the Service Manager (SM) alarms generated and the SM log files collected in
case of any failure scenarios.

5.1.1 Log files

Description

SM creates log files for multiple operations:

• Provisioning log: This log refers to Provisioning transaction log. It contains information about all
provisioning transactions performed using either Graphical User Interface (GUI) or an upstream
driving system, such as REST Application Programming Interface (API).
• Managed Element log: This log refers to Managed Element log. It contains information, such as
ME name for the requests send to Managed Elements (MEs), ME info, or error responses.
• User Access log: This log refers to User Access log. It contains all information related to user
login or number of successful/unsuccessful attempts.
• Error log: This log has all functional area logs with severity levels defined as WARN, ERROR,
and FATAL as per the requirements.
• DDESMApplication log: This is a default log that records all operations initiated by SM users not
covered by other logs.

Log Type Log Level Scenario

Provisioning.log INFO Successful creation of diameter

route.
DEBUG

TRACE

ERROR

UserAccess.log DEBUG Successful Login/Logout into

SM GUI.
INFO

ManagedElement.log DEBUG Successful registration of new

managed element on SM GUI
INFO

id9YZ-09148-MT11-PCZZA © 2020 Nokia 193

2.0
Monitoring and Troubleshooting Guide Logging functions

Log Type Log Level Scenario

Error.log ERROR ME Registration with error Bad

URL.

Table 52: Various levels of all types of SM logs

The path of all log files is /opt/tpa/logs and alarms are logged in file /var/log/alarms.

5.1.2 SM logs

Managed Element log

2018/03/03 12:48:30.544 INFO (service-manager-job-1) [com.nokia.dde.svcmgr.server.services.MEServiceClient.

registerManagedElement:111] register 10.75.147.40
2018/03/03 12:49:30.126 INFO (service-manager-job-1) [com.nokia.dde.svcmgr.server.services.MEServiceClient.
registerManagedElement:111] register 10.75.147.40
2018/03/03 12:49:31.564 INFO (service-manager-job-1) [com.nokia.dde.svcmgr.server.services.MEServiceClient.
registerManagedElement:111] register 10.194.183.11
2018/03/03 12:50:30.253 INFO (service-manager-job-1) [com.nokia.dde.svcmgr.server.services.MEServiceClient.
registerManagedElement:111] register 10.194.183.11
2018/03/03 12:50:31.041 INFO (service-manager-job-1) [com.nokia.dde.svcmgr.server.services.MEServiceClient.
registerManagedElement:111] register 10.75.147.40
2018/03/03 12:51:15.488 INFO (IO-ThreadPool-64) [com.nokia.dde.svcmgr.server.providers.
ManagementServiceProviderImpl.muteME:1548] Un Muted ME: DDE-CI1

Error log

2018/03/03 14:08:29.966 ERROR (service-manager-job-1) [com.nokia.dde.svcmgr.server.dao.MsDao.update:269] null

2018/03/03 14:08:30.260 ERROR (service-manager-job-1) [com.nokia.dde.svcmgr.server.dao.MsDao.update:269] null
2018/03/03 14:09:30.314 ERROR (service-manager-job-1) [com.nokia.dde.svcmgr.server.dao.MsDao.update:269] null
2018/03/03 14:09:30.547 ERROR (service-manager-job-1) [com.nokia.dde.svcmgr.server.dao.MsDao.update:269] null

Provisioning log- INFO level

2018/03/09 12:19:13.081 ERROR (IO-ThreadPool-29306) [com.nokia.dde.svcmgr.server.dao.MsDao.getMEList:728] Managed

element list is empty
2018/03/09 12:20:36.202 INFO (IO-ThreadPool-29335) [com.nokia.dde.svcmgr.common.util.QueryTaskOssi.
getManagedObject:97] get Managed Object AddressGroupConfig/dde
2018/03/09 12:21:39.689 INFO (IO-ThreadPool-29361) [com.nokia.dde.svcmgr.common.util.QueryTaskOssi.
getManagedObject:97] get Managed Object AddressGroupConfig/netco
2018/03/09 12:32:21.199 INFO (IO-ThreadPool-29601) [com.nokia.dde.svcmgr.common.util.ProvRequest.
getProvStatus:1422] Provisioning Success for Object class com.nokia.dde.svcmgr.par.DiameterPeerConfig for action
UPDATE
2018/03/09 12:33:08.397 ERROR (IO-ThreadPool-29623) [com.nokia.dde.svcmgr.server.dao.MsDao.getByID:386]
RoutingProfile is null.
2018/03/09 12:33:28.106 INFO (IO-ThreadPool-29636) [com.nokia.dde.svcmgr.common.util.ProvRequest.
getProvStatus:1422] Provisioning Success for Object class com.nokia.dde.svcmgr.par.dde.RoutingProfile for action
UPDATE
2018/03/09 12:33:47.029 INFO (IO-ThreadPool-29639) [com.nokia.dde.svcmgr.common.util.ProvRequest.
getProvStatus:1422] Provisioning Success for Object class com.nokia.dde.svcmgr.par.dde.RoutingPluginConfig for
action UPDATE

id9YZ-09148-MT11-PCZZA © 2020 Nokia 194

2.0
Monitoring and Troubleshooting Guide Logging functions

Provisioning log- DEBUG level

2018/03/14 11:11:43.636 DEBUG (IO-ThreadPool-22501) [com.nokia.dde.svcmgr.server.dao.MsDao.getByID:400] Request

processed successfully for MEStatus object against id : com.nokia.dde.svcmgr.par.MEStatusPK@a1840430
2018/03/14 11:11:44.382 DEBUG (IO-ThreadPool-22503) [com.nokia.dde.svcmgr.server.dao.MsDao.getByID:396] Get
UserConfig object against id : smadmin
2018/03/14 11:11:44.384 DEBUG (IO-ThreadPool-22503) [com.nokia.dde.svcmgr.server.dao.MsDao.getByID:400] Request
processed successfully for UserConfig object against id : smadmin
2018/03/14 11:11:44.385 DEBUG (IO-ThreadPool-22503) [com.nokia.dde.svcmgr.server.dao.MsDao.getByID:396] Get
SmRole object against id : administrator
2018/03/14 11:11:44.386 DEBUG (IO-ThreadPool-22503) [com.nokia.dde.svcmgr.server.dao.MsDao.getByID:400] Request
processed successfully for SmRole object against id : administrator
2018/03/14 11:11:44.388 DEBUG (IO-ThreadPool-22503) [com.nokia.dde.svcmgr.server.dao.MsDao.getList:757] Request
successfully executed for object : LoadBalancingPool

User access log- INFO level

2018/03/09 12:19:12.299 INFO (IO-ThreadPool-29299) [com.nokia.dde.svcmgr.core.filter.AuthenticatorFilter.

filter:220] User Credential is valid for login id: smadmin and the requested URI is :/svcmgr-api/
nonssoauthenticate for method type: GET
2018/03/09 12:20:33.849 INFO (IO-ThreadPool-29335) [com.nokia.dde.svcmgr.core.filter.AuthenticatorFilter.
filter:220] User Credential is valid for login id: smadmin and the requested URI is :/svcmgr-api/register for
method type: POST
2018/03/09 12:31:45.072 INFO (IO-ThreadPool-29592) [com.nokia.dde.svcmgr.core.filter.AuthenticatorFilter.
filter:257] User Access is valid for login id: smadmin and the requested URI is :/svcmgr-api/addressgroups for
method type: GET
2018/03/09 12:31:45.196 INFO (IO-ThreadPool-29583) [com.nokia.dde.svcmgr.core.filter.AuthenticatorFilter.
filter:220] User Credential is valid for login id: smadmin and the requested URI is :/svcmgr-api/diampeer for
method type: GET
2018/03/09 12:31:45.197 INFO (IO-ThreadPool-29583) [com.nokia.dde.svcmgr.core.filter.AuthenticatorFilter.
filter:257] User Access is valid for login id: smadmin and the requested URI is :/svcmgr-api/diampeer for method
type: GET
2018/03/09 12:31:47.247 INFO (IO-ThreadPool-29586) [com.nokia.dde.svcmgr.core.filter.AuthenticatorFilter.
filter:220] User Credential is valid for login id: smadmin and the requested URI is :/svcmgr-api/dde/throttling
for method type: GET
2018/03/09 12:31:47.249 INFO (IO-ThreadPool-29587) [com.nokia.dde.svcmgr.core.filter.AuthenticatorFilter.
filter:220] User Credential is valid for login id: smadmin and the requested URI is :/svcmgr-api/dde/throttling
for method type: GET
2018/03/09 12:31:47.249 INFO (IO-ThreadPool-29586) [com.nokia.dde.svcmgr.core.filter.AuthenticatorFilter.
filter:257] User Access is valid for login id: smadmin and the requested URI is :/svcmgr-api/dde/throttling for
method type: GET
2018/03/09 12:33:49.756 INFO (IO-ThreadPool-29628) [com.nokia.dde.svcmgr.core.filter.AuthenticatorFilter.
filter:257] User Access is valid for login id: smadmin and the requested URI is :/svcmgr-api/dde/msgfilter/
listcontextsforactions for method type: GET
2018/03/09 12:33:49.962 INFO (IO-ThreadPool-29638) [com.nokia.dde.svcmgr.core.filter.AuthenticatorFilter.
filter:220] User Credential is valid for login id: smadmin and the requested URI is :/svcmgr-api/dde/
routingprofile for method type: GET

User access log- DEBUG level

2018/03/14 11:11:44.382 DEBUG (IO-ThreadPool-22503) [com.nokia.dde.svcmgr.core.filter.AuthenticatorFilter.

isPilotActive:540] ASR machine not found. Authenticating in DSC machine...
2018/03/14 11:11:44.382 DEBUG (IO-ThreadPool-22503) [com.nokia.dde.svcmgr.core.filter.AuthenticatorFilter.
filter:144] Request received from Host Address: 135.248.252.192

id9YZ-09148-MT11-PCZZA © 2020 Nokia 195

2.0
Monitoring and Troubleshooting Guide Logging functions

DDESMApplication.log

2018.03.08 14:02:10 440 +0530 D smbvnf-oame-0 FelixDispatchQueue jetty.webconnector:78 com.alcatel.as.

webconnector.jetty [120].init(com.alcatel.as.webconnector.jetty)
2018.03.08 14:02:11 243 +0530 I smbvnf-oame-0 FelixDispatchQueue com.nokia.dde.extnotification.server.
NotificationDispatchService:38 Starting NotificationScheduler
2018.03.08 14:02:19 421 +0530 I smbvnf-oame-0 FelixDispatchQueue com.nokia.dde.common.db.aerospike.
ASHealthMonitorImpl:188 Starting health monitor task
2018.03.08 14:02:19 423 +0530 I smbvnf-oame-0 FelixDispatchQueue com.nokia.dde.common.db.aerospike.
ASHealthMonitorImpl:270 Waiting for the database to become online.
2018.03.08 14:02:19 428 +0530 I smbvnf-oame-0 AS Health Monitor-1 com.nokia.dde.common.db.aerospike.
ASHealthMonitorImpl:544 Restoring service at startup...
2018.03.08 14:02:19 428 +0530 I smbvnf-oame-0 FelixDispatchQueue com.nokia.dde.common.db.aerospike.
ASHealthMonitorImpl:270 Waiting for the database to become online.
2018.03.08 14:02:19 429 +0530 I smbvnf-oame-0 FelixDispatchQueue com.nokia.dde.common.db.aerospike.
ASHealthMonitorImpl:291 Health monitor task started.
2018.03.08 14:02:22 135 +0530 I smbvnf-oame-0 FelixDispatchQueue com.nokia.dde.policy.common.
systemconfig.PolicySystemConfigCache:107 Preference "Rule Trace Logging Enabled" updated to: false
2018.03.08 14:02:45 010 +0530 I smbvnf-oame-0 RMI TCP Connection(10)-192.168.4.10 com.nokia.dde.common.
server.hacluster.HATopologyMonitor:723 New topology status is: MASTER
2018.03.08 14:02:45 044 +0530 I smbvnf-oame-0 HATopologyMonitor-ListenerNotifier-1 com.nokia.dde.
extnotification.server.NotificationPreferenceListener:158 Dedicated Notification Processing Threads - SOAP
changed to: 0
2018.03.08 14:02:45 045 +0530 I smbvnf-oame-0 HATopologyMonitor-ListenerNotifier-1 com.nokia.dde.
extnotification.server.NotificationPreferenceListener:167 Dedicated Notification Processing Threads - SMS
changed to: 0
2018.03.08 14:02:45 045 +0530 I smbvnf-oame-0 HATopologyMonitor-ListenerNotifier-1 com.nokia.dde.
extnotification.server.NotificationPreferenceListener:176 Dedicated Notification Processing Threads - EMAIL
changed to: 0
2018.03.08 14:02:45 045 +0530 I smbvnf-oame-0 HATopologyMonitor-ListenerNotifier-1 com.nokia.dde.
extnotification.server.NotificationPreferenceListener:185 Dedicated Notification Processing Threads - REST
changed to: 0
2018.03.08 14:02:45 092 +0530 I smbvnf-oame-0 HATopologyMonitor-ListenerNotifier-1 com.nokia.dde.
extnotification.server.NotificationPreferenceListener:213 Notification Date and Time Format changed to: yyyy-MM-
dd'T'HH:mm:ss.SSSXXX
2018.03.08 14:02:45 155 +0530 I smbvnf-oame-0 HATopologyMonitor-ListenerNotifier-1 com.nokia.dde.policy.
server.initializer.BaseRuleSystemVersionInitializer:53 No default rule system version found. Creating the
default rule system version.

5.2 CSD logs

5.2.1 CSD logging functions

5.2.1.1 About logging functions

CSD logs events for many internal functions.

Node Location

OAM node For all alarms - /var/log/calm/alma.log

For counter related information - /appdata/

ganglia/meas/CBAM-

*/<VM name>

id9YZ-09148-MT11-PCZZA © 2020 Nokia 196

2.0
Monitoring and Troubleshooting Guide Logging functions

Node Location
For events, alerts

• /opt/gangalia/log/gen3GPPXmlPM.log
• /var/log/calm/alma.log

Installation logs:

/opt/tpa/logs/SPI.*.log

/opt/tpa/logs/install_DDE.*.log

IO node /opt/tpa/logs/DDESMApplication.log

App Node /opt/tpa/logs/DDESMApplication.log

DB nodes /opt/tpa/logs/aerospike.log

Table 53: Location of the log files

See the following procedures for more information:

• Changing CSD logging levels on page 197

• Enabling logs from OAM VM on page 198
• To view CSD log files on page 200

5.2.1.2 Changing CSD logging levels

The default logging level for many functions is informational (or INFO), which minimizes the number

of log entries. The logging level for many internal functions can be configured to provide more

information for troubleshooting purposes, i.e. setting the logging level to DEBUG. To view this

procedure, see To change logging levels on individual VM on page 199

You can specify one of the following case-sensitive logging levels for each functional area:

Level Description

ERROR An unexpected system condition that occurred is

automatically recoverable;

for example, an out-of-memory condition that

results in a process restart or null pointer
exception.

WARN A recoverable misconfiguration or badly formed

user requests.

INFO System change events.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 197

2.0
Monitoring and Troubleshooting Guide Logging functions

Level Description

DEBUG Extremely verbose logs that provide extra

information for code-centric investigations.

TRACE Provides more detailed control than the debug

logs; splits low-level logs.

5.2.1.3 To Change the logging levels of CSD functions

Lowering a logging level creates additional log entries and requires more system resources, which can
affect system performance. Ensure that you lower the logging level for only the required functions and
for only the time required to collect the troubleshooting information. For example,

/opt/tpa/bin/DDE_logLevel com.nokia.dde DEBUG

Note:

Note that enabling DEBUG-level logging will generate an extremely large number of logs,
which can affect system performance. Ensure that you lower the logging level for only the
required functions and for only the time required to collect the necessary information.

Also, changing the log level does not persist after an application server restart. For a
persistent change, you must modify the log4j.xml file accordingly.

5.2.1.4 Enabling logs from OAM VM

Enter the following commands to manage log levels from /opt/tpa/bin

1. To get the current log level, enter

log-level [-n <hostname> | -m <mod name>] <logger name>

A value of NULL means that a level has not been explicitly set for the specified logger.

Example: log-level -n ddebvnf-oame-1 com.nokia.dde

log-level -m oam/ddeio/ddeapp com.nokia.dde, here mod name <oam/ddeio>

changes the log level of active OAM or IO respectively and ddeapp or db changes the log-level of
ddeapp or db VMs.

2. To modify the log level, enter:

log-level [-n <hostname> | -m <mod name>] <logger name> <level>

If a level has not been set for the specified logger, a previous level of INFO is assumed.

Example: log-level -n ddebvnf-oame-1 com.nokia.dde DEBUG

3. To reset the level of a logger back to default, enter:

log-level [-n <hostname> | -m <mod name>] -r <logger name>

id9YZ-09148-MT11-PCZZA © 2020 Nokia 198

2.0
Monitoring and Troubleshooting Guide Logging functions

Example: log-level -n ddebvnf-oame-1 -r com.nokia.dde

5.2.1.5 To change logging levels on individual VM

1. To get the current log level, enter

DDE_logLevel <logger name>

Example: DDE_logLevel com.nokia.dde

2. To modify the log level, enter:

DDE_logLevel <logger name> <level>

Example:DDE_logLevel com.nokia.dde DEBUG

3. To reset the level of a logger back to default, enter:

DDE_logLevel <logger name> reset

Example: DDE_logLevel com.nokia.dde reset

4. To reset the level(s) of all logger(s) back to default, enter:

DDE_logLevel reset_all

5.2.1.6 To enable OSSI log collection

Note: OSSI log collection requires the use of system resources that may affect system
performance. Ensure that you only collect the logs with the troubleshooting information
required. The system does not automatically delete OSSI logs. You must periodically delete
the logs manually.

1. Open a console window and log in to the CSD as root.

2. Open the log4j.xml file in /opt/tpa/logs/ directory using a text editor.

3. Navigate to the following:

<logger name="com.nokia.dde">

<level value="INFO"/>

</logger>

4. Enter the following line above or below the lines to which you navigated in step 3

</logger
<!-- Warning, the following entry can be use for debugging OSSI

id9YZ-09148-MT11-PCZZA © 2020 Nokia 199

2.0
Monitoring and Troubleshooting Guide Logging functions

problems. It will produce a large amount of ossi logs

files in the log directory and directly impact the
performance
of the system. Please ensure to turn it back to the INFO
level when not debugging.
<logger name="com.nokia.dde.common.server.ossi.servlet.
XmlApiServlet">
<level value="DEBUG"/>
</logger>

5. Save and close the log4j.xml file.

5.2.1.7 To enable logs on APP nodes

1. Open a console window and log in to the application blade as root user.

2. Navigate to /opt/tpa/bin/DDE_loglevel.

3. For debugging purposes set the log level to verify the traffic behavior.

For more information, refer the section Tracing and Debugging on page 144.

5.2.1.8 To view CSD log files

1. Open a console and log in to the CSD as root.

2. Navigate to the /opt/tpa/logs directory.

3. Use a text editor to open a .log file to view the file. For example, view the
DDESMApplication.log file by typing:

# vi DDESMApplication.log

5.2.2 Log rotation on CSD and SM

Overview

On both CSD and SM nodes, logs related to CSD and CSF both get rotated either when they reach
their maximum size configured or maximum time reached. On CSD, these configurations of CSD logs
are mentioned in /opt/tpa/logs/log4j.xml (present on all nodes including APP, IO and OAM
except DB) and for CSF logs, configuration is mentioned in /etc/logrotate_syslog.conf on all
nodes.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 200

2.0
Monitoring and Troubleshooting Guide Logging functions

For example, CSD logs at /opt/tpa/logs directory:

[root@ddebvnf-oame-0 logs]# ls -1
ddeLogrotate.conf
DDEOperations.log
DDESecurity.log
DDESMApplication.0.log.gz
DDESMApplication.1.log.gz
DDESMApplication.2.log.gz
DDESMApplication.3.log.gz
DDESMApplication.log
DDE_system.log
DiameterInterceptor.log
gc.log.0
gc.log.1
gc.log.2
gc.log.3
gc.log.4.current
HADiagnosticsDefault.log
install_DDE.2018.09.11-14-28.log
log4j.xml
PolicyDecision.log
Server.0.log.gz
Server.1.log.gz
Server.2.log.gz
Server.log
SPI.2018.09.11-14-27.log
SubscriberTrace.log

For DB, we have /opt/tpa/aerospike/etc/logrotate.d config file which has the

configurations for aerospike logs present at /opt/tpa/logs path.

For example, /opt/tpa/logs

[root@ddebvnf-db-1 logs]# ls -1
aerospike.log
aerospike.log.1
aerospike.log.10.gz
aerospike.log.11.gz
aerospike.log.12.gz
aerospike.log.2.gz
aerospike.log.3.gz
aerospike.log.4.gz
aerospike.log.5.gz
aerospike.log.6.gz
aerospike.log.7.gz
aerospike.log.8.gz
aerospike.log.9.gz
aerospike_warning.log
ddeLogrotate.conf
id9YZ-09148-MT11-PCZZA © 2020 Nokia 201
2.0
Monitoring and Troubleshooting Guide Logging functions

DDE_system.log
install_DDE.2018.09.11-14-44.log
SPI.2018.09.11-14-43.log

For example, CSF logs at /var/logs on all nodes.

[root@ddebvnf-oame-0 logs]# cd /var/log/

[root@ddebvnf-oame-0 log]# ls -1
aide
alarms
audit
boot.log
boot.log-20180912
boot.log-20180913
boot.log-20180914
boot.log-20180915
boot.log-20180916
boot.log-20180917
boot.log-20180918
btmp
cloud-init.log
cron
cron.1
cron.2.gz
cron.3.gz
cron.4.gz
ddelogs
dmesg
firewalld
Gen3GPPXML.log
Gen3GPPXML.log.1
grafana
grubby
grubby_prune_debug
ha
ha-sync
httpd
journal
lastlog
logging
logging.log
logstash
lost+found
maillog
messages
messages.1
nrpe
ntpstats
pluto

id9YZ-09148-MT11-PCZZA © 2020 Nokia 202

2.0
Monitoring and Troubleshooting Guide Logging functions

sa
sdc_alarm.log
sdc.log
secpam-boot.log
secure
secure.1
secure.2.gz
secure.3.gz
secure.4.gz
snmptt
spooler
syncer
tallylog
tuned
watson
wtmp
yum.log

Rotation of files containing counters-related information that are present on each VM at /

appdata/ganglia/meas/CBAM-*/<VM name> is handled by /opt/ganglia/3gpptool/
measCollecXml.conf.

The default configuration to delete these XML files is 7 days.

For example, /appdata/ganglia/meas/CBAM-1b68110798e74424ac8e99e41b518701/

ddebvnf-ss7io-1

-rw-r--r--. 1 measurement meas 9091 Mar 15 16:00 A20180315.1545+0800-

20180315.1600+0800_ddebvnf-ss7io-1_-_1.xml
-rw-r--r--. 1 measurement meas 1409 Mar 15 16:02 A20180315.1545+0800-
20180315.1600+0800_ddebvnf-ss7io-1.xml
-rw-r--r--. 1 measurement meas 9600 Mar 15 16:15 A20180315.1600+0800-
20180315.1615+0800_ddebvnf-ss7io-1_-_1.xml
-rw-r--r--. 1 measurement meas 1409 Mar 15 16:17 A20180315.1600+0800-
20180315.1615+0800_ddebvnf-ss7io-1.xml

5.2.3 Deletion of User or a Group

5.2.3.1 Performing Deletion of User or a Group

To delete any unwanted user or a group from VM, use the following commands:

• To delete a user from VM: userdel <user name>

• To delete group from VM: groupdel <group name>

id9YZ-09148-MT11-PCZZA © 2020 Nokia 203

2.0
Monitoring and Troubleshooting Guide Network Troubleshooting

6 Network Troubleshooting
Overview

This chapter describes how to manage network functions and troubleshoot common network problems
that arises in the CSD.

The following table lists the common network problems in CSD along with the solution.

Problem Solution

Host unreachable when CBAM and VNF are in Before installation, ensure that CSD or SM VNF
different subnets. gateway are reachable through CBAM.

Diameter Peer not reachable due to missing Add appropriate routes towards remote peer on
Routes. When the CSD and the remote peer IO nodes for reachability.
are on different networks, CSD fails to reach the
remote peer due to the route unavailability.

When the netstat command is executed, the

SYN_SENT status for the newly added peer
does not update as ESTABLISHED, since the
connection is not established between CSD and
remote peer.

Upgrade or rollback failure with the error Re-trigger upgrade or rollback when you
Destination Host Unreachable. encounter with the error Destination Host
Unreachable.

Table 54: Problem and Solution

id9YZ-09148-MT11-PCZZA © 2020 Nokia 204

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

7 Changes in CSD release

This chapter contains the key differences between this release and previous release of CSD.

Note: For details on release changes prior to 18.8 SP1, refer the Monitoring and
Troubleshooting Guide, 9YZ-08354-MT11-PCZZA.

7.1 Scripts

Overview

The following are the list of scripts which are present in /opt/tpa/bin.

• configure-geo-redundancy
• db-collect-logs
• distribute-diameter-dictionaries
• list-diameter-routes
• config-backup
• db-backup
• db-restore
• list-diameter-peers
• log-level

7.2 Changes from previous release to this release

7.2.1 Software name of Bare-metal variant

The software name of bare-metal variant has now changed to:

Prior to 18.8 Post 18.8

DDE_BM_18.2_Ixxx.tar.gz NOKIAcdrom-CSD_18_8_Ixxx-1.x86_64.rpm

Table 55: Software name of Bare-metal variant

7.2.2 Output of DDE_status

The Diameter Stack Status value as part of DDE_status output is changed as follows:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 205

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

Prior to 18.8 Post 18.8

Diameter Stack Status: LISTENING or NOT Diameter Stack Status: STARTED or NOT STARTED.
LISTENING.

Table 56: Diameter Stack Status

7.2.3 Changes in Provisioning log

The Provisioning.log in the following path enables you to identify the user who has performed the
provisioning related transactions using the SM GUI.

/opt/tpa/logs

Prior to 20.2 Post 20.2

The following is the sample provisioning log without The following is the sample provisioning log indicating
username. the username who performed the provisioning
transactions.
Status of provisioning the entity
DiameterPeerConfig
smadmin has provisioned the entity
{id=DiameterPeerConfigID
DiameterPeerConfig
{diamPeerId=529163938},
{id=DiameterPeerConfigID
portNumber=0, active=true,
{diamPeerId=529163938},
addressGroupName='Tcp1',
portNumber=0, active=true,
secondaryAddresses={}, secure=false,
addressGroupName='Tcp1',
description='null',
secondaryAddresses={},
locallyInitiated=false, requestTimeout=0,
secure=false, description='null',
numberOfSendAttempts=0, sourcePortNumber=0,
locallyInitiated=false,
primaryAddress='',
requestTimeout=0, numberOfSendAttempts=0,
protocol='TCP',
sourcePortNumber=0, primaryAddress='',
ingressPeerThrottlingProfileName='null',
protocol='TCP',
egressPeerThrottlingProfileName='null',
ingressPeerThrottlingProfileName='null',
peerConnectionProfileName='null',
egressPeerThrottlingProfileName='null',
downStreamOverLoadProfileName='null'
peerConnectionProfileName='null',
, sctpAssociationProfileName='null',
downStreamOverLoadProfileName='null',
requestTimoutProfileName='null',
sctpAssociationProfileName='null',
originHost='peer',
requestTimoutProfileName='null',
peerType='Fqdn', createdDate='1580121875577',
originHost='peer', peerType='Fqdn',
rank='0'}
createdDate='1580121875577', rank='0'} and
to Managed Element : ME is SUCCESS
Status of provisioning
to Managed Element : ME is SUCCESS

Table 57: Provisioning log changes

7.2.4 Changes in message filter criteria

The following unsupported actions or criteria are removed from the message filters in the SM GUI.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 206

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

Prior to 20.5 Post 20.5

The following criteria were persent in the message The following unsupported criteria are removed from
filters of the SM GUI. the message filters in SM GUI.

• Any Diameter-Answer (For example, GxCCA, • Any Diameter-Answer (For example, GxCCA,
ShPUA, S6aULA, and so on). ShPUA, S6aULA, and so on).
• Diameter-CER • Diameter-CER
• Diameter-CEA • Diameter-CEA
• Diameter-DPR • Diameter-DPR
• Diameter-DPA • Diameter-DPA

Table 58: Changes in message filter criteria

7.2.5 Auth-Application-Id AVP changes in Diameter error responses

Auth-Application-Id AVP is removed from diameter 3xxx error responses sent by CSD.

Prior to 20.2 Post 20.2

In case of routing failure or peer is in down state, CSD For any 3xxx error responses the Auth-
sends Auth-Application-Id as a single AVP or Application-Id is not sent either as a single AVP
as part of Vendor-Specific-Application-Id or as part of Vendor-Specific-Application-Id.
in the 3xxx error responses. However in scenarios
However, if the 3xxx response is sent by the remote
like Request Timeout and Loop Detection, Auth-
peer containing Auth-Application-Id, then it
Application-Id is not included in the 3xxx error
is relayed back to the client as it is with the Auth-
responses.
Application-Id.

Table 59: Auth-Application-Id

7.2.6 Change in DDE_control status

With the implementation of Active-Active IO feature applicability of DDE_control become-slave
command is changed:

Prior to 20.0 Post 20.0

DDE_control status command performs graceful DDE_control status command now performs
switchover of IO and OAM nodes. graceful switchover of OAM nodes.

The command execution to perform switchover of IO

nodes is achieved now by following commands:

1. DDE_control close-all-connections

id9YZ-09148-MT11-PCZZA © 2020 Nokia 207

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

Prior to 20.0 Post 20.0

2. DDE_control close-all-ha-connections

Table 60: DDE_control become-slave

7.2.7 Active-Active IO load balancer changes

With the introduction of Active-Active IO load balancer feature, following is the command behavior.

Prior to 20.0 Post 20.0

Execution of DDE_Status on both IO and OAM Execution of DDE_Status on both IO and OAM
nodes: nodes:

App Server HA Topology Status:MASTER App Server HA Topology Status:MASTER

App Server HA Topology Status:SLAVE App Server HA Topology Status:MASTER

Switchover of IO node is performed using the following Switchover of IO node is performed using the following
command: command:

DDE_control become-slave DDE_control close-all-ha-connections

Stateful closing of all connections (static and floating) Stateful closing of all connections (static and floating)
cannot be performed. can now be performed using the following command:

DDE_control close-all-connections

Table 61: Active-Active IO load balancer changes

7.2.8 Route-Record AVP changes

Route-Record AVP is removed from Diameter Answer messages sent by CSD.

Prior to 20.0 Post 20.0

Route-Record AVP gets added to Diameter-Answer Route-Record AVP will not be added to Diameter-
messages sent by CSD. Answer message sent by CSD.

Table 62: Route Record AVP

7.2.9 Changes in XinetD service

The XinetD service is removed from CSD.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 208

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

Prior to 20.0 Post 20.0

The XinetD service enables you to start programs The XinetD service is removed as this service can be
which provide internet services. a security vulnerability.

Table 63: XinetD service

7.2.10 Location of 3gpp.xml file for application counters

The location of 3gpp.xml file location for application counters is changed. No change to system
counters file.

Prior to 18.8 Post 18.8

/appdata/ganglia/meas/ /appdata/ganglia/meas/cpro
CBAM-51328829b76e4fd4a8ced03173fa548c

Table 64: Location of 3gpp.xml file for application counters

7.2.11 healthMachines
This tool lists all the nodes with its HA and service status. The HA status of shutdown VMs is changed.

You can access this tool from the /opt/tpa/bin directory as a root user.

Prior to 18.8 Post 18.8

HA Status HA Status

Unknown NA

Prior to 20.0 Post 20.0

healthMachines is a standalone command. In addition to the standalone execution,

healthMachines supports the following arguements.

• -d: Display detailed health status of all the VMs.

• -v: Display detailed health report of a specific VM.

For example, healthMachines -d

Table 65: healthMachines

7.2.12 In-Service Upgrade Changes

CSD supports single click upgrade (Automated upgrade and fallback package preparation for Cloud
variant).

id9YZ-09148-MT11-PCZZA © 2020 Nokia 209

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

For more details, refer the Upgrade section in the Installation and Upgrade Operations Guide for Cloud
Deployments.

7.2.13 Metrics Configuration form

The Metrics Configuration feature enables you to capture the required data in Transaction Data
Record (TDR) including Origin Host, Origin Realm, Destination Host, Destination Realm, and so on.

For more details, see the Metrics configuration section in the CSD User Guide.

7.2.14 Change of Metrics in CSV format - Transaction Data Record (TDR)

The Transaction Data Records (TDR) are generated in a CSV format.

The following table lists the changes from previous release to this release.

Prior to 18.8 Post 18.8

metrics.n.csv <startdate.timestamp+timezone>-<enddate.timestamp
+timezone>_sitename.csv

(A20181026.085500+0000-20181026.090000+0000_
v700DDE.csv)

Fields with reduced number of columns.

Logs recorded in application log (DDESMApplication. metrics-log-file.log and /var/log/

log) messages

metrics.csv file path metrics.csv file path

/opt/tpa/logs /opt/tpa/logs/metrics/csvfiles

Prior to 19.5 Post 19.5

<startdate.timestamp+timezone>-<enddate.timestamp <startdate.timestamp+localtimezone>-<enddate.
+timezone>_sitename.csv timestamp+localtimezone>_sitename.csv

(A20181026.085500+0000-20181026.090000+0000_ (A20190701.122500+0000-20190701.123000+0000_
v700DDE.csv) netco.csv)

Fields with reduced number of columns. Fields with reduced number of columns.

Note: The timezone is in UTC.

Table 66: Transaction Data Records

Note: The value of Site in the metric file displays the geoSitename configured in the
vnfd.scalable.tosca.yaml file.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 210

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

For more details, see the Transaction Data Record (TDR) sections in the CSD User Guide, Installation
and Upgrade Operations Guide for Cloud Deployments.

7.2.15 Dashboard is now Analytics

The Dashboard is now Analytics.

For more, details refer the Analytics section in the CSD User Guide.

Prior to 20.0 Post 20.0

Dashboard Analytics

• Grafana - Service Monitoring

• Prometheus - Performance Monitoring
• Alertmanager - Fault Management
• Jaeger - Traceability

7.2.16 Support for Ganglia GUI

The support for Ganglia GUI is now discontinued.

7.2.17 Improper disk utilization of all the VMs during OAM node reboot
This section provides information on common errors or issues encountered during reboot of OAM
nodes using the following command:

reboot -f

Problem Solution

The counters related to disk utilization in the Update the value of collect_every and time_
3GPP system counter file for all the VMs remains threshold of disk.total to 180 seconds in
improper during OAM VM reboot duration. the following file for all the VMs.

/etc/ganglia/gmond.conf

7.2.18 External SLF timeout value

The minimum value of AGE OUT TIME in the Configuration > Subscriber Location Function > SLF
Configuration Form window of the SM GUI is changed for external SLF.

Minimum Value: 60 seconds.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 211

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

7.2.19 Changes in SCTP Association Profile

The SCTP Association Profile now has the Selective Acknowledgement (SACK) of the time-out
interval.

Prior to 19.5 Post 19.5

SCTP Association Profile form SCTP Association Profile form

SACK Time-out:

When there are inbound messages from the

peers, CSD sends selective acknowledgement
on the basis of time-out interval configured in the
SCTP association profile.

Note: This feature is applicable only from release

19.5 and onwards.

For more details see, To create an SCTP association profile section in the CSD User Guide.

7.2.20 Changes in Peer connection status dashboard

The Peer connection status dashboard is changed to Peer Management dashboard.

Prior to 19.5 Post 19.5

Peer Connection Status Peer Management Dashboard

• Peer Status: • Peer Status:

– Total Peers – Total Peers - No change.

– Peers-Up – Peers-Up - No change.
– Peers-Down – Peers-Down - No change.
– Peers-Blocked -Introduced.

Flush All

This enables you to remove the selected records

from the Peer Management Dashboard.

Note: The Flush All operation can be performed

only on the blocked inbound peers.

Peer Connection Status List: Peer List:

• Peer Name • Peer Name - No change.

• Peer Realm • Peer Realm - No change.
• Connection Type • Connection Type - No change.
• Application • Application - No change.
• Protocol • Protocol - No change.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 212

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

Prior to 19.5 Post 19.5

• IP Address • IP Address - No change.
• Status • Status - Blocked status is introduced in the
• Last Update Time Status drop-down list.
• Connections - Enables you to view the
number of identical inbound connections.
• Last Update Time - No change.
• Actions - Enables you to perform the
following operations on a diameter peer.

– Block
– Unblock
– Disconnect

The Correlation feature enables you to view

the peer details corresponding to the Peer
Management Dashboard in the Diameter Peer
Configuration form.

For more details, see the Peer Management Dashboard section in the CSD User Guide.

7.2.21 Changes in CSD and SM internal communication matrices

The following ports in the CSD and SM communication matrices are introduced.

Prior to 19.5 Post 19.5

SM internal communication matrix: SM internal communication matrix:

• Port 53 • Port 53 - with transport type TCP provides

• Port 111 DNS services for zone transfers.
• Port 111 - with transport type UDP provides
rpcbind service.
• Port 9000 - with transport type TCP provides
grafana services.

CSD internal communication matrix: CSD internal communication matrix:

• Port 53 • Port 53 - with transport type TCP provides

• Port 111 DNS services for zone transfers.
• Port 111 - with transport type UDP provides
rpcbind service.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 213

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

Prior to 19.5 Post 19.5

• Port 9000 - with transport type TCP provides
Grafana services.

Table 67: Changes in CSD and SM internal communication matrices

Ports removed in CSD 20.0 Ports added in CSD 20.0

CSD external communication matrix: CSD external communication matrix:

• Port 22 - with the transport typeTCP of SSH • Port 8443 - with transport type TCP provides
service on IO nodes. HTTPD service OAM node.
• Port 8444 - with transport type TCP provides
prometheus service to OAM node.

CSD internal communication matrix: CSD internal communication matrix:

• Port 8888 - with the transport type TCP of • Port 5775 - with transport type UDP provides
XinetD is removed. Jaeger service to OAM node.
• Port 8890 - with the transport type TCP of • Port 6831 - with transport type UDP provides
XinetD is removed. Jaeger service to OAM node.
• Port 6832 - with transport type UDP provides
Jaeger service to OAM node.
• Port 14250 - with transport type TCP
provides Jaeger service to OAM node.
• Port 14267 - with transport type TCP
provides Jaeger service to OAM node.
• Port 14268 - with transport type TCP
provides Jaeger service to OAM node.
• Port 14269 - with transport type TCP
provides Jaeger service to OAM node.
• Port 16686 - with transport type TCP
provides Jaeger service to OAM node.
• Port 9093 - with transport type TCP provides
alert manager service to OAM node.

SM external communication matrix: SM external communication matrix:

• Port 8443 - with transport type TCP provides

HTTPD service to OAM node.
• Port 8444 - with transport type TCP provides
prometheus service to OAM node.

SM internal communication matrix: SM internal communication matrix:

id9YZ-09148-MT11-PCZZA © 2020 Nokia 214

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

Ports removed in CSD 20.0 Ports added in CSD 20.0

• Port 4369 - with the transport typeTCP of
RabbitMQ peer discovery is removed.
• Port 5672 - with the transport type TCP of
Messaging service is removed.
• Port 25672 - with the transport typeTCP of
RabbitMQ Command Line Interface (CLI) is
removed
• Port 8888 - with the transport typeTCP of
XinetD is removed.
• Port 8890 - with the transport type TCP of
XinetD is removed.

Ports removed in CSD 20.2 Ports added in CSD 20.2

CSD external communication matrix: CSD external communication matrix:

• No change.

CSD internal communication matrix: CSD internal communication matrix:

• Port 3306 - with the transport type TCP of • Port 15672 - with transport type TCP
MariaDB is removed. provides RabbitMQ service to OAM node.
• Port 10050 - with the transport type TCP of • Port 15692 - with transport type TCP
Zabbix is removed. provides RabbitMQ service to OAM node.
• Port 10051 - with the transport type TCP of
Zabbix is removed.

SM external communication matrix: SM external communication matrix:

• No change.

SM internal communication matrix: SM internal communication matrix:

• Port 3306 - with the transport typeTCP of

MariaDB is removed.
• Port 10050 - with the transport typeTCP of
Zabbix is removed.
• Port 10051 - with the transport typeTCP of
Zabbix is removed.

Ports removed in CSD 20.5 Ports added in CSD 20.5

• 8649 (TCP, UDP) • 5778, 14250, 14267, 14268, 14269, 16686,

• 8651 15672, 15692,25672 - Jaeger.
• 8652 (TCP) • 8444 - Prometheus.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 215

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

Ports removed in CSD 20.5 Ports added in CSD 20.5

• 8888 • 4369, 5672 - RabbitMQ.
• 8890 • 1161
• 1162
• 8182 - For calm-alma.service.
• 2379, 2380 - For ETCD service.
• 5140, 8514, 24224, 24231 - FluentD.
• 514 - rsyslog.

7.2.22 Database internal IP address

The database internal IP address is changed. IP address changes depending on the number of
databases installed.

Prior to 19.2 Post 19.2

Database IP address ranges: Database IP address ranges:

192.168.3.40, 192.168.3.41,... 192.168.3.50, 192.168.3.51,...

Table 68: Database internal IP address

7.2.23 VMware Changes

VMware independent disks

By default, the OVF templates consists of a root disk. Any other disks are known as independent
disks.

With this enhancement in VMware, data can be restored after performing heal (rebuild) operation.

7.2.24 Changes in Diameter Peer form

Prior to 20.2 Post 20.2

Sorting of regular expression records is Rank - Sorting of regular expression records is

performed based on Created date. performed based on rank.

Table 69: Changes in Diameter Peer form

Prior to 20.5 Post 20.5

Not applicable. The DWR Timer enables you to configure

Device-Watchdog-Request (DWR) timer for per
peer level connection.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 216

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

Prior to 20.5 Post 20.5

When a Diameter Peer Form is exported at a
object or ME level, the exported file contains
DWR Timer parameter.

For more details on DWR timer, refer the

Importing and exporting section in the CSD User
Guide.

Table 70: DWR timer

7.2.25 Changes in Peer Management Dashboard

Prior to 20.2 Post 20.2

Total number of connected peers was not Connections - Displays the total number of
displayed. connections.

Clicking the peer name displays the peers from Clicking the peer name displays the peer name
the diameter peer form. and the profiles attached to the peer.

Table 71: Changes in Peer Management Dashboard

7.2.26 Changes in SLF Identity to server pool form

Prior to 20.2 Post 20.2

Sorting of regular expression records is Rank - Sorting of regular expression records is

performed based on Created date. performed based on rank.

Table 72: Changes in SLF Identity to server pool form

7.2.27 Changes in SLF Lookup Table Form

Prior to 20.5 Post 20.5

Status - The criteria in Status supports only AND Status - Enables you to select the combination of
criteria. the following actions.

The query parameter across the parameter

supports AND criteria (that is same as previous
releases).

The criteria in Status now supports the following

actions with the OR criteria.

• Provisioned
• Partial Provision

id9YZ-09148-MT11-PCZZA © 2020 Nokia 217

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

Prior to 20.5 Post 20.5

• Failed
• New
• Delete Attempted

For more details, refer the To create an SLF

lookup table section in the CSD User Guide.

Table 73: Changes in SLF Lookup Table Form

7.2.28 SCTP link overload handling mechanism is introduced

With the SCTP link overload handling mechanism, when the application send buffer is beyond
configured thresholds, the diameter requests are responded with configured error code so that the
client can retry.

7.2.29 Changes in Grafana

Prior to 20.2 Post 20.2

CSD Diameter Routing Advanced panel CSD Diameter Routing Advanced panel

• EPR • Egress Realm (Previously EPR).

• IPR • Ingress Realm (Previously IPR).
• Time Period • Time Period is not applicable currently
instead it is dashboard selected time range.

CSD Health Summary CSD Health Summary

• Result code is introduced.

CSD Diameter Routing Status CSD Diameter Routing Status

The following are panels are introduced.

• Total Request Received

• Total Responses Sent.

Table 74: Changes in Grafana

7.2.30 Changes in output of listDiameterPeers.sh

Prior to 20.2 Post 20.2

For a SCTP multihomed peer connection, For a SCTP multihomed peer connection, both
single IP address was displayed under the configured IP addresses are displayed in
the output of listDiameterPeers.sh, if the

id9YZ-09148-MT11-PCZZA © 2020 Nokia 218

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

Prior to 20.2 Post 20.2

Connected-Address in the output of connection is established using both the IP
listDiameterPeers.sh. addresses.

Note: The order in which the IP addresses are

displayed is not fixed.

Table 75: Changes in output of listDiameterPeers.sh

7.2.31 Change in alarm log storage path

Prior to 20.5 Post 20.5

The alarm logs are stored in the following path: The alarm logs are stored in the following path:

/var/log/alarms /var/log/calm/alma.log

/var/log/calm/history.log

7.2.32 Change in alarm structure

Prior to 20.5 Post 20.5

Change in Alarm type. Now Event type

Sample Alarm format: Sample Alarm format:

local3.alert: Mar 6 12:02:18 alarm = {id=51003,

NepalDDE-oame-1 alarmagtd: name=PeerDisconnectAlarm,
{"severity": severity=MAJOR, task=NOTIFY,
"minor", key=Active-IO#DDE#Peer
"objectName": "NepalDDE-ddeapp- disconnect detected peer=server1
0#null", "probableCause": ipaddr=135.2.76.235 port=5002
"communicationsProtocolError", peerid=378456044 protocol=TCP,
"additionalText": text=ORIGINHOST=server1
"ORIGINHOST=diahost.diarealm\ PEERIP=135.2.76.235
nPROTOCOL=unknown;51002;DDE- PEERID=378456044
DIAMETER#;Peer Connection PEERPORT=5002
Detected", PROTOCOL=TCP raised by Active-
"ackStatus": false, "time": "2018- IO;51003;DDE-DIAMETER, event-
03-06 12:02:18", type=1, probable-cause=305,
"VMUUID": "e4fc1f7e-1c49-4d2f- data=ORIGINHOST=server1
b434-59cbf81fa984 LeaseId PEERIP=135.2.76.235
2015468559486589523", PEERID=378456044
"specificProblem": PEERPORT=5002
"PeerConnectAlarm", "VMName": PROTOCOL=TCP raised by Active-
"NepalDDE-ddeapp-0", IO;51003;DDE-DIAMETER}
"alarmType": "communicationAlarm"} extension =
{Enterprise=Enterprise, VM_

id9YZ-09148-MT11-PCZZA © 2020 Nokia 219

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

Prior to 20.5 Post 20.5

Name=N/A, VM_UUID=N/A, MOC_
ID=MOC_ID, MOC_NAME=MOC_
NAME, ves_nfcNamingCode=EXT-
ves_nfcNamingCode, ves_
nfNamingCode=EXT-ves_nfNamingCode,
ves_sourceType=port, ves_
category=EXT-ves_category,
ves_alarminterface=EXT-ves_
alarminterface, Routing_Key=, MOC_
Name=, source=}
system = "CSD"
host = "Active-IO#DDE"
service = "service"
timezone = "Asia/Kolkata"
time = "2020-05-27T22:49:33.
048Z"
type = "alarm"
level = "1"
facility = "informational"
process = "Active-IO#DDE#Peer
disconnect detected peer=server1
ipaddr=135.2.76.235 port=5002
peerid=378456044 protocol=TCP"
systemid = "systemid"
container = "container"
version = "1.0"
isResolvable = true
event = RAISED
cause =
alarmProbableCauseUndefined
alarmText = null
timeStamp = 1
]

7.2.33 Changes in Peer disconnect detected alarm

Severity of the Peer disconnect detected alarm is now Major.

Prior to 20.5 Post 20.5

Severity - Minor. Severity - Major.

Table 76: Severity of Peer disconnect disconnected alarm

Note: From the release 20.5 onwards, the Peer disconnect detected alarm with the severity
Minor is deprecated.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 220

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

7.2.34 Changes in Brevity control for Peer disconnect detected alarm

Prior to 20.2 Post 20.2

Brevity control - Whenever the peer bounce Brevity control - Whenever the peer bounce
is observed for the same peer for more than is observed for the same peer for more than
two times in the duration a five-minutes, then ten times in the duration of two minutes, then
only two Peer disconnect detected alarms and only ten Peer disconnect detected alarms and
its corresponding clear alarm aretriggered. its corresponding clear alarms are triggered.
Next subsequent alarms for the same peer are Next subsequent alarms for the same peer are
suppressed due to brevity control. suppressed for the duration of two minutes due to
brevity control.

Table 77: Changes in Brevity control for Peer disconnect detected alarm

7.2.35 MariaDB and Zabbix alarms are deprecated

The following MariaDB and Zabbix alarms are deprecated:

• Local Database Connection Failure

• MariaDB Node Failure
• Process is down: mariadb.service

7.3 Troubleshooting and debugging commands

This section lists all the commands used for debugging and troubleshooting the issues in CSD.

Command Description

healthMachines Enables you to fetch the process status.

DDE_logLevel Enables you to collect the application logs.

aladmin list Enables you to verify the alarms.

systemctl | egrep "failed| Enables you to view the inactive or failed

activating|inactive" processes.

df -h Enables you to verify the cinder space.

SaveLogFiles Enables you to collect the logs and prometheus

data.

list-diameter-peers -s -a Enables you to view the peers connected from IO

and App nodes.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 221

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

Command Description

list-diameter-routes -d Enables you to view the details of available

routes.

log in to the DB node as a asadm and get the Enables you to verify the database cinder and
output of info memory usage.

Grafana To verify the peers connected, TPS, VMs, CPU

or memory usage, and peer latency.

/opt/tpa/java/bin/jmap - Enables you to get the heapdump from IO and

dump:live,file=/opt/tpa/logs/xyz. App nodes.
bin `cat /opt/tpa/osgi/var/tmp/pids/
Note: It is not recommended to collect on
instance.pid`
production unless there is emergency.

Table 78: Debugging commands

Command Description

/var/log/cbam/workflows.log Enables you to fetch the VNFC workflow log form

CBAM.

/var/log/ha/ha.log Specifies the exact time of switchover.

/tmp/UpgradeBackups/db_backup.log Provides logs for upgrade or fallback failures.

/tmp/UpgradeBackups/uninstall.log Provides logs for upgrade or fallback failures.

/tmp/UpgradeBackups/install.log Provides logs for upgrade or fallback failures.

/tmp/UpgradeBackups/db_restore.log Provides logs for upgrade or fallback failures.

/opt/tpa/aerospike/etc/aerospike/ Provides logs for database related issues from

aerospike.conf active node.

/opt/tpa/logs/aerospike.log Provides logs for database related issues from

active node.

/opt/tpa/logs/aerospike_warning.log Provides logs for database related issues from

active node.

/var/log/messages Provides the logs for switchover related issues

from both the nodes.

Provisioning.log Provides the logs for SMGUI issues related to

provisioning.

UserAccess.log Provides the logs for SMGUI issues related to

user login.

id9YZ-09148-MT11-PCZZA © 2020 Nokia 222

2.0
Monitoring and Troubleshooting Guide Changes in CSD release

Command Description

ImportExport.log Provides the logs for SMGUI issues related to

Import Export function.

ManagedElement.log Provides the logs for SMGUI issues related to

managed elements.

/opt/tpa/logs Provides complete logs for SM GUI related

issues.

/var/log/calm/alma.log Enables you to view the alarm history.

DDESMApplication.log Enables you to view the active exceptions or

errors.

jstatWD.log Enables you to debug the heap memory growth

or issues.

gc.log Enables you to verify the behavior and graph of

garbage collection.

Table 79: Commands to collect logs

id9YZ-09148-MT11-PCZZA © 2020 Nokia 223

2.0