NMS Redundancy and Failover Technical Note iDX 4.1.

NMS Redundancy and Failover

Technical Note

iDX Release 4.1.x

November 15, 2017

Copyright © 2017, Inc. All rights reserved. Reproduction in whole or in part without permission is prohibited.
Information contained herein is subject to change without notice. The specifications and information regarding the
products in this document are subject to change without notice. All statements, information and recommendations
in this document are believed to be accurate, but are presented without warranty of any kind, express, or implied.
Users must take full responsibility for their application of any products. Trademarks, brand names and products
mentioned in this document are the property of their respective owners. All such references are used strictly in an
editorial fashion with no intent to convey any affiliation with the name or the product's rightful owner.

VT iDirect is a global leader in IP-based satellite communications providing technology and solutions that enable our
partners worldwide to optimize their networks, differentiate their services and profitably expand their businesses.
Our product portfolio, branded under the name iDirect, sets standards in performance and efficiency to deliver
voice, video and data connectivity anywhere in the world. VT iDirect is the world’s largest TDMA enterprise VSAT
manufacturer and is the leader in key industries including mobility, military/government and cellular backhaul.

VT iDirect
Company Web site: http://www.idirect.net ~ Main Phone: 703.648.8000
TAC Contact Information: Phone: 703.648.8151 ~ Email: [email protected] ~ Web site: http://tac.idirect.net

iDirect Government™, created in 2007, is a wholly owned subsidiary of iDirect and was formed to better serve the
U.S. government and defense communities.

iDirect Government™
Company Web site: http://www.idirectgov.com ~ Main Phone: 703.648.8118
TAC Contact Information: Phone: 703.648.8111 ~ Email: [email protected] ~ Web site: http://tac.idirectgov.com

Document Name: TN_NMS_Redundancy_T0000959_iDX 4.1.x_Rev A_11152017.pdf

Document Part Number: T0000959

ii NMS Redundancy and Failover Technical Note

iDX 4.1.x | T0000959 | Rev A
 Revision History

Revision History

The following table shows all revisions for this document. To determine if this is the latest
revision, check the Technical Assistance Center (TAC) Web site. Refer to Getting Help on
page ix for TAC access information.

Revision Date Updates

A 11/15/2017 Initial release of document.

NMS Redundancy and Failover Technical Note iii

iDX 4.1.x | T0000959 | Rev A
Revision History

iv NMS Redundancy and Failover Technical Note

iDX 4.1.x | T0000959 | Rev A
 Contents

Contents

Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii

About . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Document Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .viii
Getting Help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Document Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Chapter 1 Single NMS Redundancy and Failover . . . . . . . . . . . . . . . . . . . .1

1.1 Configure Single NMS Server Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Configure CRONTAB on the Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1.1 Configure CRONTAB on the Primary NMS . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1.2 Configure CRONTAB on Backup NMS . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.2 Verify System Backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Configure Single NMS Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2.1 Configure Network Scripts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Chapter 2 Distributed NMS Redundancy and Failover . . . . . . . . . . . . . . . . 9

2.1 Distributed NMS Redundancy and Replication . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Configure Distributed NMS Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.1 Configure CRONTAB on the Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.1.1 Configure CRONTAB on the Primary NMS . . . . . . . . . . . . . . . . . . . . . 10
2.2.1.2 Configure CRONTAB on NMS 2 and NMS 3. . . . . . . . . . . . . . . . . . . . . 11

NMS Redundancy and Failover Technical Note v

iDX 4.1.x | T0000959 | Rev A
Contents

2.2.1.3 Configure CRONTAB on the Backup NMS . . . . . . . . . . . . . . . . . . . . . 12

2.2.2 Verify System Backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3 Configure Distributed NMS Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3.1 Prepare Backup NMS Server to Take Over. . . . . . . . . . . . . . . . . . . . . . . . 14
2.3.2 Configure Backup to Take Over for Primary . . . . . . . . . . . . . . . . . . . . . . 15
Edit eth0 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3.3 Backup NMS Server Takes Over for Auxiliary Server . . . . . . . . . . . . . . . . . 16

vi NMS Redundancy and Failover Technical Note

iDX 4.1.x | T0000959 | Rev A
 About

About

Purpose
This technical note presents the following procedures:
• Configuring the crontab files for redundancy in a single NMS network.
• Configuring the crontab files for redundancy in a distributed NMS network.
• Bringing the Backup NMS server online in single NMS network.
• Bringing the Backup NMS server online in a distributed NMS network.

Audience
This technical note is written for network operators using the iDX 4.1.x software suite
(iBuilder/iMonitor/iSite), network architects, and any other personnel who operate or
monitor an iDirect network. A basic knowledge of TCP/IP concepts, satellite communications,
Linux and MS Windows operating systems and tools (including WinSCP and PuTTY) is assumed.
Prior experience with the operation and monitoring of an iDirect network is desirable but not
required.

Contents
This document contains the following major sections:
• Single NMS Redundancy and Failover
• Distributed NMS Redundancy and Failover

NMS Redundancy and Failover Technical Note vii

iDX 4.1.x | T0000959 | Rev A
About

Document Conventions
This section illustrates and describes the conventions used throughout this document.

Convention Description Example

Command Used when the user is required to Type the command:
type a command at a command cd /etc/snmp/
line prompt or in a console.
Terminal Used when showing resulting crc report all
Output output from a command that was 8350.3235 : DATA CRC [ 1]
entered at a command line or on a 8350.3502 : DATA CRC [5818]
console. 8350.4382 : DATA CRC [ 20]
Screen Used when referring to text that 1. To add a remote to an inroute group, right-click
Reference appears on the screen on a the Inroute Group and select Add Remote.
Graphical User Interface (GUI). The Remote dialog box has a number of user-
Used when specifying names of selectable tabs across the top. The Information
commands, menus, folders, tabs, tab is visible when the dialog box opens.
dialogs, list boxes, and options.
Hyperlink Used to show all hyperlinked text For instructions on adding a line card to the
within a document or external network tree, see Adding a Line Card on
links such as web page URLs. page 108.

WARNING: A warning highlights an essential operating or maintenance procedure,

practice, condition, or statement which, if not strictly observed, could result in
injury, death, or long term health hazards.

CAUTION: A caution highlights an essential operating or maintenance procedure,

practice, condition, or statement which, if not strictly observed, could result in
damage to, or destruction of, equipment or a condition that adversely affects
system operation.

NOTE: A note is a statement or other notification that adds, emphasizes, or

clarifies essential information of special importance or interest.

viii NMS Redundancy and Failover Technical Note

iDX 4.1.x | T0000959 | Rev A
 About

Getting Help
The iDirect Technical Assistance Center (TAC) and the iDirect Government Technical
Assistance Center (TAC) are available to provide assistance 24 hours a day, 365 days a year.
Software user guides, installation procedures, FAQs, and other documents that support iDirect
and iDirect Government products are available on the respective TAC Web site:
• Access the iDirect TAC Web site at http://tac.idirect.net
• Access the iDirect Government TAC Web site at http://tac.idirectgov.com
The iDirect TAC may be contacted by telephone or email:
• Telephone: 703.648.8151
• E-mail: [email protected]
The iDirect Government TAC may be contacted by telephone or email:
• Telephone: 703.648.8111
• Email: [email protected]
iDirect and iDirect Government produce documentation that are technically accurate, easy to
use, and helpful to our customers. Please assist us in improving this document by providing
feedback. Send comments to:
• iDirect: [email protected]
• iDirect Government: [email protected]
For sales or product purchasing information contact iDirect Corporate Sales at the following
telephone number or e-mail address:
• Telephone: 703.648.8000
• E-mail: [email protected]

Document Set
The following iDirect documents are available on the TAC Web site and contain information
relevant to installing and using iDirect satellite network software and equipment.
• iDX 4.1.x Release Notes
• iDX 3.5.x and Later to iDX 4.1.x Network Upgrade Procedures
• iDX iBuilder User Guide
• iDX iMonitor User Guide
• iDX Technical Reference Guide
• iDX Satellite Router Commissioning and Installation Guide
• iDX Features and Chassis Licensing Guide
• iDX Upgrade/New Installation Survey
• iDX Link Budget Analysis Guide
• iDX Hardware Matrix
• iDX Software Release Matrix

NMS Redundancy and Failover Technical Note ix

iDX 4.1.x | T0000959 | Rev A
About

x NMS Redundancy and Failover Technical Note

iDX 4.1.x | T0000959 | Rev A
1 Single NMS Redundancy
and Failover

This chapter describes how to configure the Primary NMS to back up the databases and how to
bring the Backup NMS server online in the event that the Primary NMS fails. It includes the
following major topics:
• Configure Single NMS Server Redundancy
• Configure Single NMS Failover
Beginning with iDX Release 3.1, iDirect supports near real-time replication of NMS databases
from the Primary NMS server to one or more Backup NMS servers in addition to the traditional
idsBackup and idsRestore. When the NMS Database Replication feature is enabled, changes to
the NMS databases on the Primary NMS are automatically replicated to the Backup NMS server
and the idsBackup script is automatically configured to run on the Backup server. For a
complete description of the NMS Database Replication feature, including configuration, see
the Technical Reference Guide for iDX 4.1.x.
If the NMS Database Replication feature is currently being used, the sections in this document
on configuring single and distributed NMS redundancy are not applicable. However, there are
slight differences to the procedures to bring the Backup NMS server on line in case of a
Primary NMS failure. These differences are noted in Configure Single NMS Failover on page 5
and in Configure Distributed NMS Failover on page 14.

NMS Redundancy and Failover Technical Note 1

iDX 4.1.x | T0000959 | Rev A
Configure Single NMS Server Redundancy

1.1 Configure Single NMS Server Redundancy

This section describes how to configure redundancy in a single NMS network.

NOTE: This procedure is not applicable if NMS Database Replication feature is

enabled on the NMS Servers.

1.1.1 Configure CRONTAB on the Servers

The crontab file must be configured in order to correctly run the idsBackup and idsRestore
scripts.

1.1.1.1 Configure CRONTAB on the Primary NMS

Perform the following on the Primary NMS:
1. Log on to the Primary NMS and switch to root.
2. Edit the crontab file by entering the command:
crontab -e
The crontab file opens in the Vi editor.
3. On the Primary NMS, remove the comment symbol (#) from the beginning of all lines in
the crontab file, except for this line:
#30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --
keep 2
4. Add the following line to the crontab file, replacing the IP address shown with the Backup
NMS server’s address, and, if desired, filename tokens may be personalized:
30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --inst
[email protected]:/var/idirect/backup/\%ip-\%date --keep 2

NOTE: The idsBackup command creates a local backup, and when using the
“--inst” option, it creates a remote backup and installs it on the specified
NMS.

NOTE: A backslash character (\) is required before the token “%” character
for the crontab to execute properly. For example, use “\%ip” and not “%ip”

5. Optionally, change the time and frequency with which backups are performed by
modifying the string 30 0 * * * (highlighted above). This is a five-field string set by
default to run a backup every night at 12:30am. The time and frequency of the backups
can be modified by changing these fields according to the following definitions. The fields
below are numbered left to right.
• Field 1 (00 by default): Minute of the specified Hour (0 - 59)
• Field 2 (1 by default): Hour of the Day (0 - 23)
• Field 3 (* by default): Day of the Month (1 - 31)
• Field 4 (* by default): Month (1 - 12)
• Field 5 (* by default): Day of the week (0 - 6, with Sunday = 0)

2 NMS Redundancy and Failover Technical Note

iDX 4.1.x | T0000959 | Rev A
 Configure Single NMS Server Redundancy

6. Optionally, the number of local backup copies kept can be changed by modifying
--keep 2, for example, to --keep 1 to save disk space.
7. Save changes to the file and exit by pressing Esc and entering:
:wq!
An example of an edited crontab file for the Primary NMS is shown below.
0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >>
/opt/idirect/nms/utils/db_maint/cons.output 2>&1
0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >>
/opt/idirect/nms/utils/db_maint/check_db.output 2>&1
#30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --
keep 2
30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --inst
[email protected]:/var/idirect/backup/\%ip-\%date --keep 2

1.1.1.2 Configure CRONTAB on Backup NMS

Perform the following steps on the Backup NMS:
1. Log on to the Backup NMS and switch to root.
2. Edit the crontab file by entering the command:
crontab -e
The crontab file opens in the Vi editor.
3. Remove the comment symbol (#) from the following line:
0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >>
/opt/idirect/nms/utils/db_maint/check_db.output 2>&1
This command performs a recommended, daily maintenance check on the database.
4. Ensure comment symbols (#) appear in front of the following lines:
#0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >>
/opt/idirect/nms/utils/db_maint/cons.output 2>&1
#30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --
keep 2
5. Save changes to the file and exit by pressing Esc and entering:
:wq!
An example of an edited crontab file for the Backup NMS is shown below:
#0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >>
/opt/idirect/nms/utils/db_maint/cons.output 2>&1
0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >>
/opt/idirect/nms/utils/db_maint/check_db.output 2>&1
#30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --
keep 2

NMS Redundancy and Failover Technical Note 3

iDX 4.1.x | T0000959 | Rev A
Configure Single NMS Server Redundancy

1.1.2 Verify System Backup

This procedure provides all necessary steps to verify the system backup script is in working
condition and that the appropriate backups are made on the Backup NMS server.
The CRONTAB time setting can be modified to run the commands 1-3 minutes after the
current time. Be sure to check the idsBackup.log to see if the backup and install is running
properly.
Perform the following:
1. On the Primary NMS, log on as idirect and switch to root.
2. Enter the following command to push SSH keys to the Backup NMS:
pushSSHKeys <Backup NMS IP address>
The following sample output displays:
Pushing SSH key for user root to [email protected]:
The authenticity of host ’172.20.4.253 (172.20.4.253)’ can’t be
established.
RSA key fingerprint is 3e:28:f2:a5:50:22:a5:2f:51:33:69:39:ca:a4:8d:63.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ’172.20.4.253’ (RSA) to the list of known hosts.
[email protected]’s password: remote_users_password
SSH key for user root successfully pushed to [email protected]
3. On the Primary NMS, get the current time.
date
Tue Aug 31 15:21:45 UTC 2010
4. In the crontab file, set the idsBackup command to run two minutes from the current
time.
crontab -e
0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >>
/opt/idirect/nms/utils/db_maint/cons.output 2>&1
0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >>
/opt/idirect/nms/utils/db_maint/check_db.output 2>&1
#30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --
keep 2
23 15 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --
inst [email protected]:/var/idirect/backup/\%ip-\%date --keep 2
5. Check the idsBackup.log to monitor if the commands ran as expected.
tail –f /var/idirect/log/idsBackup.log
6. Restore the original settings in the crontab file.

4 NMS Redundancy and Failover Technical Note

iDX 4.1.x | T0000959 | Rev A
 Configure Single NMS Failover

1.2 Configure Single NMS Failover

NOTE: This procedure is not applicable if NMS Database Replication feature is

enabled on the NMS Servers.

This procedure describes how to bring a Backup NMS on line if the Primary NMS fails.
Perform the following:
1. If the primary NMS is still accessible, reconfigure the primary NMS eth0 interface with
another unused IP address and physically unplug the eth0 interface. If the interface has
been reconfigured, the NMS server must be rebooted. Restart the network services by
entering the following command at the prompt:
service network restart
2. If the primary NMS remains connected to the network, the iDirect NMS services MUST be
shutdown by entering the following commands and the IP address must be different.
Additionally, these services MUST remain down while the other new (backup) NMS server
is online.
service idirect_nms stop
service idirect_nms stop nms_config
cd /etc/init.d
chkconfig --level 2345 idirect_nms off
3. Log on to the Backup NMS server as idirect and switch to root.
4. Verify that the routing configuration on the Backup NMS server is correct. Any static
routes that were configured on the Primary NMS server should also be configured on the
Backup NMS server. Make sure they are configured as persistent routes so they remain in
effect after a reboot.
5. If NMS Database Replication is enabled, enter the following command on the Backup NMS
server to stop the server from being a MySQL slave:
cr8DbSlave -d
6. Proceed to Configure Network Scripts.

NMS Redundancy and Failover Technical Note 5

iDX 4.1.x | T0000959 | Rev A
Configure Single NMS Failover

1.2.1 Configure Network Scripts

Perform the following steps to configure network interface configuration:
1. Go to the /etc/sysconfig/network-scripts directory.
cd /etc/sysconfig/network-scripts
2. Using the Vi editor, open the ifcfg-eth0 file for editing by entering the following
command:
vi ifcfg-eth0
3. Change the command lines in the ifcfg-eth0 file as shown in Table 1-1 (do not change any
other values, and retain all other lines within this file).

NOTE: Command lines are case-sensitive.

Table 1-1. ifcfg-eth0 File Entries

Command Description
DEVICE eth0
BOOTPROTO none
DHCPCLASS
HWADDR *DO NOT CHANGE*
IPADDR upstream IP address of the Primary NMS server
NETMASK subnet mask of the NMS server
GATEWAY IP address of the upstream router interface
ONBOOT yes
MTU 1500

An example of a correctly configured network-script file for eth0 is shown below.

# Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express
DEVICE=eth0
BOOTPROTO=none
DHCPCLASS=
HWADDR=00:14:5E:7E:84:0C
IPADDR=192.168.76.81
NETMASK=255.255.255.128
GATEWAY=192.168.76.126
ONBOOT=yes
MTU=1500

CAUTION: Do not delete any lines from the existing file. Doing so may have an
adverse effect on network performance.

6 NMS Redundancy and Failover Technical Note

iDX 4.1.x | T0000959 | Rev A
 Configure Single NMS Failover

4. Record the hardware address (HWADDR) as displayed in the example above.

5. Save the file and exit the Vi editor.
:wq!
6. Contact the TAC to see if a new license file is needed. Typically, the primary and backup
NMS server MAC addresses are listed in the license file, so a new license file is not
required.
7. Return to the /etc/sysconfig directory.
cd /etc/sysconfig
8. Using the vi editor, open the network file for editing by entering the following command:
vi network
An example of the network file is shown below:
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=Primary_NMS
GATEWAYDEV=eth0
9. Change the command lines in the network-scripts file as shown in Table 1-2 (do not change
any other values, and retain all other lines within this file).

Table 1-2. Network File Commands

Command Description
NETWORKING yes
NETWORKING_IPV6 no
HOSTNAME COMPANY_NMS_Primary
GATEWAYDEV eth0

10. Save the file and exit the editor by entering the following command:
:wq!
11. Go to the /etc directory by entering the following command:
cd /etc
12. Using the Vi editor, open the hosts file for editing by entering the following command:
vi hosts
13. Add the following lines to the hosts file, as applicable to the IP addressing scheme being
implemented. The two lines for 127.0.0.1 must be present. Enter the following command
lines:
127.0.0.1 COMPANY_NMS_Primary localhost.localdomain localhost
192.168.1.3 COMPANY_NMS_Primary
The hosts file displays as shown in the example.
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 COMPANY_NMS_Primary localhost.localdomain localhost
192.168.1.3 COMPANY_NMS_Primary
14. Save the file and exit the editor by entering the following command:
:wq!

NMS Redundancy and Failover Technical Note 7

iDX 4.1.x | T0000959 | Rev A
Configure Single NMS Failover

15. Configure the NMS services to start automatically at boot time by entering the following
command:
chkconfig idirect_nms on
16. Reboot the Backup NMS.
The backup NMS becomes primary upon boot up. It may take a few minutes for the upstream
router to update the ARP tables. After this update is complete, IP connectivity to the server
is restored.

8 NMS Redundancy and Failover Technical Note

iDX 4.1.x | T0000959 | Rev A
 Distributed NMS Redundancy and Replication

2 Distributed NMS
Redundancy and
Failover
Special consideration is required for hubs using a Distributed NMS. If the current system is a
Distributed NMS, then additional configuration changes must be made on the Primary NMS
server, Auxiliary servers, and Backup server (event server (evtsvr), latency server (latsvr),
and the archive server (nrdsvr). This chapter includes major topic discussion and process for:
• Distributed NMS Redundancy and Replication
• Configure Distributed NMS Redundancy
• Configure Distributed NMS Failover

2.1 Distributed NMS Redundancy and Replication

Beginning with iDX Release 3.1, iDirect supports near real-time replication of NMS databases
from the Primary NMS server to one or more Backup NMS servers in addition to the traditional
idsBackup and idsRestore. When enabling the NMS Database Replication feature, changes to
the NMS databases on the Primary NMS are automatically replicated to the Backup NMS server
and the idsBackup script is automatically configured to run on the Backup server.
On a distributed NMS with NMS Database Replication enabled, each NMS server replicates its
database to a different backup machine. Assuming a standard DNMS setup with all databases
being replicated, the system consists of one Primary NMS server (NMS 1), two Auxiliary NMS
servers (NMS 2 and NMS 3), and three Backup NMS servers. During normal operation, each of
the three on line Servers (NMS 1, NMS 2 and NMS 3) replicates its database to the
corresponding backup server. (For details, see “NMS Database Replication on a Distributed
NMS” in the “NMS Database Replication” chapter of the Technical Reference Guide
for iDX 4.1.x).
If a Primary or an Auxiliary NMS server fails, it is possible to switch to the corresponding
Backup NMS server by disabling the failed server and reconfiguring the Backup server to take
its place. This procedure is identical to the procedure for a Single NMS failure. Therefore,
follow the procedure in Configure Distributed NMS Failover on page 14.

NOTE: Do not forget to execute Step 5 of Section 1.2, Configure Single NMS
Failoverto disable Database Replication feature on the Backup NMS server before
bringing it up as the active server.

NOTE: If NMS Database Replication feature is enabled on the Distributed NMS

servers, then the remaining procedures in this chapter are not applicable.

NMS Redundancy and Failover Technical Note 9

iDX 4.1.x | T0000959 | Rev A
Configure Distributed NMS Redundancy

For a complete description of the NMS Database Replication feature see the Technical
Reference Guide for iDX 4.1.x.

2.2 Configure Distributed NMS Redundancy

This section describes how to configure redundancy in a distributed NMS network.

CAUTION: Do not perform this procedure if Database Replication feature is enable

on the NMS servers. See Distributed NMS Redundancy and Replication on page 9.

2.2.1 Configure CRONTAB on the Servers

The crontab file must be reconfigured in order to correctly run the idsBackup and idsRestore
scripts.

2.2.1.1 Configure CRONTAB on the Primary NMS

On the Primary NMS, perform the following:
1. Log on to the Primary NMS and switch to the root account.
2. Edit the crontab file by entering the command:
crontab -e
The crontab file opens in the Vi editor.
3. On the Primary NMS, remove the comment symbol (#) from the beginning of all lines in
the crontab file, except for this line:
#30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --
keep 2
4. Add the following line to the crontab file, replacing the IP address shown with the Backup
NMS server’s address, and, if desired, filename tokens shown may be personalized:
30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --inst
[email protected]:/var/idirect/backup/\%ip-\%date --keep 2

NOTE: The idsBackup command creates a local backup, and when using the
“inst” option, it creates a remote backup and installs it on the specified
NMS.

NOTE: A backslash character (\) is required before the token “%” character
for the crontab to execute properly. For example, use “\%ip” and not “%ip”

10 NMS Redundancy and Failover Technical Note

iDX 4.1.x | T0000959 | Rev A
 Configure Distributed NMS Redundancy

5. Optionally, change the time and frequency with which backups are performed by
modifying the string 30 0 * * * (highlighted above). This is a five-field string set by
default to run a backup every night at 12:30am. The time and frequency of the backups
can be modified by changing these fields according to the following definitions. The fields
below are numbered left to right.
• Field 1 (00 by default): Minute of the specified Hour (0 - 59)
• Field 2 (1 by default): Hour of the Day (0 - 23)
• Field 3 (* by default): Day of the Month (1 - 31)
• Field 4 (* by default): Month (1 - 12)
• Field 5 (* by default): Day of the week (0 - 6, with Sunday = 0)
6. Optionally, the number of local backup copies kept can be changed by modifying --keep
2, for example, to --keep 1 to save disk space.
7. Save changes to the file and exit by pressing Esc and entering:
:wq!
An example of an edited crontab file for the Primary NMS is shown below.
0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >>
/opt/idirect/nms/utils/db_maint/cons.output 2>&1
0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >>
/opt/idirect/nms/utils/db_maint/check_db.output 2>&1
#30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --
keep 2
30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --inst
[email protected]:/var/idirect/backup/\%ip-\%date --keep 2

2.2.1.2 Configure CRONTAB on NMS 2 and NMS 3

A distributed NMS has a special backup configuration that must be configured in the crontab
file.
Perform the following steps on NMS 2 and NMS 3 of the Distributed NMS:
1. Log on to the Auxiliary NMS (NMS 2 or NMS 3) as idirect and switch to the root account.
2. Edit the crontab file by entering the command:
crontab -e
The crontab file opens in the Vi editor.
3. On the Auxiliary NMS, remove the comment symbol (#) from the beginning of all lines in
the crontab file, except for this line:
#30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --
keep 2
4. Add the following line to the crontab file, replacing the IP address shown with the NMS
server’s address, and, if desired, filename tokens may be personalized:
30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --bkup
[email protected]:/var/idirect/backup/\%ip-\%date --keep 2

NMS Redundancy and Failover Technical Note 11

iDX 4.1.x | T0000959 | Rev A
Configure Distributed NMS Redundancy

5. Save changes to the file and exit by pressing Esc and entering:
:wq!
An example of an edited crontab file is shown below.
0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >>
/opt/idirect/nms/utils/db_maint/cons.output 2>&1
0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >>
/opt/idirect/nms/utils/db_maint/check_db.output 2>&1
#30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --
keep 2
30 2 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --bkup
[email protected]:/var/idirect/backup/\%ip-\%date --keep 2

2.2.1.3 Configure CRONTAB on the Backup NMS

Perform the following:
1. Log on to the Backup NMS and switch to the root account.
2. Edit the crontab file by entering the command:
crontab -e
The crontab file opens in the Vi editor.
3. Remove the comment symbol (#) from the following line:
0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >>
/opt/idirect/nms/utils/db_maint/check_db.output 2>&1
This command performs a recommended, daily maintenance check on the database.
An example of an edited crontab file for the Backup NMS is shown below.
#0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >>
/opt/idirect/nms/utils/db_maint/cons.output 2>&1
0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >>
/opt/idirect/nms/utils/db_maint/check_db.output 2>&1
#30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --
keep 2

12 NMS Redundancy and Failover Technical Note

iDX 4.1.x | T0000959 | Rev A
 Configure Distributed NMS Redundancy

2.2.2 Verify System Backup

This procedure provides all necessary steps to verify the system backup script is working and
that the appropriate backups are made on the Backup NMS server.
The time in crontab can be modified to run the commands 1-3 minutes after the current time,
and then check the idsBackup.log to see if the backup and install is running properly.
1. Log on to the Primary NMS and switch to the root account.
2. Enter the following command to push SSH keys to the Backup NMS:
pushSSHKeys <Backup NMS IP address>
The following sample output displays:
Pushing SSH key for user root to [email protected]:
The authenticity of host ’172.20.4.253 (172.20.4.253)’ can’t be
established.
RSA key fingerprint is 3e:28:f2:a5:50:22:a5:2f:51:33:69:39:ca:a4:8d:63.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ’172.20.4.253’ (RSA) to the list of known hosts.
[email protected]’s password: remote_users_password
SSH key for user root successfully pushed to [email protected]
3. Get the current time by entering the following command:
date
Tue Aug 31 15:21:45 UTC 2010
4. In the crontab file, set the idsBackup command to run two minutes from the current time
by entering the following command:
crontab -e
0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >>
/opt/idirect/nms/utils/db_maint/cons.output 2>&1
0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >>
/opt/idirect/nms/utils/db_maint/check_db.output 2>&1
#30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --
keep 2
23 15 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --
inst [email protected]:/var/idirect/backup/\%ip-\%date --keep 2
5. Check the idsBackup.log to monitor if the commands executed successfully, by entering
the following command:
tail –f /var/idirect/log/idsBackup.log
6. Restore the original settings in the crontab file.
7. In a Distributed NMS configuration, repeat Step 1 through Step 6 on each Auxiliary NMS.

NOTE: On the Auxiliary NMS servers, the line to edit ends in --bkup rather than
--inst as shown below.
23 15 * * * /usr/bin/idsBackup --bkup

NMS Redundancy and Failover Technical Note 13

iDX 4.1.x | T0000959 | Rev A
Configure Distributed NMS Failover

2.3 Configure Distributed NMS Failover

This section describes how to assign the Backup NMS server to take over for either the Primary
NMS Server or for an Auxiliary NMS Server.

CAUTION: Do not perform this procedure if Database Replication feature is

enabled on the NMS servers. See Distributed NMS Redundancy and Replication on
page 9.

2.3.1 Prepare Backup NMS Server to Take Over

Perform the following to prepare for the Backup NMS to take over:
1. If the primary NMS is still accessible, reconfigure the primary NMS eth0 interface with
another unused IP address and physically unplug the eth0 interface. If the interface has
been reconfigured, the NMS server must be rebooted. Additionally, the network services
can be restarted by entering the following command at the prompt, by entering the
following command:
service network restart
2. If the primary NMS remains connected to the network, the iDirect NMS services MUST be
shut down by entering the following command and the IP address must be different.
Additionally, these services MUST remain down while the other new (backup) NMS server
is online:
service idirect_nms stop
service idirect_nms stop nms_config
cd /etc/init.d
chkconfig --level 2345 idirect_nms off
3. Verify that the routing configuration on the Backup NMS server is correct. If RIPv2 was
running on the Primary NMS server, it has to be running on the backup. Any static routes
that were configured on the Primary NMS Server need to be configured on the Backup NMS
server. Make sure they are configured as persistent routes so they remain in effect after a
reboot.
4. If NMS Database Replication is enabled, enter the following command on the Backup NMS
server to stop the server from being a MySQL slave:
cr8DbSlave -d

14 NMS Redundancy and Failover Technical Note

iDX 4.1.x | T0000959 | Rev A
 Configure Distributed NMS Failover

5. Update the database to enable archiving real-time data in the NRD_archive bys entering
the following commands:
a. Open the NMS database using MySQL:
mysql nms
b. Enter the following MySQL command
UPDATE nms.config SET ip_archive_flag = '1', timeplan_archive_flag =
'1', ucp_archive_flag = '1', rem_stats_archive_flag = '1',
modem_state_change_archive_flag = '1', event_msg_archive_flag = '1',
hub_status_msg_archive_flag = '1', pp_state_change_archive_flag = '1',
lat_stats_archive_flag = '1', chassis_state_change_archive_flag = '1',
ota_archive_flag = '1', sl_qos_archive_flag = '1',
group_qos_archive_flag = '1', dvbs2_hub_archive_flag = '1',
dvbs2_pp_archive_flag = '1', dvbs2_remote_archive_flag = '1',
remote_rx_archive_flag = '1';
a. Exit out of MySQL.
exit;
b. Enter the following command
/home/nms/utils/db_maint/NMS-domain-commands.pl -exec="restart"

2.3.2 Configure Backup to Take Over for Primary

This section provides steps to bring the Backup NMS server on-line to replace the Primary
server.

Edit eth0 Configuration

Configure the Backup NMS Server eth0 interface with the primary NMS original eth0 interface
address. The reason for this is to match the original primary NMS IP address that is configured
in the configuration (options) files of the iDirect network elements.
1. Log on to the Backup NMS and switch to the root account.
2. Change the IP address and hostname, and configure the NMS services to start
automatically at boot time by performing Configure Network Scripts on page 6.
3. Rebuild the Distributed NMS configuration. Refer to Appendix A of the iBuilder User Guide
(Section A.4 “Setting UP a Distributed NMS Environment” through Section A.6, “Granting
Read Permissions to NMS 2 and NMS 3”).
4. The network should now be running on the Backup NMS Server. The status on the Line
Cards and the Remotes should be normal in iMonitor.
5. If needed, apply the Protocol Processor level and network level configuration to all
networks via iBuilder.

NMS Redundancy and Failover Technical Note 15

iDX 4.1.x | T0000959 | Rev A
Configure Distributed NMS Failover

2.3.3 Backup NMS Server Takes Over for Auxiliary Server

To bring the backup NMS server on-line in as an Auxiliary server, perform the following:
1. Log on to the Backup NMS and switch to the root account.
2. Find the restore file by entering the following commands to change to the backup
directory and list the files:
cd /var/idirect/backup
ll
This example shows a list of files in the backup directory:
-rw-r--r-- 4 idirect idirect 49519268 Feb 27 00:31 192.168.76.81-
2012.02.27-00.30.01
-rw-r--r-- 4 idirect idirect 49653853 Feb 28 00:31 192.168.76.81-
2012.02.28-00.30.01
-rw-r--r-- 4 idirect idirect 45100228 Feb 27 00:31 192.168.76.82-
2012.02.27-00.30.01
-rw-r--r-- 4 idirect idirect 45100803 Feb 28 00:31 192.168.76.82-
2012.02.28-00.30.01
-rw-r--r-- 4 idirect idirect 29449298 Feb 27 00:31 192.168.76.83-
2012.02.27-00.30.01
-rw-r--r-- 4 idirect idirect 29443853 Feb 28 00:31 192.168.76.83-
2012.02.28-00.30.01
3. Identify the backup file to restore, and enter the following command:
/usr/bin/idsRestore --bkup /var/idirect/backup/<filename>

Where, <filename> is the name of the backup file.

When this commands executes, the idsRestore script runs and the Backup NMS server
becomes the Auxiliary server. In this example, it becomes NMS2. During the takeover
process, the IP address is changed to that of the Auxiliary server. It may take a few
minutes for the Upstream router ARP tables to update. When the update is complete, IP
connectivity to the server is restored.
4. Configure the NMS services to start automatically at boot time by entering the following
commands:
cd /etc/init.d
chkconfig idirect_nms on
5. Reboot the Backup NMS.
6. If needed, apply the Protocol Processor level and network level configuration to all
networks via iBuilder.
7. Ensure backup is working by performing Verify System Backup on page 13.

16 NMS Redundancy and Failover Technical Note

iDX 4.1.x | T0000959 | Rev A
iDirect
13861 Sunrise Valley Drive, Suite 300
Herndon, VA 20171-6126
+1 703.648.8000
+1 866.345.0983
www.idirect.net
Advancing a Connected World