Symantec NetBackup Troubleshooting Guide

UNIX, Windows, and Linux

Release 7.5

21220066

Symantec NetBackup Troubleshooting Guide

The software described in this book is furnished under a license agreement and may be used only in accordance with the terms of the agreement. Documentation version: 7.5 PN: 21220066

Legal Notice
Copyright 2012 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. This Symantec product may contain third party software for which Symantec is required to provide attribution to the third party (Third Party Programs). Some of the Third Party Programs are available under open source or free software licenses. The License Agreement accompanying the Software does not alter any rights or obligations you may have under those open source or free software licenses. Please see the Third Party Legal Notice Appendix to this Documentation or TPIP ReadMe File accompanying this Symantec product for more information on the Third Party Programs. The product described in this document is distributed under licenses restricting its use, copying, distribution, and decompilation/reverse engineering. No part of this document may be reproduced in any form by any means without prior written authorization of Symantec Corporation and its licensors, if any. THE DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID. SYMANTEC CORPORATION SHALL NOT BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES IN CONNECTION WITH THE FURNISHING, PERFORMANCE, OR USE OF THIS DOCUMENTATION. THE INFORMATION CONTAINED IN THIS DOCUMENTATION IS SUBJECT TO CHANGE WITHOUT NOTICE. The Licensed Software and Documentation are deemed to be commercial computer software as defined in FAR 12.212 and subject to restricted rights as defined in FAR Section 52.227-19 "Commercial Computer Software - Restricted Rights" and DFARS 227.7202, "Rights in Commercial Computer Software or Commercial Computer Software Documentation", as applicable, and any successor regulations. Any use, modification, reproduction release, performance, display or disclosure of the Licensed Software and Documentation by the U.S. Government shall be solely in accordance with the terms of this Agreement.

Symantec Corporation 350 Ellis Street Mountain View, CA 94043 http://www.symantec.com Printed in the United States of America. 10 9 8 7 6 5 4 3 2 1

Technical Support
Symantec Technical Support maintains support centers globally. Technical Supports primary role is to respond to specific queries about product features and functionality. The Technical Support group also creates content for our online Knowledge Base. The Technical Support group works collaboratively with the other functional areas within Symantec to answer your questions in a timely fashion. For example, the Technical Support group works with Product Engineering and Symantec Security Response to provide alerting services and virus definition updates. Symantecs support offerings include the following:

A range of support options that give you the flexibility to select the right amount of service for any size organization Telephone and/or Web-based support that provides rapid response and up-to-the-minute information Upgrade assurance that delivers software upgrades Global support purchased on a regional business hours or 24 hours a day, 7 days a week basis Premium service offerings that include Account Management Services

For information about Symantecs support offerings, you can visit our Web site at the following URL: www.symantec.com/business/support/ All support services will be delivered in accordance with your support agreement and the then-current enterprise technical support policy.

Contacting Technical Support

Customers with a current support agreement may access Technical Support information at the following URL: www.symantec.com/business/support/ Before contacting Technical Support, make sure you have satisfied the system requirements that are listed in your product documentation. Also, you should be at the computer on which the problem occurred, in case it is necessary to replicate the problem. When you contact Technical Support, please have the following information available:

Product release level

Hardware information Available memory, disk space, and NIC information Operating system Version and patch level Network topology Router, gateway, and IP address information Problem description:

Error messages and log files Troubleshooting that was performed before contacting Symantec Recent software configuration changes and network changes

Licensing and registration

If your Symantec product requires registration or a license key, access our technical support Web page at the following URL: www.symantec.com/business/support/

Customer service
Customer service information is available at the following URL: www.symantec.com/business/support/ Customer Service is available to assist with non-technical questions, such as the following types of issues:

Questions regarding product licensing or serialization Product registration updates, such as address or name changes General product information (features, language availability, local dealers) Latest information about product updates and upgrades Information about upgrade assurance and support contracts Information about the Symantec Buying Programs Advice about Symantec's technical support options Nontechnical presales questions Issues that are related to CD-ROMs, DVDs, or manuals

Support agreement resources

If you want to contact Symantec regarding an existing support agreement, please contact the support agreement administration team for your region as follows:
Asia-Pacific and Japan Europe, Middle-East, and Africa North America and Latin America [email protected] [email protected] [email protected]

Contents

Technical Support ............................................................................................... 4 Chapter 1 Introduction .......................................................................... 13

Troubleshooting a problem ............................................................ 13 Problem report for Technical Support .............................................. 15 About gathering information for NetBackup-Java applications ............. 17

Chapter 2

Troubleshooting procedures ............................................. 19

About troubleshooting procedures .................................................. Troubleshooting NetBackup problems ............................................. Verifying that all processes are running on UNIX servers .............. Verifying that all processes are running on Windows servers ........................................................................... Troubleshooting installation problems ............................................. Troubleshooting configuration problems .......................................... Device configuration problem resolution .......................................... Testing the master server and clients .............................................. Testing the media server and clients ................................................ Resolving network communication problems with UNIX clients ........... Resolving network communication problems with PC clients ............... About troubleshooting networks and hostnames ................................ Verifying host name and service entries in NetBackup ........................ Example of host name and service entries on UNIX master server and client ....................................................................... Example of host name and service entries on UNIX master server and media server .............................................................. Example of host name and service entries on UNIX PC clients ............................................................................ Example of host name and service entries on UNIX clients in multiple networks ............................................................ Example of host name and service entries on UNIX server that connects to multiple networks ............................................ About the bpclntcmd utility ........................................................... Using the Host Properties window to access configuration settings ................................................................................ 20 22 24 26 28 29 31 34 38 41 47 51 55 60 61 63 64 66 68 70

Contents

Resolving full disk problems .......................................................... 71 Frozen media troubleshooting considerations ................................... 72 Logs for troubleshooting frozen media ....................................... 72 About conditions that cause media to freeze ............................... 73 Resolving PBX problems ................................................................ 76 Checking PBX installation ........................................................ 77 Checking that PBX is running ................................................... 77 Checking that PBX is set correctly ............................................. 78 Accessing the PBX logs ............................................................ 79 Troubleshooting PBX security .................................................. 80 Determining if the PBX daemon or service is available .................. 81 About troubleshooting Auto Image Replication .................................. 82 Troubleshooting Auto Image Replication .................................... 82 About troubleshooting automatic import jobs .............................. 89 Troubleshooting network interface card performance ......................... 93 About SERVER entries in the bp.conf file .......................................... 94 About unavailable storage unit problems .......................................... 94 About troubleshooting NetBackup in a SAN environment .................... 95 NetBackup enterprise lifecycle best practices .............................. 96 About using CommandCentral Storage to troubleshoot NetBackup in a SAN environment ....................................................... 97 Using CommandCentral Storage to troubleshoot the inability to access drives or robots in a SAN environment ....................... 98 Using CommandCentral Storage to troubleshoot the inability to discover a drive or robot in a SAN environment ..................... 98 Using CommandCentral Storage to troubleshoot an intermittent drive failure in a SAN environment .................................... 100

Chapter 3

Using logs ............................................................................ 101

About logs ................................................................................. About UNIX system logs .............................................................. About unified logging .................................................................. Gathering unified logs for NetBackup ....................................... Types of unified logging messages ........................................... File name format for unified logging ........................................ Originator IDs for the entities that use unified logging ................ About changing the location of unified log files .......................... About rolling over unified log files ........................................... About recycling unified log files .............................................. About using the vxlogview command to view unified logs ............ About query strings used with the vxlogview command ............... Examples of using vxlogview to view unified logs ....................... 101 103 103 104 106 107 108 114 114 116 117 118 121

Contents

Examples of using vxlogmgr to manage unified logs ................... Examples of using vxlogcfg to configure unified logs .................. About legacy logging ................................................................... UNIX client processes that use legacy logging ............................ PC client processes that use legacy logging ................................ File name formats for legacy logging ........................................ Directory names for legacy debug logs for servers ..................... Directory names for legacy debug logs for media and device management ................................................................. How to control the amount of information written to legacy logging files ................................................................... About limiting the size and the retention of legacy logs ............... Configuring legacy log rotation ............................................... Creating legacy log directories to accompany problem reports for synthetic backup ....................................................... About global logging levels ........................................................... Changing the logging level ..................................................... Changing the logging level on Windows and NetWare clients .......................................................................... Setting debug logging to a higher level ..................................... Logs to accompany problem reports for synthetic backups ................. Setting retention limits for logs on clients ...................................... Logging options with the Windows Event Viewer ............................. Troubleshooting error messages in the NetBackup Administration Console for UNIX .................................................................. About extra disk space required for logs and temporary files ............................................................................. Enabling detailed debug logging ..............................................

122 125 127 128 130 133 134 136 137 138 139 140 141 143 143 144 144 145 146 148 149 150

Chapter 4

Using NetBackup utilities ................................................ 153

About NetBackup troubleshooting utilities ...................................... About the analysis utilities for NetBackup debug logs ........................ About network troubleshooting utilities ......................................... About the NetBackup support utility (nbsu) ..................................... Output from the NetBackup support utility (nbsu) ...................... Status code information gathered by the NetBackup support utility (nbsu) .................................................................. Example of a progress display for the NetBackup support utility (nbsu) ........................................................................... About the NetBackup consistency check utility (NBCC) ...................... Output from the NetBackup consistency check utility (NBCC) .......................................................................... 153 154 158 159 161 164 164 165 167

10

Contents

Example of an NBCC progress display ....................................... About the NetBackup consistency check repair (NBCCR) utility ........... About the nbcplogs utility ............................................................ About the robotic test utilities ...................................................... Robotic tests on UNIX ........................................................... Robotic tests on Windows ......................................................

168 174 176 177 178 179

Chapter 5

Disaster recovery ............................................................... 181

About disaster recovery ............................................................... Recommended backup practices .................................................... About disk recovery procedures for UNIX and Linux ......................... Recovering the master server disk for UNIX and Linux ................ About recovering the NetBackup media server disk for UNIX and Linux ........................................................................... Recovering the system disk on a UNIX client workstation ............ About clustered NBU server recovery for UNIX and Linux .................. Replacing a failed node on a UNIX or Linux cluster ..................... Recovering the shared disk on a UNIX or Linux cluster ................ Recovering the entire UNIX or Linux cluster .............................. About disk recovery procedures for Windows .................................. About recovering the master server disk for Windows ................. About recovering the NetBackup media server disk for Windows ....................................................................... Recovering a Windows client disk ............................................ About clustered NBU server recovery for Windows ........................... Replacing a failed node on a Windows VCS cluster ...................... Recovering the shared disk on a Windows VCS cluster ................ Recovering the entire Windows VCS cluster .............................. About recovering the NetBackup catalog ......................................... About catalog recovery and OpsCenter ..................................... About recovering the entire NetBackup catalog .......................... About recovering the NetBackup catalog image files ................... About recovering the NetBackup relational databases ................. Recovering the NetBackup catalog when NetBackup Access Control is configured ...................................................... Recovering the NetBackup catalog from a nonprimary copy of a catalog backup ............................................................... Recovering the NetBackup catalog without the disaster recovery file ............................................................................... Recovering a NetBackup user-directed online catalog from the command line ................................................................ Restoring files from a NetBackup online catalog backup ............. 181 182 184 185 190 190 191 191 193 194 194 195 201 201 204 204 205 206 207 208 209 220 233 241 243 244 248 251

Contents

11

Unfreezing the NetBackup online catalog recovery media ........... 251

Appendix A

Backup and restore functional overview ...................... 253

About backup and restore functional overview ................................. Backup and restore startup process ............................................... Backup and archive processes ....................................................... Job scheduling ..................................................................... EMM server and master server ................................................ Backups and archives - UNIX clients .............................................. Backup process .................................................................... Backup with multiple data streams .......................................... Snapshot backup and Windows open file backups ....................... SAN client ........................................................................... Backups and archives - Windows ............................................. Backups and archives - NetWare clients .................................... Synthetic backups ................................................................ NetBackup online, hot catalog backup ...................................... About UNIX and Linux client restoration ........................................ About SAN client (UNIX or Windows) restoration ............................. About Windows client restoration ................................................. About NetWare client restoration .................................................. About catalog backup restoration .................................................. NetBackup directories and files ..................................................... NetBackup directory structure - UNIX ...................................... Contents of /usr/openv/netbackup .......................................... NetBackup programs and daemons ................................................ NetBackup catalogs ..................................................................... 253 254 254 255 255 255 256 260 261 264 267 268 269 272 274 278 281 282 284 286 287 288 290 303

Appendix B

Media and device management functional description .................................................................... 305

Media and device management startup process ................................ Media and device management process .......................................... Shared Storage option management process .................................... Barcode operations ..................................................................... Media and device management components .................................... 305 307 309 311 313

Index ................................................................................................................... 323

12

Contents

Chapter

Introduction
This chapter includes the following topics:

Troubleshooting a problem Problem report for Technical Support About gathering information for NetBackup-Java applications

Troubleshooting a problem
The following steps offer general guidelines to help you resolve any problems you may encounter while you use NetBackup. The steps provide links to more specific troubleshooting information. Table 1-1 Step
Step 1

Steps for troubleshooting NetBackup problems Description

Action

Remember the error message Error messages are usually the vehicle for telling you something went wrong. If you dont see an error message in an interface, but still suspect a problem, check the reports and logs. NetBackup provides extensive reporting and logging facilities. These can provide an error message that points you directly to a solution. The logs also show you what went right and the NetBackup operation that was ongoing when the problem occurred. For example, a restore operation needs media to be mounted, but the required media is currently in use for another backup. Logs and reports are essential troubleshooting tools. See About logs on page 101.

14

Introduction Troubleshooting a problem

Table 1-1 Step

Step 2

Steps for troubleshooting NetBackup problems (continued) Description

Action

Identify what you were doing Ask the following questions: when the problem occurred What operation was tried? What method did you use? For example, more than one way exists to install software on a client. Also more than one possible interface exists to use for many operations. Some operations can be performed with a script. What type of server platform and operating system was involved?

If your site uses both the master server and the media server, was it a master server or a media server? If a client was involved, what type of client was it?

Have you performed the operation successfully in the past? If so, what is different now? What is the service pack level?

Do you use operating system software with the latest fixes supplied, especially those required for use with NetBackup? Is your device firmware at a level, or higher than the level, at which it has been tested according to the posted device compatibility lists?

Step 3

Record all information

Capture potentially valuable information:

NetBackup progress logs NetBackup Reports NetBackup Utility Reports NetBackup debug logs Media and Device Management debug logs

On UNIX NetBackup servers, check for error or status messages in the system log or standard output. Error or status messages in dialog boxes

On Windows, NetBackup servers, check for error or status information in the Event Viewer Application and System log.

Record this information for each try. Compare the results of multiple tries. A record of tries is also useful for others at your site and for Technical Support in the event that you cannot solve the problem. You can get more information about logs and reports. See About logs on page 101.

Introduction Problem report for Technical Support

15

Table 1-1 Step

Step 4

Steps for troubleshooting NetBackup problems (continued) Description

After you define the problem, use the following information to correct it: Take the corrective action recommended by the status code or message. See the Status Code Reference Guide. If no status code or message exists, or the actions for the status code do not solve the problem, use additional troubleshooting procedures to isolate common problems. See Troubleshooting NetBackup problems on page 22.

Action
Correct the problem

Step 5

Complete a problem report for Technical Support

If your troubleshooting is unsuccessful, prepare to contact Technical Support by filling out a problem report. See Problem report for Technical Support on page 15. See About gathering information for NetBackup-Java applications on page 17. On UNIX systems, the /usr/openv/netbackup/bin/goodies/support script creates a file containing data necessary for Technical Support to debug any problems you encounter. For more details, consult the usage information of the script by using support -h.

Step 6

Contact Technical Support

The Symantec Technical Support Web site has a wealth of information that can help you solve NetBackup problems. Access Technical Support at the following URL: www.symantec.com/business/support/

Note: The term media server may not apply to the NetBackup server product. It depends on the context. When you troubleshoot a server installation, be aware that only one host exists: The master and the media server are one and the same. Ignore references to a media server on a different host.

Problem report for Technical Support

Fill out the following information before you contact support to report a problem. Date: _________________________ Record the following product, platform, and device information:

Product and its release level. Server hardware type and operating system level.

16

Introduction Problem report for Technical Support

Client hardware type and operating system level, if a client is involved. Storage units being used, if it is possible that storage units are involved. If it looks like a device problem, be ready to supply the following device information: The types of robots and drives and their version levels along with Media and Device Management and system configuration information. Software patches to the products that were installed. The service packs and hotfixes that were installed. ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________

Define the problem. ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ What were you doing when the problem occurred? (for example, a backup on a Windows client) ______________________________________________________________________ ______________________________________________________________________ What were the error indications? (for example, status code, error dialog box) ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ Did this problem occur during or shortly after any of the following: _____ Initial installation _____ Configuration change (explain) _____ System change or problem (explain) _____ Have you observed the problem before? (If so, what did you do that time?) ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________

Introduction About gathering information for NetBackup-Java applications

17

Logs or other failure data you have saved: _____ All log entries report _____ Media and Device Management debug logs _____ NetBackup debug logs _____ System logs (UNIX) _____ Event Viewer Application and System logs (Windows) Ways that you can communicate with us: _____ ftp _____ telnet _____ email _____ WebEx

About gathering information for NetBackup-Java applications

If you encounter problems with the NetBackup-Java applications, use the following methods to gather data for support. The following scripts are available for gathering information:
Logs data in a log file in /usr/openv/netbackup/logs/user_ops/nbjlogs. At startup, ( NetBackup-Java administration application the script tells you which file in this directory it logs to. Normally, startup script) this file does not become very large (usually less than 2 KB). Consult the file /usr/openv/java/Debug.properties for the options that can affect the contents of this log file. jnbSA NetBackup-Java administration application Logs data in a log file if NetBackup is installed on the computer where on Windows the application was started. It logs on install_path\NetBackup\logs\user_ops\nbjlogs. If NetBackup was not installed on this computer, then no log file is created. To produce a log file, modify the last java.exe line in the following to redirect output to a file: install_path\java\nbjava.bat. . /usr/openv/java/get_trace Provides a Java virtual machine stack trace for support to analyze. This stack trace is written to the log file that is associated with the instance of execution.

18

Introduction About gathering information for NetBackup-Java applications

/usr/openv/netbackup/bin/goodies/support Creates a file containing data necessary for customer support to debug any problems you encounter. For more details, consult the usage information of the script by using support -h.

The following example describes how you can gather troubleshooting data for Symantec Technical Support to analyze.
An application does not respond. Wait for several minutes before you assume that the operation is hung. Some operations can take quite a while to complete, especially operations in the Activity Monitor and Reports applications. Run /usr/openv/java/get_trace under the account where you started the Java application. This script causes a stack trace to write to the log file. For example, if you started jnbSA from the root account, start /usr/openv/java/get_trace as root. Otherwise, the command runs without error, but fails to add the stack trace to the debug log. This failure occurs because root is the only account that has permission to run the command that dumps the stack trace. Get data about your configuration. Run /usr/openv/netbackup/bin/goodies/support. Run this script after you complete the NetBackup installation and every time you change the NetBackup configuration.

Still no response after several minutes.

Contact Symantec Technical Provide the log file and the output of the support script Support for analysis.

Chapter

Troubleshooting procedures
This chapter includes the following topics:

About troubleshooting procedures Troubleshooting NetBackup problems Troubleshooting installation problems Troubleshooting configuration problems Device configuration problem resolution Testing the master server and clients Testing the media server and clients Resolving network communication problems with UNIX clients Resolving network communication problems with PC clients About troubleshooting networks and hostnames Verifying host name and service entries in NetBackup About the bpclntcmd utility Using the Host Properties window to access configuration settings Resolving full disk problems Frozen media troubleshooting considerations Resolving PBX problems About troubleshooting Auto Image Replication Troubleshooting network interface card performance

20

Troubleshooting procedures About troubleshooting procedures

About SERVER entries in the bp.conf file About unavailable storage unit problems About troubleshooting NetBackup in a SAN environment

About troubleshooting procedures

These procedures for finding the cause of NetBackup errors are general in nature and do not try to cover every problem that can occur. They do, however, recommend the methods that usually result in successful problem resolution. The Symantec Technical Support site has a wealth of information that can help you solve NetBackup problems. See the following site for comprehensive troubleshooting details: http://www.symantec.com/business/support/ When you perform these procedures, try each step in sequence. If you already performed the action or it does not apply, skip to the next step. If it branches you to another topic, use the solutions that are suggested there. If you still have a problem, go to the next step in the procedure. Also, alter your approach according to your configuration and what you have already tried. Troubleshooting procedures can be divided into the following categories:
Preliminary troubleshooting The following procedures describe what to check first. They branch off to other procedures as appropriate. See Troubleshooting NetBackup problems on page 22. See Verifying that all processes are running on UNIX servers on page 24. See Verifying that all processes are running on Windows servers on page 26. Installation troubleshooting Problems that apply specifically to installation. See Troubleshooting installation problems on page 28. Configuration troubleshooting Problems that apply specifically to configuration. See Troubleshooting configuration problems on page 29.

Troubleshooting procedures About troubleshooting procedures

21

General test and troubleshooting

These procedures define general methods for finding server and client problems and should be used last. See Testing the master server and clients on page 34. See Testing the media server and clients on page 38. See Resolving network communication problems with UNIX clients on page 41. See Resolving network communication problems with PC clients on page 47. See Verifying host name and service entries in NetBackup on page 55. See About the bpclntcmd utility on page 68. See Verifying host name and service entries in NetBackup on page 55.

Other troubleshooting procedures

See Resolving full disk problems on page 71. See Frozen media troubleshooting considerations on page 72. See About conditions that cause media to freeze on page 73. See Troubleshooting network interface card performance on page 93. See About troubleshooting NetBackup in a SAN environment on page 95.

A set of examples is also available that shows host name and service entries for UNIX systems.

See Example of host name and service entries on UNIX master server and client on page 60. See Example of host name and service entries on UNIX master server and media server on page 61. See Example of host name and service entries on UNIX PC clients on page 63. See Example of host name and service entries on UNIX clients in multiple networks on page 64. See Example of host name and service entries on UNIX server that connects to multiple networks on page 66.

22

Troubleshooting procedures Troubleshooting NetBackup problems

Troubleshooting NetBackup problems

If you have problems with NetBackup, perform these actions first. This preliminary NetBackup troubleshooting procedure explains what to check first and branches to other procedures as appropriate. These procedures do not try to cover every problem that can occur. However, they do recommend the methods that usually result in successful problem resolution. When you perform these procedures, try each step in sequence. If you already performed the action or it does not apply, skip to the next step. If you branch to another topic, use the solutions that are suggested there. If you still have a problem, go to the next step in the procedure. Also, alter your approach according to your configuration and what you have already tried. Table 2-1 Step
Step 1

Steps for troubleshooting NetBackup problems Description

Action

Verify operating systems and Ensure that your servers and clients are running supported operating peripherals. system versions and that any peripherals you use are supported. Refer to the NetBackup release notes and the NetBackup compatibility lists on the following Web site: http://www.symantec.com/docs/TECH59978

Step 2

Use reports to check for errors.

Use the All Log Entries report and check for NetBackup errors for the appropriate time period. This report can show the context in which the error occurred. Often it provides specific information, which is useful when the status code can result from a variety of problems. See the Reports information in the NetBackup Administrators Guide, Volume I. If the problem involved a backup or archive, check the Status of Backups report. This report gives you the status code. If you find a status code or message in either of these reports, perform the recommended corrective actions. See the Status Codes Reference Guide.

Troubleshooting procedures Troubleshooting NetBackup problems

23

Table 2-1 Step

Step 3

Steps for troubleshooting NetBackup problems (continued) Description

Check the system log on UNIX or the Event Viewer Application and System log on Windows if the problem pertains to media or device management and one of the following is true:

Action
Check the operating system logs.

NetBackup does not provide a status code.

You cannot correct the problem by following the instructions in NetBackup status codes and messages. You cannot correct the problem by following the instructions in media and device management status codes and messages. These logs can show the context in which the error occurred. The error messages are usually descriptive enough to point you to a problem area. Step 4 Review the debug logs. Read the applicable enabled debug logs and correct any problems you detect. If these logs are not enabled, enable them before you retry the failed operation. See About logs on page 101. Step 5 Retry the operation. If you performed corrective actions, retry the operation. If you did not perform corrective actions or if the problem persists, continue with the next step. If you see the problem during a new installation, during an upgrade installation, or after you make changes to an existing configuration, see the following procedures: See Troubleshooting installation problems on page 28. See Troubleshooting configuration problems on page 29. Step 7 Ensure that the servers and clients are operational. If you experienced a server or a client disk crash, procedures are available on how to recover the files that are critical to NetBackup operation.

Step 6

Get more information for installation problems.

24

Troubleshooting procedures Troubleshooting NetBackup problems

Table 2-1 Step

Step 8

Steps for troubleshooting NetBackup problems (continued) Description

Verify that you have enough space available in the disk partitions that NetBackup uses. If one or more of these partitions is full, NetBackup processes that access the full partition fail. The resulting error message depends on the process. Possible error messages: "unable to access" or "unable to create or open a file." On UNIX systems, use the df command to view disk partition information. On Windows systems, use Disk Manager or Explorer. Check the following disk partitions:

Action
Ensure that the partitions have enough disk space.

The partition where NetBackup software is installed.

On the NetBackup master or media server, the partition where the NetBackup databases reside. The partition where the NetBackup processes write temporary files.

The partition where NetBackup logs are stored. The partition where the operating system is installed.

Step 9

Increase the logging level.

Enable verbose logging either for everything or only for areas that you think are related to the problem. See Changing the logging level on page 143. See How to control the amount of information written to legacy logging files on page 137. See Changing the logging level on Windows and NetWare clients on page 143.

Step 10

Determine which daemons or processes are running.

Follow the procedures for UNIX or Windows NetBackup servers. See Verifying that all processes are running on UNIX servers on page 24. See Verifying that all processes are running on Windows servers on page 26.

Verifying that all processes are running on UNIX servers

For NetBackup to operate properly, the correct set of processes (daemons) must be running on your UNIX servers. This procedure determines which processes are running and shows how to start processes that may not be running.

Troubleshooting procedures Troubleshooting NetBackup problems

25

To verify that all processes are running on UNIX servers

To see the list of processes (daemons) running on the server and on the Media Manager, enter the following command:
/usr/openv/netbackup/bin/bpps -x

If the master server is also the EMM server, ensure that the nbemm and the nbrb services are running. If neither service is running, start them by entering the following two commands. If only one of the services is running, start the other service by using the appropriate command.
/usr/openv/netbackup/bin/nbemm /usr/openv/netbackup/bin/nbrb

The nbpem and the nbjm services must be running on the master server. If neither service is running, start them by entering the following two commands. If only one of the services is running, start the other service by using the appropriate command.
/usr/openv/netbackup/bin/nbjm /usr/openv/netbackup/bin/nbpem

If either the NetBackup request daemon (bprd) or database manager daemon (bpdbm) is not running, start them by entering the following command:
/usr/openv/netbackup/bin/initbprd

Make sure that the following media and device management processes are running:

ltid (needs to be running only if drives are configured on the server) vmd (volume) avrd (automatic volume recognition), only if drives are configured on the

server

Processes for all configured robots

If any of these processes are not running, stop the device daemon ltid by running the following command:
/usr/openv/volmgr/bin/stopltid

26

Troubleshooting procedures Troubleshooting NetBackup problems

To verify that the ltid, avrd, and robotic control daemons are stopped, run the following command:
/usr/openv/volmgr/bin/vmps

If you use ACS robotic control, the acsssi and the acssel daemons may continue to run when ltid is terminated. Stop any robot control daemons that may continue to run by entering the following command:
/usr/openv/netbackup/bin/bp.kill_all

Then, start all daemons by running the following command:

/usr/openv/volmgr/bin/ltid

For debugging, start ltid with the -v (verbose) option.

Verifying that all processes are running on Windows servers

Use the following procedure to make sure that all the processes that need to run on Windows server are actually running.

Troubleshooting procedures Troubleshooting NetBackup problems

27

Table 2-2

Steps to ensure that all necessary processes are running on Windows servers

Step
Step 1

Action
Start all services.

Description
The following services must be running. If these services are not running, start them by using the NetBackup Activity Monitor or the Services application in the Windows Control Panel. To start all of the services, run install_path\NetBackup\bin\bpup.exe. Services on master servers:

NetBackup Request Manager service NetBackup Policy Execution Manager service NetBackup Job Manager service NetBackup database manager service NetBackup Device Manager service (if the system has configured devices) NetBackup Volume Manager service NetBackup Client service

Services on EMM servers:

NetBackup Enterprise Media Manager service NetBackup Resource Broker service

Services on media servers:

NetBackup Device Manager service (if the system has configured devices) NetBackup Volume Manager service NetBackup Client service

Services on clients (including NetBackup Remote Administration Consoles):

NetBackup Client service

Step 2

Start avrd and processes for robots.

Use the NetBackup Activity Monitor to see if the following processes are running: avrd (automatic media recognition), only if drives are configured on the server Processes for all configured robots. See the NetBackup Administrators Guide, Volume I.

If these processes are not running, stop and restart the NetBackup Device Manager service. Use the NetBackup Activity Monitor or the Services application in the Windows Control Panel.

28

Troubleshooting procedures Troubleshooting installation problems

Table 2-2

Steps to ensure that all necessary processes are running on Windows servers (continued)

Step
Step 3

Action
Restart the operation or do additional troubleshooting.

Description
If you had to start any of the processes or services in the previous steps, retry the operation. If the processes and services are running or the problem persists, you can try to test the servers and clients. See Testing the master server and clients on page 34. See Testing the media server and clients on page 38. If you cannot start any of these processes or services, check the appropriate debug logs for NetBackup problems. See About logs on page 101. When these processes and services start, they continue to run unless you stop them manually or a problem occurs on the system. On Windows systems, we recommend that you add commands for starting them to your startup scripts, so they restart in case you have to reboot.

Troubleshooting installation problems

Use the following steps to troubleshoot installation problems. Table 2-3 Step
Step 1

Steps for troubleshooting installation problems.

Action

Description

Determine if you can Some reasons for failure are as follows: install the software on Not logged on as an administrator on a Windows system (you must have the master server and permission to install services on the system) the media servers by Permission denied (ensure that you have permission to use the device and to using the release write the directories and files being installed) media. Bad media (contact Technical Support)

Defective drive (replace the drive or refer to vendors hardware documentation) Improperly configured drive (refer to the system and the vendor documentation)

Troubleshooting procedures Troubleshooting configuration problems

29

Table 2-3 Step

Step 2

Steps for troubleshooting installation problems. (continued)

Action

Description

Determine if you can Note: Before you install or use NetBackup on a Linux client, verify that the inetd install NetBackup (or xinetd) service is started on that computer. This service ensures proper client software on the communication between the NetBackup master and the Linux client. clients. Note: You cannot install PC client software from a UNIX NetBackup server. Do the following:

For an install to a trusting UNIX client, verify the following: The correct client name is in your policy configuration

The correct server name is in the client /.rhosts file

If the installation hangs, check for problems with the shell or the environment variables for the root user on the client. The files that you check depend on the platform, operating system, and shell you use. For example, your .login on a Sun system runs an stty (such as stty ^erase) before it defines your terminal type. If this action causes the install process to hang, you can modify the .login file to define the terminal before you run the stty. Or, move the client .login to another file until the install is complete.

For an installation to a secure UNIX client, check your ftp configuration. For example, you must use a user name and password that the client considers valid.

Step 3

Resolve network problems.

Determine if the problem is related to general network communications. See Resolving network communication problems with UNIX clients on page 41. See Resolving network communication problems with PC clients on page 47.

Troubleshooting configuration problems

Use the following steps to check for problems after an initial installation or after changes are made to the configuration.

30

Troubleshooting procedures Troubleshooting configuration problems

Table 2-4 Step

Step 1

Steps for troubleshooting configuration problems

Action
Check for device configuration problems.

Description
Check for the following device configuration problems:

Configuration for robotic drive does not specify the robot. Drive is configured as wrong type or density. Incorrect Robotic Drive Number.

SCSI ID for the robotic control is specified instead of the logical Robot Number that is assigned to the robot. The same robot number is used for different robots.

SCSI ID for the drive is specified instead of a unique Drive Index number. A platform does not support a device or was not configured to recognize it.

Robotic device is not configured to use LUN 1, which some robot hardware requires. On UNIX, drive no-rewind device path is specified as a rewind path. On UNIX, tape devices are not configured with "Berkeley style close." NetBackup requires this feature which is configurable on some platforms. Further explanation is available. On UNIX, tape devices (other than QIC) are not configured as "variable mode." NetBackup requires this feature which is configurable on some platforms. When this condition exists, you can frequently perform backups but not restores. Further explanation is available. See NetBackup status code 174 in the Status Codes Reference Guide. On UNIX, pass-through paths to the tape drives have not been established.

More description is available on device configuration problems: See the NetBackup Device Configuration Guide. Step 2 Check the daemons or Check for the following problems with the daemons or services: services. Daemons or services do not start during reboot (configure system so they start). Wrong daemons or services are started (problems with media server start-up scripts). Configuration was changed while daemons or services were running. On Windows, the %SystemRoot%\System32\drivers\etc\services file does not have an entry for vmd, bprd, bpdbm, and bpcd. Also, ensure that the processes have entries for configured robots. A list of these processes is available. See the NetBackup Device Configuration Guide. On UNIX, the /etc/services file (or NIS or DNS) does not have an entry for vmd, bprd, bpdbm, or robotic daemons.

Troubleshooting procedures Device configuration problem resolution

31

Table 2-4 Step

Step 3

Steps for troubleshooting configuration problems (continued)

Action
Retry the operation and check for status codes and messages.

Description
If you found and corrected any configuration problems, retry the operation and check for NetBackup status codes or messages in the following: Check the All Log Entries report for NetBackup errors for the appropriate time period. This report can show the context in which the error occurred. Often it provides specific information, which is useful when the error can result from a variety of problems. If the problem involved a backup or archive, check the Status of Backups report. This report gives you the status code. If you find a status code or message in either of these reports, perform the recommended corrective actions. See the Status Codes Reference Guide. Check the system log on UNIX or the Event Viewer Application and System log on Windows if the following is true: The problem pertains to media or device management, and NetBackup does not provide a status code, or you cannot correct the problem by following the instructions in the status codes. Check the appropriate enabled debug logs. Correct any problems you detect. If these logs are not enabled, enable them before your next try. See About logs on page 101.

Step 4

Retry the operation and do additional troubleshooting.

If you performed corrective actions, retry the operation. If you did not perform corrective actions or the problem persists, go to one of the following procedures. See Resolving full disk problems on page 71. See Frozen media troubleshooting considerations on page 72. See About conditions that cause media to freeze on page 73. See Troubleshooting network interface card performance on page 93. See About troubleshooting NetBackup in a SAN environment on page 95.

Device configuration problem resolution

An auto-configuration warning message appears in the second panel of the Device Configuration Wizard if the selected device meets any of the following conditions:

Not licensed for NetBackup server Exceeds a license restriction Has some inherent qualities that make it difficult to auto-configure

The following messages relate to device configuration, along with their explanations and recommended actions.

32

Troubleshooting procedures Device configuration problem resolution

Table 2-5 Message Explanation

Recommended actions for device configuration messages Recommended action

Ask the manufacturer for a newer firmware version that returns serial numbers (if available), or manually configure and operate the drive without a serial number.

Drive does not support The drive does not return its serial number. serialization Note that some manufacturers do not support serial numbers. Although automatic device configuration does not function optimally, the drive can be manually configured and operated without its serial number. Robot does not support The robot does not return its serial number serialization or the serial numbers of the drives that are contained within it. Note that some manufacturers do not support serial numbers. Although automatic device configuration does not function optimally, the robot and drives can be manually configured and operated without serial numbers. No license for this robot type No license for this drive type Unable to determine robot type NetBackup server does not support the robotic type that is defined for this robot.

Ask the manufacturer for a newer firmware version that returns serial numbers (if available). Or manually configure and operate the robot and drives without serial numbers.

Define a different robot. Use only the robotic libraries that NetBackup server supports.

The drive type that is defined for this drive Define a different drive. Use only the drives that the NetBackup server does not support. that NetBackup supports NetBackup does not recognize the robotic library. The robotic library cannot be auto-configured. Do the following: Download a new device_mapping file from the Symantec support Web site, and try again. Configure the robotic library manually.

Use only the robotic libraries that NetBackup supports.

Drive is stand-alone or Either the drive is stand-alone, or the drive in unknown robot or robot does not return a serial number. Note that some manufacturers do not support serial numbers. Although automatic device configuration does not function optimally, the drive or robot can be manually configured and operated without a serial number.

Ask the manufacturer for a newer firmware version that returns serial numbers (if available), or manually configure and operate the drive robot without serial numbers.

Troubleshooting procedures Device configuration problem resolution

33

Table 2-5

Recommended actions for device configuration messages (continued) Recommended action

Ask the manufacturer for a newer firmware version that returns serial numbers (if available). Or manually configure and operate the drive and robot without serial numbers.

Message

Explanation

Robot drive number is Either the drive or robot does not return a unknown serial number. Note that some manufacturers do not support serial numbers. Although automatic device configuration does not function optimally, the drive or robot can be manually configured and operated without a serial number. Drive is in an unlicensed robot The drive is in a robotic library that cannot be licensed for NetBackup server. Since the robot cannot be licensed for NetBackup server, any drives that were configured in that robot are unusable.

Configure a drive that does not reside in the unlicensed robot.

Drives SCSI adapter A drive was found that does not have a SCSI does not support pass-through path configured. The possible pass-thru (or pass-thru causes are: path does not exist) The drive is connected to an adapter that does not support SCSI pass-through. The pass-through path for this drive has not been defined. No configuration device file exists A device has been detected without the corresponding device file necessary to configure that device.

Change the drives adapter or define a pass-through path for the drive. SCSI adapter pass-through information is available. See the NetBackup Device Configuration Guide.

Refer to NetBackup Device Configuration Guide for information on how to create device files.

Unable to determine drive type

The NetBackup server does not recognize the Do the following: drive. The drive cannot be auto-configured. Download a new device_mapping file from the Symantec support Web site, and try again. Configure the drive manually.

Use only the drives that NetBackup supports.

34

Troubleshooting procedures Testing the master server and clients

Table 2-5

Recommended actions for device configuration messages (continued) Recommended action

If you do not need hardware data compression, no action is necessary. The drive can be operated without hardware data compression. Hardware data compression and tape drive configuration help are available. Refer to NetBackup Device Configuration Guide for information on how to create device files.

Message
Unable to determine compression device

Explanation
A drive was detected without the expected compression device file that is used to configure that device. Automatic device configuration tries to use a device file that supports hardware data compression. When multiple compression device files exist for a drive, automatic device configuration cannot determine which compression device file is best. It uses a non-compression device file instead.

Testing the master server and clients

If the NetBackup, installation, and configuration troubleshooting procedures do not reveal the problem, perform the following procedure. Skip those steps that you have already performed. The procedure assumes that the software was successfully installed, but not necessarily configured correctly. If NetBackup never worked properly, you probably have configuration problems. In particular, look for device configuration problems. You may also want to perform each backup and restore twice. On UNIX, perform them first as a root user and then as a nonroot user. On Windows, perform them first as a user that is a member of the Administrators group. Then perform them as a user that is not a member of the Administrator group. In all cases, ensure that you have read and write permissions on the test files. The explanations in these procedures assume that you are familiar with the functional overview information. See About backup and restore functional overview on page 253. Several steps in this procedure mention the All Log Entries report. To access more information on this report and others, refer to the following: See the NetBackup Administrators Guide, Volume I.

Troubleshooting procedures Testing the master server and clients

35

Table 2-6 Step

Step 1

Steps for testing the master server and clients

Action
Enable debug logs.

Description
Enable the appropriate debug logs on the master server. See About logs on page 101. See About unified logging on page 103. See About legacy logging on page 127. If you do not know which logs apply, enable them all until you solve the problem. Delete the debug log directories when you have resolved the problem.

Step 2

Configure a test policy. Configure a test policy and set the backup window to be open while you test. Name the master server as the client and a storage unit that is on the master server (preferably a nonrobotic drive). Also, configure a volume in the NetBackup volume pool and insert the volume in the drive. If you dont label the volume by using the bplabel command, NetBackup automatically assigns a previously unused media ID. Verify the daemons and services. To verify that the NetBackup daemons or services are running on the master server, do the following:

Step 3

To check the daemons on a UNIX system, enter the following command: /usr/openv/netbackup/bin/bpps -a

To check the services on a Windows system, use the NetBackup Activity Monitor or the Services application of the Windows Control Panel.

Step 4

Backup and restore a policy.

Start a manual backup of a policy by using the manual backup option in the NetBackup administration interface. Then, restore the backup. These actions verify the following: NetBackup server software is functional, which includes all daemons or services, programs, and databases. NetBackup can mount the media and use the drive you configured.

Step 5

Check for failure.

If a failure occurs, first check the NetBackup All Log Entries report. For the failures that relate to drives or media, verify that the drive is in an UP state and that the hardware functions. To isolate the problem further, use the debug logs. A functional overview sequence of events is available. See About backup and restore functional overview on page 253.

36

Troubleshooting procedures Testing the master server and clients

Table 2-6 Step

Step 6

Steps for testing the master server and clients (continued)

Action

Description

Consult information If the debug logs do not reveal the problem, check the following: besides the debug logs. Systems Logs or Event Viewer System logs

Event Viewer Application and System logs on Windows systems vmd debug logs on the EMM database host for the device bptm debug logs

See the vendor manuals for information on hardware failures. Step 7 Verify robotic drives. If you use a robot and the configuration is an initial configuration, verify that the robotic drive is configured correctly. In particular, verify the following: The same robot number is used both in the Media and Device Management and storage unit configurations. Each robot has a unique robot number.

On a UNIX NetBackup server, you can verify only the Media and Device Management part of the configuration. To verify, use the tpreq command to request a media mount. Verify that the mount completes and check the drive on which the media was mounted. Repeat the process until the media is mounted and unmounted on each drive from the host where the problem occurred. If this works, the problem is probably with the policy or the storage unit configuration. When you are done, tpunmount the media. Step 8 Include a robot in the test policy. If you previously configured a nonrobotic drive and your system includes a robot, change your test policy now to specify a robot. Add a volume to the robot. The volume must be in the NetBackup volume pool on the EMM database host for the robot. Return to step 3 and repeat this procedure for the robot. This procedure verifies that NetBackup can find the volume, mount it, and use the robotic drive. Step 9 Use the robotic test utilities. If you have difficulties with the robot, try the test utilities. See About the robotic test utilities on page 177. Do not use the Robotic Test Utilities when backups or restores are active. These utilities prevent the corresponding robotic processes from performing robotic actions, such as loading and unloading media. The result is that it can cause media mount timeouts and prevent other robotic operations like robotic inventory and inject or eject from working. Step 10 Enhance the test policy. Add a user schedule to your test policy (the backup window must be open while you test). Use a storage unit and media that was verified in previous steps.

Troubleshooting procedures Testing the master server and clients

37

Table 2-6 Step

Step 11

Steps for testing the master server and clients (continued)

Action
Backup and restore a file.

Description
Start a user backup and restore of a file by using the client-user interface on the master server. Monitor the status and the progress log for the operation. If successful, this operation verifies that the client software is functional on the master server. If a failure occurs, check the NetBackup All Log Entries report. To isolate the problem further, check the appropriate debug logs from the following list. On a UNIX system, the debug logs are in the /usr/openv/netbackup/logs/ directory. On a Windows system, the debug logs are in the install_path\NetBackup\logs\ directory. Debug log directories exist for the following processes:

bparchive (UNIX only) bpbackup (UNIX only) bpbkar bpcd bplist bprd bprestore nbwin (Windows only) bpinetd (Windows only)

Explanations about which logs apply to specific client types are available. See About logs on page 101. See About unified logging on page 103. See About legacy logging on page 127. Step 12 Reconfigure the test policy. Reconfigure your test policy to name a client that is located elsewhere in the network. Use a storage unit and media that has been verified in previous steps. If necessary, install the NetBackup client software.

38

Troubleshooting procedures Testing the media server and clients

Table 2-6 Step

Step 13

Steps for testing the master server and clients (continued)

Action
Create debug log directories.

Description
Create debug log directories for the following processes:

bprd on the server bpcd on the client bpbkar on the client nbwin on the client (Windows only) bpbackup on the client (except Windows clients) bpinetd (Windows only)

Explanations about which logs apply to specific client types are available. See About logs on page 101. See About unified logging on page 103. See About legacy logging on page 127. Step 14 Verify communication Perform a user backup and then a restore from the client that is specified in step between the client and 8. These actions verify communications between the client and the master server, the master server. and NetBackup software on the client. If an error occurs, check the All Log Entries report and the debug logs that you created in the previous step. A likely cause for errors is a communications problem between the server and the client. Step 15 Test other clients or storage units. When the test policy operates satisfactorily, repeat specific steps as necessary to verify other clients and storage units.

Step 16

Test the remaining When all clients and storage units are functional, test the remaining policies and policies and schedules. schedules that use storage units on the master server. If a scheduled backup fails, check the All Log Entries report for errors. Then follow the recommended actions as is part of the error status code.

Testing the media server and clients

If you use media servers, use the following steps to verify that they are operational. Before testing the media servers, eliminate all problems on the master server. See Testing the master server and clients on page 34.

Troubleshooting procedures Testing the media server and clients

39

Table 2-7 Step

Step 1

Steps for testing the media server and clients

Action
Enable legacy debug logs.

Description
Enable appropriate legacy debug logs on the servers See About logs on page 101. See About legacy logging on page 127. If you are uncertain which logs apply, enable them all until you solve the problem. Delete the legacy debug log directories when you have resolved the problem.

Step 2

Configure a test policy.

Configure a test policy with a user schedule (set the backup window to be open while you test) by doing the following: Name the media server as the client and a storage unit that is on the media server (preferably a nonrobotic drive). Add a volume on the EMM database host for the devices in the storage unit. Ensure that the volume is in the NetBackup volume pool. Insert the volume in the drive. If you do not pre-label the volume by using the bplabel command, NetBackup automatically assigns a previously unused media ID.

Step 3

Verify the daemons and services.

Verify that all NetBackup daemons or services are running on the master server. Also, verify that all Media and Device Management daemons or services are running on the media server. To perform this check, do one of the following:

On a UNIX system, run: /usr/openv/netbackup/bin/bpps -a

On a Windows system, use the Services application in the Windows Control Panel.

Step 4

Backup and restore a file.

Perform a user backup and then a restore of a file from a client that has been verified to work with the master server. This test verifies the following:

NetBackup media server software

NetBackup on the media server can mount the media and use the drive that you configured Communications between the master server processes nbpem, nbjm, nbrb, EMM server process nbemm, and media server processes bpcd and bpbrm

Communications between media server process bpbrm and client processes bpcd and bpbkar

For the failures that relate to drives or media, ensure that the drive is in an UP state and that the hardware functions.

40

Troubleshooting procedures Testing the media server and clients

Table 2-7 Step

Step 5

Steps for testing the media server and clients (continued)

Action
Verify communication between the master server and the media servers.

Description
If you suspect a communications problem between the master server and the media servers, check the debug logs for the pertinent processes. If the debug logs dont help you, check the following:

On a UNIX server, the System log On a Windows server, the Event Viewer Application and System log vmd debug logs

Step 6

Ensure that the hardware runs correctly.

For the failures that relate to drives or media, ensure that the drive is running and that the hardware functions correctly. See the vendor manuals for information on hardware failures. If you use a robot in an initial configuration condition, verify that the robotic drive is configured correctly. In particular, verify the following: The same robot number is used both in the Media and Device Management and storage unit configurations. Each robot has a unique robot number.

On a UNIX server, you can verify only the Media and Device Management part of the configuration. To verify, use the tpreq command to request a media mount. Verify that the mount completes and check the drive on which the media was mounted. Repeat the process until the media is mounted and unmounted on each drive from the host where the problem occurred. Perform these steps from the media server. If this works, the problem is probably with the policy or the storage unit configuration on the media server. When you are done, use tpunmount to unmount the media.

Troubleshooting procedures Resolving network communication problems with UNIX clients

41

Table 2-7 Step

Step 7

Steps for testing the media server and clients (continued)

Action
Include a robotic device in the test policy.

Description
If you previously configured a non-robotic drive and a robot was attached to your media server, change the test policy to name the robot. Also, add a volume for the robot to the EMM server. Verify that the volume is in the NetBackup volume pool and in the robot. Start with step 3 to repeat this procedure for a robot. This procedure verifies that NetBackup can find the volume, mount it, and use the robotic drive. If a failure occurs, check the NetBackup All Log Entries report. Look for any errors that relate to devices or media. See the NetBackup Administrators Guide, Volume I. If the All Log Entries report doesnt help, check the following:

On a UNIX server, the system logs on the media server vmd debug logs on the EMM server for the robot On a Windows system, the Event Viewer Application and System log

In an initial configuration, verify that the robotic drive is configured correctly. Do not use a robot number that is already configured on another server. Try the test utilities. See About the robotic test utilities on page 177. Do not use the Robotic Test Utilities when backups or restores are active. These utilities prevent the corresponding robotic processes from performing robotic actions, such as loading and unloading media. The result is that it can cause media mount timeouts and prevent other robotic operations like robotic inventory and inject or eject from working. Step 8 Test other clients or storage units. When the test policy operates satisfactorily, repeat specific steps as necessary to verify other clients and storage units.

Step 9

Test the remaining When all clients and storage units are in operation, test the remaining policies and policies and schedules that use storage units on the media server. If a scheduled backup fails, schedules. check the All Log Entries report for errors. Then follow the suggested actions for the appropriate status code.

Resolving network communication problems with UNIX clients

The following procedure is for resolving NetBackup communications problems, such as those associated with NetBackup status codes 54, 57, and 58. This

42

Troubleshooting procedures Resolving network communication problems with UNIX clients

procedure consists of two variations: one for UNIX clients and another for PC clients. Note: In all cases, ensure that your network configuration works correctly outside of NetBackup before trying to resolve NetBackup problems. For UNIX clients, perform the following steps. Before you start this procedure, add the VERBOSE option to the /usr/openv/netbackup/bp.conf file. Also, create a bpcd debug log directory on your server and clients and a bprd log directory on the server. During subsequent retries, the debug logs provide detailed debug information, which can help you analyze the problem. Table 2-8 Steps for resolving network communication problems with UNIX clients

Step
Step 1

Action
Test a new or modified confitguration.

Description
If this configuration is a new or a modified configuration, do the following:

Check any recent modifications to ensure that they did not introduce the problem.

Ensure that the client software was installed and that it supports the client operating system. Check the client names, server names, and service entries in your NetBackup configuration as explained in the following topic: See Verifying host name and service entries in NetBackup on page 55. You can also use the hostname command on the client to determine the host name that the client sends with requests to the server. Check the bprd debug log (verbose) on the server to determine what occurred when the server received the request. Note the required NIS or DNS updates. Failure to update these services properly is a common source of network problems.

Troubleshooting procedures Resolving network communication problems with UNIX clients

43

Table 2-8

Steps for resolving network communication problems with UNIX clients (continued)

Step
Step 2

Action
Verify network connectivity.

Description
Verify network connectivity between client and server by trying to ping the client from the server. # ping clientname Where clientname is the name of the client as configured in the NetBackup policy configuration, in /etc/hosts, and in NIS and DNS (if applicable). For example, to ping a client that is named ant: # ping ant ant.nul.nul.com: 64 byte packets 64 bytes from 199.199.199.24: icmp_seq=0. time=1. ms ----ant.nul.nul.com PING Statistics---2 packets transmitted, 2 packets received, 0% packet loss round-trip (ms) min/avg/max = 1/1/1 Also, try ping from the client to the server. If ping succeeds in both instances, it verifies connectivity between the server and client. If ping fails, you have a network problem outside of NetBackup that must be resolved before you proceed. Some forms of the ping command let you ping the bpcd port on the client as in the following command: # ping ant 13782 Or # ping ant bpcd

44

Troubleshooting procedures Resolving network communication problems with UNIX clients

Table 2-8

Steps for resolving network communication problems with UNIX clients (continued)

Step
Step 3

Action
Ensure that the client listens on the correct port for the bpcd connections.

Description
Run one of the following commands (depending on platform and operating system): netstat -a | grep bpcd netstat -a | grep 13782 rpcinfo -p | grep 13782 If no problem occurs with the port, the results are similar to: tcp 0 0 *.13782 *.* LISTEN LISTEN indicates that the client listens for connections on this port. If a problem occurs, this line does not appear. One of the following may exist:

/etc/services (or applicable NIS file) does not have the correct bpcd entry. The correct /etc services entry is: bpcd 13782/tcp bpcd

/etc/inetd.conf (or applicable NIS or DNS file) does not have the correct bpcd entry. The correct /etc/inetd.conf entry is: bpcd stream tcp nowait root /usr/openv/netbackup/bin/bpcd

/etc/inetd.conf was changed but was not re-read. Correct this condition by running one of the following (whichever works): /bin/ps -ef | grep inetd kill -HUP the_inetd_pid Or /bin/ps -aux | grep inetd kill -HUP the_inetd_pid

On a Hewlett-Packard platform, use inetd -c to send a SIGHUP to inetd. On an AIX client, use SMIT to verify that the InetServ object policy was updated with information about the bpcd process (/etc/inetd.conf and /etc/services information). If you use SMIT to modify the InetServ object policy, the inetexp command automatically runs. If you edit the InetServ object policy, run the inetexp command to export the InetServ object policy to the /etc/inetd.conf and /etc/services files. This command keeps these files in sync with the InetServ object policy. Run the following command to inform the inetd daemon of the changes to its configuration file: refresh -s inetd or kill -1 InetdPID

Troubleshooting procedures Resolving network communication problems with UNIX clients

45

Table 2-8

Steps for resolving network communication problems with UNIX clients (continued)

Step
Step 4

Action
Connect to the client through telnet.

Description
telnet to bpcd on the client. If it succeeds, keep the connection until after you perform step 5, then terminate it with Ctrl-c. telnet clientname 13782 Where clientname is the name of the client as configured in the NetBackup policy configuration, /etc/hosts, and also in NIS and DNS (if applicable). For example, # telnet ant bpcd Trying 199.999.999.24 ... Connected to ant.nul.nul.com. Escape character is ^]. In this example, telnet can establish a connection to the client ant. If the telnet succeeds, then inetd on the client is configured correctly. It can pass its connection to bpcd and NetBackup should also be able to establish a connection. If telnet doesnt work, ensure that the inetd.conf file and /etc/services files on both the server and client are correct and match. By default, these are as follows: In /etc/services:

bpcd

13782/tcp

bpcd

In /etc/inetd.conf: bpcd stream tcp nowait root /usr/openv/netbackup/bin/bpcd bpcd Then, run kill -HUP to reread the /etc/inetd.conf file as explained in step 3. Also, update the applicable NIS or DNS files. If these files are correct and you cannot connect to the client, you may have problems with the network routing or the port assignment. (See the next step.)

46

Troubleshooting procedures Resolving network communication problems with UNIX clients

Table 2-8

Steps for resolving network communication problems with UNIX clients (continued)

Step
Step 5

Action
Ensure that the client listens on the correct port for the telnet connection to bpcd.

Description
Run one of the following commands (depending on platform and operating system). netstat -a | grep bpcd netstat -a | grep 13782 rpcinfo -p | grep 13782 The value 13782 could also be the value that is specified during installation. One of the following conditions occurs:

If the port is not the problem, you see the following: tcp 0 0 ant.nul.nul.com.13782 whale...com.1516 ESTABLISHED tcp 0 0 *.13782 *.* LISTEN

Where ESTABLISHED indicates that the telnet connection was established to bpcd through port 13782 on the client. LISTEN indicates that the client listens for further connections on this port. Change the port number for bpcd or other NetBackup services only if there is no alternative. All NetBackup servers and clients in the configuration must use this new port assignment. If a process other than bpcd uses the port, try to reboot the client to clear the problem. If the problem is still not fixed, you may need to change one of the service numbers (preferably for the other service). To change a service number, modify the /etc/services files Then send SIGHUP signals to the inetd processes on your clients. /bin/ps -ef | grep inetd kill -HUP the_inetd_pid Or /bin/ps -aux | grep inetd kill -HUP the_inetd_pid On a Hewlett-Packard platform, use inetd -c to send a SIGHUP to inetd. Also make applicable NIS or DNS updates. If the problem is with an AIX client and you make changes to the /etc/inetd.conf and /etc/services information, use SMIT to verify that the InetServ object policy was updated. See step 3.

Troubleshooting procedures Resolving network communication problems with PC clients

47

Table 2-8

Steps for resolving network communication problems with UNIX clients (continued)

Step
Step 6

Action
Verify communication between the client and the master server.

Description
To verify client to master server communications, use the bpclntcmd utility. When -pn and -sv run on a NetBackup client, they initiate inquiries to the NetBackup master server (as configured in the client bp.conf file). The master server then returns information to the requesting client. More information is available about bpclntcmd. See About the bpclntcmd utility on page 68.

Resolving network communication problems with PC clients

The following procedure is for resolving NetBackup communications problems, such as those associated with NetBackup status codes 54, 57, and 58. This procedure consists of two variations: one for UNIX clients and another for PC clients. Note: In all cases, ensure that your network configuration works correctly outside of NetBackup before trying to resolve NetBackup problems. This procedure helps you resolve network communication problems with PC clients. To resolve network communication problems

Before you retry the failed operation, do the following:

Increase the logging level on the client (see the clients user guide). On the NetBackup server, create a bprd debug log directory and on the clients create a bpcd debug log. On the NetBackup server, set the Verbose level to 1. See Changing the logging level on Windows and NetWare clients on page 143.

If this client is new, verify the client and the server names in your NetBackup configuration. See Verifying host name and service entries in NetBackup on page 55.

Verify basic network connectivity between client and server by pinging from the server to the client and vice versa. Use the following command:

48

Troubleshooting procedures Resolving network communication problems with PC clients

# ping hostname

Where hostname is the name of the host as configured in the following:

NetBackup policy configuration WINS DNS (if applicable).

hosts file in system directory %SystemRoot%\system32\drivers \etc\hosts (Windows XP or 2003)

If ping succeeds in all instances, it verifies basic connectivity between the server and client. If ping fails, you have a network problem outside of NetBackup that must be resolved before you proceed. As a first step, verify that the workstation is turned on. A workstation that is not turned on is a common source of connection problems with PC workstations.

On Microsoft Windows or NetWare clients, check the NetBackup Client service. Do one of the following tasks:

Ensure that the service is active by checking the logs or by doing one of the following:
Windows XP or Use the Services application in the Control Panel to verify Windows Server 2003 that the NetBackup Client service is running. Start it if clients necessary. NetWare clients Enter modules bpcd.nlm from the NetWare server console to verify that the NetBackup client daemon is running. If necessary, type bpstart.ncf from the NetWare server console to start the NetBackup client daemon.

Check the bpcd debug logs for problems or errors. Instructions are available on how to enable and use these logs. See About legacy logging on page 127. Verify that the same NetBackup client service (bpcd) port number is specified on both the NetBackup client and server (by default, 13782). Do one of the following:

Troubleshooting procedures Resolving network communication problems with PC clients

49

Windows

Check the NetBackup client service port number. Start the Backup, Archive, and Restore interface on the client. On the File menu, click NetBackup Client Properties. In the NetBackup Client Properties dialog box on the Network tab, check the NetBackup client service port number. Verify that the setting on the Network tab matches the one in the services file. The services file is located in: %SystemRoot%\system32\drivers\etc\services (Windows) The values on the Network tab are written to the services file when the NetBackup client service starts.

NetWare clients

See the BPCD setting in the SYS:VERITAS\NBUCLT\NetBack\BP.INI file. The bpcd port number is in the /etc/services file. On Windows NetBackup servers, see the Client Properties dialog box in the Host Properties window. See Using the Host Properties window to access configuration settings on page 70.

UNIX NetBackup servers

Correct the port number if necessary. Then, on Windows clients and servers, stop and restart the NetBackup Client service. On NetWare clients, stop and restart the NetBackup client daemon (bpcd). Do not change NetBackup port assignments unless it is necessary to resolve conflicts with other applications. If you do change them, do so on all NetBackup clients and servers. These numbers must be the same throughout your NetBackup configuration.

50

Troubleshooting procedures Resolving network communication problems with PC clients

Verify that the NetBackup Request Service (bprd) port number on Microsoft Windows and NetWare clients is the same as on the server (by default, 13720). Do one of the following:
Windows clients Check the NetBackup client service port number. Start the Backup, Archive, and Restore interface on the client. On the File menu, click NetBackup Client Properties. In the NetBackup Client Properties dialog box on the Network tab, check the NetBackup client service port number. Verify that the setting on the Network tab matches the one in the services file. The services file is located in: %SystemRoot%\system32\drivers\etc\services (Windows) The values on the Network tab are written to the services file when the NetBackup client service starts. NetWare clients See the BPRD setting in the SYS:VERITAS\NBUCLT\NetBack\BP.INI file. The bprd port number is in the /etc/services file. See Using the Host Properties window to access configuration settings on page 70. Set these numbers in the Client Properties dialog box in the Host Properties window. See Using the Host Properties window to access configuration settings on page 70.

UNIX NetBackup servers

Windows NetBackup servers

Verify that the hosts file or its equivalent contains the NetBackup server name. The hosts files are the following:
Windows XP or 2003 NetWare UNIX %SystemRoot%\system32\drivers\etc\hosts SYS:etc\hosts /etc/hosts

Verify client-to-server connectability by using ping or its equivalent from the client (step 3 verified the server-to-client connection).

Troubleshooting procedures About troubleshooting networks and hostnames

51

If the clients TCP/IP transport allows telnet and ftp from the server, try these services as additional connectivity checks. For a NetWare client, ensure that the server does not try to connect when a backup or restore is already in progress on the client. If you try more than one job at a time on these clients, it results in a "cant connect" or similar error.

Use the bpclntcmd utility to verify basic client to master server communications. When -pn and -sv run on a client, they initiate inquiries to the master server (as configured in the server list on the client). The master server then returns information to the requesting client. See About the bpclntcmd utility on page 68.

10 Use the bptestbpcd utility to try to establish a connection from a NetBackup

server to the bpcd daemon on another NetBackup system. If successful, it reports information about the sockets that are established. See About the bpclntcmd utility on page 68.

11 Verify that the client operating system is one of those supported by the client
software.

About troubleshooting networks and hostnames

In a configuration with multiple networks and clients with more than one hostname, NetBackup administrators must configure the policy entries carefully. They must consider the network configuration (physical, hostnames and aliases, NIS/DNS, routing tables, and so on). If administrators want to direct backup and restore data across specific network paths, they especially need to consider these things. For a backup, NetBackup connects to the host name as configured in the policy. The operating systems network code resolves this name and sends the connection across the network path that is defined by the systems routing tables. The bp.conf file is not a factor making this decision. For restores from the client, the client connects to the master server. For example, on a UNIX system, the master server is the first one named in the /usr/openv/netbackup/bp.conf file. On a Windows system, the master server is specified on the Server to use for backups and restores drop-down of the Specify NetBackup Machines and Policy Type dialog box. To open this dialog, start the NetBackup Backup, Archive, and Restore interface and click Specify NetBackup Machines and Policy Type on the File menu. The clients network code that maps the server name to an IP address determines the network path to the server.

52

Troubleshooting procedures About troubleshooting networks and hostnames

Upon receipt of the connection, the server determines the clients configured name from the peername of its connection to the server. The peername is derived from the IP address of the connection. This means that the address must translate into a host name (using the gethostbyaddr() network routine). This name is visible in the bprd debug log when a connection is made as in the line:
Connection from host peername ipaddress ...

The clients configured name is then derived from the peername by querying the bpdbm process on UNIX systems. On Windows systems, you must query the NetBackup Database Manager service. The bpdbm process compares the peername to a list of client names that are generated from the following:

All clients for which a backup has been attempted All clients in all policies

The comparison is first a simple string comparison. The comparison is verified by comparing hostnames and aliases that are retrieved by using the network function gethostbyname(). If none of the comparisons succeed, a more brute force method is used, which compares all names and aliases using gethostbyname(). The configured name is the first comparison that succeeds. Note that other comparisons might also have succeeded if aliases or other "network names" are configured. If the comparison fails, the clients hostname as returned by the gethostname() function on the client is used as the configured name. An example of a failed comparison is when the client had changed its hostname but its new hostname is not yet reflected in any policies. These comparisons are logged in the bpdbm debug log if VERBOSE is set. You can determine a clients configured name by using the bpclntcmd command on the client. For example:
# /usr/openv/netbackup/bin/bpclntcmd -pn (UNIX) # install_path\NetBackup\bin\bpclntcmd -pn (Windows) expecting response from server wind.abc.me.com danr.abc.me.com danr 194.133.172.3 4823

Where the first output line identifies the server to which the request is directed and the second output line is the servers response in the following order:

Troubleshooting procedures About troubleshooting networks and hostnames

53

Peername of the connection to the server Configured name of the client IP address of the connection to the server Port number that is used in the connection

When the client connects to the server, it sends the following three names to the server:

browse client requesting client destination client

The browse client name is used to identify the client files to list or restore from. The user on the client can modify this name to restore files from another client. For example, on a Windows client, the user can change the client name by using the Backup, Archive, and Restore interface. (See the NetBackup online Help for instructions). For this change to work, however, the administrator must also have made a corresponding change on the server. See the NetBackup Administrators Guide, Volume I. The requesting client is the value from the gethostname() function on the client. The destination client name is a factor only if an administrator pushes a restore to a client from a server. For a user restore, the destination client and the requesting client are the same. For an administrator restore, the administrator can specify a different name for the destination client. By the time these names appear in the bprd debug log, the requesting client name has been translated into the clients configured name. The name that used to connect back to the client to complete the restore is either the clients peername or its configured name. The type of restore request (for example, from root on a server, from a client, to a different client, and so on) influences this action. When you modify client names in NetBackup policies to accommodate specific network paths, the administrator needs to consider:

The client name as configured on the client. For example, on UNIX the client name is CLIENT_NAME in the clients bp.conf file. On a Windows client, it is on the General tab of the NetBackup Client Properties dialog box. To open this dialog box, select NetBackup Client Properties from the File menu in the Backup, Archive, and Restore interface. The client as currently named in the policy configuration.

54

Troubleshooting procedures About troubleshooting networks and hostnames

Client backup and archive images that already exist as recorded in the images directory on the master server. On a UNIX or Linux server, the images directory is /usr/openv/netbackup/db/. On a Windows NetBackup server, the images directory is install_path\NetBackup\db\images.

Any of these client names can require manual modification by the administrator if the following: a client has multiple network connections to the server and restores from the client fail due to a connection-related problem. On UNIX, the public domain program traceroute (not included with NetBackup) often can provide valuable information about a networks configuration. Some system vendors include this program with their systems. The master server may be unable to reply to client requests, if the Domain Name Services (DNS) are used and the following is true: the name that the client obtains through its gethostname() library (UNIX) or gethostbyname() network (Windows) function is unknown to the DNS on the master server, The client and the server configurations can determine if this situation exists. gethostname() or gethostbyname()on the client may return an unqualified host name that the DNS on the master server cannot resolve. Although you can reconfigure the client or the master server DNS hosts file, this solution is not always desirable. For this reason, NetBackup provides a special file on the master server. This file is as follows:
/usr/openv/netbackup/db/altnames/host.xlate (UNIX and Linux) install_path\NetBackup\db\altnames\host.xlate (Windows)

You can create and edit this file to force the desired translation of NetBackup client host names. Each line in the host.xlate file has three elements: a numeric key and two hostnames. Each line is left-justified, and a space character separates each element of the line.
key hostname_from_client client_as_known_by_server

The following describes the preceding variables:

key is a numeric value used by NetBackup to specify the cases where the translation is to be done. Currently this value must always be 0, which indicates a configured name translation. hostname_from_client is the value to translate. This value must correspond to the name that is obtained by the clients gethostname() function and sent to the server in the request.

Troubleshooting procedures Verifying host name and service entries in NetBackup

55

client_as_known_by_server is the name to substitute for hostname_from_client when responding to requests. This name must be the name that is configured in the NetBackup configuration on the master server. It must also be known to the master servers network services.

This following is an example:

0 danr danr.eng.aaa.com

When the master server receives a request for a configured client name (numeric key 0), the name danr is always replaced by the name danr.eng.aaa.com. The problem is resolved, assuming the following:

The clients gethostname() function returns danr. The master servers network services gethostbyname() function did not recognize the name danr. The client was configured and named in the NetBackup configuration as danr.eng.aaa.com and this name is also known to network services on the master server.

Verifying host name and service entries in NetBackup

This procedure is useful if you encounter problems with host names or network connections and want to verify that the NetBackup configuration is correct. Several examples follow the procedure. More information about host names is available.

See About troubleshooting networks and hostnames on page 51. See the NetBackup Administrators Guide, Volume II.

To verify the host name and service entries in NetBackup

Verify that the correct client and server host names are configured in NetBackup. The action you take depends on the computer that you are checking.

56

Troubleshooting procedures Verifying host name and service entries in NetBackup

On Windows servers, Windows clients, and NetWare nontarget clients

Do the following:

On the Server to use for backups and restores drop-down list, ensure that a server entry exists for the master server and each media server. Start the Backup, Archive, and Restore interface on the client. On the File menu, click Specify NetBackup Machines and Policy Type. In the Specify NetBackup Machines and Policy Type dialog box, click the Server to use for backups and restores drop-down list. On Windows systems, the correct server must be designated as the current master server in the list. If you add or modify server entries on the master server, stop and restart the NetBackup Request service and NetBackup database manager services. On UNIX systems, if you add or modify SERVER entries on the master server, stop and restart bprd and bpdbm.

On the General tab, verify that the client name setting is correct and matches what is in the policy client list on the master server. Start the Backup, Archive, and Restore interface on the client. On the File menu, click NetBackup Client Properties. In the NetBackup Client Properties dialog box, click the General tab. On a master or a media server, ensure that a server entry exists for each Windows administrative client to use to administer that server. Ensure that host names are spelled correctly in the bp.conf file (UNIX) or in the servers list (Windows) on the master server. If a host name is misspelled or cannot be resolved by using gethostbyname, the following error messages are logged on the NetBackup error log:

Gethostbyname failed for <host_name>:<h_errno_string> (<h_errno>) One or more servers was excluded from the server list because gethostby name() failed. You can also make these changes on the appropriate tabs in the properties dialog boxes on a Windows NetBackup server See Using the Host Properties window to access configuration settings on page 70.

Troubleshooting procedures Verifying host name and service entries in NetBackup

57

On UNIX NetBackup Check the server and the client name entries in the bp.conf file by doing the following: servers and clients and Ensure that a SERVER entry exists for the master server and each media server in the Macintosh clients configuration. The master server must be the first name in the list. If you add or modify SERVER entries on the master server, stop and restart bprd and bpdbm before the changes take effect.

The bp.conf of the master server does not require the addition of other clients, other than the master server as CLIENT_NAME = master server name. The name is added by default.

The bp.conf file is in the /usr/openv/netbackup directory on UNIX clients and it is in the Preferences:NetBackup folder on Macintosh clients. Users on UNIX clients can also have a personal bp.conf file in their home directory. A CLIENT_NAME option in $HOME/bp.conf overrides the option in /usr/openv/netbackup/bp.conf. On NetWare clients Check the SYS:VERITAS\NBUCLT\NetBack\BP.INI file to ensure the following: A SERVER entry exists for the master server and each media server in the configuration. The master server must be the first name in the list. The ClientName entry and the entries in the [clients] section are correct and match what is in the policy client list on the master server.

58

Troubleshooting procedures Verifying host name and service entries in NetBackup

On the master server

Verify that you have created any of the following required files:

/usr/openv/netbackup/db/altnames files (UNIX) install_path\NetBackup\db\altnames files (Windows)

Pay particular attention to requirements for host.xlate file entries.

Verify that each server and client have the required entries for NetBackup reserved port numbers. The following examples show the default port numbers. See Example of host name and service entries on UNIX master server and client on page 60. See Example of host name and service entries on UNIX master server and media server on page 61. See Example of host name and service entries on UNIX PC clients on page 63. See Example of host name and service entries on UNIX clients in multiple networks on page 64. See Example of host name and service entries on UNIX server that connects to multiple networks on page 66. Do not change NetBackup port assignments unless it is necessary to resolve conflicts with other applications. If you do change them, do so on all NetBackup clients and servers. These numbers must be the same throughout your NetBackup configuration.

On NetBackup servers, check the services files to ensure that they have entries for the following:

bpcd and bprd vmd bpdbm

Processes for configured robots (for example, tl8cd). See the NetBackup Device Configuration Guide.

Verify the NetBackup client daemon or service number, and the request daemon or service port number. The action you take depends on whether the client is UNIX, Microsoft Windows, or NetWare.
On UNIX clients Check the bprd and the bpcd entries in the /etc/services file.

Troubleshooting procedures Verifying host name and service entries in NetBackup

59

On Microsoft Windows clients

Verify that the NetBackup Client Service Port number and NetBackup Request Service Port number match settings in the services file by doing the following: Start the Backup, Archive, and Restore interface on the client. On the File menu, click NetBackup Client Properties. In the NetBackup Client Properties dialog box on the Network tab, select the following: The NetBackup Client Service Port number and NetBackup Request Service Port number. The values on the Network tab are written to the services file when the NetBackup Client service starts. The services file is in the following location: %SystemRoot%\system32\drivers\etc\services

On NetWare clients

Check the BPCD and the BPRD entries in the SYS:VERITAS\NBUCLT\NetBack\BP.INI file.

On UNIX servers and clients, check the /etc/inetd.conf file to ensure that it has the following entry:
bpcd stream tcp nowait root /usr/openv/netbackup/bin/bpcd bpcd

5 6 7

On Windows servers and clients, verify that the NetBackup Client service is running. If you use NIS in your network, update those services to include the NetBackup information that is added to the /etc/services file. NIS, WINS, or DNS host name information must correspond to what is in the policy configuration and the name entries. On Windows NetBackup servers, Microsoft Windows clients, and NetWare nontarget clients, do the following:

Check the General tab: Start the Backup, Archive, and Restore interface on the client. On the File menu, click NetBackup Client Properties. In the NetBackup Client Properties dialog box, click the General tab. Check the Server to use for backups and restores drop-down list: Start the Backup, Archive, and Restore interface on the client. On the File menu, click Specify NetBackup Machines and Policy Type. In the Specify NetBackup Machines and Policy Type dialog box, click the Server to use for backups and restores drop-down list. The bp.conf file on UNIX servers and clients and Macintosh clients. The \veritas\nbuclt\netback\bp.ini file on NetWare clients.

60

Troubleshooting procedures Verifying host name and service entries in NetBackup

Also, verify that reverse DNS addressing is configured.

To confirm the setup, use the NetBackup bpclntcmd utility: the IP addresses and hostnames in DNS, NIS, and local hosts files on each NetBackup node. See About the bpclntcmd utility on page 68.

Example of host name and service entries on UNIX master server and client
The following illustration shows a UNIX master server with one UNIX client. Figure 2-1 UNIX master server and client

UNIX Master Server

jupiter

Ethernet Policy Client List jupiter mars usr/openv/netbackup/bp.conf SERVER=jupiter CLIENT_NAME=jupiter usr/openv/netbackup/bp.conf /etc/inetd.conf bpcd ... (see note 1) /etc/services bpcd ... (see note 1) # NetBackup services bpcd 13782/tcp bpcd bprd 13720/tcp bprd bpdbm 13721/tcp bpdbm # Volume Manager services # vmd 13701/tcp vmd tl8cd 13705/tcp tl8cd . . /etc/services # NetBackup services bpcd 13782/tcp bpcd bprd 13720/tcp bprd SERVER=jupiter CLIENT_NAME=mars /etc/inetd.conf

mars

UNIX Client

Consider the following notes about Figure 2-1:

Troubleshooting procedures Verifying host name and service entries in NetBackup

61

The following is the complete inetd.conf entry:

bpcd stream tcp nowait root /usr/openv/netbackup/bin/bpcd bpcd

All other applicable network configuration must also be updated to reflect the NetBackup information. For example, this information could include the /etc/hosts file and NIS, and DNS (if used).

Example of host name and service entries on UNIX master server and media server
The following illustration shows a UNIX NetBackup media server named saturn. Note the addition of a SERVER entry for saturn in the bp.conf files on all the systems. This entry is second, beneath the one for the master server jupiter.

62

Troubleshooting procedures Verifying host name and service entries in NetBackup

Figure 2-2
UNIX Master Server

UNIX master and media servers

UNIX Media Server

jupiter

saturn

Ethernet Policy Client List jupiter mars saturn usr/openv/netbackup/bp.conf SERVER=jupiter SERVER=saturn CLIENT_NAME=jupiter /etc/inetd.conf bpcd ... (see note 1) /etc/services # NetBackup services bpcd 13782/tcp bpcd bprd 13720/tcp bprd bpdbm 13721/tcp bpdbm # Volume Manager services # vmd 13701/tcp vmd tl8cd 13705/tcp tl8cd odld 13706/tcp odld . . usr/openv/netbackup/bp.conf SERVER=jupiter SERVER=saturn CLIENT_NAME=mars /etc/inetd.conf bpcd ... bpcd (see note 1) /etc/services # NetBackup services bpcd 13782/tcp bpcd bprd 13720/tcp bprd usr/openv/netbackup/bp.conf SERVER=jupiter SERVER=saturn CLIENT_NAME=saturn /etc/inetd.conf bpcd ... bpcd (see note 1) /etc/services # NetBackup services bpcd 13782/tcp bpcd bprd 13720/tcp bprd bpdbm 13721/tcp bpdbm # Volume Manager services # vmd 13701/tcp vmd tl8cd 13705/tcp tl8cd odld 13706/tcp odld . .

mars

UNIX Client

Consider the following notes about Figure 2-2:

The following is the complete inetd.conf entry:

bpcd stream tcp nowait root /usr/openv/netbackup/bin/bpcd bpcd

All other applicable network configuration must also be updated to reflect the NetBackup information. For example, this information could include the /etc/hosts file and NIS, and DNS (if used).

Troubleshooting procedures Verifying host name and service entries in NetBackup

63

Example of host name and service entries on UNIX PC clients

The following illustration shows a NetBackup master server with PC clients, defined here as Windows, NetWare, or Macintosh clients. Server configuration is the same as it is for UNIX clients. These clients do not have inetd.conf entries. Figure 2-3
UNIX Master Server

UNIX PC clients

jupiter
Ethernet

Policy Client List jupiter mars saturn pluto bp.ini usr/openv/netbackup/bp.conf SERVER=jupiter CLIENT_NAME=jupiter /etc/inetd.conf bpcd ... (see note 1) /etc/services # NetBackup services bpcd 13782/tcp bpcd bprd 13720/tcp bprd bpdbm 13721/tcp bpdbm # Volume Manager services # vmd 13701/tcp vmd tl8cd 13705/tcp tl8cd odld 13706/tcp odld . .

mars

NetWare Target Client

saturn

Windows Client

[bp] ClientName=mars [servers] master=jupiter [clients] browser=jupiter [tcpip] bpcd=13782 bprd=13720 NetBackup Client Properties dialog box Servers Server List: jupiter General Client Name: saturn Network NetBackup Client Service Port 13782 NetBackup Request Service Port 13720

Consider the following notes about Figure 2-3:

The following is the complete inetd.conf entry:

bpcd stream tcp nowait root /usr/openv/netbackup/bin/bpcd bpcd

64

Troubleshooting procedures Verifying host name and service entries in NetBackup

All other applicable network configuration must also be updated to reflect the NetBackup information. For example, this information could include the /etc/hosts file and NIS, and DNS (if used).

Example of host name and service entries on UNIX clients in multiple networks
The following illustration shows a client that is a router to clients in another network. The client host name on the master server side is mars and the host name that is presented to the client pluto is meteor. Figure 2-4
UNIX Master Server

UNIX clients in multiple networks

UNIX Media Server Ethernet

jupiter

saturn

Policy Client List jupiter mars saturn pluto usr/openv/netbackup/bp.conf SERVER=jupiter SERVER=saturn CLIENT_NAME=jupiter

mars meteor
UNIX Client Ethernet

pluto
usr/openv/netbackup/bp.conf

UNIX Client

/etc/inetd.conf bpcd ... bpcd (see note 1) /etc/services # NetBackup services bpcd 13782/tcp bpcd bprd 13720/tcp bprd bpdbm 13721/tcp bpdbm # Volume Manager services # vmd 13701/tcp vmd tl8cd 13705/tcp tl8cd odld 13706/tcp odld . .

SERVER=jupiter SERVER=saturn CLIENT_NAME=mars /etc/inetd.conf bpcd ... bpcd (see note 1) /etc/services # NetBackup services bpcd 13782/tcp bpcd bprd 13720/tcp bprd

usr/openv/netbackup/bp.conf SERVER=jupiter SERVER=saturn CLIENT_NAME=pluto /etc/inetd.conf bpcd ... bpcd (see note 1) /etc/services # NetBackup services bpcd 13782/tcp bpcd bprd 13720/tcp bprd

Troubleshooting procedures Verifying host name and service entries in NetBackup

65

Consider the following notes about Figure 2-4:

The following is the complete inetd.conf entry:

bpcd stream tcp nowait root /usr/openv/netbackup/bin/bpcd bpcd

All other applicable network configuration must also be updated to reflect the NetBackup information. For example, this information could include the /etc/hosts file and NIS, and DNS (if used).

The policy client list shows the configuration of the router system as mars because that is the name of the interface to the master server. Other than the client name setting, this setup has no special configuration. This name must be set to mars, because mars is the name that the master server recognizes. The second client, pluto, is also configured no differently than if it were in the same network as the master server. If all the standard networking files (hosts, NIS, DNS, WINS, and routing tables) are set up correctly, all the required network connections can be made. However, to restore files from pluto would be a problem in the following situation: the mars-meteor system is a type of router that hides the name of the originating host when it routes requests between the two networks. For example, a router between an Ethernet and a token ring network exhibits this behavior. To illustrate what occurs, assume that pluto is on FDDI (token ring) and the server is on Ethernet. Then a user on pluto starts a restore. The router can use the name of its network interface to pluto (meteor) as the peer name when it forwards the request to the server. The server interprets the request as coming from a host that is named meteor. It does not allow the restore because meteor is not in the client list. To resolve this problem, the administrator creates an altnames directory on the master server and adds a file for meteor to that directory. On a Windows NetBackup server, the file path is:
install_path\netbackup\db\altnames\meteor

On a UNIX NetBackup server, the file path is:

/usr/openv/netbackup/db/altnames/meteor

Then, the administrator adds the following line to this file:

pluto

The master server now recognizes as legitimate any of the restore requests with a peer name of meteor and client name of pluto.

66

Troubleshooting procedures Verifying host name and service entries in NetBackup

For more information, see the NetBackup Administrators Guide, Volume I. Regardless of the type of router, the configuration for the media server, saturn, is the same as in another example. See Example of host name and service entries on UNIX master server and media server on page 61. If a media server is involved in a backup or restore for pluto, the master server provides the following: the correct peer name and client name for the media server to use to establish connections.

Example of host name and service entries on UNIX server that connects to multiple networks
The following illustration shows an NBU server with two Ethernet connections and clients in both networks. The server host name is jupiter on one and meteor on the other.

Troubleshooting procedures Verifying host name and service entries in NetBackup

67

Figure 2-5
UNIX Client

UNIX server connects to multiple networks

mars
Ethernet

saturn

UNIX Media Server

jupiter meteor

UNIX Master Server Ethernet

Policy Client List jupiter mars saturn pluto usr/openv/netbackup/bp.conf usr/openv/netbackup/bp.conf SERVER=jupiter SERVER=meteor SERVER=saturn CLIENT_NAME=mars /etc/inetd.conf bpcd ... bpcd (see note 1) /etc/services # NetBackup services bpcd 13782/tcp bpcd bprd 13720/tcp bprd SERVER=jupiter SERVER=meteor SERVER=saturn CLIENT_NAME=jupiter /etc/inetd.conf bpcd ... (see note 1) /etc/services # NetBackup services bpcd 13782/tcp bpcd bprd 13720/tcp bprd bpdbm 13721/tcp bpdbm # Volume Manager services # vmd 13701/tcp vmd tl8cd 13705/tcp tl8cd odld 13706/tcp odld . . /etc/inetd.conf bpcd ... bpcd (see note 1) /etc/services # NetBackup services bpcd 13782/tcp bpcd bprd 13720/tcp bprd usr/openv/netbackup/bp.conf SERVER=jupiter SERVER=meteor SERVER=saturn CLIENT_NAME=pluto

pluto

UNIX Client

Consider the following notes about Figure 2-5:

The complete inetd.conf entry is:

bpcd stream tcp nowait root /usr/openv/netbackup/bin/bpcd bpcd

68

Troubleshooting procedures About the bpclntcmd utility

All other applicable network configuration must also be updated to reflect the NetBackup information. For example, this information could include the /etc/hosts file and NIS, and DNS (if used).

This example illustrates a UNIX server that connects to multiple networks. The NetBackup policy client list specifies jupiter as the client name for the master server. The list can show either jupiter or meteor but not both. The NetBackup server list on the master server has entries for both jupiter and meteor. The reason for both is that when the server does a backup, it uses the name that is associated with the client it backs up. For example, it uses the meteor interface when it backs up pluto and the jupiter interface when it backs up mars. The first server entry (master server name) is jupiter because that is the name used to back up the client on the master server. The NetBackup server list for the other systems also has entries for both the jupiter and the meteor interfaces. This setup is recommended to keep the server entries the same on all clients and servers in the configuration. It would be adequate to list only the master-server name for the local network interface to the client system or media server. (For example, list meteor for pluto.) For the network that is shown, the only configurations that are required are the differences for the policy client list and the server list. If all the standard networking files (hosts, WINS, NIS, DNS, and routing tables) are set up correctly, all required network connections can be made. A problem exists to restore the files in the following situation: the master server system is a router that hides the originating host name when it routes requests between networks. For example, if pluto were on FDDI (token ring), the master server would use meteor as the peer name when it forwards the request to NetBackup. NetBackup would then interpret the request as coming from a host that is named meteor, which was not in the client list. The restore would fail. The solution, in this case, is identical to the solution that is discussed in the following:

About the bpclntcmd utility

The bpclntcmd utility resolves IP addresses into host names and host names into IP addresses. It uses the same system calls as the NetBackup application software. The following directory contains the command that starts the utility:
Windows UNIX install_path\NetBackup\bin /usr/openv/netbackup/bin

Troubleshooting procedures About the bpclntcmd utility

69

On Windows, run this bpclntcmd command in an MS-DOS command window so you can see the results. The bpclntcmd options that are useful for testing the functionality of the host name and IP address resolution are -ip, -hn, -sv, and -pn. The following topics explain each of these options:
-ip bpclntcmd -ip IP_Address The -ip option lets you specify an IP address. bpclntcmd uses gethostbyaddr() on the NetBackup node and gethostbyaddr() returns the host name with the IP address as defined in the following: the nodes DNS, WINS, NIS, or local hosts file entries. No connection is established with the NetBackup server. -hn bpclntcmd -hn Hostname The -hn option specifies a host name. bpclntcmd uses gethostbyname() on the NetBackup node to obtain the IP address that is associated with the host name defined in the following: the nodes DNS, WINS, NIS, or local hosts file entries. No connection is established with the NetBackup server. -sv bpclntcmd -sv The -sv option displays the NetBackup version number on the master server. -pn When the -pn option is run on a NetBackup client, it initiates an inquiry to the NetBackup master server. The server then returns information to the requesting client. First, the server is the Current Server in the server list. Then it displays the information that the server returns. For example: bpclntcmd -pn expecting response from server rabbit.friendlyanimals.com dove.friendlyanimals.com dove 123.145.167.3 57141 The following is true of this command example:

expecting response from server rabbit.friendlyanimals.com is the master server entry from the server list on the client. dove.friendlyanimals.com is the connection name (peer name) returned by the master server. The master server obtained this name through gethostbyaddress().

dove is the client name configured in the NetBackup policy client list.

123.145.167.3 is the IP address of the client connection at the master server. 57141 is the port number of the connection on the client.

70

Troubleshooting procedures Using the Host Properties window to access configuration settings

Use -ip and -hn to verify the ability of a NetBackup node to resolve the IP addresses and host names of other NetBackup nodes. For example, to verify that a NetBackup server can connect to a client, do the following:

On the NetBackup server, use bpclntcmd -hn to verify the following: The operating system can resolve the host name of the NetBackup client (as configured in the client list for the policy) to an IP address. The IP address is then used in the nodes routing tables to route a network message from the NetBackup server. On the NetBackup client, use bpclntcmd -ip to verify that the operating system can resolve the IP address of the NetBackup server. (The IP address is in the message that arrives at the clients network interface.)

Using the Host Properties window to access configuration settings

The Host Properties window in the NetBackup Administration Console provides access to many configuration settings for NetBackup clients and servers. For example, you can modify the server list, email notification settings, and various timeout values for servers and clients. The following are general instructions for using this window. Many procedures in this guide also refer to the NetBackup Client Properties dialog box in the Backup, Archive, and Restore interface on Microsoft Windows clients. This dialog box lets you change NetBackup configuration settings only for the local system where you are running the interface. Most settings in the NetBackup Client Properties dialog box are also available in the Host Properties window. To use the Host Properties window to access configuration settings

1 2 3 4

In the NetBackup Administration Console, in the left pane, expand NetBackup Management > Host Properties. Depending on the host to be configured, select Master Servers, Media Servers, or Clients. On the Actions menu, select Properties. In the Properties dialog box, in the left pane, click the appropriate property and make your change.

Troubleshooting procedures Resolving full disk problems

71

Resolving full disk problems

If the NetBackup installation directory fills up, such as with logging files, a number of problems can result. NetBackup may become unresponsive. For example, NetBackup jobs may remain queued for long periods, even though all NetBackup processes and services are running. To resolve full disk problems

The NetBackup Resource Broker (nbrb) log may have database connection errors in it. These errors indicate failed tries to establish connections to the nbemm database. The following is an example of such errors in the nbrb log:

7/20/2005 12:33:47.239 [RBDatabase::connectDatabase()] ODBC connection failed. ErrMsg: [Sybase][ODBC Driver][Adaptive Server Anywhere]Disk write failure 'Fatal error: disk write failure C:\Program Files\VERITAS\NetBackupDB\data\NBDB.log' -transaction rolled back ErrCode: -1Sqlstate: HY000

The nbrb log (originator ID 118) is written in /usr/openv/logs (UNIX) or install_path\NetBackup\logs (Windows). More information is available about unified logging. See About logs on page 101.

To correct the situation, clear up disk space in the directory where NetBackup is installed by doing the following:

You may need to delete log files manually, reduce logging levels, and adjust log retention to have log files automatically deleted sooner. More information is available about logging levels, log file retention, and how to configure unified logging. See About logs on page 101. Consider moving the NetBackup unified logging files to a different file system. See About changing the location of unified log files on page 114.

Use the Activity Monitor to verify that the NetBackup relational database service is running. This service is the NB_dbsrv daemon on UNIX and the "Adaptive Server Anywhere - Veritas_NB" service on Windows. If the NetBackup relational database service is stopped, note the following:

Do not stop the nbrb service. If you stop the nbrb service while the NetBackup relational database service is down, it can result in errors. Restart the NetBackup relational database service.

72

Troubleshooting procedures Frozen media troubleshooting considerations

Verify that the NetBackup relational database service is running. If it is not and you remove files to free up disk space, you may not fix the problem. The relational database service must be restarted to allow the Resource Broker (nbrb) to allocate job resources.

Frozen media troubleshooting considerations

Frozen media can cause a number of problems including one of the following status codes: 84, 85, 86, 87 and 96. When troubleshooting frozen media, be aware of the following:

Be sure that the media server that freezes the media stores the actual FROZEN status of that media in its media database (MediaDB). Every media server including the master server has its own unique media database. Use the bpmedialist command to access the MediaDB information including the media status (Frozen, Full, or Active). To unfreeze the media, use the bpmedia command. Specify the media server that contains that frozen record in the command syntax. Unfreeze the media one at a time. Frozen media does not necessarily mean that the media is defective. NetBackup may freeze media as a safety measure to prevent further errors, drive damage, or data loss. Investigate any patterns to the media IDs, tape drives, or media servers that are involved when media is frozen.

Logs for troubleshooting frozen media

The following logs are useful when troubleshooting frozen media:
UNIX and Linux

The bptm log from the media servers that froze the media: /usr/openv/netbackup/logs/bptm

The Admin messages or syslog from the operating system. The bptm log from the media servers that froze the media: install_dir\VERITAS\NetBackup\logs\bptm

Windows

The Windows Event Viewer System Log The Windows Event Viewer Application Log

Troubleshooting procedures Frozen media troubleshooting considerations

73

Set the verbosity of the bptm process log to 5 to troubleshoot any media and drive-related issues. This log does not use excessive drive space or resources even at an elevated verbosity. When media is frozen, the bptm logs may contain more detailed information that the Activity Monitor or Problems Report. Set the verbosity for bptm on individual media servers by changing their logging levels under Host Properties on the NetBackup Administration Console. See Frozen media troubleshooting considerations on page 72. See About conditions that cause media to freeze on page 73.

About conditions that cause media to freeze

The following conditions can cause media to freeze:

The same media has excessive errors during backup. An example of the log entry is as follows:
FREEZING media id E00109, it has had at least 3 errors in the last 12 hour(s)

Causes and resolutions for this problem include:

Dirty drives Clean the drives that are freezing the media according to the manufacturer's suggestions.. One of the first symptoms of a dirty drive is frozen media. Check for tape device errors that are reported by the operating system logs or by the device driver. If any are found, follow the hardware manufacturer's recommendations for this type of error.

The drive itself

Communication issues Check for SCSI or HBA device errors that are reported by the at the SCSI or host bus operating system logs or by their driver. If any are found, adapter (HBA) level follow the hardware manufacturer's recommendations for this type of error. Drive not supported Ensure that the tape drives appear on the hardware compatibility list as supported for NetBackup. This list is located on the following Symantec support Web site: http://www.symantec.com/business/support/overview.jsp?pid=15143 Media not supported Ensure that the media is supported for use with the tape drive by the tape drive vendor.

An unexpected media is found in the drive. An example of the log entry is as follows:

74

Troubleshooting procedures Frozen media troubleshooting considerations

Incorrect media found in drive index 2, expected 30349, found 20244, FREEZING 30349

The following conditions can cause this error:

NetBackup requests a media ID to be mounted in a drive. If the media ID that is physically recorded on the tape is different than the NetBackup media ID, the media freezes. This error occurs if the robot needs to be inventoried, or if bar codes have been physically changed on the media. Another NetBackup installation previously wrote to the media with different barcode rules. The drives in the robot are not configured in order within NetBackup, or they are configured with the wrong tape paths. The correct robot drive number is important to the proper mounting and use of media. The robot drive number is normally based on the relationship of the drive serial number with the drive serial number information from the robotic library. Validate this number before you consider that the device configuration is complete.

The media contain a non-NetBackup format. An example of the log entry is as follows:
FREEZING media id 000438, it contains MTF1-format data and cannot be used for backups FREEZING media id 000414, it contains tar-format data and cannot be used for backups FREEZING media id 000199, it contains ANSI-format data and cannot be used for backups

These library tapes may have been written outside of NetBackup. By default, NetBackup only writes to a blank media or other NetBackup media. Other media types (DBR, TAR, CPIO, ANSI, MTF1, and recycled Backup Exec BE-MTF1 media) are frozen as a safety measure. Change this behavior by using the following procedure:

Troubleshooting procedures Frozen media troubleshooting considerations

75

On UNIX and Linux

To allow NetBackup to overwrite foreign media, add the following to the bp.conf file that is located at /usr/openv/netbackup/bp.conf for the related media server: ALLOW_MEDIA_OVERWRITE ALLOW_MEDIA_OVERWRITE ALLOW_MEDIA_OVERWRITE ALLOW_MEDIA_OVERWRITE ALLOW_MEDIA_OVERWRITE ALLOW_MEDIA_OVERWRITE = = = = = = DBR TAR CPIO ANSI MTF1 BE-MTF1

Stop and restart the NetBackup daemons for the changes to take effect. On Windows On the Administration Console, proceed to Host Properties | Media Server Open the properties for the media server in question. Select the Media tab. The Allow Media Overwrite property overrides the NetBackup overwrite protection for specific media types. To disable the overwrite protection, select one or more of the listed media formats. Then stop and restart the NetBackup services for the changes to take effect. Do not select a foreign media type for overwriting unless you are sure that you want to overwrite this media type. More details are available on each media type. For more information, see the NetBackup Device Configuration Guide.

The media is a tape formerly used for the NetBackup catalog backup. For example, the log entry may be the following:
FREEZING media id 000067: it contains Symantec NetBackup (tm) database backup data and cannot be used for backups.

The media is frozen because it is an old catalog backup tape which NetBackup does not overwrite by default. The bplabel command must label the media to reset the media header.

The media is intentionally frozen. You can use the bpmedia command to manually freeze media for a variety of administrative reasons. If no record exists of a specific job freezing the media, the media may have been frozen manually.

76

Troubleshooting procedures Resolving PBX problems

The media is physically write protected. If the media has a write-protect notch that is set for write protection, NetBackup freezes the media.

To unfreeze frozen media, enter the following bpmedia command:

# bpmedia -unfreeze -m mediaID -h media_server

The media_server variable is the one that froze the media. If this item is unknown, run the bpmedialist command and note the "Server Host:" listed in the output. The following example shows that media server denton froze media div008:
# bpmedialist -m div008 Server Host = denton ID rl images allocated last updated density kbytes restores vimages expiration last read <------- STATUS -------> -----------------------------------------------------------------------DIV08 1 1 1 04/22/2010 10:12 04/22/2010 10:12 05/06/2010 10:12 04/22/2010 10:25 hcart FROZEN 35 5

Resolving PBX problems

The Enterprise Media Manager (EMM) services and other services of NetBackup require a common services framework that is called Private Branch Exchange (PBX). Like vnetd, PBX helps limit the number of TCP/IP ports that the CORBA services of NetBackup use. To resolve PBX problems

Check that the PBX is properly installed. If PBX is not installed, NetBackup is unresponsive. Refer to the following procedure: See Checking PBX installation on page 77.

Check that PBX is running, and initiate PBX if necessary by using the following procedure: See Checking that PBX is running on page 77.

Check that PBX is correctly configured. If PBX is incorrectly configured, NetBackup is unresponsive. Refer to the following procedure: See Checking that PBX is set correctly on page 78.

Access and check the PBX logs by using the following procedure: See Accessing the PBX logs on page 79.

Troubleshooting procedures Resolving PBX problems

77

Check the PBX security and correct any problem by using the following procedure: See Troubleshooting PBX security on page 80.

Check that the required NetBackup daemon or service is running. If necessary, start the needed daemon or service by using the following procedure: See Determining if the PBX daemon or service is available on page 81.

Checking PBX installation

NetBackup requires the Symantec Private Branch Exchange service (PBX). PBX can be installed before NetBackup or during NetBackup installation. See the NetBackup Installation Guide. If you uninstall PBX, you must reinstall it. To check PBX installation

Look for the following directory on the NetBackup master server:

On UNIX: /opt/VRTSpbx On Windows: install_path\VxPBX

To check the version of PBX, enter the following:

On UNIX: /opt/VRTSpbx/bin/pbxcfg -v On Windows: install_path\VxPBX\bin\pbxcfg -v

Checking that PBX is running

After you know that PBX is installed on the NetBackup master server, you need to verify that it is running. To see if PBX is running

On UNIX, check for the PBX process:

ps | grep pbx_exchange

To start PBX on UNIX, type the following:

/opt/VRTSpbx/bin/vxpbx_exchanged start

On Windows, make sure the Symantec Private Branch Exchange service is started. (Go to Start > Run and enter services.msc.)

78

Troubleshooting procedures Resolving PBX problems

Checking that PBX is set correctly

Two settings are vital to the correct functioning of PBX: Auth User (authenticated user) and Secure Mode. When PBX is installed, they are automatically set as required. To check that PBX is set correctly

To display the current PBX settings, do one of the following:

On UNIX, type the following:

/opt/VRTSpbx/bin/pbxcfg -p

Example output:
Auth User:0 : root Secure Mode: false Debug Level: 10 Port Number: 1556 PBX service is not cluster configured Auth User must be root and Secure Mode must be false.

On Windows, type the following:

install_path\VxPBX\bin\pbxcfg -p

Example output:
Auth User:0 : localsystem Secure Mode: false Debug Level: 10 Port Number: 1556 PBX service is not cluster configured Auth User must be localsystem and Secure Mode must be false.

Reset Auth User or Secure Mode as needed:

To add the correct user to the authenticated user list (UNIX example):
/opt/VRTSpbx/bin/pbxcfg -a -u root

To set Secure Mode to false:

/opt/VRTSpbx/bin/pbxcfg -d -m

For more information on the pbxcfg command, refer to the pbxcfg man page.

Troubleshooting procedures Resolving PBX problems

79

Accessing the PBX logs

PBX uses unified logging. PBX logs are written to the following:

/opt/VRTSpbx/log (UNIX) install_path\VxPBX\log (Windows)

The unified logging originator number for PBX is 103. More information is available about unified logging. See About unified logging on page 103. Error messages regarding PBX may appear in the PBX log or in the unified logging logs for nbemm, nbpem, nbrb, or nbjm. The following is an example of an error that is related to PBX:
05/11/10 10:36:37.368 [Critical] V-137-6 failed to initialize ORB: check to see if PBX is running or if service has permissions to connect to PBX. Check PBX logs for details

To access the PBX logs

Use the vxlogview command to view PBX and other unified logs. The originator ID for PBX is 103. For more information, see the vxlogview man page. You can also refer to the following topic: See About unified logging on page 103.

To change the logging level for PBX, enter the following:

pbxcfg -s -l debug_level

where debug_level is a number from 0 to 10, where the settings 10 is the most verbose. PBX may log messages by default to the UNIX system logs (/var/adm/messages or/var/adm/syslog) or to the Windows Event Log. As a result, the system logs may fill up with unnecessary PBX log messages, since the messages are also written to the PBX logs (/opt/VRTSpbx/log on UNIX and <install_path>\VxPBX\log on Windows).

To disable PBX logging to the system or event logs, enter the following command:
# vxlogcfg -a -p 50936 -o 103 -s LogToOslog=false

You do not have to restart PBX for this setting to take effect.

80

Troubleshooting procedures Resolving PBX problems

Troubleshooting PBX security

The PBX Secure Mode must be set to false. If Secure Mode is true, NetBackup commands such as bplabel and vmoprcmd do not work. PBX messages similar to the following appear in /opt/VRTSpbx/log (UNIX) or install_path\VxPBX\log (Windows).
5/12/2008 16:32:17.477 [Error] V-103-11 User MINOV\Administrator not authorized to register servers 5/12/2008 16:32:17.477 [Error] Unauthorized Server

To troubleshoot PBX security

Set Secure Mode to false by entering the following:

On UNIX:
/opt/VRTSpbx/bin/pbxcfg -d -m

On Windows:
install_path\VxPBX\bin\pbxcfg -d -m

Verity the PBX security settings by entering the following:

pbxcfg -p

Stop NetBackup:

On UNIX:
/usr/openv/netbackup/bin/bp.kill_all

On Windows:
install_path\NetBackup\bin\bpdown

Stop PBX:

On UNIX:
/opt/VRTSpbx/bin/vxpbx_exchanged stop

On Windows: Go to Start > Run, enter services.msc, and stop the Symantec Private Branch Exchange service.

Start PBX:

Troubleshooting procedures Resolving PBX problems

81

On UNIX:
/opt/VRTSpbx/bin/vxpbx_exchanged start

On Windows: Go to Start > Run, enter services.msc, and start the Symantec Private Branch Exchange service.

Start NetBackup:

On UNIX:
/usr/openv/netbackup/bin/bp.start_all

On Windows:
install_path\NetBackup\bin\bpup

Determining if the PBX daemon or service is available

If NetBackup does not work as configured, a required NetBackup service may have stopped. For example, backups may not be scheduled or may be scheduled but are not running. The type of problem depends on which process is not running. When a NetBackup service is not running and another process tries to connect to it, messages similar to the following appear in /usr/openv/logs for PBX. (The unified logging originator for PBX is 103.)
05/17/10 9:00:47.79 05/17/10 9:00:47.79 05/17/10 9:00:47.79 extension=EMM 05/17/10 9:00:47.80 05/17/10 9:00:47.80 05/17/10 9:00:47.80 [Info] PBX_Manager:: handle_input with fd = 4 [Info] PBX_Client_Proxy::parse_line, line = ack=1 [Info] PBX_Client_Proxy::parse_line, line = [Info] hand_off looking for proxy for = EMM [Error] No proxy found. [Info] PBX_Client_Proxy::handle_close

To determine if the PBX daemon or service is available

Start the needed service. In this example, the missing NetBackup service is EMM. To start the needed service, enter the nbemm command (UNIX) or start the NetBackup Enterprise Media Manager service (Windows; Start > Run, enter services.msc).

If necessary, stop and restart all NetBackup services.

On UNIX:

82

Troubleshooting procedures About troubleshooting Auto Image Replication

/usr/openv/netbackup/bin/bp.kill_all /usr/openv/netbackup/bin/bp.start_all

On Windows:
install_path\NetBackup\bin\bpdown install_path\NetBackup\bin\bpup

About troubleshooting Auto Image Replication

Auto Image Replication operations are characterized by storage lifecycle policies in at least two NetBackup master server domains. Verify that the two master servers follow these rules:

The name of the storage lifecycle policy in the source master server domain must match the name of the storage lifecycle policy in the target master server domain. The names are case sensitive. The name of the data classification used by the storage lifecycle policy in the source master server domain must match the name of the data classificationn in the storage lifecycle policy in the target master server domain. The names are case sensitive. The duplicate-to-remote-master copy in the source storage lifecycle policy must use hierarchical duplication and specify a source copy with a residence capable of replication. (The disk pool replication column must show Source.) The storage lifecycle policy in the target domain must specify an import for its first copy. The residence for the import must include the device that is the replication partner of the source copy in the source storage lifecycle policy. The import copy may specify a storage unit group or a storage unit but not Any Available. The storage lifecycle policy in the target domain must have at least one copy that specifies the Remote Retention type.

Troubleshooting Auto Image Replication

Auto Image Replication is the process of replicating backups generated in one NetBackup domain to another media server in one or more NetBackup domains. Note: Although Auto Image Replication supports replication across different master server domains, the Replication Director does not.

Troubleshooting procedures About troubleshooting Auto Image Replication

83

Auto Image Replication operates like any duplication job except that its job contains no write side. The job must run on a media server that runs NetBackup 7.1 or higher. It also must consume a read resource from the disk volume on which the duplicated images reside. If no media server is available with NetBackup 7.1 or higher, the job fails with status 800. The Auto Image Replication job operates at a disk volume level. Within the storage unit specified in the storage lifecycle policy for the source copy, some disk volumes may not support replication and some media servers may not be running NetBackup 7.1 or higher. Use the Disk Pools interface of the System Administration Console to verify that the image is on a disk volume that supports replication. If the interface shows that the disk volume is not a replication source, click Update Replication to update the disk volumes in the disk pool. If the problem persists, check your disk device configuration. The following procedure is based on NetBackup that operates in an OpenStorage configuration. This configuration communicates with an media server deduplication pool (MSDP) that uses Auto Image Replication.

84

Troubleshooting procedures About troubleshooting Auto Image Replication

To troubleshoot Auto Image Replication jobs

Display the storage server information by using the following command:

# bpstsinfo -lsuinfo -stype PureDisk -storage_server woodridge LSU Info: Server Name: PureDisk:woodridge.min.veritas.com LSU Name: PureDiskVolume Allocation : STS_LSU_AT_STATIC Storage: STS_LSU_ST_NONE Description: PureDisk storage unit (/woodridge.min.veritas.com#1/2) Configuration: Media: (STS_LSUF_DISK | STS_LSUF_ACTIVE | STS_LSUF_STORAGE_NOT_FREED | STS_LSUF_REP_ENABLED | STS_LSUF_REP_SOURCE) Save As : (STS_SA_CLEARF | STS_SA_OPAQUEF | STS_SA_IMAGE) Replication Sources: 0 ( ) Replication Targets: 1 ( PureDisk:bayside:PureDiskVolume ) ...

This output shows the logical storage unit (LSU) flags STS_LSUF_REP_ENABLED and STS_LSUF_REP_SOURCE for PureDiskVolume. PureDiskVolume is enabled for Auto Image Replication and is a replication source.

To verify that NetBackup recognizes these two flags, run the following command:
# nbdevconfig -previewdv -stype PureDisk -storage_server woodridge -media_server woodridge -U Disk Pool Name : Disk Type : PureDisk Disk Volume Name : PureDiskVolume ... Flag : ReplicationSource ...

The ReplicationSource flag confirms that NetBackup recognizes the LSU flags.

Troubleshooting procedures About troubleshooting Auto Image Replication

85

To display the replication targets by using the raw output, run the following command:
# nbdevconfig -previewdv -stype PureDisk -storage_server woodridge -media_server woodridge V7.0 DiskVolume < "PureDiskVolume" "PureDiskVolume" 46068048064 46058373120 0 0 0 16 1 > V7.0 ReplicationTarget < "bayside:PureDiskVolume" >

The display shows that the replication target is a storage server called bayside and the LSU (volume) name is PureDiskVolume.

To ensure that NetBackup captured this configuration correctly, run the following command:
# nbdevquery -listdv -stype PureDisk -U Disk Pool Name : PDpool Disk Type : PureDisk Disk Volume Name : PureDiskVolume ... Flag : AdminUp Flag : InternalUp Flag : ReplicationSource Num Read Mounts : 0 ...

The listing shows that disk volume PureDiskVolume is configured in disk pool PDPool, and that NetBackup recognizes the replication capability.

If NetBackup does not recognize the replication capability, run the following command:
# nbdevconfig -updatedv -stype PureDisk -dp PDpool

To ensure that you have a storage unit that uses this disk pool, run the following command:
# bpstulist PDstu 0 _STU_NO_DEV_HOST_ 0 -1 -1 1 0 "*NULL*" 1 1 51200 *NULL* 2 6 0 0 0 0 PDpool *NULL*

The output shows that storage unit PDstu uses disk pool PDpool.

86

Troubleshooting procedures About troubleshooting Auto Image Replication

Check the settings on the disk pool by running the following command:
nbdevquery -listdp Disk Pool Name : Disk Pool Id : Disk Type : Status : Flag : ... Flag : Raw Size (GB) : Usable Size (GB) : Num Volumes : High Watermark : Low Watermark : Max IO Streams : Comment : Storage Server : -stype PureDisk -dp PDpool -U PDpool PDpool PureDisk UP Patchwork OptimizedImage 42.88 42.88 1 98 80 -1 woodridge.min.veritas.com (UP)

Max IO Streams is set to -1, which means the disk pool has unlimited input-output streams.

To check the media servers, run the following command:

# tpconfig -dsh -all_host ============================================================== Media Server: woodridge.min.veritas.com Storage Server: woodridge.min.veritas.com User Id: root Storage Server Type: BasicDisk Storage Server Type: SnapVault Storage Server Type: PureDisk ==============================================================

This disk pool only has one media server, woodridge. You have completed the storage configuration validation.

Troubleshooting procedures About troubleshooting Auto Image Replication

87

The last phase of validation is the storage lifecycle policy configuration. To run Auto Image Replication, the source copy must be on storage unit PDstu. Run the following command:
nbstl woodridge2bayside -L Name: Data Classification: Duplication job priority: State: Version: Destination 1 Use for: Storage: Volume Pool: Server Group: Retention Type: Retention Level: Alternate Read Server: Preserve Multiplexing: Enable Automatic Remote Import: State: Source: Destination ID: Destination 2 Use for: Storage: Volume Pool: Server Group: ... Preserve Multiplexing: Enable Automatic Remote Import: State: Source: Destination ID: woodridge2bayside (none specified) 0 active 0 backup PDstu (none specified) (none specified) Fixed 1 (2 weeks) (none specified) false true active (client) 0 Auto Image Replication Remote Master (none specified) (none specified) false false active Destination 1 (backup:PDstu) 0

To troubleshoot the Auto Image Replication job flow, use the same command lines as you use for other storage lifecycle policy managed jobs. For example, to list images which have been duplicated to remote master, run the following:
nbstlutil list -copy_type replica -U -copy_state 3

To list images which have not been duplicated to remote master (either pending or failed), run the following:
nbstlutil list -copy_type replica -U -copy_incomplete

88

Troubleshooting procedures About troubleshooting Auto Image Replication

10 To list the target storage devices that complete Auto Image Replication copies
(replication destination), run the following command:
nbstlutil repllist Image: Master Server Backup ID Client Backup Time Policy Client Type Schedule Type Storage Lifecycle Policy Storage Lifecycle State Time In Process Data Classification ID Version Number OriginMasterServer OriginMasterServerID Import From Replica Time Required Expiration Date Created Date Time Copy: Master Server Backup ID Copy Number Copy Type Expire Time Expire LC Time Try To Keep Time Residence Copy State Job ID Retention Type MPX State Source Destination ID Last Retry Time

: : : : : : : : : : : : : : : : :

woodridge.min.veritas.com woodridge_1287610477 woodridge 1287610477 (Wed Oct 20 16:34:37 2010) two-hop-with-dup 0 0 woodridge2bayside2pearl_withdup 3 (COMPLETE) 1287610545 (Wed Oct 20 16:35:45 2010) (none specified) 0 (none specified) 00000000-0000-0000-0000-000000000000 0 (Wed Dec 31 18:00:00 1969) 0 (Wed Dec 31 18:00:00 1969) 1287610496 (Wed Oct 20 16:34:56 2010)

: : : : : : : : : : : : : : :

woodridge.min.veritas.com woodridge_1287610477 102 3 1290288877 (Sat Nov 20 15:34:37 2010) 1290288877 (Sat Nov 20 15:34:37 2010) 1290288877 (Sat Nov 20 15:34:37 2010) Remote Master 3 (COMPLETE) 25 0 (FIXED) 0 (FALSE) 1 1287610614

Replication Destination: Source Master Server: woodridge.min.veritas.com Backup ID : woodridge_1287610477

Troubleshooting procedures About troubleshooting Auto Image Replication

89

Copy Number Target Machine Target Info Remote Master

: : : :

102 bayside PureDiskVolume (none specified)

About troubleshooting automatic import jobs

The automatic import jobs managed by the storage lifecycle policy components are different than legacy import jobs. Automatic import jobs asynchronously notify NetBackup that an image needs to be imported. Also, an Auto Image Replication job gives the catalog entries for this copy to the storage device so that the job does not have to read the entire image. Automatic import jobs simply read the catalog record off the storage device and add it into its own catalog. This process is so fast that NetBackup batches images for import for efficiency. A pending import is the state where NetBackup has been notified, but the import has not yet occurred. More information is available about automatic import and how to tune the batch interval of the import manager process. See the NetBackup Administrator's Guide, Volume I. The notify event from the storage server provides the following: the image name, where on the storage server to read the catalog for this image, and the name of the storage lifecycle policy that processes the image. Images for automatic import jobs are batched by storage lifecycle policy name and disk volume. The import job consumes an input-output stream on the disk volume. To view images that are pending import, run the following command:
# nbstlutil pendimplist -U Image: Master Server : Backup ID : Client : Backup Time : Policy : Client Type : Schedule Type : Storage Lifecycle Policy : Storage Lifecycle State : Time In Process : Data Classification ID : Version Number : OriginMasterServer :

bayside.min.veritas.com gdwinlin04_1280299412 gdwinlin04 1280299412 (Wed Jul 28 01:43:32 2010) (none specified) 0 0 (none specified) 1 (NOT_STARTED) 0 (Wed Dec 31 18:00:00 1969) (none specified) 0 master_tlk

90

Troubleshooting procedures About troubleshooting Auto Image Replication

OriginMasterServerID Import From Replica Time Required Expiration Date Created Date Time Copy: Master Server Backup ID Copy Number Copy Type Expire Time Expire LC Time Try To Keep Time Residence Copy State Job ID Retention Type MPX State Source Destination ID Last Retry Time Fragment: Master Server Backup ID Copy Number Fragment Number Resume Count Media ID Media Server Storage Server Media Type Media Sub-Type Fragment State Fragment Size Delete Header Fragment ID

: : : :

00000000-0000-0000-0000-000000000000 0 (Wed Dec 31 18:00:00 1969) 0 (Wed Dec 31 18:00:00 1969) 1287678771 (Thu Oct 21 11:32:51 2010)

: : : : : : : : : : : : : : :

bayside.min.veritas.com gdwinlin04_1280299412 1 4 0 (Wed Dec 31 18:00:00 1969) 0 (Wed Dec 31 18:00:00 1969) 0 (Wed Dec 31 18:00:00 1969) (none specified) 1 (NOT_STARTED) 0 0 (FIXED) 0 (FALSE) 0 0

: : : : : : : : : : : : : :

bayside.min.veritas.com gdwinlin04_1280299412 1 -2147482648 0 @aaaab bayside.min.veritas.com bayside.min.veritas.com 0 (DISK) 0 (DEFAULT) 1 (ACTIVE) 0 1 gdwinlin04_1280299412_C1_IM

The action taken on the automatic import job and the automatic import event depends on several conditions as shown in the following table.

Troubleshooting procedures About troubleshooting Auto Image Replication

91

Action
Automatic import jobs queue

Condition
No media server or I/O stream is available for this disk volume.

Automatic import jobs never start (copy The storage lifecycle policy is inactive. stays at storage lifecycle state 1) The storage lifecycle policy import destination is inactive. The storage lifecycle policy is between sessions. The image has exceeded the extended retry count and the extended retry time has not passed. Automatic import event is discarded and the image is ignored The event specifies a backup ID that already exists in this master server catalog. The event specifies a disk volume that is not configured in NetBackup for this storage server.

Automatic import job is started but the The storage lifecycle policy that is specified image is expired and deleted to clean in the event does not contain an import up disk space in some cases. The event destination. logs an error in the Problems Report or The storage lifecycle policy that is specified bperror output. An import job runs, but in the event has an import destination with a the import for this image fails showing residence that does not include the disk a status code in the range 15321535. volume specified by the event The storage lifecycle policy that is specified does not exist. This is default behavior. More information is available for the storage lifecycle policy configuration options. See the NetBackup Administrator's Guide, Volume I.

Look at the Problems Report or the bperror list for these cases. For troubleshooting job flow for automatic import jobs, use the same command lines as you would for other storage lifecycle policy managed jobs. To list images for which NetBackup has received notification from storage but not yet initiated import (either pending or failed), use the commands noted above or run the following command:
# nbstlutil list -copy_type import -U -copy_incomplete

To list images that have been automatically imported, run the following command:

92

Troubleshooting procedures About troubleshooting Auto Image Replication

# nbstlutil list -copy_type Master Server : Backup ID : Client : Backup Time : Policy : Client Type : Schedule Type : Storage Lifecycle Policy : Storage Lifecycle State : Time In Process : Data Classification ID : Version Number : OriginMasterServer : OriginMasterServerID : Import From Replica Time : Required Expiration Date : Created Date Time :

import -U -copy_state 3 -U bayside.min.veritas.com woodridge_1287610477 woodridge 1287610477 (Wed Oct 20 16:34:37 2010) two-hop-with-dup 0 0 woodridge2bayside2pearl_withdup 3 (COMPLETE) 1287610714 (Wed Oct 20 16:38:34 2010) (none specified) 0 woodridge.min.veritas.com f5cec09a-da74-11df-8000-f5b9412d8988 1287610672 (Wed Oct 20 16:37:52 2010) 1290288877 (Sat Nov 20 15:34:37 2010) 1287610652 (Wed Oct 20 16:37:32 2010)

The OriginMasterServer, OriginMasterServerID, Import From Replica Time, and Required Expiration Date are not known until after the image is imported so a pending record may look like this:
Image: Master Server Backup ID Client Backup Time Policy Client Type Schedule Type Storage Lifecycle Policy Storage Lifecycle State Time In Process Data Classification ID Version Number OriginMasterServer OriginMasterServerID Import From Replica Time Required Expiration Date Created Date Time

: : : : : : : : : : : : : : : : :

bayside.min.veritas.com gdwinlin04_1280299412 gdwinlin04 1280299412 (Wed Jul 28 01:43:32 2010) (none specified) 0 0 (none specified) 1 (NOT_STARTED) 0 (Wed Dec 31 18:00:00 1969) (none specified) 0 master_tlk 00000000-0000-0000-0000-000000000000 0 (Wed Dec 31 18:00:00 1969) 0 (Wed Dec 31 18:00:00 1969) 1287680533 (Thu Oct 21 12:02:13 2010)

The OriginMasterServer here is not empty, although it may be in some cases. In cascading Auto Image Replication, the master server sends the notification.

Troubleshooting procedures Troubleshooting network interface card performance

93

Troubleshooting network interface card performance

If backup or restore jobs are running slowly, verify that the network interface cards (NIC) are set to full duplex. Half duplex often causes poor performance. Note: If the NIC in a NetBackup master or media server is changed, or if the server IP address changes, CORBA communications may be interrupted. To address this situation, stop and restart NetBackup. For help on how to view and reset duplex mode for a particular host or device, consult the manufacturers documentation. If the documentation is not helpful, perform the following procedure. To troubleshoot network interface card performance

1 2

Log on to the host that contains the network interface card whose duplex mode you want to check. Enter the following command to view the current duplex setting.
ifconfig -a

On some operating systems, this command is ipconfig. The following is an example output from a NAS filer:
e0: flags=1948043<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500 inet 10.80.90.91 netmask 0xfffff800 broadcast 10.80.95.255 ether 00:a0:98:01:3c:61 (100tx-fd-up) flowcontrol full e9a: flags=108042<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500 ether 00:07:e9:3e:ca:b4 (auto-unknown-cfg_down) flowcontrol full e9b: flags=108042<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500 ether 00:07:e9:3e:ca:b5 (auto-unknown-cfg_down) flowcontrol full

In this example, the network interface that shows "100tx-fd-up" is running in full duplex. Only interface e0 (the first in the list) is at full duplex. A setting of "auto" is not recommended, because devices can auto-negotiate to half duplex.

94

Troubleshooting procedures About SERVER entries in the bp.conf file

The duplex mode can be reset by using the ifconfig (or ipconfig) command. For example:
ifconfig e0 mediatype 100tx-fd

For most hosts, you can set full-duplex mode permanently, such as in the hosts /etc/rc files. Refer to the hosts documentation for more information.

About SERVER entries in the bp.conf file

On Solaris and Linux systems, every SERVER entry in a client bp.conf file must be a NetBackup master or media server. That is, each system that is listed as a SERVER must have either NetBackup master or media server software installed. The client service on some clients cannot be started if the client name is incorrectly listed as a server. If a bp.conf SERVER entry specifies a NetBackup client-only computer, SAN client backups or restores over Fibre Channel may fail to start. In this case, determine if the nbftclnt process is running on the client. If it is not running, check the nbftclnt unified logging file (OID 200) for errors. You may see the following in the nbftclnt log:
The license is expired or this is not a NBU server. Please check your configuration. Note: unless NBU server, the host name can't be listed as server in NBU configuration.

Remove or correct the SERVER entry in the bp.conf file, restart nbftclnt on the client, and retry the operation. Note: The nbftclnt process on the client must be running before you start a SAN client backup or restore over Fibre Channel.

About unavailable storage unit problems

NetBackup jobs sometimes fail because storage units are unavailable, due to drives that are down or configuration errors, such as referencing an incorrect robot number. NetBackup processes log messages to the NetBackup error log that help you pinpoint and resolve these types of issues. In addition, the Job Details dialog box available from the Activity Monitor contains messages that describe the following:

The resources that the job requests

Troubleshooting procedures About troubleshooting NetBackup in a SAN environment

95

The granted (allocated) resources.

If a job is queued awaiting resources, the Job Details dialog lists the resources for which the job waits. The three types of messages begin with the following headers:
requesting resource ... awaiting resource ... granted resource ...

About troubleshooting NetBackup in a SAN environment

NetBackup administrators may encounter any or all of the following common problems in a SAN (storage area network) environment:

Intermittent backup failures Connectivity issues (drives that are down) SAN configuration changes

If the SAN administrator rezones the network or masks an array in use by NetBackup, some of the devices that NetBackup needs may be unavailable. Either action causes backups to fail and drives to go down. The only information available to the NetBackup administrator is an error 83 (media open error) or error 84 (media write error) status code. You can use Veritas CommandCentral Storage to check elements of the SAN configuration. For example, you can check whether a particular device is connected as well as the zoning and masking on the SAN. Sometimes a switch or a Windows box is interrupted and sends out a reset command. Since NetBackup doesnt automatically maintain persistent bindings, the reset command can cause drives to be mapped differently. CommandCentral Storage can help find the problem by showing the changes in the drive mappings, although it cannot automatically fix the problem. For information on how to implement persistent bindings, refer to the NetBackup Device Configuration Guide. NetBackup lets you launch CommandCentral Storage in context. The CommandCentral Storage Web GUI precisely displays the area of the SAN configuration you plan to troubleshoot.

96

Troubleshooting procedures About troubleshooting NetBackup in a SAN environment

NetBackup enterprise lifecycle best practices

SAN-related problems generally involve the use of Shared Storage Option (SSO). The two types of NetBackup users generally are as follows:

Operators who have limited access to hosts and to the fabric of the SAN System administrators who have administrator privileges, but no access to the fabric

The SAN administrator generally operates outside the NetBackup domain entirely. Troubleshooting NetBackup is difficult when it involves the SAN because administrative responsibility tends to be spread out. No one person has a clear picture of the overall backup structure. CommandCentral Storage provides a consistent view of the entire SAN against which to measure performance. It gives NetBackup administrators the data they need to request changes of and collaborate with the SAN administrators. It helps NetBackup administrators when they design, configure, implement, or modify solutions in response to changes in backup environments (hardware, applications, demand). CommandCentral Storage can help those responsible for managing a backup system in a SAN environment by integrating SAN management and backup operation information. CommandCentral Storage can provide support during the following backup lifecycle stages:

Design Use CommandCentral Storage during the design phase to determine the following:

Where to deploy a backup system on the SAN If SAN redesign is required to meet backup windows at minimum hardware cost and application impact For example, a backup design may not require the purchase of additional switches if it takes into account the following: the performance trending reports that CommandCentral Storage keeps to determine the pattern of fabric utilization. Or perhaps if you re-zone the fabric through CommandCentral Storage, it may provide sufficient bandwidth for meeting backup window requirements. In addition, CommandCentral Storage can provide visibility into recovery designs and fabric performance in the event of large restores that critical business operations require.

Configuration, testing

Troubleshooting procedures About troubleshooting NetBackup in a SAN environment

97

Generally, backup systems are tested before implementation to obtain benchmarks and adjust (tune) the system for maximum efficiency. CommandCentral Storage can provide the performance metrics for end-to-end I/O capabilities for all elements in the backup path. Additionally, CommandCentral Storage can provide valuable environmental information for qualifying the backup environment as well as a baseline for future troubleshooting configuration management.

Implementation, reconfiguration, production CommandCentral Storage can help to determine whether a host can see through the entire I/O path to the target backup device by pinpointing connectivity issues.

About using CommandCentral Storage to troubleshoot NetBackup in a SAN environment

You can use CommandCentral Storage in the following ways to troubleshoot NetBackup in a SAN environment:
In-context launch The ability to launch CommandCentral Storage and access an overview of the SAN from NetBackup in context is valuable for determining root cause problems quickly. In addition, because NetBackup administrators and SAN administrators are often in different groups, the fragmented operations that lead to resolution delays may be avoided. With CommandCentral Storage, the NetBackup administrator has a view of the overall health of the SAN as part of the initial troubleshooting process. The CommandCentral Storage view of the SAN environment can help you detect any failure in the topology. In addition, having an environment inventory to provide to support for troubleshooting is valuable to the support process. General troubleshooting tools Some ways to investigate a backup failure are as follows: Launch CommandCentral Storage in context from NetBackup to check fabric health. Check reports for fabric events occurring around the time NetBackup generated the error log.

Connectivity and device check

98

Troubleshooting procedures About troubleshooting NetBackup in a SAN environment

Using CommandCentral Storage to troubleshoot the inability to access drives or robots in a SAN environment
The following use case demonstrates how CommandCentral Storage can be integrated into a NetBackup troubleshooting procedure to investigate the SAN context of a backup system. Most common NetBackup problems on SANs revolve around connectivity issues. Typically found as an error 213 (no storage units available for use) in NetBackup, this problem represents a loss of connectivity. This issue is a problem because NetBackup freezes tapes with two write failures, even when SAN problems cause the failures. Symptom: Backup jobs fail To use CommandCentral Storage to troubleshoot the inability to access drives or robots in a SAN environment

1 2

Check the NetBackup device monitor to see whether a device is down. If a device is down, try to bring it back up. If the drive is still down, check the following for status 219 (the required storage unit is unavailable) and 213 (no storage units available for use) on the media server:

Syslog Device logs NetBackup logs

3 4

Check the NetBackup logs for status 83, 84, 85, and 86. These codes relate to write, read, open, position failures to access the drive. Try a robtest to determine connectivity. If there is no connectivity, the likely problem is with hardware.

5 6 7

From the master server, select the robot or device the storage unit is associated with. Launch CommandCentral Storage for a view of the media server and devices. Check the fabric connectivity (whether any I/O path devices are down).

Using CommandCentral Storage to troubleshoot the inability to discover a drive or robot in a SAN environment
The following use case demonstrates how CommandCentral Storage can be integrated into a NetBackup troubleshooting procedure to investigate the SAN

Troubleshooting procedures About troubleshooting NetBackup in a SAN environment

99

context of a backup system. Most common NetBackup problems on SANs revolve around connectivity issues. The NetBackup administrator installs a new device and runs the Device Configuration Wizard to discover and configure it. The wizard does not see the newly installed device. CommandCentral Storage topology is a good visual tool to check connectivity between the hosts and the devices. You can use it to see if a network cable was dislodged or if some other problem exists. This use case may be encountered when you configure off-host backups. Off-host backups require the media server to be able to see all devices with which it conducts the backup: disk array, disk cache, data mover, library, and drive. Connectivity must be correct. In addition, the bptpcinfo command in NetBackup Snapshot Client generates a 3pc.conf configuration file for running the backup. Often the WWN (world wide name) for some devices is incorrect. You can use CommandCentral Storage to verify that the contents of the 3pc.conf file correlate to the actual fabric configuration. For a description of off-host backup, the bptpcinfo command, and the 3pc.conf file, refer to the NetBackup Snapshot Client Configuration document. For help accessing this document, see "Snapshot Client Assistance" in the NetBackup Snapshot Client Administrators Guide. Symptom: After you run the Device Configuration Wizard, the new device does not appear in the discovered devices list. To use CommandCentral Storage to troubleshoot the inability to discover a drive or robot in a SAN environment

1 2 3

Run device discovery again. If the new device is still not seen, the likely problem is with hardware. Launch CommandCentral Storage. If the new device does not appear in the CommandCentral Storage topology, check the SAN hardware connections to determine whether or not the device is connected. If the new device shows up as disconnected or offline, contact the SAN administrator and check switch configuration. Compare this troubleshooting procedure to a similar problem without the benefit of CommandCentral Storage, such as Robotic status code 214: robot number does not exist. See Robotic status code 214 in the Status Codes Reference Guide.

Rerun the Device Configuration Wizard.

100

Troubleshooting procedures About troubleshooting NetBackup in a SAN environment

Using CommandCentral Storage to troubleshoot an intermittent drive failure in a SAN environment

The following use case demonstrates how CommandCentral Storage can be integrated into a NetBackup troubleshooting procedure to investigate the SAN context of a backup system. Most common NetBackup problems on SANs revolve around connectivity issues. A drive fails and causes a backup to fail, but on examination the drive looks fine. Sometimes a problem with a switch or bridge either before or during the backup job causes the job to fail and takes down the drive. This problem is one of the most difficult to diagnose. By the time the NetBackup administrator looks at the SAN everything may be fine again. To use CommandCentral Storage to troubleshoot this issue, do the following: check for alerts around the time of the job failure and see if a SAN problem occurred that would have caused the job to fail. Another possibility is that another application reserved the device. A SCSI device monitoring utility is required to resolve this issue, which neither CommandCentral Storage nor NetBackup currently supplies. Symptom: The backup job fails intermittently and the drive is down intermittently. No errors appear in the error log other than that the job failed. To use CommandCentral Storage to troubleshoot an intermittent drive failure in a SAN environment

1 2

Select a drive inside the NetBackup Device Monitor. Launch CommandCentral Storage in the drive context to see whether the drive is connected to the SAN. Check CommandCentral Storage alert reports to see whether a SAN problem existed that would have affected the drive during the time the backup job failed.

Chapter

Using logs
This chapter includes the following topics:

About logs About UNIX system logs About unified logging About legacy logging About global logging levels Logs to accompany problem reports for synthetic backups Setting retention limits for logs on clients Logging options with the Windows Event Viewer Troubleshooting error messages in the NetBackup Administration Console for UNIX

About logs
NetBackup uses several different logs and reports to help you troubleshoot any problems that you encounter. Users need to know where log and report information is on their systems. Figure 3-1 shows the location of the log and report information on the client and the server and the processes that make the information available.

102

Using logs About logs

Figure 3-1

Logs in the NetBackup Enterprise system

Master Server

Client
Progress Files Client Debug Logs

Error Catalog

NetBackup Database Manager

Server Programs Media Catalog Storage Server

Client Programs

NetBackup Administration Interface

Progress Files

Server Debug Logs

System Logs System Messages Windows Event Log

Master Server or Media Server

You can review a functional overview that describes the programs and daemons that are mentioned in this figure. You can also use NetBackup reports to help troubleshoot problems. NetBackup reports give information about status and errors. To run reports, use the NetBackup Administration Console. See the NetBackup Administrators Guide, Volume I for detailed descriptions of NetBackup reports. Note: The log-entry format in the NetBackup logs is subject to change without notice.

Using logs About UNIX system logs

103

About UNIX system logs

The NetBackup server daemons and programs occasionally log information through the syslogd man page. syslogd then shows a message or writes the information in an appropriate system log or the console log. On UNIX, NetBackup automatically records robotic and network errors in the system logs by using syslogd. On Windows, NetBackup records robotic and drive errors in the Event Viewer Application log. On both operating systems, log entries are also made when robotically controlled drives change between UP and DOWN states. Note: On HP-UX, the sysdiag tool may provide additional information on hardware errors. To enable system logs, use one of the following:

Use the ltid command that started the device management processes. If the -v option is included on the ltid command, all daemons that were started as a result also have the -v option in effect. Use a command to start a specific daemon (for example, acsd -v).

On UNIX, enable debug logging to the system logs by including the verbose option (-v) on the command that you use to start a daemon. To troubleshoot ltid or robotic software, you must enable system logging. See the syslogd(8) man page for information on setting up system logs. Errors are logged with LOG_ERR, warnings with LOG_WARNING, and debug information with LOG_NOTICE. The facility type is daemon. See the syslogd man page for the locations of system log messages on your system.

About unified logging

Unified logging and legacy logging are the two forms of debug logging used in NetBackup. Unified logging creates log file names and messages in a standardized format. All NetBackup processes use either unified logging or legacy logging. Unlike the files that are written in legacy logging, unified logging files cannot be viewed with a text editor. The unified logging files are in binary format, and some of the information is contained in an associated resource file. See About legacy logging on page 127. Server processes and client processes use unified logging.

104

Using logs About unified logging

See Originator IDs for the entities that use unified logging on page 108. Unlike legacy logging, unified logging does not require that you create logging subdirectories. Log files for originator IDs are written to a subdirectory with the name specified in the log configuration file. All unified logs are written to subdirectories in the following directory:
UNIX Windows /usr/openv/logs install_path\NetBackup\logs

You can access logging controls in the NetBackup Administration Console. In the left pane, expand NetBackup Management > Host Properties > Master Servers or Media Servers. Double-click the server you want to change. In the left pane of the dialog box, click Logging. You can also manage unified logging by using the following commands:
vxlogcfg Modifies the unified logging configuration settings. See Examples of using vxlogmgr to manage unified logs on page 122. vxlogmgr Manages the log files that are generated by the products that support unified logging. See Examples of using vxlogcfg to configure unified logs on page 125. vxlogview Displays the logs that unified logging generates. See Examples of using vxlogview to view unified logs on page 121.

A complete description of the vxlogcfg, vxlogmgr, and vxlogview commands are provided in the NetBackup Commands Reference Guide. These commands are located in the following directory:
UNIX Windows /usr/openv/netbackup/bin install_path\NetBackup\bin

Gathering unified logs for NetBackup

This topic uses an example to describe how to gather unified logs for NetBackup.

Using logs About unified logging

105

To gather unified logs for NetBackup

Create a directory named /upload by using the following command.

# mkdir /upload

Copy unified logs (for NetBackup only) to the /upload directory by using the following command:
# vxlogmgr -p NB -c --dir /upload

Example output:
Following are the files that were found: /usr/openv/logs/bmrsetup/51216-157-2202872032-050125-0000000.log /usr/openv/logs/nbemm/51216-111-2202872032-050125-0000000.log /usr/openv/logs/nbrb/51216-118-2202872032-050125-0000000.log /usr/openv/logs/nbjm/51216-117-2202872032-050125-0000000.log /usr/openv/logs/nbpem/51216-116-2202872032-050125-0000000.log /usr/openv/logs/nbsl/51216-132-2202872032-050125-0000000.log Total 6 file(s) Copying /usr/openv/logs/bmrsetup/51216-157-2202872032-050125-0000000.log ... Copying /usr/openv/logs/nbemm/51216-111-2202872032-050125-0000000.log ... Copying /usr/openv/logs/nbrb/51216-118-2202872032-050125-0000000.log ... Copying /usr/openv/logs/nbjm/51216-117-2202872032-050125-0000000.log ... Copying /usr/openv/logs/nbpem/51216-116-2202872032-050125-0000000.log ... Copying /usr/openv/logs/nbsl/51216-132-2202872032-050125-0000000.log ...

106

Using logs About unified logging

Change to the /upload directory and list its contents.

# cd /upload ls

Example output:
51216-111-2202872032-050125-0000000.log 51216-116-2202872032-050125-0000000.log 51216-117-2202872032-050125-0000000.log 51216-118-2202872032-050125-0000000.log 51216-132-2202872032-050125-0000000.log 51216-157-2202872032-050125-0000000.log

Tar the log files.

# tar -cvf file_name.logs ./*

Types of unified logging messages

The following message types can appear in unified logging files:
Application log messages Application log messages include informational, warning, and error messages. They are always logged and cannot be disabled. These messages are localized. An example of an application message follows: 05/02/10 11:02:01.717 [Warning] V-116-18 failed to connect to nbjm, will retry

Diagnostic log messages

Diagnostic log messages are the unified logging equivalent of the legacy debug log messages. They can be issued at various levels of detail (similar to verbose levels in legacy logging). These messages are localized. Diagnostic messages can be disabled with the vxlogcfg command. An example of a diagnostic message follows: 05/05/09 14:14:30.347 V-116-71 [JobScheduler::doCatIncr] no configured session based incremental catalog schedules

Using logs About unified logging

107

Debug log messages

Debug log messages are intended primarily for Symantec engineering. Like diagnostic messages, they can be issued at various levels of detail. These messages are not localized. Debug messages can be disabled with the vxlogcfg command. An example of a debug message follows: 10/29/09 13:11:28.065 [taolog] TAO (12066|1) Transport_Cache_Manager::bind_i, 0xffbfc194 -> 0x7179d0 Transport[12]

File name format for unified logging

Unified logging uses a standardized naming format for log files. The following is an example of a log file name.
/usr/openv/logs/nbpem/51216-116-2201360136-041029-0000000000.log

Table 3-1 describes each part of the log file name. Table 3-1 Example
51216

Description of the file name format for unified logging Description Details
Product ID Identifies the product. The NetBackup product ID is 51216. The product ID is also known as the entity ID. Identifies the log writing entity, such as a process, service, script, or other software. The number 116 is the originator ID of the nbpem process (the NetBackup policy execution manager). Identifies the host that created the log file. Unless the file was moved, this ID is the host where the log resides.. Shows the date when the log was written in YYMMDD format. Identifies the numbered instance of a log file for a given originator. The rollover number (rotation) indicates the instance of this log file. By default, log files roll over (rotate) based on file size. If the file reaches maximum size and a new log file is created for this originator, the new file is designated 0000000001. See About rolling over unified log files on page 114.

116

Originator ID

2201360136 Host ID

041029

Date

0000000000 Rotation

108

Using logs About unified logging

The log configuration file specifies the name of the directories where the log files for originator IDs are written. These directories and the log files that they hold are written to the following directory, except as noted in the following: See Originator IDs for the entities that use unified logging on page 108..
UNIX Windows /usr/openv/logs install_path\NetBackup\logs

Originator IDs for the entities that use unified logging

Many server processes, services, and libraries use unified logging. Also, UNIX, Linux, and Windows clients use unified logging. An originator identifier (OID) corresponds to a NetBackup process, service, or library. An OID identifies a process, a service, or a library. A process creates entries in its own log file. The process can call a library that also creates entries in the same file but with an OID unique to the library. Hence, a log file can contain entries with different OIDs. Multiple processes can use the same library, so a library OID may appear in several different log files. Table 3-2 lists the NetBackup server and NetBackup client processes, services, and libraries that use unified logging. Table 3-2 Originator ID Entity
18 nbatd

Originator IDs for the server entities that use unified logging Description
The Authentication Service (nbatd) directory is created under the usr/netbackup/sec/at/bin directory (UNIX and Linux) or the install_path\NetBackup\sec\at\bin directory (Windows). The Private Branch Exchange (PBX) service provides single-port access to clients outside the firewall that connect to Symantec product services. Service name: VRTSpbx. It writes logs to /opt/VRTSpbx/log (UNIX and Linux)) or install_path\VxPBX\log (Windows). The Enterprise Media Manager (EMM) is a NetBackup service that manages the device and the media information for NetBackup. It runs only on the EMM server. The NetBackup Policy Execution Manager (nbpem) creates policy and client tasks and determines when jobs are due to run. It runs only on the master server.

103

pbx_exchange

111

nbemm

116

nbpem

Using logs About unified logging

109

Table 3-2

Originator IDs for the server entities that use unified logging (continued) Description
The NetBackup Job Manager (nbjm) accepts the jobs that the Policy Execution Manager submits and acquires the necessary resources. It.runs only on the master server. The NetBackup Resource Broker locates storage units, tape drives, and client reservations for jobs, then starts the jobs. It.works with EMM and runs only on the EMM server. The NetBackup Bare Metal Restore (BMR) master server daemon. The BMR Save Configuration is a data collection utility that runs on the NetBackup client, not the server. The BMR Client Utility originates on the BMR boot server and runs on the restoring client. UNIX clients use it to communicate to the BMR master server during a restore. The BMR Server Utility. The BMR Create Floppy utility is used by the BMR commands that create floppy disks. It runs on the BMR boot server and is Windows only. The BMR Create SRT utility creates a shared resource tree. It runs on the BMR boot server. The BMR Prepare to Restore utility prepares the BMR servers for a client restoration. The BMR Setup Commands utility sets up BMR installation, configuration, and upgrade processes. The BMR Libraries and Common Code catalog provides log messages to the BMR libraries. The BMR Edit Configuration utility modifies the client configuration. The BMR Create Package utility adds Windows drivers, service packs, and hot fixes to the BMR master server for restore operations. The BMR Restore utility restores Windows BMR clients. It runs on the restoring client for Windows systems only.

Originator ID Entity
117 nbjm

118

nbrb

119 121

bmrd bmrsavecfg

122

bmrc

123 124

bmrs bmrcreatefloppy

125

bmrsrt

126

bmrprep

127

bmrsetup

128

bmrcommon

129 130

bmrconfig bmrcreatepkg

131

bmrrst

110

Using logs About unified logging

Table 3-2

Originator IDs for the server entities that use unified logging (continued) Description
The NetBackup Service Layer facilitates the communication between the NetBackup graphical user interface and NetBackup logic. nbsl is required to run Symantec OpsCenter, an application that manages and monitors multiple NetBackup environments. This process runs only on the master server. The NDMP agent daemon manages NDMP backups and restores. It runs on the media server. The libraries control the logging level in the NetBackup libraries. The application and diagnostic messages are for customer use; debug messages are intended for Symantec engineering. The media server user interface is used for the Enterprise Media Manager (EMM). The BMR External Procedure process manages the BMR external procedures that are used during a restore operation. The EMM Media and Device Selection process manages the media selection component and device selection component of the Enterprise Media Manager (EMM). The EMM Device Allocator is used for shared drives. TheSymantec OpsCenter reporting service is part of Symantec OpsCenter. The Symantec OpsCenter Client is part of Symantec OpsCenter. The Symantec OpsCenter Server is part of Symantec OpsCenter The NDMP message log (ndmp) handles NDMP protocol messages, avrd, and robotic processes. The BMR Override Table Admin Utility manages the custom override functions for Bare Metal Restore.

Originator ID Entity
132 nbsl

134

ndmpagent

137

libraries

140

mmui

142

bmrepadm

143

mds

144 146

da NOMTRS

147 148 151

NOMClient NOMServer ndmp

154

bmrovradm

Using logs About unified logging

111

Table 3-2

Originator IDs for the server entities that use unified logging (continued) Description
The NBACE process controls the logging level in the (ACE/TAO) CORBA components for any process that uses a CORBA interface. The default level is 0 (only important messages are logged). This logging is intended for Symantec engineering. If Symantec Technical Support instructs you to increase the logging level, increase the level for originator ID 137 to 4 or higher.

Originator ID Entity
156 ace

Warning: A debug logging level greater than 0 generates large amounts

of data. 158 159 163 ncfrai ncftfi nbsvcmon Remote access interface for NetBackup clients. Transmater for NetBackup clients. The NetBackup Service Monitor monitors the NetBackup services that run on the local computer and tries to restart a service that unexpectedly terminates. The NetBackup Vault Manager manages NetBackup Vault. nbvault must be running on the NetBackup Vault server during all NetBackup Vault operations. The Disk Service Manager (DSM) performs set and get operations on disk storage and disk storage units. The Fibre Transport (FT) server process runs on media servers that are configured for NetBackup Fibre Transport. On the server side of the FT connection, nbftsrvrcontrols data flow, processes SCSI commands, manages data buffers, and manages the target mode driver for the host bus adapters. nbftsrvr is part of SAN client. The Fibre Transport (FT) client process runs on the client and is part of SAN Client. The FT Service Manager (FSM) is a component of the Enterprise Media Manager (EMM) and is part of SAN Client. The Storage Service manages the storage server and runs on the media server. Exchange Firedrill Wizard for NetBackup clients.

166

nbvault

178

dsm

199

nbftsrvr

200

nbftclnt

201

fsm

202

stssvc

210

ncfive

112

Using logs About unified logging

Table 3-2

Originator IDs for the server entities that use unified logging (continued) Description
The Resource Event Manager (REM) is a CORBA loadable service that runs inside nbemm. REM works with the Disk Polling Service to monitor free space and volume status, and to watch for disk-full conditions. Disk polling service for NetBackup clients. The Media Performance Monitor Service (MPMS) runs on every media server within RMMS and gathers CPU load and free memory information for the host. Remote Monitoring and Management Service (RMMS) is the conduit through which EMM discovers and configures disk storage on media servers. The Storage Services controls the lifecycle image duplication operations. The Remote Disk Service Manager interface (RDSM) runs within the Remote Manager and Monitor Service. RDMS runs on media servers. The Event Manager Service provides asynchronous event management services for cooperating participants. The BMR Launcher Utility in the Windows BMR Fast Restore image configures the BMR environment. Recovery Assistant for SharePoint Portal Server for NetBackup clients. Artifact Generator Generated Source. The NetBackup Administration Console for Windows Legacy error codes. The Expiration Manager handles the capacity management and the image expiration for storage lifecycle operations. The Encryption Key Management Service is a master server based symmetric service that provides encryption keys to the media server NetBackup Tape Manager processes. NetBackup Audit Manager. NetBackup Audit Messages. NetBackup Client Framwork.

Originator ID Entity
219 rsrcevtmgr

220 221

dps mpms

222

nbrmms

226

nbstserv

230

rdsm

231

nbevtmgr

248

bmrlauncher

254 261 263 271 272

SPSV2RecoveryAsst aggs wingui nbecmsg expmgr

286

nbkms

293 294 309

nbaudit nbauditmsgs ncf

Using logs About unified logging

113

Table 3-2

Originator IDs for the server entities that use unified logging (continued) Description
NetBackup Client/Server Communications. NetBackup Client Beds Plug-in. NetBackup Client Windows Plug-in. NetBackup Relational Database access library. NetBackup Client Oracle Plug-in. Live Browse Client. Granular Restore. NetBackup TAR Plug-in. NetBackup Client VxMS Plug-in. NetBackup Restore. NetBackup Browser. NetBackup Client Oracle utility. NetBackup Client DB2 Plug-in. NetBackup Agent Request Services. Database Agent Request Server process call NetBackup Client Service. NetBackup ImportManager. Image Manager. Hold Service. NetBackup Indexing Service.

Originator ID Entity
311 317 318 321 348 351 352 355 356 357 359 360 361 362 363 366 369 371 372 373 375 377 380 381 ncfnbservercom ncfbedspi ncfwinpi dbaccess ncforaclepi ncflbc ncfgre ncftarpi ncfvxmspi ncfnbrestore ncfnbbrowse ncforautil ncfdb2pi nbars dars ncfnbcs importmgr nbim nbhsm nbism

ncfnbusearchserverpi NetBackup Client Search Server Plug-in ncfnbdiscover ncfnbquiescence ncfnbdboffline NetBackup Client Component Discovery. NetBackup Client Component Quiescence/Unquiescence. NetBackup Client Component Offline/Online.

114

Using logs About unified logging

Table 3-2

Originator IDs for the server entities that use unified logging (continued) Description
NetBackup Content Indexer. NetBackup NCF VMware Plug-in. NetBackup Remote Network Transport. STS Event Manager. NetBackup Utilities. NetBackup Search Enterprise Vault Ingest. NetBackup Discovery. NetBackup Client MSSQL Plugin. NetBackup Client Exchange Plugin. NetBackup Client SharePoint Plugin. NetBackup Client FileSystem Plugin.

Originator ID Entity
385 386 387 395 396 398 400 401 402 403 412 ncfnbci ncfvmwarepi nbrntd stsem nbutils nbevingest nbdisco ncfmssqlpi ncfexchangepi ncfsharepointpi ncffilesyspi

About changing the location of unified log files

The unified logging files can consume a lot of disk space. If necessary, enter the following to direct them to a different location.
UNIX /usr/openv/netbackup/bin/vxlogcfg -a -p NB -o Default -s LogDirectory=new_log_path Where new_log_path is a full path, such as /bigdisk/logs. Windows install_path\NetBackup\bin\vxlogcfg -a -p NB -o Default -s LogDirectory=new_log_path Where new_log_path is a full path, such as D:\logs.

About rolling over unified log files

To prevent log files from becoming too large, or to control when or how often logs are created, you can set a log rollover option. When a file size or time setting is reached, the current log file is closed. New log messages for the logging process are written or rolled over to a new log file.

Using logs About unified logging

115

You can set log file rollover to occur based on file size, time of day, or elapsed time. Set the conditions by using the vxlogcfg command with the options described in Table 3-3. Table 3-3 Option
MaxLogFileSizeKB

vxlogcfg options that control the rollover of unified log files Description
Specifies the maximum size that is allowed for the log file (in kilobytes) before rollover occurs, if the RolloverMode is set to FileSize. Specifies the time of day at which the log file is rolled over, if the RolloverMode is set to LocalTime.

RolloverAtLocalTime

RolloverPeriodInSeconds Specifies a period of time in seconds after which the log file is rolled over, if the RolloverMode is set to Periodic. MaxLogFileSizeKB or RolloverAtLocalTime Specifies that the log file rollover occurs whenever the file size limit or the local time limit is reached, whichever is first. An example of the command: vxlogcfg -a -p 51216 -g Default MaxLogFileSizeKB=256 RolloverAtLocalTime=22:00

MaxLogFileSizeKB or Specifies that the log file rollover occurs whenever the file RolloverPeriodInSeconds size limit or the periodic time limit is reached, whichever is first.

A complete description of the vxlogcfg command is provided in the NetBackup Commands Reference Guide. By default, log file rollover is based on a file size of 5120 KB. When a log file reaches 5120 KB in size, the file closes and a new log file opens. The following example sets the NetBackup (prodid 51216) rollover mode to Periodic.
# vxlogcfg -a --prodid 51216 --orgid 116 -s RolloverMode=Periodic RolloverPeriodInSeconds=86400

The previous example uses the vxlogcfg command with the RolloverMode option. It sets rollover mode for nbpem (originator ID 116) to Periodic. It also sets the interval until the next nbpem log file rollover to 24 hours (86400 seconds).

116

Using logs About unified logging

In the following example, the file names show the log file rollover with the rotation ID incremented:
/usr/openv/logs/nbpem/51216-116-2201360136-041029-0000000000.log /usr/openv/logs/nbpem/51216-116-2201360136-041029-0000000001.log /usr/openv/logs/nbpem/51216-116-2201360136-041029-0000000002.log

In addition, you can use log file rotation with the following:

Logs for the server processes that use unified logging See Originator IDs for the entities that use unified logging on page 108. Certain legacy logs The unified logging files that the Bare Metal Restore process bmrsavecfg creates

About recycling unified log files

Deleting the oldest log files is referred to as recycling. You can recycle unified logging files in the following ways.
Limit the number of log files Specify the maximum number of log files that NetBackup retains. When the number of log files exceeds the maximum, the oldest log files become eligible for deletion during log cleanup. The NumberOfLogFiles option for the vxlogcfg command defines that number. The following example sets to 8000 the maximum number of log files that are allowed for all unified logging originators in the NetBackup (product ID 51216). When the number of log files exceeds 8000 for a particular originator, the oldest log files become eligible for deletion during log cleanup. # vxlogcfg -a -p 51216 -o ALL -s NumberOfLogFiles=8000 See Examples of using vxlogcfg to configure unified logs on page 125. Specify the number of Use the Keep logs property to specify the maximum number of days the log files are days logs are kept. When the maximum number of days is reached, kept the unified logs and legacy logs are automatically deleted. In the NetBackup Administration Console, in the left pane, expand NetBackup Management > Host Properties > Master Servers. Double-click the server you want to change. A new dialog box appears. In the left pane, click Clean-up > Keep logs.

Using logs About unified logging

117

Explicitly delete the log files

To initiate recycling and delete the log files, run the following command: # vxlogmgr -a -d If you cannot manually delete or move files with vxlogmgr, the Keep logs property removes the old logs for both unified logging and legacy logging. See Examples of using vxlogmgr to manage unified logs on page 122.

If the vxlogcfg LogRecycle option is ON (true), the Keep logs setting is disabled for unified logs. In this case, unified logging files are deleted when their number (for a particular originator) exceeds the number that the NumberOfLogFiles option specifies on the vxlogcfg command.

About using the vxlogview command to view unified logs

Use the vxlogview command to view the logs that unified logging creates. These logs are stored in the following directory.
UNIX Windows /usr/openv/logs install_path\logs

Unlike the files that are written in legacy logging, unified logging files cannot be easily viewed with a text editor. The unified logging files are in binary format, and some of the information is contained in an associated resource file. Only the vxlogview command can assemble and display the log information correctly. You can use vxlogview to view NetBackup log files as well as PBX log files. To view PBX logs using the vxlogview command, do the following:

Ensure that you are an authorized user. For UNIX and Linux, you must have root privileges. For Windows, you must have administrator privileges. Specify the PBX product ID by entering -p 50936 as a parameter on the vxlogview command line.

vxlogview searches all the files, which can be a slow process. Refer to the following

topic for an example of how to display results faster by restricting the search to the files of a specific process. See Examples of using vxlogview to view unified logs on page 121.

118

Using logs About unified logging

About query strings used with the vxlogview command

Use the vxlogview command to display the logs that unified logging generates. The vxlogview command includes the following option: -w (- -where) QueryString.
QueryString represents a text expression similar to a database WHERE clause.

The query string expression is used to retrieve log entries from the unified logging system. The expression is a combination of relational operators, constant integers, constant strings, and names of log fields that evaluate to a single value. Expressions are grouped by logical operators such as AND and OR. The supported relational operators are as follows:
< > <= >= = != less than greater than less than and equal to greater than and equal to equal to not equal to

The supported logical operators are as follows:

&& || logical AND logical OR

Table 3-4 shows data types for specific fields as well as description and an example. When more than one example is listed, both examples produce the same results. Table 3-4 Field name
PRODID

Data types for fields Description

Provide the product ID or the abbreviated name of product. Provide the originator ID or the abbreviated name of the component. Provide the process ID

Type
Integer or string

Example
PRODID = 51216 PRODID = 'NBU' ORGID = 116 ORGID = 'nbpem' PID = 1234567

ORGID

Integer or string

PID

Long Integer

Using logs About unified logging

119

Table 3-4 Field name

TID STDATE

Data types for fields (continued) Description

Provide the thread ID

Type
Long Integer Long Integer or string

Example
TID = 2874950

Provide the start date in seconds or STDATE = 98736352 in the locale-specific short date and STDATE = '4/26/11 11:01:00 time format. For example, a locale AM' may have format 'mm/dd/yy hh:mm:ss AM/PM' Provide the end date in seconds or ENDATE = 99736352 in the locale-specific short date and ENDATE = '04/27/11 10:01:00 time format. For example, a locale AM' may have format 'mm/dd/yy hh:mm:ss AM/PM' Provide the hours in 'hh:mm:ss' PREVTIME = '2:34:00' format. This field should be used only with operators =, <, >, >= and <= Provide one of the following possible severity types: 0 = INFO 1 = WARNING 2 = ERR 3 = CRIT 4 = EMERG SEV = 0 SEV = INFO

ENDATE

Long Integer or string

PREVTIME

String

SEV

Integer

MSGTYPE

Integer

Provide one of the following possible message types: 0 = DEBUG (debug messages) 1 = DIAG (diagnostic messages) 2 = APP (application messages) 3 = CTX (context messages) 4 = AUDIT (audit messages)

MSGTYPE = 1 MSGTYPE = DIAG

120

Using logs About unified logging

Table 3-4 Field name

CTX

Data types for fields (continued) Description Example

Type
Integer or string

Provide the context token as string CTX = 78 identifier or 'ALL' to get all the CTX = 'ALL' context instances to be displayed. This field should be used only with the operators = and !=.

Consider the following when writing a query string.

Case sensitivity Field names, severity types, and message types are not case sensitive. For example, the following are valid entries:

sev = info msgtype = diag

String constants

String constants should be given in single quotes. For example, PRODID = 'NBU' Start and end dates can be provided in the following formats: A string constant that corresponds to the regional display short date format A UNIX long value of number of seconds that elapsed since midnight January 1, 1970.

Dates

Table 3-5 provides examples of query strings. Table 3-5 Example

(PRODID == 51216) && ((PID == 178964) || ((STDATE == '2/5/09 00:00:00 AM') && (ENDATE == '2/5/03 12:00:00 PM'))

Examples of query strings Description

((prodid = 'NBU') && ((stdate >= 11/18/09 0:0:0 AM) && (endate <= 12/13/09 13:0:0 AM))) || ((prodid = 'BENT') && ((stdate >= 12/12/09 0:0:0 AM) && (endate <= 12/25/09 25:0:0 PM)))

Using logs About unified logging

121

Table 3-5 Example

Examples of query strings (continued) Description

Retrieves the log messages that were logged on or before 2009-05-04 for all the installed Symantec products.

(STDATE <= 04/05/09 0:0:0 AM)

Examples of using vxlogview to view unified logs

The following examples demonstrate how to use the vxlogview command to view unified logs. Table 3-6 Item
Display all the attributes of the log messages Display specific attributes of the log messages

Example uses of the vxlogview command Example

vxlogview -p 51216 -d all

Display the log messages for NetBackup (51216) that show only the date, time, message type, and message text: vxlogview --prodid 51216 --display D,T,m,x

Display the latest log messages

Display the log messages for originator 116 (nbpem) that were issued during the last 20 minutes. Note that you can specify -o nbpem instead of -o 116: # vxlogview -o 116 -t 00:20:00

Display the log messages from a specific time period

Display the log messages for nbpem that were issued during the specified time period: # vxlogview -o nbpem -b "05/03/05 06:51:48 AM" -e "05/03/05 06:52:48 AM"

122

Using logs About unified logging

Table 3-6 Item

Example uses of the vxlogview command (continued) Example

You can use the -i option to specify an originator for a process: # vxlogview -i nbpem The vxlogview -i option searches only the log files that the specified process (nbpem) creates. By limiting the log files that it has to search, vxlogview returns a result faster. By comparison, the vxlogview -o option searches all unified log files for the messages that the specified process are logged by.

Display results faster

Note: If you use the -i option with a process that is not a service,
vxlogview returns the message "No log files found." A process that is not a service has no originator ID in the file name. In this case, use the -o option instead of the -i option. The -i option displays entries for all OIDs that are part of that process including libraries (137, 156, 309, etc.). Search for a job ID You can search the logs for a particular job ID: # vxlogview -i nbpem | grep "jobid=job_ID" The jobid= search key should contain no spaces and must be lowercase. When searching for a job ID, you can use any vxlogview command option. This example uses the -i option with the name of the process (nbpem). The command returns only the log entries that contain the job ID. It misses related entries for the job that do not explicitly contain the jobid=job_ID.

Examples of using vxlogmgr to manage unified logs

The following examples show how to use the vxlogmgr command to manage unified logging files. Log file management includes actions such as deleting or moving the log files.

Using logs About unified logging

123

Table 3-7 Item

List the log files

Example uses of the vxlogmgr command

Example
List all unified log files for the nbrb service: # vxlogmgr -s -o nbrb /usr/openv/logs/nbrb/51216-118-1342895976-050503-00.log /usr/openv/logs/nbrb/51216-118-1342895976-050504-00.log /usr/openv/logs/nbrb/51216-118-1342895976-050505-00.log Total 3 file(s)

Delete the oldest log files

If the vxlogcfg NumberOfLogFiles option is set to 1, the following example deletes the two oldest log files for the nbrb service: # vxlogcfg -a -p 51216 -o nbrb -s NumberOfLogFiles=1 # vxlogmgr -d -o nbrb -a Following are the files that were found: /usr/openv/logs/nbrb/51216-118-1342895976-050504-00.log /usr/openv/logs/nbrb/51216-118-1342895976-050503-00.log Total 2 file(s) Are you sure you want to delete the file(s)? (Y/N): Y Deleting /usr/openv/logs/nbrb/51216-118-1342895976-050504-00.log ... Deleting /usr/openv/logs/nbrb/51216-118-1342895976-050503-00.log ...

Delete the newest log files

Delete all the unified log files that NetBackup created in the last 15 days: # vxlogmgr -d --prodid 51216 -n 15 Make sure that you roll over (rotate) the log files before you recycle them.

Delete the log files for Delete all unified log files for originator nbrb: a specific originator # vxlogmgr -d -o nbrb Make sure that you roll over (rotate) the log files before you recycle them. Delete all the log files Delete all unified log files for NetBackup:. # vxlogmgr -d -p NB Make sure that you roll over (rotate) the log files before you recycle them.

124

Using logs About unified logging

Table 3-7 Item Example

Example uses of the vxlogmgr command (continued)

Control the number of You can use the vxlogmgr command with the vxlogcfg commands NumberOfLogFiles log files option to manually delete log files. For example, the NumberOfLogFiles option is set to 2, you have 10 unified logging files, and cleanup has not occurred. Enter the following to keep the two most recent log files and delete the rest for all originators: # vxlogmgr -a -d The following command keeps the two most recent log files of all PBX originators: # vxlogmgr -a -d -p ics The following deletes the older log files for the nbrb service only: # vxlogmgr -a -d -o nbrb

Control disk space usage

Periodically run the vxlogmgr -a -d command (such as through a cron job) to delete logs and monitor the disk space that unified logging uses. The disk space that a given originator uses can be calculated as follows: NumberOfFiles for originator * MaxLogFileSizeKB for originator The total disk space that unified logs consume is the sum of the disk space that each originator consumes. If none of the originators overrides the NumberOfFiles and MaxLogFileSizeKB settings, then the total disk space that unified logging consumes is as follows: Number of originators * default MaxLogFileSizeKB * default NumberOfFiles Use the vxlogcfg command to list the current unified logging settings. For example, assume the following:

vxlogmgr -a -d -p NB is configured as a cron job with a frequency of one hour. No originators override default settings for MaxLogFileSizeKB or NumberOfFiles.

The number of active NetBackup originators on the host is 10. (Typical of a NetBackup master server that is not running BMR or NDMP.) The default MaxLogFileSizeKB is equal to 5120.

The default NumberOfFiles is equal to 3.

To calculate the total disk space that unified logging consumes, insert the values from the example into the previous formula. The results are as follows: 10 * 5120 * 3 KB = 15,360 KB of additional disk space used each hour.

Using logs About unified logging

125

For a complete description of the vxlogmgr command, see the vxlogmgr man page or the NetBackup Commands Reference Guide.

Examples of using vxlogcfg to configure unified logs

Use the vxlogcfg command to change logging levels and rollover settings. The vxlogcfg command has the following characteristics:

The vxlogcfg command is the only way to turn off diagnostic and debug messages in unified logging. In legacy logging, the writing of messages cannot be turned off, only minimized. The vxlogcfg options for robust file logging (MaxLogFileSizeKB and NumberOfLogFiles) also affect certain legacy logs. See About limiting the size and the retention of legacy logs on page 138. Absolute paths must be specified. Do not use relative paths.

The following examples show how to use the vxlogcfg command to configure unified logging settings.

126

Using logs About unified logging

Table 3-8 Item

Example uses of the vxlogcfg command Example

By default, the maximum log file size in unified logging is 5120 KB. When a log file reaches 5120 KB, the file closes and a new log file opens. You can change the maximum file size with the MaxLogFileSizeKB option. The following command changes the default maximum log size to 2048 KB for the NetBackup product: # vxlogcfg -a -p 51216 -o Default -s MaxLogFileSizeKB=2048 For MaxLogFileSizeKB to be effective, the RolloverMode option must be set to FileSize : # vxlogcfg -a --prodid 51216 --orgid Default -s RolloverMode=FileSize MaxLogFileSizeKB can be set per originator. An originator that is not configured uses the default value. The following example overrides the default value for service nbrb (originator 118). # vxlogcfg -a -p 51216 -o nbrb -s MaxLogFileSizeKB=1024

Set the maximum log file size

Set log recycling

The following example sets automatic log file deletion for nbemm logs (originator ID 111): # vxlogcfg -a --prodid 51216 --orgid 111 -s RolloverMode=FileSize MaxLogFileSizeKB=5120 NumberOfLogFiles=999999 LogRecycle=TRUE This example sets nbemm rollover mode to file size, and turns on log recycling. When the number of log files exceeds 999999, the oldest log file is deleted. EXAMPLE 5 shows how to control the number of log files.

Set debug level and diagnostic level

The following example sets the debug level and diagnostic level for all the originators of product ID NetBackup (51216): # vxlogcfg -a --prodid 51216 --orgid ALL -s DebugLevel=0 DiagnosticLevel=1

Using logs About legacy logging

127

Table 3-8 Item

Example uses of the vxlogcfg command (continued) Example

List the unified logging The following vxlogcfg example shows how to list the active settings unified logging settings for a given originator (the nbrb service). Note that MaxLogFileSizeKB, NumberOfLogFiles, and RolloverMode are included in the output. # vxlogcfg -l -o nbrb -p NB Configuration settings for originator 118, of product 51,216... LogDirectory = /usr/openv/logs/ DebugLevel = 5 DiagnosticLevel = 5 LogToStdout = False LogToStderr = False LogToOslog = False RolloverMode = FileSize MaxLogFileSizeKB = 5120 RolloverPeriodInSeconds = 43200 RolloverAtLocalTime = 0:00 NumberOfLogFiles = 4 ...

For a complete description of the vxlogcfg command, see the vxlogcfg man page or the NetBackup Commands Reference Guide.

About legacy logging

Legacy logging and unified logging are the two forms of debug logging used in NetBackup. In legacy debug logging, each process creates logs of debug activity in its own logging directory. All NetBackup processes use either unified logging or legacy logging. See About unified logging on page 103. To enable legacy debug logging on NetBackup servers, you must first create the appropriate directories for each process.
UNIX /usr/openv/netbackup/logs /usr/openv/volmgr/debug

128

Using logs About legacy logging

Windows

install_path\NetBackup\logs install_path\Volmgr\debug

After the directories are created, NetBackup creates log files in the directory that is associated with each process. A debug log file is created when the process begins. To enable debug logging for the NetBackup Status Collection Daemon (vmscd), create the following directory before you start nbemm. As an alternative, you can stop and restart nbemm after creating the following directory:
UNIX Windows /usr/openv/volmgr/debug/reqlib install_path\Volmgr\debug\reqlib\

Tables are available that list the log directories that you must create. See Directory names for legacy debug logs for servers on page 134. See Directory names for legacy debug logs for media and device management on page 136. Note: On a Windows server, you can create the debug log directories at once, under install_path\NetBackup\Logs, by running the following batch file: install_path\NetBackup\Logs\mklogdir.bat. Media servers have only the bpbrm, bpcd, bpdm, and bptm debug logs.

UNIX client processes that use legacy logging

Most UNIX client processes use legacy logging. To enable legacy debug logging on UNIX clients, create the appropriate subdirectories in the following directory.
/usr/openv/netbackup/logs

Note: Create the directories with access modes of 777 so that user processes can write to the log files. Table 3-9 describes the directories for the legacy debug logs that apply to UNIX clients..

Using logs About legacy logging

129

Table 3-9 Directory

bp bparchive bpbackup bpbkar bpcd bphdb

UNIX client processes that use legacy logging Associated process

Menu driven client-user interface program. Archive program. Also useful for debugging bp. Backup program. Also useful for debugging bp. Program that is used to generate backup images. NetBackup client daemon or manager. Program that starts a script to back up a database on a NetBackup database agent client. See the system administrator's guide for the appropriate NetBackup database agent for more information. The NetBackup-Java application server authentication service that inetd starts during startup of the NetBackup Java interface applications. This program authenticates the user that started the application. The NetBackup program that bpjava-msvc starts upon successful logon through the logon dialog box that is presented when a NetBackup-Java interface is started. This program services all requests from the Java administration and user interfaces on the host where bpjava-msvc is running. Program that lists backed up and archived files. Also useful for debugging bp. Program that determines local mount points and wildcard expansion for multiple data streams. Command-line program on clients to export Oracle data in XML format. Communicates with bprd on server. 64-bit command-line program on clients to export Oracle data in XML format. Communicates with bprd on server. Command-line program on clients to import Oracle data in XML format. Communicates with bprd on server. 64-bit command-line program on clients to import Oracle data in XML format. Communicates with bprd on server. Restore program. Also useful for debugging bp.

bpjava-msvc

bpjava-usvc

bplist

bpmount

bporaexp

bporaexp64

bporaimp

bporaimp64

bprestore

130

Using logs About legacy logging

Table 3-9 Directory

db_log

UNIX client processes that use legacy logging (continued) Associated process
For more information on these logs, see the NetBackup guide for the database-extension product that you use. These logs have information about the mtfrd process, which is used for phase 2 imports and restores of Backup Exec media. tar process during restores. The user_ops directory is created during the install of NetBackup on all servers and clients. The NetBackup Java interface programs use it for the following: temporary files and for job and progress log files that the Backup, Archive, and Restore program (jbpSA) generates. This directory must exist for successful operation of any of the Java programs and must have public read, write, and run permissions. This directory contains a directory for every user that uses the Java programs. In addition, on NetBackup-Java capable platforms, the NetBackup Java interface log files are written in a subdirectory that is called nbjlogs. All files in the user_ops directory hierarchy are removed according to the setting of the KEEP_LOGS_DAYS configuration option.

mtfrd

tar user_ops

PC client processes that use legacy logging

Most PC client processes use legacy logging. To enable detailed legacy debug logging on Windows clients or NetWare target clients, create the directories in the following location. The directory names that you create correspond to the processes you want to create logs for.
Windows clients NetWare clients C:\Program Files\VERITAS\NetBackup\Logs\ SYS:VERITAS\NBUCLT\NetBack\logs\

Note: These are the default locations in which to place these directories. You can specify another location during client installation. Table 3-10 lists the legacy debug log directories that apply to these clients.

Using logs About legacy logging

131

Table 3-10 Directory

bp

PC client processes that use legacy logging NetBackup client

NetWare target

Associated process
Client-user interface program for NetWare. Client service logs. These logs have information on the bpinetd32 process. Archive program that is run from the command line. The backup program that is run from the command line. Backup and archive manager. These logs have information on the bpbkar32 process. NetBackup client daemon or manager. These logs have information on communications between the server and client. On NetWare clients, these logs also contain the log information for the backup and restore processes.

bpinetd

Windows2003

bparchive

Windows 2003

bpbackup

Windows 2003

bpbkar

Windows 2003

bpcd

All Windows and NetWare clients

bpjava-msvc

The NetBackup-Java bpjava-msvc application server authentication service that the Client Services service starts during startup of the NetBackup Java interface applications. This program authenticates the user that started the application. (On all Windows platforms.)

132

Using logs About legacy logging

Table 3-10 Directory

bpjava-usvc

PC client processes that use legacy logging (continued) NetBackup client Associated process

NetBackup program that bpjava-usvc bpjava-msvc starts upon successful logon through the logon dialog box that is presented when a NetBackup-Java interface is started. This program services all requests from the Java administration and user interfaces on the NetBackup host where bpjava-msvc is running. (On all Windows platforms.) Windows 2003 List program that is run from the command line. The program that is used to collect drive names on the client for multistreaming clients. The restore program that is run from the command line. NetBackup service utility. This program allows the system with the user interface to communicate with the NetBackup for NetWare client. tar process. These logs have information about the tar32 process.

bplist

bpmount

Windows 2003

bprestore

Windows 2003

bpsrv

NetWare nontarget

tar

Windows 2003

Using logs About legacy logging

133

Table 3-10 Directory

user_ops

PC client processes that use legacy logging (continued) NetBackup client

Windows 2003

Associated process
The user_ops directory is created during the install of NetBackup on all servers and clients. The NetBackup Java interface programs use it for the following: temporary files and for job and progress log files that the Backup, Archive, and Restore program (jbpSA) generates. This directory must exist for successful operation of any of the Java programs and must have public read, write, and run permissions. user_ops contains a directory for every user that uses the Java programs. In addition, on NetBackup-Java capable platforms, the NetBackup Java interface log files are written in a subdirectory that is called nbjlogs. All files in the user_ops directory hierarchy are removed according to the setting of the KEEP_LOGS_DAYS configuration option.

File name formats for legacy logging

In the standard legacy log system, a single NetBackup process creates one debug log file per day. In the legacy logging system with Enable robust logging enabled, a NetBackup process creates a certain number of log files. Each file grows to a certain size before it closes and a new one is created. Legacy logging uses two formats for log file names. The format that is used depends on whether the log uses the standard system or file rotation (robust logging). Table 3-11 Type
Standard legacy logging

File name formats for different types of legacy logging File name format

On UNIX: log.mmddyy For example: log.040805 On Windows: mmddyy.log For example: 040105.log

134

Using logs About legacy logging

Table 3-11 Type

File name formats for different types of legacy logging (continued) File name format
mmddyy_nnnnn.log For example: 040105_00001.log Where nnnnn is a counter or a rotation number for the log file. When the counter exceeds the setting for number of log files, the oldest log file is deleted. The NumberOfLogFiles option on the vxlogcfg command sets the number of log files.

Legacy logging with robust logging enabled

For compatibility with existing scripts, the debug log file naming format does not change. If robust file logging is enabled after standard legacy logs have been created, only the log files for the processes that robust logging governs use the file rotation naming format. Any mixture of new and old log file names in a legacy debug log directory is managed according to the Keep logs setting and the robust logging settings.

Directory names for legacy debug logs for servers

Table 3-12 describes the directories you need to create to support legacy debug logs for servers. Each directory corresponds to a process. Unless it is noted, each directory should be created under the following directory.
UNIX Windows /usr/openv/netbackup/logs install_path\NetBackup\logs

Table 3-12 Directory

admin bpbrm bpcd bpdbjobs bpdm bpdbm

Directory names for legacy debug logs

Associated process
Administrative commands. NetBackup backup and restore manager. NetBackup client daemon or manager. The NetBackup Client service starts this process NetBackup jobs database manager program. NetBackup disk manager. NetBackup database manager. This process runs only on master servers. On Windows systems, it is the NetBackup database manager service.

Using logs About legacy logging

135

Table 3-12 Directory

bpjava-msvc

Directory names for legacy debug logs (continued)

Associated process
The NetBackup-Java application server authentication service that is started when the NetBackup Java interface applications start. On UNIX servers, inetd starts it. On Windows servers, the Client Services service starts it. This program authenticates the user that started the application.

bpjava-susvc

The NetBackup program that bpjava-msvc starts upon successful logon through the logon dialog box that is presented when a NetBackup-Java interface starts. This program services all requests from the Java user interfaces on the NetBackup master or media server host where the bpjava-msvc program is running (all Windows platforms). NetBackup request daemon or manager. On Windows systems, this process is called the NetBackup Request Manager service. The NetBackup process for synthetic backup. nbjm starts bpsynth. bpsynth runs on the master server. NetBackup tape management process. Authentication daemon (UNIX and Linux) or service (Windows). nbatd authenticates access to interfaces of NetBackup services or daemons. Authorization daemon (UNIX and Linux) or service (Windows). nbazd authorizes access to interfaces of NetBackup services or daemons. System log. You must enable system logging to troubleshoot ltid or robotic software. See the syslogd man page.

bprd

bpsynth

bptm nbatd

nbazd

syslogs

user_ops

The user_ops directory is created during the install of NetBackup on all servers and clients. NetBackup Java interface programs use it for the following: temporary files and for job and progress log files that the Backup, Archive, and Restore program (jbpSA) generates. This directory must exist for successful operation of any of the Java programs and must have public read, write, and execute permissions. user_ops contains a directory for every user that uses the Java programs. In addition, on NetBackup-Java capable platforms, the NetBackup Java interface log files are written in the nbjlogs subdirectory. All files in the user_ops directory hierarchy are removed according to the setting of the KEEP_LOGS_DAYS configuration option.

136

Using logs About legacy logging

Table 3-12 Directory

vnetd

Directory names for legacy debug logs (continued)

Associated process
The Symantec network daemon, used to create firewall-friendly socket connections. Started by the inetd(1M) process.

Note: Logging occurs in either the /usr/openv/logs directory or the

/usr/openv/netbackup/logs if the vnetd directory exists there. If the vnetd directory exists in both locations, logging occurs only in /usr/openv/netbackup/logs/vnetd.

More information is available on the programs and daemons that write the logs. See About backup and restore functional overview on page 253. On UNIX systems, also refer to the README file in the /usr/openv/netbackup/logs directory.

Directory names for legacy debug logs for media and device management
The debug log directories enable logging for the media management processes and device management processes. Table 3-13 describes the directories you need to create to support legacy debug logs for media and device management. Each directory corresponds to a process. Table 3-13 Directory
acsssi

Media and device management legacy debug logs Associated process

UNIX only. Debug information on transactions between NetBackup and the Storage Tek ACSLS server. Debug information for vmd (NetBackup Volume Manager service, Windows) and its associated processes (oprd and rdevmi). Stop and restart vmd after creating the directory. Debug information on ltid, the Media Manager device daemon (UNIX), or on the NetBackup Device Manager service (Windows), and on avrd. Stop and restart ltid after creating the directory. Debug information on the processes that request media management services from vmd or EMM. Stop and restart vmd after creating the directory. Debug information on all robotic daemons, which includes tldcd, tl8cd, and tl4d daemons. Stop and restart robotic daemons.

daemon

ltid

reqlib

robots

Using logs About legacy logging

137

Table 3-13 Directory

tpcommand

Media and device management legacy debug logs (continued) Associated process
Debug information for device configuration, including the tpconfig and the tpautoconf commands and the NetBackup Administration Console. Debug information for the NetBackup Status Collection daemon. Stop and restart vmscd after creating the directory.

vmscd

Unless it is noted, each directory should be created under the following directory.
UNIX Windows /usr/openv/volmgr/debug install_path\Volmgr\debug

NetBackup creates one log per day in each of the debug directories. You can disable debug logging by deleting or renaming the following directory:
UNIX: vmd command /usr/openv/volmgr/debug/daemon

Windows: NetBackup Volume install_path\Volmgr\debug\daemon Manager service

See File name formats for legacy logging on page 133. See About limiting the size and the retention of legacy logs on page 138. See Directory names for legacy debug logs for media and device management on page 136.

How to control the amount of information written to legacy logging files

You can set legacy logging levels to increase the amount of information that NetBackup processes write in the logs. The following settings affect legacy logging, except media and device management.

Increase the Global logging level. See Changing the logging level on page 143. Note: This setting also affects unified logging.

138

Using logs About legacy logging

On UNIX, add a VERBOSE entry in the /usr/openv/netbackup/bp.conf file. If you enter VERBOSE without a value, the verbose value defaults to 1. For more log detail, enter VERBOSE = 2 or a higher value. This setting affects legacy logging only. Warning: High verbose values can cause debug logs to become very large. Set the logging level for individual processes. In Host Properties, change logging levels for individual processes in the Logging dialog box. Or, specify the verbose flag (if available) when you start the program or daemon. See the NetBackup Administrators Guide, Volume I.

Media and device management legacy logging has two levels: not verbose (the default) and verbose. To set the verbose (higher) level, add the word VERBOSE to the vm.conf file. Create the file if necessary. Restart ltid and vmd after you add the VERBOSE entry. This entry affects logging levels in the Event Viewer Application and System log. The vm.conf file is located in the following directory:
UNIX Windows /usr/openv/volmgr/ install_path\Volmgr\

About limiting the size and the retention of legacy logs

Certain NetBackup processes write legacy debug logs. Because legacy debug logs can grow very large, enable them only if unexplained problems exist. Delete the logs and the associated directories when they are no longer needed. To limit the time NetBackup retains legacy debug logs, specify the number of days in the Keep logs field. The default is 28 days. You can specify the number under Host Properties in the Clean-up dialog box. See the NetBackup Administrators Guide, Volume I. To limit the amount of disk space that the logs consume, use robust logging. Robust logging involves file rotation, like that which is used in unified logging. Robust logging does not apply to media and device management logging. See About rolling over unified log files on page 114. Specify the maximum size for a log file and the maximum number of log files to keep in a logging directory. When a log file grows to its maximum size, it closes and a new file opens. If the number of log files exceeds the number that is allowed for the directory, the oldest file is deleted.

Using logs About legacy logging

139

Logs created by the following NetBackup processes can use log rotation (robust logging):

bpbkar (client process only) bpbrm bpcd bpdbm bpdm bprd bptm

For the legacy logs created by other NetBackup processes (except media and device management logs), use the Keep logs property. The Keep logs property may override the robust file logging settings. If Keep logs is set to 10 days and robust file logging settings allow more than 10 days, the logs are deleted on day 11. For media and device management legacy logs, use the DAYS_TO_KEEP_LOGS setting in the vm.conf file to control log file rotation. The default is infinite retention. The vm.comf file is located in the following directory:
UNIX Windows /usr/openv/volmgr/ install_path\Volmgr\

To retain logs for three days, enter the following in the vm.conf file:
DAYS_TO_KEEP_LOGS = 3

For instructions on how to use this entry, see the NetBackup Administrators Guide, Volume II. See Directory names for legacy debug logs for media and device management on page 136.

Configuring legacy log rotation

You can specify the maximum file size for a legacy log and the maximum number of log files to retain.

140

Using logs About legacy logging

To configure the legacy log rotation

1 2 3

In the NetBackup Administration Console, in the left pane, expand NetBackup Management > Host Properties > Master Servers. In the right pane, double-click the server you wan to modify. In the dialog box that appears, in the left pane, select Logging and check Enable robust logging. Robust logging applies only to legacy logs. Robust logging is also known as log rotation. By default, the maximum file size is 5120 KB and the maximum number of files that are kept per log directory is 3. If Enable robust logging is disabled, the standard behavior remains in effect. A single log file is created per log directory per day, and log deletion is based on the Keep logs property.

To change the maximum file size or the maximum number of log files per directory, use the MaxLogFileSizeKB and the NumberOfLogFiles options. These options are part of the vxlogcfg command, which is located in the following directory:
UNIX Windows /usr/openv/netbackup/bin install_path\NetBackup\bin

Use the following example to set the maximum file size to 2048 and the maximum number of log files per log directory to 10:
vxlogcfg -a -p 51216 --orgid Default -s MaxLogFileSizeKB=2048,NumberOfLogFiles=10

The example sets the default values for all unified logging processes and for all legacy processes for NetBackup (product ID 51216). : See the vxlogcfg man page or the NetBackup Commands Reference Guide.

Creating legacy log directories to accompany problem reports for synthetic backup
If the legacy log directories have not been created, you must create them. If the directories do not exist, the logs cannot be written to disk.

Using logs About global logging levels

141

Table 3-14 Step

Step 1

Creating legacy log directories

Action

Description

Create directories Create the following directories: on the master install_path/netbackup/logs/bpsynth server. install_path/netbackup/logs/bpdbm install_path/netbackup/logs/vnetd

Step 2

Create directories Create the following directories: on the media install_path/netbackup/logs/bpcd server. install_path/netbackup/logs/bptm install_path/netbackup/logs/bpdm

Step 3

Change the Global In Host Properties, select a master server and set the Global logging level to 5. logging level. See Changing the logging level on page 143. See About global logging levels on page 141. See Using the Host Properties window to access configuration settings on page 70.

Step 4

Rerun the job.

Rerun the job and gather the logs from the directories that you created. The bptm logs are required only if the images are read from or written to a tape device. The bpdm logs are needed only if the images are read from or written to disk. If the images are read from multiple media servers, the debug logs for bptm or bpdm must be collected from each media server.

See Logs to accompany problem reports for synthetic backups on page 144.

About global logging levels

The logging level determines how much information is included in the log message. The log range is 0-5. The higher the level number, the greater the amount of detail is in the log message. The following table describes the detail that each level includes.

142

Using logs About global logging levels

Table 3-15 Logging level

0

Global logging levels Description

Includes very important, low-volume diagnostic messages and debug messages Adds verbose diagnostic messages and debug messages Adds the progress messages Adds the informational dumps Adds the function entry and exits Includes everything. The finest detail.

1 2 3 4 5

Unified logging is enabled by default to log debug messages at level 0 and application messages at level 5. The following actions affect logging levels:

In the Global logging level list, a zero (0) level specifies the minimum level of logging for both legacy and unified logging. However, for diagnostic and debug messages in unified logging, the logging level can be turned off completely. No diagnostic messages or debug messages are logged. This level cannot be set with the Global logging level list in the NetBackup Administration Console. You can set it with the vxlogcfg command. See Changing the logging level on page 143. See Examples of using vxlogcfg to configure unified logs on page 125.

A change to the Global logging level list affects the logging level of all NetBackup and Enterprise Media Manager (EMM) processes on the server or client. (The exceptions are PBX and media and device management logging.) This setting overrides any previous settings. If you make a change to the VERBOSE level in the bp.conf file or the vm.conf file, it only affects the legacy logging level. See How to control the amount of information written to legacy logging files on page 137. If you make a change with the vxlogcfg command, it only affects the unified logging level.

A change to the Global logging level list does not affect the level of the following logging processes:

PBX logging See Accessing the PBX logs on page 79.

Using logs About global logging levels

143

Media and device management logging (vmd, ltid, avrd, robotic daemons, media manager commands) See Directory names for legacy debug logs for media and device management on page 136.TAG Any unified logging process whose debug level has been changed from the default setting

Changing the logging level

The logging level determines how much information is included in the log message. The log range is 0-5. The higher the level number, the greater the amount of detail is in the log message. To change the logging level

1 2 3 4 5

In the NetBackup Administration Console, in the left pane, expand NetBackup Management > Host Properties. Select Master Servers, Media Servers, or Clients. In the right pane, click the server or client to view the version and platform. Then, double-click to view the properties. In the properties dialog box, in the left pane, click Logging. In the Global logging level list, select a value from 0 to 5. Changes affect the logging level of both unified logging and legacy logging. See About global logging levels on page 141.

Click OK.

Changing the logging level on Windows and NetWare clients

You can increase the amount of information that client processes write in the logs. To change the logging level on Windows clients

1 2 3

In the NetBackup Administraion Console, on the File menu, click Backup, Archive, and Restore. In the Backup, Archive, and Restore interface, on the File menu, click NetBackup Client Properties. In the NetBackup Client Properties dialog box, select the Troubleshooting tab.

144

Using logs Logs to accompany problem reports for synthetic backups

In the Verbose property field, enter a debug level from 0 to 5. Use the default level of 0 unless advised otherwise by Technical Support. Higher levels can cause the logs to accumulate large amounts of information.

Click OK.

You can also change logging levels in the following ways:

On NetWare clients, change the value of the level and the tcp parameters in the debug section of the bp.ini file. For the unified logging files that the Bare Metal Restore process bmrsavecfg creates, you also can control the logging level with the vxlogcfg command. See Examples of using vxlogcfg to configure unified logs on page 125.

An increase in the log level can cause the logs to grow very large; increase the logging level only if unexplained problems exist.

Setting debug logging to a higher level

To solve many error conditions, set debug logging to a higher level. Then retry the operation and examine the debug logs. To set debug logging to a higher level

1 2

Enable legacy debug logging by creating the necessary directories and folders. Increase the level of verbosity for media and device management processes by adding the VERBOSE option in the vm.conf file. This file is located in /usr/openv/volmgr/ (UNIX and Linux) or install_path\Volmgr\ (Windows). Restart the daemons and services or run the command verbose option, if available.

Logs to accompany problem reports for synthetic backups

To debug problems with synthetic backups, you must include a complete set of logs in the problem report and additional items. Send all the information to Symantec Technical Support. Include the following log types:

Log files that unified logging creates See Gathering unified logs for NetBackup on page 104. Log files that legacy logging creates

Using logs Setting retention limits for logs on clients

145

See Creating legacy log directories to accompany problem reports for synthetic backup on page 140. Include the following additional items:
Try file The try file is located in the following directory: install_path/netbackup/db/jobs/trylogs/jobid.t If the job ID of the synthetic backup job was 110, the try file is named 110.t. Policy attributes Use the following command to capture the policy attributes: install_path/netbackup/bin/admincmd/bppllist policy_name -L where policy_name is the name of the policy for which the synthetic backup job was run. List of storage units Capture the list of storage units from the following command: install_path/netbackup/bin/admincmd/bpstulist -L

See Creating legacy log directories to accompany problem reports for synthetic backup on page 140.

Setting retention limits for logs on clients

You can specify the numbers of days that NetBackup retains client logs on UNIX, Windows, and NetWare. To set retention limits for logs on UNIX clients

1 2 3 4 5

In the NetBackup Administration Console, in the left pane, expand Host Properties > Clients. In the right pane, double-click the client you want to modify. In the properties dialog box, click UNIX Client. In the Client Settings dialog box, find the Keep status of user-directed backups, archives, and restores for field. Enter the number of days you want to retain the log files, and click OK.

146

Using logs Logging options with the Windows Event Viewer

To set the retention limits for logs on Windows clients

1 2 3 4 5

In the NetBackup Adminsistration Console, on the File menu, click Backup, Archive, and Restore. In the Backup, Archive, and Restore interface, on the File menu, click NetBackup Client Properties. In the NetBackup Client Properties dialog box, select the General tab. In the Keep status of user-directed backups, archives, and restores for field, enter the number of days you want to retain the log files. Click OK.

To set retention limits for the logs on NetWare clients

Open the following file:

\veritas\nbuclt\netback\bp.ini

2 3

Under Keep_Logs_Days, specify the number of days to keep the logs. Save the file with your changes.

Logging options with the Windows Event Viewer

NetBackup Windows master servers can be configured so messages from NetBackup reports are written to the Windows Event Viewer Application log. You can see these messages in the Application log and also use third-party tools to monitor the Application log for these messages. To route unified logging application and diagnostic messages for an originator to the Application log, set the LogToOslog value to true for that originator. The following example routes the application and diagnostic messages for nbrb to the Windows event log:
vxlogcfg -a -o nbrb -p NB -s "LogToOslog=true"

Note: For this setting to take effect, restart NetBackup services. To enable the logging tool, do the following:

Create the following file on the NetBackup master serve.

install_path\NetBackup\db\config\eventlog

Optionally, add an entry to the eventlog file. The following is an example:

Using logs Logging options with the Windows Event Viewer

147

56 255

The parameters in the eventlog represent severity and type. The parameters have the following characteristics:
Severity

Listed as the first parameter.

Controls the messages that NetBackup writes to the Application log. If the file is empty, the default severity is Error (16).

If the file has only one parameter, it is used for the severity level. Listed as the second parameter.

Type

Controls the type of messages that NetBackup writes to the Application log If the file is empty, the default type is Backup Status (64).

Both parameters are specified as decimal numbers and equate to a bitmap that expresses the following values:
Severity 1 = Unknown 2 = Debug 4 = Info 8 = Warning 16 = Error 32 = Critical Type 1 = Unknown 2 = General 4 = Backup 8 = Archive 16 = Retrieve 32 = Security 64 = Backup Status 128 = Media Device

You can configure the eventlog file to log the messages that include several different severities and types. Consider the results that the following entry in the eventlog file produces:
56 255

148

Using logs Troubleshooting error messages in the NetBackup Administration Console for UNIX

Entry 56

Produces a log with messages that have a severity of warning, error, and critical. ( 56 = 8 + 16 + 32) Produces a log with messages for all types. (225 = 1 + 2 + 4 + 8 + 16 + 32 + 64 +128)

Entry 255

Consider the following example message that is written in the Windows Event Viewer Application log:
16 4 10797 -1 cacao bush nbpem backup of client bush exited with status 71

The definition of each value is as follows (left to right):

Severity = 16 (Error) Type = 4 (Backup) Job ID = 10797 Job group ID = 1 Server = cacao Client = bush Process = nbpem Text = backup of client bush exited with status 71

Troubleshooting error messages in the NetBackup Administration Console for UNIX

Most error messages in the NetBackup Administration Console for UNIX appear in the following locations:

An attention dialog box An error message pane in the lower right area of the console

If the errors appear elsewhere, they are Java exception errors. They may appear in the status line (bottom) of the NetBackup Administration Console window. They also may appear in the log file that contains the stdout or the stderr messages that the Java APIs or the NetBackup Administration Console write. Symantec does not document Java exception errors. Four types of error messages appear in the NetBackup Administration Console.

Using logs Troubleshooting error messages in the NetBackup Administration Console for UNIX

149

NetBackup status codes and messages

The operations that are performed in the NetBackup Administration Console can result in errors that are recognized in other parts of NetBackup. These errors usually appear exactly as documented in the NetBackup status codes and messages.

Note: A status code does not always accompany the error message.
To find the status code, look up the NetBackup message in the alphabetical listing and click the link to see a full description. See the Status Codes Reference Guide. NetBackup Administration Console: application server status codes and messages These messages have status codes in the 500 range. Messages with status codes 500, 501, 502, 503 and 504 begin with "Unable to login, status:". Messages with status codes 511 and 512 may or may not begin with "Unable to login, status:".

Note: A status code does not always accompany the error message.
See the Status Codes Reference Guide.

Java exceptions

Either the Java APIs or NetBackup Administration APIs generate these exceptions. These messages begin with the name of the exception. For example: java.lang.ClassCastException or vrts.nbu.NBUCommandExecutionException Java exceptions usually appear in one of the following places: The status line (bottom) of the NetBackup Administration window The log file that the jnbSA or jbpSA commands generate

The output file of the Windows Display Console .bat file if it is set up See Troubleshooting error messages in the NetBackup Administration Console for UNIX on page 148.

Operating system errors

Any messages that do not match those in the NetBackup documentation are most likely messages from the operating system.

About extra disk space required for logs and temporary files
For successful operation, the NetBackup Administration Console requires extra disk space to store logs and temporary files. The disk space should be available in the following locations.

150

Using logs Troubleshooting error messages in the NetBackup Administration Console for UNIX

On the host that is specified in the logon dialog box In /usr/openv/netbackup/logs/user_ops On the host where the console was started In /usr/openv/netbackup/logs/user_ops/nbjlogs

If space is not available in the respective file systems, you may experience the following:

Long waits for application response Incomplete data No response during logon Reduced functionality in the NetBackup interface, for example, only the Backup, Archive, and Restore and Files System Analyzer nodes appear in the tree Unexpected error messages:

"Cannot connect" socket errors during logon to the NBJava application server "Unable to login, status: 35 cannot make required directory" "/bin/sh: null: not found (1) " "An exception occurred: vrts.nbu.admin.bpmgmt.CommandOutputException: Invalid or unexpected class configuration data: <the rest of the message will vary>" Empty warning dialog boxes

Enabling detailed debug logging

The NetBackup Administration Console is a distributed application that allows administration of remote NetBackup servers. All administration is accomplished through the application server of the NetBackup Administration Console. This application server is made up of an authentication service and a user service. The logon request from the logon dialog box is sent to the authentication service for validation. The user name and password have to be valid in the Windows/UNIX authentication files and process. After validation, the authentication service starts a user service under the users account. Thereafter, all NetBackup administrative tasks are performed through an instance of the user service. Additional user service processes are initiated to process requests from the console.

Using logs Troubleshooting error messages in the NetBackup Administration Console for UNIX

151

On both UNIX and Windows, the authentication service is the bpjava-msvc application. The user service is the bpjava-susvc or bpjava-usvc application. To enable detailed debug logging, you must first create logging directories for these applications. Table 3-16 Step
Step 1

Enabling detailed debug logging Description

On the NetBackup client or server that is specified in the logon dialog box, create the following directories:

Action
Create logging directories

bpjava-msvc bpjava-susvc (if a NetBackup server) bpjava-usvc (if a NetBackup client)

Create the directories in the following locations:

/usr/openv/netbackup/logs (UNIX) install_path\NetBackup\logs (Windows)

See About unified logging on page 103. See About legacy logging on page 127. Step 2 Edit the Debug.properties file Add the following line to the Debug.properties file: debugMask=2 The Debug.properties file can be found in the following locations:

/usr/openv/java Change the file on the UNIX machine where you run the jnbSA or jbpSA commands. The log file name is displayed in the xterm window where you ran the jnbSA or jbpSA commands.

install_path\VERITAS\java Change the file at this location if you use the NetBackup Java Windows Display Console.

Step 3

Edit the nbjava.bat file

Perform this step if you use the Windows Display Console on a host where NetBackup is not installed. Edit the nbjava.bat file to redirect output to a file. The nbjava.bat file is located in install_path\VERITAS\java See the nbjava.bat file for details.

152

Using logs Troubleshooting error messages in the NetBackup Administration Console for UNIX

Chapter

Using NetBackup utilities

This chapter includes the following topics:

About NetBackup troubleshooting utilities About the analysis utilities for NetBackup debug logs About network troubleshooting utilities About the NetBackup support utility (nbsu) About the NetBackup consistency check utility (NBCC) About the NetBackup consistency check repair (NBCCR) utility About the nbcplogs utility About the robotic test utilities

About NetBackup troubleshooting utilities

Several utilities are available to help diagnose NetBackup problems. The analysis utilities for NetBackup debug logs and the NetBackup support utility (nbsu) are especially useful in troubleshooting. Table 4-1 Utility Troubleshooting utilities Description

Analysis utilities for NetBackup debug They enhance NetBackups existing debug logs capabilities by providing a consolidated view of a job debug log. See About the analysis utilities for NetBackup debug logs on page 154.

154

Using NetBackup utilities About the analysis utilities for NetBackup debug logs

Table 4-1 Utility

Troubleshooting utilities (continued) Description

They verify various aspects of the network configuration inside and outside NetBackup to ensure that there is no misconfiguration See About network troubleshooting utilities on page 158.

Network troubleshooting utilities

NetBackup support utility (nbsu)

It queries the host and gathers appropriate diagnostic information about NetBackup and the operating system. See About the NetBackup support utility (nbsu) on page 159.

NetBackup consistency check utility (NBCC)

It analyzes the integrity of portions of the NetBackup configuration and catalog and database information as they pertain to tape media. See About the NetBackup consistency check utility (NBCC) on page 165.

NetBackup consistency check repair (NBCCR) utility

It processes database-catalog repair actions and automates the application of approved suggested repair actions. See About the NetBackup consistency check repair (NBCCR) utility on page 174.

nbcplogs utility

It simplifies the process of gathering logs for deliver to Symantec technical support. See About the nbcplogs utility on page 176.

About the analysis utilities for NetBackup debug logs

The debug log analysis utilities enhance NetBackups existing debug capabilities by providing a consolidated view of a job debug log. NetBackup jobs span multiple processes that are distributed across servers. You can get more information about legacy logging and unified logging. See About logs on page 101. To trace a NetBackup job you must view and correlate messages in multiple log files on multiple hosts. The log analysis utilities provide a consolidated view of

Using NetBackup utilities About the analysis utilities for NetBackup debug logs

155

the job debug logs. The utilities scan the logs for all processes that are traversed or run for the job. The utilities can consolidate job information by client, job ID, start time for the job, and policy that is associated with the job. Table 4-2 describes the log analysis utilities. To see the parameters, limitations, and examples of use for each utility, use the command with the -help option. All the commands require administrative privileges. The log analysis utilities are available for all platforms that are supported for NetBackup servers. Note: The utilities must be initiated on supported platforms. However, the utilities can analyze debug log files from most NetBackup client and server platforms for UNIX and Windows. Table 4-2 Utility
backupdbtrace

Analysis utilities for NetBackup debug logs

Description
Consolidates the debug log messages for specified NetBackup database backup jobs and writes them to standard output. It sorts the messages by time. backupdbtrace attempts to compensate for time zone changes and clock drift between remote servers and clients. At a minimum, you must enable debug logging for admin on the master server, and for bptm and bpbkar on the media server. For best results, set the verbose logging level to 5 and enable debug logging for the following: bpdbm on the master server and bpcd on all servers in addition to the processes already identified. A complete description of the backupdbtrace command is provided in the NetBackup Commands Reference Guide.

backuptrace

Copies to standard output the debug log lines relevant to the specified backup jobs, including online (hot) catalog backups. The backuptrace utility can be used for regular file system, database extension, and alternate backup method backup jobs. It consolidates the debug logs for specified NetBackup jobs. The utility writes the relevant debug log messages to standard output and sorts the messages by time. backuptrace attempts to compensate for time zone changes and clock drift between remote servers and clients. The format of the output makes it relatively easy to sort or grep by timestamp , program name, and server or client name. The backuptrace utility works with the nbpem, nbjm, and nbrb logs on the master server. You should enable debug logging for bpbrm and bptm or bpdm on the media server and for bpbkar on the client. For best results, set the verbose logging level to 5. Enable debug logging for the following: bpdbm and bprd on the master server and for bpcd on all servers and clients in addition to the processes already identified. A complete description of the backuptrace command is provided in the NetBackup Commands Reference Guide.

156

Using NetBackup utilities About the analysis utilities for NetBackup debug logs

Table 4-2 Utility

bpgetdebuglog

Analysis utilities for NetBackup debug logs (continued)

Description
A helper program for backuptrace and restoretrace. It can also be useful as a stand-alone program and is available for all NetBackup server platforms. bpgetdebuglog prints to standard output the contents of a specified debug log file. If only the remote machine parameter is specified, bpgetdebuglog prints the following to standard output: the number of seconds of clock drift between the local computer and the remote computer. A complete description of the bpgetdebuglog command is provided in the NetBackup Commands Reference Guide.

duplicatetrace

Consolidates the debug logs for the specified NetBackup duplicate jobs and writes them to standard output. It sorts the messages by time. duplicatetrace attempts to compensate for time zone changes and clock drift between remote servers and clients. At a minimum, you must enable debug logging for admin on the master server and for bptm or bpdm on the media server. For best results, set the verbose logging level to 5 and enable debug logging for the following: bpdbm on the master server and bpcd on all servers and clients in addition to the processes already identified. A complete description of the duplicatetrace command is provided in the NetBackup Commands Reference Guide.

importtrace

Consolidates the debug log messages for the specified NetBackup import jobs and writes them to standard output. It sorts the messages by time. importtrace attempts to compensate for time zone changes and clock drift between remote servers and clients. At a minimum, you must enable debug logging for admin on the master server. And for bpbrm, you must enable debug logging for bptm and tar on the media server. For best results, set the verbose logging level to 5 and enable debug logging for the following: bpdbm on the master server and bpcd on all servers and clients in addition to the processes already identified. A complete description of the importtrace command is provided in the NetBackup Commands Reference Guide.

Using NetBackup utilities About the analysis utilities for NetBackup debug logs

157

Table 4-2 Utility

restoretrace

Analysis utilities for NetBackup debug logs (continued)

Description
Copies to standard output the debug log lines relevant to the specified restore jobs. The restoretrace utility consolidates the debug logs for specified NetBackup restore jobs. The utility writes debug log messages relevant to the specified jobs to standard output and sorts the messages by time. restoretrace attempts to compensate for time zone changes and clock drift between remote servers and clients. The format of the output makes it relatively easy to sort or grep by timestamp, program name, and server or client name. At a minimum, you must enable debug logging for bprd on the master server. Enable debug logging for bpbrm and bptm or bpdm on the media server and tar on the client. For best results, set the verbose logging level to 5. Enable debug logging for bpdbm on the master server and for bpcd on all servers and clients. A complete description of the restoretrace command is provided in the NetBackup Commands Reference Guide.

verifytrace

Consolidates the debug log messages for the specified verify jobs and writes them to standard output. It sorts the messages by time. verifytrace attempts to compensate for time zone changes and clock drift between remote servers and clients. At a minimum, you must enable debug logging as follows: for admin on the master server and for bpbrm, bptm (or bpdm) and tar on the media server. For best results, set the verbose logging level to 5 and enable debug logging for the following: bpdbm on the master server and bpcd on all servers and clients in addition to the processes already identified. A complete description of the verifytrace command is provided in the NetBackup Commands Reference Guide.

The analysis utilities have the following limitations:

Media and device management logs are not analyzed. The legacy debug log files must be in standard locations on the servers and clients.
UNIX Windows /usr/openv/netbackup/logs/<PROGRAM_NAME>/log.mmddyy install_path/NetBackup/Logs/<PROGRAM_NAME>/mmddyy.log

An option may be added later that allows the analyzed log files to reside on alternate paths. Note: For the processes that use unified logging, no log directories must be created.

158

Using NetBackup utilities About network troubleshooting utilities

The consolidated debug log may contain messages from unrelated processes. You can ignore messages with timestamps outside the duration of the job from the following: bprd, nbpem, nbjm, nbrb, bpdbm, bpbrm, bptm, bpdm, and bpcd.

An output line from the log analysis utilities uses the following format:
daystamp.millisecs.program.sequence machine log_line daystamp millisecs program sequence machine log_line The date of the log that is in the format yyyymmdd. The number of milliseconds since midnight on the local computer. The name of program (BPCD, BPRD, etc.) being logged. Line number within the debug log file. The name of the NetBackup server or client. The line that appears in the debug log file.

For more information, see the NetBackup Commands Reference Guide.

About network troubleshooting utilities

A set of utility programs (commands) verifies various aspects of the network configuration inside and outside NetBackup to ensure that there is no misconfiguration. The utilities also provide user-friendly messages for any errors they find. Network configuration broadly falls into the following categories:

Hardware, operating system, and NetBackup level settings. Examples include correct DNS lookups, firewall port openings, and network routes and connections. The NetBackup Domain Network Analyzer (nbdna) verifies this configuration. A set of utilities that verify the NetBackup level settings. The utilities include bptestcd and bptestnetconn, and the settings they verify include CONNECT_OPTIONS and CORBA endpoint selection.

Using NetBackup utilities About the NetBackup support utility (nbsu)

159

Table 4-3 Utility

bptestbpcd

Network troubleshooting utilities Description

Tries to establish a connection from a NetBackup server to the bpcd daemon on another NetBackup system. If successful, it reports information about the sockets that are established. A complete description of the bptestbpcd command is provided in the NetBackup Commands Reference Guide.

bptestnetconn

Performs several tasks that aid in the analysis of DNS and connectivity problems with any specified list of hosts. This list includes the server list in the NetBackup configuration. To help troubleshoot connectivity problems between the services that use CORBA communications, bptestnetconn can perform and report on CORBA connections to named services. A complete description of the bptestnetconn command is provided in the NetBackup Commands Reference Guide.

nbdna (NetBackup Domain Network Analyzer)

Evaluates the hostnames in the NetBackup Domain. The nbdna utility self-discovers the NetBackup domain and evaluates hostname information, then tests connectivity to these hostnames and validates their network relationship status. Network connectivity evaluation in a NetBackup domain is difficult. NetBackup domains can scale to hundreds of servers, and thousands of clients across complex network topologies. A complete description of the nbdna command is provided in the NetBackup Commands Reference Guide.

About the NetBackup support utility (nbsu)

The NetBackup support utility (nbsu) is a command line tool. It queries the host and gathers appropriate diagnostic information about NetBackup and the operating system. nbsu provides a wide range of control over the types of diagnostic information gathered. For instance, you can obtain information about NetBackup configuration settings, about specific troubleshooting areas, or about NetBackup or media management job status codes. The NetBackup support utility (nbsu) resides in the following location:
UNIX Windows /usr/openv/netbackup/bin/support/nbsu install_path\NetBackup\bin\support\nbsu.exe

160

Using NetBackup utilities About the NetBackup support utility (nbsu)

Symantec recommends that you run the NetBackup support utility (nbsu) in the following circumstances:

To obtain baseline data on your NetBackup installation. If you encounter problems later, this data can be useful. To document changes in your NetBackup or operating system environment. Run nbsu periodically to keep your baseline data up to date. To help isolate a NetBackup or operating system issue. To report issues to Symantec technical support.

The following suggestions can help you run the nbsu utility more effectively:

For an nbsu description, examples, and how to gather diagnostic information to send to Symantec Technical Support, refer to the nbsu command. For troubleshooting, run nbsu when the system is in the same state as when the problem occurred. For example, do not stop and restart the NetBackup processes after the error occurs or make a change to the server or network. If you do, nbsu may not be able to gather key information about the problem. If a NetBackup component is not operational (for example, bpgetconfig does not return information), nbsu may be unable to properly report on the system. For these cases, use the -nbu_down command line option to bypass the need for NetBackup to be operational. A complete description of the nbsu command is provided in the NetBackup Commands Reference Guide.

If nbsu does not perform as expected, try the following:

By default, nbsu sends error messages to standard error (STDERR) and also includes the messages in its output files under the header STDERR. Note the following alternate ways to view nbsu error messages:
To redirect the nbsu error messages to standard output (STDOUT) Enter the following: UNIX /usr/openv/netbackup/bin/support/nbsu 2>&1

Windows install_path\NetBackup\bin\support\nbsu.exe 2>&1

To send all nbsu screen output including error messages to a file

Enter the following: nbsu 2>&1 > file_name Where 2>&1 directs standard error into standard output, and file_name directs standard output into the designated file.

Using NetBackup utilities About the NetBackup support utility (nbsu)

161

To generate the debug messages that relate to nbsu, enter the following:
# nbsu -debug

The messages are written to the nbsu_info.txt file. The nbsu_info.txt file provides an overview of the environment where nbsu is run. It contains the following:

General operating system and NetBackup information on the environment that nbsu detects A list of diagnostics that were run A list of diagnostics that returned a non-zero status

The information in nbsu_info.txt may indicate why nbsu returned particular values, or why it did not run certain commands. If nbsu does not produce adequate information or if it seems to perform incorrectly, run nbsu with the -debug option. This option includes additional debug messages in the nbsu_info.txt file. A complete description of the nbsu command is provided in the NetBackup Commands Reference Guide.

Output from the NetBackup support utility (nbsu)

The NetBackup support utility (nbsu) writes the information it gathers to text files in the following directory:
UNIX /usr/openv/netbackup/bin/support/output/nbsu /hostname_timestamp

Windows

install_path\NetBackup\bin\support\output\nbsu \hostname_timestamp

The NetBackup environment where nbsu runs determines the particular files that nbsu creates. nbsu runs only those diagnostic commands that are appropriate to the operating system and the NetBackup version and configuration. For each diagnostic command that it runs, nbsu writes the command output to a separate file. As a rule, the name of each output file reflects the command that nbsu ran to obtain the output. For example, nbsu created the NBU_bpplclients.txt by running the NetBackup bpplclients command and created the OS_set.txt file by running the operating systems set command.

162

Using NetBackup utilities About the NetBackup support utility (nbsu)

Each output file begins with a header that identifies the commands that nbsu ran. If output from more than one command was included in the file, the header identifies the output as an internal procedure. Figure 4-1 shows the actual commands and output follow the header. Figure 4-1 Example nbsu output file: ipconfig command (excerpt)

--------------------- Network ipconfig information report ------------------------------------------- Command used ---------------------------> "C:\WINDOWS\system32\ipconfig" /all Windows IP Configuration Host Name . . . . . . . Primary Dns Suffix . . Node Type . . . . . . . IP Routing Enabled. . . WINS Proxy Enabled. . . DNS Suffix Search List. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . : : : : : : host1 Hybrid No No company.com

Figure 4-2 shows an example of part of the nbsu output file for the bpgetconfig command. Figure 4-2 Example nbsu output file: bpgetconfig command (excerpt)

------------------ NetBackup bpgetconfig information report ---------------------------- nbsu diagnostic name and internal procedure used ------------NBU_bpgetconfig - NBU_get_bpgetconfig_info ------------------------------- Command Used ------------------------------> "C:\Program Files\VERITAS\netbackup\bin\admincmd\bpgetconfig" -g host1 -L Client/Master = Master NetBackup Client Platform = PC, Windows2000 NetBackup Client Protocol Level = 6.5.0 Product = NetBackup Version Name = 6.5Alpha Version Number = 650000 NetBackup Installation Path = C:\Program Files\VERITAS\NetBackup\bin Client OS/Release = Windows2003 5 ------------------------------- Command Used ------------------------------> "C:\Program Files\VERITAS\netbackup\bin\admincmd\bpgetconfig" SERVER = host1 SERVER = host2 SERVER = host3 SERVER = host4 SERVER = host5 SERVER = host6 SERVER = host7

Using NetBackup utilities About the NetBackup support utility (nbsu)

163

If the executed command returned a non-zero status, an EXIT STATUS header indicates the status. For example:
----------------------- EXIT STATUS = 227 -------------------------

As part of the internal processing of each command that a diagnostic command runs, nbsu redirects each commands STDERR to an internal file. If the command writes information to STDERR, nbsu captures this information and includes a STDERR header along with the information. For example:
----------------------------- STDERR -----------------------------bpclient: no entity was found (227)

If a supported archive program is available on the host where nbsu runs, nbsu bundles its output files into an archive file. If a supported compression utility is available, nbsu compresses the archive file. Otherwise, the individual output files remain unarchived and uncompressed. An example of a compressed archive file that nbsu created is as follows:
/usr/openv/netbackup/bin/support/output/nbsu/host1_master_20060814_ 164443/host1_master_20060814_164443.tar.gz

where host1 is the name of the host on which nbsu ran. master indicates that the host is a NetBackup master server.
nbsu supports tar for archive and gzip for compression. Symantec may add support

for other archive and compression utilities in the future. For an up-to-date list of supported archive and compression utilities, run the nbsu -H command on your installed version of NetBackup. Note: Archiving and compression utilities are usually available on UNIX and Linux systems. On Windows, it may be necessary to install these programs. Note that the archiving utility must be referenced in the system PATH environment variable. If no archive utility is installed on your system, use the -xml option of the nbsu command. This option lets you create a single .xml file in place of the individual output files. The single .xml file contains all the diagnostic information that the individual files contain. Use this command to conveniently bundle nbsu output for Symantec technical support. A complete description of the nbsu command is provided in the NetBackup Commands Reference Guide.

164

Using NetBackup utilities About the NetBackup support utility (nbsu)

Status code information gathered by the NetBackup support utility (nbsu)

You can use nbsu to gather diagnostic information about certain NetBackup or Media Manager status codes. nbsu gathers this information by running one or more NetBackup commands whose output may indicate the cause of the problem. You can get more information about the commands that nbsu runs by looking under Recommended Actions for the topics that describe the status codes. See the Status Codes Reference Guide. See nbsu in the Commands Reference Guide. The following are examples of the results you can get when you enter specific commands:
To gather diagnostic information about Enter nbsu -nbu_e 25. status code 25 This command runs only the diagnostic commands that are related to NetBackup status code 25. Since fewer commands are run, the result may be a smaller set of output files. To determine what information nbsu Enter nbsu -l -nbu_e 25. can collect for a particular status code

You can get more information about the output files that nbsu generates. See Output from the NetBackup support utility (nbsu) on page 161. Note: You can also use a NetBackup exit script to call nbsu. The script passes the NetBackup status code to nbsu to gather associated diagnostics for a job. A complete description of the nbsu command is provided in the NetBackup Commands Reference Guide.

Example of a progress display for the NetBackup support utility (nbsu)

By default, the NetBackup support utility (nbsu) displays its progress to standard output. First, it lists environment queries, and then it lists the diagnostic commands that it runs as in the following example:
C:\Program Files\VERITAS\NetBackup\bin\support>nbsu 1.0 Determining initial nbsu settings 1.1 Determining OS environment

Using NetBackup utilities About the NetBackup consistency check utility (NBCC)

165

1.2 1.3 1.4 2.0 2.1 3.0

Determining OS host services Determining identified network interface hostnames Determining NetBackup environment Querying nbsu diagnostic lists Determining nbsu diagnostics to run Executing nbsu diagnostics Executing diagnostic DEV_scsi_reg Registry query of HKEY_LOCAL_MACHINE\hardware\DeviceMap\Scsi\ Executing diagnostic MM_ndmp "C:\Program Files\VERITAS\volmgr\bin\set_ndmp_attr" -list "C:\Program Files\VERITAS\volmgr\bin\set_ndmp_attr" -probe <hostname> "C:\Program Files\VERITAS\volmgr\bin\set_ndmp_attr" -verify <hostname> Executing diagnostic MM_tpconfig "C:\Program Files\VERITAS\\Volmgr\Bin\tpconfig" -d

4.0 nbsu successfully completed the identified diagnostic commands. Creating support package... Microsoft (R) Cabinet Maker - Version 5.2.3790.0 Copyright (c) Microsoft Corporation. All rights reserved.. 770,201 bytes in 36 files Total files: 36 Bytes before: 770,201 Bytes after: 105,503 After/Before: 13.70% compression Time: 0.67 seconds ( 0 hr 0 min 0.67 sec) Throughput: 1119.27 Kb/second Cleaning up output files... The results are located in the .\output\nbsu\lou4_master_20070409_160403 directory...

A complete description of the nbsu command is provided in the NetBackup Commands Reference Guide.

About the NetBackup consistency check utility (NBCC)

The NetBackup consistency check utility (NBCC) is a command line utility. It is used to analyze the integrity of portions of the NetBackup configuration and

166

Using NetBackup utilities About the NetBackup consistency check utility (NBCC)

catalog and database information as they pertain to tape media. This analysis includes review of NetBackup storage units, the EMM server, volume pools, tape media, and backup images that are associated with tape media.
NBCC does the following:

Queries the EMM database to obtain the primary hostname, associated hostnames, and server attributes for hostname normalization Through examination of the NetBackup configuration, identifies, cluster, application cluster and servers Gathers database and catalog information Analyzes the consistency of the gathered configuration and database and catalog information Creates a packaged bundle for Symantec technical support to review

NBCC resides in the following location: UNIX Windows /usr/openv/netbackup/bin/support/NBCC install_path\NetBackup\bin\support\NBCC.exe

Symantec recommends that you run NBCC in the following circumstances:

To check the consistency of the NetBackup configuration and catalog and database information from a tape media perspective To gather and create a package bundle when directed to do so by Symantec technical support

The following items can help you run the NBCC utility:

For an NBCC description, examples, and how to gather NetBackup catalog and database information to send to Symantec technical support , refer to the NBCC -help command.
NBCC is designed to be run on NetBackup master servers.

In some cases, a non-functioning operating system or NetBackup process or service can prevent NBCC from running properly or completing. As NBCC progresses through the interrogation of various operating system or NetBackup components, it outputs what processes to STDOUT. As NBCC processes catalog and database components, it displays how many records have been processed. The number of records that are processed is in direct relationship to the size of the catalog and database being processed. If NBCC detects a failure, related information is output to STDERR. Information to STDOUT or STDERR are also output to the nbcc-info.txt file (if available).

Using NetBackup utilities About the NetBackup consistency check utility (NBCC)

167

If NBCC does not perform as expected, try the following:

Use a text editor to look for error notices in the nbcc-into.txt file. By default, NBCC sends error messages to standard error (STDERR) and also includes the messages in its output files under the header STDERR. If NBCC does not produce adequate information or if it seems to perform incorrectly, run NBCC with the -debug option to include additional debug messages in the nbcc-info.txt file. For troubleshooting, run NBCC when the system is in the same state as when the problem occurred. For example, do not stop and restart the NetBackup processes after the error occurs or make a change to the server or network. NBCC may not be able to gather key information about the problem.

The nbcc-info.txt file provides an overview of the environment where NBCC is run, and contains the following:

General operating system and NetBackup configuration information on the environment that NBCC detects A copy of the NBCC processing information that was displayed to STDOUT or STDERR.

This information would indicate the processing that NBCC had done. The Processing detected NetBackup server entries section of the nbcc-info.txt contains a Summary of NBCC server processing. This information summarizes the results of the processing of detected server entries. See Example of an NBCC progress display on page 168. A complete description of the NBCC command is provided in the NetBackup Commands Reference Guide.

Output from the NetBackup consistency check utility (NBCC)

NBCC writes the information it gathers to packaged files in the following directory. UNIX and Linux /usr/openv/netbackup/bin/support/output /nbcc/hostname_NBCC_timestamp

Windows

install_path\NetBackup\bin\support\output \nbcc\hostname_NBCC_timestamp

If a supported archive program is available on the host where NBCC runs, NBCC bundles its output files into an archive file. If a supported compression utility is

168

Using NetBackup utilities About the NetBackup consistency check utility (NBCC)

available, NBCC compresses the archive file. Otherwise, the individual output files remain unarchived and uncompressed. An example of a compressed (UNIX) archive file that NBCC created is as follows:
/usr/openv/netbackup/bin/support/output/NBCC/host1_NBCC_20060814_ 164443/host1_NBCC_20060814_164443.tar.gz

where host1 is the name of the host where NBCC had been run. On UNIX platforms, NBCC supports the tar , compress, and gzip utilities for UNIX file archiving and compression. On Windows platforms, NBCC supports the tar, Makecab, and gzip utilities for Windows file archiving and compression. A complete description of the NBCC command is provided in the NetBackup Commands Reference Guide.

Example of an NBCC progress display

By default, NetBackup consistency check utility (NBCC) displays its progress numerically to standard output. The name of the output file is nbcc-info.txt. The following example of NBCC output has been edited for brevity:
1.0 1.1 Gathering initial NBCC information Obtaining initial NetBackup configuration information

NBCC is being run on NetBackup master server mlbnbu NBCC version 7.5 Gather mode = full NBCC command line = C:\Veritas\NetBackup\bin\support\NBCC.exe -nozip \ -nocleanup OS name = MSWin32 OS version = Microsoft Windows [Version 6.0.6002] NetBackup Install path = C:\Veritas\ > dir output\nbcc\mlbnbu_NBCC_20110707_094057 2>&1 Parsed output for "bytes free" 4 Dir(s) 2.0 862,367,666,176 bytes free

Gathering required NetBackup configuration information If NBCC is unable to determine the NetBackup version for ANY detected media server, is there a SINGLE version of NetBackup you would like associated to these media servers? [Y/y,N/n] N

Using NetBackup utilities About the NetBackup consistency check utility (NBCC)

169

If NBCC detects images that were written by media servers that are not known to NetBackup, would you like NBCC to: 1. Stop now, since I know of such media servers, and I wish to resolve these before running NBCC again Prompt me again, after NBCC has processed all images, so that I can designate a media server known to NetBackup that full analysis will use to INHERIT ALL of the image copies associated to the unknown media servers Prompt me again, after NBCC has processed ALL images, so that I can select the course of action to take for EACH media server Flag ALL of the unknown media servers so that full analysis will mark all of their related image copies to be EXPIRED Flag ALL of the unknown media servers so that full analysis will generate COMMENTED out repairs that can be reviewed

2.

3.

4.

5.

2.1 2.2

2.3

2.4

2.5

2.6

[1,2,3,4,5] 4 Determining the date format to use with NetBackup commands... Using the date format /mm/dd/yyyy Building EMM host configuration information... Detected the EMM Server hostname mlbnbu ... Obtaining EMM server aliases... EMM aliases for detected EMM Server mlbnbu mlbnbu.min.veritas.com ... Setting internal EMM Cluster member state... Cluster mlbnbu Member american is inactive Member national is active Building NetBackup storage unit list... Detected tape media storage unit host gath.min.veritas.com ... Obtaining Disk Pool information... Detected Disk Pool

170

Using NetBackup utilities About the NetBackup consistency check utility (NBCC)

2.7

2.8

2.9

xena2-pool host xenabl2.min.veritas.com Detected Disk Pool xena2-pool member xenabl2.min.veritas.com Obtaining tpconfig Storage credential information... Detected the media server hostname xenabl2.min.veritas.com and associated Storage server hostname xenabl2.min.veritas.com Evaluating the relationship of EMM Cluster members and configured Storage Units or Disk Pools... Evaluating Cluster mlbnbu Storage Units are properly configured for this cluster Analyzing EMM master and/or media servers and configured Storage Units... The following EMM server entries do not have configured Storage Units or Disk Pools: Media server - mann.min.veritas.com

2.10 Obtaining NetBackup unrestricted media sharing status... Configuration state = NO 2.11 Obtaining NetBackup Media Server Groups... No Server Groups configured 2.12 Building NetBackup retention level list... 3.0 Obtaining NetBackup version from media servers mlbnbu... janebl6.lab.rmn.dcmg.symantec.com... ... 3.1 Gathering required NetBackup catalog information Start time = 2011-07-07 09:41:07 3.2 Gathering NetBackup EMM conflict table list Found 0 EMM conflict records 3.3 Gathering list of all tapes associated with any Active Jobs Building NetBackup bpdbjobs list 3.4 Gathering all TryLog file names from the E:\Veritas\netbackup\db\jobs\trylogs directory Found 10 TryLogs for 10 active jobs. TryLogs found for all Active Jobs 3.5 Building NetBackup Image database contents list Reading Image number 1000

Using NetBackup utilities About the NetBackup consistency check utility (NBCC)

171

Reading Image number 2000 Reading Image number 3000 Reading Image number 4000 Found 4014 images in the Image database 3.6 Unloading the NBDB database Reading the reload.sql file... Reading the Machine table file... Counting the number of records in the Image table file... Counting Image record 1000 ... 3.7 Building EMM database Media and Device configuration attribute lists Obtaining the EMM database Media attribute list for disk virtual server mlbnbu ... There were 0 bpmedialist records detected for media server mlbnbu Getting device configuration data from server mlbnbu ... ... 3.8 Building EMM database Unrestricted Sharing Media attribute lists Found 0 Unrestricted Sharing media records in the EMM database 3.9 Building the EMM database Volume attribute list... Getting the EMM database Volume attributes from EMM server mlbnbu ... Found 43 Volume attribute records in the EMM database 3.10 Building NetBackup volume pool configuration list EMM Server mlbnbu 3.11 Building NetBackup scratch pool configuration list EMM Server mlbnbu 3.12 Gathering NetBackup EMM merge table list Found 0 EMM merge table records Summary of gathered NetBackup catalog information End time = 2011-07-07 09:44:16 Number of Images gathered = 4014 Number of database corrupt images gathered = 0 Number of EMM database Media attribute records gathered = 38 Number of EMM database Volume attribute records gathered = 43 Catalog data gathering took 189 seconds to complete ***WARNING***

172

Using NetBackup utilities About the NetBackup consistency check utility (NBCC)

** ** ** ** ** ** ** ** ** ** **

It took more than 60 seconds to collect all of the DB data. If backups are running, there is a risk that the data collected might not be consistent, because the DBs may have been updated while the data was being collected. If backups are running and it is possible, please stop all backups and run NBCC again. If it is not possible to stop all backups, just let NBCC complete.

dir results for created NBCC files: 07/07/2011 09:42 AM 8 nbcc-active-tapes 07/07/2011 07/07/2011 ... 4.0 5.0 5.1 5.3 09:42 AM 09:43 AM 752,698 nbcc-bpdbjobs-most_columns 2,211,811 nbcc-bpimagelist-l

Verifying required catalog components were gathered Beginning NetBackup catalog consistency check Start time = 2011-07-07 09:44:18 Loading list of tape media involved in active NetBackup jobs Processed 1 active tape media involved with active NetBackup jobs. Processing EMM database Volume attribute records, pass 1 (of 2), 43 records to be processed Processed 43 EMM database Volume attribute records. Checking for duplicate EMM server host names in Volume attribute data Processing NBDB Image table records, pass 1 (of 1), 6981 NBDB Image table records to be processed 6981 NBDB Image table records processed 6981 NBDB Image table records for this NetBackup domain processed Processing NBDB Image Copy table records, pass 1 (of 1), 4341 NBDB Image Copy table records to be processed 4341 NBDB Image Copy table records processed Processing NBDB Image Fragment table records, pass 1 (of 1), 8605 NBDB Image Fragment table records to be processed 8605 NBDB Image Fragment table records processed Processing Image DB, pass 1 (of 2), 4014 images to be processed

5.4 5.5

5.6

5.7

5.8

Using NetBackup utilities About the NetBackup consistency check utility (NBCC)

173

4014 images processed on pass 1 Processing EMM database Media attribute records, pass 1 (of 3), 38 records to be processed Processed 38 EMM database Media attribute records. 5.11 Check for duplicate media server names in the EMM database Media attribute data 5.12 Processing EMM database Media attribute records, pass 2 (of 3), 38 records to be processed CONSISTENCY_ERROR Oper_16_4 5.9 5.13 NetBackup catalog consistency check completed End time = 2011-07-07 09:44:25 5.14 Checking for the latest NBCCR repair output directory C:\Veritas\netbackup\bin\support\output\nbccr No repair file output directory detected. Summary of NBCC EMM Server processing ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + Primary and associated alias hostnames: + + mlbnbu mlbnbu.min.veritas.com + + Sources: + + nbemmcmd vmoprcmd + + EMM Server = yes + + EMM NetBackup version = 7.5 + + NBCC NetBackup version = 7.5 + ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ... ***NBCC DETECTED A NetBackup CATALOG INCONSISTENCY!*** Report complete, closing the .\output\nbcc\mlbnbu_NBCC_20110707_094057\nbcc-info.txt output file.

A complete description of the NBCC command is provided in the NetBackup Commands Reference Guide.

174

Using NetBackup utilities About the NetBackup consistency check repair (NBCCR) utility

About the NetBackup consistency check repair (NBCCR) utility

The NetBackup consistency check repair (NBCCR) utility is a command line tool that processes database-catalog repair actions. It automates the application of approved suggested repair actions. Symantec technical support analyzes the data that is collected by the NBCC utility and site-specific configuration information. This analysis results in the generation of a suggested repair actions (SRA) file. Before NBCCR is run, Symantec technical support interacts with the customer to determine which repairs are needed. Undesirable repair actions are deleted or commented out of the SRA file. Each line of the SRA file contains one repair action that is paired with an associated parameter. The NBCCR utility executes each repair action in several stages. Table 4-4 Stage
Stage 1

Stages of repair Name

Data collection

Description
NBCCR first collects the information that is required to perform a repair. Immediately before the suggested repair is applied, NBCCR verifies that the current status of the tape still qualifies for the requested repair. It recognizes that time has passed and the environment may have changed since the data was collected. If so, it reports in a history file that the repair is not qualified. Finally, NBCCR performs up to three steps of repair for every repair entry in the SRA file. An element may be modified to enable the repair and steps may be necessary after the repair. If the repair fails during the repair operation, NBCCR tries to roll back the repair so that the corrective action does not introduce any new errors.

Stage 2

Repair qualification

Stage 3

Repair

NBCCR resides in the following location:

UNIX Windows /usr/openv/netbackup/bin/support/NBCCR install_path\NetBackup\bin\support\NBCCR.exe

NBCCR accepts one input file, creates two output files, and uses one temporary file.

Using NetBackup utilities About the NetBackup consistency check repair (NBCCR) utility

175

Input file

NBCCR accepts as input the Suggested Repair Action (SRA) file named mastername_NBCCA_timestamptxt. Technical Support analyzes the NBCC support package and generates this file which is sent to the end user. This file is placed in the following directory for NBCCR processing: On Unix: /usr/openv/netbackup/bin/support/input/nbccr/SRA On Windows: install_path\NetBackup\bin\support\input\nbccr\SRA

Output files

NBCCR automatically creates a separate directory for each SRA file processed. The file name is based on the contents of the SRA file. The name of the directory is as follows: On UNIX: /usr/openv/netbackup/bin/support/output/ nbccr/mastername_nbccr_timestamp On Windows: install_path\NetBackup\bin\support\output\ nbccr\mastername_nbccr_timestamp. After repair processing is complete, NBCCR relocates the SRA file to the same directory. NBCCR also creates the following output files and places them in the same directory. NBCCR creates NBCCR.History.txt, which is a history file of all the repair actions attempted. NBCCR creates NBCCR.output.txt.

Temporary file

While it runs, the NBCCR utility uses KeepOnTruckin.txt, which appears in the same location as the output files described above. To terminate NBCCR while it processes repairs, delete this file. This action causes NBCCR to complete the current repair, then shut down. Any other interruption causes undetermined results.

The following sample NBCCR.output.txt files show the results of two MContents repairs. One where all images were found on tape and one where one or more images were not found on the tape:

Example 1: NBCCR found all images on the tape. The MContents repair action is successful.
MContents for ULT001 MediaServerExpireImagesNotOnTapeFlag ExpireImagesNotOnTape flag not set ULT001 MContents - All images in images catalog found on tape MContents ULT001 status: Success

176

Using NetBackup utilities About the nbcplogs utility

Example 2: NBCCR did not find one or more images on the tape. The MContents repair action was not performed.
MContents for ULT000 MediaServerExpireImagesNotOnTapeFlag ExpireImagesNotOnTape flag not set Did NOT find Backup ID winmaster_123436 Copy 1 AssignTime 2011-02-11 01:19:13 (123436) on ULT000 Leaving winmaster_123436 Copy 1 on ULT000 in ImageDB ULT000 MContents - One or more images from images catalog NOT found on tape MContents ULT000 status: ActionFailed

A complete description of the NBCCR command is provided in the NetBackup Commands Reference Guide.

About the nbcplogs utility

When you troubleshoot a customer problem, you must gather and copy the correct logs to debug the issue. The log types (NBU, vxul, vm, pbx,...) may be in many places. The process of getting the logs to Symantec technical support can be difficult and time consuming. By default, nbcplogs (NetBackup log uploader) now runs the nbsu utility and uploads nbsu information for the host system. This capability improves the end-user experience with Technical Support by saving time and keystrokes to gather and upload information. The utility also gathers additional log information for clusters and pack history information.
nbcplogs uses file transfer protocol (FTP) to upload its support package to

Technical Support. This process requires temporary disk space to build the compressed bundle that it transfers. You can configure this temporary space by setting up an environment variable (TMPDIR) and using a nbcplogs command line option (--tmpdir) as follows: On Windows:
# nbcplogs --tmpdir=C:\temp -f ###-###-###

On UNIX: In /bin/sh, enter the following:

# TMPDIR=/tmp # export TMPDIR # nbcplogs -f ###-###-###

Using NetBackup utilities About the robotic test utilities

177

In /bin/bash, enter the following:

# export TMPDIR=/tmp # nbcplogs -f ###-###-###

In /bin/csh or /bin/tcsh, enter the following:

# nbcplogs --tmpdir=/tmp -f ###-###-###

This utility supports three types of search algorithms. These are command options that are part of the nbcplogs command line.

--filecopy. File copy is the default condition. It copies the entire log file. File

copy with compression is usually enough to get the job done.

--fast. Fast search uses a binary search to strip out lines that are outside the

time frame of the file. This mechanism is useful when copying extremely large log files such as bpdbm. This option is rarely needed and should be used with caution. The default condition is the file copy, which copies the entire log file. A fast search algorithm uses a binary search to strip out lines that are outside the time frame of the file. This mechanism is useful when copying extremely large log files such as bpdbm. The nbcplogs utility is intended to simplify the process of copying logs by specifying the following options:

A time frame for the logs. The log types that you want to collect. Bundling and in-transit data compression.

In addition, you can preview the amount of log data to be copied. A complete description of the nbcplogs command is provided in the NetBackup Commands Reference Guide.

About the robotic test utilities

Each of the robotic software packages includes a robotic test utility for communicating directly with robotic peripherals. The tests are for diagnostic purposes and the only documentation is the online Help that you can view by entering a question mark (?) after starting the utility. Specify -h to display the usage message.

178

Using NetBackup utilities About the robotic test utilities

Note: Do not use the robotic test utilities when backups or restores are active. The tests lock the robotic control path and prevent the corresponding robotic software from performing actions, such as loading and unloading media. If a mount is requested, the corresponding robotic process times out and goes to the DOWN state. This usually results in a media mount timeout. Also, be certain to quit the utility when your testing is complete.

Robotic tests on UNIX

If the robot has been configured (that is, added to the EMM database), start the robotic test utility by using the robtest command. This action saves time, since robotic and drive device paths are passed to the test utility automatically. The procedure is as follows: To use the robtest command, do the following (in the order presented):

Execute the following command:

/usr/openv/volmgr/bin/robtest

The test utility menu appears.

Select a robot and press Enter. The test starts.

If the robot is not configured, you cannot use robtest and must execute the command that applies to the robot you test.
ACS /usr/openv/volmgr/bin/acstest -r ACSLS_hostpath for acstest to work on UNIX and Linux, acssel and acsssi must be running ODL TL4 TL8 TLD TLH TLM TSH /usr/openv/volmgr/bin/odltest -r roboticpath /usr/openv/volmgr/bin/tl4test -r roboticpath /usr/openv/volmgr/bin/tl8test -r roboticpath /usr/openv/volmgr/bin/tldtest -r roboticpath /usr/openv/volmgr/bin/tlhtest -r robotic_library_path /usr/openv/volmgr/bin/tlmtest -r DAS_host /usr/openv/volmgr/bin/tshtest -r roboticpath

More information on ACS, TLH, and TLM robotic control is available.

Using NetBackup utilities About the robotic test utilities

179

See the NetBackup Device Configuration Guide. In the previous list of commands, roboticpath is the full path to the device file for the robotic control (SCSI). You can review the section for your platform to find the appropriate value for roboticpath. An optional parameter specifies the device file path for the drives so that this utility can unload the drives using the SCSI interface.

Robotic tests on Windows

If the robot has been configured (that is, added to the EMM database), start the robotic test utility by using the robtest command. This action saves time, since robotic and drive device paths are passed to the test utility automatically. To use the robtest command, do the following (in the order presented):

Execute the following command:

install_path\Volmgr\bin\robtest.exe

The test utility menu appears.

Select a robot and press Enter. The test starts.

Note: If the robot is not configured, you cannot use robtest and must execute the command that applies to the robot you are testing (see following list).
ACS TL4 TL8 TLD TLH install_path\Volmgr\bin\acstest -r ACSLS_HOST install_path\Volmgr\bin\tl4test -r roboticpath install_path\Volmgr\bin\tl8test -r roboticpath install_path\Volmgr\bin\tldtest -r roboticpath install_path\Volmgr\bin\tlhtest -r robotic_library_name install_path\Volmgr\bin\tlmtest -r DAS_Hostname

TLM

More information on ACS, TLH, and TLM robotic control is available. See the NetBackup Device Configuration Guide.

180

Using NetBackup utilities About the robotic test utilities

In the previous list of commands, roboticpath is the full path to the device file for the robotic control (SCSI). You can review the section for your platform to find the appropriate value for roboticpath. An optional parameter specifies the device file path for the drives so that this utility can unload the drives using the SCSI interface. Usage is:
install_path <-p port -b bus -t target -l lan | -r roboticpath>

where: roboticpath is the changer name (e.g., Changer0).

Chapter

Disaster recovery
This chapter includes the following topics:

About disaster recovery Recommended backup practices About disk recovery procedures for UNIX and Linux About clustered NBU server recovery for UNIX and Linux About disk recovery procedures for Windows About clustered NBU server recovery for Windows About recovering the NetBackup catalog

About disaster recovery

Data backup is essential to any data protection strategy, especially a strategy that is expected to assist in disaster recovery. Regularly backing up data and therefore being able to restore that data within a specified time frame are important components of recovery. Regardless of any other recovery provisions, backup protects against data loss from complete system failure. And off-site storage of backup images protects against damage to your on-site media or against a disaster that damages or destroys your facility or site. To perform recovery successfully, the data must be tracked. Knowing at what point in time the data was backed up allows your organization to assess the information that cannot be recovered. Configure your data backup schedules to allow your organization to achieve its recovery point objective (RPO). The RPO is the point in time before which you cannot accept lost data. If your organization can accept one days data loss, your backup schedule should be at least daily. That way you can achieve an RPO of one day before any disaster.

182

Disaster recovery Recommended backup practices

Your organization also may have a recovery time objective (RTO), which is the expected recovery time or how long it takes to recover. Recovery time is a function of the type of disaster and of the methods that are used for recovery. You may have multiple RTOs, depending on which services your organization must recover when. High availability technologies can make the recovery point very close or even identical to the point of failure or disaster. They also can provide very short recovery times. However, the closer your RTO and RPO are to the failure point, the more expensive it is to build and maintain the systems that are required to achieve recovery. Your analysis of the costs and benefits of various recovery strategies should be part of your organizations recovery planning. Effective disaster recovery requires procedures specific to an environment. These procedures provide detailed information regarding preparation for and recovering from a disaster. Use the disaster recovery information in this chapter as a model only; evaluate and then develop your own disaster recovery plans and procedures. Warning: Before you try any of the disaster recovery procedures in this chapter, Symantec recommends that you contact technical support. This topic provides information about NetBackup installation and (if necessary), catalog recovery after a system disk failure. Symantec assumes that you recover to the original system disk or one configured exactly like it.

Warning: NetBackup may not function properly if you reinstall and recover to a different partition or to one that is partitioned differently due to internal configuration information. Instead, configure a replacement disk with partitioning that is identical to the failed disk. Then reinstall NetBackup on the same partition on which it was originally installed. The specific procedures that replace failed disks, build partitions and logical volumes, and reinstall operating systems can be complicated and time consuming. Such procedures are beyond the scope of this manual. Appropriate vendor-specific information should be referenced.

Recommended backup practices

The following backup practices are recommended:

Disaster recovery Recommended backup practices

183

Selecting files to back up

In addition to backing up files on a regular basis, it is important to select the correct files to back up. Include all files with records that are critical to users and the organization. Back up system and application files, so you can quickly and accurately restore a system to normal operation if a disaster occurs. Include all Windows system files in your backups. In addition to the other system software, the Windows system directories include the registry, which is needed to restore the client to its original configuration. If you use a NetBackup exclude list for a client, do not specify any Windows system files in that list. Do not omit executables and other application files. You may want to save tape by excluding these easy-to-reinstall files. However, backing up the entire application ensures that it is restored to its exact configuration. For example, if you have applied software updates and patches, restoring from a backup eliminates the need to reapply them.

Bare Metal Restore

NetBackup Bare Metal Restore (BMR) protects client systems by backing them up with a policy configured for BMR protection. A complete description of BMR backup and recovery procedures is available. See the NetBackup Bare Metal Restore Administrator's Guide.

Critical policies

When you configure a policy for online catalog backup, designate certain NetBackup policies as critical. Critical policies back up systems and data deemed critical to end-user operation. During a catalog recovery, NetBackup verifies that all of the media that is needed to restore critical policies are available. If the configuration contains Windows clients that have incremental backup configurations set to Perform Incrementals Based on Archive Bit, run a full backup of these clients as soon as possible after a catalog recovery. The archive bit resets on the files that were incrementally backed up after the catalog backup that was used for the catalog recovery. If a full backup of these clients is not run after a catalog recovery, these files could be skipped and not backed up by subsequent incremental backups. Online, hot catalog backup is a policy-driven backup that supports tape-spanning and incremental backups. It allows for restoring catalog files from the Backup, Archive, and Restore interface. Online catalog backups may be run while other NetBackup activity occurs, which provides improved support for environments in which continual backup activity is typical. Symantec recommends saving the disaster recovery files that are created by the online catalog backup to a network share or removable device. Do not save the disaster recovery files to the local computer. Catalog recovery from an online catalog backup without the disaster recovery image file is a more complex procedure and time-consuming procedure.

Full backup after catalog recovery

Online catalog backups

Online catalog backup disaster recovery files

184

Disaster recovery About disk recovery procedures for UNIX and Linux

Automated recovery

The catalog disaster recovery file (created during an online catalog backup) is intended to automate the process of NetBackup recovery. If you recover a system other than the one that originally made the backups, it should be identical to the original system. For example, the system that performs the recovery should include NetBackup servers with identical names to those servers where the backups were made. If not, the automated recovery may not succeed. Configure the online catalog backup policy to email a copy of the disaster recovery information to a NetBackup administrator in your organization. Configure this policy as part of every catalog backup. Do not save the disaster recovery information emails to the local computer. Catalog recovery without the disaster recovery image file or the disaster recovery information email available is exceedingly complex, time consuming, and requires assistance. You may tailor the disaster recovery email process by using the mail_dr_info notify scrpt. More details are available. Seethe NetBackup Administrator's Guide, Volume II.

Online catalog disaster recovery information email

Identifying the correct catalog backup

Ensure that you identify and use the appropriate catalog backup for your recovery. For example, if you recover from your most recent backups, use the catalog from your most recent backups. Similarly, if you recover from a specific point in time, use the catalog backup from that specific point in time. System environment, catalog size, location, and backup configuration (full and incremental policy schedules) all help determine the time that is required to recover the catalog. Carefully plan and test to determine the catalog backup methods that result in the desired catalog recovery time. The NetBackup catalog backup protects your configuration data and catalog data. Set up backup schedules for the master servers and media servers in your NetBackup installation. These schedules protect the operating systems, device configurations, and other applications on the servers. Master or media server recovery procedures when the system disk has been lost assume that the servers are backed up separately from the catalog backup. Backups of master and media servers should not include NetBackup binaries, configuration or catalog files, or relational database data.

Catalog recovery time

Master and media server backups

About disk recovery procedures for UNIX and Linux

The three different types of disk recovery for UNIX and Linux are as follows:

Master server disk recovery procedures See Recovering the master server disk for UNIX and Linux on page 185. Media server disk recovery procedures

Disaster recovery About disk recovery procedures for UNIX and Linux

185

See About recovering the NetBackup media server disk for UNIX and Linux on page 190.

Client disk recovery procedures See Recovering the system disk on a UNIX client workstation on page 190.

The disk-based images that reside on AdvancedDisk or on OpenStorage disks cannot be recovered by means of the NetBackup catalog. These disk images must be recovered by means of the NetBackup import feature. For information on import, See the topic on importing NetBackup images in the NetBackup Administrators Guide, Volume I. When the disk image is imported, NetBackup does not recover the original catalog entry for the image. Instead, a new catalog entry is created.

Recovering the master server disk for UNIX and Linux

Two procedures explain how to recover data if the system disk fails on a UNIX or Linux NetBackup master server, as follows:

The root file system is intact. The operating system, NetBackup software and some (if not all) other files are assumed to be lost. See Recovering the master server when root is intact on page 186. The root file system is lost along with everything else on the disk. This situation requires a total recovery. This recovery reloads the operating system to an alternate boot disk and boots from this disk during recovery. This operation lets you recover the root partition without risking a crash that is caused by overwriting the files that the operating system uses during the restore. See Recovering the master server when the root partition is lost on page 187.

For NetBackup master and media servers, the directory locations of the NetBackup catalog become an integral part of NetBackup catalog backups. Any recovery of the NetBackup catalog requires identical directory paths or locations be created during the NetBackup software reinstallation. Disk partitioning, symbolic links, and NetBackup catalog relocation utilities may be needed. NetBackup Bare Metal Restore (BMR) protects client systems by backing them up with a policy configured for BMR protection. Information is available that describes BMR backup and recovery procedures. See the NetBackup Bare Metal Restore System Administrator's Guide.

186

Disaster recovery About disk recovery procedures for UNIX and Linux

Recovering the master server when root is intact

The following procedure recovers the master server by reloading the operating system, restoring NetBackup, and then restoring all other files. To recover the master server when root is intact

Verify that the operating system works, that any require patches are installed, and that specific configuration settings are made. Take corrective action as needed. Reinstall NetBackup software on the server you want to recover. See the NetBackup Installation Guide for UNIX for instructions.

Install any NetBackup patches that had been previously installed. See the documentation that was included with the patch software. Note: Symantec does not support the recovery of a catalog image that was backed up using an earlier version of NetBackup.

If any of the default catalog directories have changed that may be reflected in the NetBackup catalog backups, recreate those directories before the catalog recovery. The following are examples:

Use of symbolic links as part of the NetBackup catalog directory structure. Use of the NetBackup nbdb_move command to relocate parts of the NetBackup relational database catalog.

If the recovery scenario involves restoring policy or catalog backups, the appropriate recovery device(s) must be configured, which may involve the following tasks:

Install and configure the robotic software for the devices that read backups of the NetBackup catalog and regular backups of the disk being restored. If a non-robotic drive is available that can read these backups, then no robot is required. Although manual intervention is required if multiple pieces of media are required. See the NetBackup Device Configuration Guide. Use the NetBackup Device Configuration Wizard to discover and configure the recovery device in NetBackup. See the NetBackup Administrator's Guide, Volume I. Use the NetBackup command tpautoconf to discover and configure the recovery device in NetBackup.

Disaster recovery About disk recovery procedures for UNIX and Linux

187

See the NetBackup Commands Reference Guide.

Update the device mapping files. See the NetBackup Administrators Guide, Volume I.

If you must restore from the policy backups or catalog backups that were done to media, the appropriate media may have to be configured in NetBackup. See the NetBackup Administrators Guide, Volume I. Configuring the media may require some or all of the following tasks:

Manually load the required media into a stand-alone recovery device. Use the NetBackup utilities such as robtest or vendor-specific robotic control software to load media into the required recovery device or devices. Use the NetBackup Volume Configuration Wizard to inventory the media contents of a robotic device. Use the vendor-specific robotic control software to load the media into the required recovery device(s).

Recover the NetBackup catalogs to the server you are recovering. The NetBackup catalogs can be recovered only to the same directory structure from which they were backed up (alternate path recovery is not allowed).

Stop and restart all NetBackup daemons. Use the following NetBackup commands, or use the Activity Monitor in the NetBackup Administration Console. Your configuration may include an EMM server that is separate from the master server. If so, start NetBackup on the EMM server before starting NetBackup on the master server.
/usr/openv/netbackup/bin/bp.kill_all /usr/openv/netbackup/bin/bp.start_all

Start the NetBackup Backup, Archive, and Restore interface (or the bp command) and restore other files to the server as desired. When the files are restored, you are done.

Recovering the master server when the root partition is lost

The following procedure assumes that the root file system is lost along with everything else on the disk. This procedure reloads the operating system to an alternate boot disk and boots from that disk during recovery. This operation lets you recover the root partition without risking a crash that is caused by overwriting the files that the operating system uses during the restore.

188

Disaster recovery About disk recovery procedures for UNIX and Linux

To recover the master server when the root partition is lost

1 2

Load the operating system on an alternate boot disk, using the same procedure as you would normally use for the server type. On the alternate disk, create the partition and directory where NetBackup, its catalogs (if applicable), and databases resided on the original disk. By default, they reside under the /usr/openv directory. Verify that the operating system works, that any required patches are installed, and that specific configuration settings are made. Take corrective action as needed. Install NetBackup on the alternate disk. Install only the robotic software for the devices that are required to read backups of the NetBackup catalogs and regular backups of the disk being restored. If a non-robotic drive can read these backups, no robot is required. Install any NetBackup patches that had been previously installed. See the documentation that was included with the patch software. If the catalog directories differ from those in the NetBackup catalog backups, recreate that directory structure on disk before you recover the catalog. Examples of those directories are the following:

5 6

Use of symbolic links as part of the NetBackup catalog directory structure. Use of the NetBackup nbdb_move command to relocate parts of the NetBackup relational database catalog.

If the recovery scenario involves restoring policy or catalog backups, the appropriate recovery device(s) must be configured. Device configuration may include the following tasks:

Install and configure the robotic software for the devices that read backups of the NetBackup catalog and regular backups of the disk being restored. If a non-robotic drive is available that can read these backups, then no robot is required. Although manual intervention is required if multiple pieces of media are required. See the NetBackup Device Configuration Guide. Use the NetBackup Device Configuration Wizard to discover and configure the recovery device in NetBackup. See the NetBackup Administrator's Guide, Volume I. Use the NetBackup command tpautoconf to discover and configure the recovery device in NetBackup. See the NetBackup Commands Reference Guide manual.

Disaster recovery About disk recovery procedures for UNIX and Linux

189

Update the device mapping files. See the NetBackup Administrators Guide, Volume I.

If you must restore from the policy backups or catalog backups that were done to media, the appropriate media may have to be configured in NetBackup. See the NetBackup Administrators Guide, Volume I. Configuring the media may require some or all of the following tasks:

Manually load the required media into a stand-alone recovery device. Use the NetBackup utilities such as robtest or vendor-specific robotic control software to load media into the required recovery device or devices. Use the NetBackup Volume Configuration Wizard to inventory the media contents of a robotic device. Use the vendor-specific robotic control software to load the media into the required recovery device(s).

Recover the NetBackup catalogs to the alternate disk. See About recovering the NetBackup catalog on page 207. The catalogs can be recovered only to the same directory structure from which they were backed up (alternate path recovery is not allowed).

10 Start the NetBackup Backup, Archive, and Restore interface (or the bp
command). Restore the latest backed up version of all files. You restore these files from the backup of the master server, not from the NetBackup catalog backup. Be sure to specify the disk that you recover as the alternate recovery location. Warning: Do not restore files to the /usr/openv/var, /usr/openv/db/data, or /usr/openv/volmgr/database directories (or relocated locations) or the directories that contain NetBackup database data. This data was recovered to the alternate disk in step 9 and is copied back to the recovery disk in step 12.

11 Stop all NetBackup processes that you started from NetBackup on the alternate
disk. Use the Activity Monitor in the NetBackup Administration Console or the following:
/usr/openv/netbackup/bin/bp.kill_all

190

Disaster recovery About disk recovery procedures for UNIX and Linux

12 Maintaining the same directory structure, copy the NetBackup catalogs from
the alternate disk to the disk that you recover. These are the catalogs recovered in step 9.

13 Make the recovered disk the boot disk again and restart the system. 14 Start and test the copy of NetBackup on the disk that you have recovered.
If your configuration includes an Enterprise Media Manager (EMM) server that is separate from the master server, start NetBackup on the EMM server before starting NetBackup on the master server.
/usr/openv/netbackup/bin/bp.start_all

Try the NetBackup Administration utilities. Also, try some backups and restores.

15 When you are satisfied that the recovery is complete, delete the NetBackup
files from the alternate disk. Or, unhook that disk, if it is a spare.

About recovering the NetBackup media server disk for UNIX and Linux
NetBackup 6.0 and later media servers store information in the NetBackup relational database. If you need to recover the system disk on a NetBackup media server, the recommended procedure is similar to disk recovery for the client. See Recovering the system disk on a UNIX client workstation on page 190. Note: A separate computer that functions as a NetBackup 6.0 or later media server is available only on NetBackup Enterprise Server. For NetBackup server installations, the master server and the media server are installed on the same system and have the same host name. Therefore, recovering the master server disk also recovers the media server.

Recovering the system disk on a UNIX client workstation

The following procedure recovers the client by reloading the operating system, installing NetBackup client software, and then restoring all other files. The procedure assumes that the host name does not change.

Disaster recovery About clustered NBU server recovery for UNIX and Linux

191

To recover the system disk on a client workstation

1 2 3

Install the operating system as you normally would for a client workstation of that type. Install NetBackup client software and patches. Use the NetBackup Backup, Archive, and Restore interface to select and restore user files.

About clustered NBU server recovery for UNIX and Linux

NetBackup server clusters do not protect against catalog corruption, loss of the shared disk, or loss of the whole cluster. Regular catalog backups must be performed. More information is available about configuring catalog backups and system backup policies in a clustered environment. Refer to topics about configuring NetBackup in the NetBackup High Availability Guide. Warning: Before attempting any of the recovery procedures in this topic, contact technical support.

Replacing a failed node on a UNIX or Linux cluster

Cluster technology-specific information is available about how to bring the NetBackup resource group online and offline. Also, information about how to freeze and unfreeze (that is, disable and enable monitoring for) the NetBackup Resource group. Refer to topics about configuring NetBackup in the NetBackup High Availability Guide. The following procedure applies when the shared disk and at least one configured cluster node remain available. To replace a failed node on a UNIX or Linux cluster

1 2 3

Configure the hardware, system software, and cluster environment on the replacement node. Verify that the device configuration matches that of the surviving nodes. Ensure that the NetBackup Resource group is offline on all nodes before installing NetBackup on the replacement node.

192

Disaster recovery About clustered NBU server recovery for UNIX and Linux

4 5 6

Ensure that the NetBackup shared disks are not mounted on the node on which NetBackup is to be installed. Freeze the NetBackup service. Reinstall NetBackup on the new mode or replacement node. Be sure to use the NetBackup Virtual Name as the name of the NetBackup server. Follow the instructions for installing the NetBackup server software. Refer to the NetBackup Installation Guide.

7 8 9

Install any Maintenance Packs and patches that are required to bring the newly installed node to the same patch level as the other cluster nodes. Bring the NetBackup Resource group online on a node other than the freshly installed node. Log onto the node on which the NetBackup resource group is online and run the following command:
/usr/openv/netbackup/bin/cluster/cluster_config -s nbu -o add_node -n node_name

node_name is the name of the freshly installed node.

10 Switch the NetBackup resource group to the replacement node. 11 Freeze the NetBackup group. 12 Ensure that the appropriate low-level tape device and robotic control device
configuration necessary for your operating system has been performed. Information is available for your operating system. Refer to the NetBackup Device Configuration Guide.

13 Run the Device Configuration Wizard to configure the devices. You do not
have to rerun the device configuration on the pre-existing nodes. Configuration information on your particular cluster is available. See the NetBackup Administrator's Guide, Volume I.

14 Check that the robot numbers and robot drive numbers for each robot are
consistent across all nodes of the cluster. Repeat for any other servers that are connected to that robot and correct if necessary. See the NetBackup Administrators Guide , Volume 1.

15 Test the ability of NetBackup to perform restores using the configured devices
on the replacement node.

16 Unfreeze the NetBackup resource group.

Disaster recovery About clustered NBU server recovery for UNIX and Linux

193

Recovering the shared disk on a UNIX or Linux cluster

The following procedure is applicable in situations where the configured cluster nodes remain available but files on the shared disk have been corrupted or lost. These files can include the NetBackup catalog, the database files, or both. The following conditions must be true to proceed with this procedure:

The shared storage hardware is restored to a working state, so that the shared disk resource can be brought online with an empty shared directory. Valid online catalog backups exist.

To recover the shared disk on a UNIX or Linux cluster

1 2

Clear the faulted NetBackup resource group, disable monitoring, and bring up the shared disk and virtual name resources on a functioning node. Manually create the following directories on the shared disk: <shared disk path>/netbackup/db <shared disk path>/db/data <shared disk path>/var/global <shared disk path>/volmgr/misc/robotic_db

If this server is an EMM server, enter the following to bring up the database server and EMM, then run tpext to initialize the EMM db:
# SHARED_DISK=<top-level shared disk mount point> # dataDir=${SHARED_DISK}/db/data # /usr/openv/netbackup/bin/nbdbms_start_stop start /usr/openv/db/bin/create_nbdb \ -data ${dataDir} \ -index ${dataDir} \ -tlog ${dataDir} \ -mlog ${dataDir} \ -staging ${dataDir}/staging \ -drop # /usr/openv/volmgr/bin/tpext -loadEMM

Configure required devices and media and recover the NetBackup catalogs. See Recovering the master server when root is intact on page 186.

Manually shut down and restart NetBackup on the active node.

194

Disaster recovery About disk recovery procedures for Windows

6 7

Re-enable monitoring of the NetBackup resource group. Verify that the NetBackup server can now be brought online on all configured nodes.

Recovering the entire UNIX or Linux cluster

The following procedure applies to the clustered NetBackup server environment that must be re-created from scratch. Before you proceed, ensure that you have valid online catalog backups. To recover the entire UNIX or Linux cluster

1 2

Configure the hardware, system software, and cluster environment on the replacement cluster. Ensure that the appropriate low-level tape device and robotic control device configuration necessary for your operating system has been performed. Refer to the NetBackup Device Configuration Guide.

Reinstall NetBackup on each of the cluster nodes. Be sure to use the NetBackup Virtual Name as the name of the NetBackup server. Follow the instructions for installing NetBackup server software. Refer to the NetBackup Installation Guide.

Configure the clustered NetBackup server. Refer to the NetBackup High Availability Guide.

Install any Maintenance Packs and patches that are required to bring the newly installed NetBackup server to the same patch level as the server being replaced. Configure required devices and media and recover the NetBackup catalogs. See Recovering the master server when root is intact on page 186.

Bring the NetBackup resource group on each node in turn and run the Device Configuration Wizard to configure the devices. Configuration information on your particular cluster is available. Refer to the NetBackup High Availability Guide.

About disk recovery procedures for Windows

The three different types of disk recovery for Windows are as follows:

Master server disk recovery procedures

Disaster recovery About disk recovery procedures for Windows

195

See About recovering the master server disk for Windows on page 195.

Media server disk recovery procedures See About recovering the NetBackup media server disk for Windows on page 201. Client disk recovery procedures See Recovering a Windows client disk on page 201.

The disk-based images that reside on AdvancedDisk or on OpenStorage disks cannot be recovered by means of the NetBackup catalog. These disk images must be recovered by means of the NetBackup import feature. For information on import, refer to the section on importing NetBackup images in the following manual: See NetBackup Administrators Guide, Volume I. Note: When the disk image is imported, NetBackup does not recover the original catalog entry for the image. Instead, a new catalog entry is created.

About recovering the master server disk for Windows

The procedure in this section explains how to recover data if one or more disk partitions are lost on a Windows NetBackup master server. The following two scenarios are covered:

Windows is intact and not corrupted. The system still starts Windows, but some or all other partitions are lost. NetBackup software is assumed to be lost. See Recovering the master server with Windows intact on page 195. All disk partitions are lost. Windows must be reinstalled, which is a total recovery. These procedures assume that the NetBackup master disk was running a supported version of Windows and that the defective hardware has been replaced. See Recovering the master server and Windows on page 198.

For NetBackup master and media servers, the directory locations of the NetBackup catalog become an integral part of NetBackup catalog backups. Any recovery of the NetBackup catalog requires the identical directory paths or locations be created before the catalog recovery.

Recovering the master server with Windows intact

This procedure shows how to recover the NetBackup master server with the Windows operating system intact.

196

Disaster recovery About disk recovery procedures for Windows

To recover the master server with Windows intact

1 2 3

Determine the install_path in which NetBackup is installed. By default, NetBackup is installed in the C:\Program Files\VERITAS directory. Determine if any directory paths or locations need to be created for NetBackup catalog recovery. Partition any disks being recovered as they were before the failure (if partitioning is necessary). Then reformat each partition as it was before the failure. Reinstall NetBackup software on the server. Refer to the NetBackup Installation Guide for Windows.

5 6

Install any NetBackup patches that had been previously installed. See the documentation that was included with the patch software. If the catalog directories differ from those in the NetBackup catalog backups, recreate that directory structure on disk before you recover the catalog. For example, use the NetBackup nbdb_move command to relocate parts of the NetBackup relational database catalog. If the recovery scenario involves restoring policy or catalog backups, the appropriate recovery devices must be configured. You may have to do some or all of the following:

Install and configure the robotic software for the devices that read backups of the NetBackup catalog and regular backups of the disk being restored. If a non-robotic drive is available that can read these backups, then no robot is required. Although manual intervention is required if multiple pieces of media are required. See the NetBackup Device Configuration Guide. Use the NetBackup Device Configuration Wizard to discover and configure the recovery device in NetBackup. See the NetBackup Administrator's Guide, Volume I. Use the NetBackup command tpautoconf to discover and configure the recovery device in NetBackup. See the NetBackup Commands Reference Guide manual. Update the device mapping files. See the NetBackup Administrators Guide, Volume I.

If the recovery scenario involves restoring the policy backups or catalog backups that were done to media, the appropriate recovery device(s) must be configured.

Disaster recovery About disk recovery procedures for Windows

197

Configuring the media may involve the following actions:

Manually load the required media into a stand-alone recovery device. Use NetBackup utilities such as robtest or vendor-specific robotic control software to load media into the required recovery devices. Use the NetBackup Volume Configuration Wizard to inventory the media contents of a robotic device. Use the vendor-specific robotic control software to load the media into the required recovery device(s).

Recover the NetBackup catalogs. See About recovering the NetBackup catalog on page 207.

10 When catalog recovery is complete, stop and restart the NetBackup services.
Use the following bpdown and bpup commands, the Activity Monitor in the NetBackup Administration Console, or the Services application in the Windows Control Panel.
install_path\NetBackup\bin\bpdown install_path\NetBackup\bin\bpup

Your configuration may include an EMM server that is separate from the master server. If so, start NetBackup on the EMM server before starting NetBackup on the master server. Warning: In step 11, do not restore files to the install_path\NetBackup\db, install_path\NetBackupDB, install_path\NetBackup\var, or install_path\Volmgr\database directories. The catalogs were recovered in step 9 and overwriting them with regular backups leave them in an inconsistent state. If the NetBackup relational database files were relocated using nbdb_move from install_path\NetBackupDB\data, they are recovered in step 9 and should not be restored in step 11.

11 To restore all other files, do the following actions in the order shown:

Start the NetBackup Administration interface on the master server. Start the Backup, Archive, and Restore utility. Browse for restores and select only the partitions that were lost. Select the system directory (typically C:\Winnt), which ensures that all registry files are restored.

198

Disaster recovery About disk recovery procedures for Windows

Deselect the install_path\NetBackup\db, install_path\NetBackupDB, install_path\NetBackup\var, and install_path\Volmgr\database directories (see the caution in step 10). If you reinstall Windows, select the Overwrite existing files option, which ensures that existing files are replaced with the backups. Start the restore.

12 Reboot the system, which replaces any files that were busy during the restore.
When the boot process is complete, the system is restored to the state it was in at the time of the last backup.

Recovering the master server and Windows

This procedure assumes that all disk partitions in Windows are lost. To recover the master server and Windows

Install a minimal Windows operating system (perform the Express install).

Install the same type and version of Windows software that was used previously. Install Windows in the same partition that was used before the failure. Install any required patches. Take corrective action as needed. Specify the default workgroup. Do not restore the domain. Install and configure special drivers or other software that is required to get the hardware operational (for example, a special driver for the disk drive). Install SCSI or other drivers as needed to communicate with the tape drives on the system. Follow any hardware manufacturer's instructions that apply, such as loading SSD on a Compaq system. Reboot the system when Windows installation is complete.

2 3 4 5

Determine the install_path in which NetBackup is installed. By default, NetBackup is installed in the C:\Program Files\VERITAS directory. Determine if any directory paths or locations need to be created for NetBackup catalog recovery. If necessary, partition any disks being recovered as they were before the failure. Then reformat each partition as it was before the failure. Reinstall NetBackup software on the server being recovered. Do not configure any NetBackup policies or devices at this time.

Disaster recovery About disk recovery procedures for Windows

199

6 7

Install any NetBackup patches that had been previously installed. See the documentation that was included with the patch software. If the catalog directories differ from those in the NetBackup catalog backups, recreate that directory structure on disk before you recover the catalog. For example, use the NetBackup nbdb_move command to relocate parts of the NetBackup relational database catalog. If the recovery scenario involves restoring policy or catalog backups, the appropriate recovery device or devices have to be configured. You may have to do all or some of the following tasks:

Install and configure the robotic software for the devices that read backups of the NetBackup catalog and regular backups of the disk being restored. If a non-robotic drive is available that can read these backups, then no robot is required. Although manual intervention is required if multiple pieces of media are required. See the NetBackup Device Configuration Guide. Use the NetBackup Device Configuration Wizard to discover and configure the recovery device in NetBackup. See the NetBackup Administrator's Guide, Volume I. Use the NetBackup command tpautoconf to discover and configure the recovery device in NetBackup. See the NetBackup Commands Reference Guide manual. Update the device mapping files. See the NetBackup Administrators Guide, Volume I.

If you must restore from the policy backups or catalog backups that were done to media, the appropriate media may have to be configured in NetBackup. See the NetBackup Administrators Guide, Volume I. When you configure the media, you may have to do some or all of the following:

Manually load the required media into a stand-alone recovery device. Use the NetBackup utilities such as robtest or vendor-specific robotic control software to load media into the required recovery devices. Use the NetBackup Volume Configuration Wizard to inventory the media contents of a robotic device. Use the vendor-specific robotic control software to load the media into the required recovery devices.

200

Disaster recovery About disk recovery procedures for Windows

10 Recover the NetBackup catalogs.

See About recovering the NetBackup catalog on page 207.

11 When catalog recovery is complete, stop and restart the NetBackup services.
Use the following bpdown and bpup commands, the Activity Monitor in the NetBackup Administration Console, or the Services application in the Windows Control Panel.
install_path\NetBackup\bin\bpdown install_path\NetBackup\bin\bpup

If your configuration includes an Enterprise Media Manager (EMM) server that is separate from the master server, start NetBackup on the EMM server first. Warning: In step 12, do not restore files to the install_path\NetBackup\db, install_path\NetBackupDB, install_path\NetBackup\var, or install_path\Volmgr\database directories. These directories were recovered in step 10 and overwriting them with regular backups leaves the catalogs in an inconsistent state. If the relational database files were relocated using nbdb_move from install_path\NetBackupDB\data, they are recovered in step 10 and should not be restored in step 12.

12 To restore all other files, do the following steps in the order presented:

Start the NetBackup Administration interface on the master server. Start the Backup, Archive, and Restore client interface. Browse for restores and select only the partitions that were lost. Select the system directory (typically C:\Winnt), which ensures that all registry files are restored. Deselect the install_path\NetBackup\db, install_path\NetBackupDB (or relocated NetBackup relational database path), install_path\NetBackup\var, or install_path\Volmgr\database directories. See the caution in this procedure. If you reinstall Windows, select the Overwrite existing files option, which ensures that existing files are replaced with the backups.

Disaster recovery About disk recovery procedures for Windows

201

Start the restore.

13 Restart the system, which replaces any files that were busy during the restore.
When the boot process is complete, the system is restored to the state it was in at the time of the last backup.

About recovering the NetBackup media server disk for Windows

A separate computer that functions as a NetBackup 6.0 or later media server is available only on NetBackup Enterprise Server. For NetBackup server installations, the master server and the media server are installed on the same system and have the same host name. Therefore, recovering the master server disk also recovers the media server. NetBackup media servers store their information in the NetBackup relational database. If you need to recover the system disk on a NetBackup media server, the recommended procedure is similar to disk recovery for the client. See Recovering a Windows client disk on page 201.

Recovering a Windows client disk

The following procedure explains how to perform a total recovery of a Windows NetBackup client in the event of a system disk failure. NetBackup Bare Metal Restore (BMR) protects client systems by backing them up with a policy configured for BMR protection. A complete description of BMR backup and recovery procedures is available. See the Bare Metal Restore System Administrator's Guide. This procedure assumes that the Windows operating system and NetBackup are reinstalled to boot the system and perform a restore. The following are additional assumptions:

The NetBackup client was running a supported Microsoft Windows version. The NetBackup client was backed up with a supported version of NetBackup client and server software. The NetBackup master server to which the client sent its backups is operational. You request the restore from this server. The backups included the directory where the operating system and its registry resided. If the backups excluded any files that resided in the directory, you may not be able to restore the system identically to the previous configuration. Defective hardware has been replaced.

202

Disaster recovery About disk recovery procedures for Windows

Before starting, verify that you have the following:

Windows system software to reinstall on the NetBackup client that being restored. Reinstall the same type and version of software that was previously used. NetBackup client software to install on the client that being restored. Special drivers or other software that is required to make the hardware operational (for example, a special driver for the disk drive). IP address and host name of the NetBackup client. IP address and host name of the NetBackup master server. The partitioning and formatting scheme that was used on the system to be restored. You must duplicate that scheme during Windows installation.

To recover a Windows client disk

Install a minimal Windows operating system (perform the Express install). During the installation, do the following tasks:

Partition the disk as it was before the failure (if partitioning is necessary). Then, reformat each partition as it was before the failure. Install the operating system in the same partition that was used before the failure. Specify the default workgroup. Do not restore to the domain. Follow any hardware manufacturers instructions that apply.

2 3

Reboot the system when the installation is complete. Configure the NetBackup client system to re-establish network connectivity to the NetBackup master server. For example, if your network uses DNS, the configuration on the client must use the same IP address that was used before the failure. Also, it must specify the same name server (or another name server that recognizes both the NetBackup client and master server). On the client, configure DNS in the Network dialog, accessible from the Windows Control Panel.

Install NetBackup client software. Refer to the NetBackup Installation Guide for Windows for instructions. Ensure that you specify the correct names for the client server and master server.

To specify the client name, start the Backup, Archive, and Restore interface on the client and click NetBackup Client Properties on the File menu. Enter the client name on the General tab of the NetBackup Client Properties dialog.

Disaster recovery About disk recovery procedures for Windows

203

To specify the server name, click Specify NetBackup Machines and Policy Type on the File menu.

5 6

Install any NetBackup patches that had previously been installed. Enable debug logging by creating the following debug log directories on the client:
install_path\NetBackup\Logs\tar install_path\NetBackup\Logs\bpinetd

NetBackup creates logs in these directories.

Stop and restart the NetBackup Client service. This action enables NetBackup to start logging to the bpinetd debug log.

Use the NetBackup Backup, Archive, and Restore interface to restore the system files and user files to the client system. For example, if all files are on the C drive, restoring that drive restores the entire system. To restore files, you do not need to be the administrator, but you must have restore privileges. For instructions, refer to the online Help or refer to the following: See the NetBackup Backup, Archive, and Restore Getting Started Guide. NetBackup restores the registry when it restores the Windows system files. For example, if the system files are in the C:\Winnt directory, NetBackup restores the registry when it restores that directory and its subordinate subdirectories and files.

Check for ERR or WRN messages in the log files that are in the directories you created in step 6. If the logs indicate problems with the restore of Windows system files, resolve those problems before proceeding.

10 Stop the NetBackup Client service and verify that the bpinetd program is no
longer running.

11 Restart the NetBackup client system.

When the boot process is complete, the system is restored to the state it was in at the time of the last backup.

204

Disaster recovery About clustered NBU server recovery for Windows

About clustered NBU server recovery for Windows

NetBackup server clusters do not protect against catalog corruption, loss of the shared disk, or loss of the whole cluster. Regular catalog backups must be performed. More information is available about configuring catalog backups and system backup policies in a clustered environment. Refer to topics about configuring NetBackup in the NetBackup High Availability Guide. Warning: Contact technical support before you try these recovery procedures.

Replacing a failed node on a Windows VCS cluster

Cluster technology-specific information is available about how to bring the NetBackup resource group online and offline. Also, it is available on how to freeze and unfreeze (disable and enable the monitoring for) the resource group. Refer to topics about configuring NetBackup in the NetBackup High Availability Guide. Check the following conditions before you proceed with this procedure:

The hardware, system software, and cluster environment on the replacement node have been configured. The reconfigured node or replacement node has been made a member of the cluster and has the same name as the failed node.

The following procedure applies when the shared disk and at least one configured cluster node remain available. To replace a failed node on a Windows cluster using VCS

1 2 3

Freeze the NetBackup service. Ensure that the NetBackup shared disks are not mounted on the node on which NetBackup is to be installed. Reinstall NetBackup on the new node or replacement node. Be sure to use the NetBackup Virtual Name as the name of the NetBackup server. Follow the instructions for installing the NetBackup server software. Refer to the NetBackup Installation Guide.

Ensure that the node is a member of an existing cluster and that it performs the necessary configuration automatically.

Disaster recovery About clustered NBU server recovery for Windows

205

5 6

Install any Maintenance Packs and patches that are required to bring the newly installed node to the same patch level as the other cluster nodes. Unfreeze the NetBackup service and verify that it can be brought up on the replacement node.

Recovering the shared disk on a Windows VCS cluster

The following procedure is applicable in situations where the configured cluster nodes remain available but the NetBackup catalog, database files, or both on the shared disk have been corrupted or lost. Check the following conditions before you proceed with this procedure:

The shared storage hardware is restored to a working state, so that the shared disk resource can be brought online with an empty shared directory. Valid online catalog backups exist.

To recover the shared disk on a Windows cluster that uses VCS

1 2 3

Clear the faulted NetBackup resource group, disable monitoring, and bring up the shared disk and virtual name resources on a functioning node. Ensure that all NetBackup shared disks are assigned the same drive letters that were used when NetBackup was originally installed and configured. To reconfigure NetBackup for the cluster, initialize the database by running the following commands in sequence on the active node:
bpclusterutil -ci tpext bpclusterutil -online

Use the appropriate NetBackup catalog recovery procedure to restore the NetBackup catalog information on the shared disk. See Recovering the master server and Windows on page 198.

If the clustered NetBackup server is a media server, verify that the restored vm.conf file contains the correct host-specific MM_SERVER_NAME configuration entry for the active node. If MM_SERVER_NAME is different from the local host name, edit the file and change the server name to the local host name: MM_SERVER_NAME=<local host name>

Use NetBackup to restore any data on the shared disks. Details are available on how to perform a restore. Refer to the NetBackup Backup, Archive, and Restore Getting Started Guide.

206

Disaster recovery About clustered NBU server recovery for Windows

7 8 9

Configure required devices and media and recover the NetBackup catalogs. Manually shut down and restart NetBackup on the active node. Re-enable monitoring of the NetBackup resource group. nodes.

10 Verify that the NetBackup server can now be brought online on all configured

Recovering the entire Windows VCS cluster

The following procedure applies to the clustered NetBackup server environment that must be re-created from scratch. Before you proceed, ensure that you have valid online catalog backups. To recover the entire Windows VCS cluster

1 2

Configure the hardware, system software, and cluster environment on the replacement cluster. Ensure that the appropriate low-level tape device and robotic control device configuration necessary for your operating system has been performed. Refer to the NetBackup Device Configuration Guide.

Reinstall NetBackup on each of the cluster nodes. Be sure to use the NetBackup Virtual Name as the name of the NetBackup server. Follow the instructions for installing NetBackup server software. Refer to the NetBackup Installation Guide.

Configure the clustered NetBackup server. Refer to the NetBackup High Availability Guide.

Install any Maintenance Packs and patches that are required to bring the newly installed NetBackup server to the same patch level as the server that is being replaced. Configure required devices and media and recover the NetBackup catalogs. See Recovering the master server and Windows on page 198.

Bring the NetBackup resource group on each node in turn and run the Device Configuration Wizard to configure the devices. Configuration information on your cluster (MSCS or VCS) is available. Refer to the NetBackup High Availability Guide.

Disaster recovery About recovering the NetBackup catalog

207

About recovering the NetBackup catalog

The NetBackup catalog consists of the following parts:

The image database, which contains information about the data that has been backed up. It is the largest part of the catalog. NetBackup configuration files. The databases.conf and server.conf configuration files are flat files that contain instructions for the SQL Anywhere services. NetBackup data that is stored in relational database files. The data includes media and volume data that describes media usage and volume information, which is used during the backups.

Several options to recover the catalog exist, as follows:

Recover the entire catalog Symantec recommends that you recover the entire catalog. Doing so helps ensure consistency among the various parts of the catalog. See About recovering the entire NetBackup catalog on page 209. Recover the catalog image files and configuration files Symantec recommends that you recover the catalog images files only in the following scenarios: The NetBackup relational database is valid, but NetBackup policy, backup image, or configuration files are lost or corrupt. You want to restore part of the NetBackup catalog before you restore the entire catalog. This procedure recovers only the catalog images and configuration files.

See About recovering the NetBackup catalog image files on page 220. Recover the relational database files Recover the relational database if it is corrupt or lost but the catalog image files exist and are valid. See About recovering the NetBackup relational databases on page 233.

Recovery of the entire catalog or the catalog image files relies on the disaster recovery information that is saved in a file during the online, hot catalog backup. The location of the disaster recovery file is configured in the catalog backup policy. See Recovering the NetBackup catalog without the disaster recovery file on page 244.

208

Disaster recovery About recovering the NetBackup catalog

Note: After a catalog recovery, NetBackup changes the state of the media that contains the catalog backup to frozen. This operation prevents a subsequent accidental overwrite action on the final catalog backup image on the media. This final image pertains to the actual catalog backup itself and its recovery is not part of the catalog recovery. You can unfreeze the media. Catalog recovery may be part of a larger recovery procedure. See About disk recovery procedures for UNIX and Linux on page 184. See About disk recovery procedures for Windows on page 194. Other procedures exist for special use cases. See Recovering the NetBackup catalog when NetBackup Access Control is configured on page 241. Catalog recovery may affect NetBackup OpsCenter. See About catalog recovery and OpsCenter on page 208.

About catalog recovery and OpsCenter

When the NetBackup catalog is recovered, NetBackup resets the job ID to 1. NetBackup starts assigning job numbers beginning with 1. If you use NetBackup OpsCenter to monitor NetBackup activity, you may see duplicate job IDs in OpsCenter after a catalog recovery. To prevent duplicate job IDs, you can specify the job ID number in NetBackup. Specify a number that is one higher than the highest job number in OpsCenter. See Specifying the NetBackup job ID number after a catalog recovery on page 208.

Specifying the NetBackup job ID number after a catalog recovery

You can specify the NetBackup job ID number after a catalog recovery. If you use OpsCenter to monitor NetBackup activity, doing so prevents duplicate job ID numbers in OpsCenter. See About catalog recovery and OpsCenter on page 208. To specify the NetBackup job ID number after a catalog recovery

1 2 3

If necessary, restore the OpsCenter database from a backup. Determine the last job ID number that is recorded in OpsCenter. Edit the NetBackup jobid file and set the value to one higher than the number from step 2. The following is the pathname to the jobid file:

Disaster recovery About recovering the NetBackup catalog

209

UNIX: /usr/openv/netbackup/db/jobs/jobid Windows: install_path\Veritas\NetBackup\db\jobs\jobid

Because the recovery consumes job numbers, you must specify the number before the catalog recovery.

Recover the NetBackup catalog.

About recovering the entire NetBackup catalog

Symantec recommends that you recover the entire catalog. Doing so helps ensure consistency among the various parts of the catalog. Recovery includes the catalog image files and configuration files that are in the catalog backups that are identified by the disaster recovery file, as follows:
Full backup The NetBackup relational database files identified by the DR file are restored. The images and configuration files that are identified by the disaster recovery file are restored. The NetBackup relational database files identified by the DR file are restored. All catalog backup image files back to the last full catalog backup are automatically included in an incremental catalog backup. Therefore, only catalog images and configuration files that changed since the last full backup are restored. You can then use the Backup, Archive, and Restore user interface to restore all backup images.

Incremental backup

You can use either of the following methods to recover the entire catalog:

The Catalog Recovery Wizard in the NetBackup Administration Console. See Recovering the entire NetBackup catalog using the Catalog Recovery Wizard on page 209. The text-based wizard launched by the bprecover -wizard command and option. See Recovering the entire NetBackup catalog using bprecover -wizard on page 216.

The relational database transaction log is not applied during full catalog recovery.

Recovering the entire NetBackup catalog using the Catalog Recovery Wizard
This procedure shows you how to recover the entire catalog using the Catalog Recovery Wizard. You must have root (administrative) privileges. The relational database transaction log is not applied during full catalog recovery.

210

Disaster recovery About recovering the NetBackup catalog

Note: You must have root (administrative) privileges to perform these procedures.

Note: You must be logged on to the master server on which you want to recover the catalog. You cannot change server while running the NetBackup Administration Console a a different host and then run the wizard.

Note: During the catalog recovery process, services may be shut down and restarted. If NetBackup is configured as a highly available application (cluster or global cluster), freeze the cluster before starting the recovery process to prevent a failover. Then unfreeze the cluster after the recovery process is complete.

Warning: Do not run any client backups before you recover the NetBackup catalog. To recover the entire catalog

1 2

If recovering the catalog to a new NetBackup installation, such as at a disaster recovery site, go to step 4. Your configuration may include an Enterprise Media Manager (EMM) server that is separate from the master server. If so, start NetBackup on the EMM server before starting NetBackup on the master server. Start all of the NetBackup services by entering the following: On UNIX and Linux:
/usr/openv/netbackup/bin/bp.start_all

On Windows:
install_path\NetBackup\bin\bpup

4 5 6

Start the NetBackup Administration Console. If the necessary devices are not already configured, configure them in NetBackup. Make available to NetBackup the media that contains the catalog backup.

Disaster recovery About recovering the NetBackup catalog

211

Click Recover the catalogs on the NetBackup Administration Console to start the Catalog Recovery Wizard. The Welcome panel appears.

212

Disaster recovery About recovering the NetBackup catalog

Click Next on the Welcome panel to display the Catalog Disaster Recovery File panel.

Specify where the disaster recovery file is stored by entering the fully qualified path to the disaster recovery file. In most cases, you specify the most recent disaster recovery information file available. If the most recent catalog backup is an incremental backup, use the disaster recovery file from the incremental backup. (There is no need to first restore the full backup and then follow with the incremental backup.) If some form of corruption has occurred, you may want to restore to an earlier state of the catalog.

Disaster recovery About recovering the NetBackup catalog

213

The wizard waits while NetBackup searches for the necessary media sources. The wizard then informs you if the necessary backup ID of the disaster recovery image is located. If the media is not located, the wizard lists which media is needed to update the database.

If necessary, follow the wizard instructions to insert the media that is indicated and run an inventory to update the NetBackup database. The information that is displayed on this panel depends on whether the recovery is from a full backup or an incremental backup.

214

Disaster recovery About recovering the NetBackup catalog

10 When the required media sources are all found, click Next to display the
Disaster Recovery Method panel. The Recover entire NetBackup catalog option is selected.

11 If desired, select a Job Priority then click Next to initiate the recovery of the
entire NetBackup catalog. NetBackup restores the entire NetBackup relational database, which includes the following:

NBDB database (including the EMM database) BMR database (if applicable) NetBackup policy files Backup image files Other configuration files

If the EMM server is located on a remote computer, the NBDB database is recovered on the remote computer.

Disaster recovery About recovering the NetBackup catalog

215

12 The wizard displays the recovery progress and announces when the catalog
has been recovered.

If the recovery is not successful, consult the log file messages for an indication of the problem. When the recovery job is finished, each image file is restored to the proper image directory, and the NetBackup relational databases (NBDB and optionally BMRDB) have been restored and recovered. If you recovered the catalog from removable media, NetBackup freezes the catalog media. See Unfreezing the NetBackup online catalog recovery media on page 251. NetBackup does not run scheduled backup jobs until NetBackup is stopped and restarted. You can manually submit backup jobs before you stop and restart NetBackup. Be aware that if you have not protected the media that contains the backups done after the catalog backup, the media may be overwritten. Before you restart NetBackup, protect the media that contains the backups that were successfully performed after the catalog backup that was used to recover the catalog. Click Next to continue to the final panel

13 Click Finish in the final panel

216

Disaster recovery About recovering the NetBackup catalog

14 Stop and restart NetBackup on all the servers.

If a remote EMM server is used, restart NetBackup on it before you start NetBackup on the master server. On UNIX and Linux:
/usr/openv/netbackup/bin/bp.kill_all /usr/openv/netbackup/bin/bp.start_all

On Windows:
install_path\NetBackup\bin\bpdown install_path\NetBackup\bin\bpup

15 If the catalog recovery is part of a server recovery procedure, complete the

remaining steps in the appropriate recovery procedure. Recovery can include the following:

Importing the backups from the backup media into the catalog. Write protecting the media. Ejecting the media and setting it aside. Freezing the media.

Recovering the entire NetBackup catalog using bprecover -wizard

The bprecover -wizard command is an alternate way to recover an entire catalog that is backed up using the online catalog backup method. This method does not require the NetBackup Administration Console. You must have root (administrative) privileges to perform this procedure. The relational database transaction log is not applied during full catalog recovery. Note: You must be logged on locally to the master server that being recovered.

Note: During the catalog recovery process, services may be shut down and restarted. If NetBackup is configured as a highly available application (cluster or global cluster), freeze the cluster before starting the recovery process to prevent a failover. Then unfreeze the cluster after the recovery process is complete.

Disaster recovery About recovering the NetBackup catalog

217

To recover the entire catalog using bprecover -wizard

If recovering the catalog to a new NetBackup installation, such as at a disaster recovery site, do the following:

Install NetBackup. Configure the devices that are required for the recovery. Add the media that are required for the recovery to the devices.

Start NetBackup by entering the following: If your configuration includes an Enterprise Media Manager (EMM) server that is separate from the master server, first start NetBackup on the EMM server. Then start NetBackup on the master server. UNIX and Linux: /usr/openv/netbackup/bin/bp.start_all Windows:install_path\NetBackup\bin\bpup.exe

Run the following command: UNIX: /usr/openv/netBbckup/bin/admincmd/bprecover.exe -wizard Windows: install_path\Veritas\NetBackup\bin\admincmd\bprecover.exe

-wizard

The following is displayed:

Welcome to the NetBackup Catalog Recovery Wizard! Please make sure the devices and media that contain catalog disaster recovery data are available Are you ready to continue?(Y/N)

Enter Y to continue. The following prompt appears:

Please specify the full pathname to the catalog disaster recovery file:

218

Disaster recovery About recovering the NetBackup catalog

Enter the fully qualified pathname to the disaster recovery file for the backup that you want to restore. For example:
/mnt/hdd2/netbackup/dr-file/Backup-Catalog_1318222845_FULL

If the pathname is to a valid DR file, a message similar to the following is displayed:

vm2.symantecs.org_1318222845 All media resources were located Do you want to recover the entire NetBackup catalog? (Y/N)

If the DR file or the pathname is not valid, the command-line wizard exits.

Enter Y to continue. The following is displayed:

Do you want to startup the NetBackup relational database (NBDB) after the recovery?(Y/N)

The image file is restored to the proper image directory and the NetBackup relational databases (NBDB and optionally BMRDB) are restored and recovered.

Enter Y or N to continue. The following is displayed while the restore is in progress:

Catalog recovery is in progress. Please wait... Beginning recovery of NBDB. Please wait... Completed successful recovery of NBDB on vm2.symantecs.org INF - Catalog recovery has completed. WRN - NetBackup will not run scheduled backup jobs until NetBackup is restarted. For more information, please review the log file: /usr/openv/netbackup/logs/user_ops/root/logs/Recover1318344410.log

Disaster recovery About recovering the NetBackup catalog

219

When the recovery job is finished, each image file is restored to the proper image directory, and the NetBackup relational databases (NBDB and optionally BMRDB) have been restored and recovered. If you recovered the catalog from removable media, NetBackup freezes the catalog media. See Unfreezing the NetBackup online catalog recovery media on page 251. NetBackup does not run scheduled backup jobs until NetBackup is stopped and restarted. You can manually submit backup jobs before you stop and restart NetBackup. Be aware that if you have not protected the media that contains the backups done after the catalog backup, the media may be overwritten. Before you restart NetBackup, protect the media that contains the backups that were successfully performed after the catalog backup that was used to recover the catalog.

Stop and restart NetBackup. If a remote EMM server is used, restart NetBackup on it before you start NetBackup on the master server. On UNIX and Linux:
/usr/openv/netbackup/bin/bp.kill_all /usr/openv/netbackup/bin/bp.start_all

On Windows:
install_path\NetBackup\bin\bpdown install_path\NetBackup\bin\bpup

10 If the catalog recovery is part of a server recovery procedure, complete the

remaining steps in the appropriate recovery procedure. This procedure can include the following tasks:

Importing the backups from the backup media into the catalog Write protecting the media Ejecting the media and setting it aside Freezing the media

220

Disaster recovery About recovering the NetBackup catalog

About recovering the NetBackup catalog image files

The catalog image files contain information about all the data that has been backed up. This information constitutes the largest part of the NetBackup catalog. The partial catalog recovery does the following:

Recovers the image .f files. Recovers the configuration files (databases.conf and server.conf). Restores the NetBackup relational database (NBDB) to the staging directory so that it available for further processing if required. See About processing the relational database in staging on page 240. Optionally, recovers the policy and the licensing data.

Table 5-1 is a list of the files included in a partial recovery. Note: Beginning with the NetBackup 7.5 release, the images files are now stored in the NetBackup relational database. The images files contain the metadata the describes the backups. Symantec recommends that you recover the catalog images files only in the following scenarios:

The NetBackup relational database is valid, but NetBackup policy, backup image, or configuration files are lost or corrupt. You want to restore part of the NetBackup catalog before you restore the entire catalog. This procedure recovers only the catalog images and configuration files.

Recovery includes the catalog image files and configuration files that are in the catalog backups that are identified by the disaster recovery file, as follows:
Full backup The images and configuration files that are identified by the disaster recovery file are restored.

Disaster recovery About recovering the NetBackup catalog

221

Incremental backup

Two recover scenarios exist, as follows: The catalog contains no information about the corresponding full backup and other incremental backups. NetBackup restores only the backup image .f files, configuration files, and NetBackup policy files that are backed up in that incremental backup. However, all of the catalog backup image .f files up to the last full catalog backup are restored. Therefore, you can restore the rest of the policy, image .f files, and configuration files from the Backup, Archive and Restore interface. The catalog contains information about the corresponding full backup and other incremental backups. NetBackup restores all backup image .f files and configuration files that were included in the related set of catalog backups.

Table 5-1 lists the files that comprise a partial catalog recovery. Table 5-1 UNIX and Linux
/usr/openv/netbackup/bp.conf /usr/openv/netbackup/db/* /usr/openv/netbackup/db/class/* (optional) /usr/openv/netbackup/vault/ sessions* /usr/openv/var/* (optional)

Catalog image files Windows

Not applicable install_path\NetBackup\db\* install_path\NetBackup\db\class\* (optional) install_path\NetBackup\vault\sessions\*

install_path\NetBackup\var\* (optional) install_path\Volmgr\database\* install_path\Volmgr\vm.conf

/usr/openv/volmgr/database/* /usr/openv/volmgr/vm.conf

You can use either of the following methods to recover the catalog image files:

The Catalog Recovery Wizard in the NetBackup Administration Coinsole. See Recovering the NetBackup catalog image files using the Catalog Recovery Wizard on page 222. The text-based wizard launched by the bprecover -wizard command and option.

222

Disaster recovery About recovering the NetBackup catalog

See Recovering the NetBackup catalog image file using bprecover -wizard on page 228.

Recovering the NetBackup catalog image files using the Catalog Recovery Wizard
You must have root (administrative) privileges to perform this procedure. Note: The Catalog Recovery Wizard does not work after you perform a change server operation. You must be logged on locally to the master server that being recovered.

Disaster recovery About recovering the NetBackup catalog

223

To recover the catalog image files using the Catalog Recovery Wizard

Start NetBackup by entering the following: If your configuration includes an EMM server that is separate from the master server, do the following: start NetBackup on the EMM server before starting NetBackup on the master server. On UNIX and Linux:
/usr/openv/netbackup/bin/bp.start_all

On Windows:
install_path\NetBackup\bin\bpup

Click Recover the Catalogs in the NetBackup Administration Console to start the Catalog Recovery Wizard. Warning: Do not run any client backups before you recover the NetBackup catalog.

224

Disaster recovery About recovering the NetBackup catalog

This wizard relies on the disaster recovery information that was generated during the online, hot catalog backup. Part of configuring the catalog backup included the indication of where the disaster recovery information was to be stored and sent. Indicate where the disaster recovery file is stored by entering the fully qualified path to the disaster recovery file. For example:

This wizard relies on the disaster recovery information that is saved in a file during the online, hot catalog backup. The location of the disaster recovery file was configured in the catalog backup policy. In most cases, you specify the most recent disaster recovery information file available. If some form of corruption has occurred, then you may want to restore to an earlier state of the catalog. If the most recent catalog backup was an incremental backup, use the disaster recovery file from the incremental backup. (There is no need to first restore the full backup and then follow with the incremental backup.) Specify where the disaster recovery file is stored by entering the fully qualified path to the disaster recovery file. More information is available on the email that is sent and the attached disaster recovery file.

Disaster recovery About recovering the NetBackup catalog

225

See Recovering the NetBackup catalog without the disaster recovery file on page 244.

The wizard waits while NetBackup searches for the necessary media sources, then tells you if the necessary backup ID of the disaster recovery image was located. If the media is not located, the wizard lists which media is needed to update the database.

Follow the wizard instructions to insert the indicated media and run an inventory to update the NetBackup database.

226

Disaster recovery About recovering the NetBackup catalog

Click Next to display the Disaster Recovery Method dialog. Select the Recover only NetBackup catalog image and configuration files radio option and click Next.

Disaster recovery About recovering the NetBackup catalog

227

The wizard displays the recovery progress and announces when the catalog has been recovered.

If the recovery is not successful, consult the log file messages for an indication of the problem. When the recovery job is finished, the NetBackup relational databases (NBDB and optionally BMRDB) have been restored and recovered. If you recovered the catalog from removable media, NetBackup freezes the catalog media. See Unfreezing the NetBackup online catalog recovery media on page 251. NetBackup does not run scheduled backup jobs until NetBackup is stopped and restarted. You can manually submit backup jobs before you stop and restart NetBackup. Be aware that if you have not protected the media that contains the backups done after the catalog backup, the media may be overwritten. Before you restart NetBackup, protect the media that contains the backups that were successfully performed after the catalog backup that was used to recover the catalog. Click Next to continue to the final panel

228

Disaster recovery About recovering the NetBackup catalog

Stop and restart NetBackup on all the servers. On UNIX and Linux:
/usr/openv/netbackup/bin/bp.kill_all /usr/openv/netbackup/bin/bp.start_all

On Windows:
install_path\NetBackup\bin\bpdown install_path\NetBackup\bin\bpup

If a remote EMM server is used, start NetBackup on it before you start NetBackup on the master server.

If the catalog recovery is part of a server recovery procedure, complete the remaining steps in the appropriate recovery procedure. Recovery can include the following:

Importing the backups from the backup media into the catalog. Write protecting the media. Ejecting the media and setting it aside. Freezing the media.

Recovering the NetBackup catalog image file using bprecover -wizard

You must have root (administrative) privileges to perform this procedure.

Disaster recovery About recovering the NetBackup catalog

229

To recover the catalog image file using bprecover -wizard

Start NetBackup by entering the following: If your configuration includes an EMM server separate from the master server, start NetBackup on the EMM server before starting NetBackup on the master server. On UNIX and Linux:
/usr/openv/netbackup/bin/bp.start_all

On Windows:
install_path\NetBackup\bin\bpup

Run the following command:

bprecover -wizard

The following is displayed:

Welcome to the NetBackup Catalog Recovery Wizard! Please make sure the devices and media that contain catalog disaster recovery data are available Are you ready to continue?(Y/N)

Enter Y to continue. The following prompt appears:

Please specify the full pathname to the catalog disaster recovery file:

230

Disaster recovery About recovering the NetBackup catalog

Enter the fully qualified pathname to the disaster recovery file for the backup that you want to restore. For example:
/mnt/hdd2/netbackup/dr-file/Backup-Catalog_1318222845_FULL

If you specified a DR file for a full backup, a message similar to the following is displayed:
vm2.symantecs.org_1318222845 All media resources were located Do you want to recover the entire NetBackup catalog? (Y/N)

If you specified a DR file for an incremental backup, a message similar to the following is displayed:

vm2.symantec.org_1318309224 All media resources were located The last catalog backup in the catalog disaster recovery file is an incremental. If no catalog backup images exist in the catalog, a PARTIAL catalog recovery will only restore the NetBackup catalog files backed up in that incremental backup. However, all of the catalog backup images up to the last full catalog backup are restored. Then you can restore the remaining NetBackup catalog files from the Backup, Archive, and Restore user interface. If catalog backup images already exist, all files that were included in the related set of catalog backups are restored. Do you want to recover the entire NetBackup catalog? (Y/N)

Enter N to continue. The following is displayed:

A PARTIAL catalog recovery includes the images directory containing the dotf files and staging of the NetBackup relational database (NBDB) for further processing. Do you also want to include policy data?(Y/N)

Enter Y or N to continue. The following is displayed:

Do you also want to include licensing data?(Y/N)

Disaster recovery About recovering the NetBackup catalog

231

Enter Y or N to continue. The following is displayed:

Catalog recovery is in progress. Please wait... Completed successful recovery of NBDB in staging directory on vm2.symantecs.org This portion of the catalog recovery has completed. Because this was a PARTIAL recovery of the NetBackup catalog, any remaining files included in the catalog backup can be restored using the the Backup, Archive, and Restore user interface. The image metadata that is stored in NBDB in the staging directory can be exported using "cat_export -staging", and, imported using "cat_import". The "nbdb_unload -staging" command can be used to unload one or more database tables from NBDB in the staging directory. The "nbdb_restore -recover -staging" command can be used to replace NBDB in the data directory with the contents from the staging directory. WRN - NetBackup will not run scheduled backup jobs until NetBackup is restarted. For more information, please review the log file: /usr/openv/netbackup/logs/user_ops/root/logs/Recover1318357550.log

232

Disaster recovery About recovering the NetBackup catalog

When the recovery job is finished, each image file is restored to the proper image directory and the configuration files are restored. If you chose to recover the policy data or licensing data, it is restored also. If you recovered the catalog from removable media, NetBackup freezes the catalog media. See Unfreezing the NetBackup online catalog recovery media on page 251. NetBackup does not run scheduled backup jobs until NetBackup is stopped and restarted. You can manually submit backup jobs before you stop and restart NetBackup. Be aware that if you have not protected the media that contains the backups done after the catalog backup, the media may be overwritten. Before you restart NetBackup, protect the media that contains the backups that were successfully performed after the catalog backup that was used to recover the catalog. Because this operation is a partial recovery, any remaining portions of the catalog must be restored using Backup, Archive, and Restore. You can now recover the NetBackup database if necessary.

Stop and restart NetBackup on all the servers. If a remote EMM server is used, start NetBackup on it before you start NetBackup on the master server. On UNIX and Linux:
/usr/openv/netbackup/bin/bp.kill_all /usr/openv/netbackup/bin/bp.start_all

On Windows:
install_path\NetBackup\bin\bpdown install_path\NetBackup\bin\bpup

10 If the catalog recovery is part of a server recovery procedure, complete the

remaining steps in the appropriate recovery procedure. This procedure can include the following tasks:

Importing the backups from the backup media into the catalog Write protecting the media Ejecting the media and setting it aside Freezing the media

Disaster recovery About recovering the NetBackup catalog

233

About recovering the NetBackup relational databases

You can recover the NetBackup relational databases independently of an entire catalog backup.
Recover from a backup See Recovering NetBackup relational database files from a backup on page 233. See Recovering the NetBackup NBDB relational database from staging on page 237. Several NetBackup commands let you process the NetBackup NBDB relational database in the staging directory. See About processing the relational database in staging on page 240.

Recover from the staging directory

Recovering NetBackup relational database files from a backup

If the NetBackup (NBDB) or Bare Metal Restore (BMRDB) relational database files need to be recovered, perform this procedure. The full procedure is necessary only if the NBDB database has been corrupted. The procedure includes creating a temporary database, which is required before you can recover the from the catalog backup. If the NBDB database is available and the SQL Anywhere server is running, do only steps 11 and 12. These steps replace the existing database with the copy from the catalog backup. Note: If your configuration includes a remote EMM server, perform step 1 through step 7 on the EMM server.

234

Disaster recovery About recovering the NetBackup catalog

To recover relational database files from an online catalog backup

If NetBackup is running, stop it. On UNIX and Linux:

/usr/openv/netbackup/bin/bp.kill_all

On Windows:
install_path\NetBackup\bin\bpdown

Move the following set of existing database files from their current location to a temporary directory. NBDB:
nbdb.db, nbdb.log, emm_index.db, emm_data.db

BMRDB:
bmrdb.db, bmrdb.log

Change databases.conf so SQL Anywhere does not try to automatically start them when the server is started. On UNIX and Linux:
/usr/openv/db/bin/nbdb_admin -auto_start NONE

On Windows:
install_path\NetBackup\bin\nbdb_admin -auto_start NONE

Start the SQL Anywhere server. UNIX and Linux:

/usr/openv/netbackup/bin/nbdbms_start_stop start

Windows:
install_path\NetBackup\bin\bpup -e SQLANYs_VERITAS_NB

Disaster recovery About recovering the NetBackup catalog

235

Re-create an empty database. UNIX and Linux:

/usr/openv/db/bin/create_nbdb -drop

Windows:
install_path\NetBackup\bin\create_nbdb -drop

If the database has been moved or the environment is clustered, add -staging staging_dir to the end of the create_nbdb command line. If the database has been moved or the environment is clustered, and space constraints force you to create this temporary database in the final location, use the following command: UNIX and Linux:
/usr/openv/db/bin/create_nbdb -drop -data <data_dir> -index \ <index_dir> -tlog <tlog_dir> -staging <staging_dir>

Windows:
install_path\NetBackup\bin\create_nbdb -drop -data <data_dir> -index <index_dir> -tlog <tlog_dir> -staging <staging_dir>

Where the <data_dir>, <index_dir>, <tlog_dir>, and <staging_dir> values are defined in the vxdbms.conf file as VXDBMS_NB_DATA, VXDBMS_NB_INDEX, VXDBMS_NB_TLOG, and VXDBMS_NB_STAGING.

Stop and restart NetBackup. UNIX and Linux:

/usr/openv/netbackup/bin/bp.kill_all /usr/openv/netbackup/bin/bp.start_all

Windows:
install_path\NetBackup\bin\bpdown install_path\NetBackup\bin\bpup

236

Disaster recovery About recovering the NetBackup catalog

Run tpext: UNIX and Linux:

/usr/openv/volmgr/bin/tpext

Windows:
install_path\Volmgr\bin\tpext

If you used the nbdb_move command to relocate NetBackup database files, re-create the directories where the files were located when you backed up the catalog. The default location is the following: UNIX and Linux:
/usr/openv/db/data

Windows:
install_path\NetBackupDB\data

Start the device manager: UNIX and Linux:

/usr/openv/volmgr/bin/ltid -v

Windows: start the device manager service.

10 Configure the necessary recovery device in NetBackup. 11 Make available to NetBackup the media that contains the catalog backup.
Inventory the robot or add the media for stand-alone drives.

Disaster recovery About recovering the NetBackup catalog

237

12 For online catalog recovery, run the following command on the master server:
UNIX and Linux:
/usr/openv/netbackup/bin/admincmd/bprecover -r -nbdb

Windows:
install_path\NetBackup\bin\admincmd\bprecover -r -nbdb

13 Stop and restart NetBackup.

UNIX and Linux:
/usr/openv/netbackup/bin/bp.kill_all /usr/openv/netbackup/bin/bp.start_all

Windows:
install_path\NetBackup\bin\bpdown install_path\NetBackup\bin\bpup

If a remote EMM server is used, start NetBackup on it before you start NetBackup on the master server.

Recovering the NetBackup NBDB relational database from staging

The option that recovers the image files and the configuration files restores the relational database files to the staging directory. You can recover the NetBackup NBDB relational database files from the staging directory. This procedure applies the current online transaction log during the recovery. In this case, the current online transaction log should be applied, by default, to make the database as consistent as possible with the current db/images directory. In addition to recovering NBDB from staging, NetBackup commands let you further process the NBDB. See About processing the relational database in staging on page 240. Note: If your configuration includes a remote EMM server, perform step 1 through step 7 on the EMM server.

238

Disaster recovery About recovering the NetBackup catalog

To recover relational database files from an online catalog backup

If NetBackup is running, stop it. On UNIX and Linux:

/usr/openv/netbackup/bin/bp.kill_all

On Windows:
install_path\NetBackup\bin\bpdown

Move the following set of existing database files from their current location to a temporary directory. NBDB:
nbdb.db, nbdb.log, emm_index.db, emm_data.db

Change databases.conf so SQL Anywhere does not try to automatically start them when the server is started. On UNIX and Linux:
/usr/openv/db/bin/nbdb_admin -auto_start NONE

On Windows:
install_path\NetBackup\bin\nbdb_admin -auto_start NONE

Start the SQL Anywhere server. UNIX and Linux:

/usr/openv/netbackup/bin/nbdbms_start_stop start

Windows:
install_path\NetBackup\bin\bpup -e SQLANYs_VERITAS_NB

Re-create an empty database. UNIX and Linux:

/usr/openv/db/bin/create_nbdb -drop

Windows:
install_path\NetBackup\bin\create_nbdb -drop

Disaster recovery About recovering the NetBackup catalog

239

Stop and restart NetBackup. UNIX and Linux:

/usr/openv/netbackup/bin/bp.kill_all /usr/openv/netbackup/bin/bp.start_all

Windows:
install_path\NetBackup\bin\bpdown install_path\NetBackup\bin\bpup

Run tpext: UNIX and Linux:

/usr/openv/volmgr/bin/tpext

Windows:
install_path\Volmgr\bin\tpext

If you used the nbdb_move command to relocate NetBackup database files, re-create the directories where the files were located when you backed up the catalog. The default location is the following: UNIX and Linux:
/usr/openv/db/data

Windows:
install_path\NetBackupDB\data

Start the device manager: UNIX and Linux:

/usr/openv/volmgr/bin/ltid -v

Windows: start the device manager service.

240

Disaster recovery About recovering the NetBackup catalog

10 Run the following command on the master server to recover NBDB from
staging: UNIX and Linux:
/usr/openv/netbackup/bin/admincmd/ nbdb_restore dbn NBDB recover -staging

Windows:
install_path\NetBackup\bin\admincmd\ nbdb_restore dbn NBDB recover -staging

11 Stop and restart NetBackup.

UNIX and Linux:
/usr/openv/netbackup/bin/bp.kill_all /usr/openv/netbackup/bin/bp.start_all

Windows:
install_path\NetBackup\bin\bpdown install_path\NetBackup\bin\bpup

If a remote EMM server is used, start NetBackup on it before you start NetBackup on the master server.

About processing the relational database in staging

A recovery of the NetBackup image files and configuration files also restores the NetBackup relational database (NBDB) to the staging directory. You can use the following NetBackup commands to further process the NBDB database if required:
cat_import Use cat_import to import the image metadata that is in the legacy flat file format into an NBDB relational database. The NBDB database can be the actual production DB or an NBDB in a different NetBackup domain. Use cat_export staging to extract the image metadata from the relational database. It writes the data the db.export directory in the legacy flat file format. You can export all of the image metadata or a subset of the image metadata by client or backup ID. Then, you can use the cat_import command to insert the data into another NBDB database. Another NBDB can be the actual production DB or an NBDB in a different NetBackup domain.

cat_export

Disaster recovery About recovering the NetBackup catalog

241

nbdb_restore -staging

See Recovering the NetBackup NBDB relational database from staging on page 237.

nbdb_unload -staging Use nbdb_unload staging to unload the media table and related tables to a set of flat files. Then, you can use SQL tools to insert the subset of data into another NBDB. Another NBDB can be the actual production DB or an NBDB in a different NetBackup domain.

Warning: Symantec recommends that you do not manipulate or process the NetBackup relational database unless directed to do so by a Symantec Support Representative. For help with NetBackup domain merges and splits, contact the Symantec Information Management Consulting Services. More information about the commands is available. See the NetBackup Commands Reference Guide.

Recovering the NetBackup catalog when NetBackup Access Control is configured

If you have configured NetBackup Access Control (NBAC), the online, hot catalog backup automatically backs up your authentication information and authorization configuration information. Both the Operate and Configure permission sets are required on the catalog object to successfully back up and recover NBAC authentication and authorization data. Separate recovery procedures exist based on operating system, as follows:

UNIX: See Table 5-2 on page 241. Windows: See Table 5-3 on page 242. To recover the NetBackup catalog on UNIX when NetBackup Access Control is configured Procedure

Table 5-2

Step
Step 1

Task

If recovering to a master server on which NBAC See the NetBackup Security and Encryption Guide. is configured and operational, disable NBAC (that is, set it to PROHIBITED mode). Recover the NetBackup catalog from the online See About recovering the entire NetBackup catalog backup using the Catalog Recovery Wizard catalog on page 209. or the bprecover command.

Step 2

242

Disaster recovery About recovering the NetBackup catalog

Table 5-2

To recover the NetBackup catalog on UNIX when NetBackup Access Control is configured (continued) Procedure

Step
Step 3

Task

Configure NetBackup to use NBAC by setting it to See the NetBackup Security and Encryption Guide. AUTOMATIC or REQUIRED as per the security level desired. Restart NetBackup. /usr/openv/netbackup/bin/bp.kill_all /usr/openv/netbackup/bin/bp.start_all

Step 4

Table 5-3

To recover the NetBackup catalog on Windows when NetBackup Access Control is configured Procedure

Step
Step 1

Task

If recovering to a master server on which See the NetBackup Security and Encryption Guide. NBAC is configured and operational, disable NBAC (that is, set it to PROHIBITED mode). Stop the NetBackup services. install_path\Veritas\NetBackup\bin\bpdown.exe

Step 2 Step 3

In Windows, change the start-up type of the See the Microsoft documentation. NetBackup Authentication Service and NetBackup Authorization Service to Disabled. Start the NetBackup services. Recover the NetBackup catalog from the online catalog backup using the Catalog Recovery Wizard or the bprecover command. install_path\Veritas\NetBackup\bin\bpup.exe See About recovering the entire NetBackup catalog on page 209.

Step 4 Step 5

Step 6

In Windows, change the start-up type of the See the Microsoft documentation. NetBackup Authentication Service and NetBackup Authorization Service to Automatic. Configure NetBackup to use NBAC. The procedure depends on the environment, as follows: For recovery in an existing environment, set NBAC to AUTOMATIC or REQUIRED as per the security level desired. For recovery to a new installation, configure NBAC by using the bpnbaz -setupmaster command and option.

Step 7

See the NetBackup Security and Encryption Guide.

Disaster recovery About recovering the NetBackup catalog

243

Table 5-3

To recover the NetBackup catalog on Windows when NetBackup Access Control is configured (continued) Procedure
install_path\Veritas\NetBackup\bin\bpdown.exe install_path\Veritas\NetBackup\bin\bpup.exe

Step
Step 8

Task
Restart NetBackup.

See About recovering the NetBackup catalog on page 207.

Recovering the NetBackup catalog from a nonprimary copy of a catalog backup

By default, the online, hot catalog backup can have multiple copies, and the catalog is recovered from the primary backup copy. The primary copy is the first or the original copy. However, you can recover from a copy other than the primary. Note: You must be logged on to the master server on which you want to recover the catalog. You cannot change server while running the NetBackup Administration Console a a different host and then run the wizard.

Note: You must have root (administrative) privileges to perform these procedures. To recover the catalog from a non-primary copy

If the copy of the catalog backup is on a medium other than tape, do the following:
BasicDisk Make sure that the disk that contains the backup is mounted against the correct mount path (as displayed in the disaster recovery file).

244

Disaster recovery About recovering the NetBackup catalog

Disk pool

For a catalog backup file in a disk pool, do the following: Create the disk storage server for the storage by using the Storage Server Configuration Wizard. Create the disk pool for the storage by using the Disk Pool Configuration Wizard. Run the following command to synchronize the disaster recovery file to the new disk pool. nbcatsync -sync_dr_file disaster_recovery_file

Run the following NetBackup command to recover the catalog:

bprecover wizard copy N

N is the number of the copy from which you want to recover.

Recovering the NetBackup catalog without the disaster recovery file

If the disaster recovery file has been lost, consult the email that was sent to the administrator when the catalog was backed up. The disaster recovery file is written to the location you specify in the catalog backup policy and is appended to the backup stream itself. To recover the catalog without the disaster recovery file

The email identifies the media that contains the disaster recovery file, and the media that was used to back up critical policies. Ensure that this media is available. Follow up the normal catalog recovery steps until the point where the Catalog Recovery Wizard or bprecover command is called for. Run the following command to retrieve all disaster recovery files from the catalog backup media:
bpimport -drfile media_id -drfile_dest fully_qualified_dir_name

2 3

This command recovers all disaster recovery files from the specified media ID and places them in the specified directory. The ID can be either a tape media ID or the fully qualified location of a disk storage unit.

Disaster recovery About recovering the NetBackup catalog

245

4 5

Verify that the correct disaster recovery file is available in the specified directory and that it is available from the NetBackup master server. Continue with the normal catalog recovery procedure by running the Catalog Recovery Wizard or bprecover command, providing the disaster recovery file location when prompted. Refer to the email as your primary source for recovery instructions, because they are the most current instructions for recovering your catalog. The instructions are sent when the catalog backup is completed, or when a catalog backup image is duplicated. Note: If you restore catalog files directly by using bprestore on a Solaris system, use the following path: /opt/openv/netbackup/bin/bprestore. The name of the online catalog backup policy is CatalogBackup. The email is written to the following file:
/storage/DR/CatalogBackup_1123605764_FULL.

The file name itself indicates if the backup was full or not. See NetBackup disaster recovery email example on page 245.

NetBackup disaster recovery email example

The following is an example of a disaster recovery email after a successful catalog backup:
From: Netbackup.HostName@HostName Sent: Monday, January 2, 2012 1:26 PM To: NetBackup Administrator Subject: NetBackup Catalog Backup successful on host HostName status 0 Server HostName Date Sun Jan 1 13:05:44 2012 Policy hot Catalog Backup Status

246

Disaster recovery About recovering the NetBackup catalog

the requested operation was successfully completed (status 0). DR image file: /backup/dr/hot_1305655567_FULL To ensure that the NetBackup catalog data is protected through Tue May 17 13:05:44 2011 , retain a copy of the attached file, and the media or files listed below: Catalog Recovery Media Media Server Disk Image Path * HostName /backup/nb/HostName_1305655547_C1_F1 * HostName /backup/nb/HostName_1305655567_C1_F1 * HostName /backup/nb/HostName_1305655567_C1_TIR DR file written to /backup/dr/hot_1305655567_FULL * - Primary Media

Image File Required hot_1305655547_FULL hot_1305655567_FULL hot_1305655567_FULL

Catalog Recovery Procedure for the Loss of an Entire Catalog You should create a detailed disaster recovery plan to follow should it become necessary to restore your organization's data in the event of a disaster. A checklist of required tasks can be a tremendous tool in assisting associates in triage. For example, after the facility is safe for data to be restored, the power and data infrastructure need to be verified. When these tasks are completed, the following scenarios will elp to quickly restore the NetBackup environment, and in turn, restore applications and data. Disaster Recovery Procedure using the DR Image File In the event of a catastrophic failure, use the following procedure to rebuild the previous NetBackup environment. Note: If new hardware is required, make sure that the devices contain drives capable of reading the media and that the drive controllers are capable of mounting the drives. 1. Install NetBackup. 2. Configure the devices necessary to read the media listed above.

Disaster recovery About recovering the NetBackup catalog

247

3. Inventory the media. 4. Make sure that the master server can access the attached DR image file. Start the NetBackup Recovery Wizard from the NetBackup Administration Console. Or, start the wizard from a command line by entering bprecover -wizard. Disaster Recovery Procedure without the DR Image File NOTE: ONLY attachment to recover the Shared step 2 and 1. 2. 3. 4. ATTEMPT THIS AS A LAST RESORT If you do not have the included with this email, use the following instructions your catalog (If using OpenStorage disk pools, refer to Storage Guide to configure the disk pools instead of 3 below):

Install NetBackup. Configure the devices necessary to read the media listed above. Inventory the media. Run: bpimport -create_db_info [-server name] -id /backup/nb 5. Run: cat_export -client HostName 6. Go to the following directory to find the DR image file hot_1305655567_FULL: /usr/openv/netbackup/db.export/images/HostName/1305000000 7. Open hot_1305655567_FULL file and find the BACKUP_ID (for example: HostName_1305655567). 8. Run: bpimport [-server name] -backupid HostName_1305655567 9. Run: bprestore -T -w [-L progress_log] -C HostName -t 35 -p hot -X -s 1305655567 -e 1305655567 / 10. Run the BAR user interface to restore the remaining image database if the DR image is a result of an incremental backup. 11. To recover the NetBackup relational database, run: bprecover -r -nbdb 12. Stop and Start NetBackup 13. Configure the devices if any device has changed since the last backup. 14. To make sure the volume information is updated, inventory the media to update the NetBackup database.

248

Disaster recovery About recovering the NetBackup catalog

Recovering a NetBackup user-directed online catalog from the command line

This procedure recovers the catalog manually through the command line interface (CLI) without a Phase 1 import when the disaster recovery (DR) file is available. You must have root (administrative) privileges to perform this procedure. Note: Use this procedure only if you want to restore the minimal NetBackup catalog information that lets you begin to recover critical data. To recover the user-directed online catalog from the command line interface

Verify the location of the disaster recovery files that are created from Full and Incremental Hot Catalog backups. These files can be stored in a specified path of the file system on the master server and in email attachments to the NetBackup administrator. Set up each master server and media server in the same configuration as the configuration that is used during the last catalog backup. The master server and media servers have the following same properties as the backed up catalog configuration: name, NetBackup version, operating system patch level, and path to storage devices. Configure any devices and volumes you may need for the recovery.

Locate the latest DR image file corresponding to the backup that are used for recovery. Open the file in an editor and find values for the following:
master_server Use the exact name that is specified in NetBackup configuration for the master server . The location of the robot or disk storage unit that is used for catalog backup. The four most significant digits in the DR file name and six zeroes attached. The location of the catalog backup media as specified by the disaster recovery file under the FRAGMENT keyword. Found in the DR file under BACKUP_ID.

media_server

timestamp

media

backup_id

Example: file: Hot_Backup_1122502016_INCR timestamp: 1122000000

Disaster recovery About recovering the NetBackup catalog

249

Create the DR recovery directory on the master server. UNIX and Linux:
/usr/openv/netbackup/db/images/master_server/timestamp/tmp

Windows:
C:\Program Files\VERITAS\NetBackup\db\images\master_server \timestamp\tmp

Copy the DR file to the newly created directory.

Edit the DR file in netbackup/db/images/master_server/timestamp/tmp as follows:

Change the value of IMAGE_TYPE to 1. Change the value of TIR_INFO to 0. Change the value of NUM_DR_MEDIAS to 0. Remove ALL lines containing DR_MEDIA_REC.

If your catalog recover media is on tape, run the vmquery command to assign the media to the media server.
vmquery -assigntohost media timestamp master_server

Example:
vmquery -assigntohost DL005L 1122000000 klingon

To recover the catalog .f file from the hot catalog backup, run a Phase II import on the media that is specified by the disaster recovery file .
bpimport -server master_server -backupid backup_id

If your catalog backup was incremental, recover all the other catalog backup images up to and including the most recent Full Catalog backup.

Open the Backup, Archive, and Restore client interface for NetBackup. Select NBU-Catalog as the policy type. Set the source clients and destination clients to your master server. Search the backups and restore all files that are located in the following directory:
install_path/netbackup/db/images/master_server

Verify that all files are restored successfully on the master server.

250

Disaster recovery About recovering the NetBackup catalog

Restore your critical data by using the Backup, Archive, and Restore client interface or the command line.

Restore the catalog backup images for each media server which requires data recovery. To restore the backup images, select NBU-Catalog as the policy type. Source and destination clients should be your master server. Refresh your view in the BAR GUI. Traverse the file system for the master server to the following:
install_path/netbackup/db/images

Restore the images for each configured media server. Verify that your images are present by searching for them in the catalog.

10 Recover backup data from each media server in the previous step. Change
the Policy Type, Source, and Destination client to match the client that is used to back up the desired data. Select the desired files from the Backup, Archive, and Restore client interface and restore them.

11 To recover the NetBackup relational database, run the following:

bprecover -r -nbdb

This command restores NetBackup media usage information, ensure that media containing backups are not overwritten, and restore the storage unit configuration. You cannot recover the NetBackup relational database to a configuration that is not identical to the configuration on which the catalog was backed up. Instead, you must import each piece of backup media.

12 If your catalog recovery media is on tape, freeze the media that contains the
catalog backup that is used for recovery. This action protects the media from being reused:
bpmedia -freeze -m media -h master_server

Run bpmedialist to verify that the media is frozen.

Disaster recovery About recovering the NetBackup catalog

251

13 Recover your policies and configuration data on each master server and media
server. Before recovering NetBackup policy files, ensure that you have recovered all of your critical data, or protected the media that contains your critical data. When policy information is recovered, NetBackup starts to run the scheduled jobs that may overwrite the media that was written after the last catalog backup. Open the Backup, Archive, and Restore client interface for NetBackup and select NBU-Catalog as the policy type. For each server to be restored, set the source clients and destination clients to your server, starting with the master server. Restore all files that are backed up by the hot catalog backup on each server.

14 Stop and restart the NetBackup services.

Restoring files from a NetBackup online catalog backup

Because the online catalog backup uses the standard backup format, you may recover specific files using the NetBackup Backup, Archive, and Restore user interface. Restoring catalog files directly to their original location may cause inconsistencies in the NetBackup catalog or cause NetBackup to fail. Instead, you should restore catalog files to an alternate location. To restore files from an online catalog backup

1 2 3

From the Specify NetBackup Machines and Policy Type menu, select the NBU-Catalog policy type. Specify the master server as the source client for the restore. Select the catalog files to restore.

Unfreezing the NetBackup online catalog recovery media

This procedure shows how to unfreeze your online catalog recovery media. To unfreeze the online catalog recovery media

On the master server, go to the image database. In the master server's portion of the image catalog, locate the catalog backup image file from which the recovery was done.

Identify the associated catalog backup parent image file by viewing the PARENT_IMAGE_ID value. Identify the media that the catalog backup was written to by viewing the second to last field in the DR_MEDIA_REC line(s).

252

Disaster recovery About recovering the NetBackup catalog

Save the catalog backup parent image file that was identified in the first substep. Relocate or remove all other image files that relate to the catalog backup policy.

If the NetBackup configuration includes a remote EMM server, on the master server, go to the image database for the remote EMM server. Relocate or remove any images that relate to the catalog backup policy. On the master server, for each media that is identified in step 1b, run the following command:
bpimport -create_db_info -server server_name -id media_id

On the master server, run the following command:

bpimport

On the master server, for each media that is identified in step 1b, run the following command:
bpmedia -unfreeze -m media_id -h server_name

Appendix

Backup and restore functional overview

This appendix includes the following topics:

About backup and restore functional overview Backup and restore startup process Backup and archive processes Backups and archives - UNIX clients About UNIX and Linux client restoration About SAN client (UNIX or Windows) restoration About Windows client restoration About NetWare client restoration About catalog backup restoration NetBackup directories and files NetBackup programs and daemons NetBackup catalogs

About backup and restore functional overview

This appendix provides a functional overview of NetBackup backup and restore operations for both UNIX and Windows. The discussions include descriptions of important services or daemons and programs, and the sequence in which they

254

Backup and restore functional overview Backup and restore startup process

execute during backup and restore operations. The databases and the directory structure of the installed software are also described. Note that this appendix does not describe the NetBackup products for backing up relational databases (such as NetBackup for ORACLE). The guides for those products have information regarding their operation.

Backup and restore startup process

When the NetBackup master server starts up, a script automatically starts all services, daemons, and programs that are required by NetBackup. (The start-up commands that are used by the script vary according to the platform.) The same is true on a media server. NetBackup automatically starts additional programs as required, including robotic daemons. For more information, see the following topic: Information is available on SAN client and Fibre Transport startup processes. See the NetBackup Administrators Guide for UNIX and Linux, Volume II, or See the NetBackup Administrators Guide for Windows, Volume II Note: No daemons or programs need to be explicitly started. The necessary programs are started automatically during the backup or restore operation. A daemon that executes on all servers and clients is the NetBackup client daemon, bpcd. On UNIX clients, inetd starts bpcd automatically so no special actions are required. On Windows clients, bpinetd performs the same functions as inetd. Netware clients do not use inetd or bpinetd but are configured to start the bpcd NLM (bpcd.nlm) automatically. An NLM is similar to a service; NLM stands for NetWare Loadable Module. Note that all NetBackup processes can be started manually by running the following:
/usr/openv/netbackup/bin/bp.start_all

Backup and archive processes

The backup processes and archive processes vary depending on the type of client. The following explains the variations and describes the synthetic backup process. A description is included about how NetBackup operates when backing up its catalogs.

Backup and restore functional overview Backups and archives - UNIX clients

255

Job scheduling
The scheduler process bpsched consists of the following services:

The nbpem service (Policy Execution Manager) does the following: creates policy/client tasks and determines when jobs are due to run. It starts the job and upon job completion, determines when the next job should run for the policy-client combination. The nbjm service (Job Manager) accepts requests from nbpem to run backup jobs, or to run media jobs from commands such as bplabel and tpreq. nbjm acquires resources for each job, such as storage unit, drives, media, and client and policy resources, and executes the job. The nbrb service (Resource Broker) allocates resources in response to requests from nbjm. nbrb acquires physical resources from nbemm (the Enterprise Media Manager service). It also manages logical resources such as multiplex groups, maximum jobs per client, and maximum jobs per policy. nbrb is also responsible for initiating drive unloads and manages pending request queues.

EMM server and master server

The NetBackup master server and the Enterprise Media Manager (EMM) server can be on the same physical host or on different hosts. The master server is responsible for running jobs as configured in NetBackup policies. The nbpem and nbjm services run only on the master server. The EMM server allocates resources for one or more master servers. The EMM server is the repository for all device configuration information. The nbemm service and the nbrb service run only on the EMM server. The nbemm service centralizes resource selection and maintains devices, media, and storage units in a relational database.

Backups and archives - UNIX clients

For UNIX clients, NetBackup supports scheduled, immediate manual, and user-directed backups of both files and raw partitions. User-directed archives of files are also supported (you cannot archive raw partitions). When the operations start, they are all similar to the extent that the same daemons and programs execute on the server. Each type of backup is started differently as follows:

256

Backup and restore functional overview Backups and archives - UNIX clients

Scheduled backups begin when the nbpem service detects that a job is due. nbpem checks the policy configurations for the scheduled client backups that are due. Immediate manual backups begin if the administrator chooses this option in the NetBackup Administration Console or runs the bpbackup command with the i option. This action causes bprd to contact nbpem, which then processes the policy, client, and schedule that are selected by the administrator. User-directed backups or archives begin when a user on a client starts a backup or archive through the user interface on the client. The user can also enter the bpbackup or bparchive commands on the command line. This action invokes the clients bpbackup or bparchive program, which sends a request to the request daemon bprd on the master server. When bprd receives the user request, it contacts nbpem, which checks the policy configurations for schedules. By default nbpem chooses the first user-directed schedule that it finds in a policy that includes the requesting client. For user-directed backups or archives, it is also possible to specify a policy and schedule. A description is available of the UNIX BPBACKUP_POLICY and BPBACKUP_SCHED options in bp.conf and the Windows equivalents. See the NetBackup Administrators Guide, Volume I I.

Backup process
This topic uses a diagram and a table to describe each step of a backup process. PBX (not shown in the diagram) must be running for NetBackup to operate. See Resolving PBX problems on page 76. Figure A-1 illustrates the various operations that comprise the backup process.

Backup and restore functional overview Backups and archives - UNIX clients

257

Figure A-1

Backup or archive to tape or disk

Master server
Configuration Database File Database NetBackup Policy Management bprd EMM Database

UNIX client

NetBackup user interface

Command line

bpdbm

nbproxy

nbpem Bpbackup or bparchive

nbproxy

Mo

t un

re

t es qu

nbemm
Ca

nbrb

nbjm

tal

og

nbrmms

Master or media server

bptm (parent) Tape request
Ba ck

Inf

bpcd

up

ck

Ba

Mount Disk volume

ag

bpbrm
up
Shar e mem d ory

Cat

bpcd
alo g In fo

Im

Im

ag e

bptm (child)**

bpbkar Backup Image

Tape Mount

ltid*

Notes: * For details on these components, see the Media and Device Management Functional Description later in this chapter. Itid is for tape backup only . ** If the media server is backing up itself (server and client on same host), there is no bptm child: bpbkar sends the data directly to shared memory.

Client disk

Table A-1 shows the sequence of operation of a backup process. Table A-1 Agent
Start-up script

Backup to tape or disk sequence of operation Action

Launches bprd on the master server and ltid on the master server and all media servers. All other daemons and programs are started as necessary including nbpem, nbjm, nbrb, and nbemm.

258

Backup and restore functional overview Backups and archives - UNIX clients

Table A-1 Agent

Policy execution manager service (nbpem)

Backup to tape or disk sequence of operation (continued) Action

Gets the policy list from bpdbm. Creates a policy-client task for all policy-client combinations specified in the policy list. Computes the due time for each policy-client task (policy priority is honored for internal processing). Submits to nbjm all jobs as policy-client tasks become due. When a job finishes, it recomputes the due time of that policy-client task.

Job manager service (nbjm)

Issues a single request (with a request ID) to nbrb, for all resources that are required by a job. nbrb gets the storage unit, tape drive, and media id information from nbemm and allocates client and policy resources. nbrb returns to nbjm an allocation sequence that contains one allocation for each resource (each allocation contains a unique ID). nbrb also returns allocation data for the specific resource type. nbrb also returns the request ID along with the allocations so that nbjm can correlate the response with the right request (and job). Note that nbrb allocates all resources that are included in a request. If the resources are temporarily unavailable the request is queued in nbrb. If the resource cannot be allocated, nbrb fails the request. nbjm starts the backup by using the client daemon bpcd to start the backup and restore manager bpbrm. For normal backup (not snapshots), nbjm starts bpbrm on the media server, which may or may not be the same system as the master server.

Backup and restore manager (bpbrm)

Starts bptm. Starts the actual backup (or archive) by using the client daemon bpcd to start the backup program and archive program bpbkar on the client.

Backup and restore functional overview Backups and archives - UNIX clients

259

Table A-1 Agent

Backup to tape or disk sequence of operation (continued) Action

Sends the information about files within the image to bpbrm, which directs the file information to the NetBackup file database. The information is sent by means of bpdbm on the master server. Transmits the backup image to bptm depending on one of the following: whether the media server backs up itself (bptm and bpbkar are on the same host) or back ups a client that resides on a different host. If the media server backs up itself, bpbkar stores the image block-by-block in shared memory on the media server. If the media server backs up a client on a different host, the bptm process on the server creates a child process of itself. The child receives the image from the client by means of socket communications and then stores the image block-by-block in shared memory on the server. Use the NOSHM file to force a media server that backs up itself to do the following: create a child process and use socket communications, as though the client is on a different host. More information on the NOSHM file is available. See the NetBackup Backup Planning and Performance Tuning Guide.

Backup and archive manager (bpbkar)

260

Backup and restore functional overview Backups and archives - UNIX clients

Table A-1 Agent

Backup to tape or disk sequence of operation (continued) Action

The bptm process on the server takes the image from shared memory and directs it to the storage media, which can be disk, tape, or both disk and tape. The bpdm process manages and cleans up the disk images. If the storage media is tape, bptm requests information for the first media and drive to use, by exchanging information with nbjm. bptm sends mount requests for specific media and drives to the NetBackup Device Manager (ltid). This action causes the media to be mounted on the appropriate devices. If, during the backup, a tape span is required, bptm again exchanges information with nbjm to release the correct tape and to get another one. nbjm exchanges information with nbrb to accomplish this function. For AdvancedDisk and OpenStorage, bptm requests the volume from nbjm. nbjm then passes the request to nbemm to choose the volume server and media server to use. nbemm calls nbrmms on the media server that was chosen to mount the volume. If, during the backup, a tape span is required, bptm again exchanges information with nbjm to release the correct tape and to get another one. nbjm exchanges information with nbrb to accomplish this function. For BasicDisk, bptm writes the images to the path that is configured in the disk storage unit. The system disk manager controls the actual writing of data. In the case of an archive, bpbrm deletes the files from the client disk after the files are successfully backed up.

backup manager for tape (bptm) or disk (bpdm)

Job manager service (nbjm)

Receives the completion status of the job from bpbrm. Releases the resources to nbrb and returns the status to nbpem.

Backup with multiple data streams

For multiplexed backups, the process is essentially the same. An exception is that a separate bpbrm process and bptm process is created for each backup image being multiplexed onto the media. NetBackup also allocates a separate set of shared memory blocks for each image.

Backup and restore functional overview Backups and archives - UNIX clients

261

The other client and server processes are the same as shown in Figure A-1. Figure A-2 shows multiplexed images from two clients. Figure A-2 Multiplexed backups example (two streams)

NetBackup server
Only on maser server bpdbm bprm (parent)

UNIX client
bpcd bprm (child)
File in

File info

fo

bpbkar See Figure A-1 for process details leading to nbemm. nbemm Mount request nbrmms bptm (parent)
Ba ck

bptm (child)**
S m har em ed or y

Bac

ku

age p Im

Fi le inf o

Client disk

UNIX client

Im a

bprm (child)
up

ge

bpcd

up

ck

Ba

Mount Disk volume

Sha mem red ory

Im ag e

File
bptm (child)**

ltid*

Notes: * For details on these components, see the Media and Device Management Functional Description later in this chapter. . ** If the server is backing up itself (server and client on same host), there is no bptm child: bpbkar sends the data directly to shared memory.

Snapshot backup and Windows open file backups

Figure A-3 shows the overall snapshot backup process. PBX (not shown in the diagram) must be running for NetBackup to operate.

Tape request

info
bpbkar

Backup Image

Tape Mount

Client disk

262

Backup and restore functional overview Backups and archives - UNIX clients

Figure A-3

Snapshot backup and Windows open file backup using multiple data streams

Master server
Configuration Database File Database Backup Policy Management bprd EMM Database

UNIX client
NetBackup user interface or command line

bpdbm

bpbackup or bparchive nbproxy nbpem bpfis Create Snapshot

nbproxy

Mo

t un

st ue eq

nbemm
Ca

nbrb

nbjm

bpcd

tal

og

nbrmms

Master or media server

bptm (parent)

Inf

bpbrm bpcd

Client disk

bpcd bpbrm
Sha r mem ed ory
ag e

ag

up

ck

Ba

Mount Disk volume

Ba

Cat

Tape request

Im

ck

alog

up

Im

bptm (child)**

Info
bpbkar

Backup Image

Tape Mount

ltid*

Notes: * For details on these components, see the Media and Device Management Functional Description later in this chapter. . ** If the media server is backing up itself (server and client on same host), there is no bptm child: bpbkar sends the data directly to shared memory.

A separate parent job creates all snapshots followed by a child job that backs up the snapshot. An exception is when Windows opens file backups that do not use multiple data streams. The following sequence of operation is for snapshot creation and backup that includes Windows open file backups that employ multiple data streams:

The NetBackup master server or primary client initiates the backup. This action causes the NetBackup request daemon bprd to submit a backup request

Backup and restore functional overview Backups and archives - UNIX clients

263

to the Policy Execution Manager nbpem. nbpem processes the policy configurations.

nbpem (through nbjm) starts a parent job to create the snapshot. This job is

separate from the job that backs up the snapshot.

nbjm starts an instance of bpbrm through bpcd on the media server, and bpbrm

starts bpfis through bpcd on the client.

bpfis creates a snapshot of the clients data by means of a snapshot method.

When bpfis is finished, it sends snapshot information and completion status to bpbrm and exits. bpbrm, in turn, reports the snapshot information and status to nbjm and exits. nbjm relays the information and status to nbpem.
nbpem submits a child job for the backup to nbjm, with a file list derived from

the snapshot information. nbjm starts bpbrm to back up the snapshot.

bpbrm starts bpbkar on the client. bpbkar sends the file catalog information

to bpbrm, which relays it to the NetBackup file database bpdbm on the master server.

bpbrm starts the process bptm (parent) on the media server.

The next step depends on whether the media server backs up itself (bptm and bpbkar are on the same host) or the media server backs up a client that resides on a different host. If the media server backs up itself, bpbkar stores the snapshot-based image block by block in shared memory on the media server. If the media server backs up a client that resides on a different host, bptm on the server creates a child process of itself. The child receives the snapshot-based image from the client by means of socket communications and then stores the image block-by-block in shared memory. The original bptm process then takes the backup image from shared memory and sends it to the storage device (disk or tape). Information is available on how the tape request is issued. See Media and device management process on page 307.
bptm sends backup completion status to bpbrm, which passes it to nbjm.

When nbpem receives backup completion status from nbjm, nbpem tells nbjm to delete the snapshot. nbjm starts a new instance of bpbrm on the media server, and bpbrm starts a new instance of bpfis on the client. bpfis deletes the snapshot on the client, unless the snapshot is of the Instant Recovery type, in which case it is not automatically deleted. bpfis and bpbrm report their status and exit. For more information on snapshot backups involving Snapshot Client, refer to the following:

264

Backup and restore functional overview Backups and archives - UNIX clients

See the NetBackup Snapshot Client Administrators Guide. Note that Windows open file backups do not require Snapshot Client.

SAN client
For backups to disk, the SAN Client feature provides high speed data movement between NetBackup media servers and NetBackup SAN-attached clients. SAN-attached clients send backup data to the media server by means of fibre channel connections. As part of SAN Client, the FT Service Manager (FSM) is a domain layer service that resides on the EMM server. The FSM provides discovery, configuration, and event monitoring of SAN Client resources. The FSM collects fibre channel information from the client and from the media server; FSM then populates the EMM database with the information. (FSM runs in the same process as EMM.) FSM interacts with the nbftclnt process on NetBackup clients and with the nbftsrvr process on media servers. The initial stages of a backup are the same as shown in Figure A-1 Figure A-4 shows the server and client components that are unique to SAN client backup over Fibre Channel.

Backup and restore functional overview Backups and archives - UNIX clients

265

Figure A-4

SAN client backup over Fibre Transport

EMM Server

nbjm

FT Service Manager (FSM - Part of EMM)

Master or Media Server

bpcd

UNIX Client bpcd

Bptm (parent)
Backup Im age

bpbrm

catalog Info
bpbkar

Shared Memory

Shared Memory
Client disk nbftsrvr Backup Image sent over Fiber Channel nbftclnt

Storage Disk

The process flow for a SAN Client backup is as follows (in the order presented):

A start-up script launches bprd on the master server and ltid on the master server and all media servers. All other daemons and programs are started as necessary including nbpem, nbjm, nbrb, and nbemm. The policy execution manager service (nbpem) does the following:

Gets the policy list from bpdbm. Builds a work list of all scheduled jobs. Computes the due time for each job. Sorts the work list in order of due time. Submits to nbjm all jobs that are currently due. Sets a wakeup timer for the next due job.

266

Backup and restore functional overview Backups and archives - UNIX clients

When the job finishes, re-computes the due time of the next job and submits to nbjm all jobs that are currently due.

The job manager service (nbjm) requests backup resources from the resource broker (nbrb). nbrb returns information on the use of shared memory for SAN Client. nbjm starts the backup by means of the client daemon bpcd, which starts the backup and restore manager bpbrm. bpbrm starts bptm. bptm does the following:

Requests SAN Client information from nbjm. Sends a backup request to the FT server process (nbftsrvr). Sends a backup request to the FT Client process on the client (nbftclnt). nbftclnt opens a fibre channel connection to nbftsrvr on the media server, allocates shared memory, and writes shared memory information to the backup ID file.

bpbrm starts bpbkar by means of bpcd. bpbkar does the following:

Reads the shared memory information from the BID file (waits for the file to exist and become valid). Sends the information about files in the image to bpbrm. Writes the file data to tar, optionally compresses it, and writes the data to the shared buffer. When the buffer is full or the job is done, sets buffer flag.

The FT Client process nbftclnt waits for the shared memory buffer flag to be set. nbftclnt then transfers the image data to the FT Server (nbftsrvr) shared memory buffer, and clears the buffer flag. nbftsrvr waits for data from nbftclnt; the data is written to the shared memory buffer. When the transfer completes, nbftsrvr sets the buffer flag. bptm waits for the shared memory buffer flag to be set, writes data from the buffer to the storage device, and clears the buffer flag. At the end of the job:

bpbkar informs bpbrm and bptm that the job is complete. bptm sends bpbrm the final status of the data write. bptm directs nbftclnt to close the fibre channel connection. nbftclnt closes the fibre channel connection and deletes the BID file.

Backup and restore functional overview Backups and archives - UNIX clients

267

Backups and archives - Windows

NetBackup supports the same types of operations on Windows clients as it does on UNIX clients. Figure A-5 shows the Windows client processes. In this figure, the following items applies:

NBWIN is the user interface program on the client. The bpbackup function and

the bparchive function are merged into NBWIN.

BPINETD serves the same purpose as inetd on UNIX clients.

The NetBackup client daemon is called BPCD.

BPBKAR32 serves the same purpose as bpbkar on UNIX clients.

The server processes are the same as described for UNIX. Figure A-5 Backup and archive - Windows clients

Server For details on the server processes, see Backups and Archives - UNIX Clients earlier in this chapter.

Windows 2000 Client NetBackup User Interface

bprd

request

NBWIN

bpbrm
File Inf

BPINETD

bptm
Backup Image

om

atio

BPCD
n

BPBKAR32

Client Disk

268

Backup and restore functional overview Backups and archives - UNIX clients

Backups and archives - NetWare clients

NetBackup supports the same types of operations on NetWare clients as it does on UNIX clients, with the following exceptions:

Raw partition backups are not supported. NetBackup for NetWare does not support archiving.

Figure A-6 shows the NetWare client processes. In this figure, the following item applies:

For NetWare nontarget operations, the Windows-based user interface program is called NBNWNT. For NetWare target operations, the user interface program is called BP.NLM on the Netware console. The bpbackup, bparchive, and bplist functions are merged into the user interface programs on the clients. The NetBackup NetWare client daemon is called BPCD. The bpbkar functions are merged into BPCD.

The server processes are the same as described for UNIX.

Backup and restore functional overview Backups and archives - UNIX clients

269

Figure A-6

Backup and archive -- NetWare clients

Server
For details on the server processes, see Backups and Archives UNIX Clients earlier in this chapter.

NetWare client

NetBackup user interface

bprd

Request

NBNWNT (NetWare nontarget) BP (NetWare target)

bpbrm
Ima ge

File information

bpcd

up ack

bptm Client disk

.

Synthetic backups
The typical NetBackup backup process accesses the client to create a backup. A synthetic backup is a backup image created without using the client. Instead, a synthetic backup process creates a full or a cumulative incremental image by using only previously created backup images, called component images. Note: Synthetic archives do not exist. For example, an existing full image and subsequent differential incremental images may be synthesized to create a new full image. The previous full image and the incrementals are the component images. The new synthetic full image behaves like a backup that is created through the traditional process. The new

270

Backup and restore functional overview Backups and archives - UNIX clients

synthetic full image is a backup of the client that is as current as the last incremental. The synthetic image is created by copying the most current version of each file from the most recent component image that contain the file. A synthetic backup must be created in a policy with the True Image Restore with Move Detection option selected. This option enables the synthetic backup to exclude the files that have been deleted from the client file system from appearing in the synthetic backup. Like a traditional backup, nbpem typically initiates a synthetic backup. nbpem submits a request to nbjm to start the synthetic backup job. nbjm starts bpsynth. bpsynth executes on the master server. It controls the creation of the synthetic backup image and the reading of the files that are needed from the component images. If directory bpsynth exists in the debug log directory, additional debug log messages are written to a log file in that directory.
bpsynth makes a synthetic image in several phases:

Phase 1 - Prepare catalog information and extents

In phase 1, bpsynth makes a synthetic backup request to the database manager, bpdbm. bpdbm uses the entries and the TIR information from the catalogs of the component images to build the catalog for the new synthetic image. It also builds the extents to be copied from the component images to the synthetic image. Bpdbm returns the list of extents to bpsynth. (An extent is the starting block number and the number of contiguous blocks within a specific component image.) A set of extents must usually be copied from each component image onto the new synthetic image. Figure A-7 shows how phase 1 operates. Figure A-7
nbpem

Synthetic backup -- preparation phase

nbjm Request to make synthetic backup bpsynth Extents and media needed to form the synthetic backup bpdbm Catalog

Backup and restore functional overview Backups and archives - UNIX clients

271

Phase 2 - Obtain resources

In phase 2, bpsynth obtains write resources (storage unit, drive, and media) for the new image. It also reserves all the read media containing component images and obtains the drive for the first media to be read. When the component images reside on BasicDisk, no resource reservation is done.

Phase 3 - Copy data

In phase 3, bpsynth starts the writer bptm (for tape and disk) on the media server to write the new synthetic image. bpsynth starts a reader bptm (tape) or bpdm (disk) process for each component image on a media server that can access the component image. The reader process reads all extents for the component image. Figure A-8 illustrates how phase 3 operates. Figure A-8
Master server

Synthetic backup -- copy data phase

Media server parent bptm child bptm new image

bpsynth

data flow

child bptm or bpdm parent bptm or bpdm component image(s)

Note that bpsynth only starts the parent bptm (writer) and bpdm (reader) process on the media server. The parent in turn starts a child process. The parent and child communicate by means of buffers in shared memory. The bpsynth process sends the extents (starting block and count) for each component image to the corresponding child bptm or bpdm reader process. The parent bptm or bpdm reader process reads the data from the appropriate media into the shared buffers. The child bptm or bpdm reader process sends the data in

272

Backup and restore functional overview Backups and archives - UNIX clients

the shared buffers to the child bptm writer process over a socket. The child bptm writer process writes the data into the shared buffers. The parent bptm writer process copies the data from the shared buffers to the media and notifies bpsynth when the synthetic image is complete.

Phase 4 - Validate the image

In phase 4, the bpsynth process validates the image. The new image is now visible to NetBackup and can be used like any other full or cumulative incremental backup. Synthetic backup requires the following:

That True Image Restore (TIR) with move detection be selected for each component image. That the component images are synthetic images.

NetBackup online, hot catalog backup

Online, hot catalog backup. This type of catalog backup is policy-based, with all of the scheduling flexibility of a regular backup policy. This backup type is designed for highly active NetBackup environments where other backup activity usually takes place. The catalog backup is performed online, meaning that the catalog is not turned off. More details are available. See Hot catalog backup process on page 272. You can use an option in the Administration Console to start a manual backup of the NetBackup catalogs. Or, you can configure a NetBackup policy to automatically back up its catalogs.

Hot catalog backup process

Figure A-9 shows the hot catalog backup that is followed by the backup process itself.

Backup and restore functional overview Backups and archives - UNIX clients

273

Figure A-9
Master Server

Hot catalog backup process

Backup Policy Management 1 bpbackup bprd Command line

nbpem 2 nbjm

Back up Relational Database Files

bpdbm

Back up NetBackup Database Files bprd

Relational database files

Sybase ASA database agent 3a 3b

/usr/openv/db/s taging

bprd

See Backup to tape or disk on page591. Note: the master server backs up itself.

See Backup to tape or disk on page591. Note: the master server backs up the EMM server.

A hot catalog backup consists of the following jobs that run on the master server:

A parent job that is started manually by the administrator or by a catalog backup policy schedule. A child job that backs up the NetBackup relational database files. A child job that copies the NetBackup database files on pre-6.0 media servers, if any. A child job that backs up the NetBackup database files (all files in /usr/openv/netbackup/db).

A hot catalog backup process is as follows (in the order presented):

A manual backup or a catalog backup policy initiates the backup.

274

Backup and restore functional overview About UNIX and Linux client restoration

nbpem submits a parent job to nbjm; nbjm sends a request to bpdbm. bpdbm handles the backup of the relational database files, in two steps:

The SQL Anywhere files database agent makes an online copy of the relational database files to /usr/openv/db/staging. See the Disaster Recovery chapter for a list of the relational database files. After the files are in the staging area, the SQL Anywhere database agent backs them up in the same manner as is used for an ordinary backup.

NetBackup backs up the database files that are in /usr/openv/netbackup/db and important NetBackup files to the master server. NetBackup creates the disaster recovery file, and emails it to the administrator if the email option was selected in the policy.

Consult the following logs for messages on hot catalog backup:

bpdbm, bpbkar, bpbrm, bpcd, bpbackup, bprd

Note: If the EMM server is on its own host (separate from the master server), consult this log on the EMM server: /usr/openv/netbackup/logs/admin (UNIX), or install_path\NetBackup\logs\admin (Windows). For messages pertaining only to the relational database files, see the progress log file in the following directory:

/usr/openv/netbackup/logs/user_ops/dbext/logs (UNIX) install_path\NetBackup\logs\user_ops\dbext\logs (Windows)

About UNIX and Linux client restoration

Before starting a restore, a user browses the file catalog to list the files available in the backup images. The desired files can then be selected from the list. The browsing is done through the bplist program on the client. The bplist program can be started directly from the command line and the NetBackup user interface programs can use it.
bplist obtains the file list by sending a query to the request daemon, bprd, on

the master server (see Figure A-10). The request daemon, in turn, queries bpdbm for the information and transmits it to bplist on the client.

Backup and restore functional overview About UNIX and Linux client restoration

275

Figure A-10

List operation - UNIX and Linux client

Master Server File Database

UNIX Client

NetBackup User Interface

Command line

query
bpdbm

File list

bprd

File list
bplist

Refer to one of the following topics as you read through the restore process. See Figure A-11 on page 277. See Figure A-12 on page 278. The following are the processing steps in a restore (in the order presented):

When the user starts a restore, NetBackup invokes the clients bprestore program which sends a request to the request daemon, bprd. This request identifies the files and client. The request daemon then uses bpcd (client daemon) to start the backup and restore manager (bpbrm). Note: To restore Backup Exec images, bpbrm initiates mtfrd instead of tar on the clients. The server processes are the same as those used for NetBackup restores. If the disk device or tape device on which the data resides attaches to the master server, the following occurs: bprd starts the backup and restore manager on the master server. If the disk unit or tape unit connects to a media server, bprd starts the backup and restore manager on the media server. The backup and restore manager starts bptm and uses the client daemon (bpcd) to establish a connection between the NetBackup tar program on the client and bptm on the server. The bptm process identifies which media (disk or tape) is needed for the restore, based on the image catalog. bptm then requests the allocation of the required

276

Backup and restore functional overview About UNIX and Linux client restoration

media from nbrb through nbjm. nbjm then asks mds (part of nbemm)for the resources. nbemm allocates the media and selects and allocates an appropriate drive (for tape media). For tape: bptm asks ltid to mount the tape in the drive. For disk: (such as AdvancedDisk or OpenStorage), nbrb tells nbemm to issue the mount by means of nbrmms, after nbemm allocates the resources. For restore from non-shared disk (BasicDisk, PureDisk, SnapVault), bptm does not need to ask nbrb for an allocation, because disk inherently supports concurrent access. bptm uses the file path in a read request to the system disk manager.

When the allocation is granted to it, bptm starts retrieving data. bptm stores the image block-by-block in shared memory.
bptm directs the image to the client in one of two ways. If the server restores

itself (server and client are on the same host), tar reads the data directly from shared memory. If the server restores a client that resides on a different host, it creates a child bptm process which transmits the data to tar on the client. Note: Only the part of the image that is required to satisfy the restore request is sent to the client, not necessarily the entire backup image. The NetBackup tar program writes the data on the client disk.

PBX must be running for NetBackup to operate (PBX is not shown in the next diagram). See Resolving PBX problems on page 76. Figure A-11 shows how to restore from tape in the UNIX and Linux environments:

Backup and restore functional overview About UNIX and Linux client restoration

277

Figure A-11

Restore from tape (UNIX and Linux)

Master server

UNIX client

NetBackup user interface

Command line

EMM Database bprd bprestore

nbemm

nbrb

nbjm

Master or media server

bptm (parent) Tape request ltid*
Ba

bpcd

bpbrm
ck

bpcd

up

Shar e mem d ory

Im ag e

bptm (child)**

Backup Image

NetBackup tar

Tape

Mount

Notes: * For details on this component, see the Media and Device Management Functional Description later in this chapter. Itid is for tape backup only . ** If the media server is restoring its own data (server and client on same host), there is no bptm child: tar reads the data directly from shared memory.

Client disk

Figure A-12 shows how to restore from disk in the UNIX and Linux environments:

278

Backup and restore functional overview About SAN client (UNIX or Windows) restoration

Figure A-12

Restore from disk (UNIX and Linux)

Master server UNIX client

EMM Database

NetBackup user interface

Command line

nbemm

nbrb

nbjm

bprd

bprestore

ag

up

Im

Disk volume
Note: * If the server is restoring its own data (server and client on same host), there is no . bptm child: tar reads the data directly from shared memory.

Ba

ck

About SAN client (UNIX or Windows) restoration

Figure A-13 shows the server and client components that are used in a restore of a SAN client over Fibre Channel.

Mount request nbrmms Mount

Master or media server

bptm

bpcd

bpbrm
Shar e mem d ory

bpcd

bptm* Backup Image

NetBackup tar

Client disk

Backup and restore functional overview About SAN client (UNIX or Windows) restoration

279

Figure A-13

SAN client restore with Fibre Transport

Master server

UNIX client

NetBackup user interface

Command line

nbjm

bprd

bprestore

Master or media server

bpcd Backup Image bptm Storage device (tape or disk) Backup Image Shared memory bpbrm bpcd

Shared memory

bptm child

NetBackup tar UNIX TAR32 Windows

nbftsrvr Backup image sent over Fibre Channel .

nbftclnt

Client disk

The process flow for a SAN Client restore is as follows (in the order presented).

When the user starts a restore, NetBackup invokes the clients bprestore program which sends a request to the request daemon, bprd. This request identifies the files and client. The request daemon then uses bpcd (client daemon) to start the backup and restore manager (bpbrm). Note: To restore Backup Exec images, bpbrm invoke mtfrd instead of tar on the clients. The server processes are the same as those used for NetBackup restores.

280

Backup and restore functional overview About SAN client (UNIX or Windows) restoration

If the disk device or tape device on which the data resides attaches to the master server, then bprd starts the backup and restore manager on the master server. If the disk unit or tape unit connects to a media server, bprd starts the backup and restore manager on the media server. bpbrm starts bptm and provides bptm with the backup ID and the shmfat (shared memory) flag. bptm does the following:

Requests SAN Client information from nbjm. Sends a restore request to the FT server process (nbftsrvr). Sends a restore request to the FT Client process on the client (nbftclnt). nbftclnt opens a fibre channel connection to nbftsrvr on the media server, allocates shared memory, and writes shared memory information to the backup ID file.

bpbrm starts tar by means of bpcd and provides tar with the backup ID, socket information, and the shmfat (shared memory) flag. bptm does the following:

Reads the image from the storage device. Creates a bptm child process. This process filters the backup image so that only the files that are selected for the restore are sent to the client. Writes the image data to the shared buffer on the server. When buffer is full or job is done, sets buffer flag (partial buffers may be sent to the client).

tar does the following:

Sends the status and control information to bpbrm. Reads the shared memory information from the local backup ID file (waits for the file to exist and become valid). Waits for the buffer flag that indicates the data is ready to be read. Reads data from the buffer, extracts files and restores them. When the shmfat (shared memory) flag is provided, tar considers the data to be already filtered.

The FT Server process nbftsrvr waits for the shared memory buffer flag to be set. nbftsrvr then transfers the image data to the FT Client (nbftclnt) shared memory buffer, and clears the buffer flag. The FT Client (nbftclnt) waits for the data from nbftsrvr and writes the data to the shared memory buffer on the client. nbftclnt then sets the buffer flag.

Backup and restore functional overview About Windows client restoration

281

At the end of the job:

bptm informs tar and bpbrm that the job is complete. bptm directs nbftclnt to close the fibre channel connection. nbftclnt closes the fibre channel connection and deletes the BID file.

About Windows client restoration

NetBackup supports the same types of operations on Windows clients as it does for UNIX clients. The following are the Windows processes involved in restore operations:

NBWIN is the user interface program on the client. The bpbackup function and

the bparchive function are merged into NBWIN.

BPINETD serves the same purpose as inetd on UNIX clients.

The NetBackup client daemon is called BPCD.

TAR32 is part of NetBackup for Windows and serves the same purpose as

NetBackup tar on UNIX. Note: To restore Backup Exec images, bpbrm invokes mtfrd.exe instead of tar32.exe on the clients. The server processes are the same as those used for NetBackup restores. The server processes are the same as described for UNIX. Figure A-14 shows the client processes involved in these operations.

282

Backup and restore functional overview About NetWare client restoration

Figure A-14

Restore - Windows client

Server For details on the server processes, see Backups and archives - UNIX clients earlier in this chapter.

Windows Client NetBackup User Interface

bprd

Request

NBWIN

BPINETD bpbrm

BPCD bptm

Backup Im

age

TAR32

Client Disk

About NetWare client restoration

NetBackup supports the same types of restore operations on NetWare clients as it does on UNIX clients.Figure A-15 shows the client processes involved in these operations. In this figure, the following applies:

The NetWare nontarget user interface program is called NBNWNT. The NetWare target user interface program is BP on the Netware console. The bprestore function and the bplist function are merged into the user interface programs on the clients. The NetBackup NetWare client daemon is called BPCD. The NetBackup tar functions are merged into BPCD.

Backup and restore functional overview About NetWare client restoration

283

mtfrd functionality (used to restore Backup Exec images) has been merged

into BPCD. The server processes involved in import and restore operations for Backup Exec images are the same as those involved for NetBackup restores. The server processes are the same as described for UNIX. Figure A-15 shows the restore operation for a NetWare client Figure A-15 Restore - NetWare client

Server

NetWare client

For details on the server processes, see Backups and Archives UNIX Clients earlier in this chapter.

NetBackup user interface

bprd

Request

NBNWNT (NetWare nontarget)

Backup Image bptm BPCD (NetWare)

Client disk
.

284

Backup and restore functional overview About catalog backup restoration

About catalog backup restoration

A catalog restore can be initiated by the NetBackup Catalog Recovery Wizard in the Administration Console, or by manual use of the bprecover command. More information is available in the following topic: See About disaster recovery on page 181. Figure A-16 illustrates the catalog restore and recovery process.

Backup and restore functional overview About catalog backup restoration

285

Figure A-16

Catalog restore and recovery

NetBackup Catalog Recovery Wizard

Command line

bprd

bprecover

Restore NetBackup Database Files See Restore from tape (UNIX) or Restore from disk, depending on the catalog backup policy.

bprd

Sybase ASA database agent

2 Restore Relational Database Files See Restore from tape (UNIX) or Restore from disk, depending on the catalog backup policy.

/usr/openv/db/ staging

Relational database files

A restore of the NetBackup database and relational database files from a hot catalog backup consists of the following steps (in the order presented):

The NetBackup database files are restored by means of the standard NetBackup restore procedure.

286

Backup and restore functional overview NetBackup directories and files

The relational database files are restored by means of the standard NetBackup restore procedure. The database files are restored to /usr/openv/db/staging (UNIX and Linux), or to install_path\NetBackupDB\staging (Windows). After the files are restored to the staging directory, the relational database is recovered. Each transaction log in the staging area is applied in order, one by one. The relational database files are moved from the staging directory to a location determined by the following: the bp.conf file VXDBMS_NB_DATA setting on UNIX or Linux and by the corresponding registry key on Windows. The default location is /usr/openv/db/data on UNIX and Linux, and install_path\NetBackupDB\data on Windows. If the relational database files are relocated, they are moved from the staging directory to the /usr/openv/db/data/vxdbms.conf file (UNIX) or the install_path\NetBackupDB\data\vxdbms.conf file (Windows). A description is available of how the NetBackup relational database files can be relocated after installation. See "NetBackup Relational Database" in the NetBackup Administrators Guide, Volume I.

Messages that are related to this catalog recovery process are divided into the following three areas:

For messages that are related to all catalog recovery steps, consult the /usr/openv/netbackup/logs/admin logs (UNIX and Linux), or install_path\NetBackup\logs\admin (Windows). For messages that are related to the first two bulleted items, consult the tar, bpbrm, and bpcd logs. For messages pertaining only to the relational database files, see the progress logs in the following directory: /usr/openv/netbackup/logs/user_ops/root/logs (UNIX and Linux), or install_path\NetBackup\logs\user_ops\root\logs (Windows).

NetBackup directories and files

Figure A-17 shows the NetBackup file and directory structure on UNIX servers and clients. If a host is only a client and not a server, only the files in the Client portion are present. If a host is both a client and a server, the client shares files as necessary from those in the Server portion. A Windows NetBackup server has equivalent files and folders that are located where NetBackup is installed (C:\Program Files\VERITAS by default).

Backup and restore functional overview NetBackup directories and files

287

NetBackup directory structure - UNIX

Figure A-17 lists the items that are described in tables on the following pages. Figure A-17 NetBackup directories and files

NetBackup server /usr/openv/ bin/ man/ tmp/ db/ msg/ var/ java/ netbackup/ volmgr/ lib/ resources/ logs/ share/

/usr/openv/netbackup/ bin/ help/ remote_versions/ bp.conf logs/ version client/ nblog.conf version_master db/ nblog.conf.template dbext/ nbsvcmon.conf

NetBackup client /usr/openv/ bin/ java/ lib/ msg/ netbackup/ resources/ share/ tmp/ var/

/usr/openv/netbackup/ bin/ bp.conf dbext/ help/ logs/ nblog.conf1 nblog.conf.template

Table A-2 describes the /usr/openv/ files and directories. Table A-2 Directories and files in /usr/openv/ - servers and UNIX clients Contents

File or directory in
/usr/openv/ bin/

Contains miscellaneous executable binaries including the vnetd daemon and utilities for legacy enhanced authentication. Contains the NetBackup Relational Database Manager (SQL Anywhere) and database data file.

db/

288

Backup and restore functional overview NetBackup directories and files

Table A-2

Directories and files in /usr/openv/ - servers and UNIX clients (continued) Contents

File or directory in
/usr/openv/ java/

Contains the NetBackup-Java Administration Console and the Backup, Archive and Restore user interface. Contains shared libraries that are required for NetBackup operation. Contains all logs that are written by unified logging. You do not have to create subdirectories for these logs. Contains man pages for NetBackup commands. Contains the message files and a configuration file for all installed languages of NetBackup. A tar file that contains the NetBackup-Java interfaces. See Table A-3 on page 289. Contains the NetBackup message catalogs that are used by unified logging (VxUL). Contains static configuration files. These files are normally unchanged between NetBackup releases. Contains the NetBackup Relational Database Manager (SQL Anywhere) installation trace files, and the log files regarding to database start and stop. Contains the variable configuration files. These files, which are related to licensing, authentication, authorization, and networking, may change while NetBackup is running. /usr/openv/var/global contains various static and variable configuration files. In a cluster, the /global directory is shared between nodes. Contains the media and device management directories and files. See NetBackup directory structure - UNIX on page 287.

lib/

logs/

man/ msg/

NB-Java.tar.Z netbackup/ resources/

share/

tmp/sqlany

var/

volmgr/

Contents of /usr/openv/netbackup
Table A-3 describes the /usr/openv/netbackup files and directories.

Backup and restore functional overview NetBackup directories and files

289

Table A-3

Directories and files in /usr/openv/netbackup/ - servers and UNIX clients Contents

File or Directory in
/usr/openv/netbackup/ bin/

Commands, scripts, programs, daemons, and files that are required for NetBackup operation and administration. On a server, there are two subdirectories under bin. admincmd: Contains various commands that used internally by NetBackup. Use these commands ONLY if they are documented. Most of these commands are not documented and should not be used directly. goodies (UNIX only): Contains scripts and information that may be useful to the administrator. These subdirectories are not present on clients.

bp.conf

Configuration file containing options for NetBackup operation. A detailed explanation is available about each option and how to set it. See the NetBackup Administrators Guide, Vol II. On a Windows server, these options are set in the NetBackup Administration Console.

client/

NetBackup client software that is installed on the clients during installation. Do not install this directory on a media server. NetBackup catalogs. See Table A-5 on page 304.

db/

dbext/

For NetBackup database agent software, contains the version file, compressed tar file, and install_dbext script. Help files that are used by NetBackup programs. These files are in ASCII format. Legacy debug logs for NetBackup processes. You must create the necessary subdirectories in order for these log files to be written. See About legacy logging on page 127. See Table A-4 on page 291. for an explanation of the processes that produce the logs.

help/

logs/

290

Backup and restore functional overview NetBackup programs and daemons

Table A-3

Directories and files in /usr/openv/netbackup/ - servers and UNIX clients (continued) Contents

File or Directory in
/usr/openv/netbackup/ nblog.conf

Specifies the settings for unified logging.

Note: Do not edit this file manually: use the vxlogcfg

command instead. See About unified logging on page 103. nblog.conf.template Specifies the settings for unified logging.

Note: Do not edit this file manually: use the vxlogcfg

command instead. See About unified logging on page 103. nbsvcmon.conf Configuration file for the NetBackup Service Monitor. It tells the Service Monitor what services to monitor and how to restart them if they fail unexpectedly. A cache of the versions of other media servers in the system. Version and release date of the software. Identifies the NetBackup master server.

remote_versions/ version version_master

NetBackup programs and daemons

Table A-4 describes the programs and daemons that provide most of the control for backup, archive, and restore operations. The explanations include what starts and stops the program or daemon, and the debug log subdirectory (if any) where it records its activities. You must create legacy logging directories manually; see "logs" in the previous table. More information is available. See About legacy logging on page 127.

Backup and restore functional overview NetBackup programs and daemons

291

Table A-4

NetBackup daemons and programs Description

On UNIX clients, this menu-driven, character-based interface program has options for starting user-directed backups, restores, and archives. Started By: /usr/openv/netbackup/bin/bp command on the client. Stopped By: Exiting the interface program. Debug Log: /usr/openv/netbackup/logs/bp on the client. The debug logs for bpbackup, bparchive, bprestore, and bplist also have information about bp activities.

Program/Daemon
bp

BP.NLM

On NetWare target clients, BP.NLM is the NetWare Loadable Module that starts the client-user interface. Started By: LOAD BP command. Stopped By: Choosing Quit Utility from the main menu. Debug Log: SYS:\VERITAS\NBUCLT\NETBACK\LOGS\BP\mmddyy.log file on the client.

bpadm

On a UNIX master server, this administrator utility has a menu-driven, character-based, interface with options for configuring and managing NetBackup. Started By: /usr/openv/netbackup/bin/bpadm command on the master server. Stopped By: Quit option from within bpadm. Debug Log: admin legacy log directory on the server.

bparchive

On UNIX clients, this program communicates with bprd on the master server when a user starts an archive. Started By: Starting an archive by using the client-user interface or by executing the /usr/openv/netbackup/bin/bparchive command on the client. Stopped By: Completion of operation. Debug Log: bparchive legacy log directory on the client.

292

Backup and restore functional overview NetBackup programs and daemons

Table A-4

NetBackup daemons and programs (continued) Description

On UNIX clients, this program communicates with bprd on the master server when a user starts a backup. Started By: Starting a backup by using the client-user interface or by executing the /usr/openv/netbackup/bin/bpbackup command on the client. Stopped By: Completion of operation Debug Log: bpbackup legacy log directory on the client.

Program/Daemon
bpbackup

bpbkar

On UNIX clients the Backup/Archive Manager generates the backup images. Started By: bpbrm on the server with the storage unit. Stopped By: Completion of operation. Debug Log: bpbkar legacy log directory on the client.

BPBKAR32

On Windows clients, the Backup/Archive Manager generates the backup images. Started By: BPCDW32 on the client. Stopped By: Completion of operation. Debug Log: BPBKAR legacy log directory in the NetBackup logs directory on the client.

bpbrm

On master and media servers, the Backup/Restore Manager manages the client and bptm or bpdm process. It also uses error status from the client and from bptm or bpdm to determine the final status of backup or restore operations. Started By: For each backup or restore, nbjm starts an instance of bpbrm on the server with the appropriate storage unit. Stopped By: Completion of operation. Debug Log: bpbrm legacy log directory on the server.

Backup and restore functional overview NetBackup programs and daemons

293

Table A-4

NetBackup daemons and programs (continued) Description

On UNIX clients, bpcd is the NetBackup client daemon and lets NetBackup start programs on remote hosts (can be UNIX clients or other servers). For example, the server can connect to UNIX clients without requiring /.rhosts entries on the remote host. The program is used when nbjm starts bpbrm and when bpbrm communicates with the client. (For a description of the NetBackup client daemon on PC clients, see BPCDW32.EXE and BPCD.NLM in this table.) Started By: inetd. Stopped By: Completion of operation. Debug Log: bpcd legacy log directory on both client and server.

Program/Daemon
bpcd

BPCD.NLM

On NetWare clients, BPCD.NLM is the executable file that starts the NetBackup client daemon. Started By: When you enter BPSTART.NCF at the NetWare Server console. Or, add BPSTART.NCF to your autoexec.ncf file. Stopped By: UNLOAD BP command Debug Log: BPCD legacy log directory on the client.

BPCDW32.EXE

On Windows clients, BPCDW32.EXE is the executable file that starts the NetBackup client daemon. Started By: When Windows starts if the daemon is in the Startup group. Otherwise, by double clicking on its icon. Stopped By: On Windows, you can stop it through the Services application in the Control Panel. Debug Log: BPCD legacy log directory on the client.

bpdbjobs

On UNIX master servers, this program is used to clean up the NetBackup jobs database. Started By: /usr/openv/netbackup/bin/admincmd/bpdbjobs. When bprd starts, it runs this command automatically. The administrator can also execute it manually or with a cron job. Stopped By: No terminate option exists for this command outside of using kill. Debug Log: bpdbjobs legacy log directory on the server.

294

Backup and restore functional overview NetBackup programs and daemons

Table A-4

NetBackup daemons and programs (continued) Description

On master servers, the NetBackup database manager program that manages the configuration, error, and file databases. Started By: bprd (also by /usr/openv/netbackup/bin/initbpdbm on UNIX) Stopped By: /usr/openv/netbackup/bin/bpdbm -terminate command on UNIX and by stopping the NetBackup Database Manager service on Windows. Debug Log: bpdbm legacy log directory on the server.

Program/Daemon
bpdbm

bpdm

On master and media servers, bpdm is used for the following disk operations: read phase of disk duplication, read phase of synthetic backups, disk verify and disk import, true image restore from disk, disk image deletion. Started By: For each backup or restore, bpbrm starts an instance of bpdm, on the server with the storage unit. Stopped By: Completion of operation. Debug Log: bpdm legacy log directory on the server.

bpfis

On clients, bpfis creates and deletes snapshots. Note that bpfis is part of the Snapshot Client add-on product. Started By: bpbrm. Stopped By: Completion of operation. Debug Log: bpfis legacy log directory on the client or alternate client.

bphdb

On SQL, Oracle, Informix, Sybase, DB2, and SAP database clients, bphdb executes scripts to back up the database. Started By: Client-user interface when the user starts a database backup operation. Stopped By: Completion of operation. Debug Log: bphdb legacy log directory on the client.

Backup and restore functional overview NetBackup programs and daemons

295

Table A-4

NetBackup daemons and programs (continued) Description

NetBackup-Java master server application program. This program runs on all NetBackup UNIX systems and authenticates the users that start the NetBackup-Java interface programs. Started By: inetd during startup of the NetBackup Java interfaces. Stopped By: When authentication is complete. Debug Log: bpjava-msvc legacy log directory on the server.

Program/Daemon
bpjava-msvc

bpjava-usvc

NetBackup-Java user server application program. This program services all requests from the NetBackup-Java user and administration interfaces. Started By: bpjava-msvc upon successful login through the Login dialog box that is presented when a NetBackup-Java interface is started. Stopped By: When the interface program is terminated. Debug Log: bpjava-usvc legacy log directory.

bplist

On UNIX clients, this program communicates with bprd on the master server when a user browses the database during a restore operation. Started By: Starting a search of the image database by using the client-user interface or by executing the /usr/openv/netbackup/bin/bplist command on the client. Stopped By: Completion of operation Debug Log: bplist legacy log directory on the client.

296

Backup and restore functional overview NetBackup programs and daemons

Table A-4

NetBackup daemons and programs (continued) Description

On master servers, the request daemon responds to client and administrative requests for the following:

Program/Daemon
bprd

Restores Backups (scheduled and user-directed) Archives List that is backed up or archived files Manual immediate backups (started through the NetBackup administration interface manual backup option)

Started By: Initiate Request Daemon option on the Special Actions menu in bpadm (also the /usr/openv/netbackup/bin/initbprd command). Stopped By: Terminate Request Daemon option on the Special Actions menu in bpadm. Debug Log: bprd legacy log directory on the server. bprestore On UNIX clients, this program communicates with bprd on the master server when a user starts a restore. Started By: Starting restore by using the client-user interface (or by executing the /usr/openv/netbackup/bin/bprestore command on the client). Stopped By: Completion of operation Debug Log: bprestore legacy log directory on the client. BPSVR.NLM On NetWare nontarget clients, BPSVR.NLM is the program that allows the system that has the client-user interface to communicate with the Netware server that is the NetBackup client. Started By: Enter bpstart.ncf. Stopped By: Enter bpstop.ncf. Debug Log: SYS:VERITAS\NBUCLT\NetBack\logs\bpsrv\ directory on the client. BPSYS.EXE On Windows clients, BPSYS.EXE is the NetBackup System Registry Replacement utility. Started By: NetBackup as required. Stopped By: Completion of operation. Debug Log: BPSYS legacy log directory on the client.

Backup and restore functional overview NetBackup programs and daemons

297

Table A-4

NetBackup daemons and programs (continued) Description

On master and media servers, bptm manages both disk and tape backup and restore. It is used when the storage unit type is either disk or Media Manager. This program manages the transfer of images between the client and the storage device. Started By: For each backup or restore, bpbrm starts an instance of bptm on the server that has the storage unit. Stopped By: Completion of operation. Debug Log: bptm legacy log directory on the server.

Program/Daemon
bptm

jbpSA

A Java-based program for performing backups, archives, and restores of UNIX clients. Started By: On UNIX, the /usr/openv/netbackup/bin/jbpSA command. Debug Log: None, although the logs for the bpbackup, bparchive, bplist, and bprestore commands on the client can be useful. Also, check the bpjava-msvc and bpjava-usvc logs.

jnbSA

A Java-based administration utility for managing NetBackup on UNIX. In addition, administration of supported UNIX systems can be performed by using the NetBackup-Java Windows Display Console on a Windows system. Started By: On UNIX, the /usr/openv/netbackup/bin/jnbSA command. On a NetBackup-Java Windows Display console, the NetBackup - Java on host menu item on the Programs/NetBackup menu. Stopped By: Exit option in jnbSA. Debug Log: None, although the logs for bpjava-msvc and bpjava-usvc can be helpful.

298

Backup and restore functional overview NetBackup programs and daemons

Table A-4

NetBackup daemons and programs (continued) Description

On the server that is defined as the EMM server, nbemm manages devices, media, and storage unit configuration, and performs resource selection. Replaces vmd as the device allocator. Started By: Started when NetBackup starts. Stopped By: /usr/openv/netbackup/bin/nbemmm -terminate Debug Log: On the server, /usr/openv/logs (UNIX) or install_path\logs (Windows). See About unified logging on page 103.

Program/Daemon
nbemm

nbaudit

On the master server, the audit daemon accepts audit requests from other NetBackup components and persists the audit records in the database. It also queries and returns the audit records from the database to display to the user. Started By: Started when NetBackup starts. Stopped By: /usr/openv/netbackup/bin/nbaudit -terminate. Debug Log: On the server, /usr/openv/logs/nbaudit (UNIX) or install_path\logs\nbaudit (Windows).

nbfdrv64

On a media server that is enabled for SAN Client backup over fibre channel, nbfdrv64 is the following: a user mode component that is used for both backup and restore. nbfdrv64 uses a windrvr6 proxy to move fibre channel data between nbftclnt and bptm buffers. Started By: /usr/openv/netbackup/bin/nbftsrvr Stopped By: /usr/openv/netbackup/bin/nbftsrvr -terminate Debug Log: On the server, /usr/openv/logs (UNIX) or install_path\logs (Windows). See About unified logging on page 103.

Backup and restore functional overview NetBackup programs and daemons

299

Table A-4

NetBackup daemons and programs (continued) Description

On clients that are enabled for SAN Client backup over fibre channel, nbftclnt transfers the backup image over fibre channel to nbftsrvr on the media server. Started By: Started when NetBackup starts. Stopped By: /usr/openv/netbackup/bin/nbftclnt -terminate. Debug Log: On the client, /usr/openv/logs (UNIX) or install_path\logs (Windows). See About unified logging on page 103.

Program/Daemon
nbftclnt

nbftsrvr

On a media server that is enabled for SAN Client backup over fibre channel, nbftsrvr does the following: reads the backup image from nbftclnt and transfers it to shared memory on the media server. Started By: Started when NetBackup starts. Stopped By: /usr/openv/netbackup/bin/nbftsrvr -terminate. Debug Log: On the server, /usr/openv/logs (UNIX) or install_path\logs (Windows). See About unified logging on page 103.

nbjm

On master servers, the nbjm service accepts job requests from nbpem and from media commands such as bplabel and tpreq. nbjm acquires job resources from nbrb, and runs the jobs once resources are available. Started By: Started when NetBackup starts. Stopped By: /usr/openv/netbackup/bin/nbjm -terminate Debug Log: On the server, /usr/openv/logs (UNIX) or install_path\logs (Windows). See About unified logging on page 103.

NBNWNT.EXE

For NetWare nontarget clients, NBNWNT.EXE is the executable file that starts the client-user interface on Windows systems. Started By: From the Windows Start menu, under Programs/ NetBackup. Stopped By: Exiting the client-user interface. Debug Log: none.

300

Backup and restore functional overview NetBackup programs and daemons

Table A-4

NetBackup daemons and programs (continued) Description

On master servers, the nbpem service uses nbproxy to get the policy list from bpdbm, creates the policy-client tasks, determines when jobs are due to run, and starts due jobs. Started By: Started when NetBackup starts. Stopped By: /usr/openv/netbackup/bin/nbpem -terminate Debug Log: On the server, /usr/openv/logs (UNIX) or install_path\logs (Windows). See About unified logging on page 103.

Program/Daemon
nbpem

nbproxy

Runs on the master server and the media server as a child of the process it serves. nbproxy provides a thread-safe API for the libraries that are not yet thread safe. Started By: the process that uses nbproxy as a proxy. Stopped By: stops the process that uses nbproxy. Debug Log: nbproxy legacy log directory on the server.

nbrb

On the server that is defined as the EMM server, the nbrb service accepts resource requests from nbjm, acquires physical resources from nbemm, and manages logical resources. Started By: Started when NetBackup starts. Stopped By: /usr/openv/netbackup/bin/nbrb -terminate Debug Log: On the server, /usr/openv/logs (UNIX) or install_path\logs (Windows). See About unified logging on page 103.

ndmpagent

Controls backup and restore operations on a NAS server. ndmpagent is for remote NDMP: backing up NDMP data to a drive that is configured in a Media Manager storage unit on a NetBackup media server. Started By: bpbrm. Stopped By: Completion of backup or restore. Debug Log: On the server, /usr/openv/logs (UNIX) or install_path\logs (Windows). See About unified logging on page 103.

Backup and restore functional overview NetBackup programs and daemons

301

Table A-4

NetBackup daemons and programs (continued) Description

Runs on the master server. The nbstserv service manages lifecycle operations including duplication, staging, and image expiration. Started By: Started when NetBackup starts. Stopped By: /usr/openv/netbackup/bin/nbstserv -terminate Debug Log: On the server, /usr/openv/logs (UNIX) or install_path\logs (Windows). For more information about OID 226 and 272, see the following topic: See Originator IDs for the entities that use unified logging on page 108.

Program/Daemon
nbstserv

NBWIN.EXE

For Windows clients, NBWIN.EXE is the executable file that starts the client-user interface on Windows systems. Started By: From the Windows Start menu, under Programs/ NetBackup. Stopped By: Exiting the client-user interface. Debug Log: NBWIN legacy log directory on the client.

nbrmms

Remote Manager and Monitor Service (nbrmms) is the conduit through which EMM discovers and configures storage on media servers. In addition to configuration management, nbrmms provides all access to media server resources for monitoring and event notifications. Started By: Started when NetBackup starts, or by /usr/openv/netbackup/bin/nbrmms Stopped By: Stopped when NetBackup stops, or by /usr/openv/netbackup/bin/nbrmms -terminate Debug Log: On the server, /usr/openv/logs (UNIX) or install_path\logs (Windows). See About unified logging on page 103.

302

Backup and restore functional overview NetBackup programs and daemons

Table A-4

NetBackup daemons and programs (continued) Description

Private Branch Exchange (PBX) is a common services framework that helps limit the number of TCP/IP ports that the CORBA services of NetBackup use. Started By: Started when NetBackup starts, or by /opt/VRTSpbx/bin/vxpbx_exchanged start Stopped By: Stopped when NetBackup stops, or by /opt/VRTSpbx/bin/vxpbx_exchanged stop Debug Log: On the server, /opt/VRTSpbx/log (UNIX) or install_path\VxPBX\log (Windows). See Accessing the PBX logs on page 79.

Program/Daemon
pbx_exchange

ql2300_stub

On a Solaris media server that is enabled for SAN Client transfers over fibre channel: ql2300_stub is a device driver used to read and write to the NVRAM on a target mode Fibre Channel Host Bus Adapter. On Linux, it also prevents initiator mode drivers from binding to the target mode fibre channel HBAs. Started By: Device driver that is started by the operating system on a reboot after nbftsrv_config -nbhba on Linux and Solaris. On Linux, it is also started on all reboots after nbftsrv_config. Stopped By: Device driver that is stopped by nbfdrv64 on Linux and nbftsrv_config on Solaris. Debug Log: The host operating system handles the logging for the device driver in the system messages log: /var/adm/messages (Solaris) or /var/log/messages (Linux).

tar

On UNIX clients, the Tape ARchive program is a special version of tar provided with NetBackup and used to restore images. Started By: For each restore, bpbrm starts an instance of tar on the client. Stopped By: Completion of restore operation. Debug Log: tar legacy log directory on the client.

Backup and restore functional overview NetBackup catalogs

303

Table A-4

NetBackup daemons and programs (continued) Description

On Windows clients, the TAR32 program is a special version of tar provided with NetBackup and used to restore images. Started By: For each restore, NetBackup starts an instance of TAR32 on the client. Stopped By: Completion of restore operation. Debug Log: TAR legacy log directory on the client.

Program/Daemon
TAR32

windrvr6

On a Media Server that is enabled for SAN Client transfers using fibre channel: windrvr6 is a kernel device driver used to communicate through the PCI bus to the target mode Fibre Channel Host Bus Adapters. Started By: Device driver that is started by the operating system at boot (Solaris) or by nbfdrv64 (Linux). Stopped By: Device driver that is stopped by the operating system at shutdown. Debug Log: The host operating system handles the logging in the system messages log log: /var/adm/messages (Solaris) or /var/log/messages (Linux).

NetBackup catalogs
The NetBackup catalogs contain the information that is used internally by NetBackup and reside in the /usr/openv/netbackup/db directory on UNIX servers and in the install_path\NetBackup\db directory on Windows NetBackup servers. The /usr/openv/netbackup/db/class directory (install_path\NetBackup\db\class on Windows) has a subdirectory for each NetBackup policy that contains information about the policy. Table A-5 describes the NetBackup catalogs.

304

Backup and restore functional overview NetBackup catalogs

Table A-5 Database

config

NetBackup catalogs Contents

Configuration information. This database resides on the master server and has three parts: policy: Contains the information about each NetBackup policy. config: Contains the information about global attributes, storage units, and database backups. altnames: Contains the information about client names for restores.

error

Error and status information about NetBackup operations. This database resides on the master server and has two parts: error: Contains the information that is recorded during backup operations and used in the NetBackup reports. failure_history: Contains the daily history of backup errors.

images

Information about the backup images and resides only on the master server. One of the files in the images directory is the file database. The file database is the one that NetBackup accesses when a user browses for files to restore. Job information that is used by the NetBackup job monitor (UNIX NetBackup server) and activity monitor (Windows NetBackup server). The Jobs database is on the master server. Media related information that is used by bptm. Also has an errors file that contains error history information for media and devices.

jobs

media

The NetBackup Search feature uses the catalog to help locate backup files, hold them, and then release them. It backs up the NetBackup client data to the media server and backs up the catalog metadata to the master server. The NetBackup indexing server indexes the metadata in the catalog on the master server.

Appendix

Media and device management functional description

This appendix includes the following topics:

Media and device management startup process Media and device management process Shared Storage option management process Barcode operations Media and device management components

Media and device management startup process

Media and device management processes are automatically initiated during NetBackup startup. To start these processes manually, run bp.start_all (UNIX) or bpup (Windows). ltid automatically starts other daemons and programs as necessary. The daemons should be running after initial startup. See Figure B-1 on page 307. In the case of robotic daemons, such as tl8d and tlhd, the associated robot must also be configured for the daemon to run. There are also additional ways to start and stop daemons. See Table B-1 on page 314. TL8, TLH, and TLD require following types of daemons:

306

Media and device management functional description Media and device management startup process

robotic

Each host with a robotic drive attached must have a robotic daemon. These daemons provide the interface between ltid and the robot or, if different drives within a robot can attach to different hosts, the robotic daemon communicates with a robotic-control daemon (see below). Robotic-control daemons centralize the control of robots when drives within a robot can connect to different hosts. A robotic-control daemon receives mount and unmount requests from the robotic daemon on the host to which the drive is attached and then communicates these requests to the robot.

robotic control

You must know the hosts involved in order to start all the daemons for a robot.

Media and device management functional description Media and device management process

307

Figure B-1

Starting media and device management

,

At system startup, the server automatically starts ltid which starts applicable robotic daemons.

To start the processes manually, enter: On UNIX: /usr/openv/netbackup/bin/bp.start_all On Windows: install_path \NetBackup\bin\bpup Automated Cartridge System

acsd

acsssi

acsel ltid odld Optical Disk Library

tl4d

Tape Library 4mm

tl8d

tl8cd

Tape Library 8mm

tldd vmd avrd tlhd

tldcd

Tape Library DLT

tlhcd

Tape Library Half-inch Tape Library Multimedia Tape Stacker Half-inch

tlmd

tshd

Media and device management process

When the media and device management daemons are running, NetBackup, Storage Migrator (UNIX only), Storage Migrator for Microsoft Exchange (Windows only), or users can request data storage or retrieval. The request is initially handled by the scheduling services. See Backup and archive processes on page 254.

308

Media and device management functional description Media and device management process

The resulting request to mount a device is passed from nbjm to nbrb, which acquires the physical resources from nbemm (the Enterprise Media Manager service). If the backup requires media in a robot, ltid sends a mount request to the robotic daemon that manages the drives in the robot that are configured on the local host. The robotic daemon then mounts the media, and sets a drive busy status in memory shared by itself and ltid. Drive busy status also appears in the Device Monitor. See Figure B-2 on page 309. Assuming that the media is physically in the robot, the media is mounted and the operation proceeds. If the media is not in the robot, nbrb creates a pending request, which appears as a pending request in the Device Monitor. An operator must then insert the media in the robot and use the appropriate Device Monitor command to resubmit the request so the mount request can occur. A mount request is also issued if the media is for a nonrobotic (standalone) drive and the drive does not contain media that meets the criteria in the request. If the request is from NetBackup and the drive does contain appropriate media, then that media is automatically assigned and the operation proceeds. More information is available on NetBackup media selection for nonrobotic drives. See the NetBackup Administrators Guide, Volume II. Note: On UNIX systems, when a tape is being mounted, the drive_mount_notify script is called. This script is in the /usr/openv/volmgr/bin directory. Information on the script can be found within the script itself. A similar script is called for the unmount process (drive_unmount_notify, in the same directory). When a robotic volume is added or removed through the media access port, the media management utility communicates with the appropriate robotic daemon to verify the volume location and/or barcode. The media management utility (through a library or command-line interface) also calls the robotic daemon for robot inventory operations. Figure B-2 shows an example of the media and device management process.

Media and device management functional description Shared Storage option management process

309

Figure B-2
User

Media and device management example process

Backup and archive processes

nbjm EMM Database bptm Request tape mount Itid

Mo t un me dia ID

Device monitor

Devicemanagement utility

nbemm

Media-management utility Inventory barcodes or inject/eject

SDLT600

LT0-3

tl8d

tl8cd Robotic control Tape library TL8

Non-robotic drives

Shared Storage option management process

Shared Storage Option (SSO) is an extension to tape drive allocation and configuration for media and device management. SSO allows individual tape drives (stand-alone or in a robotic library) to be dynamically shared between multiple NetBackup media servers or SAN media servers. More information is available on the Shared Storage Option. See the NetBackup Administrators Guide for UNIX and Linux, Volume II, or See the NetBackup Administrators Guide for Windows Volume II

310

Media and device management functional description Shared Storage option management process

The following shows the shared storage option management process in the order presented:

NetBackup, Storage Migrator, or users can initiate backups. nbjm makes a mount request for the backup. nbrb tells the EMM server to obtain a drive for the backup. nbrb tells the device allocator (DA) in the EMM server to stop scanning the selected drive. nbemm tells the appropriate media server (the scan host for the selected drive) to stop scanning the drive. The stop scan request is carried out by means of oprd, ltid, and avrd in the media servers shared memory. nbemm informs nbrb when scanning on the selected drive has stopped. nbrb informs nbjm that the selected drive (A) is available for the backup. nbjm conveys the mount request and drive selection to bptm, which proceeds with the backup. To protect the integrity of the write operation, bptm uses SCSI reservations. See How NetBackup reserves drives in the NetBackup Administrators Guide, Volume II. The mount-media operation is initiated. bptm makes position checks on the drive to ensure that the drive has not been rewound by another application. bptm also does the actual write to the tape. When the backup is complete, nbjm tells nbrb to release resources. nbrb de-allocates the drive in EMM. EMM tells the scan host to resume scanning the drive. The scan request is carried out by means of oprd, ltid, and avrd in the media servers shared memory.

Figure B-3 illustrates the shared storage option management process.

Media and device management functional description Barcode operations

311

Figure B-3

Media and device management process flow showing SSO components

Device monitor EMM Database Devicemanagement utility

User

Backup and archive processes

nbjm

nbrb

nbemm/DA

Request tape mount

ltid

St o

ltid

sc

bptm

an

ltid

ltid

bptm

avrd

avrd

bptm

6.5 Media Server 1 Scan host for drive A

6.5 Media Server 2 Scan host for drive B

Shared drive A Note: Shaded area represents shared memory on the media server.

Shared drive B

Barcode operations
Barcode reading is mainly a function of the robot hardware rather than media and device management. When a robot has a barcode reader, it scans any barcode

312

Media and device management functional description Barcode operations

that may be on a tape and stores the code in its internal memory. This associates the slot number and the barcode of the tape in that slot. NetBackup determines that association for its own use by interrogating the robot. If a robot supports barcodes, NetBackup automatically compares a tapes barcode to what is in the EMM database as an extra measure of verification before mounting the tape. A request for media that is in a robot that can read barcodes begins in the same manner as other requests. See Figure B-4 on page 313. ltid includes the media ID and location information in a mount request to the robotic daemon for the robot that has the media ID. This request causes the robotic daemon to query the robotic-control daemon or the robot for the barcode of the tape in the designated slot. (This is a preliminary check to see if the correct media is in the slot.) The robot returns the barcode value it has in memory. The robotic daemon compares this barcode with the value it received from ltid and takes one of the following actions:

If the barcodes dont match, and the mount request is not for a NetBackup backup job, the robotic daemon informs ltid and a pending action request (Misplaced Tape) appears in the Device Monitor. An operator must then insert the correct tape in the slot. If the barcodes dont match and the mount request is for a NetBackup backup job, the robotic daemon informs ltid and the mount request is canceled. NetBackup (bptm) then requests a new volume from nbjm and from EMM. If the barcodes match, the robotic daemon requests the robot to move the tape to a drive. The robot then mounts the tape. At the start of the operation, the application (for example, NetBackup) checks the media ID and if it also matches what should be in this slot, the operation proceeds. For NetBackup, a wrong media ID results in a media manager found wrong tape in drive error (NetBackup status code 93).

Media and device management functional description Media and device management components

313

Figure B-4
Devicemanagement utility User

Barcode request
NetBackup EMM Database

Request media ID mount

Itid

nbemm

vmd

Media-management utility

Mount media ID Robot inventory request or inject/eject tl8d tl8cd

What is Barcode Mount barcode tape

Tape library TL8

Media and device management components

This topic shows the file and directory structure and the programs and daemons assoicated with the media and device management. Figure B-5 shows the file and directory structure for media and device management on a UNIX server. A Windows NetBackup server has equivalent files and directories that are located in the directory where NetBackup is installed (by default, C:\Program Files\VERITAS).

314

Media and device management functional description Media and device management components

Figure B-5
NetBackup server

Media and device management directories and files

/usr/openv/volmgr/ bin/ debug/ 1 help/ NetBackup_DeviceConfig_Guide.tx misc/ vm.conf2

/usr/openv/volmgr/bin/ driver/ format/ goodies/ avrd/1 robots/1 1. Created by administrator to enable legacy debug logging.

1 /usr/openv/volmgr/debug/

daemon/1 tpcommand/1

ltid/1

reqlib/1

/vmscd

2. Created by administrator or automatically by media management utilities.

Table B-1 describes the directories and files that are of special interest. Table B-1 File or directory
bin

Media and device management directories and files Contents

Commands, scripts, programs, daemons, and files required for media and device management. There are three subdirectories under bin. driver: Contains SCSI drivers used on various platforms to control robotics. format: Disk format information for optical platters on Solaris (SPARC only) platforms. goodies: Contains vmconf script and scan utility.

debug

Legacy debug logs for the Volume Manager daemon, vmd, and all requesters of vmd, ltid, and device configuration. The administrator must create these directories for debug logging to occur. Help files used by media and device management programs. These files are in ASCII format. Lock files and temporary files required by various components of media and device management.

help

misc

Media and device management functional description Media and device management components

315

Table B-1 File or directory

vm.conf

Media and device management directories and files (continued) Contents

Media and device management configuration options.

Table B-2 describes the media and device management programs and daemons. The explanations include what starts and stops the program or daemon, and the log (if any) where it records its activities. On UNIX, all of the components discussed in this table reside under /usr/openv/volmgr/bin. On Windows, they reside under install_path\volmgr\bin. Note: The following table contains references to the system log. This log is managed by syslog on UNIX (the facility is daemon). On Windows the Event Viewer manages the system log (the log type is Application). Table B-2 Media and device management daemons and programs

Program or daemon Description

acsd The Automated Cartridge System daemon interfaces with the Automated Cartridge System. It communicates with the server that controls the ACS robotics through the acsssi process (UNIX) or the STK Libattach Service (Windows). Also, for UNIX, see the acsssi and acssel programs. Started By: Starting ltid (or on UNIX, independently by using the /usr/openv/volmgr/bin/ascd command. Stopped By: Stopping ltid (or on UNIX, independently by finding the PID (process id) and then using the kill command). Debug Log: Errors are logged in the system log and robots debug log. Debug information is included by adding VERBOSE to the vm.conf file. On UNIX, debug information is also included by starting the daemon with the -v option: this option can also be used through ltid, or by putting VERBOSE in the vm.conf file. acssel Available only on UNIX. See the NetBackup Device Configuration Guide. acsssi Available only on UNIX. See the NetBackup Device Configuration Guide.

316

Media and device management functional description Media and device management components

Table B-2

Media and device management daemons and programs (continued)

Program or daemon Description

avrd The automatic-volume-recognition daemon controls automatic volume assignment and label scanning. This lets NetBackup read labeled tape and optical disk volumes and to automatically assign the associated removable media to requesting processes. Started By: Starting ltid (or on UNIX, independently by using the /usr/openv/volmgr/bin/avrd command). Stopped By: Stopping ltid, (or on UNIX, independently by finding the PID (process id) and then using the kill command). Debug Log: All errors are logged in the system log. Debug information is included by adding VERBOSE to the vm.conf file. On UNIX, debug information is also included by aborting avrd and starting the daemon with the -v option. ltid The device demon (UNIX) or NetBackup Device Manager service (Windows) controls the reservation and assignment of tapes and optical disks. Started By: /usr/openv/volmgr/bin/ltid command on UNIX or Stop/Restart Device Manager Service command in Media and Device Management window on Windows. Stopped By: /usr/openv/volmgr/bin/stopltid command on UNIX or Stop/Restart Device Manager Service command in the Media and Device Management window on Windows. Debug Log: Errors are logged in the system log and ltid debug log. Debug information is included if the daemon is started with the -v option (available only on UNIX) or adding VERBOSE to the vm.conf file. odld The Optical Disk Library daemon interfaces with the Optical Disk Library, communicating with the robotics through a SCSI interface. This library is not supported on Windows. Started By: Starting ltid or independently by using the /usr/openv/volmgr/bin/odld command. Stopped By: Stopping ltid or independently by finding the PID (process id) and then using the kill command. Debug Log: All errors are logged in the system log. Debug information is included if the daemon is started with the -v option (either by itself or through ltid) or adding VERBOSE to the vm.conf file.

Media and device management functional description Media and device management components

317

Table B-2

Media and device management daemons and programs (continued)

Program or daemon Description

tl4d The Tape Library 4MM daemon is the interface between ltid and the Tape Library 4MM and communicates with the robotics through a SCSI interface. Started By: Starting ltid (or on UNIX, independently by using the /usr/openv/volmgr/bin/tl4d command). Stopped By: Stopping ltid (or on UNIX, independently by finding the PID (process id) and then using the kill command). Debug Log: All errors are logged in the system log. Debug information is included by adding VERBOSE to the vm.conf file. On UNIX, debug information is also included by starting the daemon with the -v option (either by itself or through ltid). tl8d The Tape Library 8MM daemon provides the robotic control for a TL8 robot (Tape Library 8mm or Tape Stacker 8mm). The Tape Library 8MM daemon drives in the same TL8 robot may be attached to different hosts than the robotic control. tl8d is the interface between the local ltid and the robotic control. If a host has a device path for a drive in a TL8 robot, then mount or unmount requests for that drive go first to the local ltid and then to the local tl8d (all on the same host). tl8d then forwards the request to tl8cd on the host that is controlling the robot (could be on another host). Started By: Starting ltid (or on UNIX, independently by using the /usr/openv/volmgr/bin/tl8d command). Stopped By: Stopping ltid (or on UNIX, independently by finding the PID (process id) and then using the kill command. Debug Log: Errors are logged in the system log and robots debug log. Debug information is included by adding VERBOSE to the vm.conf file. On UNIX, debug information is also included by starting the daemon with the -v option (either by itself or through ltid).

318

Media and device management functional description Media and device management components

Table B-2

Media and device management daemons and programs (continued)

Program or daemon Description

tl8cd The Tape Library 8MM Control daemon provides the robotic control for a TL8 robot and communicates with the robotics through a SCSI interface. tl8cd receives mount and unmount requests from tl8d on the host to which the drive is attached and then communicates these requests to the robot. Started By: Starting ltid (or on UNIX, independently by using the /usr/openv/volmgr/bin/tl8cd command). Stopped By: Stopping ltid or by using the tl8cd -t command. Debug Log: Errors are logged in the system log and robots debug log. Debug information is included by adding VERBOSE to the vm.conf file. On UNIX, debug information is also included by starting the daemon with the -v option (either by itself or through ltid). tldd The Tape Library DLT daemon works in conjunction with tldcd to handle requests to TLD robots (Tape Library DLT and Tape Stacker DLT). tldd provides the interface between the local ltid and the robotic control (tldcd) in the same manner as explained previously for tl8d. Started By: Starting ltid (or on UNIX, independently by using the /usr/openv/volmgr/bin/tldd command). Stopped By: Stopping ltid (or on UNIX, independently by finding the PID (process id) and then using the kill command). Debug Log: Errors are logged in the system log and robots debug log. Debug information is included by adding VERBOSE to the vm.conf file. On UNIX, debug information is also included by starting the daemon with the -v option (either by itself or through ltid).

Media and device management functional description Media and device management components

319

Table B-2

Media and device management daemons and programs (continued)

Program or daemon Description

tldcd The Tape Library DLT Control daemon provides robotic control for a TLD robot in the same manner as explained previously for tl8cd. Started By: Starting ltid (or on UNIX, independently by using the /usr/openv/volmgr/bin/tldcd command). Stopped By: Using the tldcd -t command. Stopping ltid or by using the tldcd -t command. Debug Log: Errors are logged in the system log and robots debug log. Debug information is included by adding VERBOSE to the vm.conf file. On UNIX, debug information is also included by starting the daemon with the -v option (either by itself or through ltid). tlhd The Tape Library Half-inch daemon works in conjunction with tlhcd to handle requests to TLH robots that are in an IBM Automated Tape Library (ATL). tlhd provides the interface between the local ltid and the robotic control (tlhcd) in the same manner as explained previously for tl8d. Started By: Starting ltid (or on UNIX, independently by using the /usr/openv/volmgr/bin/tlhd command). Stopped By: Stopping ltid (or on UNIX, independently by finding the PID (process id) and then using the kill command). Debug Log: Errors are logged in the system log and robots debug log. Debug information is included by adding VERBOSE to the vm.conf file. On UNIX, debug information is also included by starting the daemon with the -v option (either by itself or through ltid).

320

Media and device management functional description Media and device management components

Table B-2

Media and device management daemons and programs (continued)

Program or daemon Description

tlhcd The Tape Library Half-inch Control daemon provides robotic control for a TLH robot that is in an IBM Automated Tape Library (ATL) in a similar manner to that which was explained previously for tl8cd. Started By: Starting ltid (or on UNIX, independently by using the /usr/openv/volmgr/bin/tlhcd command). Stopped By: Stopping ltid or by using the tlhcd -t command. Debug Log: Errors are logged in the system log and robots debug log. Debug information is included if the daemon is started with the -v option (either by itself or through ltid). The -v option is available only on UNIX. Also, add the VERBOSE option to the vm.conf file. tlmd The Tape Library Multimedia daemon is the interface between ltid and a TLM robot that is in an ADIC Distributed AML Server (DAS). This daemon communicates with the TLM robotics through a network API interface. Started By: Starting ltid or independently by using the /usr/openv/volmgr/bin/tlmd command. Stopped By: Stopping ltid or independently by finding the PID (process id) and then using the kill command. Debug Log: Errors are logged in the system log and robots debug log. Debug information is included if the daemon is started with the -v option (either by itself or through ltid). The -v option is available only on UNIX. Also, add the VERBOSE option to the vm.conf file.

Media and device management functional description Media and device management components

321

Table B-2

Media and device management daemons and programs (continued)

Program or daemon Description

tpconfig The Tape Library Multimedia daemon is the interface between ltid and a TLM robot that is in an ADIC Distributed AML Server (DAS). This daemon communicates with the TLM robotics through a network API interface. Started By: Starting ltid or independently by using the /usr/openv/volmgr/bin/tlmd command. Stopped By: Stopping ltid or independently by finding the PID (process id) and then using the kill command. Debug Log: Errors are logged in the system log and robots debug log. Debug information is included if the daemon is started with the -v option (either by itself or through ltid). The -v option is available only on UNIX. Also, add the VERBOSE option to the vm.conf file. tshd The Tape Stacker Half-inch daemon is the interface between ltid and the half-inch-cartridge stacker and communicates with the robotics through a SCSI interface. This robot is not supported on Windows. Started By: Starting ltid (or on UNIX, independently by using the /usr/openv/volmgr/bin/tshd command). Started By: tpconfig command. Stopped By: Quit option from within the utility on UNIX. On Windows, tpconfig is only a command-line interface that runs to completion (no quit option). Debug Log: tpcommand debug logs. vmd The Volume Manager daemon (NetBackup Volume Manager service on Windows) allows remote administration and control of Media and Device Management. vmd provides a proxy to EMM for pre-6.0 NetBackup servers. Started By: Starting ltid (or on UNIX, independently by using the Initiate Media Manager Volume Daemon option in vmadm) Stopped By: Terminate Media Manager Volume Daemon option in vmadm). Debug Log: System log and also a debug log if the daemon or reqlib debug directories exist

322

Media and device management functional description Media and device management components

Table B-2

Media and device management daemons and programs (continued)

Program or daemon Description

vmadm Available only on UNIX. An administrator utility with options for configuring and managing volumes under control of media and device management. It has a menu-driven, character-based interface that can be used from workstations that do not have graphical display capabilities. Started By: /usr/openv/volmgr/bin/vmadm command Stopped By: Quit option from within the utility. Debug Log: /usr/openv/volmgr/debug/reqlib vmscd The Media Manager Status Collector Daemon keeps the EMM server database up-to-date with the actual status of drives attached to 5.x servers. Started By: the EMM server. Stopped By: the EMM server. Debug Log: /usr/openv/volmgr/debug/vmscd (UNIX), install_path\Volmgr\debug\vmscd (Windows)

Index

A
acssel, description 315 acsssi, description 315 acstest 179 Adaptive Server Anywhere 71 admin log 134 admincmd directory 289 administration interface activity logging 150 errors 148 AdvancedDisk 185, 195 Alternate client restores host.xlate file 54 altnames file 304 application server status codes (Java interface) 149 archiving for NBCC 168 for nbsu 163 ascd, description 315 Auth User for PBX 78 auto-configuration problems 31 avrd, description 316

B
backup NetBackup catalogs 272 process files 254 multiplexing 260 NetWare clients 268 Windows clients 267 process overview 257, 265 snapshot overview 261 synthetic processes 269 UNIX clients 255 Bare Metal Restore 183, 185, 201 bin Media and Device Management 314 UNIX client 287, 289

BP 282 bp description 291 log 131 UNIX client log 129 bp.conf file 256 UNIX client/server 289 SERVER entries 94 bp.kill_all 8081 BP.NLM 268, 291 bp.start_all 81 bpadm description 291 bparchive description 291 log 129, 131 bpbackup description 292 log 129, 131 bpbackup log 131 BPBACKUP_POLICY 256 BPBACKUP_SCHED 256 bpbkar description 292 log 129, 131 bpbkar log 131 BPBKAR32 267, 292 bpbrm 263 description 292 bpbrm log 134 BPCD 282 bpcd description 293 server log 134 UNIX client log 129, 131 BPCD.NLM 293 BPCDW32.EXE 293 bpdbjobs description 293 bpdbjobs log 134

324

Index

bpdbm description 294 bpdbm log 134 bpdm description 294 bpdm log 134 bpdown command 80, 82, 197, 200 bpfis 263, 294 bphdb description 294 log 129 BPINETD 267, 281 bpinetd log 131 bpinetd.log 131 bpjava-msvc 295 bpjava-msvc log 135, 151 bpjava-usvc log 151 bplist description 295 log 129, 132 bplist log 132 bpmount log 129 bpmount log 132 bporaexp log 129 bporaexp64 log 129 bporaimp log 129 bporaimp64 log 129 bpps 25 bprd description 296 bprd log 135 bprestore description 296 log 129, 132 bprestore log 132 bpsched see also nbpem 300 bpsrv log 132 bpsrv log 132 BPSVR.NLM 296 bpsynth 270 BPSYS.EXE 296 bptm description 297 bptm log 135 bptpcinfo 99 bpup command 82, 197, 200

bundling NBCC output 168 nbsu output 163

C
catalog backup 272 class database file 304 client NetBackup configured name 52 debug logs. See UNIX clients. See Windows and NetWare clients installation problems 29 multiple hostnames 51 peername 52 software location. See UNIX clients testing configuration 35, 38 Client Properties dialog 70 client, NetBackup Windows disk recovery 201 CommandCentral Storage 9596 communications problems PC clients 47 UNIX clients 42 compression for NBCC 168 for nbsu 163 config file 304 configuration database 304 configuration problems 29

D
daemons robotic 305 robotic control 305 database backup (see catalog backup) 272 database extension 254 DAYS_TO_KEEP_LOGS vm.conf setting 139 db directory NetBackup 287, 289 debug level 143 debug logs 150 analysis utilities 154 NetBackup 314 vmd 136, 314 debug.properties file 151 debugging NBCC 167

Index

325

debugging (continued) nbsu 160 device configuration problems 31 Device Configuration Wizard 196 directory structure Media and Device Management 313 disaster recovery preparing for disaster 181 disk full 71 disk recovery Windows client 201 disk space for logs files 124 drive_mount_notify script 308 drive_unmount_notify script 308 driver directory 314 duplex mode and performance 93

functional overview (continued) Media and Device Management (continued) directories and files 313 volume management 307 NetBackup backup and archive 254 restores 274 startup 254

G
Global Logging Level 137 Global logging level 141142 goodies directory 289 goodies directory 314

H
Half duplex and poor performance 93 help files Media and Device Management 314 UNIX client 289 host name entries checking 55 Host Properties 70 host.xlate file 54 hostID unified logging 107

E
E-mail 184 EMM server 255 enable debug logging 136 Enable robust logging 140 Enterprise Media Manager 190 Enterprise Media Manager (EMM) 255 error database 304 Event viewer logging option 146 eventlog 146 file entries 147 exception errors in Java admin interface 148

I
ifconfig for checking NIC duplex mode 93 images database 304 inetd 29 Information E-mail 184 installation Linux 29 installation problems 28 ipconfig for checking NIC duplex mode 93

F
failure_history file 304 fibre channel 264 file database 304 files archive process 254 backup process 254 restore process 274 format directory 314 FSM 264 FT Service Manager 264 full disk 71 full duplex mode 93 functional overview introduction 254 Media and Device Management device management 307

J
Java interface debug logging 150 troubleshooting background 148 jbpSA overview 297 job ID search in unified logs 122

326

Index

jobs queued for long periods 71 jobs database 304

K
Keep logs For setting 116 Keep Logs setting 138

L
legacy logging 127 client logs 128 configuring rotation 140 controlling size of 138 directories 128 file name format 133 locations 127 PC clients 130 rotation of 138 levels for logging 141 Linux 29 log analysis utilities debug logs 154 limitations 157 output format 158 Log level Windows and NetWare clients 143 logging changing location of 114 levels 141 see legacy logging 127 setting level on PC clients 143 synthetic backup 144 logs debug enabling detailed 150 event viewer logging option 146 file retention 116 overview[Logs aaa] 101 PC client activity bp 131 bparchive 131 bpbackup 131 bpbkar 131 bpcd 131 bpinetd 131 bplist 132 bpmount 132

logs (continued) PC client activity (continued) bprestore 132 bpsrv 132 tar 132 user_ops 133 reports NetBackup 102 server activity acssi 136 admin 134 bpbrm 134 bpcd 134 bpdbjobs 134 bpdbm 134 bpdm 134 bpjava-susvc 135 bprd 135 bpsynth 135 bptm 135 daemon 136 ltid 136 nbatd 135 nbazd 135 nbjm 109 nbpem 108 reqlib 136 robots 136 syslogs 135 tpcommand 137 setting retention period 138 system 103 UNIX client activity bp 129 bparchive 129 bpbackup 129 bpbkar 129 bpcd 129 bphdb 129 bpjava-msvc 135 bplist 129 bpmount 129 bprestore 129 obackup_tape 130 tar 130 user_ops 130 logs directory UNIX client/server 289 ltid 138

Index

327

ltid, description 316

M
master server test procedure 35, 39 MaxLogFileSizeKB 124, 126127, 140 media database 304 media server test procedure 38 misc file 314 mklogdir.bat 128 moving log locations 114 multiplexed backups 260

N
name format legacy logging 133 NB_dbsrv daemon 71 nbatd log 135 nbaudit 298 nbazd log 135 NBCC archiving and compression 168 does the following 166 introduction 166 location of 166 nbcc-info.txt file 167 Notes on running 166 output 167 progress display 168 troubleshooting 167 when to use 166 nbcc-info.txt file 167 nbdb_move 196 nbemm 25, 255, 298 nbfdrv64 298 nbftclnt 264, 266, 280, 299 and bp.conf 94 nbftsrvr 264, 266, 280, 299 nbjm 25, 109, 255, 263, 270, 299300 NBNWNT 268, 282 NBNWNT.EXE 299 nbpem 25, 108, 255256, 263, 270, 300 nbproxy 300 nbrb 25, 71, 255, 300 nbrmms 301 nbstserv 301

nbsu and status codes 164 archiving and compression 163 bundling 163 creating xml output file 163 introduction 159 location of 159 nbsu_info.txt file 161 output files 161 progress display 164 troubleshooting 160 when to use 160 nbsu_info.txt file 161 NBWIN 267, 281 NBWIN.EXE 301 ndmpagent overview 300 NearStore 271 NetBackup if unresponsive 71 product ID 107 NetBackup Administration Console debug logging 150 errors 148 NetBackup Client Service start and stop 27 NetBackup consistency check see NBCC 166 NetBackup Database Manager service start and stop 27 NetBackup Device Manager service start and stop 27 NetBackup Enterprise Media Manager service start and stop 27 NetBackup Job Manager service start and stop 27 NetBackup Policy Execution Manager service start and stop 27 NetBackup Request Manager service start and stop 27 NetBackup Resource Broker service start and stop 27 NetBackup Status Collection daemon.. See vmscd NetBackup Support Utility see nbsu 159 NetBackup Volume Manager service start and stop 27 network connections multiple 51

328

Index

network daemon (vnetd) 136 network interface cards 93 network problems PC clients 47 UNIX clients 42 NIC cards and full duplex 93 NumberOfFiles 124 NumberOfLogFiles 127, 140

processes (see functional overview) 254 product ID for NetBackup 107 productID unified logging 107

Q
ql2300_stub 302 query string 118 queued jobs 71

O
obackup_tape log 130 odld, description 316 odltest 178 off-host backup 99 ogs server activity nbatd 108 OpenStorage 185, 195 operating system errors 149 originator IDs list of 108 originatorID unified logging 107

R
raw partitions backup process 254 restore process 274 recording information 14 recovery procedures Windows client disk 201 RedHat 29 relational database 71 reports NetBackup 102 reqlib directory 128 restore process 274 NetWare client 282 Windows 2000 client 281 retention of logs 116 robot drive selection 308 robotic control daemons 306 robotic daemons 306 robotic test utility 177 acstest 179 odltest 178 tl4test 178179 tl8test 178179 tldtest 178179 tlhtest 179 tshtest 178 robtest 178179 robust file logging 125 RolloverMode 127 rotation legacy logging 138 of logs 114 unified logging 107

P
patches (installing during recovery) 203 PBX Auth User 78 logging 79 Secure Mode 78, 80 starting 77 starting/stopping 80 troubleshooting 76 pbx_exchange 77, 302 pbxcfg 78 preliminary troubleshooting procedure 22 Private Branch Exchange (PBX) 76 procedures recovery Windows client disk 201 troubleshooting communications problems 42, 47 host names and services 55 installation and configuration 28 introduction 20 master server and clients 35 media server and clients 38 preliminary 22

S
SAN Client 264

Index

329

SAN client and bp.conf 94 SANPoint Control 95 Secure Mode for PBX 78 server installation problems 28 NetBackup debug logs 128 test procedure for master 35, 39 test procedure for media server 38 SERVER entries bp.conf 94 services entries checking 55 SharedDisk 185, 195 slow performance and NIC cards 93 snapshot backup process overview 262 software version determining UNIX client/server 290 starting NetBackup processes 81 startup NetBackup 254 status codes and nbsu 164 Status Collection Daemon 128 stderr 148 stdout 148 stopping NetBackup processes 8081 storage units 94 SuSE 29 synthetic backup 269 logs 144 syslogd 103 system logs 103

tl8test 178179 tldcd, description 319 tldd, description 318 tldtest 178179 tlhcd, description 320 tlhd, description 319 tlhtest 179 tlmd, description 320 tpautoconf 137, 188 tpconfig 137 tpconfig, overview 321 traceroute 54 troubleshooting procedure communication problems PC clients 47 UNIX clients 42 general master server and clients 35, 39 media server and clients 38 host name and services entries 55 installation 28 preliminary 22 try file 145 tshd, overview 321 tshtest 178

U
unavailable 94 unified logging 103 changing location of 114 client logs 128 configuring settings 125 controlling disk space usage 124 controlling number of log files 124 controlling size of 126 deleting logs 123 file name format 107 file rotation 114 format of files 117 listing settings 127 location 104 message types 106 NetBackup product ID 107 processes using 108 retention 116 setting level on PC clients 143 settings levels 141 submitting to Technical Support 105 tar log files 106

T
tar log 132 log files 106 NetBackup 302303 TAR32 281 test utility robotic 177 tl4d, description 317 tl4test 178179 tl8cd, description 318 tl8d, description 317

330

Index

upload directory 106 user-directed backups 256 user_ops log 130, 133, 135 utility robotic test 177

V
VERBOSE 138 verbose flag 138 VERBOSE level 142 vm.conf 138 vm.conf file 315 vmadm, overview 322 vmd 136 debug logging 136 overview 321 vmscd 128 logging 137 vmscd, overview 322 vnetd log 136 Volume Configuration Wizard 197 vxlogcfg 114, 140 vxlogcfg command 125, 127, 142 vxlogmgr command 122, 124 vxlogview command 117 query string overview 118 with job ID option 122 vxpbx_exchanged 80

W
Windows open file backup 262 windrvr6 303

X
xinetd 29 XML 129 xml for nbsu 163