Replacing PCIe cards in a 32xx system

To replace a PCIe card or riser in the system, you must perform a specific sequence of tasks.
About this task

You can use this procedure with all versions of Data ONTAP supported by your system.
In this procedure, a Cluster-Mode system refers to a system running Data ONTAP 8.x Cluster-Mode. A 7-Mode system
refers to a system running Data ONTAP 8.x 7-Mode or releases prior to Data ONTAP 8.0.
This procedure refers to HA pairs, which in releases prior to Data ONTAP 8.0 were called active/active configurations.
All other components in the system must be functioning properly; if not, you must contact technical support.

1.
2.
3.
4.
5.
6.
7.
8.

Shutting down the node on page 1

Opening the system on page 3
Removing a PCIe card on page 4
Installing a PCIe card on page 5
Reinstalling the controller module and booting the system on page 5
Running diagnostics on a PCIe card on page 6
Configuring newly added Ethernet ports for cluster or management use (Data ONTAP Cluster-Mode) on page 8
Completing the replacement process on page 8

Shutting down the node

You shut down a node using different procedures, depending on whether it is a stand-alone system or part of an HA pair.
Next topics

Shutting down a node in an HA pair on page 1

Shutting down the node in a stand-alone system on page 2

Shutting down a node in an HA pair

To shut down the node, you must determine the status of the node and, if necessary, take over the node so that the partner
continues to serve data from the node's storage.
About this task

As part of this procedure, you leave the power supplies or turn them off depending on your configuration:

If you have two controller modules in the same chassis, you must leave the power supplies turned on to provide power to the
partner node.
If you have one controller module in the chassis, but is part of an HA pair, you should turn off the power supplies in the
target node chassis.

Steps

1. Check the status of the target node (the node you want to perform maintenance on) by entering the following command at
the system console of either node:

215-05027_B0

Copyright 2011 NetApp, Inc. All rights reserved.

If your system is configured in...

Then issue this command...

7-Mode

cf status

Cluster-Mode

storage failover show

2. Take one of the following actions, depending on the result of the cf status or storage failover show command:
If...

Then...

Neither node is in takeover mode

Go to the next step in this procedure.

The partner node took over the

target node

The target node is in a state where you can begin removing it from the system chassis.

The target node took over the

partner node

a. Correct the problem that caused the takeover.

b. Run the cf giveback command (if in a 7-Mode system) or storage failover
giveback command (if in a Cluster-Mode system) from the target node console.
c. Go back to the beginning of this procedure.

3. Take over the target node by entering one of the following commands from the partner nodes console:
If your system is configured in...

Then issue this command...

7-Mode

cf takeover

Cluster-Mode

storage failover takeover -bynode node

4. Take one of the following actions depending on your system configuration:

If your system has...

Then...

Two controller modules in the chassis

Do not shut off the power supplies.

One controller module in the chassis

Turn off the power supplies and unplug them from the power sources.

Shutting down the node in a stand-alone system

For a node that is in a stand-alone configuration, you must perform a clean shutdown (ensuring that all data has been written to
disk) and disconnect the power supplies.
Steps

1. Enter one of the following commands from the system console:

If your system is configured in...

Then issue this command...

7-Mode

halt

Cluster-Mode

halt local

Attention: You must perform a clean system shutdown before replacing system components to avoid losing unwritten

data in the nonvolatile memory (NVMEM). The NVMEM LED is located on the controller module to the right of the
network ports, marked with a battery symbol. If the NVMEM LED is flashing, there is content in the NVMEM that has
not been saved to disk. You need to reboot the controller module and proceed from the beginning of this procedure. If
repeated attempts to cleanly shut down the controller module fail, be aware that you might lose any data that was not
saved to disk.

Replacing PCIe cards in a 32xx system

0a

c0a

0c

c0b

0d

e0a

0b

LNK

LNK

!
e0b

NVMEM LED

2. If you are not already grounded, properly ground yourself.

3. Turn off the power supplies and unplug both power cords from the power source:
If your system uses...

Then...

AC power supplies

Unplug the power cords from the power source, and then remove the power cords.

DC power supplies

Remove the power at the DC source, and remove the DC wires, if necessary.

Opening the system

To access components inside the controller module, you must open the system.
Steps

1. If you are not already grounded, properly ground yourself.

2. Unplug the system cables from the controller module, as needed, and keep track of where the cables were connected.
Leave the cables in the cable management arm so that when you reinstall the cable management arm in the controller
module, the cables will be organized.
3. Loosen the thumbscrew on the cam handle.
4. Pull the cam handle downward and slide the controller module out of the system.

Removing a PCIe card

To remove a PCIe card from the system, you must perform a specific sequence of steps.
Steps

1. Loosen the thumbscrew on the controller module side panel.

2. Swing the side panel open until it comes off the controller module.

1
2

1
2

Controller module side panel with thumbscrew

Controller module PCIe card

3. Remove the PCIe card from the controller module and set it aside.
4. Repeat these steps as needed.

Replacing PCIe cards in a 32xx system

Installing a PCIe card

To install a PCIe card in the system, you must perform a specific sequence of steps.
Steps

1. Open the controller module side panel, if necessary, slide off the PCIe card filler plate, as needed, and install the PCIe card.
Be sure that you properly align the card in the slot and exert even pressure on the card when seating it in the socket. The
adapter must be fully and evenly seated in the slot.
Note: If you are installing a card in the bottom slot and cannot see the card socket well, remove the top card so that you
can see the card socket, install the card, and the reinstall the card you removed from the top slot.

2. Repeat these steps as needed for additional PCIe cards.

3. Close the side panel and tighten the thumbscrew.

Reinstalling the controller module and booting the system

After you replace a component within the controller module, you must reinstall the controller module in the system chassis and
boot it to a state where you can run diagnostic tests on the replaced component.
About this task
Note: For HA pairs with two controller modules in the same chassis, the sequence in which you reinstall the controller
module is especially important because it attempts to reboot as soon as you completely seat it in the chassis.
Steps

1. Align the end of the controller module with the opening in the chassis, if necessary, and then gently push the controller
module halfway into the system.
2. Reinstall the cable management arms and recable the controller module, as needed.
When recabling, remember to reinstall the media converters (SFPs) if you are using fiber cables.
3. Reinstall the controller module:
a. Push the controller module all the way into the system.
b. Push the cam handle to finish seating the controller module.
c. Close the cam and tighten the thumbscrew.
d. Halt the boot process.

For systems in an HA pair, press Ctrl-c.

For stand-alone systems, reconnect the power cables to the power supplies and to the power sources, turn on the
power to start the boot process, and then press Ctrl-c.

The boot process is halted at either the Boot menu or the Loader prompt.

Running diagnostics on a PCIe card

After installing a PCIe card, you should run diagnostics.
Before you begin

Before running diagnostics tests on your PCIe card you need to have it cabled according to the following guidelines:

NIC interfaces must cabled be in a pair-wise manner, with adjacent ports connected together or connected through a switch.
Ports e0M and e0P must not be connected together because of the internal switch connection.
SAS card ports must be connected to storage or connected to the adjacent SAS ports.
If the SAS cards or systems with more than two ports are not connected to storage, they must be connected to ports A to B,
C to D, and so forth.
The FC card ports must be connected to storage or the ports must be terminated with loopback plugs.

Steps

1. Complete the applicable step, depending on where the node halted during the boot process.
If the node halted at the...

Then...

Loader prompt

Continue with the procedure.

Boot menu

a. Select the Maintenance mode option from the displayed menu.

b. Enter the following command at the prompt:
halt
c. Continue with the procedure.

2. Enter the following command at the Loader prompt:

boot_diags
Note: You must run this command from the Loader prompt for system-level diagnostics to function properly. The
boot_diags command starts special drivers designed specifically for system-level diagnostics.

The Maintenance mode prompt (*>) appears.

3. Enter the following command at the Maintenance mode prompt:
sldiag

For details about the sldiag command, see the sldiag man page.
4. Clear the status logs by entering the following command:
sldiag device clearstatus

5. Verify that the log is cleared by entering the following command:

sldiag device status

The following default response is displayed:

SLDIAG: No log messages are present.

6. View the types of devices available for checking by entering the following command:
sldiag device show

7. Run the test for the type of PCIe card you installed by entering the following command at the Maintenance mode prompt:
sldiag device run -dev fcal|sas|ata|nic|toe

Replacing PCIe cards in a 32xx system

See the System-Level Diagnostics Guide for a description of other types of tests you can run.
8. View the status of the test by entering the following command:
sldiag device status

Your storage system provides the following output while the tests are still running:
There are still test(s) being processed.

After all the tests are complete, the following response appears by default:
*> <SLDIAG:_ALL_TESTS_COMPLETED>

9. Verify that no hardware problems resulted from the addition or replacement of hardware components on your system by
entering the following command:
sldiag device status [-dev devtype] [-name device] -long -state failed

System-level diagnostics returns you to the prompt if there are no test failures, or lists the full status of failures resulting
from testing the component.
10. Proceed based on the result of the preceding step.
If the system-level
diagnostics tests...

Then...

Were completed
a. Clear the status logs by entering the following command:
without any failures
sldiag device clearstatus
b. Verify that the log is cleared by entering the following command:
sldiag device status
The following default response is displayed:
SLDIAG: No log messages are present.
c. Exit Maintenance mode by entering the following command:
halt
d. Enter the following command at the firmware prompt to reboot the storage system:
boot
e. If your system is in an HA pair, run the cf giveback command (if in a 7-Mode system) or storage
failover giveback command (if in a Cluster-Mode system) from the partner node console.
You have completed system-level diagnostics.
Resulted in some
test failures

Determine the cause of the problem.

a. Exit Maintenance mode by entering the following command:
halt
b. Perform a clean shutdown and disconnect the power supplies.
c. Verify that you have observed all the considerations identified for running system-level diagnostics, that cables
are securely connected, and that hardware components are properly installed in the storage system.
d. Reconnect the power supplies and power on the storage system.
e. Rerun the system-level diagnostics tests.

Related information

System-Level Diagnostics Guide: now.netapp.com/NOW/knowledge/docs/hardware/NetApp/diag/sldiag.pdf

Configuring newly added Ethernet ports for cluster or management use

(Data ONTAP Cluster-Mode)
When you install an Ethernet card in a Data ONTAP Cluster-Mode system, the ports are automatically configured as data ports.
To use the ports as cluster ports or management ports, you must manually reconfigure them.
Step

1. Enter the following command to change the port configuration:

network port modify -node node -port port-name -role port-role
port-role can be mgmt, cluster, or data.

For more information, see the Data ONTAP Cluster-Mode Network and File Access Management Guide on the NOW site.

Completing the replacement process

After you replace the part, you can return the failed part to NetApp, as described in the RMA instructions shipped with the kit.
Contact NetApp technical support at 888-463-8277 (North America), 00-800-44-NETAPP (Europe), or +800-800-80-800 (Asia/
Pacific) if you need the RMA number or additional help with the replacement procedure.

Trademark information
NetApp, the NetApp logo, Network Appliance, the Network Appliance logo, ApplianceWatch, ASUP, AutoSupport, Bycast,
Campaign Express, ComplianceClock, Cryptainer, CryptoShred, Data ONTAP, DataFabric, DataFort, Decru, Decru DataFort,
FAServer, FilerView, FlexCache, FlexClone, FlexScale, FlexShare, FlexSuite, FlexVol, FPolicy, GetSuccessful, gFiler, Go
further, faster, Imagine Virtually Anything, Lifetime Key Management, LockVault, Manage ONTAP, MetroCluster, MultiStore,
NearStore, NetCache, NOW (NetApp on the Web), ONTAPI, OpenKey, RAID-DP, ReplicatorX, SANscreen, SecureAdmin,
SecureShare, Select, Shadow Tape, Simulate ONTAP, SnapCopy, SnapDirector, SnapDrive, SnapFilter, SnapLock,
SnapManager, SnapMigrator, SnapMirror, SnapMover, SnapRestore, Snapshot, SnapSuite, SnapValidator, SnapVault,
StorageGRID, StoreVault, the StoreVault logo, SyncMirror, Tech OnTap, The evolution of storage, Topio, vFiler, VFM, Virtual
File Manager, VPolicy, WAFL, and Web Filer are trademarks or registered trademarks of NetApp, Inc. in the United States,
other countries, or both.
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in
the United States, other countries, or both. A complete and current list of other IBM trademarks is available on the Web at
www.ibm.com/legal/copytrade.shtml.
Apple is a registered trademark and QuickTime is a trademark of Apple, Inc. in the U.S.A. and/or other countries. Microsoft is a
registered trademark and Windows Media is a trademark of Microsoft Corporation in the U.S.A. and/or other countries.
RealAudio, RealNetworks, RealPlayer, RealSystem, RealText, and RealVideo are registered trademarks and RealMedia,
RealProxy, and SureStream are trademarks of RealNetworks, Inc. in the U.S.A. and/or other countries.
All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such.
NetApp, Inc. is a licensee of the CompactFlash and CF Logo trademarks.
NetApp, Inc. NetCache is certified RealSystem compatible.

Replacing PCIe cards in a 32xx system