ECS ™ Procedure Generator

Solution for Validating your engagement

ECS EX-Series Switch S5148 OS10 Upgrade Guide

Topic
ECS Miscellaneous 'How To' Service Procedures
Selections
Choose Activity: Update EX-Series Dell Switch S5148F OS10

Generated: July 7, 2022 5:43 PM GMT

REPORT PROBLEMS

If you find any errors in this procedure or have comments regarding this application, send email to
[email protected]

Copyright © 2022 Dell Inc. or its subsidiaries. All Rights Reserved. Dell Technologies, Dell, EMC, Dell
EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be
trademarks of their respective owners.

The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of
any kind with respect to the information in this publication, and specifically disclaims implied warranties of
merchantability or fitness for a particular purpose.

Use, copying, and distribution of any software described in this publication requires an applicable
software license.

This document may contain certain words that are not consistent with Dell's current language guidelines.
Dell plans to update the document over subsequent future releases to revise these words accordingly.

This document may contain language from third party content that is not under Dell's control and is not
consistent with Dell's current guidelines for Dell's own content. When such third party content is updated
by the relevant third parties, this document will be revised accordingly.

Publication Date: July, 2022

Dell Technologies Confidential Information version: 2.3.6.91

Page 1 of 42
 Contents
Preliminary Activity Tasks .......................................................................................................3
Read, understand, and perform these tasks.................................................................................................3

Dell EMC ECS EX-Series Switch OS10 Upgrade Guide (EX300, EX500, and EX3000 only)5

Dell Technologies Confidential Information version: 2.3.6.91

Page 2 of 42
Preliminary Activity Tasks
This section may contain tasks that you must complete before performing this procedure.

Read, understand, and perform these tasks

1. Table 1 lists tasks, cautions, warnings, notes, and/or knowledgebase (KB) solutions that you need to
be aware of before performing this activity. Read, understand, and when necessary perform any
tasks contained in this table and any tasks contained in any associated knowledgebase solution.

Table 1 List of cautions, warnings, notes, and/or KB solutions related to this activity

2. This is a link to the top trending service topics. These topics may or not be related to this activity.
This is merely a proactive attempt to make you aware of any KB articles that may be associated with
this product.

Note: There may not be any top trending service topics for this product at any given time.

ECS Top Service Topics

Dell Technologies Confidential Information version: 2.3.6.91

Page 3 of 42
Dell Technologies Confidential Information version: 2.3.6.91

Page 4 of 42
Dell EMC ECS EX-Series Switch OS10 Upgrade Guide (EX300,
EX500, and EX3000 only)

Note: The next section is an existing PDF document that is inserted into this procedure. You may see
two sets of page numbers because the existing PDF has its own page numbering. Page x of y on the
bottom will be the page number of the entire procedure.

Dell Technologies Confidential Information version: 2.3.6.91

Page 5 of 42
Dell EMC ECS EX-Series Switch OS10
Upgrade Guide (EX300, EX500, and EX3000
only)
10.3 and later

June 2021
Rev. 2.0

Page 6 of 42
 Contents

Chapter 1: Switch OS10 upgrade................................................................................................... 4

Overview................................................................................................................................................................................4
Pre-Requisites Required:................................................................................................................................................... 4
Caveats.................................................................................................................................................................................. 4
EX-Series Switch Resources............................................................................................................................................ 5
Connect a service laptop to the EX Series rack.................................................................................................... 5
Install OS10 Package..................................................................................................................................................... 7
Update All Switches (one at a time) from 10.3.2 or 10.4.3.6 to 10.4.3.6C1.....................................................9
Updating Multiple (2 or 4) Switches ...................................................................................................................... 19
Post Switch Upgrade Verification........................................................................................................................... 25

Appendix A: Appendix.................................................................................................................. 28
Next steps when an exception error is hit.................................................................................................................. 28
How to manually set VLT MAC......................................................................................................................................29
Downgrading one switch from 10.4.3.6C1 to 10.3.2. ................................................................................................ 31
Manual ONIE Install...........................................................................................................................................................34

2 Contents

Page 7 of 42
Notes, cautions, and warnings

NOTE: A NOTE indicates important information that helps you make better use of your product.

CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid
the problem.

WARNING: A WARNING indicates a potential for property damage, personal injury, or death.

© 2021 Dell Inc. or its subsidiaries. All rights reserved. Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other
trademarks may be trademarks of their respective owners.

Page 8 of 42
 1
Switch OS10 upgrade
This procedure describes how to upgrade switch OS10 on ECS EX Series (EX300, EX3000, and EX500 only).
Topics:
• Overview
• Pre-Requisites Required:
• Caveats
• EX-Series Switch Resources

Overview
ECS EX-Series Appliance implements Dell switches for Front-End (optional) and Back-End networks in a High Availability (HA -
two switches) configuration.
ECS solution implements "qualified and supported" versions of OS10 released by Dell networking. Not all versions of OS10
released by Dell Networking is supported on ECS EX-Series.
This guide covers the following use cases for OS10 update 10.3.2 or 10.4.3.6 to 10.4.3.6C1:
● Upgrading 2 or 4 switches in a rack - single switch at a time
○ Rack in production: One switch at a time 10.3.2 or 10.4.3.6 to 10.4.3.6C1
● Upgrading 2 or 4 switches - Multiple switches at a time
○ Rack not in production: Initial installation or VDC/rack in maintenance window.

Pre-Requisites Required:
This section describes the pre-requisites for Switch OS10 upgrade.

● Service Laptop (onsite)

● Serial DB–9 to RJ–45 connector adapter cable. (This cable is provided with every replacement switch.) If the service laptop
has no 9-pin connector, you will need a USB to serial port adapter. (onsite for troubleshooting - Optional)
● PuTTY and SCP tools installed on your service laptop. This software allows you to establish SSH terminal sessions and copy
firmware files from laptop to ECS nodes.
● Determine the service login user/password (default) on nodes and switches required engagement. See the ECS Service User
Access document for more information. If the CLI user or password is changed from the default, contact the customer for
details.
● ECS EX-Series Switch OS10 Package
○ Download the latest ECS EX-Series Switch OS10 Package from Dell EMC Support required for the tasks in this guide.
This package is not available by default in the local ECS Solve desktop repository.
○ See ECS EX-Series Switch OS10 Matrix, provided in this generated ECS Solve procedure, for the latest support versions
and compatibility for EX-Series.
● “updateDell” utility is provided with the ECS OS and the latest version in the ecs-os-firmware-dell-<version>.zip package.

Caveats
This section provides the warnings, deviations, or limitations of this procedure.
NOTE: OS10.4.3.6 has a certificate issue (Search for000185544 KB article in https://www.dell.com/support/kbdoc/en-in
to renew the certificate).
● IMPORTANT: On upgrade from OS10 10.3.2 to 10.4.3.6C1 when the primary switch is restarted, during VLT synchronization
with its partner switch there may be up to a 10 second network blackout for both switches in the HA pair.

4 Switch OS10 upgrade

Page 9 of 42
● IMPORTANT: If you are updating the switches remotely and a switch does not restart successfully, service personnel will
have to be dispatched to the site for on-site troubleshooting and remediation.
● IMPORTANT: For upgrade on ECS 3.3 and earlier versions, a maintenance window and maintenance mode is required. A node
may become unreachable through private interfaces. If this situation occurs, then reset the node using the iDRAC IP from
getrackinfo with:

ipmitool -H idracIP -U root -P passwd -I lanplus power reset

EX-Series Switch Resources

Table 1 provides information on EX-Series switch details which can be used to update switch OS10.

Switch Name Rack Location (Dell Dell Model Node network Default Configuration
EMC Rack) interfaces filename
Front-End Switches (Optional)

hare EX300: U22 S5148F-ON slave-1 ECSv3-hare-dell-

s5148-<version>.xml
(IP: 192.168.219.253) EX500: U22

(Hostname: hare.rack) U3000: U40

rabbit EX300: U20 S5148F-ON slave-0 ECSv3-rabbit-dell-

s5148-<version>.xml
(IP 192.168.219.252) EX500: U20
(Hostname: rabbit.rack) EX3000: U39

Back-End Switches (Required)

hound EX300: U18 S5148F-ON pslave-1 ECSv3-hound-dell-
s5148-<version>.xml
(IP 192.168.219.250) EX500: U18
(Hostname: EX3000: U21
hound.rack)

fox EX300: U16 S5148F-ON pslave-0 ECSv3-fox-dell-

s5148-<version>.xml
(IP 192.168.219.251) EX500: U16
(Hostname: fox.rack) EX3000: U22

Connect a service laptop to the EX Series rack

Access an ECS EX-Series rack using the private (192.168.219.XXX) network from a laptop.

Prerequisites
● Access to private network IP addresses (192.168.219.1 to 16 and 192.168.219.101 to 116) are limited to the nodes connected in
the rack backend 1/10/25GbE fox management switch.
● Private.4 (NAN) network IP addresses (169.254.x.x) of all nodes in all racks in the ECS Virtual Data Center (VDC) are
accessible from any node in the ECS VDC once you SSH in to a node using a private IP address (192.168.219.x).
● If security lock down is not enabled, access to public network IP addresses for all ECS racks are available once you SSH to
one of the ECS nodes .
● Two Switches Fox and Hound are used for the private network, or Nile Area Network (NAN). For example, node 8 must
connect to Hound port 8 and Fox port 8. For more information, see ECS EX Series Hardware Guide.

Steps
1. Connect your service laptop to the VDC.

Switch OS10 upgrade 5

Page 10 of 42
 Option Description
If the cabinet contains a service Open the service shelf, and connect the red network cable to the service laptop.
shelf with a red network cable...
The red cable connects to port 34 on the fox switch. The fox switch is the bottom
back-end switch in a dual switch configuration.

If the cabinet does not contain a From the rear of the rack, connect directly to either port 34 or 36 on the fox
service shelf with a red network switch, whichever port contains a 1GB SFP.
cable...
If you want to connect a service Locate port 36 on the fox switch. The fox switch is the bottom back-end switch in
laptop to the rear of the rack... a dual switch configuration.
Port 36 has a 1GB SFP that you can connect your service laptop to with a Cat6
cable.

Figure 1. Fox switch

1 - Port 34 for service tray connection

2 - Port 36 for connection from rear

2. Set the network interface on the laptop to the static address 192.168.219.99, subnet mask 255.255.255.0, with no gateway
required.
3. Verify that the temporary network between the laptop and rack's private management network is functioning by using the
ping command.
NOTE: If 192.168.219.1 does not answer, try 192.168.219.2. If neither responds, verify the laptop IP/subnet mask,
network connection, and switch port connection. If the service laptop is connected to Dell's VPN, ping to 192.168.219.x
may not return a response.

C:\>ping 192.168.219.1
Pinging 192.168.219.1 with 32 bytes of data:
Reply from 192.168.219.1: bytes=32 time<1ms TTL=64
Reply from 192.168.219.1: bytes=32 time<1ms TTL=64
Reply from 192.168.219.1: bytes=32 time<1ms TTL=64
Reply from 192.168.219.1: bytes=32 time<1ms TTL=64

Ping statistics for 192.168.219.1:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

6 Switch OS10 upgrade

Page 11 of 42
Install OS10 Package
This task verifies if latest OS10 version package is already installed on node 1. if not, provides steps to download and install the
OS10 package on node 1 of the rack.

Steps
1. If you are not connected, perform one of the following:
● On-Site: Establish a secure shell (SSH) session to node 1 (192.168.219.1) in the target rack.
● Remote: Establish a WebEx or SRS remote connection, then a secure shell (SSH) session to node 1 in the target rack.
2. If not already in a screen session, enter into one.

# screen -S OS10

To avoid screen session inactivity timeout, run:

# unset TMOUT

To verify if you are in a screen session:

# screen -list

There is a screen on:

178214.install (Attached) 1 Socket in /var/run/screen/S-admin.

If you lose the local or remote connection, the login sessions you establish through
the screen session don't go away. You can resume your screen sessions with the
following command:

# screen -dr <screen session name>

3. Execute the following commands to display Dell OS10 package loaded and if so the version. Refer to ECS EX-Series Switch
OS10 Matrix for package versions.

Examples:

admin :~> rpm -qa | grep emc-dell

emc-dell-DELLOS_10_3_xx-3.6-3955.cc9b92d9.x86_64
emc-dell-DELLOS_10_4_xx-3.6-3955.cc9b92d9.x86_64
emc-dell-firmware-3.6-3955.cc9b92d9.x86_64
emc-dell-DELLOS_10_5_xx-3.6-3955.cc9b92d9.x86_64

ls /usr/share/emc-dell-firmware/firmware/latest
PKGS_OS10-Enterprise-10.4.3.6C1.244stretch-installer-x86_64.bin
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

NOTE: The latest directory displays the last OS10 package installed on the server to be used for updating the switches.
4. If the version required is installed, go to step 12, else continue with the next step to load and install the required OS10
version.
5. Use SCP or WinSCP to copy the ecs-os-firmware-dell-<version>.zip file to the /home/admin directory on node 1 of the rack
containing the switches that require firmware update:

admin:~> scp ecs-os-firmware-dell-<version>.zip [email protected]:/home/admin

When prompted, enter the password for the ECS system admin user.

Output similar to the following displays:

ecs-os-firmware-dell-3.6.1.2-1590.d5b699f.61.zip 100% 1202MB 11.1MB/s 01:48

6. Unzip the ecs-os-firmware-<version>.zip to the /home/admin/ directory:

admin:~> unzip ecs-os-firmware-dell-3.6.1.2-1590.d5b699f.61.zip

Archive: ecs-os-firmware-dell-3.6.1.2-1590.d5b699f.61.zip
creating: 3.6.1.2-1590.d5b699f.61/
inflating: 3.6.1.2-1590.d5b699f.61/fabfit
inflating: 3.6.1.2-1590.d5b699f.61/VERSION.firmware
inflating: 3.6.1.2-1590.d5b699f.61/MD5SUMS.firmware

Switch OS10 upgrade 7

Page 12 of 42
 inflating: 3.6.1.2-1590.d5b699f.61/ecs-os-
setup.x86_64-3.6.1.2-1590.d5b699f.61.firmware.tbz

7. Change to the directory created in the previous step:

The following shows an example command. This command reflects the directory name created based on the latest available
firmware package for example

3.6.1.2-1590.d5b699f.61
admin:~> cd 3.6.1.2-1590.d5b699f.61

8. Create a target node file that includes the IP address of node 1 only:

admin:~> echo 192.168.219.1 > ~/FIRMWARE

9. Run the fabfit command to install the firmware files on node 1 in the rack: This install process takes a few minutes.

3.6.1.2-1590.d5b699f.61> ./fabfit -f ~/FIRMWARE install

20210602-195531 Info logging to /opt/emc/firmware.d/3.6.1.2-1590.d5b699f.61/log/
20210602-195531.push-status.log
20210602-195531 Info Checking checksums locally
20210602-195537 Info Distributing files to MACHINES list of nodes
20210602-195537 Info Make /opt/emc/firmware.d/3.6.1.2-1590.d5b699f.61 on MACHINES
list of nodes
20210602-195538 Info pushing /home/admin/kala/
3.6.1.2-1590.d5b699f.61/3.6.1.2-1590.d5b699f.61
20210602-195550 Info Running fabfit on MACHINES list of nodes

/opt/emc/bin/domulti -f /home/admin/FIRMWARE /opt/emc/firmware.d/

3.6.1.2-1590.d5b699f.61/fabfit install

192.168.219.1
========================================
20210602-195550 Info logging to /opt/emc/firmware.d/3.6.1.2-1590.d5b699f.61/log/
20210602-195550.local-status.log
20210602-195550 Info Obtaining lock...
20210602-195550 Info Obtained fabfit lock.
20210602-195550 Info Checking checksums of pushed files
20210602-195556 Info Expanding tarball in /opt/emc/firmware.d/
3.6.1.2-1590.d5b699f.61
20210602-195919 Info Upgrade: emc-dell-firmware-3.6-3967.ca5ad563.x86_64.rpm
emc-dell-DELLOS_10_3_xx-3.6-3967.ca5ad563.x86_64.rpm emc-
dell-DELLOS_10_4_xx-3.6-3967.ca5ad563.x86_64.rpm emc-dell-
DELLOS_10_5_xx-3.6-3967.ca5ad563.x86_64.rpm
/opt/emc/firmware.d/3.6.1.2-1590.d5b699f.61/fabfit: line 216: /etc/ecs-os-release: No
such file or directory

Done: running /opt/emc/bin/domulti -f /home/admin/FIRMWARE /opt/emc/firmware.d/

3.6.1.2-1590.d5b699f.61/fabfit install

10. Verify that the latest OS10 version is installed correctly:

a. Verify that the rpms are installed properly:

admin:~> rpm -qa | grep emc-dell

emc-dell-DELLOS_10_3_xx-3.6-3967.ca5ad563.x86_64
emc-dell-DELLOS_10_4_xx-3.6-3967.ca5ad563.x86_64
emc-dell-firmware-3.6-3967.ca5ad563.x86_64
emc-dell-DELLOS_10_5_xx-3.6-3967.ca5ad563.x86_64

b. Verify that the OS directories are installed:

admin:~> ls /usr/share/emc-dell-firmware/firmware

Example displays what version exist in the directory structure:

10.3.2.3 10.4.3.6 10.5.1.2 latest

8 Switch OS10 upgrade

Page 13 of 42
 NOTE: When OS10 is extracted, previous versions of OS10 and versions that are supported for other hardware
platforms are also listed.

admin:~> ls /usr/share/emc-dell-firmware/firmware/10.4.3.6
Example displays the latest versions loaded:
PKGS_OS10-Enterprise-10.4.3.6C1.244stretch-installer-x86_64.bin

NOTE: The Fabfit tool is designed to auto-detect hardware platform and place the correct OS10 in 10.4.3.6 directory
based on the switch model supported for the underlying hardware.
● For ECS 3.3 and below, fabfit places the correct OS10 in 10.4.3.6 directory and
● ECS 3.4 and above will have the correct OS10 in 10.4.3.6, as well as the latest directories based on the switch
models.
See ECS EX-Series Switch OS10 Matrix Guide to determine the latest supported Switch OS10 version for the target
switch models in your ECS rack and confirm that the latest directory has the correct version.
11. Clean up the firmware installation temporary files:

admin:~> viprexec -f ~/MACHINES 'rm -rf /opt/emc/firmware.d'

12. Verify target switch(s) version to assist in determining which switches require updating.
a. Execute the following command to display switch OS10 version.
Example of displaying “fox” switch OS10 version
● For ECS 3.3 and lower:

admin:~> ssh [email protected] 'show version | grep Version'

admin@fox's password: <Enter switch password here>

OS Version: 10.3.2E(R3)
Build Version: 10.3.2E(R3).169

● For ECS 3.4 and later:

admin@provo-auburn:~> ssh [email protected] 'show version | grep Version'

Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:
WARNING: Cluster manager is still using default credentials
OS Version: 10.4.3.6
Build Version: 10.4.3.6.244

b. Repeat the above step for each switch in the rack (hound, rabbit, hare) as required by changing the host target name.
Example: hound = [email protected]
13. Installation task of packages required for Switch OS update is completed. Based on implementation, go to either Update All
Switches one at a time or Update All Switches (2 or 4) at once tasks.

Update All Switches (one at a time) from 10.3.2 or 10.4.3.6 to

10.4.3.6C1
This task is used when the rack is in "production" and ONLY one switch OS10 is updated at a time to not cause any unavailability
in data access. This is based on updating switches in a "rack".

Prerequisites
NOTE: For upgrade on ECS 3.3 and earlier versions, a maintenance window and maintenance mode is required. A node
may become unreachable through private interfaces. If this situation occurs, then reset the node using the iDRAC IP from
getrackinfo with:

ipmitool -H idracIP -U root -P passwd -I lanplus power reset

Pre-Switch Update

Switch OS10 upgrade 9

Page 14 of 42
● If Front-End switch (rabbit and hare) being updated, verify with customer the uplinks to customer network are configured
for each switch and are healthy.
● If Front-End switch (rabbit and hare) being updated, notify the customer there may be up to a 10 second network outage
when the primary switch of the HA (High Availability) pair is restarted.
● Execute xDoctor to verify no node to switch interfaces (Front-End or Back-End) are down.
● Verify network interfaces bonding state
Switch OS 10 Update:
● Use updateDell utility to update target switch
Post successful switch update / reboot/ on-line with new OS10 version:
● Execute xDoctor to verify no node to switch interfaces (Front-End or Back-End) are down.
● Verify network interfaces bonding state.
● ssh to switch to execute commands to verify High Availability health.

About this task

WARNING: The procedures in this section are based on having the front end switches connected to the
customer switch in a high-availability (HA) manner (i.e. connection to both ECS front-end switches). If the
switches are NOT connected in HA, it is likely that a data unavailability event may occur during the upgrade
process. It is highly recommended to verify customer uplinks prior to proceeding with the switch OS upgrade.

Steps
1. While still in the existing screen session from previous section (connected to node 1 of target rack, execute xdoctor RAP025,
and RAP092 to validate both the Front-End (FE) and Back-End (BE).
NOTE: Before running this step, it is recommended to update xDoctor to latest available version.

admin:~> # sudo xdoctor --rap=RAP025,RAP092

Example:
admin:~> # sudo xdoctor --rap=RAP025,RAP092
2020-01-15 13:27:22,440: xDoctor_4.8-57 - INFO: Initializing xDoctor v4.8-57 ...
2020-01-15 13:27:23,382: xDoctor_4.8-57 - INFO: Removing orphaned session -
session_1579094724.085
2020-01-15 13:27:23,384: xDoctor_4.8-57 - INFO: Starting xDoctor
session_1579094842.267 ... (SYSTEM)
2020-01-15 13:27:23,384: xDoctor_4.8-57 - INFO: Master Control Check ...
2020-01-15 13:27:23,518: xDoctor_4.8-57 - INFO: xDoctor Composition -
Analyzer(s):ac_interface_check, ac_bond_check, ac_dell_switch
.
.
.

2020-01-15 13:28:01,550: xDoctor_4.8-57 - INFO: Analysis Limit: 0:08:00

2020-01-15 13:28:01,550: xDoctor_4.8-57 - INFO: ANALYZER [ac_interface_check]
2020-01-15 13:28:01,594: xDoctor_4.8-57 - INFO: ANALYZER [ac_bond_check]
2020-01-15 13:28:01,605: xDoctor_4.8-57 - INFO: ANALYZER [ac_dell_switch]
2020-01-15 13:28:01,617: xDoctor_4.8-57 - INFO: All data analyzed in 0:00:00
2020-01-15 13:28:01,620: xDoctor_4.8-57 - INFO: --------------------
2020-01-15 13:28:01,620: xDoctor_4.8-57 - INFO: Diagnosis Summary
2020-01-15 13:28:01,620: xDoctor_4.8-57 - INFO: --------------------
2020-01-15 13:28:01,620: xDoctor_4.8-57 - INFO: PSNT: FNM00181300308
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: --------------------
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of CRITICAL: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of FIXED: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of ERROR: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of WARNING: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of STATUS: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of VERBOSE: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of REPORT: 0
2020-01-15 13:28:10,009: xDoctor_4.8-57 - INFO: ----------------
.
.
.
2020-01-15 13:28:10,018: xDoctor_4.8-57 - INFO: --------------------------
2020-01-15 13:28:10,018: xDoctor_4.8-57 - INFO: Session Report - xdoctor --report --
archive=2020-01-15_132722

10 Switch OS10 upgrade

Page 15 of 42
 2020-01-15 13:28:10,018: xDoctor_4.8-57 - INFO: ---------------
2020-01-15 13:28:10,019: xDoctor_4.8-57 – INFO: Session Cleaner
2020-01-15 13:28:10,019: xDoctor_4.8-57 - INFO: ---------------
2020-01-15 13:28:10,029: xDoctor_4.8-57 - INFO: Nothing to cleanup
2020-01-15 13:28:10,030: xDoctor_4.8-57 - INFO: ------------------------------
2020-01-15 13:28:10,030: xDoctor_4.8-57 - INFO: xDoctor session_1579094842.267
finished in 0:00:48
2020-01-15 13:28:16,405: xDoctor_4.8-57 - INFO: Successful Job:1579094842 Exit
Code:128

2. Determine the report archive for the xDoctor session executed in the previous step.

admin:~> # sudo xdoctor -r | grep -a1 Latest

For example,

Latest Report:
xdoctor -r -a 2019-08-14_190658

3. View the latest xDoctor report using the output from the command in previous step. If no errors reported continue with next
step.
If the errors are on the FE or BE switch pair being updated the issues should be addressed prior to updating
OS10.
The following example shows a clean report with no events.

admin:~> # sudo xdoctor -r -a 2019-08-14_190658

Displaying xDoctor Report (2019-08-14_190658) Filter:['CRITICAL’, ERROR 'WARNING'] ...

The following example shows a report with an error on Node 3, slave-0 = Front-End rabbit switch network connection down.
If the firmware was being updated on hare this would cause a loss of access to node 3 via public interfaces.

admin:~> # sudo xdoctor -r -a 2019-08-14_190658

Displaying xDoctor Report (2019-08-14_190658) Filter:['CRITICAL', 'WARNING'] ...

Timestamp = 2019-08-14_190658
Category = platform
Source = ip show
Severity = CRITICAL
Node = 169.254.10.3
Message = One or more network interfaces are down or missing
Extra = {'169.254.10.3': ['slave-0']} << Node 3, rabbit switch connection
RAP = RAP025
Solution = 207810

The following example shows a report with an error on Node 4, pslave-1 = Back-End hound switch network connection
down. If the firmware was being updated on fox this would cause a loss of access to node 4 via private interface.

admin:~> xdoctor -r -a 2019-08-14_190658

Displaying xDoctor Report (2019-08-14_190658) Filter:['CRITICAL', 'WARNING'] ...

Timestamp = 2019-08-14_191606
Category = platform
Source = ip show
Severity = CRITICAL
Node = 169.254.10.4
Message = One or more network interfaces are down or missing
Extra = {'169.254.10.4': ['pslave-1']
RAP = RAP025
Solution = 207810

4. The Fabfit tool is designed to auto-detect hardware platform and place the correct OS10 in the 10.4.3.6 directory based on
the switch model supported for the underlying hardware. Confirm that the 10.4.3.6 directory has the correct version:

ls /usr/share/emc-dell-firmware/firmware/10.4.3.6
PKGS_OS10-Enterprise-10.4.3.6C1.244stretch-installer-x86_64.bin

Switch OS10 upgrade 11

Page 16 of 42
5. Execute the update from /home/admin directory

admin:~> cd /home/admin

6. Update OS10 on first target switch installed in the rack using updateDell with full domain name (i.e fox.rack) or IP
(192.168.219.x). Update should be performed in such manner so that pairs are upgraded in sequence:
a. # sudo updateDell hound.rack
● Check for this message after the update is done:

Upgrade completed on hound.rack. Run updateDell -r if both switches in pair are

upgraded.

b. # sudo updateDell fox.rack

● Check for this message after the update is done:

Upgrade completed on fox.rack. Run updateDell -r if both switches in pair are

upgraded.

c. # sudo updateDell -r hound.rack fox.rack (This restarts both, one at time)

d. # sudo updateDell -p hound.rack fox.rack (to apply post-upgrade configuration)

For example, if starting with hound, then fox should be upgrade next (to complete BE pair), and then rabbit followed by hare
(FE pair).
● For ECS software versions 3.3 and lower:

# sudo updateDell <Switch IP address or name> -f /usr/share/emc-dell-firmware/

firmware/10.4.3.6/PKGS_OS10-Enterprise-10.4.3.6C1.244stretch-installer-x86_64.bin

● For ECS software version 3.4:

# sudo updateDell <switch IP address or name>

● For ECS software versions 3.5 and later:

# getrackinfo -c /home/admin/MACHINES

# sudo updateDell <switch IP address or name> -m /home/admin/MACHINES

NOTE: This process will take up to 15 minutes and stages will look like nothing is happening.
● Image transfer fox/hound: 15 minutes
● Image transfer rabbit/hare: 8 minutes
● Image install any switch: 10 minutes

NOTE: A certificate is not installed on 10.3.2 and hence you see the message:

Started fwUpdate for hound.rack.

Skipping. 10.3.2.3 is not an applicable base version

Example of updating OS10 on “hound” switch running on ECS version 3.3 and lower:

admin:~> # sudo updateDell hound.rack -f /usr/share/emc-dell-firmware/firmware/

10.4.3.6/PKGS_OS10-Enterprise-10.4.3.6C1.244stretch-installer-x86_64.bin
Enter admin password for hound.rack switch:
Connected to hound.rack
Switch IP Model# Current Ver License Status Upgrade Needed
hound.rack S5148F-ON 10.3.2E(R3).169 Active True
Target firmware version: 10.4.3.6C1
Do you want to upgrade the firmware now? [y/N/q] y
Enter password for local ECS user (admin):
Started fwUpdate for hound.rack
Skipping. 10.3.2.3 is not an applicable base version
Starting download of startup-configuration
Download to startup-configuration.hound.rack.2021-05-27.2014.xml completed.
Starting download of running-configuration
Download to running-configuration.hound.rack.2021-05-27.2015.txt completed.
Starting download of license

12 Switch OS10 upgrade

Page 17 of 42
 Download to license.hound.rack.2021-05-27.2015.lic completed.
Transferring image to hound.rack.
Image transfer completed on hound.rack.
Image installing on hound.rack...
Install successfully completed on hound.rack.
Upgrade completed on hound.rack. Please run updateDell -r if both switches in pair
are upgraded.

NOTE: Post upgrade commands added to the switch configuration file will not run if a peer switch is still at old version.
This is expected and happens if you upgraded only the first switch of the BE or FE pair. Additional step in this procedure
will be executed later to initiate post upgrade add of the commands for configuration file for this target switch.
7. Proceed to update fox switch using one of the following commands based on ECS version.
● For ECS software versions 3.3 and lower:

# sudo updateDell <Switch IP address or name> -f /usr/share/emc-dell-firmware/

firmware/10.4.3.6/PKGS_OS10-Enterprise-10.4.3.6C1.244stretch-installer-x86_64.bin

● For ECS software version 3.4:

# sudo updateDell <switch IP address or name>

● For ECS software versions 3.5 and later:

# getrackinfo -c /home/admin/MACHINES

# sudo updateDell <switch IP address or name> -m /home/admin/MACHINES

NOTE: This process will take up to 15 minutes and stages will look like nothing is happening.
● Image transfer fox/hound: 15 minutes
● Image transfer rabbit/hare: 8 minutes
● Image install any switch: 10 minutes

NOTE: A certificate is not installed on 10.3.2 and hence you see the message:

Started fwUpdate for fox.rack

Skipping. 10.3.2.3 is not an applicable base version

Example of updating OS10 on “fox” switch running on ECS version 3.3 and lower:

admin:~> # sudo updateDell fox.rack -f /usr/share/emc-dell-firmware/firmware/10.4.3.6/

PKGS_OS10-Enterprise-10.4.3.6C1.244stretch-installer-x86_64.bin
Enter admin password for fox.rack switch:
Connected to fox.rack
Switch IP Model# Current Ver License Status Upgrade Needed
fox.rack S5148F-ON 10.3.2E(R3).169 Active True
Target firmware version: 10.4.3.6C1
Do you want to upgrade the firmware now? [y/N/q] y
Enter password for local ECS user (admin):
Started fwUpdate for fox.rack
Skipping. 10.3.2.3 is not an applicable base version
Starting download of startup-configuration
Download to startup-configuration.fox.rack.2021-05-27.2014.xml completed.
Starting download of running-configuration
Download to running-configuration.fox.rack.2021-05-27.2015.txt completed.
Starting download of license
Download to license.fox.rack.2021-05-27.2015.lic completed.
Transferring image to fox.rack.
Image transfer completed on fox.rack.
Image installing on fox.rack...
Install successfully completed on fox.rack.
Upgrade completed on fox.rack. Please run updateDell -r if both switches in pair are
upgraded.

8. Restart both switches with the command:

# sudo updateDell -r hound.rack fox.rack

Switch OS10 upgrade 13

Page 18 of 42
9. The switch will take 2 to 3 minutes to reload. Verify the switch has rebooted and is responding using the target switch
hostname or IP.

Example of ping command responding on hound.rack.

admin:~> ping hound.rack

PING hound.rack (192.168.219.250) 56(84) bytes of data.
64 bytes from hound.rack (192.168.219.250): icmp_seq=1 ttl=64 time=5.11 ms
64 bytes from hound.rack (192.168.219.250): icmp_seq=2 ttl=64 time=1.29 ms
64 bytes from hound.rack (192.168.219.250): icmp_seq=3 ttl=64 time=1.24 ms

<CTL C to exit>

--- hound.rack ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 1.245/2.551/5.118/1.815 ms

10. The switch will take 2 to 3 minutes to reload. Verify the switch has restarted and is responding using the target switch
hostname or IP.

Example of ping command responding on fox.rack.

admin:~> ping fox.rack

PING fox.rack (192.168.219.251) 56(84) bytes of data.
64 bytes from turtle.rack (192.168.219.251): icmp_seq=1 ttl=64 time=4.37 ms
64 bytes from turtle.rack (192.168.219.251): icmp_seq=2 ttl=64 time=5.24 ms
64 bytes from turtle.rack (192.168.219.251): icmp_seq=3 ttl=64 time=5.80 ms
64 bytes from turtle.rack (192.168.219.251): icmp_seq=4 ttl=64 time=4.13 ms

<CTL C to exit>

--- fox.rack ping statistics ---

4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 4.134/4.890/5.806/0.676 ms

NOTE: You may see “turtle.rack” in above output. It is expected (turtle is another alias for fox switch).
11. After completing upgrade of a switch pair (and both BE switches are at a new version, perform the following step to
complete post upgrade configuration on that switch:

admin:~> # sudo updateDell -p fox.rack hound.rack

# sudo updateDell -p fox.rack hound.rack -m /home/admin/MACHINES
Single password for all switches? [Y/n/q] Y
Enter admin password for Dell switch:
Connected to fox.rack
Connected to hound.rack
Switch IP Model# Current Ver License Status Upgrade Needed
fox.rack S5148F-ON 10.4.3.6C1.244 Active False
hound.rack S5148F-ON 10.4.3.6C1.244 Active False
Apply post upgrade config changes? [y/N/q] y
Finished sending config commands to fox.rack
Finished sending config commands to hound.rack

12. Verify state of bonded interfaces are good:

admin:~> viprexec "grep -e stat -e Interface /proc/net/bonding/*"

192.168.219.1
========================================
Last failed login: Fri Jul 26 03:52:12 UTC 2019 from 10.246.144.54 on ssh:notty
There were 8 failed login attempts since the last successful login.
/proc/net/bonding/private:Slave Interface: pslave-0
/proc/net/bonding/private: port state: 61
/proc/net/bonding/private: port state: 61
/proc/net/bonding/private:Slave Interface: pslave-1
/proc/net/bonding/private: port state: 61
/proc/net/bonding/private: port state: 61
/proc/net/bonding/public:Slave Interface: slave-0
/proc/net/bonding/public: port state: 61
/proc/net/bonding/public: port state: 61

14 Switch OS10 upgrade

Page 19 of 42
 /proc/net/bonding/public:Slave Interface: slave-1
/proc/net/bonding/public: port state: 61
/proc/net/bonding/public: port state: 61

.
.
.
.

192.168.219.16
========================================
/proc/net/bonding/private:Slave Interface: pslave-0
/proc/net/bonding/private: port state: 61
/proc/net/bonding/private: port state: 61
/proc/net/bonding/private:Slave Interface: pslave-1
/proc/net/bonding/private: port state: 61
/proc/net/bonding/private: port state: 61
/proc/net/bonding/public:Slave Interface: slave-0
/proc/net/bonding/public: port state: 61
/proc/net/bonding/public: port state: 61
/proc/net/bonding/public:Slave Interface: slave-1
/proc/net/bonding/public: port state: 61
/proc/net/bonding/public: port state: 61

NOTE: If “port state:” is not 61 on any interface, contact ECS Remote Support to resolve this issue before proceeding
further.
13. Check the VLT-MAC was set on all the switches
The tool will use the switch mac of fox for the fox and hound switch pairing.
The tool will use the switch mac of rabbit for the rabbit and hare switch pairing
a. Check the system MAC of fox

admin:~> ssh fox.rack "show system | grep MAC"

Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:
WARNING: Cluster manager is still using default credentials
MAC : d8:9e:f3:bf:01:00
Number of MACs : 256

b. Verify that both fox and hound both have the above MAC set for the VLT-MAC.

admin:~> ssh fox.rack " show running-configuration | grep vlt-mac"

Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:
WARNING: Cluster manager is still using default credentials
vlt-mac d8:9e:f3:bf:01:00

admin:~> ssh hound.rack " show running-configuration | grep vlt-mac"

Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:
WARNING: Cluster manager is still using default credentials
vlt-mac d8:9e:f3:bf:01:00

14. The BE switch pair update has been completed.

15. If the FE pair are S5148F-ON provided by Dell EMC, proceed to next step to update FE switches. Otherwise, go to Post
Upgrade Switch Verification.
16. Update rabbit front end switch. Update should be performed in such a manner so that pairs are upgraded in a sequence:
a. # sudo updateDell hare.rack
● Check for this message after the update is done:

Upgrade completed on fox.rack. Run updateDell -r if both switches in pair are

upgraded.

Switch OS10 upgrade 15

Page 20 of 42
 b. # sudo updateDell rabit.rack
● Check for this message after the update is done:

Upgrade completed on fox.rack. Run updateDell -r if both switches in pair are

upgraded.

c. # sudo updateDell -r hare.rack rabit.rack (This restarts both, one at time)

d. # sudo updateDell -p hare.rack fox.rack (to apply post-upgrade configuration)
● For ECS software versions 3.3 and lower:

# sudo updateDell <Switch IP address or name> -f /usr/share/emc-dell-firmware/

firmware/10.4.3.6/PKGS_OS10-Enterprise-10.4.3.6C1.244stretch-installer-x86_64.bin

● For ECS software version 3.4:

# sudo updateDell <switch IP address or name>

● For ECS software versions 3.5 and later:

# getrackinfo -c /home/admin/MACHINES

# sudo updateDell <switch IP address or name> -m /home/admin/MACHINES

NOTE: This process will take up to 15 minutes and stages will look like nothing is happening.
● Image transfer fox/hound: 15 minutes
● Image transfer rabbit/hare: 8 minutes
● Image install any switch: 10 minutes

NOTE: A certificate is not installed on 10.3.2 and hence you see the message:

Started fwUpdate for rabbit.rack.

Skipping. 10.3.2.3 is not an applicable base version

Example of updating OS10 on “rabbit” switch running on ECS version 3.3 and lower:

admin:~> # sudo updateDell rabbit.rack -f /usr/share/emc-dell-firmware/firmware/

10.4.3.6/PKGS_OS10-Enterprise-10.4.3.6C1.244stretch-installer-x86_64.bin
Enter admin password for rabbit.rack switch:
Connected to rabbit.rack
Switch IP Model# Current Ver License Status Upgrade Needed
rabbit.rack S5148F-ON 10.3.2E(R3).169 Active True
Target firmware version: 10.4.3.6C1
Do you want to upgrade the firmware now? [y/N/q] y
Enter password for local ECS user (admin):
Started fwUpdate for rabbit.rack
Skipping. 10.3.2.3 is not an applicable base version
Starting download of startup-configuration
Download to startup-configuration.rabbit.rack.2021-05-27.2014.xml completed.
Starting download of running-configuration
Download to running-configuration.rabbit.rack.2021-05-27.2015.txt completed.
Starting download of license
Download to license.rabbit.rack.2021-05-27.2015.lic completed.
Transferring image to rabbit.rack.
Image transfer completed on rabbit.rack.
Image installing on rabbit.rack...
Install successfully completed on rabbit.rack.
Upgrade completed on rabbit.rack. Please run updateDell -r if both switches in pair
are upgraded.

NOTE: Post upgrade commands added to the switch configuration file will not run if a peer switch is still at old version.
This is expected and happens if you upgraded only the first switch of the BE or FE pair. Additional step in this procedure
will be executed later to initiate post upgrade add of the commands for configuration file for this target switch.
17. Proceed to update hare switch.

16 Switch OS10 upgrade

Page 21 of 42
 ● For ECS software versions 3.3 and lower:

# sudo updateDell <Switch IP address or name> -f /usr/share/emc-dell-firmware/

firmware/10.4.3.6/PKGS_OS10-Enterprise-10.4.3.6C1.244stretch-installer-x86_64.bin

● For ECS software version 3.4:

# sudo updateDell <switch IP address or name>

● For ECS software versions 3.5 and later:

# getrackinfo -c /home/admin/MACHINES

# sudo updateDell <switch IP address or name> -m /home/admin/MACHINES

NOTE: This process will take up to 15 minutes and stages will look like nothing is happening.
● Image transfer fox/hound: 15 minutes
● Image transfer rabbit/hare: 8 minutes
● Image install any switch: 10 minutes

NOTE: A certificate is not installed on 10.3.2 and hence you see the message:

Started fwUpdate for hare.rack.

Skipping. 10.3.2.3 is not an applicable base version

Example of updating OS10 on “hare” switch running on ECS version 3.3 and lower:

admin:~> # sudo updateDell hare.rack -f /usr/share/emc-dell-firmware/firmware/

10.4.3.6/PKGS_OS10-Enterprise-10.4.3.6C1.244stretch-installer-x86_64.bin
Enter admin password for hare.rack switch:
Connected to hare.rack
Switch IP Model# Current Ver License Status Upgrade Needed
hare.rack S5148F-ON 10.3.2E(R3).169 Active True
Target firmware version: 10.4.3.6C1
Do you want to upgrade the firmware now? [y/N/q] y
Enter password for local ECS user (admin):
Started fwUpdate for hare.rack
Skipping. 10.3.2.3 is not an applicable base version
Starting download of startup-configuration
Download to startup-configuration.hare.rack.2021-05-27.2014.xml completed.
Starting download of running-configuration
Download to running-configuration.hare.rack.2021-05-27.2015.txt completed.
Starting download of license
Download to license.hare.rack.2021-05-27.2015.lic completed.
Transferring image to hare.rack.
Image transfer completed on hare.rack.
Image installing on hare.rack...
Install successfully completed on hare.rack.
Upgrade completed on hare.rack. Please run updateDell -r if both switches in pair are
upgraded

18. Restart both switches with the command:

# sudo updateDell -r rabbit.rack hare.rack

19. The switch will take 2 to 3 minutes to reload. Verify the switch has restarted and is responding using the target switch
hostname or IP.

Example of ping command responding on rabbit.rack.

admin:~> ping rabbit.rack

PING rabbit.rack (192.168.219.252) 56(84) bytes of data.
64 bytes from rabbit.rack (192.168.219.252): icmp_seq=1 ttl=64 time=0.107 ms
64 bytes from rabbit.rack (192.168.219.252): icmp_seq=2 ttl=64 time=0.100 ms
64 bytes from rabbit.rack (192.168.219.252): icmp_seq=3 ttl=64 time=0.097 ms
64 bytes from rabbit.rack (192.168.219.252): icmp_seq=4 ttl=64 time=0.099 ms

Switch OS10 upgrade 17

Page 22 of 42
 <CTL C to exit>

--- rabbit.rack ping statistics ---

4 packets transmitted, 4 received, 0% packet loss, time 2997ms
rtt min/avg/max/mdev = 0.097/0.100/0.107/0.012 ms

20. The switch will take 2 to 3 minutes to reload. Verify the switch has restarted and is responding using the target switch
hostname or IP.

Example of ping command responding on hare.rack.

admin:~> ping hare.rack

PING hare.rack (192.168.219.253) 56(84) bytes of data.
64 bytes from hare.rack (192.168.219.253): icmp_seq=1 ttl=64 time=0.100 ms
64 bytes from hare.rack (192.168.219.253): icmp_seq=2 ttl=64 time=0.086 ms
64 bytes from hare.rack (192.168.219.253): icmp_seq=3 ttl=64 time=0.104 ms
64 bytes from hare.rack (192.168.219.253): icmp_seq=4 ttl=64 time=0.095 ms

<CTL C to exit>

--- hare.rack ping statistics ---

4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.086/0.096/0.104/0.009 ms

21. After completing upgrade of a switch pair (and both FE switches are at a new version, perform the following step against
the both switch of the pair (rabbit), to complete adding configuration file commands post upgrade configuration on that
switch:

admin:~> # sudo updateDell -p rabbit.rack hare.rack

# sudo updateDell -p rabbit.rack hare.rack -m /home/admin/MACHINES
Single password for all switches? [Y/n/q] Y
Enter admin password for Dell switch:
Connected to rabbit.rack
Connected to hare.rack
Switch IP Model# Current Ver License Status Upgrade Needed
fox.rack S5148F-ON 10.4.3.6C1.244 Active False
hound.rack S5148F-ON 10.4.3.6C1.244 Active False
Apply post upgrade config changes? [y/N/q] y
Finished sending config commands to rabbit.rack
Finished sending config commands to hare.rack

22. Verify state of bonded interfaces are good:

admin:~> viprexec "grep -e stat -e Interface /proc/net/bonding/*"

192.168.219.1
========================================
Last failed login: Fri Jul 26 03:52:12 UTC 2019 from 10.246.144.54 on ssh:notty
There were 8 failed login attempts since the last successful login.
/proc/net/bonding/private:Slave Interface: pslave-0
/proc/net/bonding/private: port state: 61
/proc/net/bonding/private: port state: 61
/proc/net/bonding/private:Slave Interface: pslave-1
/proc/net/bonding/private: port state: 61
/proc/net/bonding/private: port state: 61
/proc/net/bonding/public:Slave Interface: slave-0
/proc/net/bonding/public: port state: 61
/proc/net/bonding/public: port state: 61
/proc/net/bonding/public:Slave Interface: slave-1
/proc/net/bonding/public: port state: 61
/proc/net/bonding/public: port state: 61

.
.
.
.
.

192.168.219.16
========================================
/proc/net/bonding/private:Slave Interface: pslave-0

18 Switch OS10 upgrade

Page 23 of 42
 /proc/net/bonding/private: port state: 61
/proc/net/bonding/private: port state: 61
/proc/net/bonding/private:Slave Interface: pslave-1
/proc/net/bonding/private: port state: 61
/proc/net/bonding/private: port state: 61
/proc/net/bonding/public:Slave Interface: slave-0
/proc/net/bonding/public: port state: 61
/proc/net/bonding/public: port state: 61
/proc/net/bonding/public:Slave Interface: slave-1
/proc/net/bonding/public: port state: 61
/proc/net/bonding/public: port state: 61

NOTE: If “port state:” is not 61 on any interface, contact ECS Remote Support to resolve this issue before proceeding
further.
23. Check the VLT-MAC was set on all the switches
The tool will use the switch mac of fox for the fox and hound switch pairing.
The tool will use the switch mac of rabbit for the rabbit and hare switch pairing.
a. Check the system MAC of rabbit:

admin@provo-manatee:~> ssh rabbit.rack "show system | grep MAC"

Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:
WARNING: Cluster manager is still using default credentials
MAC : d8:9e:f3:c9:dd:00
Number of MACs : 256

b. Verify that both fox and hound both have the above MAC set for the VLT-MAC.

admin:~> ssh rabbit.rack " show running-configuration | grep vlt-mac"

Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:
WARNING: Cluster manager is still using default credentials
vlt-mac d8:9e:f3:c9:dd:00

admin:~> ssh hare.rack " show running-configuration | grep vlt-mac"

Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:
WARNING: Cluster manager is still using default credentials
vlt-mac d8:9e:f3:c9:dd:00
ssh hare.rack " show running-configuration | grep vlt-mac"
Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:
WARNING: Cluster manager is still using default credentials
vlt-mac d8:9e:f3:c9:dd:00

24. Go to Post Switch Upgrade section.

Updating Multiple (2 or 4) Switches

This section helps you update multiple switches.

Prerequisites
Pre-Switch Update
● If Front-End switch (rabbit and hare) being updated, verify with customer the uplinks to customer network are configured
for each switch and are healthy.
● Execute xDoctor to verify no node to switch interfaces (Front-End or Back-End) are down.
● Verify network interfaces bonding state

Switch OS10 upgrade 19

Page 24 of 42
Switch OS 10 Update:
● Use updateDell utility to update target switches.
Post successful switch update / reboot/ on-line with new OS10 version:
● Execute xDoctor to verify no node to switch interfaces (Front-End or Back-End) are down.
● Verify network interfaces bonding state
● ssh to switch to execute commands to verify High Availability health.
NOTE: For upgrade on ECS 3.3 and earlier versions, a maintenance window and maintenance mode is required. A node
may become unreachable through private interfaces. If this situation occurs, then reset the node using the iDRAC IP from
getrackinfo with:

ipmitool -H idracIP -U root -P passwd -I lanplus power reset

About this task

This Task is used when the rack is NOT in "production", enabling OS10 to be updated on ALL switches (2 or 4) and will cause
unavailability in data access if FE (rabbit/hare) are being updated. This is based on updating switches in a "rack".

Steps
1. While still in the existing screen session from previous section (connected to node 1 of target rack, execute xdoctor to
validate both the Front-End (FE) and Back-End (BE).
NOTE: Before running this step, it is recommended to update xDoctor to latest available version.

Example:
admin:~> sudo xdoctor --rap=RAP025,RAP092
2019-08-14 18:27:16,534: xDoctor_4.7-52 - INFO : Initializing xDoctor v4.7-52 ...
2019-08-14 18:27:17,224: xDoctor_4.7-52 - INFO : Removing orphaned session -
session_1565806279.817
2019-08-14 18:27:17,226: xDoctor_4.7-52 - INFO : Removing orphaned session -
session_1565806424.040
2019-08-14 18:27:17,227: xDoctor_4.7-52 - INFO : Starting xDoctor
session_1565807236.398 ... (SYSTEM)
2019-08-14 18:27:17,227: xDoctor_4.7-52 - INFO : Master Control Check ...
2019-08-14 18:27:17,359: xDoctor_4.7-52 - INFO : xDoctor Composition -
Analyzer(s):ac_interface_check, ac_network
2019-08-14 18:27:17,360: xDoctor_4.7-52 - INFO : Session limited to 0:40:00
.
.
2019-08-14 18:27:36,988: xDoctor_4.7-52 - INFO : Number of CRITICAL: 1
2019-08-14 18:27:36,988: xDoctor_4.7-52 - INFO : Number of FIXED: 0
2019-08-14 18:27:36,988: xDoctor_4.7-52 - INFO : Number of ERROR: 0
2019-08-14 18:27:36,988: xDoctor_4.7-52 - INFO : Number of WARNING: 0
2019-08-14 18:27:36,988: xDoctor_4.7-52 - INFO : Number of STATUS: 0
2019-08-14 18:27:36,988: xDoctor_4.7-52 - INFO : Number of VERBOSE: 0
2019-08-14 18:27:36,988: xDoctor_4.7-52 - INFO : Number of REPORT: 0
2019-08-14 18:27:36,996: xDoctor_4.7-52 - INFO : ---------------------
.
.
2019-08-14 18:27:39,967: xDoctor_4.7-52 - INFO : --------------------------
2019-08-14 18:27:39,967: xDoctor_4.7-52 - INFO : Session Report -
xdoctor --report --archive=2019-08-14_182716
2019-08-14 18:27:39,967: xDoctor_4.7-52 - INFO : ---------------
2019-08-14 18:27:39,967: xDoctor_4.7-52 - INFO : Session Cleaner
2019-08-14 18:27:39,967: xDoctor_4.7-52 - INFO : ---------------
2019-08-14 18:27:39,984: xDoctor_4.7-52 - INFO : Nothing to cleanup
2019-08-14 18:27:39,985: xDoctor_4.7-52 - INFO : ------------------------------
2019-08-14 18:27:39,985: xDoctor_4.7-52 - INFO : xDoctor session_1565807236.398
finished in 0:00:23
2019-08-14 18:27:42,916: xDoctor_4.7-52 - INFO : Successful Job:1565807236 Exit
Code:132

20 Switch OS10 upgrade

Page 25 of 42
2. Determine the report archive for the xDoctor session executed in the previous step.

admin:~> sudo xdoctor -r | grep -a1 Latest

For example,

Latest Report:
xdoctor -r -a 2019-08-14_190658

3. View the latest xDoctor report using the output from the command in previous step. If no errors reported continue with next
step. If the errors are on the FE or BE switch pair being updated the issues should be addressed prior to updating OS10.
The following example shows a clean report with no events.

admin:~> sudo xdoctor -r -a 2019-08-14_190658

Displaying xDoctor Report (2019-08-14_190658) Filter:['CRITICAL’, 'WARNING'] ...

The following example shows a report with an error on Node 3, slave-0 = Front-End rabbit switch network connection down.
If the firmware was being updated on hare this would cause a loss of access to node 3 via public interfaces.

admin:~> xdoctor -r -a 2019-08-14_190658

Displaying xDoctor Report (2019-08-14_190658) Filter:['CRITICAL', 'WARNING'] ...

Timestamp = 2019-08-14_190658
Category = platform
Source = ip show
Severity = CRITICAL
Node = 169.254.10.3
Message = One or more network interfaces are down or missing
Extra = {'169.254.10.3': ['slave-0']} << Node 3, rabbit switch connection
RAP = RAP025
Solution = 207810

The following example shows a report with an error on Node 4, pslave-1 = Back-End hound switch network connection
down. If the firmware was being updated on fox this would cause a loss of access to node 4 via private interfaces.

admin:~> xdoctor -r -a 2019-08-14_190658

Displaying xDoctor Report (2019-08-14_190658) Filter:['CRITICAL', 'WARNING'] ...

Timestamp = 2019-08-14_191606
Category = platform
Source = ip show
Severity = CRITICAL
Node = 169.254.10.4
Message = One or more network interfaces are down or missing
Extra = {'169.254.10.4': ['pslave-1']
RAP = RAP025
Solution = 207810

4. Execute the update from /home/admin directory

admin:~> cd /home/admin

5. Update OS10 on ALL (2 or 4) Dell switches installed in the rack with one updateDell execution using the hostnames or IPs.
● For ECS software versions 3.3 and lower:

# sudo updateDell <Switch IP address or name> -f /usr/share/emc-dell-firmware/

firmware/10.4.3.6/PKGS_OS10-Enterprise-10.4.3.6C1.244stretch-installer-x86_64.bin

● For ECS software version 3.4:

# sudo updateDell <switch IP address or name>

Switch OS10 upgrade 21

Page 26 of 42
 ● For ECS software versions 3.5 and later:

# getrackinfo -c /home/admin/MACHINES

# sudo updateDell <switch IP address or name> -m /home/admin/MACHINES

NOTE: Enter multiple entries by leaving a space in between IPs or names. UpdateDell utility updates and restarts one
switch at a time when multiple switches are provided. At stages in the process will look like nothing is happening.

NOTE: This process will take up to 15 minutes and stages will look like nothing is happening:
● Image transfer fox/hound: 15 minutes
● Image transfer rabbit/hare: 8 minutes
● Image install any switch: 10 minutes
● It is possible that an error exception, like shown below, may occur while upgrading
● Print "DEBUG [%s] %s" % (type(err), err)
● Print "Image transfer started but unable to determine its status."
● Print "Please manually check image status on %s" % self.host
● The utility is having difficulty checking image transfer of the file. If this occurs, continue on and then refer to the
Appendix on next steps for an error exception switch run.
When updating multiple switches, the questions and output vary as compared to when one switch at a time is updated:
a. Type Y, when asked for single password for all switches.
b. Enter password when asked.
c. Type Y, when asked if you would like to update the firmware on each switch.
d. Type Y, when asked to restart each switch.

admin@provo-auburn:~> sudo updateDell rabbit.rack hound.rack fox.rack hare.rack

Single password for all switches? [Y/n/q] Y
Enter admin password for Dell switch:
Connected to rabbit.rack
Connected to hound.rack
Connected to fox.rack
Connected to hare.rack
Switch IP Model# Current Ver License Status Upgrade Needed
rabbit.rack S5148F-ON 10.4.3.6.244 Active True
hound.rack S5148F-ON 10.4.3.6.244 Active True
fox.rack S5148F-ON 10.4.3.6.244 Active True
hare.rack S5148F-ON 10.4.3.6.244 Active True
Target firmware version: 10.4.3.6C1
Do you want to upgrade the firmware now? [y/N/q] y
Enter password for local ECS user (admin):
Started fwUpdate for rabbit.rack
Started fwUpdate for hound.rack
Started fwUpdate for fox.rack
Started fwUpdate for hare.rack
Transferring file to switch, please wait..
Transferring file to switch, please wait..
Transferring file to switch, please wait..
Transferring file to switch, please wait..
Upload from /usr/share/emc-dell-firmware/config/newdell.crt completed.
Upload from /usr/share/emc-dell-firmware/config/newdell.crt completed.
Upload from /usr/share/emc-dell-firmware/config/newdell.crt completed.
Upload from /usr/share/emc-dell-firmware/config/newdell.crt completed.
Starting download of startup-configuration
Starting download of startup-configuration
Starting download of startup-configuration
Starting download of startup-configuration
Download to startup-configuration.rabbit.rack.2021-06-02.2047.xml completed.
Download to startup-configuration.hare.rack.2021-06-02.2047.xml completed.
Download to startup-configuration.hound.rack.2021-06-02.2047.xml completed.
Starting download of running-configuration
Starting download of running-configuration
Download to startup-configuration.fox.rack.2021-06-02.2047.xml completed.
Starting download of running-configuration
Starting download of running-configuration

22 Switch OS10 upgrade

Page 27 of 42
 Download to running-configuration.rabbit.rack.2021-06-02.2047.txt completed.
Download to running-configuration.hare.rack.2021-06-02.2047.txt completed.
Download to running-configuration.hound.rack.2021-06-02.2047.txt completed.
Starting download of license
Starting download of license
Download to running-configuration.fox.rack.2021-06-02.2047.txt completed.
Starting download of license
Starting download of license
Download to license.rabbit.rack.2021-06-02.2048.lic completed.
Download to license.hare.rack.2021-06-02.2048.lic completed.
Download to license.hound.rack.2021-06-02.2048.lic completed.
Download to license.fox.rack.2021-06-02.2048.lic completed.
Transferring image to rabbit.rack.
Transferring image to hare.rack.
Transferring image to hound.rack.
Transferring image to fox.rack.
Image transfer completed on rabbit.rack.
Image transfer completed on hare.rack.
Image installing on rabbit.rack...
Image installing on hare.rack...
Install successfully completed on rabbit.rack.
Install successfully completed on hare.rack.
Image transfer completed on hound.rack.
Image transfer completed on fox.rack.
Image installing on hound.rack...
Image installing on fox.rack...
Install successfully completed on hound.rack.
Install successfully completed on fox.rack.
Upgrade completed on hare.rack. Please run updateDell -r if both switches in pair are
upgraded.
Upgrade completed on rabbit.rack. Please run updateDell -r if both switches in pair
are upgraded.
Upgrade completed on hound.rack. Please run updateDell -r if both switches in pair
are upgraded.
Upgrade completed on fox.rack. Please run updateDell -r if both switches in pair are
upgraded.

6. Reload or restart each switch at a time:

updateDell -r rabbit.rack
Absolute path to 'updateDell' is '/usr/sbin/updateDell', so running it may require
superuser privileges (eg. root).
admin@provo-auburn:~> sudo updateDell -r rabbit.rack
Enter admin password for rabbit.rack switch:
Connected to rabbit.rack
Switch IP Model# Current Ver License Status Upgrade Needed
rabbit.rack S5148F-ON 10.4.3.6.244 Active True
Check public on 192.168.219.1
Check public on 192.168.219.2
Check public on 192.168.219.3
Check public on 192.168.219.4
No unsaved configs found on rabbit.rack
Reloading rabbit.rack
admin@provo-auburn:~> sudo updateDell -r fox.rack
Enter admin password for fox.rack switch:
Connected to fox.rack
Switch IP Model# Current Ver License Status Upgrade Needed
fox.rack S5148F-ON 10.4.3.6.244 Active True
Check private on 192.168.219.1
Check private on 192.168.219.2
Check private on 192.168.219.3
Check private on 192.168.219.4
No unsaved configs found on fox.rack
Reloading fox.rack
admin@provo-auburn:~> sudo updateDell -r hound.rack
Enter admin password for hound.rack switch:
Connected to hound.rack
Switch IP Model# Current Ver License Status Upgrade Needed
hound.rack S5148F-ON 10.4.3.6.244 Active True
Check private on 192.168.219.1
Check private on 192.168.219.2
Check private on 192.168.219.3

Switch OS10 upgrade 23

Page 28 of 42
 Check private on 192.168.219.4
No unsaved configs found on hound.rack
Reloading hound.rack
admin@provo-auburn:~> sudo updateDell -r hare.rack
Enter admin password for hare.rack switch:
Connected to hare.rack
Switch IP Model# Current Ver License Status Upgrade Needed
hare.rack S5148F-ON 10.4.3.6.244 Active True
Check public on 192.168.219.1
Check public on 192.168.219.2
Check public on 192.168.219.3
Check public on 192.168.219.4
No unsaved configs found on hare.rack
Reloading hare.rack

7. Check the VLT-MAC was set on all the switches.

The tool will use the switch mac of fox for the fox and hound switch pairing.
The tool will use the switch mac of rabbit for the rabbit and hare switch pairing.
a. Check the system MAC of fox

admin:~> ssh fox.rack "show system | grep MAC"

Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:
WARNING: Cluster manager is still using default credentials
MAC : d8:9e:f3:bf:01:00
Number of MACs : 256

b. Verify that both fox and hound both have the above MAC set for the VLT-MAC.

admin:~> ssh fox.rack " show running-configuration | grep vlt-mac"

Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:
WARNING: Cluster manager is still using default credentials
vlt-mac d8:9e:f3:bf:01:00

admin:~> ssh hound.rack " show running-configuration | grep vlt-mac"

Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:
WARNING: Cluster manager is still using default credentials
vlt-mac d8:9e:f3:bf:01:00

c. Check the system MAC of rabbit

admin@provo-manatee:~> ssh rabbit.rack "show system | grep MAC"

Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:
WARNING: Cluster manager is still using default credentials
MAC : d8:9e:f3:c9:dd:00
Number of MACs : 256

d. Verify that both rabbit and hare both have the above MAC set for the VLT-MAC.

admin:~> ssh rabbit.rack "show running-configuration | grep vlt-mac"

Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:
WARNING: Cluster manager is still using default credentials
vlt-mac d8:9e:f3:c9:dd:00

admin:~> ssh hare.rack "show running-configuration | grep vlt-mac"

Debian GNU/Linux 9

24 Switch OS10 upgrade

Page 29 of 42
 Dell EMC Networking Operating System (OS10)
[email protected]'s password:
WARNING: Cluster manager is still using default credentials
vlt-mac d8:9e:f3:c9:dd:00

8. Go to Post Switch Upgrade section.

Post Switch Upgrade Verification

This task verifies the switch upgrade.

Steps
1. If not connected to node 1, Establish a secure shell (SSH) session to node 1 (192.168.219.1) in the target rack.
2. Verify the switches which were updated successfully
NOTE: Execute command below against all switches that have been upgraded in the rack

a. Type Y, when asked for single password for all switches.

b. Enter password when asked

admin:~> sudo updateDell fox.rack hound.rack rabbit.rack hare.rack

Single password for all switches? [Y/n/q] Y
Enter admin password for Dell switch: <Enter switch admin user password>
Conneceted to fox.rack
Conneceted to hound.rack
Conneceted to rabbit.rack
Conneceted to hare.rack
Switch IP Model# Current Ver License Status Upgrade Needed
fox.rack S5148F-ON 10.4.3.6C1.244 Active False
hound.rack S5148F-ON 10.4.3.6C1.244 Active False
rabbit.rack S5148F-ON 10.4.3.6C1.244 Active False
hare.rack S5148F-ON 10.4.3.6C1.244 Active False

3. Ensure the SSH Cipher has been reset to default state.

admin:~> sudo reduce_sshd_mac -d

4. Clear xDoctor cache before you remove any stale failure information. Run the command:

# sudo xdoctor --clear

You are about to clear all xDoctor Cache files.

Are you sure you want to proceed? [No]: Yes
2020-11-01 18:43:21,485: xDoctor_4.8-67 - INFO : xDoctor Cache files successfully
cleared.

5. While still in the existing screen session from previous section (connected to node 1 of target rack, execute xdoctor RAP025
and RAP092 to validate both the Front-End (FE) and Back-End (BE).
NOTE: Before running this step, it is recommended to update xDoctor to latest available version.

admin:~> sudo xdoctor --rap=RAP025,RAP092

Example:
admin:~> sudo xdoctor --rap=RAP025,RAP092
2020-01-15 13:27:22,440: xDoctor_4.8-57 - INFO: Initializing xDoctor v4.8-57 ...
2020-01-15 13:27:23,382: xDoctor_4.8-57 - INFO: Removing orphaned session -
session_1579094724.085
2020-01-15 13:27:23,384: xDoctor_4.8-57 - INFO: Starting xDoctor
session_1579094842.267 ... (SYSTEM)
2020-01-15 13:27:23,384: xDoctor_4.8-57 - INFO: Master Control Check ...
2020-01-15 13:27:23,518: xDoctor_4.8-57 - INFO: xDoctor Composition -
Analyzer(s):ac_interface_check, ac_bond_check, ac_dell_switch

Switch OS10 upgrade 25

Page 30 of 42
 .
.
.

2020-01-15 13:28:01,550: xDoctor_4.8-57 - INFO: Analysis Limit: 0:08:00

2020-01-15 13:28:01,550: xDoctor_4.8-57 - INFO: ANALYZER [ac_interface_check]
2020-01-15 13:28:01,594: xDoctor_4.8-57 - INFO: ANALYZER [ac_bond_check]
2020-01-15 13:28:01,605: xDoctor_4.8-57 - INFO: ANALYZER [ac_dell_switch]
2020-01-15 13:28:01,617: xDoctor_4.8-57 - INFO: All data analyzed in 0:00:00
2020-01-15 13:28:01,620: xDoctor_4.8-57 - INFO: --------------------
2020-01-15 13:28:01,620: xDoctor_4.8-57 - INFO: Diagnosis Summary
2020-01-15 13:28:01,620: xDoctor_4.8-57 - INFO: --------------------
2020-01-15 13:28:01,620: xDoctor_4.8-57 - INFO: PSNT: FNM00181300308
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: --------------------
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of CRITICAL: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of FIXED: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of ERROR: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of WARNING: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of STATUS: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of VERBOSE: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of REPORT: 0
2020-01-15 13:28:10,009: xDoctor_4.8-57 - INFO: ----------------
.
.
.
2020-01-15 13:28:10,018: xDoctor_4.8-57 - INFO: --------------------------
2020-01-15 13:28:10,018: xDoctor_4.8-57 - INFO: Session Report - xdoctor --report --
archive=2020-01-15_132722
2020-01-15 13:28:10,018: xDoctor_4.8-57 - INFO: ---------------
2020-01-15 13:28:10,019: xDoctor_4.8-57 – INFO: Session Cleaner
2020-01-15 13:28:10,019: xDoctor_4.8-57 - INFO: ---------------
2020-01-15 13:28:10,029: xDoctor_4.8-57 - INFO: Nothing to cleanup
2020-01-15 13:28:10,030: xDoctor_4.8-57 - INFO: ------------------------------
2020-01-15 13:28:10,030: xDoctor_4.8-57 - INFO: xDoctor session_1579094842.267
finished in 0:00:48
2020-01-15 13:28:16,405: xDoctor_4.8-57 - INFO: Successful Job:1579094842 Exit
Code:128

6. Determine the report archive for the xDoctor session executed in the previous step.

admin:~> sudo xdoctor -r | grep -a1 Latest

For example,

Latest Report:
xdoctor -r -a 2019-08-14_190658

7. View the latest xDoctor report using the output from the command in previous step. If no errors reported continue with next
step. If the errors are on the FE or BE switch pair being updated the issues should be addressed post updating OS10.
The following example shows a clean report with no events.

admin:~> sudo xdoctor -r -a 2019-08-14_190658

Displaying xDoctor Report (2019-08-14_190658) Filter:['CRITICAL’, 'WARNING'] ...

The following example shows a report with an error on Node 3, slave-0 = Front-End rabbit switch network connection down.
If the firmware was being updated on hare this would cuase a loss of access to node 3 via public interfaces.

admin:~> xdoctor -r -a 2019-08-14_190658

Displaying xDoctor Report (2019-08-14_190658) Filter:['CRITICAL', 'WARNING'] ...

Timestamp = 2019-08-14_190658
Category = platform
Source = ip show
Severity = CRITICAL
Node = 169.254.10.3
Message = One or more network interfaces are down or missing
Extra = {'169.254.10.3': ['slave-0']} << Node 3, rabbit switch connection
RAP = RAP025
Solution = 207810

26 Switch OS10 upgrade

Page 31 of 42
 The following example shows a report with an error on Node 4, pslave-1 = Back-End hound switch network connection
down. If the firmware was being updated on fox this would cause a loss of access to node 4 via private interfaces.

admin:~> xdoctor -r -a 2019-08-14_190658

Displaying xDoctor Report (2019-08-14_190658) Filter:['CRITICAL', 'WARNING'] ...

Timestamp = 2019-08-14_191606
Category = platform
Source = ip show
Severity = CRITICAL
Node = 169.254.10.4
Message = One or more network interfaces are down or missing
Extra = {'169.254.10.4': ['pslave-1']
RAP = RAP025
Solution = 207810

8. You may close out of any remaining screen sessions (Ctrl-] d).
9. Any issues or failures contact ECS Remote Support, else switch OS10 upgrade(s) are successfully complete.

Switch OS10 upgrade 27

Page 32 of 42
 A
Appendix
Enter the lead-in sentence for the mini-TOC. Trisoft automatically adds the mini-TOC when you output to PDF. See the
Structured Authoring How-To Guide for best practices and guidance.
Topics:
• Next steps when an exception error is hit
• How to manually set VLT MAC
• Downgrading one switch from 10.4.3.6C1 to 10.3.2.
• Manual ONIE Install

Next steps when an exception error is hit

Prerequisites

Steps
1. SSH into the switch that was reported to have the error expectation.
2. Check the image status

fox # show image status

3. If the transfer is listed as failed, exit out of the switch and follow the single switch process to upgrade. Otherwise, wait until
Installation State is listed as Complete. Once it is listed as complete

fox # show image status

Image Upgrade State: idle
==================================================
File Transfer State: idle
--------------------------------------------------
State Detail: Completed: No error
Task Start: 2019-09-06T17:00:46Z
Task End: 2019-09-06T17:01:14Z
Transfer Progress: 100 %
Transfer Bytes: 502807409 bytes
File Size: 502807409 bytes
Transfer Rate: 18582 kbps

Installation State: idle

--------------------------------------------------
State Detail: Completed: Success
Task Start: 2019-09-06T17:01:14Z
Task End: 2019-09-06T17:06:56Z

4. Check the boot status. If the Next-Boot is listed for the Standby 10.4.3.6C1 partition, then skip to step 7.

fox # show boot

Current system image information:
===================================
Type Boot Type Active Standby Next-Boot
-----------------------------------------------------------------------------------
Node-id 1 Flash Boot [A] 10.3.2E(R3) [B] 10.4.3.6C1 [A] active

5. Change the next-boot partition to be for the standby 10.4.3.6C1 partition.

fox # boot system standby

28 Appendix

Page 33 of 42
6. Check the boot status. If the Standby partition is not listed as Next-Boot, redo the previous step.

fox # show boot

Current system image information:
===================================
Type Boot Type Active Standby Next-Boot
-----------------------------------------------------------------------------------
Node-id 1 Flash Boot [A] 10.3.2E(R3) [B] 10.4.3.6C1 [B] standby

7. Exit out from switch

fox # exit
8. Check bond states on all nodes prior to rebooting the next switch . Ensure all ports report 61 prior to continuing.

admin:~> viprexec "grep -e stat -e Interface /proc/net/bonding/*"

NOTE: If “port state:” is not 61 on any interface, contact ECS Remote Support to resolve this issue before proceeding
further. Otherwise, continue with the process.
9. Reload the switch using the updateDell tool
● For ECS software version 3.4 and lower:

# sudo updateDell <switch IP address or name>

● For ECS software versions 3.5 and later:

# getrackinfo -c /home/admin/MACHINES

# sudo updateDell <switch IP address or name> -m /home/admin/MACHINES

10. Run the update dell tool on the switch and its pair to apply post upgrade settings.

admin:~> sudo updateDell -p fox.rack hound.rack

11. Verify that both fox and hound both have the above MAC set for the VLT-MAC.

admin:~> ssh fox.rack " show running-configuration | grep vlt-mac"

Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:
WARNING: Cluster manager is still using default credentials
vlt-mac d8:9e:f3:bf:01:00

admin:~> ssh hound.rack " show running-configuration | grep vlt-mac"

Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:
WARNING: Cluster manager is still using default credentials
vlt-mac d8:9e:f3:bf:01:00

12. Task complete.

How to manually set VLT MAC

This section helps you manually set VLT MAC.

Prerequisites
Start by collecting the MAC that is expected to be used for the static VLT-MAC:
● Use the switch mac of fox for the fox and hound switch pairing.
● Use the switch mac of rabbit for the rabbit and hare switch pairing.

Appendix 29

Page 34 of 42
Steps
1. Check the system MAC of fox:

admin:~> ssh fox.rack "show system | grep MAC"

Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:
WARNING: Cluster manager is still using default credentials
MAC : d8:9e:f3:bf:01:00
Number of MACs : 256

2. Use the MAC highlighted above to apply to both fox and hound VLT configurations.
3. SSH to Fox and apply the VLT MAC.

admin:~> ssh fox.rack

password:
Debian GNU/Linux 8

Dell EMC Networking Operating System (OS10)

[email protected]'s password:

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent

permitted by applicable law.

-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
-* Dell EMC Network Operating System (OS10) *-
-* *-
-* Copyright (c) 1999-2017 by Dell Inc. All Rights Reserved. *-
-* *-
-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-

This product is protected by U.S. and international copyright and

intellectual property laws. Dell EMC and the Dell EMC logo are
trademarks of Dell Inc. in the United States and/or other
jurisdictions. All other marks and names mentioned herein may be
trademarks of their respective companies.

4. Enter the following commands to set the VLT MAC

fox # configure terminal

fox (config)# vlt-domain 1

Example of MAC, enter the value from step 1

fox (conf-vlt-1)# vlt-mac d8:9e:f3:bf:01:00

5. Repeat this step 3 to apply the VLT MAC for hound.

6. Verify that both fox and hound both have the above MAC set for the VLT-MAC.

admin:~> ssh fox.rack " show running-configuration | grep vlt-mac"

Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:
WARNING: Cluster manager is still using default credentials
vlt-mac d8:9e:f3:bf:01:00

admin:~> ssh hound.rack " show running-configuration | grep vlt-mac"

Debian GNU/Linux 9

Dell EMC Networking Operating System (OS10)

[email protected]'s password:

30 Appendix

Page 35 of 42
 WARNING: Cluster manager is still using default credentials
vlt-mac d8:9e:f3:bf:01:00

7. Save configuration on the switches with:

copy running-configuration startup-configuration

8. Ensure the SSH Cipher has been reset to default state

admin:~> sudo reduce_sshd_mac -d

9. Task complete.

Downgrading one switch from 10.4.3.6C1 to 10.3.2.

This section helps you downgrade switch version.

Prerequisites
● Service Laptop (onsite)
● Serial DB–9 to RJ–45 connector adapter cable. (This cable is provided with every replacement switch.) If the service laptop
has no 9-pin connector, you will need a USB to serial port adapter. (onsite for troubleshooting - Optional)
● PuTTY and SCP tools installed on your service laptop. This software allows you to establish SSH terminal sessions and copy
firmware files from laptop to ECS nodes.
● Determine the service login user/password (default) on nodes and switches required engagement. See the ECS Service User
Access document for more information. If the CLI user or password is changed from the default, contact the customer for
details.10.3.2 or 10.4.3.6C1 switch config file:
Default: Tools -> ECS_switch_configfiles_vx,x.zip file in ECS Solve Desktop or on-line https://solveonline.emc.com/solve/
home/15 version.
Or
Backed Up: configuration xml files provided by customer.
Ensure to use the v1.0 configs for the appropriate switches for 10.3.2 version.

About this task

This procedure is for a case where switch was replaced in a production environment where remaining switches are running
10.3.2.x version, but a replacement switch was added with version 10.4.3.6C1 already installed by manufacturer.
This process requires a user to be onsite to serial connect to the switches. Recommend maintenance window to perform
procedure.

Steps
1. Connect service laptop to Serial connection of the switch. Create PuTTy serial connection with the following settings
Data Rate: 115200
Data Bits: 8
Parity: None
Stop bits: 1
Hardware flow control: none

2. Log into the switch and run the following command to ensure that the standby partition is still at 10.3.2:

OS10# show boot

Appendix 31

Page 36 of 42
 If the standby partition is NOT at 10.3.2, DO NOT PROCEED with the following steps. Go to, Manual ONIE Install section and
downgrade to 10.3.2.

1. [ ] Log into the switch. Ensure that the standby partition is still at
10.3.2. If the standby partition is NOT at 10.3.2, DO NOT PROCEED with the following
steps. Go to, ONIE install must be performed to downgrade back to 10.3.2.

3. Instruct the switch to boot from the standby partition on next boot.

hound# boot system standby

4. Delete the existing startup-configuration

hound # delete startup-configuration

5. Proceed to delete startup-configuration [confirm yes/no(default)]:y

6. Issue the following command to clean up and reboot the switch

hound# system "sudo sh -c 'rm -vf /config/var/lib/redis/dump.rdb; sync; reboot -f'"

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

#1) Respect the privacy of others.

#2) Think before you type.
#3) With great power comes great responsibility.

[sudo] password for admin: <Enter admin password>

removed '/config/var/lib/redis/dump.rdb'
Failed to read reboot parameter file: No such file or directory
Rebooting.

7. Once the switch has booted back into OS10. Log in with the default OS10 switch credentials (admin/admin)

OS10# show boot

Current system image information:
===================================
Type Boot Type Active Standby Next-Boot
-----------------------------------------------------------------------------------
Node-id 1 Flash Boot [A] 10.3.2E(R3) [B] 10.4.3.6C1 [A] active

8. Copy the default template xml file to replace the default config://startup.xml file.

OS10# copy usb://<filename>.xml config://startup.xml

9. Reload switch

# reload
Proceed to reboot the system? [confirm yes/no]: yes

10. Connect service laptop to fox switch port 34 or 36, establish a secure shell (SSH) session to node 1 (192.168.219.1) in the
target rack.
11. While still in the existing screen session from previous section (connected to node 1 of target rack, execute xdoctor RAP025
and RAP092 to validate both the Front-End (FE) and Back-End (BE).
NOTE: Before running this step, it is recommended to update xDoctor to latest available version.

admin:~> sudo xdoctor --rap=RAP025,RAP092

Example:
admin:~> sudo xdoctor --rap=RAP025,RAP092
2020-01-15 13:27:22,440: xDoctor_4.8-57 - INFO: Initializing xDoctor v4.8-57 ...
2020-01-15 13:27:23,382: xDoctor_4.8-57 - INFO: Removing orphaned session -
session_1579094724.085
2020-01-15 13:27:23,384: xDoctor_4.8-57 - INFO: Starting xDoctor
session_1579094842.267 ... (SYSTEM)
2020-01-15 13:27:23,384: xDoctor_4.8-57 - INFO: Master Control Check ...
2020-01-15 13:27:23,518: xDoctor_4.8-57 - INFO: xDoctor Composition -

32 Appendix

Page 37 of 42
 Analyzer(s):ac_interface_check, ac_bond_check, ac_dell_switch
.
.
.

2020-01-15 13:28:01,550: xDoctor_4.8-57 - INFO: Analysis Limit: 0:08:00

2020-01-15 13:28:01,550: xDoctor_4.8-57 - INFO: ANALYZER [ac_interface_check]
2020-01-15 13:28:01,594: xDoctor_4.8-57 - INFO: ANALYZER [ac_bond_check]
2020-01-15 13:28:01,605: xDoctor_4.8-57 - INFO: ANALYZER [ac_dell_switch]
2020-01-15 13:28:01,617: xDoctor_4.8-57 - INFO: All data analyzed in 0:00:00
2020-01-15 13:28:01,620: xDoctor_4.8-57 - INFO: --------------------
2020-01-15 13:28:01,620: xDoctor_4.8-57 - INFO: Diagnosis Summary
2020-01-15 13:28:01,620: xDoctor_4.8-57 - INFO: --------------------
2020-01-15 13:28:01,620: xDoctor_4.8-57 - INFO: PSNT: FNM00181300308
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: --------------------
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of CRITICAL: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of FIXED: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of ERROR: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of WARNING: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of STATUS: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of VERBOSE: 0
2020-01-15 13:28:01,621: xDoctor_4.8-57 - INFO: Number of REPORT: 0
2020-01-15 13:28:10,009: xDoctor_4.8-57 - INFO: ----------------
.
.
.
2020-01-15 13:28:10,018: xDoctor_4.8-57 - INFO: --------------------------
2020-01-15 13:28:10,018: xDoctor_4.8-57 - INFO: Session Report - xdoctor --report --
archive=2020-01-15_132722
2020-01-15 13:28:10,018: xDoctor_4.8-57 - INFO: ---------------
2020-01-15 13:28:10,019: xDoctor_4.8-57 – INFO: Session Cleaner
2020-01-15 13:28:10,019: xDoctor_4.8-57 - INFO: ---------------
2020-01-15 13:28:10,029: xDoctor_4.8-57 - INFO: Nothing to cleanup
2020-01-15 13:28:10,030: xDoctor_4.8-57 - INFO: ------------------------------
2020-01-15 13:28:10,030: xDoctor_4.8-57 - INFO: xDoctor session_1579094842.267
finished in 0:00:48
2020-01-15 13:28:16,405: xDoctor_4.8-57 - INFO: Successful Job:1579094842 Exit
Code:128

12. Determine the report archive for the xDoctor session executed in the previous step.

admin:~> sudo xdoctor -r | grep -a1 Latest

For example,

Latest Report:
xdoctor -r -a 2019-08-14_190658

13. View the latest xDoctor report using the output from the command in previous step. If no errors reported continue with next
step. If the errors are on the FE or BE switch pair being updated the issues should be addressed post updating OS10.
The following example shows a clean report with no events.

admin:~> sudo xdoctor -r -a 2019-08-14_190658

Displaying xDoctor Report (2019-08-14_190658) Filter:['CRITICAL’, 'WARNING'] ...

The following example shows a report with an error on Node 3, slave-0 = Front-End rabbit switch network connection down.
If the firmware was being updated on hare this would cuase a loss of access to node 3 via public interfaces.

admin:~> xdoctor -r -a 2019-08-14_190658

Displaying xDoctor Report (2019-08-14_190658) Filter:['CRITICAL', 'WARNING'] ...

Timestamp = 2019-08-14_190658
Category = platform
Source = ip show
Severity = CRITICAL
Node = 169.254.10.3
Message = One or more network interfaces are down or missing
Extra = {'169.254.10.3': ['slave-0']} << Node 3, rabbit switch connection

Appendix 33

Page 38 of 42
 RAP = RAP025
Solution = 207810

The following example shows a report with an error on Node 4, pslave-1 = Back-End hound switch network connection
down. If the firmware was being updated on fox this would cause a loss of access to node 4 via private interfaces.

admin:~> xdoctor -r -a 2019-08-14_190658

Displaying xDoctor Report (2019-08-14_190658) Filter:['CRITICAL', 'WARNING'] ...

Timestamp = 2019-08-14_191606
Category = platform
Source = ip show
Severity = CRITICAL
Node = 169.254.10.4
Message = One or more network interfaces are down or missing
Extra = {'169.254.10.4': ['pslave-1']
RAP = RAP025
Solution = 207810

14. Close out of any Putty sessions and disconnect serial and network cables.
15. Any issues or failures contact ECS Remote Support, else switch OS10 downgrade successfully complete.

Manual ONIE Install

This method should only be used when all other options are not available, and switches have to be reinstalled.

Prerequisites
● Service Laptop (onsite)
● Serial DB–9 to RJ–45 connector adapter cable. (This cable is provided with every replacement switch.) If the service laptop
has no 9-pin connector, you will need a USB to serial port adapter. (onsite for troubleshooting - Optional)
● PuTTY and SCP tools installed on your service laptop. This software allows you to establish SSH terminal sessions and copy
firmware files from laptop to ECS nodes.
● Determine the service login user/password (default) on nodes and switches required engagement. See the ECS Service User
Access document for more information. If the CLI user or password is changed from the default, contact the customer for
details.
● 10.3.2 or 10.4.3.6C1 switch config file:
Default: Tools -> ECS_switch_configfiles_vx,x.zip file in ECS Solve Desktop or on-line https://solveonline.emc.com/solve/
home/15 version.
Or
Backed Up: configuration xml files provided by customer.
Ensure to use the v1.0 configs for the appropriate switches for 10.3.2 version, and v2.0 config for
10.4.3.6C1 version.
● Dell Networking OS10 10.3.2 OR 10.4.3.6C1 binary. Requires unpacking of the PKGS_OS10-Enterprise-10.x.x.x.x-installer-
x86_64.bin to obtain the bin file using Linux commands or a utility like 7-Zip.
● How to obtain a copy of the switch OS10 license based on Dell Service Tag (DST), https://support.emc.com/kb/528508

Steps
1. Using the service laptop, format the USB partition to FAT/FAT32.
2. Copy the required PKGS_OS10-Enterprise-10.x.x.x.x-installer-x86_64.bin file, switch default or customer provided
<config>.xml file for target switch and <servicetag>.file to the USB.
3. Connect service laptop to Serial connection of the switch. Create PuTTy serial connection with the following settings.
● Data Rate: 115200
● Data Bits: 8
● Parity: None
● Stop bits: 1

34 Appendix

Page 39 of 42
 ● Hardware flow control: none

GNU GRUB version 2.02~beta2+e4a1fe391

+--------------------------------------------------------------------------+
| OS10-A |
| OS10-B |
| EDA-DIAG |
|*ONIE |
| |
| |
| |
| |
| |
| |
| |
| |
+--------------------------------------------------------------------------+
+--------------------------------------+
|*ONIE: Install OS |
| ONIE: Rescue |
| ONIE: Uninstall OS |
| ONIE: Update ONIE |
| ONIE: Embed ONIE |
| ONIE: Diag ONIE |
+--------------------------------------+

4. Stop the ONIE discovery process if the device boots to ONIE: Install.

$ onie-discovery-stop

5. Insert USB into target switch USB port on the rear of the switch.
6. Create a USB mount location on the system.

$ mkdir /mnt/media

7. Save the configuration on the switches with copy running-configuration startup-configuration.

8. Mount the USB media plugged in the USB port on the device.

$ mount —t vfat /dev/sdb1 /mnt/media

9. Install the software from the USB, where /mnt/media specifies the path where the USB partition is mounted.

$ onie-nos-install /mnt/media/<Dell OS10 10.x.x.x binary>

10. Once the switch has booted back into OS10. Log in with the default OS10 switch credentials (admin/admin)
11. Copy the default template xml file to replace the default config://startup.xml file.

OS10# copy usb://<filename>.xml config://startup.xml

12. Install switch license

OS10# license install usb://<servicetag>.lic

13. Verify license status

# show license status

System Information
---------------------------------------------------------
Vendor Name : Dell EMC
Product Name : S5148F-ON
Hardware Version: X02
Platform Name : x86_64-dellemc_s5100_c2538-r0
PPID : CN0VY710CES007C80003
Service Tag : BVSSG02
License Details
----------------
Software : OS10-Enterprise

Appendix 35

Page 40 of 42
 Version : 10.3.2E(R3)
License Type : PERPETUAL
License Duration: Unlimited
License Status : Active
License location: /mnt/license/BVSSG02.lic
---------------------------------------------------------

14. Reload switch

# reload
Proceed to reboot the system? [confirm yes/no]: yes

a. Close out of any Putty sessions, disconnect USB and serial cables.
b. Any issues or failures contact ECS Remote Support, else switch OS10 loading successfully complete.

36 Appendix

Page 41 of 42
Dell Technologies Confidential Information version: 2.3.6.91

Page 42 of 42