ECS ™ Procedure Generator

Solution for Validating your engagement

Topic
ECS Upgrade Procedures

Selections
What ECS Version Are You Upgrading To?: ECS 3.8.x.x
Select ECS Upgrade Version: Upgrade to 3.8.x.x using Rolling upgrade guide

Generated: September 12, 2023 4:59 AM GMT

REPORT PROBLEMS

If you find any errors in this procedure or have comments regarding this application, send email to
[email protected]

Copyright © 2023 Dell Inc. or its subsidiaries. All Rights Reserved. Dell Technologies, Dell, EMC, Dell
EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be
trademarks of their respective owners.

The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of
any kind with respect to the information in this publication, and specifically disclaims implied warranties of
merchantability or fitness for a particular purpose.

Use, copying, and distribution of any software described in this publication requires an applicable
software license.

This document may contain certain words that are not consistent with Dell's current language guidelines.
Dell plans to update the document over subsequent future releases to revise these words accordingly.

This document may contain language from third party content that is not under Dell's control and is not
consistent with Dell's current guidelines for Dell's own content. When such third party content is updated
by the relevant third parties, this document will be revised accordingly.

Publication Date: September, 2023

Dell Technologies Confidential Information version: 2.3.6.182

Page 1 of 76
 Contents
Preliminary Activity Tasks .......................................................................................................6
Read, understand, and perform these tasks.................................................................................................6

Notes, cautions, and warnings................................................................................................7

Rolling Upgrade Procedure for each VDC ..............................................................................7
Revision History.......................................................................................................................................7
Introduction ..............................................................................................................................................7
Service Console.......................................................................................................................................8
Security fix for Management API .............................................................................................................9
Workflow for the Rolling upgrade with the Service Console ....................................................................9
Workflow Flowchart ...............................................................................................................................10
Required software files for ECS Upgrade..............................................................................................10
Prepare for the upgrade.........................................................................................................................11
Connect to the VDC...............................................................................................................................12
Verify and upgrade xDoctor before the upgrade....................................................................................13
Verify the downloaded files needed to install ECS ................................................................................14
Check for configuration changes ...........................................................................................................15
Download the required software ............................................................................................................15
Install the Service Console and generate the cluster.ini file ..................................................................18
Update the DataNodes file for each VDC ..............................................................................................20
Run the preupgrade health check..........................................................................................................21
Perform rolling upgrade .........................................................................................................................22
Begin post upgrade tasks for a single VDC ...........................................................................................24
Run the post upgrade health check .......................................................................................................24
Run xDoctor health check using Service Console.................................................................................25
Renew certificates .................................................................................................................................25
Optimize system configuration...............................................................................................................26
Verify the data path functionality............................................................................................................26
Finish postupgrade tasks for a single VDC............................................................................................27
Disconnect from the node......................................................................................................................27
Connecting to a node and staying connected.......................................................................28
Connecting to a node and staying connected.......................................................................28
Access ECS remotely ............................................................................................................................28
Connect a service laptop to U or D series rack .....................................................................................28
Locate and download a GP from the patchDB......................................................................29

Dell Technologies Confidential Information version: 2.3.6.182

Page 2 of 76
 Locate and download a GP from the patchDB ......................................................................................30
Troubleshooting and Known Issues......................................................................................31
Error received when executing the Cluster_Config command...............................................................31
Problem .................................................................................................................................................31
Solution..................................................................................................................................................31
Related JIRA..........................................................................................................................................32
Several Service Console commands do not work after upgrade...........................................................32
Problem .................................................................................................................................................32
Solution..................................................................................................................................................32
Related JIRA..........................................................................................................................................32
Health_Check command fails after node replacement ..........................................................................32
Problem .................................................................................................................................................32
Solution..................................................................................................................................................33
Related JIRA..........................................................................................................................................33
A node upgrade failed because more than one node was offline during the upgrade ..........................33
Problem .................................................................................................................................................33
Solution..................................................................................................................................................34
Verify the xDoctor version fails due to blocked FTP site .......................................................................34
Problem .................................................................................................................................................34
Solution..................................................................................................................................................35
All VDCs are not upgraded after final teardown ....................................................................................35
Blocked communication causes upgrade teardown to fail.....................................................................36
Problem .................................................................................................................................................36
Workaround ...........................................................................................................................................37
Related JIRA..........................................................................................................................................37
Incorrect fabric-syslog owner.................................................................................................................37
Low capacity exceed alert post upgrade ...............................................................................................38
Node stuck during operating system upgrade .......................................................................................38
Related JIRAs:.......................................................................................................................................38
Health_check fails for inactive RGs .......................................................................................................38
Related JIRAs........................................................................................................................................39
Update the DataNodes file for each VDC ..............................................................................................39
Rolling_Upgrade_Setup failed to validate object-main goal states consistency for Agent and Lifecycle40
Problem .................................................................................................................................................40
Solution..................................................................................................................................................40
Related JIRA..........................................................................................................................................41
Error when running the getclusterinfo command ...................................................................................41
Problem .................................................................................................................................................41
Workaround ...........................................................................................................................................41

Dell Technologies Confidential Information version: 2.3.6.182

Page 3 of 76
 Related JIRA..........................................................................................................................................41
Transform svc service on replaced node...............................................................................................41
Problem .................................................................................................................................................41
Workaround ...........................................................................................................................................41
Related JIRA..........................................................................................................................................41
Transformsvc disabled by default ..........................................................................................................41
Problem .................................................................................................................................................41
Workaround ...........................................................................................................................................42
Upgrade only a few nodes of the cluster ...............................................................................................42
Operating system upgrade is stuck .......................................................................................................42
Problem .................................................................................................................................................42
Workaround ...........................................................................................................................................42
Related JIRA..........................................................................................................................................42
Preupgrade health check for switch operating system version..............................................................42
Related JIRA..........................................................................................................................................43
Upgrade gets stuck updating on various nodes due to unresponsive systemctl ...................................43
Workaround ...........................................................................................................................................43
Related JIRA..........................................................................................................................................43
Upgrade fails due to issues with NAN ...................................................................................................43
Problem .................................................................................................................................................43
Workaround ...........................................................................................................................................43
Related JIRA..........................................................................................................................................43
Checksum preventing new node from rejoining cluster .........................................................................43
Problem .................................................................................................................................................43
Workaround ...........................................................................................................................................43
Related JIRA..........................................................................................................................................43
Resourcesvc keeps restarting ...............................................................................................................43
Problem .................................................................................................................................................43
Workaround ...........................................................................................................................................44
Related JIRA..........................................................................................................................................44
VIP checks in preupgrade SC checks ...................................................................................................44
Related JIRA..........................................................................................................................................44
List objects request fails with 500 server error on a partially upgraded cluster .....................................44
Workaround ...........................................................................................................................................44
Related JIRA..........................................................................................................................................44
Upgrade fails when there is no VDC......................................................................................................44
Workaround ...........................................................................................................................................44
Related JIRA..........................................................................................................................................45
Rolling Upgrade Output Example..........................................................................................45

Dell Technologies Confidential Information version: 2.3.6.182

Page 4 of 76
 Upgrade setup and Node 2 upgrade from Node 1.................................................................................45
Upgrade on remaining nodes, example output from Node 2 .................................................................55
Document feedback ..............................................................................................................76

Dell Technologies Confidential Information version: 2.3.6.182

Page 5 of 76
Preliminary Activity Tasks
This section may contain tasks that you must complete before performing this procedure.

Read, understand, and perform these tasks

1. Table 1 lists tasks, cautions, warnings, notes, and/or knowledgebase (KB) solutions that you need to
be aware of before performing this activity. Read, understand, and when necessary perform any
tasks contained in this table and any tasks contained in any associated knowledgebase solution.

Table 1 List of cautions, warnings, notes, and/or KB solutions related to this activity

2. This is a link to the top trending service topics. These topics may or not be related to this activity.
This is merely a proactive attempt to make you aware of any KB articles that may be associated with
this product.

Note: There may not be any top trending service topics for this product at any given time.

ECS Top Service Topics

IMPORTANT: Please take our survey! ECS Procedure Quality Survey

Dell Technologies Confidential Information version: 2.3.6.182

Page 6 of 76
Notes, cautions, and warnings
NOTE: A NOTE indicates important information that helps you make better use of your product.

CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells
you how to avoid the problem.

WARNING: A WARNING indicates a potential for property damage, personal injury, or death.

Rolling Upgrade Procedure for each VDC

This guide describes how to upgrade to the current ECS version using the Rolling upgrade method with
the Service Console.

Revision History
Table 1. Revision history

Revision Changes

August 2023 Initial release of the document.

Introduction
This guide describes how to upgrade to the current ECS version using the Rolling upgrade method with
the Service Console.
In the Rolling upgrade (sequential mode) process, the full stack (operating system, fabric, and object
services) is upgraded on one node before proceeding to the next node (node by node). The Service
Console (SC) is used to upgrade both the operating system and the services.
You can upgrade to ECS 3.8.x.x from previous releases listed here with the latest Service Console.

• ECS 3.5 to 3.8.x.x

• ECS 3.5.0.1 to 3.8.x.x
• ECS 3.5.0.2 to 3.8.x.x
• ECS 3.5.1.0 to 3.8.x.x
• ECS 3.5.1.1 to 3.8.x.x
• ECS 3.5.1.2 to 3.8.x.x
• ECS 3.5.1.3 to 3.8.x.x
• ECS 3.5.1.4 to 3.8.x.x
• ECS 3.5.1.5 to 3.8.x.x
• ECS 3.5.1.6 to 3.8.x.x
• ECS 3.5.1.7 to 3.8.x.x

Dell Technologies Confidential Information version: 2.3.6.182

Page 7 of 76
 • ECS 3.6 to 3.8.x.x
• ECS 3.6.1 to 3.8.x.x
• ECS 3.6.0.1 to 3.8.x.x
• ECS 3.6.1.1 to 3.8.x.x
• ECS 3.6.1.2 to 3.8.x.x
• ECS 3.6.1.3 to 3.8.x.x
• ECS 3.6.2 to 3.8.x.x
• ECS 3.6.2.1 to 3.8.x.x
• ECS 3.6.2.2 to 3.8.x.x
• ECS 3.6.2.3 to 3.8.x.x
• ECS 3.6.2.4 to 3.8.x.x
• ECS 3.6.2.5 to 3.8.x.x
• ECS 3.7.0.0 to 3.8.x.x
• ECS 3.7.0.1 to 3.8.x.x
• ECS 3.7.0.2 to 3.8.x.x
• ECS 3.7.0.3 to 3.8.x.x
• ECS 3.7.0.4 to 3.8.x.x
• ECS 3.7.0.5 to 3.8.x.x
• ECS 3.7.0.6 to 3.8.x.x
• ECS 3.8.0.1 to 3.8.x.x
• ECS 3.8.0.2 to 3.8.x.x

NOTE: Upgrade from GPs are also supported.

NOTE: Use the latest version of Service Console to perform an upgrade.

NOTE:
ECS 3.8 is not available for download. Upgrade to GA version 3.8 is not supported. To upgrade from older releases,
upgrade to the latest supported interim release and then to 3.8.0.1.

NOTE:
Upgrade to ECS 3.8.0.3 is supported only from ECS 3.6 and above.

NOTE: ECS 3.8.0.3 is upgradable only using rolling upgrade.

Service Console
The Service Console is a tool that simplifies ECS upgrades and provides the following improvements:

• Automates preupgrade and post upgrade health checks

Dell Technologies Confidential Information version: 2.3.6.182

Page 8 of 76
 • Updates the operating system and upgrades the services with a few simple commands
• Provides detailed, human-readable logs that are intended to speed up troubleshooting

To perform an ECS upgrade, you must use the latest version of the Service Console. You can download
the Service Console under the Product Tool category and it is also available in Solve desktop Tools and
downloads section.
The Service Console logfiles are collected in the /opt/emc/caspian/service-console/log
directory on the node where the Service Console commands are run. The logs help you to monitor the
command execution and provide the corresponding output which you can access from anywhere.

Security fix for Management API

ECS 3.8 is enhanced to allow only the registered external servers like load balancers or proxy to post
any management operations. If you do not register the load balancers or proxy servers IP address to the
accepted list, the management operations that are posted by those servers fail with error 403-
forbidden.

Before you use a load balancer or attempt to access ECS by proxy, ensure that you register the IP
address of the server you are using to the trusted server names. After upgrade to 3.8, register load
balancers and proxy IP for management operations.
https://www.dell.com/support/kbdoc/en-us/000205031 provides information.

You do not have to register external servers making Management API calls directly to the ECS nodes or
using the ECS nodes IPs. The procedure is required only for connection that pass through a proxy server
or a Load Balancer.
This does not impact data operations.

Workflow for the Rolling upgrade with the Service Console

Rolling upgrade with Service Console follows this workflow. All the following tasks are performed for
each VDC, except the last task.
1. Complete the prerequisite tasks:
o Prepare for the upgrade.
o Connect to the VDC.
o Verify and upgrade to the latest version of XDoctor.
o Check for installed GP.
2. Download ECS operating system .zip files, the ECS binaries, and the service console to the
installer node. The installer node is the first node in the first rack at the site.
3. Install or upgrade the Service Console and generate the cluster.ini file.
4. Run the preupgrade health checks.
5. If the preupgrade health check shows BMC and API credential error, update datanodes file
with nondefault BMC and API credentials. Perform this task if you have nondefault BMC or API
credentials.
6. Perform the rolling upgrade.
7. Perform post upgrade tasks:

Dell Technologies Confidential Information version: 2.3.6.182

Page 9 of 76
 o Run post upgrade health check.
o Optimize system configuration.
o Verify the data path functionality.
8. If there is a multi-hop upgrade of a Geo-federate environment, upgrade all VDCs in the
federation to an interim version of ECS. Then, upgrade all the VDCs to the final ECS version.
9. To upgrade peer VDCs, repeat the previous steps.
10. When all other VDCs in the federation have been upgraded, Service Console sets the
upgrade_complete flags for all VDCs in the federation.

Workflow Flowchart
The diagrammatic representation of the rolling upgrade with Service Console workflow is as shown:

Figure 1. Workflow Flowchart

Required software files for ECS Upgrade

The table in the section lists the required files for upgrade.

NOTE: Depending on the requirement, you can download the ECS Appliance Software without

Dell Technologies Confidential Information version: 2.3.6.182

Page 10 of 76
 Encryption or the ECS Appliance Software with Encryption.

Table 1. Required software files for services upgrade The following table provides details of the
required files for upgrade:

File Description

ECS Download the Service Console from the Support site.

Service
Console https://www.dell.com/support/search/en-sg#q=ecs-os-update&sort=rel
software evancy&f:iuxType=[Drivers%20%26%20Downloads]&f:langFacet=
[en,fa]

ECS The upgrade to the ECS major releases requires an operating system update. Download the
operating operating system binary file ecs-os-update-<version>.zip from
system https://www.dell.com/support. The ECS operating system is under Update section.

ECS • For countries that do not enable encryption (non-D@RE environments), ecs-
productio <version>-production-no-encryption.tgz. (the no-encryption applies only
n to the Russian Federation and the People's Republic of China.). Download the
software software from ECS Appliance Software without Encryption.
• For countries that require encryption (D@RE environments), ecs-<version>-
production.tgz. (All countries other than the Russian Federation and the
People's Republic of China must use this version.). Download the required software
from ECS Appliance Software with Encryption.
Contains the ECS software and associated files that are used to upgrade the ECS software.
The production.tgz file can be used for upgrade, only when you have used the
production.tgz file for the initial install of ECS. If you used the production-no-
encryption.tgz file for the initial install, you must use the production-no-
encryption.tgz file for the upgrade.

CAUTION: If you use Internet Explorer to download ecs-<version>-

production.tgz, the file name extension is changed to .gz. Rename the file to
.tgz.

xDoctor Download the xDoctor software from the Support site. Refer to the xDoctor Release notes
for more information.

ecs-os- Download the file from the Support site.

hotfix-
<version>

Prepare for the upgrade

Before starting the upgrade procedure, ensure that you follow the recommendations in this section.

CAUTION: You can perform any management operations on storage pools, replication groups, or
namespaces even after some of the VDCs have been upgraded and upgrade of the other VDCs
are in progress. This also applies to VDCs where a few nodes have been upgraded. Do not
perform Extend, IP change, network separation, or disk replacement during upgrade. Do not
perform Geo operations like PSO and add new site while performing upgrade. Also, the upgrade

Dell Technologies Confidential Information version: 2.3.6.182

Page 11 of 76
 may have enabled new bucket-level features. It is advisable not to use new bucket-level features
until all VDCs in a geo-federation have been upgraded.

NOTE: The 9211 port is bi-directional between all nodes of VDC, including the inter-rack.

Complete the recommended tasks that are listed before starting the upgrade procedure.

• Review the current Release Notes.

• Review KB 522405 for possible configuration changes required to complete an upgrade.
• Ensure that the VDC is created, else the upgrade fails.
• If you are using content addressable storage (CAS), ensure that you have two IPs in the
applications connection string to avoid extended disruption.
• You cannot upgrade from an encrypted environment to a nonencrypted Environment, and you
cannot upgrade from a nonencrypted environment to An encrypted environment.
• Use svc_version command to determine if installation has General Patches and consult the
list of supported upgrade paths in the Introduction section.
• For all installations, use svc_version command to determine if there are configuration
changes. If configuration changes are found, contact ECS Remote Support to determine if these
configuration changes need to be preserved.
• Obtain administrative user credentials for the ECS Portal and for the SSH login on the installer
node/Node 1 and Node 2 of the cluster.
• Obtain the Baseboard Management Controller (BMC) user, management API monitor, and
service account information.

NOTE: The installer node is the first node of the first rack in each VDC. Node 2 is the second
lowest indexed node in the same rack as the installer node.

• To optimize the ECS release system configuration, obtain and complete the post upgrade
configuration instructions in the following Knowledge Base (KB) article: KB 485299. If no KBs are
required, the article says so.
https://www.dell.com/support/search/en-sg#q=10.4.3.6C1&sort=relevancy&f:iuxType=[Drivers%20%26%20Downloads]&f:langFacet=[en,fa]
• Check the version of the switch operating system and the Network interface card (NIC) firmware
(FW). In 3.8.x OS, upgrade requires NIC FW version has to be upgraded. See preupgrade
health check for hardware-specific NIC and switch operating system details.
• If there is any unfinished transformation information, do not perform the upgrade operation.

CAUTION: Expect possible I/O timeouts on all nodes when one node is being upgraded. When a
node that is being upgraded enters maintenance mode, all the other nodes in the cluster
experience an I/O time-out of approximately 30 s–90 s. The time-out occurs even when a load
balancer is in use. Applications accessing the cluster must be able to handle the I/O time-out.

Connect to the VDC

Communicate with the VDC by connecting to the installer node, which is the first node in the first rack in
the VDC.

Dell Technologies Confidential Information version: 2.3.6.182

Page 12 of 76
 Connect the service laptop to the service port on the private (Fox) switch; ports 34 or 36.
Configure your laptop with the following network parameters:
IP: 192.168.219.99
Netmask: 255.255.255.0
No Gateway
Start a command prompt on your laptop and verify connectivity to iDRAC.
# ping 192.168.219.1(Master/Installer Node)

Open an SSH session using Putty to 192.168.219.1:

m.Username: admin
n. Password: ChangeMe
Steps

1. Depending on whether you are remote or local, follow the steps in the linked procedure to
connect to the node:
o If you are connecting from on site, follow these instructions:
Connecting the service laptop to the ECS appliance

o If you are connecting from remote site, follow these instructions:

Accessing ECS remotely

2. Confirm that you have established an SSH connection to the node.

Verify and upgrade xDoctor before the upgrade

Verify that xDoctor is installed and is at the latest version across the ECS systems.
About this task

CAUTION: Once you complete the operating system update, the installed version of xDoctor may
revert to a version earlier than the latest. To recheck the xDoctor version and upgrade it to the
latest version before proceeding.

Perform this task for each rack in the current VDC that is being upgraded.
Steps

1. Log in to the current master rack.

ssh master.rack

2. Ensure that the same version of xDoctor is installed on all nodes in the rack by running the
following command:

sudo -i xdoctor --sysversion

In the following example, the xDoctor version is uniform on all nodes:

admin@provo-fern:~> sudo -i xdoctor --sysversion

xDoctor Uniform on all nodes: 4.8-64

In the following example, the xDoctor version is not uniform on all nodes:

admin@provo-pineapple:~> sudo -i xdoctor --sysversion

xDoctor not uniform across all nodes ...
Trying xDoctor Resync ...
Resync failed: No xDoctor package found for re-installation
[4.8-64] -> ['169.254.130.1', '169.254.130.2']
[4.8-58] -> ['169.254.130.3', '169.254.130.4']

Dell Technologies Confidential Information version: 2.3.6.182

Page 13 of 76
 Ensure you have the latest xDoctor package on the node to initiate automatic re-installation. See
the ECS xDoctor Release note available in ECS SolVe for details.
Next steps

If all nodes have the latest version, go to the next task in this procedure.

Verify the downloaded files needed to install ECS

Steps

1. On a local Linux workstation, verify the signatures file.

ecs-3.8.0.3-8416.5a924dd1-production.tgz
ecs-3.8.0.3-8416.5a924dd1-production-no-encryption.tgz hdfsclient-
3.8.0.3.138685.3a0a9b6.zip
hdfsclientrpm-3.8.0.3.138685.3a0a9b6.3.1.rpm
ecs-os-setup.x86_64-3.8.0.3-2206.44cc40a.62.install.iso
ecs-os-update-3.8.0.3-2206.44cc40a.62.zip
ECS-MIB-v2.mib

When successful, the signature validation returns Verified OK.

2. Download the signature files and binaries into the same directory:
a. Make a new subdirectory named signatures.

mkdir signatures

b. Download the files into this subdirectory.

c. Navigate to the subdirectory.

cd signatures

As an example, the directory contents for the ECS 3.8.0.3 release is shown:

ecs-3.8.0.3-8416.5a924dd1-production.tgz ecs-3.8.0.3-8416.5a924dd1-
production.tgz.signed.bin ecs-3.8.0.3-8416.5a924dd1-production-no-
encryption.tgz ecs-3.8.0.3-8416.5a924dd1-production-no-
encryption.tgz.signed.bin hdfsclient-3.8.0.3.138685.3a0a9b6.zip hdfsclient-
3.8.0.3.138685.3a0a9b6.zip.signed.bin hdfsclientrpm-
3.8.0.3.138685.3a0a9b6.3.1.rpm hdfsclientrpm-
3.8.0.3.138685.3a0a9b6.3.1.rpm.signed.bin ecs-os-setup.x86_64-3.8.0.3-
2206.44cc40a.62.install.iso ecs-os-setup.x86_64-3.8.0.3-
2206.44cc40a.62.install.iso.signed.bin ecs-os-update-3.8.0.3-
2206.44cc40a.62.zip ecs-os-update-3.8.0.3-2206.44cc40a.62.zip.signed.bin ECS-
MIB-v2.mib ECS-MIB-v2.mib.signed.bin

3. Verify the public key file:

Dell Technologies Confidential Information version: 2.3.6.182

Page 14 of 76
 openssl dgst -sha256 -verify ecs-3.8.0.3-public-key.pem -signature ecs-3.8.0.3-
8416.5a924dd1-production.tgz.signed.bin ecs-3.8.0.3-8416.5a924dd1-production.tgz

When successful, the signature validation returns Verified OK.

4. Once you have validated that the ecs-3.8.0.3-public-key.pem is valid, you can use that
file to verify each of the other downloaded files:

openssl dgst -sha256 -verify ecs-3.8.0.3-public-key.pem -signature

<SIGNED.BIN_FILENAME> <DOWNLOADED_FILENAME>

For example, to verify the ECS-MIB-v2.mib file, you would use the following command:

openssl dgst -sha256 -verify ecs-3.8.0.3-public-key.pem -signature ECS-MIB-

v2.mib.signed.bin ECS-MIB-v2.mib

Repeat this process for all other downloaded files.

Check for configuration changes

Use the svc_version tool to determine if there are configuration changes.
Steps

From the latest xDoctor, run the command:

svc_version -uc

NOTE: Command svc_version -uc prints configuration changes that must be preserved on
the new version and ignores those that can be overwritten. Use the command svc_version -c
to print all configuration changes, including the ones that can be overwritten and preserved.
Ensure you have the xDoctor version that supports the -uc option.

Download the required software

Use the procedure to download the required software package on the installer node (the first node of the
first rack in the VDC).
About this task

WARNING: Type or paste the long command as a single line of text, with no breaks.

Steps

1. Download ECS OS, ECS production.tgz, Service Console, and xDoctor software:
o Go to Software with encryption or Software without Encryption.
o Click Drivers and Downloads.
o Select the ECS version in the version panel.

Dell Technologies Confidential Information version: 2.3.6.182

Page 15 of 76
 o Obtain the ECS software required.

NOTE: You can install the software in both encryption-enabled and nonencryption environments.

2. Use Putty to SSH to the installer node of the first rack of the VDC by using the public IP address
of the node. The default credentials are Username: admin and Password: ChangeMe.
3. Extract the OS update binary file:
a. Browse to the /tmp/Upgrade directory, if the directory exists.

cd /tmp/Upgrade/

b. If the /tmp/Upgrade directory does not exist, create it:

mkdir /tmp/Upgrade/

c. If the /tmp/Upgrade directory does exist, delete all the contents under the directory.

WARNING: Directories that are removed with the rm -rf command cannot be recovered.

rm -rf /tmp/Upgrade/*

d. Copy the downloaded OS update.zip binary file to the /tmp/Upgrade directory on

the installer node by using pscp.exe, or a similar copy tool. The OS update zip file is
in a format similar to ecs-os-update-<version>.zip:

pscp ecs-os-update-<version>.zip [email protected]:/tmp/Upgrade/ecs-os-

update-<version>.zip

4. Download the ECS services upgrade installer:

a. Browse to the /tmp/Upgrade directory.

cd /tmp/Upgrade

b. IF GP Patch is required, download it from https://www.dell.com/support or from

http://ecsee.isus.emc.com/patchdb/generalpatches.php.
c. If GP is not required, download the code from online support by Product at
https://www.dell.com/support/home/us/en/04/product-support/product/ecs-appliance-
software-without-encryption/overview.
▪ ecs-<version>-production.tgz (for D@RE encryption environments)
▪ ecs-<version>-production-no-encryption.tgz (for non-D@RE
environments: Russian Federation and People's Republic of China only.)

WARNING: Do not upgrade from an encrypted environment to a nonencrypted environment and

do not upgrade from a nonencrypted environment to an encrypted environment.

Dell Technologies Confidential Information version: 2.3.6.182

Page 16 of 76
 d. Copy the appropriate production .tgz file to the /tmp/Upgrade directory on the
installer node by using a secure copy tool, such as pscp.exe. For example:

pscp ecs-<version>-production.tgz [email protected]:/tmp/Upgrade/ecs-

<version>-production.tgz

If you are upgrading to GP version, to download the production.tgz follow appendix.

e. Compare the checksum of the distribution file from Online Support or
http://ecsee.isus.emc.com/patchdb/generalpatches.php with the checksum of the copied
file:

sha256sum /tmp/Upgrade/ecs-<version>-production.tgz

Or

sha256sum /tmp/Upgrade/ecs-<version>-production-no-encryption.tgz

5. In the downloaded Service Console software file:

a. Check to see if the /tmp/service_console directory exists.
▪ If the directory exists, remove old files from the directory by running the following
command:

WARNING: Directories that are removed with the rm -rf command cannot be recovered.

[ -d /tmp/service_console ] && rm -rf /tmp/service_console/*

▪ If the directory does not exist, create it by running the following command:

mkdir /tmp/service_console

b. Change to the Service Console directory by running the following command:

cd /tmp/service_console

c. Copy the Service Console installer .tgz file from the download directory to the
/tmp/service_console directory using a secure copy tool, such as pscp.exe.
For example, you would enter a command similar to the following:

pscp service-console.tgz admin@<ip_address>:/tmp/service_console/service-

console-<service_console_version><service_console_build_number>.tgz

Dell Technologies Confidential Information version: 2.3.6.182

Page 17 of 76
 Install the Service Console and generate the cluster.ini file
Install the Service Console and generate the cluster.ini file on the installer node of Rack 1 of the
VDC.
Prerequisites

If you have an existing cluster.ini file, Service Console takes a backup of the existing file to retain
any customizations and for comparison with the generated cluster.ini file after you run the
Cluster_Config command.
Steps

1. On the installer node of Rack 1 of the VDC, extract the Service Console package by running the
following command:

cd /tmp/service_console
tar -xf service-console-<service_console_version><service_console_build_number>.tgz

2. Upgrade the Service Console by running the following command:

./service-console upgrade

If the Service Console is not installed, it installs when you run this upgrade command.
3. On the installer node of Rack 1 of the VDC, run the following command to generate the
cluster.ini file:

service-console run Cluster_Config

The generated cluster.ini file is on the installer node at

/opt/emc/config/local/cluster.ini and includes the private.4 IP addresses for all
nodes in all racks in the VDC.

NOTE: If the credentials are incorrect, expect a minute delay per node for command execution as
observed in /opt/emc/caspian/service-console/log/ logs.

4. Verify whether the nodes are grouped correctly in the rack, and that the rack names in the
cluster.ini file correspond to rack colors. The cluster.ini file contains the list of node
Private.4 (NAN) addresses, rack names, and VDC names that might be called in commands as
part of the procedure.
5. Verify that the cluster.ini file does not contain any WARNING and vdc_<x>_unknown
entries by running the following command.

sudo cat /opt/emc/config/local/cluster.ini

Sample output: WARNING and vdc_<x>_unknown errors display similar to the following:

######
# This file was automatically generated by the Service Console.
# Please verify that it reflects the actual cluster topology.

Dell Technologies Confidential Information version: 2.3.6.182

Page 18 of 76
 # Credentials (BMC, Mgmt API, etc) should be set in separate files.
# Use file group_vars/datanodes to set cluster-wide variables.
# Use file host_vars/HOST_IP to set node-specific variables.
######

[datanodes:children]
vdc_1

[vdc_1:children]
red

[red:vars]
rack_id=1
rack_name=red
rack_psnt=PSNT
rack_dns_server=8.8.8.8
rack_dns_search=ecs.test.com,test.com
rack_ntp_server=9.9.9.9,10.10.10.10
rack_ns_switch=files,mdns4_minimal,[NOTFOUND=return],dns,mdns4
sc_collected=True

[red:children]
node_169_254_1_1 # Installer / SC node
node_169_254_1_2
node_169_254_1_3
node_169_254_1_5

[node_169_254_1_1]
169.254.1.1

[node_169_254_1_1:vars]
bmc_ip=192.168.219.101
public_ip=10.10.200.50
public_fqdn=provo-red.ecs.test.com
public_subnet=255.255.255.0
public_gateway=10.10.200.1
public_vlan=None
public_interface=None
.
.
.
[node_169_254_1_5]
169.254.1.5

[node_169_254_1_5:vars]
bmc_ip=192.168.219.105
public_ip=10.10.200.66
public_layton-red.ecs.test.com
public_subnet=255.255.255.0
public_gateway=10.10.200.1
public_vlan=None
public_interface=None

Dell Technologies Confidential Information version: 2.3.6.182

Page 19 of 76
 #
# WARNING: The Service Console failed to find
# the corresponding rack(s) for nodes
# in the [vdc_1_unknown] section.
# Edit the cluster.ini file and map
# these nodes to rack(s) manually.
#
[vdc_1_unknown]
169.254.1.4

Update the DataNodes file for each VDC

This task is optional and required only if you are working on an installation with nondefault BMC
credentials or nondefault Management API credentials.
About this task

The BMC credentials are used for IPMItool to remotely power the node on or off as needed. The default
credentials are, Username is root, and the Password is passwd.

The management API credentials are used for logging into the GUI, the default users are emcservice
and emcmonitor the default password for both the accounts is ChangeMe. These credentials are used
for checking the health of the API, Enabling or Disabling DT load balancing, and other parameter
changes during the upgrade.

To use nondefault BMC or Management API passwords during service console command execution, edit
a separate configuration file to include the passwords.

NOTE: If custom credentials are used for the BMC interface, the Service Console automatically
restores these BMC credentials on the replaced node.

The password that was updated with customer password in the previous Service Console installation,
continues to be valid even after the upgrade.
Perform the procedure for each VDC:
Steps

1. To ensure that the owner of the file is root, create a backup of DataNodes file using the
following command before editing:

sudo cp -p /opt/emc/config/local/group_vars/datanodes
/opt/emc/config/local/group_vars/datanodes.backup

Edit the DataNodes using the following command:

sudo vi /opt/emc/config/local/group_vars/datanodes

2. If the BMC credentials (username or password) have been modified, open the datanodes
file and remove the comment indicator (#) in front of the line that was modified and specify the
new value.

Dell Technologies Confidential Information version: 2.3.6.182

Page 20 of 76
 Example of BMC credentials:

BMC credentials
bmc_username: root
bmc_password:

If the BMC password was changed to test, remove the comment indicator (#) and specify the
new password. The BMC credentials look as shown:

BMC credentials
bmc_username: root
bmc_password: test

If the BMC username was changed to Admin, and the password was changed to test, remove
the comment indicators (#) in front of both lines and specify the new values. The BMC
credentials look as shown:

BMC credentials
bmc_username: Admin
bmc_password: test

3. If the Mgmt API monitor account credentials (username or password) have been
modified, remove the comment indicator (#) in front of the line that was modified and specify the
new value.
4. If the Mgmt API service account credentials (username or password) have been
modified, remove the comment indicator (#) in front of the line that was modified and specify the
new value.
5. Switch credentials are used to communicate with Public and Private switches. If the Switch
credentials have been modified, remove the comment indicator (#) and specify the new
value. The Switch credential looks as shown:

Switch credentials
switch_username: admin
switch_password:

6. Ensure that the DataNodes file is protected by restricting access to the file. Set the file
permission to 600.

sudo chmod 600 /opt/emc/config/local/group_vars/datanodes

Run the preupgrade health check

The health check command performs all checks necessary to qualify the VDC for an upgrade.
Steps

1. Run the health check from the installer node:

Dell Technologies Confidential Information version: 2.3.6.182

Page 21 of 76
 service-console run Health_Check --tags pre_upgrade

Service Console upgrades xDoctor to the most recent version as part of this command.

Service Console runs xDoctor on all racks in the VDC, so you do not have to run this command
manually on each master rack. Because Service Console runs xDoctor on the entire VDC,
budget extra time for this command to complete in a multirack environment.
2. Check the health check output for switch operating system version for all Gen 3 hardware
(Validate switch operating system version). If the check reports a warning for wrong version, see
troubleshooting section to update the switch operating system version.

20190730 10:02:50.334: | Validate switch OS version

WARNING: Switch fox.rack OS version 10.3.2 REQUIRES updating on node 169.254.49.1
prior to upgrading to 3.4 and above.
WARNING: Switch hound.rack OS version 10.3.2 REQUIRES updating on node 169.254.49.1
prior to upgrading to 3.4 and above.

For the customized switches, use the following commands to check the backend and front end
switches.

# service-console run Health_Check --specific-check "Validate BE switch OS version"

# service-console run Health_Check --specific-check "Validate FE switch OS version"

3. Address any failures that the health check output indicates. See Troubleshooting and Known
Issues. Contact ECS Remote Support for assistance if needed.
Service Console outputs two reports following the preupgrade health check: one check contains
Service Console issue findings, and the other contains xDoctor issue findings.

NOTE: There is a possibility for Service Console to report a PASS status when xDoctor reports
issues. So ensure that you review both reports and address all issues before you continue.

Example

NOTE: The service console health check may fail when the datanode file has a wrong password
for emcservice, or BMC or emcmonitor or all of them. See KB 43725 for troubleshooting
information.

Perform rolling upgrade

Upgrade the nodes using the rolling upgrade procedure given here.
Prerequisites

NOTE: In cases where an upgrade must be postponed because of errors that need time to be
addressed, run the service-console run Pause_Upgrade command to pause the upgrade.
This command reverts the CF to default values. You can reattempt or reschedule the upgrade to
some other time when there is an error.

Dell Technologies Confidential Information version: 2.3.6.182

Page 22 of 76
 • Ensure the operating system update binary file (OS update.zip) is in the /tmp/Upgrade
directory.
• Ensure the ECS services upgrade installer (production .tgz) is in the /tmp/Upgrade directory.
• Ensure the cluster.ini file has been generated and reflects the current VDC state.

NOTE: It is mandatory to type or paste the following long command as a single line of text, with no
breaks.

NOTE: In a multirack setup where the life-cycle nodes are distributed over racks, the first upgrade
happens on the next life-cycle node other than the installer node, even if it is on the next rack.
Check for service console output to find out the upgraded node and use the same for upgrading
the rest of the nodes.

Steps

1. Use Putty to SSH to the installer node of the first rack of the site by using the public IP address
of the node. The default credentials are admin/ChangeMe.
2. From <Node 1>, perform the upgrade using the ecs-os-update and production.tgz files. For
example, to upgrade to version 3.8.0.3 run the following command:

service-console run Upgrade --upgrade-type rolling --os-bundle /tmp/Upgrade/ecs-os-

update-3.8.0.3-2206.44cc40a.62.zip --production-package /tmp/Upgrade/production.tgz

NOTE: Use the Upgrade_To_361 command even for 3.6.2 upgrade.

This command performs the following actions:

o Runs upgrade setup
o Installs Service Console on <Node 2> to assist with upgrade of all the remaining nodes
in the VDC
o Upgrades the operating system, HAL, fabric, and object on <Node 2>

NOTE: If the service-console.tgz file is downloaded on <Node 1> in a path that is different from
the default path, add --service-console-bundle <full path to service-
console.tgz> to the end of the command.

3. Once Node 2 is upgraded successfully, log out from the installer node.

logout

4. Use Putty to SSH to the next node <Node 2> by using the public IP address of the node. The
default credentials are admin/ChangeMe.
5. From <Node 2>, run the following command:

service-console run Upgrade --upgrade-type rolling

This command performs the following actions:

Dell Technologies Confidential Information version: 2.3.6.182

Page 23 of 76
 o Runs rolling upgrade on all the nodes
o Runs the upgrade teardown

NOTE: For upgrading a specific node, use the command with --target-node <private.4
ip of node> option. For excluding a particular node from upgrade, see Upgrade only a few
nodes of the cluster in the troubleshooting section.

Results

All nodes in the site are successfully upgraded. Continue to the next stage. If the criteria is not met for
even one node in the site, see the Troubleshooting and Known Issues section. If the issue persists
contact ECS Remote Support.

NOTE: If there is a rolling upgrade failure, resolve the cause of the failure and then run the
command again to complete the upgrade. If you want to upgrade or exclude a particular node,
see the troubleshooting section.

Begin post upgrade tasks for a single VDC

You have just completed the upgrade of a single VDC.
Before upgrading other VDCs, you must perform some post upgrade tasks on this VDC.

NOTE: Do not perform these tasks until all nodes in the VDC are upgraded.

Perform the tasks in this VDC immediately after the upgrade of each VDC.

Run the post upgrade health check

The health check command performs all checks necessary to certify the upgrade that is completed
successfully.
Steps

1. From Node 1 (Installer node) of the VDC that was upgraded, run the health check:

service-console run Health_Check --tags post_upgrade

The command contains all the checks that are needed for the upgrade procedure, but has fewer
checks compared to the default health check. It reduces the time that is spent on the post
upgrade health checks and isolates the issues that are not related to the upgrade.
2. Address any failures that the health check output indicates. See Troubleshooting and Known
Issues. Contact ECS Remote Support for assistance if needed.
Results

The health check command completes successfully. The VDC has been upgraded successfully and is in
a healthy state. Continue with the next task.

NOTE: If all the VDCs in the federation are not yet upgraded at this stage, all new features the
upgrade introduces are not available yet. Do not try to exercise new features.

Dell Technologies Confidential Information version: 2.3.6.182

Page 24 of 76
Ignore the instance where the health check reports a warning: [WARN] Upgrade is not complete,
but local vdc is upgraded, skip completing upgrade. After one site is upgraded, and
upgrade for other sites is still pending.

Run xDoctor health check using Service Console

Steps

1. To verify that xDoctor is configured correctly and is able to send alerts and reports to xDoctor
portal, use this command.

sudo xdoctor --rap RAP069

Use the Session Report value in the output of the above command to see if any issues
are found

sudo xdoctor --report --archive=2018-08-08_173737

If output looks similar to this use KB520782 to address the issue or escalate to
support
xDoctor 4.6-42
FC6ND154300217 - ECS 3.2 SP1 GP 4 (??)

Displaying xDoctor Report (2018-08-08_173737) Filter:[] ...

Timestamp = 2018-08-08_173737
Category = connecthome
Source = os
Severity = WARNING
Message = RAP069 - xDoctor needs to be Kickstarted (CACHED)
Extra = No Connected ESRS

2. If the report returns any Warning, Errors, or Critical events resolve those events before
continuing this procedure.
All xDoctor reported Warning, Error, and Critical events must be resolved before proceeding.
Contact ECS Remote Support for assistance as required.

Renew certificates
The fabric installer supports non-expired certificates renewal and old certificates restoration.

NOTE: It is mandatory to run the renew certificate procedure to avoid data unavailability due to
certificate expiration.

Certificates should be renewed only if they expire in less than two years. To check the date run the
following command:

#service-console run Health_Check --specific-check "Verify Fabric Certificate"

Health Check
20210317 12:26:57.091: Execute Specific Health Checks
20210317 12:26:57.092: | Verify Fabric certificate
Certificate expiration date: Feb 16 16:04:05 2026 GMT

Dell Technologies Confidential Information version: 2.3.6.182

Page 25 of 76
 20210317 12:27:34.402: | | PASS (37 sec)
20210317 12:27:34.403: | PASS (37 sec)

You can find the How to renew certificate using Service Console procedure guide in SolVe Desktop or
SolVe online under How To Procedures.

Optimize system configuration

Instructions to optimize the system configuration.
To optimize system configuration, obtain the instructions in the ECS KB article KB 485299.

Perform each relevant KB task at this stage unless the KB task instructs you to perform the task after all
VDCs in a geo-federation have been upgraded.

NOTE: If the system has static routes that are configured, see the Troubleshooting and Known
Issues for some mandatory post upgrade tasks.

Verify the data path functionality

Use the instructions provided here to verify the data path functionality.
Prerequisites

• Obtain the S3 browser tool.

• Create or locate a text file on the laptop that you can upload to the appliance.
About this task

To determine if there is an object user, a secret key, and a bucket that you can use to test the data path,
consult the customer. If they do not have them, you must create them. Consult the ECS documentation
for instructions about how to create an object user, a secret key, and a
bucket:https://www.dell.com/support.
Steps

1. Start the S3 Browser and set up an account for the ECS Appliance with the following settings for
the options provided:

Table 1. S3 browser settings The following table lists the S3 browser settings:

Option Setting

Storage Type S3 Compatible Storage

REST Endpoint IP address of an ECS Appliance node using port 9020 or 9021. For example:
198.51.100.244:9021

Access Key ID ecsuser

Secret Access Object Secret Access Key

Key
NOTE: The path for Object Secret Access Key is found in ECS User Interface,
Manage > Users..

Dell Technologies Confidential Information version: 2.3.6.182

Page 26 of 76
 You should now see the bucket that you provided or created in the browser.
2. To configure S3 for a user, complete the following tasks:
a. Click Edit next to the specific user.
b. To view the secret key, click the check box.
c. Copy and paste the key in the S3 browser.
3. Use the S3 browser to upload the test file from the laptop to verify that you can write to the
appliance.

Finish postupgrade tasks for a single VDC

You have completed the postupgrade steps for a single VDC.

If it is an interim upgrade version, upgrade all VDCs to this version before starting an upgrade to the
target version of ECS.
If you have another VDC to upgrade, repeat the upgrade procedure for that VDC.

After completing the upgrade, remove the directory containing the files that were used for the upgrade on
the installer node using the # rm -rf /tmp/Upgrade/* command.

WARNING: Directories that are removed with the rm -rf command cannot be recovered.

Disconnect from the node

After completing the maintenance procedures, disconnect from the ECS virtual data center (VDC).
Steps

1. End the SSH session:

logout

2. Disconnect from the hardware.

For Gen2 hardware:

o If you were connected using the service shelf, disconnect the red cable from the laptop.
o If you were connected directly to the Turtle switch, disconnect the cable from the Turtle
switch port 24.
o If port 50 was used, disconnect the cable from the port and reconnect the white cable.
For Gen3 hardware:

o If you were connected through red cable, disconnect it from the laptop.
o If you were connected directly to the Fox switch, disconnect the cable from Port 34 or
36.

Dell Technologies Confidential Information version: 2.3.6.182

Page 27 of 76
Connecting to a node and staying connected
This appendix covers the following:

Connecting to a node and staying connected

This appendix covers the following:

Access ECS remotely

Remote access to ECS VDC/Rack is obtainable through Dell Secure Remote Services or WebEx to
customer jump box.
Steps

Using ServiceLink, select a device which enables remote connection to the target Rack or VDC, or
establish a WebEx support session to a customer jump box.

• Once an SSH session through Dell Secure Remote Services is established, determine which
node you have connected to. If needed connect SSH to Node1 Rack1. If that node is not
available, then connect SSH to Node2 Rack1.
• Shutting down a rack or VDC remotely requires an onsite personnel or remote access to RMMs
to power on the nodes in the rack.
• Remote Power off/on is available to the node being shutdown using ipmitool commands from
an operational node within the rack.

Connect a service laptop to U or D series rack

Access the ECS rack using the private (192.168.219.xxx) network from a laptop.
Prerequisites

• Access to private network IP addresses (192.168.219.1 to 24 and 192.168.219.101 to 124) are

limited to the nodes connected in the rack to the 1 GbE management switch.

• Private.4 (NAN) network IP addresses (169.254.x.x) of all nodes in all racks in the ECS VDC are
accessible from any node. Access is obtained once you SSH in to a node using a private IP
address (192.168.219.x).

• If security lock down is not enabled, access to public network IP addresses for all ECS racks is
available once you SSH in to one of the ECS nodes.
Steps

1. If cabinet contains a service shelf, open the shelf and connect the network cable (red) to the
service laptop.
2. From the rear of the rack, locate the 1 GbE private switch network ports by opening the rear
door.
3. On the 1 GbE (turtle) switch, attach a network cable from the laptop to port 24 on the switch.
If port 24 is occupied, then disconnect port 50 network cable (white) temporarily and attach the
network cable from the service laptop.

Dell Technologies Confidential Information version: 2.3.6.182

Page 28 of 76
 Figure 1. Locate port 24 on the private 1 GbE switch

4. Set the network interface on the laptop to the static address 192.168.219.99, subnet mask
255.255.255.0, with no gateway required.
5. Verify that the temporary network between the laptop and rack's private management network is
functioning by using the ping command.

NOTE: If 192.168.219.1 does not answer, try 192.168.219.2. If neither responds, verify the laptop
IP/subnet mask, network connection, and switch port connection. If the service laptop is
connected to Dell's VPN, ping to 192.168.219.x may not return a response.

C:\>ping 192.168.219.1
Pinging 192.168.219.1 with 32 bytes of data:
Reply from 192.168.219.1: bytes=32 time<1ms TTL=64
Reply from 192.168.219.1: bytes=32 time<1ms TTL=64
Reply from 192.168.219.1: bytes=32 time<1ms TTL=64
Reply from 192.168.219.1: bytes=32 time<1ms TTL=64
Ping statistics for 192.168.219.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

6. Establish an SSH session into 192.168.219.1 using any SSH tool such as PuTTY. Default
credentials are admin/ChangeMe #ssh 192.168.219.1
Results

You have established a connection to the installer node and can go ahead with the procedure.

Locate and download a GP from the patchDB

This appendix covers the following topic:

Dell Technologies Confidential Information version: 2.3.6.182

Page 29 of 76
Locate and download a GP from the patchDB
Follow the instructions in this section for locating the GP code only when it is required.
Steps

1. Visit the General Patch landing page in the ECS PatchDB, at

http://ecsee.isus.emc.com/patchdb/generalpatches.php.

Figure 1. General Patch landing page

2. Check the Latest <version> General Patch field.

A link is provided to the latest approved patch and a page with more detailed information.
3. From this Patch Details page, select either the Download D@RE or non-D@RE patch option (in
the upper-right corner) as appropriate.

Figure 2. D@RE or Non-D@RE Download

This downloads the production.tgz file that is used with the Patch Upgrade procedure.

NOTE: The View list of included Jiras link can provide more detailed info about changes in this
patch release.

Dell Technologies Confidential Information version: 2.3.6.182

Page 30 of 76
Troubleshooting and Known Issues
The first place to look when troubleshooting an upgrade issue is the ECS Service Console log files.
Service console logs are automatically logged in the /opt/emc/caspian/service-console/log
directory. Also, information about audit logs for upgrade activity on ECS nodes is provided in the ECS
3.7 Monitoring Guide.
This appendix covers the following topics:

Error received when executing the Cluster_Config command

Problem
When you create a cluster.ini file using the Cluster_Config command, the following error
appears:

service-console run Cluster_Config

You receive an error similar to the following:

Error response from daemon: Cannot start container
fb41bb5a10a60327e4b47e3b5f733e541d54e867b299395e00e884d660dd692d: [8] System
error: read parent: connection reset by peer

Solution
If you are having problems using the Cluster_Config command, create the cluster.ini file
manually in the /opt/emc/config/local directory on the node (for example, Node 1). Once you
create the cluster.ini file manually on Node 1, you can go ahead with the
Rolling_Upgrade_Setup and Rolling_Upgrade steps.

The following is an example of manually creating the cluster.ini file.

Create cluster.ini file

Below applicable since service console build # VS-vipr-object-qa-ECS-Automation-master/408/
(1.0.0.0-13453.ff071c8)
Create /config/local folder:
viprexec "mkdir -p /opt/emc/config/local"
Edit /opt/emc/config/local/cluster.ini file to update with private.4 Ips of all nodes in the
site on node1.
getrackinfo -v | grep private.4
private.4(NAN) : 00:1e:67:d8:3d:f3 169.254.130.5/255.255.0.0
private.4(NAN) : 00:1e:67:d6:24:51 169.254.130.6/255.255.0.0
private.4(NAN) : 00:1e:67:d8:8e:e8 169.254.130.7/255.255.0.0
private.4(NAN) : 00:1e:67:d6:1c:c7 169.254.130.8/255.255.0.0

vi /opt/emc/config/local/cluster.ini
Add private.4 IPs of all nodes to this file in the below format:
[datanodes]
169.254.89.5
169.254.89.6
Note:

Dell Technologies Confidential Information version: 2.3.6.182

Page 31 of 76
 Look up comment in FABRIC-4472 to see how BMC IPs and BMC cipher need to be added in case they
are different than what is there in ipmitool lan print output.
Look up comment in FABRIC-4619 to see how BMC password is to be specified if it is anything
but the default (root/passwd)

Related JIRA
• FABRIC-5033

Several Service Console commands do not work after upgrade

Problem
When upgrading, if due to any issues the object container is not running, a failure is injected on a node
by stopping the fabric agent on the node. The upgrade failed. Then, to continue upgrade normally on the
node, the container is brought back. However, after the upgrade, some of the executed fcli commands
on the node such as lock/unlock nodes and management.tasks, fail with the ERROR_INTERNAL
message, as shown in the following example.

service-console run Node_Maintenance_Exit

In the ECS Portal, the Platform Locking page displays unknown status for the node

Solution
The node is in maintenance mode. If the upgrade completes successfully, perform the following steps.

1. Check to see if the node is in maintenance mode by performing the following command:

service-console run Node_Maintenance_List

2. Exit maintenance mode.

service-console run Node_Maintenance_Exit

If the upgrade fails or is incomplete, retry upgrade with the force flag set to true. Use --force-
upgrade-fabric or --force-upgrade-object depending on the upgrade stage where it failed.

NOTE: Contact ECS Remote Support ECS remote support before running this command

Related JIRA
FABRIC-5205

Health_Check command fails after node replacement

Problem
After performing a node replacement, the Service Console Health_Check command fails on the
replaced node with an error similar to the following:

Dell Technologies Confidential Information version: 2.3.6.182

Page 32 of 76
 # 'Validate services owner and that goalstates are equal on LM and agents' failed:
Application: object owner for agent: a4a76db4-3059-4d75-9ace-8c6485a6ebca differs with
expected: {"application": "fabric", "instance": "lifecycle"}, the diff is set([u'installer'])

Solution
1. Restart the LM and fabric-agent on all nodes.

sudo service fabric-agent stop;sudo docker stop fabric-lifecycle;sleep 30; sudo

service fabric-agent start

2. Run the Service Console Health_Check to verify that the error no longer appears:

service-console run Health_Check

If restarting the LM/fabric-agent fails and the error appears in the health check, continue with the
upgrade which completes successfully.

Related JIRA
FABRIC-5153

A node upgrade failed because more than one node was offline during the upgrade

Problem
WARNING: Contact ECS Remote Support before proceeding with this operation. If more than one
node is offline during upgrade, it can lead to DU.

During the upgrade, the node from which the upgrade was being run (Node 1) was in maintenance
mode. As a result, when the process continued to upgrade Node 2 (initiated from Node 1), the upgrade
failed with the following output:

2018-03-01 12:45:56 main#1304 INFO: Redirecting console output to /opt/emc/caspian/service-

console/log/runRollingUpgradeupgradeosfalset_20180301_124556_0/output.log
Service console version: 2.0.0.0-18287.f505db5a9
Debug log: /opt/emc/caspian/service-
console/log/runRollingUpgradeupgradeosfalset_20180301_124556_0/dbg_robot.log
================================================================================
20180301 12:45:57.651: Service Console Setup
20180301 12:46:04.102: | PASS (6 sec)
============================================================
Rolling Upgrade
20180301 12:46:04.104: Rolling upgrade setup
20180301 12:46:04.105: | PASS
20180301 12:46:04.106: Pre-upgrade Service Check
[WARN] Node 169.254.130.18 is not in ACTIVE mode.
WARNING: Failed to check DT status.
20180301 12:46:07.132: | PASS (3 sec)

Dell Technologies Confidential Information version: 2.3.6.182

Page 33 of 76
 20180301 12:46:07.133: Get Rolling Upgrade Cli Parameters
20180301 12:46:07.134: | PASS
20180301 12:46:07.134: ${target_node} IN [ @{TARGET_NODES} ]
20180301 12:46:07.135: | ${target_node} = 169.254.130.17
20180301 12:46:07.135: | | OS and Node Upgrade
20180301 12:46:07.136: | | | Check if product components should be upgraded
Object is not upgraded on host 169.254.130.17
Fabric is upgraded on host 169.254.130.17
Product components to be upgraded on node 169.254.130.17: OS - False, Fabric - False, Object -
True
20180301 12:48:43.983: | | | | PASS (2 min 36 sec)
20180301 12:48:43.984: | | | Fail if too many down nodes
Node 169.254.130.18 is not in ACTIVE mode
20180301 12:48:53.149: | | | | FAIL (9 sec)
20180301 12:48:53.150: | | | FAIL (2 min 46 sec)
20180301 12:48:53.150: | | FAIL (2 min 46 sec)
20180301 12:48:53.151: | FAIL (2 min 46 sec)
20180301 12:48:53.154: Run Keywords
20180301 12:48:53.154: | Check FI execution
20180301 12:48:53.155: | | PASS
20180301 12:48:53.156: | Pass execution if not connected
20180301 12:48:53.156: | | PASS
20180301 12:48:53.156: | Service Console Teardown
20180301 12:48:57.122: | | PASS (3 sec)
20180301 12:48:57.123: | PASS (3 sec)
================================================================================
Status: FAIL
Time Elapsed: 2 min 59 sec
Debug log: /opt/emc/caspian/service-
console/log/runRollingUpgradeupgradeosfalset_20180301_124556_0/dbg_robot.log
HTML log: /opt/emc/caspian/service-
console/log/runRollingUpgradeupgradeosfalset_20180301_124556_0/log.html
================================================================================
Messages:
More than one node will be offline on this site. Proceeding with this operation could lead to
potential DU.
Provide option '--skip-down-node-check true' to override this check and proceed with this
operation.
================================================================================

Please bring node out of MM and continue with the upgrade.

Solution
In case of more than one node down, contact ECS Remote Support team before proceeding to upgrade.

Verify the xDoctor version fails due to blocked FTP site

Problem
When verifying the xDoctor version, the sudo –i xdoctor –u -A command fails because the EMC
FTP site is not accessible from the customer's data center.

Dell Technologies Confidential Information version: 2.3.6.182

Page 34 of 76
 Solution
Download the latest version of xDoctor from Solve Desktop to the service laptop and reinstall the latest
version of xDoctor.

1. Download the latest version of xDoctor from Solve Desktop to the service laptop.
2. Use Putty to SSH to the installer node of the first rack of the VDC by using the public IP address
of the node. The default credentials are admin/ChangeMe.
3. Copy xDoctor to the /tmp directory on the installer node on Rack 1 of the VDC.
4. From the installer node, run this command to reinstall the latest version of xDoctor (for example,
4.4.-32):

# sudo -i xdoctor -uL /tmp/xDoctor4ECS-4.4-30.noarch.rpm --reinstall

All VDCs are not upgraded after final teardown

After you finish upgrading the last VDC in the federation, the Service Console will automatically set
the upgrade_complete flags on all VDCs. If flags are not set for every VDC in the federation, you
can manually run the Cluster_Upgrade_Completion command on each VDC to set the
upgrade_complete flags.
Prerequisites

• All VDCs in the geo-federation are upgraded. Do not start the following task until each peer VDC
in the geo-federation has been upgraded.
About this task

If an error occurs while upgrade_complete flags are set, you see an error similar to the following in the
Upgrade_Teardown command output:

Set upgrade complete if all sites upgraded

Skip: upgrade is not complete

If you see this error in the output, manually set the flags.
Steps

1. Run the following command from <Node 1> to check that all VDCs are at the target version.

svc_version

2. Run the following command from <Node 1> of each VDC to set the flag.

service-console run Cluster_Upgrade_Completion

Sample output:

# 2017-10-26 08:50:53 INFO: Redirecting console output to /opt/emc/caspian/service-

console/log/runClusterUpgradeCompletion_20171026_085053_0/output.log
Service console version: 1.3.0.0-17392.a839643a8

Dell Technologies Confidential Information version: 2.3.6.182

Page 35 of 76
 ==============================================================================
ECS
==============================================================================
ECS.Service Console
==============================================================================
20171026 08:50:54.269: Service Console Setup
20171026 08:50:55.732: | PASS (1 sec)
Cluster Upgrade Completion :: Performs post-all sites upgrade actions
20171026 08:50:55.734: Set Upgrade Complete
Set com.emc.ecs.upgrade.3_1_upgrade_complete = true
Got expected status state: SYNCHRONIZED
20171026 08:51:31.345: | PASS (35 sec)
| PASS |
------------------------------------------------------------------------------
20171026 08:51:31.347: Service Console Teardown
20171026 08:51:31.566: | PASS
ECS.Service Console | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
ECS | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Debug: /opt/emc/caspian/service-
console/log/runClusterUpgradeCompletion_20171026_085053_0/dbg_robot.log
Output: /opt/emc/caspian/service-
console/log/runClusterUpgradeCompletion_20171026_085053_0/output.xml
Log: /opt/emc/caspian/service-
console/log/runClusterUpgradeCompletion_20171026_085053_0/log.html
Report: /opt/emc/caspian/service-
console/log/runClusterUpgradeCompletion_20171026_085053_0/report.html
admin@provo-pineapple:~>

Results

If the Cluster_Upgrade_Completion command succeeds for all VDCs in the geo-federation, the
upgrade engagement is complete and all new functionality is available at all VDCs. If the
Cluster_Upgrade_Completion command continues to fail for any VDC, contact ECS Remote
Support for assistance.

Blocked communication causes upgrade teardown to fail

Problem
In an environment that has multiple sites, if communication is blocked between Virtual Data Centers
(VDCs) outside of replication ports, the teardown reports a failure and does not set the upgrade
completion flag. Also, the subsequent steps like DT LoadBalancing (DTLB) are not enabled and the state
of the cluster does not move to normal mode.

Dell Technologies Confidential Information version: 2.3.6.182

Page 36 of 76
 Workaround
Login to each upgraded VDC separately and run the service-console run
Cluster_Upgrade_Completion --target-vdc <vdc_name> command manually. If the command
continues to fail for any VDC, contact ECS Remote Support for assistance.

Related JIRA
SCONSOLE-591

Incorrect fabric-syslog owner

Following a rolling upgrade, the cluster level goal state contains an incorrect owner of fabric-syslog.
About this task

To address this issue, you must dump the current goal state to a JSON file, edit the JSON file to correct
the goal state owner, then update the goal state with the corrected JSON file.
Steps

1. Dump the current goal state to a JSON file by running the following command:

./bin/fcli lifecycle application.goalstate --id fabric > /tmp/faric.json

2. Create a backup of the JSON file before you make any edits by running the following command:

cp /tmp/fabric.json /tmp/fabric.json.orig

3. Correct the JSON by making the following edits:

a. Remove the status header:

{"status":"OK","etag":0,"goal_state":

b. Remove the closing brace at the end of the file that corresponds to the header you just
removed.

c. At the end of the file between the ] and the }, insert the following, including the leading
comma:

,"application":"fabric"

d. Change the goal state from installer to lifecycle by running the following
command:

sed -i "s/installer/lifecycle/g" /tmp/fabric.json

Dell Technologies Confidential Information version: 2.3.6.182

Page 37 of 76
 4. Update the syslog owner with the modified JSON file by running the following command:

cat /tmp/fabric.json | fcli lifecycle application.update --id fabric --body

Results

Related JIRAs:

FABRIC-6674

Low capacity exceed alert post upgrade

About this task

For a VDC with a low capacity license, that exceeds the low capacity limit (10 TB) following an upgrade,
the system does not generate a Licensed Capacity Entitlement Exceeded alert.
Steps

Upload the license again.

Results

Related JIRAs:

ECSDOC-412

Node stuck during operating system upgrade

During operating system upgrade phase, the node may get stuck at below stage.

20180417 00:32:17.054: | | | Node Enter SSM MM before upgrade

Skip: SSM MM is not supported
20180417 00:32:17.056: | | | | PASS
20180417 00:32:17.059: | | | Node Enter Maintenance Mode
20180417 00:32:20.113: | | | | PASS (3 sec)
20180417 00:32:20.121: | | | OS Upgrade
20180417 00:32:20.126: | | | | Run OS Upgrade

NOTE: The swapoff -a may take hours to complete. Wait until this operation completes.

Related JIRAs:
CONSOLE-922

Health_check fails for inactive RGs

The upgrade the Health_Check, checks for the TSO status of inactive RGs.

20180416 04:34:41.681: | | Check that the system is not in TSO state

Dell Technologies Confidential Information version: 2.3.6.182

Page 38 of 76
 [ERROR] Detected site in TSO state, failed zones:
[u'urn:storageos:VirtualDataCenterData:a96a892c-333a-4de5-83b3-a9a9f6b5a773']
20180416 04:34:44.583: | | | FAIL (2 sec)
[ERROR] 'Check that the system is not in TSO state' failed: with above errors

As these RGs are inactive, their TSO status does not affect the working. However, the Heath_Check
reports failure.

Related JIRAs
CONSOLE-894

Update the DataNodes file for each VDC

This task is optional and required only if you are working on an installation with nondefault BMC
credentials or nondefault Management API credentials.
About this task

The BMC credentials are used for IPMItool to remotely power the node on or off as needed. The default
credentials are, Username is root, and the Password is passwd.

The management API credentials are used for logging into the GUI, the default users are emcservice
and emcmonitor the default password for both the accounts is ChangeMe. These credentials are used
for checking the health of the API, Enabling or Disabling DT load balancing, and other parameter
changes during the upgrade.

To use nondefault BMC or Management API passwords during service console command execution, edit
a separate configuration file to include the passwords.

NOTE: If custom credentials are used for the BMC interface, the Service Console automatically
restores these BMC credentials on the replaced node.

The password that was updated with customer password in the previous Service Console installation,
continues to be valid even after the upgrade.
Perform the procedure for each VDC:
Steps

1. To ensure that the owner of the file is root, create a backup of DataNodes file using the
following command before editing:

sudo cp -p /opt/emc/config/local/group_vars/datanodes
/opt/emc/config/local/group_vars/datanodes.backup

Edit the DataNodes using the following command:

sudo vi /opt/emc/config/local/group_vars/datanodes

2. If the BMC credentials (username or password) have been modified, open the datanodes
file and remove the comment indicator (#) in front of the line that was modified and specify the
new value.

Dell Technologies Confidential Information version: 2.3.6.182

Page 39 of 76
 Example of BMC credentials:

BMC credentials
bmc_username: root
bmc_password:

If the BMC password was changed to test, remove the comment indicator (#) and specify the
new password. The BMC credentials look as shown:

BMC credentials
bmc_username: root
bmc_password: test

If the BMC username was changed to Admin, and the password was changed to test, remove
the comment indicators (#) in front of both lines and specify the new values. The BMC
credentials look as shown:

BMC credentials
bmc_username: Admin
bmc_password: test

3. If the Mgmt API monitor account credentials (username or password) have been
modified, remove the comment indicator (#) in front of the line that was modified and specify the
new value.
4. If the Mgmt API service account credentials (username or password) have been
modified, remove the comment indicator (#) in front of the line that was modified and specify the
new value.
5. Switch credentials are used to communicate with Public and Private switches. If the Switch
credentials have been modified, remove the comment indicator (#) and specify the new
value. The Switch credential looks as shown:

Switch credentials
switch_username: admin
switch_password:

6. Ensure that the DataNodes file is protected by restricting access to the file. Set the file
permission to 600.

sudo chmod 600 /opt/emc/config/local/group_vars/datanodes

Rolling_Upgrade_Setup failed to validate object-main goal states consistency for Agent and Lifecycle

Problem
While trying to upgrade an eight node single site setup, the setup is failing when one node is shutdown.

Solution
When one node is down during the upgrade, and the upgrade fails, contact ECS Remote Support.

Dell Technologies Confidential Information version: 2.3.6.182

Page 40 of 76
Related JIRA
FABRIC-6879

Error when running the getclusterinfo command

Problem
After the upgrade, when you run the getclusterinfo command, you get an error: Failed to
create client object: Daemon not running.

Following is the error code:

admin@dallas-pansy:~> getclusterinfo
Failed to create client object: Daemon not running

Failed to create client object: Daemon not running

Warning: This service is unavailable on a slave node of an unregistered rack.
Login via ssh to the master node or set the color for this rack before you execute this
command

Workaround
To avoid the error, manually delete the process identification (PID) file in /var/run/avahi-daemon so
that it restarts and works in order.

Related JIRA
OS-1832

Transform svc service on replaced node

Problem
Transform svc service on replaced node does not start. After node replacement is done, the file
data/servicecontrol/transformsvc.on is cleared. This is used to indicate whether to start
transformsvc by default.

Workaround
Enable transformsvc on the GUI page again after completing node replacement.

Related JIRA
CONSOLE-1455

Transformsvc disabled by default

Problem
When upgrading from lower versions, the transformsvc is disabled by default. There is DU untill the
service is enabled.

Dell Technologies Confidential Information version: 2.3.6.182

Page 41 of 76
Workaround
If you are using the transformation service, enable the transformsvc service in the user interface.

Upgrade only a few nodes of the cluster

To Upgrade selected nodes in a cluster, use the target node command.

service-console run Upgrade --upgrade-type rolling --target-node <private.4 of node n>

Use --exclude command to exclude a node:

service-console run Upgrade --upgrade-type rolling --exclude <private.4 of node n>

To initiate Upgrade setup, and Upgrade a different node from node 2, as the first node, you can use the
--upgrade-first-node flag

service-console run Upgrade --upgrade-type rolling --upgrade-os false --target-node

<private.4 of node1> --production-package /tmp/Upgrade/production.tgz --os-bundle /tmp/
Upgrade/ecs-osupdate-<version>.zip --upgrade-first-node <private.4 ip of node>

NOTE: The node should have ZK container running on it.

Operating system upgrade is stuck

Problem
If during 3.3 operating system upgrade, the upgrade is stuck for a long time, from refit logs contain the
following message:

(196/211) Installing: nfs-client-1.3.0-34.8.3.x86_64 [.................." rpm is stuck

Workaround
When this issue occurs contact ECS Remote Support

Related JIRA
OS-1832

Preupgrade health check for switch operating system version

The preupgrade health check verifies the switch operating system versions on EX300, EX3000, and
EX500 clusters. If switch is not running with the correct version of Switch OS, refer to ECS EX-Series
Switch OS10 Matrix and ECS EX-Series Switch OS10 Upgrade Guide. The document can be found in
SolVe Desktop or SolVe online under How To Procedures > How to Update EX-Series Switch OS.

Dell Technologies Confidential Information version: 2.3.6.182

Page 42 of 76
Related JIRA
CONSOLE-1923

Upgrade gets stuck updating on various nodes due to unresponsive systemctl

When you upgrade from the current main release to the current patch, the quickfit upgrade could get
stuck on various nodes when the systemctl stops responding. Similarly when the systemctl stops
responding, the rolling upgrade might also get stuck.

Workaround
See KB article KB 535898 for more information.

Related JIRA
OS-2241

Upgrade fails due to issues with NAN

Problem
When upgrading, if there is an issue with NAN (EMC Network Area Network), the upgrade fails.

Workaround
Restart the emc.nan service "sudo service nan restart" and complete the upgrade.

Related JIRA
OS-2345

Checksum preventing new node from rejoining cluster

Problem
During mid upgrade node replacement, there are issues with 3.4.0.1 checksum preventing new node
from rejoining the cluster.

Workaround
After node replacement in mid upgrade, change the communicator upgrade_complete value to false or
reimage replaced node with the pre upgrade version.

Related JIRA
STORAGE-26544

Resourcesvc keeps restarting

Problem
When nodes are upgraded to 3.5, resourcesvc keeps restarting on one of the nodes.

Dell Technologies Confidential Information version: 2.3.6.182

Page 43 of 76
Workaround
Restart the container.

Related JIRA
STORAGE-27083

VIP checks in preupgrade SC checks

The VIP configurations are not supported from ECS 3.5 onwards. You must run the preupgrade health
check on all sites of the federation as a precursor to upgrade. If any site reports that it has VIP
configuration, do not proceed with the upgrade.
When VIP configured or enabled system is present, you get the following error message during
preupgrade health check:

20200528 15:45:07.179: | Check for VIPs

20200528 15:45:07.181: | | FAIL
[ERROR] 'Check for VIPs' failed: Network Separation with VIPs detected: mgmt, repl, data,
data2. Unable to proceed. Please refer to KB 541413 for details.

When VIP not configured or a disabled system is present, you get the following Warning message:

20200529 16:27:58.169: | Check for VIPs

No network Separation with VIPs detected.
[WARN] Important note! This pre-upgrade health check needs to be run on all sites of the
federation as a precursor to upgrade and if any site reports it has VIP configuration, should
not proceed with upgrade.

Related JIRA
CONSOLE-2571

List objects request fails with 500 server error on a partially upgraded cluster
In a partially upgraded environment, list objects request fails with 500 server error, even when
metering shows that bucket has objects.

Workaround
Contact ECS Remote Support.

Related JIRA
CONSOLE-3099

Upgrade fails when there is no VDC

Upgrade fails when there is no VDC created.

Workaround
Create a VDC before upgrade or contact ECS Remote Support.

Dell Technologies Confidential Information version: 2.3.6.182

Page 44 of 76
Related JIRA
CONSOLE-3314

Rolling Upgrade Output Example

The examples present here are for reference only, the SC version and other parameters could vary. This
appendix covers the following topic:

Upgrade setup and Node 2 upgrade from Node 1

The following is an example output for a Rolling upgrade (upgrading the operating system, fabric, and
object services node by node).

/opt/emc/bin/service-console run Upgrade --upgrade-type rolling --os-bundle /tmp/Upgrade/ecs-

os-update-3.8.0.3-2206.44cc40a.62.zip --production-package /tmp/Upgrade/production.tgz
Service Console is running on node 169.254.186.1 (suite 20230804_084534_Upgrade)
Service console version: 8.2.0.1-22389.6502d4d05b
Debug log: /opt/emc/caspian/service-console/log/20230804_084532_run_Upgrade/dbg_robot.log
================================================================================
Upgrade Procedure Setup
20230804 08:45:55.118: Warn if target VDC provided
20230804 08:45:55.119: | PASS
20230804 08:45:55.120: Set Upgrade Started Flag
20230804 08:45:58.139: | PASS (3 sec)
20230804 08:45:58.140: Set Upgrade Paused Flag
20230804 08:46:01.163: | PASS (3 sec)
20230804 08:46:01.164: Set Pause Requested Flag
20230804 08:46:04.195: | PASS (3 sec)
Upgrade Setup
20230804 08:46:13.382: Check if upgrade is resumed after pause
20230804 08:46:13.383: | Get Upgrade Paused Flag
20230804 08:46:16.402: | | PASS (3 sec)
20230804 08:46:16.404: | PASS (3 sec)
Proceeding with rolling upgrade
20230804 08:46:47.793: Update system clock
20230804 08:46:52.504: | PASS (4 sec)
20230804 08:46:52.506: OS Update Bundle Deployment
20230804 08:51:08.767: | PASS (4 min 16 sec)
20230804 08:51:08.769: Pre-upgrade OS checks
20230804 08:51:08.771: | Check if local DNS is used
20230804 08:51:08.941: | | PASS
20230804 08:51:08.942: | Validate swap space on installer node
20230804 08:51:09.109: | | PASS
20230804 08:51:09.110: | PASS
20230804 08:51:09.112: Restart fabric-lifecycle containers before upgrading if needed
20230804 08:51:09.321: | PASS
20230804 08:51:09.324: Check SSH Host Keys
20230804 08:51:11.343: | PASS (2 sec)
20230804 08:51:11.344: Prepare directory for package extraction
20230804 08:51:11.680: | PASS
20230804 08:51:11.686: Extract package
20230804 08:51:33.178: | PASS (21 sec)

Dell Technologies Confidential Information version: 2.3.6.182

Page 45 of 76
 20230804 08:51:33.179: Install service tools
20230804 08:52:11.171: | PASS (37 sec)
20230804 08:52:11.188: Check production manifest compatibility for upgrade
We are about to upgrade object on nodes from version 3.8.0.1.138598.3d5db7c96f2 to 3.8.0.3-
138685.3a0a9b6bf3a
20230804 08:52:21.469: | PASS (10 sec)
20230804 08:52:21.472: Check if execution rate parameter was changed
20230804 08:52:21.472: | PASS
20230804 08:52:21.473: Update source object version in ZK
20230804 08:52:26.839: | PASS (5 sec)
20230804 08:52:26.840: Update target version in ZK
20230804 08:52:30.085: | PASS (3 sec)
20230804 08:52:30.086: Cleanup lossless config
Skip: needed when upgrade from 3.6.1 or later to 3.8.0.3 or later
20230804 08:52:36.130: | PASS (6 sec)
20230804 08:52:36.132: Verify the Upgrade Procedure
20230804 08:52:39.157: | PASS (3 sec)
20230804 08:52:39.158: Check for multiple storage pools before upgrade to 3.7
20230804 08:52:42.178: | PASS (3 sec)
20230804 08:52:42.179: Update source OS version in ZK
20230804 08:52:45.369: | PASS (3 sec)
20230804 08:52:45.371: Check object version consistency during upgrade across the cluster
sites
20230804 08:52:53.190: | PASS (7 sec)
20230804 08:52:53.191: Pause DTA scan if needed
Skip: needed when upgrade from 3.3 or 3.4 to 3.4.0.1 or later
20230804 08:52:59.234: | PASS (6 sec)
20230804 08:52:59.235: Check OS version
20230804 08:52:59.923: | PASS
20230804 08:52:59.924: Check missing disks znode
20230804 08:53:08.545: | PASS (8 sec)
20230804 08:53:08.548: Check fabric configuration files
20230804 08:53:08.549: | Check net_interfaces.json
20230804 08:53:09.206: | | PASS
20230804 08:53:09.207: | Check fabric-agent password file
20230804 08:53:10.541: | | PASS (1 sec)
20230804 08:53:10.542: | PASS (1 sec)
20230804 08:53:10.545: Check for partial GC tasks
Skip - not needed for ECS version 3.8.0.1.138598.3d5db7c96f2
20230804 08:53:10.545: | PASS
20230804 08:53:10.546: Disable partial GC
Skip - not needed for upgrade from 3.8.0.1.138598.3d5db7c96f2 to 3.8.0.3-138685.3a0a9b6bf3a
20230804 08:53:10.547: | PASS
20230804 08:53:10.548: Check for partial GC tasks
Skip - not needed for ECS version 3.8.0.1.138598.3d5db7c96f2
20230804 08:53:10.549: | PASS
20230804 08:53:10.550: Validate partition names
20230804 08:53:26.267: | PASS (15 sec)
20230804 08:53:26.268: Validate preset.cfg file
20230804 08:53:29.372: | PASS (3 sec)
20230804 08:53:29.373: Validate NIC FW version
Skip: validate only on ECS 3.4
20230804 08:53:35.668: | PASS (6 sec)

Dell Technologies Confidential Information version: 2.3.6.182

Page 46 of 76
 20230804 08:53:35.670: Validate metering reconstruction status and ensure it is not running
Skip: validate only when upgrade from 3.4 (or lower) to 3.5 (or upper)
20230804 08:53:41.709: | PASS (6 sec)
20230804 08:53:41.710: Validate SP status and ensure it is ready
Skip: validate only when upgrade to 3.5
20230804 08:53:44.730: | PASS (3 sec)
20230804 08:53:44.731: Check for VIPs if needed
Skip: upgrade to 3.4.0.0 / 3.4.0.1 only
20230804 08:53:47.750: | PASS (3 sec)
20230804 08:53:47.751: Validate Switch OS version
Skipping FE switches
20230804 08:54:33.241: | PASS (45 sec)
20230804 08:54:33.243: Update FIPS
20230804 08:54:39.283: | PASS (6 sec)
20230804 08:54:39.285: Run Remove Grafana
20230804 08:54:40.076: | PASS
20230804 08:54:40.077: Cleanup disks for CRU
20230804 08:54:46.112: | PASS (6 sec)
20230804 08:54:46.113: Re-enable Compliance monitoring if required
Cluster compliance monitoring is enabled
Re-enabled compliance monitoring in /opt/emc/caspian/installer/conf/settings.conf
20230804 08:54:47.275: | PASS (1 sec)
20230804 08:54:47.277: Temporary enable DT query port
20230804 08:54:47.446: | PASS
20230804 08:54:47.447: Handle local DNS workaround
20230804 08:54:52.063: | PASS (4 sec)
20230804 08:54:58.107: Disable SSDr read cache
Skip - not needed for upgrade from 3.8.0.1.138598.3d5db7c96f2
20230804 08:54:58.107: | PASS
20230804 08:54:58.108: Disable Callhome
20230804 08:54:58.108: | Set CallHomeEnabled
CallHome is disabled
20230804 08:55:01.118: | | PASS (3 sec)
20230804 08:55:01.119: | PASS (3 sec)
20230804 08:55:01.120: Pre-upgrade BTree GC Cleanup
Skip: target object version 3.8.0.3-138685.3a0a9b6bf3a is higher than or equal to 3.1.0.1.
Use option '--btree-gc-cleanup force' to force BTree GC cleanup.
20230804 08:55:01.121: | PASS
20230804 08:55:01.121: Pre-upgrade GC disable
20230804 08:55:01.122: | PASS
20230804 08:55:04.147: Pre-upgrade duplicated page table journal
20230804 08:55:04.148: | PASS
20230804 08:55:04.149: Switch Object to Upgrade Mode if required
Skip: target object version 3.8.0.3-138685.3a0a9b6bf3a is higher than or equal to 3.2
20230804 08:55:07.171: | PASS (3 sec)
20230804 08:55:07.173: Disable Load Balancing
Disabling Load Balancing, reason: Disabled by Service Console before upgrade
LB is disabled
20230804 08:55:09.844: | PASS (2 sec)
20230804 08:55:09.846: Disable autologout
20230804 08:55:10.025: | PASS
20230804 08:55:10.028: Change override.conf if needed
Skip: source object version is 3.8.0.1.138598.3d5db7c96f2

Dell Technologies Confidential Information version: 2.3.6.182

Page 47 of 76
 20230804 08:55:13.055: | PASS (3 sec)
20230804 08:55:13.058: Create stig-rules.config if required
Skip: source OS version is ecs-os-base-3.8.0.0-2076.a7e36fa.36.noarch (not 3.3)
20230804 08:55:13.234: | PASS
20230804 08:55:13.235: Backup production package on upgrade
20230804 08:56:42.944: | PASS (1 min 29 sec)
20230804 08:56:42.947: Handle Page Table and BTree SR if required
20230804 08:56:42.948: | PASS
20230804 08:56:42.949: Reload Fabric Registry
Not found fabric registry with blobs size overflow. Skip.
20230804 08:56:43.925: | PASS
20230804 08:56:43.926: Set Upgrade Started Flag
20230804 08:56:46.946: | PASS (3 sec)
20230804 08:57:07.755: Check for all SSD case
Skip: upgrade to 3.5 only
20230804 08:57:10.774: | PASS (3 sec)
20230804 08:57:10.775: Disable VNEST reconfiguration
20230804 08:57:17.758: | Check if VNEST is reconfiguring
20230804 08:57:18.446: | | PASS
20230804 08:57:20.691: | Set VNEST configuration parameter value
20230804 08:57:20.696: | | Write item to CF
20230804 08:57:22.973: | | | PASS (2 sec)
20230804 08:57:22.976: | | PASS (2 sec)
20230804 08:57:22.976: | PASS (12 sec)
Patch Commit Config Changes
Next Node Upgrade
Service Console was installed on node 169.254.186.2
The OS bundle is copied to 169.254.186.2
20230804 08:58:44.516: Check object version consistency during upgrade across the cluster
sites
20230804 08:58:52.309: | PASS (7 sec)
20230804 08:58:52.320: Pre-upgrade Service Check
Checking DT status (with timeout 5 min).
20230804 08:58:57.089: | PASS (4 sec)
20230804 08:58:57.091: ${target_node} IN [ @{target_nodes} ]
20230804 08:58:57.092: | ${target_node} = 169.254.186.2
20230804 08:58:57.092: | | OS and Node Upgrade
20230804 08:58:57.095: | | | Check that enough nodes are online
20230804 08:59:01.786: | | | | PASS (4 sec)
20230804 08:59:01.787: | | | Check fabric configuration files
20230804 08:59:01.789: | | | | Check net_interfaces.json
20230804 08:59:01.956: | | | | | PASS
20230804 08:59:01.957: | | | | Check fabric-agent password file
20230804 08:59:02.292: | | | | | PASS
20230804 08:59:02.293: | | | | PASS
20230804 08:59:02.295: | | | OS Update Bundle Deployment
Skip OS update bundle deployment: already deployed
20230804 09:03:01.000: | | | | PASS (3 min 58 sec)
20230804 09:03:01.001: | | | Check if product components should be upgraded
OS update bundle is not installed on host 169.254.186.2
Object is not running on host 169.254.186.2
Product components to be upgraded on node 169.254.186.2: OS - True, Fabric - True, Object -
True

Dell Technologies Confidential Information version: 2.3.6.182

Page 48 of 76
 20230804 09:05:55.349: | | | | PASS (2 min 54 sec)
20230804 09:05:55.354: | | | Disable node recovery before upgrade
Disabled data recovery for node 169.254.186.2
20230804 09:05:55.852: | | | | PASS
20230804 09:05:55.859: | | | Check if VNEST is reconfiguring
20230804 09:05:56.519: | | | | PASS
20230804 09:05:56.525: | | | Node Enter Maintenance Mode
20230804 09:06:53.505: | | | | PASS (56 sec)
20230804 09:06:53.512: | | | Run OS Upgrade
Nodes ['169.254.186.2'] have been upgraded and are going to be rebooted
20230804 09:10:33.728: | | | | PASS (3 min 40 sec)
20230804 09:10:33.729: | | | Reboot node for upgrade
Node 169.254.186.2 reboot initiated with 'shutdown --reboot now' command
[WARN] Node 169.254.186.2 has not rebooted yet; waiting 3343 seconds more;
[ INFO ] Node 169.254.186.2 has rebooted successfully
20230804 09:18:59.193: | | | | PASS (8 min 25 sec)
20230804 09:18:59.194: | | | Apply required OS upgrade workarounds
20230804 09:18:59.362: | | | | PASS
20230804 09:18:59.364: | | | Check DNS after upgrade
20230804 09:18:59.536: | | | | PASS
20230804 09:18:59.537: | | | Reinstall syscfg if needed
20230804 09:19:02.560: | | | | PASS (3 sec)
20230804 09:19:02.563: | | | Reboot node for FW upgrade
20230804 09:19:02.738: | | | | PASS
20230804 09:19:02.740: | | | Validate node NIC FW version for upgrade
Skip: validate only on ECS 3.4
20230804 09:19:05.764: | | | | PASS (3 sec)
20230804 09:19:05.767: | | | Fix ECS 3.4 NAN
20230804 09:19:08.788: | | | | PASS (3 sec)
20230804 09:19:08.794: | | | Verify that OS version is correct
OS version is correct on node 169.254.186.2
20230804 09:19:09.130: | | | | PASS
20230804 09:19:09.133: | | | Sync network separation and static routing
20230804 09:19:10.460: | | | | PASS (1 sec)
20230804 09:19:10.463: | | | Validate network separation
20230804 09:19:11.374: | | | | PASS
20230804 09:19:11.376: | | | Check fabric configuration files
20230804 09:19:11.377: | | | | Check net_interfaces.json
20230804 09:19:11.544: | | | | | PASS
20230804 09:19:11.545: | | | | Check fabric-agent password file
20230804 09:19:11.882: | | | | | PASS
20230804 09:19:11.883: | | | | PASS
20230804 09:19:11.884: | | | Verify that the node has rootFS space available for DT
engine
Skip - this check is valid for upgrade to 3.4 only
20230804 09:19:17.922: | | | | PASS (6 sec)
20230804 09:19:17.924: | | | Node Upgrade
Initializing...

Executing Program: NODE_UPGRADE

|-Save Package version

| +-[0.0.0.0] SavePackageVersion PASS (1/104)

Dell Technologies Confidential Information version: 2.3.6.182

Page 49 of 76
 |-Disable CallHome
| +-[0.0.0.0] SetCallHomeEnabled PASS (2/104)
|-Repair Validation Phase
| |-Validators
| | |-[0.0.0.0] ValidateTopology PASS (3/104)
| | +-[0.0.0.0] ValidateRegisteries PASS (4/104)
| |-Check docker health
| | +-[169.254.186.2] DockerHealth PASS (5/104)
| +-Validate Docker Options
| +-[169.254.186.2] ValidateDockerOptions PASS (6/104)
|-Repair HAL
| |-Check Java Runtime environment
| | +-[169.254.186.2] JavaRuntimeHealth PASS (7/104, 6 sec)
| |-Check Installer tool
| | +-[169.254.186.2] InstallerToolHealth PASS (8/104, 3 sec)
| |-Check HAL Rpm dependencies
| | +-[169.254.186.2] CheckDependencies PASS (9/104, 1 sec)
| |-Stop Agent service
| | +-[169.254.186.2] StopService PASS (10/104)
| |-Force upgrade HAL Rpm
| | +-[169.254.186.2] Update PASS (11/104, 4 sec)
| |-Configure HAL
| | +-[169.254.186.2] ConfigureHAL PASS (12/104, 1 sec)
| +-Start fabric agent service
| +-[169.254.186.2] StartService PASS (13/104)
|-Take Fabric Ownership
| |-[0.0.0.0] TakeOwnership PASS (14/104)
| |-[0.0.0.0] TakeOwnership PASS (15/104)
| |-[0.0.0.0] TakeOwnership PASS (16/104)
| +-[0.0.0.0] TakeOwnership PASS (17/104)
|-Repair FABRIC AGENT
| |-Check Java Runtime environment
| | +-[169.254.186.2] JavaRuntimeHealth PASS (18/104)
| |-Check Installer tool
| | +-[169.254.186.2] InstallerToolHealth PASS (19/104)
| |-Check Agent Rpm dependencies
| | +-[169.254.186.2] CheckDependencies PASS (20/104, 1 sec)
| |-Check FCLI Rpm dependencies
| | +-[169.254.186.2] CheckDependencies PASS (21/104, 1 sec)
| |-Configure Agent
| | +-[169.254.186.2] PrepareAgentConfiguration PASS (22/104, 1 sec)
| |-Install Agent Rpm
| | +-[169.254.186.2] Install PASS (23/104, 24 sec)
| |-Install FCLI Rpm
| | +-[169.254.186.2] Install PASS (24/104, 13 sec)
| |-Set up SSL
| | +-[169.254.186.1] SetupRepairSsl PASS (25/104, 1 min 36
sec)
| |-Verify Agent Health
| | +-[169.254.186.2] AgentHealth PASS (26/104)
| |-Validate Disks
INFO: [169.254.186.2] Number of found disks is 25
| | +-[169.254.186.2] StorageHealth PASS (27/104)

Dell Technologies Confidential Information version: 2.3.6.182

Page 50 of 76
 | |-Emit Topology Info
| | +-[169.254.186.2] EmitTopologyInfo PASS (28/104)
| +-Backup Encryption Keys
| +-[169.254.186.2] BackupFile PASS (29/104)
|-Join Repair Nodes Into Cluster
| |-Join Repair Node
| | +-[169.254.186.2] JoinRepairNode PASS (30/104, 2 sec)
| +-[0.0.0.0] GenerateClusterPassword PASS (31/104, 1 sec)
|-Node Upgrade Registries If Needed
| |-Load Registry image
| | +-[169.254.186.1] LoadImage 'registry' PASS (32/104, 4 sec)
| |-Node Upgrade Registry
| |-Push All Images to Head Registry
| | |-[169.254.186.1] PushImage 'registry' PASS (33/104, 5 sec)
| | |-[169.254.186.1] LoadImage 'zookeeper' PASS (34/104, 24 sec)
| | |-[169.254.186.1] LoadImage 'fabric' PASS (35/104, 22 sec)
| | |-[169.254.186.1] LoadImage 'syslog' PASS (36/104, 22 sec)
| | |-[169.254.186.1] LoadImage 'object' PASS (37/104, 1 min 14
sec)
| | |-[169.254.186.1] LoadImage 'influxdb' PASS (38/104, 10 sec)
| | |-[169.254.186.1] LoadImage 'fluxd' PASS (39/104, 7 sec)
| | |-[169.254.186.1] LoadImage 'throttler' PASS (40/104, 7 sec)
| | |-[169.254.186.1] LoadImage 'telegraf' PASS (41/104, 7 sec)
| | |-[169.254.186.1] LoadImage 'grafana' PASS (42/104, 13 sec)
| | |-[169.254.186.1] PushImage 'zookeeper' PASS (43/104, 31 sec)
| | |-[169.254.186.1] PushImage 'fabric' PASS (44/104, 2 sec)
| | |-[169.254.186.1] PushImage 'syslog' PASS (45/104, 5 sec)
| | |-[169.254.186.1] PushImage 'object' PASS (46/104, 46 sec)
| | |-[169.254.186.1] PushImage 'influxdb' PASS (47/104, 8 sec)
| | |-[169.254.186.1] PushImage 'fluxd' PASS (48/104, 3 sec)
| | |-[169.254.186.1] PushImage 'throttler' PASS (49/104, 2 sec)
| | |-[169.254.186.1] PushImage 'telegraf' PASS (50/104, 3 sec)
| | |-[169.254.186.1] PushImage 'grafana' PASS (51/104, 6 sec)
| +-Push All Images to Remote Registries
|-Push Images to Custom Registries
| |-Load Images
| +-Push Images
|-Upgrade Fabric On Specified Nodes
| |-Initiate Registry Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (52/104)
| |-Initiate ZooKeeper Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (53/104)
| |-Initiate LM Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (54/104)
| |-Initiate Syslog Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (55/104)
| |-Load Fabric images directly from installer node
| +-Load Registry image
|-Return Fabric Ownership to Lifecycle Manager
| |-[0.0.0.0] TakeOwnership PASS (56/104)
| |-[0.0.0.0] TakeOwnership PASS (57/104)
| |-[0.0.0.0] TakeOwnership PASS (58/104)
| +-[0.0.0.0] TakeOwnership PASS (59/104)

Dell Technologies Confidential Information version: 2.3.6.182

Page 51 of 76
 |-Update Fabric Ownership To Lifecycle Manager
| |-Update Ownership For Registry
| | +-[169.254.186.1] UpdateOwnership PASS (60/104)
| |-Update Ownership For ZooKeeper
| | |-[169.254.186.1] UpdateOwnership PASS (61/104)
| | |-[169.254.186.2] UpdateOwnership PASS (62/104)
| | +-[169.254.186.3] UpdateOwnership PASS (63/104)
| |-Update Ownership For LM
| | |-[169.254.186.3] UpdateOwnership PASS (64/104)
| | |-[169.254.186.2] UpdateOwnership PASS (65/104)
| | +-[169.254.186.1] UpdateOwnership PASS (66/104)
| +-Update Ownership For Syslog
| +-[169.254.186.2] UpdateOwnership PASS (67/104)
|-Post-check Fabric Services Health on Specified Nodes
| |-Validate Registry Service on Specified Nodes
| | +-[169.254.186.2] ServiceHealth PASS (68/104)
| |-Validate ZooKeeper Service on Specified Nodes
| | +-[169.254.186.2] ServiceHealth PASS (69/104, 1 min 22
sec)
| |-Validate LM Service on Specified Nodes
| | +-[169.254.186.2] ServiceHealth PASS (70/104)
| +-Validate Syslog Service on Specified Nodes
| +-[169.254.186.2] ServiceHealth PASS (71/104)
|-Upgrade Service On Specified Nodes
| |-Upgrade Metrics On Specified Nodes
| | |-Initiate Influxdb Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (72/104, 2 sec)
| | |-Initiate Fluxd Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (73/104)
| | |-Initiate Throttler Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (74/104)
| | |-Initiate Telegraf Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (75/104)
| | |-Initiate Grafana Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (76/104)
| | |-Load Influxdb image
| | |-Load Fluxd image
| | |-Load Throttler image
| | |-Load Telegraf image
| | +-Load Grafana image
| +-Upgrade Object On Specified Nodes
| |-Initiate Object Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (77/104, 3 sec)
| +-Load Object image
|-Update Services Ownership To Lifecycle Manager
| |-Update Ownership For Object
| | +-[169.254.186.2] UpdateOwnership PASS (78/104)
| |-Update Ownership For Influxdb
| | +-[169.254.186.2] UpdateOwnership PASS (79/104)
| |-Update Ownership For Fluxd
| | +-[169.254.186.2] UpdateOwnership PASS (80/104)
| |-Update Ownership For Throttler
| | +-[169.254.186.2] UpdateOwnership PASS (81/104)

Dell Technologies Confidential Information version: 2.3.6.182

Page 52 of 76
 | |-Update Ownership For Telegraf
| | +-[169.254.186.2] UpdateOwnership PASS (82/104)
| +-Update Ownership For Grafana
| +-[169.254.186.2] UpdateOwnership PASS (83/104)
|-Post-check Services Health
| |-Post-check Object Health
| | +-Validate Object Service on Specified Nodes
| | +-[169.254.186.2] ServiceHealth PASS (84/104, 1 min 33
sec)
| +-Post-check Metrics Health
| |-Validate Influxdb Service on Specified Nodes
| | +-[169.254.186.2] ServiceHealth PASS (85/104, 1 sec)
| |-Validate Fluxd Service on Specified Nodes
| | +-[169.254.186.2] ServiceHealth PASS (86/104)
| |-Validate Throttler Service on Specified Nodes
| | +-[169.254.186.2] ServiceHealth PASS (87/104)
| |-Validate Telegraf Service on Specified Nodes
| | +-[169.254.186.2] ServiceHealth PASS (88/104)
| +-Validate Grafana Service on Specified Nodes
| +-[169.254.186.2] ServiceHealth PASS (89/104)
|-Enable CallHome
| +-[0.0.0.0] SetCallHomeEnabled PASS (90/104)
|-Cleanup Obsolete Dockers
| +-Cleanup Obsolete Docker Containers and Images
|-[169.254.186.2] Removed image emcvipr/object:3.8.0.3-138685.3a0a9b6bf3a
|-[169.254.186.2] Removed image caspian/fabric:3.8.0.3-4354.47090e4
|-[169.254.186.2] Removed image caspian/fabric-zookeeper:3.8.0.3-125.63fdda3
|-[169.254.186.2] Removed image ecs-monitoring/fluxd:3.8.0.2-1551.f3ade7de
|-[169.254.186.2] Removed image ecs-monitoring/throttler:3.8.0.2-1551.f3ade7de
|-[169.254.186.2] Removed image ecs-monitoring/grafana:3.8.0.2-1551.f3ade7de
|-[169.254.186.2] Removed image ecs-monitoring/influxdb:3.8.0.2-1551.f3ade7de
|-[169.254.186.2] Removed image ecs-monitoring/telegraf:3.8.0.2-1551.f3ade7de
|-[169.254.186.2] Removed image emcvipr/object:3.8.0.1-138598.3d5db7c96f2
|-[169.254.186.2] Removed image fabric/syslog:3.8.0.0-4343.878ca95
|-[169.254.186.2] Removed image caspian/fabric:3.8.0.0-4343.878ca95
|-[169.254.186.2] Removed image ecs-monitoring/fluxd:3.8.0.0-1548.8e548035
|-[169.254.186.2] Removed image ecs-monitoring/throttler:3.8.0.0-1548.8e548035
|-[169.254.186.2] Removed image ecs-monitoring/grafana:3.8.0.0-1548.8e548035
|-[169.254.186.2] Removed image ecs-monitoring/influxdb:3.8.0.0-1548.8e548035
|-[169.254.186.2] Removed image ecs-monitoring/telegraf:3.8.0.0-1548.8e548035
|-[169.254.186.2] Removed image caspian/fabric-zookeeper:3.8.0.0-119.78667ce
| +-[0.0.0.0] CleanupDockers PASS (91/104, 11 sec)
+-Init alert for low endurance level
+-[0.0.0.0] InitLowEnduranceLvlAlert PASS (92/104)
Elapsed time is 11 min 14 sec.

NODE_UPGRADE completed successfully

Collecting data from cluster

Information has been written to the
"/opt/emc/caspian/installer/discover/NODE_UPGRADE.2023-08-04T09:31:03/provisioning.txt"
Information has been written to the
"/opt/emc/caspian/installer/discover/NODE_UPGRADE.2023-08-04T09:31:03/topology.txt"

Dell Technologies Confidential Information version: 2.3.6.182

Page 53 of 76
 Executing /configure.sh --start action in object-main container which may take up to 1000
seconds.
20230804 09:31:19.148: | | | | PASS (12 min 1 sec)
20230804 09:31:19.154: | | | Update manifest from production in LM
20230804 09:31:22.369: | | | | PASS (3 sec)
20230804 09:31:22.370: | | | Check if all product components are upgraded
Object is upgraded on host 169.254.186.2
Fabric is upgraded on host 169.254.186.2
20230804 09:31:52.824: | | | | PASS (30 sec)
20230804 09:31:52.830: | | | Node Exit Maintenance Mode
20230804 09:31:56.457: | | | | PASS (3 sec)
20230804 09:31:56.463: | | | Perform Service Check
Executing /configure.sh --start action in object-main container which may take up to 1000
seconds.
Checking DT status (with timeout 5 min).
20230804 09:32:11.990: | | | | PASS (15 sec)
20230804 09:32:11.991: | | | Enable node data recovery after upgrade
Start node recovery permission polling for nodes 169.254.186.2 ...
Data recovery enabled for node(s) 169.254.186.2
20230804 09:32:12.641: | | | | PASS
20230804 09:32:12.643: | | | Disable Load Balancing for Node
Disabling Load Balancing, reason: Disabled by Service Console after Node Upgrade
LB is disabled
20230804 09:32:15.292: | | | | PASS (2 sec)
20230804 09:32:15.294: | | | Apply required post-upgrade workarounds
Service Console was installed on node 169.254.186.2 on previous run
The OS bundle is copied to 169.254.186.2
20230804 09:32:46.588: | | | | PASS (31 sec)
20230804 09:32:46.594: | | | Add firewall to Fabric agent capabilities if needed
20230804 09:32:46.929: | | | | PASS
20230804 09:32:46.930: | | | PASS (33 min 49 sec)
20230804 09:32:46.931: | | Check if the node should have installer
20230804 09:32:46.949: | | | PASS
Service Console was installed on node 169.254.186.2 on previous run
Extracting /tmp/Upgrade/production.tgz to
169.254.186.2:/tmp/service_console_production_package
20230804 09:36:11.725: | | Check Pause Requested Flag
20230804 09:36:11.727: | | | Get Pause Requested Flag
20230804 09:36:14.745: | | | | PASS (3 sec)
20230804 09:36:14.755: | | | PASS (3 sec)
20230804 09:36:14.755: | | PASS (37 min 17 sec)
20230804 09:36:14.756: | PASS (37 min 17 sec)
Upgrade
Local Upgrade Teardown
Global Upgrade Teardown
================================================================================
Status: PASS
Time Elapsed: 51 min 1 sec
Debug log: /opt/emc/caspian/service-console/log/20230804_084532_run_Upgrade/dbg_robot.log
HTML log: /opt/emc/caspian/service-console/log/20230804_084532_run_Upgrade/log.html
================================================================================
Messages:

Dell Technologies Confidential Information version: 2.3.6.182

Page 54 of 76
 To proceed with the next stage of upgrade run 'service-console run Upgrade --upgrade-type
rolling' on the node 169.254.186.2
================================================================================
/opt/emc/service_console/ECSAutomation/sc/tools/version_checker.py:38: YAMLLoadWarning:
calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please
read https://msg.pyyaml.org/load for full details.
allowed_versions_map = yaml.load(f)

Upgrade on remaining nodes, example output from Node 2

The following is an example output for a Rolling upgrade (upgrading the operating system, fabric, and
object services node by node).

/opt/emc/bin/service-console run Upgrade --upgrade-type rolling

Service Console is running on node 169.254.186.2 (suite 20230804_094043_Upgrade)
Service console version: 8.2.0.1-22389.6502d4d05b
Debug log: /opt/emc/caspian/service-console/log/20230804_094041_run_Upgrade/dbg_robot.log
================================================================================
Upgrade Procedure Setup
Proceeding with rolling upgrade
Upgrade Setup
20230804 09:41:11.226: Check if upgrade is resumed after pause
20230804 09:41:11.227: | Get Upgrade Paused Flag
20230804 09:41:14.243: | | PASS (3 sec)
20230804 09:41:14.245: | PASS (3 sec)
Action Succeeded on Previous Run
Patch Commit Config Changes
Next Node Upgrade
Action Succeeded on Previous Run
Upgrade
20230804 09:41:36.472: Check object version consistency during upgrade across the cluster
sites
20230804 09:41:44.216: | PASS (7 sec)
20230804 09:41:44.219: Pre-upgrade Service Check
Checking DT status (with timeout 5 min).
20230804 09:41:49.012: | PASS (4 sec)
20230804 09:41:49.014: ${target_node} IN [ @{target_nodes} ]
20230804 09:41:49.014: | ${target_node} = 169.254.186.2
20230804 09:41:49.015: | | OS and Node Upgrade
Skip - the operation on the node 169.254.186.2 has succeeded on previous run
20230804 09:41:49.024: | | | PASS
20230804 09:41:49.024: | | Check if the node should have installer
20230804 09:41:49.036: | | | PASS
20230804 09:41:49.039: | | Check Pause Requested Flag
20230804 09:41:49.039: | | | Get Pause Requested Flag
20230804 09:41:52.058: | | | | PASS (3 sec)
20230804 09:41:52.068: | | | PASS (3 sec)
20230804 09:41:52.068: | | PASS (3 sec)
20230804 09:41:52.068: | ${target_node} = 169.254.186.1
20230804 09:41:52.069: | | OS and Node Upgrade
20230804 09:41:52.080: | | | Check that enough nodes are online
20230804 09:41:56.788: | | | | PASS (4 sec)

Dell Technologies Confidential Information version: 2.3.6.182

Page 55 of 76
 20230804 09:41:56.790: | | | Check fabric configuration files
20230804 09:41:56.791: | | | | Check net_interfaces.json
20230804 09:41:56.955: | | | | | PASS
20230804 09:41:56.956: | | | | Check fabric-agent password file
20230804 09:41:57.287: | | | | | PASS
20230804 09:41:57.288: | | | | PASS
20230804 09:41:57.290: | | | OS Update Bundle Deployment
Skip OS update bundle deployment: already deployed
20230804 09:45:53.872: | | | | PASS (3 min 56 sec)
20230804 09:45:53.873: | | | Check if product components should be upgraded
OS update bundle is not installed on host 169.254.186.1
Object is not running on host 169.254.186.1
Product components to be upgraded on node 169.254.186.1: OS - True, Fabric - True, Object -
True
20230804 09:48:49.356: | | | | PASS (2 min 55 sec)
20230804 09:48:49.364: | | | Disable node recovery before upgrade
Disabled data recovery for node 169.254.186.1
20230804 09:48:49.849: | | | | PASS
20230804 09:48:49.855: | | | Check if VNEST is reconfiguring
20230804 09:48:50.517: | | | | PASS
20230804 09:48:50.523: | | | Node Enter Maintenance Mode
20230804 09:49:45.145: | | | | PASS (54 sec)
20230804 09:49:45.151: | | | Run OS Upgrade
Nodes ['169.254.186.1'] have been upgraded and are going to be rebooted
20230804 09:53:29.108: | | | | PASS (3 min 43 sec)
20230804 09:53:29.110: | | | Reboot node for upgrade
Node 169.254.186.1 reboot initiated with 'shutdown --reboot now' command
[WARN] Node 169.254.186.1 has not rebooted yet; waiting 3287 seconds more;
[ INFO ] Node 169.254.186.1 has rebooted successfully
20230804 10:02:05.323: | | | | PASS (8 min 36 sec)
20230804 10:02:05.324: | | | Apply required OS upgrade workarounds
20230804 10:02:05.492: | | | | PASS
20230804 10:02:05.493: | | | Check DNS after upgrade
20230804 10:02:05.663: | | | | PASS
20230804 10:02:05.664: | | | Reinstall syscfg if needed
20230804 10:02:08.682: | | | | PASS (3 sec)
20230804 10:02:08.684: | | | Reboot node for FW upgrade
20230804 10:02:08.856: | | | | PASS
20230804 10:02:08.858: | | | Validate node NIC FW version for upgrade
Skip: validate only on ECS 3.4
20230804 10:02:11.875: | | | | PASS (3 sec)
20230804 10:02:11.877: | | | Fix ECS 3.4 NAN
20230804 10:02:14.896: | | | | PASS (3 sec)
20230804 10:02:14.901: | | | Verify that OS version is correct
OS version is correct on node 169.254.186.1
20230804 10:02:15.233: | | | | PASS
20230804 10:02:15.236: | | | Sync network separation and static routing
20230804 10:02:16.570: | | | | PASS (1 sec)
20230804 10:02:16.572: | | | Validate network separation
20230804 10:02:17.497: | | | | PASS
20230804 10:02:17.499: | | | Check fabric configuration files
20230804 10:02:17.500: | | | | Check net_interfaces.json
20230804 10:02:17.665: | | | | | PASS

Dell Technologies Confidential Information version: 2.3.6.182

Page 56 of 76
 20230804 10:02:17.668: | | | | Check fabric-agent password file
20230804 10:02:17.996: | | | | | PASS
20230804 10:02:17.997: | | | | PASS
20230804 10:02:17.998: | | | Verify that the node has rootFS space available for DT
engine
Skip - this check is valid for upgrade to 3.4 only
20230804 10:02:24.033: | | | | PASS (6 sec)
20230804 10:02:24.035: | | | Node Upgrade
Initializing...

Executing Program: NODE_UPGRADE

|-Save Package version

| +-[0.0.0.0] SavePackageVersion PASS (1/105)
|-Disable CallHome
| +-[0.0.0.0] SetCallHomeEnabled PASS (2/105)
|-Repair Validation Phase
| |-Validators
| | |-[0.0.0.0] ValidateTopology PASS (3/105)
| | +-[0.0.0.0] ValidateRegisteries PASS (4/105)
| |-Check docker health
| | +-[169.254.186.1] DockerHealth PASS (5/105)
| +-Validate Docker Options
| +-[169.254.186.1] ValidateDockerOptions PASS (6/105)
|-Repair HAL
| |-Check Java Runtime environment
| | +-[169.254.186.1] JavaRuntimeHealth PASS (7/105)
| |-Check Installer tool
| | +-[169.254.186.1] InstallerToolHealth PASS (8/105, 1 sec)
| |-Check HAL Rpm dependencies
| | +-[169.254.186.1] CheckDependencies PASS (9/105, 1 sec)
| |-Stop Agent service
| | +-[169.254.186.1] StopService PASS (10/105)
| |-Force upgrade HAL Rpm
| | +-[169.254.186.1] Update PASS (11/105, 4 sec)
| |-Configure HAL
| | +-[169.254.186.1] ConfigureHAL PASS (12/105, 1 sec)
| +-Start fabric agent service
| +-[169.254.186.1] StartService PASS (13/105)
|-Take Fabric Ownership
| |-[0.0.0.0] TakeOwnership PASS (14/105)
| |-[0.0.0.0] TakeOwnership PASS (15/105)
| |-[0.0.0.0] TakeOwnership PASS (16/105)
| +-[0.0.0.0] TakeOwnership PASS (17/105)
|-Repair FABRIC AGENT
| |-Check Java Runtime environment
| | +-[169.254.186.1] JavaRuntimeHealth PASS (18/105)
| |-Check Installer tool
| | +-[169.254.186.1] InstallerToolHealth PASS (19/105)
| |-Check Agent Rpm dependencies
| | +-[169.254.186.1] CheckDependencies PASS (20/105, 1 sec)
| |-Check FCLI Rpm dependencies
| | +-[169.254.186.1] CheckDependencies PASS (21/105, 1 sec)

Dell Technologies Confidential Information version: 2.3.6.182

Page 57 of 76
 | |-Configure Agent
| | +-[169.254.186.1] PrepareAgentConfiguration PASS (22/105, 1 sec)
| |-Install Agent Rpm
| | +-[169.254.186.1] Install PASS (23/105, 24 sec)
| |-Install FCLI Rpm
| | +-[169.254.186.1] Install PASS (24/105, 13 sec)
| |-Set up SSL
| | +-[169.254.186.2] SetupRepairSsl PASS (25/105, 1 min 36
sec)
| |-Verify Agent Health
| | +-[169.254.186.1] AgentHealth PASS (26/105)
| |-Validate Disks
INFO: [169.254.186.1] Number of found disks is 25
| | +-[169.254.186.1] StorageHealth PASS (27/105)
| |-Emit Topology Info
| | +-[169.254.186.1] EmitTopologyInfo PASS (28/105)
| +-Backup Encryption Keys
| +-[169.254.186.1] BackupFile PASS (29/105)
|-Join Repair Nodes Into Cluster
| |-Join Repair Node
| | +-[169.254.186.1] JoinRepairNode PASS (30/105, 2 sec)
| +-[0.0.0.0] GenerateClusterPassword PASS (31/105, 1 sec)
|-Node Upgrade Registries If Needed
| |-Load Registry image
| | +-[169.254.186.1] LoadImage 'registry' PASS (32/105, 6 sec)
| |-Node Upgrade Registry
| | +-[169.254.186.1] InstallUpgradeService PASS (33/105, 1 min 35
sec)
| |-Push All Images to Head Registry
| | |-[169.254.186.1] PushImage 'registry' PASS (34/105, 1 sec)
| | |-[169.254.186.1] LoadImage 'zookeeper' PASS (35/105, 21 sec)
| | |-[169.254.186.1] LoadImage 'fabric' PASS (36/105, 24 sec)
| | |-[169.254.186.1] LoadImage 'syslog' PASS (37/105, 43 sec)
| | |-[169.254.186.1] LoadImage 'object' PASS (38/105, 1 min 13
sec)
| | |-[169.254.186.1] LoadImage 'influxdb' PASS (39/105, 8 sec)
| | |-[169.254.186.1] LoadImage 'fluxd' PASS (40/105, 7 sec)
| | |-[169.254.186.1] LoadImage 'throttler' PASS (41/105, 7 sec)
| | |-[169.254.186.1] LoadImage 'telegraf' PASS (42/105, 7 sec)
| | |-[169.254.186.1] LoadImage 'grafana' PASS (43/105, 11 sec)
| | |-[169.254.186.1] PushImage 'zookeeper' PASS (44/105)
| | |-[169.254.186.1] PushImage 'fabric' PASS (45/105)
| | |-[169.254.186.1] PushImage 'syslog' PASS (46/105, 5 sec)
| | |-[169.254.186.1] PushImage 'object' PASS (47/105)
| | |-[169.254.186.1] PushImage 'influxdb' PASS (48/105)
| | |-[169.254.186.1] PushImage 'fluxd' PASS (49/105)
| | |-[169.254.186.1] PushImage 'throttler' PASS (50/105)
| | |-[169.254.186.1] PushImage 'telegraf' PASS (51/105)
| | |-[169.254.186.1] PushImage 'grafana' PASS (52/105)
| +-Push All Images to Remote Registries
|-Push Images to Custom Registries
| |-Load Images
| +-Push Images

Dell Technologies Confidential Information version: 2.3.6.182

Page 58 of 76
 |-Upgrade Fabric On Specified Nodes
| |-Initiate Registry Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (53/105)
| |-Initiate ZooKeeper Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (54/105)
| |-Initiate LM Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (55/105)
| |-Initiate Syslog Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (56/105)
| |-Load Fabric images directly from installer node
| +-Load Registry image
|-Return Fabric Ownership to Lifecycle Manager
| |-[0.0.0.0] TakeOwnership PASS (57/105)
| |-[0.0.0.0] TakeOwnership PASS (58/105)
| |-[0.0.0.0] TakeOwnership PASS (59/105)
| +-[0.0.0.0] TakeOwnership PASS (60/105)
|-Update Fabric Ownership To Lifecycle Manager
| |-Update Ownership For Registry
| | +-[169.254.186.1] UpdateOwnership PASS (61/105)
| |-Update Ownership For ZooKeeper
| | |-[169.254.186.1] UpdateOwnership PASS (62/105)
| | |-[169.254.186.3] UpdateOwnership PASS (63/105)
| | +-[169.254.186.2] UpdateOwnership PASS (64/105)
| |-Update Ownership For LM
| | |-[169.254.186.2] UpdateOwnership PASS (65/105)
| | |-[169.254.186.1] UpdateOwnership PASS (66/105)
| | +-[169.254.186.3] UpdateOwnership PASS (67/105)
| +-Update Ownership For Syslog
| +-[169.254.186.1] UpdateOwnership PASS (68/105)
|-Post-check Fabric Services Health on Specified Nodes
| |-Validate Registry Service on Specified Nodes
| | +-[169.254.186.1] ServiceHealth PASS (69/105)
| |-Validate ZooKeeper Service on Specified Nodes
| | +-[169.254.186.1] ServiceHealth PASS (70/105, 1 min 14
sec)
| |-Validate LM Service on Specified Nodes
| | +-[169.254.186.1] ServiceHealth PASS (71/105)
| +-Validate Syslog Service on Specified Nodes
| +-[169.254.186.1] ServiceHealth PASS (72/105)
|-Upgrade Service On Specified Nodes
| |-Upgrade Metrics On Specified Nodes
| | |-Initiate Influxdb Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (73/105)
| | |-Initiate Fluxd Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (74/105)
| | |-Initiate Throttler Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (75/105)
| | |-Initiate Telegraf Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (76/105)
| | |-Initiate Grafana Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (77/105)
| | |-Load Influxdb image
| | |-Load Fluxd image

Dell Technologies Confidential Information version: 2.3.6.182

Page 59 of 76
 | | |-Load Throttler image
| | |-Load Telegraf image
| | +-Load Grafana image
| +-Upgrade Object On Specified Nodes
| |-Initiate Object Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (78/105, 3 sec)
| +-Load Object image
|-Update Services Ownership To Lifecycle Manager
| |-Update Ownership For Object
| | +-[169.254.186.1] UpdateOwnership PASS (79/105)
| |-Update Ownership For Influxdb
| | +-[169.254.186.1] UpdateOwnership PASS (80/105)
| |-Update Ownership For Fluxd
| | +-[169.254.186.1] UpdateOwnership PASS (81/105)
| |-Update Ownership For Throttler
| | +-[169.254.186.1] UpdateOwnership PASS (82/105)
| |-Update Ownership For Telegraf
| | +-[169.254.186.1] UpdateOwnership PASS (83/105)
| +-Update Ownership For Grafana
| +-[169.254.186.1] UpdateOwnership PASS (84/105)
|-Post-check Services Health
| |-Post-check Object Health
| | +-Validate Object Service on Specified Nodes
| | +-[169.254.186.1] ServiceHealth PASS (85/105, 1 min 13
sec)
| +-Post-check Metrics Health
| |-Validate Influxdb Service on Specified Nodes
| | +-[169.254.186.1] ServiceHealth PASS (86/105)
| |-Validate Fluxd Service on Specified Nodes
| | +-[169.254.186.1] ServiceHealth PASS (87/105)
| |-Validate Throttler Service on Specified Nodes
| | +-[169.254.186.1] ServiceHealth PASS (88/105)
| |-Validate Telegraf Service on Specified Nodes
| | +-[169.254.186.1] ServiceHealth PASS (89/105)
| +-Validate Grafana Service on Specified Nodes
| +-[169.254.186.1] ServiceHealth PASS (90/105)
|-Enable CallHome
| +-[0.0.0.0] SetCallHomeEnabled PASS (91/105)
|-Cleanup Obsolete Dockers
| +-Cleanup Obsolete Docker Containers and Images
|-[169.254.186.1] Removed image emcvipr/object:3.8.0.3-138685.3a0a9b6bf3a
|-[169.254.186.1] Removed image fabric/syslog:3.8.0.3-4354.47090e4
|-[169.254.186.1] Removed image caspian/fabric:3.8.0.3-4354.47090e4
|-[169.254.186.1] Removed image caspian/fabric-zookeeper:3.8.0.3-125.63fdda3
|-[169.254.186.1] Removed image ecs-monitoring/fluxd:3.8.0.2-1551.f3ade7de
|-[169.254.186.1] Removed image ecs-monitoring/throttler:3.8.0.2-1551.f3ade7de
|-[169.254.186.1] Removed image ecs-monitoring/grafana:3.8.0.2-1551.f3ade7de
|-[169.254.186.1] Removed image ecs-monitoring/influxdb:3.8.0.2-1551.f3ade7de
|-[169.254.186.1] Removed image ecs-monitoring/telegraf:3.8.0.2-1551.f3ade7de
|-[169.254.186.1] Removed image emcvipr/object:3.8.0.1-138598.3d5db7c96f2
|-[169.254.186.1] Removed image fabric/syslog:3.8.0.0-4343.878ca95
|-[169.254.186.1] Removed image caspian/fabric:3.8.0.0-4343.878ca95
|-[169.254.186.1] Removed image ecs-monitoring/fluxd:3.8.0.0-1548.8e548035

Dell Technologies Confidential Information version: 2.3.6.182

Page 60 of 76
 |-[169.254.186.1] Removed image ecs-monitoring/throttler:3.8.0.0-1548.8e548035
|-[169.254.186.1] Removed image ecs-monitoring/grafana:3.8.0.0-1548.8e548035
|-[169.254.186.1] Removed image ecs-monitoring/influxdb:3.8.0.0-1548.8e548035
|-[169.254.186.1] Removed image ecs-monitoring/telegraf:3.8.0.0-1548.8e548035
|-[169.254.186.1] Removed image caspian/fabric-registry:2.3.1.0-82.c8163d2
|-[169.254.186.1] Removed image caspian/fabric-zookeeper:3.8.0.0-119.78667ce
| +-[0.0.0.0] CleanupDockers PASS (92/105, 11 sec)
+-Init alert for low endurance level
+-[0.0.0.0] InitLowEnduranceLvlAlert PASS (93/105)
Elapsed time is 10 min 44 sec.

NODE_UPGRADE completed successfully

Collecting data from cluster

Information has been written to the
"/opt/emc/caspian/installer/discover/NODE_UPGRADE.2023-08-04T10:13:38/provisioning.txt"
Information has been written to the
"/opt/emc/caspian/installer/discover/NODE_UPGRADE.2023-08-04T10:13:38/topology.txt"
Executing /configure.sh --start action in object-main container which may take up to 1000
seconds.
20230804 10:13:54.281: | | | | PASS (11 min 30 sec)
20230804 10:13:54.305: | | | Update manifest from production in LM
20230804 10:13:57.494: | | | | PASS (3 sec)
20230804 10:13:57.495: | | | Check if all product components are upgraded
Object is upgraded on host 169.254.186.1
Fabric is upgraded on host 169.254.186.1
20230804 10:14:25.999: | | | | PASS (28 sec)
20230804 10:14:26.004: | | | Node Exit Maintenance Mode
20230804 10:14:29.769: | | | | PASS (3 sec)
20230804 10:14:29.774: | | | Perform Service Check
Executing /configure.sh --start action in object-main container which may take up to 1000
seconds.
Checking DT status (with timeout 5 min).
20230804 10:14:45.313: | | | | PASS (15 sec)
20230804 10:14:45.315: | | | Enable node data recovery after upgrade
Start node recovery permission polling for nodes 169.254.186.1 ...
Data recovery enabled for node(s) 169.254.186.1
20230804 10:14:45.944: | | | | PASS
20230804 10:14:45.946: | | | Disable Load Balancing for Node
Disabling Load Balancing, reason: Disabled by Service Console after Node Upgrade
LB is disabled
20230804 10:14:48.631: | | | | PASS (2 sec)
20230804 10:14:48.632: | | | Apply required post-upgrade workarounds
20230804 10:14:48.642: | | | | PASS
20230804 10:14:48.647: | | | Add firewall to Fabric agent capabilities if needed
20230804 10:14:48.973: | | | | PASS
20230804 10:14:48.974: | | | PASS (32 min 56 sec)
20230804 10:14:48.975: | | Check if the node should have installer
20230804 10:14:48.994: | | | PASS
Service Console was installed on node 169.254.186.1 on previous run
Checking production package /tmp/service_console_production_package/production.tgz
Extracting /tmp/service_console_production_package/production.tgz to
169.254.186.1:/tmp/service_console_production_package

Dell Technologies Confidential Information version: 2.3.6.182

Page 61 of 76
 20230804 10:18:50.338: | | Check Pause Requested Flag
20230804 10:18:50.339: | | | Get Pause Requested Flag
20230804 10:18:53.357: | | | | PASS (3 sec)
20230804 10:18:53.367: | | | PASS (3 sec)
20230804 10:18:53.367: | | PASS (37 min 1 sec)
20230804 10:18:53.367: | ${target_node} = 169.254.186.3
20230804 10:18:53.368: | | OS and Node Upgrade
20230804 10:18:53.378: | | | Check that enough nodes are online
20230804 10:18:58.092: | | | | PASS (4 sec)
20230804 10:18:58.095: | | | Check fabric configuration files
20230804 10:18:58.096: | | | | Check net_interfaces.json
20230804 10:18:58.259: | | | | | PASS
20230804 10:18:58.260: | | | | Check fabric-agent password file
20230804 10:18:58.587: | | | | | PASS
20230804 10:18:58.587: | | | | PASS
20230804 10:18:58.589: | | | OS Update Bundle Deployment
Skip OS update bundle deployment: already deployed
20230804 10:22:52.655: | | | | PASS (3 min 54 sec)
20230804 10:22:52.657: | | | Check if product components should be upgraded
OS update bundle is not installed on host 169.254.186.3
Object is not running on host 169.254.186.3
Product components to be upgraded on node 169.254.186.3: OS - True, Fabric - True, Object -
True
20230804 10:24:09.518: | | | | PASS (1 min 16 sec)
20230804 10:24:09.523: | | | Disable node recovery before upgrade
Disabled data recovery for node 169.254.186.3
20230804 10:24:10.024: | | | | PASS
20230804 10:24:10.029: | | | Check if VNEST is reconfiguring
20230804 10:24:10.501: | | | | PASS
20230804 10:24:10.506: | | | Node Enter Maintenance Mode
20230804 10:25:07.610: | | | | PASS (57 sec)
20230804 10:25:07.617: | | | Run OS Upgrade
Nodes ['169.254.186.3'] have been upgraded and are going to be rebooted
20230804 10:28:43.909: | | | | PASS (3 min 36 sec)
20230804 10:28:43.910: | | | Reboot node for upgrade
Node 169.254.186.3 reboot initiated with 'shutdown --reboot now' command
[WARN] Node 169.254.186.3 has not rebooted yet; waiting 3335 seconds more;
[ INFO ] Node 169.254.186.3 has rebooted successfully
20230804 10:36:56.480: | | | | PASS (8 min 12 sec)
20230804 10:36:56.481: | | | Apply required OS upgrade workarounds
20230804 10:36:56.645: | | | | PASS
20230804 10:36:56.646: | | | Check DNS after upgrade
20230804 10:36:56.814: | | | | PASS
20230804 10:36:56.815: | | | Reinstall syscfg if needed
20230804 10:36:59.835: | | | | PASS (3 sec)
20230804 10:36:59.838: | | | Reboot node for FW upgrade
20230804 10:37:00.012: | | | | PASS
20230804 10:37:00.015: | | | Validate node NIC FW version for upgrade
Skip: validate only on ECS 3.4
20230804 10:37:03.032: | | | | PASS (3 sec)
20230804 10:37:03.035: | | | Fix ECS 3.4 NAN
20230804 10:37:06.056: | | | | PASS (3 sec)
20230804 10:37:06.061: | | | Verify that OS version is correct

Dell Technologies Confidential Information version: 2.3.6.182

Page 62 of 76
 OS version is correct on node 169.254.186.3
20230804 10:37:06.396: | | | | PASS
20230804 10:37:06.398: | | | Sync network separation and static routing
20230804 10:37:07.704: | | | | PASS (1 sec)
20230804 10:37:07.706: | | | Validate network separation
20230804 10:37:08.604: | | | | PASS
20230804 10:37:08.606: | | | Check fabric configuration files
20230804 10:37:08.607: | | | | Check net_interfaces.json
20230804 10:37:08.772: | | | | | PASS
20230804 10:37:08.773: | | | | Check fabric-agent password file
20230804 10:37:09.099: | | | | | PASS
20230804 10:37:09.100: | | | | PASS
20230804 10:37:09.101: | | | Verify that the node has rootFS space available for DT
engine
Skip - this check is valid for upgrade to 3.4 only
20230804 10:37:15.140: | | | | PASS (6 sec)
20230804 10:37:15.142: | | | Node Upgrade
Initializing...

Executing Program: NODE_UPGRADE

|-Save Package version

| +-[0.0.0.0] SavePackageVersion PASS (1/104)
|-Disable CallHome
| +-[0.0.0.0] SetCallHomeEnabled PASS (2/104)
|-Repair Validation Phase
| |-Validators
| | |-[0.0.0.0] ValidateTopology PASS (3/104)
| | +-[0.0.0.0] ValidateRegisteries PASS (4/104)
| |-Check docker health
| | +-[169.254.186.3] DockerHealth PASS (5/104)
| +-Validate Docker Options
| +-[169.254.186.3] ValidateDockerOptions PASS (6/104)
|-Repair HAL
| |-Check Java Runtime environment
| | +-[169.254.186.3] JavaRuntimeHealth PASS (7/104, 6 sec)
| |-Check Installer tool
| | +-[169.254.186.3] InstallerToolHealth PASS (8/104, 3 sec)
| |-Check HAL Rpm dependencies
| | +-[169.254.186.3] CheckDependencies PASS (9/104, 1 sec)
| |-Stop Agent service
| | +-[169.254.186.3] StopService PASS (10/104)
| |-Force upgrade HAL Rpm
| | +-[169.254.186.3] Update PASS (11/104, 4 sec)
| |-Configure HAL
| | +-[169.254.186.3] ConfigureHAL PASS (12/104, 1 sec)
| +-Start fabric agent service
| +-[169.254.186.3] StartService PASS (13/104)
|-Take Fabric Ownership
| |-[0.0.0.0] TakeOwnership PASS (14/104)
| |-[0.0.0.0] TakeOwnership PASS (15/104)
| |-[0.0.0.0] TakeOwnership PASS (16/104)
| +-[0.0.0.0] TakeOwnership PASS (17/104)

Dell Technologies Confidential Information version: 2.3.6.182

Page 63 of 76
 |-Repair FABRIC AGENT
| |-Check Java Runtime environment
| | +-[169.254.186.3] JavaRuntimeHealth PASS (18/104)
| |-Check Installer tool
| | +-[169.254.186.3] InstallerToolHealth PASS (19/104)
| |-Check Agent Rpm dependencies
| | +-[169.254.186.3] CheckDependencies PASS (20/104, 1 sec)
| |-Check FCLI Rpm dependencies
| | +-[169.254.186.3] CheckDependencies PASS (21/104, 1 sec)
| |-Configure Agent
| | +-[169.254.186.3] PrepareAgentConfiguration PASS (22/104, 1 sec)
| |-Install Agent Rpm
| | +-[169.254.186.3] Install PASS (23/104, 24 sec)
| |-Install FCLI Rpm
| | +-[169.254.186.3] Install PASS (24/104, 13 sec)
| |-Set up SSL
| | +-[169.254.186.2] SetupRepairSsl PASS (25/104, 1 min 36
sec)
| |-Verify Agent Health
| | +-[169.254.186.3] AgentHealth PASS (26/104)
| |-Validate Disks
INFO: [169.254.186.3] Number of found disks is 23
| | +-[169.254.186.3] StorageHealth PASS (27/104)
| |-Emit Topology Info
| | +-[169.254.186.3] EmitTopologyInfo PASS (28/104)
| +-Backup Encryption Keys
| +-[169.254.186.3] BackupFile PASS (29/104)
|-Join Repair Nodes Into Cluster
| |-Join Repair Node
| | +-[169.254.186.3] JoinRepairNode PASS (30/104, 2 sec)
| +-[0.0.0.0] GenerateClusterPassword PASS (31/104, 1 sec)
|-Node Upgrade Registries If Needed
| |-Load Registry image
| | +-[169.254.186.1] LoadImage 'registry' PASS (32/104, 4 sec)
| |-Node Upgrade Registry
| |-Push All Images to Head Registry
| | |-[169.254.186.1] PushImage 'registry' PASS (33/104)
| | |-[169.254.186.1] LoadImage 'zookeeper' PASS (34/104, 22 sec)
| | |-[169.254.186.1] LoadImage 'fabric' PASS (35/104, 25 sec)
| | |-[169.254.186.1] LoadImage 'syslog' PASS (36/104, 25 sec)
| | |-[169.254.186.1] LoadImage 'object' PASS (37/104, 1 min 15
sec)
| | |-[169.254.186.1] LoadImage 'influxdb' PASS (38/104, 10 sec)
| | |-[169.254.186.1] LoadImage 'fluxd' PASS (39/104, 7 sec)
| | |-[169.254.186.1] LoadImage 'throttler' PASS (40/104, 7 sec)
| | |-[169.254.186.1] LoadImage 'telegraf' PASS (41/104, 7 sec)
| | |-[169.254.186.1] LoadImage 'grafana' PASS (42/104, 12 sec)
| | |-[169.254.186.1] PushImage 'zookeeper' PASS (43/104)
| | |-[169.254.186.1] PushImage 'fabric' PASS (44/104)
| | |-[169.254.186.1] PushImage 'syslog' PASS (45/104)
| | |-[169.254.186.1] PushImage 'object' PASS (46/104)
| | |-[169.254.186.1] PushImage 'influxdb' PASS (47/104)
| | |-[169.254.186.1] PushImage 'fluxd' PASS (48/104)

Dell Technologies Confidential Information version: 2.3.6.182

Page 64 of 76
 | | |-[169.254.186.1] PushImage 'throttler' PASS (49/104)
| | |-[169.254.186.1] PushImage 'telegraf' PASS (50/104)
| | |-[169.254.186.1] PushImage 'grafana' PASS (51/104)
| +-Push All Images to Remote Registries
|-Push Images to Custom Registries
| |-Load Images
| +-Push Images
|-Upgrade Fabric On Specified Nodes
| |-Initiate Registry Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (52/104)
| |-Initiate ZooKeeper Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (53/104)
| |-Initiate LM Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (54/104)
| |-Initiate Syslog Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (55/104)
| |-Load Fabric images directly from installer node
| +-Load Registry image
|-Return Fabric Ownership to Lifecycle Manager
| |-[0.0.0.0] TakeOwnership PASS (56/104)
| |-[0.0.0.0] TakeOwnership PASS (57/104)
| |-[0.0.0.0] TakeOwnership PASS (58/104)
| +-[0.0.0.0] TakeOwnership PASS (59/104)
|-Update Fabric Ownership To Lifecycle Manager
| |-Update Ownership For Registry
| | +-[169.254.186.1] UpdateOwnership PASS (60/104)
| |-Update Ownership For ZooKeeper
| | |-[169.254.186.1] UpdateOwnership PASS (61/104)
| | |-[169.254.186.2] UpdateOwnership PASS (62/104)
| | +-[169.254.186.3] UpdateOwnership PASS (63/104)
| |-Update Ownership For LM
| | |-[169.254.186.3] UpdateOwnership PASS (64/104)
| | |-[169.254.186.2] UpdateOwnership PASS (65/104)
| | +-[169.254.186.1] UpdateOwnership PASS (66/104)
| +-Update Ownership For Syslog
| +-[169.254.186.3] UpdateOwnership PASS (67/104)
|-Post-check Fabric Services Health on Specified Nodes
| |-Validate Registry Service on Specified Nodes
| | +-[169.254.186.3] ServiceHealth PASS (68/104)
| |-Validate ZooKeeper Service on Specified Nodes
| | +-[169.254.186.3] ServiceHealth PASS (69/104, 3 min 24
sec)
| |-Validate LM Service on Specified Nodes
| | +-[169.254.186.3] ServiceHealth PASS (70/104)
| +-Validate Syslog Service on Specified Nodes
| +-[169.254.186.3] ServiceHealth PASS (71/104)
|-Upgrade Service On Specified Nodes
| |-Upgrade Metrics On Specified Nodes
| | |-Initiate Influxdb Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (72/104)
| | |-Initiate Fluxd Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (73/104)
| | |-Initiate Throttler Upgrade if Required

Dell Technologies Confidential Information version: 2.3.6.182

Page 65 of 76
 | | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (74/104)
| | |-Initiate Telegraf Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (75/104)
| | |-Initiate Grafana Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (76/104)
| | |-Load Influxdb image
| | |-Load Fluxd image
| | |-Load Throttler image
| | |-Load Telegraf image
| | +-Load Grafana image
| +-Upgrade Object On Specified Nodes
| |-Initiate Object Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (77/104, 4 sec)
| +-Load Object image
|-Update Services Ownership To Lifecycle Manager
| |-Update Ownership For Object
| | +-[169.254.186.3] UpdateOwnership PASS (78/104)
| |-Update Ownership For Influxdb
| | +-[169.254.186.3] UpdateOwnership PASS (79/104)
| |-Update Ownership For Fluxd
| | +-[169.254.186.3] UpdateOwnership PASS (80/104)
| |-Update Ownership For Throttler
| | +-[169.254.186.3] UpdateOwnership PASS (81/104)
| |-Update Ownership For Telegraf
| | +-[169.254.186.3] UpdateOwnership PASS (82/104)
| +-Update Ownership For Grafana
| +-[169.254.186.3] UpdateOwnership PASS (83/104)
|-Post-check Services Health
| |-Post-check Object Health
| | +-Validate Object Service on Specified Nodes
| | +-[169.254.186.3] ServiceHealth PASS (84/104, 1 min 33
sec)
| +-Post-check Metrics Health
| |-Validate Influxdb Service on Specified Nodes
| | +-[169.254.186.3] ServiceHealth PASS (85/104, 1 sec)
| |-Validate Fluxd Service on Specified Nodes
| | +-[169.254.186.3] ServiceHealth PASS (86/104, 1 sec)
| |-Validate Throttler Service on Specified Nodes
| | +-[169.254.186.3] ServiceHealth PASS (87/104)
| |-Validate Telegraf Service on Specified Nodes
| | +-[169.254.186.3] ServiceHealth PASS (88/104, 1 sec)
| +-Validate Grafana Service on Specified Nodes
| +-[169.254.186.3] ServiceHealth PASS (89/104)
|-Enable CallHome
| +-[0.0.0.0] SetCallHomeEnabled PASS (90/104)
|-Cleanup Obsolete Dockers
| +-Cleanup Obsolete Docker Containers and Images
|-[169.254.186.3] Removed image emcvipr/object:3.8.0.3-138685.3a0a9b6bf3a
|-[169.254.186.3] Removed image ecs-monitoring/fluxd:3.8.0.2-1551.f3ade7de
|-[169.254.186.3] Removed image ecs-monitoring/grafana:3.8.0.2-1551.f3ade7de
|-[169.254.186.3] Removed image ecs-monitoring/influxdb:3.8.0.2-1551.f3ade7de
|-[169.254.186.3] Removed image ecs-monitoring/telegraf:3.8.0.2-1551.f3ade7de
|-[169.254.186.3] Removed image emcvipr/object:3.8.0.1-138598.3d5db7c96f2

Dell Technologies Confidential Information version: 2.3.6.182

Page 66 of 76
 |-[169.254.186.3] Removed image fabric/syslog:3.8.0.0-4343.878ca95
|-[169.254.186.3] Removed image caspian/fabric:3.8.0.0-4343.878ca95
|-[169.254.186.3] Removed image ecs-monitoring/fluxd:3.8.0.0-1548.8e548035
|-[169.254.186.3] Removed image ecs-monitoring/grafana:3.8.0.0-1548.8e548035
|-[169.254.186.3] Removed image ecs-monitoring/influxdb:3.8.0.0-1548.8e548035
|-[169.254.186.3] Removed image ecs-monitoring/telegraf:3.8.0.0-1548.8e548035
|-[169.254.186.3] Removed image caspian/fabric-zookeeper:3.8.0.0-119.78667ce
| +-[0.0.0.0] CleanupDockers PASS (91/104, 10 sec)
+-Init alert for low endurance level
+-[0.0.0.0] InitLowEnduranceLvlAlert PASS (92/104)
Elapsed time is 11 min 28 sec.

NODE_UPGRADE completed successfully

Collecting data from cluster

Information has been written to the
"/opt/emc/caspian/installer/discover/NODE_UPGRADE.2023-08-04T10:49:14/provisioning.txt"
Information has been written to the
"/opt/emc/caspian/installer/discover/NODE_UPGRADE.2023-08-04T10:49:14/topology.txt"
Executing /configure.sh --start action in object-main container which may take up to 1000
seconds.
20230804 10:49:30.065: | | | | PASS (12 min 14 sec)
20230804 10:49:30.091: | | | Update manifest from production in LM
20230804 10:49:34.291: | | | | PASS (4 sec)
20230804 10:49:34.292: | | | Check if all product components are upgraded
Object is upgraded on host 169.254.186.3
Fabric is upgraded on host 169.254.186.3
20230804 10:50:03.797: | | | | PASS (29 sec)
20230804 10:50:03.803: | | | Node Exit Maintenance Mode
20230804 10:50:07.245: | | | | PASS (3 sec)
20230804 10:50:07.251: | | | Perform Service Check
Executing /configure.sh --start action in object-main container which may take up to 1000
seconds.
Checking DT status (with timeout 5 min).
20230804 10:50:22.800: | | | | PASS (15 sec)
20230804 10:50:22.801: | | | Enable node data recovery after upgrade
Start node recovery permission polling for nodes 169.254.186.3 ...
Data recovery enabled for node(s) 169.254.186.3
20230804 10:50:23.434: | | | | PASS
20230804 10:50:23.436: | | | Disable Load Balancing for Node
Disabling Load Balancing, reason: Disabled by Service Console after Node Upgrade
LB is disabled
20230804 10:50:26.340: | | | | PASS (2 sec)
20230804 10:50:26.342: | | | Apply required post-upgrade workarounds
20230804 10:50:26.351: | | | | PASS
20230804 10:50:26.355: | | | Add firewall to Fabric agent capabilities if needed
20230804 10:50:26.702: | | | | PASS
20230804 10:50:26.703: | | | PASS (31 min 33 sec)
20230804 10:50:26.704: | | Check if the node should have installer
20230804 10:50:26.723: | | | PASS
20230804 10:50:26.725: | | Check Pause Requested Flag
20230804 10:50:26.725: | | | Get Pause Requested Flag
20230804 10:50:29.742: | | | | PASS (3 sec)

Dell Technologies Confidential Information version: 2.3.6.182

Page 67 of 76
 20230804 10:50:29.754: | | | PASS (3 sec)
20230804 10:50:29.754: | | PASS (31 min 36 sec)
20230804 10:50:29.754: | ${target_node} = 169.254.186.4
20230804 10:50:29.755: | | OS and Node Upgrade
20230804 10:50:29.771: | | | Check that enough nodes are online
20230804 10:50:34.511: | | | | PASS (4 sec)
20230804 10:50:34.513: | | | Check fabric configuration files
20230804 10:50:34.514: | | | | Check net_interfaces.json
20230804 10:50:34.679: | | | | | PASS
20230804 10:50:34.680: | | | | Check fabric-agent password file
20230804 10:50:35.015: | | | | | PASS
20230804 10:50:35.016: | | | | PASS
20230804 10:50:35.018: | | | OS Update Bundle Deployment
Skip OS update bundle deployment: already deployed
20230804 10:54:28.919: | | | | PASS (3 min 53 sec)
20230804 10:54:28.921: | | | Check if product components should be upgraded
OS update bundle is not installed on host 169.254.186.4
Object is not running on host 169.254.186.4
Product components to be upgraded on node 169.254.186.4: OS - True, Fabric - True, Object -
True
20230804 10:55:45.880: | | | | PASS (1 min 16 sec)
20230804 10:55:45.885: | | | Disable node recovery before upgrade
Disabled data recovery for node 169.254.186.4
20230804 10:55:46.386: | | | | PASS
20230804 10:55:46.394: | | | Check if VNEST is reconfiguring
20230804 10:55:46.897: | | | | PASS
20230804 10:55:46.902: | | | Node Enter Maintenance Mode
20230804 10:56:45.249: | | | | PASS (58 sec)
20230804 10:56:45.259: | | | Run OS Upgrade
Nodes ['169.254.186.4'] have been upgraded and are going to be rebooted
20230804 11:00:21.558: | | | | PASS (3 min 36 sec)
20230804 11:00:21.560: | | | Reboot node for upgrade
Node 169.254.186.4 reboot initiated with 'shutdown --reboot now' command
[WARN] Node 169.254.186.4 has not rebooted yet; waiting 3362 seconds more;
[ INFO ] Node 169.254.186.4 has rebooted successfully
20230804 11:08:40.713: | | | | PASS (8 min 19 sec)
20230804 11:08:40.714: | | | Apply required OS upgrade workarounds
20230804 11:08:40.883: | | | | PASS
20230804 11:08:40.884: | | | Check DNS after upgrade
20230804 11:08:41.052: | | | | PASS
20230804 11:08:41.054: | | | Reinstall syscfg if needed
20230804 11:08:44.075: | | | | PASS (3 sec)
20230804 11:08:44.078: | | | Reboot node for FW upgrade
20230804 11:08:44.254: | | | | PASS
20230804 11:08:44.257: | | | Validate node NIC FW version for upgrade
Skip: validate only on ECS 3.4
20230804 11:08:47.280: | | | | PASS (3 sec)
20230804 11:08:47.282: | | | Fix ECS 3.4 NAN
20230804 11:08:50.301: | | | | PASS (3 sec)
20230804 11:08:50.306: | | | Verify that OS version is correct
OS version is correct on node 169.254.186.4
20230804 11:08:50.642: | | | | PASS
20230804 11:08:50.644: | | | Sync network separation and static routing

Dell Technologies Confidential Information version: 2.3.6.182

Page 68 of 76
 20230804 11:08:51.958: | | | | PASS (1 sec)
20230804 11:08:51.960: | | | Validate network separation
20230804 11:08:52.868: | | | | PASS
20230804 11:08:52.870: | | | Check fabric configuration files
20230804 11:08:52.871: | | | | Check net_interfaces.json
20230804 11:08:53.038: | | | | | PASS
20230804 11:08:53.040: | | | | Check fabric-agent password file
20230804 11:08:53.372: | | | | | PASS
20230804 11:08:53.373: | | | | PASS
20230804 11:08:53.374: | | | Verify that the node has rootFS space available for DT
engine
Skip - this check is valid for upgrade to 3.4 only
20230804 11:08:59.410: | | | | PASS (6 sec)
20230804 11:08:59.412: | | | Node Upgrade
Initializing...

Executing Program: NODE_UPGRADE

|-Save Package version

| +-[0.0.0.0] SavePackageVersion PASS (1/104)
|-Disable CallHome
| +-[0.0.0.0] SetCallHomeEnabled PASS (2/104)
|-Repair Validation Phase
| |-Validators
| | |-[0.0.0.0] ValidateTopology PASS (3/104)
| | +-[0.0.0.0] ValidateRegisteries PASS (4/104)
| |-Check docker health
| | +-[169.254.186.4] DockerHealth PASS (5/104)
| +-Validate Docker Options
| +-[169.254.186.4] ValidateDockerOptions PASS (6/104)
|-Repair HAL
| |-Check Java Runtime environment
| | +-[169.254.186.4] JavaRuntimeHealth PASS (7/104, 6 sec)
| |-Check Installer tool
| | +-[169.254.186.4] InstallerToolHealth PASS (8/104, 3 sec)
| |-Check HAL Rpm dependencies
| | +-[169.254.186.4] CheckDependencies PASS (9/104, 1 sec)
| |-Stop Agent service
| | +-[169.254.186.4] StopService PASS (10/104)
| |-Force upgrade HAL Rpm
| | +-[169.254.186.4] Update PASS (11/104, 4 sec)
| |-Configure HAL
| | +-[169.254.186.4] ConfigureHAL PASS (12/104, 1 sec)
| +-Start fabric agent service
| +-[169.254.186.4] StartService PASS (13/104)
|-Take Fabric Ownership
| |-[0.0.0.0] TakeOwnership PASS (14/104)
| |-[0.0.0.0] TakeOwnership PASS (15/104)
| |-[0.0.0.0] TakeOwnership PASS (16/104)
| +-[0.0.0.0] TakeOwnership PASS (17/104)
|-Repair FABRIC AGENT
| |-Check Java Runtime environment
| | +-[169.254.186.4] JavaRuntimeHealth PASS (18/104)

Dell Technologies Confidential Information version: 2.3.6.182

Page 69 of 76
 | |-Check Installer tool
| | +-[169.254.186.4] InstallerToolHealth PASS (19/104)
| |-Check Agent Rpm dependencies
| | +-[169.254.186.4] CheckDependencies PASS (20/104, 1 sec)
| |-Check FCLI Rpm dependencies
| | +-[169.254.186.4] CheckDependencies PASS (21/104, 1 sec)
| |-Configure Agent
| | +-[169.254.186.4] PrepareAgentConfiguration PASS (22/104, 1 sec)
| |-Install Agent Rpm
| | +-[169.254.186.4] Install PASS (23/104, 24 sec)
| |-Install FCLI Rpm
| | +-[169.254.186.4] Install PASS (24/104, 13 sec)
| |-Set up SSL
| | +-[169.254.186.2] SetupRepairSsl PASS (25/104, 1 min 36
sec)
| |-Verify Agent Health
| | +-[169.254.186.4] AgentHealth PASS (26/104)
| |-Validate Disks
INFO: [169.254.186.4] Number of found disks is 25
| | +-[169.254.186.4] StorageHealth PASS (27/104)
| |-Emit Topology Info
| | +-[169.254.186.4] EmitTopologyInfo PASS (28/104)
| +-Backup Encryption Keys
| +-[169.254.186.4] BackupFile PASS (29/104)
|-Join Repair Nodes Into Cluster
| |-Join Repair Node
| | +-[169.254.186.4] JoinRepairNode PASS (30/104, 3 sec)
| +-[0.0.0.0] GenerateClusterPassword PASS (31/104, 1 sec)
|-Node Upgrade Registries If Needed
| |-Load Registry image
| | +-[169.254.186.1] LoadImage 'registry' PASS (32/104, 4 sec)
| |-Node Upgrade Registry
| |-Push All Images to Head Registry
| | |-[169.254.186.1] PushImage 'registry' PASS (33/104)
| | |-[169.254.186.1] LoadImage 'zookeeper' PASS (34/104, 22 sec)
| | |-[169.254.186.1] LoadImage 'fabric' PASS (35/104, 25 sec)
| | |-[169.254.186.1] LoadImage 'syslog' PASS (36/104, 23 sec)
| | |-[169.254.186.1] LoadImage 'object' PASS (37/104, 1 min 15
sec)
| | |-[169.254.186.1] LoadImage 'influxdb' PASS (38/104, 9 sec)
| | |-[169.254.186.1] LoadImage 'fluxd' PASS (39/104, 7 sec)
| | |-[169.254.186.1] LoadImage 'throttler' PASS (40/104, 7 sec)
| | |-[169.254.186.1] LoadImage 'telegraf' PASS (41/104, 7 sec)
| | |-[169.254.186.1] LoadImage 'grafana' PASS (42/104, 12 sec)
| | |-[169.254.186.1] PushImage 'zookeeper' PASS (43/104)
| | |-[169.254.186.1] PushImage 'fabric' PASS (44/104)
| | |-[169.254.186.1] PushImage 'syslog' PASS (45/104)
| | |-[169.254.186.1] PushImage 'object' PASS (46/104)
| | |-[169.254.186.1] PushImage 'influxdb' PASS (47/104)
| | |-[169.254.186.1] PushImage 'fluxd' PASS (48/104)
| | |-[169.254.186.1] PushImage 'throttler' PASS (49/104)
| | |-[169.254.186.1] PushImage 'telegraf' PASS (50/104)
| | |-[169.254.186.1] PushImage 'grafana' PASS (51/104)

Dell Technologies Confidential Information version: 2.3.6.182

Page 70 of 76
 | +-Push All Images to Remote Registries
|-Push Images to Custom Registries
| |-Load Images
| +-Push Images
|-Upgrade Fabric On Specified Nodes
| |-Initiate Registry Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (52/104)
| |-Initiate ZooKeeper Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (53/104)
| |-Initiate LM Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (54/104)
| |-Initiate Syslog Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (55/104)
| |-Load Fabric images directly from installer node
| +-Load Registry image
|-Return Fabric Ownership to Lifecycle Manager
| |-[0.0.0.0] TakeOwnership PASS (56/104)
| |-[0.0.0.0] TakeOwnership PASS (57/104)
| |-[0.0.0.0] TakeOwnership PASS (58/104)
| +-[0.0.0.0] TakeOwnership PASS (59/104)
|-Update Fabric Ownership To Lifecycle Manager
| |-Update Ownership For Registry
| | +-[169.254.186.1] UpdateOwnership PASS (60/104)
| |-Update Ownership For ZooKeeper
| | |-[169.254.186.1] UpdateOwnership PASS (61/104)
| | |-[169.254.186.3] UpdateOwnership PASS (62/104, 1 sec)
| | +-[169.254.186.2] UpdateOwnership PASS (63/104, 1 sec)
| |-Update Ownership For LM
| | |-[169.254.186.2] UpdateOwnership PASS (64/104)
| | |-[169.254.186.1] UpdateOwnership PASS (65/104)
| | +-[169.254.186.3] UpdateOwnership PASS (66/104)
| +-Update Ownership For Syslog
| +-[169.254.186.4] UpdateOwnership PASS (67/104)
|-Post-check Fabric Services Health on Specified Nodes
| |-Validate Registry Service on Specified Nodes
| | +-[169.254.186.4] ServiceHealth PASS (68/104)
| |-Validate ZooKeeper Service on Specified Nodes
| | +-[169.254.186.4] ServiceHealth PASS (69/104)
| |-Validate LM Service on Specified Nodes
| | +-[169.254.186.4] ServiceHealth PASS (70/104)
| +-Validate Syslog Service on Specified Nodes
| +-[169.254.186.4] ServiceHealth PASS (71/104)
|-Upgrade Service On Specified Nodes
| |-Upgrade Metrics On Specified Nodes
| | |-Initiate Influxdb Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (72/104)
| | |-Initiate Fluxd Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (73/104)
| | |-Initiate Throttler Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (74/104)
| | |-Initiate Telegraf Upgrade if Required
| | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (75/104)
| | |-Initiate Grafana Upgrade if Required

Dell Technologies Confidential Information version: 2.3.6.182

Page 71 of 76
 | | | +-[0.0.0.0] UpdateApplicationOnNodes PASS (76/104)
| | |-Load Influxdb image
| | |-Load Fluxd image
| | |-Load Throttler image
| | |-Load Telegraf image
| | +-Load Grafana image
| +-Upgrade Object On Specified Nodes
| |-Initiate Object Upgrade if Required
| | +-[0.0.0.0] UpdateApplicationOnNodes PASS (77/104, 4 sec)
| +-Load Object image
|-Update Services Ownership To Lifecycle Manager
| |-Update Ownership For Object
| | +-[169.254.186.4] UpdateOwnership PASS (78/104)
| |-Update Ownership For Influxdb
| | +-[169.254.186.4] UpdateOwnership PASS (79/104)
| |-Update Ownership For Fluxd
| | +-[169.254.186.4] UpdateOwnership PASS (80/104)
| |-Update Ownership For Throttler
| | +-[169.254.186.4] UpdateOwnership PASS (81/104)
| |-Update Ownership For Telegraf
| | +-[169.254.186.4] UpdateOwnership PASS (82/104)
| +-Update Ownership For Grafana
| +-[169.254.186.4] UpdateOwnership PASS (83/104)
|-Post-check Services Health
| |-Post-check Object Health
| | +-Validate Object Service on Specified Nodes
| | +-[169.254.186.4] ServiceHealth PASS (84/104, 1 min 43
sec)
| +-Post-check Metrics Health
| |-Validate Influxdb Service on Specified Nodes
| | +-[169.254.186.4] ServiceHealth PASS (85/104, 1 sec)
| |-Validate Fluxd Service on Specified Nodes
| | +-[169.254.186.4] ServiceHealth PASS (86/104)
| |-Validate Throttler Service on Specified Nodes
| | +-[169.254.186.4] ServiceHealth PASS (87/104)
| |-Validate Telegraf Service on Specified Nodes
| | +-[169.254.186.4] ServiceHealth PASS (88/104)
| +-Validate Grafana Service on Specified Nodes
| +-[169.254.186.4] ServiceHealth PASS (89/104)
|-Enable CallHome
| +-[0.0.0.0] SetCallHomeEnabled PASS (90/104)
|-Cleanup Obsolete Dockers
| +-Cleanup Obsolete Docker Containers and Images
|-[169.254.186.4] Removed image emcvipr/object:3.8.0.1-138598.3d5db7c96f2
|-[169.254.186.4] Removed image fabric/syslog:3.8.0.0-4343.878ca95
|-[169.254.186.4] Removed image ecs-monitoring/fluxd:3.8.0.0-1548.8e548035
|-[169.254.186.4] Removed image ecs-monitoring/throttler:3.8.0.0-1548.8e548035
|-[169.254.186.4] Removed image ecs-monitoring/grafana:3.8.0.0-1548.8e548035
|-[169.254.186.4] Removed image ecs-monitoring/influxdb:3.8.0.0-1548.8e548035
|-[169.254.186.4] Removed image ecs-monitoring/telegraf:3.8.0.0-1548.8e548035
| +-[0.0.0.0] CleanupDockers PASS (91/104, 9 sec)
+-Init alert for low endurance level
+-[0.0.0.0] InitLowEnduranceLvlAlert PASS (92/104)

Dell Technologies Confidential Information version: 2.3.6.182

Page 72 of 76
 Elapsed time is 8 min 10 sec.

NODE_UPGRADE completed successfully

Collecting data from cluster

Information has been written to the
"/opt/emc/caspian/installer/discover/NODE_UPGRADE.2023-08-04T11:17:21/provisioning.txt"
Information has been written to the
"/opt/emc/caspian/installer/discover/NODE_UPGRADE.2023-08-04T11:17:21/topology.txt"
Executing /configure.sh --start action in object-main container which may take up to 1000
seconds.
20230804 11:18:22.489: | | | | PASS (9 min 23 sec)
20230804 11:18:22.513: | | | Update manifest from production in LM
20230804 11:18:25.696: | | | | PASS (3 sec)
20230804 11:18:25.697: | | | Check if all product components are upgraded
Object is upgraded on host 169.254.186.4
Fabric is upgraded on host 169.254.186.4
20230804 11:18:54.173: | | | | PASS (28 sec)
20230804 11:18:54.178: | | | Node Exit Maintenance Mode
20230804 11:18:57.993: | | | | PASS (3 sec)
20230804 11:18:58.001: | | | Perform Service Check
Executing /configure.sh --start action in object-main container which may take up to 1000
seconds.
Checking DT status (with timeout 5 min).
20230804 11:19:13.540: | | | | PASS (15 sec)
20230804 11:19:13.541: | | | Enable node data recovery after upgrade
Start node recovery permission polling for nodes 169.254.186.4 ...
Data recovery enabled for node(s) 169.254.186.4
20230804 11:19:14.171: | | | | PASS
20230804 11:19:14.174: | | | Disable Load Balancing for Node
Disabling Load Balancing, reason: Disabled by Service Console after Node Upgrade
LB is disabled
20230804 11:19:16.846: | | | | PASS (2 sec)
20230804 11:19:16.847: | | | Apply required post-upgrade workarounds
20230804 11:19:16.856: | | | | PASS
20230804 11:19:16.860: | | | Add firewall to Fabric agent capabilities if needed
20230804 11:19:17.186: | | | | PASS
20230804 11:19:17.187: | | | PASS (28 min 47 sec)
20230804 11:19:17.188: | | Check if the node should have installer
20230804 11:19:17.204: | | | PASS
20230804 11:19:17.205: | | Check Pause Requested Flag
20230804 11:19:17.206: | | | Get Pause Requested Flag
20230804 11:19:20.224: | | | | PASS (3 sec)
20230804 11:19:20.235: | | | PASS (3 sec)
20230804 11:19:20.236: | | PASS (28 min 50 sec)
20230804 11:19:20.236: | PASS (1 hour(s) 37 min 31 sec)
20230804 11:19:20.237: Get upgraded nodes list
20230804 11:19:20.238: | PASS
Local Upgrade Teardown
20230804 11:19:25.542: Ensure data recovery is enabled for all nodes after upgrade
Start node recovery permission polling for nodes 169.254.186.2, 169.254.186.1, 169.254.186.3,
169.254.186.4 ...
Data recovery enabled for node(s) 169.254.186.2, 169.254.186.1, 169.254.186.3, 169.254.186.4

Dell Technologies Confidential Information version: 2.3.6.182

Page 73 of 76
 20230804 11:19:26.013: | PASS
20230804 11:19:26.014: Check if the local site is upgraded
20230804 11:19:42.994: | PASS (16 sec)
20230804 11:19:42.996: Restore VNEST reconfiguration state
20230804 11:19:49.665: | Set VNEST diagnostic mode
20230804 11:20:17.515: | | PASS (27 sec)
20230804 11:20:17.516: | Set VNEST configuration parameter value
20230804 11:20:17.522: | | Write item to CF
20230804 11:20:19.693: | | | PASS (2 sec)
20230804 11:20:19.695: | | PASS (2 sec)
20230804 11:20:19.700: | PASS (36 sec)
20230804 11:20:19.703: Switch Object to Normal Mode
Skip: target object version 3.8.0.3-138685.3a0a9b6bf3a is higher than or equal to 3.2
20230804 11:20:22.722: | PASS (3 sec)
20230804 11:20:24.306: Enable Load Balancing
LB is enabled
20230804 11:20:26.878: | PASS (2 sec)
20230804 11:20:26.880: Upgrade static routes
20230804 11:20:27.050: | PASS
20230804 11:20:27.052: Disable MPD for VDC if required
20230804 11:20:34.095: | PASS (7 sec)
20230804 11:20:34.097: Check Storage pools threshold alert
20230804 11:20:44.464: | PASS (10 sec)
20230804 11:20:44.466: Check manifest from production in LM
20230804 11:20:47.651: | PASS (3 sec)
20230804 11:20:47.652: Trigger Listing conversion verification scanner if required
Skipped: not needed when upgrading from ECS version 3.8.0.1.138598.3d5db7c96f2
20230804 11:20:51.719: | PASS (4 sec)
20230804 11:20:51.721: Enable autologout
20230804 11:20:52.163: | PASS
20230804 11:20:52.165: Post-upgrade cold run stigmgr
Skip: source os version is ecs-os-base-3.8.0.0-2076.a7e36fa.36.noarch (not 3.3)
20230804 11:20:56.191: | PASS (4 sec)
20230804 11:20:56.192: Post-upgrade restart fabric agent if needed
20230804 11:21:04.174: | PASS (7 sec)
20230804 11:21:05.092: Remove nan patch postinstall.tgz file if available
20230804 11:21:05.915: | PASS
20230804 11:21:06.480: Post-upgrade GC enable
20230804 11:21:06.481: | PASS
20230804 11:21:06.482: Post-upgrade BTree GC Cleanup
Skip: target object version 3.8.0.3.138685.3a0a9b6bf3a is higher than or equal to 3.1.0.1.
Use option '--btree-gc-cleanup force' to force BTree GC cleanup.
20230804 11:21:06.482: | PASS
20230804 11:21:06.483: Enable CAS SR if required
20230804 11:21:06.483: | PASS
20230804 11:21:06.484: Enable Journal SR if required
20230804 11:21:13.612: | PASS (7 sec)
20230804 11:21:13.613: Check if Journal GC is enabled
20230804 11:21:15.745: | PASS (2 sec)
20230804 11:21:15.746: Does page table patch exist
20230804 11:21:15.748: | PASS
20230804 11:21:15.750: Enable page table
Page table is already enabled.

Dell Technologies Confidential Information version: 2.3.6.182

Page 74 of 76
 20230804 11:21:17.914: | PASS (2 sec)
20230804 11:21:17.916: Enable BTree SR if disabled by SC
Checking BTree SR.
BTree SR is already enabled.
20230804 11:21:20.033: | PASS (2 sec)
20230804 11:21:20.035: Handle Page Table and BTree SR if required
20230804 11:21:20.036: | PASS
20230804 11:21:23.057: Enable partial GC after upgrade
Skip - not needed for ECS version 3.8.0.1.138598.3d5db7c96f2
20230804 11:21:23.058: | PASS
20230804 11:21:23.058: Check if SR is enabled
20230804 11:21:25.445: | PASS (2 sec)
20230804 11:21:25.447: Check fabric certificate expiration
Certificate expiration date: Aug 4 05:25:38 2028 GMT
20230804 11:21:34.788: | PASS (9 sec)
20230804 11:21:34.791: Check com.emc.ecs.chunk.suicide_prevention.threshold.user
20230804 11:21:40.832: | PASS (6 sec)
20230804 11:21:40.835: Check that log4j patch is applied
Skip: check only when upgrade to 3.6
20230804 11:21:43.855: | PASS (3 sec)
20230804 11:21:43.856: Get SSH Host Keys Metadata
20230804 11:21:43.865: | PASS
20230804 11:21:43.867: Check SSH Host Keys
20230804 11:21:45.873: | PASS (2 sec)
20230804 11:21:45.875: Enable Callhome
20230804 11:21:45.876: | Set CallHomeEnabled
CallHome is enabled
20230804 11:21:48.885: | | PASS (3 sec)
20230804 11:21:48.886: | PASS (3 sec)
20230804 11:21:48.887: Enable SSDr read cache
20230804 11:21:51.055: | PASS (2 sec)
20230804 11:21:51.056: VDC key upgrade post action
20230804 11:21:52.955: | PASS (1 sec)
20230804 11:21:52.957: Set Upgrade Started Flag
20230804 11:21:55.977: | PASS (3 sec)
20230804 11:21:55.980: Add SECURITY_ADMIN role if needed
Skip - needed only when upgrading from pre-3.6 to 3.6 or higher.
20230804 11:22:03.020: | PASS (7 sec)
20230804 11:22:03.023: Update memory limits for monitoring containers
20230804 11:22:07.611: | PASS (4 sec)
20230804 11:22:07.614: Verify PBR ports for replication
20230804 11:22:12.223: | PASS (4 sec)
Global Upgrade Teardown
20230804 11:22:17.469: Check if all sites are upgraded
Object upgrade is complete on all sites. Completing upgrade...
20230804 11:22:28.039: | PASS (10 sec)
20230804 11:22:28.042: Set Upgrade Complete with check
Completing upgrade on VDC VDC1...
[VDC VDC1] com.emc.ecs.timeFormat.rfc822_date_time_format is already 'true'
[VDC VDC1] com.emc.ecs.objheadsvc.enable_rlisting_support is already 'true'
[VDC VDC1] com.emc.ecs.objheadsvc.allow_s3_multipart_download is already 'true'
[VDC VDC1] com.emc.ecs.upgrade.2_1_upgrade_complete is already 'true'
[VDC VDC1] com.emc.ecs.upgrade.2_2_upgrade_complete is already 'true'

Dell Technologies Confidential Information version: 2.3.6.182

Page 75 of 76
 [VDC VDC1] com.emc.ecs.upgrade.2_2_1_upgrade_complete is already 'true'
[VDC VDC1] com.emc.ecs.upgrade.3_0_upgrade_complete is already 'true'
[VDC VDC1] com.emc.ecs.upgrade.3_1_upgrade_complete is already 'true'
[VDC VDC1] com.emc.ecs.upgrade.3_2_upgrade_complete is already 'true'
[VDC VDC1] com.emc.ecs.upgrade.3_2_1_upgrade_complete is already 'true'
[VDC VDC1] com.emc.ecs.upgrade.3_2_2_upgrade_complete is already 'true'
[VDC VDC1] com.emc.ecs.upgrade.3_3_upgrade_complete is already 'true'
[VDC VDC1] com.emc.ecs.upgrade.3_4_upgrade_complete is already 'true'
[VDC VDC1] com.emc.ecs.upgrade.3_4_0_1_upgrade_complete is already 'true'
[VDC VDC1] com.emc.ecs.upgrade.3_5_upgrade_complete is already 'true'
[VDC VDC1] com.emc.ecs.upgrade.3_6_upgrade_complete is already 'true'
[VDC VDC1] com.emc.ecs.upgrade.3_7_upgrade_complete is already 'true'
[VDC VDC1] com.emc.ecs.upgrade.3_8_upgrade_complete is already 'true'
[VDC VDC1] Set com.emc.ecs.upgrade.3_8_0_3_upgrade_complete = true
[VDC VDC1] Got CF state: UPDATING (expected SYNCHRONIZED), retrying in 30 sec...
[VDC VDC1] Got CF state: UPDATING (expected SYNCHRONIZED), retrying in 30 sec...
[VDC VDC1] Got expected CF state: SYNCHRONIZED
20230804 11:23:45.503: | PASS (1 min 17 sec)
20230804 11:23:45.505: Enable partial GC
Checking Partial GC.
Partial GC is already enabled.
Checked all VDCs
20230804 11:24:04.630: | PASS (19 sec)
20230804 11:24:04.632: Disable DT query port
20230804 11:24:06.062: | PASS (1 sec)
================================================================================
Status: PASS
Time Elapsed: 1 hour(s) 43 min 42 sec
Debug log: /opt/emc/caspian/service-console/log/20230804_094041_run_Upgrade/dbg_robot.log
HTML log: /opt/emc/caspian/service-console/log/20230804_094041_run_Upgrade/log.html
================================================================================
Messages:
Object version: 3.8.0.3.138685.3a0a9b6bf3a
ECSOS version: ecs-os-base-3.8.0.3-2206.44cc40a.62.noarch
================================================================================
/opt/emc/service_console/ECSAutomation/sc/lib/service_console_keywords.py:6282:
YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader
is unsafe. Please read https://msg.pyyaml.org/load for full details.
limits = yaml.load(f)

Document feedback
To provide any feedback or suggestions on the document, go to Content Feedback Router portal. For
more information, see Content Feedback Router - Support.

Dell Technologies Confidential Information version: 2.3.6.182

Page 76 of 76