ACI Multi-Pod Upgrade MOP- Adecco

Cisco Systems

Date 10.03.2020

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Highly Confidential
Agenda
• Prerequisite
• Release Overview
• MOP Upgrade ACI Fabric
o Maintenance Groups Overview
o Pre-Operation Check
o Snapshot Information Collection
o Fault Snapshot Information Collection
o Configuration Backup
o APIC Upgrade
o APIC Upgrade Verification
o Firmware Group
o Maintenance Group Creation
o ACI switches upgrade
o ACI switches upgrade Verification
• MOP Rollback. Downgrade ACI Fabric
• MOP Upgrade CIMC
• MOP Rollback. Downgrade CIMC
• Bringing Up Standby APIC
Adecco ACI Upgrade Prerequisite before MW
 Configuration Backup (confirm)
• APIC export policy for configuration backup to external server with AES encryption.

 CIMC
• Restore access to CIMC. Configure Network Settings and credentials and interconnect with OOB.
• Verify the CIMC firmware version.
• It may be required to upgrade CIMC (upon Cisco confirmation)

 Console Access to Spine and Leaf switches

 Confirm that the /firmware partition is not filled beyond 75%. Check APIC disk utilization (System ->
Controllers -> APIC<id> -> Storage).

 Analyze and Resolve ACI Fabric critical/major faults. Ensure APIC Cluster Healthy.
Output of faults on APIC:
moquery -c faultInst
 Current Release and Hardware Overview
APIC ACI switch CIMC
3.1(2o) 13.1(2o) 3.0(4d)

Pod in AMS :
o APIC: 2 x active APIC-SERVER-L2
o Spine: 2 x 9364C
o Border Leaf : 2 x Cisco Nexus 93180YC-EX
o Leaf : 10 x Cisco Nexus 93180YC-EX, 4 x Cisco Nexus 93108TC-EX
Pod in DUS :
o APIC: 1 x active APIC-SERVER-L2 and 1 standby APIC-SERVER-L2
o Spine: 2 x 9364C
o Border Leaf : 2 x Cisco Nexus 93180YC-EX
o Leaf : 8 x Cisco Nexus 93180YC-EX, 4 x Cisco Nexus 93108TC-EX
 Recommended Release
APIC ACI switch CIMC
3.2(9h) 13.2(9h) 4.0(2g)

For APIC image download go to:

[Link]

For ACI switches image download go to:

[Link]
ACI Fabric Upgrade
 ACI Multi-Pod Maintenance Groups
Maintenance Group defines the target firmware version, when to upgrade firmware, which nodes to
upgrade and how to handle failures. ACI nodes part of the same maintenance group upgraded jointly in parallel.

Pod1 (18 nodes) Pod2 (16 nodes)

Pod1-Spine-Even Pod2-Spine-Even
Pod1-Spine-Odd Pod2-Spine-Odd
Pod1-Leaf-Even Pod2-Leaf-Even
Pod1-Leaf-Odd Pod2-Leaf-Odd
From ACI firmware 3.2(4d), there are changes made to SSD overprovisioning on spine switches (N9K-C9364C) to
preserve lifespan of SSDs. After the upgrade, SSD over-provisioning might take up to an hour per spine switch to
complete.
Maintenance Group Table
Group Name Node ID Concurrent Nodes
Upgraded
Pod1-Spine-Even 102 1

Pod1-Spine-Odd 101 1

Pod1-Leaf-Even 112,122,124,126,128,130,152,154 8

Pod1-Leaf-Odd 111,121,123,125,127,129,151,153 8

Pod2-Spine-Even 202 1

Pod2-Spine-Odd 201 1

Pod2-Leaf-Even 212,222,224,226,228,252,254 7

Pod2-Leaf-Odd 211,221,223,225,227,251,253 7
Pre-Operational Check
 Make sure that critical services have resilient interconnection over ACI Fabric:

 Verify all critical servers are dual homed to ACI fabric.

 Verify Firewall cluster portchannels are up, no link member is down.
 Verify L3OUT connectivity have redundancy over pair of Border Leafs

 Check all the current faults in the system and evaluate them for potential impact on upgrade.

 Verify that all APICs are in “fully fit” state.

Snapshot Information Collection
Take configuration snapshot local to APIC:

From Admin>Config Rollbacks Create the Configuration Snapshot. You will need to put the distinctive Description that indicates that the configuration
before the upgrade.
 Collect the Log from CLI
• ###APIC####
• terminal length 0

• acidiag rvread
• acidiag fnvread
• acidiag avread
• acidiag verifyapic
• bash -c "ip link"

• show clock
• show version
• show endpoints
• show port-channel map
• show vpc map
• show vlan-domain detail
• show tenant
• show vrf
• show bridge-domain
• show epg
• show switch
• show controller
• show controller detail
• show vmware domain
• moquery -c fabricNode -x query-target=self | egrep model\|serial\|'name '

• ###SPINES###
• fabric node_id show lldp neighbors
• fabric node_id show interface status | grep connected
• fabric node_id show bgp vpnv4 unicast summary vrf all
• fabric node_id show bgp vpnv4 unicast vrf all
• fabric node_id show isis adjacency vrf all
• fabric node_id show coop internal info ip-db
Fault Snapshot information Collection
Take screenshot of faults from GUI and CLI
System>Faults

###APIC###

show faults detail

show faults severity major detail
Export the configuration to the file server

• Admin>Export Policies>Configuration
• Enable AES Encryption, to export the security properties as passwords and
certificated. While enabling it, a passphrase is required and customer
needs to define and to keep it safe.
Upload Firmware to APIC
Upload from Laptop: Select Admin>Firmware>Download Task>Upload Firmware to APIC
Alternatively, if there is no direct access, it can be uploaded from the HTTP or SCP server
Select Admin>Firmware>Download Task>Create Firmware Download Task
In the firmware repository, ACI > Admin > Firmware > Firmware Repository, there should be 3.2(9b),
13.2(9b) and 3.1(2o), 13.1(2o) images.
 Open Firmware Repository: Click Admin > Firmware > Firmware Repository (All images will be located
here).
Double click on each image to see Firmware Details
Confirm that MD5 checksum is correct

Standard connectivity check and application accessibility tests should be performed to confirm that
services/applications are available before ACI fabric upgrade
 Upgrade APIC Controllers
~15 minutes per APIC, totally 45 minutes

Select target firmware version from Admin>Firmware>Controller Firmware>Controller Upgrade

When upgrade policy for APIC is launched, APICs are upgraded one by one without any intervention. Upon all active APICs upgrade,
standby upgrades automatically. Acidiag avread provides appliance vector information for all apic including standby.
After the upgrade, you can see the current firmware equal to the target. Status should be “Upgraded
successfully” and the process to 100%.
APIC Controllers Upgrade Verification
1. Verify that all nodes (Leafs and Spines) are visible in the topology. Go to Fabric – Inventory –
Topology and select Topology in the Work Pane
2. Click on Controllers in the left Pane and switch to table view
3. Verify the operational status and Health status in the right pane.
4. All the 3 APIC controllers should be Available and Fully fit.
5. Verify that it’s possible to SSH to all APICs with local and TACACs credentials:
Username: apic#LOGIN_DOMAIN\\LOCAL_USERNAME
6. Verify that “acidiag rvread” doesn’t report any unhealthy shard on all 3 APICs
7. Verify if any new faults generated (There will be one new fault “F3057 APIC Controller
product is not registered with CSSM and the product is in 90 days evaluation period.” But it’s
expected as Smart Licensing feature was introduced).
8. If any new fault is generated, it should be assessed and decision about further upgrade
should be made based on the potential impact of the Fault.
Define Firmware Group
Create the new Firmware Group and name it Target_Release:

From Admin>Firmware>Fabric Node Firmware>Firmware Groups select Create Firmware Group, where specify 13.2(9b) as
the target release.
Define Maintenance Group
Create Maintenance Groups per agreed approach distributing the devices across the different upgrade groups.

From Admin>Firmware>Maintenance Group create group name with node id that are part of this group
Upgrade ACI switches
~20 minutes to upgrade one maintenance group, the switches in the same maintenance group upgraded in parallel.

From ACI > Admin > Firmware > Fabric Node Firmware > Maintenance Group, select Maintenance Group > Upgrade Now

After the upgrade, you can see the current firmware equal to the target. Status should be
“Upgraded successfully” and the process to 100%.
 If N9K-C9364C or N9K-C9336C-FX2 switches present in the fabric the fault may
be generated with the requirements to perform reload and SSD reformat/repartition. It’s
expected and it’s consequence of SSD over-provisioning feature introduced in 3.2(4)
releases. The reload and SSG reformat/repartition may take up to one hour. As it’s not
suggested to put the switch in the maintenance mode due-to CSCvk03229, the reload of the
switch will lead to traffic drops
In case if the fault F2972, the reload of the switch should be performed with SSD
reformat checkbox set
ACI switches Upgrade Verification
1. Verify that all nodes (Leafs and Spines) are visible in the topology. Go to Fabric – Inventory –
Topology and select Topology in the Work Pane
2. Verify that it’s possible to SSH to upgraded switches with local and TACACs credentials:
Username: apic#domain\\username

3. Verify if any new fault generated

4. If any new fault is generated, it should be assessed and decision about further upgrade
should be made based on the potential impact of the Fault.

5. Standard connectivity check and application accessibility tests should be performed to

confirm that services/applications are available after ACI fabric upgrade
ACI Fabric Upgrade Rollback
Downgrade APIC Controllers
~15 minutes per APIC, totally 45 minutes

Select target firmware version from Admin>Firmware>Controller Firmware>Controller Upgrade

After the downgrade, you can see the current firmware equal to the target. Status should be
“Upgraded successfully” and the process to 100%.
APIC Controllers Downgrade Verification
1. Verify that all nodes (Leafs and Spines) are visible in the topology. Go to Fabric – Inventory –
Topology and select Topology in the Work Pane
2. Click on Controllers in the left Pane and switch to table view
3. Verify the operational status and Health status in the right pane.
4. All the 3 APIC controllers should be Available and Fully fit.
5. Verify that it’s possible to SSH to all APICs with local and TACACs credentials:
Username: apic#LOGIN_DOMAIN\\LOCAL_USERNAME
6. Verify that “acidiag rvread” doesn’t report any unhealthy shard on all 3 APICs
7. Verify if any new faults generated
8. If any new fault is generated, it should be assessed and decision about further upgrade
should be made based on the potential impact of the Fault.
Change Firmware Group
Change Firmware Group Target_Release:

From Admin>Firmware>Fabric Node Firmware>Firmware Groups>Target_Release, where specify 13.1(2o) as

the target release.
Downgrade ACI switches
~20 minutes to upgrade one maintenance group, the switches in the same maintenance group upgraded in parallel.

From ACI > Admin > Firmware > Fabric Node Firmware > Maintenance Group, select Maintenance Group > Upgrade Now

After the upgrade, you can see the current firmware equal to the target. Status should be
“Upgraded successfully” and the process to 100%.
ACI switches Downgrade Verification
1. Verify that all nodes (Leafs and Spines) are visible in the topology. Go to Fabric – Inventory –
Topology and select Topology in the Work Pane
2. Verify that it’s possible to SSH to upgraded switches with local and TACACs credentials:
Username: apic#domain\\username

3. Verify if any new fault generated

4. If any new fault is generated, it should be assessed and decision about further upgrade
should be made based on the potential impact of the Fault.

5. Standard connectivity check and application accessibility tests should be performed to

confirm that services/applications are available after ACI fabric upgrade
CIMC Upgrade
Upgrade Prerequisites - CIMC
• Obtain Software images from [Link] for versions 4.0(2g) and
3.0(4d)
• Confirm that the MD5 Checksum of the image matches the one
published on [Link]
• Time needed for the process of upgrading a CIMC version varies
based on the speed of the link between the Local Machine and the
UCS-C chassis, and source/target software image and other
internal component versions. From experience it can take between
1h and 1h30 per server.
• Upgrading the CIMC version does not affect the production network
as APICs are not in the Data Path of the traffic.
Upgrade Prerequisites - CIMC
• Please understand that changing CIMC version might also require
changes to the Internet Browser, and Java Software version to run
the vKVM. In the case of ADECCO, HTML 5 based vKVM is
possible.
• Obtaining HUU Firmware
• From Cisco Website Download Software page, Navigate to
Downloads, and follow:
• Select Product > Servers - Unified Computing > Cisco UCS C-
Series Rack-Mount Standalone Server Software > Choose UCS
C220 M4 Rack Server Software based on the generation of the
APIC.
[Link]
Upgrade Prerequisites - CIMC

APIC requires TPM enabled and active, otherwise, APIC will fail to decrypt root
filesystems
Launch vKVM
Upgrade Procedure - CIMC

Activate Virtual Devices from: Virtual Media > Activate Virtual Devices
Upgrade Procedure - CIMC

Map the downloaded ISO image for CIMC software 4.0(2g) as an CD/DVD: Virtual
Media > Map CD/DVD
Upgrade Procedure - CIMC

Choose the ISO image from the local machine:

Upgrade Procedure - CIMC

Reboot: Power > Reset System (warm boot)

Upgrade Procedure - CIMC

Get into Boot menu, from the screen below, press F6

Upgrade Procedure - CIMC

Insert Password - Default Password in case the password has not been changed
is "password“
Upgrade Procedure - CIMC

Choose vKVM mapped vDVD

Upgrade Procedure - CIMC

Cisco UCS Host Upgrade Utility starts:

Upgrade Procedure - CIMC
Upgrade Procedure - CIMC

Agree on License Agreement

Upgrade Procedure - CIMC

Choose to upgrade all components by pressing on Update All Current Version will be
upgrading to Update Version, note that, during the upgrade the APIC will show
disconnections.
Upgrade Procedure - CIMC
Upgrade Procedure - CIMC

Do not enable Cisco IMC secure boot

Upgrade Procedure - CIMC post checks

HUU will start upgrading each component individually, once it is done it will reboot the
server.
We recommend to boot a second time on the HUU ISO to launch the upgrade
verification utility (see next slide)
Upgrade Procedure - CIMC Verification utility
• Reboot again the server on the USB key, but choose the option “LAST UPDATE VERIFY” on the HUU menu.
• This option will you tell if everything went fine. Then, click on “EXIT”, the controller will then boot on the ACI image.
ACI Verification after CIMC Upgrade
1. Verify that all nodes (Leafs and Spines) are visible in the topology. Go to Fabric – Inventory –
Topology and select Topology in the Work Pane
2. Click on Controllers in the left Pane and switch to table view
3. Verify the operational status and Health status in the right pane.
4. All the 3 APIC controllers should be Available and Fully fit.
5. Verify that it’s possible to SSH to all APICs with local and TACACs credentials:
Username: apic#LOGIN_DOMAIN\\LOCAL_USERNAME
6. Verify that “acidiag rvread” doesn’t report any unhealthy shard on all 3 APICs
CIMC Upgrade Rollback
Launch vKVM
Downgrade Procedure - CIMC

Activate Virtual Devices from: Virtual Media > Activate Virtual Devices
Downgarde Procedure - CIMC

Map the downloaded ISO image for software version 3.0(4d) as an CD/DVD: Virtual
Media > Map CD/DVD

Procceed with further the steps described in CIMC Upgarde Procedure

ACI Verification after CIMC Downgrade
1. Verify that all nodes (Leafs and Spines) are visible in the topology. Go to Fabric – Inventory –
Topology and select Topology in the Work Pane
2. Click on Controllers in the left Pane and switch to table view
3. Verify the operational status and Health status in the right pane.
4. All the 3 APIC controllers should be Available and Fully fit.
5. Verify that it’s possible to SSH to all APICs with local and TACACs credentials:
Username: apic#LOGIN_DOMAIN\\LOCAL_USERNAME
6. Verify that “acidiag rvread” doesn’t report any unhealthy shard on all 3 APICs
Bringing Up Standby APIC
Use this procedure to switch over an active APIC with a standby APIC (please ensure that the
GUI session is opened with the APIC which is not expected to be replaced)

1) On the menu bar, choose System > Controllers.

2) In the Navigation pane, expand Controllers > APIC NOT to be replaced > Cluster as Seen
by Node.
3) In the Work pane, verify that the Health State in the Active Controllers summary table
indicates the active controller is Fully Fit before continuing.
4) Right click an “APIC to be replaced” that you want to switch over and select Replace. The
Replace dialog box displays.
5) Choose the Backup Controller from the drop-down list and click Submit. Wait until the new
active APIC is in Fully Fit state.
6) Add Testing configuration
7) Disconnect the replaced APIC