Quickstart guide

HPE M-Series switches with

NVIDIA Cumulus v5.9.x, 5.10.0,
and 5.11.0 NOS
April 2025
Quickstart guide

Contents
Abstract .......................................................................................................................................................................................................................................................................................................................................... 3
Introduction ................................................................................................................................................................................................................................................................................................................................. 3
Configuration backup and restore.......................................................................................................................................................................................................................................................................... 3
Uninstalling the switch NOS......................................................................................................................................................................................................................................................................................... 4
Installing the switch NOS ................................................................................................................................................................................................................................................................................................ 5
NOS update methods......................................................................................................................................................................................................................................................................................................... 8
Switch NOS upgrade .................................................................................................................................................................................................................................................................................................... 8
NVIDIA repository upgrade.................................................................................................................................................................................................................................................................................... 8
NOS update using MLAG ............................................................................................................................................................................................................................................................................................... 9
Configuring the switch ................................................................................................................................................................................................................................................................................................... 10
In-band management configuration ...........................................................................................................................................................................................................................................................11
MLAG ......................................................................................................................................................................................................................................................................................................................................11
MLAG setup ...................................................................................................................................................................................................................................................................................................................... 11
Layer 2 port setup .............................................................................................................................................................................................................................................................................................................13
Quality of Service (QoS)................................................................................................................................................................................................................................................................................................15
Switch array port configuration .......................................................................................................................................................................................................................................................................16
Configuration example ................................................................................................................................................................................................................................................................................................... 18
Example 1: Single spine and dual leaf .......................................................................................................................................................................................................................................................18
Example 2: Dual spine and dual leaf ...........................................................................................................................................................................................................................................................24
Conclusion .................................................................................................................................................................................................................................................................................................................................33
Quickstart guide Page 3

Abstract
This document provides a step-by-step procedure that can be used to configure HPE M-Series switches running NVIDIA® Cumulus v5.9.x,
v5.10.0, or v5.11.0 NOS with initiators and storage from Hewlett Packard Enterprise. Note that this document is specific to the NVIDIA
Cumulus v5.9.x., v5.10.0, and v5.11.0 NOS and will contain certain NVUE commands that are not available in the NVIDIA Cumulus v5.6.0 NOS.

Introduction
The HPE Quickstart guide is a document that provides a step-by-step procedure to configure NVIDIA Cumulus Linux® on all supported SN
Ethernet switches. It is recommended that the network operating system (NOS) of all new switches be updated to the newest HPE supported
NOS listed on the Cumulus SPOCK document before any data cables are inserted or data ports configured.

Out of the box, the default switch username and password are both cumulus (lowercase). After the first-time login, the switch will prompt for
the current password to be reentered and a new password to then be specified. Once logged in, the current switch system version can be
verified with the command nv show system |grep build:

Switches that have NVIDIA Cumulus v5.6.0 NOS cannot be upgraded to the newest supported version directly. Instead, the customer must
back up the existing switch configuration, perform a new binary install of the NVIDIA Cumulus v5.9.x, v5.10.0, or v5.11.0 NOS, and then
restore the switch configuration. If this switch is part of a production setup then it is recommended for all work to be performed during a
maintenance window with the I/O load at a minimum or stopped completely. Before starting, a switch cl-support file must be generated and
copied to a safe location along with /etc/nvue.d/startup.yaml. This is a very important step that must not be ignored.

Configuration backup and restore

Saving the switch configuration
HPE recommends using only Cumulus NVUE commands to configure all Cumulus switches. This includes using only NVUE commands to
change the passwords of root (if enabled), cumulus, and all individual user accounts. Any Linux commands that are used will not be saved in
the /etc/nvue.d/startup.yaml file and, therefore, will not be restored.

Customers can use the following procedure to save the startup.yaml file:
1. Issue the commands nv config apply and then nv config save. These two steps may be combined into one command by
issuing nv set system config auto-save state enabled and then nv config apply. Applying a configuration in the
future will now automatically save it to /etc/nvue.d/startup.yaml.
2. Copy the configuration file to the /home/cumulus directory on the switch using the command sudo cp
/etc/nvue.d/startup.yaml /home/cumulus
3. Customers are required to modify the permissions of the copied configuration file using the command sudo chmod 777
/home/cumulus/startup.yaml. It is recommended to change the name of the startup file to reflect which switch it was copied from.
For example, mv startup.yaml startup-switch1.yaml
4. Now, we can SCP/SFTP to the switch as the “cumulus” user and copy the /home/cumulus/startup.yaml file to a safe location off
the switch. The file can now be used to restore a switch in the future.

Restoring the switch configuration

Customers can use the following procedure to restore the configuration to a new switch or an existing switch that has been rebuilt/factory reset:
1. If the switch is unconfigured then it is necessary to assign the IP address/subnet/gateway/DNS information such that an SFTP/SCP
session can be made. Please note that DHCP is enabled on eth0 by default therefore you can connect to the switch using the
dynamically obtained IP address.
2. SFTP or SCP the startup.yaml file that was saved off the switch previously back to the /home/cumulus directory
3. Issue the command nv config patch /home/cumulus/startup.yaml and then nv config apply
Note: If the following error message is seen after the configuration is applied then it simply means that DHCP has to be disabled on eth0
via the command ‘nv unset interface eth0 ip address dhcp’ and then ‘nv config apply:
Invalid config [rev_id: 2]
Config invalid at
interface.eth0.ip.address: {'dhcp': {},
Quickstart guide Page 4

'172.17.5.100/16': {}} has too many properties

4. Reboot the switch via nv action reboot system

5. Ensure that the switch is now accessible
It is recommended to periodically backup the startup.yaml file and generate support files on all switches. These files should be moved to a
safe location off of the switch.

Uninstalling the switch NOS

1. Enter the ONIE menu by restarting the switch and using the cursor keys to select the ONIE option

2. Select the option ONIE: Uninstall OS

You will see the following. Please note that the uninstallation process may take a while to complete so please be patient:
Quickstart guide Page 5

After the uninstallation process completes then the switch will reboot. You may now move to the section of this document entitled “Installing
the switch NOS”.

Installing the switch NOS

The NVIDIA Cumulus v5.9.x, v5.10.0, and v5.11.0 NOS can be installed from a USB memory stick, a web server, from within ONIE, or directly
from the Cumulus NOS.
Web server NOS installation
Installing NOS directly from NVIDIA Cumulus
This option is most useful if there is a need to reinstall NVIDIA Cumulus or upgrade the NOS version to a newer one. Ensure that the current
switch configuration is backed up and moved off the switch before reinstalling the NOS.
1. Copy the NVIDIA Cumulus binary file to a directory on the web server
2. Issue the command sudo onie-install
-a -i http://172.17.4.1/directory/cumulus-linux-5.11.0-mlx-amd64.bin && sudo reboot

Installing NOS from within ONIE

1. Copy the NVIDIA Cumulus binary file to a directory on the web server
2. Enter the ONIE menu by restarting the switch and using the cursor keys to select the ONIE option
Quickstart guide Page 6

3. Select the option ONIE: Install OS

4. The fast-scrolling stream of commands can be halted with the command onie-stop

5. Assign an IP address to the switch eth0 using the command ip addr add 172.17.5.101/24 dev eth0
6. Start the NVIDIA Cumulus NOS install using the command onie-nos-install
http://172.17.4.1/directory/cumulus-linux-5.11.0-mlx-amd64.bin
7. The OS will copy the NVIDIA Cumulus binary file to ONIE and proceed with the installation
Quickstart guide Page 7

USB installation
Note
Ensure that the USB drive contains the appropriate connector for the switch before attempting an installation through USB. The
HPE SN3700cM accepts a Micro USB Type B 2.0 connector, the HPE M-Series SN2100M accepts a Mini USB 2.0 Type B connector, and all
other switches accept a USB 2.0 Type A connector. Only the HPE M-Series SN3420M switch contains a USB 3.0 Type A interface.

1. Insert the USB stick into the switch

2. Enter the ONIE menu by restarting the switch and using the cursor keys to select the ONIE option

3. Select the option ONIE: Install OS

4. The fast-scrolling stream of commands can be halted with the command onie-stop

5. Issue the following commands at the ONIE prompt to begin the NOS install:
mkdir /mnt/tmpusb

mount /dev/sdb1 /mnt/tmpusb

sync

cd /mnt/tmpusb

onie-nos-install [full name of file.bin]

Quickstart guide Page 8

NOS update methods

Switch NOS upgrade
As mentioned previously, it is recommended that the NOS of all switches be upgraded before any data cables are attached. If the switches
have external internet access, then utilizing NVIDIA repository is the preferred upgrade method. If the switches are located in a dark site or
have other security concerns, then utilizing the ONIE USB upgrade method or a localized TFTP server is preferable. Although not required, it
is recommended for the most recent NOS release to be freshly installed on all new switches via USB or another method before being put into
production.

NVIDIA repository upgrade

Newly ordered NVIDIA switches may come with NVIDIA Cumulus v5.6.0 NOS installed and this version cannot be directly upgraded to
Cumulus v5.9.x (5.9.2 is recommended), v5.10.0, or v5.11.0 through the repository. A binary install of NVIDIA Cumulus v5.9.x NOS, NVIDIA
Cumulus v5.10.0, or NVIDIA Cumulus v5.11.0 NOS is required. Once switches are at a minimum of NVIDIA Cumulus v5.9.x NOS then the
NVIDIA repository can be used for an easy transition to future Cumulus NOS versions. It is important to verify which NOS versions are
supported by HPE before moving to the newest available one on the repository.
Note that the repository will not stop an upgrade from older NOS versions to the newest available; however, for the purposes of support,
HPE recommends only moving forward two versions in one upgrade iteration.
1. Issue the commands nv config apply and then nv config save. These two steps may be combined into one command by
issuing nv set system config auto-save state enabled and then nv config apply. Applying a configuration in the
future will now automatically save it to /etc/nvue.d/startup.yaml.
2. Generate a cl-support of the switch using the command nv action generate system tech-support. Move the cl-support file
from /var/support to a safe location off the switch along with /etc/nvue.d/startup.yaml. If multiple switches are being upgraded then
ensure that each startup.yaml file is appropriately labeled with a descriptive name.
3. Issue the command sudo nano /etc/apt/sources.list. The file will look like the following:

4. If the switch is already at NVIDIA Cumulus v5.9.x, modify the top two lines in preparation for upgrading to NVIDIA Cumulus v5.11.0 NOS
(file name must match exactly in terms of letter capitalization):
deb https://apt.cumulusnetworks.com/repo CumulusLinux-5.11.0 cumulus upstream netq

deb-src https://apt.cumulusnetworks.com/repo CumulusLinux-5.11.0 cumulus upstream netq

5. Issue the NVUE command nv action upgrade system packages to latest dry-run to download the newest metadata
from the NVIDIA repository and verify that no upgrade issues are encountered. Address all issues seen before moving forward with the
actual upgrade.
6. Issue the NVUE command nv action upgrade system packages to latest to upgrade all packages to the specified
distribution.
7. Once the upgrade is complete, issue the NVUE command nv action reboot system to reboot the switch.
Quickstart guide Page 9

NOS update using MLAG

1. Save the switch running configuration on all switches using the nv config save command. This will write all changes to
/etc/nvue.d/startup.yaml.
2. Run the command nv action generate system tech-support. or cl-support on all switches to create a support file in the
/var/support directory. Move these files and /etc/nvue.d/startup.yaml from each switch to a secure location away from the switch that is
accessible once the switches have a new NOS installed. Ensure that the support files and startup.yaml file for each switch are adequately
labeled so that they will be copied to the correct switch after the NOS upgrade for configuration restoration purposes.
3. The current secondary switch in the MLAG pair will be upgraded first and the current primary switch will be upgraded last. Run the
command nv show mlag to identify the respective switch roles through the local-role parameter, which will display as secondary.
4. Identify the secondary switch uplink connections and shut the individual ports down using the command nv set interface swp1-
2 link state down (replace swp1-2 with the correct range) and then nv config apply.
5. Shut down the secondary switch peer link using the command nv set interface peerlink link state down and then nv
config apply. Check the status of MLAG on the primary switch using the command nv show mlag to ensure that the local-role is
still primary and the peer-alive is now displaying false.
6. Reference the section in this guide entitled Installing the switch NOS for instructions on how to install a new binary image of the NVIDIA
Cumulus NOS.
7. Verify that the NOS is updated correctly once the switch powers backup using the command nv show system |grep build. The
version on the switch should be displayed as follows:

Contact HPE Support for assistance if this is not the case.

8. Restore the prior switch configuration to the secondary switch that’s now running NVIDIA Cumulus v5.11.0 NOS by copying the saved
switch startup.yaml file to /home/cumulus and running the command nv config patch /home/cumulus/startup.yaml. Apply
the configuration using the command nv config apply. Reboot the switch through the command nv action reboot system.
9. Verify that all configured ports on the switch, including those that belong to the uplink bond and the peer link, have their admin and
operational statuses as up using the command nv show interface. This may take a few minutes depending on the total number of
ports in use.
10. If Spanning Tree Protocol (STP) is used, then it’s important to verify that the convergence process is complete on both switches in the
MLAG pair. If certain links take a long time to transition to the forwarding state, then certain edge ports may be configured improperly.
11. The final step is to ensure that the MLAG itself is reestablished using the command nv show mlag and verify that the local-role is set
to secondary and the peer-alive is displaying true. More time is necessary if the peer-alive value is still set to false.
At this time, the secondary switch is at the newer NOS version, which means that it needs to be temporarily made the primary switch for the
purposes of upgrading. Ensure that the current primary switch has a valid cl-support file that has been moved to a safe location along with
the /etc/nvue.d/startup.yaml file.
12. Issue the command nv set mlag priority 2048 on the secondary switch with the newer Cumulus NOS to temporarily force it into
the primary role within the MLAG pair. Confirm that the switch running the older NOS has now taken on the secondary role. Both roles
can be verified with nv show mlag.
13. Identify the switch uplink connections on the now secondary switch (running the older NVIDIA Cumulus NOS) and shut them down
using the command nv set interface swp1-2 link state down (substitute the proper port range) and then nv config
apply.
14. Shut down the secondary switch peer link using the command nv set interface peerlink link state down and then nv
config apply. Check the status of MLAG on the primary switch using the command nv show mlag to ensure that the local-role is
still primary and the peer-alive is now displaying false. Do not proceed further until this is the case.
15. Update the switch NOS using the same method as in Step 6. If the installation method is through USB, then ensure that the USB drive
has been inserted into the switch USB port.
16. Verify that the NOS has updated correctly to NVIDIA Cumulus v5.11.0 NOS using the command nv show system |grep build.
17. Restore the prior switch configuration to the secondary switch that’s now running NVIDIA Cumulus v5.11.0 NOS by copying the saved
switch startup.yaml file to /home/cumulus and running the command nv config patch /home/cumulus/startup.yaml.
Apply the configuration using the command nv config apply and then reboot the switch via the command ‘nv action
reboot system’.
Quickstart guide Page 10

18. Verify that all configured ports on the switch, including those that belong to the uplink bond and the peer link, have their admin and
operational statuses as up using the command nv show interface. This may again take a few minutes depending on the total
number of ports in use.
19. Once STP completes the convergence process (verified through mstpctl showall) then ensure that MLAG itself is reestablished
using the command nv show mlag. The local-role of the switch should be set to secondary and the peer-alive should display true.
This may take a few minutes.
20. The upgrade process is now complete and the individual switch roles can be reverted to their defaults. Set the primary switch
MLAG priority back to its original value with the command nv set mlag priority 32768. Both switches should have this
MLAG value set.
21. Issue the command nv show mlag consistency-checker global to make sure that no conflicts are listed.

Configuring the switch

All switches come with a DB9 serial cable for initial configuration purposes and the baud rate used is 115200. Eth0, the out-of-band (OOB)
management port, is set to DHCPv4 by default; however, it is recommended for all switches to utilize static addresses for the IP, subnet, DNS,
and gateway. Additionally, all switches should have unique hostnames assigned. The following example shows the commands to set the
management information:
nv unset interface eth0 ip address dhcp
nv config apply
nv set interface eth0 ip address 172.17.12.20/16
nv set interface eth0 ip gateway 172.17.0.2
nv set service dns mgmt server 172.17.0.5
nv set system hostname spine1
nv config apply

The appropriate time zone and NTP server should be set. Note that NTP utilizes the default VRF and may need to be changed to use the
mgmt VRF:
nv unset service ntp default
nv set service ntp mgmt
nv config apply

The NTP server can now be specified:

nv set service ntp mgmt server 172.17.0.1 iburst on
nv set system timezone US/Eastern
nv config apply

Confirm that the clock synchronization is taking place through the Linux date command at the command prompt.
Quickstart guide Page 11

In-band management configuration

In specific instances, customers may wish to utilize the data ports of the switch for management functionality instead of the switch’s mgmt0
port. If the switch is an SN2010M or SN3420M then only the R9Q45A RJ45 transceiver is required for usage in an SFP28 port, else the
845970-B21 adapter is additionally required for operation in a QSFP28 port. A new static IP must also be specified along with the subnet
and gateway. The configuration process is accomplished by issuing the following commands through an SSH session:
nv set interface swp4 ip address 172.17.5.136/16
nv set interface swp4 ip gateway 172.17.0.2
nv set interface swp4 link mtu 1500
nv set interface swp4 link speed auto
nv set interface swp4 link state up
nv set interface swp4 type swp
nv config apply

Once the intended switch port is configured for in-band management access, a standard Ethernet cable with an RJ45 connector can be
safely connected to a management switch. If the configured IP address is pingable then a regular SSH session can be established as normal.

MLAG
MLAG allows a maximum of two Ethernet switches of the same model (for example SN3700cM with SN3700cM) to operate as a unified
switch to improve redundancy and boost bandwidth capacity. The two peer switches in the MLAG pair, Switch-1 and Switch-2, must be
directly connected to each other and will appear as a single device to an uplink switch or an initiating device. MLAG cannot be used with
switches that have more than one VLAN-aware bridge configured.

A pair of NVIDIA Cumulus leaf switches in an MLAG pair can be uplinked to a single spine switch or to another pair of NVIDIA Cumulus spine
switches that are also configured in an MLAG pair, connected through an LACP bond to the uplink switches. These uplink switches can be
NVIDIA Onyx switches configured in an MLAG pair or any other vendor switches that are presented as a virtualized pair.

Normally in an MLAG setup, the switch with the lower MAC address assumes the primary role while the remaining switches become the
secondary. If needed, the switch roles can be changed by setting a priority to the switches in the range of 0 and 65535 where the switch
with the lower priority becomes the primary switch:
nv set mlag priority 2084
nv config apply

Unless there is a specific reason to do so, HPE does not recommend that customers modify the switch MLAG priorities and instead leave
them at the default value of 32768.

MLAG setup
The most common MLAG interchassis links consist of two short 0.5m direct attach copper (DAC) cables that are connected between the
final two QSFP ports on the switches. However, the connections can also be active optical cables (AOC) or optical transceivers with MPO
cables. MLAG requires high-bandwidth links for the interchassis links, therefore a minimum of two 40GbE QSFP+ redundant connections
shall be used. An aggregated set of SFP+ or SFP28 connections shall not be used.

The uplink switch cable connections can be DAC, AOC, or optical transceivers connected with MPO cables; however, shorter cables are
recommended if the racked switches are close in proximity to each other. It is recommended for all switch-to-switch connections to also be
configured with redundancy in mind. The following example includes the NVUE commands to create a redundant LACP bond on two leaf
switches to an uplink spine switch for connectivity purposes. Note the requirement for a unique MLAG MAC address between
44:38:39:ff:00:00 and 44:38:39:ff:ff:ff and MLAG ID ranging from 1 to 65535 that shall be assigned to the bonds configured on both
switches within a common network. But you can have multiple configurations in a network for switch-to-switch MLAG. The allowed range of
MAC addresses that can be utilized for MLAG is between 44:38:39:ff:00:00 and 44:38:39:ff:ff:ff. Using a MAC address outside this range may
lead to conflicts with other device interfaces within the network. The bonds can be disabled by setting the MLAG ID to 0.
Quickstart guide Page 12

Switch-1
nv set interface bond11 bond member swp1-2
nv set interface bond11 description Leaf-to-Spine-Connection
nv config apply

nv set interface bond11 bond mlag id 1

nv config apply

It is important that the bond is included in the bridge configuration to ensure that network traffic can be rerouted through the peer link if the
bond fails.
nv set interface bond11 bridge domain br_default
nv config apply

The interchassis switch ports that comprise the peer link bond are now added along with a unique MAC address for the MLAG pair and a
backup IP address that is the management eth0 IP of the other switch in the MLAG pair. Note that the backup IP must be in the mgmt VRF
and not the default VRF:
nv set interface peerlink bond member swp59-60
nv set mlag mac-address 44:38:39:FF:00:AA
nv set mlag backup 172.17.1.2 vrf mgmt
nv set mlag peer-ip linklocal
nv config apply

Switch-2
The port cabling for Switch-2 should mirror what was used for Switch-1. If switch ports swp59 and swp60 are used for the interchassis
switch connections on Switch-1 then the same numerical ports should be used on Switch-2:
nv set interface bond11 bond member swp1-2
nv set interface bond11 description Leaf-to-Spine-Connection
nv config apply

nv set interface bond11 bond mlag id 1

nv config apply

Make sure that the bond is included in the bridge configuration to ensure that network traffic can be rerouted through the peer link if the
bond fails.
nv set interface bond11 bridge domain br_default
nv config apply

The interchassis switch ports that comprise the peer link bond are now added along with the same unique MAC address that was used for
Switch-1, and a backup IP address that is Switch-1’s eth0 IP. Ensure that the backup IP is in the mgmt VRF and not the default VRF:
nv set interface peerlink bond member swp59-60
nv set mlag mac-address 44:38:39: FF4:00:AA
nv set mlag backup 172.17.1.1 vrf mgmt
nv set mlag peer-ip linklocal
nv config apply
Quickstart guide Page 13

Once both switches are fully configured, run the nv show mlag command to verify that MLAG is active. In particular, local-role and peer-role
should be different, peer-alive should be true, and backup-active should also be true:

The command nv show mlag consistency-checker global should also be run to find MLAG inconsistencies. These will be listed
in the Conflict column:

Layer 2 port setup

Switch port setup
Speed auto-negotiation is enabled by default for all switch ports and HPE recommends keeping it this way for all initiators, targets, and
switch uplink ports. All switch-to-switch uplink ports must be configured in a bond set for the purposes of fault tolerance and to increase
overall throughput.

VLAN setup
The Ethernet bridge of a switch allows Layer 2 communication among nodes by forming a single domain consisting of physical and logical
interfaces. In NVIDIA Cumulus, users are allowed to utilize a traditional bridge or a VLAN-aware bridge, the latter of which is more
appropriate for larger Layer 2 environments. Unless there is a specific reason to do so, HPE recommends that customers configure a single
VLAN-aware bridge to prevent running into specific limitations with multiple VLAN-aware bridges.

Although possible, it is not recommended for traditional and VLAN-aware bridges to be used together on the same switch. Also, be aware
that the NVIDIA Cumulus NOS reserves VLANs 3725-3999 for internal operations but will not prevent VLAN overlap.
Quickstart guide Page 14

Tagged VLAN setup

Switch ports that accept only tagged traffic are called trunk ports, and all switch ports can be configured to access all predefined VLANs.
NVUE contains a default bridge, br_default, which must have VLANs defined before those VLANs can be assigned to switch ports.
nv set bridge domain br_default vlan 5-7
nv set interface swp1-2 bridge domain br_default
nv config apply

The example command set mentioned previously creates VLANs 5, 6, and 7 on the default switch bridge, br_default, and then assigns all of
the VLANs to switch ports 1 and 2. The default untagged VLAN 1 is also assigned to all ports. If only VLAN 5 is required on switch ports 1
and 2, then the following command set would be used:
nv set bridge domain br_default vlan 5-7
nv set interface swp1-2 bridge domain br_default vlan 5
nv config apply

Note that just assigning a bridge to a port or a range of ports will result in those ports having full access to all tagged and untagged VLANs
defined on that bridge.

Access port setup

A switch port that accepts only untagged traffic and egresses all packets to a specified VLAN is called an access port. All tagged packets
that enter an access port are ignored and VLANs specified in access ports must be defined on the switch default bridge prior to use. Note
that switch ports set as access ports can only utilize a single VLAN. The following set of commands shows how to define switch port swp25
as an access port with VLAN 20:
nv set interface swp25 bridge domain br_default
nv set bridge domain br_default vlan 20
nv set interface swp25 bridge domain br_default access 20
nv config apply

Once the switches are fully configured, the following command can be used to verify that all VLANs are mapped to each bridge port as
expected:
nv show bridge domain br_default port vlan

Untagged VLAN setup

A Cumulus VLAN-aware bridge is allowed to have just one untagged VLAN configured. By default the Primary VLAN Identifier is set to ‘1’
however the value can be modified via the following command ‘nv set bridge domain br_default untagged 30’.

VLAN-aware bridges can also be configured to drop all untagged frames on a per-port basis via the command ‘nv set interface
swp25 bridge domain br_default untagged none’.

The VLAN change may be verified via the command ‘nv show bridge domain br_default port vlan’.
Quickstart guide Page 15

Quality of Service (QoS)

Buffer pool configuration and QoS profile creation
The switch buffer pool needs to be configured for lossy and lossless flows. HPE recommends that 80% of memory be allocated for the
lossless pool that’s utilized by iSCSI Priority 4 and 20% be allocated for lossy flows. The NVUE commands are as follows and should be set on
each switch:
nv set qos traffic-pool default-lossless memory-percent 80
nv set qos traffic-pool default-lossy memory-percent 20
nv config apply
Additionally, specific profiles must be created on all switches and declared on all interfaces that will be using Priority Flow Control (PFC),
Enhanced Transmission Selection (ETS), and link-level pause.
Example PFC profile
nv set qos pfc my_pfc_ports switch-priority 4
nv set qos pfc my_pfc_ports tx enable
nv set qos pfc my_pfc_ports rx enable
nv set qos pfc my_pfc_ports cable-length 50
nv config apply
Example ETS profile
nv set qos egress-scheduler default-ets traffic-class 0,1,2,3,4,5 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 0 bw-percent 20
nv set qos egress-scheduler default-ets traffic-class 1,2,3,5 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 4 bw-percent 80
nv set qos egress-scheduler default-ets traffic-class 6 mode strict
nv set qos egress-scheduler default-ets traffic-class 7 mode strict
nv config apply
Example link-level pause profile
nv set qos link-pause my_pause_ports tx enable
nv set qos link-pause my_pause_ports rx enable
nv set qos link-pause my_pause_ports cable-length 50
nv config apply

nv set qos mapping customports trust port

nv set qos mapping customports port-default-sp 4

nv config apply
Application Priority TLV
NVIDIA Cumulus v5.9.x, v5.10.0, and v5.11.0 NOS provides support for the DCBx Application Priority TLV that allows the transmission of
priority values per application within LLDP data packets. All iSCSI data will be sent through iSCSI priority 4 utilizing the lossless switch queue.
The following command is required for the priority assignment and needs to be specified just once on each switch:
nv set service lldp application-tlv app iSCSI priority 4

All switch ports of initiator and target devices that utilize PFC must have the Application Priority TLV enabled in their LLDP PDUs using the
following command:
nv set interface swp10 lldp application-tlv app iSCSI

Refer to Table 1 for a list of storage arrays that support PFC. Do not enable the Application TLV on the ports of devices that only support
Link-level Flow Control.
Quickstart guide Page 16

Switch array port configuration

The following table shows the currently supported arrays on NVIDIA Cumulus along with the various technologies that can be used with
each one respectively:
Table 1. Switch features supported with individual arrays
Supported storage arrays with HPE M-Series with NVIDIA Cumulus
HPE Storage arrays HPE MSA HPE Nimble Storage HPE Alletra 6000 HPE Primera HPE Alletra 9000 HPE Alletra MP
1060/2060/2070
Switch features

VLAN tagging Not supported Supported Supported Supported Supported Supported

PFC Not supported Not supported Not supported Supported Supported Not supported
ETS Not supported Not supported Not supported Supported Supported Not supported

Link-level Flow Control Supported Supported Supported Supported Supported Supported

All array devices except for the HPE MSA support VLAN tagging. In the case of the HPE MSA devices, the switch ports must be set as
access ports to direct all untagged packets to a specific VLAN. Note that link level pause is supported with all arrays but cannot be set at the
same time as PFC; only one flow control technology must be used at a time. Attempting to set PFC for devices that do not support it will lead
to unexpected behavior.
HPE Primera or HPE Alletra 9000 array switch port configuration
Configuring QoS on target switch ports is dependent on the array type. If the switch ports of an HPE Primera or HPE Alletra 9000 array are
being configured, then the following command set must be used for each switch port since both VLAN tagging and PFC are supported:
nv set interface swp10 lldp application-tlv app iSCSI
nv set interface swp10 lldp dcbx-pfc-tlv on
nv set interface swp10 lldp dcbx-ets-config-tlv on
nv set interface swp10 lldp dcbx-ets-recomm-tlv on
nv set interface swp10 qos pfc profile my_pfc_ports
nv set interface swp10 qos egress-scheduler profile default-ets
nv config apply

HPE MSA 2060 array switch port configuration

If the switch ports of an HPE MSA 2060 are being configured, then the following command set must be used for each switch port.
Remember that VLAN tagging is not supported with this array; therefore, the switch port must be configured as an access port:
nv set interface swp10 bridge domain br_default access 10
nv set interface swp10 qos mapping profile customports
nv set interface swp10 qos link-pause profile my_pause_ports
nv config apply
Quickstart guide Page 17

HPE Alletra MP, HPE Alletra 6000, or HPE Nimble Storage HF40 array switch port configuration
If the switch ports of an HPE Alletra 6000 or HPE Nimble Storage HF40 are being configured, then the following command set must be
used for each switch port:
nv set interface swp10 qos mapping profile customports
nv set interface swp10 qos link-pause profile my_pause_ports
nv config apply

In the case of adapters, it is important to reference the individual device online QuickSpecs to verify whether PFC and ETS are supported.
Switch port configuration for adapters that support PFC and ETS
nv set interface swp5 lldp application-tlv app iSCSI
nv set interface swp5 lldp dcbx-pfc-tlv on
nv set interface swp5 lldp dcbx-ets-config-tlv on
nv set interface swp5 lldp dcbx-ets-recomm-tlv on
nv set interface swp5 qos pfc profile my_pfc_ports
nv set interface swp5 qos egress-scheduler profile default-ets
nv config apply

Switch port configuration for adapters that support link-level pause

nv set interface swp5 qos mapping profile customports
nv set interface swp5 qos link-pause profile my_pause_ports
nv config apply
Quickstart guide Page 18

Configuration example
Example 1: Single spine and dual leaf
The following set of configuration commands are available in NVIDIA Cumulus v5.9.x, 5.10.0, and 5.11.0 NOS and may not be present in
older or newer versions. In the following example, the adapter within Server 1 only supports Link-level Flow Control:

Figure 1. Single spine to dual leaf configuration

HPE SN4600cM—NVIDIA Cumulus-Spine

nv unset interface eth0 ip address dhcp
nv config apply

nv set system config auto-save enable on

nv set interface eth0 ip address 172.17.3.40/16
nv set interface eth0 ip gateway 172.17.0.1
nv set interface eth0 ip vrf mgmt
nv set interface eth0 type eth
nv set service dns mgmt server 172.17.0.5
nv set system hostname SN4600cM-Cumulus-Spine
nv set system timezone US/Eastern
nv config apply

nv unset service ntp default

nv set service ntp mgmt
nv config apply

nv set service ntp mgmt server 172.17.0.3 iburst on

nv config apply

nv set service lldp application-tlv app iSCSI priority 4

nv set qos traffic-pool default-lossless memory-percent 80
nv set qos traffic-pool default-lossy memory-percent 20
nv set qos egress-scheduler default-ets traffic-class 0 bw-percent 20
nv set qos egress-scheduler default-ets traffic-class 0 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 1 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 1 mode dwrr
Quickstart guide Page 19

nv set qos egress-scheduler default-ets traffic-class 2 bw-percent 0

nv set qos egress-scheduler default-ets traffic-class 2 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 3 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 3 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 4 bw-percent 80
nv set qos egress-scheduler default-ets traffic-class 4 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 5 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 5 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 6 mode strict
nv set qos egress-scheduler default-ets traffic-class 7 mode strict
nv set qos pfc my_pfc_ports cable-length 50
nv set qos pfc my_pfc_ports rx enable
nv set qos pfc my_pfc_ports tx enable
nv set qos pfc my_pfc_ports switch-priority 4
nv config apply
nv set bridge domain br_default stp priority 4096
nv set bridge domain br_default vlan 10
nv set bridge domain br_default vlan 20
nv set interface bond1 bond member swp1
nv set interface bond1 bond member swp2
nv set interface bond1 bond member swp60
nv set interface bond1 bond member swp61
nv set interface bond1 type bond
nv set interface bond1 description MLAG-SN3420M-LEAF-SWITCHES
nv set interface bond1 bond lacp-rate fast
nv set interface bond1 bridge domain br_default stp
nv set interface bond1 qos pfc profile my_pfc_ports
nv set interface bond1 qos egress-scheduler profile default-ets
nv set interface bond1 bridge domain br_default vlan 10
nv set interface bond1 bridge domain br_default vlan 20
nv set interface bond1, swp1-64 link state up
nv set interface swp1 description 'Link to SN3420M-CUMULUS-LEAF1-SWP1'
nv set interface swp2 description 'Link to SN3420M-CUMULUS-LEAF1-SWP2'
nv set interface swp60 description 'Link to SN3420M-CUMULUS-LEAF2-SWP1'
nv set interface swp61 description 'Link to SN3420M-CUMULUS-LEAF2-SWP2'
nv set interface swp1-2,60-61 link speed auto
nv set interface swp1-2,60-61 lldp dcbx-ets-config-tlv on
nv set interface swp1-2,60-61 lldp dcbx-ets-recomm-tlv on
nv set interface swp1-2,60-61 lldp dcbx-pfc-tlv on
nv set interface swp1-2,60-61 lldp application-tlv app iSCSI
nv set interface swp1-2,60-61 type swp
nv config apply
Quickstart guide Page 20

HPE SN3420M—NVIDIA Cumulus-Leaf1

nv unset interface eth0 ip address dhcp
nv config apply

nv set system config auto-save enable on

nv set interface eth0 ip address 172.17.5.104/16
nv set interface eth0 ip gateway 172.17.0.1
nv set interface eth0 type eth
nv set service dns mgmt server 172.17.0.5
nv set service ntp mgmt server 172.17.0.3 iburst on
nv set system hostname SN3420-Cumulus-Leaf1
nv set system timezone US/Eastern
nv config apply
nv set service lldp application-tlv app iSCSI priority 4
nv set qos traffic-pool default-lossless memory-percent 80
nv set qos traffic-pool default-lossy memory-percent 20
nv config apply

nv set qos egress-scheduler default-ets traffic-class 0 bw-percent 20

nv set qos egress-scheduler default-ets traffic-class 0 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 1 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 1 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 2 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 2 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 3 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 3 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 4 bw-percent 80
nv set qos egress-scheduler default-ets traffic-class 4 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 5 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 5 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 6 mode strict
nv set qos egress-scheduler default-ets traffic-class 7 mode strict
nv config apply

nv set qos link-pause my_pause_ports cable-length 50

nv set qos link-pause my_pause_ports rx enable
nv set qos link-pause my_pause_ports tx enable
nv config apply
nv set qos mapping customports trust port
nv set qos mapping customports port-default-sp 4
nv config apply
Quickstart guide Page 21

nv set qos pfc my_pfc_ports cable-length 50

nv set qos pfc my_pfc_ports rx enable
nv set qos pfc my_pfc_ports switch-priority 4
nv set qos pfc my_pfc_ports tx enable
nv set bridge domain br_default stp priority 32768
nv set bridge domain br_default vlan 10
nv set bridge domain br_default vlan 20
nv set interface bond1 bond member swp49
nv set interface bond1 bond member swp50
nv set interface bond1 description UPLINK-TO-SN4600cM-Cumulus-Spine
nv set interface bond1 bond lacp-rate fast
nv config apply

nv set interface bond1 bond mlag id 1

nv config apply

nv set interface bond1,peerlink,swp1-2 bridge domain br_default vlan 10

nv set interface bond1,peerlink,swp1-2 bridge domain br_default vlan 20
nv set interface peerlink bond member swp59-60
nv set mlag mac-address 44:38:39:ff:00:00
nv set mlag backup 172.17.5.105 vrf mgmt
nv set mlag peer-ip linklocal
nv config apply

nv set interface bond1,peerlink.4094 link mtu 9216

nv set interface bond1,swp1-2,49-50 link state up
nv set interface bond1,swp1-2 bridge domain br_default stp
nv set interface bond1 qos egress-scheduler profile default-ets
nv set interface bond1 qos pfc profile my_pfc_ports
nv set interface swp1-2 bridge domain br_default stp auto-edge on
nv set interface swp1-2,49-50,59-60 type swp
nv set interface swp1-2 link speed auto
nv set interface swp1 description Server1-AdapterPort1
nv set interface swp2 description Alletra6K-iSCSI-Port1
nv set interface swp1-2 qos link-pause profile my_pause_ports
nv set interface swp1-2 qos mapping profile customports
nv config apply
Quickstart guide Page 22

HPE SN3420M—NVIDIA Cumulus-Leaf2

nv unset interface eth0 ip address dhcp
nv config apply

nv set system config auto-save enable on

nv set interface eth0 ip address 172.17.5.105/16
nv set interface eth0 ip gateway 172.17.0.1
nv set interface eth0 type eth
nv set service dns mgmt server 172.17.0.5
nv set service ntp mgmt server 172.17.0.3 iburst on
nv set system hostname SN3420-Cumulus-Leaf12
nv set system timezone US/Eastern
nv config apply
nv set service lldp application-tlv app iSCSI priority 4
nv set qos traffic-pool default-lossless memory-percent 80
nv set qos traffic-pool default-lossy memory-percent 20
nv config apply

nv set qos egress-scheduler default-ets traffic-class 0 bw-percent 20

nv set qos egress-scheduler default-ets traffic-class 0 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 1 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 1 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 2 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 2 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 3 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 3 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 4 bw-percent 80
nv set qos egress-scheduler default-ets traffic-class 4 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 5 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 5 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 6 mode strict
nv set qos egress-scheduler default-ets traffic-class 7 mode strict
nv config apply

nv set qos link-pause my_pause_ports cable-length 50

nv set qos link-pause my_pause_ports rx enable
nv set qos link-pause my_pause_ports tx enable
nv config apply

nv set qos mapping customports trust port

nv set qos mapping customports port-default-sp 4
nv config apply
Quickstart guide Page 23

nv set qos pfc my_pfc_ports cable-length 50

nv set qos pfc my_pfc_ports rx enable
nv set qos pfc my_pfc_ports switch-priority 4
nv set qos pfc my_pfc_ports tx enable
nv set bridge domain br_default stp priority 32768
nv set bridge domain br_default vlan 10
nv set bridge domain br_default vlan 20
nv set interface bond1 bond member swp49-50
nv set interface bond1 description UPLINK-TO-SN4600cM-Cumulus-Spine
nv set interface bond1 bond lacp-rate fast
nv config apply

nv set interface bond1 bond mlag id 1

nv config apply

nv set interface bond1,peerlink,swp1-2 bridge domain br_default vlan 10

nv set interface bond1,peerlink,swp1-2 bridge domain br_default vlan 20
nv set interface peerlink bond member swp59-60
nv set mlag mac-address 44:38:39:ff:00:00
nv set mlag backup 172.17.5.104 vrf mgmt
nv set mlag peer-ip linklocal
nv config apply

nv set interface bond1,peerlink.4094 link mtu 9216

nv set interface bond1,swp1-2,49-50 link state up
nv set interface bond1,swp1-2 bridge domain br_default stp
nv set interface bond1 qos egress-scheduler profile default-ets
nv set interface bond1 qos pfc profile my_pfc_ports
nv set interface swp1-2 bridge domain br_default stp auto-edge on
nv set interface swp1-2,49-50,59-60 type swp
nv set interface swp1-2 link speed auto
nv set interface swp1 description Server1-AdapterPort2
nv set interface swp2 description Alletra6K-iSCSI-Port2
nv set interface swp1-2 qos link-pause profile my_pause_ports
nv set interface swp1-2 qos mapping profile customports
nv config apply
Quickstart guide Page 24

Example 2: Dual spine and dual leaf

In the following figure, the adapter within Server 1 supports PFC and ETS:

Figure 2. Dual spine to dual leaf configuration

HPE SN4600cM—NVIDIA Cumulus-Spine1

nv unset interface eth0 ip address dhcp
nv config apply

nv set system config auto-save enable on

nv set interface eth0 ip address 172.17.5.106/16
nv set interface eth0 ip gateway 172.17.0.1
nv set interface eth0 type eth
nv set service dns mgmt server 172.17.0.5
nv set system hostname SN4600cM-Cumulus-Spine1
nv set system timezone US/Eastern
nv config apply

nv unset service ntp default

nv set service ntp mgmt
nv config apply

nv set service ntp mgmt server 172.17.0.3 iburst on

nv config apply

nv set service lldp application-tlv app iSCSI priority 4

nv set qos traffic-pool default-lossless memory-percent 80
nv set qos traffic-pool default-lossy memory-percent 20
nv config apply

nv set qos egress-scheduler default-ets traffic-class 0 bw-percent 20

nv set qos egress-scheduler default-ets traffic-class 0 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 1 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 1 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 2 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 2 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 3 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 3 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 4 bw-percent 80
Quickstart guide Page 25

nv set qos egress-scheduler default-ets traffic-class 4 mode dwrr

nv set qos egress-scheduler default-ets traffic-class 5 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 5 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 6 mode strict set
nv qos egress-scheduler default-ets traffic-class 7 mode strict config
nv apply
nv set qos link-pause my_pause_ports cable-length 50
nv set qos link-pause my_pause_ports rx enable
nv set qos link-pause my_pause_ports tx enable
nv config apply

nv set qos mapping customports trust port

nv set qos mapping customports port-default-sp 4
nv config apply

nv set qos pfc my_pfc_ports cable-length 50

nv set qos pfc my_pfc_ports rx enable
nv set qos pfc my_pfc_ports switch-priority 4
nv set qos pfc my_pfc_ports tx enable config apply
nv set bridge domain br_default stp priority 4096
nv set bridge domain br_default vlan 10
nv set bridge domain br_default vlan 20
nv set interface bond1 bond member swp1-2
nv set interface bond1 description BOND-TO-SN3420M-SWITCHES
nv set interface bond1 bond lacp-rate fast
nv config apply

nv set interface bond1 bond mlag id 1

nv set interface bond1 bond mlag enable on
nv config apply

nv set interface bond1,peerlink,swp1-2 bridge domain br_default vlan 10

nv set interface bond1,peerlink,swp1-2 bridge domain br_default vlan 20
nv set interface peerlink bond member swp63-64
nv set mlag mac-address 44:38:39:ff:00:00
nv set mlag backup 172.17.5.107 vrf mgmt
nv set mlag peer-ip linklocal
nv config apply
nv set interface bond1,peerlink.4094 link mtu 9216
nv set interface bond1,swp1-2 link state up
nv set interface bond1 bridge domain br_default stp
nv set interface bond1 qos egress-scheduler profile default-ets
nv set interface bond1 qos pfc profile my_pfc_ports
nv set interface swp1-2 bridge domain br_default stp auto-edge on
nv config apply
Quickstart guide Page 26

nv set interface swp1-2,63-64 type swp

nv set interface swp1-2,63-64 link speed auto
nv set interface swp1-2 lldp application-tlv app iSCSI
nv set interface swp1-2 lldp dcbx-ets-config-tlv on
nv set interface swp1-2 lldp dcbx-ets-recomm-tlv on
nv set interface swp1-2 lldp dcbx-pfc-tlv on

HPE SN4600cM—NVIDIA Cumulus-Spine2

nv unset interface eth0 ip address dhcp
nv config apply

nv set system config auto-save enable on

nv set interface eth0 ip address 172.17.5.107/16
nv set interface eth0 ip gateway 172.17.0.1
nv set interface eth0 type eth
nv set service dns mgmt server 172.17.0.5
nv set system hostname SN4600cM-Cumulus-Spine2
nv set system timezone US/Eastern
nv config apply

nv unset service ntp default

nv set service ntp mgmt
nv config apply

nv set service ntp mgmt server 172.17.0.3 iburst on

nv config apply

nv set service lldp application-tlv app iSCSI priority 4

nv set qos traffic-pool default-lossless memory-percent 80
nv set qos traffic-pool default-lossy memory-percent 20
nv config apply

nv set qos egress-scheduler default-ets traffic-class 0 bw-percent 20

nv set qos egress-scheduler default-ets traffic-class 0 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 1 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 1 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 2 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 2 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 3 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 3 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 4 bw-percent 80
nv set qos egress-scheduler default-ets traffic-class 4 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 5 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 5 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 6 mode strict
nv set qos egress-scheduler default-ets traffic-class 7 mode strict
nv config apply
Quickstart guide Page 27

nv set qos link-pause my_pause_ports cable-length 50

nv set qos link-pause my_pause_ports rx enable
nv set qos link-pause my_pause_ports tx enable
nv config apply

nv set qos mapping customports trust port

nv set qos mapping customports port-default-sp 4
nv config apply

nv set qos pfc my_pfc_ports cable-length 50

nv set qos pfc my_pfc_ports rx enable
nv set qos pfc my_pfc_ports switch-priority 4
nv set qos pfc my_pfc_ports tx enable
nv set bridge domain br_default stp priority 4096
nv set bridge domain br_default vlan 10
nv set bridge domain br_default vlan 20
nv set interface bond1 bond member swp1-2
nv set interface bond1 description BOND-TO-SN3420M-SWITCHES
nv set interface bond1 bond lacp-rate fast
nv config apply

nv set interface bond1 bond mlag id 1

nv set interface bond1 bond mlag enable on
nv config apply

nv set interface bond1,peerlink,swp1-2 bridge domain br_default vlan 10

nv set interface bond1,peerlink,swp1-2 bridge domain br_default vlan 20
nv set interface peerlink bond member swp63-64
nv set mlag mac-address 44:38:39:ff:00:00
nv set mlag backup 172.17.5.106 vrf mgmt
nv set mlag peer-ip linklocal
nv config apply

nv set interface bond1,peerlink.4094 link mtu 9216

nv set interface bond1,swp1-2 link state up
nv set interface bond1 bridge domain br_default stp
nv set interface bond1 qos egress-scheduler profile default-ets
nv set interface bond1 qos pfc profile my_pfc_ports
nv set interface swp1-2 bridge domain br_default stp auto-edge on
nv config apply
Quickstart guide Page 28

nv set interface swp1-2,63-64 type swp

nv set interface swp1-2,63-64 link speed auto
nv set interface swp1-2 lldp application-tlv app iSCSI
nv set interface swp1-2 lldp dcbx-ets-config-tlv on
nv set interface swp1-2 lldp dcbx-ets-recomm-tlv on
nv set interface swp1-2 lldp dcbx-pfc-tlv on
nv config apply

HPE SN3420M—NVIDIA Cumulus-Leaf1

nv unset interface eth0 ip address dhcp
nv config apply

nv set system config auto-save enable on

nv set interface eth0 ip address 172.17.5.104/16
nv set interface eth0 ip gateway 172.17.0.1
nv set interface eth0 type eth
nv config apply

nv set system hostname SN3420M-Cumulus-Leaf1

nv set system timezone US/Eastern
nv config apply

nv unset service ntp default

nv set service ntp mgmt
nv config apply

nv set service ntp mgmt server 172.17.0.3 iburst on

nv config apply

nv set service lldp application-tlv app iSCSI priority 4

nv set qos traffic-pool default-lossless memory-percent 80
nv set qos traffic-pool default-lossy memory-percent 20
nv config apply

nv set qos egress-scheduler default-ets traffic-class 0 bw-percent 20

nv set qos egress-scheduler default-ets traffic-class 0 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 1 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 1 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 2 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 2 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 3 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 3 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 4 bw-percent 80
nv set qos egress-scheduler default-ets traffic-class 4 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 5 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 5 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 6 mode strict
nv set qos egress-scheduler default-ets traffic-class 7 mode strict
nv config apply
Quickstart guide Page 29

nv set qos link-pause my_pause_ports cable-length 50

nv set qos link-pause my_pause_ports rx enable
nv set qos link-pause my_pause_ports tx enable
nv config apply

nv set qos mapping customports trust port

nv set qos mapping customports port-default-sp 4
nv config apply

nv set qos pfc my_pfc_ports cable-length 50

nv set qos pfc my_pfc_ports rx enable
nv set qos pfc my_pfc_ports switch-priority 4
nv set qos pfc my_pfc_ports tx enable
nv config apply

nv set bridge domain br_default stp priority 32768

nv set bridge domain br_default vlan 10
nv set bridge domain br_default vlan 20
nv set interface bond1 bond member swp49-50
nv set interface bond1 type bond
nv set interface bond1 description UPLINK-TO-SN4600cM
nv set interface bond1 bond lacp-rate fast
nv config apply

nv set interface bond1 bond mlag enable on

nv set interface bond1 bond mlag id 1
nv config apply

nv set interface bond1,peerlink bridge domain br_default vlan 10

nv set interface bond1,peerlink bridge domain br_default vlan 20
nv set interface peerlink bond member swp59-60
nv set mlag mac-address 44:38:39:ff:00:01
nv set mlag backup 172.17.5.105 vrf mgmt
nv set mlag peer-ip linklocal
nv config apply

nv set interface bond1,peerlink.4094,swp1-2 link mtu 9216

nv set interface bond1,swp1-2,59-60 link state up
nv set interface bond1,swp1-2 bridge domain br_default stp
nv set interface bond1,swp1-2 qos egress-scheduler profile default-ets
nv set interface bond1,swp1-2 qos pfc profile my_pfc_ports
nv set interface swp1-2 bridge domain br_default stp auto-edge on
nv config apply

nv set interface swp1-2,59-60 type swp

nv set interface swp1-2 link speed auto
nv set interface swp59-60 link speed auto
Quickstart guide Page 30

nv set interface swp1 description Server1-AdapterPort1

nv set interface swp2 description Alletra9K-Port1
nv set interface swp1-2 lldp application-tlv app iSCSI
nv set interface swp1-2 lldp dcbx-ets-config-tlv on
nv set interface swp1-2 lldp dcbx-ets-recomm-tlv on
nv set interface swp1-2 lldp dcbx-pfc-tlv on
nv config apply
Quickstart guide Page 31

HPE SN3420M—NVIDIA Cumulus-Leaf2

nv unset interface eth0 ip address dhcp
nv config apply

nv set system config auto-save enable on

nv set interface eth0 ip address 172.17.5.105/16
nv set interface eth0 ip gateway 172.17.0.1
nv set interface eth0 type eth
nv config apply

nv set system hostname SN3420M-Cumulus-Leaf2

nv set system timezone US/Eastern
nv config apply

nv unset service ntp default

nv set service ntp mgmt
nv config apply

nv set service ntp mgmt server 172.17.0.3 iburst on

nv config apply

nv set service lldp application-tlv app iSCSI priority 4

nv set qos traffic-pool default-lossless memory-percent 80
nv set qos traffic-pool default-lossy memory-percent 20
nv config apply

nv set qos egress-scheduler default-ets traffic-class 0 bw-percent 20

nv set qos egress-scheduler default-ets traffic-class 0 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 1 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 1 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 2 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 2 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 3 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 3 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 4 bw-percent 80
nv set qos egress-scheduler default-ets traffic-class 4 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 5 bw-percent 0
nv set qos egress-scheduler default-ets traffic-class 5 mode dwrr
nv set qos egress-scheduler default-ets traffic-class 6 mode strict
nv set qos egress-scheduler default-ets traffic-class 7 mode strict
nv config apply

nv set qos link-pause my_pause_ports cable-length 50

nv set qos link-pause my_pause_ports rx enable
nv set qos link-pause my_pause_ports tx enable
nv config apply
Quickstart guide Page 32

nv set qos mapping customports trust port

nv set qos mapping customports port-default-sp 4
nv config apply

nv set qos pfc my_pfc_ports cable-length 50

nv set qos pfc my_pfc_ports rx enable
nv set qos pfc my_pfc_ports switch-priority 4
nv set qos pfc my_pfc_ports tx enable
nv config apply
nv set bridge domain br_default stp priority 32768
nv set bridge domain br_default vlan 10
nv set bridge domain br_default vlan 20
nv set interface bond1 bond member swp49-50
nv set interface bond1 type bond
nv set interface bond1 description UPLINK-TO-SN4600cM
nv set interface bond1 bond lacp-rate fast
nv config apply

nv set interface bond1 bond mlag enable on

nv set interface bond1 bond mlag id 1
nv config apply

nv set interface bond1,peerlink bridge domain br_default vlan 10

nv set interface bond1,peerlink bridge domain br_default vlan 20
nv set interface peerlink bond member swp59-60
nv set mlag mac-address 44:38:39:ff:00:01
nv set mlag backup 172.17.5.104 vrf mgmt
nv set mlag peer-ip linklocal
nv config apply

nv set interface bond1,peerlink.4094,swp1-2 link mtu 9216

nv set interface bond1,swp1-2,59-60 link state up
nv set interface bond1,swp1-2 bridge domain br_default stp
nv set interface bond1,swp1-2 qos egress-scheduler profile default-ets
nv set interface bond1,swp1-2 qos pfc profile my_pfc_ports
nv set interface swp1-2 bridge domain br_default stp auto-edge on
nv config apply

nv set interface swp1-2,59-60 type swp

nv set interface swp1-2 link speed auto
nv set interface swp59-60 link speed auto
nv set interface swp1 description Server1-AdapterPort1
nv set interface swp2 description Alletra9K-Port1
nv set interface swp1-2 lldp application-tlv app iSCSI
nv set interface swp1-2 lldp dcbx-ets-config-tlv on
nv set interface swp1-2 lldp dcbx-ets-recomm-tlv on
nv set interface swp1-2 lldp dcbx-pfc-tlv on
nv config apply
Quickstart guide

Conclusion
The HPE M-Series with NVIDIA Cumulus v5.9.x, v5.10.0, and v5.11.0 NOS quickstart guide is intended to provide customers with a
basic framework to set up switches and is not intended to be inclusive of every available configuration option. Users can reference the
online NVIDIA Cumulus Linux 5.11.0 User Guide.l

Learn more at
HPE.com/us/en/Storage/Networking.html

© Copyright 2025 Hewlett Packard Enterprise Development LP. The information contained herein is subject to
change without notice. The only warranties for Hewlett Packard Enterprise products and services are set forth in the
express warranty statements accompanying such products and services. Nothing herein should be construed as
constituting an additional warranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors or
omissions contained herein.

Linux is the registered trademark of Linus Torvalds in the U.S. and other countries. NVIDIA is a trademark and/or
registered trademark of NVIDIA Corporation in the U.S. and other countries. All third-party marks are property of
their respective owners.

a00143358ENW, Rev. 4