H3C Data Center Network Solution Underlay

Network Design Guide

Document version: 6W100-20221206

Copyright © 2022 New H3C Technologies Co., Ltd. All rights reserved.
No part of this manual may be reproduced or transmitted in any form or by any means without prior written consent of New
H3C Technologies Co., Ltd.
Except for the trademarks of New H3C Technologies Co., Ltd., any trademarks that may be mentioned in this document
are the property of their respective owners.
The information in this document is subject to change without notice.
Contents
Overview ······································································································· 1
Solution architecture ······················································································ 1
Features ····················································································································································· 1
Network design principles ·························································································································· 1
Leaf gateway design ·································································································································· 2
Routing protocol selection ·························································································································· 5
Capacity planning······································································································································· 8
Multi-fabric expansion ······························································································································ 10
Recommended hardware ························································································································· 12
Access design ····························································································· 12
Server access design ······································································································································· 12
Connecting border devices to external gateways ···························································································· 14
M-LAG topology ······································································································································· 14
Triangle topology······································································································································ 15
Square topology ······································································································································· 16
IP planning principles ······································································································································· 17
Deployment modes ······················································································ 17
About the deployment modes ·························································································································· 17
Management network configuration ················································································································· 18
Overview
A data center network interconnects servers in a data center, distributed data centers, and data
center and end users. Data center underlay connectivity protocols have gradually evolved from
primarily Layer 2 protocols to primarily IP routing protocols. Driven by the scale of computing, the
physical topology of data centers has evolved from an access-aggregation-core three-tier network
architecture to a CLOS-based two-tier spine-leaf architecture.
This document describes the underlay network that uses the spine-leaf architecture.

Solution architecture
The underlay network uses OSPF or EBGP for communication between servers or between servers
and the external network, and uses M-LAG between leaf nodes for access reliability.
Select appropriate switch models and configure spine and leaf nodes as needed based on the
access server quantity, interface bandwidth and type, and convergence ratio to build a tiered DC
network for elastic scalability and service deployment agility.

Features
A typical data center fabric network offers the following features:
• Support for one or multiple spine-leaf networks.
• Flexible configuration and elastic scaling of spine and leaf nodes.
• VLAN deployment for isolation at Layer 2, and VPN deployment for isolation at Layer 3.
As a best practice, use the following network models in a spine-leaf network:
• Configure M-LAG for leaf nodes for access availability.
• Interconnect leaf and spine devices at Layer 3, and configure OSPF or EBGP for equal-cost
multi-path load balancing and link backup.

Network design principles

As a best practice, deploy S68xx switch series, S98xx switch series, and S12500 switch series in
the data center, and configure spine and leaf nodes based on the network scale.

1
 Figure 1 Spine-leaf architecture

Spin e 1 Spin e 2
(BGP E VPN RR) (BGP E VPN RR)

ECMP

Peer link Peer link

Lea f 1 Lea f 2 Lea f 3 Lea f 4

Virtuali zation Bar e metal Virtuali zation Bar e metal

server server server server

Spine design
Use Layer 3 Ethernet interfaces to interconnect spine and leaf nodes to build an all-IP fabric.
Leaf design
• Configure M-LAG, S-MLAG, or IRF on the leaf nodes to enhance availability and avoid single
points of failure. As a best practice, configure M-LAG on the leaf nodes.
• Connect each leaf node to all spine nodes to build a full mesh mode network.
• Leaf nodes are typically Top of Rack (ToR) devices. As a best practice to decrease deployment
complexity, use the controller to deploy configuration automatically or use zero-touch
provisioning (ZTP) for deployment.
ZTP automates loading of system software, configuration files, and patch files when devices
with factory default configuration or empty configuration start up.

Leaf gateway design

Gateway deployment scheme overview
As a best practice, configure M-LAG in conjunction with VLAN or VRRP on leaf nodes to provide
redundant gateways for servers.
Table 1 Gateway deployment scheme

Gateway type Description

• A VLAN interface is configured on each M-LAG member device,
and both M-LAG member devices can respond to ARP packets
and perform Layer 3 forwarding.
• Attached servers require Layer 3 connectivity to the M-LAG
system in some scenarios, K8s containers are deployed on the
Dual-active VLAN interfaces servers for example. To fulfil this requirement, perform one of the
(recommended) following tasks:
 Configure static routes.
 Assign a virtual IPv4 or IPv6 address to each gateway VLAN
interface by using the port m-lag virtual-ip or
port m-la ipv6 virtual-ip command.

2
 • Both the VRRP master and backup devices perform Layer 3
forwarding, but only the master device responds to ARP packets.
VRRP group
• The server-side devices can set up dynamic routing neighbor
relationships with the M-LAG member devices.

M-LAG + dual-active VLAN interfaces

Configure a VLAN interface on each M-LAG member device, as shown in Figure 2. Both M-LAG
member devices can respond to ARP packets and perform Layer 3 forwarding.
Figure 2 Network diagram
Spine

Vlan-int 100 Dual-active VLAN Vlan-int 100

IPv4: 100.1.1.100/24 interfaces IPv4: 100.1.1.100/24
IPv6: 100::100/96 IPv6: 100::100/96
MAC: 0000-0010-0010 MAC: 0000-0010-0010
Peer link
Leaf1 Leaf2

Server

To configure dual-active VLAN interfaces on the M-LAG system, perform the following tasks:
1. Create a gateway VLAN interface on each M-LAG member device for the same VLAN.
2. Assign the same IPv4 and IPv6 addresses and MAC address to the gateway VLAN interfaces.
The dual-active VLAN interfaces operate as follows:
• Each M-LAG member device forwards traffic locally instead of forwarding traffic towards the
M-LAG peer over the peer link. For example, Leaf 1 will directly respond to an ARP request
received from the server.
• When one uplink fails, traffic is switched to the other uplink. For example, when Leaf 1 is
disconnected from the spine device, traffic is forwarded as follows.
 Downlink traffic is switched to Leaf 2 for forwarding.
 When receiving uplink traffic destined for the spine device, Leaf 2 forwards the traffic locally.
When Leaf 1 receives uplink traffic, it sends the traffic to Leaf 2 over the peer link, and Leaf
2 forwards the traffic to the spine device.
• Traffic is load shared between the access links to increase bandwidth utilization.
The dual-active VLAN interfaces use the same IP address and MAC address. As shown in Figure 3
and Table 2, for the M-LAG member devices to set up routing neighbor relationships with the
server-side network device, perform the following tasks:
• Use the port m-lag virtual-ip or port m-lag ipv6 virtual-ip command to
assign an M-LAG virtual IP address to the dual-active VLAN interfaces.
• Configure routing protocols.
The dual-active VLAN interfaces will use the virtual IP addresses to establish routing neighbor
relationships.
3
 Figure 3 Routing neighbor relationship setup
Spine

32.1.1.2/24 33.1.1.2/24

Vlan-int 101 BGP Vlan-int 101

101.1.1.1/24 32.1.1.1/24 33.1.1.1/24 101.1.1.2/24
Peer link
Leaf1 Leaf2
Dual-active VLAN
Vlan-int 100 Vlan-int 100
IPv4: 100.1.1.100/24 interfaces IPv4: 100.1.1.100/24
IPv6: 100::100/96 IPv6: 100::100/96
MAC: 0000-0010-0010
OSPF/BGP MAC: 0000-0010-0010
M-LAG Virtual IP: 100.1.1.101/24 M-LAG Virtual IP: 100.1.1.102/24

Server
100.1.1.103/24

Table 2 Configuration tasks

Tasks Forwarding
• Dual-active VLAN interface configuration:
a. Create a VLAN interface on each M-LAG
member device for the same VLAN.
b. Assign the same IP address and MAC address
to the VLAN interfaces.
The traffic sent to the M-LAG system is load
c. Assign a unique virtual IP address from the shared between the uplinks or downlinks. The
same subnet to each of the VLAN interfaces M-LAG member devices forward traffic as
with the same VLAN ID. follows:
The VLAN interfaces use the virtual IP
addresses for BGP or OSPF neighbor • For Layer 2 traffic sent by the server, the
relationship setup. M-LAG member devices look up the MAC
address table and forward the traffic locally.
• Create a VLAN interface on each M-LAG member
• For Layer 3 traffic sent by the server, the
device for another VLAN, assign the peer-link
M-LAG member devices perform Layer 3
interfaces to this VLAN, and assign a unique IP
forwarding based on the FIB table.
address from the same subnet to each of the VLAN
interfaces. • For external traffic destined for the server,
The M-LAG member devices use these VLAN the M-LAG member devices perform
interfaces to forward Layer 3 traffic between them forwarding based on the FIB table.
when a link to the upstream spine device is failed.
• Use Layer 3 interfaces to connect the M-LAG
member devices to the upstream spine device, and
configure ECMP routes for load sharing between
the uplinks.

M-LAG + VRRP gateways

You can configure VRRP groups on an M-LAG system to provide gateway services for the attached
server, as shown in Figure 4 and Table 3.

4
 Figure 4 Network diagram
Spine

10.39.244.12/31 10.39.244.44/31

BGP
Vlan-int 4094 Vlan-int 4094
100.100.100.1/30 100.100.100.2/30

10.39.244.13/31 10.39.244.45/31
Peer link
Leaf1 Leaf2

Vlan-int 100 VRRP Vlan-int 100

IP: 10.40.196.61/26 IP: 10.40.196.62/26
Sub IP: 100.127.128.253/24 OSPF/BGP Sub IP: 100.127.128.254/24
VRRP virtual IP: 10.40.196.1 VRRP virtual IP: 10.40.196.1

VLAN 100

Server
100.127.128.3/24

Table 3 Configuration tasks

Tasks Forwarding
• Configure a VRRP group on the M-LAG member devices
and use the VRRP virtual IP address as the gateway for the
attached server.
• VLAN interface configuration:
a. Create a VLAN interface for the VLAN where the
M-LAG interface resides on each M-LAG member The traffic sent to the M-LAG system is
device. load shared between the uplinks or
b. Assign a unique primary IP address from the same downlinks. The M-LAG member devices
subnet to each of the VLAN interfaces. forward traffic as follows:
c. Assign a unique secondary IP address from another • For the Layer 3 traffic sent by the
subnet to each of the VLAN interfaces. server, both M-LAG member
devices perform Layer 3
• Use the primary or secondary IP addresses of the VLAN forwarding.
interfaces to set up BGP or OSPF neighbor relationships
with the server-side network device. • For the external traffic destined for
the server, the M-LAG member
• Set up a Layer 3 connection over the peer link between the devices make forwarding decisions
M-LAG member devices. based on local routes.
The M-LAG member devices use the Layer 3 connection to
forward traffic between them when a link to the upstream
spine device is failed.
• Use Layer 3 interfaces to connect the M-LAG member
devices to the upstream spine device, and configure ECMP
routes for load sharing between the uplinks.

Routing protocol selection

About underlay routing protocol selection
As a best practice, select OSPF or EBGP as the underlay routing protocol based on the network
scale.
5
 Table 4 About underlay routing protocol selection

Protocol Advantages Disadvantages Applicable scenarios

 Simple deployment.
 Fast convergence.  Single area in small- and
 Because OSPF packets in 
medium-sized networks,
Limited OSPF routing
the underlay and BGP and multiple areas in
domain.
OSPF packets in the overlay are in large networks.
 Relatively large fault
different queues, the VPN  Recommended number
domain.
instance and route entries of neighbors: Less than
are isolated, implementing 200.
fault isolation.
 Independent routing domain
per AS. Controllable fault  Medium-sized and large
domain. networks.
EBGP  Flexible route control and Complicated configuration.  Recommended number
flexible scalability. of neighbors: Less than
 Applicable to large 500.
networks.

Detailed OSPF characteristics

• Network scale
 Applicable to a wide scope, and supports various network scales.
 Easy deployment. As best practice, deploy OSPF to small-sized networks.
• Area-based OSPF network partition
 In large OSPF routing domains, SPF route computations consume too many storage and
CPU resources. To resolve these issues, OSPF splits an AS into multiple areas. Each area
is identified by an area ID.
 Area-based network partition can reduce the LSDB size, memory consumption and CPU
load, as well as network bandwidth usage because of less routing information transmitted
between areas.
 For a single-fabric network, you can deploy internal devices to the same OSPF area.
 For a multi-fabric network, deploy each area to a different fabric, and connect the areas
through the backbone area.
• ECMP: Supports multiple equal-cost routes to the same destination.
• Fast convergence: Advertises routing updates instantly upon network topology changes.
Detailed BGP characteristics
BGP is more advantageous in large networks with the following characteristics:
• Reduces bandwidth consumption by advertising only incremental updates.
• Reduces the number of routes by using route aggregation, route dampening, community
attributes, and route reflection.
When more than 200 leaf nodes exist, as a best practice, use EBGP to deploy underlay network
routes. For recommended deployment solutions, see "Recommended BGP deployment solution 1"
and "Recommended BGP deployment solution 2."

6
Recommended BGP deployment solution 1
Figure 5 Recommended EBGP route planning within a single fabric
PE PE

AS 65505
Border Border

Spine
AS 65504 Spine

IBGP
Leaf1 Leaf2 Leaf3 Leaf4 Service Leaf Service Leaf
AS 65501 AS 65502 AS 65503

Characteristics of recommended BGP deployment solution 1:

• Assign an AS number to each group of leaf devices, and you do not need to disable BGP
routing loop prevention.
• Establish IBGP peers between leaf devices in a group over the peer link, and establish EBGP
peers between leaf devices and spine devices.
• An uplink failure can trigger a switchover through the peer link. The current solution supports
switchover by establishing neighbors through the peer link between the M-LAG devices.
Switchover through Monitor Link will be added in the future as planned.

7
Recommended BGP deployment solution 2
Figure 6 Recommended EBGP route planning within a single fabric
PE PE

AS 65501
Border Border

AS 65504 Spine
Spine

IBGP
Leaf Leaf Leaf1 Leaf2 Service Leaf Service Leaf
AS 65501 AS 65501 AS 65501

Characteristics of recommended BGP deployment solution 2:

• Assign the same AS number to all leaf and border devices.
• Establish EBGP peers between leaf or border devices in each group with the spine devices.
Establish IBGP peers between the two devices in each group.
• Because all leaf and border devices use the same AS number, you need to disable BGP
routing loop prevention on leaf and border nodes. Alternatively, configure a routing policy on
the spine nodes for AS-PATH replacement (recommended) as follows:
 route-policy AS_Replace permit node node-number
 apply as-path 65504 replace
 peer group-name route-policy AS_Replace import
• The current solution supports switchover by establishing neighbors through the peer link
between the M-LAG devices. Switchover through Monitor Link will be added in the future as
planned.

Capacity planning
This section introduces the server capacity design within a single fabric.
Design principles
The number of spine downlink interfaces is the number of leaf devices. Number of servers =
Number of leaf devices × number of leaf downlink interfaces / 2.
Typical design
• M-LAG access
When the access switches adopt M-LAG, the leaf devices that form an M-LAG system use two
high-speed ports to establish the peer link, and four ports in the uplink. Among the four ports,
each two ports are connected to one spine device.

8
 Table 5 The leaf devices adopt 10-G interfaces to connect to the servers and 40-G uplink
interfaces

Spine Number of Spine convergence Number of

Number of access
device spine ratio access
switches
model devices (uplink/downlink) servers
432 / 2 = 216
Evaluation based on 36 ×
40-G interface card: Convergence ratio (1:3)
That is, 160:480 (4 × 216 × 48 / 2
Convergence ratio (1:3)
40-G uplink ports and 48 = 5184
That is, 144 uplink ports
× 10-G downlink ports
and 432 downlink ports.
per switch).
S12516X-AF 2
288 / 2 = 144
Evaluation based on 36 ×
40-G interface card: Convergence ratio (1:3)
That is, 160:480 (4 × 144 × 48 / 2
Convergence ratio (1:1)
40-G uplink ports and 48 = 3456
That is, 288 uplink ports
× 10-G downlink ports
and 288 downlink ports.
per switch).

Table 6 The leaf devices adopt 25-G interfaces to connect to the servers and 100-G
uplink interfaces

Number
Spine Number
Spine convergence ratio Number of of
device of spine
(downlink/uplink) access switches access
model devices
servers
432 / 2 = 216
Evaluation based on 36 × 100-G Convergence ratio
interface card: (1:3)
That is, 400:1200 (4 216 × 48 /
Convergence ratio (1:3)
× 100-G uplink ports 2 = 5184
That is, 144 uplink ports and 432
and 48 × 25-G
downlink ports.
downlink ports per
switch).
S12516X-AF 2
288 / 2 = 144
Evaluation based on 36 × 100-G Convergence ratio
interface card: (1:3) 144 × 48 /
Convergence ratio (1:1) That is, 400:1200 (4 2 = 3456
× 100-G uplink ports
That is, 288 uplink ports and 288
and 48 × 25-G
downlink ports.
downlink ports per
switch).

• S-MLAG access
When the access switches adopt S-MLAG access, each access switch uses six or eight uplink
ports. Each port corresponds to one spine device.

9
 Table 7 The leaf devices use 10-G interfaces to connect to the servers and 100-G uplink
interfaces

Spine Number of Spine convergence Number of

Number of access
device spine ratio access
switches
model devices (uplink/downlink) servers
96
Convergence ratio (1:3) Convergence ratio (5:4)
That is, 600:480 (6 × 96 × 48 / 2 =
That is, 32 uplink ports 2304
and 96 downlink ports. 100-G uplink ports and 48
× 10-G downlink ports per
switch).
S9820-8C 6
64
Convergence ratio (1:1) Convergence ratio (5:4)
That is, 600:480 (6 × 64 × 48 / 2 =
That is, 64 uplink ports 1536
and 64 downlink ports. 100-G uplink ports and 48
× 10-G downlink ports per
switch).

Table 8 The leaf devices use 25-G interfaces to connect to the servers and 100-G uplink
interfaces

Spine Number of Spine convergence Number of

Number of access
device spine ratio access
switches
model devices (uplink/downlink) servers
96
Convergence ratio (1:3) Convergence ratio (2:3)
That is, 800:1200 (8 × 96 × 48 / 2 =
That is, 32 uplink ports 2304
and 96 downlink ports. 100-G uplink ports and 48
× 25-G downlink ports per
switch).
S9820-8C 8
64
Convergence ratio (1:1) Convergence ratio (2:3)
That is, 800:1200 (8 × 64 × 48 / 2 =
That is, 64 uplink ports 1536
and 64 downlink ports. 100-G uplink ports and 48
× 25-G downlink ports per
switch).

Multi-fabric expansion
You can add more fabrics to further expand the data center network. Connect the fabrics with EDs.

10
Figure 7 Single fabric network diagram

Fabric1
Internet VPN
FW1 FW2

Peer link
Border1 Border2

Spine1 Spine2

Peer link Peer link

Server- Server- Service- Service-
Leaf1 Leaf2 Leaf1 Leaf2

Backup

Server1 Server2 FW3 F5

Figure 8 Multiple fabrics connected with EDs

11
Recommended hardware
Device
Scenario Device model
role
• Type H modules for the S12500X-AF
switches
Medium-sized and large networks
Border/ED • All modules supported by the S12500G-AF
switches
Small-sized networks Same as the leaf devices

Type H modules for the S12500X-AF switches

More than 3000 servers
All modules supported by the S12500G-AF
switches
Spine
1000 to 3000 servers S9820-8C

Less than 1000 servers S68xx

• S6800
• S6860
• S6805
10-GE access • S6850-2C/S9850-4C with 10-GE interface
modules installed
• S6812/S6813
• S6880-48X8C
• S6825
• S6850-56HF
25-GE access • S6850-2C/S9820-4C with 25-GE interface
Leaf
modules installed
• S6880-48Y8C
• S6800
40-GE access • S6850-2C/S9850-4C with 40-GE interface
modules installed
• S9850-32H
• S6850-2C/S9850-4C with 100-GE interface
100-GE access modules installed
• S9820-8C
• S9820-64H

Access design
Server access design
Every two access devices form an M-LAG system of leaf devices to provide redundant and backup
access for servers.

12
Figure 9 Typical data center network

Internet

peer-link

Border1 Border2
L3GW L3GW

Spine1 Spine2

Leaf1 Leaf6
peer-link peer-link peer-link
Leaf2 Leaf3 Leaf4 Leaf5
L3GW L3GW L3GW L3GW L3GW L3GW

Virtualization Bare-metal Virtualization Container Legacy L4-L7

server server server network service

As a best practice, use one of the following server access solutions:

• A server accesses an M-LAG system of leaf devices in primary/backup mode. Physical
interfaces act as common trunk access interfaces, and are not assigned to the M-LAG group.
When the primary link is operating normally, the backup link does not receive or send packets.
When the primary link fails, a primary/backup link switchover occurs for the server, and ARP
and ND entries are updated to trigger remote route updates.
• A server accesses an M-LAG system of leaf devices in load sharing mode. An aggregate
interface acts as a trunk access interface, and is assigned to the M-LAG group. Both links of
the server operate at the same time.

13
 Figure 10 Server access methods

peer-link peer-link

Border1 Border2 Border1 Border2

Spine1 Spine2 Spine1 Spine2

peer-link peer-link
Leaf1 Leaf2 Leaf1 Leaf2

Du

e
al-

iv
ac

ct
tive

l-a
ua
Primary Backup Backup Primary

D
Eth0 Eth1 Eth0 Eth1 Eth0 Eth1 Eth0 Eth1

bond0 bond0 bond0 bond0

Server1 Server2 Server1 Server2

Sever access in primary/backup mode Sever access in load-sharing mode

Connecting border devices to external gateways

Border devices can be connected to external gateways by using the following methods:
• M-LAG topology
• Triangle topology
• Square topology

M-LAG topology
Network topology
As shown in Figure 11, two border devices form an M-LAG system, and two external gateways also
form an M-LAG system. The physical cables are cross-connected among border devices and
external gateways. Four links among border devices and external gateways are aggregated to form
a logical link through M-LAG. Because an external network specifies only one gateway IP for an
external gateway and the M-LAG network requires only one interconnect IP for one aggregate link,
this network topology is well compatible with the cloud network model. As a best practice, use the
M-LAG network if possible.

14
 Figure 11 M-LAG topology for border devices and external gateways

peer-link
External External
gateway 1 gateway 2

peer-link

Border1 Border2

Reliability
On the M-LAG network, you do not need to deploy failover links between border devices. The traffic
will not be interrupted only if one of the four physical links among the border devices and external
gateways is normal.
• When one link between Border1 and external gateways fails, LACP automatically excludes the
faulty link and switches traffic to the normal link, which is transparent for routing.
• When both links between Border1 and external gateways fail, M-LAG keeps the aggregate
interface up, and the traffic is switched to Border2 over the peer link. Then, the traffic is
forwarded to external gateways through the normal aggregate interface, which is transparent
for routing.
• When Border1 fails, the underlay routing protocol senses the failure. On the spine device, the
routing protocol withdraws the route with the next hop VTEP IP as Border1, and switches to
Border2 the traffic previously load-shared to Border1 in equal cost mode.
Benefits and restrictions
The external gateways must also support M-LAG. An M-LAG system supports only two member
devices.
Fewer IP addresses and VLAN resources are consumed. The M-LAG topology is well compatible
with the cloud network model.

Triangle topology
Network topology
As shown in Figure 12, two border devices form a multi-active device group, which supports more
than two devices. Border devices use four Layer 3 interfaces to connect to eternal gateways, and
the physical cables are cross-connected. The border devices and external gateways can
communicate through static routes and OSPF routes.
Figure 12 Triangle topology for border devices and external gateways

External External
gateway 1 gateway 2

Border1 Border2

15
Reliability
In the triangle topology, failover links between Border1 and Border2 are not required. The failover
links between border devices are needed only when all links between a border device and external
gateways fail.
• When one link between Border1 and external gateways fails, the corresponding static routes
are invalidated or the corresponding dynamic routes are withdrawn. Then, the traffic previously
load-shared to the faulty link in equal cost mode can be switched to the other normal link.
• When both links between Border1 and external gateways fail, failover links are needed
between border devices for the network to operate normally. In this case, the corresponding
static routes are invalidated or the corresponding dynamic routes are withdrawn. Then, the
traffic on Border1 can pass through the failover links to reach Border2, and then be forwarded
to external gateways through Border2.
• When Border1 fails, the underlay routing protocol senses the failure. On the spine device, the
routing protocol withdraws the route with the next hop VTEP IP as Border1, and switches to
Border2 the traffic previously load-shared to Border1 in equal cost mode. In this case, the
corresponding static routes are invalidated or the corresponding dynamic routes are withdrawn
on the external gateways. Then, the traffic in the north-to-south direction can be switched to
Border2.
Benefits
Two or more border devices can be active. More IP addresses and VLAN resources are consumed.

Square topology
Network topology
As shown in Figure 13, border devices use two Layer 3 interfaces to connect to external gateways.
You must deploy a failover link between the two border devices, and make sure the preference of a
failover route is lower than that of a route for normal forwarding. Typically, dynamic routing protocols
are deployed between two external gateways and two border devices to automatically generate
routes for normal forwarding and failover routes.
Figure 13 Square topology for border devices and external gateways

External External
gateway 1 gateway 2

Border1 Border2

Reliability
In the square topology, you must deploy a failover link between Border1 and Border2. When the
interconnect link between a border device and external gateways fails, the failover link is required
for service continuity.
• When the interconnect link between Border1 and external gateways fails, the corresponding
static routes are invalidated or the corresponding dynamic routes are withdrawn. Then, the
traffic on Border1 can pass through the failover link to reach Border2, and then be forwarded to
external gateways through Border2.
• When Border1 fails, the underlay routing protocol senses the failure. On the spine device, the
routing protocol withdraws the route with the next hop VTEP IP as Border1, and switches to
16
 Border2 the traffic previously load-shared to Border1 in equal cost mode. In this case, the
corresponding static routes are invalidated or the corresponding dynamic routes are withdrawn
on the external gateways. Then, the traffic in the north-to-south direction can be switched to
Border2.
Benefits
The square topology reduces the number of links between border devices and external gateways.
This topology is applicable when the border devices are far from the external gateways or hard to
connect to external gateways.

IP planning principles
On a single-fabric network, you must plan the spine-leaf interconnect interface addresses and
router ID addresses.
Spine-leaf interconnect interface IP addresses
As a best practice, configure the spine-leaf interconnect interfaces to borrow IP addresses from
loopback interfaces. On a Layer 3 Ethernet interface, execute the ip address unnumbered
interface LoopBack0 command to configure the interface to borrow an IP address from a
loopback interface.
Router ID address planning on an M-LAG network
For a routing protocol, two member devices of an M-LAG system are standalone and must be
configured with different router IDs. Please manually configure router IDs. If you do not do that, a
device will automatically select a router ID, which might cause router ID conflicts. In an
EVPN+M-LAG environment, as a best practice, configure the network as follows:
• Two member devices in the same M-LAG system use loopback0 interface addresses as the
local VTEP IP address and router ID of the M-LAG system, and cannot be configured with the
same address.
• Two member devices in the same M-LAG system use loopback1 interface addresses as the
virtual VTEP address (configured by using the evpn m-lag group command) and must be
configured with the same address.

Deployment modes
About the deployment modes
The underlay network contains spine and leaf nodes.
You can deploy the underlay network in the following modes:
• Manual deployment
• Automated deployment
Automated deployment can be implemented in the following ways:
• Automatic configuration deployment by the controller. You configure templates on the
controller and devices can be incorporated by the controller automatically when they start up
with factory default configuration. No underlay device provisioning is required.
• Automatic configuration (recommended in scenarios without a controller deployed)
a. The administrator saves the software image files (including startup image files and patch
packages) and configuration files for the device to an HTTP, TFTP, or FTP server. The
software image files and configuration files for the device can be identified by the SN of the
device.

17
 b. The device obtains the IP address of the TFTP server through DHCP at startup and then
obtains the script for automatic configuration.
c. The device runs the script and obtains device information (for example, SN) and matches
the software image file and configuration file for the device.
d. The device downloads the software image file and configuration file from the file server,
and loads the image and deploys configuration automatically.
Output from a Python script for automatic configuration enables you to monitor the operations
during execution of that script, facilitating fault location and troubleshooting.
With the automatic configuration feature, the device can automatically obtain a set of
configuration settings at startup. This feature simplifies network configuration and
maintenance.

Management network configuration

In a data center network, a management switch connects to the management network of each
device.
Use Layer 2 or Layer 3 networking mode for the management network in the single-fabric scenario.
Use Layer 3 networking mode for the management network in the multi-fabric scenario. In Layer 3
networking mode, you must assign different VLANs to different fabrics, and manual configure a
gateway and DHCP relay. As a best practice for future expansion of a fabric, use Layer 3
networking mode in the single-fabric scenario.
Figure 14 shows the management network in Layer 3 networking mode in the multi-fabric scenario.
Figure 14 Management network
Component

SeerEngine-DC Links

Mgmt network (VLAN 10)

VLAN 10
Mgmt network (VLAN 20)
Mgmt switch
DHCP relay Mgmt network (VLAN 30)
VLAN 20 VLAN 30

Peer link Peer link

Border 1 Border 2 Border 1 Border 2

Spine 2
Spine 1 Spine 1 Spine 2

Peer link Peer link Peer link Peer link

Leaf 1 Leaf 2 Leaf 3 Leaf 4 Leaf 1 Leaf 2 Leaf 3 Leaf 4

Server 1 Server 2 Server 3 Server 4 Server 1 Server 2 Server 3 Server 4

Fabric 1 Fabric 2

18