Open Network Linux

A Programmer’s View

ROB SHERWOOD, JEFF TOWNSEND, STEVE NOBLE

BIG SWITCH NETWORKS
 AGENDA / OUTLINE

•  What is Open Network Linux? What is it good for?

•  ONL source code and build system overview

•  Writing to ONL: ONLP and ONLSwitch APIs

•  General discussion
–  Pro's and Con's of different hardware platforms
–  Other discussion

2 © 2015 BIG SWITCH NETWORKS, INC.

 ONL: OPEN NETWORK LINUX
A Linux Distribution for Bare Metal Switches

•  Reference NOS for the Open Compute Project (OCP)

–  Collection of software packages, utilities, drivers, and abstractions
to run on OCP, bare metal, “brite box” hardware
–  i.e., a “NOS” that ONIE would install

•  OCP Networking Community/Ecosystem

http://opennetlinux.org
3 © 2015 BIG SWITCH NETWORKS, INC.
 ONL: EXAMPLE USE-CASES

•  Reference hardware testing platform, e.g., OCP Certification

–  Common core in open source for verification, reference

–  Shared platform code ensures consistency and test coverage in areas like cable/optics compatibility,
operational statistics, front panel LED behavior, SNMP behavior, environmental sensors, fan drivers, etc.

•  DIY packet forwarding platform, e.g., for academic research

–  Platform driver code too complicated for most hobbyists
–  Pluggable forwarding agents enable innovation in dataplane features and network designs

•  Building Block for Commercial or Production-grade Software

–  Pull ONL in to a larger, commercially supported solution – e.g. Big Switch products
–  Interest in using ONL as the basis for web-scale OS’es – e.g. NTT Labs / Facebook

4 © 2015 BIG SWITCH NETWORKS, INC.

 BIG SWITCH PORTFOLIO – OPEN SDN FABRICS
BIG MONITORING BIG CLOUD
FABRIC FABRIC
CONTROLLER CONTROLLER

1G/10G/40G 10G/40G
(Trident-II)
Alternative to NPB:
SWITCH LIGHT™ OS
ONIE Installer Alternative to ACI:

Big Switch uses ONL as the base for Switch Light™ OS

5 © 2015 BIG SWITCH NETWORKS, INC.
 WHY OPEN NETWORK LINUX?

•  Why not use an existing Linux distribution?

–  Switches are very similar to servers, but not quite
–  Does build on existing distribution – Debian Wheezy, Jessie
–  Need to create ONIE installers for many platforms
–  Need to manage switch-specific hardware (e.g.,SFPs)

•  Distribution or ‘A La Carte’?
–  ONL supports both
–  This talk focuses on the ‘A La Carte’ pieces of ONL

6 © 2015 BIG SWITCH NETWORKS, INC.

 ONL: SUPPORTED HARDWARE
http://opennetlinux.org/hcl

•  ONL Supports 25+ Switches

–  x86, PowerPC, and ARM Architectures
–  1g, 10g, 40g, and 100g Switches
–  Many Open Compute Project switches
–  Many non-OCP switches

•  Platform Ports in Progress

–  Freescale T2080RDB
–  Accton AS4610 (cost optimized)
–  Facebook Wedge 100
–  Celestica Redstone XP
–  Accton AS7512 w/ Cavium processor

Updated: October 2015

7 © 2015 BIG SWITCH NETWORKS, INC.
 ONL: THREE+ FORWARDING AGENTS; 20+ PLATFORMS
ADDITIONAL / FUTURE APPLICATIONS HERE

Indigo
Microsoft Facebook
OpenFlow
SONiC FBOSS
Agent
FORWARDING AGENT PROGRAMMING APIS

SAI
Interface
OF-DPA
Open Network Linux Platform Abstraction Layer Open NSL
PLATFORM

(Platform specific drivers including Optics)

Platform
Driver Platform Specific ASIC Drivers
Installer
(ONIE)
ONL Linux Kernel Broadcom SDK
(Includes extra drivers: I2C, MUX, management Ethernet, …) (others coming soon)

CPU Miscellaneous Hardware Packet Forwarding Chip

HARDWARE

(PowerPC, x86) (Fans, LED controllers, SFP sensors, Power…) (ASIC)

OCP Switch Hardware

(Facebook Wedge, IM Niagara, Accton 6712, 7512, Dell S6000-ON, Quanta LY6, ...)

8 © 2015 BIG SWITCH NETWORKS, INC.

 AGENDA / OUTLINE

•  What is Open Network Linux? What is it good for?

•  ONL source code and build system overview

•  Writing to ONL: ONLP and ONLSwitch APIs

•  General discussion
–  Pro's and Con's of different hardware platforms
–  Other discussion

9 © 2015 BIG SWITCH NETWORKS, INC.

 ONL: BUILD SYSTEM UNDER THE HOOD
Anatomy of `make onl-$arch`

Linux 3.2.65 SWI: Installer:

Kernel Patches
`make –C $ONL/builds/ `make –C $ONL/builds/
Linux 3.9.6 $arch/swi` $arch/installer`
Architecture ONL-Specific
Infrastructure Example Agents .debs
(SONiC, FBOSS, Indigo)
Module (AIM) $ONL/REPO/$arch
Switch Image
FaultD
(.swi)
Open
BigCode
Libraries Platform-* Network
(VPI, uCLI, ELS) (installers, etc.)
Generic .debs Linux
(Emdebian.org &
Specific Components:
ONLP/ONLSwitch
(Platform API Lib) Debian.org) Installer Script +
+
`make –C $ONL/packages/base/ via SHAR Wrapper ONIE
$arch/$component deb` OpenNSL/OFDPA apt-cacher-ng
Installer
ONLP-SNMP
Busybox
Buildroot ONL Loader ONL Loader
Etc.
New from ONL 3rd Party OSS Combination

10 © 2015 BIG SWITCH NETWORKS, INC.

 ONL BUILD LAYOUT

•  cd $ONL && `make docker`

–  Use a docker image ($ONL/docker/images/*) to fix build
environment, e.g., cross compiles, libraries, etc.
–  Checks out $ONL/sm/infra (for AIM) and $ONL/sm/bigcode
–  Builds everything at once
•  Interactive building:
–  cd $ONL
–  ./docker/tools/onlbuilder # add “-8” for jessie, defaults to wheezy
–  . setup.env # source the $ONL/setup.env file into your shell
•  ./RELEASE/$suite/$arch/ONL-*.{installer, swi}
–  Resulting images on a successful build

11 © 2015 BIG SWITCH NETWORKS, INC.

 ONL CODE LAYOUT

•  $ONL/packages/base/$arch
–  $arch = ‘any’ à “common” across multiple architectures
–  $arch = ‘all’à really no arch dependence (e.g., python code)
•  $ONL/packages/platforms/$vendor/$arch/$ONIE_ID
–  packages/platforms/accton/x86-64/x86-64-accton-as7712-32x
–  ./onlp – the ONLP sub-driver for this platform
–  ./platform-config/r$version/ – installation, booting, initialization
–  More info in $ONL/docs/PortingGuide.md
•  $ONL/packages/platforms-closed
–  For switch vendors who have not yet open sourced their code
–  E.g., Dell S4048-ON, Dell S6000-ON
12 © 2015 BIG SWITCH NETWORKS, INC.
 ONL DISTRIBUTION INFORMATION

•  Default root fs contents:

–  $ONL/builds/any/$suite/common/*-packages.yml
•  $ONL/tools
–  onlpm -- used by build system to build packages, extract deps
–  onlrfs -- used to create root file system for distro from .yml file
•  $ONL/make
–  Many useful Makefile functions: kernels, .deb pkgs, etc.
–  e.g., set RFS_CONFIG=your-pkgs.yml, include $ONL/make/rfs.mk
•  $ONL/REPO/$suite/packages/$arch à resulting .deb files
–  Great for ‘a la carte’ usage

©2015 BIG SWITCH NETWORKS, INC. WWW.BIGSWITCH.COM PROPRIETARY AND

13
CONFIDENTIAL
 UNDERSTANDING JTABS

•  Much of our code has a structured format – affectionately “JTABS”

–  $package/src/module/{auto,src,inc/$package}
•  $package/src/module/auto/$package.yml
–  Auto-generate code (parameters, enumerations, etc.) for this module
–  Inline auto-gen code into files via taglets, e.g., /* <auto.start.VAR(PARM) > */
•  $package/src/module/src
–  $package_ucli.c: mixin for uCLI command line options
–  $package_log.c: mixin for AIM logging – per-module log levels, etc.
–  $package_config.c: auto-gen, defaults for compile time parameters
•  $package/src/utest
–  unit tests for this module
•  $(DEPENDMODULES) + dependmodules.mk
–  Makefile variable to pull lists of modules together into a single package

©2015 BIG SWITCH NETWORKS, INC. WWW.BIGSWITCH.COM PROPRIETARY AND

14
CONFIDENTIAL
 STRATEGIES FOR ROLLING YOUR OWN DISTRO

1.  Add the ONL repo as a submodule to your existing code

2.  Create a your-pkgs.yml file with custom package lists
–  Mix and match with existing ONL packages as desired
–  Optionally transitively include existing lists, e.g., $arch-common.yml
3.  Call the $ONL/tools/onlrfs.py tool to make the root file system
4.  Convert the root file system into an ONIE shar (using ONL tools)
5.  Possibly re-use existing Makefile magic from $ONL/make

Result: Custom ONIE installer - your-custom-$version.installer

©2015 BIG SWITCH NETWORKS, INC. WWW.BIGSWITCH.COM PROPRIETARY AND
15
CONFIDENTIAL
INSTALL USING ONIE THEN BOOT ONL

Boot Logic: ~2GB

ONL
1.  uBoot/grub POSTs Loader
2.  $nos_boot_cmd is read from ENVs
ONL
3.  run $nos_boot_cmd SWI #1
~64MB •  If $nos_boot_cmd returns, run ONIE (cached)

uBoot/grub •  On install, ONIE sets

ENVs $nos_boot_cmd to load ONL loader ONL
SWI #2
ONIE 4.  Loader downloads specified SWI URL (cached)
Free if not cached
Space 5.  Loader mounts rootfs as ramdisk with
Boot overlayfs Mass
Flash 6.  ONL loader kexec’s SWI kernel Storage
 AGENDA / OUTLINE

•  What is Open Network Linux? What is it good for?

•  ONL source code and build system overview

•  Writing to ONL: ONLP and ONLSwitch APIs

•  General discussion
–  Pro's and Con's of different hardware platforms
–  Other discussion

17 © 2015 BIG SWITCH NETWORKS, INC.

 LIBONLP.SO: PLATFORM ABSTRACTION LAYER

•  Manages platform peripherals, FRUs, etc.

–  e.g., SFP’s, Fans, Temp Sensors, Power supplies, system details
–  Not any of the Broadcom chip specifics (that’s ONLSwitch.so)
•  Full “application facing” API in
–  $ONL/packages/base/any/onlp/src/onlp/module/inc/ONLP/*.h
–  e.g., int onlp_sfp_eeprom_read(int port, uint8_t** rv);
–  e.g., int onlp_fan_info_get(onlp_oid_t id, onlp_fan_info_t* rv);
•  onlp_init():
–  Required before any other onlp calls
•  onlp_sys_platform_manage_start(int block)
–  Starts thread to manage fans, FRU insertion, etc.

18 © 2015 BIG SWITCH NETWORKS, INC.

 LIBONLP.SO IMPLEMENTATION
Platform Abstraction Layer

•  /lib/$platform/libonlp.so depends on:

–  /lib/$platform/libonlp-platform.so # platform specific driver
–  /lib/$platfrom/libonlp-defaults.so # default implementations
•  libonlp-platform.so is a boot-time symlink to
–  /lib/platform-config/$platform/onl/lib/libonlp-$platform.so
•  Writing an ONLP driver for a new platform is easy:
–  Driver-side libonlp API is in onlp/inc/ONLP/platformi/*.h
–  Driver can be user-space or kernel-space
–  For more information, see ./docs/PortingGuide.md

19 © 2015 BIG SWITCH NETWORKS, INC.

 LIBONLS.SO: LOW-LEVEL DATAPLANE ABSTRACTION LAYER

•  Runs on top of SDK; Provides add-on APIs for your app

•  Handles all of the ASIC/SDK initialization
–  Equivalent to config.bcm and rc.soc for all supported platforms
–  Lane mappings, warp core programming, PHY+power settings, etc.
•  Provides a unified Port Lifecycle/Management API
–  Operational and Admin state: “link down” versus “admin down”
–  SFF state management: automatic PHY program on SFF insertion
–  Standardized capabilities, capacities, break-out + 1G support
–  LED programming by port state

20 © 2015 BIG SWITCH NETWORKS, INC.

 LIBONLS.SO: PORT MANAGEMENT ABSTRACTIONS

•  All port management through serialized event manager

–  Avoids concurrency issues
–  Decouples forwarding logic from platform-level details
•  “Switch” Port versus “Platform” port abstraction
–  1:1 unless breakout cables
–  “Switch” port: logical switching interface
–  “Platform” port: physical port interface
•  Application registers for and posts port events to ONLS
–  Status: Link up/down, SFF insert/removal, breakout, auto-neg, etc.
–  Operational: Admin up/down, disable auto-neg, etc.

21 © 2015 BIG SWITCH NETWORKS, INC.

 LIBONLS.SO: EXAMPLE API SEQUENCE

•  Initialize platform:
–  int onls_init();
–  Same for all platforms

•  Register for port events:

–  int onls_port_event_register(brcm_port_event_handler_f, void* cookie);

•  Post event to ONLS (e.g., ADMIN up a port):

–  ONLS_SWITCH_PORT_EVENT(port, ADMIN_ENABLE);

•  Full list of implemented port events in Appendix

22 © 2015 BIG SWITCH NETWORKS, INC.

 AGENDA / OUTLINE

•  What is Open Network Linux? What is it good for?

•  ONL source code and build system overview

•  Writing to ONL: ONLP and ONLSwitch APIs

•  General discussion
–  Pro's and Con's of different hardware platforms
–  Other discussion

23 © 2015 BIG SWITCH NETWORKS, INC.

 ONL BRINGS EXPANDED AND UNIFORM PLATFORM SUPPORT
ONL will bring OF-DPA and OpenNSL to 12+ new platforms

Switch OF-DPA OpenNSL ONL ONL also adds

Support? Support? Support? support for:
Accton AS4600,
4610, 5600, 5610, •  Break-out cables
5710, 5712, 6700,
6712, 7712 •  1G optics
Accton Wedge
•  Dynamic re-
Celestica
Redstone XP
cabling
IM Niagara
(breakout to
standard)
Quanta LB9, LY2,
LY2R, LY6, LY8
•  Additional
Dell S4048, S4810,
S6000, Z9100
cables, optics

24 Partial; Full Support in Progress ©2016 BIG SWITCH NETWORKS, INC. WWW.BIGSWITCH.COM PROPRIETARY AND
CONFIDENTIAL
 DISCUSSION TOPICS

•  Roll your own SDK or use our’s?

•  OpenNSL versus the direct SDK?
•  First target project?
•  Is our Indigo top-half of value?
•  Interaction model?
–  Is the public mailing list enough?
•  Testing and support model?
•  Tour of the data center downstairs for testing

25 © 2015 BIG SWITCH NETWORKS, INC.

CONCLUSION AND HOW TO GET INVOLVED

•  Join the mailing list and provide feedback

–  Compile the code
–  Try a new platform
–  “Feedback is fine, but patches are preferred” J
•  Growing support for OCP and non-OCP switches
•  Forwarding Agents: Fboss, ORC, OpenFlow and more
•  Find out more at http://opennetlinux.org
•  Documentation/Videos
•  Tutorials: http://opennetlinux.org/docs/routingtutorial
•  Pre-compiled binaries
27 © 2015 BIG SWITCH NETWORKS, INC.
 LIBONLS.SO: LIST OF PORT EVENT

•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_CONFIGURE, •  ONLS_SWITCH_PORT_EVENT(port, ADMIN_ENABLE) •  ONLS_PORT_EVENT_SWITCH_PORT_ACTION_LINK_FAILED,

•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_ADMIN_ENABLE, •  ONLS_PORT_EVENT_PLATFORM_PORT_AUTO_BREAKOUT_ENABLED, •  ONLS_PORT_EVENT_SWITCH_PORT_LINK_FAILED,

•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_AUTO_BREAKOUT_DISABLE,
•  ONLS_PORT_EVENT_PLATFORM_PORT_ADMIN_ENABLED, •  ONLS_PORT_EVENT_SWITCH_PORT_ACTION_AUTONEG_ENABLE,
•  ONLS_PORT_EVENT_PLATFORM_PORT_AUTO_BREAKOUT_DISABLED,
•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_ADMIN_DISABLE, •  ONLS_PORT_EVENT_SWITCH_PORT_AUTONEG_ENABLED,
•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_SFP_FORCE_ABSENT_ENABLE,
•  ONLS_PORT_EVENT_PLATFORM_PORT_ADMIN_DISABLED, •  ONLS_PORT_EVENT_SWITCH_PORT_ACTION_AUTONEG_DISABLE
•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_SFP_FORCE_ABSENT_DISABLE,
•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_SFP_INSERT, •  ONLS_PORT_EVENT_SWITCH_PORT_AUTONEG_DISABLED,
•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_FORCE_ONLINE_ENABLE,
•  ONLS_PORT_EVENT_PLATFORM_PORT_SFP_INSERTED, •  ONLS_PORT_EVENT_SWITCH_PORT_ACTION_FORCE_ONLINE_ENABLE,
•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_FORCE_ONLINE_DISABLE,
•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_SFP_REMOVE, •  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_SFP_ALWAYS_ENABLED_ENABLE, •  ONLS_PORT_EVENT_SWITCH_PORT_ACTION_FORCE_ONLINE_DISABLE

•  ONLS_PORT_EVENT_PLATFORM_PORT_SFP_REMOVED, •  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_SFP_ALWAYS_ENABLED_DISABLE,

•  ONLS_PORT_EVENT_PLATFORM_PORT_SFP_IDENTIFIED, •  ONLS_PORT_EVENT_SWITCH_PORT_ACTION_ADD

•  ONLS_PORT_EVENT_SWITCH_PORT_ACTION_REMOVE,
•  ONLS_PORT_EVENT_PLATFORM_PORT_SFP_SUPPORTED,
•  ONLS_PORT_EVENT_SWITCH_PORT_REMOVED,
•  ONLS_PORT_EVENT_PLATFORM_PORT_SFP_UNSUPPORTED,
•  ONLS_PORT_EVENT_SWITCH_PORT_ACTION_CONFIGURE,
•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_SFP_QUALIFY,
•  ONLS_PORT_EVENT_SWITCH_PORT_ACTION_ADMIN_ENABLE,
•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_SFP_UNKNOWN,
•  ONLS_PORT_EVENT_SWITCH_PORT_ADMIN_ENABLED,
•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_POLL,
•  ONLS_PORT_EVENT_SWITCH_PORT_ACTION_ADMIN_DISABLE,
•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_AUTO_BREAKOUT_CONFIGURE, •  ONLS_PORT_EVENT_SWITCH_PORT_ADMIN_DISABLED,
•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_BREAKOUT_MODE_1X40, •  ONLS_PORT_EVENT_SWITCH_PORT_ACTION_PLATFORM_ENABLE,

•  ONLS_PORT_EVENT_PLATFORM_PORT_MODE_1X40_ENABLED, •  ONLS_PORT_EVENT_SWITCH_PORT_ACTION_PLATFORM_DISABLE,

•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_BREAKOUT_MODE_4X10, •  ONLS_PORT_EVENT_SWITCH_PORT_ACTION_LINKSCAN

•  ONLS_PORT_EVENT_SWITCH_PORT_ACTION_LINK_UP,
•  ONLS_PORT_EVENT_PLATFORM_PORT_MODE_4X10_ENABLED,
•  ONLS_PORT_EVENT_SWITCH_PORT_LINK_UP,
•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_BREAKOUT_MODE_1X100,
•  ONLS_PORT_EVENT_SWITCH_PORT_ACTION_LINK_DOWN,
•  ONLS_PORT_EVENT_PLATFORM_PORT_MODE_1X100_ENABLED,
•  ONLS_PORT_EVENT_SWITCH_PORT_LINK_DOWN,
•  ONLS_PORT_EVENT_PLATFORM_PORT_ACTION_AUTO_BREAKOUT_ENABLE,
•  ONLS_PORT_EVENT_SWITCH_PORT_ACTION_LINK_REMOTE_FAULT,
•  ONLS_PORT_EVENT_SWITCH_PORT_REMOTE_FAULT,

28 © 2015 BIG SWITCH NETWORKS, INC.

 ONL: EXAMPLE INDUSTRY USE CASES

Project Atrium with Indigo OpenFlow Agent

FBOSS – Built on Broadcom’s OpenNSL Library

L3 Routing using GoBGP + ORC

Details in following slides.

29 © 2015 BIG SWITCH NETWORKS, INC.
 ONL/OpenFlow on Project Atrium
Quagga BGP
vlan
vlan x y Peering Application
E-BG
E-BGP P ONOS
vlan z OFDPA Driver
OpenFlow 1.3
Indigo OF Agent
OCP
Software OF-DPA API
- OF-DPA
ONL
ONIE
BRCM SDK API
BRCM ASIC

OCP Bare Metal Hardware

OCP: Open Compute Project; ONL: Open Network Linux; ONIE: Open Network Install Env;
BRCM: Broadcom Merchant Silicon ASICs; OF-DPA: OpenFlow Datapath AbstracPon
 FACEBOOK FBOSS SUPPORT ONL IN ALPHA

Demo From
FBOSS
OCP
40G OpenNSL
Workshop
Breakout ONL
10/9 in
Cable
Wedge Switch
Boston
1 1
1 2 40G 5 6

Interfaces
Vlan Vlan
#1 #2 10G
SFP+
Cables

Hosts 1 2 3 4
NTT’S EVPN USE CASE
ospfd GoBGP
zapi
ORC
zebra driver
netlink
Linux Networking
OpenNSL
Subsystem
ONL

ONIE

Accton’s Edgecore 5712-54x

Copyright©2015 NTT Corp. All Rights Reserved. 32

NTT GOBGP demo – orc running
NTT GOBGP DEMO – GOBGP RUNNING
NTT GOBGP DEMO – TCPDUMP
NTT GOBGP demo – add routes
NTT GOBGP demo – Iperf
NTT GOBGP demo – 9.41 Gbits/sec
 Early Adoption: REANNZ

•  Research and Education Advanced Network New Zealand

–  R&E WAN Provider for NZ’s University Systems
•  REANNZ is currently trialing OF-DPA on Open Network Linux
–  Previously trialed many SDN platforms
–  CARDIGAN, Front Line Assembly and Treehouse
•  “A Linux for Networking”
•  REANNZ is working on new SDN deployment options

39 ©2015 BIG SWITCH NETWORKS, INC. WWW.BIGSWITCH.COM PROPRIETARY AND CONFIDENTIAL

 ONL Contributors/Supporters (So Far)

Adds
Forwarding
Agents

Open
Network
Linux

Provides
Platform
Drivers
40 © 2015 BIG SWITCH NETWORKS, INC.
 Early Adoption: REANNZ

•  Research and Education Advanced Network New Zealand

–  R&E WAN Provider for NZ’s University Systems
•  REANNZ is currently trialing OF-DPA on Open Network Linux
–  Previously trialed many SDN platforms
–  CARDIGAN, Front Line Assembly and Treehouse
•  “A Linux for Networking”
•  REANNZ is working on new SDN deployment options

41 ©2015 BIG SWITCH NETWORKS, INC. WWW.BIGSWITCH.COM PROPRIETARY AND CONFIDENTIAL

OPEN NETWORK INSTALL ENVIRONMENT (ONIE)
•  Open source project to install/uninstall network OS
•  hYp://github.com/onie/onie or hYp://onie.github.io/onie/
•  Think of it like a hybrid PC BIOS and Grub/LILO/Sysimage
•  Co-operaTve project: OCP, Cumulus, Big Switch, Others
•  In pracPce: Curt Brune from Cumulus Networks does almost all of
the work
•  Allows a network admin to install/uninstall a network OS
•  In pracPce, it is itself a ~4MB mini-Linux installaPon

©2014 BIG SWITCH NETWORKS, INC. WWW.BIGSWITCH.COM 42

 Traditional: ONL + Netlink w/Open Route Cache

Quagga/Xorp/Etc. This API

looks a lot
like SAI
S Open Route Cache
Applications SensorD ONLP N
Dump M ORC Driver API
P Orc-brcm Driver
ONLP APIs
ONL Platform APIs
BRCM
Platform SDK Closed
ONL Distribution (Linux + stuff)
Source
CPU Misc Hardware
BRCM
Hardware (x86,PPC) (fans, LEDs, SFP, sensors)

Per Switch Stack

Basic L3 Routing With Open Route Cache (ORC)
1.  ORC creates a orcXX interface for each port
Quagga/Xorp/Etc.
2.  Quagga (or Xorp, etc.) installs routes into Linux
kernel
Open Route Cache
3.  Linux kernel sends RT Netlink Updates to ORC
Orc-brcm Driver
4.  ORC translates to SDK calls
•  Hardware accelerated routing!
BRCM
•  Only unicast IPv4 now SDK
•  No IPv6, Multicast, ECMP support

5.  Linux kernel is slow path to ASIC’s fast path

BRCM
 Lots of open source controllers
• But what are they controlling?
Open
Source
SDN
Controllers

???
This talk

SDN OSS Software Switches SDN OSS Hardware Switches

 ONL w/ OpenFlow Agent
OpenFlow Controller
OpenFlow Protocol:
v1.0-v1.3

S Indigo OpenFlow Agent

Applications SensorD ONLP N
Indigo API
Dump M
P Indigo OF-DPA Driver

ONL Platform APIs OF-DPA Driver Platform/

Platform BRCM ASIC s
ONL Distribution (Linux + stuff) SDK
Closed
CPU Misc Hardware Source
BRCM
Hardware (x86,PPC) (fans, LEDs, SFP, sensors)

Per Switch Stack

OpenFlow Datapath AbtracPon: OF-DPA

Indigo
open
driver OF-DPA API
OF-DPA
SDK closed
Chip
SWITCH SCALING: UNDERSTANDING TABLES
Classical OpenFlow versus Modern: ProacPve and MulP-table OpenFlow
Processing Pipeline
Early OpenFlow Modern OpenFlow
implementaPons only used implementaPon leverages
the most flexible table. all tables
•  Didn’t scale well •  Scales like tradiPonal
L2 L3 •  Scale forced reacPve •  Allows proacPve
Table: Table: populaPon populaPon
~100K+ ~100K+ •  ReacPve caused •  ProacPve reduces
Entries Entries controller load and controller load and
addiPon scale problems allows for headless
control

VLAN ACL Table:

Egress Table
Table ~2k Entries

OpenFlow does NOT imply Flow-based Networking

©2014 BIG SWITCH NETWORKS, INC. WWW.BIGSWITCH.COM 48 48
DISCUSSION: SAI VS. OPENNSL VS. OFDPA

•  Short story: this is all evolving

•  Closed versus Open Source
•  SAI gets you chip vendor independence (in theory)
–  Broadcom, Mellanox, Cadvium, Barefoot (via software)
•  OpenNSL is closest to existing BCM APIs
•  OF-DPA exposes other functionality like OAM and MPLS
•  There is a move to merge these APIs
–  e.g., SAI implemented on OpenNSL, OF-DPA implemented on
OpenNSL
Thank You
HTTP://OPENNETLINUX.ORG