Network Automation Journey, A systems engineer NetOps perspective

Network Automation Journey, A systems engineer NetOps perspective

• 1.
  Network Automation Journey Asystems engineer NetOps perspective Walid Shaari @walidshaari https://www.linkedin.com/in/walidshaari FOSDEM 2018 4th February 2018 Brussels background image credit: https://commons.wikimedia.org/wiki/File:Social_Network_Analysis_Visualization.png
• 2.
  Walid Shaari @walidshaari https://www.linkedin.com/in/walidshaari ● Systemengineer supporting HPC Linux clusters ● Configuration management evaluation and deployment project in 2014 ● Advocating open source, automation, containers and Kubernetes ● Husband and father of 3 lovely kids ● Last 3 months in short work assignment with Network management team
• 3.
  Incentives
• 4.
  • Open sourceand standards • Pure Layer 3 network implementation • Lightweight IP to IP encapsulation • Policy based secure networking • Scalable and simple • Scale out SDN controller • Openstack , Docker, Kubernetes, Mesos and CNI
• 5.
  “In 2018, wewill see more demands placed on the continuous delivery of changes to networking setups due to pressure from containerisation, distributed systems, and security needs. Thus, networking must become as flexible and automation-friendly as the software that runs over it, and become less of a bottleneck.” https://www.itproportal.com/features/what-do-organisations-need-to-prepare-for-in-2018/ Nigel Kersten, Chief Technical Strategist at Puppet
• 6.
  How to buildan event driven, dynamically reconfigurable microservices platform by Sven Beauprez: https://youtu.be/1D8hyLWMtfM
• 8.
  Team: handful network engineerssupported by handful infrastructure & cabling Network: Campus several building and Data Centers Medium scale less than 200 switches heterogeneous platforms and models: existing tooling in place CMDB IPAM NMS : SNMP based setup: Traditional Three tier architecture “Core-Distribution-Access” with extended layer 2 Frequent operational activities: Changes Troubleshooting New applications constraints
• 9.
  Enterprise Network managementtrends CLI Cut & Paste NPA Notepad Automation Frameworks & Controllers ● Excel ● Python jinja2 ● Templating engines ● Ansible, ● Puppet ● Chef Event Driven Automation ● Sensor triggered events ● napalm_logs ● Salt ● IFTTT ○ StackStorm Ansible AWX Intent Based Networking Declarative Network Intent Composition Aspen Boulder
• 10.
  Enterprise Network management •Manual • Cut & Paste • Serial • Inconsistent •Error prone 10https://www.slideshare.net/opennetsummit/ons2013-guido-appenzellerbig-switch-networks
• 12.
  Why we arenot automating? • Just a fad: All these are is vendor driven buzzwords, exaggerated, will fade away • Not relevant to our setup, • Mixed diverse vendor and legacy platform environment, no solution can handle this, it will be cost prohibitive. • Do not need a bazooka to handle a mosquito. We are small • Busy, shortage of resources • None of my acquaintances in the industry is doing it. • It is hard, steep learning curve. we need to learn a lot of new things • Vendor advanced training e.g. CCIE teaches us differently • This will affect our job security, automate us out of the job • Automation pushes mistakes faster, can bring everything down • Do not know where to start? 12
• 13.
  Example of in-houseVisual Basic Template App 13
• 14.
  Kickstart: Brainstorm ● Whatis NetOps, NetDevOps? ● What problems we are trying to solve? ● How much visibility into operations we have? ● What are your responsibilities? ● Tasks/processes you are responsible for ? ● how much time you can set aside to look into automation? ● What current automations already in place? (processes or tools) ● Why would you like to have automations processes/tools? ● What automations you are aware of, or would like to have? processes/tools ● sharing medium, how should we enhance activity communications? ( knowledge, process, updates, documentation, in-job training) ● How should we start?
• 15.
  Pressing Needs: - Scale:increasing environment size, scale and diversity - Optimization: more to do with less engineers - Failure management: solve new problems, guarantee level of performance and consistency. Opportunities: - 4IR: fourth Industrial revolution - IoT - Big Data - Digital transformations and cloud initiatives - Artificial intelligence and machine learning - Services “e.g. microservices” demand growing infrastructure and hence network scale
• 16.
  What to Automate? https://docs.google.com/forms/d/e/1FAIpQLSdiBNMK0ZUmgBSNEaOWa-YHGQ4AlZo7EhB52_dXzvMqic3eHA/viewanalytics
• 17.
  Top traditional networkissues Spanning Tree misconfigurations VLAN(s) misconfiguration and more
• 18.
  Priorities 18 ImplementationRisk Complexity Perceived Value NTP DNS SNMP syslog reporting baseline validation Don’tpursue Pursue in Time Best efforts Low hanging fruits Security policy vlan configurations backup software qualifications
• 19.
  What Not toAutomate? https://imgs.xkcd.com/comics/automation.png
• 20.
  Ansible CFEngine Chef Foreman DevOps Day Kubernetes Terraform sysdig consul Puppet Git CapistranoScripting Vagrant Rundeck Salt Rudder Fabric Docker etcd cfgmgmtcamp Stackstorm Packer ELKstack mgmt Habitat Envoy Openfaas Puppet Bolt Istio Brigade Nomad Helm
• 21.
  The Road toSDN: https://queue.acm.org/detail.cfm?id=2560327
• 22.
  White Box Pica8 batfish ONIE Trigger RANCID NetFlow ssh OpenWRT JunosStdlib Junos Ansible Junos Chef Facebook Wedge gobgpsflow SNMP NetConf Ethernet Fabric (CLOS)** SDN Junos Puppet Cumulus Networks OpenDayLight Apstra AOS Ansible 2.0 SaltStack Snaproute SONIC Intention Based Networks FaceBook Open/RNetworktoCode ntc N.A.P.A.L.M. OpenConfig * Dates may be inaccurate. they were collected from initial release of standard, commit, or project info and other talks ** Research has roots back from 1953 Yang 2010 monitoring managment container validation device hardware topology RMON 2000
• 23.
  vrnetlab
• 24.
  Technology: Toolset https://interestingtraffic.nl/2017/03/27/insights-from-the-netdevops-fall-2016-survey/
• 25.
  Ansible! however, From thebook “Automating Junos Administration” by Jonathan Looney and Stacy Smith: “Ansible configurations can grow to become somewhat complex. There are multiple files for inventory, variables, playbooks, and roles. Like with any critical system, it’s a good idea to keep all of these files under a revision control system such as Git. You may also want to couple revision control with a review and testing process to ensure any changes to the Ansible configurations are thoroughly verified before applying them to a production network.” In other Words • Collaboration • Version control • How to manage scale and growth • Testing 25
• 26.
  Small wins • Dowhat you do everyday However try to improve one thing a time Don’t try to learn more than one thing at a time Automation will not stand in your way • Think and do simple: Start simple use cases Stay simple handling generic cases
• 27.
  The Modern CLI ●syntax highlighting ● Validation, linting, indentation ● the Automation UX Automation Platform ● Configuration management ● Orchestration ● Role Based Access ● Scheduling ● Remote Execution ● Event based triggers Version Control System ● History tracking ● Peer review ● Collaboration Engine ● Live Documentation ● Integration with issue tracker 03 01 02
• 28.
  Footer Text 28
• 29.
  Recommendations ●Favor open solutionsover proprietary ●Gain yearly saving of over 25% in 2023 ●Invest back the savings into the people ●PP-RPC ●Create cross functional assignments https://www.gartner.com/doc/3446727/time-shift-network-spend-premium
• 30.
  Think ahead Devices, processes,and automation • Standardize network elements as much as possible • Have standardized configurations and processes • Avoid massive variations in vendors, platforms, and OS versions • Avoid massive variations in topologies and feature use ( e.g. virtual router vs. zones) • Insist on hardware that does have usable API (e.g. Netconf) and avoid to relying on screen-scraping for automation • Hardware that does have good commit, rollback, and diff mechanism. • Hardware that virtual images to be able to test and validate changes. 30
• 32.
  Desired Policies: - Devicestate(s) - Topology state - External input Contellor Data Model: - Configuration - Operational Network Device(s) Data Model: - Configuration - Operational Topology (protocols) -..-.-..-.-.-..-. Configuration difference (e.g. NetConf) =-...-.-=.. State: - Telemetry - SNMP - NetConf - RestConf - Syslog Connection:: - telnet*/ssh - https
• 33.
  CLI best practices SNMP experience Operations requirements Netconf, RestConf and YANG Charles Eckel,Cisco DevNet: https://fosdem.org/2018/schedule/event/opendaylight/
• 34.
  Data model language usedto model Configuration and operational state data easy to extend on existing models IETF standard defined in RFC 6020 Data model language for both Configuration and operational state data A solution to the multi-vendor device data discrepancy Not All vendors yet serious about it Ivan Pepelnjak on Monday, January 29,2018 blogged:: http://blog.ipspace.net/2018/01/use-yang-data-models-to-configure.html
• 35.
  ● Defined inRFC 4741 (2006), updated by RFC 6241 (2011) ● Provides mechanisms to install, manipulate, and delete the configuration of network devices ● Model driven APIs ● Distinguishes between configuration and operational/state dat ● Multiple configuration datastores (candidate, running, startup) ● Configuration change validation and transactions ● Selective data retrieval via filtering ● Streaming and playback of event notifications
• 36.
  ● IETF RFC8040 ● Configuration data and state data exposed as resources ● How to access the data using REST verbs (GET / PUT / POST/ …) ● How to construct URIs to access the data ● HTTP instead of SSH for transport ● JSON in addition to XML for data encoding
• 38.
  Ansible netconf module JunosManaged Device IOS-XR Managed Device EOS Managed Device netconf over ssh port 830 RestConf* over https port 443 YANG based XML YANG based XMl/JSON Start Run Candi date data store * Ansible does not support Restconf yet, most likely will support gRPC transport
• 39.
  Ansible netconf_config module
• 40.
  Ansible vendor modules AristaEOS - name: set ntp server in the device eos_commands: - “ntp server {{ ntp_servername }}” CISCO IOS - name: set ntp server in the device ios_commands: - “ntp server {{ ntp_servername }}” Juniper Junos - name: set ntp server in the device junos_commands: - “set system ntp server {{ ntp_servername }}”
• 41.
  To declare ornot to declare vendor_config - Playbooks becomes operating manuals. easy to understand and replicate in the CLI - Gradual step toward declarative, declarative network modules coverage is not complete. e.g. Junos syslog - favor the human interactive CLI over the cut & paste machine structures.
• 42.
  Ansible supported modules: ●netconf: ○ netconf_config ● vendor_config ○ ios_config ● vendor_cmmand ○ ios_command Minimum Viable Platform Agnostic modules: e.g. net_interface Vendor/Community supported modules: ● netconf: ○ junos_netconf ○ ce_netconf ● Network to Code: ○ ntc_install_os ○ ntc_get_facts ● N.A.P.A.L.M: ○ napalm_diff_yang ○ napalm_get_facts Custom built module: https://www.ansible.com/ansible-module-development-101
• 43.
  Hit Refresh • ContinuousImprovements • Review and Improve what has been done • Improve one thing at a time • Learn and review past work • Document and prioritize new problems challenges
• 44.
  Networktocode slack channelhttp://networktocode.herokuapp.com/ *** Absolutely Gorgeous ***** SDN & NFV: https://fosdem.org/2018/schedule/event/opendaylight/ Blogs: Csilla Bessenyei Networker and coder https://networkerandcoder.wordpress.com/ Kirk Byers “Python for network engineers” https://pynet.twb-tech.com/ Mircea Ulinic https://mirceaulinic.net Jason Edelman http://jedelman.com/ David Lore http://ipengineer.net/ netmiko https://github.com/ktbyers/netmiko N.A.P.A.L.M https://napalm-automation.net/ Training: gns3 Academy http://academy.gns3.com/ Ansible network automation examples: https://github.com/network-automation Salt: https://docs.saltstack.com/en/develop/topics/network_automation/index.html Net survey: https://docs.google.com/forms/d/e/1FAIpQLSdiBNMK0ZUmgBSNEaOWa-YHGQ4AlZo7EhB52_dXzvMqic3eHA/viewanalytics https://interestingtraffic.nl/2017/03/27/insights-from-the-netdevops-fall-2016-survey/ ipspace blog and podcast: https://www.ipspace.net packetpushers podcast: http://packetpushers.net/
• 45.
  Thank you