Bridging the
Dev-OPS Gap
DevOPS Overview
Omri Spector
• Owner of deveLeap
• 20+ years in development, always
active
• Former director of Business
Solutions in Actimize
• Engineering Practices Consultant to
many companies of varying sizes
What is DevOPS
• It’s a Fashionable term 
• It originated as multiple practices in
several unrelated organizations:
– Netflix, Twitter, Facebook, Google,
Amazon, Flickr
• Everyone is looking for “devops
people” (?)
• Everyone says it has great ROI
• But what is it!?
There is no “formal” definition
“Help me define it…”
Adrian
Cockroft,
Netflix
My take: DevOPS is…
(1) A Culture shift in responsibility &
communication
(2) A set of:
– Architecture & Design patterns
– Tools
– Practices
That allow this shift to happen
In this presentation we will discuss all these
aspects
Important Caveat
• DevOPS movement mostly originated in the
SAAS arena
• Many of the concepts, tools and practices
apply well to any development environment
BUT
• In some cases adaptation to the Enterprise
Software arena is a challenge
When starting to implement DevOPS:
• Be open minded, not dogmatic
• Align with your business and challenges
Why did DevOPS originate at
“Consumer” SAAS?
• Fierce competition and low entry
barriers
• Direct deploy to production is an
internal negotiation
• Risk is limited – limited liability
Dev vs. OPS
• Most of failures can be tracked to dev changes
• Most of the changes can be tracked to business need
OPS – is in
charge of
stability
Dev – is
expected to
change fast
& furious
DevOPS Tenets
Common Goal
• Acknowledge the common goal:
To enable the business
• Reflect that in the R&R
– Developers that do OPS
– All OPS is code
• Reflect that in common metrics:
– Operational metrics
– Live, visible, accessible
Everything is code
• There is 0 manual OPS
• Code is:
– Version controlled
– Visible to all – use shared repo!
– Deterministic
• Build & deploy must be fully
automated
Continuous Integration
• Fully automated from commit to
deploy
• Automated stability tests with visible
results
• Anyone can prepare a release
Automated Deployment
• Single click from release to
deployment
• Continuous Deployment as an option
• Clear deployment log – Who, When,
Why
• In an ES company this may map into
customer visible staging area
CD in an ES Context
Automated Testing
• If you didn’t test it – it doesn’t work
• Your release is as good as your CI
• Continuous Release relies on
massive automated coverage
Release Early, Release Often
Release should be a “non-event”
• Anything that is done 4 times a year
is complex, risky and inherently error
prone
• Anything that is done multiple times a
day is a “non-event”
Do You Have The
Guts To Deploy 10
Times A Day?
Does it take guts?
• Deploying often will lower risk if:
– Concerns are indeed separated
– Everything is automated
– Changes are small and managed
– Occasional failure is accounted for
• Toggles
• Easy Rollbacks
• Openness to failure
Fail Fast, Fix Fast
• Daniel Ek: "We aim to make mistakes
faster than anyone else.”
Some common
techniques
Small Changes, Smaller Risks
We rapidly prototype our satellites.
We use "release early, release
often" on our software. We take a
different risk approach: We take
them outside and test them.
We even put satellites in space
just to test
Will Marshal, planet-labs
Simplify Deployment Unit
• Separate concerns into services:
– Lower bubble effect
– Simplify deployment – just define roles
– Fast deploy – rollback – roll forward
• Other “Side effects”
– Flexible scaling and auto-scaling
– Easier to Focus development efforts
– Complex overall architecture
• This is the trend to “micro services”
Use Toggle Switches
• Each feature (and less) can be
toggled in & out at will
• This allows:
– Pushing partial work for higher level of
CI
– Partial rollouts
– Fast back out
Toggle Implementation
Toggle Dashboard
Apply Canary Releases
• Release gradually:
– Specific user group
– Specific GEO / BU
– Percentage
• Monitor situation
– Load
– Errors
– User satisfaction
• All by/with DEV
Multiple Concurrent Versions
• A direct result of Canary release
• Has impacts on design
– All communication must be backward
compatible
– DB Upgrades must be backward
compatible
• Will sometimes lead to multi-phased
release
– E.g. Migrating database structure
• In the ES world this also presents
marketing and support challenges!
Dark Launch
• Toggle-in work load,
Toggle-out functionality
• Minimal impact on failure
– just toggle out
• Gradual Dark launch
makes release an
anti climax
Ongoing Incident Management
• Track all deployments:
– Type (code, db, cm, …)
– Who, When, Why
• Post mortem all incidents:
– Dev & OPS – together
We are in the same boat
Central Operational Dashboard
• Operational status is the highest non-
business metric
• Dev exposed to “OPS” metrics on
daily basis:
– Uptime, Downtime, Incidents
– Resource consumption
– Performance metrics
– And even cost
Measurability in every feature
• Measuring feature impact, technical
& functional, is part of a feature spec
• Feature released with clear set of
metrics
• Metrics are monitored by feature dev
(at least at the beginning)
Typical Tools
Using the correct tools
• DevOPS is implemented with many
tools & tool sets. There is no silver
bullet.
• Following are some of the more
popular/relevant ones
• Often people mix “knowledge of tool
X” with “knowledge of DevOPS”
Configuration Management
• Pioneered by CFEngine
• CM Principles:
– Descriptive
– Idempotent
– Continuous
• Currently hyped: Puppet, Chef,
Ansible
Example: Ansible Playbook
• Defines a set of “roles”: appserver,
db, …
• Defines the configuration per role
using descriptive yaml
• Allows mapping of a role to a host
• Upon call – applies the role to the
host
Virtualization, a.k.a. Cloud
Computing
• Services: AWS, Google Cloud, Azure
• Products: OpenStack,OpenShift
• Enables us to:
– Spin versions in and out,
– Scale system components based on
load
– Support multiple environments for dev
and QA needs
APM & OPS Analytics
• Application Performance Monitors:
– AppDynamics
– New Relic
• Log Analytics
– ELK Stack (ElasticSearch, LogStash,
Kibabna)
– Splunk
Example: AppDynamics Dashboard
Example: ELK Dashboard
Source Control
• Well, you all know SVN, but…
• GIT is where the devops world
currently resides:
– Distributed
– Free
– Low install overhead
– Integrated with everything
Continuous Integration Servers
• Jenkins
• TeamCity
• Travis
• Bamboo
• :
Putting it all together
DevOPS is…
• DevOPS is a cultural change, it is a
mindset that must be accompanied
by technical changes in architecture,
procedures and tooling
• Remove any of these legs
at your own peril…
Wrong mindset
• Manager: It is all about responsibility
– so lets enforce it!
• Being responsible requires more than
good will – it requires a context that
allows you to practice this
responsibility in an effective manner
Effective context
Supportive Architecture
• Easy swap in/out of components,
• Limited bubbling effect,
• Fast reload times,
• …
Tools in place
• Configuration Management
• Automated deployment
• Toggle switches,
• Online monitoring,
• Easy deploy & rollback
• …
Acceptable risk
• The worst case is manageable,
• Most glitches are low risk,
• Occasional failure is accepted,
• Reasonable failures are perceived as
a learning opportunity, a price for fast
paced innovation
Cooperative Culture
• Dev & OPS cooperation:
– Before – content visibility,
– During – timeline visibility
– After – environment visibility
• OPS is part of every design
• Dev is part of every incident handling
and Post Mortem
How long will it take?
• DevOPS is another aspect of
becoming a “learning organization”
• And in that sense – it is a journey
Bon Voyage 
Omri Spector, deveLeap www.develeap.com

DevOps Overview

  • 1.
  • 2.
    Omri Spector • Ownerof deveLeap • 20+ years in development, always active • Former director of Business Solutions in Actimize • Engineering Practices Consultant to many companies of varying sizes
  • 3.
    What is DevOPS •It’s a Fashionable term  • It originated as multiple practices in several unrelated organizations: – Netflix, Twitter, Facebook, Google, Amazon, Flickr • Everyone is looking for “devops people” (?) • Everyone says it has great ROI • But what is it!? There is no “formal” definition
  • 4.
    “Help me defineit…” Adrian Cockroft, Netflix
  • 5.
    My take: DevOPSis… (1) A Culture shift in responsibility & communication (2) A set of: – Architecture & Design patterns – Tools – Practices That allow this shift to happen In this presentation we will discuss all these aspects
  • 6.
    Important Caveat • DevOPSmovement mostly originated in the SAAS arena • Many of the concepts, tools and practices apply well to any development environment BUT • In some cases adaptation to the Enterprise Software arena is a challenge When starting to implement DevOPS: • Be open minded, not dogmatic • Align with your business and challenges
  • 7.
    Why did DevOPSoriginate at “Consumer” SAAS? • Fierce competition and low entry barriers • Direct deploy to production is an internal negotiation • Risk is limited – limited liability
  • 8.
    Dev vs. OPS •Most of failures can be tracked to dev changes • Most of the changes can be tracked to business need OPS – is in charge of stability Dev – is expected to change fast & furious
  • 10.
  • 11.
    Common Goal • Acknowledgethe common goal: To enable the business • Reflect that in the R&R – Developers that do OPS – All OPS is code • Reflect that in common metrics: – Operational metrics – Live, visible, accessible
  • 12.
    Everything is code •There is 0 manual OPS • Code is: – Version controlled – Visible to all – use shared repo! – Deterministic • Build & deploy must be fully automated
  • 13.
    Continuous Integration • Fullyautomated from commit to deploy • Automated stability tests with visible results • Anyone can prepare a release
  • 14.
    Automated Deployment • Singleclick from release to deployment • Continuous Deployment as an option • Clear deployment log – Who, When, Why • In an ES company this may map into customer visible staging area
  • 15.
    CD in anES Context
  • 16.
    Automated Testing • Ifyou didn’t test it – it doesn’t work • Your release is as good as your CI • Continuous Release relies on massive automated coverage
  • 17.
    Release Early, ReleaseOften Release should be a “non-event” • Anything that is done 4 times a year is complex, risky and inherently error prone • Anything that is done multiple times a day is a “non-event”
  • 18.
    Do You HaveThe Guts To Deploy 10 Times A Day?
  • 19.
    Does it takeguts? • Deploying often will lower risk if: – Concerns are indeed separated – Everything is automated – Changes are small and managed – Occasional failure is accounted for • Toggles • Easy Rollbacks • Openness to failure
  • 20.
    Fail Fast, FixFast • Daniel Ek: "We aim to make mistakes faster than anyone else.”
  • 21.
  • 22.
    Small Changes, SmallerRisks We rapidly prototype our satellites. We use "release early, release often" on our software. We take a different risk approach: We take them outside and test them. We even put satellites in space just to test Will Marshal, planet-labs
  • 23.
    Simplify Deployment Unit •Separate concerns into services: – Lower bubble effect – Simplify deployment – just define roles – Fast deploy – rollback – roll forward • Other “Side effects” – Flexible scaling and auto-scaling – Easier to Focus development efforts – Complex overall architecture • This is the trend to “micro services”
  • 24.
    Use Toggle Switches •Each feature (and less) can be toggled in & out at will • This allows: – Pushing partial work for higher level of CI – Partial rollouts – Fast back out
  • 25.
  • 26.
  • 27.
    Apply Canary Releases •Release gradually: – Specific user group – Specific GEO / BU – Percentage • Monitor situation – Load – Errors – User satisfaction • All by/with DEV
  • 28.
    Multiple Concurrent Versions •A direct result of Canary release • Has impacts on design – All communication must be backward compatible – DB Upgrades must be backward compatible • Will sometimes lead to multi-phased release – E.g. Migrating database structure • In the ES world this also presents marketing and support challenges!
  • 29.
    Dark Launch • Toggle-inwork load, Toggle-out functionality • Minimal impact on failure – just toggle out • Gradual Dark launch makes release an anti climax
  • 30.
    Ongoing Incident Management •Track all deployments: – Type (code, db, cm, …) – Who, When, Why • Post mortem all incidents: – Dev & OPS – together We are in the same boat
  • 31.
    Central Operational Dashboard •Operational status is the highest non- business metric • Dev exposed to “OPS” metrics on daily basis: – Uptime, Downtime, Incidents – Resource consumption – Performance metrics – And even cost
  • 32.
    Measurability in everyfeature • Measuring feature impact, technical & functional, is part of a feature spec • Feature released with clear set of metrics • Metrics are monitored by feature dev (at least at the beginning)
  • 33.
  • 34.
    Using the correcttools • DevOPS is implemented with many tools & tool sets. There is no silver bullet. • Following are some of the more popular/relevant ones • Often people mix “knowledge of tool X” with “knowledge of DevOPS”
  • 35.
    Configuration Management • Pioneeredby CFEngine • CM Principles: – Descriptive – Idempotent – Continuous • Currently hyped: Puppet, Chef, Ansible
  • 36.
    Example: Ansible Playbook •Defines a set of “roles”: appserver, db, … • Defines the configuration per role using descriptive yaml • Allows mapping of a role to a host • Upon call – applies the role to the host
  • 37.
    Virtualization, a.k.a. Cloud Computing •Services: AWS, Google Cloud, Azure • Products: OpenStack,OpenShift • Enables us to: – Spin versions in and out, – Scale system components based on load – Support multiple environments for dev and QA needs
  • 38.
    APM & OPSAnalytics • Application Performance Monitors: – AppDynamics – New Relic • Log Analytics – ELK Stack (ElasticSearch, LogStash, Kibabna) – Splunk
  • 39.
  • 40.
  • 41.
    Source Control • Well,you all know SVN, but… • GIT is where the devops world currently resides: – Distributed – Free – Low install overhead – Integrated with everything
  • 42.
    Continuous Integration Servers •Jenkins • TeamCity • Travis • Bamboo • :
  • 43.
  • 44.
    DevOPS is… • DevOPSis a cultural change, it is a mindset that must be accompanied by technical changes in architecture, procedures and tooling • Remove any of these legs at your own peril…
  • 45.
    Wrong mindset • Manager:It is all about responsibility – so lets enforce it! • Being responsible requires more than good will – it requires a context that allows you to practice this responsibility in an effective manner
  • 46.
  • 47.
    Supportive Architecture • Easyswap in/out of components, • Limited bubbling effect, • Fast reload times, • …
  • 48.
    Tools in place •Configuration Management • Automated deployment • Toggle switches, • Online monitoring, • Easy deploy & rollback • …
  • 49.
    Acceptable risk • Theworst case is manageable, • Most glitches are low risk, • Occasional failure is accepted, • Reasonable failures are perceived as a learning opportunity, a price for fast paced innovation
  • 50.
    Cooperative Culture • Dev& OPS cooperation: – Before – content visibility, – During – timeline visibility – After – environment visibility • OPS is part of every design • Dev is part of every incident handling and Post Mortem
  • 51.
    How long willit take? • DevOPS is another aspect of becoming a “learning organization” • And in that sense – it is a journey
  • 52.
    Bon Voyage  OmriSpector, deveLeap www.develeap.com

Editor's Notes

  • #9 BUT note that these changes WERE NEEDED. Our whole business model is based on our ability to adapt software to changing need.
  • #10 This is highly un-productive
  • #13 Shared repo is the first step to shared responsibility and trust – I can help because I can see. No magic.
  • #30 Dark launch is not limited to UI – it is often applied to business rules and backend logic