Why don'we just say "No" - Devopsdays Silicon Valley Jun 2014
6 likes2,091 views
The document discusses strategies for learning to say "no" more often. It provides several reasons why it can be difficult for people and teams to say no, such as not wanting to disappoint others, being a team player, or fearing those in positions of power. It then describes how one operations team evolved their ability to say no by establishing work-in-progress limits, prioritizing work, decreasing lead times, and giving customers visibility into their capacity constraints and priorities. Saying no requires clear definitions of roles and responsibilities, continuous improvement efforts, and establishing rules or limits to provide fair ways to determine what work can be taken on or will need to wait.
Why don'we just say "No" - Devopsdays Silicon Valley Jun 2014
- 1. Why don’t We just say no? ! ! ! Ge$ng!things!done!doesn't!always! start!with!"Yes". @dominicad! www.ddegrandis.com!
- 2. @dominicad! www.ddegrandis.com! For a number of reasons, it’s hard to say no.!
- 9. Why do Development teams seem to be able to say no more than Ops teams? !
- 10. Stuff Ops deals with that Dev traditionally doesn’t: • Live prod issues • DDOS attacks • 100% disk utilization • access requests at 2:00 am • opening up firewalls
- 11. Over the last 11 years, a lot of Dev teams have been given permission to say, “We’ve already done the planning for this week. Your next opportunity to get your request considered is next week.” Saying “No” has been institutionalized by using timeframes.
- 12. This doesn’t work for Ops very well… “Sorry - that live production fix will have to wait until next week” What if instead of using timeframes to say “No”, we used limits to say “No”?
- 13. A!look!at!one!team’s!evoluCon! 40 Ops Engineers (SysAdmin, DBA, Network, Mon, Sec) Tasked to build out/retrofit 6 data-centers across 6 diff countries. AND…. • keep the lights on four existing data centers • build out a new platform architecture • support live issues (on-call) • roll out a new configuration management tool • deploy a crap load of new features • deal with 3 reorgs over a 6 month period !! ! !
- 14. !!Problems: • Conflicting priorities • risky dependencies • over-commitments • interrupt-driven context switching resulting in missed commitments. • Confusion around the new org structure !! ! !
- 15. How!we!got!started!on!road!to! improving!–!looked!at!3!data!pts! !! ! open closed #1 Can we keep up with the demand?
- 16. 0! 10! 20! 30! 40! 50! 60! 1! 2! 3! 4! 5! 6! 7! 8! 9! 10! 20! 30! 40! 50! 60! 70! 80! 90! 100! 110! 120! 130! 140! 150! 160! 170! 180! 190! 200! 250! 300! More! #"#ckets" # days it took for ticket to go from created to closed #2 Lead time – how long does it take to get work done?
- 18. Customer!Mtg!Cme! !! Invited!customers!to!a!meeCng!and!showed!them!the!data.!! – showed!them!the!demand!and!what!was!ge$ng!done!!! – showed!them!the!Lead!Cme!! – showed!!where!work!got!stuck! !! Customers!appreciated!the!visibility!into!Ops! !We!took!advantage!of!that!by!humbly!asking!for!their!help.! !beginning!with!all!of!those!Cckets!si$ng!in!the!validate!state.!!
- 20. and!then!there!were!reorgs! The!1st!!org!restructure!change!created!the!A!Team!!`!to! focus!on!compleCng!projects!close!to!being!done,!but!sCll! hanging!on.! ! • This!team!didn’t!have!to!respond!to!one!off!requests! and!wasn’t!supposed!to!be!on`call.! !! • 2nd!org!structure!change!split!Ops!into!3!teams!(live! Ops,!Build,!architecture)! • Live`Ops!with!25%!of!team!and!60%!of!the!work!! ! !
- 21. • access requests for systems, non-Zabbix monitor • hardware investigation/verification/fixes - vlan/port changes - data retrieval (i.e. logs, network stats, etc) • configuration triage - firewalls, load balancers, OS settings • (small) capacity expansion • verification of configs/services across shards • database development consultation • security compliance mitigation Live Ops tasks
- 24. Live!Ops!SRE!Changes! • Socialized wip limit idea over 6 months and gradually lowered it from 10 to 7 – out of 18 guys, average is 5-7. • Hired 4 more people, although 2 got stolen by another team • Closed out all tickets with no activity > 90 days • Started saying “No” to last minute requests. !
- 25. Hi D.C., Team SRE has a very large number of changes scheduled for today already, and an even larger number of requests in our backlog that this request will displace if moved to the front of the queue. It would not be fair to other teams if we jumped on this immediately while planned work is pushed off. Monitoring should be a requirement for a service to go live, not a last minute addition. For us to fully support a live service, please implement monitoring before going live. For future requests, please give us as much notice as possible, and make sure to create a ticket (xxx.com) so we can prioritize and schedule the changes as necessary. Here's the ticket for this work…. Respectfully, A.H !
- 26. Live!Ops!SRE!changes!con’t! • Took time during standups to focus on kaizen improvements. • Reduced validate state from 7 to 5 to 3 days. • Found creative way to deal with walkups, and work done via personal relationships • 15 min daily sync up at 3pm instead of interrupting. • 5 min videos to present Ops review data to customers
- 28. Clear definitions of done between queues
- 29. kaikzen/ ! noun, continuous improvement A business philosophy or system based on making positive changes on a regular basis.
- 30. “Disney thinks we can’t do it. Let’s prove them wrong.” -Steve Jobs
- 31. “Asking this much of people, even when they wanted to give it, was not acceptable.” ! !! ! ! ! !!`!Ed!Catmull!
- 32. “WIP it down to the personal safety level”!
- 33. “No” is an honorable reply to someone asking too much from you.
- 35. ! But,!what!about!when!the!show!must! go!on?! ! World!Cup!finals!`!!Sunday,!July!13! !! fifa regulations ensure they have capacity. “pg 8 e) ensuring the presence of a sufficient number of ground staff and security stewards to guarantee safety.” Let’s not expect day shift workers to also cover the night shift.
- 36. @dominicad! www.ddegrandis.com! Honey-do list rules for saying no to the spouse…