Automated Server Administration for DevSecOps

Automated Server Administration for DevSecOps

• 1.
  VSHN - TheDevOps Company Automated Server Administration for DevSecOps Aarno Aukia, CTO @ VSHN - The DevOps Company DevSecOps Forum 12.3.2019
• 2.
  VSHN - TheDevOps Company ● Who is Aarno and VSHN - The DevOps Company ? ● Managing 1200 servers ○ Challenges ○ Our implementation & user interface ○ Demo ○ Lessons learned 22 Agenda
• 3.
  VSHN - TheDevOps Company @aarnoaukia http://about.me/aarno [email protected] ETH → Google → Atrila → VSHN VSHN - The DevOps Company Since 2014, currently 35 VSHNeers in Zürich, Switzerland Helping Developers run applications on any infrastructure making both visitors happy with stability and developers happy with agility 33 About Aarno & VSHN.ch
• 4.
  VSHN - TheDevOps Company 5 Operations = Firefighting-as-a-Service ? 5
• 5.
  VSHN - TheDevOps Company Capability Maturity Model Integration (CMMI) 66 Operations 2014 How to get to this level?
• 6.
  VSHN - TheDevOps Company DevOps: CMMI Level 5: People, Processes & Tools 77
• 7.
  VSHN - TheDevOps Company ● Interface Dev <-> Ops ○ DevOps, CI/CD, Containers, etc ● Interface Ops <-> Infra ○ Hypervisors, APIs, Metrics, etc ● Manage VMs on any infrastructure ● Service Monitoring, Alerting, Metrics, Logs ● Updates, Backups, Configuration Management 88 VSHN Managed Service
• 8.
  VSHN - TheDevOps Company ● you don’t like proper (software) release management ○ or else you’ll be in dependency hell with 200 modules in 200 environments ● you practice “don’t touch a running system” ○ you’ll be updating all servers all the time - both good and bad ● standardizing/optimizing ○ no more handcrafted snowflake pets, lots and lots of cattle ● you don’t actively develop monitoring/checks ○ or else you’ll get alert fatigue from noisy checks 99 Don’t do automatic server management if...
• 9.
  VSHN - TheDevOps Company ● Web-based self-service for customers ● CRUD server definitions -> CMDB 1010 VSHN Control Panel
• 10.
  VSHN - TheDevOps Company 11 Demo 11
• 11.
  VSHN - TheDevOps Company 1212 Infrastructure as Code Single source of truth: Configuration YAML in GIT --- classes: - profile_certificates - profile_webhosting packages: - nagios-plugins-wordpress - nagios-plugins-wordpress-config profile_php::php_settings: date.timezone: 'Europe/Zurich' acmetool::use_production_endpoint: true profile_php::php_version: '7.0' profile_php::extensions: - mysql - gd - mbstring - curl profile_icinga2::hiera_services: check_wordpress_dev: check_command: 'wordpress_updates' display_name: 'Wordpress Updates test.vshn.net' vars: wordpress_instance: 'vshn_dev' wordpress_db_name: 'vshn_dev' wordpress_db_user: 'vshn_dev' wordpress_db_pass: 'replaced' check_wordpress_prod: check_command: 'wordpress_updates' display_name: 'Wordpress Updates vshn.net' vars: wordpress_instance: 'vshn_prod' wordpress_db_name: 'vshn_prod' wordpress_db_user: 'vshn_prod' wordpress_db_pass: 'replaced' profile_mariadb::db_root_pw: 'replaced' profile_webhosting::sites: vshn_dev: stack_type: 'php' db_type: 'mariadb' db_password: 'replaced' php_admin_values: memory_limit: '300M' post_max_size: '20M' upload_max_filesize: '20M' manage_tls: letsencrypt
• 12.
  VSHN - TheDevOps Company ● who ● when ● what ● tests successful? ● why 1313 GIT: changelog
• 13.
  VSHN - TheDevOps Company 1414 Logic # https check if $manage_tls and $manage_tls != 'false' { ::profile_icinga2::resources::check { "${name}_nginx-${main_domain}-https": check_command => 'http', display_name => "nginx https ${name}", check_zone => $monitoring_check_zone, notes => 'https check on port 443, supports authentication and URI', notes_url => 'http', production_level => $monitoring_production_level, vars => merge({ 'http_address' => $::fqdn, 'http_vhost' => $main_domain, 'http_ssl' => true, 'http_uri' => $monitor_path, 'http_sni' => true, }, $_auth_var, $monitor_extra_check_vars), } } # https certificate check if ($manage_tls and $manage_tls != 'false' and $manage_tls != 'snakeoil') { if $manage_tls == 'trusted' { $_days = 30 } else { $_days = 20 } ::profile_icinga2::resources::check { "${name}_nginx-${main_domain}-certificate": check_command => 'http', display_name => "nginx certificate ${main_domain}", notes => "https certificate check, checks days (${_days}) left until expire", notes_url => 'https_certificate', production_level => $monitoring_production_level, check_interval => '2h', vars => { 'http_address' => $::fqdn, 'http_vhost' => $main_domain, 'http_tls' => true, 'http_certificate' => $_days, 'http_sni' => true, }, } } } e.g. automatically monitor all HTTPS sites & certificates
• 14.
  VSHN - TheDevOps Company ● Trade-off between multi-tenancy (separation of customers/environments) and common code base development -> Tools & Software-Release-Processes (e.g. https://github.com/vshn/crmngr) ● Monitoring & Backup by default: value > cost ● Hierarchical CMDB: configuration for policy groups ● Staged/canary rollouts (e.g. for intrusive things like firewall changes) ● Local tests (pacco: puppet-in-docker-in-vagrant: https://github.com/vshn/puppet-in-docker) ● Good and robust process for regular & frequent updates 1515 Lessons learned
• 15.
  Come visit usfor a coffee! VSHN AG - Neugasse 10 - CH-8005 Zürich - +41 44 545 53 00 - https://vshn.ch/ - [email protected] https://vshn.ch/kontakt/ Follow us on Twitter! @vshn_ch 16