MoldCamp - multidimentional testing workflow. CIBox.

MoldCamp - multidimentional testing workflow. CIBox.

• 1.
  https://github.com/propeoplemd/cibox Multidimensional testing workflowbefore merge to master. CIBox suite. @podarok Andrii Podanenko DevOps Architect, FFWco https://www.drupal.org/u/podarok
• 2.
  Oldschool development workflow
• 3.
  Development infrastructure ● singledevelopment environment ● separate company dev server (multiple vhosts one per project) ● SaaS solutions (Acquia, Pantheon, …)
• 4.
  Coding process ● allcommits directly to master ● master deployed to dev site (sometimes automatically) ● code review on dev site after deployment
• 5.
  Configuration process ● masterdatabase, get pulled to local dev environments (backup & migrate) ● changes to configuration happens on dev site manually
• 6.
  Problems ● shared resourceson dev environment -> dev conflicts (cache, solr, mysql, apache) ● too much time to configure all services on local dev desktop (varnish, solr) ● large gap between Dev and Ops (only one guru able to do production release)
• 7.
  “Newschool” development workflow
• 8.
  Local development ● localvirtualized environment (vagrant) ● started from puphpet.com but switched to ansible ● based on ubuntu 12.04 (upgrading to 14.04)
• 9.
  Database ● no configurationon demo/stage/prod sites ● code driven development ● database/profile workflow
• 10.
  Code process ● Githubpull requests workflow ● code review before merge to master ● code style checks, test runs ● security tests ● QA on builds
• 11.
  Extra tools ● osmonitoring to see resources consuming operations (multinode munin) ● automated complex deployments Acquia deployment ● visual regression testing -> ● urls health checks ->
• 12.
  Visual regression ● http://backtrac.ioSaaS ● screenshots before / after release, diff ● automated scheduled screenshots / diffs ● authenticated user ● register and start using now!
• 14.
  URLs health checks ●https://github.com/ygerasimov/website-size-scan ● scans all URLs in the file, checks sizes of images ● logs 404s, 5xx, etc. ● written on golang
• 15.
  Welcome CIBox http://bit.ly/ffw-cibox
• 16.
  CI for projectand VM with Drupal initial codebase Project initial creation playbook github.yml Continuous Integration for a project jenkinsbox.yml CIBox code structure
• 17.
  Steps for gettingstarted 2. Repository initialization ● Make needed changes to github.yml ● Run ansible-playbook github.yml ● push generated codebase folder to github repo ● check Pull Request builder with newly created change to readme.md ● profit 1. CI server ● Get virtual or real server from hosting provider (Ubuntu LTS 64 bit only for now) ● Set root password ● Make initial config changes to jenkinsbox.yml and inventory ● Run ansible-playbook jenkinsbox.yml from a shell ● Make changes to jenkins UI with credentials to github repo How to deploy whole CI system
• 18.
  ansible playbook forinstalling CI server - Jenkins powered install - Needed Jenkins’s plugins - LAMP stack + SSL - PHP Code Sniffer, scss-lint - Java JDK - Jetty && Apache Solr - Selenium && Behat packages - Optimized and preconfigured configs jenkinsbox.yml
• 19.
  A bunch ofjobs with scripts for running playbooks Preconfigured Jenkins - Pull Request Builder - Skeleton for Backup production database - Demo site builder - Disk space cleaner
• 20.
  During run ofgithub.yml you’ll get a codebase that has latest drupal in drupal folder and scripts for future CI builds and tests with Vagrant VM. Latest drupal, adminer, devops scripts, basic profile Main project code structure
• 21.
  Vagrant + virtualbox(optional lxc) + ansible provisioner We are using trick for sharing ansible roles between CI server and VM provisioning scripts for making sure we have equal environments for both. Basic stuff for now (all are inside splitted ansible roles): composer, pear, ansible, apache, memcached, mysql, php, sdebug, shprof, selenium, behat, drush, jetty solr, phpdaemon, php codesniffer, apache ssl, custom swap. Just vagrant up and you are ready to go coding. Vagrant box
• 22.
  how to workwith CIBox Developer point of view
• 23.
  Single task flow
• 24.
  Comments by CIbot
• 25.
  Development phases 1. Reinstallfrom scratch every build by reinstall.yml playbook 2. Update path, content can be edited at stage by reinstall.yml playbook and pp_environment: staging 3. SLA (production) update path with QA testing on stage
• 26.
  technical information How CIworks
• 27.
  Profile based flow ReinstallsDrupal from scratch every builder time
• 28.
  SQL based flow ImportsSQL dump every build and prepares it to codebase
• 29.
  Team rules ● Nevermerge own Pull Request ● Never push directly to main repo master branch ● master branch is stable ● There should be two siblings for every role (optional) ● Bugs introduced from specific PR would be nice to assign to its reviewer. (optional)
• 30.
  Responsibility Due to thefact all DevOps scripts are in the same repo with a project itself - any developer can change workflow at any point. Team does manage all the steps for DevOps scripts, no need to involve Ops into the team.
• 31.
  Flow Bottlenecks ● Dependencyfrom github(gitlab, bitbucket) ● If CI server down - team get stopped on code review step ● New developers should follow new rules. (Coder is tough) ● DevOps must be a team member(s) ● Code Review get hurt ● Builds are slow on huge projects ● Decent desktops for a team (SSD is a must) ● Minimal task >=1 hour ● Overall system is pretty complex ● Not so easy to start for new teams or companies
• 32.
  How to startusing CIBox in your team ● Contribute* to CIBox and get more familiar with its subsystems. ● Install the flow for some internal project without tough deadline ● Organize codesprint with CIBox as workflow ● Start to use 1-2 parts of the system and add new parts every following project ○ sniffers.yml ○ tests.yml ○ reinstall.yml ○ Vagrant box ○ ansible scripting(playbooks) ○ Jenkins ○ Github PR for manual code review *CIBox is opensource, based on popular technologies...
• 33.
  Usefull linksDocumentation ● https://github.com/propeoplemd/cibox ●https://github.com/propeoplemd/cibox/blob/master/README.md ● https://github.com/propeoplemd/cibox/blob/master/github/files/drupal7/scripts/README.reinstall.m d Presentations ● http://events.drupal.org/losangeles2015/sessions/multidimensional-testing-workflow-merge-master ● http://druler.com/node/888 ● http://www.slideshare.net/podarok/drupal-continuous-integration-workflow ● http://www.slideshare.net/podarok/start-using-vagrant-now ● http://www.slideshare.net/podarok/live-deployment-ci-drupal ● http://www.slideshare.net/ygerasimov/ci-drupal-camp-berlin-2014 ● http://www.slideshare.net/ygerasimov/vagrant-stanford-drupalcamp-2014 ● http://www.slideshare.net/ygerasimov/continuous-integration-stanford-2014 Blog posts ● http://wearepropeople.com/blog/how-we-use-vagrant-in-our-drupal-development-workflow
• 34.
  Questions? http://bit.ly/ffw-cibox Andrii Podanenko DevOps Architect https://www.drupal.org/u/podarok