ONE COMMIT
ONE RELEASE
CONTINUOUSLY DELIVERING A PROJECT
Javier López @loalf
Senior Platform Engineer @Rightster!
formerly Software Architect @TimeOut
Certified Symfony Developer
Zend Certified PHP
Co-organizer @desymfony
DISCLAIMER
AGENDA
CONTINUOUS DELIVERY IN A NUTSHELL
BUILDING THE PIPELINE
BEFORE AND AFTER
1
2
3
CONTINUOUS DELIVERY …
… IN A NUTSHELL
1
COMMIT BUILD TEST DEPLOY
When and how a change in your code is
going to trigger the pipeline
COMMIT BUILD TEST DEPLOY
Checks the integerity of your code.
Produces an artifact.
COMMIT BUILD TEST DEPLOY
Checks the integerity of your project.
!
After this step you should be 100% confident that
you could deploy your artifact.
COMMIT BUILD TEST DEPLOY
Does what it says in the tin. Triggering this
step should be “a click away”.
Manual
COMMIT BUILD TEST DEPLOY
Does what it says in the tin. Triggering this
step should be “a click away”.
Manual
THE BOOK
THE PIPELINE
BUILDING
2
TOOLS!
OF THE TRADE
BUILDING!
THE PIPELINE
COMMIT BUILD TEST DEPLOY
master
6ebb017
COMMIT BUILD TEST DEPLOY
master
featureA
6ebb017
COMMIT BUILD TEST DEPLOY
master
featureA
6ebb017
PULL REQUEST
COMMIT BUILD TEST DEPLOY
master
featureA
6ebb017 9046c48
COMMIT BUILD TEST DEPLOY
master
featureA
6ebb017 9046c48
BUILD TEST DEPLOY
COMMIT BUILD TEST DEPLOY
1. Checkout out master branch
COMMIT BUILD TEST DEPLOY
1. Checkout out master branch
2. Generate parameters.ini for each environment
COMMIT BUILD TEST DEPLOY
1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
COMMIT BUILD TEST DEPLOY
1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
COMMIT BUILD TEST DEPLOY
—optimizer-­‐autoload  —prefer-­‐dist
1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
COMMIT BUILD TEST DEPLOY
1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
4. Run PHPUnit tests
COMMIT BUILD TEST DEPLOY
1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
4. Run PHPUnit tests
5. Fetch JS dependencies
COMMIT BUILD TEST DEPLOY
1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
4. Run PHPUnit tests
5. Fetch JS dependencies
6. Generate assets (JS, CSS)
COMMIT BUILD TEST DEPLOY
1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
4. Run PHPUnit tests
5. Fetch JS dependencies
6. Generate assets (JS, CSS)
7. Generate artifact (zip file)
COMMIT BUILD TEST DEPLOY
1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
4. Run PHPUnit tests
5. Fetch JS dependencies
6. Generate assets (JS, CSS)
7. Generate artifact (zip file)
COMMIT BUILD TEST DEPLOY
discard unneeded files
1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
4. Run PHPUnit tests
5. Fetch JS dependencies
6. Generate assets (JS, CSS)
7. Generate artifact (zip file)
COMMIT BUILD TEST DEPLOY
1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
4. Run PHPUnit tests
5. Fetch JS dependencies
6. Generate assets (JS, CSS)
7. Generate artifact (zip file)
8. Upload artifact to S3
COMMIT BUILD TEST DEPLOY
1. Checkout out master branch
2. Generate parameters.ini for each environment
3. Fetch PHP dependencies
4. Run PHPUnit tests
5. Fetch JS dependencies
6. Generate assets (JS, CSS)
7. Generate artifact (zip file)
8. Upload artifact to S3
COMMIT BUILD TEST DEPLOYbuild.xml
<?xml  version="1.0"  encoding="UTF-­‐8"  ?>  
<project  name="Time  Out  Miyagi"  description="Time  Out  Website  V4"  default="build"    
!
    <!-­‐-­‐  Install  PHP  dependencies  (composer)  -­‐-­‐>  
    <target  name="build:php-­‐dependencies"  depends="build:params">  
        <echo  msg="Installing  PHP  dependencies  (composer  install)"  />  
        <exec  command="/opt/composer/composer.phar  install  -­‐-­‐optimize-­‐autoloader  -­‐-­‐prefer-­‐dist"    
                    logoutput="true"    
                    checkreturn=“true"  
        />  
    </target>  
!
    <!-­‐-­‐  Test  JS  &  Generate  JS/CSS  assets  (grunt)  -­‐-­‐>  
    <target  name="build:frontend-­‐dependencies">  
        <echo  msg="Test  JS  and  Generate  JS/CSS  assets  (grunt)"  />  
        <exec  command="npm  install"    
                    logoutput="true"  checkreturn="true"  dir="./web-­‐src"  />  
        <exec  command="xvfb-­‐run  grunt  -­‐-­‐env=dist"    
                    logoutput="true"  checkreturn="true"  dir="./web-­‐src"  />      
    </target>  
     
</project>
build.xml
COMMIT BUILD TEST DEPLOY
Split into two jobs in Jenkins
1. Test deployment script
2. Run automated tests
COMMIT BUILD TEST DEPLOY
TEST DEPLOYMENT SCRIPT
COMMIT BUILD TEST DEPLOY
TEST DEPLOYMENT SCRIPT
1. SSH to QA server
COMMIT BUILD TEST DEPLOY
TEST DEPLOYMENT SCRIPT
1. SSH to QA server
2. Fetch artifact from S3
COMMIT BUILD TEST DEPLOY
TEST DEPLOYMENT SCRIPT
1. SSH to QA server
2. Fetch artifact from S3
3. Unzip artifact
COMMIT BUILD TEST DEPLOY
TEST DEPLOYMENT SCRIPT
1. SSH to QA server
2. Fetch artifact from S3
3. Unzip artifact
4. Set right permissions
COMMIT BUILD TEST DEPLOY
TEST DEPLOYMENT SCRIPT
1. SSH to QA server
2. Fetch artifact from S3
3. Unzip artifact
4. Set right permissions
5. Update virtual host
COMMIT BUILD TEST DEPLOY
TEST DEPLOYMENT SCRIPT
1. SSH to QA server
2. Fetch artifact from S3
3. Unzip artifact
4. Set right permissions
5. Update virtual host
6. Reload PHP-FPM and nginx
-­‐-­‐-­‐  
-­‐  hosts:  qa:beta:prod  
    sudo:  true  
    vars:  
        amazon_bucket  :  "releases-­‐miyagi"  
        base_dir            :  "/var/www/v4.timeout.com"  
        project_dir      :  "{{  base_dir  }}/{{  git_hash  }}"  
        web_dir              :  "{{  project_dir  }}/web"  
    tasks:  
        -­‐  name:  Fetch  artefact  from  S3  
            s3:  bucket={{  amazon_bucket  }}  aws_access_key={{  amazon_key  }}  
aws_secret_key={{  amazon_secret  }}  object={{  git_hash  }}.gzip  dest="{{  project_dir  
}}/{{artefact_zip}}"  mode="get"  
!
        -­‐  name:  Uncompress  zip  file  
            command:  chdir="{{  project_dir  }}"  tar  -­‐xf  {{  artefact_zip  }}  
!
        -­‐  name:  Apply  right  permissions  to  project  root  
            file:  dest="{{  project_dir  }}"  state=directory  mode=0755  group=nginx  
owner=nginx  recurse=true  
          
        -­‐  name:  Restart  PHP-­‐FPM    
            service:  name=php-­‐fpm  state=reloaded
deploy.yml
COMMIT BUILD TEST DEPLOY
RUN AUTOMATED TESTS
COMMIT BUILD TEST DEPLOY
RUN AUTOMATED TESTS
1. Checkout 9046c48
COMMIT BUILD TEST DEPLOY
RUN AUTOMATED TESTS
1. Checkout 9046c48
2. Fetch PHP dependencies
COMMIT BUILD TEST DEPLOY
RUN AUTOMATED TESTS
1. Checkout 9046c48
2. Fetch PHP dependencies
3. Run behat against QA box
COMMIT BUILD TEST DEPLOY
Split into two jobs in Jenkins
1. Run deployment script (production)
2. Run automated tests agains production
BUILD DEPLOY
AUTOMATED!
TESTS
6ebb017
6ebb017
QA
6ebb017
QA
DEPLOY
AUTOMATED!
TESTS
6ebb017
PROD
6ebb017
PROD
6ebb017
6ebb017.zip
BUILD DEPLOY
AUTOMATED!
TESTS
6ebb017
6ebb017
QA
6ebb017
QA
DEPLOY
AUTOMATED!
TESTS
6ebb017
STAGING
6ebb017
STAGING
DEPLOY
AUTOMATED!
TESTS
6ebb017
PROD
6ebb017
PROD
6ebb017
6ebb017.zip
SAFETY NET
PARAMETERS.INI!
DILEMMA
1. One parameters.ini per environment
2. Inject environment specific parameters on virtual host
http://symfony.com/doc/current/cookbook/configuration/external_parameters.html
TIME
BUILD DEPLOY
AUTOMATED!
TESTS
~3 mins ~30 secs ~5 mins
VISIBILITY
EVERYBODY IN THE TEAM SHOULD BE
AWARE OF THE STATUS OF THE PIPELINE
BUILD DEPLOY
AUTOMATED!
TESTS
9046c48
9046c48
QA
9046c48
QA
9046c48
BUILD DEPLOY
AUTOMATED!
TESTS
6ebb017
6ebb017
QA
6ebb017
QA
6ebb017
BUILD DEPLOY
AUTOMATED!
TESTS
b0b325
b0b325
QA
b0b325
QA
b0b325
BUILD DEPLOY
AUTOMATED!
TESTS
99e6d6
99e6d6
QA
99e6d6
QA
99e6d6
TRACEABILITY
EVERYBODY SHOULD KNOW WHAT VERSION
IS DEPLOYED IN WHICH ENVIRONMENT
curl  -­‐I  http://www.timeout.com/las-­‐vegas  
!
HTTP/1.1  200  OK  
Server:  nginx/1.4.7  
Vary:  Accept-­‐Encoding  
Cache-­‐Control:  no-­‐cache  
Content-­‐Type:  text/html;  charset=UTF-­‐8  
Date:  Fri,  19  Sep  2014  06:07:29  GMT  
Transfer-­‐Encoding:  chunked  
Access-­‐Control-­‐Allow-­‐Origin:  http://media.timeout.com  
Connection:  Keep-­‐Alive  
X-­‐TIMEOUT-­‐V:  d645127afb423e543d90ab5a7b8eae94f248b137  
X-­‐Powered-­‐By:  PHP/5.5.14
VERSION NUMBER
https://github.com/symfony/symfony/commits/a469c56
ROLLING!
BACK
YOU’LL NEED!
ALLIES
YOUR ALLIES
DEV TEAM
YOUR ALLIES
DEV TEAM
QA TEAM
YOUR ALLIES
DEV TEAM
QA TEAM
DEVOPS
YOUR ALLIES
DEV TEAM
QA TEAM
DEVOPS
PRODUCT
YOUR ALLIES
DEV TEAM
QA TEAM
DEVOPS
PRODUCT
THE BRASS
YOUR ALLIES
BEFORE AFTER&
3
ONE QA BOX PER FEATURE
SAME QA BOX FOR EVERYONE
BEFORE
AFTER
~ !WEEKS FROM DEVELOPMENT TO RELEASE
~ !DAYS FROM DEVELOPMENT TO RELEASE
BEFORE
AFTER
5 PEOPLE TO RELEASE TO PRODUCTION
1 PERSON TO RELEASE TO PRODUCTION
BEFORE
AFTER
~30 MINUTES TO RUN DEPLOYMENT SCRIPT
~30 SECONDS TO RUN DEPLOYMENT SCRIPT
BEFORE
AFTER
RELEASING WAS AN EVENT
RELEASING WAS A NO EVENT
BEFORE
AFTER
@loalf

One commit, one release. Continuously delivering a Symfony project.

  • 1.
  • 2.
    Javier López @loalf SeniorPlatform Engineer @Rightster! formerly Software Architect @TimeOut Certified Symfony Developer Zend Certified PHP Co-organizer @desymfony
  • 3.
  • 4.
    AGENDA CONTINUOUS DELIVERY INA NUTSHELL BUILDING THE PIPELINE BEFORE AND AFTER 1 2 3
  • 5.
  • 6.
    COMMIT BUILD TESTDEPLOY When and how a change in your code is going to trigger the pipeline
  • 7.
    COMMIT BUILD TESTDEPLOY Checks the integerity of your code. Produces an artifact.
  • 8.
    COMMIT BUILD TESTDEPLOY Checks the integerity of your project. ! After this step you should be 100% confident that you could deploy your artifact.
  • 9.
    COMMIT BUILD TESTDEPLOY Does what it says in the tin. Triggering this step should be “a click away”. Manual
  • 10.
    COMMIT BUILD TESTDEPLOY Does what it says in the tin. Triggering this step should be “a click away”. Manual
  • 11.
  • 12.
  • 13.
  • 19.
  • 20.
    COMMIT BUILD TESTDEPLOY master 6ebb017
  • 21.
    COMMIT BUILD TESTDEPLOY master featureA 6ebb017
  • 22.
    COMMIT BUILD TESTDEPLOY master featureA 6ebb017 PULL REQUEST
  • 23.
    COMMIT BUILD TESTDEPLOY master featureA 6ebb017 9046c48
  • 24.
    COMMIT BUILD TESTDEPLOY master featureA 6ebb017 9046c48 BUILD TEST DEPLOY
  • 25.
  • 26.
    1. Checkout outmaster branch COMMIT BUILD TEST DEPLOY
  • 27.
    1. Checkout outmaster branch 2. Generate parameters.ini for each environment COMMIT BUILD TEST DEPLOY
  • 28.
    1. Checkout outmaster branch 2. Generate parameters.ini for each environment 3. Fetch PHP dependencies COMMIT BUILD TEST DEPLOY
  • 29.
    1. Checkout outmaster branch 2. Generate parameters.ini for each environment 3. Fetch PHP dependencies COMMIT BUILD TEST DEPLOY —optimizer-­‐autoload  —prefer-­‐dist
  • 30.
    1. Checkout outmaster branch 2. Generate parameters.ini for each environment 3. Fetch PHP dependencies COMMIT BUILD TEST DEPLOY
  • 31.
    1. Checkout outmaster branch 2. Generate parameters.ini for each environment 3. Fetch PHP dependencies 4. Run PHPUnit tests COMMIT BUILD TEST DEPLOY
  • 32.
    1. Checkout outmaster branch 2. Generate parameters.ini for each environment 3. Fetch PHP dependencies 4. Run PHPUnit tests 5. Fetch JS dependencies COMMIT BUILD TEST DEPLOY
  • 33.
    1. Checkout outmaster branch 2. Generate parameters.ini for each environment 3. Fetch PHP dependencies 4. Run PHPUnit tests 5. Fetch JS dependencies 6. Generate assets (JS, CSS) COMMIT BUILD TEST DEPLOY
  • 34.
    1. Checkout outmaster branch 2. Generate parameters.ini for each environment 3. Fetch PHP dependencies 4. Run PHPUnit tests 5. Fetch JS dependencies 6. Generate assets (JS, CSS) 7. Generate artifact (zip file) COMMIT BUILD TEST DEPLOY
  • 35.
    1. Checkout outmaster branch 2. Generate parameters.ini for each environment 3. Fetch PHP dependencies 4. Run PHPUnit tests 5. Fetch JS dependencies 6. Generate assets (JS, CSS) 7. Generate artifact (zip file) COMMIT BUILD TEST DEPLOY discard unneeded files
  • 36.
    1. Checkout outmaster branch 2. Generate parameters.ini for each environment 3. Fetch PHP dependencies 4. Run PHPUnit tests 5. Fetch JS dependencies 6. Generate assets (JS, CSS) 7. Generate artifact (zip file) COMMIT BUILD TEST DEPLOY
  • 37.
    1. Checkout outmaster branch 2. Generate parameters.ini for each environment 3. Fetch PHP dependencies 4. Run PHPUnit tests 5. Fetch JS dependencies 6. Generate assets (JS, CSS) 7. Generate artifact (zip file) 8. Upload artifact to S3 COMMIT BUILD TEST DEPLOY
  • 38.
    1. Checkout outmaster branch 2. Generate parameters.ini for each environment 3. Fetch PHP dependencies 4. Run PHPUnit tests 5. Fetch JS dependencies 6. Generate assets (JS, CSS) 7. Generate artifact (zip file) 8. Upload artifact to S3 COMMIT BUILD TEST DEPLOYbuild.xml
  • 39.
    <?xml  version="1.0"  encoding="UTF-­‐8" ?>   <project  name="Time  Out  Miyagi"  description="Time  Out  Website  V4"  default="build"     !    <!-­‐-­‐  Install  PHP  dependencies  (composer)  -­‐-­‐>      <target  name="build:php-­‐dependencies"  depends="build:params">          <echo  msg="Installing  PHP  dependencies  (composer  install)"  />          <exec  command="/opt/composer/composer.phar  install  -­‐-­‐optimize-­‐autoloader  -­‐-­‐prefer-­‐dist"                        logoutput="true"                        checkreturn=“true"          />      </target>   !    <!-­‐-­‐  Test  JS  &  Generate  JS/CSS  assets  (grunt)  -­‐-­‐>      <target  name="build:frontend-­‐dependencies">          <echo  msg="Test  JS  and  Generate  JS/CSS  assets  (grunt)"  />          <exec  command="npm  install"                        logoutput="true"  checkreturn="true"  dir="./web-­‐src"  />          <exec  command="xvfb-­‐run  grunt  -­‐-­‐env=dist"                        logoutput="true"  checkreturn="true"  dir="./web-­‐src"  />          </target>       </project> build.xml
  • 40.
    COMMIT BUILD TESTDEPLOY Split into two jobs in Jenkins 1. Test deployment script 2. Run automated tests
  • 41.
    COMMIT BUILD TESTDEPLOY TEST DEPLOYMENT SCRIPT
  • 42.
    COMMIT BUILD TESTDEPLOY TEST DEPLOYMENT SCRIPT 1. SSH to QA server
  • 43.
    COMMIT BUILD TESTDEPLOY TEST DEPLOYMENT SCRIPT 1. SSH to QA server 2. Fetch artifact from S3
  • 44.
    COMMIT BUILD TESTDEPLOY TEST DEPLOYMENT SCRIPT 1. SSH to QA server 2. Fetch artifact from S3 3. Unzip artifact
  • 45.
    COMMIT BUILD TESTDEPLOY TEST DEPLOYMENT SCRIPT 1. SSH to QA server 2. Fetch artifact from S3 3. Unzip artifact 4. Set right permissions
  • 46.
    COMMIT BUILD TESTDEPLOY TEST DEPLOYMENT SCRIPT 1. SSH to QA server 2. Fetch artifact from S3 3. Unzip artifact 4. Set right permissions 5. Update virtual host
  • 47.
    COMMIT BUILD TESTDEPLOY TEST DEPLOYMENT SCRIPT 1. SSH to QA server 2. Fetch artifact from S3 3. Unzip artifact 4. Set right permissions 5. Update virtual host 6. Reload PHP-FPM and nginx
  • 48.
    -­‐-­‐-­‐   -­‐  hosts: qa:beta:prod      sudo:  true      vars:          amazon_bucket  :  "releases-­‐miyagi"          base_dir            :  "/var/www/v4.timeout.com"          project_dir      :  "{{  base_dir  }}/{{  git_hash  }}"          web_dir              :  "{{  project_dir  }}/web"      tasks:          -­‐  name:  Fetch  artefact  from  S3              s3:  bucket={{  amazon_bucket  }}  aws_access_key={{  amazon_key  }}   aws_secret_key={{  amazon_secret  }}  object={{  git_hash  }}.gzip  dest="{{  project_dir   }}/{{artefact_zip}}"  mode="get"   !        -­‐  name:  Uncompress  zip  file              command:  chdir="{{  project_dir  }}"  tar  -­‐xf  {{  artefact_zip  }}   !        -­‐  name:  Apply  right  permissions  to  project  root              file:  dest="{{  project_dir  }}"  state=directory  mode=0755  group=nginx   owner=nginx  recurse=true                    -­‐  name:  Restart  PHP-­‐FPM                service:  name=php-­‐fpm  state=reloaded deploy.yml
  • 49.
    COMMIT BUILD TESTDEPLOY RUN AUTOMATED TESTS
  • 50.
    COMMIT BUILD TESTDEPLOY RUN AUTOMATED TESTS 1. Checkout 9046c48
  • 51.
    COMMIT BUILD TESTDEPLOY RUN AUTOMATED TESTS 1. Checkout 9046c48 2. Fetch PHP dependencies
  • 52.
    COMMIT BUILD TESTDEPLOY RUN AUTOMATED TESTS 1. Checkout 9046c48 2. Fetch PHP dependencies 3. Run behat against QA box
  • 53.
    COMMIT BUILD TESTDEPLOY Split into two jobs in Jenkins 1. Run deployment script (production) 2. Run automated tests agains production
  • 54.
  • 55.
  • 56.
  • 57.
    1. One parameters.iniper environment 2. Inject environment specific parameters on virtual host http://symfony.com/doc/current/cookbook/configuration/external_parameters.html
  • 58.
  • 59.
  • 60.
    VISIBILITY EVERYBODY IN THETEAM SHOULD BE AWARE OF THE STATUS OF THE PIPELINE
  • 61.
    BUILD DEPLOY AUTOMATED! TESTS 9046c48 9046c48 QA 9046c48 QA 9046c48 BUILD DEPLOY AUTOMATED! TESTS 6ebb017 6ebb017 QA 6ebb017 QA 6ebb017 BUILDDEPLOY AUTOMATED! TESTS b0b325 b0b325 QA b0b325 QA b0b325 BUILD DEPLOY AUTOMATED! TESTS 99e6d6 99e6d6 QA 99e6d6 QA 99e6d6
  • 63.
    TRACEABILITY EVERYBODY SHOULD KNOWWHAT VERSION IS DEPLOYED IN WHICH ENVIRONMENT
  • 64.
    curl  -­‐I  http://www.timeout.com/las-­‐vegas  ! HTTP/1.1  200  OK   Server:  nginx/1.4.7   Vary:  Accept-­‐Encoding   Cache-­‐Control:  no-­‐cache   Content-­‐Type:  text/html;  charset=UTF-­‐8   Date:  Fri,  19  Sep  2014  06:07:29  GMT   Transfer-­‐Encoding:  chunked   Access-­‐Control-­‐Allow-­‐Origin:  http://media.timeout.com   Connection:  Keep-­‐Alive   X-­‐TIMEOUT-­‐V:  d645127afb423e543d90ab5a7b8eae94f248b137   X-­‐Powered-­‐By:  PHP/5.5.14 VERSION NUMBER
  • 65.
  • 66.
  • 67.
  • 68.
  • 69.
  • 70.
  • 71.
  • 72.
  • 73.
  • 74.
  • 75.
    ONE QA BOXPER FEATURE SAME QA BOX FOR EVERYONE BEFORE AFTER
  • 76.
    ~ !WEEKS FROMDEVELOPMENT TO RELEASE ~ !DAYS FROM DEVELOPMENT TO RELEASE BEFORE AFTER
  • 77.
    5 PEOPLE TORELEASE TO PRODUCTION 1 PERSON TO RELEASE TO PRODUCTION BEFORE AFTER
  • 78.
    ~30 MINUTES TORUN DEPLOYMENT SCRIPT ~30 SECONDS TO RUN DEPLOYMENT SCRIPT BEFORE AFTER
  • 79.
    RELEASING WAS ANEVENT RELEASING WAS A NO EVENT BEFORE AFTER
  • 80.