UNIT-IV

Software Builds in DevOps

• Definition: The process of transforming code from one form
to another, including compiling, linting, and testing.

• Phases:
- Compilation: Converting source code to native or
bytecode.
- Linting: Static analysis for code errors, originated from
Unix's Lint tool.
- Unit Testing: Running code in controlled environments to
catch errors early.
- Artifacts: Generating deployable code.
 Build Systems and Their Types
• Software build tools are used to automate the
creation of executable applications from source code.
• They script or automate a variety of tasks, such as
downloading dependencies, compiling and packaging
code, running tests, and deployment.
• They can be triggered through the command line,
inside an IDE, or by continuous integration tools after
checking the code out of a repository and onto a build
machine.
• Different build systems exist for different languages.
Usually, build systems use either a domain-specific
language or XML to specify a build.
 Build Systems and Their Types
• Different build systems for different languages:
- Java: Maven, Gradle, Ant.
- C/C++: Make, Autotools.
- JavaScript: Grunt.
- Clojure: Leiningen.
- Ruby: Rake.
Challenges of standardizing build tools across an organization:
• - Organizations may use multiple systems due to varying
requirements.
Example: Grunt is more efficient for JavaScript, while Maven suits
Java projects.
• Rule of thumb: Developers should be able to build code locally
with minimal surprises.
The Role of Jenkins in Build Management
• Jenkins: A popular build server forked from Hudson,
created by Kohsuke Kawaguchi.
• Jenkins is versatile, supporting builds in Java and
other languages.
• Setup: Easy to install (e.g., on Fedora via dnf),
accessible at http://localhost:8080.
• Job Types:
- Freestyle project: Basic job type with build steps
like running shell commands (e.g., fortune or date).
- Other job types allow more complex setups (e.g.,
chaining jobs, pipelines).
• Jenkins builds and tests our software projects which continuously
making it easier for developers to integrate changes to the project,
and making it easier for users to obtain a fresh build.
• It also allows us to continuously deliver our software by
integrating with a large number of testing and deployment
technologies.
• Possible steps executed by Jenkins are for example:
• Perform a software build using a build system like Gradle or Maven
Apache
• Execute a shell script
• Archive a build result
• Running software tests
Work Flow:
Advantages of Jenkins

• It is an open source tool.

• It is free of cost.
• It does not require additional installations or components. Means it is easy to
install.
• Easily configurable.
• It supports 1000 or more plugins to ease your work. If a plugin does not exist,
you can write the script for it and share with community.
• It is built in java and hence it is portable.
• It is platform independent. It is available for all platforms and different
operating systems. Like OS X, Windows or Linux.
• Easy support, since it open source and widely used.
• Jenkins also supports cloud based architecture so that we can deploy Jenkins in
cloud based platforms.
Disadvantages of Jenkins

• Its interface is out dated and not user friendly

compared to current user interface trends.
• Not easy to maintain it because it runs on a
server and requires some skills as server
administrator to monitor its activity.
• CI regularly breaks due to some small setting
changes. CI will be paused and therefore requires
some developer's team attention.
 Managing Build Dependencies
• Automatic dependency fetching: Maven, Grunt, and Go
can automatically fetch dependencies if not present.
• Challenges with C/C++ builds:
- Use of Autotools: Adapts to available dependencies, but
this flexibility can cause issues in enterprise environments.
- RPM (Red Hat Package Manager)spec files: Lists
dependencies and steps to build C/C++ software
consistently.
• Importance of clear dependency management to avoid
missing functionality on production servers.
 Jenkins plugins
• Jenkins has a plugin system to add
functionality to the build server. There are
many different plugins available, and they can
be installed from within the Jenkins web
interface.
• Many of them can be installed without even
restarting Jenkins. This screenshot shows a list
of some of the available plugins:
 File system layout
• In DevOps, the file system layout refers to how files and directories
are organized and structured within a server or system.
• A well-organized file system is crucial for efficient operations,
version control, CI/CD pipelines, and maintenance.
• Here's an overview of a typical file system layout in DevOps
environments:
• 1. Project Repositories
• /srv/repos/
Stores code repositories, typically managed with Git, Subversion
(SVN), or other version control systems.
– project-1/
– project-2/
– project-n/
2. Build Directories
/var/builds/
Contains all build-related files, including
compiled binaries and logs generated by CI
tools like Jenkins or GitLab CI.
– project-1/
– project-2/
– project-n/
– build_logs/ (stores logs related to build status)
3. Deployment Directories
/var/deploy/
Includes files and configurations for the current
and previous releases of applications. Helps with
rollbacks and maintaining the deployment history.
– project-1/
– project-2/
– releases/ (contains versions of the application
deployed on the server)
– artifacts/ (stores packaged versions ready for
deployment)
4. Configuration Files
/etc/
Holds configuration files for the system, build tools, and
deployed applications.
– jenkins/ (configuration for Jenkins)
– nginx/ (configuration for Nginx server)
– application-config/ (project-specific configurations)

5. Containerization Layout
/var/lib/docker/
Used for container storage when working with Docker. It stores
images, containers, volumes, and other Docker-related data.
– containers/
– images/
– volumes/
6. Logs and Monitoring
/var/log/
Stores log files, crucial for debugging, performance
monitoring, and audits.
– syslog (system logs)
– jenkins/ (CI tool logs)
– nginx/ (web server logs)
– application/ (custom logs from applications)
7. Package/Artifact Storage
/var/artifacts/
Houses compiled binaries, Docker images, and other
deployable artifacts produced by the build process.
– docker/ (Docker images)
– maven/ (Maven-built JAR files)
8. Scripts
/usr/local/bin/
Contains executable scripts or binaries related to the DevOps
environment, including deployment scripts, monitoring scripts, or helper
tools.
deployment-scripts/ (automated scripts for deployments)
cleanup-scripts/ (scripts for managing resources, cleaning old builds)
9. Temporary Files
/tmp/
Temporary files used during builds or deployments. These files are usually
removed after a certain period or at the end of the build process.
ci_tmp/ (temporary files from CI/CD tools)
build_tmp/ (files created during build processes)
10. Virtual Environments and Dependency Management
/var/virtualenvs/ OR
/home/devops/.venvs/
For projects using Python or other interpreted languages, virtual
environments or environment management files are stored here.
project-1-env/
12. Shared Libraries
/usr/local/lib/
Stores shared libraries, including dependencies
and shared components used across multiple
applications or services.
custom-libraries/
13. CI/CD Tool Configurations
/etc/jenkins/ (or the respective directory for the CI
tool you are using)
Stores the configuration files for your CI/CD tools.
jobs/ (defines the pipeline configurations)
plugins/ (manages the plugins used in CI/CD tools)
Considerations:
• Security:
Sensitive files (e.g., secrets, keys, and passwords)
should be stored securely, often in a separate directory
with restricted permissions or using a secrets manager.
• Access Control:
Ensure proper file and directory permissions to restrict
access to critical parts of the system, especially in a
shared environment.
• Consistency:
Keep the layout consistent across environments (dev,
test, prod) to ensure smooth transitions and
deployment processes.
 The host server
• The build server is usually a pretty important
machine for the organization.
• Builds shouldn't take too long, so you will need a
server with good specifications for the build server
—with lots of disk space, processor cores, and RAM.
• The build server also has a kind of social aspect: it is
here that the code of many different people and
roles integrates properly for the first time.
• Machines are cheaper than people, so don't let this
particular machine be the area you save money on.
 Build Slaves and Build Pipelines
• Build slaves reduce queues by distributing
workloads to different machines.
• Example: Linux master, Windows/Mac slaves for
cross-platform builds.
• Upstream and downstream jobs:
- Upstream: Initial job that triggers further builds.
-Downstream: Jobs triggered after upstream
completion.
• Pipelines: Chaining jobs for complex workflows,
using plugins like Multijob and Workflow
 Software on the host
• Depending on the complexity of your builds,
you might need to install many different types
of build tool on your build server.
• Remember that Jenkins is mostly used to
trigger builds, not perform the builds
themselves.
• Most of the build systems are available in the
distribution repositories, so it's very
convenient to install them from there.
 Triggers
• You can either use a timer to trigger builds, or you can poll the code
repository for changes and build if there were changes.

• It can be useful to use both methods at the same time

• Git repository polling can be used most of the time so that every
check in triggers a build.

• Nightly builds can be triggered, which are more stringent than

continuous builds, and thus take a longer time. Since these builds
happen at night when nobody is supposed to work, it doesn't matter
if they are slow.

• An upstream build can trigger a downstream build. You can also let
the successful build of one job trigger another job.
 Job chaining and build pipelines

• It's often useful to be able to chain jobs

together. In its simplest form, this works by
triggering a second job in the event that the
first job finishes successfully.

• Sometimes, a nicer visualization of the build

steps as well as greater control over the
details of the chain is desired.
• In Jenkins terminology, the first build in a
chain is called the upstream build, and the
second one is called the downstream build.

• Such a build chain is often called a pipeline or

workflow.

• There are many plugins that create improved

pipelines for Jenkins, and the fact that there
are several shows that there is indeed a great
desire for improvements in this area.
 Alternative Build Servers
• Jenkins alternatives:
- Travis CI: Popular for open source projects.
- Buildbot: Python-based, customizable.
- Go server (by ThoughtWorks): CI/CD focus.
- Bamboo (by Atlassian): Integrates with Jira for
agile teams.
- GitLab: Built-in CI/CD tools, more integrated
infrastructure-as-code approach.
• Each tool has its strengths; Jenkins excels in
flexibility and plugin availability.
 Build servers and infrastructure as code

• Infrastructure as Code is the management of infrastructure

(networks, virtual machines, load balancers, and connection
topology) in a descriptive model, using the same versioning as
DevOps team uses for source code”.

• Put simply, IAC is the practice of managing the infrastructure

using code. It uses machine-readable files to configure and
manage the required IT infrastructure.
The 4 best practices of IaC include:
• Manage infrastructure through source control
to offer detailed audit tracks for modifications.
• Testing the infrastructure for any flaws or
errors using unit testing, integration testing,
and functional testing.
• Allow collaboration around infrastructure
configuration and provisioning.
• Avoid written documentation as the code
itself will document the state of the machine.
 Build phases

• One of the principal benefits of the Maven build tool is that

it standardizes builds.
• This is very useful for a large organization, since it won't
need to invent its own build standards.
• Other build tools are usually much more lax regarding how
to implement various build phases.
• The rigidity of Maven has its pros and cons. Sometimes,
people who get started with Maven reminisce about the
freedom that could be had with tools such as Ant.
• You can implement these build phases with any tool, but it's
harder to keep the habit going when the tool itself doesn't
enforce the standard order: building, testing, and
deploying.
 Alternative build servers
• While Jenkins appears to be pretty dominant in the build
server scene in my experience, it is by no means alone.
• Travis CI is a hosted solution that is popular among open
source projects.
• Buildbot is a buildserver that is written in, and
configurable with, Python.
• The Go server is another one, from ThoughtWorks.
• Bamboo is an offering from Atlassian. GitLab also supports
build server functionality now.
• Do shop around before deciding on which build server
works best for you. When evaluating different solutions,
be aware of attempts at vendor lock-in.
Collating Software Quality Metrics with Jenkins

• Jenkins collects and visualizes quality metrics

(e.g., Java unit test results).
• Advanced option: SonarQube integration for
deeper code analysis and visualizations.
• SonarQube highlights code issues like bugs,
vulnerabilities, and code smells.
• Downside: Running Sonar builds during nightly
builds to avoid slowing regular build cycles.