Testing SAP:
Modern Testing
Methodologies
Ethan Jewett
But first!
Whats the problem?
Enterprise project trade-offs
Software development
trade-offs
Client requests
Build Faster
And produce more Value
Cheaper
with less Risk
Remember
Are there hidden
assumptions?
Yes.
Methodology
People and Culture
Technology
Any other assumptions?
The topic
Over the next hour and a half, well talk
about how testing methodologies,
techniques, and technologies can help us
Change these underlying assumptions.
Enable modern project methodologies.
Control the basic project trade-off.
The focus
The goal
More value
Faster
With less risk
(and more fun)
For example
V-Model methodology
http://en.wikipedia.org/wiki/File:V-model.JP
More parallel, less repetition
http://en.wikipedia.org/wiki/File:V-model.JP
Faster
http://en.wikipedia.org/wiki/File:V-model.JP
More iterative
http://en.wikipedia.org/wiki/File:V-model.JP
How do we get there?
Modern methodology
Modernquality assurance and testing
Leading practice in custom
development, open source, and agile
projects
Test driven development (TDD), behavior driven
development (BDD), readable tests, executable
requirements, continuous integration
Overview of the SAP test automation
technologies
Road we're going to take through testing
SAP
Leading Practice
Context
Questions and problems
Approaches
Tools
Leading Practice
Context
Questions and problems
Approaches
Tools
Context
The context of modern software
development is above all open,
agile and decentralized.
Open source projects are usually the
furthest along this path, and as such
these projects use (or are forced to
invent) tools that fit their processes.
Openness
Requirements, specifications, issues
reports, test cases, source code, and
project tools and metrics are readily
open and available to all participants
in the project.
(Not necessarily the same as open
source.)
Open Collaboration within Corporations Using Software Forges http://www.riehle.org/publications/2009/open-collaboration-within-corporations-using
Agility
The values of the Agile Manifesto:
Individuals and interactions over
processes and tools
Working software over comprehensive
documentation
Customer collaboration over contract
negotiation
Responding to change over following a
plan
http://agilemanifesto.org/
Decentralization
Open-source projects are decentralized
by nature, but most organizations
today have some element of
decentralization
Challenges:
Quality control
Communication
Maintaining commitment
Leading Practice
Context
Questions and problems
Approaches
Tools
Questions and problems
How do we
enforce
disciplined
Do we use unit,
testing?
functional,
integration, or
acceptance
testing? Or all of
them?
How do we ensure
that the build
complies with the
tests?
How do we track
100s or 1000s of
bugs (including
How do we duplicates)?
make tests
How does
relevant?
testing
integrate with
issue
How do we
tracking?
determine the
Manual or
cause of test
automatic
failure?
testing?
Leading Practice
Context
Questions and problems
Approaches
Tools
Approaches
Test Coverage
Test Automation
Test Driven Development
Behavior Driven Development
Spec as Test (the test/spec
convergence)
Exploratory Testing
Continuous Integration
Test coverage
The percentage of code or
development that is tested.
In code, this might be measured by
determining if every branch of code
can result in a failing test condition
(Branch coverage)
Wikipedia http://en.wikipedia.org/wiki/Code_coverage
Test automation
Accepted as gospel modern dev
communities
Regardless of how good they are at
testing, accept that they should
automate as much as possible
Can lead to ignoring non-automatable
testing
In the SAP world we havent even
accepted that full test coverage is
desirable, much less automation
Test automation pushback http://railspikes.com/2008/7/11/testing-is-overrated
http://michaelfeathers.typepad.com/michael_feathers_blog/200
8/06/the-flawed-theo.html
Automated testing story on SDN (never completed) https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/4103
A note on the following
slides
Im using bowling as an example because
Im copying from the internet. I dont
know much about bowling, so the
examples are probably wrong. My
assumption is that a game is usually 10
frames of two balls per frame. If the last
frame is a spare or a strike, you get at
most two additional frames.
Or something like that.
Test Driven Development
(TDD)
Technical specification
TDD Example
require test/unit
require test/unit/assertions
require bowling
class TC_Bowling <
Test::Unit::TestCase
def setup
bowling = Bowling.new
end
def gutter_game
20.times { bowling.hit(0) }
assert bowling.score == 0
end
end
Behavior Driven Development
(BDD)
Focus on the external behavior of the
code
Abstract away implementation
details
Additionally, BDD libraries tend to
codify the best practices of unit
testing and TDD
BDD example
require spec
require bowling
describe Bowling do
it "should score 0 for gutter
game" do
bowling = Bowling.new
20.times { bowling.hit(0) }
bowling.score.should == 0
end
end
Spec as Test or writing features
Recently, the Ruby community has
begun developing testing frameworks
that are even closer to natural
language.
Cucumber
These frameworks wrap TDD and
BDD frameworks and allow for
business-writable, businessreadable, executable test scripts.
Feature example
Feature (visible to the user)
Scenario: Gutter game
Given I have bowled 20 balls
and I have knocked over 0 pins per ball
When I check the score
Then I should have 0 points
Implementation (not visible to user)
Feature example cont.
Note that we can now implement
many more tests with no more
code/implementation:
Scenario:
Perfect game
Given I have bowled 12 balls
and I have knocked over 10 pins per ball
When I check the score
Then I should have 300 points
Scenario: Bad game
Given I have bowled 20 balls
and I have knocked over 2 pins per ball
When I check the score
Then I should have 40 points
Scenario: Lots of spares
Side-by-side
TDD
require test/unit
require test/unit/assertions
require bowling
BDD
require spec/expectations
require bowling
describe Bowling do
class TC_Bowling <
Test::Unit::TestCase
def setup
bowling = Bowling.new
end
def gutter_game
20.times { bowling.hit(0) }
assert bowling.score == 0
end
end
it "should score 0 for gutter
game" do
bowling = Bowling.new
20.times { bowling.hit(0) }
bowling.score.should == 0
end
end
Spec as Test
Scenario: Gutter game
Given I have bowled 20 balls
and I have knocked over 0 pins
per ball
When I view the score
Then I should have 0 points
TDD, BDD, and Test-as-Spec
References
Test Driven Design/Development
http://en.wikipedia.org/wiki/Test-driven
_development
http://www.agiledata.org/essays/tdd.html
Behavior Driven Development
http://behaviour-driven.org/
Spec as test
Cucumber - http://cukes.info/
Rspec - http://rspec.info/
http://www.pragprog.com/titles/achbd/th
e-rspec-book
Exploratory Testing
The practice of trying to break things
Career security for klutzes
Exploratory testing appears informal,
but can be structured and is a very
important aspect of software testing.
Probably the most neglected form of
testing in open source projects
http://www.kohl.ca/blog/archives/000185.html
http://www.kohl.ca/articles/ExploratoryTesting_Musicof
Continuous Integration
The practice of automating not only
your tests but your full commit-buildtest cycle
1. A commit new or changed code
2. Triggers a full system build
3. And an execution of the entire test
suite
Cruisecontrol.rb http://rubyforge.org/projects/cruisecontrolrb
http://cruisecontrolrb.thoughtworks.com/
Hudson https://hudson.dev.java.net/
http://www.softwarebloat.com/2008/11/19/continuous-integratio
n-blueprints-how-to-build-an-army-of-killer-robots-with-hudso
Leading Practice
Context
Questions and problems
Approaches
Tools
Tools
Source version control (svn, git)
Testing libraries (rspec, cucumber,
junit, ABAPunit, jspec)
Continuous Integration tools
(cruisecontrol.rb, Hudson)
Issue tracking (Sourceforge, trac,
Lighthouse, Google Code)
Agile methodologies (Scrum, XP,
etc.)
Modern methodology
Modernquality assurance and testing
Leading practice in custom development,
open source, and agile projects
Test driven development (TDD), behavior driven
development (BDD), readable tests, executable
requirements, continuous integration
Overview of the SAP test automation
technologies
Road we're going to take through testing
SAP
The SAP World
Manual
Unit testing
ABAP Unit
Java
Functional testing
eCATT
ABAP Unit
Modeled on Java Unit
Excellent (if a bit unwieldy) for
traditional unit test automation or
classes
Works well for TDD
ABAP Objects (object-oriented ABAP)
TDD Demo ZBOWLING &
ZBOWLING_TEST
http://help.sap.com/saphelp_nw70ehp1/helpdata/en/a
2/8a1b602e858645b8aac1559b638ea4/frameset.htm
Non-SAP
We can use just about anything via
RFCs
For Web Dynpros, BSPs, or Portal
applications, there are a lot of
options using web-drivers that can
automatically drive browsers
Address in depth in future sessions
Modern methodology
Modernquality assurance and testing
Leading practice in custom development,
open source, and agile projects
Test driven development (TDD), behavior driven
development (BDD), readable tests, executable
requirements, continuous integration
Overview of the SAP test automation
technologies
Road we're going to take through
testing SAP
The Road to the Future
Developing techniques to support
agile, open, decentralized
development in an SAP landscape
Using SAP tools and 3rd party tools
Shorter, more focused and hands-on
sessions
Start with this
http://en.wikipedia.org/wiki/File:V-model.JP
Get more parallel, less
repetitive
Spec = Test
http://en.wikipedia.org/wiki/File:V-model.JP
Get faster
Test
Automati
on
http://en.wikipedia.org/wiki/File:V-model.JP
Get iterative
Continuo
us
Integrati
on
http://en.wikipedia.org/wiki/File:V-model.JP
And end up here
(or somewhere similar)
http://en.wikipedia.org/wiki/File:Scrum_pro
Thanks
General references for
inclusion
Open Collaboration within Corporations Using Software Forges - http://www.riehle.org/publications/2009/open-collaboration-within-corporations-using-software-forges/
Continuous integration tools
http://rubyforge.org/projects/cruisecontrolrb - http://cruisecontrolrb.thoughtworks.com/
https://hudson.dev.java.net/
http://www.softwarebloat.com/2008/11/19/continuous-integration-blueprints-how-to-build-an-army-of-killer-robots-with-hudson-and-cucumber
Exploratory testing
http://www.kohl.ca/blog/archives/000185.html
http://www.kohl.ca/articles/ExploratoryTesting_MusicofInvestigation.pdf
The ongoing revolution in software testing
http://www.kaner.com/pdfs/TheOngoingRevolution.pdf
ABAP Unit
Spotlight on ABAP Unit Part 1 - https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/1088
Load testing
http://dannorth.net/2007/02/monkey-business-value
Joel Spolsky 12 steps to better code - http://www.joelonsoftware.com/articles/fog0000000043.html
Automated testing story on SDN - https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/4103
JUnit - http://www.junit.org/
Open Source
http://www.paulgraham.com/opensource.html (What business can learn from open source)
Watir
http://wtr.rubyforge.org/
Spec as test
Cucumber - http://cukes.info/
Rspec - http://rspec.info/
http://www.pragprog.com/titles/achbd/the-rspec-book
SAP testing
http://www.beteoblog.com/beteo-alm-miniguides/software-quality/
http://raa.ruby-lang.org/project/saprfc/
Integration Tests in ABAP Development und Tool Support - https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/7131
XSLT - https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/12173
Test automation as panacea (not)
http://railspikes.com/2008/7/11/testing-is-overrated
http://michaelfeathers.typepad.com/michael_feathers_blog/2008/06/the-flawed-theo.html
BDD - http://behaviour-driven.org/
Blue Ruby - https://sap.na.pgiconnect.com/p16473929/ - http://www.slideshare.net/schmerdy/blue-ruby-sdn-webinar-1260181 https://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/408a9a3b-03f9-2b10-b29c-f0a3374b19d8
