Mini Guide Functional Testing :

Governance based on testware

metrics
What to do when you want to outsource
your testing service to a Test Factory?

Ton Dekkers
Ignacio Lopez Carrillo

1
 Why NESMA
About LEDAmc
The cuestion
Want to improve
Typical Problems
What to do
Examples
And in Agile

2
About Nesma
Ton Dekkers
Nesma|President The renewed website is
Interdependent | Software Cost Engineer organized into 6 themes:
ISBSG | Past President - Board
COSMIC | IAC member Benchmarking
Outsourcing
Partners Productivity
Galorath | US Project Control
Leda | ES Estimation
Metrieken.nl | NL Sizing methods
PRICE systems | UK

Vision
independent from customers and suppliers

nesma is the not-for-profit organisation in the area of predictability of the cost of the
delivery and the maintenance of software
nesma joins as much as possible existing standards with a different focus than measurement
nesma connects surrounding attention domains
nesma is independent from customers and suppliers

3
Nesma Miniguides Software metrics in contracts

1. Guideline for metrics in contracts (resume)

2. Development methodologies;
3. Maintenance;
4. Management;
5. RFP Questions;
6. Functional Quality;
7. Pricing Mechanisms;
8. Technical Quality;
9. Assessing Suppliers Performance;
10. Software Metrics in Contracts; [13] Mini
Guide:
English -- Free 10,= 20,= 40,= in progress

11. Requirements for Supplier organizations; Functional

Testing
12. Requirements for Customer organizations.
13. Functional Testing

http://nesma.org/publications/downloads/guides/software-metrics-in-contracts/

4

About LEDAmc
Spanish company
> 100 employees
Office in Madrid and Bogota
Customers in Europe, USA, LATAM and ASIA
Quantify Focussed in Outsourcing Management
Optimize > 70% FP certified consultants in Spain
ROI service commitment (47M 2015)
Projections

5
Risk SW Gobernance

Content Management

Lack of Risk Type

Methodology Complexity Schedule Over Effort Preassure
Information

Size
Persons
Technology Process Factors Risk SW Consecuences Economical
Time
Technology Gobernance
Management Imagen Performance
Experience Productivity Quality
Model

Test Design Productivity

Estimate
Test Productivity Test Execution Productivity Test Process Diagnostic
OPP Test Effectiveness Prevent Predict Agile

Projects
Score Card Testing
Test Gobernance Predictions


Decision Making Optimize Project Risk Sizing Team
Office

TMO Test Audit & Test QA

Quality Project Navigator
Strategy &Test Planning
Test Management Tracking reports
Delivery reports Gobernance Test Process Maturity
Test Design
Agile Testing
Testing Test Execution
Inside_
Estimaciones Testware
Functional Quality Benchmarking SOAPUI
ROI Pruebas

6
The question

7
The question

Esfuerzo de pruebas:
Disminucin del coste por reduccin de tarifa

Produccin de Pruebas (TFP):

Incremento de coste por perdida de
productividad de pruebas

8
Test Teams / Test Factories
Lack of this
Testware metrics
Some goes directly
enhance
?
capability
to this model

Lack of reference
values
Testing
factories Lack of results expected

enhance
?
effectiveness
RFP Testing Factory

Select wrong Testing

Factory supplier
enhance
performance ? Loss of time, money
and quality

Functional Testing
Miniguide

9
Typical Problems
Dont worry
Test Management
Different Test and Defect tracking Be busy
environment common tools
management

Different
Highly detailed Location
documentation
Organizational
Non Not enough
professional information in
profiles Quality Gates

Capability

?
High level Human Decision
of rotation resources making
Performance
No information
Effectiveness Subjetive in earlier stage
Payment Model
Savings (MH, TC)
Economical
Project
Planning delay
Over sizing
of testing
resources Duplication of Not objective
cost for getting testing estimation
good quality model Not control
Not measurable of outsourced
target of service
testing results

10
What to do to mitigate problems?

Four Stages during the testing outsourcing process to mitigate the problems :

RFP preparation: before the RFP adjudication

RFP adjudication: during the RFP adjudication

Service Operation: during the operation of the outsourced testing service

Close/Renewal of service: during the termination or renovation of the outsourced service

11
Activities

Getting reference Define testing Justify savings with

Define testing service Define Testware Sizing of human Selection of possible
values of testing ecosystem conditions outsourcing to testing
payment model estimation model Resources needed suppliers
activities of outsourced service factories

RFP preparation

Getting reference Analyze the size and

Define Non subjective Define size estimation Define outsourcing Sizing of testing Justify savings
values of testing business focus of the
payment model model of the project location resources in time
productivity suppliers needed

Getting reference Define model of

Define testing service Sizing of defect Ask for a RFQ to no Justify savings
values of testing Define size of testware outsourcing testing
ROI model removal resources more than six suppliers in Quality
effectiveness factory

Getting reference Define test

Define bonus/penalty Define testing level to Justify savings
values of current effort estimation Test design, Test execution,
scheme be outsourced in
testing cost model Test automation

Define Defect removal

Function Points tested Enhancement detail of
effort estimation
or any other functional documentation needed
model
Function Points or Designed test case , expected to be
size metrics
any other detected defects, expected incidence in
functional size production (during the guarantee period
metrics and Quality debt)
12
Activities

Define Testing Factory Define Testing Factory Prepare testing

RFP MUST RFP NTH ecosystem

RFP adjudication

Request for a Prepare organization of

Define ANS continuous operation with testing
improvement model tools Requirements, Test
Management, defect Tracking
Demand the use of a Guarantee access to
Define Quality Gates
test process reference different test
metrics
model not proprietary environments

Define protocol to start Select different blocks

Guarantee semi real
and finish the testing for two different
test data
service suppliers
ISTQB,
Facilitate communication
Request of a Control Include testing service
with developers: Defect
Panel for the TKPIs ROI model
removal

Request for a service

tracking model

13
Activities

Establish model to Extrapolate possible

Control testing metrics Audit the outsourced
reduce the testing evolution of the final
(TKPIs) testing service
resource needed quality results

Service Operation

Audit the level of Enhancement of the Related with Quality defects in the next
Control testing testing phases
details of the test case testing activities in the Gate and compared
productivity
designed testing levels with initial estimation

Audit the test Automatize tracking Propose evolution

Control testing
coverage, depth and service, quality and needed to get the
effectiveness
impact in the risk of test designed governance reports committed results
quality,
dates and quality
performance and Compare the TKPIs Propose possible new
effectiveness of the Audit the effectiveness
metrics by area (block), and more realistic
testing service of the testing service
supplier, technology, commitments

Compare the TKPIs Control Impact of Translate TKPIs

metrics by month and rotation of the hhrr evolution to economic
year assigned to the service data

14
Activities

Prepare next period of

Economical review Close Renewal
service

Close/ Renewal of service

Fix new TKPIs target

Estimate new needs for Execute transfer plan
Analyze ANS results for next period of
next period to new supplier
service

Perform continuous
Perform ROI report of Define target of
improvement model
the testing service enhacement
(TMMi)

Perform bonus/penalty
scheme

15
Examples RFP Preparation: Test-ware estimation
Estimaciones de Testware

Early test-ware size and effort estimations

16
Examples RFP Preparation: Sizing of testing resources
Estimaciones de Testware

17
Examples RFP Preparation: Testing Productivity figures
Caso de xito

18
Saving money Using the Mini Guide
Caso de xito

KPI Indicator Control No control KPI Indicator Control No control GAP %

Time to create a TC 24 39 Total TC designed by person/year 3.575 2.200 1.375 62,5%
Productivity Productivity
Time to execute a TC 32 48 Total TC executed by person/year 2.681 1.788 894 50,0%
% Bugs detected 81% 49% Total Bugs detected by person/year 203 75 127 168,6%
Effectiveness % Wrong bugs 8,0% 15,7% Efectiveness Total Wrong Bugs reported by person/year 16 6 10 168,6%
Bug Fixing 67,0% 26,7% Total Bugs fixed by person/year 136 20 116 574,8%

Project Size 5.236 PF No

# TC 28.275 Control control GAP %
Test size
Testware # Bugs 3.677
estimation TC Design 4.166 Size Test Team 8 13 -5 62%
Test effort
TD execution 5.024
Total Cost 417.600 588.120 170.520 29%

TC

28.275

17.400
Control
No control

0
1 year 1,6 years Time

19
 Software Testing metrics in Agile projects
Estimaciones de Testware

Relation between History Point and Function Point

Quality debt by team (current and future)
Quality team ranking Agile yes, but not
Sprint amount needed by 100 implemented History Point kamikaze
Testing and SW
Cooperation Test automation effectiveness
metrics always
of all team
needed
Agile Test
Team: ATT

History Dont
Factories Point with compare
different persons
criteria and teams

Average test cases designed by person [/ day] / 1 designedTest Case Agile teams
Average test cases executed by person [/ day] / 1 executed Test case Releases
Average defects checked by person [/ day] / 1 detected defect Sprints
# detected defects by History Point / 1 avoided defect
% total errors detected QA / UATs / Production / 1 tested History Point

20
Software Testing metrics in Agile pojects
Estimaciones de Testware

Be agile,
flexible, but
not fragile

21
Estimaciones de Testware

22