ISO/IEC 29119
The New International Software Testing Standard
Stuart Reid
Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500 Fax: 0207 623 8459 www.testing-solutions.com Stuart Reid, 2012
Scope
The purpose of standards
Overview of ISO 29119
Applicability Recent developments Timeline Future work
What are standards?
Guideline documentation that reflects agreements on products, practices, or operations by nationally or internationally recognized industrial, professional, trade associations or governmental bodies - ISO
Guidelines documents as they are not compulsory unless mandated by an individual or an organization Agreements because they often reflect a certain level of consensus
Pre-standardization?
Why use standards?
Consumers
Confidence in compliant products Authors provide expertise in standards
Manufacturers
Conformance and Marketing
Safety from liability
Guidelines on production
But not Best Practice
Quality and Standards
What use are standards?
Standards describe a current body of knowledge that provides the basis for a professional discipline Basis for:
Communication common terminology Professional qualifications Certification/compliance schemes Benchmark of good industry practice Contracts Interoperability and consistency
A Case in Point
Standardization Bodies
100
120
140
20 Published Maintained
40
60
80
ISO/IEC Software Standards
0
1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Motivation for ISO 29119
Conflicts in definitions, processes & procedures
plethora of standards to be replaced by one
e.g. IEEE 829, IEEE 1008, BS 7925-1/-2, IEEE 1028
practitioners do not know which standard to follow
Lacking in current standards:
Organizational Testing not covered
e.g. Test Policy and Organizational Test Strategy
no Project Test Management BS 7925 only covers unit testing common functional techniques missing poor coverage of non-functional testing
ISO/IEC SC7 WG26
ISO IEC
TC176 Quality
JTC1
TC56 Dependability
Information Technology
SC7 Software & Systems Engineering
SC22 Programming Languages
SC27 IT Security
SC32
Data Management & Interchange
WG26
SWG1 ..................
Software Testing
Representatives of National Standards Bodies
ISO/IEC 29119 Structure
Part 1 Concepts & Vocabulary
BS 7925-1
Part 4 Testing Techniques
Part 2 Processes
Part 3 Documentation
BS 7925-2
Part 5 Process Assessment
IEEE 829
Actually ISO/IEC 33063
Part 1: Concepts & Vocabulary
Scope, Conformance, Normative References
DEFINITIONS
SOFTWARE TESTING CONCEPTS
TESTING IN DIFFERENT LIFE CYCLE MODELS ROLES AND RESPONSIBILITIES IN TESTING ANNEXES Metrics, Examples, Bibliography
Part 2: Testing Processes
ORGANIZATIONAL TEST PROCESS
TEST MANAGEMENT PROCESSES
DYNAMIC TEST PROCESSES
Instantiating Testing Processes
ORGANIZATIONAL TEST PROCESS
TEST MANAGEMENT PROCESSES
DYNAMIC TEST PROCESSES
Test Management Processes
ORGANIZATIONAL TEST PROCESS
ORGANIZATIONAL TEST DOCUMENTATION FEEDBACK ON ORGANIZATIONAL TEST DOCUMENTATION
TEST MANAGEMENT PROCESSES
TEST PLAN UPDATES TEST COMPLETION REPORT TEST PLAN
TEST PLANNING
TEST MONITORING & CONTROL
TEST COMPLETION
TEST PLAN, CONTROL DIRECTIVES
TEST PLAN, TEST COMPLETION REPORT, TEST MEASURES
TEST PLAN, CONTROL DIRECTIVES
TEST MEASURES
TEST MANAGEMENT PROCESSES
DYNAMIC TEST PROCESSES
Test Planning Process
Understand Context
Scope
Organise Test Plan Development Identify & Estimate Risks
Analyzed Risks Treatment Approaches
Identify Risk Treatment Approaches
Schedule, Staffing Profile Draft Test Plan
Design Test Strategy
Test Strategy
Determine Staffing and Scheduling
Document Test Plan
Approved Test Plan Test Plan
Gain Consensus on Test Plan
Publish Test Plan
Test Monitoring & Control Process
test status report
test progress info
Report
test control info
[testing incomplete]
test plan
Set-Up
test measures
[testing complete]
Monitor
test progress info
Control
measures
control directives
...TEST PROCESSES... Dynamic/Management
Dynamic Test Processes
(LEVEL) TEST MANAGEMENT PROCESS
(LEVEL) TEST PLAN CONTROL DIRECTIVES
TEST MEASURES
DYNAMIC TEST PROCESSES
TEST DESIGN & IMPLEMENTATION
TEST SPECIFICATION
TEST EXECUTION
TEST RESULTS
[NO ISSUES NOTICED]
[ISSUE NOTICED or RETEST RESULT] TEST ENVT REQUIREMENTS
TEST ENVIRONMENT SET-UP
TEST ENVIRONMENT READINESS REPORT
TEST INCIDENT REPORTING
INCIDENT REPORT
Test Process Descriptions Example Organizational Test Process
Each testing process is described using a standard template (following ISO 24774), for example:
Purpose The purpose of the Organizational Test Process is to develop and maintain organizational test specifications, such as the Test Policy and Organizational Test Strategy. Outcomes As a result of the successful implementation of the Organizational Test Process: The organizational test specification is developed based on stakeholder requirements; The organizational test specification is agreed by stakeholders; The organizational test specification is published and accessible to stakeholders; Conformance of stakeholders with the organizational test specification is managed; The organizational test specification is reviewed on a periodic basis; Minor updates to the organizational test specification are made as necessary. Activities and tasks The tester shall implement the following activities and tasks in accordance with applicable organization policies and procedures with respect to the Organizational Test Process. Develop Test Specification This activity consists of the following tasks: Analyze any relevant source documents and the current testing practices within the organization to identify requirements for the organizational test specification. etc. Gain Consensus on Test Specification etc. Information items The output of the Organizational Test Process is the organizational test specification. The Organizational Test Policy and Organizational Test Strategy are typical examples of organizational test specifications.
Note that ISO 24774 (and this format) has changed since we started, so we have had to rewrite all the process descriptions
Part 3 Test Documentation
Scope, Conformance, Normative References
TEST DOCUMENTATION ANNEXES - EXAMPLES
Part 3: Test Documentation
Organizational test documentation
Test policy Test strategy
Project test documentation
Project test plan Test project completion report
Test Level documentation
Test plan Test specification Test results Anomaly reports Level test status report Test environment report Test level completion report
Appendices
examples of documents at each level of testing
Part 4 Test Techniques
Scope, Conformance, Normative References
TEST DESIGN TECHNIQUES
BLACK BOX WHITE BOX
TEST COVERAGE MEASUREMENT ANNEXE TESTING OF QUALITY CHARACTERISTICS
ANNEXE SELECTION OF TECHNIQUES ANNEXE TEST TECHNIQUE EFFECTIVENESS
Applicability
Intended to be a generic standard
support all domains
Some domains are more standards-oriented, e.g.
safety-related telecoms
Some domains will become more regulated, e.g.
financial banks, stock markets, etc.
Needs trialling in all domains and project types
Applicability to sequential, iterative and agile Applicability to new and maintenance projects Applicability to IS and embedded E.g. when wouldnt you be selecting and using techniques? When wouldnt you plan?
When can you start using it and when can you get involved?
Typical ISO standards take over 7 years
ISO 12207 was conceived in 1988 and published in 1995 and represents 17,000 person hours (8.5 person years)
BS 7925-1 & -2 took 8 years to develop
IEEE estimates 2-4 years to develop a standard, at a cost of between $2,000 and $10,000 per page [1998]
Timelines all going to plan!
Working Draft (WD) Committee Draft (CD) Draft International Standard (DIS) Final Draft International Standard (FDIS) Final International Standard (FIS)
Parts 2 & 3 Parts 1 & 4
WD
CD1
CD2
CD3
DIS
FDIS
FIS
FIS CD3 DIS FDIS
WD
CD1
CD2
May 10
May 11
May 12
May 13
Do you want to be involved?
Join the working group
representing your national standards body 6 day meetings, every 6 months contribute between meetings
Join a mirror group
for your national standards body
Contribute materials
Review drafts
Current status
Drafts are out for review
Unsurprisingly (for a testing standard) we get plenty of comments (1000s)
Countries attending meetings include:
Denmark, India, China, UK, South Africa, Hong Kong, Spain, Germany, USA, Finland, Australia, Korea, Canada, France, Poland, Japan, Russian Federation, Columbia, Sweden, Brazil, Malaysia, New Zealand, Argentina, India, Peru, Colombia, Turkey
Last meeting Mumbai, Nov 2011 Next meeting Korea, May 2012
Conclusions
International standard will provide practitioners with guidelines for testing that cover all aspects of the life cycle Provides a consistent set of definitions, processes, procedures & techniques for software testing Will be adopted by IEEE, BSI, ISO and other national standards bodies Currently has representation from 18 nations & is being reviewed by software testing professionals world-wide We still need to ensure widespread applicability
Any Questions? Any Volunteers?
For more information
[email protected] if interested in trialling the standard on a project, reviewing drafts or writing examples
http://softwaretestingstandard.org/
WG26 website
http://www.jtc1-sc7.org/
access to official documents released by WG 26
