Software Quality Assurance

Quality Concepts - 1

Variation control is the heart of quality control Software engineers strive to control the

process applied resources expended end product quality attributes

Quality of design
refers

to characteristics designers specify for the end product to be constructed

Quality Concepts - 2

Quality of conformance
degree

to which design specifications are followed in manufacturing the product

of inspections, reviews, and tests used to ensure conformance of a work product to its specifications and reporting procedures used to provide management with data needed to make proactive decisions
3

Quality control
series

Quality assurance
auditing

Quality Costs

Prevention costs
quality

planning, formal technical reviews, test equipment, training

and inter-process inspection, equipment calibration and maintenance, testing repair, failure mode analysis

Appraisal costs
in-process

Failure costs
rework,

External failure costs

complaint

resolution, product return and replacement, help line support, warranty work
4

Total Quality Management - 1

Kaizen
develop

a process that is visible, repeatable, and measurable the intangibles that affect the process and work to optimize their impact on the process

Atarimae hinshitsu
examine

Total Quality Management - 2

Kanse
examine

the way the product is used by the customer with an eye to improving both the product and the development process product use in the market place to uncover new product applications and identify new products to develop
6

Miryokuteki hinshitsu
observe

Software Quality Assurance

Conformance to software requirements is the foundation from which software quality is measured. Specified standards are used to define the development criteria that are used to guide the manner in which software is engineered. Software must conform to implicit requirements (ease of use, maintainability, reliability, etc.) as well as its explicit requirements.
7

Software Quality Assurance

Software Quality Assurance (SQA) as a planned and systematic pattern of all actions necessary to provide adequate confidence that the item or product conforms to established technical requirements'. SQA does this by checking that:

plans are defined according to standards

procedures are performed according to plans products are implemented according to standards. A procedure defines how an activity will be conducted. Procedures are defined in plans, such as a Software Configuration Management Plan. A product is a deliverable to a customer. Software products are code, user manuals and technical documents, such as an Architectural Design Document. Examples of product standards are design and coding standards, and the standard document templates. SQA is not the only checking activity in a project. Whereas SQA checks procedures against plans and output products against standards,Software Verification and Validation (SVV) checks output products against input products.

Differences between SQA and SVV

Standards Plans

Input Products

Development Activity

Output Products

SQA Check output product against standard and plans

SVV Check Output product against input products

SVV Reports

Quality Assurance

Quality assurance is an activity that establishes and evaluates the processes that produce software products. Quality assurance is a planned and systematic set of activities necessary to provide adequate confidence that products and services will conform to specified requirements and meet user needs. Quality Assurance is responsible for implementing the quality policy defined through the development and continuous improvement of software development processes. Quality assurance helps establish processes and sets up measurement programs to evaluate processes. Quality assurance identifies weaknesses in processes to improve them and it is concerned with all of the products that will ever be produced by a process. Quality Assurance tries to bring the balance between the producer's and the customer's view point of quality within a specified limit.
10

Quality Assurance And Quality Control

Quality Assurance: A set of activities designed to ensure that the development and/or maintenance process is adequate to ensure a system will meet its objectives.
Quality Control: A set of activities designed to evaluate a developed work product.

QA is process oriented QC is product oriented.

Quality Assurance makes sure you are doing the right things, the right way.
Quality Control makes sure the results of what you've done are what you expected.
11

SQA Group Activities - 1

Prepare SQA plan for the project. Participate in the development of the project's software process description. Review software engineering activities to verify compliance with the defined software process.

12

SQA Group Activities - 2

Audit designated software work products to verify compliance with those defined as part of the software process. Ensure that any deviations in software or work products are documented and handled according to a documented procedure. Record any evidence of noncompliance and reports them to management.
13

Software Reviews

Purpose is to find defects (errors) before they are passed on to another software engineering activity or released to the customer. Software engineers (and others) conduct formal technical reviews (FTR) for software engineers. Using formal technical reviews (walkthroughs or inspections) is an effective means for improving software quality.

14

Formal Technical Reviews - 1

Involves 3 to 5 people (including reviewers) Advance preparation (no more than 2 hours per person) required Duration of review meeting should be less than 2 hours Focus of review is on a discrete work product Review leader organizes the review meeting at the producer's request.
15

Formal Technical Reviews - 2

Reviewers ask questions that enable the producer to discover his or her own error (the product is under review not the producer) Producer of the work product walks the reviewers through the product Recorder writes down any significant issues raised during the review Reviewers decide to accept or reject the work product and whether to require additional reviews of product or not.
16

Formality and Timing

Analysis is complete. Design is complete. After first compilation. After first test run. After all test runs. Any time you complete an activity that produce a complete work product.

17

Formal SQA Approaches

1.
2. 3.

Proof of correctness. Statistical quality assurance. Cleanroom process combines items 1 & 2.

18

Statistical Quality Assurance

Information about software defects is collected and categorized Each defect is traced back to its cause Using the Pareto principle (80% of the defects can be traced to 20% of the causes) isolate the "vital few" defect causes Move to correct the problems that caused the defects
19

Benefits of Quality Assurance

Clarification of what services a client expects the practice to provide. Reduction in loss of time due to re-work or ineffective and/or inefficient practices. Better communications structure. Higher repeat business/referrals from a satisfied client base. Problems are highlighted and resolved in an open manner. Increased confidence that controls are in place and the risk of error is reduced. Training for staff in performing their roles. Morale enhanced by having an effective, well-run practice.

20

Software Reliability

Defined as the probability of failure free operation of a computer program in a specified environment for a specified time period Can be measured directly and estimated using historical and developmental data (unlike many other software quality factors) Software reliability problems can usually be traced back to errors in design or implementation.
21

Software Reliability Metrics

Reliability metrics are units of measure for system reliability System reliability is measured by counting the number of operational failures and relating these to demands made on the system at the time of failure A long-term measurement program is required to assess the reliability of critical systems
22

Reliability Metrics - part 1

Probability of Failure on Demand (POFOD)

POFOD

= 0.001 For one in every 1000 requests the service fails per time unit

Rate of Fault Occurrence (ROCOF)

ROCOF

= 0.02 Two failures for each 100 operational time units of operation

23

Reliability Metrics - part 2

Mean Time to Failure (MTTF)

average

time between observed failures (aka

MTBF)

Availability = MTBF / (MTBF+MTTR)

MTBF

= Mean Time Between Failure MTTR = Mean Time to Repair

Reliability = MTBF / (1+MTBF)

24

Time Units

Raw Execution Time

non-stop

system

Calendar Time
If

the system has regular usage patterns type transaction systems

Number of Transactions
demand

25

Software Safety

SQA activity that focuses on identifying potential hazards that may cause a software system to fail. Early identification of software hazards allows developers to specify design features to can eliminate or at least control the impact of potential hazards. Software reliability involves determining the likelihood that a failure will occur without regard to consequences of failures.
26

Validation Perspectives

Reliability validation
Does

measured system reliability meet its specification? Is system reliability good enough to satisfy users?

Safety validation
Does

system operate so that accidents do not occur? Are accident consequences minimized?

Security validation
Is

system secure against external attack?

27

Validation Techniques

Static techniques
design

reviews and program inspections mathematical arguments and proof

Dynamic techniques
statistical

testing scenario-based testing run-time checking

Process validation
SE

processes should minimize the chances of introducing system defects

28

Static Validation Techniques

Concerned with analysis of documentation Focus is on finding system errors and identifying potential problems that may arise during system operation Documents may be prepared to support static validation
structured

arguments mathematical proofs

29

Static Safety Validation Techniques

Demonstrating safety by testing is difficult Testing all possible operational situations is impossible Normal reviews for correctness may be supplemented by specific techniques intended to make sure unsafe situations never arise

30

SQA Plan 1

Management section
describes

the place of SQA in the structure of the organization

each work product produced as part of the software process

Documentation section
describes

Standards, practices, and conventions section

lists

all applicable standards/practices applied during the software process and any metrics to be collected as part of the software engineering work
31

SQA Plan - 2

Reviews and audits section

provides

an overview of the approach used in the reviews and audits to be conducted during the project the test plan and procedure document and defines test record keeping requirements
32

Test section
references

SQA Plan - 3

Problem reporting and corrective action section

defines

procedures for reporting, tracking, and resolving errors or defects, identifies organizational responsibilities for these activities SQA methods, change control, record keeping, training, and risk management

Other
tools,

33