A Day in the Life of a

Verification Requirement

Presented at the Systems and

Software Technology Conference
Salt Lake City, Utah, 20 – 23 April,
2009
Mr Stephen Scukanec
Test and Evaluation

Cleared for Public Release, Control

No. 08-001, dated 28 January 2008
 Agenda

• Abstract
• Overall Process
• Verification Cross-Reference Matrix
• Verification Plans
• References
• Author Biographies

Cleared for Public Release, Control

No. 08-001, dated 28 January 2008
2
 Abstract

• One measure of the quality of a product requirement is that it be verifiable. Verifiability

assessment is one of the exit criteria for the Systems Requirements Review and is necessary
for requirement validity. Nomination of one or more verification methods (inspection, analysis,
modeling and simulation, demonstration or test) is often taken as the sole evidence of
verifiability A completed Verification Cross Reference Matrix is frequently considered as the
verifiability.
final verifiability assessment and responsibility for the remainder of the verification effort is
transferred to the test and evaluation and other implementing communities for completion.
• Lessons learned from many Programs have shown that a more robust application of systems
engineering
g g should include the requirements
q engineers
g (with
( detailed knowledge
g of product
p
requirement intent) working with the implementing organizations as the best combination to
define the verification requirements. Such definition should include statement of the
verification objectives, success criteria and environment. Including this information in the
”Quality Assurance” section of the requirements document allows for buy-in by the customer
well in advance of implementing the verification activities. This information is used by
verification
e ification personnel
pe sonnel to generate
gene ate one o or mo
more e verification
e ification plans and to develop
de elop the detailed
verification program. Verification requirements are planned into verification events which are
executed using the proper system elements and environments. These verification
requirements are key to establishing long lead verification facilities, tools and laboratories.
Early definition of these requirements helps prevent facility re-designs and verification re-plans
that can cause expensive delays
delays. Finally
Finally, verification data analysis is performed,
performed and the
information compiled into verification reports certifying system product requirements
compliance. This robust verification approach will provide proof of requirements satisfaction,
leading to systems that meet the customers' needs at a lower life-cycle cost.
• This paper describes these concepts and steps in detail and provides examples for a set of
generic
i aircraft
i ft requirements.
i t
Cleared for Public Release, Control
No. 08-001, dated 28 January 2008
3
 Verification Requirements – What Are They And
Why Do We Need Them?

• Verification Requirements Specify the Verification Events Needed to

P
Prove th
the S
Satisfaction
ti f ti off th
the P
Product
d tR Requirements
i t anddHHelp
l tto
Define the Verification Process and Environment
• Verification
e cat o Requirements
equ e e ts a
are
e Necessary
ecessa y for
o at Least
east Two
o Reasons:
easo s
– Existence of Verification Requirements Demonstrates Verifiability of Product
Requirements
– Agreed-to Verification Requirements Define the Verification Program by Which the
Contractor Shows that the Product is What the Customer Needed

Cleared for Public Release, Control

No. 08-001, dated 28 January 2008
4
 Verification in the Product Development Life-Cycle

System Concept Production,

Validation
Development – Deployment, Operation,
Operational Concept Support, Disposal
Document

System Verification Integrate

Requirements
System
Definition - SRD

Preliminary
Design – Verification Integrate
Segment Segments
Specifications

Detailed Design – Verification Integrate

Lower-Level Lower-Level
Specifications Components

Implementation
After Forsberg, K, Mooz, H and Cotterman, H,
Visualizing Project Management, 2nd Edition,
J Wiley and Sons
Sons, New York
York, 2000
2000.

Cleared for Public Release, Control

No. 08-001, dated 28 January 2008
5
 A Day in the Life of a Verification
Requirement
Product Requirements
Develop Verification Requirements

IPT
Establish
IPT
Establish
Certification /
IPT
Establish
Acceptance
IPT
Develop
IPT
Develop Detailed Verification Events Satisfy the
Proof of Design Verification Verification
Verification
Statements (A)
SOF Verification
Requirements
(B)
Verification
Requirements
(C) IPT / V&V
Cross Reference
Matrix (VCRM)
Requirements
(E)
Verification Requirements, NOT
IPT / V&V IPT / V&V IPT / V&V IPT / V&V

the
h Product
P d Requirements.
R i
These Documents May
IPT Develop Plan Verification
Be Combined Depending Product Design
No
Verification Plan
on Program Direction or
Product Requirements (F)
IPT
Product Requirements are
Develop Test
Develop
Demonstration
Develop
Develop
Examination
Plan Approval
Approved?
Never Complete Until The
Plan Analysis Plan IPT / V&V
Plan
Yes
Plan
A
Associated
i t d Verification
V ifi ti
Requirements are Completed
No
No
IPT IPT Execute
Develop IPT
Develop Verification
Verification
Information
Sheets ((VISs))
Submit VISs for
Coordination and Approved? Yes
Verification
Procedure
Approved? The Culmination of the
Approval
pp
(H)
(G) IPT / V&V
Yes
IPT / V&V
V ifi ti Activity
Verification A ti it off the
th
IPT
Conduct
Verification
IPT
Conduct Pre-
Verification Brief
IPT

Perform Design Requirements Results in

Review (Test and (Test and Demo Verification
Demo only)
(I)
only)
(J)
Activity a Verified Product.
IPT / V&V IPT / V&V
No
No

IPT
IPT IPT IPT Generate
Generate Archive Data Verification Data
Perform Data Verification Package With Package and
Approved? Approved?
Analysis Method Report Configuration Submit for
(K) IPT / V&V Management Approval IPT / V&V
(L)
Yes
IPT IPT
Archive Data
Report
epo Verification
e ca o Package With Submit Package
Results Configuration To Customer
Management

Cleared for Public Release, Control

No. 08-001, dated 28 January 2008
Verified Product DD-250
6
 Start with Product Requirements

• The Verification Process Begins With Authenticated

Product Requirements
• Examples
– PR-1:LRU Markings
• The product line-replaceable units shall be marked in accordance with
MIL-STD-130M.
– PR
PR-2:
2: Operational Availability
• The product shall have an operational availability (A0) of 97.5% at IOC.
– PR-3: LRU Accessibility
• Each product line-replaceable unit shall be able to be removed and
replaced without removing any other item or displacing any cables.
cables
– PR-4:Recovery Force Communication - Nominal
• The product shall provide a communications system capable of
communicating with the ground command.

Cleared for Public Release, Control

No. 08-001, dated 28 January 2008
7
 Create Verification Cross-Reference Matrix

[Link].34 Recovery Force Communications

SE – Translates Operational Objectives
The communications system shall provide a communications
system capable of communicating with the recovery forces pre-
into Product Requirements
Product
Requirement and post- landing Design – Provides assessment of
requirements
q implementation
p

Verification Objective Pass / Fail (Success Criteria) Test – Provides assessment of

requirements verifiability
Perform Integrated System Testing will show that the
Test of the communications communications system can
system capability to provide transmit and receive audio
SE – Provides compliance of the Product
a voice communications and at frequencies and ranges
beacon with recovery forces (power) represented by requirement
pre and post landing within standard ground recovery
Verification Implementation Group –
Product an integrated hardware / force communications
Verification software environment devices as defined in TBD Ensures Verification Implementation
Perform a demonstration of Demonstration will show the Feasibility
the communications systems ability for the
capability to provide voice communications systems to Advises alternatives to support
and beacon communications verbally communicate with programmatics
with recovery forces pre and the on board communication
Assess completeness
post landing while within a production configuration
representative environment equipment. The Provides verifiability assessment
and using a production demonstration will also
equipment configuration show beacon tracking within
communication ranges
established by TBD. SE – Verification Allocation and
Traceability Assurance
Verification Cross-Reference Matrix
Paragraph # N/A Exam Anal Demo Test

[Link].34 X
Traceability
[Link].34 X

Identifying a verification method is necessary, but not sufficient!

8 Cleared for Public Release, Control No. 08-001, dated 28 January 2008
 Verification Requirement Attributes

Objective
What is the purpose of this verification?

Method
What method do you need performed? What are
the verification circumstances (e.g., laboratory,
Verification
desk-top analysis, flight test)?
Requirements
Environment
•Examination Must answer
•Analysis What are the environmental conditions under
5 Questions which the item will be verified?
•Demonstration
•Test
Special Conditions (if necessary)
Are there any unique conditions (e.g., item
configurations) necessary for the execution of the
verification?

Success Criteria
Verification isn’t ONLY test! What results are to expected?

Cleared for Public Release, Control

No. 08-001, dated 28 January 2008
9
 Sample Verification Requirements - 1

• VR-1I: Compliance of product markings shall be verified by

examination
i ti off design
d i drawings
d i att the
th LRU supplier’s
li ’ location
l ti prior
i
to the LRU CDR. The inspection will show that each marking on the
LRU conforms to MIL-STD-130M.
• VR-2A: The product operational availability shall be calculated using
the results of the Government-accredited contractor-developed
reliability and maintainability analyses performed during the design
in conjunction with the Design Reference Missions documented in
Report XXXX. The analysis will show that the product, in its
operational environment, supported with its support equipment and
personnel, l across allll missions,
i i will
ill have
h an operational
ti l availability
il bilit off
at least 97.5%.

Cleared for Public Release, Control

No. 08-001, dated 28 January 2008
10
 Sample Verification Requirements - 2

• VR-3D: Removal and replacement of all LRUs shall be demonstrated

on the
th aircraft
i ft to
t show
h that
th t each
h LRU can be
b removed d and
d replaced
l d
without removing any other items or moving any cables.
• VR-4D: Perform
e o de
demonstration
o st at o to p
provide
o de a co
communications
u cat o s
system capable of communicating with the ground command team
while in a representative environment and production configuration.
Demonstration will show capability to communicate with recovery
forces at TBD distances in the TBD terrain environment.

Cleared for Public Release, Control

No. 08-001, dated 28 January 2008
11
 Sample Verification Requirements - 3

• VR-4T: Prove that the product’s communications system

is capable
p of communicating
g with the ground
g command
team by performing an integrated system test within an
integrated hardware/software environment. Testing will
show that the product can transmit and receive audio at
frequencies represented by standard ground recovery
forces communications devices defined in (TBD).
Verification Objective

Note – there are no

Verification Method Special Conditions

Environment

Cleared for Public Release, Control Success Criteria

No. 08-001, dated 28 January 2008
12
 Verification Requirements Flow and Traceability

Specification
Design
PR-1 PR-2 PR-3 PR-4
Requirements
VR-1E
VR 1E VR-2A
VR 2A VR-3D
VR 3D VR-4D
VR 4D
Verification
VR-4T Requirements

Verification Requirements
q Appear
pp
in the Same Specification as the
Product Requirements to be Verified Product N/A Exam Anal Demo Test Verification
Requirement Requirement

PR-1 X VR-1I
Master Verification Plan
PR-2 X VR-2A

PR-4 X VR-3D
Examination VR-1E PR-5 X X VR-4D
Analysis VR-2A VR-4T
Demonstration
VR-3D, VR-4D
Test
VR-4T

Cleared for Public Release, Control

No. 08-001, dated 28 January 2008
13
 Create Detailed Verification Requirements
(Verification Events)

Convert Verification Statements into

Detailed Verification Requirements
(Verification Events) by ----
Master Verification
For each verification activity identified in the
Plan (MVP)

(Verific
Verific
develo
Verific
verification matrix, a detailed description of the
ments

activity including:

cation Requirements

cation Requirements are

oped into Deta
on Requirem

cation Events))
Examination
E i i VR-1E
VR 1E
Analysis VR-2A •Verification configuration & its relationship to
Demonstration production configuration
VR-3D, VR-4D
•Associated prerequisites
Test
•Constraints
Verificatio

VR-4T
ailed •Objectives
•Procedures
•Relevant environmental conditions
•Pass/fail criteria- and necessary Data Set,
•Analysis models
models, if applicable
applicable.
A One To One Relationship Exists Between •Sequence if applicable
the Verification Requirements and the DVRs •Verification Environment (i.e.; Lab, Flight,
Production)

Cleared for Public Release, Control

No. 08-001, dated 28 January 2008
14
 Master Verification Plan

Master Verification Plan

Examination VR-1I
Analysis VR-2A
Demonstration
VR-3D, VR-4D
Test
Customer
VR-4T Concurrence

Modeling
Modeling
Modeling / Test
Test
Test /
Examination
Examination
Examination Analysis
Analysis
Analysis
Analysis Simulation
and
and Demo
and
andPlans
Plans
Plans
Plans Plans
Plans
Plans
Plans Simulation
Simulation
Plans Demonstration
Demonstration
VR-3D, VR-4D,
VR-1I
VR-1I
VR-1I VR-2A
Plans
Plans Plans
Plans
VR-4T

Cleared for Public Release, Control

15 No. 08-001, dated 28 January 2008
 Verification Execution Flow

Plans Procedures Execute Reports Results

Plan Specification

Compliance

Verified
Method Organization Early Verification Benefits

Examination QA, Manufacturing, •Examination Points Identified

Mission Assurance
•Tooling Requirements Identified Certification
Analysis Systems Engineering •Define / Build / Buy / Train Analysis Prior to
Specialty Engineering Need Date
Design •Accreditation of Analyses Tools Prior to Need
Date
Demonstration Systems Engineering •Define / Build / Buy / Train Demonstration Early Verification
Specialty Engineering Tools, Equipment and Facilities Prior to Need
Date Supports Multiple
Design, Operational
Assessment •Accreditation of Demonstration Models and
Organizational
Simulations Prior to Need Date Functions’ Long Lead
Test Ground and Flight Test •Laboratory and Lab Software Requirements Needs and Prevents
Identified
Facilities Development Costly Late Program
•Facilities Requirements Identified
Re-Work
•Long Lead Test Items Identified

Cleared for Public Release, Control

No. 08-001, dated 28 January 2008
16
 Planning for Verification Execution and Product
Verification
Rev 1 Rev 2 Rev 3 Rev 4
Requirements
Design
g
Build
Verification
C tifi ti
Certification

Long Lead Facilities

Define Verification Laboratory Design
Requirements Early Discover the Verification
Range Coordination
and in Detail to Requirements Late and Have
Establish the Entire Design Requirements
Enormous Rework to Establish
Verification Effort Software the Entire Verification Effort
Analysis Tools

… and it Costs … and it Costs a

Relatively little … Lot More …

Early Verification Is an Effective Cost Avoidance Approach

Cleared for Public Release, Control
17 No. 08-001, dated 28 January 2008
 Summary and Conclusions

• The Verification Process Begins With Authenticated Product

Requirements
• Define Verification Requirements, not Just Methods – The VCRI Is
the Last Thing Developed in the Specification
• Verification Requirements Must State the Objective, Method,
Environment, and Expected Results. There May Also be Special
Conditions.
• The Master Verification Plan is the Guidance for the Verification
Program
• Verification is Conducted Against the Product Defined by the Title
of the Specification
• Verification
V ifi ti Program
P Benefits
B fit are nott Limited
Li it d to
t Just
J t the
th Systems
S t
Engineering and Test Organizations
• Define the Verification Requirements Early to Reduce the Overall
Program Cost

Cleared for Public Release, Control

No. 08-001, dated 28 January 2008
18
Acronyms

A0 Availability
Anal Analysis
D
Demo D
Demonstration
t ti
DVR Detailed Verification Requirement
Exam Examination
IOC Initial Operational Capability
LRU Line Replaceable Unit
MVP Master Verification Plan
PR Product Requirements
QA Quality Assurance
SE Systems Engineering
SRD Systems Requirements Document
TBD To Be Determined
VCRI Verification Cross Reference Index
VCRM Verification Cross Reference Matrix
VR Verification Requirements
 Author Biographies

• Steve Scukanec has spent over 25 years as an Aerospace Engineer on

various complex programs including the B B-2,
2, B
B-22 Long Term Software
Support, F-35. With a focus on test and evaluation, Steve has been able to
participate in programs from inception to completion. This experience over
several programs has provided Steve with a rare understanding of the
values of a well executed Verification program as well as the problems
causeddbby th
the lack
l k off one. His
Hi experience
i as a “requirements
“ i t generator”,
t ”
“requirements customer”, “requirements manager” and verifier gives him
insight into the lifecycle of a requirement and a large lessons learned
knowledge base.
• Jim van Gaasbeek has 35 years experience analyzing and developing
rotary-wing and fixed-wing aircraft, launch vehicles and spacecraft, both
in the United States and European defense environments. Beginning as a
rotor aeroservoelastician, his career has progressed with experience in
constructive and virtual simulation,
simulation accident investigation,
investigation vehicle-
vehicle
management system design and systems engineering, concentrating in
risk management and requirements development, management and
verification.

Cleared for Public Release, Control

No. 08-001, dated 28 January 2008
20