NIST Special Publication 800

NIST SP 800-55v1 ipd

Measurement Guide for

Information Security
Volume 1 — Identifying and Selecting Measures

Initial Public Draft

Katherine Schroeder
Hung Trinh
Victoria Yan Pillitteri

This publication is available free of charge from:

https://doi.org/10.6028/NIST.SP.800-55v1.ipd
 NIST Special Publication 800
NIST SP 800-55v1 ipd

Measurement Guide for

Information Security
Volume 1 — Identifying and Selecting Measures

Initial Public Draft

Katherine Schroeder
Hung Trinh
Victoria Yan Pillitteri
Computer Security Division
Information Technology Laboratory

This publication is available free of charge from:

https://doi.org/10.6028/NIST.SP.800-55v1.ipd

January 2024

U.S. Department of Commerce

Gina M. Raimondo, Secretary

National Institute of Standards and Technology

Laurie E. Locascio, NIST Director and Under Secretary of Commerce for Standards and Technology
NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

Certain equipment, instruments, software, or materials, commercial or non-commercial, are identified in this
paper in order to specify the experimental procedure adequately. Such identification does not imply
recommendation or endorsement of any product or service by NIST, nor does it imply that the materials or
equipment identified are necessarily the best available for the purpose.

There may be references in this publication to other publications currently under development by NIST in
accordance with its assigned statutory responsibilities. The information in this publication, including concepts and
methodologies, may be used by federal agencies even before the completion of such companion publications.
Thus, until each publication is completed, current requirements, guidelines, and procedures, where they exist,
remain operative. For planning and transition purposes, federal agencies may wish to closely follow the
development of these new publications by NIST.

Organizations are encouraged to review all draft publications during public comment periods and provide feedback
to NIST. Many NIST cybersecurity publications, other than the ones noted above, are available at
https://csrc.nist.gov/publications.

Authority
This publication has been developed by NIST in accordance with its statutory responsibilities under the Federal
Information Security Modernization Act (FISMA) of 2014, 44 U.S.C. § 3551 et seq., Public Law (P.L.) 113-283. NIST is
responsible for developing information security standards and guidelines, including minimum requirements for
federal information systems, but such standards and guidelines shall not apply to national security systems
without the express approval of appropriate federal officials exercising policy authority over such systems. This
guideline is consistent with the requirements of the Office of Management and Budget (OMB) Circular A-130.

Nothing in this publication should be taken to contradict the standards and guidelines made mandatory and
binding on federal agencies by the Secretary of Commerce under statutory authority. Nor should these guidelines
be interpreted as altering or superseding the existing authorities of the Secretary of Commerce, Director of the
OMB, or any other federal official. This publication may be used by nongovernmental organizations on a voluntary
basis and is not subject to copyright in the United States. Attribution would, however, be appreciated by NIST.

NIST Technical Series Policies

Copyright, Use, and Licensing Statements
NIST Technical Series Publication Identifier Syntax

Publication History
Approved by the NIST Editorial Review Board on YYYY-MM-DD [Will be added in the final publication]
Supersedes NIST Series XXX (Month Year) DOI [Will be added in the final publication]

How to Cite this NIST Technical Series Publication:

Schroeder K, Trinh H, Pillitteri V (2024) Measurement Guide for Information Security: Volume 1 — Identifying and
Selecting Measures. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication
(SP) NIST SP 800-55v1 ipd. https://doi.org/10.6028/NIST.SP.800-55v1.ipd

Author ORCID iDs

Katherine Schroeder: 0000-0002-4129-9243
Hung Trinh: 0000-0002-3323-0836
Victoria Yan Pillitteri: 0000-0002-7446-7506
NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

Public Comment Period

January 17, 2024 – March 18, 2024

Submit Comments
[email protected]

National Institute of Standards and Technology

Attn: Computer Security Division, Information Technology Laboratory
100 Bureau Drive (Mail Stop 8930) Gaithersburg, MD 20899-8930

All comments are subject to release under the Freedom of Information Act (FOIA).
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

1 Abstract
2 This document provides guidance on how an organization can develop information security
3 measures to identify the adequacy of in-place security policies, procedures, and controls. It
4 explains the measures prioritization process and how to evaluate measures.

5 Keywords
6 assessment; information security; measurement; measures; metrics; performance; qualitative;
7 quantitative; reports; security controls.

8 Reports on Computer Systems Technology

9 The Information Technology Laboratory (ITL) at the National Institute of Standards and
10 Technology (NIST) promotes the U.S. economy and public welfare by providing technical
11 leadership for the Nation’s measurement and standards infrastructure. ITL develops tests, test
12 methods, reference data, proof of concept implementations, and technical analyses to advance
13 the development and productive use of information technology. ITL’s responsibilities include
14 the development of management, administrative, technical, and physical standards and
15 guidelines for the cost-effective security and privacy of other than national security-related
16 information in federal information systems. The Special Publication 800-series reports on ITL’s
17 research, guidelines, and outreach efforts in information system security, and its collaborative
18 activities with industry, government, and academic organizations.

19 Audience
20 This guide is written primarily for users with responsibilities or interest in information security
21 measurement and assessment. Government and industry can use the concepts, processes, and
22 candidate measures presented in this guide.
23

i
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

24 Call for Patent Claims

25 This public review includes a call for information on essential patent claims (claims whose use
26 would be required for compliance with the guidance or requirements in this Information
27 Technology Laboratory (ITL) draft publication). Such guidance and/or requirements may be
28 directly stated in this ITL Publication or by reference to another publication. This call also
29 includes disclosure, where known, of the existence of pending U.S. or foreign patent
30 applications relating to this ITL draft publication and of any relevant unexpired U.S. or foreign
31 patents.
32 ITL may require from the patent holder, or a party authorized to make assurances on its behalf,
33 in written or electronic form, either:
34 a) assurance in the form of a general disclaimer to the effect that such party does not hold
35 and does not currently intend holding any essential patent claim(s); or
36 b) assurance that a license to such essential patent claim(s) will be made available to
37 applicants desiring to utilize the license for the purpose of complying with the guidance
38 or requirements in this ITL draft publication either:
39 i. under reasonable terms and conditions that are demonstrably free of any unfair
40 discrimination; or
41 ii. without compensation and under reasonable terms and conditions that are
42 demonstrably free of any unfair discrimination.
43 Such assurance shall indicate that the patent holder (or third party authorized to make
44 assurances on its behalf) will include in any documents transferring ownership of patents
45 subject to the assurance, provisions sufficient to ensure that the commitments in the assurance
46 are binding on the transferee, and that the transferee will similarly include appropriate
47 provisions in the event of future transfers with the goal of binding each successor-in-interest.
48 The assurance shall also indicate that it is intended to be binding on successors-in-interest
49 regardless of whether such provisions are included in the relevant transfer documents.
50 Such statements should be addressed to: [email protected].
51

ii
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

52 Note to Reviewers
53 The initial public drafts (ipd) of NIST Special Publication (SP) 800-55, Measurement Guide for
54 Information Security, Volume 1 – Identifying and Selecting Measures and Volume 2 – Developing
55 an Information Security Measurement Program are available for comment after extensive
56 research, development, and customer engagement.
57 In response to the feedback from the pre-draft call for comment and initial working draft
58 (annotated outline), NIST continued to refine the publications by organizing the guidance into
59 two volumes and developing more actionable and focused guidance in each.
60 • Volume 1 – Identifying and Selecting Measures – is a flexible approach to the
61 development, selection, and prioritization of information security measures. This
62 volume explores both quantitative and qualitative assessment and provides basic
63 guidance on data analysis techniques as well as impact and likelihood modeling.
64 • Volume 2 – Developing an Information Security Measurement Program - is a
65 methodology for developing and implementing a structure for an information security
66 measurement program.
67 Reviewers are encouraged to comment on all or parts of draft NIST SP 800-55 Measurement
68 Guide for Information Security, Volume 1 – Identifying and Selecting Measures and Volume 2 –
69 Developing an Information Security Measurement Program. NIST request comments be
70 submitted to [email protected] by 11:59 PM Eastern Time (ET) on March 18, 2024.
71 Commenters are encouraged to use the comment template provided with the document
72 announcement.
73

iii
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

74 Table of Contents
75 1. Introduction ...................................................................................................................................1
76
77
78
79
80 2. Fundamentals ................................................................................................................................4
81
82
83
84
85 2.4.1. Measures Documentation .......................................................................................................... 10
86 2.4.2. Data Management...................................................................................................................... 11
87 2.4.3. Data Quality ................................................................................................................................ 12
88 2.4.4. Uncertainty and Errors ............................................................................................................... 12
89
90 3. Selecting and Prioritizing Measures ..............................................................................................16
91
92
93 3.2.1. Implementation Measures ......................................................................................................... 17
94 3.2.2. Effectiveness Measures .............................................................................................................. 17
95 3.2.3. Efficiency Measures .................................................................................................................... 17
96 3.2.4. Impact Measures ........................................................................................................................ 17
97 3.2.5. Comparing Measures and Assessment Results .......................................................................... 18
98
99 3.3.1. Likelihood and Impact Modeling ................................................................................................ 19
100 3.3.2. Weighing Scale ........................................................................................................................... 19
101
102 References.......................................................................................................................................22
103 Appendix A. Glossary .......................................................................................................................24
104 Appendix B. Data Analysis Dictionary ...............................................................................................27
105
106
107
108 Appendix C. Modeling Impact and Likelihood ...................................................................................30

iv
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

109
110
111
112
113 Appendix D. Change Log ...................................................................................................................32

114 List of Tables

115 Table 1. Stevens Scale of Measurement ..............................................................................................3
116 Table 2. Data analysis examples .........................................................................................................8
117 Table 3. Data cleaning methods for reducing uncertainty ..................................................................12
118 Table 4. Examples of measures and types of qualitative and semi-quantitative assessment results .... 18

119 List of Figures

120 Fig. 1. Notional process for the definition, collection, and analysis of metrics ....................................14
121

v
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

122 1. Introduction
123 Information security measurement enables organizations to describe and quantify information
124 security, allocate finite resources, and make informed and data-driven decisions. However,
125 organizations first need to know what policies, procedures, and controls they have in place at
126 any given time; whether those countermeasures are working effectively and efficiently; and
127 how the organization and its risks are impacted. By developing and monitoring measurements
128 that evaluate what an organization has in place for information security risk management and
129 how well those efforts are working, an organization can better address their goals and direct
130 resources.

131 1.1. Purpose and Scope

132 NIST Special Publication (SP) 800-55v1, r2 is a flexible guide to the development and selection
133 of information security measures at the organization, mission/business, and system levels to
134 identify the success of in-place policies, procedures, and controls.1 This document expands on
135 previous NIST work on information security measures and measurements by focusing on
136 quantitative assessment 2 and addressing organizational or program maturity.
137 The Measurement Guide for Information Security, Volume 2 – Program, provides a
138 methodology for implementing an information security measurement program. Additionally,
139 while many of the principles of information security measurement may apply to privacy, privacy
140 is out of scope for this document.

141 1.2. Relationship to Other NIST Publications

142 This document is intended to provide considerations for measuring the information security
143 program activities described in other NIST publications, including:
144 • SP 800-137A, Assessing Information Security Continuous Monitoring Programs [14]
145 • Framework for Improving Critical Infrastructure Cybersecurity, Version 2.0 (NIST
146 Cybersecurity Framework) [1]
147 • SP 800-30r1, Guide for Conducting Risk Assessments [9]
148 • SP 800-37r2, Risk Management Framework for Information Security Systems and
149 Organizations: A System Life Cycle Approach for Security and Privacy [10]
150 • SP 800-161r1, Cybersecurity Supply Chain Risk Management Practices for Systems and
151 Organizations [17]
152 • NIST Engineering Handbook [18]

1
This document uses the term controls to broadly describe identified countermeasures for managing information security risks. It is intended to
be framework- and standard-agnostic and can also apply to other existing models or frameworks.
2 SP 800-55 uses the terms quantitative assessment and measurement synonymously. Refer to Sec. 1.4, Document Terminology, for additional

information.

1
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

153 • NIST Internal Report (IR) 8286, Identifying and Estimating Cybersecurity Risk for
154 Enterprise Risk Management (ERM) [4]

155 1.3. Document Organization

156 The remaining sections of this document discuss the following:
157 • Section 2, Information Security Measurement Development Process
158 • Section 3, Measurement Development and Selection
159 • Appendix A, Glossary
160 • Appendix B, Data Analysis Dictionary
161 • Appendix C, Likelihood and Impact Models
162 • Appendix D, Change Log

163 1.4. Document Terminology

164 In the context of this document, the follow terms are defined as follows:
165 • Assessment: The action of evaluating, estimating, or judging against defined criteria.
166 Different types of assessment (qualitative, quantitative, and semi-quantitative) are used
167 to assess risk. Some types of assessment yield measures.
168 • Assessment result: The output or outcome of an assessment.
169 • Information security3: The protection of information and systems from unauthorized
170 access, use, disclosure, disruption, modification, or destruction to provide
171 confidentiality, integrity, and availability. [2]
172 • Measurement: The process of obtaining quantitative values using quantitative methods.
173 • Measures: Quantifiable and objective values that result from measurement.
174 • Metrics: Measures and assessment results designed to track progress, facilitate
175 decision-making, and improve performance with respect to a set target.
176 • Qualitative assessment: The use of a set of methods, principles, or rules for assessing
177 risk based on nonnumerical categories or levels. [9]
178 • Quantitative assessment: The use of a set of methods, principles, or rules for assessing
179 risks based on the use of numbers where the meanings and proportionality of values are
180 maintained inside and outside of the context of the assessment. [9]
181 • Semi-quantitative assessment: The use of a set of methods, principles, or rules for
182 assessing risk based on bins, scales, or representative numbers whose values and
183 meanings are not maintained in other contexts. [9]

3 The term information security can be used interchangeably with cybersecurity.

2
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

This document discusses concepts that are similar to the Stevens Scale of Measurement,
as shown in Table 1, but takes a different view on what is and is not a measurement. For
the purposes of this document, a nominal scale is considered a form of data gathering,
and an ordinal scale is considered a ranking system. Both interval and ratio scales use
variables that represent true numbers and can be used in a quantitative assessment, so
they are considered measurement [19].

184
185 Table 1. Stevens Scale of Measurement

Scale Level Definition

Nominal A nominal scale only looks at classification or identification.
Nominal scales are used in surveys and in dealings with either non-
numeric variables or numbers that do not have an assigned value.
The data collected from a nominal scale can be used for counting,
mode, or correlation contingency matrices.
Ordinal An ordinal scale is similar to a nominal scale in that it primarily uses
non-numeric values or numbers that are meant to show ranking.
Related statistics include medians and percentiles.
Interval An interval scale is used when measuring variables with equal
intervals between values. When using an interval scale, there is no
true zero. Examples of the use of interval scales are temperature or
time scales. Interval data allows for quantitative analysis, such as
descriptive statistics like frequency, averages, position, and
dispersion. Interval statistics include mean, standard deviation, and
rank-order correlation.
Ratio Ratio scales allow for the categorization and ranking of data, similar
to an interval scale, but with a true zero and no negative values.
Ratio scales allow for numbers to be used for addition, subtraction,
multiplication, and division.

186

3
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

187 2. Fundamentals
188 The terms measurement and assessment are often used interchangeably in the information
189 security field. This document provides a lexicon for key terminology and an overview of
190 foundational concepts to those looking to measure and assess information security risk and
191 clarifies the distinction between measurement and assessment. As described in Sec. 1.4,
192 assessment refers to the process of evaluating, estimating, or judging against defined criteria,
193 and measurement is the process of obtaining quantitative values. Hence, assessment is a
194 broader concept that also includes measurement.
195 Organizations perform multiple kinds of assessment when evaluating information security risk,
196 such as risk assessments, program assessments, and control assessments. Risk assessments are
197 used to identify the risks that an organization faces and can support decision-making [9].
198 Program-level assessments are used for decision-making about the strategies, policies,
199 procedures, and operations that determine the security posture of an information security
200 program. In control assessments, organizations evaluate whether specific controls are
201 performing the way they were intended and achieving the desired results. Both program
202 assessments and control assessments are in and of themselves a form of risk assessment and
203 provide a different lens for viewing information security risk. SP 800-55 is intentionally agnostic
204 on specific risk assessment models. However, many identify threat, likelihood, vulnerability,
205 and impact as areas to assess. 4

206 2.1. Types of Assessment

207 There are three types of assessment:
208 1. Qualitative assessments use non-numerical values or categories, such as high, medium,
209 and low or heat maps.
210 2. Semi-quantitative assessments use numbers, but those numbers do not maintain their
211 value outside of the assessment context. This is commonly seen in models that use
212 number rankings to show a level of organizational integration. While the assessment
213 may say that the organization is at “level 3,” that “3” represents a set of qualities rather
214 than a numerical value.
215 3. With quantitative assessments, any numbers used retain their value outside of the
216 context. For example, 98 % of authorized accounts belong to current employees, and 2
217 % belong to former employees. Here, the values “98 %” and “2 %” stay the same
218 regardless of the context. Since measurement is the process of obtaining quantifiable
219 values using quantifiable assessment methods, measures are quantitative assessment
220 results.
221 Quantitative assessments (i.e., measurements) can provide objective data that allows for
222 tracking and shows changes. However, they can be more difficult to produce since they require
223 more data and resources than qualitative assessments. In contrast, qualitative assessments

4 For additional information about risk assessment models, see [9].

4
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

224 may be more commonly used and easier to conduct, but their results can also be subjective and
225 require everyone to have an equal understanding of the scale used.
226 Organizations will first consider their motivations for measuring information security risks
227 before determining whether a quantitative or qualitative assessment is appropriate. For
228 example, an organization motivated primarily by compliance with an industry certification or
229 international standard has different measurement needs than an organization motivated by
230 cost reduction. An organization could have multiple, competing motivations that drive the
231 identification and selection of measures.

Some organizational motivations may benefit from quantitative assessments, such as trying
to determine whether the organization is patching known vulnerabilities in an acceptable
amount of time. Knowing the mean time to remediate a vulnerability provides more precise
insight into patching efficiency than simply knowing the number of vulnerabilities patched in
a year. Because the question of mean time to remediate a vulnerability deals in non-zero
numbers that are attainable to gather, a measurement can be taken, and a mathematically
derived answer can be given.
232
233 When real and attainable numbers based on gathered data can be found and analyzed, a
234 quantitative assessment may be the appropriate action. If there are proposed questions that do
235 not have measurable numbers attached to them but still need to be addressed, a qualitative
236 assessment may be the best option.
237 Commonly used qualitative methods include color scales that represent risk levels or number
238 scales that show rankings. For the purposes of this document, qualitative and semi-quantitative
239 assessments are not considered measurement, and the values produced by these types of
240 assessments are not considered measures. Most organizations will use a mixture of
241 quantitative, semi-quantitative, and qualitative assessments. Ultimately, some or all the
242 assessment results will be used to determine success.
243 In addition to measurement, organizations also utilize metrics to track progress, facilitate
244 decision-making, and improve performance. Information gained from measurement may be
245 used to identify and define new metrics. Metrics can be applied at the system level, program
246 level, 5 and organization level. System-level metrics, such as the frequency of third-party access
247 to a system or the number of communication ports open, can facilitate tactical decision-making
248 and support program-level metrics. Program-level metrics, such as the number of security
249 incidents in a year or the cost per incident, may be helpful when making organizational
250 strategic decisions. Both system- and program-level metrics can also support risk management-
251 informed decision-making.

5
SP 800-39, Managing Information Security Risk: Organization, Mission, and Information System View, includes a model of multi-level risk
management for the integration of risk management across the organization. In this model, three levels are identified to address risk: (i) the
organization level, (ii) the missions/business process level, and (iii) and the system level. For the purpose of this document, the program-level
can be synonymous with the mission/business process-level and/or the organization-level.

5
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

252 2.2. Benefits of Using Measures

253 Developing and establishing measures to capture and provide meaningful data at all levels of an
254 organization requires careful consideration. Meaningful measures take organizational
255 information security goals and objectives into account and are obtainable, repeatable, and
256 feasible to measure. Information security measurement enables organizations to quantify
257 improvements or gaps in securing systems and demonstrate quantifiable progress in
258 accomplishing strategic goals and objectives. Well-designed measurements can provide
259 information on the implementation, effectiveness, efficiency, and business impacts of controls,
260 such as the results of information security activities, events (e.g., incident data, revenue lost to
261 cyber attacks), and information security investments.
262 Measurement also provides data that can enable an organization to examine the impacts of
263 implementing information security programs, specific controls, and associated policies and
264 procedures. Such data is integral when making risk-based decisions, weighing performance
265 against designated metrics, and demonstrating compliance. Measurement can also increase
266 accountability by providing data that can facilitate the identification of the personnel
267 responsible for controls implemented within specific organizational components or systems
268 and support an environment that allows for continuous analysis and improvement.

269 2.3. Measurement and Quantitative Assessment

270 Measures are numerically expressed data that are gathered through the process of
271 measurement.6 Measures can be derived from any operations or systems that can be measured
272 with numbers. Quantitative assessments judge measures data against a set criteria or target
273 and can be used to analyze information security risks using frequency, rates, financial impacts,
274 and other numeric indicators.
275 Using quantitative assessments requires a knowledge of measurement techniques and data
276 analysis processes. One challenge of measurement is using the right measures and quantity of
277 measures to perform useful analysis. A single measure alone may not provide sufficient data to
278 make risk-based decisions, but organizations may also have restraints on resources that prevent
279 them from employing and analyzing every potential measure. An organization finds the number
280 of measures and depth of analysis that work best for their needs.
281 The ability to measure information security risk relies on data availability. Methods for
282 collecting information security data may include experimentation, observation, or sampling.
283 The NIST Engineering Statistics Handbook [18] offers detailed information on choosing a
284 sampling scheme, including the following methods:
285 • Experimentation is a systematic approach to testing new ideas, methods, or activities
286 that applies principles and techniques at the data collection stage to ensure the
287 generation of valid, defensible, and supportable conclusions. A recognizable use of
288 experimentation to collect information security data is a phishing test, which is a form of

6As described in Section 1.4, measures and quantitative assessment results can be used synonymously, as can the terms measurement and
quantitative assessment.

6
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

289 internal security testing where organizations send fake phishing emails to determine
290 which users respond to it. The rates of success are then judged against set criteria.
291 • Observational data refers to data captures through the observation of an activity or
292 behavior without the direct involvement of the subject. Observational data is often
293 gathered as part of routine information security operations, such as log management
294 tools that are used to collect and analyze network activities. Data from these logs are
295 observational and can be used for further analysis.
296 • Sampling is the process of taking samples of something for the purpose of analysis.
297 Sampling may be used when continuous observation and passive data collection are not
298 an option or when random, stratified, or systematic sampling may be preferred.
299 Random sampling is a method of sampling where each sample has an equal chance of
300 selection in hopes of gathering an unbiased representation. Stratified sampling is the
301 process of segmenting a population across levels of some factors to minimize variability
302 within those segments (e.g., taking a sample from a terminal in each department of an
303 organization). Systematic sampling is a method of sampling where samples are taken at
304 a regular interval (e.g., once an hour or from every tenth user).
305 Once the data from measurement is procured, the outputs of quantitative analysis can be used
306 in a quantitative assessment to determine whether the organization is meeting its information
307 security goals and support risk-based decision-making. Data analysis methods 7 are largely
308 based off of the type of questions that the organization is asking about their information
309 security risks, program, and controls. The NIST Engineering Statistics Handbook [18] identifies
310 three popular approaches to data analysis:
311 1. Classical — The classical data analysis approach is when data collection is directly
312 followed by modeling, and the analysis, estimation, and testing that come after focus on
313 the parameters of that model. Classical data analysis includes deterministic and
314 probabilistic models, such as regression and the analysis of variance (ANOVA).
315 2. Exploratory — Exploratory data analysis begins by inferring what model would be
316 appropriate before trying different analytic models. Identifying patterns in the data may
317 give insight as to what models would produce the most useful information. Some
318 common exploratory data analysis graphical techniques include standard deviation plots
319 and histograms.
320 3. Bayesian — Bayesian methodology consists of formally combining both the prior
321 distribution of the parameters and the collected data to jointly make inferences and/or
322 test assumptions about the model of parameters. Bayesian methods can be used for
323 expected range setting and predictive models.
324 Table 2 shows examples of quantitative analysis across risk assessment, program-level
325 assessment, and control-level assessment.

7 Appendix C describes additional examples of quantitative data analysis methods.

7
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

326 Table 2. Data analysis examples

Type of Assessment Approach Example

Risk Assessment Classical (Value at Risk) An organization conducting a risk assessment will
likely consider their value at risk (VaR) if they were
to suffer an adverse information security event.
The organization may look at potential losses from
downtime, the cost of repairing the environment,
or reputational damage.
Risk Assessment Bayesian The Bayesian method looks at prior distribution,
collected data, and set parameters to make
inferences about future outcomes. Using data
from SP 800-53 control RA-3(4), Predictive Cyber
Analytics, as part of a risk assessment, the
inferences found through the Bayesian method
allow organizations to make risk-based decisions
based on the likelihood of future events.
Program-Level Assessment Classical (Mean) At the program level, an organization may choose
to identify the mean time it takes to complete an
action. For example, using SP 800-53 control PM-
22, Personally Identifiable Information Quality
Management, the mean time to correct or delete
inaccurate or outdated personally identifiable
information is measured. The organization may
also consider the variance in that data from year
to year or see whether certain individuals are
addressing that personally identifiable information
at different rates.
Program-Level Assessment Exploratory Data Analysis An organization may want to use a scatter plot as
(Scatter Plot) part of a program-level assessment to reveal
relationships or associations between two
variables. Using data collected as part of SP 800-
53 control PM-31, Continuous Monitoring
Strategy, one can examine linear relationships
shown in a scatter plot of historical data. The
scatter plot can reveal outliers or information
about what typical uses of a system.
Program-Level Assessment Bayesian The Bayesian method can be used to influence
programmatic decisions around continuous
improvement. For example, using SP 800-53
control PM-6, Measures of Performance, and the
Bayesian method on prior historical data, an
organization can determine what future data may
look like. This information on future outcomes can
be used to set the expected results of information
security performance.
Control Assessment Classical (Linear Regression) At the control level, an organization may have
implemented continuous monitoring (i.e., SP 800-
53, control CA-7) of a specific system-level metric.
The data provided by the continuous monitoring
of a system can be used in linear regression to
learn what “normal” looks like for that system,

8
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

Type of Assessment Approach Example

which in turn allows the organization to identify
deviations from that “normal.” This is a
foundational piece of the information security
measurement and assessment process.
Control Assessment Exploratory Data Analysis At the control level, a multi-factor/comparative
box plot could be used to compare the key
characteristics or unusual data in a data set
monitoring a control.
Control Assessment Bayesian The Bayesian method may be used to make
decisions about the frequency of equipment
maintenance using SP 800-53 control MA-6(2),
Timely Maintenance | Predictive Maintenance and
historical data about organizational equipment.
327

Organizations that are early in the process of assessing their information security risks,
program, or systems may rely heavily on qualitative assessments that present non-
numerical information in place of measurement. These non-numerical methods can help
show context, examine labels, and look at behavior. A prominent example of qualitative
assessment featured in many information security measurement programs is the risk
matrix — a table that uses colored rating scales to show the impact and likelihood of
various risks. As organizations gain the ability to record and track information security
data, they are able to move away from the subjectivity of qualitative assessments and
toward the increased precision and reduced bias of quantitative assessments.
328

329 2.4. Measurement Considerations

330 Because measurement can involve large amounts of data, having a plan for data handling is
331 critical to ensure that factors such as documentation, data management, data quality, and
332 uncertainty are all considered. An organized and repeatable process that allows for the
333 consistent assessment of collected data provides much-needed context for measurements.
334 Information security measurements can be scoped to a variety of environments and needs.
335 Assets, controls, vulnerabilities, and security incidents can all be measured. Measures can be
336 applied to organizational units, sites, or other constructs. Organizations will carefully define the
337 scope of measures based on specific stakeholder needs, strategic goals and objectives,
338 operating environments, risk priorities, and resources.
339 Information security measures can be applied at the system level to provide quantifiable data
340 regarding the implementation, effectiveness, efficiency, and impact of required or desired
341 security controls. System-level measures can be used to determine the system’s security
342 posture, demonstrate compliance with organizational requirements, and identify areas of
343 improvement.

9
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

344 Measurements can be used to monitor organizational information security activities at the
345 program and organization levels. These measurements may be derived by aggregating multiple
346 system-level measures or developed by using the entire enterprise as the scope. Organization-
347 level measurements require that the processes on which the measures depend are consistent,
348 repeatable, and ensure the availability of data across the organization.
349 Perfectly measuring information security is challenging due to the gap between mathematical
350 models and practical implementations [21]. Instead, experimenting as possible with relative
351 metrics, models, and approaches over time is the best way to identify the most effective
352 performance indicators.

353 2.4.1. Measures Documentation

354 Organizations document their measures in a standard format to ensure the repeatability of
355 measures development, collection, and reporting activities. By keeping a consistent record of
356 what is being measured, where the data comes from, what formulas and calculations are being
357 used, and who interacts with the data, it becomes easier to trace data and ensure continuity of
358 the process.
359 Organizations can tailor their standard format to their unique environments and requirements
360 based on internal practices and procedures. However, the following fields offer a common
361 starting point:
362 • Unique ID: A unique identifier for tracking and sorting. The unique identifier can use an
363 organization-specific naming convention or directly reference another source.
364 • Goal: Statement of strategic and/or information security goals to guide control
365 implementation for system-level control measures as well as higher-level measures.
366 These goals are usually articulated in strategic and performance plans. When possible,
367 include both the organization-level goal and the specific information security goal
368 extracted from organization documentation, or identify an information security goal
369 that would contribute to the accomplishment of the selected strategic goal.
370 • Measure: Statement of measurement. Use a numeric statement that begins with the
371 word “percentage,” number,” “frequency,” “average,” or other similar term.
372 • Type: Statement of whether this is a record of implementation or a measure of
373 effectiveness, efficiency, or impact.
374 • Formula: Calculation that results in a numeric expression of a measure. The organization
375 may also note the information gathered in an implementation survey.
376 • Target: Threshold for a satisfactory rating for the measure (e.g., a milestone completion
377 or statistical measure) that can be expressed in percentages, time, currency, or other
378 unit of measurement. The target may be tied to a required completion timeframe. It
379 may also be useful to select and record final and interim targets to track progress
380 toward a stated goal.

10
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

381 • Implementation evidence: Evidence used to compute the measure, validate that the
382 activity is performed, and identify probable causes of unsatisfactory results for a specific
383 measure.
384 o For manual data collection, identify questions and data elements that would
385 provide the data inputs necessary to calculate the measure’s formula, qualify the
386 measure for acceptance, and validate the information provided.
387 o For automated data collection, identify data elements that would be required for
388 the formula, qualify the measure for acceptance, and validate the information
389 provided.
390 • Frequency: How often the data is collected, analyzed, and reported. Select the
391 frequency of data collection based on a rate of change that is being evaluated. Select
392 the frequency of data reporting based on external reporting requirements and internal
393 customer preferences.
394 • Responsible parties: Key stakeholders, such as:
395 o Information owner — Identify the organizational component and the individual
396 who owns the required information.
397 o Information collector — Identify the organizational component and the
398 individual responsible for collecting the data. 8
399 o Information customer — Identify the organizational component and the
400 individual who will revive the data.
401 • Data source: Location of the data to be used in calculating the measure, including
402 databases, tracking tools, logs, organizations, and specific roles within the organization
403 that can provide the required information.
404 • Reporting format: Indication of how the measure will be reported, such as a pie chart,
405 line chart, bar graph, or other format. It may also be beneficial to include a sample.

406 2.4.2. Data Management

407 Although substantial amounts of information security data may be collected, not all data will be
408 useful for the information security measurement program at any given point in time. Any data
409 collected specifically for information security measures are as nonintrusive as possible and of
410 maximum usefulness to ensure that available resources are primarily used to correct problems
411 rather than collect data.
412 The information in information security data repositories represents a significant collection of
413 operational and vulnerability data. Due to the sensitivity of this data, information security

8 When possible, the information collector will be a different individual or even a representative of a different organizational unit than the
information owner to avoid the possibility of a conflict of interest and ensure separation of duties, though this may not be feasible for smaller
organizations.

11
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

414 performance measurement data repositories are protected in accordance with applicable laws,
415 regulations, policies, and procedures.

416 2.4.3. Data Quality

417 Data collection methods and the data repositories used for measures data collection and
418 reporting (either directly or as data sources) are clearly defined to ascertain the quality and
419 validity of the data. This also helps ensure that testing is repeatable and can show changes over
420 time.
421 Data validity is suspect if the primary data source is an incident-reporting database that only
422 stores information reported by a few organizational elements or if reporting processes between
423 organizations are inconsistent. The importance of standardizing reporting processes cannot be
424 overemphasized. When organizations are developing and implementing processes that may
425 serve as inputs into an information security measurement program, while ensuring that data
426 gathering and reporting are clearly defined to facilitate valid data collection. Having a validation
427 process in place to check the integrity, accuracy, and structure of the data provides a way to
428 address potential errors before any analysis is done. By setting a standard process to validate
429 data, an organization can have a repeatable way to look at the data and ensure its quality.

430 2.4.4. Uncertainty and Errors

431 Even when measurements are intended to be precise and accurate, random and systemic
432 errors can still occur. While there is no guaranteed way to measure uncertainty in all
433 measurements, statistical information calculated from the data (e.g., standard deviation,
434 standard error of mean, and confidence intervals) can provide more insight.
435 Uncertainty can be reduced by using data cleaning methods, such as validation, normalization,
436 transformation, and imputation, as shown in Table 3.
437 Table 3. Data cleaning methods for reducing uncertainty

Data Cleaning Method Definition

Data Validation The process of determining that collected data is
acceptable according to a predefined set of tests
[15]
Normalization The conversion of information into consistent
representations and categorizations [4]
Transformation The conversion of data from one state or format
into another state or format
Imputation The replacement of unknown, unmeasured, or
missing data with a particular value. The simplest
form of imputation is to replace all missing values
with the average of that variable [18].

438 In addition to making the data itself more useable, data analysis methods (e.g., sensitivity
439 analysis and Monte Carlo analysis) can address uncertainty within the data. Organizations often
440 make quantitative projections using statistical methods, such as regression, time series analysis,

12
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

441 and machine learning methods. When looking at projections, it is helpful to consider that future
442 events and other unknown factors can cause unforeseen changes.

443 2.5. Metrics

444 Metrics are designed to track progress, facilitate decision-making, and improve performance
445 with respect to a set target. Metrics leverage measures and assessment results to provide
446 insight into how well an organization is performing at the program or system level and whether
447 the organization is reducing their information security risk. As with measures, the
448 characteristics of meaningful metrics include the value being objective, accurate, precise, tied
449 to a fixed reference or point in time, replicable, and comparable to previous measurements.
450 Metrics are set with organizational goals in mind and drive subsequent assessments whose
451 results then inform the metrics going forward. Figure 1 shows a notional process for the
452 definition, collection, and analysis of metrics.

13
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

453
454 Fig. 1. Notional process for the definition, collection, and analysis of metrics

455 When selecting measurements and metrics to focus on, it is helpful to know why the
456 measurements are being taken and their purpose. It is important that the chosen metrics tell a
457 meaningful story about organization-, program-, or system-level information security. To do so,
458 metrics are designed to be unambiguous so that their purpose and output can be more easily
459 understood. For example, when evaluating cybersecurity awareness training, consider
460 completion rates and the results of review quizzes instead of marking participation as “low,
461 medium, or high.”
462 By keeping metrics consistent over time, organizations can evaluate long-term trends and
463 expected ranges. A new metric may provide important insight, but tracking the measurements

14
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

464 related to metrics over a continuous period (e.g., quarter to quarter, year to year) will give
465 more information about the success of organization-, program-, and system-level information
466 security plans, policies, procedures, and goals. Some metrics may be gathered because of
467 outside guidance or regulations.
468 Key risk indicators (KRIs) and key performance indicators (KPIs) are examples of metrics, though
469 not all metrics fall into these categories. Organizations may find a wide variety of metrics fit
470 their needs. For example, appropriate measures at the organization level may include the cost
471 per security incident as part of the budget allocation process, whereas measurements at the
472 system level may include the frequency of virus scans across individual systems.

15
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

473 3. Selecting and Prioritizing Measures

474 Developing and selecting information security measures consists of four major activities:
475 1. Identifying and defining the current information security program
476 2. Developing specific measures to gauge the implementation, effectiveness, efficiency,
477 and impact of security controls
478 3. Prioritizing measures based on organizational needs
479 4. Evaluating collected measures data

480 3.1. Identification and Definition

481 This document focuses on the development of measures related to information security risk
482 management, which is part of a larger implementation process of information security
483 measurement.9 The identification and definition of the existing information security program
484 are important to the development of measures.
485 Identification and definition include:
486 • Stakeholders and interests: Identifying relevant stakeholders and their interests in
487 information security measurement
488 • Goals and objectives: Identifying and documenting security goals and objectives that
489 will guide control implementation
490 • Information security policies, guidelines, and procedures review: Examining existing
491 organization-specific policies, guidelines, and procedures related to information security
492 • Information security implementation review: Reviewing any existing measures and
493 data repositories that can be used to derive measures data

494 3.2. Types of Measures

495 Knowing what controls are implemented in an organization is foundational to quantitative
496 assessment. A complete understanding of the system- and program-level controls that need to
497 be tracked are needed before an organization can evaluate what kinds of measurements to
498 take or the process of prioritizing potential measures. This creates a structure for determining
499 what measurements need to be taken and what metrics are used for evaluation.
500 There are four types of measures/assessment results:
501 1. Implementation
502 2. Effectiveness
503 3. Efficiency

9 Refer to SP 800-55 Volume 2 for more information.

16
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

504 4. Impact

505 3.2.1. Implementation Measures

506 Implementation measures demonstrate the progress of specific controls. Monitoring
507 implementation may include assessment results, such as a tally of known systems or a binary
508 “yes/no” about which systems have up-to-date patches. 10 Implementation measures look at
509 quantitative outputs and are usually demonstrated in percentages. Examples of
510 implementation measures related to information security programs include the percentage of
511 systems with approved system security plans and the percentage of systems with password
512 policies that are configured as required. Implementation measures can also examine system-
513 level areas, such as the percentage of servers in a system with a standard configuration.
514 By gathering this data, an organization can understand how its goals are being implemented
515 and what tasks still need to be accomplished. Organizations never fully retire implementation
516 measures because they are a record of what exists and what needs improvement. However,
517 once implementation measures are completed, the emphasis and resources of the
518 measurement program shift away from implementation to include effectiveness, efficiency, and
519 impact measures.

520 3.2.2. Effectiveness Measures

521 Effectiveness measures evaluate how well implementation processes and controls are working
522 and whether they are meeting the desired outcome. An effectiveness assessment can either
523 concentrate on the evidence and results of a quantitative analysis of measures or be applied in
524 a qualitative “yes/no” paradigm. Effectiveness measures may require multiple data points that
525 quantify the degree to which information controls are implemented and their effects on the
526 organization’s information security posture.

527 3.2.3. Efficiency Measures

528 Efficiency measures examine the timeliness of controls by determining the speed at which they
529 give useful feedback and how quickly those issues are addressed. An efficiency assessment
530 concentrates on the evidence and results of quantitative measures analysis.

531 3.2.4. Impact Measures

532 Impact measures articulate the impact of information security on an organization’s unique
533 mission, goals, and objectives by quantifying the following:
534 • Cost savings produced by the information security program or through costs incurred
535 from addressing information security events

10 Records of these essential implementation assessment results are foundational to information security measurement and are addressed in SP

800-55 Volume 2.

17
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

536 • Business value gained or lost

537 • The degree of public trust gained or maintained by the information security program
538 • Other mission-related impacts of information security
539 These measures combine the results of control implementation with a variety of information
540 about resources. They can provide the most direct insight into the value of information security
541 to the organization and are the ones that executives seek.

542 3.2.5. Comparing Measures and Assessment Results

543 Qualitative and semi-quantitative assessments may also be useful or even necessary to assess
544 implementation, effectiveness, efficiency, and impact, as shown in Table 4.
545 Table 4. Examples of measures and types of qualitative and semi-quantitative assessment results

Examples of Qualitative or
Assessment Types Semi-Quantitative Assessment Examples of Measures
Results
Implementation: Examine the Determine whether identified The percentage of systems with
progress of specific controls. controls are in place. up-to-date patches (i.e.,
implementation of a specific
control or capability)
Effectiveness: Examine how well Use a color-coded risk matrix to A chart that shows the changes of
controls are working. demonstrate the potential risks percentage of information
involved with improperly security incidents caused by
configured access controls. improperly configured access
controls over a 5-year period
Efficiency: Examine the timeliness of Use a 1–5 scale to determine Data that compares the mean
controls. whether the organization is at an time of response to information
acceptable level of security incidents versus the cost
responsiveness in case of an of the incident
information security incident.
Impact: Examine the impact of Rank risks on a color-coded scale Data on the known costs of
information security on an to evaluate financial impacts to breaches to industry peers
organization’s mission. an organization.

546 3.3. Prioritizing Measures

547 Most organizations have constraints that prevent gathering and analyzing measurement data
548 on every possible measurement data source. For this reason, after implementation measures
549 are in place, organizations prioritize which efficiency, effectiveness, and business impact
550 measures to implement. Prioritization can be driven by a variety of factors, including an
551 organization’s risk management strategy, mission and business objectives, information from
552 risk assessments, policies, and legal, regulatory, or other requirements.

18
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

553 3.3.1. Likelihood and Impact Modeling

554 Likelihood and impact modeling are meant to work in tandem as part of a larger risk
555 assessment process.11 Simply knowing either the likelihood or the potential impact of an event
556 is not enough information to determine the importance of a potential measure to an
557 organization.
558 Identifying existing data for use in likelihood and impact modeling typically involves working
559 with stakeholders from across the organizational structure. When possible, data from existing
560 risk assessments can be utilized to reduce redundancy and enable decision-making (e.g., using
561 existing modeled data to help decide what measurements to prioritize). Organizations may also
562 have in-house knowledge from audits, interviews, surveys, or studies that can provide useful
563 data points. In addition to existing internal data, external data on likelihood and impact can be
564 useful as well. Published annual reports can provide information on threat landscapes and the
565 financial impacts of information security incidents that can be used to create models. A wide
566 range of event likelihood models can be used to assess the likelihood of adverse events when
567 determining which systems and controls to measure.
568 Organizations can also compare impact models with event probability models (e.g., expected
569 loss and statistical analysis of historical market trends) to determine their measurement
570 priorities. Controls or systems with higher likelihoods of incident or higher potential impacts if
571 affected could then be prioritized when organizations decide how to allocate measurement
572 resources. Where possible, leverage existing event likelihood and impact models (e.g., risk
573 registers12) to avoid a duplication of efforts. More information on quantitative likelihood and
574 impact models can be found in Appendix C.
575 In addition to using historical information for likelihood and impact modeling, current trends
576 may provide useful datapoints when prioritizing and selecting measures. Staying up to date on
577 current threats allows for more effective continuous measurement and assessment. At the
578 same time, organizations consider recency bias13 about current events when determining
579 courses of action and resource allocation. Outliers and unexampled events may occur over
580 time. An organization can prepare for these issues using horizon scanning, stress tests, and
581 system resilience. 14

582 3.3.2. Weighing Scale

583 Information gained from modeling likelihood and impact can be combined with knowledge
584 about organizational goals and existing controls to create a customized weighing scale to
585 prioritize potential measures. Using a weighing scale with set parameters ensures consistency

11
More information on risk assessments can be found in SP 800-30, Guide for Conducting Risk Assessments.
12 More information on risk registers can be found in [4].
13 Recency bias is the tendency to favor recent events or experiences over historical ones.
14 More information on cyber resiliency can be found in SP 800-160 Volume 2.

19
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

586 when prioritizing and selecting measures, even those that are unrelated to information
587 security. Measures that are ultimately selected are useful for:
588 • Identifying causes of unsatisfactory performance
589 • Pinpointing areas for improvement
590 • Facilitating consistent policy implementation
591 • Redefining goals and objectives
592 • Modifying security policies

593 3.4. Evaluating Methods for Supporting Continuous Improvement

594 After an organization selects its measures, the collected data is evaluated. Evaluation may look
595 different depending on the types of measures being analyzed. Quantitative data analysis
596 methods, like those in Sec. 2.3, can be used to evaluate measures.
597 For implementation measures, evaluation may be as simple as comparing the percentage of
598 controls implemented with the goal percentage of implementation. Effectiveness, efficiency,
599 and impact measures will likely be more complicated to evaluate. Both effectiveness and
600 efficiency measures often begin by establishing average data output and evaluating acceptable
601 ranges against output going forward. For example, an organization may want to know if the
602 volume of data being transferred on the network has an anomaly. To monitor for changes, the
603 average volume of data transferred is established. An organization may also set an acceptable
604 range based on a standard deviation from this average. This may mean looking for outliers in
605 the data or monitoring for changes over time. Evaluating impact measures will likely include
606 outcomes outside of information security, such as financial outcomes or even public
607 perception.
608 Various indicators and inputs can be useful to track the effectiveness and efficiency of an
609 information security program by monitoring performance and security over time, such as:
610 • False positive rate: The proportion of positive reports that were incorrectly identified
611 • Key performance indicators: A measure of progress toward intended results
612 • Key risk indicators: A metric used to measure risk
613 • Leading indicators: A predictive metric that tracks events or behaviors that precede
614 incidents
615 • Lagging indicator: A metric that tracks the outcome of events or trends
616 • Mean time to detect: A metric that tracks the average amount of time that a problem
617 exists before it is found
618 • Mean time to recovery: A metric that tracks the average amount of time it takes to
619 recover from a product or system failure

20
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

620 Access to average outputs, acceptable ranges, and long-term data makes effectiveness and
621 efficiency measures more accurate and beneficial by enabling organizations to track changes
622 over time. Even if processes are not yet consistent, average outputs and acceptable ranges help
623 organizations set metrics. Some metrics are directly related to established averages, while
624 others are set by other sources, and established ranges may not have any effect on
625 organizational goals. While inconsistent processes will not provide meaningful data,
626 measurements may still be used to establish average outputs and acceptable ranges for future
627 analysis. Data analysis for finding average outputs and acceptable ranges will typically include
628 historical data and a forecast of what that trend would continue to look like in the future if all
629 variables stay the same.
630 It is important to remember that some measures have the potential to give misleading
631 information. Inputs such as phishing test success rates or the number of knows vulnerabilities
632 depend heavily on the quality of work behind them. A poorly designed phishing test might
633 show a better success rate while giving less information about the preparedness of the
634 workforce to recognize a well-designed phishing email. This does not mean that organizations
635 need to avoid these measures altogether, but numbers alone may not always show the whole
636 story.

21
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

637 References
638 [1] National Institute of Standards and Technology (2018) Framework for Improving Critical
639 Infrastructure Cybersecurity, Version 1.1. (National Institute of Standards and
640 Technology, Gaithersburg, MD), NIST Cybersecurity White Paper (CSWP) NIST CSWP 6.
641 https://doi.org/10.6028/NIST.CSWP.6
642 [2] National Institute of Standards and Technology (2006) Minimum Security Requirements
643 for Federal Information and Information Systems. (U.S. Department of Commerce,
644 Washington, DC), Federal Information Processing Standards Publication (FIPS) 200.
645 https://doi.org/10.6028/NIST.FIPS.200
646 [3] Bowen P, Kissel RL (2007) Program Review for Information Security Management
647 Assistance (PRISMA). (National Institute of Standards and Technology, Gaithersburg,
648 MD), NIST Interagency or Internal Report (IR) 7358.
649 https://doi.org/10.6028/NIST.IR.7358
650 [4] Stine KM, Quinn SD, Witte GA, Gardner RK (2020) Integrating Cybersecurity and
651 Enterprise Risk Management (ERM). (National Institute of Standards and Technology,
652 Gaithersburg, MD), NIST Interagency or Internal Report (IR) 8286.
653 https://doi.org/10.6028/NIST.IR.8286
654 [5] Taylor BN (2011) The current SI seen from the perspective of the proposed new
655 SI. Journal of Research of the National Institute of Standards and Technology 116(6):797.
656 https://doi.org/10.6028/jres.116.022
657 [6] Software Quality Group (2021) Metrics and Measures. (National Institute of Standards
658 and Technology, Gaithersburg, MD). Available at https://www.nist.gov/itl/ssd/software-
659 quality-group/metrics-and-measures
660 [7] Thomas D (2019). Monte Carlo Tool. (National Institute of Standards and Technology,
661 Gaithersburg, MD). Available at https://www.nist.gov/services-
662 resources/software/monte-carlo-tool
663 [8] ASTM International (2018) ASTM C1012/C1012M-18a – Standard Test Method for
664 Length Change of Hydraulic-Cement Mortars Exposed to a Sulfate Solution (ASTM
665 International, West Conshohocken, PA). https://doi.org/10.1520/C1012_C1012M-18A
666 [9] Joint Task Force Transformation Initiative (2012) Guide for Conducting Risk
667 Assessments. (National Institute of Standards and Technology, Gaithersburg, MD), NIST
668 Special Publication (SP) 800-30, Rev. 1. https://doi.org/10.6028/NIST.SP.800-30r1
669 [10] Joint Task Force (2018) Risk Management Framework for Information Systems and
670 Organizations: A System Life Cycle Approach for Security and Privacy. (National Institute
671 of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) 800-37,
672 Rev. 2. https://doi.org/10.6028/NIST.SP.800-37r2
673 [11] Chew E, Swanson MA, Stine KM, Bartol N, Brown A, Robinson W (2008) Performance
674 Measurement Guide for Information Security. (National Institute of Standards and
675 Technology, Gaithersburg, MD), NIST Special Publication (SP) 800-55, Rev. 1.
676 https://doi.org/10.6028/NIST.SP.800-55r1
677 [12] Grance T, Nolan T, Burke K, Dudley R, White G, Good T (2006) Guide to Test, Training,
678 and Exercise Programs for IT Plans and Capabilities. (National Institute of Standards and

22
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

679 Technology, Gaithersburg, MD), NIST Special Publication (SP) 800-84.

680 https://doi.org/10.6028/NIST.SP.800-84
681 [13] Kent K, Souppaya M (2006) Guide to Security Log Management. (National Institute of
682 Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) 800-92.
683 https://doi.org/10.6028/NIST.SP.800-92
684 [14] Dempsey KL, Pillitteri VY, Baer C, Niemeyer R, Rudman R, Urban S (2020) Assessing
685 Information Security Continuous Monitoring (ISCM) Programs: Developing an ISCM
686 Program Assessment. (National Institute of Standards and Technology, Gaithersburg,
687 MD), NIST Special Publication (SP) 800-137A. https://doi.org/10.6028/NIST.SP.800-137A
688 [15] Barker E, Smid M, Branstad D (2015) A Profile for U.S. Federal Cryptographic Key
689 Management Systems. (National Institute of Standards and Technology, Gaithersburg,
690 MD), NIST Special Publication (SP) 800-152. https://doi.org/10.6028/NIST.SP.800-152
691 [16] Ross, R, Winstead M, McEvilley, M (2022) Engineering Trustworthy Secure Systems.
692 (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special
693 Publication (SP) 800-160v1r1. https://doi.org/10.6028/NIST.SP.800-160v1r1
694 [17] Boyens JM, Smith AM, Bartol N, Winkler K, Holbrook A, Fallon M (2022) Cybersecurity
695 Supply Chain Risk Management Practices for Systems and Organizations. (National
696 Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP)
697 800-161r1. https://doi.org/10.6028/NIST.SP.800-161r1
698 [18] Heckert N, Filliben J, Croarkin C, Hembree B, Guthrie W, Tobias P, Prinz J (2012)
699 NIST/SEMATECH e-Handbook of Statistical Methods. (National Institute of Standards
700 and Technology, Gaithersburg, MD). https://doi.org/10.18434/M32189
701 [19] Stevens SS (1946) On the Scales of Measurement. Science 103(2684):677-680.
702 http://www.jstor.org/stable/1671815
703 [20] Turan MS, Barker E, Kelsey J, McKay KA, Baish ML, Boyle M (2018) Recommendation for
704 the Entropy Sources Used for Random Bit Generation. (National Institute of Standards
705 and Technology, Gaithersburg, MD), NIST Special Publication (SP) 800-90B.
706 https://doi.org/10.6028/NIST.SP.800-90B
707 [21] Stolfo S, Bellovin S, Evans D (2011) Measuring Security. IEEE Security & Privacy 9(3):60-
708 65. https://doi.org/10.1109/MSP.2011.56
709
710
711

23
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

712 Appendix A. Glossary

713 assessment
714 The action of evaluating, estimating, or judging against defined criteria. Different types of assessment (i.e.,
715 qualitative, quantitative, and semi-quantitative) are used to assess risk. Some types of assessment yield results.

716 assessment results

717 The output or outcome of an assessment.

718 Bayesian methodology

719 Statistical approach to data analysis based on Bayes’ theorem where uncertainty is quantified by combining
720 existing information with new information to create forecast models. [18, adapted]

721 classical data analysis

722 A data analysis technique where data collection is followed by the imposition of a model and the analysis,
723 estimation, and testing that follow are focused on the parameters of that model. [18, adapted]

724 data validation

725 The process of determining that data or a process for collecting data is acceptable according to a predefined set of
726 tests and the results of those tests. [15]

727 experimentation
728 A systematic approach to the process of testing new ideas, methods, or activities that applies principles and
729 techniques at the data collection stage to ensure the generation of valid, defensible, and supportable conclusions.

730 exploratory data analysis

731 A data analysis technique where data collection is immediately followed by analysis with the goal of inferring what
732 model would be appropriate. [18, adapted]

733 false positive

734 An erroneous acceptance of the hypothesis that a statistically significant event has been observed. [20]

735 imputation
736 The replacement of unknown, unmeasured, or missing data with a particular value. The simplest form of
737 imputation is to replace all missing values with the average of that variable. [18, adapted]

738 information security

739 The protection of information and systems from unauthorized access, use, disclosure, disruption, modification, or
740 destruction to provide confidentiality, integrity, and availability. [2]

741 interval scale

742 From the Stevens Scale of Measurement, a quantitative measurement scale using variables with equal values and
743 no true zero, such as time and temperature. [19, adapted]

744 key performance indicator

745 A metric of progress toward intended results.

746 key risk indicator

747 A metric used to measure risk.

748 lagging indicator

749 A metric that tracks the outcome of events or trends.

750 leading indicator

751 A predictive metric that tracks events or behaviors that precede incidents.

24
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

752 machine learning

753 The development and use of computer systems that adapt and learn from data with the goal of improving
754 accuracy.

755 mean
756 The sum of the data points divided by the number of data points. Commonly referred to as the average. [18,
757 adapted]

758 mean time to detect

759 A metric that tracks the average amount of time that a problem exists before it is found.

760 mean time to recovery

761 A metric that tracks the average amount of time that it takes to recover from a product or system failure.

762 measurement
763 The process of obtaining quantitative values using quantitative methods.

764 measures
765 Quantifiable and objective values that result from measurement.

766 median
767 The value of the point that has half the data smaller than that point and half the data larger than that point. [18]

768 metrics
769 Measures and assessment results designed to track progress, facilitate decision-making, and improve performance
770 with respect to a set target.

771 mode
772 The value of the random sample that occurs with the greatest frequency. This value is not necessarily unique. [18]

773 Monte Carlo analysis

774 A probabilistic sensitivity analysis used to account for uncertainty. [7]

775 nominal scale

776 From the Stevens Scale of Measurement, a scale that labels named variables into classifications. [19, adapted]

777 normalization
778 The conversion of information into consistent representations and categorization. [4]

779 observational data

780 Data captured through the observation of an activity or behavior without the direct involvement of the subject.

781 ordinal scale

782 From the Stevens Scale of Measurement, a scale that orders and ranks data without establishing a degree of
783 variation between ranks. [19, adapted]

784 outliers
785 An observation that lies an abnormal distance from other values in a random sample from a population. [18]

786 qualitative assessment

787 The use of a set of methods, principles, or rules for assessing risk based on non-numerical categories or levels. [6]

788 quantitative assessment

789 The use of a set of methods, principles, or rules for assessing risk based on numbers where the meanings and
790 proportionality of values are maintained inside and outside of the context of the assessment. [6]

25
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

791 random sampling

792 A method of sampling where each sample has an equal chance of selection in hopes of gathering an unbiased
793 representation. [18, adapted]

794 ratio scale

795 From the Stevens Scale of Measurement, a quantitative measurement scale with a true zero using variables that
796 can be compared to find differences or intervals. [19]

797 regression
798 A statistical technique used to predict the value of a variable based on the relationship between explanatory
799 variables.

800 sampling
801 The process of taking samples of something for the purpose of analysis.

802 semi-quantitative assessment

803 The use of a set of methods, principles, or rules for assessing risk based on bins, scales, or representative numbers
804 whose values and meanings are not maintained in other contexts. [9]

805 stratified sampling

806 The process of segmenting a population across levels of some factors to minimize variability within those
807 segments. [18]

808 systematic stratified sampling

809 A method of sampling where samples are taken at a regular interval. [18, adapted]

810 time series analysis

811 The analysis of an ordered sequence of values of a variable at equally spaced time intervals. [18, adapted]

812 transformation
813 The conversion of one state or format into another state or format.

814

26
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

815 Appendix B. Data Analysis Dictionary

816 The following information is found in the NIST Engineering Statistics Handbook.

817 B.1. Bayesian Methodology

818 Bayesian Methodology consists of formally combining the prior distribution on the parameters
819 and the collected data to jointly make inferences and/or test assumptions about the model of
820 parameters.
821 • Bayes Formula
𝑃𝑃(𝐴𝐴, 𝐵𝐵) 𝑃𝑃(𝐴𝐴) × 𝑃𝑃(𝐵𝐵|𝐴𝐴)
822 P(A|B) = =
𝑃𝑃(𝐵𝐵) 𝑃𝑃(𝐵𝐵)
823 • Law of Probability
𝑛𝑛

824 P(B) = � 𝑃𝑃(𝑃𝑃|𝐴𝐴𝑖𝑖 )𝑃𝑃(𝐴𝐴𝑖𝑖 )

𝑖𝑖=1

825 B.2. Classical Data Analysis

826 Classical data analysis is when data collection is followed by a model, and the subsequent
827 analysis, estimation, and testing focus on the parameters of that model. Classical data analysis
828 includes deterministic and probabilistic models, such as regression and ANOVA. Some of the
829 more common relevant classical quantitative models include:
830 Location
831 • Measures of Location (mean, median, and mode)
832 • Confidence Limits for Mean and One Sample t-Test
833 • Two Sample t-Test for Equal Means
834 • One Factor Analysis of Variance
835 • Multi-Factor Analysis of Variance
836 Scale (or variability or spread)
837 • Measures of Scale
838 • Bartlett’s Test
839 • Chi-Square Test
840 • F-Test
841 • Levene Test

27
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

842 Skewness and Kurtosis

843 • Measures of Skewness and Kurtosis
844 Randomness
845 • Autocorrelation
846 • Runs Test
847 Distributional Measures
848 • Anderson-Darling Test
849 • Chi-Square Goodness of Fit Test
850 • Kolmogorov-Smirnov Test
851 Outliers
852 • Detection of Outliers
853 • Grubbs Test
854 • Tietjen-Moore Test
855 • Generalized Extreme Deviate Test
856 2-Level Factorial Designs
857 • Yates Algorithm

858 B.3. Exploratory Data Analysis

859 Exploratory data analysis emphasizes graphical techniques and inferring different analytic
860 models in order to determine what model would be appropriate. Some common exploratory
861 data analysis graphical techniques include:
862 Univariate
863 𝑦𝑦 = 𝑐𝑐 + 𝑒𝑒
864 • Run Sequence Plot
865 • Lag Plot
866 • Histogram
867 • Normal Probability Plot
868 • 4-Plot
869 • PPCC Plot
870 • Weibull Plot
871 • Probability Plot

28
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

872 • Box-Cox Linearity Plot

873 • Bootstrap Plot
874 Time Series
875 𝑦𝑦 = 𝑓𝑓(𝑡𝑡) + 𝑒𝑒
876 • Run Sequence Plot
877 • Spectral Plot
878 • Autocorrelation Plot
879 • Complex Demodulation Amplitude Plot
880 • Complex Demodulation Phase Plot
881 • Decomposition
882 1 Factor
883 𝑦𝑦 = 𝑓𝑓(𝑥𝑥) + 𝑒𝑒
884 • Scatter Plot
885 • Box Plot
886 • Bihistogram
887 • Quantile Plot
888 • Mean Plot
889 • Standard Deviation Plot
890 Multi-Factor/Comparative
891 𝑦𝑦 = 𝑓𝑓(𝑥𝑥𝑥𝑥, 𝑥𝑥1, 𝑥𝑥2, … , 𝑥𝑥𝑥𝑥) + 𝑒𝑒
892 • Block Plot
893 Multi-Factor/Screen
894 𝑦𝑦 = 𝑓𝑓(𝑥𝑥1, 𝑥𝑥2, 𝑥𝑥3, … , 𝑥𝑥𝑥𝑥) + 𝑒𝑒
895 • DOE Scatter Plot
896 • DOE Mean Plot
897 • DOE Standard Deviation Plot
898 • Contour Plot
899

29
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

900 Appendix C. Modeling Impact and Likelihood

901 This appendix is intended to provide a high-level overview of complex statistical concepts. The
902 successful application of these concepts will require further training and understanding on the
903 part of practitioners.

904 C.1. Bayesian Methodology

905 Bayes’ formula expresses the conditional probability of event A given event B written as P(A|B).
906 It can be calculated using Bayes’ Rule:
𝑃𝑃(𝐴𝐴, 𝐵𝐵) 𝑃𝑃(𝐴𝐴) × 𝑃𝑃(𝐵𝐵|𝐴𝐴)
907 P(A|B) = =
𝑃𝑃(𝐵𝐵) 𝑃𝑃(𝐵𝐵)
908 Bayesian methodology is applied when there is previous knowledge of the conditions
909 associated with an event. It can provide conditional probability estimates quickly and without
910 using significant resources. Because Bayesian methodology relies on prior information, it is
911 important to note that the use of either inaccurate or a different selection of prior information
912 may lead to results that do not provide significant insight.

913 C.2. Monte Carlo Methodology

914 The Monte Carlo method is a multiple probability simulation used to predict possible outcomes
915 of an uncertain event. The Monte Carlo method uses randomly generated outcomes within a
916 set range, and the frequencies of different outcomes generated form a normal distribution.
917 The Monte Carlo method allows for repeated modeling and can be performed using
918 spreadsheet editors or programing languages for statistical computing. When using the Monte
919 Carlo method, it is important to note that these simulations show an estimated probability and
920 not an inevitable outcome.

921 C.3. Time Series Analysis

922 Time series analysis shows the level, trend, seasonality, or noise within a series of data points in
923 a time series. Time series data is often found when monitoring a process over a period. Time
924 series analysis considers the potential for an internal structure, such as trends or seasonal
925 variations to data.
926 Time series regression models are primarily used for forecasting. Time series decomposition
927 exhibits patterns within time series data and can be useful when setting the expected range or
928 use of processes or systems.

30
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

929 C.4. Value at Risk

930 Value at risk (VaR) is a statistical analysis technique that builds a model that measures the risk
931 of loss, primarily using a probability density function. The three key elements of building a VaR
932 model are a fixed time period, a specific level of loss in value, and a confidence interval.
933 Calculating VaR can be helpful when making decisions about investments and resources. Like all
934 predictive models, VaR relies heavily on the quality of inputs and cannot effectively estimate all
935 scenarios.
936

31
 NIST SP 800-55v1 ipd (Initial Public Draft) Measurement Guide for Information Security
January 2024 Volume 1 — Measures

937 Appendix D. Change Log

938 [Upon final publication, a change log will be included that describes differences from the
939 superseded version of this publication: NIST SP 800-55r1 (2008).]
940 In <date of final publication> the following changes were made to the report:
941 • …

32