Service Measurement Index
Framework Version 2.1
July 2014
CSMIC
Carnegie Mellon University Silicon Valley
Moffett Field, CA USA
Introducing the Service
Measurement Index (SMI)
The Service Measurement Index (SMI) is a set of business-relevant Key Performance Indicators (KPI's)
that provide a standardized method for measuring and comparing a business service regardless of
whether that service is internally provided or sourced from an outside company. It is designed to
become a standard method to help organizations measure cloud-based business services based on
their specific business and technology requirements. The Service Measurement Index is currently being
developed by the Cloud Services Measurement Initiative Consortium (CSMIC).
The SMI is a hierarchical framework. The top level
divides the measurement space into 7 Categories.
Each Category is further refined by 4 or more
Attributes. Then within each Attribute a set of KPI’s
are defined that describe the data to be collected for
each measure/metric. Some of these KPI’s will be
service specific while others will apply to all services
(BPaaS, IaaS, PaaS, and SaaS).
Version 2.1 of the CSMIC SMI contains the first 2
levels of this hierarchy, the Categories and Attributes.
Work is currently ongoing on the definition of the KPI’s
and measures related to several attributes. See
Figure 1: High-Level Overview to the right for a high-
level overview of the SMI. A more detailed view
(Figure 2: Detailed View) is provided on the last page
of this document. For more information see: csmic.org
The Attribute definitions provided here are all written
using these basic terms.
Client
An organization that acquires one or more cloud-based services for its users
Cloud Service Provider (CSP)
An organization that delivers one or more cloud-based services to one or more clients.
Service
A set of functions or capabilities made available to a group of users by a cloud service
provider, in order to achieve client objectives.
SLA (Service Level Agreement)
An agreement between a cloud service provider and a client documenting quantitative
service level commitments.
User
The people that interact directly with a service.
SMI Framework Version 2.1 © 2011 - 2014 Carnegie Mellon University 2
Definitions
Accountability
This category contains attributes used to measure the properties related to the cloud service provider
organization. These properties may be independent of the service being provided.
Attribute SMI Attribute Definition
Auditability The ability of a client to verify that the cloud service provider is adhering to the
standards, processes, and policies that they follow.
Compliance Standards, processes, and policies committed to by the cloud service provider
are followed.
Contracting experience Indicators of client effort and satisfaction with the process of entering into the
agreements required to use a service.
Ease of doing business Client satisfaction with the ability to do business with a cloud service provider.
Governance The processes used by the cloud service provider to manage client
expectations, issues and service performance.
Ownership The level of rights a client has over client data, software licenses, and
intellectual property associated with a service.
Provider business stability The likelihood that the cloud service provider will continue to exist throughout
the contracted term.
Provider Certifications The cloud service provider maintains current certifications for standards
relevant to their clients' requirements.
Provider Contract/SLA The cloud service provider makes available to clients SLAs adequate to
Verification manage the service and mitigate risks of service failure.
Provider Ethicality Ethicality refers to the manner in which the cloud service provider conducts
business; it includes business practices and ethics outside the scope of
regulatory compliance. Ethicality includes fair practices with suppliers, clients,
and employees.
Provider Personnel The extent to which cloud service provider personnel have the skills,
Requirements experience, education, and certifications required to effectively deliver a
service.
Provider Supply Chain The cloud service provider ensures that any SLAs that must be supported by
its suppliers are supported.
Provider Support The extent to which the cloud service provider includes or makes available
assistance to the client in their efforts to use the service, including answering
questions about the service and working around or correcting any problems
that may arise.
Sustainability The impact on the economy, society and the environment of the cloud service
provider.
SMI Framework Version 2.1 © 2011 - 2014 Carnegie Mellon University 3
Agility
Indicates the impact of a service upon a client's ability to change direction, strategy, or tactics quickly
and with minimal disruption.
Attribute SMI Attribute Definition
Adaptability The ability of the cloud service provider to adjust to changes in client
requirements.
Elasticity The ability of a cloud service provider to adjust its resource consumption for a
service at a rapid enough rate to meet client demand.
Extensibility The ability to add new features or services to existing services.
Flexibility The ability to add or remove predefined features from a service.
Portability The ability of a client to easily move a service from one cloud service provider to
another with minimal disruption.
Scalability The ability of a cloud service provider to increase or decrease the amount of
service available to meet client requirements and agreed SLAs.
Assurance
This category includes key attributes that indicate how likely it is that the service will be available as
specified.
Attribute SMI Attribute Definition
Availability The appropriateness of the service availability window, as well as the likelihood
that the availability window will actually be provided to clients.
Maintainability Maintainability refers to the ability for the cloud service provider to make
modifications to the service to keep the service in a condition of good repair.
Recoverability Recoverability is the degree to which a service is able to quickly resume a
normal state of operation after an unplanned disruption.
Reliability Reliability reflects measures of how a service operates without failure under
given conditions during a given time period.
Resiliency/Fault Tolerance The ability of a service to continue to operate properly in the event of a failure in
one or more of its components.
Service stability The degree to which the service is resistant to change, deterioration, or
displacement.
Serviceability The ease and efficiency of performing maintenance and correcting problems with
the service.
SMI Framework Version 2.1 © 2011 - 2014 Carnegie Mellon University 4
Financial
The amount of money spent on the service by the client.
Attribute SMI Attribute Definition
Billing Process The level of integration that is available between the client and cloud service
provider’s billing systems and the predictability of periodic bills.
Cost The client’s cost to consume a service over time. This includes cost of transition
of the service along with recurring costs (e.g., monthly access fees) and usage-
based costs.
Financial Agility The flexibility and elasticity of the financial aspects of the CSP’s services
Financial Structure How responsive to the client’s needs are the cloud service provider’s pricing and
billing components.
Performance
This category covers the features and functions of the provided services.
Attribute SMI Attribute Definition
Accuracy The extent to which a service adheres to its requirements.
Functionality The specific features provided by a service.
Suitability How closely the capabilities of the proposed service match the features needed
by the client.
Interoperability The ability of a service to easily interact with other services (from the same cloud
service provider and from other cloud service providers).
Service Response Time An indicator of the time between when a service is requested and when the
response is available.
SMI Framework Version 2.1 © 2011 - 2014 Carnegie Mellon University 5
Security and Privacy
This category includes attributes that indicate the effectiveness of a cloud service provider's controls on
access to services, service data, and the physical facilities from which services are provided.
Attribute SMI Attribute Definition
Access Control & Privilege Policies and processes in use by the cloud service provider to ensure that only
Management the personnel granted appropriate privileges can make use of or modify
data/work products.
Data Geographic/Political The client's constraints on service location based on geographic or political risk.
Data Integrity Keeping the data that is created, used, and stored in its correct form so that
clients may be confident that it is accurate and valid.
Data Privacy & Data Loss Client restrictions on use and sharing of client data are enforced by the cloud
service provider. Any failures of these protections are promptly detected and
reported to the client.
Physical & Environmental Policies and processes in use by the cloud service provider to protect the
Security provider facilities from unauthorized physical access, damage or interference.
Proactive Threat & Mechanisms in place to ensure that the service is protected against know
Vulnerability Management recurring threats as well as new evolving vulnerabilities.
Retention/Disposition The cloud service provider’s data retention and disposition processes meet the
clients' requirements.
Security Management The capabilities of cloud service providers to ensure application, data, and
infrastructure security based on the security requirements of the client.
Usability
The ease with which a service can be used.
Attribute SMI Attribute Definition
Accessibility The degree to which a service is operable by users with disabilities.
Client Personnel The minimum number of personnel satisfying roles, skills, experience, education,
Requirements and certification required of the client to effectively utilize a service.
Installability Installability characterizes the time and effort required to get a service ready for
delivery (where applicable).
Learnability The effort required of users to learn to use the service.
Operability The ability of a service to be easily operated by users.
Transparency The extent to which users are able to determine when changes in a feature or
component of the service occur and whether these changes impact usability.
Understandability The ease with which users can understand the capabilities and operation of the
service.
SMI Framework Version 2.1 © 2011 - 2014 Carnegie Mellon University 6
References
In addition to earlier versions of the SMI, the following references were consulted in the creation of the
Category and Attribute definitions. Due to the specific focus of the SMI on services that are or could be
cloud-based, the referenced definitions have been modified to reflect this focus.
CoBIT Control Objectives for Information and related Technology, Version 4.1, ISACA, 2007
DSCI, Security Framework: Best Practices, 2010
Gartner, Gartner Highlights Five Attributes of Cloud Computing, available at http://www.gartner.com/it/page.jsp?id=1035013
Hefley, B. and Loetsche, E., eSourcing Capability Model for Client Organizations, Van Haren, 2009
ISO 9001:2008 Quality management systems – Requirements, 2008
ISO/IEC 27001:2005 Information technology – Security techniques – Information security management systems –
Requirements, 2005
ISO/IEC 9126-1:2001 Software Engineering - Product Quality - Part 1: Quality Model
ITILV3 Glossary, English, v1, 2007 (available from http://www.itil.org/en/glossar/glossarkomplett.php)
ITILV3, Service Design, Office of Government Commerce, TSO, London, 2007
NIST, Definition of Cloud Computing (Draft), January, 2011 and Cloud Taxonomy, February, 2011
NIST Cloud Computing Reference Architecture Cloud Service Metrics Description, rev. 2.3a, draft, July, 2013
Punter, T. Using checklists to evaluate software product quality: measurement concepts for qualitative determination of quality
characteristics
TM Forum, Enabling Cloud Services, Taxonomy draft
TM Forum, Enabling End-to-End Cloud SLA Management: A Collaboration of Standard Bodies, TR 178, v2-0921, v0.4,
September, 2012
INFORMATION & COPYRIGHT
This material was prepared by Carnegie Mellon University and the members of its Cloud Services Measurement Initiative Consortium
(“CSMIC”). It is published in the interest of providing information regarding global cloud-based services. This material is subject to U.S.
copyright law and is the property of its respective creators.
NO WARRANTY
THIS MATERIAL IS FURNISHED ON AN AS-IS BASIS. NEITHER CARNEGIE MELLON UNIVERISTY NOR ANY OTHER CONTRIBUTOR
TO THE MATERIAL MAKES ANY WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, TO ANY MATTER (INCLUDING BUT
NOT LIMITED TO, WARRANTY OF FITNESS FOR A PARTICULAR PURPOSE OR MECHANTABILITY, EXLCUSIVITY, AND/OR RESULTS
OBTAINED FROM USE OF THE MATERIAL). WITHOUT LIMITING THE GENERAL NATURE OF THE PRIOR SENTENCE, NEITHER
CARNEGIE MELLON UNIVERISTY NOR ANY OTHER CONTRIBUTOR TO THE MATERIAL MAKES ANY WARRANTIES OF ANY KIND
WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK OR COPYRIGHT INFRINGEMENT.
TRADEMARKS
Use of any trademarks in this material is not intended in any way to infringe on the rights of the trademark holder.
INTERNAL USE
Permission to reproduce this document and to prepare derivative works from this document for internal use is granted, provided that the text of
the “INFORMATION & COPYRIGHT” clause and the “NO WARRANTY” clause listed above are included on all reproductions and derivative
works.
EXTERNAL USE
This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form.
SMI Framework Version 2.1 © 2011 - 2014 Carnegie Mellon University 7
Can we count on the Can it be How likely is it that How much Does it do Is the service safe and Is it easy to learn
provider organization? changed and the service will is it? what we need? privacy protected? and to use?
how quickly? work as expected?
Accountability Agility Assurance Financial Performance Security & Privacy Usability
Auditability Adaptability Availability Billing process Accuracy Access control & privilege management Accessibility
Compliance Elasticity Maintainability Cost Functionality Data geographic/political Client personnel requirements
Contracting experience Extensibility Recoverability Financial agility Interoperability Data integrity Installability
Ease of doing business Flexibility Reliability Financial structure Service response time Data privacy & data loss Learnability
Governance Portability Resiliency/fault tolerance Suitability Physical & environmental security Operability
Ownership Scalability Service stability Proactive threat &vulnerability management Transparency
Provider business stability Serviceability Retention/disposition Understandability
Provider certifications Security management
Provider contract/SLA verification
Provider ethicality
Provider personnel requirements
Provider supply chain
Provider support
Sustainability
CSMIC Framework, v2.1
07/2014
