IEEE Power & Energy Society TECHNICAL REPORT

June 2024 PES-TR 123

IEEE Power and Energy

Technology Assessment
and Roadmap
PREPARED BY THE
IEEE PES Technical Roadmap Task Force
IEEE PES Technical Council

© IEEE (2024) The Institute of Electrical and Electronics Engineers, Inc.

No part of this publication may be reproduced in any form, in an electronic retrieval system or otherwise, without the prior written permission of the publisher.
TR-123 — IEEE PES Technical Roadmap

THIS PAGE LEFT BLANK INTENTIONALLY

ii
TR-123 — IEEE PES Technical Roadmap

IEEE PES Technical Roadmap Task Force

Chair: Hong Chen

Chapter 1
Lead: Hong Chen
Contributors: Chen-Ching Liu, Bruce Kraemer, George Becker

Chapter 2
Lead: Yingchen "YC" Zhang, Mengmeng Cai
Contributors: Bruce Kraemer, Mark Siira, Eduardo Bettoni, Gary Hoffman, Rudi
Schubert, George Becker, Hong Chen, Junbo Zhao, Bendong Tan, Ketian Ye, Anamika
Dubey, Rui Yang, Elina Spyrou, Suman Debnath, Sudipta Dutta, Kamal Garg and Patel
Manish

Chapter 3
Lead: Jianhui Wang
Contributors: Hong Chen, Joe Warner, Charlie Li, George Becker, Louis Gullo,
Fangxing (Fran) Li, Theo Laughner, Gary Hoffman

Chapter 4
Lead: Yashen Lin
Contributors: Ron Melton, Annabelle Pratt, Farrokh Rahimi, Ankur Srivastava, Luka
Strezoski, QZ Sun, Junbo Zhao, Xinyang Zhou

Chapter 5
Lead: Fangxing (Fran) Li
Contributors: Rui Bo, Hantao Cui, Jianhui Wang, Xiaochuan Luo, Y.C. Zhang,
Suman Debnath, Eduardo Bettoni, Bruce Kraemer

Chapter 6
Lead: Craig Preuss
Contributors: Aditya Ashok, Venkatesh Venkataramanan, Scott R. Mix, Theo Laughner,
Mohammad Reza Khalghani, Paul Skare, Chen-Ching Liu, Manimaran Govindarasu,
Louis Gullo, Mariana Hentea, Samuel D. Chanoski

Other Task Force Members

Vijay Vital, Diane Watkins, Jim McBride, Jeffrey Nelson, Bikash Pal, Carolyn reenwell,
Michael Thompson, Doug Edwards, Dwight Alexander, Eriks Surmanis, Jens Boemer,
Jeff Britton, John Dirkman, Jim Lau, Kathy Grise, Kai Strunz, Matthew Borst, Abira
Altvater, Maria Lattanzio, Mark Bowman, Mike Mueller, Steve Hensley, Hongbin Sun,
Steve Miller, Steve Vechy, Bill Tonti, Yingli Wen, Rakesh Kumar, Maike Luiken

iii
TR-123 — IEEE PES Technical Roadmap

ACKNOWLEDGMENTS

The Task Force sincerely acknowledges the guidance from IEEE Technology Roadmaps
Committee and the invaluable review by all PES Technical Committees, Industry
Technical Support Leadership Committee and Executive Advisory Council, especially
detailed review from Mark Lauby and NERC staff, Dean Sharafi from AEMO and
Veronika Rabl from Quanta Technology, Maike Luiken, Wei-Jen Lee from The
University of Texas at Arlington, and Diane Watkins.

KEYWORDS

Climate change, Sustainability, Decarbonization, Renewable technologies, Inverter-based

resources, System integration, Technological challenges and advancements, Market and
policy

Reliability, Resilience, Power System Hardening, Restoration, Physical Security and

Human-caused Attacks, Condition Assessment, Sourcing, Prognostics and Health
Management (PHM) , Fault Tolerance , Resource Adequacy, Resilience Assessment

Grid Edge, Grid-Integrated Buildings, Electric Vehicle (EV), Energy Storage,

Forecasting, Virtual Power Plant, Optimal Power Flow, State Estimation, Distributed
Control, Advanced Distribution Management System (ADMS), Transactive Energy,
Energy Blockchain, Energy Market, Energy Policy, Microgrids

Large-scale modeling, large-scale computation, uncertainty, parallel and distributed

computing, artificial intelligence, machine learning, neural networks, quantum
computing, GPU computing, edge computing, cloud computing, open-source tools,
synthetic test systems, testbed, digital twin

Cyber security, Cyber-Informed Engineering (CIE), operational technology, cyber-

physical system, information and communication technology (ICT), Consequence-driven
Cyber-informed Engineering (CCE)

iv
TR-123 — IEEE PES Technical Roadmap

CONTENTS

1. Executive Summary .................................................................................................... 1

1.1 Setting the Stage ................................................................................................ 1
1.1.1 State of the Industry .................................................................................... 2
1.1.2 Challenges................................................................................................... 2
1.1.3 Vision ........................................................................................................... 3
1.2 Summary of Chapters ........................................................................................ 4
1.2.1 Renewable Energy Integration .................................................................... 4
1.2.2 Reliability and Resilience............................................................................. 4
1.2.3 Grid Edge Technologies .............................................................................. 5
1.2.4 Computation ................................................................................................ 5
1.2.5 Cyber Security ............................................................................................. 5
1.3 Next Step ............................................................................................................ 6
1.4 References ......................................................................................................... 6
2. Renewable Energy Integration .................................................................................... 6
2.1 Executive Summary ........................................................................................... 6
2.2 Introduction......................................................................................................... 7
2.3 Types of Renewables ......................................................................................... 9
2.3.1 Wind Energy ................................................................................................ 9
2.3.2 Solar Energy .............................................................................................. 10
2.3.3 Hydropower and Pumped Storage ............................................................ 11
2.3.4 Geothermal Energy ................................................................................... 11
2.3.5 Biomass Energy ........................................................................................ 11
2.3.6 Ocean Energy............................................................................................ 12
2.3.7 Hydrogen ................................................................................................... 12
2.3.8 Small Modular Reactors (SMRs) ............................................................... 13
2.4 Current State .................................................................................................... 14
2.5 Future State...................................................................................................... 15
2.6 Challenges ....................................................................................................... 17
2.6.1 Variability and Uncertainty of Renewable Energy Resources ................... 17
2.6.2 Grid Stability and Reliability Issues ........................................................... 18
2.6.3 Protection Challenges ............................................................................... 19
2.6.4 Decentralization ......................................................................................... 20

v
TR-123 — IEEE PES Technical Roadmap

2.6.5 Market and Policy Hurdles......................................................................... 20

2.7 Potential Solutions ............................................................................................ 21
2.7.1 Uncertainty and Risk Assessment and Management ................................ 22
2.7.2 Transmission System Upgrades ................................................................ 23
2.7.3 Distribution System Upgrades ................................................................... 24
2.7.4 Smart Grid Technologies ........................................................................... 26
2.7.5 Transmission and Distribution Coordination .............................................. 28
2.7.6 Market and Policy Enhancement ............................................................... 29
2.7.7 Future Technologies Advancement ........................................................... 30
2.8 References ....................................................................................................... 31
3. Reliability and Resilience .......................................................................................... 38
3.1 Executive Summary ......................................................................................... 38
3.2 Reliability and Resilience Definition and Metrics .............................................. 38
3.2.1 Summary ................................................................................................... 38
3.2.2 Current State ............................................................................................. 38
3.2.3 Future State ............................................................................................... 39
3.2.4 Challenges................................................................................................. 39
3.2.5 Potential Solutions ..................................................................................... 39
3.3 Extreme Weather ............................................................................................. 40
3.3.1 Summary ................................................................................................... 40
3.3.2 Current State ............................................................................................. 40
3.3.3 Future State ............................................................................................... 40
3.3.4 Challenges................................................................................................. 40
3.3.5 Potential Solutions ..................................................................................... 41
3.4 Power System Hardening ................................................................................. 41
3.4.1 Summary ................................................................................................... 41
3.4.2 Current State ............................................................................................. 42
3.4.3 Future State ............................................................................................... 42
3.4.4 Challenges................................................................................................. 42
3.4.5 Potential Solutions ..................................................................................... 42
3.5 Restoration ....................................................................................................... 43
3.5.1 Summary ................................................................................................... 43
3.5.2 Current State ............................................................................................. 44
3.5.3 Future State ............................................................................................... 44
3.5.4 Challenges................................................................................................. 44

vi
TR-123 — IEEE PES Technical Roadmap

3.5.5 Potential Solutions ..................................................................................... 45

3.6 Physical Security and Human-caused Attacks ................................................. 46
3.6.1 Summary ................................................................................................... 46
3.6.2 Current State ............................................................................................. 46
3.6.3 Future State ............................................................................................... 46
3.6.4 Challenges................................................................................................. 46
3.6.5 Potential Solutions ..................................................................................... 47
3.7 Condition Assessment ...................................................................................... 48
3.7.1 Summary ................................................................................................... 48
3.7.2 Current State ............................................................................................. 48
3.7.3 Future State ............................................................................................... 48
3.7.4 Challenges................................................................................................. 48
3.7.5 Potential Solutions ..................................................................................... 49
3.8 Reliable Sourcing of Products and Engineering Services to Promote Nation
Security ..................................................................................................................... 49
3.8.1 Summary ................................................................................................... 49
3.8.2 Current State ............................................................................................. 50
3.8.3 Future State ............................................................................................... 50
3.8.4 Challenges................................................................................................. 50
3.8.5 Potential Solutions ..................................................................................... 51
3.9 Prognostics and Health Management (PHM) ................................................... 52
3.9.1 Summary ................................................................................................... 52
3.9.2 Current State ............................................................................................. 52
3.9.3 Future State ............................................................................................... 52
3.9.4 Challenges................................................................................................. 52
3.9.5 Potential Solutions ..................................................................................... 53
3.10 Fault Tolerance ................................................................................................ 53
3.10.1 Summary ................................................................................................... 53
3.10.2 Current State ............................................................................................. 54
3.10.3 Future State ............................................................................................... 54
3.10.4 Challenges................................................................................................. 54
3.10.5 Potential Solutions ..................................................................................... 54
3.11 Resource Adequacy and Fuel/Energy Assurance ............................................ 55
3.11.1 Summary ................................................................................................... 55
3.11.2 Current State ............................................................................................. 55

vii
TR-123 — IEEE PES Technical Roadmap

3.11.3 Future State ............................................................................................... 56

3.11.4 Challenges................................................................................................. 56
3.11.5 Potential Solutions ..................................................................................... 56
3.12 Resilience Assessment Under Climate-induced Extreme Weather Conditions 58
3.12.1 Summary ................................................................................................... 58
3.12.2 Current State ............................................................................................. 58
3.12.3 Future State ............................................................................................... 58
3.12.4 Challenges................................................................................................. 58
3.12.5 Potential Solutions ..................................................................................... 58
3.13 References ....................................................................................................... 59
4. Grid Edge Technologies ............................................................................................ 60
4.1 Executive Summary ......................................................................................... 60
4.2 Flexibility at the Grid Edge (device/plant level) ................................................ 61
4.2.1 Grid-Integrated Smart Buildings, Homes, and Communities ..................... 61
4.2.2 Electric Vehicles (EVs) and Battery ........................................................... 63
4.2.3 Forecast: Load, PV, EV pattern ................................................................. 64
4.2.4 Virtual Power Plants .................................................................................. 66
4.3 Coordination and Control of Grid Edge Assets (architecture/system level)...... 68
4.3.1 Optimal Power Flow .................................................................................. 68
4.3.2 State Estimation ........................................................................................ 70
4.3.3 Distributed Control ..................................................................................... 71
4.3.4 Advanced Distribution Management System (ADMS) and Distributed Energy
Resource Management System (DERMS) ............................................................ 73
4.3.5 Transactive Energy Systems and Energy Blockchain ............................... 76
4.3.6 DER market participation: regulatory/business model ............................... 78
4.4 Microgrids ......................................................................................................... 80
4.4.1 Current State ............................................................................................. 80
4.4.2 Future State ............................................................................................... 81
4.4.3 Challenges................................................................................................. 81
4.4.4 Potential Solutions ..................................................................................... 82
4.5 References ....................................................................................................... 82
5. Computation .............................................................................................................. 84
5.1 Executive Summary ......................................................................................... 84
5.2 Large-Scale Models and Uncertainty ............................................................... 84
5.2.1 Large-scale Modeling and Optimization .................................................... 84

viii
TR-123 — IEEE PES Technical Roadmap

5.2.2 Computation Techniques Under the Paradigm of Uncertainty and Variability

89
5.3 Parallel and Distributed Computing .................................................................. 92
5.3.1 The current state of PDC for power system analysis ................................ 93
5.3.2 Future State ............................................................................................... 96
5.3.3 Challenges................................................................................................. 97
5.3.4 Potential Solutions ..................................................................................... 97
5.4 Artificial Intelligence (AI) for Power System Computation ................................ 97
5.4.1 AI-based data initialization and scenario generation ................................. 98
5.4.2 Accelerating power system computation ................................................... 98
5.4.3 AI for problems without analytical models ............................................... 101
5.5 Emerging Computational Technologies ......................................................... 101
5.5.1 Quantum computing ................................................................................ 101
5.5.2 GPU computing ....................................................................................... 103
5.5.3 Edge intelligence ..................................................................................... 104
5.5.4 Cloud computing...................................................................................... 106
5.6 Computational Tools for the Power Systems ................................................. 109
5.6.1 Open-source Tools .................................................................................. 109
5.6.2 Synthetic Test Systems ........................................................................... 110
5.6.3 Testbeds and Digital Twins ..................................................................... 112
5.7 References ..................................................................................................... 114
6. Cybersecurity .......................................................................................................... 122
6.1 Executive Summary ....................................................................................... 122
6.2 Introduction..................................................................................................... 123
6.3 Current State .................................................................................................. 126
6.3.1 Prevention (protect) ................................................................................. 126
6.3.2 Risk minimization (identify) ...................................................................... 127
6.3.3 Detection (detect) .................................................................................... 129
6.3.4 Response (respond) ................................................................................ 129
6.3.5 Remediation/mitigation (recover) ............................................................. 130
6.3.6 People ..................................................................................................... 131
6.3.7 Processes ................................................................................................ 131
6.3.8 Technologies ........................................................................................... 132
6.4 Future State (trends) ...................................................................................... 133
6.4.1 Prevention (protect) ................................................................................. 134

ix
TR-123 — IEEE PES Technical Roadmap

6.4.2 Risk minimization (identify) ...................................................................... 134

6.4.3 Detection (detect) .................................................................................... 135
6.4.4 Response (respond) ................................................................................ 136
6.4.5 Remediation/mitigation (recover) ............................................................. 137
6.4.6 People ..................................................................................................... 137
6.4.7 Processes ................................................................................................ 137
6.4.8 Technologies ........................................................................................... 138
6.5 Challenges ..................................................................................................... 139
6.5.1 Prevention (protect) ................................................................................. 139
6.5.2 Risk minimization (identify) ...................................................................... 141
6.5.3 Detection (detect) .................................................................................... 142
6.5.4 Response (respond) ................................................................................ 142
6.5.5 Remediation/mitigation (recover) ............................................................. 143
6.5.6 People ..................................................................................................... 143
6.5.7 Processes ................................................................................................ 144
6.5.8 Technologies ........................................................................................... 145
6.6 Potential Solutions .......................................................................................... 147
6.6.1 Prevention (protect) ................................................................................. 147
6.6.2 Risk minimization (identify) ...................................................................... 148
6.6.3 Detection (detect) .................................................................................... 149
6.6.4 Response (respond) ................................................................................ 149
6.6.5 Remediation/mitigation (recover) ............................................................. 151
6.6.6 People ..................................................................................................... 151
6.6.7 Processes ................................................................................................ 152
6.6.8 Technologies ........................................................................................... 153
6.7 References ..................................................................................................... 154

x
TR-123 — IEEE PES Technical Roadmap

ACRONYM
AC OPF Alternating Current OPF
AD Anomaly Detection
ADMM Alternating Direction Method of Multipliers
ADMS Advanced Distribution Management System
AE Algebraic Equation
AGC Automatic Generation Control
AI Artificial intelligence
AIS Adaptive Instructional Systems
AMI Advanced Metering Infrastructure
AMPS Analytical Methods for Power System Committee
API Application Programming Interfaces
APTs Advanced Persistent Threats
BAS Building Automation Systems
BD Benders’ Decomposition
BES Bulk Electric System
BESS Battery Energy Storage System
BTM Behind-The-Meter
BYOL Bring Your Own License
C2M2 Cybersecurity Capability Maturity Model
CA Contingency Analysis
CCE Consequence-driven Cyber-informed Engineering
CCG Column-and-Constraint Generation
CCS Carbon Capture and Storage
CEII Critical Energy Infrastructure Information
CESER Cybersecurity, Energy Security, and Emergency Response
CIGRE International Council on Large Electric Systems
CIE Cyber-Informed Engineering
CIP Critical Infrastructure Protection
CO2 Carbon dioxide
CO2e Carbon Dioxide Equivalent
CIE Cyber-Informed Engineering
CCE Consequence-driven Cyber-informed Engineering
CFAA Computer Fraud and Abuse Act
CPPS Cyber-Physical Power System
CPS Cyber-Physical System
CRISP Cybersecurity Risk Information Sharing Program
CSF Cyber Security Framework
CSP Cloud Service Provider
CSP Concentrating Solar Power
CVaR Conditional Value at Risk
CVSS Common Vulnerability Scoring System
DAE Differential Algebraic Equation
DC OPF Direct Current OPF
DER Distributed Energy Resource

xi
TR-123 — IEEE PES Technical Roadmap

DERMS Distributed Energy Resource Management System

DMS Distribution Management System
DOE Department of Energy
DoS Denial of Service
DPI Digital Privacy Initiative
D-STATCOM Distribution Static Compensator
DT Digital Twin
ED Economic dispatch
EGCC Energy Government Coordinating Council
E-ISAC Electricity Information Sharing and Analysis Center
EM Electro-Magnetic
EMP Electromagnetic Pulses
EMS Energy Management System
EMT Electromagnetic Transient
EPA Environmental Protection Agency
ESA European Space Agency
ESCC Electricity Subsector Coordinating Council
EV Electric Vehicle
FAIR Factor Analysis of Information Risk
FERC Federal Energy Regulatory Commission
FLISR Fault Location, Isolation, and Service Restoration
GEB Grid-interactive Efficient Building
GERs Grid-Edge Resources
GHG Green House Gas
GIS Geographic Information System
GIS Gas-Insulated Substation
GIWH Grid Interactive Electric Water Heaters
GMD Geomagnetic Disturbances
GFL Grid-Following
GFM Grid-Forming
GPU Graphics Processing Unit
GUI Graphical User Interfaces
GWAC GridWise® Architecture Council
H2 Hydrogen Gas
HAWT Horizontal-Axis Wind Turbine
HBOM Hardware Bill-of-Materials
HIL Hardware-In-The-Loop
HILP High Impact Low Probability
HPC High-Performance Computing
HTB Hardware Testbed
HVDC High Voltage Direct Current
IaaS Infrastructure as a Service
IAEA International Atomic Energy Agency
IBR Inverter-Based Resource
ICT Information and Communication Technology
IDS Intrusion Detection System

xii
TR-123 — IEEE PES Technical Roadmap

IEA International Energy Agency

IOA Indicators-of-Attack
IOC Indicators-of-Compromise
IoT Internet of Things
IIoT Industrial Internet of Things
IPCC Intergovernmental Panel on Climate Change
IRA Inflation Reduction Act
ISA International Society of Automation
(ISC)² International Information System Security Certification
Consortium
ISO Independent System Operator
Li-ion Lithium-ion
LP Linear Programming
LTB Large-Scale Testbed
LTCs Load Tap Changers
MCS Monte Carlo Sampling
MMRs Micro Modular Reactors
ML Machine Learning
MOO Multi-Objective Optimization
NERC North American Electric Reliability Corporation
NICE National Initiative for Cybersecurity Education
NIST National Institute of Standards and Technology
NOx Nitrogen Oxides
ODE Ordinary Differential Equation
OMS Outage Management System
OpenFMB Open Field Message Bus
OPF Optimal Power Flow
OT Operational Technology
OTEC Ocean Thermal Energy Conversion
PaaS Platform as a Service
PDC Parallel and distributed computing
PDFs Probability Density Functions
PES Power and Energy Society
PHM Prognostics and Health Management
PLI Production Linked Incentive
PLL Phase Locked Loop
PM Particular Matter
PMU Phasor Measurement Units
PPAs Power Purchase Agreements
PSCCC Power System Communication and Cybersecurity Committee
PSRC Power System Relaying and Control Committee
QoS Quality of Service
R&D Research and Development
RC Reliability Coordinator
RES Renewable Energy with Storage
RL Reinforcement Learning

xiii
TR-123 — IEEE PES Technical Roadmap

RoCof Rate of Change of Frequency

RTO Regional Transmission Organizations
RTU Remote Terminal Unit
RUL Remaining Useful Life
SaaS Software as a Service
SAIDI System Average Interruption Duration Index
SAIFI System Average Interruption Frequency Index
SBOM Software Bill-Of-Materials
SCADA Supervisory Control And Data Acquisition
SCOPF Security-Constrained Optimal Power Flow
SCs Switched Capacitors
SE State Estimation
SEI Space Energy Initiative
SF6 Sulfur Hexafluoride
SGs Synchronous Generators
SMRs Small Modular Reactors
SRM Shared Responsibility Model
SVC Static Var Compensator
T&D Transmission and Distribution
TRMM Transmission Resiliency Maturity Model
TTPs Tactics, Techniques, and Procedures
UC Unit Commitment
UPFC Unified Power Flow Controller
V2G Vehicle-to-Grid
VAWT Vertical-Axis Wind Turbines
VOCs Volatile Organic Compounds
VPP Virtual Power Plant
VVO Volt/VAR optimization
WLS Weighted Least Squares

xiv
TR-123 — IEEE PES Technical Roadmap

THIS PAGE LEFT BLANK INTENTIONALLY

xv
TR-123 — IEEE PES Technical Roadmap

1. Executive Summary
1.1 Setting the Stage
Electricity is the lifeblood of the modern society. Most of the world’s population is
heavily dependent upon electricity as an energy source which is needed for daily support
of transportation, national security, air conditioning, data processing, communication, food
processing, health care, entertainment and a variety of other purposes. These vital services
were provided by global electric generation with a capacity of over 8,500 Gigawatts in
2022 [1]. With an increasing level of electrification, especially in the context of Net Zero
2050 with electrification of everything, electricity demand is continuing to increase
through 2050. “Clean” grid will play even bigger and more important role in the society.

Over the past two decades, we have witnessed an increasing awareness of the
potential impacts of climate change. Adverse human contributions to the climate should
be minimized by the global community.

A global topic has been how to reduce greenhouse gas emissions into the
atmosphere. Carbon dioxide (CO2) is the largest contributor to greenhouse gas effects that
contribute to climate change and the resulting weather extremes. Significant efforts are
underway to reduce or remove its emissions, i.e. “deep de-carbonization”. Many countries
and regions have committed to and already implemented policies to achieve net-zero CO2
emissions targets by 2050 or earlier.

The major challenge addressed in this document is how to achieve greenhouse gas
reduction and deep de-carbonization while “keeping the lights on.” There is the need to
phase out fossil fuels during the transition to a low-carbon energy supply system. Energy
systems impact so many aspects of society and a transition to Net Zero would have
profound implications well beyond climate and energy. It is therefore imperative that the
transformation to Net Zero is rapid, but orderly, without significant disruptions to vital
services.

The global power and energy industry and the IEEE Power and Energy Society
(PES) play a critical role in balancing the mitigation of adverse human contributions to the
climate with providing a reliable, secure, resilient and affordable power system.

Awareness is also growing that on one hand Earth’s resources are limited,
especially some critical metals and minerals, and on the other hand waste, e.g. electronic
waste is growing very significantly. The implementation of ‘Circular Economy’ principles
is expected to address these challenges.

The IEEE PES is committed to bringing technology solutions to meet the

challenges facing the industry and to educate the energy workforce. This PES Technical
Roadmap identifies current strategic directions and technology trends to achieve Net Zero.
The roadmap spans the next 10+ years to 2035 and provides global strategic goals for the
1
TR-123 — IEEE PES Technical Roadmap

power and energy industry. It serves as a professional and independent resource for energy
producers, grid operators, electricity distributors, research labs, and academia, as well as
policy makers.

1.1.1 State of the Industry

Power industry has been undergoing significant transformation for the past
decades. In many countries, the industry has been transformed from a vertically integrated
electric utility service industry into partly competitive markets for electric energy and
essential system services (formerly called ancillary services). Concerns over climate
change, and changing in economics, have led to the widespread deployment of renewable
energy, mainly solar, wind, energy storage, electric vehicles, and electrified end-users in
many countries. The implementation of these renewable energy resources also is changing
the nature of the power grid from one with concentrated large scale power plants to a more
distributed generation architecture that requires decentralized operation and control. Low-
cost and high-performance sensing, computing, and communication technologies
implemented with cyber-informed designs can enable the large-scale adoption of
distributed generation resources and diverse distribution loads. The connectivity of
components and systems has grown so much that there is an explosion of data and
information that is available from sensing technology and instrumentation.

Likewise, the implementation of renewable energy resources on the grid has and
will continue to reduce the overall combined capacity factor and baseload capacity of the
generation resources that supply the grid. The application of intermittent renewable energy
resources as replacements for baseload, high-capacity factor energy resources must be re-
examined from the standpoint of system security, reliability, and resilience. More typical
renewable energy resources, i.e. wind and solar resources not only have capacity factors in
the range of 30% to 40% and their output can be highly uncertain during extreme
environmental conditions. They also supplant large generators that provide the necessary
and important stability services to the grid such as inertia and system strength. Some of
the most effective renewable energy resources that can complement the typical variable
renewable resources, e.g. wind, solar, are hydro power/pumped storage, advanced modular
nuclear reactors, geothermal, biomass, green hydrogen storage, and ocean energy. This
roadmap will provide a pathway to the fulfillment of an “all of the above approach” and
the advancement and application of the most effective mix of renewable resources to the
grid.

1.1.2 Challenges
Energy transition is faced with existing and emerging challenges. Natural disasters
such as extreme weather events and major earthquakes are often causes for large scale
power outages. The threat from weather-related or anthropogenic (smog, smoke) events is
becoming increasingly serious. The power grid is operated and controlled by information
technology and communication equipment. The potential for cyberattacks have emerged
as a major concern for the security of the grid. In addition to cyber security, the privacy of
data and information is needed for the connected entities. The variable nature of large scale
solar and wind energy resources is a challenge for the operation, protection, and control of
2
TR-123 — IEEE PES Technical Roadmap

the power grid. The fast down-ramping nature of solar energy at sunset calls for reserves
that can become available quickly with high ramp-rates to compensate for the change. This
will most likely require a paradigm shift that changes the approach and application of wind
and solar energy systems as replacements for baseload generation, to intermediate load or
peak load generation resources.

Power system resilience and physical security challenges have become prevalent
throughout the industry. Existing overhead lines and air insulated substations are highly
susceptible to environmental factors, severe weather events and human-caused threats.
Challenges include also potential shortfalls of resources required to build infrastructure and
the goal of realizing a circular economy. This roadmap will provide the approach to
promote a robust and resilient power infrastructure that can withstand a multitude of
adversities. It will identify the need for significant infrastructure upgrades, the cost of
implementing advanced technologies, and the need for extensive training of staff for
efficient and safe restoration after a disruption. It will also identify the design strategies to
better mitigate the effects of physical attacks and criminal activities.

1.1.3 Vision
A reliable, secure, and resilient infrastructure for electric energy is critical to
maintaining vital services to society, sustaining a low carbon economy and to provide for
the security of power systems. To this end, it is not sufficient to simply plan to reduce the
frequency and outage durations of the electricity service. It is critical to ensure that the
generation resources and transmission/distribution infrastructure of the grid are highly
reliable, and increasingly resilient. New concepts and technologies, procedures, and
standards are needed to support power grids with a high penetration of renewable and
distributed energy resources. To address the cyber security and privacy issues, a full
understanding of the impact of cyber threats on the physical power grid is required, i.e.,
cyber security as well as cyber-physical security, and the associated vulnerability, risk
management and mitigation methods.

With respect to critical infrastructure, resiliency is defined as; “The ability of a

facility or an asset to anticipate, resist, absorb, respond to, adapt to, and recover from a
disturbance.” The deployment of distributed resources such as widespread microgrids
(grid-connected and not grid-connected) with the availability of renewable and energy
storage devices as well as clean baseload, high-capacity factor energy resources will be an
important goal for the future. Design basis for resource adequacy and transmission
availability must be revisited to support an economy that is increasingly dependent on the
production and delivery of electricity at all times, during normal and extreme events. The
enabler of this vision is an infrastructure with high connectivity that promotes system
stability, advanced monitoring, safe and efficient operation, and effective
protection/control in a real time environment. The cost and performance of critical
infrastructure is a key requirement for engineering technologies to be widely adopted. As
resource limitations and circular economy requirements become more important, this
infrastructure also will need to be designed for maximum re-use of materials at the end of
use.

3
TR-123 — IEEE PES Technical Roadmap

1.2 Summary of Chapters

This roadmap addresses current status, future status, gaps and challenges, as well
as potential solutions for the following key areas:

1.2.1 Renewable Energy Integration

There is a global trend of shifting towards cleaner and more sustainable energy
resources. Renewable energy integration is at forefront of de-carbonization. Many
countries have set ambitious targets to increase their share of renewable energy. The
electric power industry is working hard to integrate renewable resources at a fast pace while
maintaining reliability and resilience. The key to a successful approach and a vibrant
society is balancing the mitigation of the impact of adverse human contributions on the
climate cycle with providing an available, reliable, and resilient power system.

The system design basis needs to be improved as current approaches and their
assumptions are no longer sufficient to ensure the systems of the future. Key challenges
including variability and uncertainties introduced by renewable resources, grid stability
and reliability issues, decentralization, market and policy hurdles, must be addressed by
innovative solutions that harness advanced technologies and methods. Probabilistic
assessment approaches, driven by machine learning algorithms, enable the quantification
of uncertainties and risks in power system operations. Transmission and distribution
system upgrades, including energy storage integration, clean firm resources, HVDC
transmission, and expanded interconnections, enhance grid flexibility and stability. Smart
grid technologies, such as advanced sensors and communication systems, empower real-
time monitoring and efficient grid management. Policy enhancements are crucial in
fostering a supportive environment for renewable integration. Policies should align with
strategic energy goals, offering financial incentives and support to accelerate renewable
investments that enhance the security, reliability and resilience of the grid.

1.2.2 Reliability and Resilience

To provide the necessary electrical energy to society, it is critical to ensure that the
generation resources and transmission/distribution infrastructure of the grid are highly
reliable, available and resilient. Development of reliable electric systems ensures
continuous service under normal operating conditions and no disruptions to service under
credible contingencies. A resilient electric system can withstand disruptions, adapt, and
recover quickly under extreme operating conditions due to disruptive external threats.

Maintaining the security, reliability and resilience of the electric system amidst
energy transition balanced with the challenge of keeping the lights on during periods of
substantial electricity market volatility and climate-induced extreme weather conditions is
an arduous challenge. Some of the other challenges include capacity shortages due to
accelerated baseload resource retirements, increasing electric demand, the increased
frequency and intensity of both natural and human-caused physical threats, and the
4
TR-123 — IEEE PES Technical Roadmap

indiscriminate application of variable renewable energy resources. Reliable sourcing is

considered as a key factor in promoting energy security, along with prognostics and health
management (PHM) in improving system reliability. Developing a resilience performance
scoring matrix could be one approach to harden power systems, minimize potential damage
and as strategies for mitigation. Fault tolerance is indispensable for system reliability and
resilience. Normative requirements, clear framework, resource investment, continuous
adaptation, and knowledge sharing could facilitate implementation of dependable practices.

1.2.3 Grid Edge Technologies

With increasing distributed energy resources, EVs, batteries and controllability at

the customer side, such as Grid-integrated smart buildings, homes and communities, Grid
Edge Technologies are becoming important to reliable and efficient power grid operation,
e.g. participating as virtual power plants to respond to dispatch signals. There are various
technical, economic, and regulatory challenges and opportunities associated with these
technologies. These include system protection, power system operation, power quality,
controllable devices coordination, need for new market structure and business model as
well as data management.

These challenges involve a variety of stakeholders, including customers, system

operators, utilities, regulators, researchers, and manufacturers. Solving these challenges
needs to bring all parties to the table to work together. Smart sensors, advanced
optimization, data analytics, machine learning algorithms and distributed, hierarchical
control are among the potential solutions for these challenges.

1.2.4 Computation
With the integration of renewable resources and smart grid technologies, modern
power grids need advanced computation solutions to address emerging challenges. Among
them are increased uncertainty and variability, control complexities, time intensive
computation related to various complicated simulation and analysis, real-time optimization
and decision support as well as the need for automated smart real-time application.
Parallel and distributed computing (PDC) can serve to expedite computational
workloads and enhance efficiency. Artificial intelligence (AI) and machine learning (ML)
offer promising applications in terms of increased accuracy and acceleration of
computation intensive tasks, including prospective real-time applications. Emerging
technologies such as quantum computing, graphics processing unit (GPU) computing, edge
computing, and cloud computing, are revolutionizing and transforming power system
computation. Various computational tools such as open-source packages, synthetic test
systems, testbed and digital twins are enabling technologies for a panoramic understanding
of complicated power systems.

1.2.5 Cyber Security

5
TR-123 — IEEE PES Technical Roadmap

The cybersecurity discussion begins by recognizing that the electric power sector
includes diverse perspectives of the increased threats and attacker sophistication. A
definition of cybersecurity and the power grid itself guides the discussion so that the
perspective presented is understood within the applied constraints in these definitions. The
power grid is recognized as a cyber-physical system. Focus is placed on the operational
side of the electric utilities who operate the power grid. This also results in defining
operational technology.

With these definitions in place, the application of the emerging Cyber-Informed

Engineering (CIE) can be used to integrate cybersecurity considerations into the
conception, design, development, and operation of future power grids. Even though CIE is
not formally mapped to the National Institute of Standards and Technology (NIST) Cyber
Security Framework, the concepts of prevention, risk mitigation, detection, response, and
remediation/mitigation are related to people, process and technology. Standardization work
is already underway in the ISA99 WG14 but the challenge remains to have a regulatory
agnostic, technically sound, risk-informed and performance-based framework for
implementing a cybersecurity program for the electric grid. Adoption of CIE in the electric
sector is still emerging and quick adoption will be a challenge to gain widespread
application. These are two of the numerous challenges presented, where potential solutions
are accelerating the work completion of ISA99 WG14 and building into it CIE concepts,
amongst other solutions.

1.3 Next Step

This Technical Roadmap is a living document. It will grow, evolve, and be

updated as needed or as significant development arises that should be incorporated in the
document.

1.4 References
[1] Electricity generation capacity worldwide in 2022 with a forecast to 2050.
Available at https://www.statista.com/statistics/859178/projected-world-
electricity-generation-capacity-by-energy-source/.

2. Renewable Energy Integration

2.1 Executive Summary

This chapter presents a comprehensive overview of the opportunities and

challenges associated with the seamless integration of renewable energy into the global
power grid. The surge in renewable capacity, primarily in solar and wind power, has been
fueled by falling technology costs, government subsidies, and growing environmental
awareness. As of 2023, the global annual renewable capacity additions surpassed 510 GW,
the fastest growth rate in the past two decades. In 2023, China commissioned as much solar

6
TR-123 — IEEE PES Technical Roadmap

PV as the entire world did in 2022, while its wind additions also grew by 66% year-on-
year. Looking ahead, global renewable capacity is forecast to reach 7,300 GW by 2028.
This growth trend would see global capacity increase to 2.5 times its current level by 2030.
Solar PV and wind technologies will continue to dominate capacity additions, with
competitive auctions, policy setting, market conditions, and corporate agreements driving
the growth. Challenges related to higher-than-expected installation costs, potential
reductions in government subsidies, permitting and overextended grid infrastructure may
slow expansion, particularly in emerging economies. However, addressing these obstacles
could lead to a 25% higher renewable capacity expansion, reaching nearly 2,950 GW
globally. To achieve CO2 net-zero emissions by 2050, the pace of zero carbon capacity
expansion needs to increase by 60% globally. With supportive policies and international
collaboration, the world is poised to accelerate renewable integration, creating a cleaner
and more resilient energy future.

While this presents an exciting prospect for a cleaner energy landscape, it also
introduces complexities due to the variable nature of renewable generation. Key challenges
that must be addressed to fully leverage the potential of renewable energy integration have
been identified, including variability and uncertainty introduced by renewable resources,
grid stability and reliability issues, decentralization, and aligning market mechanisms and
policy frameworks to incentivize renewable investments. To overcome these challenges,
innovative solutions have been proposed that harness advanced technologies and
methodologies. Probabilistic assessment methods, driven by machine learning algorithms,
enable the quantification of uncertainties and risks in power system operations.
Transmission and distribution system upgrades, including energy storage integration,
HVDC transmission, advanced overhead transmission technologies, and expanded
interconnections, enhance grid flexibility and stability. Smart grid technologies, such as
advanced sensors and communication systems, empower real-time monitoring and
efficient grid management. Market and policy enhancements are crucial in fostering a
supportive environment for renewable integration. Market mechanisms must be refined to
encourage competition among suppliers and fairly value the contributions of flexibility and
grid services. Policies should align with strategic energy goals, offering financial incentives
and support to accelerate renewable investments.

By embracing innovative solutions, the integration of zero and low-carbon energy

resources will empower nations to reduce carbon emissions, combat climate change, and
achieve energy security, through coordinated efforts among policymakers, industry
stakeholders, and researchers.

2.2 Introduction

Over the past two decades, we have witnessed changes in the climate cycle of the
earth, which has caused more frequent and intense extreme weather events, such as
heatwaves, floods, droughts, wildfires, and storms, and bring devastating impacts on
human societies and natural ecosystems [8]. From 1980 to 2020, the United States
experienced an average of 7.1 events per year, which increased to 16.2 events per year in
2016-2020 [9]. According to the Intergovernmental Panel on Climate Change (IPCC),
global warming is likely to exceed 1.5°C above pre-industrial levels between 2030 and
7
TR-123 — IEEE PES Technical Roadmap

2050 if it continues to increase at the current rate [10]. Even a 1 to 2 degrees Celsius
warming would result in more intense and frequent natural disaster events, with significant
losses of life and property, and greater spending by the governments on responding to such
disasters. Every year damages from climate change become more widespread and severe
and public support for actions becomes stronger. The global community is working to
address the mounting threats climate change poses to the planet and society.

The focus has been on emission reduction and removal of CO2 equivalents,
contributing to climate change. For example, the United States emits about 6.7 billion
metric tons of carbon dioxide equivalent (Gt CO2e) each year, of which about 80 percent
is carbon dioxide (CO2). Ninety percent of global CO2 emissions are caused by fossil fuel
combustion. The global community is calling for accelerating “deep de-carbonization”, and
many countries and regions have committed to net-zero emissions targets by 2050 or
earlier, and many have already implemented policies and strategies to achieve these goals.
If this goal is reached globally, it is anticipated that future warming would be limited to a
target of 1.5 degrees Celsius. Despite that, Green House Gas (GHG) emissions continue to
rise globally and are not on track to meet the Paris Agreement's goal of limiting global
warming to well below 2°C above pre-industrial levels [11]. The major gap and challenge
in achieving deep de-carbonization is the need to phase out high-carbon energy sources
and transition to a low-carbon energy system.

The electric power sector is the second largest source of direct emission (27%),
right next to transportation (28%). The energy system impacts so many aspects of society,
a transition to net zero would have profound implication well beyond climate and energy.
Power and energy industry plays a critical role in mitigating the impact of climate change,
in terms of reducing GHG emission from electricity generation and supporting the de-
carbonization of other sectors. The electric sector is a linchpin in successful transition to
net-zero emissions. Most near-term emission reductions during a transition to net zero
would come from the electric sector and electrification.

From 2005 to 2019, carbon dioxide emissions from electricity generation declined
by a third, resulting from the replacement of the oldest, least efficient coal plants with
plants that burn natural gas. Additionally, SF6 emissions trends have been significantly
declining for the members of the SF6 Emission Reduction Partnership, in a collaborative,
voluntary effort between the United States’ EPA and the electric power industry. With
ongoing policy support and steady innovation by the private sector and growing
preferences for renewable power, the electric sector in US could deliver as much as 90
percent clean electricity by 2035 at rates comparable to today’s levels. Every year, non-
emitting energy technologies become cheaper and more available. There is a global trend
of shifting towards cleaner and more sustainable energy sources, with renewable energy
becoming increasingly competitive and cost-effective. Many countries have set ambitious
targets to increase their share of renewable energy. Renewable integration is at forefront of
de-carbonization. In the United States, the Biden administration has set a goal to produce
carbon pollution-free electricity in the U.S. by 2035 [12]. Renewables are expected to
become the largest source of global electricity generation by early 2025, surpassing coal.
The levelized costs of wind has declined by 70 percent and solar photovoltaics by almost

8
TR-123 — IEEE PES Technical Roadmap

90 percent since 2009, while the cost of lithium-ion batteries for electric vehicles dropped
by 85 percent [1]. All these can supply electricity with no direct CO2 emissions.

The electric power industry is working hard on the following challenges, to

integrate carbon-free energy sources while maintaining reliability and resilience.

● Renewable resources, such as solar and wind, are highly dependent on weather
conditions, variable in nature, posing a challenge to maintaining grid reliability. To
integrate more renewable energy, there is a need for greater flexibility of grid
services. Energy storage technologies will play an important role in mitigating these
issues. Technological advances are also being made in “clean firm” resources, such
as advanced modular nuclear reactors, geothermal generation, natural gas with
Carbon Capture and Sequestration (CCS), clean coal generation, and carbon-free
fuels, which can provide the necessary baseload resources for dependable
electricity in concert with renewables and energy storage. The question that needs
to be addressed is how quickly this research can be completed and scaled to meet
the goals set for decarbonization.

● Sufficient electric transmission capacity and substantial expansion of high-voltage

climate hardened transmission lines and substations is needed to connect renewable
generation to the grid and load centers, and to move renewable power between
regions. Distribution networks also need to be reinforced to prepare for rapid
electrification, better use of flexible load and distributed energy resources.

● Government policies will play a key role in the reliable integration of new zero-
carbon resources into power systems. Some low carbon resources have high upfront
capital costs and low operating costs while other low carbon resources have high
upfront capital costs and much longer lifetime, higher availability, and low life
cycle costs.

2.3 Types of Renewables

This section explores the major types of renewable generation, delving into their
working principles, advantages, and challenges. From solar and wind power to
hydroelectric, nuclear and geothermal energy, each form of renewable generation holds
unique potential to shape our energy landscape and propel us towards a more sustainable
world.

2.3.1 Wind Energy

Wind technology involves the conversion of kinetic wind energy into usable
electrical or mechanical energy. Wind turbines are the primary technology employed to
capture wind energy. These turbines consist of tall towers equipped with large rotor blades
affixed to a central hub. The most used wind turbine is the horizontal-axis wind turbine
(HAWT), where the rotor blades revolve around a horizontal axis. Although less frequently
employed, vertical-axis wind turbines (VAWT) represent another type. As the wind blows,
9
TR-123 — IEEE PES Technical Roadmap

it passes over the rotor blades, instigating their rotation. The blades' shape and
configuration are optimized to effectively capture the maximum available wind energy. A
shaft connects the rotor blades to a generator. As the blades rotate, they drive the generator,
which converts the mechanical energy into electrical energy.

Wind energy is a sustainable and environmentally friendly source of power,

reducing the reliance on fossil fuels and combat climate change. By reducing dependence
on fossil fuels and addressing climate change, wind energy plays a crucial role in fostering
energy adequacy and enhancing national energy security. Nonetheless, wind's variable
nature, with variations in strength and availability, necessitates supplementary scheduled
power sources or energy storage systems to ensure a continuous electricity supply,
especially during long-term, widespread environmental conditions. In addition, the
availability of wind (too much or too little) can be can affect output and plant performance.
Furthermore, the installation of wind turbines involves substantial upfront expenses and
requires significant land area or large offshore areas, while noise generation and potential
risks to wildlife are additional considerations. Despite these challenges, wind energy
continues to grow as a viable and sustainable energy source, contributing to the global
transition towards a cleaner and more sustainable future.

2.3.2 Solar Energy

Solar energy generation is typically provided using photovoltaic (PV) panels that
generate direct current (dc). Solar energy is sent to the power grid using inverters that help
convert the dc to the alternating current (ac) present in the power grid. Solar energy
generation has seen tremendous growth worldwide in the past decade with decreasing cost
of photovoltaic (PV) panels and inverters. Further exponential growth is expected in the
coming decade with the cost of utility-scale PV reducing to less than $30/MWh and the
cost of residential PV reducing to less than $130/MWh. The reduction in cost helps to
makes solar energy more cost-effective from equipment and installation perspectives. The
large footprint of solar energy generation coupled with the energy storage necessary to
achieve the required MW capacity, significantly increases the overall cost of solar
installations. Moreover, with the right setting, such as in case of microgrids, solar energy
generation on rooftops can provide resilience during restoration post extreme weather
events, reducing the duration of black-out observed by individual homes. It is also expected
to bring in new economic opportunities as significant upgrades happen in the generation
and transmission infrastructure in the power grid to introduce solar energy in the power
grid.

In the conventional power system, coal-fired or gas-fired power plants were more
predictable in terms of the energy generation and only the loads were uncertain and
variable. However, now the solar energy becomes both variable and uncertain leading to a
higher level of variability and uncertainty of the system net loads. To mitigate these
challenges, increasingly the installations in the power grid are expected to be a combination
of solar and energy storage. When there is excess generation from solar, energy storage
can sink the excess power. When there is reduced generation from solar, energy storage
can smooth the output power. In this way, the variability and uncertainty are reduced to
some extent by making the resource hybrid through use of both solar and energy storage.
10
TR-123 — IEEE PES Technical Roadmap

2.3.3 Hydropower and Pumped Storage

Hydropower is an energy source that harnesses the power of moving water to

generate electricity. It relies on the natural water cycle which is constantly replenished by
rainfall and snowmelt and doesn’t consume or deplete water resources, making it a
sustainable energy option. Unlike solar and wind, which are variable, hydropower can
generate power continuously, making it a reliable source of electricity. And it can be
quickly adjusted to meet fluctuating demand, providing flexibility to the grid. Hydropower
systems can also be designed to store water with two water reservoirs at different elevations
that can generate power as water moves down from one to the other (discharge), passing
through a turbine, acting as a form of pumped storage.

However, building new dams can be costly because of government regulations and
can have significant environmental and social impacts, including habitat disruption,
alteration of river ecosystems, and displacement of local communities. In addition, the
availability of water can be affected by droughts and other weather conditions. Therefore,
careful planning, environmental assessments, and stakeholder engagement are crucial to
mitigate those potential drawbacks and ensure sustainable hydropower deployment.

2.3.4 Geothermal Energy

Geothermal energy uses the natural heat stored within the earth’s crust to generate
electricity. They have high energy conversion efficiencies and high-capacity factors,
making it a great resource for baseload power generation. Additionally, Geothermal power
plants have a long operational life, typically exceeding 30 years, and require relatively low
operational and maintenance costs once the facility is established. This can contribute to
long-term cost stability and energy affordability.

However, geothermal energy is limited to specific geographic regions. They are

found in regions where the Earth's heat is closer to the surface, such as areas with volcanic
activity or geothermal reservoirs. Additionally, they can be expensive to develop and still
associate with environmental concerns, as deep wells and extracting hot fluids can release
gases and minerals into the atmosphere. Furthermore, geothermal resources are often
located in remote areas necessitating the construction of transmission lines to connect the
power plants to the electricity grid, which is often costly.

2.3.5 Biomass Energy

Biomass energy is derived from organic matter, such as plants, crop residues, wood,
and agricultural waste. It involves converting biomass into heat, electricity, or biofuels
through various processes, such as combustion, gasification, or biochemical conversion.
Biomass energy is considered renewable because the organic matter used for energy
production can be replenished through sustainable practices like responsible forestry and
agricultural waste management. Moreover, it can be produced locally allowing for
decentralized energy production and can operate continuously to provide the baseload
power generation.

11
TR-123 — IEEE PES Technical Roadmap

Despite being considered as renewable, Biomass energy still faces environmental

issues, as the combustion or conversion of biomass can release pollutants such as nitrogen
oxides (NOx), particular matter (PM), and volatile organic compounds (VOCs) to the
atmosphere. In addition to that, biomass energy production requires significant amounts of
biomass feedstock, which may compete with other land uses, such as food production or
natural ecosystems. Transportation and storage challenges also add overall cost and
complexity to the biomass energy production. Careful planning, sustainable practices, and
appropriate emission control technologies are crucial to maximize the benefits while
minimizing environmental impacts and potential drawbacks.

2.3.6 Ocean Energy

Ocean energy uses the natural movement or temperature differences of ocean water
to generate electricity. Different technologies, such as wave energy converters, tidal
turbines, ocean current turbines, and ocean thermal energy conversion (OTEC), capture the
kinetic energy of waves, tides, currents, or the temperature gradient between warm surface
water and cold deep water. Tides and waves are predictable and consistent, allowing for
accurate energy generation forecasts. This predictability is advantageous for grid
integration and balancing electricity supply and demand. The world's oceans have high
energy density and vast energy potential, providing a virtually limitless resource for ocean
energy. With over 70% of the Earth's surface covered by oceans, the energy available from
waves, tides, and currents is significant, offering the possibility of generating large amounts
of clean electricity.

However, ocean energy technologies are still in the early stages of development
and face technological challenges, including durability, reliability, and scalability. The
costs associated with research, development, deployment, and maintenance of ocean
energy devices can be relatively high compared to conventional energy sources. The
deployment of ocean energy devices can have environmental impacts on marine
ecosystems and may face conflicts with other ocean uses, such as fishing, shipping,
tourism, and marine conservation areas.

2.3.7 Hydrogen

Hydrogen energy is an emerging technology that utilizes hydrogen gas (H2) as a

clean and versatile energy carrier. It can be produced through various processes, such as
electrolysis, steam methane reforming, or biomass gasification. When hydrogen is used as
an energy source, it can be converted back into usable energy through combustion, fuel
cells, or other hydrogen-specific technologies. Hydrogen is considered a clean energy
carrier because when it is consumed or used in fuel cells, the only byproducts are water
vapor (potentially potable water) and potentially heat. It can serve as a means of energy
storage: excessive electricity generated during low-demand periods can be used to produce
hydrogen through electrolysis. The stored hydrogen can then be used for power generation
or other applications during periods of high electricity demand or when renewable energy
supply is low.

12
TR-123 — IEEE PES Technical Roadmap

However, the production of hydrogen typically requires energy input and water,
and the most common methods, such as steam methane reforming, rely on fossil fuels,
which can result in carbon emissions. Hydrogen has a low energy density per unit volume
compared to fossil fuels, which means larger storage volumes are required to store
equivalent energy. This presents challenges in terms of storage, transportation, and onboard
fuel storage for hydrogen-powered vehicles. Currently, hydrogen production and
infrastructure development can be relatively expensive compared to other energy sources,
which is a barrier to its widespread adoption. In addition to that, hydrogen has certain safety
considerations, primarily related to its flammability and the need for proper handling and
storage to prevent accidents. The low round-trip efficiency of hydrogen (around 40%) [85],
caused by multiple energy conversions, high energy requirements for storage (compression
or liquefaction), and fuel cell and combustion inefficiencies, remains a significant barrier
to its widespread adoption as an energy carrier, especially when compared to the more
efficient battery storage systems, whose round-trip efficiency is around 80-90% [86].

2.3.8 Small Modular Reactors (SMRs)

Small modular reactors (SMR) are potential sources of low-carbon baseload power.
They have a smaller footprint than traditional nuclear power plants and can be located
closer to electricity demand centers. The IAEA (International Atomic Energy Agency)
defines an SMR as a nuclear fission reactor design that is small (less than 300 MWe) and
modular (components and systems can be shop fabricated to expedite field construction
with multiple units that can be placed together at one location) and has separated SMR
designs into six categories, including Land-Based Water-Cooled SMRs, Marine-Based
Water-Cooled SMRs, High Temperature Gas Cooled SMRs, Fast Neutron Spectrum
SMRs, Molten Salt SMRs, Micro-Sized SMRs (MMRs).

SMRs have a greater than 90% capacity factor with a 40-to-50-year life cycle, as
compared to renewable energy sources that have less than a 50% capacity factor and 15-
to-20-year life cycles. SMRs face regulatory challenges and concerns about nuclear waste
disposal. But their inherent safety features and minimized amount of nuclear waste address
these concerns. The initial capital cost of an SMR may be comparable to an offshore or
onshore wind installation of equal capacity at full output. SMRs can be grouped to drive
larger generators that are critical to the inertial stability of the electric system. Traditional
nuclear power plants require refueling every one to two years, whereas SMRs may require
fueling only every three to seven years. The Micro Modular Reactors (MMRs) may not
require refueling at any point during their lifecycle. The fuels being contemplated for many
MMRs and SMRs are also being designed to “burn” more efficiently and more completely.
These characteristics will drive a reduction in nuclear waste.

There is an ongoing debate regarding the need for nuclear energy and power
generation to meet the short and long term decarbonization goals. Once built, the
combination of a zero-CO2 emitting generation source and its high reliability provide a
compelling argument that small modular nuclear reactors could be a major factor, if not the
major contributing factor, in helping nations and energy suppliers meet reliability and
decarbonization goals. [MC3] The challenges associated with the implementation of SMR
and MMR technologies includes:
13
TR-123 — IEEE PES Technical Roadmap

1. Negative public perceptions of nuclear power.

2. Unnecessary and overburdened amounts of government regulatory
requirements that must be removed.
3. Keeping the SMR development and manufacturing environment
competitive.
4. Ensuring that materials and components for SMRs are sourced to promote
electric system reliability, public safety/security and promote national
security.
5. Lobbying of government and marketing campaigns by promoters of other
less efficient renewable generation systems.
6. Negative perceptions about the handling of nuclear waste.

2.4 Current State

Countries around the world have been rapidly increasing their renewable energy
capacity, particularly in solar and wind power. This growth has been driven by falling costs
of renewable technologies, supportive government policies, and increased public
awareness of climate change and environmental concerns. Initial estimates indicate global
renewable capacity has reached 3,371 GW by 2022, surpassing the 2013 figure by more
than double [1]. Renewable electricity production experienced an impressive surge of
nearly 7% in 2021, marking a historic increase of 522 TWh. Wind and solar photovoltaic
(PV) technologies played a significant role in this growth, contributing to almost 90% of
the overall increase. The proportion of renewable energy in worldwide electricity
generation reached 28.7% in 2021, indicating a modest growth of 0.4 percentage points.
2022 sets the record for renewable capacity addition, with an annual capacity of 340 GW,
rose by almost 13% [2]. Among all the renewable technologies, solar PV stood out by
setting a deployment record of nearly 220 GW, representing a remarkable 35% surge
compared to 2021. Following solar PV, hydropower emerged as the second-largest
contributor to the global expansion of renewable energy at record levels. However, the
annual wind capacity additions reduced by 21% in 2022 compared to 2021, marking the
second consecutive year of reduction after achieving record-breaking growth in 2020.
Despite the decline in bioenergy production growth, the global market for geothermal and
concentrated solar power technologies maintained a modest yet stable annual growth rate
[2]. With respect to the SMR nuclear energy resources, there are more than 69 current SMR
designs at different stages of development around the world: one in operation, two in
construction, and a few in the later stages of design and permitting through their respective
regulating agency. Across countries, China has a renewable capacity of 1,207 GW (solar
capacity of 393 GW and wind capacity of 365 GW) in 2022, followed closely by the United
States, with a renewable capacity of 371 GW (solar capacity of 113 GW and wind capacity
of 140 GW) [3].

In 2022, the global manufacturing capacity of solar photovoltaic (PV) saw an

increase of over 70%, reaching nearly 450 GW. Notably, China accounted for more than
95% of the new facilities across the entire supply chain. In the meantime, governments in
the United States, Europe, and India have taken initiatives to diversify the solar PV supply
chain. India's Production Linked Incentive (PLI) scheme and the US Inflation Reduction
Act (IRA) offer direct financial incentives to domestic manufacturers. European Union’s
14
TR-123 — IEEE PES Technical Roadmap

Green Deal Industrial Plan for the Net-Zero Age and the Net-Zero Industry Act set targets
for domestic solar PV manufacturing, yet without specific incentives. In 2022, the
manufacturing capacity for the main wind power components, including nacelles, towers,
and blades, remained relatively consistent, ranging from 110 to 120 GW. It is worth noting
that China continues to spearhead the growth in wind equipment manufacturing. In 2022,
manufacturing capacity for the main wind power components, including nacelles, towers,
and blades, remained relatively stable at 110-120 GW with China again leading the growth.
Unlike solar PV manufacturing, the production of wind equipment is geographically
dispersed. This is because suppliers tend to establish manufacturing plants near demand
centers, to avoid significant expenses and risks of transporting large and delicate
components over long distances [4].

The growth in renewable capacity expansion is closely coupled with the evolution
of advanced inverters. The initial growth in solar PV applications was concurrent to the
increasing adoption of distributed energy resources. Most applications of inverters during
this transition were grid-following inverters, due to their lower initial penetration (as a %
of total resources) and the avoidance of system protection since the grid reference was
driving performance requirements of the inverters. As renewable energy resources
gradually displaced the rotating generators through DER applications, issues such as low
inertia grids and local needs for voltage support emerged resulting in the need for the first
interconnection rules (IEEE 1547-2003). Although the inverter can be made to mimic a
rotating machine, this capability is limited by the lack of stored energy and the lack of
power availability in most inverter applications, except for the Battery Energy Storage
System (BESS). An IBR without built-in energy storage, such as a solar PV plant, among
others, cannot be converted into a virtual synchronous machine, unless the IBR is designed
to inject additional energy into the grid or absorb energy from the grid. Even if an emulation
function is programmed into the inverter controller, the resulting control function would
not be related to any physical characteristic of the inverter, unlike a rotating generator.

2.5 Future State

Global renewable capacity is expected to increase by almost 2,400 GW (almost

75%) between 2022 and 2027 according to the IEA main-case forecast. This growth is
driven by more ambitious expansion policies in key markets and increased economic
attractiveness of renewable power technologies due to high fossil fuel and electricity prices.
The high prices are more pronounced due to the current global energy crisis in light of
Russia's invasion of Ukraine. In these conditions, renewable energy provides a pathway to
the globally desired energy independence and energy security. China is forecasted to install
almost half of new global renewable power capacity between 2022 and 2027, driven by its
ambitious renewable energy targets, market reforms, and strong provincial government
support. In the 14th Five-Year Plan, China sets the goal to achieve a 50 percent increase in
renewable energy generation (from 2.2 trillion kWh in 2020 to 3.3 trillion kWh in 2025),
establishes a 2025 renewable electricity consumption share of 33 percent (up from 28.8
percent in 2020) [5]. The European Union, which is the second-largest growth market,
expects its pace of expansion to more than double during the same period, propelled by
aggressive goals to eliminate Russian fossil fuel imports under the REPowerEU package
[6]. To meet its renewable integration goal of achieving 80% renewable energy generation
15
TR-123 — IEEE PES Technical Roadmap

by 2030, the United States is anticipated to undergo substantial expansion in renewable

energy. This expansion is bolstered by the passage of the IRA (Inflation Reduction Act) in
August 2022, which extends tax credits for renewables until 2032, providing crucial
support for the renewable energy sector [7]. Renewable energy expansion is gaining
momentum in India, Brazil, the Middle East and North Africa (MENA), sub-Saharan
Africa, the Association of Southeast Asian Nations (ASEAN), and other regions, thanks to
policy incentives.

Among all the renewable technologies, solar PV and wind are expected to dominate
global capacity additions. Onshore wind is expected to increase from 74 GW in 2021 to
109 GW in 2027, driven by stable policy frameworks and grid expansion plans. Offshore
wind capacity is projected to reach over 30 GW in 2027, with significant contributions
from the European Union, the United States, and China. Competitive auctions remain the
main driver of growth, with increasing contributions from corporate power purchase
agreements (PPAs), bilateral contracts, and merchant activity. Hydropower, bioenergy,
geothermal, concentrating solar power (CSP), and ocean technologies face challenges such
as higher investment costs, lack of policy support, and limited recognition of flexibility
value, which hampers their faster uptake. Hydropower additions are volatile due to
commissioning deadlines of large reservoir projects [2].

Despite the promising growth trend, challenges related to permitting and grid
infrastructure expansion remain. In emerging economies, policy and regulatory
uncertainties and implementation challenges hinder faster renewable energy expansion.
Developing countries face weak grid infrastructure and a lack of access to affordable
financing, which hampers faster commissioning of multiple projects. Addressing these
challenges could result in a 25% higher renewable capacity expansion than the main case,
reaching nearly 2,950 GW globally. To be in line with the IEA Net Zero by 2050 Scenario,
the pace of renewable capacity expansion needs to increase by 60% globally over the
forecast period. The accelerated case narrows the gap for renewable electricity growth
needed to achieve net-zero emissions by 2050.

Solar PV and wind generators, balanced with storage, such as BESS have resulted
in an increase in renewable energy integration. More generation and storage are also
installed at the distribution level close to the load to avoid the complexities associated with
deploying new transmission. Renewable energy with storage (RES) at the transmission and
distribution levels are increasingly displacing synchronous generators (SGs). The
increasing deployment of inverter-based resources (IBR) results in a gradual reduction of
overall system inertia. Reduced inertia leads to increased rate of change of frequency
(RoCof) in the grid. It may lead to larger frequency and voltage variations during
contingencies, such as the loss of generation or transmission lines and possible loss of
synchronism of both IBR and SGs. For IBR, larger frequency and voltage variations may
make the phase locked loop (PLL) unable to track the grid voltage. As the penetration of
IBR increases, particularly at the transmission level, there may be a need to transition
inverters from the Grid-Following (GFL) to the Grid Forming (GFM) modes to ensure the
stability of the grid during contingencies, particularly under momentary loss of generation
or transmission.

16
TR-123 — IEEE PES Technical Roadmap

Moreover, there is a trend towards integration of renewables using HVDC systems.

This trend can lead to integrated solutions, where PV and ESS are directly integrated with
HVDC links and transmission grids [87] .

2.6 Challenges
The transition to renewable energy resources faces several challenges and gaps,
particularly in the integration of these resources into existing energy systems. As the world
recognizes the urgent need to shift away from fossil fuels and embrace cleaner alternatives,
the integration of renewable energy poses complex and multifaceted challenges that require
careful consideration and strategic planning.

2.6.1 Variability and Uncertainty of Renewable Energy Resources

The inherent characteristics of renewable energy, such as its variability and
uncertainty, pose significant challenges for power system operations and planning [13].
These uncertainties, coupled with variations in flexible loads, create a stochastic dynamic
power system [14]. Accurate forecasting of renewable energy generation is crucial for grid
stability and energy management. However, due to the inherent uncertainty in weather
patterns and other factors influencing renewable energy resources, forecasting can be
challenging. Errors in forecasting can lead to imbalances between supply and demand,
requiring backup power sources or curtailment of renewable generation. It is therefore
essential to quantify the impacts of these uncertainties on the resource adequacy and
dynamic security of power systems. Relying solely on deterministic analysis can
potentially result in solutions that are either overestimated or underestimated [15].
Therefore, comprehensive assessment and control considering the stochastic nature of
uncertain parameters are required to guarantee the dynamic stability of the power system.
Several challenges that need to be addressed, especially for large-scale systems, include
data quality and quantity issues, algorithm effectiveness, algorithm efficiency, and model
generalizability and reliability.

In addition to that, to integrate more renewable, there is growing need in operating

flexibility to hedge the uncertainties introduced from renewables. Flexibility refers to the
ability to quickly adjust electricity generation or consumption in response to changes in
supply and demand. Energy storage technologies, such as batteries, pumped hydro storage,
and other emerging technologies, are poised to play a pivotal role in mitigating the issues
associated with renewable energy variability. Advances in energy storage technologies are
crucial for enhancing grid flexibility and facilitating the integration of renewable energy
resources. Researchers and innovators are actively working on improving the cost-
effectiveness, scalability, and efficiency of energy storage systems. This includes exploring
new materials, developing advanced battery chemistries, and optimizing storage system
designs. Moreover, alongside the rapid growth of renewable energy, there are ongoing
developments in what are known as "clean firm" resources. These resources, such as
advanced modular nuclear reactors, natural gas with carbon capture and storage (CCS),
direct air carbon capture, and carbon-free fuels, can offer a reliable and consistent source
of electricity generation that can complement the variable nature of renewables. In addition
to that, demand response programs involve incentivizing electricity consumers to adjust
17
TR-123 — IEEE PES Technical Roadmap

their power consumption in response to grid conditions. By participating in demand

response initiatives, consumers can help balance the grid by reducing or shifting their
electricity use during periods of high demand or low renewable energy generation. Demand
response providers facilitate the coordination and implementation of these programs.
Virtual Power Plants (VPPs) aggregate and optimize the output of multiple distributed
energy resources, including renewables, energy storage systems, and demand response
participants. VPP operators can actively manage and coordinate these resources to provide
flexibility and support the grid. By intelligently orchestrating the diverse portfolio of assets,
VPPs can balance supply and demand, integrate renewables effectively, and enhance grid
stability. It is important to note that despite the various options of flexible resources, there
is still a challenge of collaborating and coordinating among various stakeholders to achieve
an effective and reliable power system with high shares of renewable energy. This
objective will require effective control strategy, market and policy designs.

2.6.2 Grid Stability and Reliability Issues

The variability and uncertainty of renewable resources make it difficult to always

match supply and demand. This can cause fluctuations in frequency and voltage
destabilizing the grid and resulting in power outages. Additionally, congestion can occur
in areas where renewable energy sources are heavily concentrated, leading to grid
overloads and outages.

Renewable energy generation is typically connected to the power grid through

power electronics. For example, PV power plants connect through inverters and wind
power plants connect through back-to-back inverters to the power grid. The presence of
power electronics as an interface between PV panels or wind turbines and power grids
introduces its benefits. They include fast controllability of active power, firewalling the PV
panels or wind turbines from grid disturbances and vice-versa, and controllable essential
system services support among others. They also introduce challenges as the power grid is
accustomed to planning and operation of traditional electric machine-based generation. The
new characteristics introduced by power electronics need to be investigated at the planning
phase, and new upgrades and/or means to operate the power grid become necessary. For
example, the power electronics in the renewable energy generators can provide only limited
fault currents, are extremely sensitive to voltages and currents in the power grid. They also
introduce high-frequency harmonics and may interact with other fast-acting components
creating control interactions, sub-synchronous interactions, super-synchronous
interactions, to name a few. These challenges have been observed in multiple events
recorded by North American Electric Reliability Corporation (NERC) in the past decade
in regions with locally high penetration of renewables connecting power electronics to the
power grid. The challenges anticipated (and some observed in the past decade) include
reliability issues arising from reduction in power during a fault and post fault generated
from renewables far away from the fault location, oscillations (sub-synchronous or super-
synchronous) and relay mis-operations, to name a few. Moreover, IBR can create power
quality issues such as harmonic distortion and voltage flicker due to the integration of
renewable energy sources through power electronics. These challenges lead to the need for
new planning and interconnection studies tools that can enable study of these issues during

18
TR-123 — IEEE PES Technical Roadmap

planning phase (rather than observing them in the field) [84]. It also leads to the need for
new operational tools and control/integration methods to avoid these challenges.

Certain renewable technologies such as hydropower and pumped storage hydro as

well as advanced modular nuclear reactors (SMRs) which can be grouped to drive larger
generators, can provide the inertial stability and reliability that is so critical to the electric
system.

2.6.3 Protection Challenges

Inverter-based resources, has introduced protection challenges because the fault
current characteristic of an IBR during short circuit conditions is very different from a
conventional generators as traditional rotating synchronous machines. Current from a
synchronous source, immediately after a short circuit and within the timeframe of
protection operation, is of high magnitude, uncontrolled and driven by electrical parameters
of the source and impedance of short circuit path. In contrast, the short circuit current of
an IBR is of low magnitude, highly controlled and managed by fast switching of power
electronics devices, and dependent upon manufacturer specific and often proprietary
control system design.

Traditional protection schemes, which largely rely on high magnitude and high
inductive nature of the short circuit current, may not provide reliable protection during high
IBR penetration. Protection systems are generally designed based on monitoring the output
current, voltages, angle relationship and sequence components of power system faults.
IEEE Standard 2800 and many other industry efforts have standardized fault current
response of renewable generation, so relays can correctly determine the stressed system
conditions and fault conditions. The IEEE Std 1547-2018 remains silent on fault current
characteristics for distributed inverter-based resources. But this standard is going through
revision and may include fault current characteristic requirements. The standardized fault
response from IBRs along with modification of relaying schemes may mitigate protection
challenges in near future. In the meantime, the application of AI in identifying the
fingerprint of faults on the power system can be a promising breakthrough.

The power system relaying and control committee (PSRC) has published several
technical reports regarding protection challenges with IBRs [74-78]. Following PSRC
working groups are developing literature related to this topic:

 C38: IEEE Guide for Design of Microgrid Protection Systems.

 C45: Technical Report on Protection and short-circuit modeling of systems with
high penetration of inverter-based resources.
 C50: Technical Report on Protection of Inverter-Based Resources (Revision of PES
TR-87).
 C51: Technical Report on Under frequency mitigation strategies for low inertia
systems.
 D38: Technical Report on Impact of high SIR in Line Relaying.
 D48: Technical Report on Single Phase Trip and Reclose on Transmission Lines.

19
TR-123 — IEEE PES Technical Roadmap

2.6.4 Decentralization

Decentralization may require additional investments in distribution infrastructure

to accommodate the added complexity of integrating distributed renewable resources into
the grid.

The large-scale deployment of controllable grid-edge resources (GERs), including

DERs, Grid-interactive buildings, demand response, is requiring changes in the existing
grid operation paradigm. Traditionally, centralized decision support of Advanced
Distributed Management System (ADMS) is employed to coordinate the grid’s many
distributed resources to extract various grid support services. However, the centralized
decision-support system is vulnerable to communication and single-point failures. It is also
slow in response and poses scalability challenges when required to coordinate a large
number of controllable agents. Furthermore, the increasing deployment of non-utility
distributed energy resources (DERs) is forcing a transition away from the purely
centralized approach. The future power grid with increasing penetration of non-utility
DERs and microgrids is expected to have multiple control hierarchies and federated
architectures for information sharing and coordination. Thus, the nature of utility
distribution system operations is rapidly evolving from centralized coordination to more
distributed and decentralized coordination architecture. These advanced grid management
architectures (1) reduce the computational requirements on a single decision-making unit
by distributing the problem into several smaller sub-problems that are computationally
simpler; (2) result in a decision-making paradigm with multiple interacting agents that is
robust to single-point failures; and (3) relax the need for communication between the
central controller and all connected controllable/non-controllable assets, thus better
manage data privacy considerations which is a crucial requirement with increasing
penetration of non-utility DERs and microgrids.

However, the real-world deployment of distributed applications is at the very early

stages and the existing advanced distribution management systems (ADMS) are limited in
their ability to host distributed applications. The limited understanding and non-
standardized procedure for data transport among the distributed agents remains one of the
crucial challenges in fully incorporating a distributed coordinated architecture in the utility
operational paradigm. Moreover, several possibilities for communication and control
hierarchies make it extremely challenging to develop a platform that can support different
instantiations of distribution applications. Although data transport reference architectures
such as OpenFMB have been developed for interoperability in distributed coordination
settings, they are limited in their ability to include the specialized needs of grid operations.
This necessitates a more holistic approach to designing the distributed/decentralized grid
management approaches, which is informed and driven by the Quality of Service (QoS)
requirements for the grid operations.

2.6.5 Market and Policy Hurdles

As capital costs for wind and solar power plants have drastically reduced over the
last decade [16], investment in renewable energy technologies is rising [17]. As a result,
20
TR-123 — IEEE PES Technical Roadmap

power systems around the globe with a high share of renewables experience challenges
that extend beyond the market entry of renewable resources to the market entry of
complementary technologies such as storage and the market exit of incumbent resources
such as coal and natural gas power plants. Challenges often manifest in the form of long
wait times [18][19] and rising costs [20] for connection to the bulk power system,
curtailments of renewable electricity generation [21], and difficulty in securing finance.

To connect to the bulk power system, any asset must go through a series of studies
that aim to estimate needs for equipment and grid reinforcements. The associated costs of
such investments are charged to the party requesting the connection. As batteries, wind and
solar power plants tend to have smaller capacity per plant capacity than fossil fuel power
plants, system operators face an unprecedented number of applications for connection. In
the UK, National Grid used to receive 40-50 applications per year [22]; and it now receives
~50 applications per month [23]. The situation is equally worrisome in the USA where wait
times and costs are rising [19][20]. In addition to unprecedented demand for connections,
further complications arise when studies must be performed sequentially for one asset or
cluster of assets at a time, and when it is hard to create test cases representative of future
system conditions [24]. If major grid reinforcements are necessary, the costs will likely be
high and the wait time will increase as further permissions will be necessary before
construction begins.

Once connected to the grid, renewable resources still face challenges as their output
is subject to curtailment when there is an oversupply; and their revenue could be at risk in
cases of undersupply. Conditions of oversupply occur when the system cannot integrate
more renewable electricity when the grid is congested, when local or system-wide
electricity demand is relatively low, or when synchronous machines cannot be turned off
due to reliability concerns [21]. Transmission investments, flexible technologies such as
storage or demand response, and higher coordination with neighboring power systems
could help manage over-/under-supply and grid congestion [25][26]. However, permitting
challenges, outdated system protocols that prevent emerging resources from participating
in essential system services, and lack of products/services that value flexibility hinder
investment in transmission and flexible technologies. Knowledge gaps on the potential of
inverter-based resources to supply essential system services and the nature of services
needed in grids with high share of inverter-based resources further challenge renewable
integration [27].

Securing financing with a low cost of capital can be a challenge due to macro-
economic conditions, country-specific risks, or uncertainty with respect to future revenues
[28]. For example, in the UK, industry responses to proposed market reforms have warned
that uncertainty with future market prices and performance could result in an investment
hiatus [29]. Missing price signals or reward mechanisms for services needed by the system
[30] along with weak retail electricity prices can further exacerbate revenue risks.

2.7 Potential Solutions

To address these formidable challenges and pave the way for a sustainable energy
future, innovative solutions have emerged. Embracing these innovative solutions can
21
TR-123 — IEEE PES Technical Roadmap

accelerate the global transition to sustainable energy, fostering a cleaner and greener future
for generations to come.

2.7.1 Uncertainty and Risk Assessment and Management

To tackle the growing uncertainties and risks in the system operations under high
renewable integration, probabilistic assessment methods have been proposed as a solution.
It enables the extraction of detailed statistical information from the dynamic responses of
the power system. For instance, they can generate probability density functions (PDFs) that
enable the quantification of statistical characteristics of different quantities of interest. By
assessing the system responses across the entire range of possible values for uncertain
parameters, these probabilistic methods offer a means to identify system vulnerabilities and
enhance the understanding of the power system's behavior under uncertain conditions [15].

Monte Carlo sampling (MCS) is a widely used probabilistic approach to propagate

the uncertainties of parameters to power system dynamic response. By randomly and
repeatedly sampling from the distributions of parameters, tens of thousands of dynamic
simulations are conducted to get convergence [31]. An alternative approach to sampling-
based methods is the moments-based method, which focuses solely on the statistics of the
model output. Unlike sampling-based methods, this approach does not require the
construction of a surrogate model or the execution of a large number of samples. Instead,
it directly analyzes the moments of the model output to extract the desired statistical
information.

Once the uncertainties of the dynamic power system have been quantified,
mitigation actions can be implemented to ensure system stability. These actions involve
employing various strategies and measures to counteract the potential adverse effects
caused by uncertainties.

While the stochastic dynamic power system assessment and control methods have
made significant strides, several challenges remain to be addressed. These challenges
include:

(1) Scalability: The scalability of the methods to handle large-scale power systems
with numerous renewable energies and loads is a major challenge since it becomes an
extremely high-dimensional problem. Developing the probabilistic dynamic security
assessment and control techniques that can efficiently handle the increasing complexity
and size of modern power systems is crucial.

(2) Fast N-k Contingency Scanning: To mitigate the risk of system instability caused
by rare events, N-k contingency scanning is required. It involves more than thousands of
scanning tasks. How to quickly conduct the probabilistic dynamic security assessment and
control is still an open problem.

(3) Topology Changes: Power system topologies are subject to changes, i.e., line or
generator outages or switching due to maintenance or dispatching. The surrogate model-
based probabilistic dynamic security assessment and control become invalid in this

22
TR-123 — IEEE PES Technical Roadmap

scenario. Developing advanced computational techniques and algorithms that can quickly
update the model without compromising accuracy is a challenge that needs attention.

Addressing these challenges will further advance the field of stochastic dynamic
power system assessment and control, leading to a more robust and reliable operation of
power systems under uncertain conditions.

2.7.2 Transmission System Upgrades

Several transmission system upgrades have been implemented to address the

challenges associated with the integration of renewable energy. These upgrades aim to
ensure a reliable and efficient transmission of electricity from renewable energy resources
to consumers, overcoming the inherent variability and uncertainties of renewable
generations.

a. Energy storage integration: Energy storage technologies play a vital role in

enabling the seamless integration of renewable energy into the grid. By storing excess
renewable energy during periods of high generation and releasing it during low generation
periods, energy storage systems help balance supply and demand. Battery energy storage,
pumped hydro storage, compressed air energy storage, and other emerging storage
technologies are being integrated into the transmission system to provide grid flexibility
and stability, thereby supporting the integration of variable renewable energy sources.

b. High-voltage direct current (HVDC) transmission: HVDC transmission has

gained prominence in renewable energy integration due to its ability to efficiently transmit
electricity over long distances with lower losses compared to traditional alternating current
(AC) transmission. HVDC systems convert the renewable energy generated at the point of
generation into direct current (DC) and transmit it at high voltages to minimize
transmission losses. At the receiving end, the DC is converted back to AC for distribution.
HVDC transmission facilitates the integration of offshore wind farms, remote renewable
energy resources, and interconnecting grids, enabling the transfer of renewable energy
from resource-rich areas to population centers. As the penetration of dc increases, there is
a need to resolve several challenges like interoperability, feasibility of multi-vendor multi-
terminal dc systems, breakers needed in hybrid dc/ac systems, among others [88].

c. Interconnection and grid expansion: Interconnecting different regions and

expanding the transmission grid is essential for accommodating the increasing penetration
of renewable energy. By connecting diverse renewable energy resources across larger
geographical areas, grid operators can harness the benefits of geographic diversity,
reducing the impact of localized fluctuations in renewable generation. Enhanced
interconnections and grid expansion facilitate the sharing of renewable energy resources
and provide access to a broader range of generation options, improving grid stability and
reliability.

d. System stability enhancements (synchronous condensers, large rotating

machines): The integration of significant amounts of renewable energy can impact the
stability of the power system due to the inherent characteristics of renewable generation.
23
TR-123 — IEEE PES Technical Roadmap

To address this, system stability enhancements are implemented. Synchronous condensers,

which are rotating machines that provide reactive power support and help stabilize grid
voltage, are deployed to enhance system stability. Large rotating machines, such as
synchronous generators or flywheels, can provide inertia to the grid, improving frequency
stability and response. These technologies ensure the reliable operation of the power
system while accommodating high levels of renewable energy.

2.7.3 Distribution System Upgrades

The utility distribution systems are designed to deliver reliable electric power
economically to the electrical consumers at their place of consumption. However, over the
last decade, the electric power grid has been transforming unprecedentedly, necessitating
a significant change in how we design, operate, and control traditional power systems. High
penetration of DERs, the integration of electric vehicle technology, and bi-directional
power flow can potentially result in system-level challenges. Sensitivity of power system
protection may be affected too; especially for phase-to-ground faults due to diversity of
DER step-up transformer connections. Feeder protection may need the addition of
directional overcurrent, distance, and other directional elements. Future IEEE PSRC
standards could address this challenge. Interconnection requirements may be addressed in
IEEE 1547 and other new standards as well. Feeders are mostly radial but can be switched
in many configurations. With the increase of remote controls on switching devices, utilities
may be able to attend to contingencies and reroute DER to other substations or healthy
distribution feeders automatically. Automatic reclosing with sync-check hasn’t been
implemented yet at feeders, but it may be an option to be considered in the future as more
DER are added.

Another concept that may change in distribution networks with DER is anti-
islanding. Currently, issues at a substation or faults on feeders require to send remote trips
to DERs removing the generation. In the future, due to better control and communication
in distribution networks, it may be possible to allow islanding, or transitional islanding and
automatic reconnection of feeders without losing DER resources. Faults are more frequent
in distribution voltages (compared to transmission voltages) due to smaller electrical
clearances, vegetation within the Right of Way (ROW) corridors, and animal-caused
outages. Better availability of DER resources after contingencies is needed, as they will
become a comparable resource in capacity to traditional generation in the future. The
distribution feeder of the future should be able to identify the feeder that has an issue and
should isolate it automatically and retie feeder portions to other healthy parts of the
distribution network within cycles.

Forecast analysis for distribution networks is a field that will require new
standardization as new sensors and algorithms are placed in the network, so the utility may
be able to predict better modes of operation, loading, and shedding of circuits.

Moreover, with climate change, electric power grids are facing severe threats from
extreme weather events leading to extended outages, impacting critical services and
community wellbeing. Most of the extended outages observed in the aftermath of an
extreme weather event are due to the damage in the mid- and low-voltage power
24
TR-123 — IEEE PES Technical Roadmap

distribution systems. The impacts of such disruptions are especially severe for low-income
and underserved communities, which may lack non-electric resources and backup options.
The current capabilities to restore power mostly rely on a top-down approach. The “Last
mile” recovery, which depends on the repairs in low/mid voltage power distribution
systems, often takes several days. Fortunately, recent advances in distribution systems,
including the integration of DERs and distribution automation capabilities, provide
potential means to improve system resilience if applied in a purposeful and methodical
manner. If deployed and controlled purposefully, they can provide multiple crucial grid
services that can help improve the efficiency, reliability, and resilience of the power grid.

Due to the increased frequency, duration, and intensity of extreme weather events,
resilient planning and operational solutions for power distribution grid have drawn
significant attention. Different utilities have spent millions of dollars in deploying smart
grid technologies such as distribution automation with automated feeder switching,
intentional islanding (microgrid), as well as upgrading vulnerable feeders and substations
[45]. Solutions that augment resilience include system hardening as well operational
improvement. Examples of system hardening include conductor undergrounding,
microgrid deployment, solar and storage placement, and deployment of new sensing and
control technologies (such as smart meters and other grid-level sensors). Examples of
operational improvements include demand response and intentional islanding to reduce the
impacts of the extreme events [46][47]. Hardening the distribution grid and investing in
smart grid technologies are effective resilience-oriented designs that need to be adopted in
the utilities' portfolio for long-term infrastructure planning to improve grid's response to
extreme weather events. Likewise, the recent advances in smart grid technology that
include remote control capabilities and integration of DERs, help enable advanced grid
operations such as intentional islanding to support the critical services during disruptions
[48].

There is a broad literature on distribution system planning for resilience, which

primarily approaches planning as a high-dimensional stochastic optimization problem,
where all resources are designed simultaneously to minimize an overall expected cost
[49][50][51]. While these approaches are promising, disruption management entails
several practical challenges that require new methods. For example, investment for
disruption management needs to be driven by high impact low probability (HILP) events
rather than persistent costs that require characterizing the impacts of HILP events.
Furthermore, the distribution system planning should be driven by the need for operational
flexibility that requires integrating models for distribution systems along with advanced
system operations in computationally tractable ways. The existing planning and
operational solutions do not prioritize the use of PVs and storage for community resilience
enhancement, thus limiting the operational capabilities of the distribution grid during HILP
events. Finally, infrastructure planning for resilience is computationally challenging as it
requires: a) to include several different uncertainties (e.g., fault location, load profiles,
nature and severity of extreme events, and so forth) in the decision-making process, b)
ensure the validity of the planning decisions for the entire profile of weather events, and c)
mitigate the critical challenge of achieving a balance between computational cost and
accuracy. Thus, long-term infrastructure planning for resilience is conceptually a different
problem than the prevalent methods for operational planning solutions. This calls for new
25
TR-123 — IEEE PES Technical Roadmap

methods and contributions to systematically build resilience in active power distribution

grid infrastructures against the HILP events; such as risk-averse planning frameworks
while including operational models [52].

With all these aspects, distribution feeder design must evolve, and new standards
should address required clearances, controls, protection, and equipment specification to
address new challenges of the distribution network of the future. The design aspects should
also address impact on work methods and line worker safety. The future networks will have
many ways of re-energization and the workers will need to incorporate new procedures to
ensure circuits are safe to work on.

Regulations will need to incorporate the growing need of DER and ensure that
economic operation accounts for both traditional generation and DER. The distribution
network of the future will have a considerable local flow of energy that won’t necessarily
transit through the transmission system nor come from the traditional large generators (e.g.,
coal, natural gas, nuclear, hydroelectric). Currently, reliability regulations are more
stringent in networks 69 kV and above. More rules are needed in lower voltage feeders and
DER voltages (46 kV and under). In the past, distribution feeders were designed assuming
only loads were connected on them. Now with multiple DER in place, better lightning
performance, animal protection, wind withstand, and clearances may be required to
improve reliability. Some utilities build DER lines at a higher voltage equivalent for this
purpose (e.g., build a 15 kV circuit with 34.5 kV design criteria).

2.7.4 Smart Grid Technologies

Smart grid technologies can help manage the growing complexity and variability
through the use of advanced sensors and communication technologies that enable real-time
monitoring of the power grid, as well as the use of advanced analytics and machine learning
algorithms to analyze large amounts of data generated by the power grid.

Distribution systems include a variety of legacy control devices such as voltage

regulators, switched capacitors (SCs), switches, and feeder-level switches. Voltage
regulating elements such as load tap changers (LTCs) are available at the substation
transformers to regulate the customer end voltage. SCs are used for reactive power supply.
Devices such as circuit breakers, reclosers, sectionalizers and fuses are used for the system
and equipment protection. Switches and sectionalizers are often used to reconfigure the
distribution system feeders. Active distribution systems also include various DER
technologies, such as PVs, BESS, and Electric Vehicles (EVs), battery management
systems (BMS), etc. Several power-electronics-based devices have also been integrated
and employed for control voltage and power flow in the distribution systems [53]. Some
examples include Low-voltage Distribution Static Compensator (D-STATCOM), Static
Var Compensator (SVC), Unified power flow controller (UPFC) and Soft open points.
Along with the legacy metering equipment at substation and feeder levels, present day
distribution systems are also equipped with a communication infrastructure and the various
components that make up the Advanced Metering Infrastructure (AMI), and new power
electronics-based controllable devices.

26
TR-123 — IEEE PES Technical Roadmap

Historically, distribution system operations have employed rule-based methods to

control the feeder's few legacy voltage control devices, such as capacitor banks and voltage
regulators. These control rules were pre-designed and acted based on local measurements.
Since the loads were predictable and the system lacked any local generation resources, the
rule-based controls were sufficient to ensure desirable system operations. However, the
integration of DERs, including EVs has led to added variability and uncertainty in
distribution system operations rendering rule-based and local-control-only algorithms
inapplicable. Early studies showed that the new load types, such as EVs and photovoltaic
generation (PV) resources, might lead to multiple system-level challenges, including, but
not limited to, voltage limit violations, increased voltage variability and three-phase
voltage unbalance, and thermal limit violations [54][55][56]. It was also shown that the
local control might result in unnecessary tap changes and capacitor bank operations; these
are mechanical devices, and a higher number of operations can lead to mechanical failures
[57]. Mitigating these system-level operational challenges required a coordinated operation
of systems’ controllable devices. It was also recognized that the new grid-edge resources
could provide additional grid services, such as capacity, flexibility, ramping, voltage
support, and so on, that were previously not possible in a passive power distribution system.
These challenges and potential opportunities had resulted in the development of new
methods and advanced applications to actively manage grid-edge resources [58].

Maintaining the integrity of the power grid under adverse and highly uncertain
operating conditions requires novel grid management and response strategies.
Consequently, distribution system optimization has drawn significant attention and several
researchers have proposed formulation and algorithms to solve distribution-level optimal
power flow models. Although significant efforts have been made, a combination of
fundamental challenges needs addressing to optimally manage distribution-connected
millions of controllable grid-edge resources. The grid-edge optimization problem, which
requires real-time control and coordination of numerous DERs/DGs, is not amenable to the
current optimization algorithms. To make matters even more difficult, the problem's
nonconvexity, heterogeneity, and variety of control modes make it even more difficult to
develop scalable optimization models for grid-edge optimization. Additional challenges
arise when incorporating various grid-edge technologies, such as smart inverters, battery
energy storage systems, secondary voltage controllers and so on that introduce integer
variables into the underlying optimization problem. Furthermore, grid-edge resources
introduce uncertainties as a result of a lack of appropriate models and data or limited
forecasting capability (solar/wind generation), which should be systematically
incorporated into the optimization framework. The intertemporal constraints resulting from
the optimization of battery energy resources require a multi-time period formulation,
further increasing the problem scale, especially under uncertainty.

Given the challenges of purely model-based optimization methods, data-driven

model-free reinforcement learning (RL) approaches have recently emerged as an attractive
alternative to solving distribution-level OPF problems. A detailed survey summarizing the
applications of RL methods in power grid operation and control is provided in [59]. These
methods learn scalable operational strategies from interactions with a system-like
simulation model driven by a large amount of operating data that can be further used for
optimizing in new operating conditions. One of the applications of deep RL that has lately
27
TR-123 — IEEE PES Technical Roadmap

gained a lot of interest is the voltage regulation of the distribution systems. Specifically,
deep RL has been used to control capacitors, voltage regulators, and smart inverters to
regulate voltages and reduce power losses. Unfortunately, existing model-free RL
algorithms ignore the crucial information embedded in the physics-based model of the
power distribution systems and may thus compromise the optimizer performance and pose
scalability challenges. In more recent works, including power system model information
in neural networks has shown to improve the performance of the OPF problems [60][61].
Another recent work uses imitation learning algorithms to speed up the RL training using
model-based approximate distribution OPF (D-OPF) solutions [62]. The synergistic
integration of physics-based information into data-driven approaches is an active area of
research in power grid optimization.

2.7.5 Transmission and Distribution Coordination

Transmission and Distribution (T&D) coordination can result in increased

operational flexibility and better absorption of the renewable resources across regions as it
allows energy to be transported from areas with high generation potential to areas with
high demand.

The industry should transition the T&D model into a model of ‘make-use’ so the
generated resources, whether they are produced at a traditional power plant or a DER, can
be routed with the optimal power flow, reduced losses, and economical dispatch among
high voltage and medium voltage networks. Currently, transmission equipment has vast
controls and communication installations. Distribution is not at the same level in
communications, but as of 2023 more utilities are carrying out modernization projects and
investing billions of dollars in distribution infrastructure. Over the next decade, it will be
possible to integrate communication between transmission and distribution networks. New
standards must be developed to aid in the transition. Examples of smart distribution line
equipment include motor operated disconnect switches with remote open/close status,
automatic control of line capacitor banks, instrument transformers for power flow
monitoring, remotely operated reclosers, smart fault indicators, and else.

With the advent of smart metering devices and end-use appliances, a new era has
started where demand side resources, e.g., residential PVs and EVs, are able to sell
electricity back to the network through commercial agreements and phone applications. It
requires complex networks of communications between a vast amount of devices.
Standardization is required for determining communication protocols, best practices for
signals to measure, and what the monitoring equipment performance specifications should
be. Utilities may have access in the future to monitor and control residential, commercial,
and industrial customers owned equipment, appliances, etc. to help manage generation-
load demand. Consumers will require access to real-time usage and cost data to be able to
make evidence-based decisions on consumption (timing) to balance their need for power
with cost. Fast switching is possible for lowering peak demand, for contingency mitigation,
and for system stability.

28
TR-123 — IEEE PES Technical Roadmap

2.7.6 Market and Policy Enhancement

Present challenges for integration of renewables call for enhancements in

governance, power sector planning, connection studies, and power system operations and
markets [63].

Legal and regulatory frameworks are at the core of energy governance. They are essential
to clarify roles, responsibilities, and interactions among institutions and organizations.
Enhancements to legal and regulatory frameworks can address complexities arising in
topics falling under ambiguous or overlapping jurisdictions. For example, FERC order
1000 [64] and FERC order 2222 [65] in the USA mandated the creation of frameworks for
transmission planning across jurisdictions and for coordination among the regional grid
operator, the DER aggregator, the distribution utility and the relevant retail regulatory
authority. Interactions of power system operators with assets are governed by the grid
code. As more and more assets are connected to the grid through inverters or aggregators,
grid code updates are necessary to clarify what data and models should the system operator
have access to and how the system operator will communicate with and verify performance
of those assets [66].

Enhancements to power sector planning and connection studies that incentivize

more proactive and strategic investments can help accelerate the integration of renewable
power plants. Enhancements can bridge the gap between planning and connection studies;
focus on recognizing different types of benefits of grid reinforcements under a wide range
of future system conditions; engage stakeholders to foster plan acceptance; and allocate
costs for grid reinforcements in more equitable and robust manners [67].

So far, enhancements in markets and power system operations have focused on

removing barriers for emerging technologies by allowing participation of new technologies
in existing services [79]; designing new, sometimes technology-specific, products [80];
creating participation models that recognize unique characteristics of emerging resources
[81]; and investigating how operability requirements translate into services [82]. Future
enhancements could further refine and finalize definitions and procurement mechanisms
of services aiming for (1) technology-agnostic definitions to encourage competition among
suppliers; (2) a balance between complexity and effectiveness; and (3) price signals that
reflect the value of flexibility and other services and support reliable and cost-effective
system operations. In addition to wholesale markets, retail market enhancements could
encourage demand-side flexibility and promote affordability [68].

The impact of policies on renewable investments to date cannot be understated.

Policies that aim to spur innovation in new technologies or accelerate the adoption of
mature technologies by providing financial incentives will continue to play a key role in
the future. Future policies will be even more successful if further alignment is achieved
between strategic energy goals and planning, operations, and markets. Some call for clear
mandates in the form of carbon taxes, green power pools, clean firm energy products, and
carbon taxes; while others prioritize incremental changes that promote further alignment
[83]. No policy pathway will be suitable for every system around the globe and system-

29
TR-123 — IEEE PES Technical Roadmap

specific assessments will help decision makers choose pathways appropriate for their
systems.

Proposed policies aimed at implementing a circular economy and scarce resources

management as well as regulations, like the European Unions’ Digital Product Passport,
will impact infrastructure design and upgrades.

2.7.7 Future Technologies Advancement

The pace at which information, technology, and research and development (R&D)
advances today has increased significantly compared to the rest of human history. In 1900,
it took humanity about 100 years to double knowledge. In 2023, it takes about 13 months
to double knowledge. That said, there are many studies available that point to physical
phenomena and equipment that could be used in the power industry in the near future. The
following shows only a small part of the ongoing discoveries that will help innovate power
systems by 2050. It is said that most of the technologies needed to reduce emissions of
CO2 may need to come from technologies that are not yet in the market or near scale as of
2023.

It is usual to define human history by the prime materials discovered and used (iron,
bronze, etc.). We are currently living in the silicon age which has produced great inventions
such as transistors, electronics, communications, and computers. In the next 50 years,
power systems will see the introduction of new materials and innovation that will improve
power transfer, data encryption, and faster computing.

Research in superconductors has gone a long way and recent discoveries have shed
light on what properties of materials need to be controlled to achieve critical temperatures
closer to ambient temperature. The current drawback of some materials that achieve
superconductivity at high temperatures (70 K or above) is that they require a very high
operating pressure. Conductors with very high conductivity will be useful for overhead line
design once a material with optimal operating temperature and pressure is discovered.

Research on Graphene has increased a lot due to its mechanical strength benefits
(1100 GPa Young modulus, 125 GPa shear strength), and high conductivity (~ 1700 S/m).
It is currently used for aerospace and military applications due to its good electrical
performance and light weight. Today it is cost prohibitive for T&D power system
applications, but as manufacturing methods improve, its cost will decline, and its purity
will improve as well. Graphene can be used in manufacturing conductors for power and
instrumentation, antennas, and electro-magnetic (EM) shielding.

Also, a recent discovery by [69] and [70] showed that graphene may be used for
generating electricity, in the order of ~0.0025 (µV/mm2)/(mm/s). When water flows
through a sheet of graphene an electrical potential is generated that is proportional to flow
speed and surface area. This hydro-voltaic effect is little understood and will require future
studies and creation of large prototypes for generating at small scale at first.

30
TR-123 — IEEE PES Technical Roadmap

In 2022, the European Space Agency ESA, through the SOLARIS R&D project
[71], has planned to study if space-based solar power is feasible by 2025. After that stage
is completed, a small in-orbit demonstration platform is to be launched which will consist
of a network of solar power satellites in geostationary orbit that will harvest sunlight 24/7.
The system will convert stored energy into low power density microwaves that can be
beamed down to earth to receiver stations (rectennas). Space solar eliminates the
intermittent nature of ground-based solar alternatives, which only have a plant factor of
about 25%. The UK has a similar program known as the Space Energy Initiative SEI to
develop space-based solar as well [72]. They plan to develop the first orbital demonstrator
by 2030, and a system that will be able to generate power into the grid by 2040 in the UK.

The photoelectric effect was first discovered by Einstein in 1905 and earned him a
Nobel Prize in 1921. This effect is responsible for how solar panels operate. Since its
discovery, the field has kept evolving. Traditionally, it is based on a quasi-dc effect in
which a material absorbs photons of light and release electrons and can cause an electric
current. The efficiency of solar panels keeps improving. It is currently at 15-25% for
commercial applications (e.g., multi-crystalline silicon, thin film CdTe), however, R&D of
panel materials including technical ceramics expects to improve this to about 45% in the
near future (e.g., multi-junction cells). That will improve land area use in solar plants. Also,
in 2020, a group of scientists [73] from the Georgia Institute of Technology in USA
discovered that it is possible to generate an ac signal with solar panels. R&D is ongoing to
fully understand this phenomenon. It is believed to be a result of the realignment of Fermi
levels of semiconductors adjacent to p-n junctions in nonequilibrium condition, which
causes a potential difference between electrodes. The effect could be used for developing
solar panels that can produce dc and ac power.

2.8 References
[1] “Renewable Capacity Statistics 2022,” Apr. 11, 2022.
https://www.irena.org/publications/2022/Apr/Renewable-Capacity-Statistics-2022
(accessed Jul. 08, 2023).
[2] “Renewable Electricity – Analysis,” IEA. https://www.iea.org/reports/renewable-
electricity (accessed Jul. 08, 2023).
[3] “Renewables Data Explorer – Data Tools,” IEA. https://www.iea.org/data-and-
statistics/data-tools/renewables-data-explorer (accessed Jul. 08, 2023).
[4] “Is there enough global wind and solar PV manufacturing to meet Net Zero targets
in 2030? – Renewable Energy Market Update - June 2023 – Analysis,” IEA.
https://www.iea.org/reports/renewable-energy-market-update-june-2023/is-there-
enough-global-wind-and-solar-pv-manufacturing-to-meet-net-zero-targets-in-
2030 (accessed Jul. 08, 2023).
[5] Zhoufeng, “China’s 14th Five-Year Plans on Renewable Energy Development and
Modern Energy System —.” Accessed: Jul. 09, 2023. [Online]. Available:
https://www.efchina.org/Blog-en/blog-20220905-en
[6] “REPowerEU: affordable, secure and sustainable energy for Europe.”
https://commission.europa.eu/strategy-and-policy/priorities-2019-2024/european-
green-deal/repowereu-affordable-secure-and-sustainable-energy-europe_en
(accessed Jul. 09, 2023).
31
TR-123 — IEEE PES Technical Roadmap

[7] “Inflation Reduction Act Guidebook | Clean Energy,” The White House.
https://www.whitehouse.gov/cleanenergy/inflation-reduction-act-guidebook/
(accessed Jul. 09, 2023).
[8] “Presidential Policy Directive -- Critical Infrastructure Security and Resilience,”
whitehouse.gov, Feb. 12, 2013. https://obamawhitehouse.archives.gov/the-press-
office/2013/02/12/presidential-policy-directive-critical-infrastructure-security-
and-resil (accessed Jul. 09, 2023).
[9] A. B. Smith, “U.S. Billion-dollar Weather and Climate Disasters, 1980 - present
(NCEI Accession 0209268).” NOAA National Centers for Environmental
Information, 2020. doi: 10.25921/STKW-7W73.
[10] “Global Warming of 1.5 oC —.” https://www.ipcc.ch/sr15/ (accessed Jul. 09, 2023).
[11] “The Paris Agreement | UNFCCC.” https://unfccc.int/process-and-meetings/the-
paris-agreement (accessed Jul. 09, 2023).
[12] T. W. House, “FACT SHEET: President Biden Signs Executive Order Catalyzing
America’s Clean Energy Economy Through Federal Sustainability,” The White
House, Dec. 08, 2021. https://www.whitehouse.gov/briefing-room/statements-
releases/2021/12/08/fact-sheet-president-biden-signs-executive-order-catalyzing-
americas-clean-energy-economy-through-federal-sustainability/ (accessed Jul. 09,
2023).
[13] B. R. Prusty and D. Jena, “A critical review on probabilistic load flow studies in
uncertainty constrained power systems with photovoltaic generation and a new
approach,” Renewable and Sustainable Energy Reviews, vol. 69, pp. 1286–1302,
Mar. 2017, doi: 10.1016/j.rser.2016.12.044.
[14] B. Tan and J. Zhao, “Debiased Uncertainty Quantification Approach for
Probabilistic Transient Stability Assessment,” IEEE Trans. Power Syst., pp. 1–4,
2023, doi: 10.1109/TPWRS.2023.3276207.
[15] K. Ye, J. Zhao, N. Duan, and Y. Zhang, “Physics-Informed Sparse Gaussian
Process for Probabilistic Stability Analysis of Large-Scale Power System With
Dynamic PVs and Loads,” IEEE Trans. Power Syst., vol. 38, no. 3, pp. 2868–2879,
May 2023, doi: 10.1109/TPWRS.2022.3188182.
[16] “Renewable Power Generation Costs in 2021,” Jul. 13, 2022.
https://www.irena.org/publications/2022/Jul/Renewable-Power-Generation-Costs-
in-2021 (accessed Jul. 15, 2023).
[17] “World Energy Investment 2022,” 2022.
[18] “Lack of vision’: UK green energy projects in limbo as grid struggles to keep pace
|National Grid | The Guardian.”
https://www.theguardian.com/business/2023/may/08/uk-green-energy-projects-in-
limbo-as-grid-struggles-to-keep-pace (accessed Jul. 15, 2023).
[19] J. Rand, M. Bolinger, R. Wiser, S. Jeong, and B. Paulos, “Queued Up:
Characteristics of Power Plants Seeking Transmission Interconnection As of the
End of 2020,” None, 1784303, ark:/13030/qt5jd5x0q9, May 2021. doi:
10.2172/1784303.
[20] J. Seel and J. M. Kemp, “Generator Interconnection Costs to the Transmission
System”.
[21] EIRGRID and SONI, “Annual Renewable Constraint and Curtailment Report
2022.”
32
TR-123 — IEEE PES Technical Roadmap

[22] “Renewables projects face 10-year wait to connect to electricity grid,” May 08,
2022. https://www.ft.com/content/7c674f56-9028-48a3-8cbf-c1c8b10868ba
(accessed Jul. 15, 2023).
[23] “Queue management: the next step in accelerating grid connections | National Grid
ET.” https://www.nationalgrid.com/electricity-transmission/queue-management-
next-step-accelerating-grid-connections (accessed Jul. 15, 2023).
[24] US Department of Energy, “Queue management & cost allocation,” [Online].
Available: https://www.energy.gov/sites/default/files/2023-05/Solution e-Xchange
Managing the Bulk Power System.pdf
[25] CAISO, “Managing oversupply,” 15 2023.
http://www.caiso.com/informed/Pages/ManagingOversupply.aspx
[26] P. Denholm et al., “The challenges of achieving a 100% renewable electricity
system in the United States,” Joule, vol. 5, no. 6, pp. 1331–1352, Jun. 2021, doi:
10.1016/j.joule.2021.03.028.
[27] J. Bialek, “System Needs and Services for Systems with High IBR penetration.”
[Online]. Available: https://globalpst.org/wp-content/uploads/GPST-IBR-
Research-Team-System-Services-and-Needs-for-High-IBR-Networks.pdf
[28] IRENA, “The cost of financing for renewable power,” May 03, 2023.
https://www.irena.org/Publications/2023/May/The-cost-of-financing-for-
renewable-power (accessed Jul. 15, 2023).
[29] “Review of Electricity Market Arrangements Summary of responses to
consultation”.
[30] E. Nicholson, “The case for LMPs, EISG Meteorology and Market Design for Grid
Services Workshop 2023,” [Online]. Available:
https://www.esig.energy/event/2023-meteorology-and-market-design-for-grid-
services-workshop/
[31] J. V. Milanović, “Probabilistic stability analysis: the way forward for stability
analysis of sustainable power systems,” Philosophical Transactions of the Royal
Society A: Mathematical, Physical and Engineering Sciences, vol. 375, no. 2100,
p. 20160296, Jul. 2017, doi: 10.1098/rsta.2016.0296.
[32] “Probabilistic Voltage Stability Assessment Considering Stochastic Load Growth
Direction and Renewable Energy Generation | IEEE Conference Publication | IEEE
Xplore.” https://ieeexplore.ieee.org/document/8585868 (accessed Jul. 15, 2023).
[33] W. Wangdee and R. Billinton, “Bulk electric system well-being analysis using
sequential Monte Carlo simulation,” IEEE Transactions on Power Systems, vol. 21,
no. 1, pp. 188–193, Feb. 2006, doi: 10.1109/TPWRS.2005.862000.
[34] W. C. Wong, C. Y. Chung, K. W. Chan, and H. Chen, “Quasi-Monte Carlo Based
Probabilistic Small Signal Stability Analysis for Power Systems With Plug-In
Electric Vehicle and Wind Power Integration,” IEEE Transactions on Power
Systems, vol. 28, no. 3, pp. 3335–3343, Aug. 2013, doi:
10.1109/TPWRS.2013.2254505.
[35] M. Perninge and L. Söder, “Analysis of transfer capability by Markov Chain Monte
Carlo simulation,” in 2010 IEEE International Conference on Power and Energy,
Nov. 2010, pp. 232–237. doi: 10.1109/PECON.2010.5697582.
[36] P. N. Papadopoulos, T. Guo, and J. V. Milanović, “Probabilistic Framework for
Online Identification of Dynamic Behavior of Power Systems With Renewable
33
TR-123 — IEEE PES Technical Roadmap

Generation,” IEEE Transactions on Power Systems, vol. 33, no. 1, pp. 45–54, Jan.
2018, doi: 10.1109/TPWRS.2017.2688446.
[37] J. Liu, X. Wang, and X. Wang, “A Sparse Polynomial Chaos Expansion-Based
Method for Probabilistic Transient Stability Assessment and Enhancement,” in
2022 IEEE Power & Energy Society General Meeting (PESGM), Jul. 2022, pp. 1–
5. doi: 10.1109/PESGM48719.2022.9916882.
[38] Y. Xu, L. Mili, A. Sandu, M. R. von Spakovsky, and J. Zhao, “Propagating
Uncertainty in Power System Dynamic Simulations Using Polynomial Chaos,”
IEEE Transactions on Power Systems, vol. 34, no. 1, pp. 338–348, Jan. 2019, doi:
10.1109/TPWRS.2018.2865548.
[39] K. Ye, J. Zhao, N. Duan, and D. A. Maldonado, “Stochastic Power System
Dynamic Simulation and Stability Assessment Considering Dynamics From
Correlated Loads and PVs,” IEEE Transactions on Industry Applications, vol. 58,
no. 6, pp. 7764–7775, Nov. 2022, doi: 10.1109/TIA.2022.3195475.
[40] Z. Yue, Y. Liu, Y. Yu, and J. Zhao, “Probabilistic transient stability assessment of
power system considering wind power uncertainties and correlations,”
International Journal of Electrical Power & Energy Systems, vol. 117, p. 105649,
May 2020, doi: 10.1016/j.ijepes.2019.105649.
[41] R. Preece, K. Huang, and J. V. Milanović, “Probabilistic Small-Disturbance
Stability Assessment of Uncertain Power Systems Using Efficient Estimation
Methods,” IEEE Transactions on Power Systems, vol. 29, no. 5, pp. 2509–2517,
Sep. 2014, doi: 10.1109/TPWRS.2014.2308577.
[42] “Probabilistic transient stability constrained optimal power flow for power systems
with multiple correlated uncertain wind generations - Google Search.”
https://www.google.com/search?q=Probabilistic+transient+stability+constrained+
optimal+power+flow+for+power+systems+with+multiple+correlated+uncertain+
wind+generations&rlz=1C5GCEA_en&oq=Probabilistic+transient+stability+cons
trained+optimal+power+flow+for+power+systems+with+multiple+correlated+un
certain+wind+generations&aqs=chrome..69i57.452j0j4&sourceid=chrome&ie=U
TF-8 (accessed Jul. 15, 2023).
[43] Y. Chen, S. M. Mazhari, C. Y. Chung, and S. O. Faried, “A Preventive Dispatching
Method for High Wind Power-Integrated Electrical Systems Considering
Probabilistic Transient Stability Constraints,” IEEE Open Access Journal of Power
and Energy, vol. 8, pp. 472–483, 2021, doi: 10.1109/OAJPE.2021.3098658.
[44] T. Su, J. Zhao, and X. Chen, “Deep Sigma Point Processes-Assisted Chance-
Constrained Power System Transient Stability Preventive Control,” IEEE
Transactions on Power Systems, pp. 1–13, 2023, doi:
10.1109/TPWRS.2023.3270800.
[45] R. Moreno et al., “From Reliability to Resilience: Planning the Grid Against the
Extremes,” IEEE Power and Energy Magazine, vol. 18, no. 4, pp. 41–53, Jul. 2020,
doi: 10.1109/MPE.2020.2985439.
[46] Q. Shi, W. Liu, B. Zeng, and F. Li, “Enhancing distribution system resilience
against extreme weather events: Concept review, algorithm summary, and future
vision ☆,” International Journal of Electrical Power & Energy Systems, vol. 138,
p. 107860, Jun. 2022, doi: 10.1016/j.ijepes.2021.107860.

34
TR-123 — IEEE PES Technical Roadmap

[47] G. Liu, T. Jiang, T. B. Ollis, X. Li, F. Li, and K. Tomsovic, “Resilient distribution
system leveraging distributed generation and microgrids: a review,” IET Energy
Systems Integration, vol. 2, no. 4, pp. 289–304, 2020, doi: 10.1049/iet-
esi.2019.0134.
[48] S. Poudel, A. Dubey, and A. Bose, “Risk-Based Probabilistic Quantification of
Power Distribution System Operational Resilience,” IEEE Systems Journal, vol.
14, no. 3, pp. 3506–3517, Sep. 2020, doi: 10.1109/JSYST.2019.2940939.
[49] E. Yamangil, R. Bent, and S. Backhaus, “Resilient Upgrade of Electrical
Distribution Grids,” Proceedings of the AAAI Conference on Artificial Intelligence,
vol. 29, no. 1, Art. no. 1, Feb. 2015, doi: 10.1609/aaai.v29i1.9369.
[50] W. Yuan, J. Wang, F. Qiu, C. Chen, C. Kang, and B. Zeng, “Robust Optimization-
Based Resilient Distribution Network Planning Against Natural Disasters,” IEEE
Transactions on Smart Grid, vol. 7, no. 6, pp. 2817–2826, Nov. 2016, doi:
10.1109/TSG.2015.2513048.
[51] S. Ma, S. Li, Z. Wang, and F. Qiu, “Resilience-Oriented Design of Distribution
Systems,” IEEE Transactions on Power Systems, vol. 34, no. 4, pp. 2880–2891,
Jul. 2019, doi: 10.1109/TPWRS.2019.2894103.
[52] A. Poudyal, S. Poudel, and A. Dubey, “Risk-Based Active Distribution System
Planning for Resilience Against Extreme Weather Events,” IEEE Transactions on
Sustainable Energy, vol. 14, no. 2, pp. 1178–1192, Apr. 2023, doi:
10.1109/TSTE.2022.3220561.
[53] H. V. Padullaparti, Q. Nguyen, and S. Santoso, “Advances in volt-var control
approaches in utility distribution systems,” in 2016 IEEE Power and Energy
Society General Meeting (PESGM), Jul. 2016, pp. 1–5. doi:
10.1109/PESGM.2016.7741366.
[54] Y. T. Tan and D. S. Kirschen, “Impact on the Power System of a Large Penetration
of Photovoltaic Generation,” in 2007 IEEE Power Engineering Society General
Meeting, Jun. 2007, pp. 1–8. doi: 10.1109/PES.2007.385563.
[55] A. Dubey and S. Santoso, “Electric Vehicle Charging on Residential Distribution
Systems: Impacts and Mitigations,” IEEE Access, vol. 3, pp. 1871–1893, 2015, doi:
10.1109/ACCESS.2015.2476996.
[56] A. Dubey and S. Santoso, “On Estimation and Sensitivity Analysis of Distribution
Circuit’s Photovoltaic Hosting Capacity,” IEEE Trans. Power Syst., vol. 32, no. 4,
pp. 2779–2789, Jul. 2017, doi: 10.1109/TPWRS.2016.2622286.
[57] Y. P. Agalgaonkar, B. C. Pal, and R. A. Jabr, “Distribution Voltage Control
Considering the Impact of PV Generation on Tap Changers and Autonomous
Regulators,” IEEE Trans. Power Syst., vol. 29, no. 1, pp. 182–192, Jan. 2014, doi:
10.1109/TPWRS.2013.2279721.
[58] A. Dubey and S. Paudyal, “Distribution System Optimization to Manage
Distributed Energy Resources (DERs) for Grid Services,” EES, vol. 6, no. 3–4, pp.
120–264, Jun. 2023, doi: 10.1561/3100000030.
[59] Y. Gao and N. Yu, “Deep Reinforcement Learning in Power Distribution Systems:
Overview, Challenges, and Opportunities,” in 2021 IEEE Power & Energy Society
Innovative Smart Grid Technologies Conference (ISGT), Feb. 2021, pp. 1–5. doi:
10.1109/ISGT49243.2021.9372283.

35
TR-123 — IEEE PES Technical Roadmap

[60] Y. Chen and B. Zhang, “Learning to Solve Network Flow Problems via Neural
Decoding.” arXiv, Jun. 09, 2020. doi: 10.48550/arXiv.2002.04091.
[61] H. Lange, B. Chen, M. Berges, and S. Kar, “Learning to Solve AC Optimal Power
Flow by Differentiating through Holomorphic Embeddings.” arXiv, Dec. 16, 2020.
doi: 10.48550/arXiv.2012.09622.
[62] G. Krishnamoorthy, A. Dubey, and A. H. Gebremedhin, “Reinforcement Learning
for Battery Energy Storage Dispatch augmented with Model-based Optimizer,” in
2021 IEEE International Conference on Communications, Control, and Computing
Technologies for Smart Grids (SmartGridComm), Oct. 2021, pp. 289–294. doi:
10.1109/SmartGridComm51999.2021.9632292.
[63] “Status of Power System Transformation 2019: Power system flexibility –
Analysis,” IEA. https://www.iea.org/reports/status-of-power-system-
transformation-2019 (accessed Jul. 15, 2023).
[64] “Order No. 1000 - Transmission Planning and Cost Allocation | Federal Energy
Regulatory Commission.” https://www.ferc.gov/electric-transmission/order-no-
1000-transmission-planning-and-cost-allocation (accessed Jul. 15, 2023).
[65] “FERC Order No. 2222: Fact Sheet | Federal Energy Regulatory Commission.”
https://www.ferc.gov/media/ferc-order-no-2222-fact-sheet (accessed Jul. 15,
2023).
[66] “Scaling up Variable Renewable Power The Role of Grid Codes,” May 01, 2016.
https://www.irena.org/publications/2016/May/Scaling-up-Variable-Renewable-
Power-The-Role-of-Grid-Codes (accessed Jul. 15, 2023).
[67] J. Pfeifenberger and J. DeLosa, “Proactive, Scenario-Based, Multi-Value
Transmission Planning,” Available online at https://www.brattle.com/wp-
content/uploads/2022/06/Proactive-Scenario-Based-Multi-Value-Transmission-
Planning.pdf.
[68] A. Olson et al., “Rate Design for the Energy Transition: Getting the most out of
flexible loads on a changing grid, Energy System Integration Group White Paper,”
2023.
[69] K. Kuriya et al., “Output density quantification of electricity generation by flowing
deionized water on graphene,” Applied Physics Letters, vol. 117, no. 12, p. 123905,
Sep. 2020, doi: 10.1063/5.0018862.
[70] J. Xie, L. Wang, X. Chen, P. Yang, F. Wu, and Y. Huang, “The Emerging of
Hydrovoltaic Materials as a Future Technology: A Case Study for China,” in Green
Energy and Environment, E. Hwa Yap and A. Huey Ping Tan, Eds., IntechOpen,
2020. doi: 10.5772/intechopen.90377.
[71] “SOLARIS.”
https://www.esa.int/Enabling_Support/Space_Engineering_Technology/SOLARI
S/SOLARIS2 (accessed Jul. 15, 2023).
[72] “Space Energy Initiative, Space-Based Energy solutions to address global energy
challenges.” https://spaceenergyinitiative.org.uk/ (accessed Jul. 15, 2023).
[73] H. Zou et al., “Alternating Current Photovoltaic Effect,” Advanced Materials, vol.
32, no. 11, p. 1907249, 2020, doi: 10.1002/adma.201907249.
[74] PES TR-87: Protection of Wind Electric Plants | IEEE Power & Energy Society
Resource Center (ieee-pes.org) [resourcecenter.ieee-pes.org].

36
TR-123 — IEEE PES Technical Roadmap

[75] PES TR-78: Modification of Commercial Fault Calculation Programs for Wind
Turbine Generators | IEEE Power & Energy Society Resource Center (ieee-pes.org)
[resourcecenter.ieee-pes.org].
[76] PES TR-71: Microgrid Protection Systems | IEEE Power & Energy Society
Resource Center (ieee-pes.org) [resourcecenter.ieee-pes.org]
[77] PES TR-81: Protection Challenges and Practices for Interconnecting Inverter Based
Resources to Utility Transmission Systems | IEEE Power & Energy Society
Resource Center (ieee-pes.org) [resourcecenter.ieee-pes.org]
[78] PES TR-26: Fault Current Contributions from Wind Plants | IEEE Power & Energy
Society Resource Center (ieee-pes.org) [resourcecenter.ieee-pes.org]
[79] I. Chernyakhovskiy, S. Koebrich, V. Gevorgian, and J. Cochran,“Grid-Friendly
Renewable Energy: Solar and Wind Participation in Automatic Generation Control
Systems,” Greening the Grid Program A joint initiative by USAID/India and
Ministry of Power, 2019.
[80] EPEX SPOT introduces curtailable blocks and loop blocks on all Day-Ahead
markets, 13 Dec 2018. Available
at: https://www.epexspot.com/sites/default/files/download_center_files/181213_E
PEXSPOT_Press_Release_Loop%20and%20curtailable%20blocks.pdf
[81] FERC Order No. 841, Electric Storage Participation in Markets Operated by
Regional Transmission Organizations and Independent System Operators
[82] National Grid ESO, Operability Strategy Report, Dec. 2023. Available
at: https://www.nationalgrideso.com/document/299926/download
[83] B. Tam and A. Walker, Electricity Market Reform, UK Parliament Post, Post Note
694, May 2023
[84] Suman Debnath, Marcelo A. Elizondo, Yuan Liu, Phani Ratna Vanamali Marthi,
Wei Du, Shilpa Marti, and Qiuhua Huang. "High penetration power electronics
grid: Modeling and simulation gap analysis." (2020).
[85] “DOE ESHB Chapter 11 Hydrogen Energy Storage”, Sandia National
Laboratories,
https://www.sandia.gov/app/uploads/sites/163/2022/03/ESHB_Ch11_Hydrogen_
Headley.pdf
[86] "Utility-scale batteries and pumped storage return about 80% of the electricity they
store", U.S. Energy Information Administration,
https://www.eia.gov/todayinenergy/detail.php?id=46756#:~:text=According%20t
o%20data%20from%20the,%2Dtrip%20efficiency%20of%2079%25.
[87] Suman Debnath et al., "Renewable Integration in Hybrid AC/DC Systems Using a
Multi-Port Autonomous Reconfigurable Solar Power Plant (MARS)," in IEEE
Transactions on Power Systems, vol. 36, no. 1, pp. 603-612, Jan. 2021.
[88] “High-Voltage Direct Current (HVDC) COst REduction (CORE) Initiative”, US
Department of Energy, available at
https://www.energy.gov/sites/default/files/2023-
08/HVDC%20CORE%20Initiative_FINAL.pdf.

37
TR-123 — IEEE PES Technical Roadmap

3. Reliability and Resilience

3.1 Executive Summary
This chapter explores elements of reliability, resilience, resource adequacy, and
fuel/energy assurance within power systems. It discusses the importance of condition
assessment strategies, noting the current use of advanced diagnostics, and emphasizes the
future vision of effectively identifying asset health issues. The chapter also acknowledges
the challenges brought on by extreme weather events, the current designs of grids, and the
vision of creating resilient structures to minimize damage. The security aspect is explored
too, focusing on physical and human-caused attacks, their potential damage, and strategies
for mitigation. The chapter introduces the concept of resilience and reliability,
differentiating the two while outlining the vision to develop a resilience performance
scoring matrix. Reliable sourcing is discussed as a key factor in promoting national energy
security, along with prognostics and health management (PHM) in improving system
reliability. The chapter also details the hardening of power systems and the restoration
process, highlighting the current status, challenges, and solutions in each domain. Finally,
it underlines the importance of fault tolerance and its implementation, and how to ensure
system reliability under climate-induced extreme weather conditions.

3.2 Reliability and Resilience Definition and Metrics

3.2.1 Summary
Understanding the nuanced differences between resilience and reliability is
paramount for enhancing the robustness of critical systems like power grids. Reliability, as
outlined by IEEE-Std-100TM, quantifies the probability of a system successfully
performing its required functions within a specified timeframe and under specific
environmental conditions. In contrast, resilience encompasses a broader spectrum of
attributes, including the ability to endure, reduce the impact of, and swiftly recover from
disruptive events. This holistic concept encompasses anticipation, absorption, adaptation,
and recovery [1][2].

3.2.2 Current State

At present, reliability assessment of distribution systems in the power industry
relies heavily on well-established metrics such as the System Average Interruption
Duration Index (SAIDI) and System Average Interruption Frequency Index (SAIFI). These
metrics play a fundamental role in gauging the reliability of power distribution systems,
providing valuable insights into outage duration and frequency under normal operating
conditions.

However, while SAIDI and SAIFI are widely used and respected metrics for
evaluating reliability, the concept of resilience within power systems goes beyond these
traditional measures. Resilience encompasses not only the ability to maintain reliability but
also to recover swiftly from disruptions and adapt effectively to changing circumstances.

38
TR-123 — IEEE PES Technical Roadmap

One notable challenge in the field is the absence of a standardized and universally
accepted performance metric for measuring resilience. While metrics like SAIDI and
SAIFI serve as robust tools for assessing reliability, quantifying and standardizing
resilience metrics are ongoing areas of development and research within the power
industry.

The goal is to establish metrics that can comprehensively evaluate a power system's
capacity to withstand and rebound from adverse events while maintaining a reliable supply
of electricity.

3.2.3 Future State

The overarching objective is to establish systems that seamlessly blend reliability
and resilience. A reliable system ensures the continuous provision of services under normal
operational circumstances, while a resilient system can withstand disruptions, adapt to
changing conditions, and recover rapidly from adverse events. This vision necessitates the
development of a comprehensive resilience performance scoring matrix to systematically
assess and enhance the resilience of power grids.

3.2.4 Challenges
Several substantial challenges loom on the path to creating a robust and
comprehensive resilience scoring method. The increasing frequency and intensity of
threats, both natural and anthropogenic, make it progressively difficult to develop a method
that accounts for all possible scenarios. While reliability analysis typically focuses on
specific, well-defined conditions, resilience demands a more extensive perspective. It must
encompass the potential for known and unknown threats, a myriad of possible disruptions,
and various sources of adversity. Furthermore, enhancing resilience often involves
complex system design considerations to facilitate self-healing and recovery mechanisms,
further complicating the task.

3.2.5 Potential Solutions

Addressing these challenges requires a multi-pronged approach. Potential solutions
include the development of a well-defined resilience performance scoring system that
incorporates various aspects of resilience, such as anticipatory capabilities, adaptive
strategies, and rapid recovery measures. Creating scoring worksheets for each resilience
subcomponent can offer a structured approach to assessment. Identifying and sharing best
practices in resilience planning and implementation is another essential step.

Additionally, incorporating hazard mitigation design choices into system planning

and development can simultaneously enhance reliability and resilience. By integrating
design principles that prioritize fault tolerance and self-healing capabilities, systems can
become more resilient to disruptions. The application of resilience engineering principles,
which focus on creating systems that can withstand unexpected events and recover swiftly,
further fortifies both reliability and resilience.

39
TR-123 — IEEE PES Technical Roadmap

Finally, a continuous process of risk assessment, ongoing monitoring, and regular

system updates is vital to enhancing both system resilience and reliability. This iterative
approach allows for the identification of vulnerabilities, the implementation of
improvements, and the adaptation of strategies to address evolving threats.

In summary, achieving the goal of systems that exhibit the desired levels of both
reliability and resilience necessitates a comprehensive approach that takes into account a
diverse range of factors and anticipates various challenges. Through the development of
thorough assessment tools, the integration of best practices, and a focus on adaptive design
principles, we can strive to create a future in which critical systems possess the capability
to withstand, adapt to, and recover from even the most challenging disruptions.

3.3 Extreme Weather

3.3.1 Summary
Extreme weather events represent the primary cause of power outages and can
inflict substantial harm upon critical infrastructure, including substations and electric
facilities, leading to widespread disruptions in system operation. Understanding the
challenges and finding solutions to enhance resilience in the face of such extreme weather
conditions is of paramount importance.

3.3.2 Current State

The existing power grid is composed of both new and legacy equipment, some of
which is rapidly aging. Past engineering practices did not always account for the impact
and increasing frequency of extreme weather events. As a result, the existing power system
remains vulnerable to damage, leading to prolonged power outages and increased
restoration costs.

From a system operation perspective, the accuracy of load forecasting models

struggles under extreme weather conditions, primarily due to a lack of historical data on
such events. While current system planning criteria, operational processes, practices, and
reliability standards do acknowledge the impact of extreme weather to some extent, there
is still a pressing need to enhance these frameworks further. System operators are actively
working to refine criteria and practices to address extreme weather conditions, such as
extreme cold or extreme heat [3].

3.3.3 Future State

The envisioned future state involves a fundamental shift in the way power grids are
designed and constructed. The goal is to create a grid infrastructure that is resilient to
natural catastrophic threats, significantly reducing damage and facilitating swift recovery.
The power system of the future should be able to withstand and reduce the magnitude or
duration of disruptive extreme weather events.

3.3.4 Challenges

40
TR-123 — IEEE PES Technical Roadmap

Numerous challenges stand in the way of achieving this vision. Firstly, designing
and constructing a resilient grid often incurs increased costs. Retrofitting existing
infrastructure to enhance resilience presents its own set of financial and logistical
constraints. Additionally, the limited availability of space in urban areas can complicate
efforts to implement resilient designs effectively. However, perhaps the most substantial
challenge in terms of system planning for extreme weather is to strike the right balance
between resilience and cost-effectiveness, realizing that as the economy is being
decarbonized, the expectation is the use of higher levels of electricity, driving the need for
increased levels of both reliability and resilience.

3.3.5 Potential Solutions

Addressing these formidable challenges necessitates a strategic, multifaceted
approach. Initiatives must begin with the identification of potential threats, followed by
thorough vulnerability assessments and impact evaluations. Maintaining vigilant
awareness of evolving threats is crucial for adaptive planning and response.

For example, the development of resilient substation designs is a critical aspect of

this effort. These designs should incorporate advanced technologies and materials that can
withstand extreme weather conditions. For existing infrastructure, robust hardening
approaches are required to fortify against severe weather events effectively. Moreover, a
holistic approach to system operation under extreme weather conditions is essential. For
example, gas-insulated substation (GIS) designs can be used to provide the advantages of
GIS-enabled construction methods. These methods include a smaller substation footprint
(area and perimeter) that can be located inside a hardened building, underground structure
or elevated structure.

In conclusion, the quest for a power infrastructure that can effectively address
extreme weather conditions requires a comprehensive strategy. This strategy should
encompass resilient design, infrastructure hardening, adaptive operational practices, and
ongoing risk assessment. While balancing the imperative for resilience with budgetary
constraints remains a significant challenge, addressing vulnerabilities, assessing impacts,
and embracing evolving threat awareness are key steps in working toward a future where
extreme weather events pose fewer threats to our critical power systems. Through
collaborative efforts and innovative solutions, the power industry can move closer to a
safer, more resilient, and reliable future.

3.4 Power System Hardening

3.4.1 Summary
Power system hardening, a vital aspect of modern infrastructure management,
entails fortifying and adapting power systems to bolster their resistance and rapid recovery
capabilities in the face of adversities. These adversities encompass a wide spectrum,
including natural disasters and deliberate cyber-attacks, with the overarching goal of
maintaining continuous and uninterrupted electrical service.

41
TR-123 — IEEE PES Technical Roadmap

3.4.2 Current State

Presently, our power systems, particularly those reliant on overhead lines, remain
vulnerable to various forms of disruption, stemming from environmental factors and
external threats. This vulnerability is exacerbated by the exposed nature of these networks,
rendering them susceptible to damage from severe weather events, electromagnetic pulses
(EMP), geomagnetic disturbances (GMD), physical stresses, and potential cyber
intrusions. The existing power infrastructure may fall short of providing the robustness
needed to withstand these multifaceted challenges.

Furthermore, from a system operation perspective, infrastructure resilience

measures—a critical component of power grid management—face limitations when
confronted with extreme weather conditions. The dearth of historical data for such
situations impedes accurate forecasts. This is especially true of weather conditions
resulting from climate change. While current system planning criteria and operational
practices do account for extreme weather conditions to some extent, there is still a need for
more comprehensive criteria and practices to address extreme weather events
comprehensively, whether they involve extreme cold or extreme hot weather [4][5].

3.4.3 Future State

The vision for power system hardening revolves around establishing a future
characterized by power infrastructure that is resilient to natural catastrophic threats, thus
reducing damage and facilitating rapid recovery. In this envisioned future, power systems
demonstrate an innate ability to withstand and mitigate the impacts of extreme weather
events and other adversities.

3.4.4 Challenges
The pursuit of this vision comes with challenges:

 Infrastructure Costs: Upgrading or replacing existing infrastructure to meet

hardening standards often entails substantial costs, necessitating careful financial
planning and allocation.
 Technical Complexity: Implementing hardening techniques, integrating advanced
technologies, and ensuring seamless compatibility with existing systems are
complex endeavors demanding specialized expertise.
 Time Investment: Implementing wide-scale changes to existing power distribution
systems is a resource-intensive process that demands time and commitment.
3.4.5 Potential Solutions
To bridge the gap between the status quo and the envisioned future of power system
hardening, a multifaceted approach is necessary:

 Improving Grid Codes: Obtaining buy-in from the government and commissions of
the need to improve the reliability and resilience of the bulk power system, and to
42
TR-123 — IEEE PES Technical Roadmap

agree upon the levels and events to which they will withstand and recover. This
lays the groundwork for investments and adoption of solutions.
 Advanced Grid Technologies: Implementation of advanced grid technologies, such
as smart grids and self-healing grids, is pivotal for enhancing system resilience.
These technologies can swiftly identify and isolate faults, optimize power flow, and
expedite the restoration of service.
 Advanced Materials: The integration of advanced materials into infrastructure
construction holds the potential to significantly increase infrastructure durability
and resilience against environmental and physical stresses.
 Microgrid Integration: Integrating microgrids into the broader power infrastructure,
enabling localized power generation and distribution to enhance overall grid
resilience.
 Strategic Power Line Placement: Careful consideration of the strategic placement
of power lines, including the possibility of burying them, offers a means of
safeguarding against weather-related damage and external threats. However, cost
implications must be evaluated.
 Improved Maintenance and Disaster Response: Establishing proactive measures
such as regular infrastructure inspections, preventive maintenance, and efficient
disaster response strategies can minimize vulnerabilities and enhance overall power
system hardening.
In conclusion, power system hardening stands as a critical endeavor in modern
infrastructure management, aimed at fortifying power systems to withstand and rapidly
recover from a spectrum of adversities, including natural disasters and deliberate cyber-
attacks, to ensure continuous and uninterrupted electrical service. Presently, vulnerabilities
persist in power systems, particularly those reliant on overhead lines, exposed to various
disruptions from environmental factors and external threats. These vulnerabilities
necessitate a transformative vision for power system hardening, envisioning resilient
infrastructure capable of withstanding extreme weather events and adversities. However,
achieving this vision comes with challenges related to infrastructure costs, technical
complexity, and time investment. To bridge this gap, a multifaceted approach is essential,
including the implementation of advanced grid technologies, integration of advanced
materials, adoption of microgrids, strategic power line placement, and improvements in
maintenance and disaster response strategies. This holistic approach is vital for
transitioning from the current vulnerable status quo to a future where power systems exhibit
robust resilience and adaptability in the face of evolving threats and challenges.

3.5 Restoration
3.5.1 Summary
Restoration, as a critical facet of grid resilience and reliability, encompasses a set
of intricate actions aimed at the rapid recovery of power supply following disruptions or
outages. This function is fundamental to ensuring uninterrupted electricity access for both
individuals and industries, emphasizing the significance of efficient and effective
restoration processes within the broader context of power infrastructure management.
43
TR-123 — IEEE PES Technical Roadmap

3.5.2 Current State

At present, power restoration predominantly involves manual intervention and
conventional practices. When an outage occurs, power utility teams must engage in a labor-
intensive, step-by-step procedure to diagnose the cause, locate the fault, and perform the
necessary repairs or reconfigurations. This often requires field technicians to physically
inspect equipment, leading to time-consuming efforts that can result in prolonged service
interruptions for consumers.

The existing approach to restoration relies on the expertise of human operators who
analyze available data and make decisions based on their experience and training. While
this method has served its purpose focusing on health and safety, it has inherent limitations
in terms of speed and precision. As a result, power restoration durations can vary
significantly depending on the complexity of the issue and the availability of skilled
personnel [6][7].

3.5.3 Future State

The future state of power restoration is envisioned as a paradigm shift from manual
and time-consuming processes to automated, data-driven, and rapid recovery mechanisms.
This transformation is driven by cutting-edge technologies and innovative strategies that
have the potential to revolutionize how disruptions are managed within the power grid.

In this visionary landscape, power restoration will be characterized by the seamless

integration of advanced technologies such as self-healing grids, predictive analytics,
artificial intelligence (AI), and machine learning. These technologies will work in harmony
to swiftly detect, diagnose, and rectify issues within the grid, reducing or even eliminating
the need for human intervention in many cases.

3.5.4 Challenges

 Infrastructure Upgrades: Implementing automated restoration technologies may

necessitate significant upgrades to the existing power infrastructure. This includes
the installation of sensors, communication networks, and smart devices throughout
the grid. Such upgrades can be technically complex and financially demanding.
 Cost Considerations: The adoption of advanced technologies comes with associated
costs, including the initial investment in hardware and software, ongoing
maintenance expenses, and the need for skilled personnel to manage and operate
these systems.
 Workforce Training: The successful deployment of automated restoration solutions
requires a workforce that is well-versed in the operation and maintenance of these
technologies. Adequate training and skill development programs are essential to
ensure that staff can harness the full potential of these innovations.
 Data Security: With the increased reliance on data and communication networks,
cybersecurity becomes a paramount concern. Safeguarding the grid against cyber
44
TR-123 — IEEE PES Technical Roadmap

threats and ensuring the integrity of data is a critical component of the envisioned
future of power restoration.
 Inrush current and cold load pickup (Uncertainty on the lad estimation)

3.5.5 Potential Solutions

To bridge the gap between the current state and the visionary future of power
restoration, a multifaceted strategic approach is required:

 Self-Healing Grid Technologies: The development and implementation of self-

healing grid technologies are central to the automated restoration vision. These
technologies encompass a range of solutions, including real-time fault detection,
isolation, and service restoration. By leveraging sensors, data analytics, and
automated switching devices, the grid can respond to disruptions with
unprecedented speed and accuracy.
 Predictive Analytics and AI: Harnessing the power of predictive analytics and AI-
driven algorithms enables utilities to anticipate potential issues before they escalate
into outages. By continuously monitoring grid data, AI systems can identify
anomalies and recommend preventive actions.
 Smart Sensors and IoT: The cyber-secure deployment of smart sensors and the
Internet of Things (IoT) devices throughout the grid provides real-time data on
equipment health and grid conditions. This wealth of data enables proactive
maintenance and enhances the accuracy of fault detection.
 Communication Infrastructure: Establishing robust, secure, and resilient
communication networks is essential for the success of automated restoration.
These networks facilitate real-time data exchange between grid components and
central control centers.
 Cybersecurity Measures: Prioritizing cybersecurity measures is a must. Robust
cybersecurity protocols, encryption techniques, cyber-informed engineering, and
continuous monitoring must be in place to protect against cyber threats.
 Investment in Workforce Development: Ensuring that the workforce is well-
prepared to operate and maintain these advanced systems is crucial. Training
programs, certifications, and ongoing skill development initiatives are essential
components of this effort.
In conclusion, the transformation of power restoration from a manual and time-
consuming process into an automated, safe, and rapid procedure represents a pivotal
advancement in grid resilience and reliability. While the journey is fraught with challenges,
the potential benefits are substantial. Reduced downtime, enhanced service quality, and
improved grid management are among the rewards with the full spectrum of technological
innovations and strategic investments in the power restoration domain. By combining
cutting-edge technologies with a well-trained workforce and a commitment to
cybersecurity, the power sector can move closer to a future where disruptions are

45
TR-123 — IEEE PES Technical Roadmap

minimized, and the reliability of power supply is maximized, benefiting both society and
the economy.

3.6 Physical Security and Human-caused Attacks

3.6.1 Summary
In the realm of modern electric infrastructure, the threat of physical attacks and
criminal activities targeting substations and other vital facilities looms large. These events
have the potential to wreak havoc, causing extensive damage and posing severe challenges
to grid reliability and resilience. Therefore, understanding, mitigating, and responding to
these threats are critical in ensuring the uninterrupted flow of electricity to homes,
businesses, and industries.

3.6.2 Current State

Presently, many utilities and grid operators are diligently reassessing their
infrastructure design and protective measures to bolster their ability to withstand physical
attacks and criminal actions. The existing landscape incorporates a multitude of security
measures, both physical and procedural, aimed at minimizing the potential impact of such
events. While these measures represent a commendable effort to safeguard the grid, there
remains ample room for improvement to further fortify the defenses and response
capabilities [8][9].

3.6.3 Future State

The envisioned future state paints an inspiring picture in which electric
infrastructure is transformed into a bastion of resilience. These facilities stand strong
against human-caused physical threats, ensuring the reliability of the power grid even in
the face of malicious attacks. However, this vision goes beyond mere defense; it
encompasses swift recovery strategies, minimizing the disruption caused by deliberate acts
of destruction.

In this future, the grid's reliability remains unwavering, even when subjected to
coordinated and determined efforts to disrupt its operation. While challenges abound, the
power sector is resolute in its commitment to realizing this vision and further strengthening
the security of critical infrastructure.

3.6.4 Challenges
Realizing this vision is not without its share of challenges, each of which demands
careful consideration and innovative solutions:

 Cost Implications: Implementing resilient designs and security enhancements can

entail substantial costs. The challenge lies in judiciously allocating resources to
balance the financial implications of security measures against the broader goals of
grid security and reliability.

46
TR-123 — IEEE PES Technical Roadmap

 Infrastructure Retrofitting: Retrofitting existing facilities to meet stringent security

standards is a complex undertaking. This challenge involves the modernization and
fortification of aging infrastructure to align with modern security requirements,
often requiring meticulous planning and execution.
 Collaboration with Security Agencies: Establishing and maintaining strong
partnerships with security agencies is indispensable for deterring and responding to
threats effectively. Successful coordination between the power sector and security
agencies requires sustained commitment, communication, and training.
3.6.5 Potential Solutions
To bridge the gap between the present and the envisioned future, a multifaceted
approach is essential:

 Resilient Infrastructure Design: Embracing innovative engineering solutions and

materials to construct electric facilities that are inherently resilient to physical
threats. This encompasses the development of reinforced structures, robust access
controls, and the integration of advanced security systems. It also encompasses the
use of innovative substation designs, such as gas-insulated substation (GIS) and
indoor substation designs. These designs can be used to provide the advantages of
a smaller substation footprint (area and perimeter) that can be located inside a
hardened building, underground structure or elevated structure.
 Expanded Security Perimeters: Extending the security perimeters around critical
infrastructure to deter unauthorized access and enhance early threat detection. This
may involve the installation of intrusion detection systems, surveillance cameras,
and access control mechanisms.
 Ongoing Threat Awareness: Continuously monitoring and assessing emerging risks
and threats to the power grid. Staying ahead of evolving tactics employed by
criminals and adversaries is essential for proactive threat mitigation.
 Continuous Engagement with Security Agencies: Fostering strong relationships
with security agencies, sharing intelligence, and collaborating on security drills and
exercises. This partnership ensures a rapid and coordinated response to threats
when they occur.
 Public-Private Cooperation: Engaging in dialogue and collaboration between
public and private stakeholders to collectively address security challenges. This
includes sharing best practices, threat intelligence, and joint training initiatives.
 Research and Development: Investing in research and development to identify
cutting-edge technologies and methods for enhancing security. This includes
exploring innovations in surveillance, access control, and threat detection systems.
In conclusion, the future of protecting critical electric infrastructure from physical
attacks and criminal activities requires a comprehensive, collaborative, and forward-
thinking approach. By investing in resilient designs, expanding security measures, and
maintaining robust partnerships with security agencies, the power sector can move closer
to a future where threats are deterred, impacts are minimized, and the reliability and
47
TR-123 — IEEE PES Technical Roadmap

resilience of the grid remain unshakable. In doing so, the industry plays a vital role in
ensuring the stability and security of the power grid.

3.7 Condition Assessment

3.7.1 Summary
In power grid operation, the pivotal element for sustainable success revolves around
the establishment of a reliable condition assessment strategy. This strategy serves as the
basis upon which the seamless functioning of the power grid is built. It is underpinned by
advanced diagnostic tools, exemplified by the deployment of cutting-edge fiber optic
sensors for transformer hot-spot monitoring. These transformative diagnostics not only
function as a sentinel, offering a crucial early warning system, but they also hold the
promise of averting extended outages by proactively identifying vulnerabilities within
power assets.

3.7.2 Current State

At the present juncture, utilities demonstrate a vigilant approach to grid
management. They diligently employ state-of-the-art monitoring devices to keep an eye on
various critical components, including power transformers' cooling systems and station
batteries. However, within this state of watchfulness lies an implicit warning – the potential
consequences of failing to address emerging problems promptly. For example, a
catastrophic transformer failure underscores the need for a more proactive approach to
asset health management.

3.7.3 Future State

The future unfolds with a compelling vision in which power utilities harness
advanced technologies to pre-emptively unveil the health status of their critical assets. This
forward-looking paradigm shift aspires to elevate grid resilience and reliability to
unprecedented echelons. By continuously monitoring and carefully assessing the condition
of these vital assets, the power sector seeks to anticipate vulnerabilities and intervene
before they metamorphose into disruptive events.

In this envisioned future state, the power grid operates as a dynamic, self-aware
entity. It is capable of proactively mitigating risks, fortifying its infrastructure against
potential failures, and defying the traditional paradigms of reactive grid management. The
convergence of technology and foresight culminates in an ecosystem where disruptions are
minimized.

3.7.4 Challenges
Navigating the path toward this visionary future presents challenges that demand
innovative solutions and strategic thinking:

 Cost Considerations: The deployment of advanced technologies for comprehensive

asset monitoring can entail substantial upfront costs. Balancing the financial
48
TR-123 — IEEE PES Technical Roadmap

implications with the long-term benefits necessitates careful cost-benefit analysis

and resource allocation.
 Operational Maintenance: Ensuring the operational effectiveness of advanced
monitoring technologies over time is crucial. Regular inspections, calibration, and
updates are essential to guarantee the continued reliability of the asset health
assessment system.
3.7.5 Potential Solutions
Realizing this vision hinges on the implementation of a robust asset management
program, underpinned by the following key strategies:

 Proactive Monitoring: Establishing a continuous monitoring framework that

seamlessly integrates advanced diagnostics and real-time data analytics to assess
asset health. This proactive approach empowers grid operators with the invaluable
capability of early issue detection and intervention.
 Predictive Maintenance: Leveraging the prowess of predictive maintenance
algorithms to forecast potential asset failures. These algorithms derive insights
from historical data and real-time performance metrics, enabling operations and
maintenance personnel to proactively address issues before they escalate.
 Data-Driven Decision-Making: Harnessing the transformative power of data
analytics to extract actionable insights from the wealth of asset health information.
Informed decision-making guides resource allocation and risk mitigation efforts,
ensuring an efficient and resilient grid.
 Investment in Training: Providing comprehensive training for personnel
responsible for operating and maintaining the advanced monitoring systems. A
well-trained staff is instrumental in maximizing the effectiveness of these
technologies and in maintaining the integrity of the grid.
 Cost-Benefit Analysis: Conducting rigorous and ongoing cost-benefit analyses to
assess the economic viability of deploying advanced technologies. This ensures that
investments align harmoniously with the overarching objectives of long-term grid
resilience and reliability.
In conclusion, achieving a power grid characterized by heightened resilience and
reliability relies on the adoption of advanced diagnostics and proactive asset health
management. By integrating these innovative technologies and implementing a robust asset
management program, utilities can effectively identify vulnerabilities in advance. This
approach paves the way for a future where power outages are rare occurrences rather than
the norm.

3.8 Reliable Sourcing of Products and Engineering Services to Promote

Nation Security
3.8.1 Summary

49
TR-123 — IEEE PES Technical Roadmap

The effective procurement of equipment, commodities, and engineering services is

foundational in constructing resilient and secure substations and electrical facilities. An
efficient and dependable supply chain plays a pivotal role in enhancing efficiency and
curtailing costs, but it is not exempt from a myriad of challenges. The intricate interplay of
global geopolitics, complex logistics, economic considerations, and rapid technological
evolution introduces both opportunities and vulnerabilities.

3.8.2 Current State

In the present landscape, many electric system owners find themselves navigating
the labyrinthine pathways of procurement decisions. The prevailing criterion often leans
toward selecting the lowest-cost providers for vital infrastructure equipment and materials.
However, this seemingly cost-effective approach brings forth its own set of intricacies. The
supply chain emerges as a precarious endeavor, predominantly centered in a few
geographic regions. This concentration exacerbates the susceptibility of the supply chain
to interruptions, consequently placing critical infrastructure at potential risk. This status
quo underscores a crucial dilemma: the appeal of cost-effectiveness versus the danger of
excessive dependence. While cost-driven decisions may appear financially prudent, they
frequently overlook the intricate web of interconnections within the supply chain and the
global dynamics influencing its resilience.

3.8.3 Future State

The vision for the future harmonizes the pursuit of flexibility with an unwavering
commitment to reliability in sourcing and supply chains. The ideal state envisions a
dynamic ecosystem where continual risk assessment serves as the guiding principle for
evaluating the inherent vulnerabilities within each supply chain. In this paradigm,
alternative supply chains are prepared and poised for deployment, standing as resilient
safeguards against disruptions.

At the core of this vision lies the pursuit of a multi-dimensional supply chain that
adapts and evolves in response to a dynamic world. It envisages a landscape where security,
efficiency, and resilience coexist in harmony. A constellation of supply sources, both local
and global, ensures that vital equipment and materials flow seamlessly, resiliently
surmounting challenges as they arise.

3.8.4 Challenges
Embarking on the journey toward this visionary future necessitates confronting
challenges that demand innovative solutions:

 Raw Material Concentration: The concentration of raw materials in specific

geographic regions poses a significant challenge. Overcoming this hurdle requires
strategic diversification in sourcing and proactive risk management.
 Political Risks: Governments and corporate interests occasionally intertwine with
the supply chain, introducing political complexities. Navigating these waters
demands diplomatic finesse and a nuanced understanding of global dynamics.
50
TR-123 — IEEE PES Technical Roadmap

 Resistance to Change: Encouraging corporations to restructure their supply chains

can encounter resistance. Convincing stakeholders of the necessity and benefits of
diversification is an imperative step.
 Potential policy and regulatory changes like the proposed EU Digital Product
Passport and potential requirements aimed at implementing a circular economy,
which is infrastructure materials to be returned to the feedstock pool at the end-of-
use of said infrastructure.

3.8.5 Potential Solutions

To transform this vision into reality, a repertoire of strategic solutions emerges:

 Each organization should perform a risk assessment of their system now and into
the future, and understand where the risks are based on available supply chains and
mitigate those risks when necessary.
 Reliable Sourcing Regions: Prioritize sourcing from regions with a proven track
record of reliability. This approach minimizes political risks while ensuring a stable
supply of essential materials.
 Local Supplier Networks: Foster local supplier networks, enhancing resilience by
reducing dependence on distant sources. Supporting local industries bolsters
community engagement and reduces vulnerabilities.
 Diversified Sourcing: Broaden the sourcing scope to encompass a spectrum of
regions. A diversified supply chain is inherently resilient and nimble in the face of
disruptions.
 Government Support: Advocate for government support in the development of
alternative supply options. Collaborate with policymakers to create an environment
conducive to diversified supply chains.
 Secondary Sources and Safety Stock: Establish secondary sources and maintain
safety stock reserves. These buffers act as a bulwark against disruptions, enabling
swift response to supply chain challenges.
 Digital Archiving: Embrace digital archiving of designs, data, and templates. This
digital repository ensures that critical information remains accessible, even in the
face of unforeseen disruptions.
In conclusion, the efficient procurement of equipment, commodities, and
engineering services is fundamental to the construction of resilient electrical facilities and
substations. A reliable supply chain is instrumental in boosting efficiency and cost
reduction, yet it confronts a complex landscape filled with geopolitical, logistical,
economic, and technological intricacies. Presently, electric system owners often prioritize
cost-driven decisions, favoring low-cost providers for essential infrastructure materials.
However, this approach exposes the supply chain's vulnerability to geopolitical disruptions
in concentrated regions. The vision for the future revolves around a flexible and reliable
sourcing and supply chain ecosystem, emphasizing continuous risk assessment and
preparedness with alternative supply chains. This multi-dimensional supply chain vision

51
TR-123 — IEEE PES Technical Roadmap

envisions a harmonious coexistence of security, efficiency, and resilience, involving local

and global supply sources. However, realizing this vision entails addressing challenges
such as raw material concentration, political risks, and resistance to supply chain
restructuring. Solutions include prioritizing reliable sourcing regions, fostering local
supplier networks, diversifying sourcing, seeking government support, maintaining
secondary sources and safety stock, and embracing digital archiving for resilience against
disruptions.

3.9 Prognostics and Health Management (PHM)

3.9.1 Summary
Prognostics and Health Management (PHM) represents a pivotal capability in the
quest to elevate system reliability. By harnessing the power of sensors, machine learning
predictive analytics, and automated decision-making for condition-based maintenance,
PHM holds the promise of a future where breakdowns are foreseen and averted. However,
many challenges exist from limited visibility to a lack of uniformity and inconsistent
results.

3.9.2 Current State

PHM constitutes a multifaceted domain characterized by a plethora of methods,
techniques, and tools. Despite the proliferation of these technologies, their practical
implementation faces a large array of impediments. The absence of standardized protocols
engenders inconsistency, resulting in the cryptic nature of PHM with variable outcomes
[10].

3.9.3 Future State

The envisaged future for PHM revolves around standardization and reliability. It
envisions a harmonized ecosystem where PHM techniques are consistently and uniformly
applied, yielding dependable results across diverse applications.

3.9.4 Challenges
Embarking on the path towards this visionary future presents substantial
challenges:

 Guidance and Motivation: The demand for comprehensive and motivating

guidance assumes paramount significance. PHM practitioners need compelling
incentives to allocate resources towards standardizing techniques and achieving
dependable Remaining Useful Life (RUL) predictions.
 Uniformity and Consistency: Overcoming the absence of standardized practices
and the issue of inconsistency constitutes a critical obstacle. Standardization
initiatives must address these disparities and provide a clearly defined roadmap.

52
TR-123 — IEEE PES Technical Roadmap

 Business Case Development: Practitioners must be endowed with essential insights

to make judicious decisions concerning the value of PHM in augmenting system
reliability and performance.
3.9.5 Potential Solutions
Strategies to materialize this vision into a tangible reality emerge as guiding
principles:

 Normative Requirements: The establishment of normative requirements for PHM

systems serves as a foundational step. These requirements lay the groundwork for
uniformity and reliability, creating a shared basis for all practitioners.
 Standard Framework: The provision of a rigorously structured framework is of
critical importance. This framework aids practitioners in discerningly selecting the
most suitable approaches, methods, and strategies for PHM implementation,
functioning as a compass toward reliability.
 Information Empowerment: Equipping practitioners with essential information
emerges as a non-negotiable imperative. By furnishing the requisite insights to
construct compelling business cases, the PHM community can galvanize
augmented investments in reliability-enhancing strategies.
 Collaborative Ecosystem: Fostering a collaborative ecosystem where PHM
practitioners, industry stakeholders, and researchers seamlessly cooperate
constitutes an essential pillar. This interconnected network stimulates the exchange
of knowledge, catalyzes innovation, and fosters a unified approach toward
enhancing reliability.
In conclusion, PHM holds immense potential for elevating system reliability
through sensor technology, machine learning, and automated maintenance. However, it
faces challenges like limited visibility and a lack of standardization. The envisioned future
revolves around standardization and reliability, but achieving this goal requires addressing
issues like guidance, standardization, and business case development.

To make this vision a reality, we need normative requirements, a structured

framework, information empowerment, and a collaborative ecosystem. Despite the
challenges, by implementing these solutions, the PHM community can create a future
where systems are not just dependable but predictable, contributing to a more reliable
world.

3.10 Fault Tolerance

3.10.1 Summary
Fault tolerance, a critical design capability, is the bedrock of system reliability and
resilience. It empowers systems and products to recover gracefully from failure events,
using redundancy, error detection, correction, and other ingenious design elements. This
ability ensures that systems continue to fulfill their intended functions, even when
confronted with independent faults, effectively thwarting the risk of system failure.
53
TR-123 — IEEE PES Technical Roadmap

3.10.2 Current State

In the area of fault-tolerant architecture and design, the present landscape is one
where failures, while not uncommon, are merely transient disruptions. The mission at hand
can still be ultimately accomplished. This feat is achieved through the incorporation of
redundant hardware and software elements, woven into the intricate fabric of a serial-
parallel system configuration. Furthermore, the judicious use of prognostics and health
monitoring bolsters the system's resilience by anticipating impending failures and
orchestrating timely interventions.

3.10.3 Future State

The future envisioned for fault tolerance is one characterized by uniformity and
reliability. It paints a landscape where fault-tolerant designs are not just sporadic successes
but standardized practices, with consistency and dependability. At its core, this vision seeks
to weave fault tolerance into the system design, making it an integral part of every
endeavor. It aspires to see robust redundancy mechanisms standing guard and predictive
analytics serving as vigilant monitors, detecting faults before they can sow chaos.

3.10.4 Challenges
Embarking on this journey towards fault tolerance's uniformity and reliability is not
without hurdles:

 Uniformity Dilemma: The absence of uniformity in applying fault-tolerant

approaches raises questions. The efficacy of fault tolerance should not be a game
of chance, but rather a standard to be upheld.
 Standards Conundrum: Inconsistent use of standards creates ambiguity. A
harmonized set of standards is imperative to navigate the complex domain of fault
tolerance.
 Reliability Variation: Variation in demonstrated results poses a perplexing paradox.
For fault tolerance to be truly dependable, results should be consistently reliable.
3.10.5 Potential Solutions
Charting the course toward this fault-tolerant future involves strategic solutions:

 Normative Requirements: Establishing normative requirements for fault-tolerant

systems is the first step. These requirements serve as the foundation upon which
uniformity and reliability are built.
 Standard Framework: Providing a standard framework offers clarity. This
framework assists practitioners in selecting the most fitting approaches, methods,
and strategies, eliminating ambiguity.
 Resource Investment: Encouraging investment in the standardization of fault-
tolerant techniques and the development of RUL outputs is pivotal. Such

54
TR-123 — IEEE PES Technical Roadmap

investment catalyzes the adoption of standardized practices across diverse

application domains.
 Continuous Evolution: Acknowledging that the fault tolerance landscape is ever-
evolving, practitioners must remain agile. Continuous adaptation and refinement
are the keys to staying ahead in this dynamic field.
 Collaborative Knowledge Sharing: Fostering a culture of knowledge sharing and
collaboration within the fault tolerance community is essential. By pooling insights,
experiences, and best practices, practitioners can collectively drive the field
forward.
In conclusion, fault tolerance is indispensable for system reliability and resilience,
offering the ability to recover from failures effectively. The envisioned future seeks to
standardize fault-tolerant practices, yet challenges such as uniformity, standards, and
reliability persist. To address these hurdles, the establishment of normative requirements,
provision of a clear framework, encouragement of resource investment in standardization
and RUL development, continuous adaptation, and fostering knowledge sharing within the
fault tolerance community are essential strategies. These initiatives collectively pave the
way for a future where fault tolerance becomes a standardized, dependable practice,
enhancing system reliability and resilience across diverse domains.

3.11 Resource Adequacy and Fuel/Energy Assurance

3.11.1 Summary
Resource Adequacy and Fuel/Energy Assurance represent two critical pillars in
energy management and security. Resource Adequacy pertains to the availability of
sufficient and reliable energy resources to meet current and future demands, ensuring grid
stability and resilience. Fuel/Energy Assurance, on the other hand, focuses on guaranteeing
the uninterrupted supply of essential fuels and energy sources, safeguarding against
disruptions caused by various factors, including geopolitical tensions, extreme weather,
and natural disasters.

3.11.2 Current State

In the status quo, the energy domain is marked by a complex interplay of factors
that challenge both Resource Adequacy and Fuel/Energy Assurance. Resource Adequacy
faces uncertainties due to the increasing integration of variable energy sources, making
grid management more intricate. Additionally, environmental concerns and transitioning
to cleaner energy sources pose challenges to maintaining resource adequacy while reducing
emissions [11].

Fuel/Energy Assurance confronts geopolitical tensions that can disrupt the supply
chain of essential fuels, posing risks to energy security. Vulnerabilities in critical
infrastructure, aging power plants, and transmission networks further exacerbate
challenges in ensuring uninterrupted energy supply. The emergence of cyber threats adds
another layer of complexity, as the energy sector becomes a prime target for malicious
actors.
55
TR-123 — IEEE PES Technical Roadmap

3.11.3 Future State

The vision for Resource Adequacy and Fuel/Energy Assurance revolves around
establishing a future where energy resources are abundant, reliable, and resilient. In this
envisioned future, Resource Adequacy entails a well-balanced mix of energy sources,
including renewables, fossil fuels, nuclear and advanced technologies, all integrated into a
smart and adaptive grid. Energy transition is seamless, emissions are minimized, and the
grid can withstand fluctuations in supply and demand.

For Fuel/Energy Assurance, the future envisions a world where energy supply
chains are diversified and resilient, capable of withstanding geopolitical tensions and
natural disasters. Energy access is universal, and critical infrastructure is fortified against
cyber threats.

3.11.4 Challenges
Embarking on the journey toward this visionary future presents multifaceted
challenges:

 Resource Variability: Balancing variable energy resources, such as wind and solar,
with the need for consistent supply creates complexities in grid management.
 Environmental Concerns: Transitioning to low/no carbon energy sources while
maintaining resource adequacy requires navigating environmental regulations and
mitigating emissions.
 Infrastructure Aging: Aging power plants and grid infrastructure pose reliability
and resilience challenges, demanding substantial upgrades.
 Changes in weather patterns: Climate change is driving the frequency and severity
of weather events. Widespread, long-duration events can exacerbate the availability
of the supply chain of fuel, which must be sustained to maintain energy sufficiency.
 Geopolitical Tensions: Geopolitical conflicts can disrupt the supply chain of critical
fuels, impacting energy security.
 Cybersecurity Threats: The energy sector faces an increasing number of cyber
threats, necessitating robust cybersecurity measures.
 Energy Access Disparities: Providing energy access to underserved communities
remains a global challenge.
 Policy Complexities: Harmonizing energy policies at local and regional levels is
essential for consistency in achieving resource adequacy and energy assurance
goals.
3.11.5 Potential Solutions
Addressing the challenges and realizing the vision for Resource Adequacy and
Fuel/Energy Assurance demands strategic solutions and collaborative efforts:

56
TR-123 — IEEE PES Technical Roadmap

 Review design criteria: Namely the measures to date have been grounded from
capacity calculations (e.g., one event in ten years). New design criteria that
measure the outputs of the capacity, energy and essential reliability services
(ramping, voltage, frequency, etc.) must be developed to ensure that the system
being developed can serve the energy needs of the consumers at all times.
 Resource Diversity and Integration: Promote resource diversity, including
renewables, fossil fuels (including dual fuel) during energy transition, nuclear, and
emerging technologies. Implement advanced grid management and integration
solutions to balance variable resources effectively.
 Increased Transmission: Ensure that transfer capability enables the transfer of areas
rich with energy to those that are depleted.
 Maintain Balancing Resources: to ensure that as ramps occur with the uncertainty
of energy constraining variable resources, the system can address the potential
shortfalls.
 Environmental Stewardship: Align energy policies with environmental goals
through incentives for clean energy adoption, realization of greater energy
efficiencies, and the development of sustainable technologies.. Encourage research
and development in carbon capture and utilization to minimize emissions.
 Sustainable Procurement: Align procurement of equipment/infrastructure with
environmental goals for minimum emissions in production and operation as well as
re-use of materials at the end-of-use, which is a minimal waste footprint.
 Energy Transition Planning: Develop comprehensive energy transition plans that
ensure a seamless shift from conventional to cleaner energy sources while
maintaining resource adequacy. Invest in energy storage and grid infrastructure to
support the transition.
 Infrastructure Modernization: Allocate resources for infrastructure modernization,
focusing on upgrading aging power plants, transmission lines, and distribution
networks. Implement advanced monitoring and control systems for better resource
utilization.
 Energy Access Initiatives: Implement energy access initiatives in underserved
areas, incorporating microgrids and decentralized energy sources. Collaborate with
local communities and leverage public-private partnerships to enhance energy
access and assurance.
 Cybersecurity Resilience: Strengthen cybersecurity measures by investing in
advanced threat detection, incident response, and resilience strategies. Foster
collaboration between government agencies, energy companies, and cybersecurity
experts.
 Policy Alignment: Harmonize energy policies and regulations at regional and
international levels to ensure consistency and coordination in achieving resource
adequacy and energy assurance goals. Promote cross-border energy collaborations.
 Research and Innovation: Encourage research and innovation in energy
technologies, grid management, energy forecasting and grid
equipment/infrastructure, including realization of greater energy efficiencies.

57
TR-123 — IEEE PES Technical Roadmap

Foster partnerships between academia, industry, and government agencies to drive

technological advancements.
 Community Engagement: Engage communities in energy planning and resilience-
building efforts. Raise awareness about the importance of resource adequacy and
energy assurance and involve stakeholders in decision-making processes.
In conclusion, the pursuit of a future characterized by abundant, reliable, and
resilient energy resources and systems necessitates a multifaceted approach that addresses
resource variability, environmental concerns, energy transition challenges, infrastructure
aging, energy access disparities, cybersecurity threats, and policy complexities.

3.12 Resilience Assessment Under Climate-induced Extreme Weather

Conditions
3.12.1 Summary
The increasing frequency and severity of extreme weather events that have wide-
spread impacts on resources worldwide, attributed to climate change, has elevated the
importance of accurately assessing their impact on power systems. Resilience assessment
under climate-induced extreme weather conditions is vital for the reliable operation and
future planning of renewable-dominated power systems.

3.12.2 Current State

While several resilience assessment approaches exist, there is a critical gap in
explicitly and systematically evaluating the effects of climate-induced extreme weather on
various aspects of power systems, including generation and transmission equipment, as
well as electricity load. This lack of assessment can mask vulnerabilities within the system,
potentially leading to misguided resilience enhancement efforts and policy decisions.

3.12.3 Future State

The future of power system resilience assessment envisions a comprehensive and
integrated approach that holistically evaluates the impacts of climate-induced extreme
weather events on all aspects of the power system. This approach transcends traditional
boundaries, encompassing generation, transmission, distribution, and consumption while
considering climatic factors and their interdependencies.

3.12.4 Challenges
Significant challenges arise from the complexities of climate prediction, modeling
intricate interconnections among power systems, climate patterns, hydrology,
transportation, telecommunications, and engaging a diverse range of stakeholders.
Uncertainties in future climate patterns add further complexity to resilience assessment.

3.12.5 Potential Solutions

Addressing these challenges necessitates multifaceted solutions:
58
TR-123 — IEEE PES Technical Roadmap

 Climate Change Models: Incorporate advanced climate change models that provide
more accurate and region-specific predictions. These models can serve as a
foundation for assessing the potential impacts of climate-induced extreme weather
on power systems.
 Advanced Modeling Tools: Develop sophisticated modeling tools and methods that
integrate multiple sectors, including energy, water resources, transportation, and
telecommunications. These tools should enable a comprehensive understanding of
interdependencies and vulnerabilities.
 Stakeholder Collaboration: Foster transparent communication channels among
stakeholders, including government agencies, utilities, research institutions, and
communities. Facilitate dialogue, information sharing, and collaboration to ensure
a collective approach to resilience assessment and planning.
 Data Integration: Enhance data integration capabilities to collect and analyze data
from various sources, including weather stations, power grids, and infrastructure
sensors. This data-driven approach enables real-time monitoring and response to
extreme weather events.
 Scenario Planning: Develop scenario-based resilience planning that considers a
range of potential climate-induced standardized extreme weather scenario libraries.
This approach enables power systems to prepare for various contingencies.
 Policy Frameworks: Establish policy frameworks that incentivize resilience
investments and mandate rigorous assessments for critical infrastructure. These
policies can drive proactive resilience enhancement efforts.
 Public Awareness: Raise public awareness about the importance of resilience
assessment and preparedness for extreme weather events. Educate communities on
safety measures and the role they play in enhancing power system resilience.
In conclusion, the future of power system resilience assessment under climate-
induced extreme weather conditions includes a holistic, multidisciplinary, and
collaborative approach. By integrating advanced models, tools, data, and stakeholder
engagement, we can work towards a future where power systems are resilient and
adaptable, ensuring uninterrupted energy supply in the face of climate challenges.

3.13 References
[1] B. Chiu et al., Resilience Framework, Methods, and Metrics for the Electricity
Sector, IEEE PES Technical Report (PES -TR83), October 2020.
[2] The definition and quantification of resilience, IEEE Power and Energy Society,
Technical Report PES-TR65, Apr. 2018.
[3] Wang, Y., Chen, C., Wang, J., & Baldick, R. (2015). Research on resilience of
power systems under natural disasters—A review. IEEE Transactions on power
systems, 31(2), 1604-1613.
[4] Panteli, M., Trakas, D. N., Mancarella, P., & Hatziargyriou, N. D. (2017). Power
systems resilience assessment: Hardening and smart operational enhancement
strategies. Proceedings of the IEEE, 105(7), 1202-1213.

59
TR-123 — IEEE PES Technical Roadmap

[5] Bagheri, A., Zhao, C., Qiu, F., & Wang, J. (2018). Resilient transmission hardening
planning in a high renewable penetration era. IEEE Transactions on Power
Systems, 34(2), 873-882.
[6] Liu, Y., Fan, R., & Terzija, V. (2016). Power system restoration: a literature review
from 2006 to 2016. Journal of Modern Power Systems and Clean Energy, 4(3),
332-341.
[7] Chen, C., Wang, J., & Ton, D. (2017). Modernizing distribution system restoration
to achieve grid resiliency against extreme weather events: An integrated
solution. Proceedings of the IEEE, 105(7), 1267-1288.
[8] Humayed, A., Lin, J., Li, F., & Luo, B. (2017). Cyber-physical systems security—
A survey. IEEE Internet of Things Journal, 4(6), 1802-1831.
[9] Sridhar, S., Hahn, A., & Govindarasu, M. (2011). Cyber–physical system security
for the electric power grid. Proceedings of the IEEE, 100(1), 210-224.
[10] Zhang, L., Lin, J., Liu, B., Zhang, Z., Yan, X., & Wei, M. (2019). A review on deep
learning applications in prognostics and health management. IEEE Access, 7,
162415-162438.
[11] NERC, Ensuring Energy Adequacy with Energy Constrained Resources (2020),
Available at
https://www.nerc.com/comm/RSTC/ERATF/ERATF%20Energy%20Adequacy%
20White%20Paper.pdf
[12] A. Stankovic et al., Methods for Analysis and Quantification of Power System
Resilience, IEEE PES Technical Report (PES - TR 108), May 2023.

4. Grid Edge Technologies

4.1 Executive Summary
As increasing distributed energy resources (such as solar) and energy storage (such
as battery) are introduced to the system, and controllability at the consumer side increases
significantly, “grid edge” technologies become important to the reliable and efficient
operation of the power grids. We recognize the term “grid edge” is very broad, and the
definition will keep evolving with the grid transformation; for the purpose of this roadmap,
we focus on technologies that are at or close to the end-use consumers. It includes hardware
technologies (such as solar panels and advanced metering infrastructure), software (such
as home energy management system, grid optimization, and data-driven algorithms), and
business models (such as market participation and aggregator).

Because of the broad scope, there are various technical, economic, and regulatory
challenges and opportunities. Nevertheless, some features and challenges are shared among
many grid edge technologies. First, with the location of the resources at the grid edge, the
direction of the power flow may change, posing challenges to system protection, operation,
power quality, local and grid stability, and more. Second, the number of controllable
devices increases significantly, requiring novel coordination strategies, cyber-informed
engineering, and new time and frequency domain algorithms. Third, on the market side,
the unique properties of the grid edge resources (such as small size, large number, high
volatility, limited dispatchability) may bring the need for new market structures and
60
TR-123 — IEEE PES Technical Roadmap

business models. Fourth, cybersecurity and data management need to be handled carefully
with more control points, communication channels, considering the various available data.
All these challenges involve many stakeholders, including customers, transmission and
distribution system operators, utilities, regulators, researchers, developers, and
manufacturers. There may even be conflicting interests in some situations. Finding a
universal solution may not be practical, but one key to solving the challenges is to bring all
parties to the table and work together.

For better organization, we group the topics into the following categories: (1) the
first category, “flexibility at the grid edge,” covers technologies at the device and plant
level; (2) the second category, “coordination and control of grid edge assets,” covers
technologies at the architecture and system level; and (3) the last category covers micro-
grid technologies. In the following sections, we will discuss the current state, trends,
challenges, and potential solutions for each topic.

4.2 Flexibility at the Grid Edge (device/plant level)

4.2.1 Grid-Integrated Smart Buildings, Homes, and Communities
Current State

Grid-integrated smart buildings, also known as grid-interactive efficient buildings

(GEBs), are structures that leverage advanced technologies and automation to intelligently
interact with the grid while optimizing energy efficiency and occupant comforts. Equipped
with advanced monitoring, sensing, communication, and controls, GEBs can provide great
demand flexibility that can be used for balancing electricity supply and demand in the
power grid.
GEBs can be commercial buildings, residential homes, and communities. Many
large commercial buildings, such as office complexes and retail spaces, have Building
Automation Systems (BAS) that incorporate advanced automation, real-time monitoring,
and control systems to manage energy use and support grid interactions. Residential homes
incorporating smart thermostats, smart home appliances, rooftop solar panels, and energy
storage systems, enable homeowners to monitor and control their energy use, and generate
their own electricity, potentially selling surplus energy back to the grid. Smart and
connected communities often involve a combination of commercial buildings, residential
homes, and shared energy resources. By coordinating and optimizing energy use across
multiple buildings, grid-interactive communities aim to improve overall energy efficiency,
demand response capabilities, and renewable energy integration.
The concept of grid-integrated smart buildings, which involves both electricity
consumers and utility suppliers, has evolved over the past several decades. In the 1990s,
customers became more engaged through demand side management programs to reduce
the peak demand when the grid was stressed. In the early 2000s, smart grid initiatives
gained traction worldwide, enabling the deployment of Advanced Metering Infrastructure
(AMI) that facilitates two-way communication between buildings and the grid. Together
61
TR-123 — IEEE PES Technical Roadmap

with advanced data analytics and machine learning algorithms, buildings are becoming
more intelligent and providing services to support the reliable operation of the grid.
In recent years, the increasing affordability and efficiency of rooftop solar panels
and energy storage systems have made it feasible to generate clean electricity on site,
contribute to grid-interactive capabilities, and enable greater flexibility and resilience. In
addition, governments and regulatory bodies worldwide have begun to develop polices and
standards to promote their adoption. For example, in the US, FERC order 2222 enables
aggregators of DERs, such as GEBs, to participate in market-based mechanisms and
provide grid services. Similar regulatory provisions have emerged around the globe in
recent years in Australia, Europe, Asia, and Latin America, and are contemplated in Africa.
Future State

The future of GEBs holds significant promise as we move towards a more

sustainable and efficient energy future. Widespread adoption of GEBs is expected in both
commercial and residential sectors with more affordable technologies. GEBs may also play
a role in the development of smart cities, becoming part of energy communities that enable
the sharing and trading of energy among buildings and local renewable energy resources
(i.e., peer-to-peer trading). In addition, GEBs may integrate electric vehicle (EV) charging
infrastructure and participate in vehicle-to-grid (V2G) technologies, which could enhance
the overall flexibility and resilience of the energy system.
Challenges

While grid-interactive buildings offer various benefits, there are several challenges
that need to be addressed. First, implementing grid-interactive technologies can be
expensive due to increased costs from advanced energy management systems, energy
storage, and renewable energy generation. In addition, coordinating these systems can be
complex and present technical challenges. Second, motivating and engaging occupants to
actively participate in grid-interactive activities can be challenging. Moreover, large
amounts of data related to occupant behavior and energy usage need to be managed, stored,
and analyzed, which poses challenges in terms of data scalability, security, and privacy.
Third, it is essential to further develop standard protocols to facilitate communication
between smart buildings and the grid. The lack of interoperability can hinder seamless
integration of grid-interactive buildings. Furthermore, the policy frameworks may not
always be fully supportive. Developing clear regulatory frameworks is essential to drive
widespread adoption. Finally, a clear set of reliability standards that outline the expected
contribution to support either the grid (surplus energy, and essential reliability services) or
the GEBs (energy when none is available and essential reliability services such as
frequency and voltage support).
Potential Solutions

Addressing the aforementioned challenges requires collaboration among various

stakeholders, such as building owners, developers, utilities, policymakers, regulators, and
62
TR-123 — IEEE PES Technical Roadmap

technology providers. Economic and technological incentives can be leveraged, and

community engagement needs to be improved. To more effectively manage the customer
data, transparency and sensitivity of information needs to be carefully exercised by the
companies collecting and storing the data. In addition, advanced data analytics and
machine learning algorithms could be used to extract information without exposing
personal identifiable data. To enhance interoperability and establish common standards,
open architectures can be used among the community of stakeholders for everyone to
contribute and provide feedback.
By overcoming these obstacles, grid-interactive buildings can play a crucial role in
building a more efficient, sustainable, and resilient energy future.
4.2.2 Electric Vehicles (EVs) and Battery
Current State

In order to meet the global decarbonization goals, electrification of the

transportation sector is an important milestone to be achieved. Consumer adoption of
electric vehicles continues to increase substantially. According to the Global EV Outlook
2023 document, the global stock of electric cars has reached 26 million, while the share of
electric cars among the total car sales has gone up to 14% in 2022. At the same time, the
lithium-ion (Li-ion) battery demand used in automotive applications saw an increase of
65% and has gone up to 550 GWh in 2022 [1]. Bulk electric storage system applications
also increase the battery demand. This increase has pressured the demand for critical
materials such as lithium, cobalt, and nickel. Therefore, it is important to use innovative
approaches to reduce the dependence on critical materials to maintain global supply chain
sustainability, resilience, and security.
Future State

The EV market share is projected to keep growing in the near future, with policy
support and advancements in technology contributing to the growth. These trends will
directly impact the electric power grid in several ways such as the need to strategically
develop electric vehicle charging infrastructure, upgrade existing grid infrastructure to
accommodate increasing power demand, ensuring that, during events, charging facilities
support the grid, rather than exacerbate them, and secure operation, control, and monitoring
of the grid. Another major recent trend is the consideration and use of EVs as one of the
behind-the-meter distributed energy resources (BTM DERs). With the current growth
trend, they could significantly impact the system operation, monitoring, and control. More
importantly, EVs as BTM DERs could be used for providing grid services such as
frequency regulation, enhanced grid resilience, voltage support, etc., similar to other BTM
DER assets [2]. There has been some recent research that proposed EVs could be used for
handling power outages and critical load restorations during extreme weather events [3].
Even though the use of EVs sounds promising, it still requires more significant research
efforts to guarantee the reliability of the electricity supply and make sure that no damage
is caused to the EV batteries. EV vehicles and batteries will be designed and built with
63
TR-123 — IEEE PES Technical Roadmap

environmental stewardship in mind, for longevity and ‘no-waste’, that is, incur minimal
waste at the end-of-use.
Challenges

Some of the main grid challenges associated with the increasing number of EVs are
peak demand management, strategic placement of electric charging infrastructure,
facilitating flexibility services by EVs, flexibility revenue and associated cyber-security
concerns, fortifying distribution and transmission systems, and peer-to-peer transactions
between EV owners. Simultaneously, there are a few challenges associated with the
increasing demand for batteries such as battery degradation based on use, charging and
discharging patterns, applying sustainable and innovative methods in the mining of the
critical materials required for batteries production, developing more sustainable ways of
battery disposal after the end of use / end of their life, i.e. i.e. re-use of all battery materials
as feedstock for new product, such that batteries essentially meet the ‘no-waste’ product
challenge of a circular economy. These same challenges apply to the charging
infrastructure and the EV’s themselves: design and built for sustainability “by design”, that
is energy efficiency and full re-use of parts and/or materials at end-of-use. All these
challenges associated with EVs, and batteries are currently being addressed but more
efforts are needed to be put in to address the concerns for the long-term.
Potential Solutions

As mentioned above, the challenges are related to many stakeholders and

addressing them requires collective efforts from many of them. For example, to manage
increasing peak demand, electric system owners may need to carefully plan their system
upgrades, balancing authorities may need to plan for more flexible resources, customers
may need to participate in smart charging programs, and policy makers may need to
develop new regulations. With collaboration among all stakeholders, the integration of EV
can contribute to a more efficient and low-carbon power grid.
Design and build EVs, charging infrastructure and batteries with environmental
stewardship in mind, for longevity and ‘no-waste’, that is, incur minimal waste at the end-
of-use, thus supporting resource availability in the long term; in effect becoming part of a
Circular Economy.

4.2.3 Forecast: Load, PV, EV pattern

Current State

Load forecasting is one of the important tasks for electric system operators. It has
evolved much over time while recent years have seen extensive use of machine learning,
neural networks, and artificial intelligence techniques over time series models. The
following are the three main factors that impact load forecasting: time, weather, and
economics. Load forecasting can be further classified based on forecast horizons as very-
64
TR-123 — IEEE PES Technical Roadmap

short term, short-term, medium-term, and long-term forecasting. Along with load
forecasting, solar PV forecasting is another important task for system operators to plan and
operate their networks properly, especially with the recent growth trend of solar PV. It
typically includes the following five aspects i.e., base methods regarding data acquisition,
data acquisition post-processing, irradiance to power conversion, forecast verification, and
grid integration [4]. The impact of PV forecasting on the power grid could be manifold
such as regulation and load following, stability and control, transmission, and distribution
planning, etc.
Future State

As the number of edge devices increases, the load patterns change, and variation
increases. Large loads, such as data centers, might also grow. Forecast needs to adapt to
capture the new characteristics in load, PV, and EV. In the meantime, more advanced
sensors are deployed in the system, providing more data. Another trend is the use of
machine learning techniques, which could be a powerful tool for forecasting.
Challenges

The main challenges associated with load forecasting that need to be taken up are
enhancing algorithmic abilities for fitting non-linear constraints, improving the data quality
using modern sensors, application of adaptable data-driven methods, consideration of
multi-energy systems, etc. [5] In the case of PV forecasting, addressing atmospheric
physics problems for the accurate forecast of base irradiance, and post-processing of
forecast of base irradiance into solar PV power forecasts are the main challenges. Another
challenge is forecasting the speed and size of deployment of PV in any given geographical
location. With regard to EV, demand pattern forecasting, it is challenging to accurately
forecast the EV demand as it depends on the behavior and schedules of the EV users.
Another associated phenomenon with load and PV forecasting is network congestion
forecasting which has recently emerged as one of the big concerns for the distribution
system operators, especially for networks with high penetration of solar PV and uncertain
load [6]. Extreme weather events also impact demand patterns and need to be considered
in forecasting algorithms.
Potential Solutions

Some of the key solutions for PV forecasting with promising prospects include
satellite-based irradiance forecast methods with very fine resolution, the evolution of area-
based PV forecasting instead of location-based PV forecasting, the collaboration between
atmospheric physics and power system research domains, etc. [7]. In the case of load
forecasting, the approaches used in literature could be broadly categorized into
deterministic and probabilistic approaches. Furthermore, EV demand forecasting has seen
the application of machine learning techniques in recent years such as the ones presented
in [8]. However, future research needs to focus on factors such as generalizability, time
efficiency, etc., for the proposed methods. Regarding congestion forecasting in distribution

65
TR-123 — IEEE PES Technical Roadmap

systems, a support tool is proposed in [6] for the system operators to handle the high
penetration of solar PV and uncertain load.
4.2.4 Virtual Power Plants
Current State

The term Virtual Power Plant (VPP) refers to an aggregation of electrical energy
producing and/or consuming devices that can change their net energy injection into, and/or
withdrawal from, the electricity grid in response to remote dispatch signals, energy prices,
or local triggers.

The devices comprising a VPP are generally referred to as Distributed Energy

Resources (DER). These may include distributed generation, storage, electric vehicle
chargers, smart thermostats, Grid Interactive Electric Water Heaters (GIWH), and similar
controllable grid-edge devices and systems.

Aggregation of DERs into VPPs may be governed by a set of rules based on

prevailing regulations and jurisdictional provisions. Generally, DER programs defined by
market operators, and/or distribution utilities form the basis for aggregation of DERs into
VPPs. At a high level VPPs and their underlying DER programs may be classified into
homogeneous or heterogeneous aggregations.

 Homogeneous aggregations involve VPPs comprising DERs with similar

technological characteristics. Thermostat control programs, air conditions ON/OFF
control programs, EV Charging Station aggregations, GIWH aggregations, and
generally conventional Demand Response (load reduction) programs fall into this
category.

 Heterogeneous aggregations include DERs with different technological

characteristics. Examples are VPPs including combination of distributed generation
(e.g., solar PV), storage, and convention Demand Response included in the same
aggregation.

The eligibility of VPPs for provision of different grid services is also based on
prevailing aggregation rules, DER technological characteristics, as well as telemetry and
control requirements. For example, provision of Frequency Regulation service in the U.S.
bulk power operation generally requires telemetry and control with periodicity of a few
seconds (normally 4 seconds), and commensurate response characteristics from the
combination of DERs comprising the VPP. In contrast peak shaving or load shifting
services are amenable to dispatch or advance price notifications of a few hours to a day or
more with no special telemetry and control, but generally with interval metering capability.

Future State

66
TR-123 — IEEE PES Technical Roadmap

With proliferation of variable generation at bulk level (e.g., solar farms, wind farm),
and increasing levels of active grid-edge assets, the grid operators are facing the need for
increasing levels of conventional grid services (such as frequency regulation and
contingency reserves in North America), as well as the need for new types of grid services,
generally called Flexibility Services. Examples of the latter in the North American context
are Ramping Up/Down, Load Following, and Uncertainty Reserves to mention a few. The
electricity grid of the future can no longer rely exclusively on conventional generation
facilities for provision of increasing levels and types of needed grid services. Demand
flexibility, leveraging DERs, is recognized globally as the resource grid operators must
rely on for cost-effective provision of conventional and emerging grid services. VPPs
provide an alternative to conventional generation for cost-effective provision of grid
services.

Challenges

In the U.S., FERC Order 2222 paves the way for participation of VPPs comprising
DERs in the footprint of FERC-jurisdictional market operators. However, its ramifications
go beyond the six FERC-jurisdictional ISOs/RTOs in the U.S., and have spread to other
regions in North America, including non-market regions in the U.S., and some Canadian
markets, such as Ontario’s IESO. All of these entities are in need of proper operational
coordination solutions for operations scheduling and real-time operation involving DER
asset operators, Aggregators, Micro-grid operators, distribution operators, and bulk power
system operators. This coordination is at present a challenging endeavor. The main issues
are concerns about dual participation (in both distribution and bulk operation) by the same
DER assets; handling of control priorities (by the distribution and transmission operators),
and dual compensation (by two entities for the same response).

There is another challenge in widespread adoption of VPPs. The increasing

percentage of VPPs compared to conventional generation requires acceptance by control
room operators. Control room operators are generally much more comfortable relying on
conventional generation plants to which they have visibility and scheduling/dispatch
control compared to VPPs comprising DER assets spread throughout the distribution grid
to which they generally have limited to no visibility and control. As such proof of
performance of VPPs in response to control room dispatch signals is highly important for
acceptance of VPPs as viable resource for provision of grid services. To enable this
increased visibility and expected performance, reliability standards or grid codes are
needed to ensure the cyber security, back-up control centers, and essential reliability
services can be relied upon to support the reliable operation of the bulk power system.

Potential Solutions

The first challenge, proper operational coordination, is being addressed by a

combination of DER and DER aggregation registry and the operational coordination
platforms. The Registry provides a system of records for all DERs and DER Aggregation
(VPPs). Depending on prevailing regulatory provisions, organizational structure, and

67
TR-123 — IEEE PES Technical Roadmap

underlying grid architecture, the Registry may be designed with a hierarchical structure for
different access privileges (Read, Write, Copy, Modify) by different stakeholders.
Similarly, the information exchange, submission, approval, denial, modification, and
withdrawal, of VPP actions (registration, aggregation, bidding, scheduling, control, and
response verification) in the Operational Coordination Platform are governed by rules
agreed upon by the stakeholders and approved by appropriate regulatory entities.

To address the second challenge, adoption in the control room, a multi-pronged

approach may be needed. First, metrics and measures may be developed to assess the
effective capacity of VPPs compared to the collective capacity of its constituent DERs.
Second, measurable VPP performance metrics are defined to track performance of VPPs
in response to control room dispatch signals. Actual VPP implementations are
supplemented by a variety of pilot VPP projects to verify their performance and enhance
their acceptability by the grid operators. Third, reliability standards and grid codes need to
be enhanced to strengthen the role of VPPs in the reliable operation of the bulk power
system.

4.3 Coordination and Control of Grid Edge Assets (architecture/system

level)
4.3.1 Optimal Power Flow
Current State
The optimal power flow (OPF) [9] is an optimization problem in electrical power
systems that minimizes the pre-defined costs functions while meeting various network and
operational constraints under power flow equations, by optimally selecting the operating
points of the controllable devices. Once solved, OPF’s solutions can provide guidance to
optimal plan and operation of the power system. In practice, OPFs of various complexity
and functionalities are formulated for different implementation scenarios. We briefly
introduce the following four types of commonly applied OPFs in current and future power
systems:

1. Unit Commitment (UC) aims to minimize the total power generation costs in a
specific period by deciding when and which generating units at each power station
to start-up and shutdown to meet a given demand while considering relevant
engineering constraints of the generators. The integer decision variables related to
turning on and off the generators make the UC problem nonconvex and
computational complex. Commercial solvers are usually used to solve such mixed-
integer problem hours or days in advance.
2. Economic Dispatch (ED) focuses on determining the optimal real-power output of
generating units to meet the electricity demand at the minimum possible cost, given
the results of UC. The ED problem is typically solved over a short time horizon
(e.g., hourly) and is an essential component of the real-time operation of power
systems. The computational complexity of ED is usually low because it is a convex
optimization problem that can be efficiently solved.

68
TR-123 — IEEE PES Technical Roadmap

3. Direct Current OPF (DC OPF) integrates DC power flow equations and their
pertinent constraints into the ED problem. As a result, DC OPF respects line flow
limits in addition to meeting the power demand-supply balance as UC and ED focus
on. However, since the DC OPF problem is still convex, the additional
computational complexity from network constraints can usually be handled
efficiently by convex optimization solvers.
4. Alternating Current OPF (AC OPF) is among the most complicated forms of
OPFs in practice, featuring full-blown AC power flow equations (therefore
involving reactive power in addition to real power) and their pertinent constraints,
making the optimization problem more complex and highly non-convex, and
therefore mathematically challenging to solve. However, because reactive power
and voltage magnitudes can be involved, AC OPF is powerful, especially in modern
power systems integrated with grid edge controllability and flexibility. AC OPF
can be exactly solved; however, approximation and convex relaxation, among other
methods, are often used to efficiently obtain “good-enough” solutions in practice.

Future State
There is a growing trend of increasing integration level of dispatchable distributed
energy resources (DERs), such as roof-top PVs, batteries, and smart loads. To address this
trend, the OPFs can be equipped with more controllable grid edge devices in addition to
the conventional ones and can thus enjoy a broader feasible set to explore better solutions.

Challenges

Meanwhile, the amount of distributed controllability integrated into the modern OPFs
are leading to modeling and solving challenges that traditional OPFs do not have:

1. Modeling: In contrast to conventional utility devices whose models are usually

available, accurate models of grid edge devices---especially the behind meter
devices---are often not.
2. Solving difficulty: The amount of controllable grid edge devices adds higher
dimensionality, non-linearities, and uncertainties to the OPF problems, requiring
significantly more computational capability.
3. Practical implementation: Implementing OPF in modern power systems has
practical challenges including: computation capabilities at DERs, communication
facilities on the grid edge, accurate and timely system states monitoring by
measurement and state estimation, network security, and so on.

Potential Solutions

To address the modeling challenge, DER flexibility estimation can be leveraged to

approximately estimate the grid edge controllability for OPF based on historic data.
Alternatively, distributed computation architectures can solve OPF while preserving model
information to prosumers through well-designed communication and computation
mechanisms.

69
TR-123 — IEEE PES Technical Roadmap

To address the second challenge mentioned above, solving difficulty, advanced

optimization techniques, data analytics, and machine learning are being integrated into
modern OPF solving technologies, which simplify or accelerate the computations.
Particularly, distributed computation has been considered a promising approach to
handling the grid edge integration, as will be elaborated in next Section of “Distributed
Control”.

For practical implementation, various stakeholders need to work together,

including manufacturers, system operators, policy makers, and customers.

4.3.2 State Estimation

Current State
State estimation in power systems [10] is a critical process used to estimate the
current operating conditions and parameters of an electrical grid. It plays a fundamental
role in ensuring the reliable and stable operation of power systems, especially in large and
complex networks. State estimation involves the use of mathematical algorithms and real-
time measurements to determine the most accurate representation of the electrical state of
the system. The most common problem formulation for State Estimation is the Weighted
Least Squares (WLS) where the objective is to minimize the weighted sum of the squared
differences between the actual measurements and the measurements estimated by the state
variables, subject to power flow equations as constraints. In this sense, WLS-based state
estimation can be seen as a special case of OPF. To efficiently solve the WLS problem,
conventionally the Gauss-Newton method is usually applied.

Future State

Nowadays, as the power systems, especially power distribution systems, have

become more volatile and dynamic with integration of grid-edge DERs, conventional state
estimation may fall behind the fast-developing system states. To address this condition,
dynamic state estimation has been developed to continuously track state variables over time
as the system evolves dynamically based on, for example, Kalman Filter and Particle Filter.

Challenges

Modern power systems state estimation is faced with several major challenges for
timely and accurate estimation results:

1. Data availability and quality: Accurate and timely measurement data is crucial for
reliable state estimation but requires deployment of large amount of accurate
measurement devices and communication infrastructure, which are costly
especially in distribution system.
2. Computational complexity: Large power systems with numerous measurements
can make state estimation computationally intensive.
3. Model availability and accuracy: The accuracy of the mathematical models used in
state estimation is essential for reliable results but often unavailable or inaccurate
especially in distribution systems.
70
TR-123 — IEEE PES Technical Roadmap

Potential Solutions

From the theoretical side, several advancements will be helpful including: smart
sensor placement, algorithms that can handle large number of measurements and imperfect
models in dynamic state estimation, and more efficient computation. From practical
implementation side, more and better sensor deployment is needed to ensure data
availability and quality.
4.3.3 Distributed Control
Current State

Distributed controls are control architectures [11] where various elements or

components of a power system have localized control capabilities. These local controls
work in conjunction with each other, rather than being managed by a centralized authority.
In traditional power systems, there is usually a centralized control system that monitors
and controls all the elements such as generators, transformers, and loads. However, as
power systems evolve, especially with the integration of grid-edge distributed energy
sources (DERs) and the development of smart grids, several challenges emerge for
traditional centralized control: scalability, resilience and reliability, optimization,
efficiency, etc. On the contrary, distributed control is becoming more prevalent in
addressing these challenges:

1. Scalability: Distributed control systems are scalable, which means that as the power
system grows, the increased size and complexity of the system does not overhaul
the existing control infrastructure or impact its computational effectiveness and
efficiency. This is especially useful in the context of grid edge integration, where
the number of control units can vary widely.
2. Resilience and reliability: By distributing control functions, the power system can
become more resilient to failures. If one controller or computation node fails, others
can still operate independently to some extent, which is not the case in a single-
point-of-failure centralized system. Such resilience can be enhanced with more
DERs integrated. Moreover, distributed control can better detect local faults and
self-heal by adapting local control strategies.
3. Optimization and efficiency: Through distributed control, the power system can
reach for more local energy resource and controllability that traditional centralized
control cannot. By optimally dispatching these additional grid-edge resources,
better global efficiency and social welfare can be achieved.

Future State

There are a few typical architectures of distributed controls deployed in modern

power systems in terms of their communication structures and control decision making
mechanism:

71
TR-123 — IEEE PES Technical Roadmap

1. Local controls make control decisions based on the locally available information
only. Local controls feature simple implementation in practice without any
communication required. Decent performance can be achieved for a lot of common
scenarios. However, local controls usually cannot meet more strict global
performance requirements, e.g., network voltage magnitudes or system frequency
within firm ranges, due to the lack of system-wide information. Examples of local
controls include Volt/Var droop control, primary frequency control, voltage
regulator control, etc.
2. Neighborhood decision making makes control decisions based on not only local
information but also on that from their neighborhood through communication with
connected nodes. By having access to and leveraging more information, such a
control structure can usually achieve better local performance and meet more
specified and strict global performance criteria than local controls only. Examples
include the alternating direction method of multipliers (ADMM) and consensus
algorithms.
3. Centrally coordinated decentralization is often designed to achieve global
optimal performance through coordination between distributed controllers and
central coordinators. It is often a distributed implementation of centralized control.
But unlike centralized control systems, which have a single control center
managing the entire grid, distributed control systems distribute control tasks among
different components within the network through one-way or two-way
communication networks. Such a control structure improves scalability and
computational efficiency by strategically allocating computational loads, while
increasing the reliability and resilience of the system. Examples include numerous
distributed optimal power flow solvers, hierarchical distributed control of the
NREL Autonomous Energy System project [12], and so on.

Challenges

The challenges of developing and implementing distributed control in modern

power systems mainly lie in the following two categories:
1. Developing advanced algorithms to optimally dispatch and coordinate all
controllable devices is challenging because (1) the innate complex nature of the
non-convex power flow equations makes the problem mathematically complex, (2)
the large amount of controllable DERs make the problem highly dimensional and
computationally difficult, (3) the distributed design must account for limited
computation and communication capability at grid edge, (4) the design must
provide timely robust updates to accommodate for the uncertain, volatile, and fast-
changing DERs, and (5) the design need to be compatible with legacy components
and control systems without requiring significant changes. To design distributed
control for specific grids for various realistic scenarios and needs, some or all of
the afore-mentioned challenges need to be addressed.

72
TR-123 — IEEE PES Technical Roadmap

2. Moreover, because the distributed control architecture is noticeably different from

the traditional ones in power systems, there are real-world implementation
challenges including (1) the availability, timeliness, and quality of real-time data,
(2) communication infrastructure and security, (3) distributed computation
capability at grid edge, (4) regulatory and standardization, (5) human factor and
acceptance, and so on.

Potential Solutions

Researchers and engineers have been working to address these challenges from the
perspectives of both theoretical design and practical implementation over the years. We
have seen successful implementations of advanced distributed control architectures in labs
and small field demonstrations.

4.3.4 Advanced Distribution Management System (ADMS) and Distributed

Energy Resource Management System (DERMS)
Current State

A distribution management system (DMS) manages the distribution grid in real

time, providing monitoring, analytics, and visualization capabilities. Many DMS vendors
now offer products that integrate multiple functions, such as an outage management system
(OMS), a geographic information system (GIS), and a supervisory control and data
acquisition (SCADA) system. These integrated platforms are commonly referred to as
advanced distribution management systems (ADMS). These platforms also offer advanced
functions, such as Volt/VAR optimization (VVO) and fault location, isolation, and service
restoration (FLISR) constraint management, and even look-ahead constraint management
applications (proactively solving predicted constraint violations). Distributed energy
resource management systems (DERMS) are emerging software technologies aimed at
providing distribution utilities with a specialized set of tools to enable them to overcome
issues caused by distributed energy resources (DERs) and to maximize the benefits of the
presence of high deployment levels of these novel resources.

Current utility processes and systems are typically based on the premise that
facilities and equipment under control are owned by the utility. The expanding penetration
of DERs not owned by the utility lead to significant changes in distribution operations and
the role of a distribution utility. Customer and third-party resources are typically operated
under a defined tariff or contract and are subject to compensation for services provided
under the terms of that tariff or contract. Behind-the-meter resources are usually not
represented in detail in ADMS. In most cases, an ADMS does not deal with commercial
aspects like tariffs or contracts that result in commercial and contractual constraints in
addition to technical device and grid constraints.

Future State

73
TR-123 — IEEE PES Technical Roadmap

The transition from traditionally passive distribution grids to increasingly complex

distribution grids with large numbers of DERs—most of which are inverter-based and
increasingly grid-forming (GFM)—dispersed throughout requires the accompanying
modernization of control centers and the development of highly intelligent software
systems to enable the real-time observability, control, aggregation, constraint management,
and protection of such grids.

DERMS technology is still nascent, and despite standardization efforts [13], the
term DERMS is used to refer to very different levels of software hierarchies that have
different sets of tools and aim to provide different services to different stakeholders—
spanning from decentralized virtual power plants (VPPs), to DER aggregators, to
centralized enterprise systems, which are sometimes referred to as a utility or grid DERMS.
The main role of DER aggregators is to aggregate small-scale DERs into DER groups and
consequently to enable their services using the aggregated DER power. The role of DER
aggregators includes enabling the participation of small-scale DERs in electricity markets,
the engagement of DERs and prosumers in energy-saving and energy-efficiency programs,
the provision of demand response and load-shedding services, as well as the provision of
other mostly customer-related services. The DER aggregator structure that is being adopted
in some regions today, however, aggregators are not fully aware of the grid model, its
conditions and its technical boundaries, so they cannot guarantee avoiding issues such as
congestion, voltage violations, or protection issues. Thus, to enable the safe use of the
services offered by DER aggregators, distribution utility operators must have observability
of the real-time grid conditions as well as the ability to validate—and modify, if
necessary—the DER aggregator schedules to avoid causing constraint violations on the
grid assets [14].

In contrast, utility DERMS solutions are grid-aware software packages that enable
the full awareness, control, and optimal management of medium- to large-scale DERs and
DER groups (consisting of behind-the-meter DERs), with the goal of using all these
resources to achieve system-wide benefits without violating grid constraints. Further,
utility DERMS solutions use all available resources to solve existing violations or predicted
constraint violations and keep the system in a stable and optimal state in real time.
Therefore, grid-aware utility DERMS are well suited to manage large- and medium-scale
DERs, whose impact on the grid conditions can be significant [14].

These different DERMS software technologies can complement each other, and
together they can provide significant benefits to utilities by enabling operators to
successfully manage evolving distribution grids with high penetrations of DERs when they
are integrated together in utility control centers. One approach is to consider DERMS as a
logical entity rather than as a physical platform: DERMS functionalities can reside in other
enterprise platforms, such as ADMS. Following this logic, a “hybrid DERMS” concept
was proposed [15], as depicted in Figure 1.

74
TR-123 — IEEE PES Technical Roadmap

Figure 1 Integration of centralized and decentralized DER management solutions into a

“hybrid DERMS”[15].

Another approach is to have a separate DERMS platform that complements the

ADMS. A properly designed and implemented DERMS addresses both the grid reliability
operations and the commercial aspects of dealing with DERs and the associated
transactions with customers and other third-party DER asset owners. It interfaces and
exchanges data and controls with other enterprise and legacy systems (including
DMS/ADMS), breaking down data silos across the distribution utility organization and
serving as the system of record for all data related to DERs and their operations.
Organizationally, the DERMS bridges the gap between Information Technology (IT)
groups such as customer services, metering, and accounting and Operational Technology
(OT) groups such as distribution system operations and planning.

Challenges

Moving forward, the following areas in ADMS and DERMS development will need
to be addressed:
 ADMS operations with GFM inverters, including some that are black-start-capable
[16], especially as they relate to FLISR and the impact on both fault location and
service restoration. Studies [17] have suggested that more synchronous condensers
will be needed in systems with high levels of GFM inverters, and these will need
to be considered by ADMS.
 The ability of DERMS to scale as the number of controllable devices increases from
the tens of thousands to the millions.
 Integration of an ADMS with a DERMS from a different vendor, which requires
data sharing to allow activities such as scheduling operation around planned and

75
TR-123 — IEEE PES Technical Roadmap

unplanned outages. Network model sharing would be required if the DERMS is

expected to operate in a network-aware way.
 Integration of an ADMS and/or a DERMS with third-party DER aggregators,
VPPs, and microgrid operators.
 Managing uncertainty in ADMS and DERMS optimization algorithms as more non
dispatchable DERs come online and there is more reliance on customer-owned
equipment [18].
 Transmission and distribution operation integration and the provision of bulk grid
services by DERs [16, 19]. This would lead to the development of integrated control
centers that include operators for transmission energy management systems,
ADMS, and DERMS.
 Cloud deployment of ADMS and DERMS solutions and meeting the associated
cybersecurity requirements.
 Absence of a globally accepted communications protocol and interoperability
standard between ADMS, DERMS, DER aggregators, and individual DERs. This
is a significant obstacle toward the integration of centralized and decentralized
technologies, including the integration of various DER management platforms,
individual (large-scale and small-scale) DERs, as well as integration with bulk
system operations [15].
 The use of custom-made application programming interfaces (APIs), e.g., between
a utility DERMS and a DER aggregator. Standardization of the communication
protocols is currently taking place, and the IEEE 2030.5 protocol is a promising
solution.

Potential Solutions

Clarifying what is meant by the term DERMS and differentiating between DERMS
that are network-aware and those that are not will be very helpful to the industry, and IEEE
can play a significant role in this space.

Many have suggested distributed, hierarchical control approaches to achieve the

scalability of DERMS to millions of devices [20, 21, 22], and some DERMS products are
emerging that are more distributed in nature. Participation of ADMS, DERMS, DER
aggregators, and VPP vendors in grid architecture development activities and interactions
among groups involved in such activities would be beneficial to the industry, especially
around standardizing data and model sharing among systems.

Finally, although technical advancements are required to address the challenges,

with high integration levels of DERs, we anticipate that many innovative technical
solutions will require associated regulatory and business model changes that should be
considered in tandem with technology developments.

4.3.5 Transactive Energy Systems and Energy Blockchain

Current State

76
TR-123 — IEEE PES Technical Roadmap

From the GridWise® Architecture Council (GWAC) Transactive Energy

Framework, the definition of transactive energy is: “A system of economic and control
mechanisms that allows the dynamic balance of supply and demand across the entire
electrical infrastructure using value as a key operational parameter” [23].

The use of the term “value” in the GWAC definition is often thought of as a
surrogate for real-time prices but is more usefully thought of as recognizing the use of
economic signals as control signals. One should note that this is not new. Transactive
energy concepts trace back to Fred Schweppe’s work at MIT in the last 1970’s and early
1980’s on what he called “Homeostatic Control.” Schweppe’s ideas couldn’t be
implemented as he envisioned then due to technology limits of the time but did provide the
foundation for the current organized wholesale markets in North America.

Transactive energy systems are a distributed form of control and coordination that
may be used to orchestrate the integration of DER and other assets in distribution systems
and of the interactions between distribution systems and the bulk power system.

There have been multiple field demonstrations of Transactive Energy Systems in

the United States and Europe. One of the earliest being the Olympic Peninsula GridWise®
Demonstration conducted in Washington State in the 2006 – 2007 timeframe. In the same
time frame TNO in the Netherlands conducted similar trials using the same double-auction
market approach with what they call PowerMatcher.

Under IEEE Future Directions and within the PES Smart Buildings, Loads, and
Customer Systems Technical Committee, work has been going on for the past several years
in a subset of Energy Blockchain focused on blockchain technology in transactive energy
systems. This work identifies the potential for using blockchain technology to provide
distributed ledgering to meet bookkeeping and audit requirements for transactive energy
systems. The possible use of blockchain based smart contracts is also considered. This
work is culminating in the publication of a Blockchain in Transactive Energy Guide
(P2418.5).

More broadly, the distributed ledger technology known as blockchain has other
potential applications in energy systems. Among them are asset management and
cybersecurity applications such as key management. As energy system engineers become
more familiar with blockchain functionality there are almost certainly other uses that will
be identified.

Future State

The heart of transactive energy systems is the use of distributed control and
coordination implemented via local markets to engage flexibility to offset variability. The
flexibility is associated with DER and the variability typically is associated with variable
renewable energy assets. As we move to decarbonized energy systems with increasing
quantities of DER deployed in distribution systems the distributed control and coordination
of transactive energy systems will become increasingly important.
77
TR-123 — IEEE PES Technical Roadmap

The IEEE blockchain effort supported by IEEE Future Directions has graduated
and is now an IEEE Technical Community. IEEE PES is one of the societies sponsoring
the new technical community. Currently Smart Buildings, Loads, and Customer Systems
is the primary connection with the new technical community. Other PES technical
committees with an interest in blockchain need to engage with the new technical
community and in particular their Blockchain in Energy subcommittee. Through such
engagement further uses of blockchain in energy systems can be discovered.
Challenges

Barriers to deployment of transactive energy systems at scale include:

 Regulatory – because they are associated with real-time prices regulators are often
concerned that they expose customers to risk that should be borne by
utilities. There are also regulatory concerns about the cost of implementing
transactive energy systems.
 Integration Costs – As we scale DER and simultaneously deploy transactive energy
of other technologies for integration of the DER, the time and cost to integrate may
be prohibitive. The development of integration standards or better use of existing
standards is needed to reduce the time and cost to integrate. The overall objects
should be plug-and-play integration.
 Customer Acceptance – There are concerns about the complexity of transactive
energy systems and possible resulting customer acceptance challenges. These
systems need to be engineered to provide transparency and ease of operation to
users.

Potential Solutions

Achieving deployment of transactive energy at-scale is a multi-stage journey.

GWAC developed a transactive energy roadmap in 2018 [24], which outlines a vision and
path forward. At different DER penetration levels, various technical, economic, and
regulatory issues need to be addressed. More information can be found in the roadmap
[24].
4.3.6 DER market participation: regulatory/business model
Current State

The electricity market structure differs greatly based on geographic regions. In the
US, the electricity markets have traditionally been vertically integrated where the utility
may own or oversee the generation, transmission, and delivery of electricity to customers.
In the late 1990’s/early 2000’s, some states decided to deregulate the electricity market to
increase competition and reduce costs. In deregulated markets, utilities that serve retail
customers are only responsible for delivering electricity to their customers; the electricity
is generated by other entities. Those generating entities typically sell the electricity they
generate through competitive power markets known as independent system operators

78
TR-123 — IEEE PES Technical Roadmap

(ISOs) and regional transmission organizations (RTOs). US electricity markets also have
wholesale and retail components. Wholesale markets involve the sale of electricity among
generators and resellers, which will then be sold to consumers. Retail markets involve the
sale of electricity directly to consumers. Wholesale and retail markets can be found in
traditionally regulated states or restructured states.
DERs are relatively new participants in the electricity market. They are usually
smaller in size and spread across the grid. Traditionally, it is difficult for DERs to
participate directly in the wholesale markets, and the more common way to participate is
through retail programs with local utilities. More recently, FERC Order 2222 allows DERs
and aggregators to participate in the wholesale market, which opens more possibilities.
With the increase of grid edge resources, we envision more impacts to the markets and new
market structure and product may be needed.
Future State

Wholesale market participation: Integrating DERs into the wholesale markets could
bring multiple benefits, for example, improve market efficiency, increase flexibility,
maintain reliability, and potentially decarbonize the system. FERC Order 2222, which was
released in 2020, mandates that ISOs/RTOs revise their market rules to allow DER
aggregations to participate in their markets. While the order has the potential to unlock
significant participation of DERs in wholesale markets, with the opportunities it brings, it
also brings challenges. The list below highlights a few of these challenges:
 Eligibility to participate: ensure fairness while avoiding double counting.
 Information and data requirements: tailor requirements to meet specific needs for
different market products or DER types.
 Coordination between ISOs/RTOs, DSOs, and DER aggregators.

Emerging Market Models: The centralized paradigm of current electricity markets

may no longer be adequate to fully leverage the flexibility and grid services provided by
grid edge resources. Markets of the future may take advantage of grid modernization
initiatives and inform timely integration of grid digitalization, energy management
systems, communication networks, and the creation and widespread adoption of new
standards and protocols. Different market models have been proposed in the literature,
which can be roughly classified from total TSO to total DSO. The total TSO structure
employs a centralized paradigm which considers a single market; while the total DSO
structure employs a hierarchical paradigm which considers the separation of transmission
and distribution resources, in that dispatch decisions for DERs are completed by the DSO.
Between the two ends of the spectrum, there are also many hybrid structures with different
levels of TSO/DSO coordination.
Challenges

Market structures along all ends of the spectrum face implementation challenges.
While centralized paradigms are a natural extension of current market structures, they may
79
TR-123 — IEEE PES Technical Roadmap

not be scalable to the cases with high penetrations of DERs that are expected to be installed
in the distribution grid. Perhaps the most pertinent challenge remains that of tier bypassing:
DERs that reside in the distribution grid are directly dispatched by wholesale markets in
response to the bulk grid objectives without considering local constraints. At the other end
of the spectrum, the hierarchical paradigm is a significant shift from current market
structures and would require a significant reimagining of the roles and responsibilities of
various market agents, and significant regulatory reform. However, the hierarchical
paradigm has better scalability across grids with high DER penetration levels, and when
appropriately designed can enable plug-and-play characteristics which are highly desirable
in future operating frameworks. Furthermore, the hierarchical paradigm lends itself more
naturally to the establishment of distribution-level retail electricity markets. Retail markets
can support the creation of new DER services which take advantage of the operational
flexibility of DERs by developing distribution-specific market derivatives and prices with
a high spatial-temporal granularity. However, retail markets significantly affect all revenue
streams associated with the vertically integrated utility. Due to this complexity, retail
markets remain a theoretical market framework. Discussions of such market structures
remain in a nascent stage, and the design of such markets, the operational changes, and the
regulatory requirements are all open research and implementation questions.
Potential Solution

The electricity markets involve many participants, including power producer,

utility companies, system operators, and regulators. Each participant may have different
responsibilities, interests, limitations, and concerns. On one hand, we need research
projects to develop technical solutions to the challenges; on the other hand, it is important
to bring all participants to the table, so that related issues are considered, and a consensus
may be reached.

4.4 Microgrids
4.4.1 Current State
Many definitions for microgrids exist; in this roadmap, we adopt the DOE
definition [25]: a microgrid is a group of interconnected loads and distributed energy
resources within clearly defined electrical boundaries that acts as a single controllable
entity. A microgrid can connect and disconnect from the grid to enable it to operate in grid-
connected and island mode or as an isolated off-grid electricity network. Although the
benefits that microgrids can bring to end users are numerous, their integration into current
distribution networks is still hindered by several technical, economic, and regulatory
issues. Many microgrids demonstration and pilot projects have been built and investigated
all over the world, however, their practical application is still at the initial stage. It is also
important to note that microgrids, by design, are largely unique to the location and needs.
There are general categories of microgrid use cases that are typically considered as having
similar categories of needs or design criteria. Consideration must be made for a use case’s
primary drivers (for example, economic, resilience, and sustainability). Use cases vary in
80
TR-123 — IEEE PES Technical Roadmap

their features based on numerous additional factors, some of which include geographical
contect/region (rural, urban, remote, island, mobile/temporary), ownership model (utility
owned, private, community, public), and ownership mix (single or multiple owner).
4.4.2 Future State
The recent advancements in technology and cost reduction of DERs and energy
storage drive the trend towards a more distributed power grid. Microgrids offer a pathway
to this future by providing opportunities to reduce costs and emissions while bolstering the
resilience of national and regional electricity systems. Microgrids can be a fundamental
building block for power system planning and operations, serving simultaneously as a
coordinator for a suite of distributed energy resources, as an aggregated, nodal point of
control for bulk power system operators, and as an electrical peer networking and sharing
resources with adjacent microgrids. Furthermore, in a future with increasingly frequent and
severe climate-related natural disasters and greater electrification, microgrids may serve as
a valuable resource in support of community resilience. In the near term, progress may
primarily reside in single microgrid adoption and operation. In the longer term,
approaching 10 years out, the progress may be more heavily on adoption and operation of
networked microgrids and their role in wider electricity delivery.
4.4.3 Challenges
The following challenges may need to be addressed:
 Technical challenges: Microgrids are by their nature site-specific solutions. Each
microgrid is designed to meet a specific set of goals and is unique in its technical
configuration and hardware requirements, leading to unique challenges that may
include:
o Deciding on proper architecture,
o Design protecting schemes,
o Set up communication networks,
o Operation strategies under different modes,
o Smooth transition between on-grid and off-grid modes; and
o Black-start capability.
 Regulatory challenges: Similar to technical challenges, each microgrid may also
pose unique regulatory challenges depending on the use case. The regulatory
frameworks that govern Microgrid deployment – including aspects such as asset
ownership, compensation, investment justification, safety, consumer protection,
and others – vary by country and/or region (and oftentimes by utility service
territory), suggesting a need for highly localized knowledge by developers to build
projects. Compounding this lack of a consistent regulatory framework is a clear
disincentive for most utilities to enable non-utility owned microgrids, as many non-
utility Microgrid business models would lead to erosion of their revenue base.
 Furthermore, utilities are financially and legally responsible for the safety of the
electric system, and managing the risks associated with non-utility microgrids may
81
TR-123 — IEEE PES Technical Roadmap

be new and administratively cumbersome. As a result of these dynamics and the

resulting regulatory uncertainties, project bankability and replicability continue to
be major challenges for Microgrid market development [26], especially for multi-
property microgrids where new types of safety and consumer protection risks
necessitate the evolution of utility and state regulatory oversight to protect
consumers, workers, and the public.

4.4.4 Potential Solutions

As mentioned before, each Microgrid has its unique use case, leading to distinct
challenges and potential solutions. Many of the challenges are multifaceted, involving
technical, economic, and regulatory issues. Thus, solving them will require careful study
of the use case and collaboration among many stakeholders.

4.5 References
[1] Global EV Outlook, 2023. https://iea.blob.core.windows.net/assets/dacf14d2-eabc-
498a-8263-9f97fd5dc327/GEVO2023.pdf.
[2] Task Force report: Behind-The-Meter Distributed Energy Resources: Estimation,
Uncertainty Quantification, and Control.

[3] A. K. Erenoğlu, S. Sancar, İ. S. Terzi, O. Erdinç, M. Shafie-Khah, and J. P. Catalão,

"Resiliency-driven multi-step critical load restoration strategy integrating on-call
electric vehicle fleet management services," IEEE Transactions on Smart Grid, vol.
13, no. 4, pp. 3118-3132, 2022.
[4] Ahmad, N., Ghadi, Y., Adnan, M., & Ali, M., “Load forecasting techniques for
power system: Research challenges and survey”, IEEE Access, vol. 10, 2022.
[5] Hou, H., Liu, C., Wang, Q., Wu, X., Tang, J., Shi, Y., & Xie, C., “Review of load
forecasting based on artificial intelligence methodologies, models, and challenges”,
Electric Power Systems Research, vol. 210, pp. 108067, 2022.
[6] A. Srivastava, D. Steen, L. A. Tuan, O. Carlson et al., “Development of a DSO
support tool for congestion forecast,” IET Generation, Transmission &
Distribution, vol. 15, no. 23, pp. 3345–3359, 2021.
[7] Yang, D., Wang, W., Gueymard, C. A., Hong, T., Kleissl, J., Huang, J., et. al., “A
review of solar forecasting, its dependence on atmospheric sciences and
implications for grid integration: Towards carbon neutrality”, Renewable and
Sustainable Energy Reviews, vol. 161, pp. 112348, 2022.
[8] Mohammad, F., Kang, D. K., Ahmed, M. A., & Kim, Y. C., “Energy Demand Load
Forecasting for Electric Vehicle Charging Stations Network based on ConvLSTM
and BiConvLSTM Architectures”, IEEE Access, vol. 11, 2023.
[9] Wood, Allen J., Bruce F. Wollenberg, and Gerald B. Sheblé. Power generation,
operation, and control. John Wiley & Sons, 2013.
[10] Abur, Ali, and Antonio Gomez Exposito. Power system state estimation: theory
and implementation. CRC press, 2004.

82
TR-123 — IEEE PES Technical Roadmap

[11] Molzahn, Daniel K., et al. "A survey of distributed optimization and control
algorithms for electric power systems." IEEE Transactions on Smart Grid 8.6
(2017): 2941-2962.
[12] Kroposki, Benjamin, et al. "Autonomous energy grids: Controlling the future grid
with large amounts of distributed energy resources." IEEE Power and Energy
Magazine 18.6 (2020): 37-46.
[13] IEEE Guide for Distributed Energy Resources Management Systems (DERMS)
Functional Specification, IEEE 2030.11-2021.
[14] L. Strezoski, H. Padullaparti, F. Ding, and M. Baggu, “Integration of utility
distributed energy resource management system and aggregators for evolving
distribution system operators,” J. of Mod. Power Syst. Clean Energy, vol. 10, no.
2, Mar. 2022.
[15] L. Strezoski, “Distributed energy resource management systems—DERMS: State
of the art and how to move forward,” WIREs Energy and Environment, vol. 12, no.
1, e460, 2023. https://doi.org/10.1002/wene.460.
[16] DOE EERE Solar Energy Technologies Office Funding Notice: Operation and
Planning Tools for Inverter-Based Resource Management and Availability for
Future Power Systems (OPTIMA), April 20, 2023.
[17] M. Trujillo et al., “Operability of a power system with synchronous condensers and
grid-following inverters,” in Proc. 2022 PVSC.
[18] V. A. Evangelopoulos, I. I. Avramidis, and P. S. Georgilakis, “Flexibility services
management under uncertainties for power distribution systems: Stochastic
scheduling and predictive real-time dispatch,” IEEE Access, vol. 8, pp. 38855–
38871, 2020.
[19] F. Ding, W. Liu, J. MacDonald, J. Ogle, A. Pratt, A. Saha, J. Hagerman, and M.
Baggu, “Federated Architecture for Secure and Transactive Distributed Energy
Resource Management Solutions (FAST-DERMS),” National Renewable Energy
Laboratory, Golden, CO, Tech. Rep., NREL/TP-5D00-81566, Jan. 2022.
https://www.nrel.gov/docs/fy22osti/81566.pdf.
[20] Grid Modernization Laboratory Consortium (GMLC) Grid Architecture.
https://gmlc.doe.gov/projects/grid-architecture,
https://gridarchitecture.pnnl.gov/library.aspx
[21] B. Kroposki, “Basic research needs for autonomous energy grids: Summary report
of the workshop on autonomous energy grids,” National Renewable Energy
Laboratory, Golden, CO, Tech. Rep., NREL/TP-5D00-70428, Sept. 2017.
https://www.nrel.gov/docs/fy22osti/81566.pdf.
[22] A. Bernstein and E. Dall’Anese, “Real-time feedback-based optimization of
distribution grids: A unified approach,” IEEE Trans. Control of Netw. Syst., 2019.
[23] GridWise Transactive Energy Framework.
https://www.gridwiseac.org/pdfs/pnnl_22946_gwac_te_framework_july_2019_v1
_1.pdf.
[24] Knight, Mark R. Transactive energy systems research, development and
deployment roadmap. No. PNNL-26778. Pacific Northwest National Lab.(PNNL),
Richland, WA (United States), 2017.
[25] DOE OE Microgrid Program Strategy. https://www.energy.gov/oe/microgrid-
program-strategy.
83
TR-123 — IEEE PES Technical Roadmap

[26] Navigant. Emerging Microgrids Business Models. Research Brief Published Q1

2016.

5. Computation
5.1 Executive Summary
Computation techniques play a pivotal role across all facets of power system
planning and operation. The intricate nature of modern power grids, compounded by the
integration of cutting-edge renewable energy and smart grid technologies, underscores the
imperative for advanced computational solutions in grid applications.
This chapter begins by addressing the necessity of large-scale models essential for
power system analysis, optimization, and control, vital for both planning and operational
purposes. Subsequently, it delves into techniques designed to manage uncertainty and
variability within the contemporary power grid landscape.
Furthermore, the chapter explores parallel and distributed computing (PDC)
methods, focusing on their applications in steady-state analyses and dynamic simulations.
These PDC approaches serve to expedite computational workloads, thereby enhancing
efficiency.
Artificial intelligence (AI) and machine learning (ML) technologies are then
examined within the realm of computation. These innovations offer promising avenues for
accelerating numerous time-intensive computing tasks inherent in power system analysis
and simulation, laying the groundwork for prospective real-time applications.
Next, emerging technologies such as quantum computing, GPU computing, edge
computing, and cloud computing are presented as potential transformative forces in power
system computation, poised to revolutionize traditional approaches.
Lastly, the chapter examines various computational tools, including open-source
packages, synthetic test systems, and testbed and digital twins, all of which contribute to a
comprehensive, panoramic understanding of computational perspectives within power
systems.

5.2 Large-Scale Models and Uncertainty

5.2.1 Large-scale Modeling and Optimization
The power system size, the time resolution, and the model complexity all play
crucial roles in determining the scale of the optimization problem in power systems.

The size of the power system refers to the number of components, such as
generators, transmission lines, transformers, loads, and buses, within the system. A large
power system may involve a large-scale model of tens of thousands of buses, transmission
84
TR-123 — IEEE PES Technical Roadmap

lines, and generators. Power flow calculation, economic dispatch, and other types of
analysis become challenging and time-consuming. The characteristics of the system itself
determines this.

The time resolution means the granularity or interval at which the power system
behavior is modeled and analyzed. For example, time resolution could be represented by
hourly, sub-hourly (e.g., 15-minute intervals), or even finer time steps. Finer time
resolutions provide more detailed representations of system dynamics but also increase the
scale of the optimization problem. This is because additional variables and constraints are
introduced to capture the temporal aspects of power system operations, such as unit
commitment, economic dispatch, and load variations.

The model complexity is related to the level of detail and intricacy incorporated
into the power system model. Increased model accuracy can introduce additional
complexities in the optimization problem. Fine-grained models may include more detailed
representations of system components, their characteristics, and their interactions. This
heightened accuracy can lead to more complicated optimization formulations and
constraints, further amplifying the problem scale.

Current State
Modeling and analysis:
Modeling and analyzing large-scale power systems is a fundamental and critical
task for system owners, power developers, transmission companies, system operators,
consulting companies and trading companies. Examples include steady state and dynamic
models for state estimation, power flow study, contingency analysis, transient studies,
outage coordination study, unit commitment and economic dispatch, production cost
simulation in operation and planning and many more. The models and analyses are
typically done using commercially available software or in-house developed
software/tools.
There is limited work performed in electromagnetic transient (EMT) simulations,
but this is expected to change in future. There have been activities focused on acceleration
of EMT simulations. The scale of the system studied in the currently in EMT is in the order
of 1000s of buses, which is sufficient for most systems that is or will be studied in near
future. As the penetration of power electronics increases in the power grid, the size of the
system that needs to be studied is expected to grow in EMT simulations. For example, with
simplified distribution grid models in today’s phasor-domain transient stability (TS)
simulators, the power grid in United States has in the range of 100,000 buses. If more
detailed distribution grid models and/or inverter-based resources are modelled in detail, the
number of buses can easily reach millions. In such cases, it may not be simple to perform
splitting of the model only based on transmission lines to introduce parallelism and speed-
up. Hence, numerical methods are being researched upon to enable utilization of the
properties and features of the dynamics of the power grid to enable faster simulations [100,

85
TR-123 — IEEE PES Technical Roadmap

101, 102]. Additionally, parallelism in solvers within multi-core CPUs are being explored
for further speed-up in simulations [103, 104, 105, 106]. In addition to multi-core CPUs,
there have been recent research trends in using graphics processing unit (GPU) for scalable
simulations. It may assist with speed-up of certain types of power grids and/or inverter-
based resources [107, 108].

Optimization algorithms:
Solving large-scale optimization problems can usually be challenging. Therefore,
different algorithms have been proposed to exploit problem structures, reduce
computational requirements, and provide near-optimal solutions within reasonable time
frames. The typical algorithms include constraint generation, constraint generation,
column-and-constraint generation, primal-dual algorithm, etc.
1) Column generation
Column Generation is a technique used to solve problems with a large number of
variables. It is particularly effective for problems formulated as linear programming or
integer programming models [1]. The idea behind column generation is to start with a
reduced set of variables (columns) and iteratively generate new variables that improve the
solution until an optimal or near-optimal solution is obtained. Column Generation is
effective in situations where the number of potential variables is exponentially large or
when generating all variables upfront is not computationally feasible. It enables efficient
exploration by adding variables that are most relevant to the problem’s optimal solution.
2) Constraint generation
Constraint Generation, also known as Benders’ Decomposition, is a technique used
to solve problems with a large number of constraints [2]. It is particularly useful for
problems with mixed-integer programming formulations. Constraint Generation aims to
iteratively generate and add new constraints to the problem to improve the solution quality.
Constraint Generation helps address computational challenges by adding constraints
progressively, focusing on the most important constraints that significantly affect the
solution quality. It allows for more efficient exploration of the feasible region and can
provide tighter bounds and improved solution accuracy.
3) Column-and-constraint generation
The Column-and-Constraint Generation algorithm is a combination of Column
Generation and Constraint Generation techniques [3]. It is used to solve optimization
problems with both a large number of variables and a large number of constraints
simultaneously. By simultaneously generating new variables and constraints, it is capable
of efficient exploration of both the variable space and the constraint space. This approach
can lead to significant computational savings and improved solution quality in large-scale
optimization problems.
4) Primal-dual algorithm
Primal-dual algorithms solve the primal and dual problems simultaneously,
updating the primal and dual variables iteratively until convergence [4]. These algorithms
typically involve steps that update the primal variables based on the current dual variables
86
TR-123 — IEEE PES Technical Roadmap

(primal update) and steps that update the dual variables based on the current primal
variables (dual update). These algorithms exploit the duality theory in optimization to find
solutions that are optimal with respect to both the primal and dual objectives. They are
often used for solving large-scale optimization problems with complex constraints, such as
linear programming, quadratic programming, and convex programming.

Trends and future directions

Modeling and analysis:

A few trending future directions are listed as follows:
 Integrated operation and planning models: the node-breaker model used in operation
and bus-branch model used in planning may be integrated into one unified model to
ensure model consistency and facilitate seamless transition between studies.
 Increased coordinated studies across sectors: examples include generation-transmission
co-planning, transmission-distribution co-simulation, electric-natural gas coordination,
electric-transportation-water-environment nexus, etc.
 Increased computational needs: expansion of system footprint, rapid deployment of
renewable and energy storage assets, uncertainties in energy policies and the need for
grid resilience under extreme weather conditions increase the scope of traditional
engineering studies.
 There is an increasing need for electromagnetic transient (EMT) simulations in dynamic
simulations with increasing penetration of power electronics in the power grid. This
poses significant computational burden in terms of the small time step and higher fidelity
of models used in these simulations. The challenges are discussed in detail in [109].

Optimization algorithms:
Here are some potential future directions in large-scale modeling and optimization.
 Scalability and Efficiency: When the optimization problem becomes more complicated,
there is a need for more scalable and efficient algorithms. Future research can focus on
developing optimization methods that can handle even larger problem sizes while
maintaining computational efficiency.
 Integration of Machine Learning: Machine learning techniques, such as deep learning
and reinforcement learning, have shown great power in various domains. Integrating
machine learning technologies with large-scale optimization can lead to more intelligent
and adaptive optimization algorithms
 Multi-Objective and Multi-Level Optimization: Many real-world problems involve
multiple conflicting objectives or multiple levels of decision-making. These problems
usually require high computational resources. Therefore, future research can also focus
on developing efficient algorithms and methods for solving large-scale multi-objective
and multi-level optimization problems, considering the trade-offs between conflicting
objectives and hierarchical decision-making structures.

87
TR-123 — IEEE PES Technical Roadmap

Challenges
Modeling and analysis:
A few challenges are explained below:
 Data and model sharing: the modeling of large systems alone is a huge undertaking. It
typically involves a lengthy and comprehensive model building process, requires
coordinated efforts among multiple entities including various stakeholders and
public/private data vendors, and mandates non-disclosure agreements to protect critical
energy infrastructure information (CEII).
 Intensive computation and long analysis time: Although these engineering studies are
typically supported by commercially available software and tools, it is computationally
intensive to perform many of these studies which take hours to weeks. The unification
of models, increased scope and anticipated coordination across sectors will further
exacerbate the computational burden.
 Use of EMT simulations: This is expected to make the interconnection studies process
longer and challenging with respect to data collection, model validation and preparation,
and compute time needed [110].

Optimization algorithms:
Some challenges still exist.
 Scalability: Developing algorithms and techniques that can effectively scale to even
larger problem sizes remains challenging.
 Handling non-convexity: Developing efficient algorithms that can handle non-
convexity and provide good-quality solutions is an ongoing challenge.

Potential Solutions
Modeling and analysis:
Possible solutions are as follows:
 Advanced computing techniques such as parallel computing, cloud computing, GPU
computing may be adopted to alleviate the computational challenges.
 New theories, analytical methods and tools, which are not based on conventional
knowledge (textbook knowledge), may provide innovative and transformative ways to
solve the problems.
 Promote the development and adoption of open-source tools such as those developed by
the US Department of Energy (DOE) labs, the Electric Power Research Institute (EPRI)
or the wider global research community. Extensive testing and evaluation by third-
parties on realistic datasets will be necessary to ensure the proposed tool satisfy the
requirements for industry-grade tools. Large-scale publicly available synthetic data sets
may be used in such testing and evaluation procedures.

Optimization algorithms:

88
TR-123 — IEEE PES Technical Roadmap

The following are some possible solutions.

 Integrating domain expertise and problem-specific characteristics into optimization
algorithms can lead to more efficient solutions.
 Developing algorithms that can effectively use parallel and distributed resources.
 Utilizing machine learning technologies to train a solution model to obtain a near
optimal solution. This strategy can significantly improve computational efficiency.

5.2.2 Computation Techniques Under the Paradigm of Uncertainty and

Variability
Real-time operation and long-term planning are two important tasks in large-scale
power systems. However, the simulated conditions may differ from the reality due to
uncertainties. There are various sources of uncertainty in the power systems. The forced
outages caused by equipment failure or faults are hard to predict and yet need to be
considered in the power system planning stage to ensure reliability and security. Extreme
weather events often arise from the interaction of deterministic physical processes with
stochastic variations in the climate system, making it challenging to precisely predict
extreme weather and the induced power outages. Additionally, uncertain changes in the
natural and economic environment in the future will greatly influence the electricity load.
In addition, there is more and more renewable energy generation integrated into large-scale
power systems. Since the output power of renewable energy is influenced by weather
conditions, its actual output presents significant uncertainties. These uncertain factors have
a significant impact on the operation and planning of power systems.

Current State
For dealing with the uncertainties of large-scale power system optimization, there
are mainly four uncertainty modeling methods, i.e., stochastic optimization, chance-
constrained optimization, robust optimization, and distributionally robust optimization.

1) Stochastic optimization
Based on the response to risk that results from the uncertainty, there are mainly two
types of stochastic optimization methods. The first one is risk-neutral two-stage stochastic
optimization, whose first-stage objective is deterministic, and the second-stage objective is
an expected cost. Accurately representing the expected second-stage objective is
challenging. To overcome the challenge, sample average approximation is often used to
represent the second-stage cost. However, many samples are needed to ensure accuracy,
which results in large and difficult problems to solve. Various alternative methods are
proposed to solve this difficulty, like Lagrange relaxation, cutting plane methods, Bender’s
decomposition, stochastic dual dynamic programming, and progressive hedging.
The other model is risk-averse two-stage stochastic optimization [5]. This model
focuses on minimizing the negative impacts, where the risk can be quantified by standard
deviation, value-at-risk, etc.
89
TR-123 — IEEE PES Technical Roadmap

Stochastic optimization has been widely used to solve unit commitment problems,
transmission expansion planning problems, hydro-thermal power system planning
problems, and energy procurement problems. New decomposition algorithm has been
proposed to solve multistage stochastic unit commitment problem [6].The advantages of
using stochastic programming to solve grid planning problems have been discussed [7].
The multi-stage stochastic programming model for hydropower planning problem can be
solved using stochastic dual dynamic programming [8]. In terms of the energy procurement
problem, a real-world case study was carried out based on a multi-stage stochastic
programming model [9].

2) Chance-constrained optimization
The chance-constrained optimization, i.e., probabilistic constrained optimization,
is an optimization problem in which the constraints contain random parameters and are
satisfied under certain probability. The chance-constrained optimization can be divided
into two categories: single chance constraints and joint chance constraints. If the constraints
of optimization model are considered individually, the optimization model is single chance
constraints, otherwise, it is joint chance constraints. For measuring the size of the violation,
CVaR approximation is used to approximate the chance-constrained optimization method
[10-12]. Additionally, there is a more general risk function, i.e., weighted chance
constraints [13]. When the probability distribution of constraints is unknown, the
distributionally robust chance-constrained can be applied.
In terms of solving chance-constrained optimization problems, the potential
solution is to evaluate the violation probabilities. The common method is to reduce the
chance-constrained problems to tractable deterministic optimization problems, which are
then solved to obtain optimal results. There are mainly four methods to realize the
reduction.
The first method approximates joint chance constraints with single chance
constraints. For example, the Bonferroni approximation is used to split the joint violation
probability equally across all constraints. However, the sub-optimal results may be
obtained by the simple method. The second method is sample average approximation, and
the idea is to replace the random parameters with some samples and allow a certain
percentage of samples to violate constraints. Although flexibility is allowed in second-
stage modeling by the method, there is no guarantee that the chance constraints will be
satisfied due to the limited number of samples considered [14]. The third method is the
scenario approach, which is mainly used to solve convex optimization problems [15]. The
scenario approach has a better-solving performance than sample average approximation in
a large-scale problem. However, in power system problems, it often gives conservative
results and cannot deal with the second-stage variables related to scenarios. The last
method is moment-based reformulations relying on affine recourse policies to handle the
uncertainty.
The chance-constrained optimization method has been used to dispatch generators
where the system constraints are satisfied with a specific probability considering the
uncertainty of demand and renewable energies [16]. Multi-area reserve dimensioning
90
TR-123 — IEEE PES Technical Roadmap

problem can also be modeled by the chance-constrained optimization method [17].

Additionally, a semidefinite relaxation of a chance-constrained AC-OPF was proposed in
reference [18], which aims to ensure global optimality.
3) Robust optimization
The robust optimization model is applied to deal with situations where the
distribution of random parameters is unknown, or the outcome is unbearable caused by
uncertainty. The idea of robust optimization model is to minimize the objective function of
the worst-case in the uncertainty set. Quantifying the uncertainty to obtain the worst-case
cost is difficult to solve. The most commonly used algorithm is the Column-and-Constraint
Generation algorithm which uses a master-adversarial problem framework to identify the
worst-case scenario.
Considering the uncertain load and renewable generation, a security-constrained
optimal power flow (SCOPF) method was proposed to balance the economy [19]. The
robust optimization method can also be applied to improve the power system reliability
under N-K outages [20]. Additionally, robust ptimization introduces resilience against
missing features in energy forecasting [21].

4) Distributionally robust optimization

In the situation that the probability distribution is unknown, the worst-case
distribution in the ambiguity set can be endogenously identified and the corresponding
expected cost can be minimized by distributionally robust optimization. Affine recourse
policies are the commonly used methods to solve distributionally robust optimization
problems. Additionally, the ambiguity set can be represented by the approach based on
moment, and the Wasserstein metric [22].
The distributionally robust optimization model can deal with OPF problems.
Reference [23] proposed a distributionally robust OPF, which can handle the ambiguity of
the first and second moments. In [24], the dynamic line rating is integrated. Considering
the uncertainty of renewable generation and reserve provided by flexible loads, a moment-
based distributionally robust optimal power flow model is developed [25]. A Wasserstein
integrated model is proposed, which considers the uncertain renewable generation [26].
Similar to the model in [26], a distributionally robust optimization model with joint chance
constraints is proposed in [27].
Although these four modeling methods are effective in some scenarios, there are
still many challenges and future directions in uncertainty modeling of large-scale power
system optimization.

Future State
Here are some potential future directions.
 Consider the influence of unexpected extreme weather in the renewable energy
uncertainty model. Extreme weather events usually lead to huge loss of resources and/or
transmission/distribution of the power system, thus considering these events is expected
to hedge their impact. Also, modeling extreme events should be linked with weather
91
TR-123 — IEEE PES Technical Roadmap

forecast in the future for a better estimation of the impact of uncertainty and extremeness
from weather and climate change.
 Model the interaction between multiple energy systems under multiple uncertainties.
Different energy systems are usually correlated and uncertainty can come from different
sources, so modeling different uncertainties in a multiple energy system is also of
interest.

Challenges
The following are some challenges.
 Data Availability and Quality: Uncertainty and variability analysis most often rely on
data. However, acquiring reliable and relevant data can be a challenge. Data may be
limited, incomplete, or subject to measurement errors.
 Computational Time: Some uncertainty quantification and optimization techniques can
be computationally intensive. Performing multiple simulations, sampling from
probability distributions, or solving complex optimization problems can require
significant computational resources and time.
 Interdependent Systems: Some systems are interdependent, and uncertainties from
different systems may be correlated. Efficiently modeling these correlated uncertainties
as well as maintaining the computational efficiency is still a challenge.
 Interconnected Systems: Such as natural gas/electric systems where events on one
system impacts the performance of the other system. Modeling this interconnectedness
is critical as new energy systems emerge.

Potential Solutions
The possible solutions are discussed as follows.
 Propose specific strategies to deal with data scarcity and improve the data quality.
 Improve tractability of existing algorithms or develop new algorithms and techniques to
reduce computational time such as through approximation and linearization method,
decomposition techniques, parallel computing, etc.
 Quantify the associated risk resulting from uncertainty propagating between the electric
grid and interconnected systems.
 Improve the understanding of interdependence of weather, generation and demand, and
the tail risk of events.

5.3 Parallel and Distributed Computing

Summary
Parallel computing and distributed computing are techniques for accelerating
computation problems by decomposing computation workload into subtasks that can run
simultaneously [28]. Parallel computing refers to breaking down a larger problem into
smaller independent parts and executing them on multiple cores or processors on a shared-
memory computer. Distributed computing refers to the use of independent computers that
92
TR-123 — IEEE PES Technical Roadmap

are networked to collaborate on a common computing task. Both techniques promise

improved performance, increased efficiency, and the capability to solve very large-scale
problems.
The adoption of parallel and distributed computing is not without challenges. Some
computational problems are inherently parallel or can be dispatched to multiple cores for
so-called “embarrassingly parallel” execution. Others necessitate a meticulous redesign to
be amenable to parallel processing. To achieve effective parallelization, a profound
understanding is required of both the domain-specific problem and the underlying
computing hardware. In other words, naïvely partitioning a task into smaller subtasks does
not necessarily guarantee efficiency; it requires theory and practice to develop
parallelizable algorithms, determine the optimal granularity of tasks, manage data
dependencies, ensure synchronization, and minimize communication overheads. For
power system researchers, embracing these techniques demands not the familiarity with
algorithms but also a holistic view of translating intricate power system problems into
parallelizable computations.

Building on such understanding, it is crucial to recognize the hierarchical nature of

computing software, especially in application domains like power systems. One example
is a practical power flow program, which relies on sophisticated libraries for solving sparse
linear equations. This introduces multiple layers of potential parallelization. At a higher
level, multiprocessing can be employed to run separate instances of the power flow
program concurrently for independent scenarios. Within each execution, the solvers
themselves might be architected to leverage multiple cores for accelerated solutions. This
multi-tiered approach highlights the nuanced strategy in adapting to PDC, which is not just
at the macro level of the application but also within the foundational components upon
which the applications are built.

Parallel and distributed computing has gained momentum in power system

analysis, planning and optimization, and real-time operations. The computational patterns
in these topics are distinct, so the specific PDC applicability may differ.

5.3.1 The current state of PDC for power system analysis

Steady-state analysis
1) Power flow analysis
Power flow analysis determines the voltage magnitudes and angles, as well as the
active and reactive power flow, throughout the network. Mathematically, it is a root-finding
problem for a set of nonlinear power injection equations. The problems are typically solved
iteratively using the Newton method, but emerging methods such as holomorphic
embedding also apply. In a Newton-based iterative process, the computational loads are
concentrated on evaluating the residual equations and solving sparse linear equations. The
former has been tackled by data parallelism using single-instruction multiple data [29], and
the latter has been handled by underlying libraries such as PARDISO [30] on CPUs and
93
TR-123 — IEEE PES Technical Roadmap

preconditioned iterative methods on GPUs [31]. Nonetheless, it is worth noting that the
parallelized sparse linear solvers are not “slam dunk” winners due to the lack of structure
in power flow Jacobians. In some cases, the serial, single-threaded KLU [32] and
CXSparse [33], [34] can yield top performance.

2) Optimal power flow

OPF problems aim to optimize a certain objective, such as cost minimization while
satisfying system constraints. OPF with DC network constraints can be formulated as
quadratic optimization (or mixed-integer quadratic) problems that can leverage highly
optimized solvers. OPF with AC network constraints is nonconvex, which makes it
difficult to obtain global solutions within a reasonable timeframe. Techniques to
decompose problems have been proposed, and they can be categorized into temporal and
spatial decomposition. Temporal decomposition splits a long-duration optimization
problem into multiple small subproblems, each having extra constraints to enforce the
starting and end states. Spatial decomposition methods divide systems into zones or areas
and solve them concurrently. Among the most widely used spatial decomposition methods
are Benders decomposition [35] and the alternating direction method of multipliers
(ADMM) [36]. Although ADMM is not theoretically guaranteed for convergence, practical
successes have been observed in many nonconvex optimization problems in power
systems.

Diving deeper, many solvers used for OPF problems can be parallelized at the
algorithmic level. These include the interior-point methods and gradient-based methods. A
mix of task parallelism and data parallelism is used for accelerating matrix factorization
and matrix-vector multiplication. Open-source solvers such as HIGHs can handle parallel
solutions of sparse linear programming. Commercial solvers such as Gurobi and CPLEX
can perform parallel and distributed solutions of mix-integer optimization.

3) State estimation
State estimation determines the state of power systems from noisy and possibly
incomplete measurements. State estimation problems are formulated as optimization
problems, so the same principles for parallelizing optimization problems apply. In addition,
decomposition can be performed based on the network topology, so that regions can be
individually optimized and interfaced at the boundary. Also, recent work has used GPUs
for matrix operations used in state estimation [37].

4) Contingency analysis
Contingency analysis assesses the impact of device outages on system security. AC
Contingency analysis requires a complete solution of power flow for each scenario. Since
the scenarios are independent, multiprocessing-based parallelization can be used to solve
scenarios on multiple cores. DC contingency analysis can use shift factor-based methods
for studying line outages. Fine-grained methods have been developed to effectively
compute such distribution factors under multiple contingencies [38].

5) Expansion planning and renewable energy integration studies

94
TR-123 — IEEE PES Technical Roadmap

Both studies involve the study of optimal placement and sizing of new
infrastructure equipment, such as lines and renewable energy sites. They are formulated as
optimization problems, so the parallelization techniques for optimization problems apply.

System dynamic analysis

1) Transient stability simulation
Transient stability simulation is a practical method for understanding stability
following large disturbances. Transient stability simulation uses device models in the
timescale of milliseconds and assumes a quasi-steady state for the network. Given the
timescale, these simulations can be used for interconnection-level studies. Stability
simulation problems are formulated as differential-algebraic equations, but the algebraic
equations can be given in either power injection or current injection forms. The solution
methods can also be categorized into partitioned and simultaneous approaches. In
commercial software, the current injection model with a fixed-step, partitioned solution
method is common. In open-source software, the same formulation can be found in the
Power System Toolbox [39], while the power injection model with the simultaneous
solution approach can be found in the Power System Toolbox [40], Dome [41] , and the
CURENT ANDES simulator [42].

Existing approaches to parallelize transient stability simulation emphasize

accelerating the solution of sparse linear equations, which can also be combined with
spatial decomposition to reduce the system size. For example, GPU-based LU factorization
is coupled with topology-based relaxation [43]. Nonetheless, GPU-based methods will
encounter large overhead due to frequent data copying between the CPU and CPU. On the
other hand, temporal decomposition, such as the Parareal method [44], [45], is adopted by
using coarse-fine-grained time stamps to predict and refine the simulation.

2) Electromagnetic transient simulation

Electromagnetic transient simulation is crucial for analyzing fast transient
phenomena, such as switching transients, lightning surges, and harmonic interactions.
EMT simulation focuses on the microsecond timescale and captures the intricate interplay
between system components. Given its high-resolution requirement, EMT simulation
mandates extremely small step sizes, which form the primary bottleneck [46]. The
traditional approach for EMT acceleration is to partition the system by leveraging the delay
due to the traveling wave, in combination with a fixed time step integration. Emerging
techniques have explored new formulations such as dynamic phasor [47] and shift
frequency analysis [48] to enable the use of larger step sizes. An open-source
implementation of shift frequency analysis is available in DPSim[49]. Also, dedicated
embedded systems, such as FPGAs, have been applied in commercial products for
accelerating simulations of very small step sizes. Open-source EMT simulators, such as
ParaEMT [50], also used message passing for system partitioning and exploited the
bordered block-diagonal matrix properties to accelerate the solution of nodal equations.

3) Small-signal stability analysis

95
TR-123 — IEEE PES Technical Roadmap

Small-signal stability analysis, also known as modal analysis, studies system

response to small disturbances around an equilibrium point using linearized models. Such
stability property is determined by eigenvalue analysis, so accelerating the eigenvalue
computation is crucial for performance gains. Parallel computing of the eigenvalues is
reported by the applied mathematics community, such as block eigen-solvers [51]. After
obtaining the state matrix at the equilibrium point, small-signal analysis for power systems
modeled by differential-algebraic equations can readily leverage the advancements in
parallel eigenvalue solvers. These parallel eigenvalue solvers can be readily used by
existing software that can extract the state matrix.

Monitoring, control, and data analysis

1) Real-time monitoring, synchrophasor applications, and wide-area control
Wide-area monitoring and control technologies enabled by communication
networks and sensors have provided capabilities to gain real-time situational awareness of
power systems. Sensor data coming from multiple data sources are significant in volume
and require timely processing. Parallel processing of independent data streams can be
executed on multiple cores and aggregated for acceleration. Analysis of stored data sets
can leverage underlying software infrastructure, such as time-series databases and cache
databases [52], for acceleration. On top of these, application-specific algorithms are
required to properly leverage PDC capabilities.

2) Distributed ledger and blockchain systems

Blockchain technology, also known as distributed ledger, stores and processes data
across the network. It enables decentralized data generation, storage, verification, and
computing. While blockchains are not the standard form of synchronized distributed
computing, they enable new applications [53] such as energy transactions in a secure and
trustworthy manner.

5.3.2 Future State

Future directions for PDC are specific to the application, but the same general trends
apply:
 Heterogeneous Computing: The future of parallel and distributed computing will
involve the integration of heterogeneous computing resources, such as CPUs, GPUs,
FPGAs, and specialized accelerators. Designing algorithms and frameworks that
efficiently use diverse hardware architectures will be a key focus.
 Adaptive Granularity: Research into adaptive algorithms that can dynamically adjust
their granularity based on real-time computational loads, ensuring optimal use of
computational resources across diverse scenarios.
 Standardization and Open Platforms: Advocate for standardized platforms and
benchmarks that allow researchers and industry professionals to compare different
parallel computing methods, promoting transparency, replicability, and continuous
improvement in the field.
 As quantum computing matures, explore its potential for solving certain
computationally intensive power system problems, like optimization or eigenvalue
96
TR-123 — IEEE PES Technical Roadmap

analysis, which could revolutionize the speed and capability of power system
computations.

5.3.3 Challenges
Parallel and distributed computing require new paradigms in computational
thinking and algorithms to properly leverage new hardware. They pose significant
complexity in maintaining data consistency and integrity when decomposing a large
problem into subproblems. Specific challenges include:

 Complexity in Algorithm Design: Designing algorithms that are inherently parallel or

can be efficiently parallelized is intricate. Not all problems can be readily divided, and
challenges exist to ensure that tasks are balanced, and the workload is evenly
distributed to minimize wait time.
 Latency and Communication Overheads: As tasks are distributed across cores or
nodes, memory and communication latency will become impactful. The time taken to
transfer data or messages between tasks can sometimes negate the benefits of
parallelization, especially in real-time applications.

5.3.4 Potential Solutions

The following are some possible solutions.
 Capitalize the hierarchical structure of software and leverage efficient libraries
whenever possible. Leveraging efficient libraries is paramount as they often
encapsulate best practices and refined algorithms, and are optimized for specific
hardware architectures, ensuring maximal performance.
 Invest in new formulations and solution methods that provide fine-grained trade-offs
between accuracy and computational complexity. One example is the use of variable-
order, variable-step methods to solve differential-algebraic equations. Such methods
allow one to adapt to the details of the model when disturbances happen while being
able to increase the step size when the system stabilizes. By providing flexibility,
computation tools can be optimized for diverse scenarios.
 Explore machine learning-based solutions for pattern discovery and model reduction,
aiming for better use of parallel processing devices, such as GPUs.

5.4 Artificial Intelligence (AI) for Power System Computation

AI is revolutionizing power grid computation, offering advanced capabilities to
enhance system operations, optimization, and decision-making. Narrow AI focuses on
specific tasks within a defined domain, such as load forecasting and fault detection,
enabling accurate predictions and efficient fault identification in power systems. General
AI aims to replicate human-like intelligence across various tasks and domains, holding the
potential for adaptive, autonomous, and advanced optimization capabilities in power grid
computation. Power system computation involves solving algebraic equations (AEs),
ordinary differential equations (ODEs), and differential algebraic equations (DAEs), which
97
TR-123 — IEEE PES Technical Roadmap

inherently encompasses the processes of simplifying, solving, and analyzing these

equations. Technically speaking, AI involves power system computation in the following
specific ways.

5.4.1 AI-based data initialization and scenario generation

Current State
AI techniques can be applied to generate intelligent and informed initial conditions
or estimates for power system computation. By analyzing historical data, system
parameters, and environmental factors, AI algorithms can predict and provide accurate
initial conditions, such as load profiles, generator statuses, and network topology. This
initialization process helps improve the convergence and efficiency of subsequent
optimization or simulation, reducing the time and computational resources required for
achieving optimal results.

Future State
Data initialization and scenario generation may have the following future
directions:
 To provide scalable and credible solutions with minimum available training
dataset.
 To employ model-based physics to assist with the training process.
 To engage with digital twin technology to enhance data generation.
 To develop industrial standard to data quality for the perspective of power system
computing.

Challenges
One of the challenges in AI-based data initialization and scenario generation is the
availability of high-quality and diverse data for training accurate models. Insufficient or
unreliable data can hinder the effectiveness of AI algorithms in generating reliable initial
conditions for power system computation. Addressing data quality issues, improving data
standardization, and ensuring data accessibility are essential to overcome this challenge.

Potential Solutions
Emphasizing the data quality, collection, and availability is crucial. Investing in
data infrastructure, data management techniques, and data sharing platforms can ensure
access to high-quality, diverse, and reliable data for training AI models. Collaborations
between industry stakeholders, researchers, and regulatory bodies can facilitate data
sharing initiatives and promote standardized data formats.

5.4.2 Accelerating power system computation

98
TR-123 — IEEE PES Technical Roadmap

Current State
AI techniques have been drastically adopted in power system computation with the
following features:

1) Simplifying iteration in optimization or time-domain simulation

AI algorithms can simplify the iterative processes involved in optimization or time-
domain simulation tasks. By leveraging machine learning or optimization techniques, AI
can develop intelligent algorithms that converge more quickly or require fewer iterations
to reach an optimal solution or accurate simulation results. These algorithms can learn from
previous computations, adapt to system dynamics, and make informed decisions to
streamline the iteration process, saving computational time and resources.

2) Complete replacement of existing solvers

Arguably, AI has the potential to completely replace traditional solvers used in
power system computation, such as power flow, optimization, and time-domain simulation.
AI-based solvers leverage advanced machine learning or deep learning models to capture
complex system behaviors and relationships. By training on large datasets, these models
can learn and generalize from diverse power system scenarios, providing accurate and
efficient solutions without relying on traditional solver approaches. This complete
replacement by AI-based solvers can significantly enhance computation speed and
accuracy.

3) Effective reduction of computation scenarios

AI techniques can effectively reduce the number of computation scenarios in power
system analysis. By using machine learning algorithms, AI can identify critical scenarios
that have the most significant impact on power system behavior, such as contingencies or
extreme conditions. This reduction in computation scenarios optimizes the use of
computational resources, saving time and effort while maintaining accurate results.

4) Accelerating computation through efficient resource allocation

AI can optimize the allocation of computational resources, especially in server-side
environments. By analyzing real-time system conditions, computational workloads, and
priorities, AI algorithms can dynamically allocate computational resources to tasks that
require immediate attention or have high computational demands. This intelligent resource
allocation ensures efficient use of server-side resources, reduces idle time, and accelerates
power system computation.

Future State
AI applications in power systems have been well discussed in academic research
and implemented in some pilot projects, but AI needs to be adopted in the real-world
applications. The following directions may include:

99
TR-123 — IEEE PES Technical Roadmap

 To investigate industrial acceptance of AI applications with proper tolerance and

solid controls to improve trust.
 To engage with industrial applications, especially control room applications,
possibly through consortium with significant involvement in power system
operators.

Challenges
While AI algorithms can simplify the iteration process in optimization or simulation
tasks, their black-box nature can make it difficult to interpret and explain the decision-
making process. Achieving interpretability and explainability is crucial to gain trust and
acceptance in AI-driven power system computation. Developing techniques to understand
and interpret AI model outputs, especially in complex optimization or simulation scenarios,
is a challenge that needs to be addressed.
Completely replacing existing solvers with AI-based solvers requires robustness
and generalization of AI models. Training AI models on historical data may not adequately
capture evolving power system dynamics and emerging technologies. Ensuring the
robustness and generalization of AI models across different system configurations,
changing conditions, and unseen scenarios is essential to achieve reliable and accurate
power system computation.
For the real-time operation in power systems, it is crucial to ensure timely decision-
making to guarantee system stability and reliability. This requires AI models capable of
generating results within the required time frame. It further requires the AI models to be
scalable and acceptable to large-scale and realistic power systems. To that end, there are
big challenges during the application process that balances accuracy and real-time
performance.

Potential Solutions
Developing interpretability techniques and tools is essential for understanding the
decision-making process of AI models. Researchers can focus on developing explainable
AI frameworks that provide insights into how AI algorithms arrive at their conclusions.
This can enhance transparency, trust, and acceptance of AI-driven results in power system
computation.
AI models need to be robust and adaptable to evolving power system dynamics,
technological advancements, and changing conditions. Research efforts should focus on
developing models that can handle unseen scenarios, adapt to system changes, and learn
from real-time data. Techniques such as transfer learning, reinforcement learning, and
continual learning can enable AI models to generalize and perform effectively in diverse
power system environments.
To meet real-time requirements, the most direct approach is to accelerate
computation process for faster result generation. One way to achieve this is by optimizing
the architecture and parameters of AI models through techniques such as model
compression, parameter pruning, or quantization to improve computational efficiency.
Another highly effective way is to leverage distributed computing frameworks and parallel
100
TR-123 — IEEE PES Technical Roadmap

processing to handle multiple computations simultaneously, significantly improving

computation speed and scalability. Additionally, streamlining data processing pipelines
and leveraging approximation techniques can also minimize processing time. Cloud
platforms offer scalability by providing resources for efficient computation. These
strategies collectively contribute to enhancing the real-time performance and scalability of
AI models used in power system computation.

5.4.3 AI for problems without analytical models

Current State
Another important application of AI in power systems is to address problems that
do not have a readily available analytical model. This is typically for statistical oriented
problems such as energy forecast, weather forecast, equipment failure modeling, consumer
behavior modeling, etc. Some of these problems may have a long history in power systems,
while others may be new emerging problems.

Future State
Future directions may include the following:
 To model social and human behavior impact at the electricity consumption side.
 To model extreme weather (in the context to power grid) impact to system
components.
 To develop credible benchmark systems for new challenges like consumer
behavior, extreme weather, etc.

Challenges
The challenge is that there is no readily available or broadly accepted benchmarks
for emerging challenges like consumer behavior and extreme weather models.

Potential Solutions
The PES community may engage with other societies like meteorology or sociology
to develop joint task forces for broadly acceptable benchmark systems.

5.5 Emerging Computational Technologies

There are a number of emerging technologies for power system computation such
as quantum computing, GPU computing, cloud computing, edge computing, and block-
chain computing. These new technologies will be elaborated in this section.

5.5.1 Quantum computing

Current State

101
TR-123 — IEEE PES Technical Roadmap

The application of quantum computing in power system engineering is mostly in

the theoretical or very early experimental stages [54-56, 111-114]. Traditional
computational methods are still primarily used for tasks such as power flow analysis,
optimization, planning, and grid reliability. However, the potential of quantum computing
to vastly outperform classical computers for certain tasks has been recognized, and
preliminary research has been conducted on its possible uses in power system optimization,
simulation, fault detection, and grid security.
Future State
The future of quantum computing in power system engineering could be
revolutionary. Quantum algorithms have the potential to significantly enhance the
performance of critical tasks such as real-time system optimization, dynamic security
assessment, and power flow calculations. As quantum computers become more powerful
and accessible, we may see a shift from traditional computational methods to quantum-
enhanced algorithms. This could result in more efficient power generation and
distribution, quicker fault detection and response, and improved grid security and
resilience. There may be exploration on use of quantum computers on cloud for edge
computing purposes and/or for small-scale optimization in the field. The focus currently
is on showcasing feasibility of using quantum computers for developing applications in
the power grid.
Challenges
The primary challenge in applying quantum computing to power system
engineering is the current immaturity of quantum technology. As of now, only small-scale
quantum computers are available, and these are prone to high error rates and issues with
qubit stability. Additionally, developing quantum algorithms for power system
applications is a complex task that requires a deep understanding of both power engineering
and quantum physics.

Potential Solutions
Efforts are being made to improve the scalability and stability of quantum
computers, with advancements being made in areas like error correction and qubit
coherence times. There are also increasing efforts to develop quantum algorithms for power
system applications, but this is still a relatively new field.
However, significant gaps remain. Bridging the gap between quantum physics and
power engineering is a major challenge, as it requires training a new generation of
engineers who are versed in both fields. Additionally, while theoretical analyses have
shown that quantum algorithms could enhance certain power system tasks, more research
is needed to determine how these theories can be translated into practical applications in
the real world. Finally, developing a suite of quantum algorithms for basic functionalities
like linear solvers, optimizers, etc. will enable domain scientists to develop quantum
algorithms for power grid applications. This is an ongoing development [115-117].
Furthermore, there is a wider need to understand the operability of quantum technologies

102
TR-123 — IEEE PES Technical Roadmap

in the power grid that include computing, communication, security, and sensing as it offers
enhanced security [118].

5.5.2 GPU computing

Current State
Graphics Processing Units (GPU) are specialized computing hardware originally
designed for rendering images and graphics in video games and multimedia applications.
Owing to its highly parallel processing capability, GPUs have been applied in a wide
variety of applications including machine learning, computer vision, natural language
processing, scientific computing, etc. In the power system field, GPUs have been used in
conjunction with CPU for various scientific computing applications including parallel
computing, real-time simulation, grid operation and optimization, etc.
For scientific computing in power systems, GPUs have been primarily and
extensively studied to analyze single and batch power flow problems [57-60] and closely
related problems such as static security analysis [61] and contingency screening [62] to
achieve parallel computing and reduce computation time. Researchers have also attempted
to use GPUs in state estimation [63], small-signal stability analysis [64], transient stability
analysis [65, 108], optimal power flow [66]. In the literature, high parallelism and
significant speedup have been demonstrated in these works.
Future State
Potential future directions may include the following:
 Evaluate the performance scalability and energy efficiency in more advanced
GPU/CPU hardware, GPU clusters, cloud computing and new computing platforms
including quantum processors.
 Develop optimized solutions with hardware and software co-design.
 Develop solutions for edge devices to enable localized processing and decision-
making for critical tasks.
 Further explore the values of GPUs in other power system applications such as unit
commitment, dynamic state estimation, fault detection, cyber physical system
security, co-simulation of transmission-distribution, EMT simulations, etc.

Challenges
A few identified challenges are summarized below:
 Algorithm Parallelization: some power system algorithms may not naturally lend
themselves to parallelization and new approaches are needed.
 Data Transfer Bottlenecks: most solutions are designed for hybrid GPU/CPU
hardware and data transfer between GPU and CPU is a significant bottleneck,
especially for applications with large datasets.

103
TR-123 — IEEE PES Technical Roadmap

 GPU Memory Limitation: GPUs generally have less memory available than CPUs
and its memory bandwidth is a significant limiting factor for memory-bound
problems.
 Performance evaluation: proposed solutions in the literature have been evaluated
for different applications, power systems and GPU hardware, making it difficult to
make comparisons of the solutions.
 Real-world application: GPU-accelerated power system solutions have been
studied in the literature with convincing results. Its commercialization by software
vendors and adoption by the industry is yet to come.

Potential Solutions
Possible solutions are discussed below:
 Develop standardized test cases and benchmark studies to evaluate and compare
the performance of different solutions and GPU/CPU hardware.
 Create consensus on best practices for domain-specific problems, for example,
preconditioner and sparse techniques for power flow analysis and/or in EMT
simulations.
 Explore newer computer technologies such as system on a chip (SoC) other than
discrete GPUs for its enhanced memory and speed capability.
 Energy efficiency and cost effectiveness need to be carefully assessed with
comparison to other computing options.
 Start pilot GPU-computing project with the industry.

5.5.3 Edge intelligence

Edge intelligence is revolutionizing the landscape of power systems, emerging as a
transformative force that fundamentally changes how we manage and optimize electricity
distribution. Positioned where decisions need to be made, often directly at the source of
data production, such as close distributed energy resource assets belong to customers,
where low latency is crucial for real-time decision-making, edge intelligence presents a
significant opportunity for power grids. This approach offers the potential to enhance
operational efficiency, improve functional safety, seamlessly integrate renewable energy
sources, bolster grid resiliency, and ultimately deliver reliable and affordable power [67].
The strategic deployment of edge intelligence marks a pivotal advancement in power
system capabilities.

Current State
To comprehend the cutting-edge advancements in edge intelligence, it is imperative
to extend our focus beyond the power industry. Within the broader scope of the Internet of
Things (IoT), edge intelligence has played a pivotal role. Its application in the IoT realm
has significantly extended, particularly as data continues to burgeon at the edge of systems
[68]. Processing this data at the edge proves to be more efficient, and consequently, edge

104
TR-123 — IEEE PES Technical Roadmap

intelligence facilitates the provision of richer data for localized inference in AI models.
The use of edge computing is prominent in smart cities, where tasks such as image
recognition and video analysis, employing deep neural networks locally, have become
commonplace [69]. Beyond this, edge intelligence has demonstrated effectiveness in
enhancing the interactivity of IoTs through the sharing of cognition and predictive analytics
[70]. To achieve the pervasive connectivity essential for myriad edge applications, there is
a necessity to augment the predominantly broadband approach. The integration of 5G
technology with edge intelligence becomes indispensable for realizing this vision [71]. In
the present landscape, power systems are undergoing a transformative shift towards
increased decentralization and real-time decision-making. Edge intelligence, distinguished
by its localized processing and decision-making capabilities, has emerged as a key player.
Technologies like edge computing, advanced sensors, and machine learning algorithms are
being strategically deployed at the edge of the power grid. This deployment facilitates
quicker responses to dynamic changes and enhances overall system efficiency. The
integration of edge intelligence in power systems brings about notable advantages,
allowing for direct power transfer from distributed energy resources, reducing harmonic
current, enabling higher voltage operation, and optimizing virtual power plants based on
local data and constraints [72]. This paradigm shift signifies a crucial step towards a more
responsive, adaptive, and efficient power infrastructure [73].

Future State
The future trajectory of power systems is intricately linked to the ongoing evolution
of edge intelligence. As we progress, there is a burgeoning emphasis on elevating the
intelligence of edge devices, empowering them to locally process and analyze data. This
evolutionary leap is geared towards diminishing latency, fortifying reliability, and laying
the groundwork for autonomous decision-making within the power grid. The synergy
between edge intelligence and emerging technologies, such as 5G and the Internet of
Things (IoT), is anticipated to substantially augment the capabilities of power systems. To
render edge computing scalable, automation assumes a pivotal role. This is necessitated by
the variable resources and diverse data streams that can interact with the edge intelligence
platform. Moreover, the resolution of decision-making can vary from subsections to
minutes, or even hours. Consequently, automation becomes critical, and human
intervention should be minimized. Decisions need to be proactive and predictive in nature,
aligning with the dynamic and complex nature of the edge of power systems.

Challenges
Despite the immense potential of edge intelligence, several challenges persist.
Issues such as data security, the interoperability of diverse edge devices, and the necessity
for standardized protocols present formidable hurdles. Additionally, scaling edge solutions
to effectively address the intricacies of large-scale power grids remains a central challenge.
Coordinating real-time data extraction proves to be particularly challenging due to the
variety of data streams and the bidirectional flow of data/decision traffic. Addressing these

105
TR-123 — IEEE PES Technical Roadmap

challenges is crucial for the seamless integration and optimal performance of edge
intelligence in power systems.
Potential Solutions
Researchers and industry experts are actively addressing these challenges. Robust
encryption and authentication mechanisms are being developed to bolster data security.
Standardization initiatives, such as those by IEEE, are working towards creating
interoperable frameworks. Additionally, advancements in edge computing infrastructure
and collaborative efforts between stakeholders are enhancing scalability and paving the
way for seamless integration.

5.5.4 Cloud computing

The U.S. National Institute of Standards and Technology (NIST) defines cloud
computing as “a model for enabling ubiquitous, convenient, on-demand network access to
a shared pool of configurable computing resources (e.g., networks, servers, storage,
applications, and services) that can be rapidly provisioned and released with minimal
management effort or service provider interaction” [74]. The cloud model is composed of
five essential characteristics: on-demand self-service, broad network access, resource
pooling, rapid elasticity, and measured service.

Cloud computing can be categorized into public, private, and hybrid clouds
according to its ownership. From the service model perspective, cloud computing can be
primarily offered as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and
Software as a Service (SaaS). The combination of different ownership and service models
provide operational efficiency, resilience, and sustainability to the users, making cloud
computing an attractive IT solution for many organizations, including the power industry.

Current State

There has been growing interest in cloud adoption amongst many utility entities.
This is motivated by rapid grid transformation in de-carbonization, digitization, and
decentralization, which requires ever-growing demand for data analytics and resources
such as computing, network, and storage, while traditional on-premises facilities face
constraints to meet the fast-growing business needs. To date, key production level use cases
for the cloud technology in the power industry include high performance computing in
planning and operations power system analysis and studies, short-term and long-term load
forecast, outage management system, distributed energy resource (DER) aggregation and
management, electricity market settlement, network model management, coordinated
system restoration drill, asset condition monitoring and management, etc. [75,76].

Future State
Potential future directions may include the following:
 SaaS Offering by Power Industry Software Vendors

106
TR-123 — IEEE PES Technical Roadmap

Some power system software vendors and solution providers continue to innovate
and start offering SaaS applications to the power industry, such as Siemens PSS/E,
GE MAPS, Energy Exemplar PLEXOS, etc. This trend will continue and we
expect more and more vendors will offer SaaS to help enterprise and users make
the most of costs while being highly flexible.
 Cloud Hosted Wide Area Monitoring and Regional Collaboration
The digitization of the power industry has produced massive amount of data from
digital relays, RTUs, PMUs, AMIs, Internet of Things (IoT), and other types of
measuring devices. Most of data is being unused or underused today because of
the inflexible legacy utility computing architecture. Cloud technology, on the other
hand, can ingest a prohibitive amount of data in a scalable manner, aggregate,
transform and time-aligned them for efficient queries and analytics to achieve wide
area monitoring across regions. In addition, cloud can host situational awareness
applications for operating entities such as Reliability Coordinators (RCs) so they
can view and share information easily and securely for effective collaboration in
real time.
 Cloud Based Data Driven Modeling and Advanced Analytics
Cloud unlocks a great deal of new solution frameworks, advanced algorithms and
tools to improve business decision. This includes data ingestion, data curation,
extract, transform, and load (ETL), data visualization, and data-driven methods
such as machine learning and artificial intelligence. Utilities can use these cloud
services to quickly develop online predictive models and power system control
algorithms that were traditionally difficult from model-based methods due to lack
of accurate model.
 Bulk Electric System (BES) Real Time Operations in the Cloud
BES real time operations applications are vital for situational awareness and
operations decisions. The major benefits of leveraging cloud for real time
operations includes faster computing and efficient disaster recovery. Traditional
on-premises EMS system including SCADA, ICCP, State Estimation, Real Time
Contingency Analysis (RTCA), Automatic Generation Control (AGC), Real Time
Voltage and Transient Stability Assessment, etc., could migrate to the cloud
gradually in the next five to ten years based on its critically in real time operations.

Challenges
A few identified challenges are summarized below:
 Power system Software License Mechanism and Pricing Model
Power system commercial software applications have different licensing methods
including dongle license, node-locked license or single-use license, Bring Your
Own License (BYOL), and network license; however, very few power system
software vendors offer subscript based license to support cloud adoption. Most
software pricing model is also based on one-time provisioning fee plus a periodical
maintenance fee, instead of subscription based pricing for measured service. These
have hindered utility companies from using the cloud technology for their power
system applications.
107
TR-123 — IEEE PES Technical Roadmap

 Cloud Security
Cloud security is one of the greatest concerns for the power industry. Today Cloud
Service Provides (CSPs) uses a Shared Responsibility Model (SRM), which means
that CSP is responsible for managing the security of the cloud while the customer
is responsible for securing what is in the cloud. The responsibility also shifts
depending on the service model and deployment model. Due to the complexity and
many misconceptions of cloud security, the power industry has been conservative
and slow in adopting the cloud technology.
 Regulatory Compliance
For mission critical workload such as BES real time operations, the operating
entities are subject to regulatory compliance requirements, for example, the NERC
CIP Standards. To date, many operating entities see the current regulatory
framework as prohibiting the adoption of cloud computing resources in support of
BES reliability functions because of increasing compliance risks.
 Available Expertise and Resource
The rapid technology evolution is creating challenges of obtaining and retaining
highly skilled professionals in the electric sector broadly. It has been and will
become increasingly difficult to find skilled professionals that understand the
legacy systems, the cloud technology, and the utility business requirements to drive
the cloud migration and innovation while ensuring business continuity.

Potential Solutions
Possible solutions are discussed below:
 Power System Software
It is recommended the power system software be architected and designed to be
cloud-friendly to facilitate cloud adoption by the utilities. Power system software
vendors should consider Microservices instead of monolithic architecture, use of
containerization to improve software interoperability, develop new domain specific
parallel computing algorithm to enable HPC capabilities. In addition, power system
software vendors need to be innovative to adopt the subscription-based licensing
model, together with the pay-by-subscription or pay-per-use pricing models, which
are more cost-effective and convenient in the cloud.
 Cloud security
Education and training in cloud security in the power industry has become
important. Cloud security is increasingly a part of cyber security training curricula
for utility IT and security professionals so that they have the knowledge and
skillsets to focus more heavily on securing the systems and critical infrastructure
under their responsibility.
 Regulatory compliance
For BES operations in the cloud which must meet compliance and regulatory
requirements such as NERC CIP, compliance demonstration may require
consideration of the acceptance of third-party audit records such as SOC reports, or
third-party certifications such as FedRAMP which are typically not necessary in an

108
TR-123 — IEEE PES Technical Roadmap

on-premises environment. Regulatory entity such as NERC’s acceptance of

FedRAMP as CIP compliant for security “of” the cloud” can provide a streamlined
compliance approach for power utilities and support efficient audit assessment
while maintaining stringent security obligations. Operating Entities building on
FedRAMP authorized infrastructure and services could focus their security controls
and compliance assessment on security “in” the cloud.”

5.6 Computational Tools for the Power Systems

Summary
The integration of renewable energy sources, energy storage systems, and smart
grid technologies has introduced complexities in power systems. Computational tools have
emerged as valuable assets to address these challenges. The advent of advanced computing
technologies and the availability of vast amounts of data have facilitated the development
of computational tools tailored to power systems. These tools leverage mathematical
models, algorithms, and simulation techniques to analyze system behavior, assess
performance, and support decision-making in various aspects of power system operations,
planning, and control.
This section aims to provide a comprehensive understanding of the current
landscape of computational tools for power systems. By exploring the advancements in
this field, researchers, practitioners, and policymakers can gain insights into the state of the
art, ongoing challenges, and future opportunities.

5.6.1 Open-source Tools

Open-source tools have the advantages of relatively low-cost compared to
commercial ones, available source code for customization. With the recent development of
open-source scientific computation, there has been a growing interest in developing and
using open-source tools in the research community [77]. The open-source tools have
covered both dispatch and dynamic simulation.

Current State
MATPOWER [78], a free, open-source MATLAB toolbox, was developed for
steady-state simulation. It is widely used in the research community and thus its file
formats, dispatch formulations, and software design can be regarded as the first-generation
open-source standards in power system analysis. Later, PSAT was developed and released
as another MATLAB toolbox that covers both static and dynamic analysis, The graphical
user interfaces (GUI) further facilitate user-friendly uses. Although MATLAB requires a
license to use, the two toolboxes are GNU Octave compatible to address the license issue.
Given the advancement of Python and its prosperous and diverse community, many
other fully open-sourced tools are developed with their own specific focus, for examples,
pandapower [79] and PyPSA [80]. Additionally, OpenDSS [81] was developed to cover

109
TR-123 — IEEE PES Technical Roadmap

the distribution system analysis. Recently, ANDES [82] proposed a hybrid symbolic-
numerical modeling framework to enable intuitive modeling process and high-performance
numerical performance. Shipped with rich model library and built-in cases, it has been
actively used by power system researchers.

Future State
Given the on-going social-economic driven factors, future power system analysis
tools are required to be scalable, flexible, and interoperable.
First, the computation tools need to capable of large-scale problems efficiently and
effectively. This may require parallel computing techniques such as distributed computing
or cloud computing to leverage the computational resources available. It may also require
advanced algorithms such as decomposition methods or approximation methods to reduce
the problem size or complexity.
Second, the tools should be flexible that allows researchers to easily test their own
algorithms and controllers. This may require modular programming techniques such as
object-oriented programming or functional programming to facilitate the development and
maintenance of the code. It may also require user-friendly interfaces such as graphical user
interfaces or web-based interfaces to enhance the usability and accessibility of the tools.
Third, high-fidelity simulation results call for the interoperations of multiple tools,
which means common data formats such as JSON or XML to enable data sharing and
integration. It may also require common standards such as IEC 61970 or IEEE 1547 to
ensure data quality and compatibility.
Challenges
Despite the on-going progress, following challenges still exist. First, the diverse
quality of tools, due to varying software engineering expertise, can lead to confusion about
their use and uncertainty about their outputs. Second, coordinating multiple tools can be
challenging for a power system researcher, especially when it comes with multiple
programming languages.

Potential Solutions
The first challenge can be mitigated by developing a public database that
continuously update the open-source resources in power system. The database should
provide commonly used software quality evaluation metrics, and their benchmark with
verified results. Regarding the second one, HELICS has been developed to enable large-
scale co-simulation of multiple tools. More built-in examples can further reduce the
researchers’ learning cost.

5.6.2 Synthetic Test Systems

Synthetic test systems provide controlled and customizable environments for
evaluating computational tools. They replicate real-world system characteristics, including

110
TR-123 — IEEE PES Technical Roadmap

generation units, transmission networks, and load profiles, while offering the flexibility to
customize parameters such as system size, complexity, and operational scenarios. The test
systems enable researchers and practitioners to conduct rigorous assessments of
computational tools, compare methods, and evaluate the performance and robustness of
algorithms, especially under extreme circumstances such as equipment failures, cyber-
attacks, and extreme weather.
Current State
Researchers can generate test cases of varying sizes, ranging from small-scale
distribution networks to large-scale transmission grids. This scalability facilitates the
assessment of computational tools' scalability, efficiency, and computational requirements.
Additionally, synthetic test systems provide a basis for comparing the performance of
different tools on common benchmark problems, enabling objective evaluations. In the
field of power systems, several synthetic test systems have been developed.
The IEEE (Institute of Electrical and Electronics Engineers) has developed a series
of standard test systems that serve as benchmarks for evaluating power system analysis
tools. The IEEE test systems, such as the IEEE 14-bus, 24-bus, 30-bus, 57-bus, and 118-
bus systems, are widely used for testing and comparing various algorithms and methods.
These test systems capture the characteristics of real power systems and are particularly
useful for assessing power flow, dispatch, and dynamic stability. The International Council
on Large Electric Systems (CIGRE) has established benchmark systems to evaluate
computational tools for power system analysis. These benchmark systems encompass
transmission and distribution networks with diverse configurations and operating
conditions. The CIGRE benchmark systems focus on specific challenges, such as voltage
stability, transient stability, and fault analysis. They provide a standardized framework for
testing and comparing different algorithms and approaches. PGLib-OPF [83] offers a
collection of optimal power flow problems. It allows researchers to benchmark and
compare different OPF algorithms and techniques. There are also some other active public
test cases contributor from open-source software and system operators, such as GridLAB-
D [84] and European Network of Transmission System Operators for Electricity.
Future State
Like the computational tools, the social factors driven power grids transformation
expect following features of the synthetic cases.
First, cases with realistic high-penetration scenarios are expected to facilitate
convincing simulation results. Especially considering the renewables have geo-correlation,
which is not the case for a traditional power grid mainly relies on fossil resource.
Second, transmission-distribution integrated cases should be developed for the emerging
new market mechanism and new distribution-level functionalities.
Challenges
Following challenges should be mitigated for the test cases. First, the static nature
of synthetic cases, due to infrequent updates, can hinder alignment between industry
practice and academic progress. Second, high-quality case synthetization efforts are often
111
TR-123 — IEEE PES Technical Roadmap

underappreciated, which further prohibits the development of test cases library. Third, the
data management policy varies across system operators, regions, or countries, and it thus
confronts researchers with data availability issues.
Potential Solutions
Following approaches are proposed to address the challenges. First, promoting
industry participants to share more public test cases with updated real-world applications.
Second, encouraging collaborations between industry and academia to develop standard
evaluation matrices to assess test cases synthetization. Third, investigating essential power
grid features that allows artificial test cases with realistic performance.

5.6.3 Testbeds and Digital Twins

Current State
Testing and validating modeling, analysis, or control techniques on power systems
are costly, risky, and impractical. therefore, testbeds are often used as platforms for
conducting experiments or tests under controlled conditions, without disrupting the actual
power system. Testbeds can mimic the behavior and characteristics of real power systems
while allowing researchers and engineers to manipulate the parameters and scenarios of
interest. Currently, testbeds for power systems can be classified into software testbeds and
hardware testbeds.

Software testbeds use software to simulate power system behavior. Software

simulation ensures flexibility and scalability by avoiding the need for real power hardware
and real-time simulation, such as evaluating the effects of renewable energy sources,
distributed generation, energy storage devices, electric vehicles, demand response
programs, and other smart grid technologies on the power system operation and planning.
Software testbeds can be used for developing and testing controllers and grid components
such as Phasor Measurement Units (PMU), centralized power management systems
(PMS), protection relays, and onboard electrical subsystems, when reducing the cost and
risk of testing, as well as accelerating the innovation and education process [85]. System
dynamic characteristics wise, software testbeds are leveraged to prototype and verify novel
control strategies and algorithms [86] for power system functions such as optimal power
flow, state estimation, voltage control, frequency control, etc. CURENT large-scale testbed
(LTB) [87-89], ORNL power grid testbeds [119, 120], and NREL Sienna [90] are two
testbeds developed for power system prototyping and verification.

Hardware-in-the-loop (HIL) testbeds use real hardware devices (such as generators,

loads, sensors, controllers, etc.) connected to a simulated power system model, aiming to
check and validate the protective systems and controller software program’s quality,
functionality, performance, and safety. The hardware devices can interact with the
simulated model through analog or digital signals such as current or voltage interfaces. In
this approach, the protection equipment and controller can be tested under any operational
112
TR-123 — IEEE PES Technical Roadmap

conditions [91]. The HIL simulation-based testbed demonstrated for the case of RT-LAB
technology is introduced in [92], demonstration purposes could be equally used on RTDS
[93]. Considering the suffered accuracy and fidelity from different levels of model
reductions in purely digital simulations, CURENT hardware testbed (HTB) has developed
as a reconfigurable electric grid [94] to overcome the limitations of digital emulators. HIL
testbeds place all the high-power, high-voltage, potentially dangerous, and expensive
equipment into a real-time simulation, but they can also be expensive and difficult to scale
up.

Digital twin is a concept that refers to a digital replica of a physical entity or system
that can be updated with real-time data and used for simulation, analysis, and
optimization. Still in early-stage investigation, DT is regarded as an extension or
integration of hardware and software testbeds. In this way, a DT can combine the
scalability and flexibility from software and accuracy and high-fidelity from hardware
[95],[96]. Introduction of large-scale high-performance computing (HPC) resources will
assist with large-scale digital twins integration within hardware-in-the-loop (HIL) testbeds
that assist with integration of data and data-driven techniques with large-scale physics-
based simulations and experiments.

Future State
Therefore, the future vision of testbeds for power systems is to develop a cyber-
physical power system (CPPS) testbed [97], [98] that can realize a holistic and
comprehensive simulation of the power system across different levels. Such a testbed
would enable more effective and efficient testing and verification of new CPPS solutions,
as well as facilitate collaboration among stakeholders.
Challenges
Testbeds for power systems also face several challenges, such as the need for
a realistic and scalable representation of complex power systems, the need for flexible and
interoperable integration of different hardware/software components, and the need for
robust and secure protection of the testbed infrastructure.

Unlike conventional models for a single component or an asset, a DT in the context

of a power system distinguishes itself by diversity in composition and complexity in
operation. Continuous live data are usually required for a proper state observation and/or
prediction of the system. Moreover, DTs require a high level of data quality, model
accuracy, and cybersecurity to ensure their validity and trustworthiness [99]. In the future,
a multi-level and multi-disciplinary integration of different DTs across the power system
is expected.
Potential Solutions

113
TR-123 — IEEE PES Technical Roadmap

Considering the concept of DT of a large-scale power grid has not reach a common
understanding yet, the collaborations between experienced system operators, software
vendors, and researchers are substantial for exploring the viable ways.

5.7 References
[1] C. Barnhart, E.L. Johnson, G.L. Nemhauser, M.W. Savelsbergh, and P.H., Vance,
“Branch-and-price: Column generation for solving huge integer programs,”
Operations research, vol. 46, no. 3, pp. 316-329, 1998.
[2] R. Rahmaniani, T. G. Crainic, M. Gendreau, and W. Rei. “The Benders
decomposition algorithm: A literature review,” European Journal of Operational
Research, vol. 259, no. 3, pp.801-817, 2017.
[3] B. Zeng and L. Zhao. “Solving two-stage robust optimization problems using a
column-and-constraint generation method,” Operations Research Letters, vol. 41,
no.5, pp. 457-461, 2013.
[4] M. Liu, S. K. Tso, and Y. Cheng, “An extended nonlinear primal-dual interior-point
algorithm for reactive-power optimization of large-scale power systems with discrete
control variables,” IEEE Trans. Power Syst., vol. 17, no. 4, pp. 982-991, Nov. 2002.
[5] J. Nair, A. Wierman, B. Zwart, The fundamentals of heavy-tails: Properties,
emergence, and identification, in: Proceedings of the ACM SIGMETRICS/
International Conference on Measurement and Modeling of Computer Systems, 2013,
pp. 387–388.
[6] J. Zou, S. Ahmed and X. A. Sun, "Multistage Stochastic Unit Commitment Using
Stochastic Dual Dynamic Integer Programming," in IEEE Transactions on Power
Systems, vol. 34, no. 3, pp. 1814-1823, May 2019.
[7] B.F. Hobbs, Q. Xu, J. Ho, P. Donohoo, S. Kasina, J. Ouyang, S.W. Park, J. Eto, V.
Satyal, Adaptive transmission planning: Implementing a new paradigm for managing
economic risks in grid expansion, IEEE Power Energy Mag. 14 (4) (2016) 30–40.
[8] R.J. Pinto, C. Borges, M.E.P. Maceira, An efficient parallel algorithm for large scale
hydrothermal system operation planning, IEEE Trans. Power Syst. 28 (4) (2013)
4888–4896.
[9] Q. Zhang, A.M. Bremen, I.E. Grossmann, J.M. Pinto, Long-term electricity
procurement for large industrial consumers under uncertainty, Ind. Eng. Chem. Res.
57 (9) (2018) 3333–3347.
[10] Nemirovski, A. Shapiro, Convex approximations of chance constrained programs,
SIAM J. Optim. 17 (4) (2007) 969–996.
[11] T. Summers, J. Warrington, M. Morari, J. Lygeros, Stochastic optimal power flow
based on conditional value at risk and distributional robustness, Int. J. Electr. Power
Energy Syst. 72 (2015) 116–125.
[12] Y. Zhang, G.B. Giannakis, Robust optimal power flow with wind integration using
conditional value-at-risk, in: 2013 IEEE International Conference on Smart Grid
Communications (SmartGridComm), IEEE, 2013, pp. 654–659.

114
TR-123 — IEEE PES Technical Roadmap

[13] L. Roald, S. Misra, M. Chertkov, G. Andersson, Optimal power flow with weighted
chance constraints and general policies for generation control, in: IEEE Conference
on Decision and Control (CDC) 2015, Osaka, Japan, 2015.
[14] J. Luedtke, S. Ahmed, A sample approximation approach for optimization with
probabilistic constraints, SIAM J. Optim. 19 (2) (2008) 674–699.
[15] G.C. Calafiore, M.C. Campi, The scenario approach to robust control design, IEEE
Trans. Automat. Control 51 (5) (2006) 742–753.
[16] Line A. Roald, David Pozo, Anthony Papavasiliou, Daniel K. Molzahn, Jalal
Kazempour, Antonio Conejo, Power systems optimization under uncertainty: A
review of methods and applications, Electric Power Systems Research, Volume 214,
Part A, 2023, 108725.
[17] Papavasiliou, A. Bouso, S. Apelfröjd, E. Wik, T. Gueuning and Y. Langer, "Multi-
Area Reserve Dimensioning Using Chance-Constrained Optimization," in IEEE
Transactions on Power Systems, vol. 37, no. 5, pp. 3982-3994, Sept. 2022.
[18] Venzke, L. Halilbasic, U. Markovic, G. Hug and S. Chatzivasileiadis, "Convex
Relaxations of Chance Constrained AC Optimal Power Flow," in IEEE Transactions
on Power Systems, vol. 33, no. 3, pp. 2829-2841, May 2018.
[19] W. Zhang, Y. Xu, Z. Dong and K. P. Wong, "Robust Security Constrained-Optimal
Power Flow Using Multiple Microgrids for Corrective Control of Power Systems
Under Uncertainty," in IEEE Transactions on Industrial Informatics, vol. 13, no. 4, pp.
1704-1713, Aug. 2017.
[20] L. Huang, C. S. Lai, Z. Zhao, G. Yang, B. Zhong and L. L. Lai, "Robust N-k Security-
constrained Optimal Power Flow Incorporating Preventive and Corrective Generation
Dispatch to Improve Power System Reliability," in CSEE Journal of Power and
Energy Systems, vol. 9, no. 1, pp. 351-364, January 2023.
[21] Stratigakos, P. Andrianesis, A. Michiorri and G. Kariniotakis, "Towards Resilient
Energy Forecasting: A Robust Optimization Approach," in IEEE Transactions on
Smart Grid, doi: 10.1109/TSG.2023.3272379.
[22] W. Xie, S. Ahmed, Distributionally robust chance constrained optimal power flow
with renewables: A conic reformulation, IEEE Trans. Power Syst. 33 (2) (2018) 1860–
1867.
[23] M. Lubin, Y. Dvorkin, S. Backhaus, A robust approach to chance constrained optimal
power flow with renewable generation, IEEE Trans. Power Syst. 31 (5) (2015) 3840–
3849.
[24] Wang, R. Gao, F. Qiu, J. Wang, L. Xin, Risk-based distributionally robust optimal
power flow with dynamic line rating, IEEE Trans. Power Syst. 33 (6) (2018) 6074–
6086.
[25] Y. Zhang, S. Shen, J.L. Mathieu, Distributionally robust chance-constrained optimal
power flow with uncertain renewables and uncertain reserves provided by loads, IEEE
Trans. Power Syst. 32 (2) (2017) 1378–1388.
[26] B.K. Poolla, A.R. Hota, S. Bolognani, D.S. Callaway, A. Cherukuri, Wasserstein
distributionally robust look-ahead economic dispatch, IEEE Trans. Power Syst. 36 (3)
(2021) 2010–2022.

115
TR-123 — IEEE PES Technical Roadmap

[27] Ordoudis, V.A. Nguyen, D. Kuhn, P. Pinson, Energy and reserve dispatch with
distributionally robust joint chance constraints, Oper. Res. Lett. 49 (3) (2021) 291–
299.
[28] G. Hager and G. Wellein, Introduction to High Performance Computing for Scientists
and Engineers, 0 ed. CRC Press, 2010. doi: 10.1201/EBK1439811924.
[29] H. Cui, F. Li, and X. Fang, “Effective Parallelism for Equation and Jacobian
Evaluation in Large-Scale Power Flow Calculation,” IEEE Trans. Power Syst., vol.
36, no. 5, pp. 4872–4875, Sep. 2021, doi: 10.1109/TPWRS.2021.3073591.
[30] O. Schenk, K. Gärtner, W. Fichtner, and A. Stricker, “PARDISO: a high-performance
serial and parallel sparse linear solver in semiconductor device simulation,” Future
Gener. Comput. Syst., vol. 18, no. 1, pp. 69–78, 2001.
[31] X. Li, F. Li, H. Yuan, H. Cui, and Q. Hu, “GPU-Based Fast Decoupled Power Flow
With Preconditioned Iterative Solver and Inexact Newton Method,” IEEE Trans.
Power Syst., vol. 32, no. 4, pp. 2695–2703, Jul. 2017, doi:
10.1109/TPWRS.2016.2618889.
[32] T. A. Davis and E. P. Natarajan, “User Guide for KLU and BTF.” 2022.
[33] F. Milano, “A python-based software tool for power system analysis,” in 2013 IEEE
Power & Energy Society General Meeting, Jul. 2013, pp. 1–5. doi:
10.1109/PESMG.2013.6672387.
[34] T. A. Davis, Direct methods for sparse linear systems. SIAM, 2006.
[35] M. Shahidehopour and Y. Fu, “Benders decomposition: applying Benders
decomposition to power systems,” IEEE Power Energy Mag., vol. 3, no. 2, pp. 20–21,
Mar. 2005, doi: 10.1109/MPAE.2005.1405865.
[36] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed Optimization and
Statistical Learning via the Alternating Direction Method of Multipliers,” Found.
Trends® Mach. Learn., vol. 3, no. 1, pp. 1–122, Jul. 2011, doi: 10.1561/2200000016.
[37] H. Karimipour and V. Dinavahi, “Accelerated parallel WLS state estimation for large-
scale power systems on GPU,” in 2013 North American Power Symposium (NAPS),
IEEE, 2013, pp. 1–6.
[38] T. Guler, G. Gross, and Minghai Liu, “Generalized Line Outage Distribution Factors,”
IEEE Trans. Power Syst., vol. 22, no. 2, pp. 879–881, May 2007, doi:
10.1109/TPWRS.2006.888950.
[39] J. H. Chow and K. W. Cheung, “A toolbox for power system dynamics and control
engineering education and research,” IEEE Trans. Power Syst., vol. 7, no. 4, pp. 1559–
1564, Nov. 1992, doi: 10.1109/59.207380.
[40] F. Milano, “An open source power system analysis toolbox,” IEEE Trans. Power Syst.,
vol. 20, no. 3, pp. 1199–1206, Aug. 2005, doi: 10.1109/TPWRS.2005.851911.
[41] F. Milano, “Power System Modelling and Scripting,” in Power System Modelling and
Scripting, F. Milano, Ed., in Power Systems. , Berlin, Heidelberg: Springer, 2010, pp.
3–17.
[42] H. Cui, F. Li, and K. Tomsovic, “Hybrid Symbolic-Numeric Framework for Power
System Modeling and Analysis,” IEEE Trans. Power Syst., vol. 36, no. 2, Mar. 2021,
doi: 10.1109/TPWRS.2020.3017019.

116
TR-123 — IEEE PES Technical Roadmap

[43] V. Jalili-Marandi and V. Dinavahi, “SIMD-Based Large-Scale Transient Stability

Simulation on the Graphics Processing Unit,” IEEE Trans. Power Syst., vol. 25, no.
3, pp. 1589–1599, Aug. 2010, doi: 10.1109/TPWRS.2010.2042084.
[44] G. Gurrala, A. Dimitrovski, S. Pannala, S. Simunovic, and M. Starke, “Parareal in
Time for Fast Power System Dynamic Simulations,” IEEE Trans. Power Syst., vol.
31, no. 3, pp. 1820–1830, May 2016, doi: 10.1109/TPWRS.2015.2434833.
[45] B. Park, K. Sun, A. Dimitrovski, Y. Liu, and S. Simunovic, “Examination of Semi-
Analytical Solution Methods in the Coarse Operator of Parareal Algorithm for Power
System Simulation,” IEEE Trans. Power Syst., vol. 36, no. 6, pp. 5068–5080, Nov.
2021, doi: 10.1109/TPWRS.2021.3069136.
[46] N. R. Watson and J. Arrillaga, Power Systems Electromagnetic Transients Simulation,
1st ed. Institution of Engineering and Technology, 2003.
[47] S. R. Sanders, J. M. Noworolski, X. Z. Liu, and G. C. Verghese, “Generalized
averaging method for power conversion circuits,” IEEE Trans. Power Electron., vol.
6, no. 2, pp. 251–259, Apr. 1991, doi: 10.1109/63.76811.
[48] P. Zhang, J. R. Marti, and H. W. Dommel, “Shifted-Frequency Analysis for EMTP
Simulation of Power-System Dynamics,” IEEE Trans. Circuits Syst. Regul. Pap., vol.
57, no. 9, pp. 2564–2574, Sep. 2010, doi: 10.1109/TCSI.2010.2043992.
[49] M. Mirz, S. Vogel, G. Reinke, and A. Monti, “DPsim—A dynamic phasor real-time
simulator for power systems,” SoftwareX, vol. 10, p. 100253, Jul. 2019, doi:
10.1016/j.softx.2019.100253.
[50] J. Tan, B. Wang, J. Maack, D. Vaidhynathan, and M. Reynolds, “ParaEMT (NREL’s
EMT Simulator: An Open Source, and HPC-Compatible Large-Scale Power System
Electro-Magnetic Transient (EMT) Simulator) [SWR-22-16],” National Renewable
Energy Laboratory (NREL), Golden, CO (United States), ParaEMT, Mar. 2022. doi:
10.11578/dc.20230315.1.
[51] A. Imakura and T. Sakurai, “Block SS–CAA: A complex moment-based parallel
nonlinear eigensolver using the block communication-avoiding Arnoldi procedure,”
Parallel Comput., vol. 74, pp. 34–48, May 2018, doi: 10.1016/j.parco.2017.11.007.
[52] J. Carlson, Redis in Action. Simon and Schuster, 2013.
[53] M. L. Di Silvestre et al., “Blockchain for power systems: Current trends and future
applications,” Renew. Sustain. Energy Rev., vol. 119, p. 109585, Mar. 2020, doi:
10.1016/j.rser.2019.109585.
[54] S. Golestan, M.R. Habibi, S.Y. Mousazadeh Mousavi, J.M. Guerrero, J.C. Vasquez,
Quantum computation in power systems: An overview of recent advances, Energy
Reports, Volume 9, 2023, Pages 584-596.
[55] Hualong Liu, Wenyuan Tang, Quantum computing for power systems: Tutorial,
review, challenges, and prospects, Electric Power Systems Research, Volume 223,
2023, 109530.
[56] Y. Zhou, Z. Tang, N. Nikmehr, P. Babahajiani, F. Feng, T. Wei, H. Zheng and P.
Zhang, Quantum computing in power systems, iEnergy, Volume: 1, Issue: 2, June
2022, Pages 170 – 187.
[57] Yoon, D.-H. and Y. Han, Parallel power flow computation trends and applications: A
review focusing on GPU. Energies, 2020. 13(9): p. 2147.
117
TR-123 — IEEE PES Technical Roadmap

[58] X. Li, F. Li, H. Yuan, H. Cui, and Q. Hu, "GPU-based Fast Decoupled Power Flow
with Preconditioned Iterative Solver and Inexact Newton Method," IEEE Transactions
on Power Systems, vol. 32, no. 4, pp. 2695-2703, Jul. 2017.
[59] Li, X. and F. Li, GPU-based power flow analysis with Chebyshev preconditioner and
conjugate gradient method. Electric Power Systems Research, 2014. 116: p. 87-93.
[60] Zhou, G., R. Bo, L. Chien, X. Zhang, S. Yang, and D. Su, GPU-accelerated algorithm
for online probabilistic power flow. IEEE Transactions on Power Systems, 2017.
33(1): p. 1132-1135.
[61] Zhou, G., Y. Feng, R. Bo, L. Chien, X. Zhang, Y. Lang, Y. Jia, and Z. Chen, GPU-
accelerated batch-ACPF solution for N-1 static security analysis. IEEE Transactions
on Smart Grid, 2016. 8(3): p. 1406-1416.
[62] Zhou, G., X. Zhang, Y. Lang, R. Bo, Y. Jia, J. Lin, and Y. Feng, A novel GPU-
accelerated strategy for contingency screening of static security analysis. International
Journal of Electrical Power & Energy Systems, 2016. 83: p. 33-39.
[63] Xia, Y., Y. Chen, Z. Ren, S. Huang, M. Wang, and M. Lin. State estimation for large-
scale power systems based on hybrid cpu-gpu platform. in 2017 IEEE Conference on
Energy Internet and Energy System Integration (EI2). 2017. IEEE.
[64] Milano, F., Small-signal stability analysis of large power systems with inclusion of
multiple delays. IEEE Transactions on Power Systems, 2015. 31(4): p. 3257-3266.
[65] Zhou, Z. and V. Dinavahi, Parallel massive-thread electromagnetic transient
simulation on GPU. IEEE transactions on Power Delivery, 2014. 29(3): p. 1045-1053.
[66] Roberge, V., M. Tarbouchi, and F. Okou, Optimal power flow based on parallel
metaheuristics for graphics processing units. Electric Power Systems Research, 2016.
140: p. 344-353.
[67] Zhang, Yingchen YC, and Marc Spieler. "Bringing Artificial Intelligence to the Grid
Edge [Technology Leaders]." IEEE Electrification Magazine 10.4 (2022): 6-9.
[68] Zhou, Zhi, et al. "Edge intelligence: Paving the last mile of artificial intelligence with
edge computing." Proceedings of the IEEE 107.8 (2019): 1738-1762.
[69] Barthélemy, Johan, et al. "Edge-computing video analytics for real-time traffic
monitoring in a smart city." Sensors 19.9 (2019): 2048. Kim, H., et al. (2021).
"Machine Learning for Edge Devices: A Review." IEEE Transactions on Neural
Networks and Learning Systems.
[70] Zhang, Yin, et al. "Edge intelligence in the cognitive Internet of Things: Improving
sensitivity and interactivity." IEEE Network 33.3 (2019): 58-64.
[71] Palattella, Maria Rita, et al. "Internet of things in the 5G era: Enablers, architecture,
and business models." IEEE journal on selected areas in communications 34.3 (2016):
510-527.
[72] Chou, Hung-Ming, et al. "Power electronics intelligence at the network edge (PINE)."
2017 IEEE Energy Conversion Congress and Exposition (ECCE). IEEE, 2017.
[73] Lin, Lin, et al. "Deep reinforcement learning for economic dispatch of virtual power
plant in internet of energy." IEEE Internet of Things Journal 7.7 (2020): 6288-6301.
[74] “The NIST definition of cloud computing.” U.S. Dept. Commerce, Nat. Inst. Stand.
Technol., Gaithersburg, MD, USA, Rep. SP 800–145, 2012.
https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-145.pdf
118
TR-123 — IEEE PES Technical Roadmap

[75] S. Zhang, A. Pandey, X. Luo, M. Powell, R. Banerji, A. Parchure, L. Fan, E. Luzcando,

Practical Adoption of Cloud Computing in Power Systems – Drivers, Challenges,
Guidance, and Real-world Use Cases”, IEEE PES Technical Report (PES-TR92),
December 2021. https://resourcecenter.ieee-pes.org/publications/technical-
reports/PES_TP_TR92_AMPS_012822.html
[76] S. Zhang, A. Pandey, X. Luo, M. Powell, R. Banerji, L. Fan, A. Parchure, E. Luzcando,
Practical Adoption of Cloud Computing in Power Systems – Drivers, Challenges,
Guidance, and Real-world Use Cases, IEEE Transactions on Smart Grid, vol. 13, no.
3, May 2022, https://ieeexplore.ieee.org/document/9703493.
[77] H.-K. Ringkjøb, P. M. Haugan, and I. M. Solbrekke, “A review of modelling tools for
energy and electricity systems with large shares of variable renewables,” in Renew.
Sustain. Energy Rev., vol. 96, pp. 440–459, Nov. 2018, doi:
10.1016/j.rser.2018.08.002.
[78] C. E. Murillo-Sánchez, R. D. Zimmerman, C. L. Anderson and R. J. Thomas, "Secure
Planning and Operations of Systems With Stochastic Sources, Energy Storage, and
Active Demand," in IEEE Transactions on Smart Grid, vol. 4, no. 4, pp. 2220-2229,
Dec. 2013, doi: 10.1109/TSG.2013.2281001.
[79] L. Thurner et al., "Pandapower—An Open-Source Python Tool for Convenient
Modeling, Analysis, and Optimization of Electric Power Systems," in IEEE
Transactions on Power Systems, vol. 33, no. 6, pp. 6510-6521, Nov. 2018, doi:
10.1109/TPWRS.2018.2829021.
[80] T. Brown, J. Hörsch, D. Schlachtberger, PyPSA: Python for Power System Analysis,
2018, Journal of Open Research Software, 6(1), arXiv:1707.09913,
doi:10.5334/jors.188.
[81] R. C. Dugan and T. E. McDermott, "An open source platform for collaborating on
smart grid research," 2011 IEEE Power and Energy Society General Meeting, Detroit,
MI, USA, 2011, pp. 1-7, doi: 10.1109/PES.2011.6039829.
[82] H. Cui, F. Li and K. Tomsovic, "Hybrid Symbolic-Numeric Framework for Power
System Modeling and Analysis," in IEEE Transactions on Power Systems, vol. 36, no.
2, pp. 1373-1384, March 2021, doi: 10.1109/TPWRS.2020.3017019.
[83] S. Babaeinejadsarookolaee et al., “The Power Grid Library for Benchmarking AC
Optimal Power Flow Algorithms”.
[84] D. P. Chassin, K. Schneider and C. Gerkensmeyer, "GridLAB-D: An open-source
power systems modeling and simulation environment," 2008 IEEE/PES Transmission
and Distribution Conference and Exposition, Chicago, IL, USA, 2008, pp. 1-5, doi:
10.1109/TDC.2008.4517260.
[85] J. Bélanger, P. Venne, and J.-N. Paquin, "The what, where and why of real-time
simulation," Planet Rt, vol. 1, no. 1, pp. 25–29, 2010.
[86] "How to Use Rapid Control Prototyping to Validate Electric Motors and Power
Converters." https://www.mathworks.com/videos/series/how-to-use-rapid-control-
prototyping-to-validate-electric-motors-and-power-converters.html (accessed Jun. 05,
2023).

119
TR-123 — IEEE PES Technical Roadmap

[87] F. Li, K. Tomsovic, and H. Cui, “A Large-Scale Testbed as a Virtual Power Grid: For
Closed-Loop Controls in Research and Testing,” IEEE Power and Energy Mag., vol.
18, no. 2, Art. no. 2, Mar. 2020, doi: 10.1109/MPE.2019.2959054.
[88] H. Cui, F. Li, and K. Tomsovic, “Hybrid Symbolic-Numeric Framework for Power
System Modeling and Analysis,” IEEE Trans. Power Syst., vol. 36, no. 2, Art. no. 2,
Mar. 2021, doi: 10.1109/TPWRS.2020.3017019.
[89] N. Parsly, J. Wang, N. West, Q. Zhang, H. Cui, and F. Li, “DiME and AGVIS A
Distributed Messaging Environment and Geographical Visualizer for Large-scale
Power System Simulation,” 2022, doi: 10.48550/ARXIV.2211.11990.
[90] R. Henriquez-Auba, J. D. Lara, D. S. Callaway, and C. Barrows, “Transient
Simulations With a Large Penetration of Converter-Interfaced Generation: Scientific
Computing Challenges And Opportunities,” IEEE Electrific. Mag., vol. 9, no. 2, pp.
72–82, Jun. 2021, doi: 10.1109/MELE.2021.3070939.
[91] R. Stanev, K. Viglov, K. Nakov and T. Asenov, "A Real Time Power Hardware in the
Loop Test Bed for Power System Stability Studies," 2020 12th Electrical Engineering
Faculty Conference (BulEF), Varna, Bulgaria, 2020, pp. 1-5, doi:
10.1109/BulEF51036.2020.9326020.
[92] C. Dufour, S. Abourida and J. Belanger, "Hardware-In-the-Loop Simulation of Power
Drives with RT-LAB," 2005 International Conference on Power Electronics and
Drives Systems, Kuala Lumpur, Malaysia, 2005, pp. 1646-1651, doi:
10.1109/PEDS.2005.1619952.
[93] RTDS Technologies, https://www.rtds.com/.
[94] L. M. Tolbert et al., "Reconfigurable Real-Time Power Grid Emulator for Systems
With High Penetration of Renewables," in IEEE Open Access Journal of Power and
Energy, vol. 7, pp. 489-500, 2020, doi: 10.1109/OAJPE.2020.3030219.
[95] H. Pan, Z. Dou, Y. Cai, W. Li, X. Lei and D. Han, "Digital Twin and Its Application
in Power System," 2020 5th International Conference on Power and Renewable
Energy (ICPRE), Shanghai, China, 2020, pp. 21-26, doi:
10.1109/ICPRE51194.2020.9233278.
[96] Z. Song et al., “Digital Twins for the Future Power System: An Overview and a Future
Perspective,” Sustainability, vol. 15, no. 6, p. 5259, Mar. 2023, doi:
10.3390/su15065259. [Online]. Available: http://dx.doi.org/10.3390/su15065259.
[97] R. V. Yohanandhan, R. M. Elavarasan, R. Pugazhendhi, M. Premkumar, L. Mihet-
Popa, and V. Terzija, “A holistic review on Cyber-Physical Power System (CPPS)
testbeds for secure and sustainable electric power grid – Part – I: Background on CPPS
and necessity of CPPS testbeds,” International Journal of Electrical Power & Energy
Systems, vol. 136, p. 107718, Mar. 2022, doi: 10.1016/j.ijepes.2021.107718.
[98] R. V. Yohanandhan, R. M. Elavarasan, R. Pugazhendhi, M. Premkumar, L. Mihet-
Popa, and V. Terzija, “A holistic review on Cyber-Physical Power System (CPPS)
testbeds for secure and sustainable electric power grid – Part – II: Classification,
overview and assessment of CPPS testbeds,” International Journal of Electrical Power
& Energy Systems, vol. 137, p. 107721, May 2022, doi:
10.1016/j.ijepes.2021.107721.

120
TR-123 — IEEE PES Technical Roadmap

[99] A. Bonetti, C. Harispuru, M. Pitzer, M. Pustejovsky, N. Wetterstrand and S.

Kachelrieß, "Digital twin technology for virtual testing of power system relay
protection," 2021 3rd Global Power, Energy and Communication Conference
(GPECOM), Antalya, Turkey, 2021, pp. 154-160, doi:
10.1109/GPECOM52585.2021.9587869.
[100] S. Debnath and J. Choi, "Electromagnetic Transient (EMT) Simulation Algorithms
for Evaluation of Large-Scale Extreme Fast Charging Systems (T& D Models),"
in IEEE Transactions on Power Systems, vol. 38, no. 5, pp. 4069-4079, Sept. 2023.
[101] J. Choi and S. Debnath, "Electromagnetic Transient (EMT) Simulation Algorithm for
Evaluation of Photovoltaic (PV) Generation Systems," 2021 IEEE Kansas Power and
Energy Conference (KPEC), Manhattan, KS, USA, 2021, pp. 1-6.
[102] S. Debnath and M. Chinthavali, "Numerical-Stiffness-Based Simulation of Mixed
Transmission Systems," in IEEE Transactions on Industrial Electronics, vol. 65, no.
12, pp. 9215-9224, Dec. 2018.
[103] S. Debnath, "Real-Time Simulation of Modular Multilevel Converters," 2018 IEEE
Energy Conversion Congress and Exposition (ECCE), Portland, OR, USA, 2018, pp.
5196-5203.
[104] T. Cheng, T. Duan and V. Dinavahi, "Parallel-in-Time Object-Oriented
Electromagnetic Transient Simulation of Power Systems," in IEEE Open Access
Journal of Power and Energy, vol. 7, pp. 296-306, 2020.
[105] S. Debnath, "Parallel-in-Time Simulation Algorithm for Power Electronics: MMC-
HVdc System," in IEEE Journal of Emerging and Selected Topics in Power
Electronics, vol. 8, no. 4, pp. 4100-4108, Dec. 2020.
[106] J. Choi, P. Marthi, S. Debnath, Md Arifujjaman, N. Rexwinkel, F. Khalilpour; A.
Arana; H. Karimjee, "Hardware-based Advanced Electromagnetic Transient
Simulation for A Large-Scale PV Plant in Real Time Digital Simulator," 2023 IEEE
Energy Conversion Congress and Exposition (ECCE), Nashville, TN, USA, 2023, pp.
965-971.
[107] S. Yan, Z. Zhou and V. Dinavahi, "Large-Scale Nonlinear Device-Level Power
Electronic Circuit Simulation on Massively Parallel Graphics Processing
Architectures," in IEEE Transactions on Power Electronics, vol. 33, no. 6, pp. 4660-
4678, June 2018.
[108] J. Sun, S. Debnath, M. Saeedifard and P. R. V. Marthi, "Real-Time Electromagnetic
Transient Simulation of Multi-Terminal HVDC–AC Grids Based on GPU," in IEEE
Transactions on Industrial Electronics, vol. 68, no. 8, pp. 7002-7011, Aug. 2021.
[109] Suman Debnath, Marcelo Elizondo, Yuan Liu, Phani Marthi, Wei Du, Shilpa Marti,
Qiuhua Huang, “High Penetration Power Electronics Grid: Gap Analysis”, Technical
Report.
[110] Suman Debnath, Alireza Ghassemian, John Seuss, Aung Thant, “EMT Simulation
Workshop”, Oak Ridge National Laboratory, 2023.
[111] Y. Zhou, F. Feng and P. Zhang, "Quantum Electromagnetic Transients Program," in
IEEE Transactions on Power Systems, vol. 36, no. 4, pp. 3813-3816, July 2021, doi:
10.1109/TPWRS.2021.3067207.

121
TR-123 — IEEE PES Technical Roadmap

[112] S. Debnath, P. R. V. Marthi, J. Choi and R. Bennink, "Emerging Computing

Architectures: Simulation of Power Electronics in Power Grids," 2023 IEEE Energy
Conversion Congress and Exposition (ECCE), Nashville, TN, USA, 2023, pp. 704-
711.
[113] Koretsky, S., Gokhale, P., Baker, J.M., Viszlai, J., Zheng, H., Gurung, N., et al.:
Adapting quantum approximation optimization algorithm (qaoa) for unit commitment.
In: 2021 IEEE International Conference on Quantum Computing and Engineering
(QCE), pp. 181–187. IEEE, Piscataway (2021)
[114] Ajagekar, A., You, F.: Quantum computing for energy systems optimization:
Challenges and opportunities. Energy 179, 76–89 (2019)
[115] Oleksandr Kyriienko, Annie E. Paine, and Vincent E. Elfving, “Solving nonlinear
differential equations with differentiable quantum circuits”, Phys. Rev. A 103, 052416
– Published 17 May 2021.
[116] C Bravo-Prieto, Ryan LaRose, et-al, “Variational Quantum Linear Solver”, Quantum
7, 1188 (2023).
[117] J.C. Criado, M. Spannowsky, “Qade: Solving Differential Equations on Quantum
Annealers”, IOP Publishing Ltd Quantum Science and Technology, Volume 8,
Number 1.
[118] “QuEnergy Resilience: Exploring the role of quantum security and cyber for the
electric grid”, QED-C.
[119] Suman Debnath, “Automated Grid Interconnected Laboratory Emulation,
https://www.ornl.gov/project/automation-grid-interconnected-laboratory-emulation-
agile.
[120] S. Debnath, S. Zheng, N. Watson, S. Campbell, R. Zeng and M. Chinthavali, "Flexible
Intelligent Real-time dc-ac grid Emulator (FIRE): Power Electronic Hardware-in-the-
Loop (PE-HIL) Amplifier," 2019 IEEE Energy Conversion Congress and Exposition
(ECCE), Baltimore, MD, USA, 2019, pp. 1060-1067.

6. Cybersecurity
6.1 Executive Summary
To many electric utilities today, cybersecurity is a swirling, sucking eddy of despair
full of false hope in an ever-darkening universe where a superhero from a favorite universe
is needed to save the day. Other perspectives and experiences with cybersecurity in the
electric sector are not quite so dire: there are utilities whose self-assessment or audit reveal
a maturing OT environment that is a challenging, exhilarating world at the intersection of
cybersecurity and the world’s most complex machine ever built. This cybersecurity
roadmap can be used across this broad spectrum of perspectives to draw out some key
issues in the current state, a vision of what the future state could be, the challenges to get
there, plus some potential solutions that the electric sector could engage in with technical

122
TR-123 — IEEE PES Technical Roadmap

leadership from Standard Development Organizations (SDOs) like the IEEE, regulators,
vendors, experts, and other stakeholders.

But before a roadmap can happen, most people involved in the IEEE will recognize
that definitions create the sandbox for standards to play within. And so, the introduction
provides some key definitions for cybersecurity and Operational Technology (OT). Some
important acknowledgements follow: physical security that is integral to cybersecurity is
only addressed in an earlier chapter. A very deliberate decision was made to not cover some
areas of great interest in securing the power grid, like the Internet of Things (IoT) or
Industrial Internet of Things (IIoT). Another area not covered is privacy. These exclusions
mean that the cybersecurity roadmap does not become its own volume of and by itself,
despite the urgency in the electric sector of cybersecurity for IoT and IIoT along with
privacy needs.

With that introduction complete, the cybersecurity roadmap begins at the current
state and winds its way through a roadmap. The discussion is organized using the NIST
Cyber Security Framework, with some cross-cutting discussion for people, process, and
technology. The roadmap proposes potential solutions to the challenges faced in achieving
a future state where cybersecurity is a whitewater rafting trip through the exhilaration and
challenge of class V rapids and calm paddles through gently flowing flatwater, where each
raft has an experienced and certified guide and everyone in the raft has all of the tools,
training, and processes needed for a safe and secure journey down the river. One key to
achieving the transformational future state is the electric sector’s broad application of
Cyber-Informed Engineering (CIE); but there is recognition that it does not solve all
challenges. In total, over forty observations are made about the possible future state that
are traced through their individual challenges and potential solutions. And yes, the IEEE
can be a superhero that saves the day, along with several other superheroes who are no less
important in creating the future we envision.

6.2 Introduction

Many definitions of cybersecurity exist across many industries and sectors; here the
definition is as follows:

The continuously improved application of people, processes, and technologies to the

power grid for:
1. Preventing intentional or unintentional manual or automatic, digital or analog,
and logical or physical malevolent acts
2. Minimizing risks that compromise integrity, availability, and confidentiality of
data in transit and at rest
3. Detecting such acts as or before they occur
4. Responding to such acts
5. Implementing actions that remediate or mitigate the threat or vulnerability so
that reliability and resilience are maintained

123
TR-123 — IEEE PES Technical Roadmap

This definition of cybersecurity is different than understanding the power grid as a

cyber-physical system (CPS) [1]. Electric power systems depend on information and
communications technology (ICT) for monitoring, operation, protection, and control. The
linkage between the ICT, or cyber system, with the physical power system transforms the
power grid into a CPS. The security of a CPS refers to the ability of the integrated CPS to
withstand contingency events, including weather-based, equipment failures/malfunctions,
and cyberattacks. This is a generalization of the traditional N-K security concept to
incorporate the emerging threats from cyberattacks on a power system into the security
assessment and control. Clearly the concept of CPS security is broader than that of
cybersecurity as the former explicitly considers the impact of cyberattacks on the physical
system.

Cybersecurity of a CPS includes physical security that minimizes the physical

vulnerabilities and threats against the CPS for an acceptable level of risk. IEEE Std 1402
[2] addresses physical security of substations across their lifecycle. Chapter 3 in this report
discusses hardening of the electric grid from various events, including physical attack.
With Chapter 3 discussion of reliability and resilience that includes physical security, this
chapter will not further review the current state, trends, challenges, and solutions of
physical security.

Beyond physical security, the power grid as a CPS is comprised of two applicable
systems: the Bulk Electric System (BES) rated at 100 kV and above to align with NERC
definitions [3] and the distribution grid as follows:

1) Bulk Electric System

a) The North American Reliability Corporation (NERC) Critical Infrastructure
Protection (CIP) [4] applies in North America with its primary focus on the physical
and cybersecurity of the BES.
b) IEC 62443 via the International Society of Automation (ISA) Committee on
Industrial Automation and Control Systems Security Committee’s (ISA99)
Working Group 14 (WG14) is emerging with development of profiles for
substations, transmission, distribution, generation, and DERs that would apply
across all elements of the electric grid.
2) Distribution grid (as the balance of the electric grid components that are not part of the
BES).
a) State regulation (emerging in the USA is the potential expansion of federal
regulation for more comprehensive application of cybersecurity to the distribution
grid).
b) IEEE Std 1547.3 Guide for Cybersecurity of DERs Interconnected with Electric
Power Systems with the potential future alignment to ISA99 WG14 work .

The limits of the distribution grid for this report stops before the customer meter.
This is a deliberate choice to not address customer-owned equipment behind the meter with
the adoption of Internet-of-Things (IoT) devices. This allows the report to cover
distribution and transmission cybersecurity issues at greater depth with the

124
TR-123 — IEEE PES Technical Roadmap

acknowledgement that there are existing and rapidly evolving cybersecurity vulnerabilities
and attack vectors with IoT, exacerbated by a significant increase in attack surface.

In addition to recognizing the applicable systems, this report will focus on the
operational side of cybersecurity, not the business side. Examining NIST’s definition of
Operational Technology (OT) [5] as:

“a broad range of programmable systems or devices that interact with the physical
environment (or manage devices that interact with the physical environment).
These systems/devices detect or cause a direct change through the monitoring
and/or control of devices, processes, and events. Examples include industrial
control systems (ICS), building automation systems, transportation systems,
physical access control systems, physical environment monitoring systems, and
physical environment measurement systems.”

NIST’s definition is broad enough that it is applicable to the power grid. However,
cybersecurity involves more than just protecting OT as demonstrated by the publication of
the U.S. Department of Energy’s National Cyber-Informed Engineering (CIE) Strategy [6].
This strategy seeks to guide the energy sector efforts to incorporate the twelve principles
of CIE [7] into the design of the power grid, including an example of Consequence-driven
Cyber-informed Engineering (CCE) [8]. CIE is an emerging method to integrate
cybersecurity considerations into the conception, design, development, and operation of
any CPS, such as the power grid. CIE complements, but does not replace, the application
of cybersecurity standards to OT. Rather, CIE expands cybersecurity decisions into the
engineering practice, not by asking engineers to become cybersecurity experts, but by
calling on engineers to apply engineering tools and make engineering decisions that
improve cybersecurity outcomes. CIE examines the engineering consequences that a
sophisticated cyber- attacker could achieve, and drives engineering changes that may
provide deterministic mitigations to limit or eliminate those consequences.

Engineers and technicians that design critical energy infrastructure installations can
integrate the twelve principles of CIE into each phase of the engineering lifecycle, from
concept to retirement. These principles are aimed at system or design engineers, operators,
and technicians, rather than software engineers or operational cybersecurity practitioners,
because the engineers who design, build, operate, and maintain the physical infrastructure
are best positioned to leverage a system’s engineering design to diminish the severity of
cyber-attacks or digital technology failures. This approach creates new opportunities for
engineering teams, not just cybersecurity teams, to secure the system using the physics and
mechanics of engineering controls, not just digital monitoring and controls.

CIE is a natural and strong complement to the ongoing work of multiple PES
technical committees. Initial engagement is underway between CIE practitioners at INL
and NREL, and the Analytical Methods for Power System Committee (AMPS), Power
System Communication and Cybersecurity Committee (PSCCC), and the Power System
Relaying and Control Committee (PSRC) to identify and pursue synergies on a more
formal scale.
125
TR-123 — IEEE PES Technical Roadmap

While the CIE and CCE concepts have not yet been formally mapped to the NIST
Cyber Security Framework (CSF) [9], this chapter is organized for easy mapping to the
NIST CSF functions as follows:

1. Prevention maps to NIST CSF protect,

2. Risk minimization maps to NIST CSF identify,
3. Detection maps to NIST CSF detect,
4. Response maps to NIST CSF respond,
5. Remediation/mitigation maps to NIST CSF recover.

Each function includes a current state, future state (trends), challenges, and
potential solutions. CIE and CCE are woven into the discussion throughout. In addition,
each current state, future state (trends), challenges, and potential solutions also includes
separate sections on people, process, and technologies as these typically map horizontally
across the NIST CSF (i.e., cross-cut NIST CSF). Finally, privacy is considered mostly
separate from cybersecurity. The IEEE Digital Privacy Initiative (DPI) has been
established to address privacy issues broadly, including groups addressing electric vehicles
and energy [10].

6.3 Current State

This current state analysis of cybersecurity in the power grid provides a baseline to
compare against the trends leading to the future state.

6.3.1 Prevention (protect)

Prevention involves several controls to provide defense-in-depth, including
preventing unauthorized access to assets and information. Like other OT systems used in
CPS, many assets being protected today are still limited in access control capabilities. One
mitigation technique is to install a security gateway that provides the needed access
controls while managing the limited capabilities of the assets.

Protection of operational data-in-motion commonly relies on authentication to

protect against threats, even when incorporating encryption is a better response to the
threats [11]. Even so, common practice is to wrap protocols with limited or unimplemented
security features like authentication within encrypted VPN tunnels. Renewables, as one
form of DER, suffer from implicit trust in their protection [12]. For example, lacking in
sophistication to verify commands and ensure integrity to protect against cyberattacks.

Protection against some cyberattacks like ransomware includes backups of

configurations, firmware, and programming software as restoration is easier than re-
creation. Today there are third-party applications that support active monitoring of
configurations across multiple vendors and assets and enforcing the use of only approved
firmware and programming software.

126
TR-123 — IEEE PES Technical Roadmap

General protection of assets is complicated because of limited capabilities for host-

based firewalls and endpoint security products on proprietary or embedded operating
systems. Additional defense-in-depth techniques provide mitigation when properly
applied. Baseline or standard configurations are a matter of policy that can be bolstered by
software functionality that removes the need for basic copy/paste (and edit) engineering.
Security gateways typically include the functionality to monitor for configuration changes
and alerting when an unauthorized change is detected. Unused ports and services can
expose an asset to cyberattack that could easily be avoided by properly identifying
unnecessary ports and services and disabling them. Within the industry, this capability has
improved but some vendors need to make improvements. Protection of assets through their
complete lifecycle, from purchase to disposal, is important and there have been
improvements in recent years on this front in part due to NERC CIP-013, Supply Chain
Risk Management.

Regular updating of firmware, programming software, and operating systems can

be difficult to evaluate due to limited information provided in vendor’s software release
notes. Typically those release notes include general references to vulnerability fixes (bugs)
in patches and differentiate those fixes from enhancements or improvements. Vulnerability
management through patching also requires testing of those patches for proper validation,
a step that many utilities bypass as acceptance of the latest and greatest versions are simply
approved for deployment. Having proper lab facilities where testing can take place can be
challenging with the proliferation of hardware options and application functionalities
embedded in part numbers. Other challenges include the cost and resources needed to
perform the testing.

The NERC CIP standards align closely with the NIST Cybersecurity Framework.
A comparison between the NIST Cybersecurity Framework and NERC CIP is posted on
the NIST website [13]. The E-ISAC promotes the use of the MITRE standards and attack
framework, as well as other maturity standard frameworks.

6.3.2 Risk minimization (identify)

A Cybersecurity risk assessment is a very important part of cybersecurity planning
that identifies the threats, vulnerabilities, and risks to assets. Cyber risk is informally
defined as the probability of a malicious cyber event happening, that is a threat actor
successfully exploiting a vulnerability, multiplied by the consequence of that cyber event.
The current practice for cyber risk assessment is “qualitative” in nature based on the
knowledge or opinion of experts to determine the risk probability as low, medium, or high;
and also the categorizing of the impacts as low, medium, or high. Such a qualitative risk
assessment is subjective in nature. A complementary approach is “quantitative” risk
assessment that relies on quantitative values of probabilities and consequences. Although
quantifying consequences is well understood, the quantifying of attack probabilities is not
well understood as it relies on threat quantification that is still more of an art than science.
Therefore, quantitative cyber risk assessment is not quite practical yet due to lack of
adequate models and mathematical tools. In recent years, there have been ongoing research
and development efforts to develop pragmatic mathematic models, using stochastic models
and game theory, to quantify threats and hence quantifying the attack likelihood for given
127
TR-123 — IEEE PES Technical Roadmap

scenarios. Information sharing programs exist such as the Cybersecurity Risk Information
Sharing Program (CRISP) that is a public-private partnership between the E-ISAC and the
U.S. Department of Energy (DOE) Office of Cybersecurity, Energy Security, and
Emergency Response (CESER) [14]. CRISP collects near-real time network traffic from
the participant’s networks using a dedicated device. Data is collected and analyzed to
discover cyber threat actors, pinpoint emerging trends, and analyze correlations for a
comprehensive picture of the cyber threat landscape of the electric sector. CRISP is
celebrating its tenth anniversary and is currently comprised of utilities that cover over 85
percent of U.S. electric customers and include some of the largest utilities in the nation.
CRISP continues to provide value to its participants and U.S. government agencies and has
been used to identify and track significant cyber threats from nation state adversaries and
cyber criminals. CRISP has successfully identified activities from DragonFly and other
Russian threat actors and more recently Volt Typhoon and other Chinese threat actors.

The Electricity Information Sharing and Analysis Center (E-ISAC) supports its
members, covering over 85% of the metered electricity customers in North America. E-
ISAC members include all investor-owned utilities, large public power companies,
Canadian crown corporations, and a significant number of municipally owned and
cooperative utilities. In addition, federal agencies, and other government partners such as
DHS, DOE, FBI, and the Canadian Centre for Cyber Security are part of the E-ISAC.
NERC members regularly share information with the E-ISAC. The E-ISAC also continues
to share threat intelligence, provide curated analysis and intelligence reports, and conduct
threat briefings. Anonymization of collected data is kept to a minimum and presents no
challenge.

To help minimize risks, security tools are now available that are OT protocol aware
that can automate asset discovery providing real-time assessment of their firmware /
software and data flows. This helps automate the identification and inventory of critical
OT assets’ hardware and software using passive techniques but require proper placement
in Ethernet networks to observe east-west traffic. These tools may also have active
scanning methods to provide more detailed information about the assets and potentially the
assets connected to them that are not connected to the Ethernet network. These tools will
then compare the discovered assets to known vulnerabilities, even being sensitive to
firmware versions and part numbers. Even with these tools being available, understanding
and documenting the data flows is still uncommon even with related NERC CIP
requirements.

The vulnerabilities, threats, and risks to assets has recently transformed with recent
publicity around supply chain cyberattacks with broad impact across many industries and
governments, even with global reach (e.g., the SolarWinds attack, Colonial Pipeline attack,
and Kaseya ransomware attacks in 2020 and 2021. Mitigating these supply chain risks has
highlighted the importance of a Software Bill-Of-Materials (SBOM) and a Hardware Bill-
of-Materials (HBOM) in OT systems. Unlike these attacks, DER compromise today is
largely limited given that many parts of the country are just starting to see significant DER
growth [12]. The impact of an attack will vary depending upon how much DER has been
deployed. A supply chain attack impacting grid-scale DER installations with a larger
128
TR-123 — IEEE PES Technical Roadmap

footprint across multiple utilities in a coordinated attack could be the next-generation

SolarWinds attack for the power grid.

In addition to these more technical controls, policies and processes are still critical
in minimizing risk. This includes understanding what processes are absolutely critical for
the power grid to be resilient, establishing roles and responsibilities in policies, and
rehearsing incident response plans.

6.3.3 Detection (detect)

Detecting threats or attacks is difficult as the end devices typically focus their
detection and reporting of any anomalies that are mostly related to system health status
reporting (e.g., power supply voltage, battery status, hardware failure, software failure) as
opposed to intrusion detection. More and more systems are automating or semi-automating
their system health monitoring processes to diminish the role of persons in the loop, saving
cost of performing the monitoring functions. Systems can also support manual or automatic
running of diagnostic routines from embedded test or external support test equipment to
detect the presence of failures or anomalies within a system.

Health monitoring is typically supplemented by an additional Intrusion Detection

System (IDS) to provide a form of system status reporting to monitor and alert when
suspicious activities are occurring. A SOC (security operations center) responds to these
alerts by investigating the event, determines the cause of the issue, and takes the required
steps to eradicate the threat. Signature Detection (SD) is one form of an IDS. Signature-
based IDS uses fingerprints or early warning indicators of known threats to be situationally
aware of the potential presence of previously identified threats’ observed behaviors and
signal characteristics. Anomaly Detection (AD) is another form of an IDS. The anomaly-
based IDS detects unknown malicious attacks as new attack methods are developed
quickly. AD data is used to update the SD historical statistical databases to be alert to future
occurrences faster. A hybrid IDS uses a combination of SD and AD methods to attempt
detection of potential threats with a minimum error rate.

SD and AD methods of an IDS need constant analyst intervention to be effective

for detecting malicious activity. These methods are prone to false alarms, whether false
positives or false negative alerts. As more and more data are collected, these false alarms
and overall detection error rates are expected to decrease.

6.3.4 Response (respond)

Despite the undertaken prevention, risk minimization, and detection functions,
compromise from a cyber-incident is still likely to occur. Rapid and effective response
helps in the remediation/mitigation, where the response does not inadvertently compound
the negative impact of a cyber-incident on the power grid. Testing response plans helps
ensure preparedness and effectiveness, including outreach to third parties and legal
reporting requirements and information sharing. Good response plans account for the
highly distributed nature of the power grid, where one critical asset could be compromised
to the aggregation of many non-critical assets. Response includes several activities that
129
TR-123 — IEEE PES Technical Roadmap

take place after a cybersecurity incident is detected, but also the practice of those activities
before an event occurs with lessons-learned applied to improve the response plans.

There are different approaches to response. One proactive approach is threat

hunting (three common approaches are structured, unstructured, or situational) that
deliberately hunts networks, systems, and devices using various methods (intelligence-
based, hypothesis-based, indicators-of-attack [IOA] based upon known threat-actor
Tactics, Techniques, and Procedures [TTPs], indicators-of-compromise [IOC] based upon
known malicious activity) and likely uses log collection from the detect function. Threat
hunters will gather as much information as possible before executing the response plan to
remediation/mitigation.

Besides any proactive approaches are more offensive responses such as:

 A counterattack, counter-hack, or hacking back; all should include some measure of

proportionality, with an ongoing ethical debate around self-defense or vigilantism as
opposed to seeking justice; but even with this debate in the USA there is the Computer
Fraud and Abuse Act (CFAA) [15] that still makes it a federal crime to access a
computer in any way without authorization.
 Honeypots or honeynets that attract attackers [16] at the risk of entrapment, so their
attacks are avoided with time-wasting in search of a better hack on a little-risk system
all the while increasing knowledge about the attacker’s TTPs.
 Sandboxes that help focus evaluation on possible attacks involving unauthorized
system components could already be underway and running that software on an isolated
system that is in quarantine.
 Simulations and walk-through tabletop exercises that primarily focus on testing
recovery plans to evaluate their effectiveness and weaknesses for potential lessons-
learned.
 Challenges/hackathons to help solve the various challenges that are exposed from
exercising response plans that can require innovation to improve response plans so they
are more productive.

One final example are reactive responses such as:

 Defensive is the classic example of enhancing existing defense-in-depth techniques that
have obviously failed with additional defenses that mitigate the ongoing attack.
 Documentation simply documents an incident took place and what responses were
undertaken because there is simply no defined response plan in place due to a
combination of the lack of the following: regulatory oversight, resources to develop
and implement a response plan, industry information sharing, or knowledge of industry
best practices.

6.3.5 Remediation/mitigation (recover)

The smart grid continues to evolve, rapidly growing in recent years in the number
of smart sensors and other devices used in monitoring and/or controlling the grid in support
130
TR-123 — IEEE PES Technical Roadmap

of numerous applications that are also rapidly growing. Resilience of the electric grid
includes the ability to recover from a cyber incident in a timely manner. Utilities should be
running their own exercises of recovery processes that help maintain operations and
afterwards performing lessons learned. NERC’s Electricity Information Sharing and
Analysis Center (E-ISAC) has been holding a biennial grid security and resilience exercise,
the seventh being held in 2023. NERC publishes their lessons learned from each GridEx to
help improve the security of the power grid [17].

6.3.6 People
To ensure the security of power grids, it is essential to invest in cybersecurity
education and workforce development. According to CyberSeek [18], an initiative funded
by the National Initiative for Cybersecurity Education (NICE), the United States utilities
sector faced a shortage of approximately 6,000 cybersecurity professionals from May
2022 through April 2023. In the USA, the DoE has programs in place to help develop a
strong and diverse cybersecurity workforce in the energy sector. One is the DoE
CyberForce® Program [19] that has been developing since 2016 to add to its original
competition with additional competitions, career fairs, webinars, and a workforce portal.
Another is the establishment in 2021 of a network of university-based, regional
cybersecurity research and development (R&D) centers across the nation that will help
develop a cybersecurity workforce that is well versed in both electrical engineering and
critical infrastructure cybersecurity requirements.

6.3.7 Processes
There are two major approaches to developing and managing effective
cybersecurity operations: a maturity-based approach or a risk-based approach. Electric
utilities have been using maturity models since first available around 2012 as a process to
assess the current level of performance and, if warranted, set programmatic goals and
priorities for improvement. A maturity model provides a set of indicators based on
responses to practice statements that represent the capability of an organization in a
particular discipline and its capability progression over time. The model and associated
tools can provide a comparison to show where improvements have been made since the
last assessment, what improvements are still needed, and can help set new goals to improve
its maturity over time.

The maturity model approach has been applied in several areas and domains to assess
the maturity of a variety of practices for different sectors. Several examples that are
applicable to electric power systems include the following [20]:

 The Cybersecurity Capability Maturity Model (C2M2) [21] was developed to assess
the maturity of an energy sector organization’s cybersecurity program (it is also used
by other critical infrastructure sectors). The model can be used to identify areas where
cost-effective enhancements can quickly improve cybersecurity programs. The current
version of the C2M2 (Version 2.1) uses 356 practices to evaluate performance across
12 domains.

131
TR-123 — IEEE PES Technical Roadmap

 The Transmission Resiliency Maturity Model (TRMM) [22] is a tool for electricity
transmission organizations to objectively evaluate and benchmark their current
transmission resiliency policies, programs, and investments. The objective is to assist
the transmission organization to target and prioritize improvements and enhance the
overall resilience of the power grid.
 The Secure Design and Development C2M2 (SD2-C2M2) [23] is a tool that can be
used by hardware, software, and system developers and integrators to assess their
design and development practices and procedures against a set of best-practice concepts
to determine the maturity level of their processes. The SD2-C2M2 assessment includes
over 800 individual practice statements covering the entire product development and
delivery lifecycle.

A risk-based approach is another approach to developing and managing effective

cybersecurity operations. Tools such as MITRE ATT&CK [24]can be used to help assess
how existing defenses stack up against attacker techniques, identify the highest priority
risks in current coverage, and modify defenses, or adopt new defenses, to address those
gaps.

6.3.8 Technologies
The prevailing technology landscape within the power grid encompasses a diverse
array of capabilities and structures, including:

 Legacy non-digital systems: Represented by analog devices like electromechanical

protection relays and OT systems that rely on analog sensors and communication
methods. The development and deployment of these systems by different
manufacturers have led to fragmented ownership structures. Their limited remote
connectivity insulates them from some external threats, although they remain
susceptible to physical attacks and insider risks, where insider risks could be increased
due to the limited dissemination of system functionalities.
 Unsecurable legacy digital systems: Comprising first-generation hardwired logic
systems intricately linked with legacy analog and serial communication protocols.
These purpose-built systems lack wide-scale adoption and pose unique challenges for
cybersecurity integration. Reliance on specific vendors or service providers for
maintenance and updates perpetuates vulnerabilities if support ceases, rendering them
exposed to cyber threats.
 Unsecured legacy digital systems: Evolving into second-generation systems featuring
microprocessor logic capable of supporting a range of communication protocols, both
legacy (serial-based) and modern (IP-based). However, the presence of difficult-to-
secure legacy protocols without IEEE 1711.2 and unsecured modern protocols like
DNP3 (IEEE 1815) with disabled cybersecurity options or unimplemented
cybersecurity features creates vulnerabilities. Ownership challenges, driven by budget
constraints or operational necessities, hinder upgrades. Interconnections with critical
infrastructure components expose broader networks and create a larger attack surface.
 Secured legacy digital systems: Advancing into later second-generation systems
equipped with integrated cybersecurity features, such as individual user access control,
132
TR-123 — IEEE PES Technical Roadmap

secure logging, and tamper detection in firmware. These systems support securable
versions of communication protocols like DNP3 with Secure Authentication or use
external cybersecurity features like VPNs or IPsec. Upgraded systems are typically
owned by entities prioritizing cybersecurity and modernization, often adhering to
industry regulations and standards.
 Unsecured modern digital systems: Representing current-generation digital systems
built on commercial hardware and software platforms. These systems possess the
potential for robust cybersecurity comparable to modern IT systems, including
individual user access control, secure logging, and software tamper detection.
However, various factors such as performance concerns, diagnostic capabilities, and
resource limitations may lead to the decision to forgo cybersecurity implementation.
Ownership spans from utilities and operators to third-party contractors involved in
system deployment and maintenance. Occasionally, inherent vulnerabilities arise from
a lack of cybersecurity awareness during development.
 Secured modern digital systems: Characterized by current digital systems with fully
enabled cybersecurity features. Ownership typically involves collaborative efforts
among cybersecurity experts, engineers, and operators. Entities prioritizing
cybersecurity weave cybersecurity measures into every stage of system development,
from conceptualization to deployment.

As the lifecycle of legacy systems concludes and replacements are introduced, the
integration of modern digital systems becomes evident. The increasing need to address
cybersecurity concerns is notably emphasized in the BES, while clear guidance for
distribution remains lacking. The deployment of cybersecurity features is often influenced
primarily by the need to meet regulatory requirements such as NERC CIP in the North
American BES, rather than considering good or best practices. Challenges of compatibility
and expertise may hinder the transition, but the eventual culmination of a secured
environment is essential to power grid operation. Balancing the coexistence of secured
(modern) and unsecured (legacy) systems during this evolution is vital for achieving a
comprehensive and secure future for the power grid.

Added to diverse array of capabilities and structures are emerging technologies.

The recently published IEEE-NERC joint report [25] outlines fourteen technologies poised
to reshape the power grid. This inclusive overview covers elements like electric vehicles,
cutting-edge computing innovations including quantum computing, and advanced
techniques like AI and machine learning. Significantly, this report goes beyond just
highlighting the potential benefits; it assesses potential risks tied to these technologies. It
also proposes a methodology for assessing the cybersecurity risks of emerging
technologies.

6.4 Future State (trends)

Application of trends to the current state of cybersecurity of the power grid provides
a glimpse of the future state. Some cross-cutting trends being observed today lead to the
following observations about a future state with:

133
TR-123 — IEEE PES Technical Roadmap

 Utilities having a regulatory agnostic, technically sound, risk-informed and

performance-based framework for implementing a cybersecurity program [26]
using a continuously improving common-language around countermeasure
terminologies [27] and adversarial tactics, techniques, and common knowledge.
 Considers potential threats and vulnerabilities at every stage of the engineering
process as utilities use CIE and incorporate cybersecurity as early in the design
process, including design of systems that provide ongoing monitoring to support
adaptation to new trends in threats and vulnerabilities.

6.4.1 Prevention (protect)

The future state for prevention is where utilities:

 Proactively defend OT systems to support operational safety as part of an essential,

well-designed, defense-in-depth security program [28].
 Measure their cybersecurity performance through a standard set of security metrics
that quantify the effectiveness of cybersecurity controls [26].
 Use reference architectures that help them design and deploy key security controls
quickly and analyze and mitigate emerging vulnerabilities and attack vectors in a
timely manner [26], such as DER ransomware, DER supply chain compromise,
DER botnets, and DER worms [12].
 Deploy effective protection controls in the multi-party grid with ubiquitous DERs
[26].
 Encrypt data-at-rest and data-in-motion that is impervious to quantum computing.
 Design, build, and operate OT systems using CIE that implements a zero-trust
model where data are validated using cryptographically secure mechanisms that are
informed by standards, testing, and vulnerability assessments [12].
 Defend against attacks that use AI or “AI for adversaries” that radically changes
the threat landscape through rapid discovery and use of zero-day vulnerabilities.
 Widely adopt password-less multi-factor authentication for regular users and
privileged account management solutions for OT devices and systems.
 Defend against attacks using more sophisticated toolkits that are applied to OT
systems requiring more specific mitigations for these systems [29].
 Develop and implement the appropriate safeguards to ensure availability of critical
systems [26].
 Protect against cyberattacks as an on-going process that avoids failures that develop
over time. Power grid OT assets are less often legacy devices and IT equipment
located in substations is less troublesome to maintain.

6.4.2 Risk minimization (identify)

The future state for risk minimization is where utilities:

 Identify and track field assets and efficiently manage their configurations and
firmware by integrating automated solutions [26].

134
TR-123 — IEEE PES Technical Roadmap

 Measure their cybersecurity performance through a standard set of security metrics

that quantify the effectiveness of cybersecurity controls [26].
 Transparently and confidently implement technical and procedural solutions that
resolve security challenges while effectively complying with mandatory standards
[26].
 Implement consistent, repeatable, regulatory agnostic, risk informed processes to
assess systems, services, and assets across critical infrastructure facilities, vendors,
and suppliers [26].
 Apply CIE to minimize risk by identifying potential threats and vulnerabilities so
that systems are built with the necessary controls, such as the recognition that a
cyberattack on many thousands or more DER or the overarching systems
controlling DER can adversely impact grid operations [12].
 Cybersecurity controls are deployed based on an organizational understanding of
actual risk to systems, assets, data, and capabilities. An integrated perspective of all
power, control, and communications systems is leveraged to understand cyber risk
and operational dependencies [26].

6.4.3 Detection (detect)

The future state for detection is where utilities:

 Use system health status monitoring and intrusion detection to automatically or

autonomously detect threats, failures, and anomalous conditions so that detection
happens in a timely manner [26]. New fault monitoring and IDS functions will be
developed to learn from data collected over the history of failure or anomalous
events to determine patterns, predict performance degradation to the point in time
when the system will be inoperative, and prevent future recurrences of the causes
of downtime. These new functions will apply AI and Machine Learning (ML) data-
driven methods to apply predictive analytics to perform prognostic functions. These
functions will be ever evolving to learn from the data, develop models, improve the
accuracy and confidence of those models, and make better predictions or
estimations of where and when the next system fault or anomalous event will occur.

 Key prognostic components of this system will include diagnostic and anomaly
detection, prognostics, and health monitoring (PHM) or prognostics and health
management, and closed loop automation. With PHM, the role of a cybersecurity
professional grows to include design of system architectures and pipelines to
manage data, models, and algorithms used for predictive analytics. PHM will
enhance not only cybersecurity posture, but also the system maintainability and
availability by providing early detection and diagnosis of equipment failures and
possibly predicting their future behavior.

 Data Acquisition Systems (DAS) are used to collect diagnostic and sensor data for
use in a PHM system. The DAS collects diagnostic data about the system
performance under certain conditions, while the sensor data provides
environmental stress data about the operating conditions of the power grid or
135
TR-123 — IEEE PES Technical Roadmap

equipment, such as temperature, humidity, pressure, strain, vibration, and other

relevant parameters. These sensor data provide the measure of stress accumulation
for fatigue/damage assessments to determine Remaining Useful Life (RUL). When
ML is used, feature selection and extraction are performed to look at the collected
data and transform these data into meaningful features to train the ML models using
data-driven methods. The ML methods require some form of anomaly detection
and outlier analysis to remove noise, outliers, or other unwanted data points before
the application of anomaly detection algorithms to improve the accuracy and to
reduce false positives.

 DAS data will be further processed for classification, categorization and analysis
using a variety of techniques based on signal processing, statistical analysis or ML.
The goal will be to identify patterns, trends, and anomalies that can indicate
deviations from normal or regular system operation conditions. Anomaly detection
and outlier analysis will be used to identify patterns and events deviating from
expected system behavior. Algorithms used for anomaly detection will fall under
statistical methods (e.g., z-score, mean-shift), ML (e.g., clustering, neural
networks, regression analysis, decision trees, random forests, and support vector
machines), time-series [e.g., autoregressive integrated moving average (ARIMA)],
deep learning (e.g., autoencoders), or hybrid methods (combining statistical and
machine learning). These hybrid methods are extremely valuable to apply multiple
algorithms and techniques combining results to improve accuracy and reliability of
the detection capabilities.

 PHM and DAS will help identify cybersecurity threats in OT systems by

monitoring system behavior and detecting anomalies in real-time. Anomalies will
include unexpected data traffic patterns, unusual data values, or other suspicious
behavior. These anomalies will then be analyzed by cybersecurity experts to
determine whether they represent a genuine threat to the power grid or safety of
workers or the public. In addition, PHM will be used to monitor system logs and
other data sources to detect known threats and vulnerabilities, allowing system
administrators to take action to prevent or mitigate the effects of attacks. PHM
algorithms will be trained on historical data including temporal dimensions to
ensure they are effective at identifying and responding to threats specific to the OT
system. Data preparation, feature selection, model training, and model validation
will be performed to ensure the accuracy of the models. The necessary changes
required to integrate PHM into an existing system will be significant and likely
require a collaborative effort across a multidisciplinary team of experts in data
science, engineering, and cybersecurity.

6.4.4 Response (respond)

The following trends in response lead to a future state where utilities:
 Have the tools and capabilities, such as orchestration and automation, to conduct
effective forensics analysis in an OT environment [26].

136
TR-123 — IEEE PES Technical Roadmap

 Develop and implement the appropriate technology and processes to act during a
cyber event to mitigate impacts and maintain or recover critical operational
capabilities [26].
 Plan for new future threats [26].
 Use AI to run tabletop exercises to improve their cybersecurity response
capabilities by providing more realistic and challenging exercises specific to their
environment, cybersecurity program, and cybersecurity maturity, automating the
attack simulation, and providing real-time feedback and analysis of the exercise
results.
 Legal issues have been minimized to help utilities understand the legal boundaries
of responding to attackers.

6.4.5 Remediation/mitigation (recover)

The future state for remediation/mitigation is where utilities:
 Backup configurations from thousands of OT devices is performed regularly so that
recovery is timely and effective. Utilities understand the loss of critical
transformers that can take months or years to replace and also understand that
physically visiting thousands of locations to restore backups can take months or
years to complete. Utilities determine minimum functionality levels for OT assets
so that restoration plans are appropriately prioritized. Utilities communicate
restoration plans to third-parties and include them in the exercise of restoration
plans.
 Regularly consider technologies that can improve operations through resiliency that
accelerates remediation.
 Deploy sandboxes to gain insight into attacker behavior while also minimizing the
risk of access to unauthorized OT systems and have the funding and resources to
perform eradication when needed.
 Use tabletop exercises to practice public relations so that messaging around a cyber
event is appropriately managed and any lessons learned are documented and
improvements made; and to gain useful and actionable intelligence from the
exercise.
 Patch OT devices and systems on regular basis using a laboratory setting to first
understand the impacts of the patch on the specific implementation in their OT
systems and to understand the challenges of performing the validation.

6.4.6 People
The future state for people is where utilities:
 Have the right workforce skills to implement their cybersecurity programs such that
there is only limited shortage of qualified cybersecurity professionals.

6.4.7 Processes
The power grid will evolve to embrace emerging technologies and in so doing,
utilities will also embrace the twelve principles of CIE to help secure the power grid by
137
TR-123 — IEEE PES Technical Roadmap

using the physics and mechanics of engineering controls, not just monitoring and controls.
As noted earlier, these twelve principles crosscut protect, identify, detect, respond, and
recover and so are discussed in more detail under the future state of processes. Utilities will
adopt the following twelve principles of CIE that include processes and whose results will
be unique to each utility based on many factors such as device implementation, system
architecture, service territory, customers, and risk tolerance. The future state for processes
is where utilities:
 Determine the critical functions utilities must ensure in the power grid and what
undesired consequences must be prevented in those systems (CIE Principle 1:
Consequence-Focused Design).
 Select and implement controls to reduce attack surface or an attack’s impact by
using engineered solutions that are tied to physical processes (CIE Principle 2:
Engineered Controls).
 Prevent undesired use or misuse of data-at-rest or data-in-motion in each identified
critical system (CIE Principle 3: Secure Information Architecture).
 Determine what features of critical systems are unnecessary for achieving critical
functions (CIE Principle 4: Design Simplification).
 Determine how to best layer defenses to protect each critical function from
experiencing the undesired consequences (CIE Principle 5: Layered Defenses).
 Proactively prepare to defend each critical function from any threat (CIE Principle
6: Active Defense).
 Understand how critical functions are related to each other functions and either
adversely impact them or be adversely impacted by them (CIE Principle 7:
Interdependency Evaluation).
 Understand what digital assets are used in each critical function, their capabilities
(whether implemented or not), and any assumptions about how they operate (CIE
Principle 8: Digital Asset Awareness).
 Ensure that their providers deliver the security required to limit the impact of
compromise (CIE Principle 9: Cyber-Secure Supply Chain Controls).
 Include in their resilience calculations the resilience of their security controls from
cyber-attack (CIE Principle 10: Planned Resilience).
 Proactively practice knowledge management about each critical function and keep
it out of the wrong hands (CIE Principle 11: Engineering Information Control).
 Work to ensure that each employee’s behavior and decisions align with their
security goals (CIE Principle 12: Organizational Culture).

6.4.8 Technologies
The future state for technologies is where utilities:
 Embrace the wave of emerging technologies that are disrupting the industry today
and are adept at evaluating the cybersecurity of emerging technologies so that they
can be proactively evaluated before CIE is even implemented for them. This trend
will continue and expand, so that distinguishing between fleeting trends and
enduring technologies become crucial. Emerging technologies reshape
cybersecurity in the power grid. AI and ML take the spotlight, enhancing threat
detection and response capabilities. These smart algorithms not only spot anomalies
138
TR-123 — IEEE PES Technical Roadmap

but adapt to new attack methods, bolstering the electric sector’s defenses.
Blockchain technology guards and strengthens data integrity and transaction
accuracy within complex energy systems. By creating unchangeable records,
blockchain guarantees transparent and trustworthy data.
 Establish collaborative models that not only encompass utilities (e.g., CRISP), but
also vendors, experts, and regulators, sharing responsibility for fortifying
infrastructure. This collective approach fosters shared knowledge, best practices,
and uniform cybersecurity protocols, strengthening the electric sector’s ability to
combat evolving threats.
 Harmonize transformation with resilience. Rigorous testing will use an on-line or
off-line environment that extends the contingency classes to include cyber intrusion
events even for emerging technologies, collaboration, and risk management
strategies will secure and provide a resilient future for the power grid.

6.5 Challenges
With a glimpse of the future state and current state, gaps emerge as challenges to
achieve the desired future state are more numerous than can be highlighted here. Some
cross-cutting challenges are:

 ISA99 WG14 is only working on four profiles for ISA/IEC 62443 that do not
encompass all the interfaces/use cases defined in IEEE 2030. The challenge is an
incomplete common language/model and use cases, especially the emerging use cases
from the grid edge. This presents significant challenge to having a regulatory agnostic,
technically sound, risk-informed and performance-based framework for implementing
a cybersecurity program in the future state.
 Adoption of CIE in the electric sector is still emerging and will be a challenge to gain
widespread adoption by utilities.

6.5.1 Prevention (protect)

The challenges to achieving the future state in prevention include:

 Defense of OT systems to support NERC CIP compliance for the North American BES
is generally not proactive and does not explicitly include operational safety as part of
any requirement. The distribution grid is even more fractured with state regulations in
the United States of America varying greatly on layered defenses, let alone being
proactive or explicitly addressing safety. Defense-in-depth strategy has been extolled
by multiple organizations, such as the Cyberspace Solarium Commission report [30]
and NERC, in their Security Integration Strategy [31]. Defense-in-depth strategies are
gaining widespread acceptance and adoption, but face challenges due to
misconfigurations and inconsistent implementations.
 Measurement of cybersecurity performance is best done via maturity models that is
embraced by other standards like IEC 62443 while NERC CIP relies on utilities to
choose and adopt any framework that work best for each utility. A standard set of

139
TR-123 — IEEE PES Technical Roadmap

security metrics is unavailable that quantifies the effectiveness of cybersecurity

controls.
 Development of cybersecurity reference architectures is needed to help utilities design
and deploy key security controls quickly so that they can analyze and mitigate emerging
vulnerabilities and attack vectors in a timely manner, such as DER ransomware, DER
supply chain compromise, DER botnets, and DER worms.
 Cybersecurity controls in IEEE 1547.3 is only guidance that needs elevation to
standards to create broad impact on the cybersecurity of DERs.
 Approaches to encrypt data-at-rest and data-in-motion are needed that are impervious
to quantum computing.
 Adoption of CIE in the electric sector is still emerging and may be a challenge to gain
widespread adoption by utilities.
 Widespread adoption of MITRE ATT&CK and DEF3ND is needed so that utilities
have a common language that can speed defenses against rapidly evolving attacks that
use “AI for adversaries”. Continuous monitoring of network connectivity and
performance alerts against malicious actors can also help. Detection of anomalies as
early as possible is important in reducing attack impact and aiding in faster recovery.
Continuous monitoring across wide area networks and with coverage across all assets
in the OT systems continues to be a challenge, especially when compromised DER
could dwarf the number of installed OT devices installed in substations and at many
utilities, out in the field located on poles and towers [12].
 OT devices are still incapable of role-based access control, let alone password-less
multi-factor authentication for regular users and privileged account management
solutions. Restricting access to critical resources to only approved users at the right
time and to the right level presents several challenges. Access control at the grid edge
and for devices outside NERC CIP’s purview such as DER are an immediate concern.
Access control can be implemented in multiple ways such as physical access control
(badging to enter restricted areas), role-based access control methods (such as
Lightweight Directory Access Protocol, LDAP) via security gateways, or rule-based
access control methods (such as Virtual Private Networks, VPN). ISO 27001 and IEC
62351-8 define various levels of access control through compliance certification.
 Defending against attacks developed using more sophisticated toolkits specific to OT
systems and grid-supply scale [12] requires more specific mitigations, layered
defenses, improved segmentation, and improved risk-based methods for the expected
grid-supply scale that go beyond current cybersecurity frameworks.
 Developing and implementing the appropriate safeguards to ensure availability of
critical systems. Penetration testing of defenses presents challenges to OT systems
supporting the power grid. Abundant caution and planning that involves all
stakeholders (e.g., operations, engineering, leadership) will help minimize the risk but
not remove the inherent risk of unintended consequences during the testing [28].
 Patching or not patching legacy OT devices anywhere and IT assets in substations, is
an on-going challenge in protecting the power grid. These challenges are reported by
multiple entities, including the U.S. Government Accountability Office [32]. Power
grid OT assets are often legacy devices where new vulnerabilities are discovered on a
regular basis.

140
TR-123 — IEEE PES Technical Roadmap

6.5.2 Risk minimization (identify)

The challenges in identification include the following:
 Identification and tracking of field assets using automation improves the efficiency of
baselining and maintaining asset inventories but is not yet widely deployed and will
not track isolated systems and may not find all assets (such as those serially connected
and not network-discoverable). Managing configurations and firmware using
automation is not widely deployed in part due to the proprietary nature of configuration
and firmware files.
 Measurement of cybersecurity performance is qualitative without a standard set of
security metrics that quantify the effectiveness of cybersecurity controls.
 Implementation of technical and procedural solutions to transparently and confidently
resolve security challenges while effectively complying with mandatory standards.
 In North America the driver is the NERC CIP regulatory requirements with built-in
risk assessments for OT systems, services, and assets across critical infrastructure
facilities, vendors, and suppliers. Cyber risk is often measured by combining the factors
of vulnerabilities, threat probability, and consequence. However, this approach
presents a problem that it is not a true mathematical formula; rather a heuristic that
enumerates the factors that contribute to measuring risk. Measuring the effect of these
factors individually also poses a challenge.

The Common Vulnerability Scoring System (CVSS) provides an empirical score for
different vulnerabilities in digital devices that relies on expert opinions in grading the
severity of risk and needs to be placed in the context of the OT system.

Accurately calculating the consequence or impact of a particular threat is complicated

in the case of cyber-related failures and threats as the modeling and understanding of
these systems are not fully developed yet. For example, a cyberattack on today’s DER
may have only a local consequence on grid operations; however, more DER
connections are occurring with integrated communications, monitoring, and control.
This means the attack surface increases with broader impact whose tipping point will
differ for each utility and how the DER are deployed and used in grid operations [12].

Considering that measuring cyber risk is a complicated process, there is yet to be a

consensus on the best method to identify and minimize cyber risk. Various methods
have been proposed, such as the Factor Analysis of Information Risk (FAIR) model
[33]. Given the sheer variety of threats and OT systems under study, a more realistic
way of identifying and minimizing risk is to use different models for different threats,
such as MITRE Insider Risk Behavioral Framework [34].
 Adoption of CIE in the electric sector is still emerging and may be a challenge to gain
widespread adoption by utilities.
 An integrated approach for cyber and physical security into the planning, design, and
operational phases of the power grid is needed. The IEEE PES and NERC recently
published TR105 [25] that also includes an integrated perspective of all power, control,
and communications systems to better understand cyber risk and operational
dependencies.
141
TR-123 — IEEE PES Technical Roadmap

6.5.3 Detection (detect)

The following challenges in detection to achieve the future state are:
 Essentially the same challenges utilities face for prevention because in the rapidly
evolving landscape of detection systems that include diagnostics and PHM systems,
the intersection with observability and cybersecurity emerges as a critical frontier.

6.5.4 Response (respond)

The following challenges in response to achieve the future state are:
 Monitoring physical security, network infrastructure, communications infrastructure,
OT devices, and OT systems still requires mostly customized orchestration and
automation for situational awareness and to conduct effective forensics analysis after a
cybersecurity incident or event. There is a lack of trust in the orchestration and
automation that is further complicated by adding AI to the mix, even if to make further
improvements in the results [26].
 Technology challenges are like the previous item, with the addition of challenges in
developing processes for more proprietary OT devices and systems that help mitigate
impacts and maintain or recover critical operational capabilities [26]. The risk is that
the response involving OT systems and OT devices typically requires specialized
knowledge that includes non-traditional operating systems and can severely impact the
power grid when improperly operated [28]. Response to cyberattacks on the power grid
is dictated by the impact of the attack. The highest priority is often assigned to attacks
that have a direct physical impact, such as the cyber led failures in Ukraine in 2015 and
2016. However, response is also dictated by impact in terms of monetary value or the
number of people affected – for example, companies might choose to pay the ransom
amount in case of a ransomware attack at least temporarily, as demonstrated in the U.S.
Colonial Pipeline compromise. For North America, NERC CIP 008-5, Incident
Reporting and Response Planning, details requirements for incident response plans
being in place, regularly exercised, and regularly reviewed, and disclosure. NIST
provides an incident response lifecycle in NIST 800-61-2 [35] where there are
challenges across preparation as well as detection and response.

In the preparation phase, utilities often face challenges in understanding the adequate
levels of preparedness – is compliance with regulations sufficient, or do they need to
do more, considering the rapidly evolving threat landscape? In addition, there are
challenges in justifying the financial investments required to be completely prepared,
such as paying for additional user trainings.

In the detection & analysis phase, the biggest challenge is usually dealing with false
positives and accurately identified. In addition to the technical challenges, this is
usually the phase where the victims need to make public statements or at least legal
disclosures about the event. Managing public relations and the associated fallout from
this reporting is a challenge that requires strategic planning and careful consideration.

142
TR-123 — IEEE PES Technical Roadmap

 Planning for new future threats is challenged by constraints in funding and resources
for regular tabletop exercises, exclusion of mutual aid for cybersecurity events, and
understanding at C-level suite of the threats to OT systems and the real risks of
devastating consequences [26].
 AI tools for running tabletop exercises are emerging and require further improvement
to be more specific to the existing environment of people, processes, and technologies
so that they create realistic scenarios within the correct context.
 A legal framework for hack-back is needed to address the weaknesses in the Computer
Fraud and Abuse Act (CFAA) that was passed in 1986 well before most Americans
purchased computers.

6.5.5 Remediation/mitigation (recover)

The following challenges in remediation/mitigation to achieve the future state are:
 Backup of configurations from OT devices in substations is still a challenge today even
though the technology to do so has been available for about a decade. The limitations
are quickly encountered with the thousands of smart grid devices being installed from
unsupported vendors. Utilities have not associated the loss of thousands of OT device
configurations taking months if not years to restore, a similar impact when a critical
transformer is lost. Utilities do not have a fundamental understanding of minimal
functionality of OT assets that could allow quicker restoration. Utilities are just learning
about communicating restoration plans to third-parties and including them in their
tabletop exercises.
 Utilities are understanding now that resilience is a key consideration for stakeholders
– OT systems are rarely failure-proof and they need to prioritize faster recovery and
resilience. Challenges around resilience are multifold, including how to optimize
recovery by optimally positioning maintenance crews in case a cybersecurity event
causes damage to physical equipment. The use of technological solutions is also
gaining prominence – for example, how can SDN, NDN, and newer technologies help
with faster recovery?
 In the containment, eradication, & recovery phase, the challenges vary depending on
the type of cyber event. Sandboxing techniques can be deployed to gain further insight
into attacker behavior, but these come with risks that attackers could gain unauthorized
access to safety critical systems. Eradication might not be required for denial of service
(DoS) attacks. However, when eradication is necessary to ensure Advanced Persistent
Threats (APTs) are eradicated, the resources and time required can create financial
burdens.
 In the post-incident activity phase, utilities often face a combination of the challenges
mentioned in the other phases – mainly around (i) managing messaging around the
incident and documenting lessons learned, and (ii) resource challenges (in terms of
personnel and finances) to invest in gaining useful and actionable intelligence.
 Remediation/mitigation of a software vulnerability using a patch can expose patch
management challenges plus validation challenges, especially when a laboratory
environment is unavailable.

6.5.6 People
143
TR-123 — IEEE PES Technical Roadmap

The following challenges in people to achieve the future state are:

 The shortage in cybersecurity professionals is particularly significant when considering
that the employed cybersecurity workforce at utilities in the U.S. totals approximately
10,000. An international perspective can be found from the International Information
System Security Certification Consortium, or (ISC)², that reports a shortage of 3.4
million cybersecurity professionals across all industries worldwide [36]. In addition,
more women are needed in the cybersecurity along with other underrepresented
minorities to bring in diverse perspectives to protecting the power grid.

This global shortage serves as a resounding wakeup call for the authorities and
industries, particularly those in the electric sector, to allocate greater efforts and
resources toward workforce development in the field of power grid cybersecurity. The
challenge with training workforce in this area is one of cross-training, as often electrical
engineers need to be trained in cybersecurity, or computer science graduates need to be
trained to understand grid physics. There is a growing acknowledgement of this
challenge as reflected by the increasing number of programs addressing cyber-physical
system training, and specialized workforce development programs aimed at training
engineers on dealing with cyber risk. However, more efforts are required to ensure that
the workforce is well compensated, and to ensure that working in critical infrastructure
security becomes an attractive job profile.

6.5.7 Processes
The following challenges in processes to achieve the future state are:
 Utilities do not have a fundamental understanding of the critical functions they must
ensure in the power grid and what undesired consequences must be prevented in those
systems (CIE Principle 1: Consequence-Focused Design).
 Utilities are focused more on compliance in the North American BES than best
practices that reduce attack surface or an attack’s impact and do not use engineered
solutions that are tied to physical processes unless required to do so (refer to the
industry response to the AURORA attack [37])(CIE Principle 2: Engineered Controls).
 Protection of data-at-rest or data-in-motion in each identified critical system is typically
not performed by utilities, and typically only performed by utilities on critical systems
subject to regulations such as NERC CIP in North America (CIE Principle 3: Secure
Information Architecture).
 Utilities are hampered by vendors whose OT devices do not allow features to be
disabled when they are determined by the utility to be unnecessary for achieving critical
functions (CIE Principle 4: Design Simplification). Utilities typically only disable
unnecessary features when subject to regulations such as NERC CIP in North America.
 Utilities typically only determine how to best layer defenses to protect critical functions
from experiencing the undesired consequences on OT systems when subject to
regulations such as NERC CIP in North America. (CIE Principle 5: Layered Defenses).
 Utilities typically only proactively prepare to defend critical functions from any threat
when those OT systems are subject to regulations such as NERC CIP in North America.
(CIE Principle 6: Active Defense).

144
TR-123 — IEEE PES Technical Roadmap

 Utilities typically do not understand how critical functions are related to each other
functions and either adversely impact them or be adversely impacted by them (CIE
Principle 7: Interdependency Evaluation). This is likely caused by the bright-line
criteria in NERC CIP-002, Cyber Security — BES Cyber System Categorization, a
risk-based method for the BES. Otherwise, this kind of fundamental understanding
takes time and expertise to develop that is challenged by resource constraints typical at
most utilities (e.g., availability and expertise).
 Utilities typically do not understand what digital assets are used in each critical
function, their capabilities (whether implemented or not), and any assumptions about
how they operate (CIE Principle 8: Digital Asset Awareness). This is likely caused by
NERC CIP having its own risk-based method for the BES that all utilities in North
America. Otherwise, this kind of fundamental understanding takes time and expertise
to develop that is challenged by resource constraints typical at most utilities (e.g.,
availability and expertise).
 Utilities subject to compliance to NERC CIP in North America are becoming better at
ensuring that their providers deliver the security required to limit the impact of
compromise (CIE Principle 9: Cyber-Secure Supply Chain Controls). Otherwise,
without local regulation, this kind of fundamental supply chain security takes time and
expertise to develop that is challenged by resource constraints typical at most utilities
(e.g., availability and expertise).
 Utilities expertise in designing resilient systems has been challenged in recent years
from various high-impact low-frequency events. Ensuring that any security controls are
also resilient from cyber-attack is challenged by perceived costs and complexities (CIE
Principle 10: Planned Resilience).
 Utilities subject to compliance to NERC CIP in North America are becoming better at
proactively practicing knowledge management about each critical function and keep it
out of the wrong hands (CIE Principle 11: Engineering Information Control).
Otherwise, without local regulation, this kind of fundamental knowledge management
takes time and expertise to develop the capabilities and maintain them that is challenged
by resource constraints typical at most utilities (e.g., availability and expertise).
 Utilities subject to compliance to NERC CIP in North America are becoming better at
working to ensure that each employee’s behavior and decisions align with their security
goals (CIE Principle 12: Organizational Culture). Otherwise, without local regulation,
this kind of alignment between policy and people takes time and expertise to develop
the capabilities and maintain them that is challenged by resource constraints typical at
most utilities (e.g., availability and expertise).

6.5.8 Technologies
The following challenges in technologies to achieve the future state are:
 Scrutiny of emerging technologies needs to evaluate several factors, such as safety and
reliability, but also cybersecurity as a fundamental design issue (i.e., applying CIE) so
that securing the emerging technology is NOT more emergent than the technology
itself.

145
TR-123 — IEEE PES Technical Roadmap

The IEEE-NERC joint report [25] was just published and only generally evaluates
fourteen technologies and does not include others [28] (e.g., cloud-based services for
OT systems, including remote monitoring-only of configuration and analysis of
engineering operations telemetry, historian-in-the-cloud, and third-party managed
services such as firewall management and vulnerability scanning) have mostly
completely emerged and established; new emerging technologies need to be evaluated
using a standard process. A comprehensive set of metrics is needed to evaluate
emerging technologies.

While this landscape of emerging technologies offers immense potential to bolster grid
efficiency and reliability, the technological surge brings with it a dual-faced challenge.
On one hand, some technologies, while promising, may not yet be primed for full
integration, potentially exposing vulnerabilities that can lead to cyber anomalies. For
instance, cloud computing might heighten cybersecurity risks, whereas blockchain
contributes to heightened cybersecurity and reliability. Smart sensors, communication
networks, cellular infrastructure, and advanced detection mechanisms stand at the
forefront of grid operation monitoring. These technologies provide the means to
monitor grid operations, detect anomalies, and facilitate timely responses to reliability,
privacy, and cybersecurity threats.

Some of these groups may carry inherent cybersecurity flaws that will surface over
time. Additionally, the tradeoffs between cybersecurity, privacy, performance, agility,
operability, maintainability, resiliency, and technological and human resources must
be assessed before determining what solutions can or will be implemented at the risk
of an insecure and fragile power grid.

Among the numerous challenges faced in this technology-driven paradigm, technology

ownership emerges as a significant barrier in effective risk management. The
boundary-spanning nature of many emerging technologies introduces complexities that
transcend organizational, regional, and jurisdictional boundaries. In line with this, the
concept of system boundaries becomes paramount, such as those external and internal
boundaries defined in NIST SP 800-82r2 control SC-7. These boundaries facilitate the
monitoring and control of communications at the designated boundaries, curbing
unauthorized communications through the implementation of boundary protection
devices like gateways, routers, firewalls, guards, and encrypted tunnels.

 CRISP includes utilities, not vendors, experts, and regulators, so that responsibility is
shared for fortifying infrastructure. A more collective approach is needed that could
foster shared knowledge, best practices, and uniform cybersecurity protocols,
strengthening the electric sector’s ability to combat evolving threats.
 Creating an off-line environment for testing is expensive for just testing typical utility
automation functions and even that requires constant maintenance to remain effective.
Adding a cybersecurity capability to a laboratory that can include real-time digital
simulation dramatically increases the costs and skills needed to maintain the laboratory
equipment.

146
TR-123 — IEEE PES Technical Roadmap

6.6 Potential Solutions

With a glimpse of the challenges to achieve the desired future state some cross-
cutting potential solutions are proposed:
 Acceleration of the completion of the work of ISA99 WG14 is needed so that lessons
can be learned and applied for developing roadmaps for addressing other use cases so
that there is a regulatory agnostic, technically sound, risk-informed and performance-
based framework for implementing a cybersecurity program for the power grid.
 One way to build adoption of CIE by electric utilities is to incorporate CIE into the
standards used to build the power grid in the IEEE and other Standard Development
Organizations (SDOs). As of December 2023, the Standards Working Group of the
CIE Community of Practice [38] is considering how to work with the IEEE, ISA, and
IEC to include CIE concepts into key standards.

6.6.1 Prevention (protect)

The potential solutions to overcome the challenges and achieve the future state in
prevention include:
 Regulatory agencies such as NERC should consider explicitly including operational
safety as part of appropriate cybersecurity requirements. To address the distribution
grid, the ongoing development work by ISA WG14 developing cybersecurity profiles
for the power grid should do the same in IEC 62443. The good news here is that the
purpose of IEC 62443 “is to improve the safety, integrity, availability, and
confidentiality of components or systems” [39], defines an “essential function” as a
“function or capability that is required to maintain health, safety, the environment, and
availability for the equipment under control” [39], and adds safety to the traditional
cybersecurity triad of Confidentiality, Integrity, and Availability [39].
 Regulatory agencies such as NERC should consider explicitly including maturity
models into their compliance model for new registered entities. To address the
distribution grid, IEC 62443 already includes maturity levels to address the capabilities
of utilities to implement the specified requirements. In addition, EPRI has identified
[26] a near-term goal to identify in the next three years a common metrics framework
for quantitative measures of the effectiveness of cybersecurity controls, with metric
adoption in four-to-seven years. Adoption could be fostered by working with SDOs to
develop a standard for these metrics that utilities can adopt and vendors can develop
tools.
 ISA WG14 is developing four profiles in IEC 62443 per item 1 above that will help
utilities design and deploy key security controls quickly so that they can analyze and
mitigate emerging vulnerabilities and attack vectors in a timely manner, such as DER
ransomware, DER supply chain compromise, DER botnets, and DER worms (one IEC
62443 profile is planned to include DER).
 The IEEE P1547 working group is considering elevating the cybersecurity
recommendations in IEEE 1547.3 to requirements.
 As the GAO documents [40], work is progressing on approaches to encrypt data-at-rest
and data-in-motion that is impervious to quantum computing.
 Refer to item 2 above.

147
TR-123 — IEEE PES Technical Roadmap

 Widespread adoption of MITRE ATT&CK and DEF3ND could be accelerated by

incorporating them into existing or new standards specific to the electric sector. Adding
requirements to the upcoming cybersecurity profiles in IEC 62443 need to be
considered for continuous monitoring of network connectivity (including wide area
networks) and performance alerts.
 Adding requirements to the upcoming cybersecurity profiles in IEC 62443 needs to be
considered for OT devices to be minimally capable of role-based access control and
optional password-less multi-factor authentication for regular users and privileged
account management solutions.
 Adding requirements to the upcoming cybersecurity profiles in IEC 62443 needs to be
considered for defending against attacks with more specific mitigations, layered
defenses, improved segmentation, and improved risk-based methods.
 Adding requirements to the upcoming cybersecurity profiles in IEC 62443 needs to be
considered for developing and implementing the appropriate safeguards to ensure
availability of critical systems during penetration testing of defenses.
 DOE CESER CEDS “Patch and Update Management Program for Energy Delivery
Systems” has gone inactive and may have provided a potential solution [41]. EPRI has
also published a report and associated training for patch management guidelines that is
not publicly available [42]. The best near-term solution is for EPRI to provide these
materials to the public free of charge to help foster any technical, process, or training
to help reduce patching anxiety.

6.6.2 Risk minimization (identify)

The potential solutions to overcome the challenges and achieve the future state in
risk minimization include:
 Developing a standard approach for reporting asset attributes in an OT environment
beyond IEC 61850 and DNP3 while making those available in those standards
mandatory at a more basic level. The standard IEC 61850 provides logical nodes with
a mix of mandatory and optional attributes related to asset attributes related to their
identification and tracking (e.g., model, part number, serial number, firmware);
however, not all OT devices support IEC 61850 or if they do, support the optional
attributes. IEEE 1815 (DNP3) also provides object 0 with several variations that
include attributes related to asset inventory that are required for Level 4
implementations; however not all OT devices support DNP3 or if they do, support
object 0 variations. IEEE PES PSRC Working Group H27 developing IEEE PC37.251
“Standard for Common Protection and Control Settings or Configuration Data Format
(COMSET)” includes a method for storing configuration files that could help in
managing configurations once widely adopted by the electric sector.

Asset inventories will eventually increase beyond the ability of a utility to manage
configurations and firmware without using automation and without mandatory
requirements in regulated or unregulated segments of the electric sector. Even IEC
62443 addresses inventory management but does not require automated management
of the inventory. Developing a requirement for automated management may be the only
potential solution to drive industry acceptance of tools that support the functionality
148
TR-123 — IEEE PES Technical Roadmap

and the development of utility processes around managing the configurations and
firmware (refer to item 13 above). Using COMSET may be another tool in the toolbox
instead of or in addition to IEC 61850 and DNP3 data.
 Creating quantitative cybersecurity metrics for the effectiveness of cybersecurity
controls is addressed by item 4 above.
 Implementing technical and procedural solutions is addressed by item 4 above.
 NIST made public in 2021 some information regarding a Cyber Risk Scoring Program
but there have been no public updates made of the program. An approach from NIST
that is applicable across multiple critical infrastructure sectors could help address
making risk calculations more transparent and meaningful, including an integrated
perspective that includes all power, control, and communications systems.
 Adopting CIE is addressed by item 2 above.
 Creating an integrated approach to cyber and physical security into the planning,
design, and operational phases of the power grid is addressed by item 17 above.

6.6.3 Detection (detect)

The potential solutions to overcome the challenges and achieve the future state in
detection include:
 The potential solutions that could evolve from a symbiotic relationship between cyber-
anomaly detection system, fault diagnostic systems, fault tolerant systems, fault
recovery and resilient systems, PHM systems, and cybersecurity, require developments
into frameworks, models, technologies, and applications that fortify the reliability,
resilience, and cybersecurity of complex and intelligent systems.

Increasing adoption of miniaturization, ubiquity, and big data within and across cyber-
physical systems is needed as they become more prevalent in diverse power and energy
industrial sectors. These developments would tackle the vulnerabilities inherent in
hyperconnected environments that demand innovative solutions at the convergence of
the electric sector with cybersecurity. Additionally, these developments would delve
into human-centric security considerations, adaptive and intelligent defenses, the
interconnected nature of PHM ecosystems, regulatory compliance, and incident
response and recovery strategies. By fostering interdisciplinary collaboration and
presenting cutting-edge research, these developments would contribute to secure,
reliable, and human-centric power and energy systems across diverse applications and
industries.

One potential solution may be what the IEEE PES is already working on: implementing
integrated continuous monitoring capability with applicable logging functionality in
the PSRC’s Task Force H55 on Distributed Cyber Physical Assessment for Grid
Resilience.

6.6.4 Response (respond)

The potential solutions to overcome the challenges and achieve the future state in
response include:
149
TR-123 — IEEE PES Technical Roadmap

 Monitoring physical security, network infrastructure, communications infrastructure,

OT devices, and OT systems is similar to the potential solution discussed to item 20
above.
 SDOs like the IEEE have difficulty developing processes for proprietary OT devices
and systems to help mitigate impacts and maintain or recover critical operational
capabilities. Without utilities adopting standards-based solutions and SDOs like the
IEEE creating the requisite standard for using commonly available operating systems
that meet protection and control performance requirements with robust testing and
certification, the risk is that specialized knowledge of non-traditional operating systems
will continue to severely impact the power grid when improperly operated.

In the preparation phase, a regulatory agnostic, technically sound, risk-informed and

performance-based framework for implementing a cybersecurity program is a potential
solution as discussed in item 1 above.

In the detection & analysis phase, dealing with false positives could be addressed by
the advances discussed in item 20 above. In addition to the technical challenges, a
standard that includes how to address public statements or at least legal disclosures
about an event. Managing public relations and the associated fallout from this reporting
is a challenge that requires strategic planning and careful consideration that would be
helpful to include in standards as discussed in item 1 above. Some of the
recommendations stemming from GridEx VI included: defining roles for
communicating with roles of individuals responsible for communicating with
employees, NERC Reliability Coordinators, utilities, trade associations, the Electricity
Subsector Coordinating Council (ESCC), the Energy Government Coordinating
Council (EGCC), state/provincial and local governments, regulators, and the media;
provide the infrastructure and resources required to support these roles through an
extended 24-hour period; understanding the correct level of detail needed in
communications in those roles [43].
 Creating a regulatory that helps electric utilities plan for new future threats by providing
new methods of funding and resources for regular tabletop exercises. Emergency
response plans for cyber events need to leverage “mutual aid” or “emergency
assistance” for cybersecurity events and the restoring or maintaining electrical service
is deemed to be necessary or advisable. Educating the workforce, even at C-level suite,
is needed as addressed by item 31 below, so that threats to OT systems and the real
risks of devasting consequences are understood by all workforce members.
 With the rapid adoption of AI to solve many problems, the IEEE is busy developing
standards such as the IEEE 2247 series of standards for Adaptive Instructional Systems
(AIS) or others listed by the IEEE-SA [44] or being worked on by the IEEE Artificial
Intelligence Standards Committee that might be applied to AI tools for running tabletop
exercises.
 Developing a national legal framework for hack-back to address the weaknesses in the
Computer Fraud and Abuse Act (CFAA), but proposals continue to fail. Without
promising legislative action, compensating measures described by many potential
solutions above will relieve pressure on hack-back operations by improving

150
TR-123 — IEEE PES Technical Roadmap

cybersecurity controls, monitoring tools, funding for cybersecurity research, and

education and training.

6.6.5 Remediation/mitigation (recover)

The potential solutions to overcome the challenges and achieve the future state in
remediation/mitigation include:
 A regulatory agnostic, technically sound, risk-informed and performance-based
framework for implementing a cybersecurity program is a potential solution from item
1 above that could help improve the available technical solutions for regularly backing
up configurations from thousands of OT devices in a timely and effective manner. The
framework needs to include the determination of a minimum functionality level for OT
assets so that restoration plans are appropriately prioritized. Extending the concepts
from item 22 above, utilities need to communicate restoration plans to third-parties and
include them in the exercise of restoration plans.
 A regulatory agnostic, technically sound, risk-informed and performance-based
framework for implementing a cybersecurity program is a potential solution from item
1 above that could help regular consideration of technologies that can improve
operations through resiliency that accelerates remediation.
 Utilities should deploy sandboxes to gain insight into attacker behavior while also
minimizing the risk of access to unauthorized OT systems and have the funding and
resources to perform eradication when needed. Guidance from SDOs or NIST might
increase the deployment of sandboxes (NIST has referenced sandboxing in several
publications, but not dedicated a publication to it).
 Extending the concepts from item 22 above, utilities need to use tabletop exercises to
practice public relations so that messaging around a cyber event is appropriately
managed.
 A regulatory agnostic, technically sound, risk-informed and performance-based
framework for implementing a cybersecurity program is a potential solution from item
1 above that could help with patching of OT devices and systems on regular basis using
a laboratory setting to first understand the impacts of the patch on the specific
implementation in OT systems and to understand the challenges of performing the
validation.

6.6.6 People
The potential solutions to overcome the challenges faced in power grid security and
foster a well-prepared workforce to safeguard critical infrastructure include:
 Identifying the leading challenges and devising a comprehensive plan by improving
curriculum relevance, remediating the shortage of qualified instructors, improving
training resources, and improving awareness about cybersecurity careers. Additionally,
the evolving threat landscape necessitates continuous professional development and
upskilling. Establishing partnerships to provide internships, cooperative programs, and
research opportunities for students can greatly contribute to addressing these
challenges.

151
TR-123 — IEEE PES Technical Roadmap

Raising awareness and conducting outreach initiatives are essential for targeting
students, parents, and educators, making them aware of the criticality of cybersecurity
and hopefully inspiring more individuals to pursue careers in this field. Developing
outreach events led by cybersecurity specialists for K-12 students can be an effective
approach.

Developing curriculum and programs should focus on providing comprehensive

education that covers a broad range of engineering disciplines and the application of
security principles and practices. Given the current limited number of programs in this
area, increased collaborative efforts and resources are needed for program development
and accreditation. Additionally, fostering collaboration between academic institutions
and industries is crucial to cultivate more skilled trainers in cybersecurity.

Equipping students with industry training and certifications so they have knowledge
about cybersecurity concerns specific to power grids. Incorporating training workshops
delivered by power systems cybersecurity professionals can be effective in addressing
these concerns. Encouraging students to pursue cybersecurity certifications helps them
gain industry insights and perspectives. This practice is beneficial not only for students
but also for energy industry employees. Tailoring certifications based on expertise and
needs enables individuals to enhance their knowledge in specific areas. Continuous
learning and professional development are vital for cybersecurity professionals in the
power and energy industry to keep pace with evolving threats and technologies.
Engaging in ongoing learning through technical events, certifications, studying
recently published cybersecurity content, and joining professional associations and
communities such as the IEEE PES PSCCC enable specialists to stay informed about
the latest trends, share experiences, and seek guidance from peers in the field.

6.6.7 Processes
The potential solutions to overcome the challenges and achieve the future state in
processes include:
 Extending the training discussed in item 31 above to provide the utility workforce with
a fundamental understanding of the critical functions they must ensure in the power
grid and what undesired consequences must be prevented in those systems (CIE
Principle 1: Consequence-Focused Design).
 Developing a regulatory agnostic, technically sound, risk-informed and performance-
based framework for implementing a cybersecurity program is a potential solution as
discussed in item 1 so that utilities become more focused on best practices that reduce
attack surface or an attack’s impact and use engineered solutions that are tied to
physical processes (CIE Principle 2: Engineered Controls).
 Developing a regulatory agnostic, technically sound, risk-informed and performance-
based framework for implementing a cybersecurity program is a potential solution as
discussed in item 1 so that utilities protect data-at-rest or data-in-motion in each
identified critical system (CIE Principle 3: Secure Information Architecture).
 Developing a regulatory agnostic, technically sound, risk-informed and performance-
based framework for implementing a cybersecurity program is a potential solution as

152
TR-123 — IEEE PES Technical Roadmap

discussed in item 1 so that vendors develop OT devices that allow features to be

disabled when they are determined to be unnecessary for achieving critical functions
(CIE Principle 4: Design Simplification).
 Developing a regulatory agnostic, technically sound, risk-informed and performance-
based framework for implementing a cybersecurity program is a potential solution as
discussed in item 1 so that utilities determine how to best layer defenses to protect
critical functions from experiencing the undesired consequences on OT systems. (CIE
Principle 5: Layered Defenses).
 Developing a regulatory agnostic, technically sound, risk-informed and performance-
based framework for implementing a cybersecurity program is a potential solution as
discussed in item 1 so that utilities proactively prepare to defend critical functions from
any threat. (CIE Principle 6: Active Defense).
 Developing a regulatory agnostic, technically sound, risk-informed and performance-
based framework for implementing a cybersecurity program is a potential solution as
discussed in item 1 so that utilities understand how critical functions are related to each
other functions and either adversely impact them or be adversely impacted by them
(CIE Principle 7: Interdependency Evaluation).
 Developing a regulatory agnostic, technically sound, risk-informed and performance-
based framework for implementing a cybersecurity program is a potential solution as
discussed in item 1 so that utilities understand what digital assets are used in each
critical function, their capabilities (whether implemented or not), and any assumptions
about how they operate (CIE Principle 8: Digital Asset Awareness).
 Developing a regulatory agnostic, technically sound, risk-informed and performance-
based framework for implementing a cybersecurity program is a potential solution as
discussed in item 1 so that utilities are better at ensuring that their providers deliver the
security required to limit the impact of compromise (CIE Principle 9: Cyber-Secure
Supply Chain Controls).
 Developing a regulatory agnostic, technically sound, risk-informed and performance-
based framework for implementing a cybersecurity program is a potential solution as
discussed in item 1 so that utilities have expertise in designing resilient systems and
that any security controls are also resilient from cyber-attack (CIE Principle 10:
Planned Resilience).
 Developing a regulatory agnostic, technically sound, risk-informed and performance-
based framework for implementing a cybersecurity program is a potential solution as
discussed in item 1 so that utilities proactively practice knowledge management about
each critical function and keep it out of the wrong hands.
 Developing a regulatory agnostic, technically sound, risk-informed and performance-
based framework for implementing a cybersecurity program is a potential solution as
discussed in item 1 so that utilities work to ensure that each employee’s behavior and
decisions align with their security goals (CIE Principle 12: Organizational Culture).

6.6.8 Technologies
The potential solutions to overcome the challenges in technologies to achieve the
future state are:

153
TR-123 — IEEE PES Technical Roadmap

 SDOs developing a standard method to evaluate emerging OT technologies that

evaluates several factors, such as safety and reliability, while mitigating risks; however,
it is imperative that this evaluation include cybersecurity as a fundamental design issue
(i.e., applying CIE) so that securing the emerging technology is NOT more emergent
than the technology itself. Extending the method proposed in the original IEEE-NERC
joint report [25] may be an approach.

Assessing emerging technologies against a comprehensive set of metrics tailored to the

unique needs of each entity or business is a critical step in technology adoption that
aligns with operational requirements and avoids introducing significant risks. These
evaluations encompass various dimensions, including technological capabilities,
scalability, usability, interoperability, and readiness. Additionally, strategic alignment
with business goals and cybersecurity and privacy considerations are integral.
Extending the impact of emerging technologies to evaluating capability enhancements,
scalability, usability, technology readiness, industry adoption, market viability,
development costs, resilience, cybersecurity, privacy, and technological dependencies.
Performing extensive research and vigilant monitoring are imperative so that emerging
technologies attain a level of maturity that enables their seamless and secure
integration.

Embracing emerging OT technologies requires effectively managing their associated

challenges through a multifaceted and collaborative approach. By evaluating
technologies comprehensively, fostering collaboration, staying informed through
research and monitoring, and adhering to industry standards, the electric sector can
proactively address challenges and create a secure and resilient power grid of the future.

 Advocating for the expansion of CRISP so that it not only includes utilities, but also
vendors, experts, and regulators, so that responsibility is shared for fortifying
infrastructure. A more collective approach could foster shared knowledge, best
practices, and uniform cybersecurity protocols, strengthening the electric sector’s
ability to combat evolving threats.
 Developing a regulatory agnostic, technically sound, risk-informed and performance-
based framework for implementing a cybersecurity program is a potential solution as
discussed in item 1 so that there are more market pressures for more-cost effective off-
line environments for utilities to build so that testing becomes more cost effective.

6.7 References
[1] NIST Special Publication 1500-201, published in 2017, available online at
https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1500-201.pdf.
[2] IEEE Std 1402-2021, “IEEE Guide for Physical Security of Electric Power
Substations”.
[3] NERC Glossary of Terms, updated in March 2023, available online at
https://www.nerc.com/pa/Stand/Glossary%20of%20Terms/Glossary_of_Terms.p
df.
154
TR-123 — IEEE PES Technical Roadmap

[4] NERC Reliability Standards from NERC, available online at

https://www.nerc.com/pa/Stand/Reliability%20Standards%20Complete%20Set/R
SCompleteSet.pdf.
[5] NIST SP 800-82, Rev 3, available online at
https://csrc.nist.gov/pubs/sp/800/82/r3/ipd.
[6] The U.S. Department of Energy’s (DOE) National Cyber-Informed Engineering
(CIE) Strategy Document, available online at
https://www.energy.gov/ceser/articles/us-department-energys-doe-national-cyber-
informed-engineering-cie-strategy-document.
[7] Cyber-Informed Engineering, available online at https://inl.gov/cie/.
[8] Consequence-Driven Cyber-Informed Engineering, available online at
https://inl.gov/cce/.
[9] NIST Cybersecurity Framework, available online at
https://www.nist.gov/cyberframework.
[10] IEEE Digital Privacy Initiative, available online at https://digitalprivacy.ieee.org/.
[11] IEEE Std 1815, “IEEE Standard for Electric Power Systems Communications-
Distributed Network Protocol (DNP3)”.
[12] “Cybersecurity Considerations for Distributed Energy Resources on the U.S.
Electric Grid”, U.S. Department of Energy, October 2022, available online at
https://www.energy.gov/sites/default/files/2022-
10/Cybersecurity%20Considerations%20for%20Distributed%20Energy%20Reso
urces%20on%20the%20U.S.%20Electric%20Grid.pdf.
[13] “Benefits of an Updated Mapping between the NIST Cybersecurity Framework and
the NERC Critical Infrastructure Protection Standards”, available online at
https://csrc.nist.gov/pubs/cswp/21/updated-mapping-between-nist-csf-and-nerc-
cip-stan/final.
[14] Cybersecurity Risk Information Sharing Program (CRISP), available online at
https://www.eisac.com/s/crisp.
[15] https://www.law.cornell.edu/uscode/text/18/1030.
[16] “Power Grid Honeypot Puts Face on Attacks”, October 5, 2023, available online
at https://threatpost.com/power-grid-honeypot-puts-face-on-attacks/116217/.
[17] https://www.nerc.com/pa/CI/ESISAC/Pages/GridEx.aspx.
[18] CyberSeek, “Cybersecurity Supply/Demand Heat Map,” June 13, 2023, available
online at https://www.cyberseek.org/heatmap.html.
[19] https://cyberforce.energy.gov/.
[20] https://www.pnnl.gov/pnnl-maturity-models.
[21] https://c2m2.doe.gov/.
[22] https://trmm.labworks.org.
[23] https://sd2-c2m2.pnnl.gov.
[24] https://attack.mitre.org/.
[25] “Towards Integrating Cyber and Physical Security for a More Reliable, Resilient,
and Secure Energy Sector (TR105)”, available online at https://resourcecenter.ieee-
pes.org/publications/technical-reports/PES_TP_TR105_PSCC_120622.html.
[26] “Cyber Security Roadmap”, EPRI, December 2018.
[27] https://d3fend.mitre.org/.

155
TR-123 — IEEE PES Technical Roadmap

[28] “SANS ICS/OT Cybersecurity Survey: 2023’s Challenges and Tomorrow’s

Defenses”, available online at https://www.sans.org/white-papers/ics-ot-
cybersecurity-survey-2023s-challenges-tomorrows-defenses/.
[29] https://www.cisa.gov/news-events/cybersecurity-advisories/aa22-103a.
[30] https://cybersolarium.org/march-2020-csc-report/march-2020-csc-report/.

[31] “Security Integration Strategy”, NERC, December 2022, available online at

https://www.nerc.com/comm/Documents/NERC_Security_Integration_Strategy_2
022.pdf.
[32] “Electricity Grid Cybersecurity”, available online at
https://www.gao.gov/assets/gao-21-81.pdf.
[33] https://www.fairinstitute.org/.
[34] https://mitre-engenuity.org/cybersecurity/center-for-threat-informed-defense/our-
work/insider-threat-ttp-knowledge-base/.
[35] http://dx.doi.org/10.6028/NIST.SP.800-61r2.
[36] “(ISC)2 Cybersecurity Workforce Study, 2022” available online at
https://www.isc2.org//-/media/ISC2/Research/2022-WorkForce-Study/ISC2-
Cybersecurity-Workforce-Study.ashx.
[37]
https://cdn.selinc.com/assets/Literature/Publications/Technical%20Papers/6467_C
ommonQuestions_MZ_20101209_Web.pdf.
[38] https://inl.gov/cie/.
[39] “ISA‑62443-1-1 Security for Industrial Automation and Control Systems” , April
2022, available online at https://www.isa.org/products/isa-62443-1-1-2007-
security-for-industrial-automat.
[40] “Securing Data for a Post-Quantum World”, available online at
https://www.gao.gov/products/gao-23-106559.
[41] “Patch and Update Management Program for Energy Delivery Systems”, U.S.
Department of Energy, available online at
https://www.energy.gov/sites/default/files/2021-
04/Patch%20and%20Update%20Management%20Program%20for%20Energy%2
0Delivery%20Systems%20-%20Foxguard%20Solutions.pdf.
[42] “Patch Management Guidelines”, EPRI, available online at
https://www.epri.com/research/products/000000003002011187.
[43] “GridEx VI Lessons Learned Report”, NERC, available online at
https://www.nerc.com/pa/CI/ESISAC/GridEx/GridEx%20VI%20Public%20Repo
rt.pdf.
[44] https://standards.ieee.org/initiatives/autonomous-intelligence-systems/standards/.

156