2024 Paris Session

ID – 10304
B5 Protection & Automation
PS1 – Practical experiences and new developments of process bus

Digital substation with process bus: grid operator and PACS manufacturer
feedback 2 years after the commissioning

Marcel PAYET Gérard CHAROT Valentin BOUVIGNIES

EDM Siemens EDF
France France France
m.payet@
[email protected] [email protected]
electricitedemayotte.com

Samir EL HADI Apolline MAZAS Fouad BENSGHIR

EDM Siemens EDF
France France France
s.el-hadi@
[email protected] [email protected]
electricitedemayotte.com

SUMMARY

EDM (Electricité de Mayotte) is the grid operator and the single utility of the French overseas territory
Mayotte composed of 2 small islands located in the Indian Ocean. As their HV-MV grid has been
developing quickly in the last decade, EDM decided in 2017 to build a new substation on the main island
to secure the power supply of the south of the archipelago.

Consequently, EDM asked EDF to help them to define the technical specifications of the Protections,
Automation and Control System (PACS) of this new substation. Together, EDM and EDF decided to
take advantage of the international standard IEC 61850 to build an innovative numerical PACS with a
Station Bus and a Process Bus.

The technical specifications were written in 2018. In 2019, Siemens was chosen as the main
manufacturer for the substation and for its PACS. The studies and the building of the PACS happened
in 2019 and 2020. The Factory Acceptance Test of the whole system took place in 2020; then, the Site
Acceptance Test in Mayotte led to the commissioning of the first Siemens Process Bus in France at the
end of 2021.
All the steps of a numerical PACS building cycle successfully happened between 2017 and 2021 and
nowadays, this innovative PACS have been running for at least 2 years. What is the feedback of the grid
operator which is using it daily? Does the system run as it was expected? Are the expected benefits of a
full numerical substation useful for the operators?

On one hand, what technical choices were made by EDM and EDF during the writing specification stage
and why? On the other hand, what technical choices were made by Siemens during the studies and the
building stage and why? Finally, after a 2-years running time, were those choices pertinent?

EDM, EDF and Siemens decided to gather their viewpoints on this pioneering project and to write lesson
learnt, which is the subject of this article. The aim is to describe the main characteristics of the project
and to share the successes, and the difficulties the 3 actors – the operator EDM, the manufacturer
Siemens, and the engineering support EDF – met during each step of the project.

The first part will deal with the technical characteristics: networks architectures (Process Bus, Station
Bus), configuration, Merging Unit and IED redundancy, etc. The second part will deal with the feedback
of each actor regarding the different aspects of the project. Finally, we will end with the main
knowledges we learnt, and we want to share.

1
Introduction

The project involves the construction of an indoor modular 90 kV/20 kV substation. Below, a
simplified single-line diagram of the project is presented.

Figure 1: Simplified single line diagram of Sada 90kV/20kV substation.

The PACS aspect of this project aimed to be as standard as possible, hence the choice of the IEC61850
standard as the employed framework. Starting from this standpoint, the strategy involved identifying the
minimal technical choices for designing an IEC61850-based PACS to meet the operator's needs while
also allowing the responsible engineering team to enhance their technical specification drafting skills.
Indeed, for engineering support and grid operator, this represented their initial project involving an
IEC61850-based PACS.

Engineering support

The engineering is managed by the CI2T entity, which is EDF's unit specialized in the field of HV
electrical substations and transmission systems, as well as the HV connections from power production
plants.

The project specification phase needed to focus on critical technical choices, stemming from the
following requirements:

❖ Implementation of a Station Bus and Process Bus,

❖ Architecture requirements,
❖ Redundancy of PACS equipment requirements (IED, Operator Station, etc.),
❖ Functional requirements (protections, data acquisition, state logging, etc.).

Regarding the decision to implement a Station Bus and Process Bus, this choice is driven by the
requirement to ensure data acquisition as close to the process as possible. Implementing the Process Bus
also serves as the initial step towards integrating non-conventional current/voltage transformers. While

2
the project does not currently include these types of transformers, the specified PACS's scalability will
enable progression in that direction in the future.

The Station Bus has been specified in two rings configuration HSR (High Seamless Redundancy). The
first ring is dedicated to the HV part of the substation and the second ring for the MV part. This decision
was made to minimize the number of connections, knowing that the Station Bus's availability does not
impact the protectability of the outgoing lines.

As for the Process Bus, the perspective differs, and the requirement is maximum availability for the
protection scheme of the installations. Accordingly, the Process Bus network is logically specified in
PRP (Parallel Redundancy Protocol). However, an important clarification is that the Process Bus only
concerns the HV part of the substation. The MV switchgears are only connected to the Station Bus. This
choice is because of the close proximity between the signals to be digitized and the IEDs for the MV
part of the substation. In fact, the current and voltage transformers are located within the bay that
integrates the IEDs.

The figure below shows a simplified architectural diagram:

Figure 2: SADA Project PACS architecture

The redundancy requirement does not differ from a classic PACS. The philosophy is to have 2 IEDs
supplied with Merging Units for the HV circuits to avoid common modes. For the MV circuits, like in
existing PACS, only one IED is installed. Indeed, the choice is to maintain redundancy of tripping
through a wired bus.

We have also specified the provision of 2 redundant operator workstations and redundant telecontrol
gateways. The objective was to ensure a high level of availability for both local and remote control of
the substation.

After these structuring choices, it was also necessary to address the performance aspect of the PACS.
With the proliferation of networks and acquired data, a new risk emerges: the loss of visibility of the
workstation and/or the protectability of assets due to network saturation. To prevent this risk,
performance criteria have been set, and tests from the supplier are expected to meet these criteria, for
example:
❖ The maximum processing time for the entire chain shall be less than 20ms,

3
 ❖ The maximum error percentage of the chain shall be less than 5%,
❖ The maximum load of the Station Bus shall be less than 70%.

For engineering support, this project is part of a larger action plan in which the use of the standard is
progressive:
❖ Step 1: Specify a PACS with a minimal set of requirements to have a standardized and proven
PACS,
❖ Step 2: Specify a PACS by creating the initial configuration files (SSD),
❖ Step 3: Specify a PACS by completing the entire configuration process and specifying the data
model (SSD, SCD with an EDF built tool).
The Sada project constitutes the first step of this strategic plan. The configuration aspect of the substation
PACS is entirely prescribed within Siemens scope of supply.

Regarding data exchanges, the decision was made to ask the industrial partner Siemens to build them
based on the normative framework but without specific requirements. The goal was to learn how
exchanges are conducted while benefiting from the supplier's expertise.

In order to enhance grid operator and engineering support skills and prepare for the operational
maintenance of the PACS, the following elements were requested as part of the contract: the system
configurator, IED configurator, and various configuration files (ICD, SCD, CID).

4
Manufacturer lesson learnt :

We will address, in the first part, the major technological choices made by the manufacturer while
providing commentary. Then, we will summarize the best practices to replicate for this type of project.
The specifications of engineering support were precise regarding the protection plan to be adopted,
whether for the 20 kV grid equipment or those of the 90 kV grid. However, for defining the architecture
of the digital control-command system, the broad outlines were imposed, just like communication
according to the IEC 61850 standard for the station bus and process bus, without, however, imposing a
specific modeling. On the other hand, a strong requirement focused on the redundancy of the system at
the 90 kV level as well as the central system. The process bus was specified for the 90 kV sections (for
remind: 2 lines, 2 transformers, and 1 bus section). The 20 kV level, on the other hand, was specified
without a process bus.
The process bus is based on MU (Merging Unit) type Siprotec5, which allows great modularity in terms
of digital input-output cards, as well as the capability, with a single MU, to digitalize all analog signals
of currents and voltages from different sections to publish them, enabling various IEDs (Intelligent
Electronic Devices) to subscribe to these streams. Within the Modular Indoor Substation, the MU
cabinets are located closest to the interfaces of the CT and VT terminals, all conventional instrument
transformer for this station. The cabinets for IEDs, on the other hand, are situated in the station's control-
command room. To ensure increased availability of the installation, it was decided to have redundancy
for the MU-IED pairs and protection functions. OM3 LC duplex optical fibers ensure communication
between publisher (MU) and subscriber (IED).

If we delve into the analog part, for the lines, we digitalize 3 currents + neutral for protection functions,
but with different analog inputs for section protection and busbar protection functions, as well as a cable
mass current and the 3-line voltages. Therefore, for the lines, we are digitalizing 8 currents and 3
voltages. At this stage, it can be noted that this type of connection of various currents on test sockets
allows independent testing of functions ANSI21, ANSI51, ANSI87BB, and ANSI87T.

Figure 3: Analog signal for line bay connected on MU over test socket.

It was decided, for testing and maintenance operations, to perform analog injections at the MU level
rather than injecting Sample Values (SV) in simulated mode. This choice was dictated by the on-site
maintenance team's practices, which carry out this type of test through analog injection without
introducing new concepts compared to simulated SV injections using SV injectors. We preserved the
team's competence, ensuring operational durability without requiring additional procedures or new
training to maintain system performance and reliability. In summary, the maintenance procedures for
testing protection functions remain the same as those in a system without a process bus. The only
difference lies in performing injections on the MU through analog circuits and observing the behavior
of IEDs subscribed to these streams. For the transformer section, the same principle applies with the
measurement of 12 currents distributed across upstream and downstream currents, as well as the
transformer neutral current.

5
For the number of digitalized analog values, we opted to transmit 2 different streams: a first stream of
currents and voltages required for the protection functions of each bay, and a second stream of current
solely for differential bar protection. Thus, the stream size is not oversized, and each subscriber
subscribes to information necessary for its specific function. The streams are compliant with IEC61869-
9 standard and are configured according to the bays to F4800S1I8U3 and F4800S1I4U0. The station
size is not critical in terms of the number of SVs to be published. To ensure proper network design, we
only verified by calculation the bandwidth used by the SVs. For each line, the published streams are 7.4
Mbit/s, for each transformer 7.7 Mbit/s, and for the control bar 4.6 Mbit/s.

Figure 4: Process bus bandwidth calculator.

Then, we verified that we were below 60-70% of the communication board capacities (access point).
This rule allows for bandwidth reserves for other streams (Goose and PTP in our architecture). In this
station's architecture, the station bus and administration networks are separate from the process bus. This
means that the IEC 61850 MMS Goose streams are not on the same network as the process bus. No
other VLAN disposition, MAC address filtering, or other measures were implemented because, from a
bandwidth perspective, the process bus used in the station was not critical. During the FAT, we verified
the bandwidth used by SV streams using a network probe.

If we now focus on the architecture of the optical process bus network, we have chosen a Parallel
Redundancy Protocol (PRP) according to the IEC 62439-3 standard. Indeed, the specifications clearly
indicated that the system should be easily and simply expandable.

6
 Figure 5: Full IT network architecture overview. Purple for process bus.

This type of PRP network offers redundancy, ensuring continuous communication even in the event of
failure through high-density port switches. It guarantees high data availability and minimizes downtime
by using parallel communication paths. The PRP ensures maximum resilience by eliminating Single
Points of Failure (SPOF), ensuring high reliability in critical substation networks. It also allows
transparent recovery without data loss, crucial for protection applications, while simplifying network
management and maintenance through its robust and automated architecture. Furthermore, adding a
section to a PRP network is simple by adding additional connections of new equipment to the switches,
unlike in an HSR (High Seamless Redundancy) architecture where the optical loop must be opened to
insert new equipment (thus temporarily creating a weak point in the network), an operation that must be
coordinated between maintenance and operation teams.

The use of two redundant GNSS PTP clocks with a Best Master Clock Algorithm (BMCA) ensures
precise and reliable clock synchronization in the process bus network. This redundancy provides
increased resilience, reducing the risk of synchronization loss in case of clock or link failure. The BMCA
algorithm automatically selects the best primary clock from the two, ensuring an optimal
synchronization hierarchy to maintain temporal coherence in the network to synchronize SV flows
between MU and IED. This test has been carried out during FAT. This approach offers increased time
stability for applications in electrical substations using the process bus. The 2 GNSS clocks also serve
to synchronize PCs and timestamp alarms, the logbook, etc. The Simple Network Time Protocol (SNTP)
was used, offering time synchronization without requiring a complex infrastructure. These two
protocols, PTP and SNTP, are directly supported by the network, unlike PPS-type synchronizations that
require additional circuits. Testing of malicious activities such as jamming/spoofing was not conducted
as part of this project.

Redundancy of Man Machine Interface (MMI) and substation computers in the system ensures critical
operational continuity. In case of hardware or software failure, redundant stations guarantee without
interruption and automatic transition to a backup unit, thus avoiding interruptions. This redundancy
enhances system availability, reduces downtime, and maintains operations without disruption, ensuring

7
high reliability in the critical environment of the Supervisory Control and Data Acquisition (SCADA)
station. Moreover, this architecture provides better fault tolerance, improving system resilience and
ensuring business continuity without compromising safety or performance. It's worth noting that the two
operator stations (of which only one is active at a time in terms of order processing) are equipped with
two screens, allowing better organization and visualization of the network segments one wishes to
observe or manage, thereby enhancing productivity and efficiency.

During the FAT and SAT tests, we configured the switches to perform port mirroring on a "test" port.
This provided increased visibility of network traffic. This technique allows in-depth analysis of data
flow without disrupting the primary network. By monitoring traffic on one or more ports, we were able
to diagnose and confirm SV and Goose frames non-invasively. Port mirroring also facilitates real-time
troubleshooting, enabling quick problem resolution and serves as a valuable tool for configuring and
optimizing complex networks, thus offering improved engineering operations management. We also
conducted specific tests, of which here is a simplified list. To assess resilience in case of GMC clock
loss, we carefully analyzed SV frames and their quality bits. Additionally, we verified the robustness of
PRP strand loss detection and examined associated signaling. Furthermore, we utilized cybersecurity
tools such as “Nmap” to validate the appropriate configuration, performing network scans to identify
and secure open ports.

We also installed a router firewall in the system, intended to allow remote access, providing flexible
system management and control from any location. This capability was particularly advantageous for
taking control of the system in Mayotte island from engineering teams in France situated 8000 km away.
Remote access enables quick issue resolution by allowing engineers to diagnose and address problems
remotely, thereby reducing intervention time and travel costs. Unfortunately, despite implementing
several cybersecurity measures, this functionality could not be implemented due to authorization
difficulties in opening access through the grid operator Information Technology (IT) network. Over the
course of two years of operation, there have been no major breakdowns or issues, except for a single
equipment that went into a safe mode a few months after commissioning, requiring on-site intervention.
The problem was quickly resolved by conducting online equipment diagnostics. The fully redundant
electrical substation did not suffer from this event. In this case, remote access would have allowed
problem resolution from a distance, avoiding a several-day intervention to organize travel.

Key best practices to replicate:

❖ A thorough analysis of flows between the publisher and subscriber is necessary for this type of
project to ensure proper system operation, implementing devices if needed to meet the SV
bandwidth limit.
❖ Despite being costlier due to requiring high-density port switches, implementing the process
bus on PRP networks is preferable.
❖ Consider the skills and resources of field teams to ensure future post-SAT testing.
❖ Delve deeper and implement the potential of the IEC 61850 standard with simulated modes and
tests for SV to automate FAT and SAT.
❖ Leverage the best practices acquired by engineering teams to be more efficient during FAT and
SAT (using network probes, establishing network test points, etc.).
❖ Emphasize the need for operational remote access while maintaining effective cybersecurity
protection.

Grid operator feedback/experience

The involvement of grid operator teams has been continuous throughout all stages of the design of the
Sada substation PACS (Power Automation and Control System). This was the case during the PACS
specification phase, the Factory Acceptance Test (FAT), and also the Site Acceptance Test (SAT).

During the FAT, grid operator teams were present for one week. The feedback received is that this
duration is insufficient. For the transfer of facilities after the SAT, maximizing the presence of future

8
operators is necessary for the effective takeover of the system. The first HV maintenance deadlines have
arrived, and there is a need to enhance the team's knowledge.

After 2 years of operation, the PACS has encountered few issues. Almost all technical difficulties were
detected in the initial months of operation.

❖ Hardware Failures:
o Frozen Merging Unit: Restarted the equipment, and no further issues occurred.
o Automatic current control for compensated neutral network (connected via wired
connection with the PACS - third part equipment).
❖ Software Failures:
o Operator station crash: Quickly resolved with the deployment of a patch.
o Occurrence of an automatic disconnection information for MV line automation in the
logbook: manufacturer has been engaged and solve this issue.

Apart from these difficulties, which were mostly resolved shortly after commissioning, the operator did
not observe any issues specific to the PACS. The protection system was also not compromised and
operated according to the implemented settings.

Throughout the first operational year, grid operator organized the shifting of several MV lines. During
these shifts, operators were able to update the name of the migrated lines in the PACS particularly in the
Operator working station, manage the signals (addition, modification, deletion) directly using
manufacturer tools. They were occasionally able to modify the configuration of the remote-control
gateway (adding information to be transmitted to the telecontrol center) using the Sicam PAS software.
grid operator also independently carried out remote control modifications. These modifications were
directly performed by the operators, sometimes with manufacturer providing phone assistance. They
were executed directly at the working station but also on a specialized PC purchased by grid operator,
which they connected directly to the protections or to the network switch in the MV building.

Among the improvement possibilities, two main areas have emerged. The first involves integrating
third-party system monitoring, like the dissolve gas transformer monitoring system (which has been
added after commissioning) into the PACS. Currently, this system isn't linked to the PACS, and
operators locally check sensor levels. Enabling the addition of this data would be beneficial to centralize
data acquisition and facilitate transmission to telecontrol center.

Improvement paths related to the PACS's usability have also been identified, such as adding a function
to retrieve disturbance recording files from the protections to the operator station. This request from grid
operator is currently under review, and a tool deployment date is scheduled for 2024.

In summary, the operator's feedback is highly positive. Technical challenges are minimal and
promptly addressed by manufacturer. The main area for improvement relates to the organization of
facility transfers, requiring more comprehensive training to ensure the initial maintenance and
operation of the station are secure.

9
BIBLIOGRAPHY

[1] IEC 61869-9, “Instrument transformers - Part 9: Digital interface for instrument transformers “, April
2016

[2] IEC 61850-9-2, “Communication networks and systems for power utility automation –Part 9-2:
Specific communication service mapping (SCSM) – Sampled values over ISO/IEC 8802-3”, Edition
2.1, 2021

[3] IEEE 1588-2019 - IEEE Standard for a Precision Clock Synchronization Protocol for Networked
Measurement and Control Systems, 2021

[4] Olszewski, W., Charot, G., Presser, N., Kuschel, M., Qastalane, Z. “Customer process for technical
qualification of Non-Conventional Instrument Transformer products for high-voltage GIS applications”;
CIGRÉ Session 2016

[5] Roux, L., Charot, G., Gatzen, F., Olszewski, W., “New approach for the on-site calibration of a LPIT
in GIS and lessons learned”; CIGRÉ Session 2022

[6] Digital substation for EDF: Engineering approach and E-Monitoring development, CIGRE 2022,
B5-903 paper, 2022.

[7] User guide for the application of monitoring and diagnostic techniques for switching equipment for
rated voltages of 72,5kV and above, CIGRE 167, Working Group 13.09, 2000.

[8] Obtaining value from on-line substation condition monitoring, CIGRE 462, Working Group B3.12,
2011.

10