Ethernet Communications
and Requirements for IEC
61850 Based Systems
Dr. Alexander Apostolov
Los Angeles, CA
© OMICRON
Scope
• Discuss Ethernet architectures supporting
IEC 61850 projects
• Not defined in the standard but essential for a
real project !
• Background on Ethernet principles and
technologies
• Practical examples
© OMICRON Page: 2
Transmission Definitions
SIMPLEX
Transmission in one direction only
HALF DUPLEX
Two way means of transmission but data
can only travel in one direction at a time
FULL DUPLEX
Transmission in both directions
simultaneously
© OMICRON Page: 3
Transmission Definitions
PROTOCOL
Rules and procedures that
Communications networks use to
communicate on the communications
medium
CONNECTION
Communications are Connection
Oriented or Connectionless
© OMICRON Page: 4
Transmission Definitions
Point-to-point
Engineering Laptop
Station
Modem IED
Modem
IED
© OMICRON Page: 5
Transmission Definitions
Shared Access
Substation
Ethernet HMI IED IED IED
Ethernet
Switch
Proxy Substation
Server HMI
RS485
Ethernet
Legacy IEC 61850 IEC 61850
IED IED IED
© OMICRON Page: 6
Transmission Definitions
NETWORK ACCESS
CARRIER SENSE METHODS
CSMA (Carrier Sense Multiple Access)/CD
(Collision Detection)
Medium Access Control (MAC)
TOKEN ACCESS METHOD
The device that has the Token has access to
transmit
RESERVATION METHOD
Each device has a predefined time slot to transmit
© OMICRON Page: 7
Network Terminology
Network An interconnected group of nodes or
stations linked by communication channels
Node The interface point where one or more
functional units are connected
LAN Local area network (<5km)
WAN Wide area network
Network topology Pattern of nodes and their interconnection
© OMICRON Page: 8
Transmission Definitions
Medium - Ethernet devices attach to a common medium that
provides a path along which the electronic signals will travel:
- historically, this medium has been coaxial copper cable
- more commonly a twisted pair
- fiber optic cabling.
Segment - a single shared medium as an Ethernet segment.
Nodes - devices that attach to that segment are stations or
nodes.
Frame - The nodes communicate in short messages called
frames, which are variably sized chunks of information.
© OMICRON Page: 9
Open Systems Interconnection
(OSI) Model
Application Selects appropriate service for application
Presentation Provides code conversion, data reformatting
Coordinates interaction between end
Session
application process
Provides for end to end data integrity and
Transport
quality of service
Network Switches and routes information
Transfers unit of information to other end
Data Link
of physical link
Physical Transmits bit stream to medium
© OMICRON Page: 10
OSI Stack
PCI =Protocol Control Inform. Commands or data
PDU =Protocol Data Unit Application
H =Header Presentation
PCI PDU
T =Trailer
PCI PDU Session
PCI PDU Transport
PCI PDU Network
PCI PDU Data link
PDU PDU PDU Physical
H fragment
T H fragment
T H fragment
T
Frames (Ethernet, token ring, etc)
© OMICRON Page: 11
Communications Process
Upper Layer Upper Layer
SAP SAP
Entity with Peer-to-peer dialog Entity with
services to offer services to offer
SAP SAP
Lower Layer Lower Layer
© OMICRON Page: 12
What is Ethernet?
• Ethernet is the most widely used local area
network (LAN) technology.
• The original and most popular version of
Ethernet supports a data transmission rate
of 10 Mb/s.
• Newer versions of Ethernet called "Fast
Ethernet" and "Gigabit Ethernet" support
data rates of 100 Mb/s and 1 Gb/s (1000
Mb/s).
• An Ethernet LAN may use coaxial cable,
special grades of twisted pair wiring, or fiber
optic cable.
© OMICRON Page: 13
What is Ethernet?
• "Bus" and "Star" wiring configurations are
supported.
• Ethernet devices compete for access to the
network using a protocol called Carrier
Sense Multiple Access with Collision
Detection (CSMA/CD).
© OMICRON Page: 14
Ethernet and the ISO/OSI Model
• The standard model for networking
protocols and distributed applications is the
International Standard Organization's Open
System Interconnect (ISO/OSI) model. It
defines seven network layers.
• Layer 1 - Physical
• Physical layer defines the cable or physical
medium itself, e.g., thinnet, thicknet,
unshielded twisted pairs (UTP).
• All media are functionally equivalent. The
main difference is in convenience and cost
of installation and maintenance.
© OMICRON Page: 15
Ethernet and the ISO/OSI Model
• Layer 2 - Data Link
• Data Link layer defines the format of data on
the network. A network data frame - packet,
includes checksum, source and destination
address, and data.
• The largest packet that can be sent through
a data link layer defines the Maximum
Transmission Unit (MTU).
• The data link layer handles the physical and
logical connections to the packet's
destination, using a network interface.
© OMICRON Page: 16
Ethernet and the ISO/OSI Model
• Layer 2 - Data Link
• A host connected to an Ethernet would have
an Ethernet interface to handle connections
to the outside world, and a loopback
interface to send packets to itself.
• Ethernet addresses a host using a unique,
48-bit address called its Ethernet address or
Media Access Control (MAC) address.
© OMICRON Page: 17
Ethernet Frame
Pre SFD DA SA Lgt/Type MAC Data + Pad FCS
7 1 6 6 2 0 – n bytes 4
• Pre: The Preamble is an alternating pattern (7 bytes) of
1 and 0 that tells receiving stations that a frame is
coming
• SFD: Start-of-frame delimiter (1 byte: 10101011)
indicating that the next bit is the left-most bit in the left-
most byte of the destination address.
• DA: Destination address (6 bytes) identifies which
station(s) should receive the frame
• SA: Source addresses (6 bytes) identifies the sending
station
© OMICRON Page: 18
Ethernet Frame
Pre SFD DA SA Lgt/Type MAC Data + Pad FCS
7 1 6 6 2 0 – n bytes 4
• Length/Type: Number of MAC-client data bytes that are
contained in the data field of the frame
• MAC Client Data: A sequence of n bytes (46=< n
=<1500) of any value. (The total frame minimum is 64
bytes). The Pad contains (if necessary) extra data bytes
in order to bring the frame length up to its minimum size.
A minimum Ethernet frame size is 64 bytes from the
Destination MAC Address field through the Frame Check
Sequence.
• FCS: The Frame Check Sequence is a 32-bit cyclic
redundancy check (CRC) value
© OMICRON Page: 19
Extended Ethernet Frame
Pre SFD DA SA TagT TagC Lgt/Type MAC Data + Pad FCS
7 1 6 6 2 2 2 0 – n bytes 4
• Virtual LAN (VLAN) tags are used to identify the
VLAN to which the frame belongs. It allows
frames from stations to be assigned to logical
groups
• TagT: The first 2-bytes of the VLAN tag consist
of the "802.1Q Tag Type" - always set to a value
of 0x8100.
• TagC: VLAN Tag Control
© OMICRON Page: 20
VLAN Tag Control
Pre SFD DA SA TagT TagC Lgt/Type MAC Data + Pad FCS
7 1 6 6 2 2 2 0 – n bytes 4
• The first 3-bits are a User Priority Field
that may be used to assign a priority
level to the Ethernet frame.
• The next 1-bit is a Canonical Format
Indicator (CFI) used in Ethernet frames
to indicate the presence of a Routing
Information Field (RIF).
• The last 12-bits are the VLAN Identifier
(VID) which uniquely identifies the
VLAN to which the Ethernet frame
belongs.
© OMICRON Page: 21
Ethernet
• Media Access Control (MAC) Address -
This is the physical address of any device,
such as the NIC in a computer, on the
network.
• The MAC address has two parts, each 3
bytes long.
• The first 3 bytes identify the company that
made the NIC.
• The second 3 bytes are the serial number
of the NIC itself.
© OMICRON Page: 22
Ethernet
• Unicast - A transmission from one node
addressed specifically to another node.
• Multicast - When a node sends a
packet addressed to a special group
address. Devices that are interested in
this group register to receive packets
addressed to the group.
• Broadcast - When a node sends out a
packet that is intended for transmission
to all other nodes on the network.
© OMICRON Page: 23
Ethernet Basic Principle
1. Listen and broadcast if the wire is free
2. If there is a collision then re-transmit in an interval [0, 2N-1]
Ethernet
© OMICRON Page: 24
Switches principles
Internal Switches
Queue
© OMICRON Page: 25
Switches: Performances
Pure delay (store & forward): 5-10μs
Maximum frame size: 1536 bytes
1536 bytes @ 100 Mbps = 123 μs
Goose size << 1536 bytes
© OMICRON Page: 26
Switches: Unicast
© OMICRON Page: 27
Switches: Multicast
2 2
© OMICRON Page: 28
Switches: Broadcast
2 2 2 2
© OMICRON Page: 29
To Process
Binary Input
Wall-to-wall performances
Filtering [2-10]
Logical [1-50]
example (ms)
[5-75]
Communication [2-15]
Switching [0.1]
Communication [2-15]
Logical [1-50] [4-72]
Relaying [1-7]
Process To + 9/147
Binary Output
© OMICRON Page: 30
Switches: other features used in
Substation Automation
• Substation environment !
• Electro-magnetic compatibility
• DC power supply with 20ms voltage dips, possibly
redundant
• Priority management (802.1p):
• Capability to have priority queues in order to further
boost the communication performances
• Virtual LAN – VLAN (802.1q tag):
• Capability to create logical groups of devices in order
to filter the messages not belonging to one group
© OMICRON Page: 31
Switches: other features used in
Substation Automation
• Management
• Capability to detect a switch failure: SNMP
or watch-dog
• Redundancy management
• No standard today for hard real time
redundancy
• Redundancy between switches (not
applications)
• See further slides
© OMICRON Page: 32
Typical network architectures
STAR
ISD ISD ISD
Basic scheme since failure of the
central switch leads to the total
communication failure
© OMICRON Page: 33
Typical network architectures
RING
ISD ISD ISD
The most common scheme since good
performance/cost ratio (save 2 central switches
vs. start configuration)
Failure of a single fiber or device does affect
the system once reconfiguration is completed
© OMICRON Page: 34
Typical network architectures
DOUBLE STAR
ISD ISD ISD
The most secured scheme since multiple
failures are tolerated except the two central
switches simultaneously
© OMICRON Page: 35
Redundancy management
• A way to improve system availability and reliability
• Behavior is not defined in IEC 61850
• Typical example where the architect needs to define what
shall be done… and first check on paper that it can be
supported by the IEDs
• May be applied to:
• Communication infrastructure: tolerate the loss of an optical
fiber or/and the loss of a central switch
• Clients: capability for a client to continue the tasks initiated
by a first client (example: Graphical user interface)
• Servers: capability for a client to switch to a redundant server
if the first one is not operational (example: CT/VT sensor)
• 99.99% availability requests the 3 types of redundancy
© OMICRON Page: 36
Redundancy management at
communication level
• Defined between Ethernet switches
• If a device has two Ethernet port it must also have an integrated
switch
• Spanning tree mechanisms
• Recalculation of the route between switches using an internal
protocol between switches and avoiding loops
• Base: 802.3d. Typically 30 s reconfiguration time
• Fast: 803w. Typically 100 ms reconfiguration time, some
implementations claims 5 ms per switch (N x 5ms with a ring made of
N switches)
• Other mechanisms
• Detects the failure of the adjacent switch to reconfigure
• For ring topology, less than 1 ms reconfigurations
• Sends two signals at the same time and keep the first one arrived
• No switch-over time, but need to keep a correct buffer size (dual homing)
© OMICRON Page: 37
Example (ring)
“Repeaters”
(IEE 802.3 Ethernet switch)
+ Self Healing Manager (SHM)
Ethernet Ports 10/100 Base TX
EEPROM
Port MII
SHM N°
Failsafe Self healing ring
output relays manager
100Mbs Full duplex
opto opto
Primary Fiber
Secondary Fiber
Optical Ring
© OMICRON Page: 38
Example (ring)
X During nominal situation, Ethernet packet goes in
primary fibre always in the same direction, and
only a checking frame (4 bytes) is sent every 5ms
in secondary fibre in the opposite direction.
Primary fibre
1 2 3 4 5 6 7 8 9 10 11
switch switch switch
switch switch
RP EP
C D E
A B
ES RS
Secondary fibre
© OMICRON Page: 39
Example (ring)
If the connection between 2 switches is broken
X The Ethernet network will continue to run correctly.
X Both SHM start immediately the network self-healing.
At one side, the messages received are no more emitted to
the primary fibre but to the secondary fibre.
On the other side of the cut off, the messages received to secondary
are emitted to primary and the new topological loop is closed.
Primary fibre
1 2 3 4 5 6 7 8 9 10 11
switch switch switch
switch switch
RP EP
C D E
A B
ES RS
Secondary fibre
© OMICRON Page: 40
Other communication elements
IEC Com. Com.
OSI Model Technology Protocol Services Element
Application MMS/SNTP Report, Control, Gateway/
File Proxy
Presentation
Session
Software
Transport TCP
Network IP Router
Bridge/Switch
Data Link 802.3 GOOSE
/Proxy
Hardware
Physical 802.3
Need to design the communication infrastructure
according to the system constraints and requirements
© OMICRON Page: 41
IEC 62351
• “Data and Communication Security” series of
standard
• IEC 62351-6: Security for IEC 61850 profiles
• Relies on IEC 62351-4: Profiles including MMS
• Relies on IEC 62351-3: Profiles including TCP
IEC Com. Security
OSI Model Technology Protocol Services Services
Report, Control, Authentification,
Application MMS/SNTP
File Replay
Presentation
Session
Software
Authentification,
Transport TCP Encryption, Replay
Network IP
Authentification,
Data Link 802.3 GOOSE Replay
Hardware
Physical 802.3
© OMICRON Page: 42
