Inter-System Communications/Networking

Electric utilities require reliable data communications for power protocols and has been implemented by virtually every
system control, operations, and maintenance. Utilities are among computer manufacturer.
the largest users of data and, it has been said, the largest users of
real-time data. In recent years, there has been an effort to apply the In past years,this approach has been satisfactory because
emerging technology of data communication networking to utility data amountswere relativelylow and the structure point- of
applications for real-time data exchange. to-point links reflected the “structure” of the needs of the
Emphasis in this paper is placed on techniques for modeling utilities involved (i.e., a utility wanted data from a single
communication requirements and planning the development of
an inter-system communications network to support power appli-
neighbor or was sending and receiving information from
cations. We will also review where the industry is today with regard a p o o l over a singlelink). The need forgreater coordination
to link installations and protocol development. between utilities forsuch functions as jointly owned unit
control orreal-timedata from external systemsfor stateesti-
THE STATE OF THE INDUSTRY mation has generated a need for ever-increasing amounts
of data from multiple sources.
There is a growing requirement in the power industryto
The traditional approach of installing a physical link for
exchange data with external systems (neighborhing utili-
each new data exchangerequirement is not feasible when
ties, pools, and coordinating groups) for system security,
the number of lines exceeds four orfive. Thelengthy sched-
power brokering, and real-time operation purposes. Appli-
ule, duplicate implementationsteps, and overall life-cycle
cations such as operation of jointly owned power plants,
communication costs fora“network”ofsinglelinks
pool coordinated AGC, power brokering, and state esti-
become prohibitive insuch a scheme.
mation, by their very nature, require the implementation
Let us assumethat five utilities desiredto exchange data
of communication links between cooperating control cen-
and severalof these hadexisting Energy ManagementSys-
ters. tems (EMSs) (refer to Fig. 1). If all utilities connected directly
The recent survey of System Control Centers (by T. E.
there would be tendata links. The number of connections
DyLiacco andD. L. Rosa) [I] indicates that approximately 60 may be calculated by the formula
percent ofthe existing electric power control systems have
external data links. Most of these are point-to-point com- N X (N - 1)
munication channels using the BISYNC protocol orits vari-
C=
2
ation. Several pools and coordinating centers exist based
where n is the number of host computers in the network.
on simple network structures. A summaryof these systems
Each link could conceivably be controlled by a different
is given in Table 1. communication protocol, and there would be tremendous
pressure to use a protocol friendlyto the existingsystems
THE PROBLEM
becauseof costor limitedcapabilityand/orresources.These
In the past, there has been considerable duplicationof existing protocols might be inefficient or might not
be suit-
effort in the implementation dataoflinks, and each link was able for the data exchangein question. The complexity of
unique with regard to link access procedure, method of the software at each utilitywould incrw’se correspondingly
data polling, application coordination,and message struc- with the number of different links. and interfaces needed
ture. at each host system.
The mostcommon method of exchanging data between In our hypothetical “growth” network,each link is a sin-
two utilities (either neighbors members
or of apool), is via gle point of failure (adding redundancy would double the
a direct leased line using a link access procedure such as number of physical links). The life-cycle communications
ASCII or the IBM Binary Synchronous (BISYNC) protocol. cost for maintainingthis networkwould bequite high since
BISYNC is one of the most popular data communication each link is a unique entity with its own hardwareand soft-
ware.

Manuscript received January 13,1987; revised October 6,1987. THE NETWORK

SOLUTION
The author is with Energy and Control Consultants, Sari lose, CA
95126-2388, USA. The solution is to install a data communication network
I E E E Log Number 8718219. using logical channels in place of multiple direct physical

0 1987 IEEE
0018-9219/87/12W-167~1.lM

1670 PROCEEDINGS OF THE IEEE, VOL. 75, NO. 12, DECEMBER 1987
 Tabk 1 MajorNetworks
Control
Center
Nodes System ~

Penn.-N. Jersey-Maryland 9 nodes IBM 370's BISYNC upgrade to SDLUHDLC

I
Interconnection, System planned
Norristown, PA 7's
New York Power Pool, 8 members liitachi BISYNC upgrading to DECNET
Albany, NY AS-9000
New England 4 nodes IBM 370's BISYNC
Power Exchange System
7's
Northeast Interpool Ontario Hydro Modcomp MAXNET links the U.S. systems
Network NYPP,NEPEX, Classic with Ontario Hydroin
PIM 7810 Canada
Central and South West 4 members Harris BISYNUISO X.25 future capability
Services, Dallas, TX H800's Reference Model
Mid-Continent Area 43 nodes Honeywell ASCII upgrade toDEC computer
Power Pool, 716's DECNETby 1987
Minneapolis, MN
National Control 5 nodes SDLC
Center (Brazil)
ASELECTRICA 9 members CDUCYBER X.25 WSCC variant
(Spain)
SSPB (Sweden) 8 nodes CDUXerox TIDAS

node is connected in a circle), star (everynode connected

through a central hub), and mesh (multiple connectionsto
each node). Hierarchical structures are often seen in util-
ities with regional SCADAs, acentral control center, and a
pool center. Star networks are alsocommon whereseveral
utilities are connected t o a pool center. In the star, data are
moved betweenthe member utilities as a"database access"
or a message switch.
A meshnetwork for our five-company exampleis shown
in Fig. 2(a). Note thatthe numberof physical links has been
reduced to seven, yet the level of redundancy is very high
since several links would have to be down before com-
m S :
munications with a utility host would be lost. In addition,
OPmSICALUNKFCfiEACHEE
the given mesh structure provides for relatively fast mes-
0 EACHLINKISSINQLEPMNTOF FNLURE
sage response as there is only one node hop between the
o McKn~pRQTococsLNvoLMD furthest elements of a logical channel.
0 COMPLEX SOFTWARE FOR MULTIPLE LINKHANDLING AT EACH NODE It is also possible t o integrate a private network with a
OPOOAPERFtX"€WTWlHWjHCOST public network.Thisis illustrated in Fig.2(b). Such"hybrid"
networks are cost-effective where the ratio of high-priority
Fig. 1. Data communications system dedicated links. Typ timecritical data is relatively low in comparison to total data
ical network growth.
requirements.
In our example, frontend processors (also called com-
channels. (A logical channel is defined by thetwo end-to- munication networkprocessors) would beused as the net-
end communicating entities and is independent of the workimplementationvehiclesincethe resourcesatthe util-
communication medium.The network protocoltakes care ity E M S host computers wereinadequate for the network
of the data routing.) There are severalways to procure the communications. Thecommunication networkprocessors
network. A value-added common carrier might be used (CNPs) at eachutility company would handle the network
(e.g., a public data network such as Telenet); a proprietary protocol, message routing, communication media inter-
network couldbe purchased (e.& IBM's SNA, CDC's CNA, face, connection to avalue-added common carrier (if
DEC's DNA, Data General's ZODIAC, Modcomp's MAX- required), and dire& channels to neighboring utilities ora
NET), or a custom networkdeveloped. (Procurement alter- pool for high-volume/timecritical data. Although thissolu-
natives are discussed later in this paper.) tion requires a fair amount of hardware, it i s particularly
There are many different network "structures" possible attractive since there is only onesoftware packageper CNP
if private facilities are used, but they all are variations on (duplicated at each utility node), asingle external protocol
four forms: hierarchical (ortree), ring (where every host or is used (the CNPs would convert the local host protocolto

ROBINSON:INTER-SYSTEM COMMUNICATIONS/NETWORKING 1671

 problems with these protocols is that theyare designed to
operate in a point-to-point configuration and have no pro-
vision for functions communicating across logic channels.
Flow control (coordination of links and/or devices oper-
atingasynchronously)and e n d - t w n d message control
(including error handling and recovery) aremajor protocol
issues that must be addressed i n a networking environ-
ment.
The newer network protocols are usually based on the
International Standard Organization Reference Model
Standard of Open System Interconnection (RWOSI). The
RMIOSI breaks the commurrication and networking func-
tions down into small manageable independent parts or
“layers.” (It also defines communication terminology and
is the basis for on-going protocol development by ANSI,
CCITT, and ECW.) The layers aredefined in terms of ser-

1J GHPIL

0 EFPhL
vices and functions on behalf of adjacent layers and the
internal structures of each layer are totally independent
from other layers(“transparency” principle). An imple-
mentation based on the RWOSI has the addedadvantage
that it may be tested, implemented, and modified in parts
with minimal (if any) impact to other parts of the model
ABPlL
(“robust” principle). Thus it is possible to modify a well-
PUBLIC
designed network and take advantage of new communi-
DATA NETWORK ;‘ cationtechnologyandnewprotocols without making
extensive software or system changesthroughout the net-
_.-- work.
The seven-level RWOSI is illustrated in Fig. 3. As can be
seen from thefigure, “information” flows from one entity
at the top or Application Layer, down through adjacent lay-
ers, acrossthe connection media, up through theadjacent
layersat the remote location until the “information” is
received bythe cooperating entity at the other end.
As an example of the use of this, let us follow the flow
of a message between an entity (say, a State Estimator Pro-
gram) in an EnergyManagementSystem to acorresponding
entity in a neighboring utility. The first application program
at Layer 7 would pass the message datato its Presentation
Layer. Here themessage would be formatted (accordingto
“presentation” rules) and aLayer 6 header and trailer con-
taining control information used by the Layer 6 services
(b) would be added and themessage passeddown to the Ses-
Fig. 2. (a) Distributed network architecture. (b) Hybrid
net- sion Layer 5. (Each layer addsits control informationheader
work architecture. and trailer, takes appropriate action, and passes the mes-
sage down to the next layer, and so on, until the message
is transmitted by the Physical Layer across the interface
the network protocol), and the CNP allows parallel pro- media.) The “session” establishes and controls the data
cessing of communications andEMSlSCADA applications. exchange, checks sequencing of messages, handles
Thecommunications system life-cyclecost savings in our addressing, prioritizes messages, checks access rights, and
networkexamplewould beconsiderableoverthefullycon-
guaranteesdata integrity. The Transport Layer provides
n e e d approach.Tivo major componentsmake up the sav- media-independentcommunication services including
ings: I) the lower recurring communication media costs
“segmentation” (breaks messages down into parts) and
(between 20 to 60 percent dependingon the redundancy), “flow control” (coordinates exchange rates)for theagreed
and 2) protocol and software savings of the CNP configu- level of “service quality”(reliabi1ity). At the Network layer,
ration. interfaceto the communication “media”is established and
Implementationanddevelopment strategiesaredis- “packets” are routed through the physical channels of the
cussed in more detail later in this paper. We turn now to network. The link access procedure handlesthe exchange
a discussionof protocol issues. of information over each point-point physical circuit (the
familiar BISYNC protocol operates here).When the “mes-
NETWORK
PROTOCOLS sage‘’ is received by the other side, each layer acts accord-
Communicatioh protocolssuch as BISYNCand ASCII are ing toits control information, strips its header and trailer,
inefficient, incomlpete, andnot usable for communication and passes the information up to the next layer until it
networks without extensive modification. One of the main reaches the proper entity in theApplication Layer on the

1672 PROCEEDINGS OF THE IEEE, VOL. 75, NO. 12, DECEMBER 1987
Fig. 3. International Standards Organization’s (ISO) Reference Model of Open-Systems
Interconnection (09).

receiving side. Note that there is a”protoco1” or formalized saging and data format recommendations. Also, ASELEC-
set of proceduresfor coordination between each of thecor- TlCA in Spain hasa variationof theWSCC standard in use.
responding layers of the model.In our example, the State (CDC developed the first versions of the WSCC links,
Estimator might use polllacknowledge sequencestoassure including theASELECTRICA system.) (At this writing, only
the secure andcoordinated transfer of information between asubset of theWSCC recommendedfeatures haveactually
the two programs. been implemented andtested. All of the operational links
C C l l l X.25 is one of themost commonly used protocols are point-to-point with predefined message formats.)
for networking [2]. It has three levels which correspond The utilities in Europehave hadcommunicationnet-
roughly to the lower three layers of theISOIRM. X.25 is the works in use for some time. The TIDAS information system
protocol recommended by the WSCC [3]. It is an extremely has been operational since 1975 i n Sweden [6].This message
popular protocolused extensivelyin many countries. It was switching network is particularly interesting since it inte-
intended primarily for use in interfacing to publicdata net- grates several communication media methods including
works but has been adopted by several computer suppliers ECMA-24 digital data links, radio, and carrier frequency
as their protocol of choice forall communications. links. More recently, the Union for Coordination of Cen-
Several notable attempts at standardization for power eration and Transportof Electrical Energyin Western Europe
system use have occurred in recent years. In the U.S., the (UCPTE) has developed a data transfer protocol based on
Western System Coordinating Council published a com- the ISO/RM m.
This protocol uses X.25 at the lower three
munication standard in June of1984 [3] that allows for net- layers, EDF at the Transport Layer (the French recommen-
working via private or publicfacilities using the CCllTX.25 dation), and speciallydeveloped procedures for the upper
protocol. This standard also includes higher level protocols three layers.
based on theI S 0 OS1 Reference Model. (See[4] forafurther A recent major effort was completed by15 utilities in the
description of the WSCC effort.) Eastern US. to define “data communication capability”
At this time, several WSCClinks are operational between along with real-time data network feasibility and approach
centers atLADWPand SCE, and between Tuscon Powerand for their interconnected power systems. (See [9],Inter-Util-
Public Service of New Mexico.PC&E and the City ofSanta ity Data Exchange Committee Project.) The study effort
Clara have implemented WSCC links usingBISYNC at the included a thorough review of earlier utility projects and
Link Level with thehigher levels basedon theWSCC mes- protocols. The decision was madeto use existing I S 0 stan-

ROBINSON: INTER-SYSTEM COMMUNICATIONS/NETWORKlNC 1673

dards at the lower layers as follows: rite Interface Control Documents (ICDs) for the
definition and control of all
interfacesto thenet-
Session Protocol-IS0 8327
work.
Transport Protocol-IS0 8073
Evaluate protocols, select contractor(s)(for
Network Protocol-IS0 88781X.25 Packet Layer
equipmentandsoftwareprocurements),and
DataLink-X.25(HDLC).
write a Work Statement(s).
At the higher layers (Application, Presentation, Manage Modifyhpgrade the utility host SCADAlEMS sys-
ment), protocols and requirements were specifically devel- tems to interface to the network
and to utilize the
oped to meet IDEC needs (see [IO]) including three levels data on behalf of the given functions.
of priority and access control via Bilateral Agreements.
Data Requirements
CONSIDERATIONS
IMPLEMENTATION Perhaps the single most important step is to identify all
data needsfor the participating utilities.
Data requirements
The implementation of a communications network to
are defined in terms of the followingcharacteristics:
support EMS functions is a difficult and time-consuming
effort. Several factors combine to make the job more dif- data types (analog, status,kwh, schedules, AGC data,
ficult: etc.),
The overall complexity of the EMS system functions supplement data (quality coding,data source,manual
using the network. entry, alarm status, exception flags, etc.),
Coordination requirement between different utilities dataformat(floating-point,integer, binary,ASCII,
having differentneeds (evenfor a single link, thereare two EBCDIC, etc.),
host systems that must be "coordinated"). amount of data,
Varied datacommunication needs (real time,high vol- polling1exchange method (datarequest, periodic,
ume, high security, "on-line" versus "off-line" versus event-driven),
"interactive" processing, etc.). periodicity of exchange,
Fast changing technology i n such areasas computer allowable data delays,
hardwarelsoftware, communication,and networking tech- synchronization (method for synchronizing data or
niques, distributed processing/databases, the proliferation events).
of value-added common carriers, etc.
These data requirements directly determine the size and
Upgradeketrofitproblem-theintegrationofcom-
type of communications needed.
munications into the existing EMS/SCADA systems with
older technologies. Possible ProcurementOptions
Changing needs-the flexibility requirement to allow
growth of the communication network to support new Thereareessentially threeprocurement alternatives:
functionsandtheexpansion/modificationofexisting public data networks (PDNs), proprietary networks, and
SCADNEMS functions. customnetworks. The CustomNetwork, as the name
We turn now to a discussion of implementationstrategy implies, is builtespeciallytomeetthecommunication needs
methods, and look at ways to minimize the technical and of the utilities involved.This would be very expensiveand
coordination problems of developing a real-timedata com- is probably not feasible today considering the availability
munication network. of standard communication products and services.
In order to implement a network, following
the task steps The proprietary networking products (such as SNA, DEC-
must be completed: NET,MAXNET, ZODIAC, etc.) all share similar character-
istics:
Identify those functions which have data exchange
implications. they are layered architectures
Finalize data exchange requirements (type, rates,tim- theyprovideforprogram-to-programcommunica-
ing constraints, priority, reliability needs, accuracy, tions.
data integrity, etc.).
Some of these have enhanced features such as:
Study and evaluate the feasibility ofalternative
approaches to meeting the data exchange require protocolconversion,
ments. gateways (linking the network to a PDN, or another
Select the best overall approach based on objective network such as SNA),
criteria (such as expected functional benefit, external allow the sharingof computer system resources(such
constraints,cost,staffing,resourcedemands, and as disks, files, printers, CPUs, etc.).
schedule).
The proprietary products differ in the protocols available,
Develop an implementation plan and schedule.
the number and types of linksavailable, sofhvarehetwork
Carry out the implementation plan: support, and monitoring devices. However, they are quite
a) Write procurement specifications where neces- inexpensive considering the functions available and the
sary (even if all the work willbe done by the util- software is "thrown in" for thecost of the hardware. (The
ities,some type ofinternalspecification is average network product software runs about 1 percent
needed). above the list price for the hardwarein the network con-
b) Develop (orselect) communication, networking, figuration.)
and application protocols. Public data networks (suchas Telenet or Tymnet i n the

1674 PROCEEDINGS OF T H E IEEE, VOL. 75, NO. 12,DECEMBER1997

U.S.) are value-addedcommon carriers thatare tariffed to tiple resources (channel availability, CPU process time,
provide “added value” in the form of: memory, routing mechanisms, message queuing, etc.) at
multiple nodes all along the virtual path.
protocol conversion, It is very important to size the network properly during
data format conversion, the early phasesof thesystem design.As discussed earlier,
virtual terminal capability, this starts with aproper understanding of the data require-
message store-and-forward, ments and expected growthof thesystem. Budgetswould
speed conversion (interfacesmay operate at different beestablishedforallresourcesonafunaion(EMSdatauser)
rates), and logical channel basis.
gateways (between PDNs, i.e., a U.S. network could Budgets may beestablishedfor thenetworkusing
interface to utilities in Canada through a Telenet to queuing theory equations.It has been demonstrated that
Datapac connection), networks may be modeledconservativelyusing MIMI1
integration of private facilities (it is possible to use a equations [8].The equations, however,have to be modified
utility company microwave system and also connect to account for the error characteristics of the media, pro-
to other utilities over the public facilities), tocol overhead, and end-end response characteristics. (A
network management services (cost and billing ser- set of recommended equations is given in Fig. 4.)
vices, reverse charging, and network resource moni-
toring).
1) Maximum Line Throughput
Interfacing toa Network M (1 - P)
MJ = ~

Most computer manufacturerscan handle the X.25 net- M + D

L
work protocol and havecertified versions availablefor the
various PDNs. Also,the PDNs havesoftware interfacepack- M: Averagemessage length (bits).
ages available for lease. Another alternative is to have the
P: Probability messageis repeated (has errors).
L: Line rate (bit&).
PDN do protocol conversion. This i s done eitherwithin the D Delaybetween messages (duetopolling,modemturn-
PDN (at the local control office in. the network DCE) or in around, message synchronization, etc.).
a PDN-provided local processor. The hardware and soft- 2) Line Utilization
ware for the localprocessor is maintained by the PDN and
software changes are down-line-loaded to the local pro- u=- actual - -A X M X P
cessor. possible MJ
Thethirdmethod,whichwasdiscussedearlierinournet- A: Average # Messages/second (based on required response)
work example, is to use a FrontEnd Processor (FEP), or, as P: Protocol Overhead Factor.
it is called whenused in a network, a Communication Net- 3) Queue Size (per line) (Gives an estimate of storage require-
work Processor (CNP). The CNPwill handle all communi- ments.)
cation message queuing,protocolconversion,network U
Q=-
interfaces, error control, message retry, and network per- 1-u’
formance monitoring.The CNPcould even serveas a node
4) MessageResponse (perline)
in the network and route messages to adjacent nodeswith-
out host intervention. One of the major benefits CNP
of the M/M J
LR = -
is that it provides parallel processing of network functions 1 -u’
and related data link handling with utility EMS/SCADA 5) End-EndResponse(across the network)
functions. Therefore, separategrowth paths for these func- R = PJ+ CLR
tions exist and the network can be modified, or the EMS1 PT: Processing time at the EMS systemlCNPs.
SCADAfunctions modified,with littleimpact to other parts C L R Sum of the line responses over the logical channel.
of the system.
Fig. 4. Network performance equations.
Interface between the CNP and utility host is by some
simple, existing, “host-friendly” protocol such as ASCII,
BISYNC, a parallelinterface, or a Local AreaNetwork (e.g.,
Ethernet). Several of the network suppliers offerEthernet For the study of large networks and/or complex EMS data
or IEEE 802 LAN interfaces which operate as “intermediate exchanges, sometype of computer-assisted analysis or sim-
nodes“ at Levels 2 and 3 of the RM/OSI. These interfaces ulation is preferred. Overthe past several years,the author
are particularly attractivebecause of their high throughput has used acommunicationnetwork analysis program
and ease of expansion and modification. (CNAP-an ECC proprietary service) that provides a con-
venient way to plan, size, and budget complex networks.
Using CNAP, a systemdesignercan optimizenetwork
NETWORK REAL-TIME RESOURCEAND TECHNICALDESIGN
topology, automatically configure for backup of failed lines
CONTROL
and nodes, determine logical channel message response,
Real-Time Resourceand Technical Design Control (RTDC) examine the network impact of using different protocols
is the process of budgeting, monitoring, unit testing, and or polling techniques or message/data formats, tune the
controlling resource utilization, throughput, performance, network with regard to data priority, and determine the
and response on a real-time system. RTDC i s particularly traffic growth capability of a network.
important in a communication network where the ability As the system is built, the budgets would be verified by
to exchange datai n a timely manner is dependent on mul- performance testing and the models, the corresponding

ROBINSON: INTER-SYSTEM COMMUNICATIONS/NETWORKING 1675

utilization figuresas well as messageresponse times would Extension ofthe NetworkStandard (For Implementation and
be updated, and designs modified as necessary. After sys- Coordination)
Defines the Complete, Network-Implemented Protocol for
tem startup, Network Resource Monitoring (NRM) software All Levels
running on the networkwould provide informationfortun- Resolves All Options and Guidelines
ing the network as it grows and changes. Identifies Message Structures, Data Types, and Data For-
High throughput and lowdata response aretotally sep- mats Supported
Provides Semantics (That is, The Meaning of all Messages
arate mechanismsin a network. We can see this by exam- and Data)
ining the equations for message response (Fig. 4). As uti- Provides Complete Error Handling and Recovery Proce-
lization goesup,responseincreases dramatically.An dures (Handling of Old Data, Missed Data, Link Reset, Net-
increaseof 10 percent (or an estimatingerror!) in utilization work Restart, Application Program Actions)
(say going from 80 to 90 percent) would result in a100-per- Resolves Protocol (e.g., X.25) options:
Level 1 (X.25 or X.21 bis?)
cent increase in response. When applying RTDC proce- LevelZ(IAPorL4PB?SeIectiveReject,Tltimes?T3times?
dures, these separate mechanisms would be monitored NZRetry Count? Frame Length?Flow Control Window?
independently. The only way to assure that amessage can BSC Framing? Etc.)
be sent and received with high probability is to dedicate Level 3 (Reverse Charging? Packeflime Accounting?
Throughput Class?Packet Size? Closed User Group?
channelsand associated resourcesfor themessage. (Public etc.)
data networks, since they are subject to traffic congestion Resolves Interface (Between Host and Front-End) Features
problems, may not be suitable for networks where high Clarifies Exceptions or Alternatives
response is a key requirement.) Sizes theNetwork,Number ofLinks, Amount of Data
Exchanged, GivesTimingand Performance Parameters,and
Identifies Growth Requirements.
NETWORK CONTROL DOCUMENTS
INTERFACE Discusses NetworkCoordination and Operation Proce-
dures, Assigns Responsibilities, and GivesTesting, Startup,
The high need for coordination when developing a com- and Maintenance Requirements
munication system dictates special coordination mecha- Provides an Implementation Plan Including a Schedule
nisms. Serves as the Control Documentfor Ongoing Maintenance
Interface Control Documents (ICDs) are used to define of the Data Link
and control all major interfaces and coordination efforts. Identifies Data Link Coordinators (Utility Contact Individ-
uals)
They are particularly important in the definition of com-
munication data links and networks where interfaces at sev- Fig. 5. Checklist. Network(datdlink)InterfaceControl
eral levels areneeded and coordination between multiple Document.
departments, utilities, and computer systems is a primary
concern during the implementation process. (e.g.,
HDLC, DDCMP) with featuressuch as implied
It is important to recognize that existing network pro- acknowledges, flow control, pipelining of messages, and
tocols are complex and cover a variety of communication combined datakontrol messages. Gains of 40 percent in
situations.There are manyoptions and parametersthat are link utilization are common with the new protocols. The
not defined.ANSI X3.28 (which definesBISYNC and ASCII trend is also awayfrom point-to-point links and toward net-
protocols) has 14 Establishment andTermination Sub- works. Networking allows for efficient use of communi-
categories and 10 Message Subcategories. And all these cation media and sharing of computer resources andcom-
combinations have optional controlcharacters and various mon protocols.
error recovery procedures orcontrol procedures. X.25 and The use of packet switching technology and publicdata
other layered protocols have alternatives and options for networks is expected to increase for utility communica-
each protocol at each layer plus interfaces between adja- tions. The technology presently exists to integrate these
cent layers.One of thereasons for the ICDis to define the networks forE M S applications and use dedicated links for
implemented network communication protocols and to high-priority, high-reliability dataand to use the public net-
resolve all the options and define network parameters prior workfor lower prioritydata, terminal communications, and
to start of design by the several participating utilities. gateways.
A checklist for a network ICD is given in Fig. 5. The primary problem i s one of coordination and plan-
ning. The marketing life of computer equipment, com-
CONCLUSION munications gear, and front-end hardware averages about
There is a definite trend toward increased
the movement 3 years. Theoperational life of EMSlSCADA systemsseldom
of data between neighboring utilities forseveral reasons. exceeds 10 years. Yet it may take 3 to 5 years to builda large
One is the need for data to support functionssuch as power communication facilityto support atilityfunctions. The E M S
brokering, pooling, operation of jointly owned units, and systems in the networkwill invariably be at different points
to improve the modelingof the interconnected powerSYS- in their life-cycle.
tems for security assessment purposes. Then there is the In this paper, we have presented ways to minimize the
impact of technology growth: the maturing of communi- coordinated development problemsassociated with inter-
cation technologyand the lowercost of data link facilities1 system communications. The key is thorough and early
satellite communications, theemergence of protocolsand planning ofneeds and expansion requirements. Particular
techniques for data exchange, and the availability of the attention must be given to communication resource bud-
computer resources and inexpensive front-end processors geting and monitoring. Also, allowing independent and
to drive communication networks. separate growth paths for the communicationsystems and
The trend is away from the low-efficiency, half-duplex, the EMS/SCADA systems helps to guarantee fewer prob-
stopand-wait protocolsto the newer full-duplex protocols lems in the future.

1676 PROCEEDINGS OF THE IEEE, VOL. 75, NO. 12, DECEMBER 1987
REFERENCES [lo] Inter-Utility Data Communication Capability Protocol/Guide
line, version A.0, Apr. 1987.
[l] T. E. DyLiacco and D. L. Rosa, Survey of System Control Cen-
ters, Edison Electric Institute, Jan. 1, 1986.
[2] Packet Mode Interface to PublicData Networks, CCllT X.25.
Geneva, Switzerland, CCIlT, June, 1980, no. 7-E, app. VII. JohnT. Robinson (Member, IEEE) received the B.S. degree in math-
[3]
Western Systems CoordinatingCouncil EMS Inter-Utility ematics from the University ofWisconsin, Madison.
Communication Guidelines, June 1,1984. He is a Principal Consultant and is the General Manager for
[4] T. E. DyLiacco, W. A. Johnson, S. H. Buchey, M. D. Crouch, Energy and Control Consultants, Inc., West Coast Office, in San
Paul Emmerich, S. A. Klein, M. D. Anderson, “Inter-utility Jose, CA.He has twenty years of experience in real-time computer
computer data exchange,” lEEE Trans. Power App. Syst., vol. systems and controlapplications, the last fourteen of whichhave
PAS-100, no. 12, Dec. 1981. been primarily in electric utility SCADNEMS environments. His
[5] “Proposed data structure for exchange of powersystem ana- experience includesdirecttechnicalparticipation in software
lytical data” (Draft Working Group Rep.), /E€€ Trans. Power development as well as overall design and specification systems.
of
Syst., vol. PWRS-1, May 1986. His implementation background in real-time and computer sys-
[6] TlDAS Network, Pamphlet KG94-102E, Swedish State Power tems includes operating systems design, SCADNEMS applica-
Board, 1979. tions, control systems, and communications. Hehas participated
[A H. P. Asal, “Information exchange between control centres in over 40 electric utilitySCADNEMS and communicationsystems
in the West European high-voltage grid,” Elec. Power Energy as either the projectmanager or technical consultant. In addition,
Syst, vol. 5, no. 4, Oct. 1983. he has been involved in several major communication networking
[a] A. S. Tanenbaum, Computer Networks. Englewood Cliffs, projects andis one of the authors of the WSCC Inter-Utility Com-
NJ: Prentice-Hall, 1981. munication Standard, the lnterutility Data Exchange Communi-
[9] S. J. Wavrek and J. T. Robinson, “Inter-Utility Data Com- cation Protocol, and has participated in ANSI and IEEE commu-
munication Committee Requirements Study Project,” pre- nication standard work group activities. He has also presented
sented at Edison Electric Institute, June 1987. papers at various conferences.

ROBINSON: INTER-SYSTEM COMMUNlCATlONSlNETWORKlNC 1677