Carrier Grade Metro Ethernet Networks

Carrier Grade Metro Ethernet Networks Beitrag der ITG-Fachgruppe 5.3.3 „Photonische Netze“ Achim Autenrieth, Andreas Kirstädter, Siemens Networks GmbH&Co KG; Bernhard Edmaier, Clemens Epple, BT (Germany) GmbH&Co oHG; Gert Eilenberger, Gert Grammel, Alcatel-Lucent Deutschland AG; Andreas Gladisch, Fritz-Joachim Westphal, T-Systems; Klaus Grobe, ADVA AG Optical Networking; Kristof Obermann, FH Regensburg; Erwin Patzak, Michael Schlosser, Fraunhofer-Institut für Nachrichtentechnik, Heinrich-Hertz- Institut; Jan Späth, Ericsson GmbH Kontaktadresse: Dr. Gert J. Eilenberger, Alcatel-Lucent Deutschland AG, Lorenzstr. 10, 70435 Stuttgart Tel.: +49.711.821-32166, Fax: +49.711.821-32457, E-Mail: [email protected] Abstract This paper gives an overview on actual trends and deployments of carrier-grade Ethernet in metro, access, and core networks. This includes the related motivation, concepts, and technologies as well as open issues regarding research, development, and standardization. Ethernet as a packet-based, connection-oriented technology is deployed for metro networks worldwide today. This is driven by the massive increase of (IP-based) data traffic and the related applications. The Ethernet de- ployments aim at most cost-efficient data service provisioning and the migration of all legacy Layer-2 services towards a unified platform. The goal is a massive reduction of both, CapEx and OpEx. Network operators and service providers impose increased requirements regarding scalability, quality of service including reliability and availability, and Operations, Administration, and Maintenance (OAM) features on their metro Ethernet solutions. These requirements are usually referred to as carrier-grade or transport Ethernet. Metro Ethernet services as deployed today mainly consist of Ethernet Private Lines (EPL) or Ethernet Virtual Private LANs (EVPLAN). These can provide dedicated LAN extension or LAN-like connectivity via IP/MPLS, respectively. A different approach is Pseudowire Emulation Edge-to-Edge (PWE3) which allows MPLS transport of Ethernet and other packet services as well as synchronous TDM services. Various network architecture and protocol options exist to migrate from metro SONET/SDH and WDM networks towards even more Ethernet-centric and -optimized networks. These include Layer-2 transport like Transport MPLS (T-MPLS), Provider Backbone Transport (PBT), and Ethernet-over-SONET/SDH/WDM/OTH. These services are currently under investigation or being standardized, and they will also migrate into long-haul and backbone networks. These approaches have common requirements regarding network and control planes (e.g., ASON/GLMPS, GELS, T-MPLS). Thus, the corresponding management and control mechanisms have to have an integrated view on the lower 3 network layers. Further challenges for transport Ethernet result from upcoming technology steps like 100 Gbps Ethernet. Again, carrier-grade requirements and interworking aspects with transport networks have to be taken into account. 1 Introduction and Market the introduction of new technologies provides the Drivers chance to substantially improve the cost situation of operators. The last years have seen an enormous growth of In this situation, Ethernet gets right in the focus of at- bandwidth needs in metro networks of around 40% tention. Being a permanent success story since nearly per year. Recently, this growth has been sharply accel- two decades, scale effects have lowered the cost for erated especially from the residential side by the up- enterprise Ethernet equipment down to such levels coming Web 2.0 driven end user applications like that it appears the suitable salvation for the band- peer-to-peer and video file sharing. Traffic growth width-revenue dilemma that the network operators and rates up to 100% per year are anticipated for the their customers are facing. Consequently, we currently backbone networks. see Ethernet appearing in many locations throughout At the same time, the revenues of the carriers stay carrier networks – both as a service and an infrastruc- pretty constant or are only rising in the order of a few ture: E.g., Ethernet-based LAN interconnects are re- percent. It is commonly understood that the resulting placing classical Layer-2 services like ATM and steep decline in revenue per bandwidth unit can only Frame Relay in the form of transparent LAN point-to- be absorbed via technological discontinuities: Only point services. Their extension towards virtual private LAN services is currently finding more and more cus- counting, Performance, and Security Management). tomers. Many new applications in the sectors of health The basic functions which are necessary include (con- and education are based on Ethernet technology right tinuous) Connectivity Check (CC), Loopback (LB), from the start as are new concepts like Services grid Trace Route (TR), and functions for alarm suppres- computing. sion, discovery, performance monitoring, and surviv- At the same time the current residential access infra- ability (protection switching, restoration). This is de- structure is being migrated to architectures based on scribed in Y.1731 (ex-Y.17ethoam) in more detail. IP DSLAMs leading to a pure Ethernet connectivity The OAM functions CC, AIS/RDI signalling, ping throughout the access networks. By the way, this de- (LB) und Trace Route are schematically shown in Fig. velopment will provide even more bandwidth to the 1. end users – additionally fuelling the bandwidth explo- sion in the backbone networks. As it seems, Ethernet CC CC CC Multi-hop CC has the potential to become the new convergence plat- Signalling Signalling form for packet transport in the same way as IP has AIS/RDI or upper/lower Signalling Protocol become the convergence platform for applications and services in the Internet. However, not only the side of the capital expenditures has to be considered. Ethernet technology can only LB (Ping) provide the correct solution for packet transport if op- TR-1 erational expenditures in the network can be reduced TR-2 at the same time. Therefore, special attention has to be TR-3 concentrated on the carrier-grade design of those fea- tures of Ethernet that support the operations, admini- Fig. 1: Ethernet OAM: Connectivity Check (CC), stration, and management of networks. AIS/RDI signalling, Trace Route (TR), and ping The next section of this paper will explain in more de- (LB) tail these carrier-grade requirements for OAM, hierar- chical layering and resilience. Section 3 then explains End-to-end management is a major requirement for the most important Ethernet services and applications carrier-grade Ethernet. (The same is true for every while section 4 describes Ethernet-based network ar- other transport technology, and end-to-end manage- chitectures and technology trends. Finally, section 5 ment was one of the main drivers behind OTH, for draws some conclusions. example refer to TCM, Tandem Connection Monitor- ing.) In the Ethernet context, all network layers (core, access) and all technologies (e.g., MPLS, EFM) have 2 Carrier-grade requirements to support the related basic OAM functions. This in- on Ethernet transport cludes interworking over several carrier domains. This scenario is shown in Fig. 2, together with the most Carriers have different requirements with respect to relevant OAM functions. The Ethernet standards their networks as compared to enterprise networks. 802.1ag and 802.3ah as mentioned in Fig. 2 are de- These requirements reflect the necessity to operate scribed in more detail hereinafter. and manage complex networks and to guarantee cer- MPLS OAM: VCCV, LSP Ping/Traceroute tain Service Level Agreements (SLAs). Consequently, the carrier requirements in particular apply to the ar- Service Provider eas of Operations, Administration, and Maintenance CE Ethernet CE MPLS Core Ethernet (OAM), layered network architectures, and mecha- Access Access nisms for resilience. These are briefly discussed here- inafter, with respect to the Ethernet protocol. Operator Domain Operator Domain Operator Domain Provider Domain Customer Domain 2.1 OAM E-LMI: Automated configuration of CE based on EVC and BW profiles, 802.1ag Connectivity Fault Management: - Domains contain OAM flows + bound OAM responsibility - Per-EVC connectivity management and fault isolation 802.3ah EFM OAM: Physical connectivity management between L2 connectivity management - 3 OAM packets: CC, L2 Ping, L2 Traceroute devices, when applicable Next to the transport, the supervision of (Ethernet) Fig. 2: Ethernet end-to-end OAM (E-LMI: Eth signals is most relevant. Carrier Ethernet networks Line Management I/F) must provide OAM functionality similar to SONET/SDH. The basic Ethernet OAM mechanisms are described in the ITU-T Y.1730 and Y.1731 stan- dards. These are related to the EPL, EVPL, EPLAN, 2.2 Layered Network Architecture and EVPLAN reference models and are aligned with the ITU-T SG15 (G.8010, G.8011). Ethernet networks must be able to provide transparent In principle, OAM functions have to provide and sup- interconnection of all sites of given customers A, B, C port FCAPS management (Fault, Configuration, Ac- etc. while maintaining complete isolation between these customers. The corresponding function is known M-in-M obviously is a complete recursion instead of as VLAN tagging and is standardized in IEEE just adding further tags as is the case with Q-in-Q. 802.1ad. The major disadvantage of this standard is its lack of scalability, or the lack of providing a hierar- SP MAC DA Destination MAC address If destination unknown, then 0xFFFFFF chically layered network architecture. 802.1ad is still SP MAC SA Source MAC address SP Header limited to 4096 VLAN addresses. VLAN tagging (also Traffic ET=0x8100 Management referred to as Q-in-Q) was hence complemented by a 3 1 12 SP Q-tag1 C fully recursive, layered architecture which is referred P-bits F I Tunnel ID (XXX) ET=MiM to as M-in-M (Mac-in-Mac), or Provider Backbone SP Payload Service Tag 7 1 24 Bridge (PBB), and which is described in 802.1ah. The Reserved PT Service ID (YYY) corresponding end-to-end network concept is shown Customer Ethernet Future in Fig. 3. Frame Growth, Payload Type EVC ID Vendor- (Data or Control) 16M SP FCS specific Site Y Ethernet packet SP Ingress switch Ethernet packet SP Egress switch ET: Ethertype CFI: Canonical Field Identifier arrives from CPE at adds SP Eth header switched across SP removes SP Eth SP ingress switch Src and Dest MAC network using details header and forwards addresses are in SP Eth header original packet to CPE Ethernet UNI ports Fig. 5: MinM Data Plane Frame Format Ethernet UNI (Destination) Site X Ethernet Switches 2.3 Resilience Ethernet UNI (Source) User Enterprise SP Eth Service Provider Resilience (protection, restoration) is necessary to en- Ethernet network data Eth Header Header able a certain availability (AV) of services. This is necessary because AV is part of Service Level Agree- Fig. 3: IEEE 802.1ah MinM Principle ments (SLAs). Today, SONET/SDH and WDM (ring) The layering (i.e. the encapsulation of client Ethernet protection is used in most metro networks. These pro- frames into carrier frames) is provided in a shim vide high (service) availabilities and fast switch-over. header. Basic shim functions are mapping of 802.1ad Disadvantages include high cost due to redundant ca- S-VIDs (Service VLAN ID) into Extended Service pacity, and the fact that more and more services – in VIDs (I-SIDs), encap/decap of 802.1ad frames, learn- particular Ethernet – are not to be transported over ing and correlation of backbone POP and customer SONET/SDH networks anymore. This leads to the re- MAC addresses, and filtering of L2 control packets quirement for additional resilience mechanisms in the sourced by core relays or by provider bridge relays Ethernet layer. These can be complemented on de- (divides spanning trees). The PBB shim functions are mand by protection in the SONET/SDH, OTH, and shown in Fig. 4. IP/MPLS layers, and later by GMPLS restoration (ITU-T ASON). 802.1ad S-Shim Operations Typical examples of service (i.e. path) AVs are 98.5%
Maps S-VID from 802.1ad into larger Relay Extended Service VID (I-SID) for unprotected services, and up to 99.995% for 802.1ad MIF PB MIF
Filters L2 control packets sourced by core relays or by provider bridge relays highly available, protected services (e.g., in the SAN Relay PBB MCF (divides spanning trees) context). Here, path AV includes fibers and equip- S-shim MIF BB MIF MAC ment. Hence, high AV can be achieved by providing T-Shim Operations MCF PBB 802.3
Encap/decap of 802.1ad frame redundancy with respect to fibers and transport T-shim MAC
Learns and correlates Backbone POP and Customer MAC addresses equipment. Path AV is then influenced by the fiber Virtual MAC 802.3
Filters L2 control packets sourced by downtime, together with the availabilities of the ser- core relays or by provider bridge Backbone Edge relays (divides spanning trees) vice-affecting components. Protected ETH#A SNC Fig. 4: PBB Shim Functions (MIF: Media Inde- West ETH#A (Normal Traffic) East pendent Function, MCF: MAC Convergence Function) Working Transport Entity (for ETH#A) ETH#A (Normal Traffic) With M-in-M, each B-VLAN (Backbone VLAN) car- ries many S-VLANs (Service VLANs, i.e. 802.1ad ETH_FF SNC Protection Transport Entity (for ETH#A) ETH_FF SNC Protection Protection VLANs). S-VLANs may be carried on a subset of a Switching ETH_FP Switching Process Process B-VLAN (i.e. all P-P S-VLANs could be carried on a single multipoint B-VLAN providing connection to all Fig. 6: Ethernet 1+1 protection (ETH_FF; end points). An I-SID uniquely identifies an S-VLAN Ethernet Flow Function, ETH_FP: Ethernet Flow within the backbone. B-VLANs are addressed like Point) regular VLANs with a 12 bit B-VID. B-VID and I- Ethernet 1+1 and 1:1 point-to-point Sub-Network SID need to be separate ID spaces to allow many S- Connection (SNC) protection is currently standardized VLANs to be carried in a single B-VLAN. The result- in ITU-T G.8031 "Ethernet Protection Switching" ing MinM data plane frame format is shown in Fig. 5. (Y.1342, ex. Y.17ethps). This standard describes SNC protection for sub-networks constructed from point-to- This work essentially deals with the question on how point Ethernet VLANs. The basic functionality is to provide non-IP VPNs directly on non-IP/MPLS shown in Fig. 6. transport technology such as SDH, OTH and in future Next versions of the G.8031 will consider sub- Ethernet. Unfortunately, Ethernet is yet not addressed, networks constructed from multi-point-to-multi-point because as a pre-condition a GMPLS control Plane for Ethernet VLANs. Potentially, this will include en- Ethernet needs to be defined. As long as IEEE, which hancements to IEEE 802.17 (Resilient Packet Ring, is the 'owner' of Ethernet standardization, does not as- RPR) in order to provide ring protection for specific sign an appropriate label space which can be used for Ethernet VLANs. control, further work on this subject is blocked. A first attempt to create a working group dealing with GMPLS for Ethernet was made to achieve the follow- 3 Ethernet Applications ing goals: • Control of Ethernet switches using GMPLS proto- cols in support of point-to-point paths. • It is a non-objective of the IETF to initiate any 3.1 Virtual Private Networks Ethernet data plane work PWE3 The work on VPNs in IETF essentially started with One key Element to provide VPN Services is the ca- BGP/MPLS VPNs leading to a basic requirements pability to transport non-IP traffic over an IP/MPLS document in 1999 (RFC2547). Later work was based network. This requires communication services that on this standard and split up into activities related to can emulate the essential properties of traditional Layer 3 (IP), Layer 2 (Ethernet) and Layer 1 (SDH, communication links over a PSN. A pseudowire emu- OTH). As a consequence also RFC2547 got several lates a point-to-point link, and provides a single ser- updates and is now replaced by RFC4364. The vice which is perceived by its user as an unshared link L3VPN working group which initiated the work on or circuit of the chosen service. It is not intended that VPNs is responsible for defining provider-provisioned an emulated service will be indistinguishable from the Layer-3 (routed) Virtual Private Networks (L3VPNs). service that is being emulated. The emulation needs Ethernet as service or transport technology is not con- only be sufficient for the satisfactory operation of the sidered there but in L2VPN and L1VPN activities. service. Emulation necessarily involves a degree of L2VPN cost-performance trade-off. Switching, multiplexing, The L2VPN activity deals with the question on how to modification or other operation on the traditional ser- create and transport Ethernet services over an vice, unless required as part of the emulation, is out of IP/MPLS network providing the following services: the scope of the PWE3 WG. • Virtual Private LAN Service (VPLS): A L2 service A PW operating over a shared PSN does not necessar- that emulates LAN across an IP and an MPLS- ily have the same intrinsic security as a dedicated, enabled IP network, allowing standard Ethernet purpose built network. In some cases this is satisfac- devices to communicate with each other as if they tory, while in other cases it will be necessary to en- were connected to a common LAN segment. hance the security of the PW to emulate the intrinsic • Virtual Private Wire Service (VPWS): A L2 ser- security of the emulated service. PWE3 will work vice that provides L2 point-to-point connectivity closely with the L2VPN WG to ensure that a clear (e.g. Frame Relay DLCI, ATM VPI/VCI, point-to- demarcation is defined for where PWE3 stops and point Ethernet) across an IP and an MPLS-enabled L2VPN starts. IP network. WG Objectives are to specify the following PW types: • IP-only VPNs: A L2 service across an IP and • Ethernet, Frame Relay, PPP, HDLC, ATM, low- MPLS-enabled IP network, allowing standard IP rate TDM, SONET/SDH and Fibre Channel. devices to communicate with each other as if they • PWE3 will not specify mechanisms by which a were connected to a common LAN or with some PW connects two different access services. mesh of point-to-point circuits (not necessarily • Specify the control and management functions of fully meshed). chartered PW types, to include PW setup, configu- L2 interworking is not in the current scope. Overall, ration, maintenance and tear-down the work on above subjects is well advanced and can • Specify Operation and Management (OAM) be considered as stable from a standardization point of mechanisms for all PW types, suitable for opera- view. tion over both IP/L2TPv3 and MPLS PSNs, and capable of providing the necessary interworking L1VPN with the OAM mechanisms of the emulated ser- In contrast to the L2VPN activity which is using vice. IP/MPLS as a server layer technology for Ethernet • Define requirements for and mechanisms to pro- services, the L1VPN Working Group specifies mecha- vide protection and restoration of PWs. nisms necessary for providing layer-1 VPN over a GMPLS-enabled transport service-provider network. 3.2 Ethernet Carrier Internal Use 4 Network Architectures From carriers’ perspective, the main driver for 4.1 Architectures for the First Mile and Ethernet services will be external customers, but also aggregation networks for internal demands Ethernet will become more and more an effective and attractive solution. Ethernet in the First Mile (EFM) The driver for the use of Ethernet will be the same, as Ethernet in the First Mile (EFM) is a relevant standard described in chapter 1 of this document, namely re- in the Metro Ethernet context. It is described in the duced ports costs and scalability options. During the IEEE 802.3ah standard and promoted in the market by following years, Ethernet ports will be a standard the Ethernet in the First Mile Alliance (EFMA). An product, which will be produced in a huge amount. overview is given in [1]. For that reason, Ethernet interfaces will be much EFM describes native Ethernet access in metro net- cheaper than e.g. SDH Interfaces. The price for a works. Design goals were on consolidation of the ac- STM-16 SR (=Short Reach) SFP will be approxi- cess with respect to the dominance of the Ethernet mately equivalent to 2 x Gigabit Ethernet SFP ports protocol, and hence lowest cost for voice, data, and (1000 Base LX) or 4 x Gigabit Ethernet ports (1000 video access. It is believed that EFM will replace Base SX). This small example shows what impact to other access technologies (E1/T1, E3/T3, STM-1/OC- the costs of a network operator can happen if a change 3) over time. Using EFM, expensive protocol conver- from SDH / POS interfaces on equipment to Ethernet sions in the access can be avoided. In addition, it is interfaces will be done. Also if the carriers will use for possible to use single-ended demarcation units which their internal platform one common “protocol”, can be managed directly via the service providers’ Ethernet, it is possible to standardize the in-house in- edge routers (no unit necessary in the service provid- frastructure (e.g. cabling). These savings for interfaces ers’ PoPs). and infrastructure will become much larger if a com- In IEEE 802.3ah, three access topologies are defined plete network will be possibly changed. The benefit – copper-based (EFMC), SSMF-basiert (EFMF), and will increase if more new platforms like Voice over IP based on a passive point-to-multipoint topology or VDSL or Layer 2 networks will be rolled out. (EFMP, the EFM version of EPON). Hybrid solutions (EFMH) are also possible. For these topologies, IEEE The second advantage of Ethernet will be the scalabil- 802.3ah defines the OAM methods, i.e. performance ity effects of Ethernet. With SDH you will have a very monitoring, loopback, and fault detection and isola- rough granularity (E1, E3, STM-1, STM-4, STM-16). tion. For low bitrates SDH has a very flexible and fine For fiber access, EFMF is the relevant substandard. It granularity, but for higher bandwidth (STM-4  defines full duplex with 1 Gbps GbE via SSMF over STM-16  STM-64) the capacity increases by the at least 10 km distance. It also describes single- and factor of 4. For Ethernet it is possible to increase the dual fiber access for point-to-point at 100 Mbps. capacity with two mechanisms. One is the increase on EFMC defines access via Cat3 copper cables at 10 additional ports (e.g. n x 10 Mbps) or also the re- Mbps over 750 m. quired bandwidth can be defined by software on a cer- tain level (e.g. 23 Mbps). The advantage of this mechanism is that several platforms can share one big Ethernet Passive Optical Network (EPON) data pipe and every service will have a guaranteed The IEEE 802.3ah EFM standard also introduces the bandwidth. concept of Ethernet Passive Optical Networks (E- PONs), in which a Point-to-Multipoint (P2MP) net- The change to Ethernet was also driven by the change work topology is implemented with passive optical of the method of transport of Ethernet or data traffic. splitters, along with optical fiber Physical Medium In the beginning, backbone networks were based on Dependent sublayers (PMDs) that support this topol- SDH and there were only very difficult ways of map- ogy. In addition, a mechanism for network Operations, ping Ethernet into SDH. The at this time available so- Administration and Maintenance (OAM) is included lutions were low cost, not managed converter boxes or to facilitate network operation and troubleshooting. routers in bridge mode or the POS interfaces, which EPONs (also known as EFMPs) are supported in the do some kind of a mapping of IP data into SDH on the market by the Ethernet First Mile Alliance (EFMA) card. Now functionalities like GFP or G.709 are avail- which became part of the Metro Ethernet Forum able, which offer the opportunity for mapping (MEF) [2]. Ethernet traffic directly into SDH or wavelength/sub EPONs enable IP-based P2MP connections using pas- wavelength. Also, DWDM systems in the Wide Area sive fiber infrastructure. Up- and downstream (US, are no more only focused on SDH interfaces DS) are controlled using the Multi Point Control Pro- (e.g.STM-16/STM-64) and nowadays offer for exam- tocol (MPCP). The US makes use of TDMA. ple several Gigabit Ethernet and 10 Gigabit Ethernet EPON was mainly motivated by the disadvantages of interfaces on one wavelength transponder. ATM (APON). These include the facts that dropped cells invalidate entire IP datagrams, that ATM imposes a cell tax on variable-length IP packets, and that ATM Ethernet aggregation platforms are also used to offer in general did not live up to its promise of becoming IPTV, video on demand and voice-over-IP services. an inexpensive technology. Moreover, Ethernet techniques in the aggregation do- EPON on the other hand provides an IP data- main are also a very promising candidate to be used as optimized access network, considering the fact that common production platform for the different service Ethernet is by far the most relevant protocol in the ac- portfolios offered to residential and business custom- cess. It provides EPON encapsulation of all data in ers (Figure 8) and thereby also providing business Ethernet fames. The EPON layer stack is shown in customers with dedicated IP- and Ethernet based ser- Fig. 7, in comparison to APON and GPON. vices (e.g. LAN interconnection) over the same ag- gregation infrastructure. APON GPON EPON Higher Layer Higher Layer Higher Layer ATM Adaptation Sublayer ATM Adapter GEM Adapter MAC Client TC GTC MAC Multipoint MAC Control Layer PON Transmission Sublayer Layer GTC Framing Sublayer Layer MAC Physical Layer Physical Layer Physical Layer Fig. 7: EPON layers compared to APON and GPON. (G)TC: (GPON) Transmission Conver- gence layer. Single-mode fibers are used for EPON. Single-Fiber Working is enabled by using 1300 nm for the US and Fig. 8: Ethernet-based aggregation networks for 1500 nm for the DS, respectively. Splitting ratios of residential and business customers [3]. 4:1 to 64:1 are supported (typically 16:1). The maxi- mum optical power budget is 20 dB, enabling maxi- mum link lengths of 10…20 km. EPON provides a symmetrical bit rate of 1.25 Gbps 4.2 Backbone Ethernet Networks (connec- for Ethernet transport only. In the DS, Ethernet frames tion oriented forwarding) transmitted (broadcast) by the OLT pass through the N:1 passive splitter and reach each ONU (with own Currently, transport networks mainly use SDH-based MAC addresses). This is similar to a shared-media framing architectures like GFP for transferring network. Almost 50% of the available bandwidth is Ethernet traffic over transport networks. However, required for the protocol overhead, leaving only ~600 novel concepts arise that use packet techniques di- Mbps for revenue use. rectly above the WDM layer. However, the unmodi- In the US, data frames from any ONU will only reach fied usage of end-to-end Ethernet network concepts in the OLT due to the directional properties of the pas- general is limited by scalability issues: Based on con- sive splitter/combiner. This is similar to an Ethernet figurable IDs of switches, configurable port weights, P2P architecture. However, EPON frames from differ- and priorities, the Spanning Tree Protocol (STP) cal- ent ONUs transmitted simultaneously can still collide. culates a single tree-structure to connect any switch Hence, ONUs need to share the trunk fiber channel with each other. Although loop-less forwarding is capacity and resources. guaranteed with this mechanism, STP provides only The EPON system provides a very basic transport so- one path between two locations and a MAC address lution where cost-effective data-only services are the learning of any equipment is performed at the primary focus. EPONs are receiving a lot of attention switches. in the Far East where missing pieces of the 802.3ah When combining large networks and adding hundreds standard are being driven by NTT. There is not much of customer networks with an Ethernet-based core interest in the U.S. and parts of Europe. network, the number of MAC addresses grows rapidly. EPON as a protocol is still under work within the Thus, scalability cannot be provided and a separation IEEE EFM group. In the 802.3av Task Force of networks or an additional hierarchy between them (10GEPON) the physical layer is extended to 10 has to be introduced to allow a scalable forwarding of Gbps. data. Ethernet Aggregation Networks Also, the use of only one tree structure and with it the Besides the last mile and PON structures, Ethernet is possibility to use only one path between two locations also starting to spread out in the aggregation and hamper the use of efficient traffic engineering and re- metro area – often denoted as “second mile”. Here, silience mechanisms. Thus, three connection-oriented currently a replacement of traditional ATM-based ag- forwarding technologies are currently discussed at gregation structures is taking place: One or more standardization bodies for Carrier-Grade Ethernet Ethernet switching stages aggregate the traffic of the transport networks: VLAN Cross-Connect (VLAN- residential customers which is, at least in Europe, of- XC), Provider Backbone Transport (PBT), and Trans- ten provided by xDSL techniques on the lasst mile and port Multi-Protocol Label Switching (T-MPLS). Scal- thereby utilizing the existing copper-based- ability is provided by all three proposed forwarding infrastructure. Beside the standard IP services, the technologies via the introduction of a backbone- network hierarchy for the forwarding of traffic. Edge and is removed at the egress backbone switch. Figure switches manipulate the incoming Ethernet packets 9 illustrates the frame structure of T-MPLS. S-TAG C-TAG and add tunnel information. Instead of MAC learning GFP or Ethernet T-MPLS DA SA TP S- TPI S- ID VID D VID L/T User Data FCS 2 2 2 (which is disabled inside the core in all the technolo- 6 octets 6 octets 2 2 46 – 1500 octets 4 octets gies) the forwarding is performed along pre-defined Fig. 9: T-MPLS frame structure. tunnels. The number of tunnels that have to be pro- vided depend on the number of edge-switches, sup- To use the MPLS concepts also in transport environ- ported types of services, and the number of distin- ments, some changes were necessary. E.g., the control guishable networks (VLANs). planes are separated, i.e. T-MPLS operates independ- ently of its clients and its associated control networks VLAN Cross-Connect (VLAN-XC): (management and signalling network). The use of Pe- The main idea of VLAN-XC is to establish pre- nultimate Hop Popping is prohibited as are the merg- defined tunnels between edge switches of a network ing of tunnels as well as the equal distribution of traf- and to use these tunnels to route and differentiate traf- fic onto paths (ECMP). fic from each other. Instead of using a destination MAC address for the forwarding decision, a label (VLAN-XC Tag) is encoded in the Ethernet header to 4.4 Control/Management aspects determine the appropriate tunnel. Ingress edge- switches have to analyze incoming packets, chose one Besides already well elaborated and widely used Con- of the pre-defined tunnels, and label an Ethernet trol Plane protocols based on MPLS and GMPLS, packet accordingly. Intermediate switches route the new approaches specifically taylored for Ethernet are traffic according to the given tunnel label and are able currently under discussion. to swap the label. Finally, the tunnel label is removed In November 2005 an initiative in the Internet Engi- at an egress switch to allow the transparent transporta- neering Task Force (IETF) was started to use the tion of customer data. With this functionality, multiple GMPLS Control Plane for Ethernet switches in order paths between two edge-switches are supported. Traf- to scale Ethernet solutions beyond the limitations of a fic can be separated and distributed in the network and LAN service. This initiative called GELS (GMPLS- traffic engineering is facilitated. To avoid changing controlled Ethernet Label Switching) intended to dy- the Ethernet header structure, VLAN-XC uses the bits namically manage the Ethernet resources. The idea reserved for VLAN-IDs of IEEE 802.1Q and IEEE was to advertise the aggregate available bandwidth on 802.1ad to encode the tunnels. each wavelength-link together with the set of available Ethernet VLAN tags via OSPF-TE. Provisioning ac- Provider Backbone Transport (PBT): tions could be instantiated using RSVP-TE signalling Similar to the VLAN-XC, Provider Backbone Trans- in order to set up Label Switched Paths (LSP) with the port establishes pre-defined tunnels between edge requested bandwidth and a proper VLAN tag. Each switches. However, instead of adding a label to the Ethernet switch would then translate RSVP-TE signal- header, a MAC encapsulation is performed at the edge ling messages into local switch commands to create switches (Figure 8). the desired VLAN-ports associations along with the MAC encapsulation Same B-DA but different B-VIDs requested bandwidth guarantees. Whenever an Ethernet circuit (or LSP) is set up or torn down, the * * bandwidth and VLAN tag information would be up- dated via distribution of OSPF-TE Link State Adver- tisements (LSAs) in order to maintain proper link states across the network. This way, a scalable Ethernet network for a Wide Area Network could be Fig. 8: MAC encapsulation in the Core network in achieved including all defined resilience and mainte- PBT. nance mechanisms currently available on GMPLS im- plementations for SDH/SONET networks. Transport Multi Protocol Label Switching (T- While the underlying idea was appealing, no progress MPLS): was made so far since backwards compatibility with Transport Multi-Protocol Label Switching (T-MPLS) existing Ethernet switches is of major concern for op- is an adaptation of MPLS and is defined in ITU-T erators as well as for vendors. It is an ongoing activity G..8110.1. The main idea is to use the well established to scope the GELS activity such that compatibility is- MPLS concept known from IP routing and adapt it for sues are covered sufficiently. At this point in time the transport forwarding issues. As with VLAN-XC and following issues need to be resolved: PBT, T-MPLS establishes pre-defined tunnels. In T- 1. Ethernet VLANs have no bandwidth assigned, MPLS an additional MPLS header is pushed in front while in GMPLS bandwidth assignment would be of the client traffic that is transported transparently used to improve scaling and allow traffic engineer- inside the backbone network. Similarly to VLAN-XC ing. the 20bit label is used to encode the backbone tunnel 2. Ethernet VLAN labels are not switchable entities and size will be unchanged. Besides the serial trans- while in GMPLS an addressing entity is required mission of 100 Gbps currently the following WDM to be switched on a per port basis. options are under discussion for the realization of the 3. Alternative approaches using special identifiers or physical layer: 10x10 Gbps, 5x20 Gbps, 4x25 Gbps MAC addresses are also under consideration but and 2x50 Gbps. Up to know it is not finally clear, raise concerns about interoperability and scalabil- which versions will go into the final standardization ity of the overall solution. process. Further work is necessary to identify in collaboration Within the ITU there are activities concerning 100 with IEEE the required identifiers and switching enti- Gigabit Ethernet in the Study Group 15. One objective ties which allow the implementation of a GMPLS is the support of OTN interworking, another is the in- based control plane. vestigation of parallel interfaces (WDM) or serial in- terfaces. A further concern is the support of already installed fibre infrastructure. 4. 5 Technology trends In general, Ethernet operation at speeds of 100 Gbps is very desirable in terms of architecture-related net- 5 Conclusion and outlook work cost [4]. The transmission of high speed data We describe the advantages of Ethernet for customer rates above 100 Gbps itself is well understood and can and carriers. Furthermore, the paper gives an overview be managed. As a consequence, the knowledge to real- of several tendencies in the development and future of ize the transmission of a 100 Gbps Ethernet signal is Ethernet. But at the moment it is not completely de- present. The problem still to solve is to find efficient cided, what will be “the” solution for Carrier Grade electro-optical and opto-electrical conversion tech- Ethernet. niques. Electrical solutions are preferable to handle A common understanding of the technology and in- the data at the transmitter and receiver since OTDM terworking options of the different solutions (e.g. be- techniques are still too complex and difficult to im- tween carriers or vendors) should be available. Oth- plement in commercial products. erwise only island solutions for Ethernet networks will By using ultra-fast electronic circuits instead of elabo- be available, like it is today. rating optical methods in high-capacity optical trans- Today there are different solutions for offering mission systems cost per transmitted bit per second Ethernet Services, like e.g. conversion on a fibre or and kilometer can be reduced. Electronic circuitry for SDH bandwidth, an SDH/GFP solution, Switched 40 Gbps is already commercially available. To really Ethernet platforms or an MPLS based VPLS solution. exploit the cost advantage of an electrical receiver The problems occur, when the different solutions will compared to optical solutions a compact integrated be connected to one service. This can happen due to device is needed - preferably a single chip. different possible frame sizes (e.g. from 64 Bytes to up to Jumbo Frames with 9028 Bytes), transparency (e.g. only data transparency to transparency of VLAN- IDs, Mac-in-Mac, Q-in Q, customer specific or vendor specific signalling information, Link Aggregation, EFM, Fault Management and Multicast Frames or protocols like Spanning Tree) and alarming status Fig. 11: Photo (left) and block diagram (right) of (e.g. Link Loss Forwarding, switch off of the port, or the integrated ETDM receiver chip [5]. not defined status). Recently, as an important step towards 100 Gbps Nevertheless, Ethernet will be “the” transport protocol Ethernet an integrated ETDM receiver comprising for the future and will lead us to Ethernet based net- 1:2-demultiplexing (DEMUX) and clock & data re- works. covery (CDR) on a single chip was presented [5]. This receiver was tested in a 100 Gbps transmission ex- 6 References periment. Error-free performance (BER < 10-9) was obtained back-to-back and after transmission over 480 [1] Ethernet in the First Mile. Making Universal Broadband Access a Reality. A White Paper presented by Ethernet in the First Mile km of dispersion managed fiber. The ETDM receiver Alliance (EFMA), 2003 was initially designed for 80 Gbps operations. A re- [2] www.metroethernetforum.org design of the receiver chip is expected to enable an [3] Lehmann / Autenrieth / Derksen / Leisching: “Die neue Ether- even better performance and operation at even higher net-Generation: 100-Gigabit-Ethernet mit integrierten elektrisch- optischen Hochgeschwindigkeitsschaltkreisen“, Photonik, 2/2006. bit rates. [4] A. Schmid-Egger / A. Kirstädter: "Ethernet in Core Networks: The IEEE 802.3 Higher Speed Study Group was es- A Technical and Economical Analysis", HPSR2006 - 2006 IEEE tablished in 2006 to evaluate the extension of the pre- Workshop on High Performance Switching and Routing, Poznan, sent Ethernet standards to interface speeds of 100 Poland, June 7-9, 2006.. [5] R.H. Derksen, G. Lehmann, C.-J. Weiske, C. Schubert, R. Lud- Gbps. An objective is the support of 100 Gbps over 40 wig, S. Ferber, C. Schmidt-Langhorst, M. Möller, J. Lutz, „Inte- km on standard single mode fibers and 100 meters on grated 100 Gbit/s ETDM Receiver in a Transmission Experiment OM3 multi mode fibers. The Ethernet frame format over 480 km DMF“, Proc. of OFC 2006, PDP37.