Tower Operations Centre - Architecture & Blueprint
- 1. TOWER OPERATIONS CENTRE (TOC) - ARCHITECTURE & BLUEPRINT Shahid Abbasi 2016
- 2. • Provide high degree of automation & optimization of business processes • Support customization of business processes for individual departmental needs • Minimize management reaction times to network events • Allow for geographical dispersion of control over aspects of network operation • Share information across functional & organizational boundaries • Provide isolation mechanisms to minimize security risks • Provide mechanisms to locate and contain infrastructure faults • Proactive service assistance and interaction with customers • Allow for management of heterogeneous devices, sensors and services within a passive infrastructure environment • Allow for technological and functional changes • Include capabilities to enhance early implementation and allow future refinements (Day 0 & Final State Implementation) • Allow for Opex & Capex optimization through use of hybrid energy technologies and telemetry / remote monitoring (IoT) sensors, transducers & actuators Management System (TOC-MS) - Architecture Rqmnts
- 3. HIGH LEVEL SYSTEMS FUNCTIONALITY Business Support Systems • Asset & Inventory Management • Lease Management • Document Management • SLA Enforcement • Billing & Settlements • Financial Management (Capex & Opex tracking) • Workflow Management • Site Planning • GIS • Reporting • Management Dashboard Operations Support Systems • Site Design • Site Build-out & Implementation • Site Acceptance • Network Infrastructure Surveillance, Monitoring & Management • Fault Management • Trouble Ticketing & Tracking • Energy Management • Fuel Management • Security & Access Management • SLA Monitoring & Enforcement • Vendor Management TOC-MS High-Level System Functionality
- 4. CUSTOMER SERVICE CENTER • End-to-end remote site‘ monitoring at a single central location - a solution that is a combination of Remote Sensoring Unit & Central Application is required for the implementation of a Tower Operations Center (TOC) • A robust Business Management System (MS), which includes automation of modules like Fault Management, Trouble Ticketing, Energy Management, Fuel Management, Work Force Management, Supply Chain, Asset Management, GIS, etc • Monitor all the technical parameters of the different equipment installed at the Cell Sites • IoT Instrumentation in the form of host of Sensors, Transducers, Actuators • to collect parameters provided by these equipment and convert them into an understandable and known format • Monitoring and Control Equipment which can be hosted at each Cell Site • to collect all the inputs in different formats from the sensors, transducers and Actuators • to forward these parameters to an upstream application for centralized monitoring & management • Connectivity Mechanism • to transfer the collected information from the equipment placed at the Cell Site to the Centralized Application installed in the TOC TOC-MS Business Goals
- 5. CUSTOMER SERVICE CENTER • A Centralized Application • to process the information & events, collected from all the cell sites, generate alarms, issue trouble tickets, send alerts to the workforce, issue corrective actions back to the monitoring equipment, etc. • A Cloud-based Infrastructure • to seamlessly deliver TOC-MS features & functionality to specific locations, irrespective of MIS Infrastructure’s physical deployment, while ensuring data integrity, back-up & redundancy • A battery of users and operators • to understand the alerts and take corrective actions accordingly • Would handle all Level 1 & 2 Passive, and Level 1 Active infrastructure alarms and would deliver the MIS to monitor the performance and KPIs of the Infrastructure and the Vendors involved in various activities of the operations. • Backbone for TowerCo’s day-to-day Operations and would help to achieve the KPIs and SLAs as committed to the customers by the TowerCo. TOC Business Goals (Cont’d)
- 6. Tower Operations Centre – Initial State Day 0 SLA / KPI Site Access Fuel Site Assets Environment Power O&M Vendor’ Personnel Embedded TowerCo Personnel Embedded Network Operations Centre (Operator) Help Desk (TowerCo) CRITICAL MINOR WARNING NOTIFICATIONS Electronic Bonding (Alarm & TT) Infrastructure O&M Service Passive infrastructure surveillance First line maintenance Second line maintenance Infrastructure Planning & Optimization Infrastructure capacity management Infrastructure expansion management Routine network optimization Power / Energy & Fuel Optimization DG, fuel & power management Hybrid power systems Spare Parts Management Service Spare parts dimensioning Warehousing Repair & Return Inventory management Other Operational Services Instrument & tools management Documentation management Site access & security management SLA & KPI management Asset Management
- 7. Tower Operations Centre – Final State Integrated real-time site monitoring and reporting Telemetry Controller Tower Operations Centre (Primary) Tower Operations Centre (Back-up) Telemetry transmits all sites’ O&M data to both TOC 1 & 2 for redundancy purposes. TOC manages, monitors, controls, measures and maintains various equipment on sites. O&M Vendor’ Personnel Embedded
- 8. ENERGY EFFICIENCY MANAGEMENT ASSET MANAGEMENT ENVIRONMENTAL MANAGEMENT FUEL MANAGEMENT TOC – 360o Operations Management Support for complete business management of Tower Operations
- 9. HUMAN RESOURCE MANAGEMENT SMART SITE SECURITY FINANCIAL MANAGEMENT OPERATIONS TOC – 360o Operations Management (Cont’d) Support for complete business management of Tower Operations
- 10. TOC – Key functions and features Support for complete business management of Tower Operations ENERGY EFFICIENCY MANAGEMENT Generator and fuel management Rectifier alarms and variables Battery bank management Auto phase selector Hybrid management Grid availability & power meter’ reading verification SITE MANAGEMENT Detailed display – site assets QR barcode – individual site elements Site topology configuration Segregated maintenance procedures Real-time & historical trends ENVIRONMENTAL MANAGEMENT Free air-cooling utilization Room thermal mapping Remote configurations & settings Fire, smoke, and flood detection Cooling units consumption Temperature & HUMIDITY monitoring – different locations FUEL MANAGEMENT Fuel quality HYBRID remote on/off ability MODBUS compatibility Fuel consumption, fuel drop, & re-fuel measurement Versatile tank shape support
- 11. TOC – Key functions and features Support for complete business management of Tower Operations RESOURCE MANAGEMENT Personnel tracking Contractor working hour’ monitoring Alert and trouble ticket assignment Maintenance activity summaries per technician Real time access to human resources SITE ACCESS & SECURITY Tamper proof systems Site protection mechanisms Recorded back up systems Site access protocols Physical / guarded sites OPEX MANAGEMENT Opex reduction analysis Site behavior analysis Alternative resource site management Expense analysis & forecast SLA & KPI goal analysis Resource efficiency optimization Multi-tenant management OPERATIONS Trouble ticket management O&M supplier management Alarm management Auto / remote alarm control Business models: monthly/maintenance Site-process control Business intelligence Financial procurement
- 12. TowerCo Functional Requirements (Final State) Operations and Maintenance - Monitoring - Alerts Mgmt - Alarm Mgmt - Trouble Ticket - Fuel Mgmt - Energy Mgmt. - Env Mgmt - Site access and Security - Dispatch - Workforce Mgmt. - Help Desk - SLA Mgmt. - Site Mgmt Inventory and Asset Management - Physical Inventory - Logical Inventory - Service Inventory - GIS visual - Failure inventory - Sites plan - Hierarchical view Site Financials Management - Site Capex & Opex - NOC & Permits - Procurement - Rent Mgmt. - Expense Analysis and Forecast - Site Lease Mgmt. Enterprise Resource Planning - Company Financials - HR - Procurement - Document Mgmt Performance Management and Integration - KPI reporting - Business Intelligence - Financial planning - Modelling - Integration with O&M, I&AM, SFM, BM security integration monitoring and system management infrastructure
- 13. TowerCo Functional Requirements (Day 0) Operations and Maintenance - Monitoring - Alerts Mgmt - Alarm Mgmt - Trouble Ticket - Fuel Mgmt - Energy Mgmt. - Env Mgmt - Site access and Security - Dispatch - Workforce Mgmt. - Help Desk - SLA Mgmt. - Site Mgmt Inventory and Asset Management - Physical Inventory - Logical Inventory - Service Inventory - GIS Visual - Failure inventory - Sites plan - Hierarchical view Site Financials Management - Site Capex & Opex - NOC & Permits - Procurement - Rent Mgmt. - Expense Analysis and Forecast - Site Lease Mgmt. Enterprise Resource Planning - Company Financials - HR - Procurement - Document Mgmt Performance Management and Integration - KPI reporting - Business Intelligence - Financial Planning - Modelling - Integration with O&M, I&AM, SFM, BM security integration monitoring and system management infrastructure Capabilities required on Day 0
- 14. TOC-MS Actor & User Context TOC-MS Telemetry / IoT Sensors Field Engineers Business users Inventory / Asset users Help desk & TOC users Mgmt users Management reporting dashboard Financial reporting, site opex, supply chain, O&M vendors, payroll Inventory, asset mgmt OPCO Trouble ticketing, incident/change mgmt, alerts, events, alarms SLA reports Call dispatch, call status update Environment, fuel, security, energy data, video stream Actors (Users and instruments) interacting with TOC-MS
- 15. TOC-MS IT’ Technology Protocol Context TOC-MS Telemetry / IoT Sensors Field Engineers Business users Inventory / Asset users Help desk & TOC users Mgmt usershttps, Ethernet, 3G/LTE OPCO VPN over Internet, http/ web services/ file transfer https, 3G/LTE, SMS, SMTP, Telephone MQTT, SNMP, RTP, GPRS, 3G/LTE https, Ethernet, 3G/LTE https, Ethernet, 3G/LTE https, Ethernet, 3G/LTE, Telephone Technology Protocols interacting with TOC-MS
- 16. TOC-MS Architecture Overview Humidity Sensor Tower Site Field Office TowerCo Operations Center OPCO Fuel Level Sensor Fuel flow Meter SCM/ gateway Temp Sensor Smoke/Fire Detector Utilities Co. Smart meter Battery power Sensor Door Access Switch Security cameras + IVA 3G/LTE IVA + Video Recorder User - Browser + Mbl Head End DISCO DG Controller Alert Mgmt. System Help Desk System Fin. Mgmt. System RFID Controller/ QR Scanner Inventory/ Asset Mgmt User – Browser + Mbl User – Browser + Mbl ODS Reporting System Reporting Data Store User access - Web Browser MDMS Billing User – Browser + Mbl User Access - Web Browser + Mobile Identity Server Backup Server Security Systems management Messaging Server Messaging Client Systems and NW monitoring + mgmt Proactivebillretrieval TowerCo A/C meter Multi-channel DC Meter Rectifier
- 17. TOC-MS High Availability and DR Redundant IP links IP based data replication Devices/users/ stakeholders External Systems External Data Sources Normal operation DR operation Data replication • No SPOF configuration within DC • Disaster Recovery Site as per Industry Best Practices TowerCo Operations Centre IVA + Video Recorder User - Browser + Mbl Alert Mgmt. System Help Desk System Fin. Mgmt. System Inventory/ Asset Mgmt User – Browser + Mbl User – Browser + Mbl ODS Reporting System Reporting Data Store User – Browser + Mbl Identity Server Backup Server Security Systems management Messaging Server Systems and NW monitoring + Mgmt TowerCo Operations Centre IVA + Video Recorder User - Browser + Mbl Alert Mgmt. system Help Desk System Fin. Mgmt. system Inventory/ Asset Mgmt User – Browser + Mbl User – Browser + Mbl ODS Reporting System Reporting Data Store User – Browser + Mbl Identity Server Backup Server Security Systems management Messaging Server Systems and NW Monitoring + Mgmt
- 18. The TOC arrangement will consist of: • Main video wall display • Operators sitting (according to technology, geography or any subject such as redundancy) • Shift manager post (in the middle and elevated) • NOC manager room (tied to the main hall) • Back office support (second level experts) • Meeting room/visitors room (tied to the main hall) • Administrator room • Server room • Air conditioner / Climate control • Audio and noise planning TOC Set-up
- 19. INFRASTRUCTURE COMPONENTS - OPTIONAL • Acoustic Ceiling construction • Floating Floor (tiles, carpets) • Plaster Walls • Doors with different acoustic isolation • Windows between the different areas • Furnishings • Announcement System • Control Systems • Fire Detection System • Low voltage cabling systems Physical TOC Arrangement
- 20. MULTIMEDIA SYSTEMS - OPTIONAL • Audio Systems • Intercom System • Audio Broadcast System • Audio Recording Systems • Video Systems • Video Conference • Video Recording • TV Sets, Projectors, Screens • Smart Board • Plasma Panel / LCD • TV / Satellite Connection • Communication components • WAN/LAN Network • Servers, Computers, Printers TOC Infrastructure Components
- 21. Deployment Options Prod and DR on premises (Private Cloud): TowerCo Owned and Operated private cloud: - Heavy CAPEX Up front - Assets on TowerCo’s books - Higher time to market due to Data Centre build As a Service: - OPEX based option - Assets can be on TowerCo’s books or service provider - Still higher time to market due to DC build Prod on premise, DR on Cloud: - Relatively lower CAPEX - DR environment can be minimal until disaster happens saving on: - HW infrastructure cost - Licensing costs - Seamless movement of workloads between private and public cloud possible - Gives flexibility to burst into cloud for incidental workloads - Local vs overseas cloud providers can be evaluated Prod and DR on Cloud: - Lowest CAPEX option, possibly no assets on TowerCo’s books - As a service, SLA based operations - Endless scalability for additional workloads - No need to separate DR environment, SLA covers that - Heavy dependency on network for mission critical applications - If remote cloud, network latency could possibly impact response time, specially for telemetry based events requiring immediate action Prod on Cloud, DR on Premise: - Unusual option but has some merits for consideration - In case of network outage, failover to local site - Local resources can be re-purposed for trivial workloads unless there is a failover
- 22. Recommended Approach Prod and DR on premises (Private Cloud): TowerCo Owned and Operated private cloud: - Heavy CAPEX Up front - Assets on TowerCo’s books - Higher time to market due to Data Centre build As a Service: - OPEX based option - Assets can be on TowerCo’s books or service provider - Still higher time to market due to DC build Prod on premise, DR on Cloud: - Relatively lower CAPEX - DR environment can be minimal until disaster happens saving on: - HW infrastructure cost - Licensing costs - Seamless movement of workloads between private and public cloud possible - Gives flexibility to burst into cloud for incidental workloads - Local vs overseas cloud providers can be evaluated Prod and DR on Cloud: - Lowest CAPEX option, possibly no assets on TowerCo’s books - As a service, SLA based operations - Endless scalability for additional workloads - No need to separate DR environment, SLA covers that - Heavy dependency on network for mission critical applications - If remote cloud, network latency could possibly impact response time, specially for telemetry based events requiring immediate action Prod on Cloud, DR on Premise: - Unusual option but has some merits for consideration - In case of network outage, failover to local site - Local resources can be re-purposed for trivial workloads unless there is a failover • Start with a local cloud service provider • Build a site facility in parallel and migrate to owned facility when ready • Engage a systems integrator to build, manage and support the IT environment for the first 3 years • Build skills and capability in parallel • Take over management of IT operations
- 23. ID Fun- ction Description Metric Possible Candidate Products 1 Infra Directory Server (Active Director or LDAP) # of Users Microsoft/ Open source 2 Infra Backup and Archiving Server TB Data Veeam VBR / IBM TSM / HP / EMC Networker 3 Infra Messaging and collaboration Server (MS Exchange + Lync or Lotus Domino + ST) # of Users Microsoft Exchange+Lync/ IBM Lotus + Sametime 4 Appl Service Desk System # of Users HP Service Desk / BMC Remedy/ IBM Control Desk / Tarantula/ AIO 5 Appl Inventory, Asset and Maintenance Management System # of Users IBM / Cisco / Accruent / AIO Systems/ Tarantula 6 Appl Financial Management System Users / # of cores SAP/ MS Dynamics/ Oracle ERP 7 Appl Reporting Engine # of Users IBM Cognos/ SAS/ SAP Business Objects/ Watson Analytics 8 Appl Operations Control Center / Alert Management System # of users IBM IOC (websphere + Netcool)/ AIO Systems/ LCC/ Tarantula 9 Infra Server Virtualization Software # of proc sockes Vmware vSphere/ MS Hyper-v/ KVM 10 Infra Operating System (dependent on Application compatibility) # of VMs MS Windows / Linux 11 Infra Applications Systems and Network Management Tool # of servers+ Nw devices IBM Tivoli/ HP Netview/ EMC Smarts/ Solarwinds Products BoM (Software)
- 24. ID Fun- ction Description Metric Possible Candidate Products 12 MW Database Product # of cores MS SQL/ MySQL (OS)/ Oracle/ IBM DB2/ Informix 13 Infra Security certificates Thawte/ Verisign 14 Infra Video Management System # of sites/ cameras ACTi/ Avigilon 15 MW Integration Software # of cores IBM Integration Bus/ Tibco BW/ Oracle Fusion 16 MW Document and Workflow management # of users MS Sharepoint, IBM Filenet, EMC Documentum Products BoM (Software Cont’d)
- 25. ID Fun- ction Description Metric Possible Candidate Products 1 Infra Firewall cluster for Network security with virtualization Concurrent sessions Juniper / Cisco 2 Infra Load balancer cluster for Web based traffic with virtualization Concurrent sessions Brocade / Cisco 3 Infra Edge Routers Concurrent sessions Juniper / Cisco 4 Infra Network Fabric Switches (10 Gbps Ethernet) for LAN # of ports Juniper / Cisco 5 Infra Network Fabric Switches (10 Gbps FC over Ethernet) for SAN # of ports Juniper / Cisco 6 Infra Compute Servers (RISC or x86) for all functions # of proc cores X86 (HP, Lenovo, Dell)/ IBM Power 7 Infra SAN attached storage TB Data IBM v7000/ EMC VNX/ Dell 8 Infra Network connectivity (WAN) # of links, Data thruput Connectivity vendor 9 Infra Scanners # of units HP/ Canon 10 Infra Printers # of units HP/ Canon Products BoM (IT Hardware)
- 26. ID Fun- ction Description Metric Possible Candidate Products 1 Infra Sensor-cum-communication (SCM) module # of sites Tarantula/ AIO/ Commscope/6th Energy/LCC 2 Infra Fuel-level sensor # of fuel tanks Various 3 Infra Fuel flow meter # of fuel tanks Various 4 Infra Temperature sensor # of sites Various 5 Infra Environmental sensors # of sites Various 6 Infra Smoke sensor # of sites Various 7 Infra Door switch & Magnetic Locks / Access control # of sites Various 8 Infra Dual AC Energy meter – Mains & DG # of tenants Various 9 Infra 4-Channel DC Energy meter # of tenants Various 10 Infra Hybrid Energy Controllers (functionality may be available in SCM) # of sites Emerson/Eltek/Huawei/AIO/ Tarantula 11 Infra Security Cameras # of sites Indigovision/ RIVA/ Avigilon/ ACTi Products BoM (Telemetry, Sensors & Actuators)
Editor's Notes
- #7: Icons in green need to be changed to match presentation