Rescue your Big Data from Downtime with HP Operations Bridge and Apache Hadoop
Download as PDF, PPTX3 likes2,989 views
The document discusses the integration of HP Operations Bridge with Apache Hadoop for managing big data effectively. It highlights the benefits of Hadoop in optimizing data operations, providing scalable compute costs, and enhancing real-time data accessibility. Additionally, it outlines the features of Apache Ambari for provisioning and monitoring Hadoop clusters, along with the role of partnerships in extending big data capabilities.
More Related Content
Similar to Rescue your Big Data from Downtime with HP Operations Bridge and Apache Hadoop (20)
![Discover.hdp2.2.ambari.final[1]](https://cdn.slidesharecdn.com/ss_thumbnails/discover-150114162536-conversion-gate01-thumbnail.jpg?width=560&fit=bounds)
![Discover.hdp2.2.h base.final[2]](https://cdn.slidesharecdn.com/ss_thumbnails/discover-141218155001-conversion-gate02-thumbnail.jpg?width=560&fit=bounds)
Rescue your Big Data from Downtime with HP Operations Bridge and Apache Hadoop
- 1. We’ll get started soon… Q&A box is available for your questions Webinar will be recorded for future viewing Thank you for joining! © Hortonworks Inc. 2011 – 2014. All Rights Reserved
- 2. Rescue your Big Data from Downtime with HP Operations Bridge and Apache Hadoop © Hortonworks Inc. 2011 – 2014. All Rights Reserved We do Hadoop.
- 3. Your speakers… Ajay Singh, Director of Technical Channels Hortonworks Saumya Upadhyaya, Product Manager HP © Hortonworks Inc. 2011 – 2014. All Rights Reserved
- 4. Big Data Trends © Hortonworks Inc. 2011 – 2014. All Rights Reserved
- 5. Traditional systems under pressure RDBMS EDW MPP © Hortonworks Inc. 2011 – 2014. All Rights Reserved DATA SYSTEM APPLICATIONS Business Analytics Custom Applications Packaged Applications • Silos of Data • Costly to Scale • Constrained Schemas Clickstream Geolocation Sentiment, Web Data Sensor. Machine Data Unstructured docs, emails Server logs SOURCES Existing Sources (CRM, ERP,…) New Data Types …and difficult to manage new data
- 6. HDP is deeply integrated in the data center YARN © Hortonworks Inc. 2011 – 2014. All Rights Reserved DEV & DATA TOOLS OPERATIONAL TOOLS INFRASTRUCTURE SOURCES EXISTING Systems Clickstream Web &Social GeolocaBon Sensor & Machine Server Logs Unstructured DATA SYSTEM RDBMS EDW MPP HANA APPLICATIONS BusinessObjects BI Deep Partnerships Hortonworks engages in deep engineered relationships with the leaders in the data center, such as HP, Microsoft, Redhat, SAP, SAS & Teradata Broad Partnerships Over 600 partners work with us to certify their applications to work with Hadoop so they can extend big data to their users HDP 2.1 Governance & Integration Security Operations Data Access Data Management
- 7. 1. Unlock New Applications from New Types of Data INDUSTRY USE CASE Sentiment © Hortonworks Inc. 2011 – 2014. All Rights Reserved & Web Clickstream & Behavior Machine & Sensor Geographic Server Logs Structured & Unstructured Financial Services New Account Risk Screens ✔ ✔ Trading Risk ✔ Insurance Underwriting ✔ ✔ ✔ Telecom Call Detail Records (CDR) ✔ ✔ Infrastructure Investment ✔ ✔ Real-time Bandwidth Allocation ✔ ✔ ✔ Retail 360° View of the Customer ✔ ✔ ✔ Localized, Personalized Promotions ✔ Website Optimization ✔ Manufacturing Supply Chain and Logistics ✔ Assembly Line Quality Assurance ✔ Crowd-sourced Quality Assurance ✔ Healthcare Use Genomic Data in Medial Trials ✔ ✔ ✔ Monitor Patient Vitals in Real-Time Pharmaceuticals Recruit and Retain Patients for Drug Trials ✔ ✔ Improve Prescription Adherence ✔ ✔ ✔ ✔ Oil & Gas Unify Exploration & Production Data ✔ ✔ ✔ ✔ Monitor Rig Safety in Real-Time ✔ ✔ ✔ Government ETL Offload/Federal Budgetary Pressures ✔ ✔ Sentiment Analysis for Gov. Programs ✔
- 8. 2. Or to realize a dramatic cost savings… EDW Optimization ✚ © Hortonworks Inc. 2011 – 2014. All Rights Reserved OPERATIONS 50% ANALYTICS 20% ETL PROCESS 30% OPERATIONS 50% ANALYTICS 50% Current Reality EDW at capacity: some usage from low value workloads Older data archived, unavailable for ongoing exploration Source data often discarded Augment w/ Hadoop Free up EDW resources from low value tasks Keep 100% of source data and historical data for ongoing exploration Mine data for value after loading it because of schema-on-read Commodity Compute & Storage Hadoop Enables Scalable Compute & Storage at a Compelling Cost Structure Cloud Storage HADOOP Engineered System MPP SAN NAS $0 $20,000 $40,000 $60,000 $80,000 $180,000 Fully-loaded Cost Per Raw TB of Data (Min–Max Cost) Hadoop Parse, Cleanse Apply Structure, Transform Storage Costs/Compute Costs from $19/GB to $0.23/GB
- 9. 3. Data Lake: An architectural shift Unlocking the Data Lake SCALE SCOPE RDBM S MPP EDW © Hortonworks Inc. 2011 – 2014. All Rights Reserved Data Lake Enabled by YARN • Single data repository, shared infrastructure • Multiple biz apps accessing all the data • Enable a shift from reactive to proactive interactions • Gain new insight across the entire enterprise New Analytic Apps or IT Optimization HDP 2.1 Governance & Integration Security Operations Data Access YARN Data Management
- 10. HDP is deeply integrated in the data center YARN © Hortonworks Inc. 2011 – 2014. All Rights Reserved DEV & DATA TOOLS OPERATIONAL TOOLS INFRASTRUCTURE SOURCES EXISTING Systems Clickstream Web &Social GeolocaBon Sensor & Machine Server Logs Unstructured DATA SYSTEM RDBMS EDW MPP HANA APPLICATIONS BusinessObjects BI Deep Partnerships Hortonworks engages in deep engineered relationships with the leaders in the data center, such as Microsoft, Teradata, Redhat, HP, SAS & SAP Broad Partnerships Over 600 partners work with us to certify their applications to work with Hadoop so they can extend big data to their users HDP 2.1 Governance & Integration Security Operations Data Access Data Management
- 11. Hortonworks Data Platform © Hortonworks Inc. 2011 – 2014. All Rights Reserved
- 12. The 1st Generation of Hadoop: Batch HADOOP 1.0 Built for Web-Scale Batch Apps Single App INTERACTIVE Single App BATCH HDFS Single App BATCH HDFS © Hortonworks Inc. 2011 – 2014. All Rights Reserved • All other usage patterns must leverage that same infrastructure • Forces the creation of silos for managing mixed workloads Single App ONLINE Single App BATCH HDFS
- 13. 2006 2009 Hadoop w/ MapReduce MapReduce Largely Batch Processing HDFS MR-‐279: YARN 1 ° ° ° ° ° (Hadoop Distributed File System) ° ° ° ° ° N Hadoop2 & YARN based Architecture © Hortonworks Inc. 2011 – 2014. All Rights Reserved YARN: Data Operating System 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° N° HDFS (Hadoop Distributed File System) Silo’d clusters Largely batch system Difficult to integrate Hadoop 2 & YARN Batch Interactive Real-Time Architected & led development of YARN to enable the Modern Data Architecture October 23, 2013
- 14. A Blueprint for Enterprise Hadoop Load data and manage according to policy PRESENTATION & APPLICATION ENTERPRISE MGMT & SECURITY DATA ACCESS SECURITY Access your data simultaneously in multiple ways (batch, interactive, real-time) Provide layered YARN Data Operating System © Hortonworks Inc. 2011 – 2014. All Rights Reserved Deploy and effectively manage the platform Store and process all of your Corporate Data Assets approach to security through Authentication, Authorization, Accounting, and Data Protection DATA MANAGEMENT GOVERNANCE & INTEGRATION OPERATIONS Enable both existing and new application to provide value to the organization Empower existing operations and security tools to manage Hadoop Provide deployment choice across physical, virtual, cloud DEPLOYMENT OPTIONS
- 15. HDP delivers a comprehensive data management platform BATCH, INTERACTIVE & REAL-TIME SECURITY DATA ACCESS TezTez © Hortonworks Inc. 2011 – 2014. All Rights Reserved YARN is the architectural center of HDP • Enables batch, interactive and real-time workloads • Single SQL engine for both batch and interactive • Enable existing ISV apps to plug directly into Hadoop via YARN Provides comprehensive enterprise capabilities • Governance • Security • Operations The widest range of deployment options • Linux & Windows • On premise & cloud HDP 2.1 Hortonworks Data Platform Provision, Manage & Monitor Ambari Zookeeper Scheduling Oozie Data Workflow, Lifecycle & Governance Falcon Sqoop Flume NFS WebHDFS In-Memory Spark YARN: Data Operating System DATA MANAGEMENT GOVERNANCE & INTEGRATION Authentication Authorization Accounting Data Protection Storage: HDFS Resources: YARN Access: Hive, … Pipeline: Falcon Cluster: Knox OPERATIONS Script Pig Search Solr SQL Hive HCatalog NoSQL HBase Accumulo Stream Storm Others ISV Engines 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° N HDFS (Hadoop Distributed File System) Deployment Choice Linux Windows On-Premise Cloud
- 16. Apache Ambari © Hortonworks Inc. 2011 – 2014. All Rights Reserved
- 17. How do you Operate a Hadoop Cluster? Apache Ambari is a platform to provision, manage and monitor Hadoop clusters © Hortonworks Inc. 2011 – 2014. All Rights Reserved
- 18. Innovation at the Speed of Community An active 100% open source Community: • 90+ Contributors / 45+ Committers • 270+ Ambari User Group Members Graduates to Apache Top Level Project © Hortonworks Inc. 2011 – 2014. All Rights Reserved 2013 Dec
- 19. Apache Ambari Themes Operate Hadoop at Scale Deliver the © Hortonworks Inc. 2011 – 2014. All Rights Reserved core opera@onal capabili@es to provision, manage and monitor Hadoop clusters at scale. Integrate with the Enterprise Robust API for integra@on with exisBng enterprise systems, such as Teradata Viewpoint, HP OperaBons Bridge and MicrosoY SCOM. Extend for the Ecosystem Provide extensible pla[orm for Customers, Partners and the Community using Stacks and Views.
- 20. Hadoop Operations Platform © Hortonworks Inc. 2011 – 2014. All Rights Reserved Apache Ambari is a 100% open source platform for provisioning, managing and monitoring Apache Hadoop clusters AMBARI WEB HP Opera)ons Bridge compute & . . . storage . . . . . compute & storage . EXTEND . AMBARI REST API AMBARI SERVER PROVISION | MANAGE | MONITOR IntegraBon with ExisBng Tools OPERATE AMBARI STACKS
- 21. Cluster Operations Start, Stop and Smoke Test Services Apply configuraBon changes and inject custom properBes © Hortonworks Inc. 2011 – 2014. All Rights Reserved
- 22. Configuration Management Ability to update core-site.xml properties Add and Remove custom properties © Hortonworks Inc. 2011 – 2014. All Rights Reserved
- 23. Host Level Visibility + Controls View System Metrics and Host Information Manage Host Components, including start/stop/decommission “Add Hosts” to expand cluster capacity © Hortonworks Inc. 2011 – 2014. All Rights Reserved
- 24. Cluster Heatmaps Gain health visibility of entire cluster Quickly identify host outliers and potential issues Includes both System and Hadoop Metric Heatmaps © Hortonworks Inc. 2011 – 2014. All Rights Reserved
- 25. Hadoop Metrics Hadoop Service metrics such as HDFS, MapReduce and HBase Visibility info DFS capacity, file operations, RPC latency and JVM performance © Hortonworks Inc. 2011 – 2014. All Rights Reserved
- 26. Hadoop + System Monitoring • Watch key Hadoop operational metrics for all services • Master service availability, CPU utilization, network latency, storage capacity © Hortonworks Inc. 2011 – 2014. All Rights Reserved © Hortonworks Inc. 2014 • Host-level alerting • Component service processes and storage availability
- 27. Customizable Metrics Dashboard Watch key System and Hadoop metrics © Hortonworks Inc. 2011 – 2014. All Rights Reserved
- 28. Extensibility Features Goal: Extend Ambari without hard-coding in Ambari • To add new Services (ISV or otherwise) beyond HDP Stack • To customize a Stack for customer specific environments • To use Ambari for automating cluster installations • To share best practices on layout and cluster configuration • To extend and customize the Ambari Web UI • Add new capabilities, customize existing capabilities Stacks Blueprints Views © Hortonworks Inc. 2011 – 2014. All Rights Reserved
- 29. Bringing it all together HP Operations Bridge © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 30. No context = high risk Traditional Private Managed Public © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 30 without notice. High Opex Point Tools Uncorrelated Silos Slow Triage Inaccurate Reports Inefficient Sea of red What’s going wrong?
- 31. The typical challenge – application siloes Application performance monitored by customers 2 No link between infrastructure and application worlds 4 Can’t do automated remediation – not totally sure of root cause 5 • Slow resolution Can’t consolidate performance data to drive planning Silo-ed health management and 1 Silo-ed view of what’s happening © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 31 without notice. • Not business aligned • Not people efficient • Poor planning & optimization • Monitoring duplication 3
- 32. 3 Steps to deliver business SLAs © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 33. Solution: ACT - Aggregate • Single pane of glass across performance, events & topology • Consolidate the data across all point tools & applications Single pane of glass Access from anywhere Role driven Simple navigation Drill down to detail Integrations Biz Impact aware Traditional to New Style of IT Depth via Management Packs Breadth via Connectors & HP portfolio © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 33 without notice. Event Detection & Enrichment Event Correlation & Business Impact Incident Submission Recovery & Closure Investigation & Diagnosis Resolution Closed Loop Management Integrated in IT processes Bi-Directional integration of HP & 3rd party tools Downtime & Assets synchronized via common repository www.hp.com/go/hpln
- 34. Visibility into your Hortonworks Hadoop Environment Leverage Apache Ambari & OpsBridge interfaces for integration BSM Connector for Apache Ambari © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 34 without notice. Topology Events
- 35. Solution: ACT - Control • Correlate data to find root cause via the RTSM • Automated actions & monitoring Monitoring Self-maintaining monitoring Tool & Application Agnostic1 SME self-service portal Centralized Config & Reports Remediation Link to CI or root cause Custom Actions or Run Book Automation2 No Operator involvement © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 35 without notice. Predictive3 Automated baselining Identify anomalies Proactive alerting Across BSM data (HP & 3rd party) 1) SiS, ArcSight, OM Agents, Oracle, MS-SQL, HP Cloud, Vertica, Hadoop 2) Enabled through integration i.e. HP Operations Orchestration 3) Enabled through integration with Service Health Analyzer
- 36. Simplify IT Operations Out of the box monitoring deployment, orchestration & ticketing integrations Customizable Dashboard Topology Events Save expensive expert personnel hours by • Identifying actionable events through correlation • Automate remediation of frequent, batch events • Automate ticket creation to the right audience © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 36 without notice. Service Desk Orchestrate Analyze
- 37. Solution: ACT - Team • Collaborate across silos & teams • Gamify to engage operators and unlock efficiencies Achievements • Motivate Users • Guide desired behavior • Explore the product Experts Board • Identify experts per area • Improve collaboration • Increase efficiency © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 37 without notice. More to Come… • Exploration Map • Custom rewards • Challenges • Social features
- 38. What Hortonworks & HP Software are doing Managing Big Data for efficient Digital Business New HP Operations Bridge for Apache AMBARI • Native AMBARI API integration • Provisioning, managing and monitoring Apache Hadoop clusters • Supports HortonWorks Data Platform i.e. HDFS, Yarn, Hbase, … • Events and topology from the above Gain even more value capabilities using HP Operations Bridge Hadoop Management Pack • Events, topology, metrics for Hadoop HDFS and MapReduce • Management cockpit dashboards • OOTB SHR Content report packs © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 38 without notice. NEW BSM Connector for Apache Ambari Download free at HP Live Network
- 39. Applying Predictive IT Operations to Hadoop Automated Predictive Operations1 and HortonWorks Data Platform integrations HP Operations Bridge Topology Events Automated Remediation Automated Tickets Operations Analytics HP Operations BSriimdpglifey Unify, Automate your WebScale IT Save expensive expert personnel hours by • Identifying actionable events through correlation • Automate remediation of frequent, batch events • Automate ticket creation to the right audience 1 –Requires OpsBridge used in BSM configuration and Hadoop Management pack © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 39 without notice.
- 40. Wide Coverage of Tools & Application Coverage Out of the box | 46+ domains covered today | expanding rapidly AIX CentOS Cluster Debian HPUX HyperV IBM LPAR KVM RedHat Oracle Ent. Citrix Linux Solaris Solaris Zone Suse Ubuntu VMWare vSphere Windows XEN Infrastructure Applications Oracle DB Oracle RAC SQL Server SAP Sybase MS Exchange IBM Websphere WMQ Siebel Documentum Apache Webserver MS IIS SunOne iPlanet TIBCO JBOSS MS Lync MS Sharepoint MS Biztalk Oracle AS Informix Peoplesoft Tuxedo Big Data & Cloud HP Cloud Amazon Web SAP HANA DB Vertica DB Weblogic Services AD Ticketing Remedy Radia integrations HP ITSM Service Now Impact Manager AppDynamics Aternity CA Spectrum CollectD SIM IBM Tivoli MS SCOM 3rd Party Integrations Solarwinds VC Nagios OEM Ops Zabbix ZenOSS xMatters © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 40 without notice. Supported with OMi Supported with OM only DB2 Domino MS SCOM Apache Ambari
- 41. HP Software and Hortonworks Managing Big Data for efficient Digital Business Demo Remedial action © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 41 without notice. Predicted anomaly alert slow response marketing analysis Operations Bridge + HDP AMBARI Operator notices Hbase alert via AMBARI integration Dependency confirmed in Service tree ticket opened Hadoop admin sees ticket + context taken (manual or automatic) Marketing Analysis Performance v v
- 42. HP Operations Bridge allows you to “ACT” Simplifying your complex world Aggregate • Single pane of glass across performance, events & topology • Consolidate the data across all point tools & applications Control • Correlate data to find root cause via the RTSM • Automate actions & monitoring Team • Collaborate across silos & teams • Gamify to engage operators and unlock efficiencies © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 42 without notice.
- 43. Tangible benefits © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 43 without notice. 30% reduction in help desk call 60%+ reduction in cumulative monthly downtime 90% reduction in Time to Restore Service 30% reduced cost • Single consolidated view of domain managers • model-driven, X-domain reporting & forecasting 60% increased availability • Advanced business aware & model driven correlation • Faster identification of root cause • More efficient operations 90% improved efficiency • Single dashboard frees staff time for innovation • consolidated information • high degree of automation
- 44. Get more information Check out the OpsBridge Web Page • www.hp.com/go/OpsBridge Check out the latest OpsBridge Integration Content • https://hpln.hp.com/group/operations-bridge Check out the latest webinars / hangouts • Operations Bridge Hangout, Operations Bridge Webinar • Webinar “Learn how DIRECTV Embraced deep-dive tools for end-to-end visibility across IT” (link) • Webinar “Look No Hands - How Automation benefits IT Operations in Data Centers at Sykehuspartner and PlayTech” (Coming on Nov 19, 2014) • OpsBridge announcements at HP Discover (Coming on Dec 17, 2014) Check out the latest #HPBSM blogs © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 44 without notice.
- 45. Next Steps… More about HP & Hortonworks http://hortonworks.com/partner/HP/ Download the Hortonworks Sandbox Learn Hadoop Build Your Analytic App Try Hadoop 2 Contact us: [email protected] © Hortonworks Inc. 2011 – 2014. All Rights Reserved