Getting to What Matters: Accelerating Your Path Through the Big Data Lifecycle with CSC and Hortonworks
Download as PDF, PPTX3 likes2,499 views
The document outlines the collaboration between CSC and Hortonworks to accelerate the adoption and implementation of big data solutions through Apache Hadoop. It highlights the challenges of traditional systems and the advantages of Hadoop in managing large volumes of varied data efficiently and cost-effectively. The document also presents case studies demonstrating successful big data platform implementations across various industries, showcasing how organizations can achieve rapid insight and operational improvements.
More Related Content
Similar to Getting to What Matters: Accelerating Your Path Through the Big Data Lifecycle with CSC and Hortonworks(20)
Getting to What Matters: Accelerating Your Path Through the Big Data Lifecycle with CSC and Hortonworks
- 1. © Hortonworks Inc.2011 – 2014. All Rights Reserved Getting to What Matters: Accelerating Your Path Through the Big Data Lifecycle with CSC and Hortonworks
- 2. © Hortonworks Inc.2011 – 2014. All Rights Reserved Presenters • John Kreisa (@marked_man) VP Strategic Marketing, Hortonworks Over 20 years in data management as a developer and a marketer • Tim Gasper (@TimGasper) Global Offerings Manager, CSC Led product for Infochimps for 4 years, now called the CSC Big Data PaaS; leads product/offering management for CSC Big Data & Analytics
- 3. © Hortonworks Inc.2011 – 2014. All Rights Reserved Traditional systems under pressure Challenges • Constrains data to app • Can’t manage new data • Costly to Scale Business Value Clickstream Geolocation Web Data Internet of Things Docs, emails Server logs 2012 2.8 Zettabytes 2020 40 Zettabytes LAGGARDS INDUSTRY LEADERS 1 2 New Data ERP CRM SCM New Traditional
- 4. © Hortonworks Inc.2011 – 2014. All Rights Reserved Hadoop emerged as foundation of new data architecture Apache Hadoop is an open source data platform for managing large volumes of high velocity and variety of data • Built by Yahoo! to be the heartbeat of its ad & search business • Donated to Apache Software Foundation in 2005 with rapid adoption by large web properties & early adopter enterprises • Incredibly disruptive to current platform economics Traditional Hadoop Advantages ü Manages new data paradigm ü Handles data at scale ü Cost effective ü Open source Application Storage HDFS Batch Processing MapReduce
- 5. © Hortonworks Inc.2011 – 2014. All Rights Reserved SYSTEMS INTEGRATOR OPERATIONAL TOOLS DEV & DATA TOOLS INFRASTRUCTURE Hadoop is deeply integrated in the data centerSOURCES EXISTING Systems Clickstream Web &Social GeolocaDon Sensor & Machine Server Logs Unstructured DATASYSTEM RDBMS EDW MPP APPLICATIONS Deep Partnerships Hortonworks engages in deep engineered relationships with the leaders in the data center, such as HP, Microsoft, Red Hat, SAP, SAS & Teradata Broad Partnerships Over 600 partners work with us to certify their applications to work with Hadoop so they can extend big data to their users HDP 2.2 Governance &Integration Security Operations Data Access Data Management YARN
- 6. © Hortonworks Inc.2011 – 2014. All Rights Reserved SYSTEMS INTEGRATOR OPERATIONAL TOOLS DEV & DATA TOOLS INFRASTRUCTURE CSC and the Modern Data Architecture Modern Data Architecture • Enable applications to have access to all your enterprise data through an efficient centralized platform • Supported with a centralized approach governance, security and operations • Versatile to handle any applications and datasets no matter the size or type CSC Extends Hadoop’s Reach • Allows for multiple deployment options - including on-premise, managed or Big Data as a Service. • CSC’s global consulting services can help you architect, develop and implement your big data strategy, analytics, integrations, and platforms Clickstream Web & Social GeolocaDon Sensor & Machine Server Logs Unstructured SOURCES Existing Systems ERP CRM SCM ANALYTICS Data Marts Business Analytics Visualization & Dashboards ANALYTICS Applications Business Analytics Visualization & Dashboards ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° HDFS (Hadoop Distributed File System) YARN: Data Operating System Interactive Real-TimeBatch Partner ISVBatch Batch MPP EDW
- 7. © Hortonworks Inc.2011 – 2014. All Rights Reserved Hadoop Driver: Cost optimization Archive Data off EDW Move rarely used data to Hadoop as active archive, store more data longer Offload costly ETL process Free your EDW to perform high-value functions like analytics & operations, not ETL Enrich the value of your EDW Use Hadoop to refine new data sources, such as web and machine data for new analytical context ANALYTICS Data Marts Business Analytics Visualization & Dashboards HDP helps you reduce costs and optimize the value associated with your EDW ANALYTICSDATASYSTEMS Data Marts Business Analytics Visualization & Dashboards HDP 2.2 ELT ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° N Cold Data, Deeper Archive & New Sources Enterprise Data Warehouse Hot MPP In-Memory Clickstream Web & Social GeolocaDon Sensor & Machine Server Logs Unstructured Existing Systems ERP CRM SCM SOURCES
- 8. © Hortonworks Inc.2011 – 2014. All Rights Reserved Single View Improve acquisition and retention Predictive Analytics Identify your next best action Data Discovery Uncover new findings Financial Services New Account Risk Screens Trading Risk Insurance Underwriting Improved Customer Service Insurance Underwriting Aggregate Banking Data as a Service Cross-sell & Upsell of Financial Products Risk Analysis for Usage-Based Car Insurance Identify Claims Errors for Reimbursement Telecom Unified Household View of the Customer Searchable Data for NPTB Recommendations Protect Customer Data from Employee Misuse Analyze Call Center Contacts Records Network Infrastructure Capacity Planning Call Detail Records (CDR) Analysis Inferred Demographics for Improved Targeting Proactive Maintenance on Transmission Equipment Tiered Service for High-Value Customers Retail 360° View of the Customer Supply Chain Optimization Website Optimization for Path to Purchase Localized, Personalized Promotions A/B Testing for Online Advertisements Data-Driven Pricing, improved loyalty programs Customer Segmentation Personalized, Real-time Offers In-Store Shopper Behavior Manufacturing Supply Chain and Logistics Optimize Warehouse Inventory Levels Product Insight from Electronic Usage Data Assembly Line Quality Assurance Proactive Equipment Maintenance Crowdsource Quality Assurance Single View of a Product Throughout Lifecycle Connected Car Data for Ongoing Innovation Improve Manufacturing Yields Healthcare Electronic Medical Records Monitor Patient Vitals in Real-Time Use Genomic Data in Medical Trials Improving Lifelong Care for Epilepsy Rapid Stroke Detection and Intervention Monitor Medical Supply Chain to Reduce Waste Reduce Patient Re-Admittance Rates Video Analysis for Surgical Decision Support Healthcare Analytics as a Service Oil & Gas Unify Exploration & Production Data Monitor Rig Safety in Real-Time Geographic exploration DCA to Slow Well Declines Curves Proactive Maintenance for Oil Field Equipment Define Operational Set Points for Wells Government Single View of Entity CBM & Autonomic Logistic Analysis Sentiment Analysis on Program Effectiveness Prevent Fraud, Waste and Abuse Proactive Maintenance for Public Infrastructure Meet Deadlines for Government Reporting Hadoop Driver: Advanced analytic applications
- 9. © Hortonworks Inc.2011 – 2014. All Rights Reserved Hadoop Driver: Enabling the data lakeSCALE SCOPE Data Lake Definition • Centralized Architecture Multiple applications on a shared data set with consistent levels of service • Any App, Any Data Multiple applications accessing all data affording new insights and opportunities. • Unlocks ‘Systems of Insight’ Advanced algorithms and applications used to derive new value and optimize existing value. Drivers: 1. Cost Optimization 2. Advanced Analytic Apps Goal: • Centralized Architecture • Data-driven Business DATA LAKE Journey to the Data Lake with Hadoop Systems of Insight
- 10. © Hortonworks Inc.2011 – 2014. All Rights Reserved Case Study: 12 month Hadoop evolution at TrueCar DataPlatformCapabilities 12 months execution plan June 2013 Begin Hadoop Execution July 2013 Hortonworks Partnership May ‘14 IPO Aug 2013 Training & Dev Begins Nov 2013 Production Cluster 60 Nodes 2 PB Jan 2014 40% Dev Staff Proficient Dec 2013 Three Production Apps (3 total) Feb 2014 Three More Production Apps (6 total) 12 Month Results at TRUECar • Six Production Hadoop Applications • Sixty nodes/2PB data • Storage Costs/Compute Costs from $19/GB to $0.23/GB “We addressed our data platform capabilities strategically as a pre-cursor to IPO.”
- 11. © Hortonworks Inc.2011 – 2014. All Rights Reserved CSC Big Data & Analytics • Fastest Time to Value Proven methodologies and customer success stories achieving insight in 30 days and production rollout in 90. • Industry Analytics Expertise Experience combining horizontal analytics approaches and techniques with industry and vertical specialization. • Global Solutions Integrator Worldwide delivery capabilities and experience with a broad set of both open and proprietary technologies and vendors. • End-to-End Consulting Taking customers on a journey from strategy and roadmap, to business and technology transformation, to ongoing SLA management and as-a-Service.
- 12. © Hortonworks Inc.2011 – 2014. All Rights Reserved CSC Big Data Platform as a Service Big Data Platform as a Service Flexible Deployment Options Hadoop Queries Streams CSC Command and Control MongoDB Elasticsearch Storm Kafka PostgreSQL PostGIS Deployment Center Operations Center Support Center Application Center Knowledge Center Public Cloud Virtual Private Cloud Enterprise Private Cloud Dedicated Cluster Enterprise Grade Security Access Control Compliance Support Perimeter Security Activity Monitoring Audit Logging Encryption Malware Protection Hardened OS DataStax TitanDB ETL Data Transformation Business Intelligence Data Mining Advanced Analytics Geolocation Hive w/ Tez HBase Accumulo HDFS, YARN, MR, Spark, …
- 13. © Hortonworks Inc.2011 – 2014. All Rights Reserved Across the Industries, Clients See the Possibilities Financial Services Utilities Transportation Health and Life Sciences Retail Telecommunications • Fraud detection • Risk management • 360° view of the customer • Real-time route optimization based on traffic and weather • Maintenance optimization and asset tracking • 360° view of the customer • Click-stream analysis • Real-time promotions Law Enforcement • Real-time multimodal surveillance • Situational awareness • Cybersecurity detection • CDR processing • Churn prediction • Geomapping/marketing • Network monitoring • Epidemic early warning system • ICU monitoring • Remote healthcare monitoring • Analysis of weather impact on power generation • Transmission monitoring • Smart grid management • Predictive maintenance • Real-time parts flow monitoring • Product configuration planning Manufacturing
- 14. © Hortonworks Inc.2011 – 2014. All Rights Reserved But They Struggle With Consistent Challenges • Data complexity • Robust and scalable service • Speed of stand-up 1. Setting up and operating a big data and analytics platform 2. Applying the right data science 3. Integrating insights into their business processes • Skills shortage • Skills retention 4. Identifying and managing big data skills
- 15. © Hortonworks Inc.2011 – 2014. All Rights Reserved Time to Value, Time to Next Iteration Business Discovery Info Discovery Logical Data Model Physical Data Model System Staging Data Ingestion, Transformation, ETL Application Development Analytics Production Staging Data Warehouse Project 12-24 Months to Reach Production Big Data Project 3-6 Months to Reach Production Prod. Stag. Business Discovery Info Discovery Sys. Stag. Initial Data Ingest Schema on Read Analytics App Dev Schema on Read Analytics App Dev Schema on Read Analytics App Dev Schema on Read Analytics App Dev Schema on Read Analytics App Dev Schema on Read Analytics App Dev
- 16. © Hortonworks Inc.2011 – 2014. All Rights Reserved Following the Big Data Maturity Lifecycle and… • Determining use cases • Art of the possible • Technology evaluation & understanding • Validate business value hypothesis with real data • Quick win, low hanging fruit, rapid initial phase • Implement one key transformation or insight into business process • Longer project timelines and robust ROI tracking • Expand to other key use cases for a big data enabled department of business function • Incorporate complementary tools and technology for a broader solution • Shift from a department or function focus to a cross-org focus • Introduce insights from across silos • Implement self-service capabilities for analytics and data integration • Provide marketplaces, catalogs, and collaboration zones
- 17. © Hortonworks Inc.2011 – 2014. All Rights Reserved … Leveraging an App Reference Design Framework It’s all about the apps.
- 18. © Hortonworks Inc.2011 – 2014. All Rights Reserved Proof of Value: Food & Hospitality Retailer This Food & Hospitality Retailer has a footprint of over 650 regional hotels, 2,800 coffee shops, and a number of restaurant chains. CSC provides the infrastructure, data platform, and analytics that uncovers revenue opportunities in customer web interactions. • The client wanted to quickly evaluate the use of big data and the value that it brings as it relates to identifying new business opportunities • Ease of use was a key need in making insights and reporting more accessible to analysts… and increasing the speed with which they could analyze • Time to market was a key factor in the decision to implement a comprehensive big data platform. The client realized: – A bare platform would not be easy to manage – Their staff does not possess the skills to operate a bare platform – They needed to focus on the big data applications, rather than the platform • CSC designed and configured the solution, built and deployed it in the cloud, and developed ETL flows to transport web activity data within 90 days: – Core platform (BDPaaS) leveraging Hortonworks Data Platform, including Hive with Tez – Aggregating lots of different data sources to create one massive web log data set – Adding data science algorithms to clean up data for better insights – Providing Pentaho Business Analytics as a comprehensive reporting and dashboard suite for insight presentation • CSC managed the infrastructure, platform components, and data flows, in addition to providing continued support/consultation services to the client • The client is generating insights on how customers interact with their website, and improving their services for happier customers and more streamlined business: – Faster path to ROI with both tech and services – Creating a real-time customer insights dashboard and set of reports – Ability to prove the value of big data internally through the mining of data and generation of insights and reports for various teams – Scalability to more data sources and use cases, including plans for mobile application analytics and operational metrics, as well as operational business analytics combining internal and external data sources SOLUTIONCHALLENGE RESULTS
- 19. © Hortonworks Inc.2011 – 2014. All Rights Reserved Business Unit Strategy: Network Rail Network Rail manages the most of the rail infrastructure across Great Britain, responsible for control and maintenance of over 2,500 railway stations, 20,000 miles of track, and 40,000 bridges and tunnels. CSC provides a data and analytics hub for massive amounts of imagery and analog track monitoring data. • Network Rail needed a platform that could not only store, but also analyze petabytes of data over the long-term: – Track imagery and video data captured via drones and cameras – Vibration data captured via maintenance trains – Other forms of large file size analog data crossed with operational, structured data sets • Network Rail wanted to implement the solution quickly, and ramp up data volumes at a fast pace • Goal of leveraging combined services to assist with loading data, managing the underlying infrastructure, and working with and analyzing the data • CSC designed and configured the solution, built and deployed it in the cloud, and developed ETL flows to import massive amounts of bulk data on an ongoing basis – Core platform (BDPaaS) leveraging Hortonworks Data Platform, including Hive with Tez • CSC’s platform integrated with ESRI ArcGIS for Big Data geolocation analysis features including geotagging and geo tiles • CSC managed the infrastructure, platform components, and data flows, in addition to providing continued support/consultation services to the client • Network Rail is generating insights on how to prioritize in near real-time the improvement and maintenance of the massive railway track and infrastructure footprint – Advanced analytics of analog data, including geolocation capabilities – Ability to handle the scale required by the massive amount of data under management and data growth – Complete transformation of a business unit’s analytics capability on track for success in less than 12 months SOLUTIONCHALLENGE RESULTS
- 20. © Hortonworks Inc.2011 – 2014. All Rights Reserved Question & Answer session will be conducted electronically, using the panel to the right of your screen Get started with Hortonworks Sandbox http://hortonworks.com/sandbox Follow us: @hortonworks CSC Big Data Maturity Survey http://www.csc.com/big_data_index Learn More @CSCNews Next Steps CSC Big Data Home http://www.csc.com/big_data
Editor's Notes
- #4 Before we dive into Hadoop and its role within the modern data architecture, let’s set the context for why Hadoop has become important. Existing approaches for data management have become both technically and commercially impractical. Technically - these systems were never designed to store or process vast quantities of data Commercially – the licensing structures with the traditonal approach are no longer feasible. These two challenges combined with rate at which data is being produce predicated a need for a new approach to data systems. If we fast-forward another 3 to 5 years, more than half of the data under management within the enterprise will be from these new data sources.
- #5 Enter Hadoop. Faced with this challenge the team at yahoo conceived and created apache hadoop to address the challenge. They then were convinced that contribution of this platform into an open community would speed innovation. They open sourced the technology and did so within the governance of the Apache Software Foundation. (ASF) This introduced two distinct significant advantages. Not only could they manage new data types at scale but the now had a commercially feasible approach. However, there will still significant challenges. The first generation of Hadoop was: - designed and optimized for Batch only workloads, - it required dedicated clusters for each application, and, - it didn’t integrate easily with many of the existing technologies present in the data center. Also, like any emerging technology, Hadoop was required to meet a certain level of readiness required by the enterprise. After running Hadoop at scale at yahoo, the team spun out to form Hortonworks with the intent to address these challenges and make Hadoop enterprise ready.
- #6 The modern data architecture simply does not work unless it integrates with the systems and tools you already deploy. HDP enables your existing data platforms to expand the data you have under management through integration. The goal of HDO is to augment not replace these existing systems as we very clearly understand that you need to ruuse skills. Further, through our work within the Hadoop community to deliver YARN, we have opened up Hadoop and unlocked innovation in the community of data center ISVs can extend their applications so that they can run natively IN Hadoop as just another workload operating on the single set of data lake. They can now function as a first class citizen alongside any other workload in Hadoop.
- #7 In 2011, Hortonworks was founded with the 24 original Hadoop architects and engineers from Yahoo! This original team had been working on a technology called YARN (Yet Another Resource Negotiator) that enable multiple applications to have access to all your enterprise data through an efficient centralized platform. It is the data operating system for hadoop that provides the versatility to handle any application and dataset no matter the size or type. Moreover, YARN provided the centralized architecture around which the critical enterprise services of Security, Operations, and Governance could be centrally addressed and integrate with existing enterprise policies. This work allowed for a new approach to data to emerge, the modern data architecture. At the heart of this approach is the capability for Hadoop to unify data and processing in an efficient data platform
- #8 Lets first start with cost optimization… there are three primary drivers. First, its about storage optimization: Archive your data off the EDW to drive down costs Second is to optimize data processing: typically a large portion of EDW usage is for low value transformational workloads. Many of these can be transitioned away from the EDW and into Hadoop and in doing so, this frees up significant resources from the EDW And finally hadoop can be used to capture new types of data that can then be refined and used within the context of the analysis of your EDW, introducing wholly new analysis and insight.
- #9 On the other hand, many start with a new analytic app based on data not previously captured. These new types of data include clickstream, sentiment machine and sensor data, geolocation data, server logs and the tomes of unstructured data often found within the enterprise. While your application will vary tremendously based on your vertical, we see commonality across application patterns intended to find value in these rich data sources. We generally find there are three patterns of application SINGLE VIEW OF ENTITY The first of three common patterns in analytics applications, a single view of an entity (like a customer, product or a machine) is now possible because platforms like Hadoop can store and organize previously unmanageable columns and varieties of data. PREDICTIVE ANALYTICS As data scientists and analysts reveal patterns and correlations inside massive data sets, new models emerge to explain business performance. Most importantly, these models can reliably predict future events based on previously dissociated data. DATA DISCOVERY New, voluminous data types such as machine and sensor data, geolocation data, clickstream data and sentiment data are valuable when correlated with other data sets in a shared enterprise “data lake.” The patterns within the data lake can then fuel machine learning applications. It is in these patterns we see them unlock value form the types of data, previously not capable before.
- #10 Ultimately, most organizations that adopt Hadoop, aspire to create a data lake where multiple applications use a shared set of resources, for both storage and processing all with a consistent level of service. The value in the data lake ultimately results in delivery of “systems of insight” where advanced algorithms and applications that access multiple data sets allow organizations to derive brand new value from data that was once unable to be investigated or simply to complex to combine and analyze. Hadoop doesn’t just create a Data Lake—it opens the platform for analysts to view multiple data sources in multiple dimensions and reduce time to insight. This journey from apps to lake is only possible with HDP and its YARN based architecture.
- #11 Let’s talk about TrueCar as they are a great example of an organization who started small but grew big… they also did this very quickly. TRUECar focuses on making car buying fair and fun for everyone. They bring together a lot of messy automotive industry data from a wide range of sources with a wide range of formats. Since they make money when cars get bought, their value is in how well they’re able to drive advanced correlations across the data to deliver an interactive customer experience that accelerates an informed buying decision. At TRUECar, data is the product they sell, and they made rolling out a Hadoop-based data architecture a pre-cursor to going IPO earlier this year. With the help of Hortonworks and in just over a year, they realized their vision of a data lake and have transformed their business. At the beginning of their journey they had very limited knowledge of Hadoop. We partnered with them to train resources, then we worked with them to develop an architecture and help them through development and implementation until finally, we now provide mission critical support for their production environment. What started with a single app on HDP grew to three in a few months and in a year they had 6 business apps running on a single 60node cluster that holds over 2PB of data. Their 60-node production cluster was rolled out in Nov 2013 and grew 5X in the year after that.