Apache conbigdata2015 christiantzolov-federated sql on hadoop and beyond- leveraging apache geode to build a poor mans sap hana

Apache conbigdata2015 christiantzolov-federated sql on hadoop and beyond- leveraging apache geode to build a poor mans sap hana

• 1.
  Federated SQL onHadoop and Beyond: Leveraging Apache Geode to Build a Poor Man's SAP HANA by Christian Tzolov @christzolov
• 2.
  Whoami Christian Tzolov Technical Architectat Pivotal, BigData, Hadoop, SpringXD, Apache Committer, Crunch PMC member [email protected] blog.tzolov.net @christzolov
• 3.
  Contents • Data Systems- Principles • Use Case: OLTP and OLAP Data Systems Integration • Passive Data Synchronization (Demo) • Federated Queries With HAWQ • HAWQ Web Tables • HAWQ PXF Architecture • Geode PXF (Demo)
• 4.
  Data Systems
• 5.
  Compute Arbitrary Functions onArbitrary Data
• 6.
  Architectural Patterns • DataLake • Lambda • Kappa • Tachyon • …
• 7.
  Integration Stack Apache HDFSData Lake - PHD or HDP Hadoop Apache HAWQ SQL on Hadoop (OLAP) Apache Geode In-memory data grid (OLTP) Spring XD Integration and Streaming Runtime Apache Ambari Manages All Clusters Apache Zeppelin Web UI for interaction with Data Systems Hadoop/HDFS Geode HAWQ SpringXD Ambari Zeppelin
• 8.
  Apache Geode (OLTP) •Cache - Performance / Consistency / Resiliency • Region - Highly available, redundant, distributed Map China Railway Corporation 5,700 train stations 4.5 million tickets per day 20 million daily users 1.4 billion page views per day 40,000 visits per second Indian Railways 7,000 stations 72,000 miles of track 23 million passengers daily 120,000 concurrent users 10,000 transactions per minute
• 9.
  Apache HAWQ (OLAP) •Built around a Greenplum MPP DB (C and C++) • Hadoop Native: Parquet, HDFS and YARN • 100% ANSI SQL compliant: SQL-92/99/2003… • Extensible - Web Tables, PXF • Connectivity: ODBC and JDBC • Access internal store: HAWQ(Parquet)InputFormat
• 10.
  HAWQ - TPC-DS •Outperforms Impala by overall 454% • 344% of performance improvement over Hive/Tez • Runs 100% of the TPC-DS queries. Unlike Impala or Hive • References: http://bit.ly/1NUDcLl, https:// github.com/dbbaskette/pivbench
• 11.
  Spring XD Orchestrates andautomates all steps across multiple data stream pipelines • HTTP • Tail • File • Mail • Twitter • Gemfire • Syslog • TCP • UDP • JMS • RabbitMQ • MQTT • Kafka • Reactor TCP/UDP • Filter • Transformer • Object-to-JSON • JSON-to-Tuple • Splitter • Aggregator • HTTP Client • Groovy Scripts • Java Code • JPMML Evaluator • Spark Streaming • File • HDFS • JDBC • TCP • Log • Mail • RabbitMQ • Gemfire • Splunk • MQTT • Kafka • Dynamic Router • Counters
• 12.
  Ambari Management
• 13.
  Use Case: Join OLTPand OLAP Data Systems
• 14.
  Use Case • IntegrateGeode with HAWQ • Unified data view • Slowly Changing Dimensions (SCDs) • Keep the Operational and Historical data in Sync
• 15.
  Passive Data Synchronization
• 16.
  Passive Sync Architecture
• 17.
  Passive Sync -Demo
• 18.
  Passive Sync Improved (gpfdist)
• 19.
  Passive Sync Improved Demo
• 20.
  Federated Queries With HAWQ
• 21.
  HAWQ Web Tables •HAWQ Web Table - access dynamic data sources on a web server or by executing OS scripts • Leverage Geode REST API and OQL • SpringBoot Controller to convert JSON into TSV CREATE EXTERNAL WEB TABLE EMPLOYEE_WEB_TABLE (...) EXECUTE E'curl http://<hostname>/gemfire-api/v1/ queries/adhoc?q=<URLencoded OQL statement>' ON MASTER FORMAT 'text' (delimiter '|' null 'null' escape E'');
• 22.
  HAWQ Web Tables Architecture Accessdynamic data sources on a web server or by executing OS scripts.
• 23.
  HAWQ Web Tables Limitations •Not Scalable • No Push Down Predicates • Static • No Compression • Requires Additional Components
• 24.
  P(ivotal) Extension Framework (PXF) •Java-Based • Parallel, High Throughput Data Access • ANSI-compliant SQL On Any Dataset • Wide variety of PXF plugins
• 25.
  PXF Architecture
• 26.
  PXF Data Model •Data Source is modeled as a collection of one or more Fragments. • Each Fragment consists of many Rows that in turn are split into typed Fields. • Analyzer (optional) provides PXF statistical data for the HAWQ query optimizer • Metadata about the data source locations, access attributes, table schemas formats, SQL queries filters, etc
• 27.
  PXF Processors Plugin InputData Fragmeter getFragments() CustomAccessor CustomResolver Analyzer getEstimatedStat() CustomAnalyzer ReadResolver getFields(OneRow) WriteResolver getFields(OneRow) ReadAccessor openForRead() readNextObject() closeForRead() WriteAccessor openForWrite() writeNextObject() closeForWrite() CustomFragmeter ExtendClass Implement Interface
• 28.
  PXF Deployment Model HAWQMaster Query Dispatcher NameNode PXF Service Date Node X PXF Service Query Executor data request for Fragment X pxfwritable records Metadata request Fragment list External (Distributed) Data System Date Node Z PXF Service Query Executor data request for Fragment Z pxfwritable records Scan plan Result SQL query Result Parallelexecution
• 29.
  PXF External Tables CREATEEXTERNAL TABLE ext_table_name <Attribute list, …> LOCATION('pxf://<host>:<port>/path/to/data? FRAGMENTER=package.name.FragmenterForX& ACCESSOR=package.name.AccessorForX& RESOLVER=package.name.ResolverForX& <Other custom user options>=<Value>’ ) FORMAT ‘custom'(formatter='pxfwritable_import');
• 30.
  PXF Gallery •HdfsTextSimple •HdfsTextMulti •Hive •HiveRC •HiveText •HBase •Avro • Accumulo •Casandra • JSON • Redis • Geode/Gemfire • JDBC
• 31.
  HAWQ PXF/Geode
• 32.
  Federated Queries withPXF/ Geode - Architecture
• 33.
  PXF/Geode Table CREATE EXTERNALTABLE <GEMFIRE_TABLE_NAME> (...) LOCATION('pxf://<namenode>/<path>? PROFILE=GEMFIRE & LOCATORS=<gemfire-server:port> & REGION=<region-name>') FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import');
• 34.
  Geode Profile <profile> <name>GEMFIRE</name> <description>A profilefor reading Gemfire data</description> <plugins> <fragmenter>io.pivotal.pxf.plugins.gemfire.GemfireFragmenter</fragmenter> <accessor>io.pivotal.pxf.plugins.gemfire.GemfireAccessor</accessor> <resolver>io.pivotal.pxf.plugins.gemfire.GemfireResolver</resolver> </plugins> </profile>
• 35.
  Federated Queries With PXF/Geode- Demo
• 36.
  Stay Connected • PXFMaven Repository: https://bintray.com/big-data/maven/pxf/view • PXF Community Plugins: https://bintray.com/big-data/maven/pxf- plugins/view • Apache HAWQ: https://github.com/apache/incubator-hawq • Apache Geode: https://github.com/apache/incubator-geode • Apache Zeppelin: https://zeppelin.incubator.apache.org • Spring XD: http://projects.spring.io/spring-xd/