Introduction to Impala

Introduction to Impala

• 1.
  Impala: A Modern, Open-SourceSQL Engine for Hadoop   Mark  Grover   So+ware  Engineer,  Cloudera   May  22nd,  2014   Twi<er:  mark_grover     Slides  at     slideshare.net/markgrover/introducFon-­‐to-­‐impala    
• 2.
  •  What  is  Hadoop?   •  What  is  Impala?   •  Use-­‐cases  for  Impala   •  Architecture  of  Impala   •  Impala  comparisons  and  performance   •  Demo  (Fme  permiRng)   Agenda  
• 3.
  Intro  to  Hadoop  
• 4.
  What  is  Apache  Hadoop?   Has the Flexibility to Store and Mine Any Type of Data §  Ask questions across structured and unstructured data that were previously impossible to ask or solve §  Not bound by a single schema Excels at Processing Complex Data §  Scale-out architecture divides workloads across multiple nodes §  Flexible file system eliminates ETL bottlenecks Scales Economically §  Can be deployed on commodity hardware §  Open source platform guards against vendor lock Hadoop Distributed File System (HDFS) Self-Healing, High Bandwidth Clustered Storage MapReduce Distributed Computing Framework Apache Hadoop is an open source platform for data storage and processing that is… ü  Distributed ü  Fault tolerant ü  Scalable CORE HADOOP SYSTEM COMPONENTS
• 5.
  MapReduce  -­‐  the  good  and  the  bad   The  Good   • VersaFle   • Flexible   • Scalable   The  Bad   •  High  latency   •  Batch  oriented   •  Not  all  paradigms  fit  very   well   •  Only  for  developers  
• 6.
  •  MR  is  hard  and  only  for  developers   •  Higher  level  pla]orms  for  converFng  declaraFve   syntax  to  MapReduce   •  SQL  –  Hive   •  workflow  language  –  Pig   •  Build  on  top  of  MapReduce  (although  they  are  being   made  more  pluggable  now)   •  But  jobs  are  sFll  as  slow  as  MapReduce   What  are  Hive  and  Pig?  
• 7.
  Impala  
• 8.
  •  General-­‐purpose  SQL  engine   •  Real-­‐Fme  queries  in  Apache  Hadoop   •  Beta  version  released  since  October  2012   •  General  availability  (v1.0)  release  out  since  April  2013   •  Open  source  under  Apache  license   •  Latest  release  (v1.3.1)  released  on  May  1st,  2014   What  is  Impala?  
• 9.
  Impala  Overview:  Goals   •  General-­‐purpose  SQL  query  engine:   •  Works  for  both  for  analyFcal  and  transacFonal/single-­‐row   workloads   •  Supports  queries  that  take  from  milliseconds  to  hours   •  Runs  directly  within  Hadoop:   •  reads  widely  used  Hadoop  file  formats   •  talks  to  widely  used  Hadoop  storage  managers     •  runs  on  same  nodes  that  run  Hadoop  processes   •  High  performance:   •  C++  instead  of  Java   •  runFme  code  generaFon   •  completely  new  execuFon  engine  –  No  MapReduce  
• 10.
  User  View  of  Impala:  Overview   •  Runs  as  a  distributed  service  in  cluster:  one  Impala  daemon  on   each  node  with  data   •  Highly  available:  no  single  point  of  failure  
• 11.
  User  View  of  Impala:  Overview   •  There  is  no  ‘Impala  format’!   •  Supported  file  formats:   •  uncompressed/lzo-­‐compressed  text  files   •  sequence  files  and  RCFile  with  snappy/gzip  compression   •  Avro  data  files   •  Parquet  columnar  format  (more  on  that  later)   •  HBase  
• 12.
  User  View  of  Impala:  SQL   •  SQL  support:   •  essenFally  SQL-­‐92,  minus  correlated  subqueries   •  only  equi-­‐joins;  no  non-­‐equi  joins,  no  cross  products   •  Order  By  requires  Limit   •  (Limited)  DDL  support   •  SQL-­‐style  authorizaFon  via  Apache  Sentry  (incubaFng)   •  UDFs  and  UDAFs  are  supported  
• 13.
  User  View  of  Impala:  SQL   •  FuncFonal  limitaFons:   •  No  file  formats,  SerDes   •  no  beyond  SQL  (buckets,  samples,  transforms,  arrays,   structs,  maps,  xpath,  json)   •  Broadcast  joins  and  parFFoned  hash  joins  supported   •  Smaller  table  has  to  fit  in  aggregate  memory  of  all  execuFng   nodes  
• 14.
  Use  Cases  of  Impala  
• 15.
  Impala  Use  Cases   Interactive BI/analytics on more data Asking new questions – exploration, ML Data processing with tight SLAs Query-able archive w/full fidelity Cost-effective, ad hoc query environment that offloads/replaces the data warehouse for:
• 16.
  Global  Financial  Services  Company   Saved 90% on incremental EDW spend & improved performance by 5x Offload data warehouse for query-able archive Store decades of data cost-effectively Process & analyze on the same system Improved capabilities through interactive query on more data
• 17.
  Digital  Media  Company   20x performance improvement for exploration & data discovery Easily identify new data sets for modeling Interact with raw data directly to test hypotheses Avoid expensive DW schema changes Accelerate ‘time to answer’
• 18.
  Architecture  of  Impala  
• 19.
  Impala  Architecture   • Three  binaries:  impalad,  statestored,  catalogd   •  Impala  daemon  (impalad)  –  N  instances   •  handles  client  requests  and  all  internal  requests  related  to   query  execuFon   •  State  store  daemon  (statestored)  –  1  instance   •  Provides  name  service  and  metadata  distribuFon   •  Catalog  daemon  (catalogd)  –  1  instance   •  Relays  metadata  changes  to  all  impalad’s  
• 20.
  Impala  Architecture:  Query  ExecuFon   Request  arrives  via  odbc/jdbc   Query  Planner   Query  Executor   HDFS  DN   HBase   SQL  App   ODBC   Query  Planner   Query  Coordinator   Query  Executor   HDFS  DN   HBase   Query  Planner   Query  Executor   HDFS  DN   HBase   SQL   request   Query  Coordinator   Query  Coordinator   HiveMeta store   HDFS  NN   Statestore   +   Catalogd  
• 21.
  Impala  Architecture:  Query  ExecuFon   Planner  turns  request  into  collecFons  of  plan  fragments   Coordinator  iniFates  execuFon  on  remote  impalad's   Query  Planner   Query  Coordinator   Query  Executor   HDFS  DN   HBase   SQL  App   ODBC   Query  Planner   Query  Coordinator   Query  Executor   HDFS  DN   HBase   Query  Planner   Query  Coordinator   Query  Executor   HDFS  DN   HBase   HiveMeta store   HDFS  NN   Statestore   +   Catalogd  
• 22.
  Impala  Architecture:  Query  ExecuFon   Intermediate  results  are  streamed  between  impalad's  Query   results  are  streamed  back  to  client   Query  Planner   Query  Coordinator   Query  Executor   HDFS  DN   HBase   SQL  App   ODBC   Query  Planner   Query  Coordinator   Query  Executor   HDFS  DN   HBase   Query  Planner   Query  Coordinator   Query  Executor   HDFS  DN   HBase   query   results   HiveMeta store   HDFS  NN   Statestore   +   Catalogd  
• 23.
  Query  Planning:  Overview   •  2-­‐phase  planning  process:   •  single-­‐node  plan:  le+-­‐deep  tree  of  plan  operators   •  plan  parFFoning:  parFFon  single-­‐node  plan  to  maximize  scan  locality,   minimize  data  movement   •  ParallelizaFon  of  operators:   •  All  query  operators  are  fully  distributed  
• 24.
  Query Planning:  Single-­‐Node  Plan   •  Plan  operators:  Scan,  HashJoin,  HashAggregaFon,  Union,   TopN,  Exchange  
• 25.
  Single-­‐Node  Plan:  Example  Query   SELECT  t1.cusFd,                SUM(t2.revenue)  AS  revenue   FROM  LargeHdfsTable  t1   JOIN  LargeHdfsTable  t2  ON  (t1.id1  =  t2.id)   JOIN  SmallHbaseTable  t3  ON  (t1.id2  =  t3.id)   WHERE  t3.category  =  'Online'   GROUP  BY  t1.cusFd   ORDER  BY  revenue  DESC  LIMIT  10;  
• 26.
  Query Planning:  Single-­‐Node  Plan   HashJoin Scan: t1 Scan: t3 Scan: t2 HashJoin TopN Agg •  Single-­‐node  plan  for  example:    
• 27.
  Query  Planning:  Distributed  Plans   HashJoinScan: t1 Scan: t3 Scan: t2 HashJoin TopN Pre-Agg MergeAgg TopN Broadcast Broadcast hash t2.idhash t1.id1 hash t1.custid at HDFS DN at HBase RS at coordinator
• 28.
  Metadata  Handling   • Impala  metadata:   •  Hive’s  metastore:  logical  metadata  (table  definiFons,   columns,  CREATE  TABLE  parameters)   •  HDFS  Namenode:  directory  contents  and  block  replica   locaFons   •  HDFS  DataNode:  block  replicas’  volume  IDs  
• 29.
  Impala  ExecuFon  Engine   •  Wri<en  in  C++  for  minimal  execuFon  overhead   •  Internal  in-­‐memory  tuple  format  puts  fixed-­‐width   data  at  fixed  offsets   •  Uses  intrinsics/special  cpu  instrucFons  for  text   parsing,  crc32  computaFon,  etc.   •  RunFme  code  generaFon  for  “big  loops”  
• 30.
  Impala  ExecuFon  Engine   •  More  on  runFme  code  generaFon   •  example  of  "big  loop":  insert  batch  of  rows  into  hash  table   •  known  at  query  compile  Fme:  #  of  tuples  in  a  batch,  tuple   layout,  column  types,  etc.   •  generate  at  compile  Fme:  unrolled  loop  that  inlines  all   funcFon  calls,  contains  no  dead  code,  minimizes  branches   •  code  generated  using  llvm  
• 31.
  Comparing  Impala  to  Dremel   •  What  is  Dremel?   •  columnar  storage  for  data  with  nested  structures   •  distributed  scalable  aggregaFon  on  top  of  that   •  Columnar  storage  in  Hadoop:  Parquet   •  stores  data  in  appropriate  naFve/binary  types   •  can  also  store  nested  structures  similar  to  Dremel's  ColumnIO   •  Parquet  is  open  source:  github.com/parquet   •  Distributed  aggregaFon:  Impala   •  Impala  plus  Parquet:  a  superset  of  the  published  version  of   Dremel  (which  didn't  support  joins)  
• 32.
  32   Performance  
• 33.
  Impala  Performance  Results   • Impala’s  Latest  Milestone:   •  Comparable  commercial  MPP  DBMS  speed   •  NaFvely  on  Hadoop     • Three  Result  Sets:   •  Impala  vs  Hive  0.12  (Impala  6-­‐70x  faster)   •  Impala  vs  “DBMS-­‐Y”  (Impala  average  of  2x  faster)   •  Impala  scalability  (Impala  achieves  linear  scale)     • Background   •  20  pre-­‐selected,  diverse  TPC-­‐DS  queries  (modified  to  remove  unsupported   language)   •  Sufficient  data  scale  for  realisFc  comparison  (3  TB,  15  TB,  and  30  TB)   •  RealisFc  nodes  (e.g.  8-­‐core  CPU,  96GB  RAM,  12x2TB  disks)   •  Methodical  tesFng  (mulFple  runs,  reviewed  fairness  for  compeFFon,  etc)     •  Details:  h<p://blog.cloudera.com/blog/2014/01/impala-­‐performance-­‐dbms-­‐class-­‐speed/   33  
• 34.
  Impala  vs  Hive  0.12  (Lower  bars  are  be<er)   34  
• 35.
  Impala  vs  “DBMS-­‐Y”  (Lower  bars  are   be<er)   35  
• 36.
  Impala  Scalability:  2x  the  Hardware   (ExpectaFon:  Cut  Response  Times  in  Half)   36  
• 37.
  Impala  Scalability:  2x  the  Hardware  and  2x  Users/Data   (ExpectaFon:  Constant  Response  Times)   37   2x the Users, 2x the Hardware 2x the Data, 2x the Hardware
• 38.
  Demo   •  Uses  Cloudera’s  Quickstart  VM h<p://Fny.cloudera.com/quick-­‐start   •  Dataset/queries  from  h<ps://github.com/ markgrover/cloudcon-­‐hive  
• 39.
  I  am  co-­‐authoring  O’Reilly  book   Hadoop  ApplicaFon   Architectures   How  to  build  end-­‐to-­‐end  soluFons   using  Apache  Hadoop  and  related   tools   @hadooparchbook   www.hadooparchitecturebook.com  
• 40.
  Try  it  out!   •  Open  source!  Available  at  cloudera.com,  AWS  EMR!   •  Packages  for  many  different  Linux  flavours   •  QuesFons/comments?  community.cloudera.com   •  My  twi<er  handle:  mark_grover   •  Slides  at:  slideshare.net/markgrover/introducFon-­‐to-­‐ impala