Flink-SQL 读取hbase 问题总结

原创已于 2024-03-28 09:30:01 修改 · 1.1k 阅读

3 ·

CC 4.0 BY-SA版权

文章标签：

#hbase #flink #sql

于 2023-04-26 19:20:54 首次发布

Flink SQL 专栏收录该内容

4 篇文章

订阅专栏

本文介绍了如何使用FlinkSQL来读取和写入HBase的数据，特别是在处理维度数据时，利用HBase的特性提高并发数。配置中提到了hbase-2.2连接器，以及Zookeeper的设置。同时展示了从Kafka流表中提取数据并插入到HBase的SQL语句。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

flink-sql 读取hbase 维度数据配置

若维度数据插入后几乎不发生更新,或不发生更新,可以用hbase存储大数据量维度数据
可以同步异步读取hbase数据,提高flink与外部hbase 交互的并发数

flink-sql连接hbase 配置

hbase相关配置


CREATE TABLE dim_hljy_ord_stu_info (
      rowkey STRING, 
      cf1 ROW<id BIGINT ,order_type STRING ,order_no STRING  ,create_time timestamp(1) ,update_time timestamp(1) ,is_deleted tinyint>,
      PRIMARY KEY (rowkey) NOT ENFORCED
 ) WITH (
        'connector' = 'hbase-2.2',
      'table-name' = 'dim:dim_stu_info',
      'zookeeper.quorum' = 'localhost:2181',
      'sink.buffer-flush.max-size'='10mb',
      'sink.buffer-flush.max-rows' = '2000',
      'sink.buffer-flush.interval' = '1s',
      'sink.async' = 'true' 
 );

– flink-sql 读取hbase列簇(数据列)

  SELECT  cf1.order_type from dim_hljy_ord_stu_info limit 10 ;

flink-sql 将kafka 数据写入hbase

kafka 流表


CREATE TABLE ods_ord_stu_info` (
  `id` BIGINT COMMENT '主键',
  `order_type` STRING  COMMENT '订单类型1-报名2-择升 3-升级', 
  `order_no` STRING  COMMENT '订单号', 
  `create_time` timestamp(1) COMMENT '创建时间' ,
  `update_time` timestamp(1) COMMENT '更新时间',
  `is_deleted` tinyint COMMENT '0:未删除,1:删除',
  PRIMARY KEY (`id`) NOT ENFORCED
) WITH (
   'connector' = 'upsert-kafka',
  'topic' = 'topic_ods_ord_stu_info',
  'properties.bootstrap.servers' = 'localhost:9092',
  'key.format' = 'json',
  'value.format' = 'json',
  'key.json.ignore-parse-errors' = 'true',
  'value.json.fail-on-missing-field' = 'false'
);

– 定义在flink hbase流表


CREATE TABLE dim_hljy_ord_stu_info (
      rowkey STRING, 
      cf1 ROW<id BIGINT ,order_type STRING ,order_no STRING  ,create_time timestamp(1) ,update_time timestamp(1) ,is_deleted tinyint>,
      PRIMARY KEY (rowkey) NOT ENFORCED
 ) WITH (
        'connector' = 'hbase-2.2',
      'table-name' = 'hljy:dim_hljy_ord_stu_info',
      'zookeeper.quorum' = '10.155.0.40:2181',
      'sink.buffer-flush.max-size'='10mb',
      'sink.buffer-flush.max-rows' = '2000',
      'sink.buffer-flush.interval' = '1s',
      'sink.async' = 'true' 
 );

– flink-sql 将kafka 流表写入hbase


insert into  dim_hljy_ord_stu_info select concat_ws('_',UUID(),`sno`,`enroll_code`),row(`id`,`sno`,`enroll_code ,`create_time`,`update_time`,`pay_route`)
from ods_ord_stu_info ;