使用datax通过HbaseShell封装writer和reader同步hbase数据到hbase_踩坑_细节总结---大数据之DataX工作笔记008

    最近在做大数据相关功能,有个需求,使用datax同步hbase到hbase中,其中还是有很多细节值得记录:

首先来看一下datax的源码中,如果你使用phoenix创建的表,那么

你就需要使用对应的hbase带有sql字样的,reader和writer.

然后如果你使用datax-web来进行测试的,那么,他默认使用的是hbase11xreader和对应的writer.

这个时候,是可以读出,使用hbase shell去创建的表的,同时也可以读出phoneix创建的表.

关于适用phoenix去操作hbase常用sql:

1.创建表:
CREATE TABLE IF NOT EXISTS "qqqq_tables" ("id" varchar primary key, "source_id" varchar, "table_name" varchar, "table_desc" varchar, "remark" varchar, "create_time" varchar) COLUMN_ENCODED_BYTES=0;
 
 
CREATE TABLE IF NOT EXISTS "qqqq_tables" ("id" varchar primary key, "source_id" varchar, "table_name" varchar, "table_desc" varchar, "remark" varchar, "create_time" varchar) COLUMN_ENCODED_BYTES=0;
 
 
2.显示所有表:
show tables
 
 
3.查询表:
select * from "qdbd_table_meta"
 
 
4.命令行连接
./sqlline.py 172.19.126.120,172.19.126.121,172.19.126.122:2181
 
 
5.删除表:
drop table "userinfo";
 
 
6.插入数据:
0: jdbc:phoenix:172.19.126.120,172.19.126.121> upsert into "userinfo" values('4','xiaohuang','[email protected]','13256716111','{name:xiaohuang}');
1 row affected (0.14 seconds)
 
 
7.查询数据添加条件:对应字段要添加双引号
0: jdbc:phoenix:172.19.126.120,172.19.126.121> select * from "userinfo" where "id" = '3';
+----+--------------+---------------------+-------------+-----------------+
| id |     name     |        email        |   mobile    |    son_json     |
+----+--------------+---------------------+-------------+-----------------+
| 3  | xiaoxingxing | [email protected] | 13256716188 | {name:xingxing} |
+----+--------------+---------------------+-------------+-----------------+
 
 
8.使用upsert更新表的某个字段
0: jdbc:phoenix:172.19.126.120,172.19.126.121> upsert into "userinfo" ("id","name") values ('4','xiaohuangzine');
1 row affected (0.017 seconds)
0: jdbc:phoenix:172.19.126.120,172.19.126.121> select * from "userinfo";
+----+---------------+---------------------+-------------+------------------+
| id |     name      |        email        |   mobile    |     son_json     |
+----+---------------+---------------------+-------------+------------------+
| 3  | xiaoxingxing  | [email protected] | 13256716188 | {name:xingxing}  |
| 4  | xiaohuangzine | [email protected]   | 13256716111 | {name:xiaohuang} |
+----+---------------+---------------------+-------------+------------------+
2 rows selected (0.035 seconds)
 
 
9.update更新某个字段无法使用:
 
0: jdbc:phoenix:172.19.126.120,172.19.126.121> update "userinfo" set "name" = 'xiaohuangzi' where "id"='4';
Error: ERROR 602 (42P00): Syntax error. Missing "STATISTICS" at line 1, column 8. (state=42P00,code=602)
org.apache.phoenix.exception.PhoenixParserException: ERROR 602 (42P00): Syntax error. Missing "STATISTICS" at line 1, column 8.
        at org.apache.phoenix.exception.PhoenixParserException.newException(PhoenixParserException.java:33)
        at org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:111)
        at org.apache.phoenix.jdbc.PhoenixStatement$PhoenixStatementParser.parseStatement(PhoenixStatement.java:1778)
        at org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1861)
        at org.apache.phoenix.jdbc.Ph