Mysql SQOOP HBASE :

-archives <comma separated list of archives> specify comma separated archives to be unarchived
on the compute machines.

The general command line syntax is

bin/hadoop command [genericOptions] [commandOptions]

At minimum, you must specify --connect and --table

Arguments to mysqldump and other subprograms may be supplied

after a '--' on the command line.

usage: sqoop import [GENERIC-ARGS] [TOOL-ARGS]

Common arguments:

--connect <jdbc-uri> Specify JDBC connect

string

--connection-manager <class-name> Specify connection manager

class name

--connection-param-file <properties-file> Specify connection

parameters file

--driver <class-name> Manually specify JDBC

driver class to use

--hadoop-home <hdir> Override

$HADOOP_MAPRED_HOME_ARG

--hadoop-mapred-home <dir> Override

$HADOOP_MAPRED_HOME_ARG

--help Print usage instructions

-P Read password from console

--password <password> Set authentication

password
 --password-file <password-file> Set authentication

password file path

--username <username> Set authentication

username

--verbose Print more information

while working

Import control arguments:

--append Imports data

in append

mode

--as-avrodatafile Imports data

to Avro data

files

--as-sequencefile Imports data

to

SequenceFile

--as-textfile Imports data

as plain

text

(default)

--boundary-query <statement> Set boundary

query for

retrieving

max and min

value of the

primary key

--columns <col,col,col...> Columns to

import from

table
 --compression-codec <codec> Compression

codec to use

for import

--delete-target-dir Imports data

in delete

mode

--direct Use direct

import fast

path

--direct-split-size <n> Split the

input stream

every 'n'

bytes when

importing in

direct mode

-e,--query <statement> Import

results of

SQL

'statement'

--fetch-size <n> Set number

'n' of rows

to fetch

from the

database

when more

rows are

needed

--inline-lob-limit <n> Set the

maximum size

for an

inline LOB
-m,--num-mappers <n> Use 'n' map

tasks to

import in

parallel

--mapreduce-job-name <name> Set name for

generated

mapreduce

job

--split-by <column-name> Column of

the table

used to

split work

units

--table <table-name> Table to

read

--target-dir <dir> HDFS plain

table

destination

--validate Validate the

copy using

the

configured

validator

--validation-failurehandler <validation-failurehandler> Fully

qualified

class name

for

ValidationFa

ilureHandler

--validation-threshold <validation-threshold> Fully

qualified
 class name

for

ValidationTh

reshold

--validator <validator> Fully

qualified

class name

for the

Validator

--warehouse-dir <dir> HDFS parent

for table

destination

--where <where clause> WHERE clause

to use

during

import

-z,--compress Enable

compression

Incremental import arguments:

--check-column <column> Source column to check for incremental

change

--incremental <import-type> Define an incremental import of type

'append' or 'lastmodified'

--last-value <value> Last imported value in the incremental

check column

Output line formatting arguments:

--enclosed-by <char> Sets a required field enclosing

character

--escaped-by <char> Sets the escape character

 --fields-terminated-by <char> Sets the field separator character

--lines-terminated-by <char> Sets the end-of-line character

--mysql-delimiters Uses MySQL's default delimiter set:

fields: , lines: \n escaped-by: \

optionally-enclosed-by: '

--optionally-enclosed-by <char> Sets a field enclosing character

Input parsing arguments:

--input-enclosed-by <char> Sets a required field encloser

--input-escaped-by <char> Sets the input escape

character

--input-fields-terminated-by <char> Sets the input field separator

--input-lines-terminated-by <char> Sets the input end-of-line

char

--input-optionally-enclosed-by <char> Sets a field enclosing

character

Hive arguments:

--create-hive-table Fail if the target hive

table exists

--hive-database <database-name> Sets the database name to

use when importing to hive

--hive-delims-replacement <arg> Replace Hive record \0x01

and row delimiters (\n\r)

from imported string fields

with user-defined string

--hive-drop-import-delims Drop Hive record \0x01 and

row delimiters (\n\r) from

imported string fields

--hive-home <dir> Override $HIVE_HOME

--hive-import Import tables into Hive

 (Uses Hive's default

delimiters if none are

set.)

--hive-overwrite Overwrite existing data in

the Hive table

--hive-partition-key <partition-key> Sets the partition key to

use when importing to hive

--hive-partition-value <partition-value> Sets the partition value to

use when importing to hive

--hive-table <table-name> Sets the table name to use

when importing to hive

--map-column-hive <arg> Override mapping for

specific column to hive

types.

HBase arguments:

--column-family <family> Sets the target column family for the

import

--hbase-create-table If specified, create missing HBase tables

--hbase-row-key <col> Specifies which input column to use as the

row key

--hbase-table <table> Import to <table> in HBase

HCatalog arguments:

--hcatalog-database <arg> HCatalog database name

--hcatalog-home <hdir> Override $HCAT_HOME

--hcatalog-table <arg> HCatalog table name

--hive-home <dir> Override $HIVE_HOME

--hive-partition-key <partition-key> Sets the partition key to

use when importing to hive

--hive-partition-value <partition-value> Sets the partition value to

 use when importing to hive

--map-column-hive <arg> Override mapping for

specific column to hive

types.

HCatalog import specific options:

--create-hcatalog-table Create HCatalog before import

--hcatalog-storage-stanza <arg> HCatalog storage stanza for table

creation

Code generation arguments:

--bindir <dir> Output directory for compiled

objects

--class-name <name> Sets the generated class name.

This overrides --package-name.

When combined with --jar-file,

sets the input class.

--input-null-non-string <null-str> Input null non-string

representation

--input-null-string <null-str> Input null string representation

--jar-file <file> Disable code generation; use

specified jar

--map-column-java <arg> Override mapping for specific

columns to java types

--null-non-string <null-str> Null non-string representation

--null-string <null-str> Null string representation

--outdir <dir> Output directory for generated

code

--package-name <name> Put auto-generated classes in

this package
Generic Hadoop command-line arguments:

(must preceed any tool-specific arguments)

Generic options supported are

-conf <configuration file> specify an application configuration file

-D <property=value> use value for given property

-fs <local|namenode:port> specify a namenode

-jt <local|jobtracker:port> specify a job tracker

-files <comma separated list of files> specify comma separated files to be copied to the map
reduce cluster

-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.

-archives <comma separated list of archives> specify comma separated archives to be unarchived
on the compute machines.

The general command line syntax is

bin/hadoop command [genericOptions] [commandOptions]

At minimum, you must specify --connect and --table

Arguments to mysqldump and other subprograms may be supplied

after a '--' on the command line.

hduser@ubuntu:~$

hduser@ubuntu:~$

hduser@ubuntu:~$

hduser@ubuntu:~$

hduser@ubuntu:~$

hduser@ubuntu:~$

hduser@ubuntu:~$ mysql -u root -p

Enter password:

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 39

Server version: 5.5.62-0ubuntu0.14.04.1 (Ubuntu)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;

+--------------------+

| Database |

+--------------------+

| information_schema |

| Hitachi |

| demo |

| hadooptest |

| mysql |

| performance_schema |

| training |

+--------------------+

7 rows in set (0.00 sec)

mysql> use training;

Reading table information for completion of table and column names

You can turn off this feature to get a quicker startup with -A

Database changed

mysql> show tables;

+--------------------+

| Tables_in_training |

+--------------------+
| CUSTOMER |

| emp |

| student |

| transactions |

+--------------------+

4 rows in set (0.00 sec)

mysql> select * from CUSTOMER;

+-----------+-----------+----------+

| contactid | firstname | lastname |

+-----------+-----------+----------+

| HOO1 | Bigdata | Doug |

| SOO1 | Spark | Bricks |

| VOO1 | Hbase | Vivek |

| BOO1 | Azure | Balu |

+-----------+-----------+----------+

4 rows in set (0.00 sec)

mysql> exit;

Bye

hduser@ubuntu:~$

hduser@ubuntu:~$

hduser@ubuntu:~$

hduser@ubuntu:~$ sqoop import \

> --connect jdbc:mysql://localhost/training \

> --username root -P \

> --table CUSTOMER \

> --hbase-table customer_hbase \

> --column-family contactid \

> --hbase-row-key contactid

Warning: /usr/lib/hcatalog does not exist! HCatalog jobs will fail.

Please set $HCAT_HOME to the root of your HCatalog installation.

Warning: $HADOOP_HOME is deprecated.

Enter password:

21/01/23 19:37:43 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.

21/01/23 19:37:43 INFO tool.CodeGenTool: Beginning code generation

21/01/23 19:37:44 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM
`CUSTOMER` AS t LIMIT 1

21/01/23 19:37:44 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM
`CUSTOMER` AS t LIMIT 1

21/01/23 19:37:44 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/local/hadoop

Note: /tmp/sqoop-hduser/compile/f28f2d11f8246bc5633bc74f33d95614/CUSTOMER.java uses or

overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.

21/01/23 19:37:49 INFO orm.CompilationManager: Writing jar file:

/tmp/sqoop-hduser/compile/f28f2d11f8246bc5633bc74f33d95614/CUSTOMER.jar

21/01/23 19:37:49 WARN manager.MySQLManager: It looks like you are importing from mysql.

21/01/23 19:37:49 WARN manager.MySQLManager: This transfer can be faster! Use the --direct

21/01/23 19:37:49 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.

21/01/23 19:37:49 INFO manager.MySQLManager: Setting zero DATETIME behavior to

convertToNull (mysql)

21/01/23 19:37:49 ERROR tool.ImportTool: Error during import: No primary key could be found for
table CUSTOMER. Please specify one with --split-by or perform a sequential import with '-m 1'.

hduser@ubuntu:~$ sqoop import --connect jdbc:mysql://localhost/training --username root -P --

table CUSTOMER --hbase-table customer_hbase --column-family contactid --hbase-row-key
contactid -m 1

Warning: /usr/lib/hcatalog does not exist! HCatalog jobs will fail.

Please set $HCAT_HOME to the root of your HCatalog installation.

Warning: $HADOOP_HOME is deprecated.

Enter password:

21/01/23 19:38:03 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.

21/01/23 19:38:03 INFO tool.CodeGenTool: Beginning code generation

21/01/23 19:38:04 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM
`CUSTOMER` AS t LIMIT 1

21/01/23 19:38:04 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM
`CUSTOMER` AS t LIMIT 1

21/01/23 19:38:04 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/local/hadoop

Note: /tmp/sqoop-hduser/compile/55d9047fc632a6be018c10e68191d125/CUSTOMER.java uses or

overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.

21/01/23 19:38:07 INFO orm.CompilationManager: Writing jar file:

/tmp/sqoop-hduser/compile/55d9047fc632a6be018c10e68191d125/CUSTOMER.jar

21/01/23 19:38:07 WARN manager.MySQLManager: It looks like you are importing from mysql.

21/01/23 19:38:07 WARN manager.MySQLManager: This transfer can be faster! Use the --direct

21/01/23 19:38:07 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.

21/01/23 19:38:07 INFO manager.MySQLManager: Setting zero DATETIME behavior to

convertToNull (mysql)

21/01/23 19:38:07 INFO mapreduce.ImportJobBase: Beginning import of CUSTOMER

21/01/23 19:38:09 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-

0x4a96405e connecting to ZooKeeper ensemble=localhost:2181

21/01/23 19:38:10 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-

1569965, built on 02/20/2014 09:09 GMT

21/01/23 19:38:10 INFO zookeeper.ZooKeeper: Client environment:host.name=ubuntu

21/01/23 19:38:10 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_201

21/01/23 19:38:10 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation

21/01/23 19:38:10 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-7-

openjdk-amd64/jre

21/01/23 19:38:10 INFO zookeeper.ZooKeeper: Client

environment:java.class.path=/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/java-7-openjdk-amd64/
lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.2.1.jar:/usr/
local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.11.jar:/usr/
local/hadoop/libexec/../lib/aspectjtools-1.6.11.jar:/usr/local/hadoop/libexec/../lib/commons-
beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/
hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-
1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../
lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/
usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/
commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/
hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/
usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/
commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/
hadoop/libexec/../lib/commons-net-3.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/
local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.2.1.jar:/usr/local/hadoop/libexec/../lib/
hadoop-fairscheduler-1.2.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.2.1.jar:/usr/local/
hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-
1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../
lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/
hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/
hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/
local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/
local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/
local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/
hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/
local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-
20081211.jar:/usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar:/usr/local/hadoop/libexec/../lib/
slf4j-log4j12-1.4.3.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../
lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/sqoop/
conf::/usr/local/sqoop/lib/ant-contrib-1.0b3.jar:/usr/local/sqoop/lib/ant-eclipse-1.0-jvm1.2.jar:/
usr/local/sqoop/lib/avro-1.5.3.jar:/usr/local/sqoop/lib/avro-ipc-1.5.3.jar:/usr/local/sqoop/lib/avro-
mapred-1.5.3.jar:/usr/local/sqoop/lib/commons-io-1.4.jar:/usr/local/sqoop/lib/hsqldb-
1.8.0.10.jar:/usr/local/sqoop/lib/jackson-core-asl-1.7.3.jar:/usr/local/sqoop/lib/jackson-mapper-asl-
1.7.3.jar:/usr/local/sqoop/lib/jopt-simple-3.2.jar:/usr/local/sqoop/lib/mysql-connector-java-5.1.40-
bin.jar:/usr/local/sqoop/lib/paranamer-2.3.jar:/usr/local/sqoop/lib/snappy-java-1.0.3.2.jar:/usr/
local/hbase/conf:/usr/lib/jvm/java-7-openjdk-amd64/lib/tools.jar:/usr/local/hbase:/usr/local/
hbase/lib/activation-1.1.jar:/usr/local/hbase/lib/asm-3.1.jar:/usr/local/hbase/lib/commons-
beanutils-1.7.0.jar:/usr/local/hbase/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase/lib/
commons-cli-1.2.jar:/usr/local/hbase/lib/commons-codec-1.7.jar:/usr/local/hbase/lib/commons-
collections-3.2.2.jar:/usr/local/hbase/lib/commons-configuration-1.6.jar:/usr/local/hbase/lib/
commons-digester-1.8.jar:/usr/local/hbase/lib/commons-el-1.0.jar:/usr/local/hbase/lib/commons-
httpclient-3.1.jar:/usr/local/hbase/lib/commons-io-2.4.jar:/usr/local/hbase/lib/commons-lang-
2.6.jar:/usr/local/hbase/lib/commons-logging-1.1.1.jar:/usr/local/hbase/lib/commons-math-
2.1.jar:/usr/local/hbase/lib/commons-net-1.4.1.jar:/usr/local/hbase/lib/findbugs-annotations-1.3.9-
1.jar:/usr/local/hbase/lib/guava-12.0.1.jar:/usr/local/hbase/lib/hadoop-core-1.2.1.jar:/usr/local/
hbase/lib/hamcrest-core-1.3.jar:/usr/local/hbase/lib/hbase-annotations-0.98.19-hadoop1.jar:/usr/
local/hbase/lib/hbase-checkstyle-0.98.19-hadoop1.jar:/usr/local/hbase/lib/hbase-client-0.98.19-
hadoop1.jar:/usr/local/hbase/lib/hbase-common-0.98.19-hadoop1.jar:/usr/local/hbase/lib/hbase-
common-0.98.19-hadoop1-tests.jar:/usr/local/hbase/lib/hbase-examples-0.98.19-hadoop1.jar:/usr/
local/hbase/lib/hbase-hadoop1-compat-0.98.19-hadoop1.jar:/usr/local/hbase/lib/hbase-hadoop-
compat-0.98.19-hadoop1.jar:/usr/local/hbase/lib/hbase-it-0.98.19-hadoop1.jar:/usr/local/hbase/
lib/hbase-it-0.98.19-hadoop1-tests.jar:/usr/local/hbase/lib/hbase-prefix-tree-0.98.19-
hadoop1.jar:/usr/local/hbase/lib/hbase-protocol-0.98.19-hadoop1.jar:/usr/local/hbase/lib/hbase-
resource-bundle-0.98.19-hadoop1.jar:/usr/local/hbase/lib/hbase-rest-0.98.19-hadoop1.jar:/usr/
local/hbase/lib/hbase-server-0.98.19-hadoop1.jar:/usr/local/hbase/lib/hbase-server-0.98.19-
hadoop1-tests.jar:/usr/local/hbase/lib/hbase-shell-0.98.19-hadoop1.jar:/usr/local/hbase/lib/hbase-
testing-util-0.98.19-hadoop1.jar:/usr/local/hbase/lib/hbase-thrift-0.98.19-hadoop1.jar:/usr/local/
hbase/lib/high-scale-lib-1.1.1.jar:/usr/local/hbase/lib/htrace-core-2.04.jar:/usr/local/hbase/lib/
httpclient-4.1.3.jar:/usr/local/hbase/lib/httpcore-4.1.3.jar:/usr/local/hbase/lib/jackson-core-asl-
1.8.8.jar:/usr/local/hbase/lib/jackson-jaxrs-1.8.8.jar:/usr/local/hbase/lib/jackson-mapper-asl-
1.8.8.jar:/usr/local/hbase/lib/jackson-xc-1.8.8.jar:/usr/local/hbase/lib/jamon-runtime-2.4.1.jar:/usr/
local/hbase/lib/jasper-compiler-5.5.23.jar:/usr/local/hbase/lib/jasper-runtime-5.5.23.jar:/usr/
local/hbase/lib/jaxb-api-2.2.2.jar:/usr/local/hbase/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase/lib/
jcodings-1.0.8.jar:/usr/local/hbase/lib/jersey-client-1.8.jar:/usr/local/hbase/lib/jersey-core-1.8.jar:/
usr/local/hbase/lib/jersey-json-1.8.jar:/usr/local/hbase/lib/jersey-server-1.8.jar:/usr/local/hbase/
lib/jettison-1.3.1.jar:/usr/local/hbase/lib/jetty-6.1.26.jar:/usr/local/hbase/lib/jetty-sslengine-
6.1.26.jar:/usr/local/hbase/lib/jetty-util-6.1.26.jar:/usr/local/hbase/lib/joni-2.1.2.jar:/usr/local/
hbase/lib/jruby-complete-1.6.8.jar:/usr/local/hbase/lib/jsp-2.1-6.1.14.jar:/usr/local/hbase/lib/jsp-
api-2.1-6.1.14.jar:/usr/local/hbase/lib/jsr305-1.3.9.jar:/usr/local/hbase/lib/junit-4.11.jar:/usr/local/
hbase/lib/libthrift-0.9.0.jar:/usr/local/hbase/lib/log4j-1.2.17.jar:/usr/local/hbase/lib/metrics-core-
2.2.0.jar:/usr/local/hbase/lib/netty-3.6.6.Final.jar:/usr/local/hbase/lib/protobuf-java-2.5.0.jar:/usr/
local/hbase/lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase/lib/slf4j-api-1.6.4.jar:/usr/local/hbase/
lib/slf4j-log4j12-1.6.4.jar:/usr/local/hbase/lib/xmlenc-0.52.jar:/usr/local/hbase/lib/zookeeper-
3.4.6.jar:/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/java-7-openjdk-amd64/lib/tools.jar:/usr/
local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.2.1.jar:/usr/local/hadoop/
libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.11.jar:/usr/local/hadoop/
libexec/../lib/aspectjtools-1.6.11.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/
usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/
commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/
libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-
configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/
hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-
1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../
lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/
libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-
1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/
commons-net-3.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../
lib/hadoop-capacity-scheduler-1.2.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-
1.2.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.2.1.jar:/usr/local/hadoop/libexec/../lib/
hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/
libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-
5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../
lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/
jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/
libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/
libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/
libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../
lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/
libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/
hadoop/libexec/../lib/slf4j-api-1.4.3.jar:/usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar:/usr/
local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/
local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/sqoop/sqoop-1.4.4.jar:/usr/local/
sqoop/sqoop-test-1.4.4.jar:

21/01/23 19:38:10 INFO zookeeper.ZooKeeper: Client

environment:java.library.path=/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64

21/01/23 19:38:10 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp

21/01/23 19:38:10 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>

21/01/23 19:38:10 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux

21/01/23 19:38:10 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64

21/01/23 19:38:10 INFO zookeeper.ZooKeeper: Client environment:os.version=4.4.0-31-generic

21/01/23 19:38:10 INFO zookeeper.ZooKeeper: Client environment:user.name=hduser

21/01/23 19:38:10 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hduser

21/01/23 19:38:10 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hduser

21/01/23 19:38:10 INFO zookeeper.ZooKeeper: Initiating client connection,

connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x4a96405e0x0,
quorum=localhost:2181, baseZNode=/hbase

21/01/23 19:38:10 INFO zookeeper.ClientCnxn: Opening socket connection to server

localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

21/01/23 19:38:10 INFO zookeeper.ClientCnxn: Socket connection established to

localhost/127.0.0.1:2181, initiating session

21/01/23 19:38:10 INFO zookeeper.ClientCnxn: Session establishment complete on server

localhost/127.0.0.1:2181, sessionid = 0x177325f3cee000f, negotiated timeout = 90000

21/01/23 19:38:12 INFO zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-

hconnection-0x4a96405e connecting to ZooKeeper ensemble=localhost:2181

21/01/23 19:38:12 INFO zookeeper.ZooKeeper: Initiating client connection,

connectString=localhost:2181 sessionTimeout=90000 watcher=catalogtracker-on-hconnection-
0x4a96405e0x0, quorum=localhost:2181, baseZNode=/hbase

21/01/23 19:38:12 INFO zookeeper.ClientCnxn: Opening socket connection to server

localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

21/01/23 19:38:12 INFO zookeeper.ClientCnxn: Socket connection established to

localhost/127.0.0.1:2181, initiating session

21/01/23 19:38:12 INFO zookeeper.ClientCnxn: Session establishment complete on server

localhost/127.0.0.1:2181, sessionid = 0x177325f3cee0010, negotiated timeout = 90000

21/01/23 19:38:12 INFO zookeeper.ClientCnxn: EventThread shut down

21/01/23 19:38:12 INFO zookeeper.ZooKeeper: Session: 0x177325f3cee0010 closed

21/01/23 19:38:12 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree module jar

containing PrefixTreeCodec is not present. Continuing without it.

21/01/23 19:38:17 INFO mapred.JobClient: Running job: job_202101231910_0001

21/01/23 19:38:18 INFO mapred.JobClient: map 0% reduce 0%

21/01/23 19:39:00 INFO mapred.JobClient: map 100% reduce 0%

21/01/23 19:39:05 INFO mapred.JobClient: Job complete: job_202101231910_0001

21/01/23 19:39:05 INFO mapred.JobClient: Counters: 17

21/01/23 19:39:05 INFO mapred.JobClient: Job Counters

21/01/23 19:39:05 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=32565

21/01/23 19:39:05 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving
slots (ms)=0

21/01/23 19:39:05 INFO mapred.JobClient: Total time spent by all maps waiting after reserving
slots (ms)=0

21/01/23 19:39:05 INFO mapred.JobClient: Launched map tasks=1

21/01/23 19:39:05 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0

21/01/23 19:39:05 INFO mapred.JobClient: File Output Format Counters

21/01/23 19:39:05 INFO mapred.JobClient: Bytes Written=0

21/01/23 19:39:05 INFO mapred.JobClient: FileSystemCounters

21/01/23 19:39:05 INFO mapred.JobClient: HDFS_BYTES_READ=87

21/01/23 19:39:05 INFO mapred.JobClient: FILE_BYTES_WRITTEN=102020

21/01/23 19:39:05 INFO mapred.JobClient: File Input Format Counters

21/01/23 19:39:05 INFO mapred.JobClient: Bytes Read=0

21/01/23 19:39:05 INFO mapred.JobClient: Map-Reduce Framework

21/01/23 19:39:05 INFO mapred.JobClient: Map input records=4

21/01/23 19:39:05 INFO mapred.JobClient: Physical memory (bytes) snapshot=99610624

21/01/23 19:39:05 INFO mapred.JobClient: Spilled Records=0

21/01/23 19:39:05 INFO mapred.JobClient: CPU time spent (ms)=3700

21/01/23 19:39:05 INFO mapred.JobClient: Total committed heap usage (bytes)=31457280

21/01/23 19:39:05 INFO mapred.JobClient: Virtual memory (bytes) snapshot=755724288

21/01/23 19:39:05 INFO mapred.JobClient: Map output records=4

21/01/23 19:39:05 INFO mapred.JobClient: SPLIT_RAW_BYTES=87

21/01/23 19:39:05 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 52.795 seconds (0

bytes/sec)

21/01/23 19:39:05 INFO mapreduce.ImportJobBase: Retrieved 4 records.

hduser@ubuntu:~$ ^C

hduser@ubuntu:~$

=====================================================================

bvm@ubuntu:~$ su - hduser
Password:

hduser@ubuntu:~$ jps

2738 Jps

hduser@ubuntu:~$ cd /usr/local/

hduser@ubuntu:/usr/local$ ls -ltr

total 1625088

-rw-r--r-- 1 root root 60965956 Jun 30 2013 apache-flume-1.4.0-bin.tar.gz

drwxr-xr-x 9 hduser hadoop 4096 Jul 22 2013 sqoop

-rw-r--r-- 1 root root 5266542 Jul 30 2013 sqoop-1.4.4.bin__hadoop-1.0.0.tar.gz

-rw-r--r-- 1 root root 63851630 Nov 6 2014 hadoop-1.2.1.tar.gz

-rw-r--r-- 1 root root 52550402 May 20 2015 apache-flume-1.6.0-bin.tar.gz

-rw-r--r-- 1 root root 102487389 Jun 23 2015 hbase-1.1.1-bin.tar.gz

-rw-r--r-- 1 root root 92834839 Jun 26 2015 apache-hive-1.2.1-bin.tar.gz

drwxr-xr-x 12 hduser hadoop 4096 Dec 21 2015 spark

-rw-r--r-- 1 root root 243448482 Dec 28 2015 spark-1.6.0-bin-hadoop1.tgz

-rw-r--r-- 1 root root 75865980 Apr 22 2016 hbase-0.98.19-hadoop1-bin.tar.gz

drwxr-xr-x 16 hduser hadoop 4096 Jun 1 2016 pig

-rw-r--r-- 1 root root 177279333 Jun 7 2016 pig-0.16.0.tar.gz

drwxr-xr-x 2 root root 4096 Aug 3 2016 src

drwxr-xr-x 2 root root 4096 Aug 3 2016 sbin

drwxr-xr-x 2 root root 4096 Aug 3 2016 include

drwxr-xr-x 2 root root 4096 Aug 3 2016 games

drwxr-xr-x 2 root root 4096 Aug 3 2016 etc

drwxr-xr-x 2 root root 4096 Aug 3 2016 bin

drwxr-xr-x 4 root root 4096 Aug 3 2016 lib

drwxr-xr-x 7 root root 4096 Aug 3 2016 share

-rw-r--r-- 1 root root 3911557 Sep 24 2016 mysql-connector-java-5.1.40.tar.gz

drwxr-xr-x 4 root root 4096 Sep 24 2016 mysql-connector-java-5.1.40

-rwxrwxrwx 1 root root 419038579 Jan 30 2017 Anaconda2-4.1.1-Linux-x86_64.sh

-rw-r--r-- 1 root root 366447449 Nov 19 2018 hadoop-2.9.2.tar.gz

lrwxrwxrwx 1 root root 9 Dec 19 2019 man -> share/man

drwxr-xr-x 16 hduser hadoop 4096 Dec 27 2019 hadoop

drwxr-xr-x 9 hduser hadoop 4096 Apr 19 2020 hbase

drwxr-xr-x 7 hduser hadoop 4096 May 19 2020 flume

-rw-r--r-- 1 root root 29793 May 21 2020 spark-2.4.5-bin-hadoop2.7.tgz

drwxr-xr-x 8 hduser hadoop 4096 Jul 1 2020 hive

hduser@ubuntu:/usr/local$

hduser@ubuntu:/usr/local$

hduser@ubuntu:/usr/local$

hduser@ubuntu:/usr/local$

hduser@ubuntu:/usr/local$ start-all.sh

starting namenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hduser-namenode-

ubuntu.out

localhost: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hduser-datanode-

ubuntu.out

localhost: starting secondarynamenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-

hduser-secondarynamenode-ubuntu.out

starting jobtracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hduser-jobtracker-

ubuntu.out

localhost: starting tasktracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hduser-

tasktracker-ubuntu.out

hduser@ubuntu:/usr/local$ jps

2991 DataNode

3221 JobTracker

2849 NameNode

3416 Jps

3140 SecondaryNameNode

3363 TaskTracker

hduser@ubuntu:/usr/local$

hduser@ubuntu:/usr/local$

hduser@ubuntu:/usr/local$

hduser@ubuntu:/usr/local$

hduser@ubuntu:/usr/local$

hduser@ubuntu:/usr/local$ hbase
Usage: hbase [<options>] <command> [<args>]

Options:

--config DIR Configuration direction to use. Default: ./conf

--hosts HOSTS Override the list in 'regionservers' file

--auth-as-server Authenticate to ZooKeeper using servers configuration

Commands:

Some commands take arguments. Pass no args or -h for usage.

shell Run the HBase shell

hbck Run the hbase 'fsck' tool

hlog Write-ahead-log analyzer

snapshot Create a new snapshot of a table

snapshotinfo Tool for dumping snapshot information

hfile Store file analyzer

zkcli Run the ZooKeeper shell

upgrade Upgrade hbase

master Run an HBase HMaster node

regionserver Run an HBase HRegionServer node

zookeeper Run a Zookeeper server

rest Run an HBase REST server

thrift Run the HBase Thrift server

thrift2 Run the HBase Thrift2 server

clean Run the HBase clean up script

classpath Dump hbase CLASSPATH

mapredcp Dump CLASSPATH entries required by mapreduce

pe Run PerformanceEvaluation

ltt Run LoadTestTool

canary Run the Canary tool

version Print the version

CLASSNAME Run the class named CLASSNAME

hduser@ubuntu:/usr/local$ start-hbase.sh
localhost: starting zookeeper, logging to /usr/local/hbase/bin/../logs/hbase-hduser-zookeeper-
ubuntu.out

starting master, logging to /usr/local/hbase/logs/hbase-hduser-master-ubuntu.out

localhost: starting regionserver, logging to /usr/local/hbase/bin/../logs/hbase-hduser-regionserver-

ubuntu.out

hduser@ubuntu:/usr/local$ jps

3824 HQuorumPeer

2991 DataNode

3887 HMaster

3221 JobTracker

4055 HRegionServer

2849 NameNode

3140 SecondaryNameNode

4107 Jps

3363 TaskTracker

hduser@ubuntu:/usr/local$ hbase shell

HBase Shell; enter 'help<RETURN>' for list of supported commands.

Type "exit<RETURN>" to leave the HBase Shell

Version 0.98.19-hadoop1, r1e527e73bc539a04ba0fa4ed3c0a82c7e9dd7d15, Fri Apr 22 18:49:26 PDT

2016

hbase(main):001:0> list

TABLE

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/local/hbase/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/

StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/hadoop/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/

StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

HCCemp

ibm

mytable

sony
t1

tab2

6 row(s) in 3.9830 seconds

=> ["HCCemp", "ibm", "mytable", "sony", "t1", "tab2"]

hbase(main):002:0> create ‘customer_hbase’ , ‘contactid’

ERROR: wrong number of arguments (0 for 1)

Here is some help for this command:

Creates a table. Pass a table name, and a set of column family

specifications (at least one), and, optionally, table configuration.

Column specification can be a simple string (name), or a dictionary

(dictionaries are described below in main help output), necessarily

including NAME attribute.

Examples:

Create a table with namespace=ns1 and table qualifier=t1

hbase> create 'ns1:t1', {NAME => 'f1', VERSIONS => 5}

Create a table with namespace=default and table qualifier=t1

hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'}

hbase> # The above in shorthand would be the following:

hbase> create 't1', 'f1', 'f2', 'f3'

hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true}

hbase> create 't1', {NAME => 'f1', CONFIGURATION => {'hbase.hstore.blockingStoreFiles' => '10'}}

Table configuration options can be put at the end.

Examples:

hbase> create 'ns1:t1', 'f1', SPLITS => ['10', '20', '30', '40']
 hbase> create 't1', 'f1', SPLITS => ['10', '20', '30', '40']

hbase> create 't1', 'f1', SPLITS_FILE => 'splits.txt', OWNER => 'johndoe'

hbase> create 't1', {NAME => 'f1', VERSIONS => 5}, METADATA => { 'mykey' => 'myvalue' }

hbase> # Optionally pre-split the table into NUMREGIONS, using

hbase> # SPLITALGO ("HexStringSplit", "UniformSplit" or classname)

hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit'}

hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit', CONFIGURATION =>
{'hbase.hregion.scan.loadColumnFamiliesOnDemand' => 'true'}}

hbase> create 't1', {NAME => 'f1', DFS_REPLICATION => 1}

You can also keep around a reference to the created table:

hbase> t1 = create 't1', 'f1'

Which gives you a reference to the table named 't1', on which you can then

call methods.

hduser@ubuntu:/usr/local$ start-hbase.sh

localhost: zookeeper running as process 3824. Stop it first.

master running as process 3887. Stop it first.

localhost: regionserver running as process 4055. Stop it first.

hduser@ubuntu:/usr/local$ hbase shell

HBase Shell; enter 'help<RETURN>' for list of supported commands.

Type "exit<RETURN>" to leave the HBase Shell

Version 0.98.19-hadoop1, r1e527e73bc539a04ba0fa4ed3c0a82c7e9dd7d15, Fri Apr 22 18:49:26 PDT

2016

hbase(main):001:0> create 'customer_hbase' , 'contactid'

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/local/hbase/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/

StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/
StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

0 row(s) in 4.6590 seconds

=> Hbase::Table - customer_hbase

hbase(main):002:0> list

TABLE

HCCemp

customer_hbase

ibm

mytable

sony

t1

tab2

7 row(s) in 0.1390 seconds

=> ["HCCemp", "customer_hbase", "ibm", "mytable", "sony", "t1", "tab2"]

hbase(main):003:0> describe customer_hbase

NameError: undefined local variable or method `customer_hbase' for #<Object:0x17faa217>

hbase(main):004:0> describe 'customer_hbase'

Table customer_hbase is ENABLED

customer_hbase

COLUMN FAMILIES DESCRIPTION

{NAME => 'contactid', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION

_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FO

REVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLO

CKCACHE => 'true'}

1 row(s) in 0.1460 seconds

hbase(main):005:0>

hbase(main):006:0*

hbase(main):007:0*

hbase(main):008:0* scan customer_hbase

NameError: undefined local variable or method `customer_hbase' for #<Object:0x17faa217>

hbase(main):009:0> scan 'customer_hbase'

ROW COLUMN+CELL

BOO1 column=contactid:firstname, timestamp=1611459538394, value=Azur

BOO1 column=contactid:lastname, timestamp=1611459538394, value=Balu

HOO1 column=contactid:firstname, timestamp=1611459538394, value=Bigd

ata

HOO1 column=contactid:lastname, timestamp=1611459538394, value=Doug

SOO1 column=contactid:firstname, timestamp=1611459538394, value=Spar

SOO1 column=contactid:lastname, timestamp=1611459538394, value=Brick

VOO1 column=contactid:firstname, timestamp=1611459538394, value=Hbas

VOO1 column=contactid:lastname, timestamp=1611459538394, value=Vivek

4 row(s) in 0.5750 seconds

hbase(main):010:0>

ref: http://geoinsyssoft.com/sqoop-import-hbase/