This documents are about Apache Sqoop:
https://sqoop.apache.org/
1
This Document show how to import data from mysql to HDFS using sqoop
command.
Open the Terminal and type below command
mysql> mysql -u root –pcloudera
mysql -u root –pcloudera
Show all database in mysql
mysql> show databases;
show databases;
1
2
Used retail_db databases.
mysql> use retail_db
use retail_db
2
3
Show all tables in retail_db database.
mysql> show tables;
show tables;
Display all fields in departments table.
mysql> select * from departments;
3
4
select * from departments;
To exit mysql database
mysql> quit
quit
Import data from mysql to HDFS using sqoop import commands
Display all mysql database using sqoop list command
sqoop list-databases --connect jdbc:mysql://quickstart.cloudera --username
root --password cloudera
4
5
Import data from mysql to HDFS using sqoop import command
sqoop import --connect jdbc:mysql://localhost:3306/retail_db --table
departments --username retail_dba --password cloudera
5
6
To display imported data in Hue file browser.
6
7
To show data in HDFS.
hadoop fs -cat /user/cloudera/departments/part*
7
This document explains how to export data from HDFS to mysql using
sqoop export command.mysql
Open the terminal and login to mysql database and write below command
mysql -uroot -pcloudera
Show database in mysql database
show database;
Use retail_db database in mysql database.
1
use retail_db
Show all tables in retail_db databases
show tables;
Create departments_data table under retail_db databes
create table departments_data like departments;
2
Show all tables in retail_db
show tables;
Show all data from departments_data
select * from departments_data;
3
Exit mysql database
quit;
Export data from HDFS to mysql
sqoop export --connect jdbc:mysql://localhost:3306/retail_db --table
departments_data --username retail_dba --password cloudera --export-dir
departments
4
Login into mysql
mysql -u root –pcloudera
Used retail_ database
use retail_db;
5
Show all tables
show tables;
Show exported data in departments_data table.
select * from departments_data;
6
Sqoop incremental import
Login to mysql databases
mysql> mysql -uroot -pcloudera
mysql -u root –pcloudera
Show all database in mysql
mysql> show databases;
show databases;
1/6
Used retail_db databases.
mysql> use retail_db
use retail_db
Show all tables in retail_db database.
mysql> show tables;
show tables;
2/6
Display all fields in departments table.
mysql> select * from departments;
select * from departments;
Insert two rows in departments table
INSERT INTO departments (department_id, department_name) VALUES (21,
'NewVal');
3/6
INSERT INTO departments (department_id, department_name) VALUES (22,
'NewVal');
Display all fields in departments table.
select * from departments;
Exit mysql database
quit
To show data in HDFS.
Hadoop fs –cat /user/cloudera/departments/part*
4/6
To show data in dapartments table through sqoop eval command.
sqoop eval --connect jdbc:mysql://localhost:3306/retail_db --username
retail_dba --password cloudera --query "select * from departments"
Import new data from mysql to HDFS using sqoop import command
sqoop import --connect jdbc:mysql://localhost:3306/retail_db --username
retail_dba --password cloudera --table departments --check-column
department_id --incremental append --last-value 7
5/6
To show all imported data in hdfs
hadoop fs -cat /user/cloudera/departments/part*
6/6
This Document show how to create sqoop job using sqoop command.
Create sqoop job using below command:
sqoop job --create mynewjob -- import --connect
jdbc:mysql://localhost:3306/retail_db --username root --password cloudera --
table orders
--list’ argument is used to verify the saved jobs. The following command is used to
verify the list of saved Sqoop jobs
sqoop job --list
--show’ argument is used to inspect or verify particular jobs and their details.
sqoop job --show mynewjob
1/3
‘--exec’ option is used to execute a saved job. The following command is used to
execute a saved job called myjob.
sqoop job --exec mynewjob
2/3
3/3
