Using Sqoop to import data from MySQL to Hive

mysql hadoop hive sqoop
Nayan picture Nayan · Mar 14, 2014 · Viewed 59.7k times · Source

I am using Sqoop (version 1.4.4) to import data from MySQL to Hive. The data will be a subset of one of tables, i.e. few columns from a table. Is it necessary to create table in Hive before hand. Or importing the data will create the name specified in the command if it is not in the Hive?

Answer

Manu Eidenberger picture Manu Eidenberger · Mar 14, 2014

As mentioned in the sqoop documentation, you will not have to create any hive tables if you use the --hive-import argument in your command

example:

sqoop import --connect jdbc:mysql://mysql_server:3306/db_name --username mysql_user --password mysql_pass --table table_name --hive-import

Also... consider the --hive-overwrite argument if you want to schedule a full data import, on a daily base for example

Related questions

Sqoop: Could not load mysql driver exception

I Installed Sqoop in my local machine. Following are the config information. Bash.bashrc: export HADOOP_HOME=/home/hduser/hadoop export HBASE_HOME=/home/hduser/hbase export HIVE_HOME=/home/hduser/hive export HCAT_HOME=/home/hduser/hive/hcatalog export …

Read More
Dynamic Partitioning + CREATE AS on HIVE

I'm trying to create a new table from another table with CREATE AS and dynamic Partitioning on HiveCLI. I'm learning from Hive official wiki where there is this example: CREATE TABLE T (key int, value string) PARTITIONED BY (ds string, …

Read More
Hive connectivity to MySQL: Access denied for user 'hive'@'localhost' hive

I am setting up Hive on CentOS, and I have installed hadoop, hive and MySQL on the same server. I have also setup metastore DB on mySQL and user ID - hive has access to the database. I have the …

Read More