Hive Partition recovery

hive hdfs hql hadoop-partitioning
Balaji Boggaram Ramanarayan picture Balaji Boggaram Ramanarayan · May 26, 2016 · Viewed 7.8k times · Source

How to recover partitions in easy fashion. Here is the scenario :

  1. Have 'n' partitions on existing external table 't'
  2. Dropped table 't'
  3. Recreated table 't' // Note : same table but with excluding some column
  4. How to recover the 'n' partitions that existed for table 't' in step #1 ?

I can manually alter table to add 'n' partition by writing some script. But that's very tedious. Is there something built-in to recover these partitions ?

Answer

54l3d picture 54l3d · May 26, 2016

When the partitions directories still exist in the HDFS, simply run this command:

MSCK REPAIR TABLE table_name;

It adds the partitions definitions to the metastore based on what exists in the table directory.

Related questions

Where does Hive store files in HDFS?

I'd like to know how to find the mapping between Hive tables and the actual HDFS files (or rather, directories) that they represent. I need to access the table files directly. Where does Hive store its files in HDFS?

Read More
Hive load CSV with commas in quoted fields

I am trying to load a CSV file into a Hive table like so: CREATE TABLE mytable ( num1 INT, text1 STRING, num2 INT, text2 STRING ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ","; LOAD DATA LOCAL INPATH '/data.csv' OVERWRITE INTO …

Read More
SparkSQL - Read parquet file directly

I am migrating from Impala to SparkSQL, using the following code to read a table: my_data = sqlContext.read.parquet('hdfs://my_hdfs_path/my_db.db/my_table') How do I invoke SparkSQL above, so it can return something …

Read More