Hive loading in partitioned table

loading hive
Valery Yesypenko picture Valery Yesypenko · Oct 30, 2012 · Viewed 66.3k times · Source

I have a log file in HDFS, values are delimited by comma. For example:

2012-10-11 12:00,opened_browser,userid111,deviceid222

Now I want to load this file to Hive table which has columns "timestamp","action" and partitioned by "userid","deviceid". How can I ask Hive to take that last 2 columns in log file as partition for table? All examples e.g. "hive> LOAD DATA INPATH '/user/myname/kv2.txt' OVERWRITE INTO TABLE invites PARTITION (ds='2008-08-15');" require definition of partitions in the script, but I want partitions to set up automatically from HDFS file.

The one solution is to create intermediate non-partitioned table with all that 4 columns, populate it from file and then make an INSERT into first_table PARTITION (userid,deviceid) select from intermediate_table timestamp,action,userid,deviceid; but that is and additional task and we will have 2 very similiar tables.. Or we should create external table as intermediate.

Answer

Denny Lee picture Denny Lee · Nov 5, 2012

Ning Zhang has a great response on the topic at http://grokbase.com/t/hive/user/114frbfg0y/can-i-use-hive-dynamic-partition-while-loading-data-into-tables.

The quick context is that:

  1. Load data simply copies data, it doesn't read it so it cannot figure out what to partition
  2. Would suggest that you load data into an intermediate table first (or using an external table pointing to all the files) and then letting partition dynamic insert to kick in to load it into a partitioned table

Related questions

How to add a spinner icon to button when it's in the Loading state?

Twitter Bootstrap's buttons have a nice Loading... state available. The thing is that it just shows a message like Loading... passed through the data-loading-text attribute like this: <button type="button" class="btn btn-primary start" id="btnStartUploads" data-loading-text="@Localization.Uploading"&…

Read More
Using the "animated circle" in an ImageView while loading stuff

I am currently using in my application a listview that need maybe one second to be displayed. What I currently do is using the @id/android:empty property of the listview to create a "loading" text. <TextView android:id="@…

Read More
open resource with relative path in Java

In my Java app I need to get some files and directories. This is the program structure: ./main.java ./package1/guiclass.java ./package1/resources/resourcesloader.java ./package1/resources/repository/modules/ -> this is the dir I need to get ./…

Read More