How to save DataFrame directly to Hive?

scala apache-spark hive apache-spark-sql
Gourav picture Gourav · Jun 5, 2015 · Viewed 180.4k times · Source

Is it possible to save DataFrame in spark directly to Hive?

I have tried with converting DataFrame to Rdd and then saving as a text file and then loading in hive. But I am wondering if I can directly save dataframe to hive

Answer

Vinay Kumar picture Vinay Kumar · Feb 18, 2016

You can create an in-memory temporary table and store them in hive table using sqlContext.

Lets say your data frame is myDf. You can create one temporary table using,

myDf.createOrReplaceTempView("mytempTable")

Then you can use a simple hive statement to create table and dump the data from your temp table. 

sqlContext.sql("create table mytable as select * from mytempTable");

Related questions

SparkSQL - Read parquet file directly

I am migrating from Impala to SparkSQL, using the following code to read a table: my_data = sqlContext.read.parquet('hdfs://my_hdfs_path/my_db.db/my_table') How do I invoke SparkSQL above, so it can return something …

Read More
How to parse a csv that uses ^A (i.e. \001) as the delimiter with spark-csv?

Terribly new to spark and hive and big data and scala and all. I'm trying to write a simple function that takes an sqlContext, loads a csv file from s3 and returns a DataFrame. The problem is that this particular …

Read More
Spark final task takes 100x times longer than first 199, how to improve

I am seeing some performance issues while running queries using dataframes. I have seen in my research, that long running finally tasks can be a sign that data is not disturbed optimally, but have not found a detailed process for …

Read More