SparkSQL vs Hive on Spark - Difference and pros and cons?

apache-spark hadoop hive apache-spark-sql
Gaurav Khare picture Gaurav Khare · Jul 24, 2015 · Viewed 42k times · Source

SparkSQL CLI internally uses HiveQL and in case Hive on spark(HIVE-7292) , hive uses spark as backend engine. Can somebody throw some more light, how exactly these two scenarios are different and pros and cons of both approaches?

Answer

prajod picture prajod · Feb 2, 2016

  1. When SparkSQL uses hive

    SparkSQL can use HiveMetastore to get the metadata of the data stored in HDFS. This metadata enables SparkSQL to do better optimization of the queries that it executes. Here Spark is the query processor.

  2. When Hive uses Spark See the JIRA entry: HIVE-7292

    Here the the data is accessed via spark. And Hive is the Query processor. So we have all the deign features of Spark Core to take advantage of. But this is a Major Improvement for Hive and is still "in progress" as of Feb 2 2016.

  3. There is a third option to process data with SparkSQL

    Use SparkSQL without using Hive. Here SparkSQL does not have access to the metadata from the Hive Metastore. And the queries run slower. I have done some performance tests comparing options 1 and 3. The results are here.

Related questions

Save Spark dataframe as dynamic partitioned table in Hive

I have a sample application working to read from csv files into a dataframe. The dataframe can be stored to a Hive table in parquet format using the method df.saveAsTable(tablename,mode). The above code works fine, but I …

Read More
How to make shark/spark clear the cache?

when i run my shark queries, the memory gets hoarded in the main memory This is my top command result. Mem: 74237344k total, 70080492k used, 4156852k free, 399544k buffers Swap: 4194288k total, 480k used, 4193808k free, 65965904k cached this doesn't …

Read More
Spark SQL "Limit"

Env : spark 1.6 using Hadoop. Hortonworks Data Platform 2.5 I have a table with 10 billion records and I would like to get 300 million records and move them to a temporary table. sqlContext.sql("select ....from my_table limit 300000000").repartition(50) .write.saveAsTable("temporary_…

Read More