Hive - How to know which execution engine I am currently using

hadoop configuration hive tez
Indrajeet Gour picture Indrajeet Gour · Feb 26, 2018 · Viewed 11.9k times · Source

I want to automate my hive ETL workflow in such a way that I need to execute hive jobs on the basis of execution engine (Tez or MR) because of memory constraints.

Would you please help, as I wanted to cross-check in-between of my whole work-flow which execution engine currently I'm dealing with.

Thanks in advance.

Answer

shriyog picture shriyog · Feb 26, 2018

The Hive execution engine is controlled by hive.execution.engine property. It can be either of the following:

  • mr (Map Reduce, default)
  • tez (Tez execution, for Hadoop 2 only)
  • spark (Spark execution, for Hive 1.1.0 onward).

The property can be read & updated using hive/beeline cli

  • For reading - SET hive.execution.engine;
  • For updating - SET hive.execution.engine=tez;

If you want to programmatically get this value out, you must go for HiveClient which supports multiple ways like JDBC, Java, Python, PHP, Ruby, C++, etc.

References

Related questions

Hadoop cluster setup - java.net.ConnectException: Connection refused

I want to setup a hadoop-cluster in pseudo-distributed mode. I managed to perform all the setup-steps, including startuping a Namenode, Datanode, Jobtracker and a Tasktracker on my machine. Then I tried to run some exemplary programms and faced the java.…

Read More
What is the relation between 'mapreduce.map.memory.mb' and 'mapred.map.child.java.opts' in Apache Hadoop YARN?

I would like to know the relation between the mapreduce.map.memory.mb and mapred.map.child.java.opts parameters. Is mapreduce.map.memory.mb > mapred.map.child.java.opts?

Read More
Hadoop: ...be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation

I'm getting the following error when attempting to write to HDFS as part of my multi-threaded application could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation. …

Read More