Hadoop: specify yarn queue for distcp

hadoop yarn distcp
Havnar picture Havnar · Nov 5, 2015 · Viewed 12.5k times · Source

On our cluster we have set up dynamic resource pools.

The rules are set so that first yarn will look at the specified queue, then to the username, then to primary group ...

However with a distcp I can't seem to be able to specify a queue, it just sets it to the primary group.

This is how I run it now (which doesn't work):

 hadoop distcp -Dmapred.job.queue.name:root.default .......

Answer

Manjunath Ballur picture Manjunath Ballur · Nov 5, 2015

You are committing a mistake in the specification of the parameter.

You should not use ":" for separating the key/value pairs. You should use "=".

The command should be

 hadoop distcp -Dmapred.job.queue.name=root.default .......

Related questions

Container is running beyond memory limits

In Hadoop v1, I have assigned each 7 mapper and reducer slot with size of 1GB, my mappers & reducers runs fine. My machine has 8G memory, 8 processor. Now with YARN, when run the same application on the same machine, I …

Read More
Apache Spark: The number of cores vs. the number of executors

I'm trying to understand the relationship of the number of cores and the number of executors when running a Spark job on YARN. The test environment is as follows: Number of data nodes: 3 Data node machine spec: CPU: Core i7…

Read More
Where are logs in Spark on YARN?

I'm new to spark. Now I can run spark 0.9.1 on yarn (2.0.0-cdh4.2.1). But there is no log after execution. The following command is used to run a spark example. But logs are not found in the history server as in …

Read More