Why does my yarn application not have logs even with logging enabled?

hadoop apache-spark logging yarn
makansij picture makansij · Mar 9, 2017 · Viewed 15k times · Source

I have enabled logs in the xml file: yarn-site.xml, and I restarted yarn by doing:

sudo service hadoop-yarn-resourcemanager restart
sudo service hadoop-yarn-nodemanager restart

I ran my application, and then I see the applicationID in yarn application -list. So, I do this: yarn logs -applicationId <application ID>, and I get the following:

hdfs://<ip address>/var/log/hadoop-yarn/path/to/application/  does not have any log files

Do I need to change some other configuration? Or am I accessing the logs the wrong way?

Thank you.

Answer

franklinsijo picture franklinsijo · Mar 23, 2017 
yarn application -list

will list only the applications that are either in SUBMITTED, ACCEPTED or RUNNING state.

Log aggregation collects each container's logs and moves these logs onto the directory configured in yarn.nodemanager.remote-app-log-dir only after the completion of the application. Refer the description of yarn.log-aggregation-enable property here.

So, the applicationId listed by the command isn't completed yet and the logs are not yet collected. Thus the response when trying to access the logs of a running application

hdfs://<ip address>/var/log/hadoop-yarn/path/to/application/  does not have any log files

You can try the same command yarn logs -applicationId <application ID> to view the logs once the application has completed.

To list all the FINISHED applications, use

yarn application -list -appStates FINISHED

Or to list all the applications

yarn application -list -appStates ALL

Related questions

Where are logs in Spark on YARN?

I'm new to spark. Now I can run spark 0.9.1 on yarn (2.0.0-cdh4.2.1). But there is no log after execution. The following command is used to run a spark example. But logs are not found in the history server as in …

Read More
Spark - load CSV file as DataFrame?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with df.registerTempTable("table_name") I have tried: scala> val df = sqlContext.load("hdfs:///csv/file/dir/file.csv") Error …

Read More
How to export data from Spark SQL to CSV

This command works with HiveQL: insert overwrite directory '/data/home.csv' select * from testtable; But with Spark SQL I'm getting an error with an org.apache.spark.sql.hive.HiveQl stack trace: java.lang.RuntimeException: Unsupported language features in …

Read More