Where does Hadoop store the logs of YARN applications?

logging hadoop yarn
padmalcom picture padmalcom · Feb 7, 2014 · Viewed 66.9k times · Source

I run the basic example of Hortonworks' yarn application example. The application fails and I want to read the logs to figure out why. But I can't find any files at the expected location (/HADOOP_INSTALL_FOLDER/logs) where the logs of my mapreduce jobs are stored.

Does anybody know where yarn stores the non-mapreduce log files? Do I have to configure a special directory in the xml files?

Answer

Remus Rusanu picture Remus Rusanu · Feb 7, 2014

The container logs should be under yarn.nodemanager.log-dirs:

Where to store container logs. An application's localized log directory will be found in ${yarn.nodemanager.log-dirs}/application_${appid}. Individual containers' log directories will be below this, in directories named container_{$contid}. Each container directory will contain the files stderr, stdin, and syslog generated by that container.

Related questions

Where are logs in Spark on YARN?

I'm new to spark. Now I can run spark 0.9.1 on yarn (2.0.0-cdh4.2.1). But there is no log after execution. The following command is used to run a spark example. But logs are not found in the history server as in …

Read More
Why does my yarn application not have logs even with logging enabled?

I have enabled logs in the xml file: yarn-site.xml, and I restarted yarn by doing: sudo service hadoop-yarn-resourcemanager restart sudo service hadoop-yarn-nodemanager restart I ran my application, and then I see the applicationID in yarn application -list. So, I …

Read More
grep a file, but show several surrounding lines?

I would like to grep for a string, but also show the preceding five lines and the following five lines as well as the matched line. How would I be able to do this?

Read More