Where are the Spark logs on EMR?

scala apache-spark emr
Sean Bollin picture Sean Bollin · May 28, 2015 · Viewed 16.6k times · Source

I'm not able to locate error logs or message's from println calls in Scala while running jobs on Spark in EMR.

Where can I access these?

I'm submitting the Spark job, written in Scala to EMR using script-runner.jar with arguments --deploy-mode set to cluster and --master set to yarn. It runs the job fine.

However I do not see my println statements in the Amazon EMR UI where it lists "stderr, stdoutetc. Furthermore if my job errors I don't see why it had an error. All I see is this in thestderr`:

15/05/27 20:24:44 INFO yarn.Client: Application report from ResourceManager: 
 application identifier: application_1432754139536_0002
 appId: 2
 clientToAMToken: null
 appDiagnostics: 
 appMasterHost: ip-10-185-87-217.ec2.internal
 appQueue: default
 appMasterRpcPort: 0
 appStartTime: 1432758272973
 yarnAppState: FINISHED
 distributedFinalState: FAILED
 appTrackingUrl: http://10.150.67.62:9046/proxy/application_1432754139536_0002/A
 appUser: hadoop

`

Answer

ChristopherB picture ChristopherB · May 29, 2015

With the deploy mode of cluster on yarn the Spark driver and hence the user code executed will be within the Application Master container. It sounds like you had EMR debugging enabled on the cluster so logs should have also pushed to S3. In the S3 location look at task-attempts/<applicationid>/<firstcontainer>/*.

Related questions

java.lang.NoClassDefFoundError: org/apache/spark/sql/DataFrame

I am running my Spark code to save data to HBase in Amazon EMR 5.8.0 which has Spark 2.2.0 installed. Running in IntelliJ, it works fine but in EMR Cluster it is throwing me this error: java.lang.NoClassDefFoundError: org/apache/spark/…

Read More
How can I change column types in Spark SQL's DataFrame?

Suppose I'm doing something like: val df = sqlContext.load("com.databricks.spark.csv", Map("path" -> "cars.csv", "header" -> "true")) df.printSchema() root |-- year: string (nullable = true) |-- make: string (nullable = true) |-- model: string (nullable = …

Read More
Spark - load CSV file as DataFrame?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with df.registerTempTable("table_name") I have tried: scala> val df = sqlContext.load("hdfs:///csv/file/dir/file.csv") Error …

Read More