Spark job execution time

apache-spark apache-spark-mllib apache-spark-1.5
pranav3688 picture pranav3688 · Apr 30, 2016 · Viewed 15k times · Source

This might be a very simple question. But is there any simple way to measure the execution time of a spark job (submitted using spark-submit)?

It would help us in profiling the spark jobs based on the size of input data.

EDIT : I use http://[driver]:4040 to monitor my jobs, but this Web UI shuts down the moment my job finishes.

Answer

Ram Ghadiyaram picture Ram Ghadiyaram · Apr 30, 2016

Every SparkContext launches its own instance of Web UI which is available at

http://[master]:4040
by default (the port can be changed using spark.ui.port ).

It offers pages (tabs) with the following information:

Jobs, Stages, Storage (with RDD size and memory use) Environment, Executors, SQL

This information is available only until the application is running by default.

Tip : You can use the web UI after the application is finished by enabling spark.eventLog.enabled.

Sample web ui where you can see the time as 3.2hours: enter image description here

Related questions

AttributeError: 'DataFrame' object has no attribute 'map'

I wanted to convert the spark data frame to add using the code below: from pyspark.mllib.clustering import KMeans spark_df = sqlContext.createDataFrame(pandas_df) rdd = spark_df.map(lambda data: Vectors.dense([float(c) for c in data])) …

Read More
The value of "spark.yarn.executor.memoryOverhead" setting?

The value of spark.yarn.executor.memoryOverhead in a Spark job with YARN should be allocated to App or just the max value?

Read More
How to handle categorical features with spark-ml?

How do I handle categorical data with spark-ml and not spark-mllib ? Thought the documentation is not very clear, it seems that classifiers e.g. RandomForestClassifier, LogisticRegression, have a featuresCol argument, which specifies the name of the column of features in …

Read More