Upload zip file using --archives option of spark-submit on yarn

scala apache-spark zip yarn
Mo Tao picture Mo Tao · Jan 6, 2017 · Viewed 10.8k times · Source

I have a directory with some model files and my application has to access these models files in local file system due to some reason.

Of course I know that --files option of spark-submit can upload file to the working directory of each executor and it does work.

However, I want keep the directory structure of my files so I come up with --archives option, which is said

YARN-only:
......
--archives ARCHIVES         Comma separated list of archives to be extracted into the working directory of each executor.
......

But when I actually use it to upload models.zip, I found yarn just put it there without extraction, like what it did with --files. Have I misunderstood to be extracted or misused this option?

Answer

Mo Tao picture Mo Tao · Jan 9, 2017

Found the answer myself.

YARN does extract the archive but add an extra folder with the same name of the archive. To make it clear, If I put models/model1 and models/models2 in models.zip, then I have to access my models by models.zip/models/model1 and models.zip/models/model2.

Moreover, we can make this more beautiful using the # syntax.

The --files and --archives options support specifying file names with the # similar to Hadoop. For example you can specify: --files localtest.txt#appSees.txt and this will upload the file you have locally named localtest.txt into HDFS but this will be linked to by the name appSees.txt, and your application should use the name as appSees.txt to reference it when running on YARN.

Edit:

This answer was tested on spark 2.0.0 and I'm not sure the behavior in other versions.

Related questions

How can I change column types in Spark SQL's DataFrame?

Suppose I'm doing something like: val df = sqlContext.load("com.databricks.spark.csv", Map("path" -> "cars.csv", "header" -> "true")) df.printSchema() root |-- year: string (nullable = true) |-- make: string (nullable = true) |-- model: string (nullable = …

Read More
Spark - load CSV file as DataFrame?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with df.registerTempTable("table_name") I have tried: scala> val df = sqlContext.load("hdfs:///csv/file/dir/file.csv") Error …

Read More
How to convert rdd object to dataframe in spark

How can I convert an RDD (org.apache.spark.rdd.RDD[org.apache.spark.sql.Row]) to a Dataframe org.apache.spark.sql.DataFrame. I converted a dataframe to rdd using .rdd. After processing it I want it back in …

Read More