How to export data from a dataframe to a file databricks

apache-spark pyspark databricks
Tom Becker picture Tom Becker · Jul 27, 2016 · Viewed 17.7k times · Source

I'm doing right now Introduction to Spark course at EdX. Is there a possibility to save dataframes from Databricks on my computer.

I'm asking this question, because this course provides Databricks notebooks which probably won't work after the course.

In the notebook data is imported using command:

log_file_path = 'dbfs:/' + os.path.join('databricks-datasets', 'cs100', 'lab2', 'data-001', 'apache.access.log.PROJECT')

I found this solution but it doesn't work:

df.select('year','model').write.format('com.databricks.spark.csv').save('newcars.csv')

Answer

MrChristine picture MrChristine · Aug 1, 2016

Databricks runs a cloud VM and does not have any idea where your local machine is located. If you want to save the CSV results of a DataFrame, you can run display(df) and there's an option to download the results.

enter image description here

Related questions

get datatype of column using pyspark

We are reading data from MongoDB Collection. Collection column has two different values (e.g.: (bson.Int64,int) (int,float) ). I am trying to get a datatype using pyspark. My problem is some columns have different datatype. Assume quantity and …

Read More
Adding constant value column to spark dataframe

I am using Spark version 2.1 in Databricks. I have a data frame named wamp to which I want to add a column named region which should take the constant value NE. However, I get an error saying NameError: name 'lit' …

Read More
Exporting spark dataframe to .csv with header and specific filename

I am trying to export data from a spark dataframe to .csv file: df.coalesce(1)\ .write\ .format("com.databricks.spark.csv")\ .option("header", "true")\ .save(output_path) It is creating a file name "part-r-00001-512872f2-9b51-46c5…

Read More