Spark DataFrame: How to specify schema when writing as Avro

apache-spark spark-dataframe spark-avro
erwaman picture erwaman · Feb 21, 2018 · Viewed 8k times · Source

I want to write a DataFrame in Avro format using a provided Avro schema rather than Spark's auto-generated schema. How can I tell Spark to use my custom schema on write?

Answer

erwaman picture erwaman · Feb 21, 2018

After applying the patch in https://github.com/databricks/spark-avro/pull/222/, I was able to specify a schema on write as follows:

df.write.option("forceSchema", myCustomSchemaString).avro("/path/to/outputDir")

Related questions

Convert spark DataFrame column to python list

I work on a dataframe with two column, mvv and count. +---+-----+ |mvv|count| +---+-----+ | 1 | 5 | | 2 | 9 | | 3 | 3 | | 4 | 1 | i would like to obtain two list containing mvv values and count value. Something like mvv = [1,2,3,4] count = [5,9,3,1] So, I tried the following code: …

Read More
how to filter out a null value from spark dataframe

I created a dataframe in spark with the following schema: root |-- user_id: long (nullable = false) |-- event_id: long (nullable = false) |-- invited: integer (nullable = false) |-- day_diff: long (nullable = true) |-- interested: integer (nullable = false) |-- …

Read More
Converting Pandas dataframe into Spark dataframe error

I'm trying to convert Pandas DF into Spark one. DF head: 10000001,1,0,1,12:35,OK,10002,1,0,9,f,NA,24,24,0,3,9,0,0,1,1,0,0,4,543 10000001,2,0,1,12:36,OK,10002,1,0,9,f,NA,24,24,0,3,9,2,1,1,3,1,3,2,611 10000002,1,0,4,12:19,PA,10003,1,1,7,f,NA,74,74,0,2,15,2,0,2,3,1,2,2,691 Code: dataset = pd.read_csv("data/AS/test_v2.csv") sc = SparkContext(conf=conf) sqlCtx = SQLContext(sc) sdf = sqlCtx.createDataFrame(…

Read More