What is the benefit of using nested data types in Parquet?

apache-spark nested parquet data-files
user976850 picture user976850 · Mar 25, 2018 · Viewed 7.9k times · Source

Is there any performance benefit resulting from the usage of using nested data types in the Parquet file format?

AFAIK Parquet files are usually created specifically for query services e.g. Athena, so the process which creates those might as well simply flatten the values - thereby allowing easier querying, simpler schema, and retaining the column statistics for each column.

What benefit is there to be gained by using nested data types e.g. struct?

Answer

Avishek Bhattacharya picture Avishek Bhattacharya · Mar 25, 2018

There is a negative consequence keeping nested structure in parquet. The issue is spark predicate pushdown doesn't work properly if you have nested structure in the parquet file.

So even if you are working with few fields in your parquet dataset spark will load and materialize the entire dataset.

Here is the ticket which is opened for a long time regarding this issue.

EDIT

The issue has been resolved in spark 2.4 version.

Related questions

How to read a nested collection in Spark

I have a parquet table with one of the columns being , array<struct<col1,col2,..colN>> Can run queries against this table in Hive using LATERAL VIEW syntax. How to read this table into an RDD, …

Read More
How can I change column types in Spark SQL's DataFrame?

Suppose I'm doing something like: val df = sqlContext.load("com.databricks.spark.csv", Map("path" -> "cars.csv", "header" -> "true")) df.printSchema() root |-- year: string (nullable = true) |-- make: string (nullable = true) |-- model: string (nullable = …

Read More
Spark - load CSV file as DataFrame?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with df.registerTempTable("table_name") I have tried: scala> val df = sqlContext.load("hdfs:///csv/file/dir/file.csv") Error …

Read More