I'm trying to save dataframe in table hive.
In spark 1.6 it's work but after migration to 2.2.0 it doesn't work anymore.
Here's the code:
blocs
.toDF()
.repartition($"col1", $"col2", $"col3", $"col4")
.write
.format("parquet")
.mode(saveMode)
.partitionBy("col1", "col2", "col3", "col4")
.saveAsTable("db".tbl)
The format of the existing table project_bsc_dhr.bloc_views is HiveFileFormat
. It doesn't match the specified format ParquetFileFormat
.;
org.apache.spark.sql.AnalysisException: The format of the existing table project_bsc_dhr.bloc_views is HiveFileFormat
. It doesn't match the specified format ParquetFileFormat
.;
I have just tried to use .format("hive")
to saveAsTable
after getting the error and it worked.
I also would not recommend to use insertInto
suggested by the author, because it looks not type-safe (as much as this term can be applied to SQL API) and is error-prone in the way it ignores column names and uses position-base resolution.