Select all except particular column in spark sql

apache-spark-sql hiveql pyspark-sql spark-hive
Patel picture Patel · Apr 26, 2017 · Viewed 12.1k times · Source

I want to select all columns in a table except StudentAddress and hence I wrote following query:

select `(StudentAddress)?+.+` from student;

It gives following error in Squirrel Sql client. org.apache.spark.sql.AnalysisException: cannot resolve '(StudentAddress)?+.+' given input columns

Answer

Neeraj Malhotra picture Neeraj Malhotra · Nov 17, 2017

You can use drop() method in the DataFrame API to drop a particular column and then select all the columns.

For example:

val df = hiveContext.read.table("student")
val dfWithoutStudentAddress = df.drop("StudentAddress")

Related questions

How to export data from Spark SQL to CSV

This command works with HiveQL: insert overwrite directory '/data/home.csv' select * from testtable; But with Spark SQL I'm getting an error with an org.apache.spark.sql.hive.HiveQl stack trace: java.lang.RuntimeException: Unsupported language features in …

Read More
PySpark: withColumn() with two conditions and three outcomes

I am working with Spark and PySpark. I am trying to achieve the result equivalent to the following pseudocode: df = df.withColumn('new_column', IF fruit1 == fruit2 THEN 1, ELSE 0. IF fruit1 IS NULL OR fruit2 IS NULL 3.) I am trying …

Read More
Can we load Parquet file into Hive directly?

I know we can load parquet file using Spark SQL and using Impala but wondering if we can do the same using Hive. I have been reading many articles but I am still confused. Simply put, I have a parquet …

Read More