How can I convert a pyspark.sql.dataframe.DataFrame back to a sql table in databricks notebook

python sql apache-spark pyspark databricks
Semihcan Doken picture Semihcan Doken · Aug 20, 2016 · Viewed 9.5k times · Source

I created a dataframe of type pyspark.sql.dataframe.DataFrame by executing the following line: dataframe = sqlContext.sql("select * from my_data_table")

How can I convert this back to a sparksql table that I can run sql queries on?

Answer

Alberto Bonsanto picture Alberto Bonsanto · Aug 20, 2016

You can create your table by using createReplaceTempView. In your case it would be like:

dataframe.createOrReplaceTempView("mytable")

After this you can query your mytable using SQL.

If your a spark version is ≤ 1.6.2 you can use registerTempTable

Related questions

How to build a sparkSession in Spark 2.0 using pyspark?

I just got access to spark 2.0; I have been using spark 1.6.1 up until this point. Can someone please help me set up a sparkSession using pyspark (python)? I know that the scala examples available online are similar (here), but I …

Read More
Filtering a Pyspark DataFrame with SQL-like IN clause

I want to filter a Pyspark DataFrame with a SQL-like IN clause, as in sc = SparkContext() sqlc = SQLContext(sc) df = sqlc.sql('SELECT * from my_df WHERE field1 IN a') where a is the tuple (1, 2, 3). I am getting this error: …

Read More
How to use window functions in PySpark?

I'm trying to use some windows functions (ntile and percentRank) for a data frame but I don't know how to use them. Can anyone help me with this please? In the Python API documentation there are no examples about it. …

Read More