Difference between createOrReplaceTempView and registerTempTable

apache-spark pyspark apache-spark-sql pyspark-sql sparkr
Amogh Huilgol picture Amogh Huilgol · Jul 17, 2017 · Viewed 15.1k times · Source

I am new to spark and was trying out a few commands in sparkSql using python when I came across these two commands:

createOrReplaceTempView() and registerTempTable().

What is the difference between the two commands?. They seem to have same set of functionalities.

Answer

zero323 picture zero323 · Jul 17, 2017

registerTempTable is a part of the 1.x API and has been deprecated in Spark 2.0.

createOrReplaceTempView and createTempView have been introduced in Spark 2.0, as a replacement for registerTempTable.

Other than that registerTempTable and createOrReplaceTempView functionally equivalent and the former one calls the latter one.

Related questions

Convert pyspark string to date format

I have a date pyspark dataframe with a string column in the format of MM-dd-yyyy and I am attempting to convert this into a date column. I tried: df.select(to_date(df.STRING_COLUMN).alias('new_date')).show() and …

Read More
Split Spark Dataframe string column into multiple columns

I've seen various people suggesting that Dataframe.explode is a useful way to do this, but it results in more rows than the original dataframe, which isn't what I want at all. I simply want to do the Dataframe equivalent …

Read More
How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?

import numpy as np df = spark.createDataFrame( [(1, 1, None), (1, 2, float(5)), (1, 3, np.nan), (1, 4, None), (1, 5, float(10)), (1, 6, float('nan')), (1, 6, float('nan'))], ('session', "timestamp1", "id2")) Expected output dataframe with count of nan/null for each column Note: The previous questions I found in stack overflow …

Read More