Top "Databricks" questions

For questions about the Databricks Unified Analytics Platform

get datatype of column using pyspark

We are reading data from MongoDB Collection. Collection column has two different values (e.g.: (bson.Int64,int) (int,float) ). …

apache-spark pyspark apache-spark-sql databricks
Exploding nested Struct in Spark dataframe

I'm working through a Databricks example. The schema for the dataframe looks like: > parquetDF.printSchema root |-- department: struct (…

scala apache-spark apache-spark-sql distributed-computing databricks
Get the size/length of an array column

I'm new in Scala programming and this is my question: How to count the number of string for each row? …

scala apache-spark apache-spark-sql databricks
DATEDIFF in SPARK SQl

I am new to Spark SQL. We are migrating data from SQL server to Databricks. I am using SPARK SQL . …

apache-spark-sql datediff databricks
Adding constant value column to spark dataframe

I am using Spark version 2.1 in Databricks. I have a data frame named wamp to which I want to add …

apache-spark pyspark databricks
How to solve this error org.apache.spark.sql.catalyst.errors.package$TreeNodeException

I have two procesess each process do 1) connect oracle db read a specific table 2) form dataframe and process it. 3) save …

apache-spark datastax-enterprise cassandra-3.0 databricks
How to slice a pyspark dataframe in two row-wise

I am working in Databricks. I have a dataframe which contains 500 rows, I would like to create two dataframes on …

python pyspark spark-dataframe databricks
How to export data from a dataframe to a file databricks

I'm doing right now Introduction to Spark course at EdX. Is there a possibility to save dataframes from Databricks on …

apache-spark pyspark databricks
Spark dataframe save in single file on hdfs location

I have dataframe and i want to save in single file on hdfs location. i found the solution here Write …

csv apache-spark dataframe databricks
Read/Write single file in DataBricks

I have a file which contains a list of names stored in a simple text file. Each row contains one …

python pyspark databricks