Top "Spark-structured-streaming" questions

Spark Structured Streaming allows processing live data streams using DataFrame and Dataset APIs.

How to read records in JSON format from Kafka using Structured Streaming?

I am trying to use structured streaming approach using Spark-Streaming based on DataFrame/Dataset API to load a stream of …

Read More
scala apache-spark apache-kafka apache-spark-sql spark-structured-streaming
How to use from_json with Kafka connect 0.10 and Spark Structured Streaming?

I was trying to reproduce the example from [Databricks][1] and apply it to the new connector to Kafka and spark …

Read More
scala apache-spark apache-kafka apache-kafka-connect spark-structured-streaming
Structured Streaming exception when using append output mode with watermark

Despite the fact that I'm using withWatermark(), I'm getting the following error message when I run my spark job: Exception …

Read More
java apache-spark spark-structured-streaming
How to continuously monitor a directory by using Spark Structured Streaming

I want spark to continuously monitor a directory and read the CSV files by using spark.readStream as soon as …

Read More
scala apache-spark spark-structured-streaming
Spark streaming with python: how to add a UUID column?

I would like to add a column with a generated id to my data frame. I have tried: uuidUdf = udf(…

Read More
python dataframe spark-structured-streaming
How to write JDBC Sink for Spark Structured Streaming [SparkException: Task not serializable]?

I need a JDBC sink for my spark structured streaming data frame. At the moment, as far as I know …

Read More
scala apache-spark spark-structured-streaming