Spark Structured Streaming allows processing live data streams using DataFrame and Dataset APIs.
I'm trying to run a sample like StructuredKafkaWordCount. I started with the Spark Structured Streaming Programming guide. My code is …
scala apache-spark sbt sbt-assembly spark-structured-streamingI use HDP-2.6.3.0 with Spark2 package 2.2.0. I'm trying to write a Kafka consumer, using the Structured Streaming API, but I'm …
apache-spark apache-spark-sql spark-structured-streaming uberjarI have my timestamp in UTC and ISO8601, but using Structured Streaming, it gets automatically converted into the local time. …
java scala apache-spark apache-spark-sql spark-structured-streamingI'm reading a stream from Kafka, and I convert the value from Kafka ( which is JSON ) in to Structure. from_…
apache-spark apache-spark-sql spark-structured-streamingI read Spark Structured Streaming doesn't support schema inference for reading Kafka messages as JSON. Is there a way to …
apache-spark apache-kafka schema spark-structured-streamingI'm using a Kafka Source in Spark Structured Streaming to receive Confluent encoded Avro records. I intend to use Confluent …
apache-spark apache-kafka avro confluent-schema-registry spark-structured-streamingSpark 2.2 introduced a Kafka's structured streaming source. As I understand, it's relying on HDFS checkpoint directory to store offsets and …
apache-spark apache-kafka apache-spark-sql offset spark-structured-streamingI would like to do multiple aggregations in Spark Structured Streaming. Something like this: Read a stream of input files (…
apache-spark apache-spark-sql spark-structured-streamingI'm struggling to get the console sink working with PySpark Structured Streaming when run from Zeppelin. Basically, I'm not seeing …
apache-spark pyspark apache-zeppelin spark-structured-streamingSo I have some data I'm stream in a Kafka topic, I'm taking this streaming data and placing it into …
apache-spark pyspark apache-kafka spark-structured-streaming