spark-kafka-writer v0.4.0 released!
We’re pleased to announce version 0.4.0 of Spark Kafka Writer.
Spark Kafka Writer is a library that lets tou save your Spark data to Kafka seamlessly:
The repository is on GitHub
and you can find the latest version on maven central.
In this post, we’ll walk through the new support for writing
Datasets to Kafka.
Writing a DataFrame to Kafka
From version 0.4.0 on, you’ll be able to write
DataFrames to Kafka.
This differs from writing the output of batch queries to Kafka using the Structure Streaming API,
in the way that you control how you serialize
Rows and you can access the callback API.
Writing a Dataset to Kafka
In the same way you can write
DataFrames to Kafka, you’ll now be able to write
Version 0.4.0 also brings other changes:
- Supporting Spark 2.2.0
- Providing a way to close producers (see pull request #77)
- Dropping the support for Kafka 0.8
For version 0.5.0, we’re aiming to provide a native API for Java and Python.
If you’d like to get involved, there are different ways you can contribute to
You can also ask questions and discuss the project on the Gitter channel and check out the Scaladoc.