Type Members

1.  class DStreamKafkaWriter[T] extends KafkaWriter[T] with Serializable

   Class used for writing DStreams to Kafka

2.  class DatasetKafkaWriter[T] extends KafkaWriter[T] with Serializable

   Class used for writing Datasets to Kafka

3.  abstract class KafkaWriter[T] extends Serializable

   Class used to write DStreams, RDDs and Datasets to Kafka

   Class used to write DStreams, RDDs and Datasets to Kafka

   Example usage:

   import com.github.benfradet.spark.kafka.writer.KafkaWriter._
   import org.apache.kafka.common.serialization.StringSerializer
   
   val topic = "my-topic"
   val producerConfig = Map(
     "bootstrap.servers" -> "127.0.0.1:9092",
     "key.serializer" -> classOf[StringSerializer].getName,
     "value.serializer" -> classOf[StringSerializer].getName
   )
   
   val dStream: DStream[String] = ...
   dStream.writeToKafka(
     producerConfig,
     s => new ProducerRecord[String, String](topic, s)
   )
   
   val rdd: RDD[String] = ...
   rdd.writeToKafka(
     producerConfig,
     s => new ProducerRecord[String, String](localTopic, s)
   )
   
   val dataset: Dataset[Foo] = ...
   dataset.writeToKafka(
     producerConfig,
     f => new ProducerRecord[String, String](localTopic, f.toString)
   )
   
   val dataFrame: DataFrame = ...
   dataFrame.writeToKafka(
     producerConfig,
     r => new ProducerRecord[String, String](localTopic, r.get(0).toString)
   )

   It is also possible to provide a callback for each write to Kafka.

   This is optional and has a value of None by default.

   Example Usage:

   @transient val log = org.apache.log4j.Logger.getLogger("spark-kafka-writer")
   
   val dStream: DStream[String] = ...
   dStream.writeToKafka(
     producerConfig,
     s => new ProducerRecord[String, String](topic, s),
     Some(new Callback with Serializable {
       override def onCompletion(metadata: RecordMetadata, e: Exception): Unit = {
         if (Option(e).isDefined) {
           log.warn("error sending message", e)
         } else {
           log.info(s"write succeeded, offset: ${metadata.offset()")
         }
       }
     })
   )

4.  class RDDKafkaWriter[T] extends KafkaWriter[T] with Serializable

   Class used for writing RDDs to Kafka