Producers

• aka Publishers

• It writes data to topics
• It automatically knows to which broker and partition to write to
• Producer can choose to receive acknowledgement
  • controlled by acks config

• Syntax:

// topic and value are mandatory
sendData(String topic, int partition, K key, V value, Map headers)

Producer Configs

• https://kafka.apache.org/20/documentation.html#producerconfigs
• https://kafka.apache.org/documentation/#producerconfigs
• bootstrap.servers
• key.serializer
• value.serializer
• acks
  • acks=0 : Producer won’t wait for acknowledgement (possible data loss)
  • acks=1 (default)
    • Producer will wait for leader acknowledgment (limited data loss)
  • acks=all (same as acks=-1)
    • Leader + replicas acknowledgment (no data loss)
    • Leader will wait for the full set of in-sync replicas to acknowledge the record

• retries
  • Number of times to retry a request that fails with a transient error
• retry.backoff.ms
  • Time to wait before attempting to retry
• client.id
• enable.idempotence: true/false
  • ensures exactly one message in broker
  • requirements
    • max.in.flight.requests.per.connection <= 5
    • retries > 0
    • acks = all
• transactional.id
  • If given, implies enable.idempotence
  • By default, Tx requires a cluster of at least three brokers
• compression.type
  • none
  • gzip
  • snappy
  • lz4

Delivery Semantics for Producers

• Producer choose when to retry if failure happens
• 3 delivery semantics:
  • At least once
    • with acks=all, retries until successfully publishes to topic with no data loss
    • can cause duplicate messages
  • At most once
    • does not retry if failure happens
    • no duplicate messages
  • Exactly once
    • most difficult to achieve
    • message is delivered exactly once
    • messaging system and client must cooperate to achieve this