Kafka Compaction and Cleanup Policy
Log Compaction Policy
- Retains only the most recent value for each key in a topic, discarding older records with the same key
- Used where you need to maintain the latest state or snapshot of data
- Property:
log.cleanup.policy=compact
- Examples:
- user profiles
- configuration data
Log Retention Policy
- Controls the time or size after which messages are deleted from Kafka topics
- Time-Based Retention
- Messages older than a specified duration (e.g., 7 days) are automatically deleted
- Property:
log.retention.ms
- default:
604800000 (7 days)
- Size-Based Retention
- Messages are deleted when the total log size exceeds a configured limit
- Property:
log.retention.bytes
- Used where you only need to keep recent data
- Examples:
- event logs
- streaming analytics