Users posting videos to Youtube and need to be encoded
Users posting tweets that will be sent to “News Feed Caches” of followers
Log based message broker
Sequential writes on disk
In order processing
More durability, Messages not deleted from disk
Replayability
Partitioning done to increase throughput
Uses
Sensors metrics coming in, want to take the average of the last 20
Each write on a database that will put in a search index
Kafka
developed by LinkedIn
distributed capabilities + message queue + Pub/Sub model
When to use message queues
process jobs asynchronously because they are expensive
post processing
decoupling systems making it easier to develop them independently
examples:
order processing service can use a message queue to send the details of a new order to a payment processing service.
post process uploaded multimedia so that we can have different versions that can be served to different types of users
posting a tweet, the tweet could be instantly posted to your timeline, but it could take some time before your tweet is actually delivered to all of your followers. (the client is doing a small amount of processing to make it seem like the task has completed)
E-commerce: Message Queues are used to process orders, payments, and shipping notifications.
Gaming: Message queues are used to synchronize game servers and clients.
Social Media: Message queues are used to distribute messages and notifications to users.