Message Brokers, Message Queues, and Workers
Table of contents
Message Broker:
A message broker is software that enables applications, systems, and services to communicate with each other and exchange information.
They can validate, store, route, and deliver messages to the appropriate destinations.
They serve as intermediaries/middlewares between other applications, allowing senders to issue messages without knowing where the receivers are, whether or not they are active, or how many of them there are.
They facilitate decoupling of processes and services within systems.
Message queues and brokers are used interchangeably most of the time.
Message Queue:
In order to provide reliable message storage and guaranteed delivery, message brokers often rely on a substructure/component called a message queue that stores and orders the messages until the consuming applications can process them.
In a message queue, messages are stored in the exact order in which they were transmitted and remain in the queue until receipt is confirmed.
NOTE:
A message broker is a middleware responsible for persisting and routing messages between applications/services while allowing you to decouple your system into smaller parts. A message queue is a part of a message broker and is just a persistence mechanism.
Workers:
Workers are “background task servers”.
While your web server is responding to user requests, the worker servers can process tasks in the background. These workers can be used for sending emails, making large changes in the database, processing files, etc.
Workers are assigned tasks via a message queue.
The queue ensures that each worker only gets one task at a time and that each task is only being processed by one worker.
Example:
I have developed a job application review service API using:
- DRF -> web server
- RabbitMQ -> message broker
- Celery -> worker
=================