How to setup Django and Celery with Docker and docker-compose
This story sets up Django and adds Celery support to a docker-compose setup. I have updated this on 22 May, 2022 to use the newest version of Celery, which does foster a few changes.
Setup basic Django project
First create a virtualenv and install django and celery (5.2.6):
$ python3 -m venv ./venv
$ source venv/bin/activate
$ pip install django celery
$ pip freeze > requirements.txt
Create the Django application
$ django-admin startproject config .
Create a dockerfile and add this:
FROM python:3.10
ARG APP_USER=appuser
RUN groupadd -r ${APP_USER} && useradd --no-log-init -r -g ${APP_USER} ${APP_USER}
EXPOSE 8000
WORKDIR /app
# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
ENV DEBUG 0
RUN apt-get update && apt-get install
RUN apt-get install -y \
gcc \
git \
python3-dev \
&& apt-get clean
RUN pip install -U pip
# Install pip requirements
COPY ./requirements.txt /app/requirements.txt
RUN python -m pip install -r /app/requirements.txt
COPY . /app/
Setup Docker-Compose
Create the docker-compose.yml file:
version: '3.2'
services:
db:
image: mdillon/postgis:9.6
container_name: postgis
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=12345
- POSTGRES_DB=projectdatabase
ports:
- "5433:5432"
volumes:
- data:/var/lib/postgresql/data
web:
build: .
command: bash -c "python /code/manage.py migrate --noinput && python /code/manage.py runserver 0.0.0.0:8000"
restart: "always"
volumes:
- .:/code
ports:
- "8001:8000"
depends_on:
- db
- broker
broker:
image: rabbitmq:latest
hostname: broker
environment:
- RABBITMQ_DEFAULT_USER=admin
- RABBITMQ_DEFAULT_PASS=mypass
ports:
- "5672:5672"
worker:
build: .
restart: "no"
command: celery -A myprojectname worker
volumes:
- .:/code
depends_on:
- broker
flower:
image: mher/flower
command: ["flower", "--broker=amqp://admin:mypass@broker", "--port=5555"]
ports:
- 5555:5555
depends_on:
- broker
volumes:
data:
Update settings
Change a single line in config/settings.py:
ALLOWED_HOSTS = ["*"]
Run application
Now run docker-compose up --build:
docker-compose up --build
go to localhost:8000 and hurra... your django app is running.
Add Celery to project
Add 3 services to docker-compose to get:
version: '3.2'
services:
web:
build: .
command: bash -c "python /code/manage.py migrate --noinput && python /code/manage.py runserver 0.0.0.0:8000"
restart: "always"
volumes:
- .:/code
ports:
- "8000:8000"
rabbitmq:
image: rabbitmq:latest
hostname: broker
environment:
- RABBITMQ_DEFAULT_USER=admin
- RABBITMQ_DEFAULT_PASS=mypass
ports:
- "5672:5672"
worker:
build:
dockerfile: Dockerfile
context: .
restart: "no"
command: celery -A config worker
volumes:
- .:/code
depends_on:
- rabbitmq
- web
flower:
image: mher/flower
environment:
- CELERY_BROKER_URL=PYAMQP://admin:mypass@broker:5672//
- FLOWER_PORT=5555
ports:
- 5555:5555
depends_on:
- rabbitmq
volumes:
data:
We also need to update our settings file and add a file to our config/ folder:
celery.py in the same folder as our settings.py file:
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings')
app = Celery('config')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
@app.task(bind=True)
def debug_task(self):
print(f'Request: {self.request!r}')
Add one line to settings.py:
rabbitmq_url = 'pyamqp://admin:mypass@broker:5672'
CELERY_BROKER_URL = rabbitmq_url
and update the __init__.py file in the config folder to be:
from .celery import app as celery_app
__all__ = ('celery_app',)
Congratulations, you have a working django and celery application running in docker.
And that is pretty much it. Run $ docker-compose up --build to rebuild the Docker images with the new requirements.txt file.
On localhost:5555 we have installed Flower, which is a handy tool to monitor your celery workers.
links
https://www.revsys.com/tidbits/celery-and-django-and-docker-oh-my/
https://www.revsys.com/tidbits/brief-intro-docker-djangonauts/
https://docs.celeryq.dev/en/stable/django/first-steps-with-django.html#using-celery-with-django/