Docker, Docker Compose and Django with Celery

How to setup Django and Celery with Docker and docker-compose

This story sets up Django and adds Celery support to a docker-compose setup. I have updated this on 22 May, 2022 to use the newest version of Celery, which does foster a few changes.

Setup basic Django project

First create a virtualenv and install django and celery (5.2.6):

$ python3 -m venv ./venv
$ source venv/bin/activate
$ pip install django celery
$ pip freeze > requirements.txt

Create the Django application

$ django-admin startproject config .

Create a dockerfile and add this:

FROM python:3.10

ARG APP_USER=appuser
RUN groupadd -r ${APP_USER} && useradd --no-log-init -r -g ${APP_USER} ${APP_USER}

EXPOSE 8000
WORKDIR /app

# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
ENV DEBUG 0

RUN apt-get update && apt-get install

RUN apt-get install -y \
  gcc \ 
  git \
  python3-dev \
  && apt-get clean

RUN pip install -U pip

# Install pip requirements
COPY ./requirements.txt /app/requirements.txt
RUN python -m pip install -r /app/requirements.txt
COPY . /app/

Setup Docker-Compose

Create the docker-compose.yml file:

version: '3.2'

services:
  db:
    image: mdillon/postgis:9.6
    container_name: postgis
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=12345
      - POSTGRES_DB=projectdatabase
    ports:
      - "5433:5432"
    volumes:
      - data:/var/lib/postgresql/data

  web:
    build: .
    command: bash -c "python /code/manage.py migrate --noinput && python /code/manage.py runserver 0.0.0.0:8000"
    restart: "always"
    volumes:
      - .:/code
    ports:
      - "8001:8000"
    depends_on:
      - db
      - broker

  broker:
    image: rabbitmq:latest
    hostname: broker
    environment:
      - RABBITMQ_DEFAULT_USER=admin
      - RABBITMQ_DEFAULT_PASS=mypass
    ports:
      - "5672:5672"

  worker:
    build: .
    restart: "no"
    command: celery -A myprojectname worker
    volumes:
      - .:/code
    depends_on:
      - broker

  flower:  
    image: mher/flower
    command: ["flower", "--broker=amqp://admin:mypass@broker", "--port=5555"]  
    ports:  
      - 5555:5555 
    depends_on:
      - broker


volumes:
  data:

Update settings

Change a single line in config/settings.py:

ALLOWED_HOSTS = ["*"]

Run application

Now run docker-compose up --build:

docker-compose up --build

go to localhost:8000 and hurra... your django app is running.

Add Celery to project

Add 3 services to docker-compose to get:

version: '3.2'

services:
  web:
    build: .
    command: bash -c "python /code/manage.py migrate --noinput && python /code/manage.py runserver 0.0.0.0:8000"
    restart: "always"
    volumes:
      - .:/code
    ports:
      - "8000:8000"

  rabbitmq:
    image: rabbitmq:latest
    hostname: broker
    environment:
      - RABBITMQ_DEFAULT_USER=admin
      - RABBITMQ_DEFAULT_PASS=mypass
    ports:
      - "5672:5672"

  worker:
    build: 
      dockerfile: Dockerfile
      context: . 
    restart: "no"
    command: celery -A config worker
    volumes:
      - .:/code
    depends_on:
      - rabbitmq
      - web

  flower:  
    image: mher/flower
    environment:
      - CELERY_BROKER_URL=PYAMQP://admin:mypass@broker:5672//
      - FLOWER_PORT=5555
    ports:  
      - 5555:5555 
    depends_on:
      - rabbitmq


volumes:
  data:

We also need to update our settings file and add a file to our config/ folder: celery.py in the same folder as our settings.py file:

import os
from celery import Celery


os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings')
app = Celery('config')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()


@app.task(bind=True)
def debug_task(self):
    print(f'Request: {self.request!r}')

Add one line to settings.py:

rabbitmq_url = 'pyamqp://admin:mypass@broker:5672'
CELERY_BROKER_URL = rabbitmq_url

and update the __init__.py file in the config folder to be:

from .celery import app as celery_app

__all__ = ('celery_app',)

Congratulations, you have a working django and celery application running in docker.

And that is pretty much it. Run $ docker-compose up --build to rebuild the Docker images with the new requirements.txt file.

On localhost:5555 we have installed Flower, which is a handy tool to monitor your celery workers.

links

https://www.revsys.com/tidbits/celery-and-django-and-docker-oh-my/
https://www.revsys.com/tidbits/brief-intro-docker-djangonauts/
https://docs.celeryq.dev/en/stable/django/first-steps-with-django.html#using-celery-with-django/