Per Wagner Nielsen's Blog

Asyncio basic II

2025-01-24T00:00:00+01:00

Tutorial on how to use Asyncio in a Jupyter Notebook

This story will show the basic of using Asyncio in a Jupyter Notebook

Prerequisites

Using Python 3.12. Need to install a few packages via pip along the way, but that will be obvious once we get to that.

The Problem with Notebooks and Asyncio

It is important to understand that when running jobs in asyncio, we are still running in the same thread. Asyncio tasks are functions that run concurrently but under the control of the same event loop. Jupyter already runs in an event loop, so that causes problems, and we have to write code slightly differently for code that runs in jupyter and code that doesn't.

Examples: Running Asyncio in Jupyter Notebooks

To effectively use asyncio in Jupyter Notebooks without relying on external libraries, you can use the built-in asyncio module. Below are examples to help you get started.

Example 1: Basic Asyncio Function

import asyncio

# Define an asynchronous function
async def say_hello():
    await asyncio.sleep(1)
    print("Hello, Asyncio!")

# Run the function using asyncio.run
asyncio.run(say_hello())

This example demonstrates a simple asynchronous function that prints a message after a delay. However, this approach will not work directly in Jupyter Notebooks due to the existing event loop.

Example 2: Using Asyncio in Jupyter Notebooks

To resolve the event loop conflict in Jupyter, you can use the asyncio module's event loop directly. Here's how:

import asyncio

# Define an asynchronous function
async def say_hello():
    await asyncio.sleep(1)
    print("Hello, Asyncio in Jupyter!")

# Get the current event loop or create a new one
loop = asyncio.get_event_loop()

# Run the function using the event loop
if loop.is_running():
    # If the loop is already running (e.g., in Jupyter), use create_task
    task = loop.create_task(say_hello())
else:
    # Otherwise, run the function normally
    loop.run_until_complete(say_hello())

This approach avoids external dependencies and works seamlessly in Jupyter Notebooks.

Example 3: Running Multiple Asyncio Tasks

You can also run multiple tasks concurrently using asyncio.gather:

import asyncio

# Define multiple asynchronous functions
async def task1():
    await asyncio.sleep(1)
    print("Task 1 completed")

async def task2():
    await asyncio.sleep(2)
    print("Task 2 completed")

# Get the current event loop or create a new one
loop = asyncio.get_event_loop()

# Run the tasks concurrently
if loop.is_running():
    # If the loop is already running (e.g., in Jupyter), use create_task
    tasks = [loop.create_task(task1()), loop.create_task(task2())]
else:
    # Otherwise, run the tasks normally
    loop.run_until_complete(asyncio.gather(task1(), task2()))

This example demonstrates how to run multiple tasks concurrently, showcasing the power of asyncio in handling asynchronous operations without relying on external libraries.

Conclusion

By using the built-in asyncio module, you can overcome the event loop conflict in Jupyter Notebooks and fully leverage the capabilities of asyncio. These examples should help you get started with writing and running asynchronous code in your notebooks.

References

Python Concurrency With Asyncio

Ryan V on youtube

Jupyterlab, Python3, asyncio – asynchronous tasks in a notebook background thread

nocomplexity.com

Colab notebook

Mini agent with Langgraph

2024-12-30T00:00:00+01:00

Tutorial on how to make a mini agent using langgraph

Prerequisites

I'm working on an Ubuntu 24 with a langgraph 0.2.60 for good measure. This is also not an introduction to LangChain or LangGraph and you are expected to have a basic understanding of these two libraries.

What is a "ReAct Agent"

No official definition of "agent" exists, but one can think of an Agent as a system that uses an LLM to reason through a problem and execute a solution. This could be a system that given a question has to determine if the system already has enough information to answer the question or if a specialized tool should be used to gather more information before answering the original question.

Example mini-agent

We will make a small mini-agent that can use one of two tools to find information about Star Wars characters. We can ask questions about characters, such as "How much older is Han Solo than Luke Skywalker?" and the agent will determine which tool to use, if any, and answer the question if the system has enough information available.

Some imports are needed:

from langchain_openai import ChatOpenAI
import requests

from langchain_community.tools import DuckDuckGoSearchRun
from langgraph.graph import MessagesState, START, StateGraph
from langchain_core.messages import HumanMessage, SystemMessage
from langchain_core.tools import tool
from langgraph.prebuilt import tools_condition, ToolNode
from IPython.display import Image, display

llm = ChatOpenAI(model="gpt-4o")

We will create 2 tools, one custom that calls a StarWars API and one search tool:

@tool
def get_character_info(id: int) -> dict:
    """Returns information about a specific star wars character

    Args:
        id: The ID in the StarWarsAPI for the character

        ids of characters:
        1: Luke Skywalker
        14: Han Solo

    Returns:
        dict with keys like: name, height, mass, hair_color, homeworld, starships (as a list)
        and others.
    """
    r = requests.get(f"https://swapi.info/api/people/{id}")
    return r.json()


search = DuckDuckGoSearchRun()

An example response from the api tool:

get_character_info.invoke({"id": 12})

{'name': 'Han Solo',
 'height': '180',
 'mass': '80',
 'hair_color': 'brown',
 'skin_color': 'fair',
 'eye_color': 'brown',
 'birth_year': '29BBY',
 'gender': 'male',
 'homeworld': 'https://swapi.info/api/planets/22',
 'films': ['https://swapi.info/api/films/1',
  'https://swapi.info/api/films/2',
  'https://swapi.info/api/films/3'],
 'species': [],
 'vehicles': [],
 'starships': ['https://swapi.info/api/starships/10',
  'https://swapi.info/api/starships/22'],
 'created': '2014-12-10T16:49:14.582000Z',
 'edited': '2014-12-20T21:17:50.334000Z',
 'url': 'https://swapi.info/api/people/14'}

We will then bind the tools to the llm, that way, the llm will make use of the tools during any invocations of the llm.

tools = [get_character_info, search]
llm_with_tools = llm.bind_tools(tools)

Next up is creating our LangGraph nodes, edges and prompts:

sys_message = SystemMessage(content="You are a useful assistant tasked with search and finding information about StarWars characters")

# Node
def reasoner(state: MessagesState):
    return {
        "messages": [llm_with_tools.invoke([sys_message] + state["messages"])]
    }

builder = StateGraph(MessagesState)

builder.add_node("reasoner", reasoner)
builder.add_node("tools", ToolNode(tools))

builder.add_edge(START, "reasoner")
builder.add_conditional_edges(
    "reasoner",
    tools_condition,
)
builder.add_edge("tools", "reasoner")
react_graph = builder.compile()

# display(Image(react_graph.get_graph(xray=True).draw_mermaid_png()))  # If you wanna visualize the graph

All we need now is to run the graph:

messages = [HumanMessage(content="What is the height of Luke Skywalker and how much older is he than Han Solo?")]
messages = react_graph.invoke({"messages": messages})

for m in messages["messages"]:
    m.pretty_print()

================================[1m Human Message [0m=================================

What is the height of Luke Skywalker and how much older is he than Han Solo?
==================================[1m Ai Message [0m==================================
Tool Calls:
  get_character_info (call_1zTTl9jvOrSeO0SUPD7OvbdR)
 Call ID: call_1zTTl9jvOrSeO0SUPD7OvbdR
  Args:
    id: 1
  get_character_info (call_hzbqd5v69htk9aBoK9wo2489)
 Call ID: call_hzbqd5v69htk9aBoK9wo2489
  Args:
    id: 14
=================================[1m Tool Message [0m=================================
Name: get_character_info

{"name": "Luke Skywalker", "height": "172", "mass": "77", "hair_color": "blond", "skin_color": "fair", "eye_color": "blue", "birth_year": "19BBY", "gender": "male", "homeworld": "https://swapi.info/api/planets/1", "films": ["https://swapi.info/api/films/1", "https://swapi.info/api/films/2", "https://swapi.info/api/films/3", "https://swapi.info/api/films/6"], "species": [], "vehicles": ["https://swapi.info/api/vehicles/14", "https://swapi.info/api/vehicles/30"], "starships": ["https://swapi.info/api/starships/12", "https://swapi.info/api/starships/22"], "created": "2014-12-09T13:50:51.644000Z", "edited": "2014-12-20T21:17:56.891000Z", "url": "https://swapi.info/api/people/1"}
=================================[1m Tool Message [0m=================================
Name: get_character_info

{"name": "Han Solo", "height": "180", "mass": "80", "hair_color": "brown", "skin_color": "fair", "eye_color": "brown", "birth_year": "29BBY", "gender": "male", "homeworld": "https://swapi.info/api/planets/22", "films": ["https://swapi.info/api/films/1", "https://swapi.info/api/films/2", "https://swapi.info/api/films/3"], "species": [], "vehicles": [], "starships": ["https://swapi.info/api/starships/10", "https://swapi.info/api/starships/22"], "created": "2014-12-10T16:49:14.582000Z", "edited": "2014-12-20T21:17:50.334000Z", "url": "https://swapi.info/api/people/14"}
==================================[1m Ai Message [0m==================================

Luke Skywalker is 172 cm tall. He was born in 19BBY, while Han Solo was born in 29BBY, making Han Solo 10 years older than Luke Skywalker.

So, what actually happend here?

Create tools

In the tools section we are creating two different tools, one custom and one buildin from langchain_community.tools. Tools in Langchain are essentially just function calls, eg. they are a way for LLMs to use functions. There are different ways to create tools in Langchain, in our example we simply made a function with a docstring. The docstring then describes what the tool does and is used by the language model to determine if and how to tool should be used.

Bind tools to LLM

with the method "bind_tools" on the llm model, we can add a list of different tools to the llm. This way, the llm is made aware of these tools and will use them if appropriate. If you just run the "llm_with_tools" and a query, the LLM will not actually run the tool, it will simply create the necessary arguments for the tool calling, and then it is up to the user/caller to actually invoke the tool.

r = llm_with_tools.invoke("Tell me about Luke Skywalker")

AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_nejexqzJTsIxoEhbJ202YS5l', 'function': {'arguments': '{"id":1}', 'name': 'get_character_info'}, 'type': 'function'}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 16, 'prompt_tokens': 168, 'total_tokens': 184, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-2024-08-06', 'system_fingerprint': 'fp_d28bcae782', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-5962e772-d42b-4838-bbde-6f307d9afb73-0', tool_calls=[{'name': 'get_character_info', 'args': {'id': 1}, 'id': 'call_nejexqzJTsIxoEhbJ202YS5l', 'type': 'tool_call'}], usage_metadata={'input_tokens': 168, 'output_tokens': 16, 'total_tokens': 184, 'input_token_details': {'audio': 0, 'cache_read': 0}, 'output_token_details': {'audio': 0, 'reasoning': 0}})

One can access the "tool_calls" property on the AIMessage returned from the llm_with_tools call to see which function should be called and with which arguments.

References

Sam Witteveen - Building a LangGraph ReAct Mini Agent

Asyncio basic I

2024-02-13T00:00:00+01:00

Tutorial on how to use Asyncio in Python

This story will show the basic of using Asyncio in Python

Prerequisites

Using Python 3.10. Need to install a few packages via pip along the way, but that will be obvious once we get to that.

Concepts

Concurrency, parallelism and multitasking

Concurrency is a term used for things that happen, or are in the proceess of happening, at the same time, but is says nothing about HOW they happen. Parallelism is a form of concurrency where 2 or more tasks take place at the same time. Multitasking (preemptive and cooperative multitasking): Preemptive multitasking is when we let the OS decided which process to execute via time slicing. Cooperative multitasking is when we explicitly in the code determine when a task can be paused in order to work on another task.

Process, Thread, Multithreading and Multitasking

A Process has it own memory space and CPU "time". Multiple processes running on a CPU with multiple cores (or on mulitple processors) is called multiprocessing.
A Thread shares memory with the process it is running on. Often thought of as a lightweight process. One process can have multiple threads, and will always have minimum one (the main thread). If multiple threads are spun up to do work concurrently, we are dealing with multithreading.

GIL (Global Interpreter Lock)

The GIL prevents a Python process from executing more than one Python bytecode instruction at any given time.

Coroutines and Tasks with async def and await

Coroutines

Coroutines are just python functions defined with "async def" instead of "def". The special sauce of a coroutine is that it can pause its execution when it encounters an operation that can take some time. When this operation is done, it can wake up and continue where it let go. While the coroutine is paused, other operations can be executed and run.
Executing a coroutine will not yields it result but rater return a coroutine object. This object must be put on an event loop to be executed.

import asyncio


async def my_coroutine():
    print('My coroutine')
    return 66


r = asyncio.run(my_coroutine())
print(r)

Which will produce:

My coroutine
66

In order for us to pause the execution of a coroutine (and do something else in the meantime), we have to use the await keyword. The await keyword will cause the coroutine following it to be run (like on the event loop). The await keyword will also pause the coroutine where it is contained in until the coroutine we awaited has finished and returned a result.

Even if we do something like this:

import asyncio


async def calculate(number: int) -> int:
    return number * 100


async def say_hello_eventually(seconds: int) -> str:
    await asyncio.sleep(seconds)
    return "Hello!"


async def main() -> None:
    message = await say_hello_eventually(5)
    calculated_number = await calculate(5)

    print(calculated_number)
    print(message)


asyncio.run(main())

we will not achieve concurrency. We would still be waiting 5 seconds for both functions to be completed and the results printed. This is because the await keyword pauses the current coroutine and wont run anything else until the coroutine has finished. Thus, "say_hello_eventually()" is run and the main() coroutine is paused while "say_hello_eventually()" finishes, then main() is resumed until "calculate()" is reached. Main() is then paused until "calculate()" has done it work, and then execution returns to main that prints the results.

In order to achieve real concurrency, we need to work with tasks.

Tasks

A task is a wrapper around a coroutine that schedule a coroutine to run on an event loop as soon as possible.

A task is created by using "asyncio.create_task(...)", which returns a task object. This task object can be put in an await expression that will extract the return value once it is complete.

import asyncio


async def calculate(number: int) -> None:
    print(number * 100) 


async def say_hello_eventually(seconds: int) -> None:
    await asyncio.sleep(seconds)
    print("Hello there")


async def main() -> None:
    message_task = asyncio.create_task(say_hello_eventually(3)) 
    calculate_task = asyncio.create_task(calculate(5))

    await message_task
    await calculate_task


asyncio.run(main())

This would immediately print the result of the "calculate_task" even though this task comes after the message_task in the main() function.

Cancelling tasks and timeouts

If you need to cancel a task, then you can call the task object's "cancel()" method. This will cause the task to raise a CancelledError when it is awaited.

await_for() is a asyncio function that takes a coroutine or a task object and a timeout specified in seconds. If the coroutine or task takes more than the given timeout to complete, a TimeoutException will be raised.

Futures

A future is a Python objects that contains a single value that you expect to get in the future but may not yet have. When a Future is created it typically doesnt have a value and is considered "unresolved", "incomplete" or "not done".

The Event Loop

The function "asyncio.run()" conviently creates an event loop and runs our tasks/coroutines. If we want to manually create an event loop, this can be done by:

import asyncio


async def main() -> None:
    print("Running main")
    await asyncio.sleep(2)

loop = asyncio.new_event_loop()

try:
    loop.run_until_complete(main())
finally:
    loop.close()
    print("Done")

If we want to access the current event loop (maybe bacause you are in a jupyter notebook), you can use the function "get_running_loop()" instead.

Examples

Async web calls

Lets say we wanna call multiple web sites and get the return values. In synchronious programming we would call them one at a time, which can be time consuming. It would be better to shoot off multiple calls at the same time and then just wait for the results to come in (web calls are I/O calls, not CPU calls). For this to work, we need to work with a async enabled web library. Requests is not async enabled, so instead we will use aiohttp.

import asyncio
import aiohttp


async def make_request(url: str) -> int:
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            return response.status


async def main():
    task_1 = asyncio.create_task(make_request("https://www.google.com"))
    task_2 = asyncio.create_task(make_request("https://www.facebook.com"))
    task_3 = asyncio.create_task(make_request("https://www.twitter.com"))

    r1 = await task_1
    r2 = await task_2
    r3 = await task_3

    print(r1, r2, r3)

asyncio.run(main())

References

Python Concurrency With Asyncio

Django on the cheap

2022-06-05T00:00:00+02:00

Story on how to setup Django as cheaply as possible.
NB: Still work in progress.

This story will show how to create a Django application and deploy it to a hosting provider as cheaply as I could possible do it.

Prerequisites

I will not be free, but my aim is to have a working solution that costs less than 10 USD pr. month.

This will not be a complete from zero to hero tutorial, rather I'll assume that you know enough basic Django to be able to apply the shown yourself.

Overview

Local setup

Create Django
Dockerize Django
Postgres
NGINX and gunicorn

Github and github actions

Github repo
Github actions to test

Hosting provider

Dokku
Push to deploy
Local and dev/prod specific files

Database

ElephantHQ
Update settings

Workes (rabbitmq)

RabbitMQ instance locally
Worker locally
RabbitMQ hosted
Workers hosted

Local Setup

Create local Django application

First step is to setup a local working Django application. We will setup a virtual environment to hold our python and Django dependencies.

$ virtualenv venv -p python3
created virtual environment CPython3.10.4.final.0-64 in 257ms
  creator CPython3Posix(dest=/django_on_the_cheap/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/username/.local/share/virtualenv)
    added seed packages: pip==22.0.4, setuptools==62.1.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
$ source venv/bin/activate
(venv) /django_on_the_cheap$ python --version
Python 3.10.4
(venv) /django_on_the_cheap$

In this case we are on Python 3.10 as I'm using an Ubuntu 22.04. Let's also install Django while we are at it:

$ pip install django
Collecting django
  Using cached Django-4.0.5-py3-none-any.whl (8.0 MB)
Collecting sqlparse>=0.2.2
  Using cached sqlparse-0.4.2-py3-none-any.whl (42 kB)
Collecting asgiref<4,>=3.4.1
  Using cached asgiref-3.5.2-py3-none-any.whl (22 kB)
Installing collected packages: sqlparse, asgiref, django
Successfully installed asgiref-3.5.2 django-4.0.5 sqlparse-0.4.2
WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the '/home/username/django_on_the_cheap/venv/bin/python -m pip install --upgrade pip' command.

We will ignore the pip upgrade for now as it is not super important.

We initialize a Django project:

$ django-admin startproject conf .

I call the project "conf" for the reason that django-admin will create a folder with this name holding files like the settings.py file. Thus, I find that calling the project "conf" or "config" makes sense and avoids some nameing confusion down the line. The final "." just means that we will not create an extra layer of folders that django does by default. I find that for smaller projects this makes it a bit easier to manage.

Let's migrate the database and spin up the development server:

$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
...
...
...
$ $ python manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
June 05, 2022 - 11:20:55
Django version 4.0.5, using settings 'conf.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

And if you go to the localhost url, you should see the django index page.

We will finally do a pip freeze in order to generate a requirements file with all our dependencies:

$ pip freeze > requirements.txt

We also need to prepare for using static files. Add this to the settings.py file in the config folder:

STATICFILES_DIRS = [
    BASE_DIR / "static",
]
STATIC_ROOT = BASE_DIR / "staticfiles"

Dockerize the application

Now we want to run our Django app in a Docker container, and, we also want to use docker-compose to manage this, as we will add more containers in the future, like Nginx and a database. Docker-compose just makes this easier to manage.

Make sure you have docker and docker-compose installed:

$ docker --version
Docker version 20.10.16, build aa7e414

$ docker-compose --version
docker-compose version 1.29.2, build 5becea4c

Create a docker-compose.yml file in the root of your project and add this:

version: '3.7'

services:
  django:
    build: 
      dockerfile: ./docker/local/Dockerfile
      context: .
    environment:
      - DEBUG=1
      - PYTHONDONTWRITEBYTECODE=1
      - PYTHONUNBUFFERED=1
      - PORT=8000
    ports:
      - 8000:8000
    volumes:
      - .:/app
    command: python manage.py runserver 0.0.0.0:8000

and create a folder called docker and within that folder, another folder called local. In there (in docker/folder) create a file called Dockerfile and put this in there:

FROM python:3.10

EXPOSE 8000

# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
ENV DEBUG 0

RUN apt-get update && apt-get upgrade -y && apt-get install -y \
  && apt-get clean

RUN pip install -U pip

# Install pip requirements
COPY . /app/
RUN python -m pip install -r /app/requirements.txt

# collect static files
WORKDIR /app
RUN python manage.py collectstatic --noinput

CMD python manage.py runserver

run

$ docker-compose up

and your application should be available on localhost:8000

Postgress database

We don't want to keep using the development sqlite database but rather use a Postgres database. This will also make it easier once we deploy to a production environment in the cloud. We do this in the docker-compose yml file.

Update the docker-compose.yml file to this:

version: '3.7'

services:
  db:
    image: postgres:12
    ports:
     - "5432:5432"
    environment:
     - POSTGRES_USER=postgres
     - POSTGRES_PASSWORD=12345
     - POSTGRES_DB=local_db
    volumes:
      - postgres_data:/var/lib/postgresql/data/

  django:
    build: 
      dockerfile: ./docker/local/Dockerfile
      context: .
    environment:
      - DEBUG=1
      - PYTHONDONTWRITEBYTECODE=1
      - PYTHONUNBUFFERED=1
      - PORT=8000
    ports:
      - 8000:8000
    volumes:
      - .:/app

volumes:
  postgres_data:

We are adding a postgres service and a volume to persist data. Do a new:

$ docker-compose up --build

We now have a running postgres database instance we can make Django connect to. Update the Database key in the settings.py file to:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'local_db', # set in docker-compose.yml as POSTGRES_DB
        'USER': 'postgres', # set in docker-compose.yml as POSTGRES_USER
        'PASSWORD': "12345", # set in docker-compose.yml as POSTGRES_PASSWORD
        'HOST': 'db', # set in docker-compose.yml as the service name
        'PORT': 5432 # default postgres port as the port number
    }
}

We also need to add postgres to our dependencies. Do this by adding psycopg2-binary to the requirements.txt file and run a docker-compose up --build again.

Now, however, we are getting this output:

django_1  | You have 18 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
django_1  | Run 'python manage.py migrate' to apply them.

as we are using a new database it hasn't been migrated yet. For now, we do this manually by logging in to the django docker container and manually run migrate.

$ docker-compose run django bash
Creating django_on_the_cheap_django_run ... done
root@49efabfd6d54:/app# python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
...
...
...

If you try and go to localhost:8000 now, you will first get a warning relating to missing ssl (https), and second, if you bypass this warning, an error like:

Invalid HTTP_HOST header: '0.0.0.0:8000'. You may need to add '0.0.0.0' to ALLOWED_HOSTS.

So, we will update our ALLOWED_HOSTS in the settings.py file to include localhost.

ALLOWED_HOSTS = ["localhost", "0.0.0.0"]

Now you should be able to successfully load the django index page.

Gunicorn

For now we have used the development server that comes with Django. But we can't use this in production, so we wanna use something else, and in this case Gunicorn. So lets first add it to our requirements and rebuild our images.

requirements.txt

asgiref==3.5.2
Django==4.0.5
sqlparse==0.4.2
psycopg2-binary
gunicorn

and run docker-compose up --build

Update the Dockerfile to use the gunicorn instead of the development server:

WORKDIR /app
RUN python manage.py collectstatic --noinput

CMD gunicorn conf.wsgi --bind 0.0.0.0:8000

Nginx

Now, if you try and go to http://0.0.0.0:8000/admin you will notice that the static assets (css, js) are missing. We need to have nginx setup to serve these. Let's add nginx as a service to our docker-compose.yml file.

  nginx:
    build: 
      dockerfile: ./docker/local/nginx/Dockerfile
      context: .
    volumes:
      - static_volume:/app/staticfiles      
    ports:
      - 8000:80
    depends_on:
      - django
    networks:
      - nginx_network

We are going to make a few changes to both our Dockerfile and docker-compose.yml in order to setup nginx. The full docker-compose.yml should look like:

version: '3.7'

services:
  db:
    image: postgres:12
    ports:
     - "5432:5432"
    environment:
     - POSTGRES_USER=postgres
     - POSTGRES_PASSWORD=12345
     - POSTGRES_DB=local_db
    volumes:
      - postgres_data:/var/lib/postgresql/data/

  nginx:
    build: 
      dockerfile: ./docker/local/nginx/Dockerfile
      context: .
    volumes:
      - static_volume:/app/staticfiles      
    ports:
      - 8000:80
    depends_on:
      - django
    networks:
      - nginx_network

  django:
    build: 
      dockerfile: ./docker/local/Dockerfile
      context: .
    environment:
      - DEBUG=1
      - PYTHONDONTWRITEBYTECODE=1
      - PYTHONUNBUFFERED=1
      - PORT=8000
    expose:
      - 8000
    volumes:
      - .:/app
      - static_volume:/app/staticfiles
    depends_on:
      - db
    networks:
      - nginx_network

volumes:
  postgres_data:
  static_volume:

networks: 
  nginx_network:
    driver: bridge

First: We are adding the nginx service.
Second: We are creating a static_volume volume to hold static assets that should be served by nginx.
Third: We are explicitly adding a network, the nginx_network (this is not strictly necessary).

We will also add a nginx config file which we copy into the nginx docker container. In the docker/local folder make a subfolder called "nginx", in here, we will have two files: Dockerfile and nginx.conf.
The Dockerfile should look like:

FROM nginx:1.21-alpine

RUN rm /etc/nginx/conf.d/default.conf
COPY ./docker/local/nginx/nginx.conf /etc/nginx/conf.d

and the nginx.conf:

upstream django_on_the_cheap {
    server django:8000;
}

server {

    listen 80 default_server;
    server_name _;

    location / {
        proxy_pass http://django_on_the_cheap;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
    }

    location /static/ {
        alias /app/staticfiles/;
    }

}

We will make a minor update to the docker/local/Dockerfile as well, just to clean up a bit:

FROM python:3.10

ENV HOME=/
ENV APP_HOME=/app

# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
ENV DEBUG 0

RUN apt-get update && apt-get upgrade -y && apt-get install -y \
  && apt-get clean

RUN pip install -U pip

# Install pip requirements
COPY . $APP_HOME/
RUN python -m pip install -r $APP_HOME/requirements.txt

WORKDIR $APP_HOME/

CMD ["gunicorn", "--bind", "0.0.0.0:8000", "conf.wsgi:application"]

Now, do a docker-compose up --build and your website should show including the static assets.

Local setup conclusions

We now have a fully working Django setup running a production like infrastructure.

Github and github actions

Github repo

Lets make our folder into a git repository and make a commit.

First, we create a .gitignore file in our root folder to ensure we don't commit stuff we don't want to:

venv/
staticfiles/

*.pyc
__pycache__/
db.sqlite3

now do:

$ git init
$ git status
On branch master

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    .gitignore
    conf/
    docker-compose.yml
    docker/
    manage.py
    requirements.txt

nothing added to commit but untracked files present (use "git add" to track)

All looks well, so lets add and commit

$ git add .
$ git commit -m "Initial commit with working local docker setup"
[master (root-commit) 0989c45] Initial commit with working local docker setup
 12 files changed, 308 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 conf/__init__.py
 create mode 100644 conf/asgi.py
 create mode 100644 conf/settings.py
 create mode 100644 conf/urls.py
 create mode 100644 conf/wsgi.py
 create mode 100644 docker-compose.yml
 create mode 100644 docker/local/Dockerfile
 create mode 100644 docker/local/nginx/Dockerfile
 create mode 100644 docker/local/nginx/nginx.conf
 create mode 100755 manage.py
 create mode 100644 requirements.txt

Make sure you have created an empty Github repository, then add this repository and push:

$ git remote add origin git@github.com:perwagner/django_on_a_dime.git
$ git push -u origin master
Enumerating objects: 18, done.
Counting objects: 100% (18/18), done.
Delta compression using up to 8 threads
Compressing objects: 100% (16/16), done.
Writing objects: 100% (18/18), 4.24 KiB | 4.24 MiB/s, done.
Total 18 (delta 1), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (1/1), done.
To github.com:perwagner/django_on_a_dime.git
 * [new branch]      master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.

Github action and tests

Hosting Provider

Vultr

create new user on the vps:

$ sudo adduser username

$ sudo usermod -aG sudo username

on you local do this to setup ssh key:

ssh-copy-id username@123.321.123.32

If you get tired of remembering the IP address, you can add this to your host file (/etc/hosts):

$ sudo nano /etc/hosts

123.321.123.32     dokku

and now you can login to your server with:

$ ssh username@dokku

Dokku on VPS

$ sudo apt-get update
$ sudo apt-get upgrade

$ wget https://raw.githubusercontent.com/dokku/dokku/v0.28.1/bootstrap.sh
$ sudo DOKKU_TAG=v0.30.1 bash bootstrap.sh

Next, we add our public key from the .ssh folder

$ echo "your-public-key-contents-here" | sudo dokku ssh-keys:add admin
$ dokku domains:set-global 123.321.123.32

Now, login to the dokku server and create an app:

$ dokku apps:create myapp

and lets also allow the firewall to let through http trafic:

$ sudo ufw allow http

In your local repository, you can now add your server as a remote:

$ git remote add production dokku@123.321.123.32:myapp
$ git push production

"production" is the name of the remote. It is important that the username is "dokku", otherwise it won't work.

Set the domain for your app so it will serve on port 80 by default:

$ dokku domains:set myapp 123.321.123.32

References

Github repo for django on a dime

Wiremock on Github with Github Actions

2022-04-14T00:00:00+02:00

Story on how to use Wiremock to run CI on Github with Github Actions

This story setup a basic Django project using Docker and Docker Compose and then use Wiremock to run tests against a stubbed external API, both locally and on Github via Github Actions.

Prerequisites

I won't be showing everything in detail here but rather focus on what is relevant for this story, eg. docker-compose and github actions.

This will not be a complete from zero to hero tutorial, rather I'll assume that you know enough basic Django and Docker to be able to apply the shown yourself.

requirements.txt

asgiref==3.5.0
attrs==21.4.0
backports.zoneinfo==0.2.1
certifi==2021.10.8
charset-normalizer==2.0.12
Django==4.0.4
gunicorn==20.1.0
idna==3.3
iniconfig==1.1.1
packaging==21.3
pluggy==1.0.0
py==1.11.0
pyparsing==3.0.8
pytest==7.1.1
pytest-django==4.5.2
requests==2.27.1
sqlparse==0.4.2
tomli==2.0.1
urllib3==1.26.9

docker-compose.yml

version: '3.7'

services:
  django:
    restart: always
    build: 
      dockerfile: ./docker/Dockerfile
      context: .
    environment:
      - DEBUG=1
      - PYTHONDONTWRITEBYTECODE=1
      - PYTHONUNBUFFERED=1
      - ENV=DOCKER_LOCAL
      - PORT=8000
    ports:
      - 8000:8000
    volumes:
      - .:/app
    command: /usr/local/bin/gunicorn config.wsgi:application -w 2 -b :8000 --reload

  wiremock:
    image: wiremock/wiremock
    ports:
      - "8080:8080"
      - "8443:8443"
    volumes:
      - ./wiremock:/home/wiremock

Dockerfile

FROM python:3.8

ARG APP_USER=appuser
RUN groupadd -r ${APP_USER} && useradd --no-log-init -r -g ${APP_USER} ${APP_USER}

EXPOSE 8000
WORKDIR /app

ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
ENV DEBUG 0

RUN apt-get update && apt-get install
RUN apt-get install -y \
  libpq-dev \
  libmariadb-dev-compat \
  libmariadb-dev \
  gcc git \
  && apt-get clean

RUN pip install -U pip

COPY ./requirements.txt /app/requirements.txt
RUN python -m pip install -r /app/requirements.txt
COPY . /app/

RUN mkdir -p /app/media
RUN mkdir -p /app/staticfiles

RUN chown -R ${APP_USER}:${APP_USER} /app/media
RUN chown -R ${APP_USER}:${APP_USER} /app/staticfiles

.github/workflows/push.yml

name: Python application

on:
  push:
    branches: [ master ]

jobs:
  build_and_test:
    runs-on: ubuntu-latest
    services:
      wiremock:
        image: wiremock/wiremock
        ports:
          - 8080:8080

    steps:
    - uses: actions/checkout@v3
    - name: Set up Python 3.8
      uses: actions/setup-python@v3
      with:
        python-version: 3.8
    - name: curl
      run: sudo apt-get install -y curl
    - name: import stubs to wiremock
      run: curl -v -d @./wiremock/mappings/externalapi-stubs.json http://localhost:8080/__admin/mappings/import
    - name: install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
    - name: Run migrations
      run: python manage.py migrate
      env:
        ENV: GITHUB_ACTION_CI
    - name: Run tests with pytest
      run: pytest
      env:
        ENV: GITHUB_ACTION_CI

pytest.ini

[pytest]
DJANGO_SETTINGS_MODULE = config.settings
# -- recommended but optional:
python_files = tests.py test_*.py *_tests.py

company/tests.py

from django.test import TestCase
import requests


def test_get_the_api_response():
    url = "http://localhost:8080/helloworld"
    r = requests.get(url)

    assert r.status_code == 200
    assert r.text == "hello world"

wiremock/mappings/externalapi-stubs.json

{
  "mappings" : [  {
    "id" : "f5d37aa8-e436-491a-8e80-570f55b34d14",
    "name" : "hello world",
    "request" : {
      "url" : "/helloworld",
      "method" : "GET"
    },
    "response" : {
      "status" : 200,
      "body" : "hello world",
      "headers" : { }
    },
    "uuid" : "f5d37aa8-e436-491a-8e80-570f55b34d14",
    "persistent" : true,
    "priority" : 5
  } ],
  "meta" : {
    "total" : 1
  }
}

References

wiremock

Deploy Django to Heroku using Docker

2021-10-20T00:00:00+02:00

Story on how to use Docker and Docker Compose and deploy a Django app to Heroku

This story setup a basic Django project using Docker and Docker Compose and then deploy said application to Heroku.

Prerequisites

I'm working on an Ubuntu 20.04, Django 3.1.3.

This will not be a complete from zero to hero tutorial, rather I'll assume that you know enough basic Django and Docker to be able to apply the shown yourself.

Setup Django in Docker

Create requirements file

In a fresh directory, lets call it "lazinda", create a file named requirements.txt and add just one line:

Django>=3.2.8,<3.3

We are telling PIP that we want from 3.2.8 until 3.3; This way, we will include any security patches that might come out but not any updates that changes the API.

Dockerfile
Create a file called just Dockerfile:

touch Dockerfile

which will hold our definition of our Docker image. Add the following to the Dockerfile:

FROM python:3.10-alpine3.14
LABEL maintainer="myself"

ENV PYTHONUNBUFFERED 1

COPY ./requirements.txt /requirements.txt
COPY ./app /app

References

Deploying Django with Docker Compose on YouTube

K Nearest Neighbor

2021-04-20T00:00:00+02:00

We will look into the basics of K Nearest Neighbor

Prerequisites

We will be using Jupyter lab, so make sure you have that installed

What is KNN (K Nearest Neighbor)

KNN is a classification algorithm meaning we are trying to determine if an instance belongs to a specific group or not. For example, if a person will make a purchase or not, if a stock price will go up or not, if a student til pass a test or not; We are not trying to determine the magnitude of anything, like, predicting how much rain will fall tomorrow or what the price of something should be.

Imagine we have a sample of people looking to buy a car, some will buy, others will not. The data looks something like:

One way to figure out if someone will buy a car? We can look at people who are 'close' to someone who is also buying a car in terms of the number of kids they have. If we look at individual 3, who has 2 kids, and look at the 2 nearest neighbors, one with one kid and one with 3 kids, they both also bought cars, but if we look at the 3 nearest neighbors, then we also include the 2 people with zero kids, who didn't buy cars.

Another way to illustrate KNN is to like this:

If we look at the 3 nearest neighbors we will predict purple for our white ball, but if we take k = 5, then the prediction would be green.

References

KNN lazy programmer

Nmist dataset

Logistic Regression 1

2021-04-01T00:00:00+02:00

How does logistic regression work in Python. This is the story.

We are going to look into the theory and application of logistic regression. This story will follow the outline of the course from lazy programmer here: https://lazyprogrammer.me/deep-learning-courses/.

Theory

We are still using the formula for from linear regression $y=ax+b$, but in machine learning we often use different wording than what we typically do in statistics.

$$(x,y) -> (x_1, x_2) = \textbf{x}$$

The constant term is renamed to $w_1$ and the bias/intercept to $w_0$, so we get:

$$h(\textbf{x})=w_0 + w_1x_1 + w_2x_2$$

making h() a linear combination of the components of x. In vector form we write $h(\textbf{x})=\textbf{w}^t\textbf{x}$

Calculate the output of a neuron

The sigmoid function we will use: $\sigma(z) = \frac{1} {{1+e^{-z}}}, \in{(0,1)}$, which can be written as: $\sigma(\textbf{w}^t\textbf{x})$

To do the logistic regression in Python, we do this:

import numpy as np

N = 100
D = 2

X = np.random.randn(N, D)  # The dataset
ones = np.array([[1]*N]).T  # the bias/intercept
Xb = np.concatenate((ones, X), axis=1)  # just the dataset as a N x 3 matrix

w = np.random.randn(D + 1)  # weights, must be a 3 x 1 matrix

# Calculate the dot product between x and w
z = Xb.dot(w)  # Will be a (Nx3) * (3x1) = N x 1 matrix

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

print(sigmoid(z))

Example with ecommerce data

Example data

The data looks something like this:

import numpy as np
import pandas as pd

df = pd.read_csv('ecommerce_data.csv')
df.head()

The 'visit_duration' is a real number in minutes of how long the user was on the site.
'time_of_day" is a category label:
0: from 0 to 6
1: from 6 to 12
2: from 12 to 18
3: from 18 to 24

We will do 'one hot encoding' for this category:

0-6 /	6-12 /	12-18 /	18-24
1	0	0	0
0	1	0	0
0	0	1	0
0	0	0	1

User action is the action we want to predict:
0: bounce
1: add_to_cart
2: begin_checkout
3: finish_checkout

Normalize

Scale and distance are important so we need to normalize the n_products_viewed and the visitor_duration numbers, which can be done according to the formula $Z= \frac{(X-\mu)} {\sigma}$

Pre-process data

Normalize numerical columns

We have 2 numerical columns that we want to normalize, the first and second ones. We will create a function to do this:

def normalize_row(row):
    """Takes a row and returns a normalized row"""
    return (row - row.mean()) / row.std()

We then create a new X matrix with the first 2 rows normalized:

data = df.to_numpy()  # makes the dataframe into a numpy array

X = data[:, :-1]  # All the X variables
Y = data[:, -1]  # The dependent Y variable

# Normalize rows 1 and 2 in place
X[:,1] = normalize_row(X[:,1])
X[:,2] = normalize_row(X[:,2])

The we do the one how encoding for the time dimension:

## The 'time of day' category column. We want to use the one-hot encoding.
N, D = X.shape  # D = 5 as there are 5 columns in X
X2 = np.zeros((N, D+3))  # D + 3 = 5 + 3=8; We will have the existing 4 categories + 4 'one hot encodings' for the 'time_of_day' column
X2[:, 0:(D-1)] = X[:, 0:(D-1)]  # X2's first 4 columns just contains the original 4 X columns

# The one hot encoding
for n in range(N):
    time_of_day = int(X[n, D-1])  # D-1=4, the 4th column in the original X matrix is the 'time_of_day' column (we count from zero)
    X2[n,time_of_day + D-1] = 1  # time_of_day + (D-1) = the category used in the one hot encoding; and we set that == 1

Additionally, in logistic regression, we are only operating with 2 'categories' for the dependent variable (Y), in our case 'bounce' or 'add_to_card', which have values 0 and 1 respectively. We filter our X2 and Y variable to only include rows with these two Y categories:

X2 = X[Y <=1 ]
Y = Y[Y <= 1]

Make prediction

We start by creating the weights and bias for our predictive model. The weights will initially just be random numbers and the bias is set to zero:

D = X2.shape[1]  # Getting the number of columns in our X2. We will use this to create our initial weights
W = np.random.randn(D)  # Our weights matrix, just random numbers
b = 0  # Bias term.

We will create 2 helper functions to assist us with our prediction, a sigmoid function and a prediction function.

Sigmoid function

def sigmoid(a):
    return 1/(1 + np.exp(-a))

Prediction function

def forward(X, W, b):
    return sigmoid(X.dot(W) + b)

We make the prediction:

P_Y_given_X = forward(X, W, b)
predictions = np.round(P_Y_given_X)

The classification rate function:

def classification_rate(Y, P):
    return np.mean(Y == P)

Running all this, we can get something like this (depending on your random weights):

print("Score:", classification_rate(Y, predictions))
Score: 0.30402010050251255

Training and solving for optimal weights

The cross entropy error function

What do we want out of an error function for a logistic error function: * Should be 0 if there is no error * > 0 if there is an error * Magnitude means higher cost

The cross-entropy error function is:

$$J = -{ t * log(y) + (1-t) * log(1-y)}$$

Where:
J: is the error
t: is the target
y: is the prediction, or the output from the logistic regression

Lets try and set t = 1:

$$J = -{ t * log(y) + (1-t) * log(1-y)}$$

$$J = -{ 1 * log(y) + (1-1) * log(1-y)} <=>$$

$$J = -{ log(y) }$$

So only the first term really matter in this case. If we set t=0, we get:

$$J = -{ t * log(y) + (1-t) * log(1-y)}$$

$$J = -{ 0 * log(y) + (1-0) * log(1-y)} <=>$$

$$J = -{ log(1-y) }$$

So only the 2nd term matter.

If we try a few examples this might make more sense.
With t=1 and y=1 (our real value (target) was 1 and the prediction was also one), then our error becomes:

$$J = -{ log(1) } = 0$$

Which intuitively makes sense, we hit the target so the error becomes zero. The same thing happens if we have t=0 and y = 0.

What if our prediction is just a bit off? say t=1 but y=0.9? Then we get:

$$J = -{ 1 * log(0.9) } = -(-0.11) = 0.11$$

So a relatively small error.

What if we have a large difference between t and y, say t = 1 and y = 0.2?

$$J = -{ 1 * log(0.27) } = -(-1.31) = 1.31$$

Giving us a much larger error.

For the calculations we typically want the total cross entropy error which we get by summing over all the errors:

$$J = -\sum_{n=1}^N( t * log(y) + (1-t) * log(1-y))$$

Making a function

We are gonna make use of the fact that the cross entropy error function simplifies for t = 0 and t = 1. We can then do this:

def cross_entropy_error(T, Y):
    """Calculate the cross entropy error.
    T: Target
    Y: Prediction
    """
    N = T.shape[0]
    E = 0
    for i in range(N):
        if T[i] == 1:
            E -= np.log(Y[i])
        else:
            E -= np.log(1 - Y[i])

    return E

Calculation example in code

N = 100
D = 2

X = np.random.randn(N, D)  # Random Xs with same mean and variance

X[:50, :] = X[:50, :] - 2 * np.ones((50, D))  # make half 2*1 lower than mean
X[50:, :] = X[50:, :] + 2 * np.ones((50, D))  # make other half 2*1 higher than mean

T = np.array([0] * 50 + [1] * 50)  # Target, first 50 are 50 then next 50 one. 

# Create X matrix with constants term
ones = np.array([[1] * N]).T
Xb = np.concatenate((ones, X), axis=1)

# Randomly initialize the weights
w = np.random.randn(D + 1)

# Calculate model output
z = Xb.dot(w)
Y = sigmoid(z)

print(cross_entropy_error(T, Y))
158.89255979864652

Let's try and visualize this solution. We first import matplotlib:`

import matplotlib.pyplot as plt

# The closed form solution
w = np.array([0, 4, 4])  # bias is zero

# The line: y = -x
plt.scatter(X[:,0], X[:,1], c=T, s=100, alpha=0.5)  # c=T for the colors
x_axis = np.linspace(-6, 6, 100)
y_axis = -x_axis
plt.plot(x_axis, y_axis)
plt.show()

Cross entropy error and log likelyhood

An interesting consequense of having the cross entropy error function the way we have it is that maximizing the log likelihood of our sigmoid function is the same as minimizing the cross entropy error function.

(Will not show the math here though)

Gradient decent

Theory

The basic idea of gradient decent is that we take small steps in the direction of the derivative. So we update our weights with a small step size (choosen by us) and rerun until our derivative is so close to zero as we are comfortable with.

How to choose the learning rate? No scientific way, just try.

$$J = -\sum_{n=1}^N( t_n * log(y_n) + (1-t_n) * log(1-y_n))$$

The derivative becomes:

$$\frac{\delta J}{\delta y_n} = -t_n \frac{1}{y_n} + (1-t_n) \frac{1}{1-y_n} (-1)$$

The derivative of the sigmoid becomes:

$$y_n = \sigma(a_n) = \frac{1}{1 + e^{-a_n}}$$

$$\frac{\delta y_n}{\delta a_n} = \frac{-1}{(1+e^{-a_n})^{2}}(e^{-a_n}) (-1)=y_n(1-y_n)$$

The deriviative of a with respect to w (the activation with regards to the weights):

$$a_n = w^t x_n$$

$$a_n = w_0 x_{n_0} + w_1 x_{n_1} + w_2 x_{n_2} + ...$$

$$\frac{\delta a_n}{\delta w_i} = x_{n_i}$$

Putting it all together:

$$\frac{\delta J}{\delta w_i} = \sum_{n=1}^N (y_n - t_n) x_{n_i}$$

In vector form:

$$\frac{\delta J}{\delta w} = X^T (Y - T)$$

In code

N = 100
D = 2

X = np.random.randn(N, D)  # Random Xs with same mean and variance

X[:50, :] = X[:50, :] - 2 * np.ones((50, D))  # make half 2*1 lower than mean
X[50:, :] = X[50:, :] + 2 * np.ones((50, D))  # make other half 2*1 higher than mean

T = np.array([0] * 50 + [1] * 50)  # Target, first 50 are 50 then next 50 one. 

# Create X matrix with constants term
ones = np.array([[1] * N]).T
Xb = np.concatenate((ones, X), axis=1)

# Randomly initialize the weights
w = np.random.randn(D + 1)

# Calculate model output
z = Xb.dot(w)
Y = sigmoid(z)

learning_rate = 0.1
# Do 100 iterations of gradient decent
for i in range(100):
    if i % 10 == 0:
        ce_error = cross_entropy_error(T, Y)
        print(f"Cross entropy error: {ce_error}")

    w += learning_rate * np.dot((T - Y).T, Xb)
    Y = sigmoid(Xb.dot(w))


print(f"Final w: {w}")

References

https://lazyprogrammer.me/deep-learning-courses/

Ecommerce data

List of latex symbols

Django logging

2021-03-28T00:00:00+01:00

Tutorial on how to use logging with Django.

Prerequisites

I'm working on an Ubuntu 20.04, Django 3.1.1 and django-extensions 3.0.9 for good measure.

This will not be a complete from zero to hero tutorial, rather I'll assume that you know enough basic Django to be able to apply the shown yourself. I'll also assume that you have a barebone Django project setup, in my examples it will be called 'config' thus generating the settings file in the config.settings.py folder.

Logging flow in Django

Below is a rough graph of how a log message (log record) flows from logger to output (via a formatter).

Logging in Django

Lets setup a view with an url:

from django.http import HttpResponse
import datetime


def index(request):
    now = datetime.datetime.now()
    html = "<html><body>It is now %s.</body></html>" % now
    return HttpResponse(html)

If we reload the view, we get:

Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Warning log message
Error log message
Critical log message
[28/Mar/2021 17:28:54] "GET / HTTP/1.1" 200 63

Only the Warning, Error and Critical log levels are showing. So how do we get the other ones to show?
We could manually set the log level in the module:

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

and reloading:

Quit the server with CONTROL-C.
DEBUG:pages.views:Debug log message
INFO:pages.views:Info log message
WARNING:pages.views:Warning log message
ERROR:pages.views:Error log message
CRITICAL:pages.views:Critical log message
[28/Mar/2021 18:05:36] "GET / HTTP/1.1" 200 63

However, this is not how we would typically do in Django; In Django, we can use the LOGGING settings in the config.settings file.

Naming of loggers

In the example above, we used the __name__ for naming the logger; We don't have to. The name itself is used to get (or create) a logger and is also used to identify the logger in the configuration. Another way of naming a logger is by using dot notation, for example:

logger = logging.getLogger('config.pages.html')

which gives a slightly different logging output:

Django version 3.1.7, using settings 'config.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
DEBUG:config.pages.html:Debug log message
INFO:config.pages.html:Info log message
WARNING:config.pages.html:Warning log message
ERROR:config.pages.html:Error log message
CRITICAL:config.pages.html:Critical log message
[31/Mar/2021 07:26:45] "GET / HTTP/1.1" 200 63

The main feature of using dot notation is that it can be used to propagage logging calls 'up the tree'.

logging in settings.py

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
        },
    },
    'root': {
        'handlers': ['console'],
        'level': 'WARNING',
    },
}

I honestly don't really know what function the 'version' keyword has; I have ever only seen it with a 1, but you get an error message if it is missing.

The disable_existing_loggers is important to remember. If you miss it, it will automatically be set to True and then disable all the build in django loggers, which is rarely what you want.

We define one handler called 'console' and assign it the class 'logging.StreamHandler', which will output everything to the console. The 'root' is the name of a logger which is the 'top level' logger that collects all log messages. We set it's level to 'WARNING', meaning, it will only ouput log messages with levels higher than or equal to "WARNING":

Django version 3.1.7, using settings 'config.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Warning log message
Error log message
Critical log message
[31/Mar/2021 07:46:14] "GET / HTTP/1.1" 200 63

See, no INFO or DEBUG messages.

If we remove the root logger and just have:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
        },
    },
}

but set DEBUG = False and get the logger named 'django', we will see that Django doesn't log out DEBUG level log messages. 'django' is a special named logger in the django logging hierarchy.

DEBUG = False
# logging.basicConfig(level=logging.DEBUG)
# logger = logging.getLogger(__name__)
logger = logging.getLogger('django')

If instead, we set DEBUG = True, we get all log messages with log level INFO or above (so no DEBUG level log messages):

Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Info log message
Warning log message
Error log message
Critical log message
[31/Mar/2021 09:07:49] "GET / HTTP/1.1" 200 63

If we want to include DEBUG level log messages, we have to overwrite the configuration for the django logger:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['console'],
            'level': 'DEBUG',
        },
    },
}

But, this will display a log of logging, so I won't show that here.

Log output options

We can log to different places, not just the console, but also files. This is done by adjusting the handler to use a different type of handler:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
        },
        'file': {
            'level': 'WARNING',
            'class': 'logging.FileHandler',
            'filename': BASE_DIR / 'filelog.log',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'INFO',
        },
    },
}

This way, we are defining a new handler, the 'file' with a logging.FileHandler class and a 'filename'; We then attach it to the django logger. A file will now be created with the 'filename':

Warning log message
Error log message
Critical log message

The Python standard logging module defines several log handlers, like FileHandler and StreamHandler Logging Handlers, and one additional Django specific handler: AdminEmailHandler.

Formatters

Formatters format the output and can be handy when you want to include timestamps or loglevel information for example. We can create a formatter like this:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,

    'formatters': {
        'verbose': {
            'format': '{levelname} {asctime} {module} {message}',
            'style': '{',
        },
    },

    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'formatter': 'verbose'
        },
    },
    'loggers': {
        'django': {
            'handlers': ['console'],
            'level': 'INFO',
        },
    },
}

And the output would now look like:

Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
INFO 2021-03-31 11:17:29,346 views Info log message
WARNING 2021-03-31 11:17:29,346 views Warning log message
ERROR 2021-03-31 11:17:29,346 views Error log message
CRITICAL 2021-03-31 11:17:29,347 views Critical log message
INFO 2021-03-31 11:17:29,347 basehttp "GET / HTTP/1.1" 200 63

More information on how to 'format' formatters can be found here: Formatting formatters

References

django logging official documentation

Using Sqlite with django and zappa

2021-03-28T00:00:00+01:00

Story on how to use Sqlite as a database when deploying Django with Zappa

Prerequisites

I'm working on an Ubuntu 20.04, Django 3.1.1 and zappa 0.5.2 and django-extensions 3.0.9 for good measure.

I'm also assuming that you are setup with an AWS account and have a aws configuration file on you local machine.

This will not be a complete from zero to hero tutorial, rather I'll assume that you know enough basic Django to be able to apply the shown yourself.

Setup

Create a virtualenv and install the necessary packages:

virtualenv venv -p python3
source venv/bin/activate
pip install django zappa django_extensions
django-admin startproject config .
zappa init

Answer the questions from zappa to create the settings file (zappa_settings.json):

{
    "dev": {
        "django_settings": "config.settings",
        "profile_name": "default",
        "project_name": "django-zappa-sq",
        "runtime": "python3.8",
        "s3_bucket": "zappa-sqlite-story"
    }
}

Lets try and run it locally:

python manage.py migrate
python manage.py runserver

And it should work just fine.

First deploy to AWS

Lets just try and deploy the system as is to AWS:

zappa deploy dev

You might get this error:

...
botocore.exceptions.NoRegionError: You must specify a region.

so we will have to specify a region in our settings file. Update to:

{
    "dev": {
        "django_settings": "config.settings",
        "profile_name": "default",
        "project_name": "django-zappa-sq",
        "runtime": "python3.8",
        "s3_bucket": "zappa-sqlite-story",
        "aws_region": "eu-central-1"
    }
}

Lets try and deploy again:

(venv) $:~/github/django_zappa_sqlite_blogstory$ zappa deploy dev
Calling deploy for stage dev..
Creating django-zappa-sq-dev-ZappaLambdaExecutionRole IAM Role..
Creating zappa-permissions policy on django-zappa-sq-dev-ZappaLambdaExecutionRole IAM Role.
Downloading and installing dependencies..
 - pyyaml==5.4.1: Downloading
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 662k/662k [00:00<00:00, 6.79MB/s]
Packaging project as zip.
Uploading django-zappa-sq-dev-1616928259.zip (16.4MiB)..
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 17.1M/17.1M [00:01<00:00, 9.69MB/s]
Scheduling..
Scheduled django-zappa-sq-dev-zappa-keep-warm-handler.keep_warm_callback with expression rate(4 minutes)!
Uploading django-zappa-sq-dev-template-1616928273.json (1.6KiB)..
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.66k/1.66k [00:00<00:00, 25.3kB/s]
Waiting for stack django-zappa-sq-dev to create (this can take a bit)..
 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎                                           | 3/4 [00:09<00:03,  3.12s/res]
Deploying API Gateway..
Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.
(venv) $:~/github/django_zappa_sqlite_blogstory$

So, we got a little further, but still no luck. The problem is that sqlite doesn't work with aws lambda.

Sqlite with lambda

We need to install another package, django-s3-sqlite:

pip install django-s3-sqlite

and add it to our installed apps in config/settings.py and update the DATABASES settings as well:

ALLOWED_HOSTS = ["*"]

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]
INSTALLED_APPS += ["django_s3_sqlite"]

...
...
...

DATABASES = {
    "default": {
        "ENGINE": "django_s3_sqlite",
        "NAME": "sqlite_zappa.db",
        "BUCKET": "zappa-sqlite-story",
    }
}

Additionally, we need to download a binary to our root django folder (the project folder where our manage.py file lives). The file is called _sqlite.so and is found in the https://github.com/FlipperPA/django-s3-sqlite/tree/master/shared-objects directory. So, copy the file to your project folder.

Additionally, add this line to your zappa settings file: "use_precompiled_packages": false,, so the file becomes:

{
    "dev": {
        "django_settings": "config.settings",
        "profile_name": "default",
        "project_name": "django-zappa-sq",
        "runtime": "python3.8",
        "s3_bucket": "zappa-sqlite-story",
        "aws_region": "eu-central-1",
        "use_precompiled_packages": false
    }
}

Update again

Lets try and update our dev environment on AWS:

zappa update dev

Still getting an error though:

Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.

The problem here is, that we haven't added our database to the S3 bucket. Lets rename our database that lives in our local dir and upload it to the S3 bucket.

mv db.sqlite3 sqlite_zappa.db

Update again to lambda:

(venv) $:~/github/django_zappa_sqlite_blogstory$ zappa update dev
Calling update for stage dev..
Packaging project as zip.
Uploading django-zappa-sq-dev-1616929230.zip (18.8MiB)..
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 19.7M/19.7M [00:01<00:00, 9.95MB/s]
Updating Lambda function code..
Updating Lambda function configuration..
Uploading django-zappa-sq-dev-template-1616929237.json (1.6KiB)..
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.66k/1.66k [00:00<00:00, 26.3kB/s]
Deploying API Gateway..
Scheduling..
Unscheduled django-zappa-sq-dev-zappa-keep-warm-handler.keep_warm_callback.
Scheduled django-zappa-sq-dev-zappa-keep-warm-handler.keep_warm_callback with expression rate(4 minutes)!
Your updated Zappa deployment is live!: https://nexusuynl2.execute-api.eu-central-1.amazonaws.com/dev
(venv) $:~/github/django_zappa_sqlite_blogstory$

And now, it just works.

References

zappa

Guide to using Django with Zappa

django-s3-sqlite

Django Forms 2 Validation

2021-03-12T00:00:00+01:00

Tutorial on how to use form validation in Django

This story will show how to use form validation in Django.

Prerequisites

I'm working on an Ubuntu 20.04, Django 3.1.1 and django-extensions 3.0.9 for good measure.

This will not be a complete from zero to hero tutorial, rather I'll assume that you know enough basic Django to be able to apply the shown yourself. Additionally, this story will lean on the topics covered in the first forms story.

Setup

We have initially these two models:

from django.db import models
from django.contrib.auth import get_user_model


User = get_user_model()


class Shop(models.Model):
    name = models.CharField(max_length=100)
    updated_by = models.ForeignKey(User, on_delete=models.DO_NOTHING, default=None, blank=True, null=True)

    def __str__(self):
        return self.name


class Product(models.Model):
    shop = models.ForeignKey(Shop, related_name="products", on_delete=models.CASCADE)
    name = models.CharField(max_length=100)
    price = models.DecimalField(max_digits=10, decimal_places=2)

and a CreateView to create a shop:

class CreateShopView(CreateView):
    template_name = "create_shop.html"
    form_class = ShopForm
    success_url = "/"

    def form_valid(self, form):
        self.object = form.save(commit=False)
        self.object.updated_by = self.request.user
        self.object.save()
        return HttpResponseRedirect(self.get_success_url())

    def get_form_kwargs(self, *args, **kwargs):
        kwargs = super().get_form_kwargs(*args, **kwargs)
        kwargs['user'] = self.request.user
        return kwargs

and a ShopForm:

from django import forms
from .models import Shop


class ShopForm(forms.ModelForm):
    class Meta:
        model = Shop
        fields = ['name', ]

    def __init__(self, *args, **kwargs):
        self.user = kwargs.pop('user')
        super().__init__(*args, **kwargs)

Validation use cases

Shop name validation

Lets assume I want to restrict what names can be entered in the shop's name field. For some reason, I don't want the user to be able to create shops with the name "The Galactic Empire". How would I do this?
There are several ways to do this, one way is to use the clean_FIELDNAME(...) method:

from django import forms
from .models import Shop
from django.core.exceptions import ValidationError


class ShopForm(forms.ModelForm):
    class Meta:
        model = Shop
        fields = ['name', ]

    def __init__(self, *args, **kwargs):
        self.user = kwargs.pop('user')
        super().__init__(*args, **kwargs)

    def clean_name(self):
        cleaned_name = self.cleaned_data['name']
        if cleaned_name == "The Galactic Empire":
            raise ValidationError("You cannot create The Galactic Empire")
        return cleaned_name

Now, if you enter "The Galactic Empire" in the form and submit, you will get the error message in the template that is raised with the ValidationError.

What if we want the same validation to apply to multiple different fields, say the name field on the Shop model and the name field on the Product model, how would we do this without duplicate code? The answer is with a custom validator, which is a function that takes the cleaned field data and either raises a ValidationError or does nothing.
We update the form to:

def validate_name(value):
    if value == "The Galactic Empire":
        raise ValidationError("You cannot create The Galactic Empire")


class ShopForm(forms.ModelForm):
    name = forms.CharField(validators=[validate_name,])

    class Meta:
        model = Shop
        fields = ['name', ]

    def __init__(self, *args, **kwargs):
        self.user = kwargs.pop('user')
        super().__init__(*args, **kwargs)

And again, we can't create The Galactic Empire.

Validate multiple conditions in one go

What if we have multiple fields we wanna validate at the same time, for example 2 password fields where both fields have to have the same value? In this case, we would use the clean method. Lets first update our Shop model to have an extra field, in this case a ticker field:

class Shop(models.Model):
    name = models.CharField(max_length=100)
    ticker = models.CharField(max_length=100, blank=True)
    updated_by = models.ForeignKey(User, on_delete=models.DO_NOTHING, default=None, blank=True, null=True)

    def __str__(self):
        return self.name

And the form we update to include this field:

class ShopForm(forms.ModelForm):
    name = forms.CharField(validators=[validate_name,])

    class Meta:
        model = Shop
        fields = ['name', 'ticker', ]

...
...
...
    def clean(self):
        name = self.cleaned_data['name']
        ticker = self.cleaned_data['ticker']

        if name == "Alderaan":
            if ticker == "":
                raise ValidationError("Alderaan must have a ticker symbol")

        return self.cleaned_data

The clean method takes out the data from the two fields and makes some logic involving both fields. And this is how this kind of multiple field validation is made.

References

Django forms official documentation

Django views official documentation

Django docs on validators

Django Forms 1

2021-03-10T00:00:00+01:00

Tutorial on how to use forms in Django

This story will show how to use forms in Django.

Prerequisites

I'm working on an Ubuntu 20.04, Django 3.1.1 and django-extensions 3.0.9 for good measure.

This will not be a complete from zero to hero tutorial, rather I'll assume that you know enough basic Django to be able to apply the shown yourself.

Setup

If you wanna follow along, setup a simple django project:

python3 -m venv ./venv
source venv/bin/activate
pip install django django-extensions
django-admin startproject config .
python manage.py startapp products

update the settings.py file with the new app:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'products',
]

...

TEMPLATE_DIR = os.path.join(BASE_DIR,"templates")
TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [TEMPLATE_DIR],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

Models

We are setting up a project to handle sales from shops; To that extend, we need a Shop and Product model:

from django.db import models


class Shop(models.Model):
    name = models.CharField(max_length=100)


class Product(models.Model):
    shop = models.ForeignKey(Shop, related_name="products", on_delete=models.CASCADE)
    name = models.CharField(max_length=100)
    price = models.DecimalField(max_digits=10, decimal_places=2)

Forms

We will start by creating a simple form so we are able to create a Shop in the system. We will begin with a function based view and a manually created form, and from there gradually use more and more of Django's included functionality.

Shop form in a function based view

in a forms.py file in the app folder, create the form like this:

from django import forms


class ShopForm(forms.Form):
    name = forms.CharField(label='The shop name', max_length=100)

We also need a view:

from django.http import HttpResponseRedirect
from django.shortcuts import render

from .forms import ShopForm


def create_shop(request):
    if request.method == 'POST':
        form = ShopForm(request.POST)

        if form.is_valid():
            return HttpResponseRedirect('/')
    else:
        form = ShopForm()

    return render(request, 'create_shop.html', {'form': form})

Setup the urls in a urls.py:

from django.urls import path

from . import views


urlpatterns = [
    path('shop/create', views.create_shop, name='create_shop'),
]

And finally, a template:

<form action="/your-name/" method="post">
    {% csrf_token %}
    {{ form }}
    <input type="submit" value="Submit">
</form>

We should now have:

Right now, if you enter anything in the shop form and click submit, nothing really happens, you are simply redirected to / as specified in the:

        if form.is_valid():
            return HttpResponseRedirect('/')

But, we really wanna save the new shop and we do this after the form.is_valid() and before the return HttpResponseRedirect('/'):

        if form.is_valid():
            name = form.cleaned_data.get('name', None)
            if name:
                new_shop = Shop(name=name)
                new_shop.save()

            return HttpResponseRedirect('/')

After the form.is_valid() has been executed, the form will contain a dictionary called cleaned_data with all the form data in it, after it has been validated and 'cleaned' (eg. made into Python types). If you printed out the form.cleaned_data, you would have gotten:

{'name': 'abc'}

If we enter a name now and submit the form, we would create an entry in the database.

Just for fun, what does the request.POST contain that we initialize the form with? if we print it out we get:

<QueryDict: {'csrfmiddlewaretoken': ['mGIi8M3YJ2X21sBRDoMr1jVRHZ3ar2jl3akkwQrIgKSXAHYGW804QXVw3PqeAInj'], 'name': ['ccc']}>

So, it contains the CSRF token as well as the 'uncleaned' data.

What if instead, we wanted to have the object creation logic (eg. create the new shop) stay in the form instead of in the view? we could create a .save() method on the form and call that from the view. The form could look like:

class ShopForm(forms.Form):
    name = forms.CharField(label='The shop name', max_length=100)

    def save(self):
        name = self.cleaned_data.get('name', None)
        new_shop = Shop(name=name)
        new_shop.save()

and we update the view to be:

        if form.is_valid():
            name = form.cleaned_data.get('name', None)
            if name:
                form.save()

            return HttpResponseRedirect('/')

If we try and create a new shop, it will still work.

is_valid()

Lets take a closer look at what the is_valid() method actually does.

1 If the form has bound data, then the form.full_clean() is called.

2 form.full_clean() iterates over the form fields and each fields also iterates over:

--- to_python() is called, creating the python type of the raw data.

--- Data is validated with field specific rules which includes custom validators.

--- Any custom made clean_FIELD(...) methods are called.

3 form.full_clean() then executes the form.clean() method, where validation that require access to multiple fields can take place (for example in a password form, ensure that password1 and password2 matches)

Failures in any of these steps raises a ValidationError(...).

If no ValidationErrors are raised then the method returns True, and False otherwise.

Modelform instead of Form

Lets swap the ShopForm we created for a ModelForm. This saves us a few lines of code.

class ShopForm(forms.ModelForm):
    class Meta:
        model = Shop
        fields = ['name', ]

and we will update our view as well:

def create_shop(request):
    if request.method == 'POST':
        form = ShopForm(request.POST)

        if form.is_valid():
            form.save()
            return HttpResponseRedirect('/')
    else:
        form = ShopForm()

    return render(request, 'create_shop.html', {'form': form})

Everything will work just like before, but we don't have to reimplement the .save() method. Additionally, the form will inherent the validation logic from the model which will also save us some lines of code.

Class based FormView

Lets try and get the modelform to work with a class based view, in this case a FormView.

class CreateShopView(FormView):
    form_class = ShopForm
    template_name = "create_shop.html"
    success_url = "/"

    def form_valid(self, form):
        form.save()
        return super().form_valid(form)

We also need to update our url a bit:

urlpatterns = [
    path('shop/create', views.CreateShopView.as_view(), name='create_shop'),
]

And again, everything works as expected.

Class based CreateView

We can make it even simpler by using a CreateView, a generic view, which will make our view even simpler:

from django.views.generic.edit import CreateView

class CreateShopView(CreateView):
    template_name = "create_shop.html"
    form_class = ShopForm
    success_url = "/"

Again, everything will work as before. And with CreateView we can make it even simpler by removing the form altogether and simply adding the model to the view:

class CreateShopView(CreateView):
    template_name = "create_shop.html"
    model = Shop
    fields = "__all__"
    success_url = "/"

No need for a separate form in this case. But, typically, it makes sense to have a separate form as you probably want to have some custom logic in your form anyway.

Extra kwargs for the form with class based views

Imagine we have a slightly different model for the Shop:

from django.db import models
from django.contrib.auth import get_user_model


User = get_user_model()


class Shop(models.Model):
    name = models.CharField(max_length=100)
    updated_by = models.ForeignKey(User, on_delete=models.DO_NOTHING, default=None, blank=True, null=True)

    def __str__(self):
        return self.name

We now have a updated_by field that we want set whenever the model instance is updated or created (in our case). Now, in the form, we want this done automatically, eg. the logged in user should simply be added to the instance.

In our CreateView version, we could do this by capturing the logged in user and pass this information to the form where it will be saved.

class CreateShopView(CreateView):
    template_name = "create_shop.html"
    form_class = ShopForm
    success_url = "/"

    def form_valid(self, form):
        self.object = form.save(commit=False)
        self.object.updated_by = self.request.user
        self.object.save()
        return HttpResponseRedirect(self.get_success_url())

    def get_form_kwargs(self, *args, **kwargs):
        kwargs = super().get_form_kwargs(*args, **kwargs)
        kwargs['updated_by'] = self.request.user
        return kwargs

If we just use this without updating the form we will get an error when trying to create a Shop. This is because, the form doesn't have a 'updated_by' field but is still passed a updated_by keyword argument. So, we need to remove this updated by kwarg before the form gets initialized which can be done by overriding the form's init method:

class ShopForm(forms.ModelForm):
    class Meta:
        model = Shop
        fields = ['name', ]

    def __init__(self, *args, **kwargs):
        self.user = kwargs.pop('updated_by')
        super().__init__(*args, **kwargs)

References

Django forms official documentation

Django views official documentation

Classy Class Based Views

Blog post on formview

Setup django with multitenancy (semi-isolated)

2020-09-20T00:00:00+02:00

Setup Django with django-tenancy (semi-isolated)

This story will show how to install setup Django to work with multiple clients, like a real SAAS.

Prerequisites

I'm working on an Ubuntu 20.04, Django 3.1.1 and django-tenants 3.2.1. Also, I'm assuming you have a running postgres server running. At the bottom, I'm including a docker-compose file that you can run that will handle that part.

What is multitenancy?

Multitenancy refers to the functionality of serving multiple different customers (tenants) using the same platform, for us, Django. Think Slack or Shopify or any other SAAS where the customers share a platform but still each have their individual 'whitelabel'.

4 different models

In general, there are 4 different approaches one can take to serve multiple tenants.

0 Completely isolated deployments

This is not traditionally considered a solution as no resources at all are shared. You simply deploy a new separate installation of your django and database to a separate server, one for each tenant.

1 Isolated

In this approach, each tenant has their own database but share the Django instance. This can be easily achieved as Django supports multiple databases out of the box. It doesn't scale well though but might be attractive if you have few customers who value their data privacy.

2 Semi-isolated

In this approach, each tenant will share the same database but each will have their own schema. This is the approach we will use in this story.

3 Shared

In the shared approach, there is one database and one schema. Multi-tenancy is achieved by having each and every table have a foreignkey relationship with a Tenant table. This approach scales very well.

Hands on setup

Install and setup django

pip install django django-tenants psycopg2-binary

Create django project

django-admin startproject config .

Create app to hold public client/tenants schema:

python manage.py startapp clients

Create models

In client/models.py

from django.db import models
from django_tenants.models import TenantMixin, DomainMixin


class Client(TenantMixin):
    name = models.CharField(max_length=100)


class Domain(DomainMixin):
    pass

Update settings

import os

import os

ALLOWED_HOSTS = ["*"]

and templates dir:

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [BASE_DIR / 'templates'],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

Add to installed apps:

INSTALLED_APPS = [
    'django_tenants',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

Enable middleware:

MIDDLEWARE = [
    'django_tenants.middleware.main.TenantMainMiddleware',
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

Database engine:

DATABASES = {
    'default': {
        'ENGINE': 'django_tenants.postgresql_backend',
        'NAME': os.environ.get('DATABASE_DB', 'mytickets'),
        'USER': os.environ.get('DATABASE_USER', 'postgres'),
        'PASSWORD': os.environ.get('DATABASE_PASSWORD', '12345'),
        'HOST': os.environ.get('DATABASE_HOST', 'localhost'),
        'PORT': os.environ.get('DATABASE_PORT', '5432'),
    }
}

Add Clients app to installed apps

INSTALLED_APPS = [
    'django_tenants',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'clients',
]

Add links to models:

TENANT_MODEL = "clients.Client" # app.Model
TENANT_DOMAIN_MODEL = "clients.Domain"  # app.Model

Add routing

DATABASE_ROUTERS = (
    'django_tenants.routers.TenantSyncRouter',
)

Update shared and tenant apps:

SHARED_APPS = [
    'django_tenants',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'clients',
]

TENANT_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
]

INSTALLED_APPS = list(set(SHARED_APPS + TENANT_APPS))

Run migrations

python manage.py makemigrations
python manage.py migrate_schemas --shared

Create tenants and tenant app

python manage.py startapp products

Add model to new app:

from django.db import models


class Product(models.Model):
    name = models.CharField(max_length=127)

Make migrations and migrate

python manage.py makemigrations
python manage.py migrate_schemas

Add to admin

from django.contrib import admin

from products.models import Product


@admin.register(Product)
class ProductAdmin(admin.ModelAdmin):
    pass

python manage.py create_tenant

Remember to setup the /etc/hosts file to have the domains point to 127.0.0.1

Create superuser

(venv) $:~/mytickets$ python manage.py create_tenant_superuser
Enter Tenant Schema ('?' to list schemas): t1
Username (leave blank to use 'perwagner'): pert1
Email address: pert1@gmail.com
Password: 
Password (again): 
Superuser created successfully.
(venv) $:~/mytickets$ python manage.py create_tenant_superuser
Enter Tenant Schema ('?' to list schemas): t2
Username (leave blank to use 'perwagner'): pert2
Email address: pert2@gmail.com
Password: 
Password (again): 
Superuser created successfully.
(venv) $:~/mytickets$

Test the website for the tenenats on:

http://t1.mytickets.local:8000/

http://t2.mytickets.local:8000/

Create view

We need to make public and private URL routes in settings

In settings:

PUBLIC_SCHEMA_URLCONF = 'config.public_urls'
ROOT_URLCONF = 'config.urls'

create the public_urls.py file in /config

create view in product/views.py:

from django.views.generic import ListView

from products.models import Product


class ProductListView(ListView):
    model = Product
    template_name = 'products.html'
    context_object_name = 'products'

create "templates" folder in project root, create products.html in there:

<htm>
    <body>
        <ul>
            {% for product in products %}
                <li>{{ product.name }}</li>
            {% endfor %}
        </ul>
    </body>
</htm>

update config/urls.py:

from django.contrib import admin
from django.urls import path

from products.views import ProductListView


urlpatterns = [
    path('admin/', admin.site.urls),
    path('', ProductListView.as_view(), name='product_list_view')
]

Test

Go to t1.mytickets.local and t2.mytickets.local and test. Don't forget to update your local hosts file in etc/hosts to include:

127.0.0.1   t1.mytickets.local
127.0.0.1   t2.mytickets.local

docker-compose with postgres

version: '3.7'

services:
  db:
    container_name: postgres
    image: postgres:alpine
    ports:
      - 5432:5432
    volumes:
      - db-data:/var/lib/postgresql/mytickets
    environment:
      POSTGRES_PASSWORD: 12345
      POSTGRES_USER: postgres
      POSTGRES_DB: mytickets

volumes:
  db-data:

References

https://django-tenants.readthedocs.io/en/latest/install.html

youtube video showing how to setup django-tenants

django-multitenant on github

YouTube video on django-multitenant

Sanic Web Tutorial Part 2 - Templates

2020-02-08T00:00:00+01:00

In Part 2 we will setup Sanic to render html templates.

Install and setup Jinja2 extension

We will enable our application to render Jinja2 templates. Jinja2 is a template engine developed by Armin Ronacher (the developer of Flask and many other things) and documentation for the project can be found here: https://palletsprojects.com/p/jinja/.

In order to render templates in Sanic we need to first install an extension. Head over to https://github.com/mekicha/awesome-sanic and find sanic-jinja2 (not jinja2-sanic, which hasn't be updated for some time). The direct link is: https://github.com/lixxu/sanic-jinja2. Install into your virtualenv

$ pip install sanic-jinja2

In order to enable this extension we need to tell Sanic so. Update our application to

from sanic import Sanic
from sanic import response
from sanic_jinja2 import SanicJinja2

app = Sanic(name='Sanic Web Tutorial')
jinja = SanicJinja2(app, pkg_name="main")


@app.route("/")
async def index(request):
    return response.html("<h1>Hello World</h1>")


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8000, debug=True)

Note that we also add the debut=True to the app.run() method. This will auto-restart the development server when we make code changes and provide more debugging info.

Additionally, notice that we set 'pkg_name="main"' when instantiating the jinja object? If you don't do this, you will likely get an error when running main.py. This seems to be a bug in the sanic-jinja2 extension though.

Our first html template

First, create a folder to hold all the templates:

$ mkdir templates

The name is important for now ('templates'). Inside this folder, crate an index.html page:

$ touch templates/index.html

And inside this index.html file, create a boilerplate HTML page:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <title>Sanic Web Tutorial</title>
  </head>
  <body>
    <h1>Sanic Web Tutorial</h1>
  </body>
</html>

and update our / route render this template instead of the plain text:

@app.route("/")
async def index(request):
    return jinja.render("index.html", request)

You should now be able to see our updated website on: http://0.0.0.0:8000/

Pass variables to the template

Lets say we wan't to pass our title as a parameter in the Python code to the template, how would we do that? Well, the jinja.render() method actually takes 3 arguments, the 3rd being the variables to pass. Lets update to:

@app.route("/")
async def index(request):
    return jinja.render("index.html", request, title="Sanic Web Title")

and update the template's h1 tag to be:

    <h1>{{ title }}</h1>

The {{ }} brackets is a Jinja2 template tag and it used as a standin for variables passed to the template.
Refresh your webpage on: http://0.0.0.0:8000/ and you should see the updated title.

What if you want to pass multiple variables to the template? Well, we can add multiple variables in the jinja.render function, like this:

@app.route("/")
async def index(request):
    return jinja.render("index.html", request, title="Sanic Web Title", desc="Something here")

and update the template's h1 tag to be:

    <h1>{{ title }}</h1>
    <p>{{ desc }}</p>

This can get quite ugly to read if you have to pass many variables, so we can pass an object instead:

@app.route("/")
async def index(request):
    context = {
        "title": "Sanic Web Title",
        "desc": "Something else here",
    }
    return jinja.render("index.html", request, context=context)

and we also need to update the html template to account for this:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <title>Sanic Web Tutorial</title>
  </head>
  <body>
    <h1>{{ context.title }}</h1>
    <p>{{ context.desc }}</p>
  </body>
</html>

and if you reload/refresh your website again, you should see that it still shows correctly.

Sanic Web Tutorial Part 1

2020-02-07T00:00:00+01:00

Welcome to my Sanic Web Tutorial where you will learn to use the async web framework Sanic to create and deploy your own website.

Assumptions

I assume that you are familiar with using Python and virtualenv, but you do not need to be an expert. You should be able to use this tutorial even if you are new to web development.

I'm developing on a Ubuntu 19.10 machine running Python 3.8, so if you are on a different OS, you might need to make minor changes.

Hello World with Sanic

Start by creating a folder for your new web application and in there create a virtualenv and install Sanic with pip:

$ mkdir sanic_web_tutorial
$ virtualenv venv -p python3
$ source venv/bin/activate
$ pip install sanic

Now we will create our main application file and put some boilerplate Python code in there.

$ touch main.py

Open main.py in an editor and enter this:

from sanic import Sanic
from sanic import response

app = Sanic(name='My Sanic Web App')

@app.route("/")
async def index(request):
    return response.text("Hello World")

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8000)

Now, start the web application by entering:

$ python main.py
[2020-02-07 23:35:35 +0800] [6539] [INFO] Goin' Fast @ http://0.0.0.0:8000
[2020-02-07 23:35:35 +0800] [6539] [INFO] Starting worker [6539]

Go to http://0.0.0.0:8000 in your browser to see you web application in action with the text "Hello World" returned.

What goes on in "Hello World"

Lets dig into this tiny Hello World application to understand what exactly goes on.

First, we import two things from the sanic framework:
Sanic: The web application class and
response: which will help us create something that the application can return to the user.

Next, we instantiate the application with app = Sanic(name='My Sanic Web App'). We must provide a 'name' otherwise we will get a warning (or the instantiation will fail).

With @app.route("/") we define a HTTP endpoint, in this case the /. Once our browser hits this endpoint, the function that is defined below it will be executed:

async def index(request):
    return response.text("Hello World")

This defines an async function called 'index'. We could make this example even more simple by removing the 'async' keyword and simply have:

async def index(request):
    return response.text("Hello World")

then our endpoint wouldn't be syncronious though, not async.

The @app.route decorater for the app.add_route method, and instead of using the decorater, we could have just used this method like this:

async def index(request):
    return response.html("<h1>Hello World</h1>")

app.add_route(index, "/")

One advantage of using the add_route method instead of the decorator is that you can add multiple routes to the same function. Say, we want both / and /main to return the same page, we can do this with 2 add_route() calls:

async def index(request):
    return response.html("<h1>Hello World</h1>")

app.add_route(index, "/")
app.add_route(index, "/main")

No automatic server restart on changes
Be mindful, that the running python main.py doesn't restart automatically when you make changes in the code like you might be used to from Flask or Django. You'll have to re-run python main.py yourself.

In the index function we are simply returning a text response with the value "Hello World". If we wanted to return html instead of text, we could have written the function as

@app.route("/")
async def index(request):
    return response.html("<h1>Hello World</h1>")

The only difference between these two response types is the Content-Type, where for text it is text/plain while for the html response it is text/html.
There are other repsonse types available as well, like response.json and response.file, but we will get back to them in time.

Finally, we instantiate the application with

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8000)

This is similar to how Flask works (if you are familiar with Flask). We specify the host and port and with __name__ we can run it with python main.py.

Testing

We could write unittests using Python's unittest module, but we will use pytest (and pytest-sanic) instead as this allows us to write shorter and nicer tests. Additionally, pytest enables more advanced testing options going forward.

First, install pytest-sanic into the virtualenv and create a test file:

$ pip install pytest-sanic
$ touch test.py

In test.py, write a simple test to check that the application returns a 200 on the router /:

from main import app


def test__index__get_request__returns_200():
    request, response = app.test_client.get('/')
    assert response.status == 200

run the test by typing

$ pytest test.py
================================================ test session starts ================================================
platform linux -- Python 3.7.3, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /sanic_web_tutorial
plugins: sanic-1.1.2
collected 1 item                                                                                                    

test.py .                                                                                                     [100%]

================================================= warnings summary ==================================================
test.py::test__index__get_request__returns_200
  /sanic_web_tutorial/venv/lib/python3.7/site-packages/httpx/client.py:234: UserWarning: Passing a 'verify' argument when making a request on a client is due to be deprecated. Instantiate a new client instead, passing any 'verify' arguments to the client itself.
    "Passing a 'verify' argument when making a request on a client "

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================================== 1 passed, 1 warning in 0.02s ============================================

Ignore the warning for now. The test passes.

Save

Finally, before closing everything, do a

$ pip freeze > requirements.txt

to create a reusable requirements file. If you are on a Ubuntu system, you might have to manually open the just created requirements.txt file and delete the line pkg-resources==0.0.0. On my system (Ubuntu), this line is wrongly created when running pip freeze (bug in Ubuntu). If you don't delete it, then you will get an error the next time you try and install the requirements with pip install -r requirements.txt.

Deploy Flask app to Heroku with Docker and Gitlab

2019-11-01T00:00:00+01:00

Deploy Flask app to Heroku with Docker

This story will show how to install deploy a simple Flask app to Heroku via Docker and Gitlab CI.

Prerequisites

I'm assuming that you are familiar with the basics of Docker, Heroku, Flask and Gitlab.

Also, you have:
1 Heroku account with an app created
2 Heroku cli installed locally
3 Gitlab account with a repository created (clean)
4 Private gitlab runner to execute Docker builds
5 Docker and docker-compose installed locally

(I'm using ubuntu 19.04)

Setup Flask application

Setup virtualenv

Create a python3 virtualenv and install Flask and gunicorn:

$ virtualenv venv -p python3
$ source venv/bin/activate
$ pip install flask gunicorn
$ pip freeze > requirements.txt

(If you are on ubuntu, you might need to enter the requirementes.txt file and remove the pkg-resources==0.0.0 from that file)

Create Flask application

In your gitlab repository, create a main.py file:

from flask import Flask, escape, request

app = Flask(__name__)

@app.route('/')
def hello():
    name = request.args.get("name", "World")
    return f'Hello, {escape(name)}!'

You can test it locally by running:

$ export FLASK_APP=main.py
$ flask run

Create docker container and run via docker-compose

Create the Dockerfile and add this:

FROM python:3.8-alpine

WORKDIR /code

# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1  # Dont write pyc files
ENV PYTHONUNBUFFERED 1  # Dont buffer stdout and stderr
ENV FLASK_APP main.py
ENV FLASK_RUN_HOST 0.0.0.0

RUN apk add --no-cache gcc musl-dev linux-headers

COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt

COPY . .

CMD ["flask", "run"]

and create the docker-compose.yml file like this:

version: '3.7'

services:
  flaskapp:
    build: .
    ports:
      - "5000:5000"

Now, you can run your application by typing:

$ docker-compose up

Heroku container registry (deploy the container directly)

(much of this comes from https://devcenter.heroku.com/articles/container-registry-and-runtime)

$ heroku container:login

Build and push your docker image to the registry:

$ heroku container:push web --app YOUR_APP_NAME

After this, the docker image will be in the heroku docker registry and you can now push it to your dymo:

$ heroku container:release web --app YOUR_APP_NAME

Gitlab CI

Create a .gitlab-ci file with this:

variables:
  HEROKU_APP_NAME: YOUR_APP_NAME

stages:
  - deploy

deploy:
  stage: deploy
  tags:
    - deploy
  script:
    - apt-get update -qy
    - apt-get install -y ruby-dev
    - gem install dpl
    - dpl --provider=heroku --app=$HEROKU_APP_NAME --api-key=$HEROKU_AUTH_TOKEN

Under your Gitlab settings (settings/CI CD/) 'runners' and 'variables' you need to do two things. Under 'runner' you should enable the private docker runner, and under 'variables' you should set the HEROKU_AUTH_TOKEN to the value you get when running: heroku auth:token

Also notes, the gitlab runner we are enabling should be tagged with 'deploy'. The .gitlab-ci file specifies 'under 'tags:' which type of runner to use for this particular stage, in this case 'deploy'. If this is not specified, a runner without the correct dependencies might be used and the deployment fail.

Heroku yml file

We will also create a heroku.yml file to tell Heroku what to do with our Dockerfile

build:
  docker:
    web: Dockerfile

run:
  web: gunicorn main:app

If we include a 'run' section in the heroku.yml file, then the CMD section of the Dockerfile is ignored.

Push to Gitlab and observe deployment in action

Now, push to gitlab:

git push origin master

And if all goes well, your application will be deployed to Heroku

References

https://github.com/travis-ci/dpl#heroku-api

(testdriven.io) Deploying Django to Heroku With Docker

Install Squid proxy on Raspberry Pi

2019-07-28T00:00:00+02:00

How to install a Squid proxy on a Raspberry Pi

This story will show how to install Squid3 on a Raspberry Pi (1) and set it up as a proxy server

Squid as a proxy

Install Squid

First install squid

$ sudo apt install squid3

Enable the squid server to run on startup.

$ sudo update-rc.d squid3 enable

Lets see if it is running

$ sudo netstat -antp | grep squid
tcp6       0      0 :::3128                 :::*                    LISTEN      7662/(squid-1)

So squid is listening.

Changes to config files

Find the squid conf files

$ sudo locate squid.conf
/etc/squid3/squid.conf
/etc/squid3/squid.conf.save
/etc/squid3/squid.conf.save.1
/etc/squid3/squid.conf.save.2
/etc/squid/squid.conf.origin
/usr/share/doc/squid3-common/squid.conf.documented.gz

Update the squid3 conf file. But first we make a backup

$ sudo cp /etc/squid3/squid.conf /etc/squid3/squid.conf.origin
$ sudo vim /etc/squid3/squid.conf

We need to update some lines to allow the local network to contact the squid proxy:

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 10.0.0.0/8     # RFC1918 possible internal network
acl localnet src 172.16.0.0/12  # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network 
#acl localnet src fc00::/7       # RFC 4193 local private network range
#acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines

(uncommented the first three acl localnet ones).

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost

(removed the comment from http_access allow localnet)

Ensure that squid is listening to the correct port:

# Squid normally listens to port 3128
http_port 3128

Save the changes and update the squid (reload or restart)

$ sudo service squid3 reload

See the status of squid:

$ sudo service squid3 status
● squid3.service - LSB: Squid HTTP Proxy version 3.x
   Loaded: loaded (/etc/init.d/squid3)
   Active: active (running) since Sun 2019-07-28 04:16:01 UTC; 15min ago
  Process: 7886 ExecReload=/etc/init.d/squid3 reload (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/squid3.service
           ├─7660 /usr/sbin/squid3 -YC -f /etc/squid3/squid.conf
           ├─7662 (squid-1) -YC -f /etc/squid3/squid.conf
           ├─7894 (logfile-daemon) /var/log/squid3/access.log
           └─7895 (pinger)

Jul 28 04:16:01 raspberrypi squid3[7660]: Squid Parent: will start 1 kids
Jul 28 04:16:01 raspberrypi squid3[7628]: Starting Squid HTTP Proxy 3.x: squid3.
Jul 28 04:16:01 raspberrypi squid3[7660]: Squid Parent: (squid-1) process 7662 started
Jul 28 04:16:01 raspberrypi systemd[1]: Started LSB: Squid HTTP Proxy version 3.x.
Jul 28 04:31:03 raspberrypi systemd[1]: Reloading LSB: Squid HTTP Proxy version 3.x.
Jul 28 04:31:03 raspberrypi squid3[7886]: Reloading Squid HTTP Proxy 3.x configuration files.
Jul 28 04:31:03 raspberrypi squid3[7886]: done.
Jul 28 04:31:04 raspberrypi systemd[1]: Reloaded LSB: Squid HTTP Proxy version 3.x.

Check the log to see the connections

The below will display the squid3 access log, and you can see the activity on the proxy.

sudo tail -f /var/log/squid3/access.log

Setup cache

In the config file, find the line with #cache_dir ufs /var/spool/squid3 100 16 256. Uncomment this.
The /var/spool/squid3 is the location of the cached files.
100 is the size of the cache in MB.
16 is 256 is

We might also want to fetch images a bit more often to avoid having old images displayed from the cache. Add this line:

refresh_pattern -i \.(gif|png|jpg|jpeg|ico)$ 3600   90%   43200

Retart squid:

$ sudo service squid3 restart

(might take a bit)

To check the cache is working, do a:

$ sudo du -sh /var/spool/squid3

and check the size.. browse a bit and check again, the size should have gone up.

Docker, Docker Compose and Django with Celery

2019-04-25T00:00:00+02:00

How to setup Django and Celery with Docker and docker-compose

This story sets up Django and adds Celery support to a docker-compose setup. I have updated this on 22 May, 2022 to use the newest version of Celery, which does foster a few changes.

Setup basic Django project

First create a virtualenv and install django and celery (5.2.6):

$ python3 -m venv ./venv
$ source venv/bin/activate
$ pip install django celery
$ pip freeze > requirements.txt

Create the Django application

$ django-admin startproject config .

Create a dockerfile and add this:

FROM python:3.10

ARG APP_USER=appuser
RUN groupadd -r ${APP_USER} && useradd --no-log-init -r -g ${APP_USER} ${APP_USER}

EXPOSE 8000
WORKDIR /app

# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
ENV DEBUG 0

RUN apt-get update && apt-get install

RUN apt-get install -y \
  gcc \ 
  git \
  python3-dev \
  && apt-get clean

RUN pip install -U pip

# Install pip requirements
COPY ./requirements.txt /app/requirements.txt
RUN python -m pip install -r /app/requirements.txt
COPY . /app/

Setup Docker-Compose

Create the docker-compose.yml file:

version: '3.2'

services:
  db:
    image: mdillon/postgis:9.6
    container_name: postgis
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=12345
      - POSTGRES_DB=projectdatabase
    ports:
      - "5433:5432"
    volumes:
      - data:/var/lib/postgresql/data

  web:
    build: .
    command: bash -c "python /code/manage.py migrate --noinput && python /code/manage.py runserver 0.0.0.0:8000"
    restart: "always"
    volumes:
      - .:/code
    ports:
      - "8001:8000"
    depends_on:
      - db
      - broker

  broker:
    image: rabbitmq:latest
    hostname: broker
    environment:
      - RABBITMQ_DEFAULT_USER=admin
      - RABBITMQ_DEFAULT_PASS=mypass
    ports:
      - "5672:5672"

  worker:
    build: .
    restart: "no"
    command: celery -A myprojectname worker
    volumes:
      - .:/code
    depends_on:
      - broker

  flower:  
    image: mher/flower
    command: ["flower", "--broker=amqp://admin:mypass@broker", "--port=5555"]  
    ports:  
      - 5555:5555 
    depends_on:
      - broker


volumes:
  data:

Update settings

Change a single line in config/settings.py:

ALLOWED_HOSTS = ["*"]

Run application

Now run docker-compose up --build:

docker-compose up --build

go to localhost:8000 and hurra... your django app is running.

Add Celery to project

Add 3 services to docker-compose to get:

version: '3.2'

services:
  web:
    build: .
    command: bash -c "python /code/manage.py migrate --noinput && python /code/manage.py runserver 0.0.0.0:8000"
    restart: "always"
    volumes:
      - .:/code
    ports:
      - "8000:8000"

  rabbitmq:
    image: rabbitmq:latest
    hostname: broker
    environment:
      - RABBITMQ_DEFAULT_USER=admin
      - RABBITMQ_DEFAULT_PASS=mypass
    ports:
      - "5672:5672"

  worker:
    build: 
      dockerfile: Dockerfile
      context: . 
    restart: "no"
    command: celery -A config worker
    volumes:
      - .:/code
    depends_on:
      - rabbitmq
      - web

  flower:  
    image: mher/flower
    environment:
      - CELERY_BROKER_URL=PYAMQP://admin:mypass@broker:5672//
      - FLOWER_PORT=5555
    ports:  
      - 5555:5555 
    depends_on:
      - rabbitmq


volumes:
  data:

We also need to update our settings file and add a file to our config/ folder: celery.py in the same folder as our settings.py file:

import os
from celery import Celery


os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings')
app = Celery('config')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()


@app.task(bind=True)
def debug_task(self):
    print(f'Request: {self.request!r}')

Add one line to settings.py:

rabbitmq_url = 'pyamqp://admin:mypass@broker:5672'
CELERY_BROKER_URL = rabbitmq_url

and update the __init__.py file in the config folder to be:

from .celery import app as celery_app

__all__ = ('celery_app',)

Congratulations, you have a working django and celery application running in docker.

And that is pretty much it. Run $ docker-compose up --build to rebuild the Docker images with the new requirements.txt file.

On localhost:5555 we have installed Flower, which is a handy tool to monitor your celery workers.

links

https://www.revsys.com/tidbits/celery-and-django-and-docker-oh-my/
https://www.revsys.com/tidbits/brief-intro-docker-djangonauts/
https://docs.celeryq.dev/en/stable/django/first-steps-with-django.html#using-celery-with-django/

Docker, Docker Compose and Django

2019-04-20T00:00:00+02:00

How to setup Django with Docker and docker-compose

In this story I'll show how to setup a simple django application to use docker and docker-compose.

Install a base django project

First we need to install a basic Django app, we do this in a Python 3 virtual environment.

$ virtualenv venv -p python3
$ source venv/bin/activate
$ pip install django
$ django-admin startproject myproject .
$ pip freeze > requirements.txt

You might have to go into the requirements.txt file and delete an annoying entry with pkg-packages 0.0.0 (or something like that).

The little dot '.' at the end tells django-admin to have a more compact directory structure.

Setup docker file to run django in container

We want to run our django project from within a Docker container. For this, we need a Dockerfile with instructions on how this shold happenn. Create a file named 'Dockerfile' and put it at the root of the project (same dir as manage.py) with this content:

FROM python:3
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8

COPY ./requirements.txt ./code/requirements.txt
RUN pip install -r /code/requirements.txt

COPY . /code/
WORKDIR /code/

FROM python:3 means that the docker container will be based off of a Python 3 image.
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8: Don't quite know what this is about.
COPY ./requirements.txt ./code/requirements.txt: Copies our requirements.txt file into the container in a subdirectory.
RUN pip install -r /code/requirements.txt: Installs the requirements in the container.
COPY . /code/: Copies our project into the /code directory.
WORKDIR /code/: Sets the directory for commands like RUN, CMD, ENTRYPOINT, COPY and ADD. Better to do it this way then having multiple cd xxx & do-something.

We also want to have docker-compose start our docker container (for ease of use). So we create a docker-compose.yml file with the following content:

version: '3.2'

services:
  web:
    build: .
    command: bash -c "python /code/manage.py migrate --noinput && python /code/manage.py runserver 0.0.0.0:8000"
    restart: "always"
    volumes:
      - .:/code
    ports:
      - "8001:8000"

command: bash -c "python /code/manage.py migrate --noinput && python /code/manage.py runserver 0.0.0.0:8000": Runs a command in the in the docker continer created by build .. We run migrate and the test server on port 8000.
restart: "always": We want to restart on any possible change as this mimics the behavior if we didn't use docker but just ran on our host machine.

Add postgis database support

In order to add another service to our docker-compose setup, we only need to add another entry in the docker-compose.yml file. So update the docker-compose.yml to:

version: '3.2'

services:
  db:
    image: mdillon/postgis:9.6
    container_name: postgis
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=12345
      - POSTGRES_DB=projectdatabase
    ports:
      - "5433:5432"
    volumes:
      - data:/var/lib/postgresql/data

  web:
    build: .
    command: bash -c "python /code/manage.py migrate --noinput && python /code/manage.py runserver 0.0.0.0:8000"
    restart: "always"
    volumes:
      - .:/code
    ports:
      - "8001:8000"
    depends_on:
      - db

We also need to update our settings.py file in our project to take account of this new db service. In settings.py update the database settings to:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'projectdatabase',
        'USER': 'postgres',
        'PASSWORD': '12345',
        'HOST': 'db',
        'PORT': '5432',
    }    
}

Notice that as HOST we are specifing the service name from the docker-compose file 'db' and not 'localhost'. This is because the services in the docker-compose environment share the same network and communicate internally via the service names.

links

https://www.revsys.com/tidbits/celery-and-django-and-docker-oh-my/
https://www.revsys.com/tidbits/brief-intro-docker-djangonauts/

Vagrant, Ansible and Posgtresql

2019-04-14T00:00:00+02:00

How to setup Vagrant, Ansible and Postgresql

In this story I’ll be setting up Vagrant with a Postgresql server provisioned by Ansible. The setup will be pretty bare bone, so don’t use it as a blueprint for a production setup.

Setup Vagrant

Create a file named ‘Vagrantfile’ in the root directory of your project. This file holds all the information about the Vagrant box we are going to create.

# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure(2) do |config|
  config.vm.box = "bento/ubuntu-18.04"
  config.vm.network "forwarded_port", guest: 5432, host: 5433, host_ip: "127.0.0.1"
  config.vm.network "private_network", ip: "192.168.60.70"
  config.vm.provider "virtualbox" do |vb|
     vb.memory = "2048"
  end
end

config.vm.box = “bento/ubuntu-18.04” -> Tells Vagrant to base itself on a ubuntu 18.04 installation

config.vm.network “forwarded_port”, guest: 5432, host: 5433, host_ip: “127.0.0.1” -> Forwards the port 5432 on the Vagrant box to port 5433 on the host machine (the machine running Vagrant). We forward port 5432, which is the default port for postgresql to a different port on our host machine to avoid any collision in case we have a local installation of postgresql running (on the host machine). This is relevant when you access the Vagrant box as localhost (127.0.0.1), not when you use the “private_network”.

config.vm.network “private_network”, ip: “192.168.60.70” -> setup a private network so we can access the Vagrant box through the specified IP on your host machine (instead of using localhost).

You can now start up the Vagrant box by typing:

$ vagrant up

The first time we power up Vagrant it will take a bit of time, but subsequent start ups will be much faster as the image (ubuntu) will already be downloaded.

You can also log in to the Vagrant box by typing:

$ vagrant ssh

Setup Ansible

We want to do several things when setting up Ansible.
1. create a inventory file (hosts_vagrant) which specifies the server that Ansible should work on. 2. create a playbook (playbook.yml) with all the tasks that needs to be run on the Vagrant box. 3. create a task for updating the Vagrant box with basic packages 4. create a task for installing postgresql 5. create specific tasks to setup postgresql configuration

1 Create the inventory file “hosts_vagrant”:

[all:vars]
env=vagrant
[postgresdb]
192.168.60.70
[postgresdb:vars]
  ansible_user=vagrant 
  ansible_private_key_file="./.vagrant/machines/default/virtualbox/private_key"

We are telling Ansible the IP address of our box as well as some login information specific for Vagrant.

2 Create playbook “playbook.yml”

---
- name: postgresdb
  hosts: all
  become: yes
  gather_facts: False
  pre_tasks:
    - name: Install python for Ansible
      raw: test -e /usr/bin/python || (apt -y update && apt install -y python-minimal)
      changed_when: False
    - setup: # aka gather_facts

We setup a few basic things like creating python for Ansible.

Run the below commands to ensure that you can actually connect to the Vagrant box.

$ ansible all -m ping -i hosts_vagrant
$ ansible -i ./hosts_vagrant postgresdb -m ping
$ ansible-playbook -i ./hosts_vagrant playbook.yml

3, 4 Install base packages and Postgresql

Update the playbook to contain the 2 roles below.

---
- name: postgresdb
  hosts: all
  become: yes
  gather_facts: False
  pre_tasks:
    - name: Install python for Ansible
      raw: test -e /usr/bin/python || (apt -y update && apt install -y python-minimal)
      changed_when: False
    - setup: # aka gather_facts
roles:
    - base
    - postgresql

Now, create the file ‘main.yml’ in the subfolder specified.

(roles/base/tasks/main.yml)
- name: Install base dependencies
  apt:
    name: "{{ packages }}"
  vars:
    packages:
      - locales
      - build-essential
      - acl
      - ntp
      - htop
      - git
      - python-pip
      - python3-pip
      - libpq-dev
    state: present
  tags:
    - packages
- name: Upgrade pip
  pip: name=pip state=latest
  tags:
    - packages

Also create the task to setup the postgresql.

(roles/postgresql/tasks/main.yml)
- name: Install postgresql
  apt:
    name: "{{ packages }}"
  vars:
    packages:
      - postgresql
      - postgresql-contrib
      - postgresql-server-dev-10
      - python-psycopg2
    state: present
  tags:
    - postgres
- name: Enable passwordless sudo
  lineinfile: dest=/etc/sudoers regexp=^vagrant line="vagrant ALL=(postgres) NOPASSWD:/bin/sh"
- name: Create a PostgreSQL database user
  postgresql_user: name=project password=project role_attr_flags=CREATEDB state=present
  become: yes
  become_user: postgres
  become_method: sudo
- name: Create acme database
  become: yes
  become_user: postgres
  postgresql_db:
    name: acme
    encoding: UTF-8

5 Need to change these files in Vagrant

(pg_hba.conf), add:
host all all 0.0.0.0/0 trust
(postgresql.conf) update:
listen_addresses = '*'

Restart server:

sudo service postgresql restart

Now, to put this into Ansible we will create 2 templates and have Ansible copy them to the Vagrant box.

Create the file “pg_hba.conf.j2” in ‘roles/postgresql/templates’

# Database administrative login by Unix domain socket
local   all             postgres                                peer
# TYPE  DATABASE        USER            ADDRESS                 METHOD
# "local" is for Unix domain socket connections only
local   all             all                                     peer
host    all             all             0.0.0.0/0               trust
host    all             all             all                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
host    all             all             192.168.60.70/24        trust
# IPv6 local connections:
host    all             all             ::1/128                 ident
# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                                     peer
host    replication     all             127.0.0.1/32            md5
host    replication     all             ::1/128                 md5

And the file “postgresql.conf.j2” in ‘roles/postgresql/templates’

# The default values of these variables are driven from the -D command-line
# option or PGDATA environment variable, represented here as ConfigDir.
data_directory = '/var/lib/postgresql/10/main'  # use data in another directory
     # (change requires restart)
hba_file = '/etc/postgresql/10/main/pg_hba.conf' # host-based authentication file
     # (change requires restart)
ident_file = '/etc/postgresql/10/main/pg_ident.conf' # ident configuration file
     # (change requires restart)
# If external_pid_file is not explicitly set, no extra PID file is written.
external_pid_file = '/var/run/postgresql/10-main.pid'   # write an extra PID file
     # (change requires restart)
#------------------------------------------------------------------------------
# CONNECTIONS AND AUTHENTICATION
#------------------------------------------------------------------------------
# - Connection Settings -
listen_addresses = '*'  # what IP address(es) to listen on;
     # comma-separated list of addresses;
     # defaults to 'localhost'; use '*' for all
     # (change requires restart)
port = 5432    # (change requires restart)
max_connections = 100   # (change requires restart)
#superuser_reserved_connections = 3 # (change requires restart)
unix_socket_directories = '/var/run/postgresql' # comma-separated list of directories
     # (change requires restart)
#unix_socket_group = ''   # (change requires restart)
#unix_socket_permissions = 0777  # begin with 0 to use octal notation
     # (change requires restart)
#bonjour = off    # advertise server via Bonjour
     # (change requires restart)
#bonjour_name = ''   # defaults to the computer name
     # (change requires restart)
# - Security and Authentication -
#authentication_timeout = 1min  # 1s-600s
ssl = on
#ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers
#ssl_prefer_server_ciphers = on
#ssl_ecdh_curve = 'prime256v1'
#ssl_dh_params_file = ''
ssl_cert_file = '/etc/ssl/certs/ssl-cert-snakeoil.pem'
ssl_key_file = '/etc/ssl/private/ssl-cert-snakeoil.key'
#ssl_ca_file = ''
#ssl_crl_file = ''
#password_encryption = md5  # md5 or scram-sha-256
#db_user_namespace = off
#row_security = on
# GSSAPI using Kerberos
#krb_server_keyfile = ''
#krb_caseins_users = off
# - TCP Keepalives -
# see "man 7 tcp" for details
#tcp_keepalives_idle = 0  # TCP_KEEPIDLE, in seconds;
     # 0 selects the system default
#tcp_keepalives_interval = 0  # TCP_KEEPINTVL, in seconds;
     # 0 selects the system default
#tcp_keepalives_count = 0  # TCP_KEEPCNT;
     # 0 selects the system default
#------------------------------------------------------------------------------
# RESOURCE USAGE (except WAL)
#------------------------------------------------------------------------------
# - Memory -
shared_buffers = 128MB   # min 128kB
     # (change requires restart)
#huge_pages = try   # on, off, or try
     # (change requires restart)
#temp_buffers = 8MB   # min 800kB
#max_prepared_transactions = 0  # zero disables the feature
     # (change requires restart)
# Caution: it is not advisable to set max_prepared_transactions nonzero unless
# you actively intend to use prepared transactions.
#work_mem = 4MB    # min 64kB
#maintenance_work_mem = 64MB  # min 1MB
#replacement_sort_tuples = 150000 # limits use of replacement selection sort
#autovacuum_work_mem = -1  # min 1MB, or -1 to use maintenance_work_mem
#max_stack_depth = 2MB   # min 100kB
dynamic_shared_memory_type = posix # the default is the first option
     # supported by the operating system:
     #   posix
     #   sysv
     #   windows
     #   mmap
     # use none to disable dynamic shared memory
     # (change requires restart)
# - Disk -
#temp_file_limit = -1   # limits per-process temp file space
     # in kB, or -1 for no limit
# - Kernel Resource Usage -
#max_files_per_process = 1000  # min 25
     # (change requires restart)
#shared_preload_libraries = ''  # (change requires restart)
# - Cost-Based Vacuum Delay -
#vacuum_cost_delay = 0   # 0-100 milliseconds
#vacuum_cost_page_hit = 1  # 0-10000 credits
#vacuum_cost_page_miss = 10  # 0-10000 credits
#vacuum_cost_page_dirty = 20  # 0-10000 credits
#vacuum_cost_limit = 200  # 1-10000 credits
# - Background Writer -
#bgwriter_delay = 200ms   # 10-10000ms between rounds
#bgwriter_lru_maxpages = 100  # 0-1000 max buffers written/round
#bgwriter_lru_multiplier = 2.0  # 0-10.0 multiplier on buffers scanned/round
#bgwriter_flush_after = 512kB  # measured in pages, 0 disables
# - Asynchronous Behavior -
#effective_io_concurrency = 1  # 1-1000; 0 disables prefetching
#max_worker_processes = 8  # (change requires restart)
#max_parallel_workers_per_gather = 2 # taken from max_parallel_workers
#max_parallel_workers = 8  # maximum number of max_worker_processes that
     # can be used in parallel queries
#old_snapshot_threshold = -1  # 1min-60d; -1 disables; 0 is immediate
     # (change requires restart)
#backend_flush_after = 0  # measured in pages, 0 disables
#------------------------------------------------------------------------------
# WRITE AHEAD LOG
#------------------------------------------------------------------------------
# - Settings -
#wal_level = replica   # minimal, replica, or logical
     # (change requires restart)
#fsync = on    # flush data to disk for crash safety
     # (turning this off can cause
     # unrecoverable data corruption)
#synchronous_commit = on  # synchronization level;
     # off, local, remote_write, remote_apply, or on
#wal_sync_method = fsync  # the default is the first option
     # supported by the operating system:
     #   open_datasync
     #   fdatasync (default on Linux)
     #   fsync
     #   fsync_writethrough
     #   open_sync
#full_page_writes = on   # recover from partial page writes
#wal_compression = off   # enable compression of full-page writes
#wal_log_hints = off   # also do full page writes of non-critical updates
     # (change requires restart)
#wal_buffers = -1   # min 32kB, -1 sets based on shared_buffers
     # (change requires restart)
#wal_writer_delay = 200ms  # 1-10000 milliseconds
#wal_writer_flush_after = 1MB  # measured in pages, 0 disables
#commit_delay = 0   # range 0-100000, in microseconds
#commit_siblings = 5   # range 1-1000
# - Checkpoints -
#checkpoint_timeout = 5min  # range 30s-1d
#max_wal_size = 1GB
#min_wal_size = 80MB
#checkpoint_completion_target = 0.5 # checkpoint target duration, 0.0 - 1.0
#checkpoint_flush_after = 256kB  # measured in pages, 0 disables
#checkpoint_warning = 30s  # 0 disables
# - Archiving -
#archive_mode = off  # enables archiving; off, on, or always
    # (change requires restart)
#archive_command = ''  # command to use to archive a logfile segment
    # placeholders: %p = path of file to archive
    #               %f = file name only
    # e.g. 'test ! -f /mnt/server/archivedir/%f && cp %p /mnt/server/archivedir/%f'
#archive_timeout = 0  # force a logfile segment switch after this
    # number of seconds; 0 disables
#------------------------------------------------------------------------------
# REPLICATION
#------------------------------------------------------------------------------
# - Sending Server(s) -
# Set these on the master and on any standby that will send replication data.
#max_wal_senders = 10  # max number of walsender processes
    # (change requires restart)
#wal_keep_segments = 0  # in logfile segments, 16MB each; 0 disables
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#max_replication_slots = 10 # max number of replication slots
    # (change requires restart)
#track_commit_timestamp = off # collect timestamp of transaction commit
    # (change requires restart)
# - Master Server -
# These settings are ignored on a standby server.
#synchronous_standby_names = '' # standby servers that provide sync rep
    # method to choose sync standbys, number of sync standbys,
    # and comma-separated list of application_name
    # from standby(s); '*' = all
#vacuum_defer_cleanup_age = 0 # number of xacts by which cleanup is delayed
# - Standby Servers -
# These settings are ignored on a master server.
#hot_standby = on   # "off" disallows queries during recovery
     # (change requires restart)
#max_standby_archive_delay = 30s # max delay before canceling queries
     # when reading WAL from archive;
     # -1 allows indefinite delay
#max_standby_streaming_delay = 30s # max delay before canceling queries
     # when reading streaming WAL;
     # -1 allows indefinite delay
#wal_receiver_status_interval = 10s # send replies at least this often
     # 0 disables
#hot_standby_feedback = off  # send info from standby to prevent
     # query conflicts
#wal_receiver_timeout = 60s  # time that receiver waits for
     # communication from master
     # in milliseconds; 0 disables
#wal_retrieve_retry_interval = 5s # time to wait before retrying to
     # retrieve WAL after a failed attempt
# - Subscribers -
# These settings are ignored on a publisher.
#max_logical_replication_workers = 4 # taken from max_worker_processes
     # (change requires restart)
#max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers
#------------------------------------------------------------------------------
# QUERY TUNING
#------------------------------------------------------------------------------
# - Planner Method Configuration -
#enable_bitmapscan = on
#enable_hashagg = on
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
#enable_seqscan = on
#enable_sort = on
#enable_tidscan = on
# - Planner Cost Constants -
#seq_page_cost = 1.0   # measured on an arbitrary scale
#random_page_cost = 4.0   # same scale as above
#cpu_tuple_cost = 0.01   # same scale as above
#cpu_index_tuple_cost = 0.005  # same scale as above
#cpu_operator_cost = 0.0025  # same scale as above
#parallel_tuple_cost = 0.1  # same scale as above
#parallel_setup_cost = 1000.0 # same scale as above
#min_parallel_table_scan_size = 8MB
#min_parallel_index_scan_size = 512kB
#effective_cache_size = 4GB
# - Genetic Query Optimizer -
#geqo = on
#geqo_threshold = 12
#geqo_effort = 5   # range 1-10
#geqo_pool_size = 0   # selects default based on effort
#geqo_generations = 0   # selects default based on effort
#geqo_selection_bias = 2.0  # range 1.5-2.0
#geqo_seed = 0.0   # range 0.0-1.0
# - Other Planner Options -
#default_statistics_target = 100 # range 1-10000
#constraint_exclusion = partition # on, off, or partition
#cursor_tuple_fraction = 0.1  # range 0.0-1.0
#from_collapse_limit = 8
#join_collapse_limit = 8  # 1 disables collapsing of explicit
     # JOIN clauses
#force_parallel_mode = off
#------------------------------------------------------------------------------
# ERROR REPORTING AND LOGGING
#------------------------------------------------------------------------------
# - Where to Log -
#log_destination = 'stderr'  # Valid values are combinations of
     # stderr, csvlog, syslog, and eventlog,
     # depending on platform.  csvlog
     # requires logging_collector to be on.
# This is used when logging to stderr:
#logging_collector = off  # Enable capturing of stderr and csvlog
     # into log files. Required to be on for
     # csvlogs.
     # (change requires restart)
# These are only used if logging_collector is on:
#log_directory = 'log'   # directory where log files are written,
     # can be absolute or relative to PGDATA
#log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # log file name pattern,
     # can include strftime() escapes
#log_file_mode = 0600   # creation mode for log files,
     # begin with 0 to use octal notation
#log_truncate_on_rotation = off  # If on, an existing log file with the
     # same name as the new log file will be
     # truncated rather than appended to.
     # But such truncation only occurs on
     # time-driven rotation, not on restarts
     # or size-driven rotation.  Default is
     # off, meaning append to existing files
     # in all cases.
#log_rotation_age = 1d   # Automatic rotation of logfiles will
     # happen after that time.  0 disables.
#log_rotation_size = 10MB  # Automatic rotation of logfiles will
     # happen after that much log output.
     # 0 disables.
# These are relevant when logging to syslog:
#syslog_facility = 'LOCAL0'
#syslog_ident = 'postgres'
#syslog_sequence_numbers = on
#syslog_split_messages = on
# This is only relevant when logging to eventlog (win32):
# (change requires restart)
#event_source = 'PostgreSQL'
# - When to Log -
#client_min_messages = notice  # values in order of decreasing detail:
     #   debug5
     #   debug4
     #   debug3
     #   debug2
     #   debug1
     #   log
     #   notice
     #   warning
     #   error
#log_min_messages = warning  # values in order of decreasing detail:
     #   debug5
     #   debug4
     #   debug3
     #   debug2
     #   debug1
     #   info
     #   notice
     #   warning
     #   error
     #   log
     #   fatal
     #   panic
#log_min_error_statement = error # values in order of decreasing detail:
     #   debug5
     #   debug4
     #   debug3
     #   debug2
     #   debug1
     #   info
     #   notice
     #   warning
     #   error
     #   log
     #   fatal
     #   panic (effectively off)
#log_min_duration_statement = -1 # -1 is disabled, 0 logs all statements
     # and their durations, > 0 logs only
     # statements running at least this number
     # of milliseconds
# - What to Log -
#debug_print_parse = off
#debug_print_rewritten = off
#debug_print_plan = off
#debug_pretty_print = on
#log_checkpoints = off
#log_connections = off
#log_disconnections = off
#log_duration = off
#log_error_verbosity = default  # terse, default, or verbose messages
#log_hostname = off
log_line_prefix = '%m [%p] %q%u@%d '  # special values:
     #   %a = application name
     #   %u = user name
     #   %d = database name
     #   %r = remote host and port
     #   %h = remote host
     #   %p = process ID
     #   %t = timestamp without milliseconds
     #   %m = timestamp with milliseconds
     #   %n = timestamp with milliseconds (as a Unix epoch)
     #   %i = command tag
     #   %e = SQL state
     #   %c = session ID
     #   %l = session line number
     #   %s = session start timestamp
     #   %v = virtual transaction ID
     #   %x = transaction ID (0 if none)
     #   %q = stop here in non-session
     #        processes
     #   %% = '%'
     # e.g. '<%u%%%d> '
#log_lock_waits = off   # log lock waits >= deadlock_timeout
#log_statement = 'none'   # none, ddl, mod, all
#log_replication_commands = off
#log_temp_files = -1   # log temporary files equal or larger
     # than the specified size in kilobytes;
     # -1 disables, 0 logs all temp files
log_timezone = 'UTC'
# - Process Title -
cluster_name = '10/main'   # added to process titles if nonempty
     # (change requires restart)
#update_process_title = on
#------------------------------------------------------------------------------
# RUNTIME STATISTICS
#------------------------------------------------------------------------------
# - Query/Index Statistics Collector -
#track_activities = on
#track_counts = on
#track_io_timing = off
#track_functions = none   # none, pl, all
#track_activity_query_size = 1024 # (change requires restart)
stats_temp_directory = '/var/run/postgresql/10-main.pg_stat_tmp'
# - Statistics Monitoring -
#log_parser_stats = off
#log_planner_stats = off
#log_executor_stats = off
#log_statement_stats = off
#------------------------------------------------------------------------------
# AUTOVACUUM PARAMETERS
#------------------------------------------------------------------------------
#autovacuum = on   # Enable autovacuum subprocess?  'on'
     # requires track_counts to also be on.
#log_autovacuum_min_duration = -1 # -1 disables, 0 logs all actions and
     # their durations, > 0 logs only
     # actions running at least this number
     # of milliseconds.
#autovacuum_max_workers = 3  # max number of autovacuum subprocesses
     # (change requires restart)
#autovacuum_naptime = 1min  # time between autovacuum runs
#autovacuum_vacuum_threshold = 50 # min number of row updates before
     # vacuum
#autovacuum_analyze_threshold = 50 # min number of row updates before
     # analyze
#autovacuum_vacuum_scale_factor = 0.2 # fraction of table size before vacuum
#autovacuum_analyze_scale_factor = 0.1 # fraction of table size before analyze
#autovacuum_freeze_max_age = 200000000 # maximum XID age before forced vacuum
     # (change requires restart)
#autovacuum_multixact_freeze_max_age = 400000000 # maximum multixact age
     # before forced vacuum
     # (change requires restart)
#autovacuum_vacuum_cost_delay = 20ms # default vacuum cost delay for
     # autovacuum, in milliseconds;
     # -1 means use vacuum_cost_delay
#autovacuum_vacuum_cost_limit = -1 # default vacuum cost limit for
     # autovacuum, -1 means use
     # vacuum_cost_limit
#------------------------------------------------------------------------------
# CLIENT CONNECTION DEFAULTS
#------------------------------------------------------------------------------
# - Statement Behavior -
#search_path = '"$user", public' # schema names
#default_tablespace = ''  # a tablespace name, '' uses the default
#temp_tablespaces = ''   # a list of tablespace names, '' uses
     # only default tablespace
#check_function_bodies = on
#default_transaction_isolation = 'read committed'
#default_transaction_read_only = off
#default_transaction_deferrable = off
#session_replication_role = 'origin'
#statement_timeout = 0   # in milliseconds, 0 is disabled
#lock_timeout = 0   # in milliseconds, 0 is disabled
#idle_in_transaction_session_timeout = 0 # in milliseconds, 0 is disabled
#vacuum_freeze_min_age = 50000000
#vacuum_freeze_table_age = 150000000
#vacuum_multixact_freeze_min_age = 5000000
#vacuum_multixact_freeze_table_age = 150000000
#bytea_output = 'hex'   # hex, escape
#xmlbinary = 'base64'
#xmloption = 'content'
#gin_fuzzy_search_limit = 0
#gin_pending_list_limit = 4MB
# - Locale and Formatting -
datestyle = 'iso, mdy'
#intervalstyle = 'postgres'
timezone = 'UTC'
#timezone_abbreviations = 'Default'     # Select the set of available time zone
     # abbreviations.  Currently, there are
     #   Default
     #   Australia (historical usage)
     #   India
     # You can create your own file in
     # share/timezonesets/.
#extra_float_digits = 0   # min -15, max 3
#client_encoding = sql_ascii  # actually, defaults to database
     # encoding
# These settings are initialized by initdb, but they can be changed.
lc_messages = 'en_US.UTF-8'   # locale for system error message
     # strings
lc_monetary = 'en_US.UTF-8'   # locale for monetary formatting
lc_numeric = 'en_US.UTF-8'   # locale for number formatting
lc_time = 'en_US.UTF-8'    # locale for time formatting
# default configuration for text search
default_text_search_config = 'pg_catalog.english'
# - Other Defaults -
#dynamic_library_path = '$libdir'
#local_preload_libraries = ''
#session_preload_libraries = ''
#------------------------------------------------------------------------------
# LOCK MANAGEMENT
#------------------------------------------------------------------------------
#deadlock_timeout = 1s
#max_locks_per_transaction = 64  # min 10
     # (change requires restart)
#max_pred_locks_per_transaction = 64 # min 10
     # (change requires restart)
#max_pred_locks_per_relation = -2 # negative values mean
     # (max_pred_locks_per_transaction
     #  / -max_pred_locks_per_relation) - 1
#max_pred_locks_per_page = 2            # min 0
#------------------------------------------------------------------------------
# VERSION/PLATFORM COMPATIBILITY
#------------------------------------------------------------------------------
# - Previous PostgreSQL Versions -
#array_nulls = on
#backslash_quote = safe_encoding # on, off, or safe_encoding
#default_with_oids = off
#escape_string_warning = on
#lo_compat_privileges = off
#operator_precedence_warning = off
#quote_all_identifiers = off
#standard_conforming_strings = on
#synchronize_seqscans = on
# - Other Platforms and Clients -
#transform_null_equals = off
#------------------------------------------------------------------------------
# ERROR HANDLING
#------------------------------------------------------------------------------
#exit_on_error = off   # terminate session on any error?
#restart_after_crash = on  # reinitialize after backend crash?
#data_sync_retry = off   # retry or panic on failure to fsync data?
#------------------------------------------------------------------------------
# CONFIG FILE INCLUDES
#------------------------------------------------------------------------------
# These options allow settings to be loaded from files other than the
# default postgresql.conf.
include_dir = 'conf.d'   # include files ending in '.conf' from
     # directory 'conf.d'
#include_if_exists = 'exists.conf' # include file only if it exists
#include = 'special.conf'  # include file
#------------------------------------------------------------------------------
# CUSTOMIZED OPTIONS
#------------------------------------------------------------------------------
# Add settings for extensions here

Actually, compared to the default config file, we are only changing one thing, the line 59 to: “listen_addresses = ‘*’”.

We also want to create a handler that restarts the postgresql server after we have written the two config files to the Vagrant box.

Create the file ‘main.yml’ in a folder “roles/postgresql/handlers”:

- name: restart postgresql
  service:
    name: postgresql
    state: restarted

Now, you can run:

$ ansible-playbook -i ./hosts_vagrant playbook.yml

and you should see something like:

PLAY [postgresdb] ***********************************************************************************************************************
TASK [Install python for Ansible] *******************************************************************************************************
ok: [192.168.60.70]
TASK [setup] ****************************************************************************************************************************
ok: [192.168.60.70]
TASK [base : Install base dependencies] *************************************************************************************************
ok: [192.168.60.70]
TASK [base : Upgrade pip] ***************************************************************************************************************
ok: [192.168.60.70]
TASK [postgresql : Install postgresql] **************************************************************************************************
ok: [192.168.60.70]
TASK [postgresql : Create pg_hba.conf file in /etc/postgresql/10/main] ******************************************************************
ok: [192.168.60.70]
TASK [postgresql : Create posgtresql.conf file in /etc/postgresql/10/main] **************************************************************
ok: [192.168.60.70]
TASK [postgresql : Enable passwordless sudo] ********************************************************************************************
ok: [192.168.60.70]
TASK [postgresql : Create a PostgreSQL database user] ***********************************************************************************
ok: [192.168.60.70]
TASK [postgresql : Create acme database] ************************************************************************************************
ok: [192.168.60.70]
PLAY RECAP ******************************************************************************************************************************
192.168.60.70              : ok=10   changed=0    unreachable=0    failed=0

(instead of the green ‘ok’ you might see a different message if it the first time you run the script).

If you want to test that you can access the database on the Vagrant box, you can create this little Python3 script (call the file test_db.py):

import psycopg2
try:
    connection = psycopg2.connect(
        user = "project",
        password = "project",
        host = "192.168.60.70",
        # host = "127.0.0.1",
        port = "5432",  # use 5433 if you use localhost
        database = "acme",
)
    cursor = connection.cursor()
    # Print PostgreSQL Connection properties
    print ( connection.get_dsn_parameters(),"\n")
    # Print PostgreSQL version
    cursor.execute("SELECT version();")
    record = cursor.fetchone()
    print("You are connected to - ", record,"\n")
except (Exception, psycopg2.Error) as error :
    print ("Error while connecting to PostgreSQL", error)
    connection = None
finally:
    #closing database connection.
        if(connection):
            cursor.close()
            connection.close()
            print("PostgreSQL connection is closed")

and run it with:

$ python3 test_db.py

and you should see something like:

{'user': 'project', 'dbname': 'acme', 'host': '192.168.60.70', 'port': '5432', 'tty': '', 'options': '', 'sslmode': 'prefer', 'sslcompression': '1', 'krbsrvname': 'postgres', 'target_session_attrs': 'any'}
You are connected to -  ('PostgreSQL 10.7 (Ubuntu 10.7-0ubuntu0.18.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0, 64-bit',)
PostgreSQL connection is closed

Django's select_related

2019-03-02T00:00:00+01:00

How does Django's select_related works and when should you use it?

What problem does it solve?

Django's select_related() solves the problem when you want to access related fields on a queryset. Assume the models:

from django.db import models


class Author(models.Model):
    name = models.CharField(max_length=250)


class Book(models.Model):
    name = models.CharField(max_length=150)
    author = models.ForeignKey(Author, on_delete=models.CASCADE)

Lets say we want to select all books and subsequently want to loop over all the books and print the authors:

>>> db.reset_queries()
>>> books = Book.objects.all()
>>> for book in books:
...     print(book.author)
... 
(0.000) SELECT "books_book"."id", "books_book"."name", "books_book"."author_id" FROM "books_book"; args=()
(0.000) SELECT "books_author"."id", "books_author"."name" FROM "books_author" WHERE "books_author"."id" = 18; args=(18,)
Author object (18)
(0.000) SELECT "books_author"."id", "books_author"."name" FROM "books_author" WHERE "books_author"."id" = 17; args=(17,)
Author object (17)
(0.000) SELECT "books_author"."id", "books_author"."name" FROM "books_author" WHERE "books_author"."id" = 18; args=(18,)
Author object (18)
(0.000) SELECT "books_author"."id", "books_author"."name" FROM "books_author" WHERE "books_author"."id" = 20; args=(20,)
Author object (20)
(0.000) SELECT "books_author"."id", "books_author"."name" FROM "books_author" WHERE "books_author"."id" = 17; args=(17,)
Author object (17)
>>> print(len(db.connection.queries))
6
>>>

A total of 6 queries are run (we had 5 books and then one query for the initial select *).
Note that the original .all() query had the SQL:

SELECT 
   "books_book"."id", 
   "books_book"."name", 
   "books_book"."author_id" 
FROM 
   "books_book"

select_related to the rescue

If instead we run:

>>> db.reset_queries()
>>> books = Book.objects.select_related('author').all()
>>> for book in books:
...     print(book.author.name)
... 
(0.000) SELECT "books_book"."id", "books_book"."name", "books_book"."author_id", "books_author"."id", "books_author"."name" FROM "books_book" INNER JOIN "books_author" ON ("books_book"."author_id" = "books_author"."id"); args=()
mary poppins
Tom clancy
mary poppins
kass larasdf
Tom clancy
>>> print(len(db.connection.queries))
1
>>>

Only one query was run. This is because the select_related() creates an INNER JOIN on the related table. The SQL becomes:

SELECT 
   "books_book"."id", 
   "books_book"."name", 
   "books_book"."author_id", 
   "books_author"."id", 
   "books_author"."name" 
FROM 
   "books_book" INNER JOIN "books_author" ON ("books_book"."author_id")

Chaining order doesn't matter

For the select_related() it doesn't matter which order you do the chaining in. The following two queries are identical:

books = Book.objects.select_related('author').all()
books = Book.objects.all().select_related('author')

What should be in the select_related(XYZ)?

This should be the name of the related model from the original model. If no 'related_name' has been specified, it will simply be the lower-case version of the model name.

Related fields

Lets extend the model with a Nationality table:

from django.db import models


class Nationality(models.Model):
    country = models.CharField(max_length=150, default="DK")


class Author(models.Model):
    name = models.CharField(max_length=250)
    country = models.ForeignKey(Nationality, related_name='authors', on_delete=models.CASCADE, blank=True, default=None)

class Book(models.Model):
    name = models.CharField(max_length=150)
    author = models.ForeignKey(Author, on_delete=models.CASCADE)

If we want to include the related 'Nationality', we can do:

>>> for book in books:
...     print(book.author.country.country)
... 
(0.001) SELECT "books_book"."id", "books_book"."name", "books_book"."author_id", "books_author"."id", "books_author"."name", "books_author"."country_id", "books_nationality"."id", "books_nationality"."country" FROM "books_book" INNER JOIN "books_author" ON ("books_book"."author_id" = "books_author"."id") LEFT OUTER JOIN "books_nationality" ON ("books_author"."country_id" = "books_nationality"."id"); args=()
Denmark
Denmark
Denmark
Denmark
Denmark
>>> print(len(db.connection.queries))
1

Again, only one query. The SQL is:

SELECT 
   "books_book"."id", 
   "books_book"."name", 
   "books_book"."author_id", 
   "books_author"."id", 
   "books_author"."name", 
   "books_author"."country_id", 
   "books_nationality"."id", 
   "books_nationality"."country" 
FROM 
   "books_book" INNER JOIN "books_author" ON ("books_book"."author_id" = "books_author"."id") 
   LEFT OUTER JOIN "books_nationality" ON ("books_author"."country_id" = "books_nationality"."id")

Setup shell

>>> from books.models import Book, Author, Nationality
>>> from django.db import connection, transaction
>>> import logging
>>> from django import db
>>> l = logging.getLogger('django.db.backends')
>>> l.setLevel(logging.DEBUG)
>>> l.addHandler(logging.StreamHandler())

In short (oneliner):

from books.models import Book, Author, Nationality;from django.db import connection, transaction;import logging;l = logging.getLogger('django.db.backends');l.setLevel(logging.DEBUG);l.addHandler(logging.StreamHandler());from django import db

Seed the db

from random import choice

from django.shortcuts import render
from django.http import HttpResponse

from books.models import Book, Author, Nationality


# Create your views here.
def index(request):
    return HttpResponse("Hello, world. You're at the polls index.")


def seed_db(request):
    authors = ['Tom clancy', 'mary poppins', 'marcus aur', 'kass larasdf']
    books = ['alpha', 'beta', 'gamma', 'epsilon', 'zeta']
    countries = ["Denmark", "Hong Kong"]

    author_list = []
    country_list = []

    for country in countries:
        nationality = Nationality(country=choice(countries))
        nationality.save()
        country_list.append(nationality)

    for author in authors:
        author = Author(name=author, country=choice(country_list))
        author.save()
        author_list.append(author)

    for book in books:
        Book(name=book, author=choice(author_list)).save()



    return HttpResponse("The db was seeded")

IP address and subnets (IPv4)

2018-04-26T00:00:00+02:00

This blog post will show how IP addresses work, what subnetting is.

Binary to decimal for two byte numbers

First, it can be beneficial to memorize the following table, which is just a list of binary digits.

128  64  32  16  8  4  2  1

It makes it easy to convert between binary and decimal.

table:  128  64  32  16  8  4  2  1
binary:   1   0   0   1  1  0  0  1
gives (by using logical AND and adding the numbers):
128 + 16 + 8 + 1 = 153 as a decimal, thus:
(binary) 1001 1001 = 153 (decimal)

IP Addresses

As an example, lets look at the IP address: 192.168.1.5. Each number between the '.' is one octet (2 bytes). First, we break it down into binary numbers:

decimal\|	binary
192	1100 0000
168	1010 1000
1	0000 0001
5	0000 0101

So, an IP address can be broken up into 4 octets, which is equal to (4 * 8) 32 bits.

Subnet mask

A subnet mask is used by the TCP/IP protocol to determine whether a host is on the local subnet or on a remote network. It is not possible from the IP address alone to determine whether or not the IP refers to a local or remote network. In order to figure this out, the network mask is needed. The network mask is another 32 bit number. For example, a subnet mask could be 255.25.255.0 (in binary 11111111.11111111.11111111.00000000). By taking the logical AND between the IP address and Subnet Mask you can find the Network address and the Host address.

Network ID and subnet mask

Given a network ID: 192.168.1.0, how do we find the IP behind it?

If we can write the CIDR notation like this: 192.168.1.0/24 (CIDR=Classless Interdomain routing). The '/24' tells us how many binary digits are turned on in the subnet mask. 24 binary digits can be shown like this:

1111 1111 . 1111 1111 . 1111 1111 . 0000 0000
In digital it becomes:
255.255.255.0

This is actually the Subnet Mask.

Given a specific Subnet Mask, 255.255.248.0, how do we find the CIDR notation? From the Subnet Mask, we can see that the first 2 octets (255.255) are both = 255, so we need to figure out how many bits 248 is. We can do that by using our table from above:

table:  128  64  32  16  8  4  2  1
We add together from left to right until we hit 248:
128 + 64 = 192 (still less than 248)
128 + 64 + 32 = 224 (still less than 248)
128 + 64 + 32 + 16 = 240 (still less than 248)
128 + 64 + 32 + 16 + 8 = 248

So we know that there are 5 subnetting bits. The first two octets of our Mask (the 255.255) holds 8 bits each (255 = 1111 1111) that are turned on, we now need to add our 5 subnetting bits, so we get:

8 + 8 + 5 = 21

which represents our CIDR (255.255.248.0/21).

Lets say we have an IP address: 192.168.40.55, how do we find the Network ID? Lets write our IP as binary and do a bit of 'calculations':

IP: 192.168.40.55
Binary:     1100 0000 . 1010 1000 . 0010 1000 . 0011 0111
Subnet:     1111 1111 . 1111 1111 . 1111 1000 . 0000 0000
Do a logical AND
Network ID: 1100 0000 . 1010 1000 . 0010 1000 . 0000 0000
in digital:    192    .    168    .    40     .     0

So our Network ID is 192.168.40.0/21 (CIDR notation).
Subnet mask: 255.255.248.0

This leaves the whole 4th octet and 3 bits of the third octet as hosts bits, which allows for quite a lot of hosts. Calculating the number of available IPs in the hosts bits is done like this:

We have 3 bits of the 3rd octet and the full 4th octet:
Binary:  0111 1111 1111
Decimal: 2047

So 2047 available IP addresses.

Example 1

Mask: 255.255.248.0/21 and IP: 192.168.45.55. How do we find the Network ID?

IP: 192.168.45.55
Binary:     1100 0000 . 1010 1000 . 0010 1101 . 0011 0111
Subnet:     1111 1111 . 1111 1111 . 1111 1000 . 0000 0000
Do a logical AND
Network ID: 1100 0000 . 1010 1000 . 0010 1101 . 0000 0000
in digital:    192    .    168    .    45     .     0

So our Network ID is 192.168.45.0/21 (CIDR notation).
Subnet mask: 255.255.248.0

Example 2

Mask: 255.255.255.192.

a) What is the CIDR notation?

We know that the first 3 octets each have 8 bits turned on, so the issue becomes to find the number of bits turned on in the 4th octet (192).

table:  128  64  32  16  8  4  2  1
We add together from left to right until we hit 192:
128 + 64 = 192
digital: 192
binary:  1100 0000

So our 4th octet has 2 bits turned on (the 128 bit and the 63 bit). Adding all the turned on bits together: 8 + 8 + 8 + 2 = 26, so our CIDR notation becomes 255.255.255.192/26.

b) Given IP address 192.168.45.55, what is the Network ID?

IP: 192.168.45.55
Binary:     1100 0000 . 1010 1000 . 0010 1101 . 0011 0111
Subnet:     1111 1111 . 1111 1111 . 1111 1111 . 1100 0000
Do a logical AND
Network ID: 1100 0000 . 1010 1000 . 0010 1101 . 0000 0000
in digital:    192    .    168    .    45     .     0

So our Network ID is 192.168.45.0/26 (CIDR notation).
Subnet mask: 255.255.248.192

Example 3

a) Determine the address range covered by the network: 192.168.4.0/22?

Lets first find the subnet mask from the '/22'. The first 2 octets in the subnet masks will be 255 (adding up to the first 16 bits), this leaves (22-16) 6 bits left which (binary: 1111 1100 = 252) equals 252, so the subnet mask becomes 255.255.252.0.

In binary, the network ID is the first 22 bits of the IP in binary:

(192.168.4.0/22) '1100 0000.1010 1000.0000 01|00.0000 0000' = 1100 0000.1010 1000.0000 01

The available addresses will be:
from: 1100 0000 . 1010 1000 . 0000 0100 . 0000 0000
to:   1100 0000 . 1010 1000 . 0000 0111 . 1111 1111 
and in decimal:
from: 192.168.4.0
to: 192.168.7.255