CRUD.cloud

Debugging Django Email with Gmail in a Dockerized App

This post summarizes a troubleshooting session where I debugged email sending issues in a Django application. The app was deployed in a Docker container built with a multi-stage Dockerfile.prod file.

Impact

  • The production Django app could not send any email through Gmail SMTP, while the same code worked in local development.
  • A manual fix inside the running container worked, but was lost on every rebuild, so email kept breaking after each deployment.

Timeline

  1. Check the Django settings. I verified the email configuration in settings.py and that the correct Gmail App Password was used:
# settings.py
# EMAIL
EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"
EMAIL_HOST = "smtp.gmail.com"
EMAIL_HOST_USER = os.environ.get("EMAIL_HOST_USER")
EMAIL_HOST_PASSWORD = os.environ.get("EMAIL_HOST_PASSWORD")
EMAIL_PORT = 587
EMAIL_USE_TLS = True
EMAIL_USERS_TO_NOTIFY = os.environ.get("EMAIL_USERS_TO_NOTIFY").split(" ")
  1. Test from the server, outside Django. I wrote a small script with hardcoded credentials:
import os
import smtplib
from email.mime.text import MIMEText

EMAIL_HOST = "smtp.gmail.com"
EMAIL_PORT = 587
EMAIL_HOST_USER = "user@gmail.com"
EMAIL_HOST_PASSWORD = "psw-from-app-password"  # Generated from Gmail account: https://myaccount.google.com/apppasswords
EMAIL_TO = "test_recipient@example.com"
EMAIL_FROM = os.environ.get("EMAIL_HOST_USER")

try:
  with smtplib.SMTP(EMAIL_HOST, EMAIL_PORT) as server:
    server.starttls()
    server.login(EMAIL_HOST_USER, EMAIL_HOST_PASSWORD)

    msg = MIMEText("This is a test email.")
    msg["Subject"] = "Test Email"
    msg["From"] = EMAIL_FROM
    msg["To"] = EMAIL_TO

    server.send_message(msg)
    print("Email sent successfully!")
except Exception as e:
  print(f"Error sending email: {e}")

Running it on the server:

root@site-name:~/repo-name# python3 django-app/send_email.py
#... output showing successful email sending...

The email was sent, so the Gmail account, App Password and network were fine. The problem was in how the Dockerized Django app accessed its configuration, or something else in the container.

  1. Check environment variables in the container. I reviewed docker-compose.yml and entrypoint.sh, then checked the variables directly:
docker-compose exec django-app sh
# Inside the container:
echo $EMAIL_HOST_USER
user@gmail.com
echo $EMAIL_HOST_PASSWORD
xxxx xxxx xxxx xxxx

The variables were set correctly, yet I was getting a (334, b'UGFzc3dvcmQ6') error. 334 is an SMTP response meaning "continue with authentication", and b'UGFzc3dvcmQ6' is the base64-encoded string "Password:": Gmail was asking for the password but not getting a valid one.

  1. SSL certificate verification error. Sending a test email from Django inside the container:
docker-compose exec django-app sh
python manage.py sendtestemail user@gamil.com

returned:

user@gmail.com
Traceback (most recent call last):
  File "/app/manage.py", line 22, in <module>
  ...
    ~~~~~~~~~~~~~~~~~^^
  File "/python/cpython-3.13.2-linux-x86_64-gnu/lib/python3.13/ssl.py", line 1372, in do_handshake
    self._sslobj.do_handshake()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1028)

Python could not verify the certificate presented by smtp.gmail.com, which usually means a missing or outdated CA certificate bundle in the container.

  1. Check ca-certificates.
dpkg -l ca-certificates

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name            Version      Architecture Description
+++-===============-============-============-=================================
un  ca-certificates <none>       <none>       (no description available)

The package was in the un state: not installed.

  1. Manual workaround. I reinstalled the package as root in the running container:
docker-compose exec --user root django-app sh
apt-get purge ca-certificates
apt-get install ca-certificates

🎉 Emails were sent successfully. 🥳 This confirmed the problem was in the Docker build of Dockerfile.prod.

  1. First fix attempt. I added ca-certificates to the third build stage of Dockerfile.prod:
#... other stages...

# ###############################################################################
# Stage 3: Building environment
# ###############################################################################
FROM python-base AS builder-base

#... other commands...

# Install ca-certificates and other dependencies
RUN apt-get update && apt-get install -y \
    ca-certificates \
    #... other packages... \
    && rm -rf /var/lib/apt/lists/*

#... other commands...
  1. The issue persisted 😢. After every rebuild, email stopped working again and the package was still missing:
docker-compose -f docker-compose-prod.yml exec --user root django-app dpkg -l ca-certificates

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name            Version      Architecture Description
+++-===============-============-============-=================================
un  ca-certificates <none>       <none>       (no description available)

I dug into the verbose output of docker-compose build, used intermediate images to find where the installation failed, and considered package conflicts, base image issues, apt problems, and even copying certificates manually. Reinstalling by hand after every build was not an option 🙄.

  1. Resolution. Re-reading Dockerfile.prod revealed the stage problem (see Root cause). Moving the install to the base stage fixed it for good.

Root cause

ca-certificates was installed in the builder-base stage, but the final webapp stage starts fresh FROM linux-base. Nothing installed with apt in builder-base is carried over: only the files explicitly copied with COPY --from=builder-base are. The final image therefore had no CA bundle, and Python's SSL verification of smtp.gmail.com failed.

Actions

Move the ca-certificates installation to the linux-base stage, which all other stages inherit from:

# syntax=docker.io/docker/dockerfile:1.7-labs

# ###############################################################################
# Stage 1: General debian environment
# ###############################################################################
FROM debian:stable-slim AS linux-base

# Assure UTF-8 encoding is used.
ENV LC_CTYPE=C.utf8
# Location of the virtual environment
ENV UV_PROJECT_ENVIRONMENT="/venv"
# Location of the python installation via uv
ENV UV_PYTHON_INSTALL_DIR="/python"
# Byte compile the python files on installation
ENV UV_COMPILE_BYTECODE=1
# Python verision to use
ENV UV_PYTHON=python3.13
# Tweaking the PATH variable for easier use
ENV PATH="$UV_PROJECT_ENVIRONMENT/bin:$PATH"

ENV APP_HOME=/app/django-app

# Update debian and install base packages
RUN apt-get update && \
    apt-get upgrade -y && \
    apt-get install --no-install-recommends -y \
    tzdata \
    ca-certificates && \
    update-ca-certificates && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*


# ###############################################################################
# Stage 2: Python environment
# ###############################################################################
FROM linux-base AS python-base

# Install debian dependencies
RUN apt-get update && \
    apt-get install --no-install-recommends -y build-essential gettext && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Install uv
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv

# Create virtual environment and install dependencies
COPY pyproject.toml ./
COPY uv.lock ./
RUN uv sync --all-extras --frozen --no-dev --no-install-project


# ###############################################################################
# Stage 3: Building environment
# ###############################################################################
FROM python-base AS builder-base

WORKDIR /app
COPY . /app

# Add build-time environment variables
ENV DJANGO_ALLOWED_HOSTS="localhost 127.0.0.1"
ENV CSRF_TRUSTED_ORIGINS="http://localhost https://localhost"
ENV EMAIL_USERS_TO_NOTIFY="dummy-email-value@gmail.com"

# Then collect all static files
RUN python manage.py collectstatic --noinput -v 2


# ###############################################################################
# Stage 4: Webapp environment
# ###############################################################################
FROM linux-base AS webapp

# Create a non-root user
RUN useradd -m appuser

# Copy python, virtual env and app
COPY --from=builder-base $UV_PYTHON_INSTALL_DIR $UV_PYTHON_INSTALL_DIR
COPY --from=builder-base $UV_PROJECT_ENVIRONMENT $UV_PROJECT_ENVIRONMENT
COPY --from=builder-base /app /app

# Change ownership of the application files to the non-root user
RUN chown -R appuser:appuser /app $UV_PROJECT_ENVIRONMENT $UV_PYTHON_INSTALL_DIR \
    && chmod +x /app/entrypoint.sh

# Switch to the non-root user
USER appuser

WORKDIR /app

RUN mkdir -p $APP_HOME/mediafiles

EXPOSE 8000

ENTRYPOINT ["/app/entrypoint.sh"]

Final check, after a rebuild:

docker-compose -f docker-compose-prod.yml exec --user root django-app dpkg -l ca-certificates

returns the package as installed (ii):

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name            Version      Architecture Description
+++-===============-============-============-=================================
ii  ca-certificates 20230311     all          Common CA certificates

Lessons learned

  • Dockerfile stages and dependencies: understand what is carried over between stages in a multi-stage build. Packages installed in an intermediate stage don't reach a final stage that starts from another base.
  • SSL certificate verification: make sure your Docker images include the CA certificates they need.
  • Isolate layers when debugging: a standalone script on the host ruled out Gmail, credentials and network in one step.
  • Environment variables in Docker: check them from inside the running container rather than assuming they're passed correctly.
  • Debugging Docker builds: verbose build output and intermediate images help pinpoint the failing step.
  • Manual intervention for debugging: installing packages in a running container is a useful way to confirm a hypothesis, not a fix.