Metadata-Version: 2.4
Name: django-govuk-notify-outbox
Version: 1.5.2
Summary: A reusable Django app to send emails via GOV.UK notify robustly and with good observability
Requires-Python: >=3.13
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: celery>=5.6.3
Requires-Dist: django>=6.0.0
Requires-Dist: notifications-python-client>=10.0.1
Dynamic: license-file

<p align="center">
  <img alt="django-govuk-notify-outbox logo - a blue envelope with a tick in a green circle" src="./.assets/django-govuk-notify-outbox-logo.svg" width="170" height="122.312">
</p>


<h3 align="center">django-govuk-notify-outbox</h3>

<p align="center">
  <em>A reusable Django app to send emails via GOV.UK notify robustly and with good observability.</em>
</p>

---

### Contents

- [Features](#features)
- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [Usage](#usage)
  - [Fire and forget](#fire-and-forget)
  - [Attaching to your own models](#attaching-to-your-own-models)
- [Email status descriptions](#email-status-descriptions)
  - [Happy path](#happy-path)
  - [Happy and unhappy paths](#happy-and-unhappy-paths)
- [Configuration](#configuration)
- [Architecture](#architecture)
  - [Why the `sync` task?](#why-the-sync-task)
  - [Why background tasks at all?](#why-background-tasks-at-all)
  - [Why store data at all?](#why-store-data-at-all)
- [License](#license)

---

## Features

- Sends emails outside of the request/response cycle (via Celery)
- Automatically retries communication with GOV.UK Notify up to a maximum number / maximum amount of time
- Usage is consistent with the `send_email_notification` function from [notifications-python-client](https://github.com/alphagov/notifications-python-client)
- Emails and updates from GOV.UK Notify (including HTTP errors and Python exceptions) are stored in Django models...
- ... so can be linked (via foreign key or many-to-many relation) to other models to be used in the user-facing application
- ... and are visible in the Django admin interface

<p align="center"><img src="./.assets/admin-screenshot-email-details.png" width="617" height="392.5"></p>


## Prerequisites

- An existing (even if newly created) [Django](https://www.djangoproject.com/) project
- ... which is configured with [Celery](https://docs.celeryq.dev/) and [Celery Beat](https://docs.celeryq.dev/en/main/userguide/periodic-tasks.html) to run background tasks
- A GOV.UK Notify service API Key


## Installation

1. Install django-govuk-notify-outbox from PyPI.

   Typically you would do this by adding django-govuk-notify-outbox as a dependency to your existing Django project, for example with uv:

    ```bash
    uv add django-govuk-notify-outbox
    ```

2. Add `django_govuk_notify_outbox` to `settings.INSTALLED_APPS`

    ```python
    INSTALLED_APPS = [
        # ...
        "django_govuk_notify_outbox",
    ]

3. Add `settings.DJANGO_GOVUK_NOTIFY_OUTBOX__API_KEY` with your GOV.UK Notify API key, typically from an environment variable.

    ```python
    import os

    DJANGO_GOVUK_NOTIFY_OUTBOX__API_KEY = os.environ.get('DJANGO_GOVUK_NOTIFY_OUTBOX__API_KEY')
    ```

4. Add `django_govuk_notify_outbox.tasks.sync` to `settings.CELERY_BEAT_SCHEDULE` on a frequent schedule, for example every 5 seconds. If there is nothing to do, this will finish quickly and without connecting to GOV.UK notify.

    ```python
    CELERY_BEAT_SCHEDULE = {
        "sync": {
            "task": "django_govuk_notify_outbox.tasks.sync",
            "schedule": 5,
        },
    }

5. Run `python manage.py migrate` to create the models. Often projects are setup to do this automatically on startup.


## Usage

### Fire and forget

Once installed, emails can be sent from any part of your Django project with the `django_govuk_notify_outbox.utils.send_email_notification` function, which is designed to be consistent with the "fire and forget"-style `send_email_notification` function from [notifications-python-client](https://github.com/alphagov/notifications-python-client)

```python
from django_govuk_notify_outbox.utils import send_email_notification

send_email_notification(
    email_address="name@example.invalid",
    template_id="9d751e0e-f929-4891-82a1-a3e1c3c18ee3",
)
```

### Attaching to your own models

The return value of `send_email_notification` is an instance of the Django model `django_govuk_notify_outbox.models.EmailMessage`, and it can be attached to your own models in the usual way via `ForeignKey` or `ManyToManyField`. Each `EmailMessage` is updated through the email sending process until is is delivered or failed.

For example:

```python
# In your app's models.py
from django.db import models
from django_govuk_notify_outbox.models import EmailMessage

class MyModel(models.Model):
    email_messages = models.ManyToManyField(EmailMessage)

# Anywhere in your app
from django_govuk_notify_outbox.utils import send_email_notification

email_message = send_email_notification(
    email_address="name@example.invalid",
    template_id="9d751e0e-f929-4891-82a1-a3e1c3c18ee3",
)
my_model_instance = MyModel.create()
my_model_instance.email_messages.add(email_message)
print(email_message.status)  # See below
```


## Email status descriptions

Each `EmailMessage.status` field can be equal to one of 10 possible values. For the value that reflect a GOV.UK Notify status, see https://docs.notifications.service.gov.uk/python.html#email-status-descriptions for details.

| State                                        | Description
| :------------------------------------------- | :-----------
| `sending-to-notify`                          |  The email is about to be sent to GOV.UK Notify (or has very recently been sent).
| `retrying-send-to-notify`                    |  The email has failed sending to GOV.UK Notify at least once and will be retried (or has very recently been retried).
| `failed-send-to-notify`                      |  The email has failed sending to GOV.UK Notify and will not be retried.
| `sent-to-notify`                             |  The email has been sent to GOV.UK Notify
| `notify-created`                             | GOV.UK notify reports the email in the `created` state, which essentially means _queued_: send will be attempted in the next few seconds.
| `notify-sending`                             | GOV.UK notify reports the email in the `sending` state.
| `notify-delivered`                           | GOV.UK notify reports the email in the `delivered` state.
| `notify-temporary-failure`                   |   GOV.UK notify reports the email in the `temporary-failure` state. Note that this does not mean that the email will be retried automatically, but more reflects that if another identical email is sent, it may succeed.
| `notify-technical-failure`                   | GOV.UK Notify reports the email in the `technical-failure` state, which indicates a technical issue between Notify and its email provider. It will not be retried automatically. 
| `notify-permanent-failure`                   | GOV.UK notify reports the email in the `permanent-failure` state, which indicates it is not likely to succeed if another identical email is sent.

### Happy path

5 statuses make up the happy path when there are no problems sending an email:

```mermaid
stateDiagram-v2

    sendingToNotify: sending-to-notify
    sentToNotify: sent-to-notify
    notifyCreated: notify-created
    notifySending: notify-sending
    notifyDelivered: notify-delivered

    [*] --> sendingToNotify
    sendingToNotify --> sentToNotify
    sentToNotify --> notifyCreated
    notifyCreated --> notifySending
    notifySending --> notifyDelivered
    notifyDelivered --> [*]
```

### Happy and unhappy paths

How `EmailMessage.status` can change in all cases of both success and failure is shown in the following diagram:

```mermaid
stateDiagram-v2

    sendingToNotify: sending-to-notify
    retryingSendToNotify: retrying-send-to-notify
    sentToNotify: sent-to-notify
    failedSendToNotify: failed-send-to-notify
    notifyCreated: notify-created
    notifySending: notify-sending
    notifyDelivered: notify-delivered
    notifyPermanentFailure: notify-permanent-failure
    notifyTemporaryFailure: notify-temporary-failure
    notifyTechnicalFailure: notify-technical-failure

    [*] --> sendingToNotify

    sendingToNotify --> retryingSendToNotify
    retryingSendToNotify --> failedSendToNotify
    sendingToNotify --> sentToNotify
    retryingSendToNotify --> sentToNotify

    failedSendToNotify --> [*]

    sentToNotify --> notifyCreated
    notifyCreated --> notifySending

    notifySending --> notifyDelivered
    notifySending --> notifyPermanentFailure
    notifySending --> notifyTemporaryFailure
    notifySending --> notifyTechnicalFailure

    notifyPermanentFailure --> [*]
    notifyTemporaryFailure --> [*]
    notifyTechnicalFailure --> [*]

    notifyDelivered --> [*]
```

## Configuration

Configuration is via settings that go into your Django project's `settings.py`

| Setting                                                      | Description
| :----------------------------------------------------------- | :-----------
| `DJANGO_GOVUK_NOTIFY_OUTBOX__API_KEY`                        | The GOV.UK Notify API key for the service |
| `DJANGO_GOVUK_NOTIFY_OUTBOX__SEND_TO_NOTIFY_RETRY_INTERVALS` | A list of how long to wait in seconds between retrying send to GOV.UK Notify<br>Default: `[5,10,60,120,240,480,960,1920]` |
| `DJANGO_GOVUK_NOTIFY_OUTBOX__SEND_TO_NOTIFY_ABANDON_AFTER`   | How long in seconds to wait to abandon send to GOV.UK notify, even if there are remaining retries according to the `DJANGO_GOVUK_NOTIFY_OUTBOX__SEND_TO_NOTIFY_RETRY_INTERVALS` setting. This is used to not send emails if the background worker has been offline for a long period of time and so the emails are no longer relevant.<br>Default: `3600` |


## Architecture

### Why the `sync` task?

The sync task does two things. It retries sending to GOV.UK Notify, and it fetches updates on sent emails. There are are three reasons for this:

1. It is the most robust way to make sure retries are attempted, especially compared to a mechanism where a trigger-like mechanism is required: code can stop before the retry is triggered.
2. It offers natural throttling in the case of many emails, which is exactly a case of when errors can happen. There is little chance the servers (both ours and those of GOV.UK Notify) get overloaded by the sync task, especially if retrying with exponential backoff as is the default behaviour.
3. It forces state to be stored in Django models/the database for observability as opposed to what can be more emphemeral via the triggering of background tasks.

In terms of push-vs-pull, the happy path email is sent via a push mechansim triggering a background worker, offering the typical immediacy benefits, but retries are by the sync task "pulling" from the database, offering the typical stability benefits of that kind of architecture.

### Why background tasks at all?

Emails are sent from a background task rather than from the request/response cycle for several reasons all releated to a good user experience even in the case of problems with infrastructure:

- The connection to GOV.UK Notify can theoretically be slow; its Python API is documented to have a default timeout of 30 seconds. This is beyond what is typically acceptable in a request/response cycle from a UX point of view.
- GOV.UK Notify, just like any service, can go down.
- The network infrastructure between our server and GOV.UK Notify can go down, for example during infrastructure changes.
- And even if it didn't go down, GOV.UK Notify enforces rate limits.

### Why store data at all?

Observability without depending on another system helps debug issues, and being able to do this without engineer level access to low-level logs is incredible powerful. This includes users of your application because emails can be tied to standard Django models and surfaces as you see fit.

Also GOV.UK Notify does not track failures of sending emails due to network connection between your project and GOV.UK Notify, due to authentication issues, validation problems, or rate limiting. In order to track and debug such failures, they have to be stored _somewhere_.


## License

[MIT](LICENSE)


[def]: #
