FastAPI is a framework for building APIs with Python based on standard Python type hints. It’s fast and easy to stand up, and as a Python developer, it was relatively easy to pick up and get started with it.

As part of deploying a FastAPI instance in production, I integrated FastAPI with Entra ID for OAuth2 via this really cool library, FastAPI-Azure-Auth.

After integrating this library into my FastAPI instance, I was able to secure access to the API server. However, I quickly ran into an interesting problem. While trying to connect a PowerBI instance to the API, I found that the API settings needed some more configuration to enable interactive logging in via Microsoft SSO within PowerBI.

Some context

FastAPI needs to be configured to return a WWW-Authenticate header when a user attempts to access the API without authorization (a HTTP 401 Unauthorized) error.

When an unauthenticated access request is made to a secure API, it can be configured to return a header called WWW-Authenticate within it’s response when it returns a HTTP 401 Unauthorized error.

The WWW-Authenticate header returns information that can be used for the user to authenticate themselves and return to the API with a valid, unexpired authentication token.

As there is no element of interaction with a HTTP API, the onus is on the application/user that is querying the API to extract the information returned in the WWW-Authenticate header and to proceed to the identity provider (in this case, Microsoft Entra ID.)

In our case, when PowerBI (via Power Query) requests access to a Web API via an Organizational account, it sends a request to the endpoint to the URL with an empty bearer token. It then expects a WWW-Authenticate header along with the Microsoft Entra ID authorization URI to use. This authorization URI should return the tenant that is to be used for the OAuth2 process. Therefore, all we need to do is configure an error handler within the FastAPI instance to return a WWW-Authenticate header when it returns a HTTP 401 Unauthorized error.

The Power Query documentation documents the supported workflow when trying to access a Web API via an Organizational account.

From the documentation, this is the reponse expected by Power Query when it is given a HTTP 401 Unauthorized error.

HTTP/1.1 401 Unauthorized
Cache-Control: private
Content-Type: text/html
Server:
WWW-Authenticate: Bearer authorization_uri=https://login.microsoftonline.com/aaaabbbb-0000-cccc-1111-dddd2222eeee/oauth2/authorize
Date: Wed, 15 Aug 2018 15:02:04 GMT
Content-Length: 49

So to sum it up, we need to configure FastAPI to return a WWW-Authenticate header when it returns a HTTP 401 Unauthorized error, and this header needs to return the authorization URI associated with the tenant that is able to access the API service via OAuth2.

The Solution

Note that this guide assumes that you have already configured your FastAPI instance to work with Entra ID for OAuth2 flows.

As a general example (which may differ from your set-up), I usually have my code that handles the Entra ID authentication flow within a folder called modules in the project root, in a file called azure.py, which I can then import into my main.py file.

Following from the basic example within the FastAPI-Azure-Auth documentation, here’s what azure.py looks like:

from fastapi_azure_auth import SingleTenantAzureAuthorizationCodeBearer
from pydantic import AnyHttpUrl, computed_field
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    BACKEND_CORS_ORIGINS: list[str | AnyHttpUrl] = [
        "https://your.server.com",
    ]
    OPENAPI_CLIENT_ID: str = ""
    APP_CLIENT_ID: str = ""
    TENANT_ID: str = ""
    SCOPE_DESCRIPTION: str = "user_impersonation"

    @computed_field
    @property
    def SCOPE_NAME(self) -> str:
        return f"api://{self.APP_CLIENT_ID}/{self.SCOPE_DESCRIPTION}"

    @computed_field
    @property
    def SCOPES(self) -> dict:
        return {
            self.SCOPE_NAME: self.SCOPE_DESCRIPTION,
        }

    @computed_field
    @property
    def OPENAPI_AUTHORIZATION_URL(self) -> str:
        return (
            f"https://login.microsoftonline.com/{self.TENANT_ID}/oauth2/v2.0/authorize"
        )

    @computed_field
    @property
    def OPENAPI_TOKEN_URL(self) -> str:
        return f"https://login.microsoftonline.com/{self.TENANT_ID}/oauth2/v2.0/token"

settings = Settings()

azure_scheme = SingleTenantAzureAuthorizationCodeBearer(
    app_client_id=settings.APP_CLIENT_ID,
    tenant_id=settings.TENANT_ID,
    scopes=settings.SCOPES,
)

The Error Handler

Define a custom error handler within your FastAPI code. For example, create a folder called modules in your project root, and create a file called errors.py within modules. Put the following in that file.

from fastapi import Request, HTTPException
from fastapi.responses import JSONResponse
from modules import azure

async def custom_http_exception_handler(request: Request, exc: HTTPException):
    if exc.status_code == 401:
        auth_url = f"https://login.microsoftonline.com/{azure.settings.TENANT_ID}/oauth2/authorize"
        return JSONResponse(
            status_code=401,
            content={"detail": "Not authenticated"},
            headers={
            "WWW-Authenticate": f'Bearer authorization_uri="{auth_url}"'
            },
        )
    else:
        return JSONResponse(
            status_code=exc.status_code, 
            content={"detail": exc.detail})

In this code snippet, we are doing the following:

• When a HTTP Exception occurs, the functions checks to see if a 401 error is returned.
• We define the URL for OAuth2 authorization flow by using the tenant ID taken from the Azure config.
• We construct and return the WWW-Authenticate header to be compliant with the format expected by Power Query.
• For HTTP Errors that are not 401, the default response is passed back.

We can then import this error handler in main.py as follows:

from typing import Annotated, AsyncGenerator
from contextlib import asynccontextmanager
from starlette.middleware.cors import CORSMiddleware
from fastapi import FastAPI, HTTPException, Security
from fastapi_azure_auth.user import User
from modules import azure
from modules.errors import custom_http_exception_handler

settings = azure.settings
azure_scheme = azure.azure_scheme

@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
    """
    Load OpenID config on startup.
    """
    await azure_scheme.openid_config.load_config()
    yield

app = FastAPI(
    lifespan=lifespan,
    swagger_ui_oauth2_redirect_url="/api/oauth2-redirect",
    swagger_ui_init_oauth={
        "usePkceWithAuthorizationCodeGrant": True,
        "clientId": settings.OPENAPI_CLIENT_ID,
        "scopes": settings.SCOPE_NAME,
    },
)

if settings.BACKEND_CORS_ORIGINS:
    app.add_middleware(
        CORSMiddleware,
        allow_origins=[str(origin) for origin in settings.BACKEND_CORS_ORIGINS],
        allow_credentials=True,
        allow_methods=["*"],
        allow_headers=["*"],
    )
app.add_exception_handler(HTTPException, custom_http_exception_handler)

@app.get("/", dependencies=[Security(azure_scheme)])
async def root():
    return {"message": "Hello World"}

if __name__ == '__main__':
    uvicorn.run('main:app', reload=True)

With this error handler imported, PowerBI should be able to use the Power Query Connector to enable SSO via your Microsoft login when it is attempting to connect to your API!

References

• FastAPI-Azure-Auth documentation
• Microsoft Learn documentation on the Power Query connector and authentication flows
• Documenation on the WWW-Authenticate header standards

When deploying internal services/tools on a private intranet, these services are usually hosted on servers that do not have a public IP address interface enabled, and only have a private IP address that only accepts connections through a firewall/IP whitelist. This approach ensures that these tools are only accessible internally, and are not public facing.

An approach like this, however, has a few caveats to be aware of. If you want to make requests to these services hosted on private servers, there are two options.

• Referring to the IP address directly
• Referring to a domain name. This requires that your DNS service is correctly configured to have the domain name resolve to the IP address associated with it.

Either way, without any further configuration, if you were to simply call these services either by the IP address or the domain name, you are greeted by warnings and disclaimers that state that your connection to this service is insecure. By default, these tools will not allow you to access these services without manually allowing the connection or by permitting the use of an insecure connection.

For example, if I was to host an API server on a VM with a private IP address of 10.0.0.1, and accessed it as follows:

curl https://10.0.0.1/api

Or, if your DNS server resolves the domain name dev.company.com to the 10.0.0.1 IP address,

curl https://dev.company.com/api

This is the following output you can expect:

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

These errors occur because these services expect a SSL certificate signed by a trusted CA by default. You will run into similar warnings when accessing front-end tools hosted on these servers as well.

There are a few options here.

• Use a self-signed SSL certificate.

This approach doesn’t negate the warnings! A self-signed SSL certificate is still considered insecure by default.

• Spin up an internal CA

When you spin up an internal CA and generate certificates signed by this local CA, you will then have to install this certificate on the end user’s devices, which might not be a feasible task. Certificates can expire, and you will have to automate the renewal, propagation and installation of the new certificates on the end-user devices on a regular basis.

The first approach doesn’t fix the problem, and the second approach requires a lot of infrastructure and effort to be built.

By far the easiest approach would be to have a publicly signed SSL certificate. This would remove the need to create your own internal CA service and the need to then install these local SSL certs on end-user devices.

However, this is where we run into the problem I’m hoping to address with this blog post. Globally trusted CAs usually do not issue SSL certificates for private IP addresses.

Even if you decided to issue an SSL certificate for your private domain name, you would need to create an external A record for it associated with your server’s public IP address, which might be not be something you want to expose.

However, if your company already has a public domain name (which most companies do), with an associated DNS provider, it is entirely possible to have a publicly signed SSL certificate hosted on your private intranet server, without ever needing to expose your server’s public IP address.

In this blog post, I will demonstrate how you can set this up and then automate the process of generating publicly signed SSL certificates for your private domain without the need for publicly exposing your server’s IP address.

First, let’s go over some basic fundamentals.

Let’s Encrypt and the ACME protocol

Let’s Encrypt is a Certificate Authority that provides free TLS/SSL certificates, and has been doing so for a full decade. It is a truly fundamental part of modern internet infrastructure, and is a project of the nonprofit Internet Security Research Group. Let’s Encrypt automates the issuing of certificates through an API that is based on the ACME protocol.

The ACME (Automated Certificate Management Environment) protocol is a communication protocol that makes it possible to automatically obtain trusted certificates without human intervention, at very low cost. ACME is on version 2 of it’s API, which was released on March 13, 2018, and this protocol was designed by the Internet Security Research Group for the Let’s Encrypt Service.

Since Let’s Encrypt requires the use of the ACME protocol, you will need an ACME client to interact with it programmatically.

dehydrated, a bash based ACME client

dehydrated is a bash-based ACME Client that supports both the ACME v1 and ACME v2 protocols.

Lightweight (it’s pretty much a bash script!) and simple with an MIT license, this is what we’ll be using as our ACME client.

ACME Challenges

When requesting a signed certificate from Let’s Encrypt via an ACME client, Let’s Encrypt validates that you control the domain name by issuing challenges defined by the ACME standard. There are three types of challenges, HTTP-01, DNS-01 and TLS-ALPN-01.

TLS-ALPN-01 is highly technical and uses a custom ALPN protocol, and isn’t very relevant for this post, so we won’t go over it this time.

HTTP-01

This is the most common challenge type, where the client proves control over a domain name by proving that it can provision HTTP resources on a server accessible under that domain name.

This challenge type requires the creation of A/AAAA records with your DNS provider. During the challenge process, Let’s Encrypt servers will connect to at least one of the hosts found in the DNS A and AAAA records. For this challenge to successfully validate, the server must be reachable at the specified IP address.

This is why we won’t be going with HTTP-01 challenges in this blog post, since that would defeat the purpose of having a private intranet to begin with.

DNS-01

DNS-01 challenges require the client to provision a TXT record containing a designated value for a specific validation domain name.

After Let’s Encrypt issues the ACME client a token as part of validation, the client will create a TXT record with your DNS provider derived from that token and account key. Let’s Encrypt will then query the DNS system for that record, and will then issue a certificate if it finds a match.

This is the challenge type we will be using in this blog post, since this approach does not require us to expose the servers to the public internet. However, this approach requires that your DNS provider has API functionality to automate this process.

Prerequisites

With a brief explanation of the fundamentals completed, let’s move forward with the prerequisites you will need before following along with this post.

You will need:

• A DNS provider with API functionality.

• Your DNS server (different from your DNS provider) maps the IP address of your private server to the domain name you intend to use.

• cURL, sed, grep, awk, mktemp and openssl on your Unix OS.

Setup

Installing the dehydrated client

The dehydrated ACME client can be installed as follows:

# Downloading the Dehydrated client
cd /etc
git clone https://github.com/dehydrated-io/dehydrated

Running this code will pull the dehydrated library to the /etc. Further configuration is needed.

Configuring the dehydrated client

# Creating the necessary folders in the `dehydrated` folder
cd /etc/dehydrated
mkdir certs accounts records_backup

Running this code will create three folders in the dehydrated folder, which will now have the following content:

dehydrated/
├── CHANGELOG
├── LICENSE
├── README.md
├── accounts
├── certs
├── dehydrated
├── docs
└── records_backup

Installing a DNS hook for DNS-01 challenges

The wiki for dehydrated has a list of community-created DNS hooks for DNS providers.

For this example, we will use the NameCheap DNS hook.

This library contains two scripts, namecheap_dns_api_hook.sh and reload_services.sh. Place them in your dehydrated directory.

Configuring dehydrated

Create empty files called config and domains.txt in the dehydrated folder.

Your dehydrated directory should now have the following content:

dehydrated/
├── CHANGELOG
├── LICENSE
├── README.md
├── accounts
├── certs
├── config
├── dehydrated
├── docs
├── domains.txt
├── namecheap_dns_api_hook.sh
├── records_backup
└── reload_services.sh

config

Place the following content in config:

########################################################
# This is the main config file for the Dehydrated      #
# Namecheap DNS 01 Hook script                         #
#                                                      #
# Default values of this config are in comments        #
########################################################

# Namecheap API Credentials
# These are not default dehydrated config params!
apiusr=# Replace with your API username from NameCheap
apikey=# Replace with your API key from NameCheap
BASEDIR="/etc/dehydrated"
CHALLENGETYPE="dns-01"

# Configure notifications

## Replace these parameters with the appropriate email addresses and domain names
#SENDER="sender@example.com"
#RECIPIENT="recipient@example.com"
#DOMAIN="domain.name.com"

DEBUG=no
OPENSSL="openssl"
# Extra options passed to the curl binary (default: <unset>)
#CURL_OPTS=

# Records backup location
RECORDS_BACKUP=${BASEDIR}/records_backup
DOMAINS_TXT="${BASEDIR}/domains.txt"
CERTDIR="${BASEDIR}/certs"
HOOK="${BASEDIR}/namecheap_dns_api_hook.sh"

# Configure certificate and key locations for deployment
DEPLOYED_CERTDIR="${BASEDIR}/certs"
DEPLOYED_KEYDIR="${BASEDIR}/certs"

# Email Sending Method Options
#  + SENDMAIL (default)
#  + SMTP
#MAIL_METHOD=SENDMAIL

# SMTP options
#SMTP_DOMAIN=localhost
#SMTP_SERVER=
#SMTP_PORT=25

domains.txt

dehydrated uses the domains.txt file as the configuration for which certificates should be requested.

In this example, if your publicly accessible domain is company.com, and you want to have your private intranet have the subdomain dev.company.com, place the following in the domains.txt file.

dev.company.com
company.com www.company.com dev.company.com

The first line will create a certificate for the dev.company.com domain, and the last line will create a multi-domain certificate.

These two lines produce certificates stored under different directory names (dev.company.com vs company.com) within the certs directory.

reload-services.sh

Make the following changes to reload-services.sh:

#!/usr/bin/env bash

# Script to reloading services that use SSL certificates to ensure
# all services are using the latest versions of the certificates.

DOMAIN="${1}" KEYFILE="${2}" CERTFILE="${3}" FULLCHAINFILE="${4}" CHAINFILE="${5}" TIMESTAMP="${6}"

# Nginx
echo " + Reloading Nginx configuration"
systemctl reload nginx.service

Running dehydrated

Run dehydrated with the command /etc/dehydrated/dehydrated -c.

This runs the dehydrated client using the config file’s parameters. It should get the challenge from Let’s Encrypt, and provision a DNS TXT record with the response via the DNS hook and the API. When validated the certificates will be placed in the certs directory, and from there they can be distributed to the appropriate applications. The certificates will be valid for 90 days.

Updating nginx

The following snippet from a server block in the nginx configuration shows how nginx will be configured to use the newly generated SSL certificates and keys.

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name dev.company.com;
    ssl_certificate /etc/dehydrated/certs/company.com/fullchain.pem;
    ssl_certificate_key /etc/dehydrated/certs/company.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';

nginx can then be reloaded with sudo systemctl reload nginx. Once reloaded, services hosted by the server block should use the newly generated SSL certs, and are now hosted securely. These certificates expire in 90 days, so the process of certificate renewal should be automated.

Automating certificate renewal

Place the following entry into the root user’s crontab:

0 0   *  *  *     (/etc/dehydrated/dehydrated -c) 2>&1 | logger -t dehydrated

This should run the dehydrated client every night at 12AM, and the log output can be captured using cat /var/log/syslog | grep dehydrated.

This CRON jon checks the expiry date of the current certificate, and if the lifetime on these certificates is less than 30 days, it will renew them with Let’s Encrypt. The hook should also reload the nginx config to use the new certs after they’ve been obtained.

Citations

• Bryan, B. (2016, September 12). Intranet SSL Certificates Using Let’s Encrypt | DNS-01. https://b3n.org/intranet-ssl-certificates-using-lets-encrypt-dns-01/

References

• The Let’s Encrypt Website.
• Let’s Encrypt’s documentation on Challenge Types.
• RFC8555 documentation on DNS-01 challenges.
• RFC8555 documentation on HTTP-01 challenges.
• Let’s Encrypt’s documentation on recommended ACME clients.
• The ACME protocol on Wikipedia.
• The dehydrated ACME client.
• The repo for the NameCheap DNS hook.

PostgreSQL (and other RDBMS) have entity types such as functions, views, materialized views, triggers, and policies. These entity types do a lot of useful work, and a well designed database will use them heavily.

These entity types are not typically considered part of the SQLAlchemy ORM, and by extension, the data model itself. By default, Alembic has no functionality to detect the creation of new entity types when it autogenerates migration files. The default Alembic ORM also has no functionality to define these entity types with classes.

However, these entity types are part of a database’s schema. Creation of/changes to them should be accounted for within the ORM and as part of migration version control, in much the same way as changes in a table are recorded in an ORM as part of migration version control.

In this blog post, we will pick up from the progress in the previous entry in this series and go over how we can use an extension to Alembic called Alembic Utils to add support for autogenerating migrations for some PostgreSQL entity types.

What we’ve already covered

In part 1:

• We used SQLAlchemy to create the first iteration of a rudimentary data model, with three tables.
• We configured Alembic to connect to a fresh development database and migrated our rudimentary data model.

In part 2:

• We created a many-to-many association between orders and products, and a many-to-many association between products and tags.
• We generated migration scripts using Alembic.
• We committed the migration scripts to version control.

What we’ll cover

In this blog post, we will cover how you can install Alembic Utils and begin incorporating entity types into the commerce data model.

We will do the following:

• Update the virtual environment with Alembic Utils.
• Configure Alembic to use the newly installed extension.
• Create a function and a view in the ORM.
• Register these entities with Alembic Utils.
• Autogenerate a migration script with Alembic that we will review, and then apply the revisions.

Installing Alembic Utils

In part 1 of this series, we created a micromamba virtual environment called data_model with SQLAlchemy and Alembic installed.

We will activate this virtual envrionment and then install Alembic Utils.

First, activate the virtual environment.

micromamba activate data_model

Then, install the Alembic Utils extension within the environment. The documentation recommends using pip.

pip install alembic_utils

Export the new environment details using the following command:

micromamba env export > environment.yaml

Adding a view to the database

Creating a View in the ORM

To explain what a view is, we can start by asking a straightforward high level question. What are the names of the products associated with a particular order? The stakeholders who have access to data might need to be able to access the products associated with a single specific order, or a group of specific orders.

In our database, there is a table called order_products that maps the associations between orders and products using the primary keys from that table. However, just looking at this table alone won’t give us the contextual information we need. We will need to write an SQL query to join the order_products table with the products table in order to retrieve the name and description of a particular order’s products.

This is a query that can be easily generalizable. A query can be written to join all the products referenced in order_products with the relevant product data from products. This query can then be filtered to return the associations with a single order or group of orders.

This query can be abstracted into a view. Views are named queries that allow for data to be queried from it like a regular table. The view is not physically stored, however. The query in the view is run each time the view itself is referenced in a query.

To return the product data associated with each product_id reference in order_products, we could write a query like this.

select
  op.order_id,
  op.product_id,
  op.product_count,
  p.product_name,
  p.description,
  p.price
from
  order_products op
  join products p on op.product_id = p.product_id

We can add an adjustment to this query to return the total price of each product associated with an order when accounting for the order count.

select
  op.order_id,
  op.product_id,
  op.product_count,
  p.product_name,
  p.description,
  p.price as unit_price,
  op.product_count * p.price as order_price
from
  order_products op
  join products p on op.product_id = p.product_id

We multiply the product count with the unit price to get the total cost of the product within an order. With this query, we can now filter to a specific set of order_id values to return all the products associated with those orders and their total cost!

Creating a view from a query like this, that has the potential to be referenced multiple times, is great database design. We can now add this view to our ORM.

First, we will create a script called db_views.py in the root of the project. This script will be where we store the views that we create for our ORM.

db_views.py:

from alembic_utils.pg_view import PGView

order_line_items = PGView(
    schema="public",
    signature="order_line_items",
    definition="""
select op.order_id ,
	op.product_id,
	op.product_count,
	p.product_name,
	p.description,
	p.price as unit_price,
	op.product_count * p.price as order_price
	from order_products op
join products p
on op.product_id = p.product_id
""",
)

With this code, we first create a PGView class object called order_line_items. This defines a PostgreSQL view also called order_line_items that references the query that we wrote to return the line items (products) associated with orders.

Registering the newly defined entity

We will update alembic/env.py to then register our newly created entity with alembic_utils.

alembic/env.py:

from logging.config import fileConfig

from sqlalchemy import engine_from_config
from sqlalchemy import pool
from alembic_utils.replaceable_entity import register_entities
from alembic import context
from models import Base
from db_views import order_line_items

register_entities([order_line_items])

# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config

# Interpret the config file for Python logging.
# This line sets up loggers basically.
if config.config_file_name is not None:
    fileConfig(config.config_file_name)

# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata
target_metadata = Base.metadata

# other values from the config, defined by the needs of env.py,
# can be acquired:
# my_important_option = config.get_main_option("my_important_option")
# ... etc.


def run_migrations_offline() -> None:
    """Run migrations in 'offline' mode.

    This configures the context with just a URL
    and not an Engine, though an Engine is acceptable
    here as well.  By skipping the Engine creation
    we don't even need a DBAPI to be available.

    Calls to context.execute() here emit the given string to the
    script output.

    """
    url = config.get_main_option("sqlalchemy.url")
    context.configure(
        url=url,
        target_metadata=target_metadata,
        literal_binds=True,
        dialect_opts={"paramstyle": "named"},
    )

    with context.begin_transaction():
        context.run_migrations()


def run_migrations_online() -> None:
    """Run migrations in 'online' mode.

    In this scenario we need to create an Engine
    and associate a connection with the context.

    """
    connectable = engine_from_config(
        config.get_section(config.config_ini_section, {}),
        prefix="sqlalchemy.",
        poolclass=pool.NullPool,
    )

    with connectable.connect() as connection:
        context.configure(connection=connection, target_metadata=target_metadata)

        with context.begin_transaction():
            context.run_migrations()


if context.is_offline_mode():
    run_migrations_offline()
else:
    run_migrations_online()

The additions to the code import the newly defined view and register it. This view should now be detected by alembic revision --autogenerate.

Autogenerating a migration

We now autogenerate a revision with the command:

alembic revision --autogenerate -m "Created a view called order_line_items that returns order line items."

This should create a new migration script located at /path_to_your_project/alembic/versions.

/path_to_your_project/alembic/versions/e2ba9d70fc4d_created_a_view_called_order_line_items_.py:

"""Created a view called order_line_items that returns order line items.

Revision ID: e2ba9d70fc4d
Revises: 6e58a26e5c70
Create Date: 2026-02-07 19:26:28.816521

"""
from typing import Sequence, Union

from alembic import op
import sqlalchemy as sa
from alembic_utils.pg_view import PGView
from sqlalchemy import text as sql_text

# revision identifiers, used by Alembic.
revision: str = 'e2ba9d70fc4d'
down_revision: Union[str, None] = '6e58a26e5c70'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None


def upgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    public_order_line_items = PGView(
        schema="public",
        signature="order_line_items",
        definition='select op.order_id , \n\top.product_id, \n\top.product_count,\n\tp.product_name, \n\tp.description, \n\tp.price as unit_price, \n\top.product_count * p.price as order_price\n\tfrom order_products op \njoin products p \non op.product_id = p.product_id'
    )
    op.create_entity(public_order_line_items)

    # ### end Alembic commands ###


def downgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    public_order_line_items = PGView(
        schema="public",
        signature="order_line_items",
        definition='select op.order_id , \n\top.product_id, \n\top.product_count,\n\tp.product_name, \n\tp.description, \n\tp.price as unit_price, \n\top.product_count * p.price as order_price\n\tfrom order_products op \njoin products p \non op.product_id = p.product_id'
    )
    op.drop_entity(public_order_line_items)

    # ### end Alembic commands ###

This migration script now creates a PostgreSQL view with the name order_line_items.

Applying the migration

We now apply the migration with the command:

alembic upgrade head

Connecting to the database shows us that the view has been migrated to the database.

Expanding the data model’s scope

We would like to now expand the data model to track when the attributes of a particular product have been edited. To do this, we will do the following:

• Add a last_edited timestamp column to the products table.
• Create a function that will update the last_edited column of a table.
• Create a trigger that will call the aforementioned function whenever a row of products is edited.

Adding a timestamp column to the products table

Amend the data model in models.py to have a last_edited timestamp column.

from sqlalchemy import (
    Column,
    Integer,
    String,
    Float,
    ForeignKey,
    Identity,
    Text,
    DateTime,
    Table,
)
from sqlalchemy.orm import declarative_base, relationship
from sqlalchemy.sql import func

Base = declarative_base()


class Customer(Base):
    __tablename__ = "customers"

    customer_id = Column(Integer, Identity(always=True), primary_key=True)
    first_name = Column(String, nullable=False)
    last_name = Column(String, nullable=False)
    email_address = Column(String, nullable=False)

    orders = relationship(
        "Order",
        back_populates="customer",
        cascade="all, delete-orphan",
    )


class OrderProduct(Base):
    __tablename__ = "order_products"
    order_id = Column(
        Integer,
        ForeignKey("orders.order_id", ondelete="CASCADE"),
        primary_key=True,
    )

    product_id = Column(
        Integer,
        ForeignKey("products.product_id", ondelete="CASCADE"),
        primary_key=True,
    )

    product_count = Column(Integer, server_default="1", nullable=False)
    order = relationship("Order", back_populates="products")
    product = relationship("Product", back_populates="orders")


class Order(Base):
    __tablename__ = "orders"

    order_id = Column(Integer, Identity(always=True), primary_key=True)
    customer_id = Column(
        Integer, ForeignKey("customers.customer_id", ondelete="CASCADE"), nullable=False
    )
    order_date = Column(
        DateTime(timezone=True),
        server_default=func.now(),
        nullable=False,
    )
    customer = relationship(
        "Customer",
        back_populates="orders",
    )
    products = relationship(
        "OrderProduct",
        back_populates="order",
        cascade="all, delete-orphan",
    )


product_tags = Table(
    "product_tags",
    Base.metadata,
    Column(
        "product_id",
        ForeignKey("products.product_id", ondelete="CASCADE"),
        primary_key=True,
    ),
    Column("tag_id", ForeignKey("tags.tag_id", ondelete="CASCADE"), primary_key=True),
)


class Product(Base):
    __tablename__ = "products"

    product_id = Column(Integer, Identity(always=True), primary_key=True)
    product_name = Column(String, nullable=False)
    description = Column(Text)
    price = Column(Float, nullable=False)
    last_edited = Column(
        DateTime(timezone=True), server_default=func.now(), onupdate=func.now()
    )
    orders = relationship(
        "OrderProduct",
        back_populates="product",
        cascade="all, delete-orphan",
    )
    tags = relationship("Tag", secondary=product_tags, back_populates="products")


class Tag(Base):
    __tablename__ = "tags"

    tag_id = Column(Integer, Identity(always=True), primary_key=True)
    name = Column(String, nullable=False, unique=True)
    products = relationship("Product", secondary=product_tags, back_populates="tags")

Generate a revision with alembic --autogenerate.

"""Added a column to products to track timestamps for row-level edits.

Revision ID: eebe110ef9d2
Revises: e2ba9d70fc4d
Create Date: 2026-02-08 16:29:55.536261

"""
from typing import Sequence, Union

from alembic import op
import sqlalchemy as sa


# revision identifiers, used by Alembic.
revision: str = 'eebe110ef9d2'
down_revision: Union[str, None] = 'e2ba9d70fc4d'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None


def upgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    op.add_column('products', sa.Column('last_edited', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=True))
    # ### end Alembic commands ###


def downgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    op.drop_column('products', 'last_edited')
    # ### end Alembic commands ###

We will then apply this migration with the alembic upgrade head command.

Creating a function to update the last_edited column of products

Create a script called db_functions.py in the project root.

db_functions.py:

from alembic_utils.pg_function import PGFunction

## Creating a function that updates the `last_edited` column of a table
timestamp_update_on_edit = PGFunction(
    schema="public",
    signature="timestamp_update_on_edit()",
    definition="""
RETURNS TRIGGER AS $$ BEGIN NEW.last_edited = now();
RETURN NEW;
END;
$$ language PLPGSQL
""",
)

With this code, we create a PGFunction class object that defines a PostgreSQL function called timestamp_update_on_edit(). When this function is called, the last_edited column is updated to have the time and date of the point in time when it was called.

Creating a trigger to call the timestamp function on row edits

Create a script called db_triggers.py in the project root.

db_triggers.py:

from alembic_utils.pg_trigger import PGTrigger

## Creating a trigger to apply `timestamp_update_on_edit()` to `products` on a row being updated
timestamp_update_apply_products_trigger = PGTrigger(
    schema="public",
    signature="timestamp_update_apply_products_trigger",
    on_entity="public.products",
    is_constraint=False,
    definition="""
BEFORE
UPDATE
  on products FOR EACH ROW EXECUTE FUNCTION timestamp_update_on_edit()
""",
)

This code defines a PGTrigger class object with the signature timestamp_update_apply_products_trigger. When a row is updated on products, this trigger is activated and calls the timestamp_update_on_edit() function, which will update the last_edited column.

Registering the newly defined entities

We will update alembic/env.py to then register our newly created entities with alembic_utils.

alembic/env.py:

from logging.config import fileConfig

from sqlalchemy import engine_from_config
from sqlalchemy import pool
from alembic_utils.replaceable_entity import register_entities
from alembic import context
from models import Base
from db_views import order_line_items
from db_functions import timestamp_update_on_edit
from db_triggers import timestamp_update_apply_products_trigger

register_entities(
    [
        order_line_items,
        timestamp_update_on_edit,
        timestamp_update_apply_products_trigger,
    ]
)

# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config

# Interpret the config file for Python logging.
# This line sets up loggers basically.
if config.config_file_name is not None:
    fileConfig(config.config_file_name)

# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata
target_metadata = Base.metadata

# other values from the config, defined by the needs of env.py,
# can be acquired:
# my_important_option = config.get_main_option("my_important_option")
# ... etc.


def run_migrations_offline() -> None:
    """Run migrations in 'offline' mode.

    This configures the context with just a URL
    and not an Engine, though an Engine is acceptable
    here as well.  By skipping the Engine creation
    we don't even need a DBAPI to be available.

    Calls to context.execute() here emit the given string to the
    script output.

    """
    url = config.get_main_option("sqlalchemy.url")
    context.configure(
        url=url,
        target_metadata=target_metadata,
        literal_binds=True,
        dialect_opts={"paramstyle": "named"},
    )

    with context.begin_transaction():
        context.run_migrations()


def run_migrations_online() -> None:
    """Run migrations in 'online' mode.

    In this scenario we need to create an Engine
    and associate a connection with the context.

    """
    connectable = engine_from_config(
        config.get_section(config.config_ini_section, {}),
        prefix="sqlalchemy.",
        poolclass=pool.NullPool,
    )

    with connectable.connect() as connection:
        context.configure(connection=connection, target_metadata=target_metadata)

        with context.begin_transaction():
            context.run_migrations()


if context.is_offline_mode():
    run_migrations_offline()
else:
    run_migrations_online()

With this update, our newly created function and trigger have been registered. They should now be detected when using autogenerate.

Autogenerating and applying a revision to migrate the newly added entities

Create a revision script with:

alembic revision --autogenerate -m "Created a function to update the last_edited timestamp column, and created a trigger to call this function for the products table."

This should generate a revision file with the following content:

"""Created a function to update the last_edited timestamp column, and created a trigger to call this function for the products table.

Revision ID: 172f9e67a364
Revises: eebe110ef9d2
Create Date: 2026-02-08 16:41:47.178179

"""
from typing import Sequence, Union

from alembic import op
import sqlalchemy as sa
from alembic_utils.pg_function import PGFunction
from sqlalchemy import text as sql_text
from alembic_utils.pg_trigger import PGTrigger
from sqlalchemy import text as sql_text

# revision identifiers, used by Alembic.
revision: str = '172f9e67a364'
down_revision: Union[str, None] = 'eebe110ef9d2'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None


def upgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    public_timestamp_update_on_edit = PGFunction(
        schema="public",
        signature="timestamp_update_on_edit()",
        definition='RETURNS TRIGGER AS $$ BEGIN NEW.last_edited = now();\nRETURN NEW;\nEND;\n$$ language PLPGSQL'
    )
    op.create_entity(public_timestamp_update_on_edit)

    public_products_timestamp_update_apply_products_trigger = PGTrigger(
        schema="public",
        signature="timestamp_update_apply_products_trigger",
        on_entity="public.products",
        is_constraint=False,
        definition='BEFORE \nUPDATE \n  on products FOR EACH ROW EXECUTE FUNCTION timestamp_update_on_edit()'
    )
    op.create_entity(public_products_timestamp_update_apply_products_trigger)

    # ### end Alembic commands ###


def downgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    public_products_timestamp_update_apply_products_trigger = PGTrigger(
        schema="public",
        signature="timestamp_update_apply_products_trigger",
        on_entity="public.products",
        is_constraint=False,
        definition='BEFORE \nUPDATE \n  on products FOR EACH ROW EXECUTE FUNCTION timestamp_update_on_edit()'
    )
    op.drop_entity(public_products_timestamp_update_apply_products_trigger)

    public_timestamp_update_on_edit = PGFunction(
        schema="public",
        signature="timestamp_update_on_edit()",
        definition='RETURNS TRIGGER AS $$ BEGIN NEW.last_edited = now();\nRETURN NEW;\nEND;\n$$ language PLPGSQL'
    )
    op.drop_entity(public_timestamp_update_on_edit)

    # ### end Alembic commands ###

The migration script should create the function and the trigger that we defined in the ORM within the database. After reviewing it, we apply the migration with the command alembic upgrade head.

Once these changes have been migrated, run the following SQL code in the database to get the triggers associated with the products table.

SELECT
    tgname AS trigger_name
FROM
    pg_trigger
WHERE
    tgrelid = 'public.products'::regclass
ORDER BY
    trigger_name;

This should give you the output:

RI_ConstraintTrigger_a_16447
RI_ConstraintTrigger_a_16448
RI_ConstraintTrigger_a_16471
RI_ConstraintTrigger_a_16472
timestamp_update_apply_products_trigger

Indicating the trigger has been successfully migrated. Run the following code to get the functions associated with the public schema:

SELECT
    routine_name
FROM
    information_schema.routines
WHERE
    routine_type = 'FUNCTION'
AND
    routine_schema = 'public';

This should give you the output:

routine_name
timestamp_update_on_edit

showing that our new function has been migrated to the database as well!

We’ve successfully added a view, a function and a trigger to our database via migrations applied from our SQLAlchemy ORM.

Browsing through alembic/versions should give us an idea of the growth of our data model and how it evolves to match the change in scope of our e-commerce workflow.

alembic/versions/
├── 172f9e67a364_created_a_function_to_update_the_last_.py
├── 6e58a26e5c70_created_a_table_for_tags_and_created_a_.py
├── 960969e75871_created_orderproduct_as_an_association_.py
├── a0cf7f5d705d_initial_commit_created_customers_orders_.py
├── e2ba9d70fc4d_created_a_view_called_order_line_items_.py
└── eebe110ef9d2_added_a_column_to_products_to_track_.py

Next Steps

In subsequent blog posts, we will cover:

• Applying our migrations to a production environment
• Using the ORM as a module when coding a FastAPI server

References

• The repo for Alembic Utils.
• The documentation for Alembic Utils.

As the scope of a data model (and by extension, its downstream APIs) changes, it will need to be updated and expanded to account for this new scope.

When changes are made to a data model, especially to a model that is split across development and production environments, schema drift becomes a constant problem that looms in the background.

Typical coding workflows are managed by version control as an industry standard. Database schemas, however, are typically not submitted to a version control system. As different team members collaborate on a data model split across different environments, the possibility of schema drift gradually grows in an environment where the data model is not tracked in a central repository.

This is where Alembic shines! It is the de-facto migration tool for SQLAlchemy, and is the ideal tool to version control a data model’s development.

In this blog post, we will pick up from the progress in the previous entry in this series and continue to develop our data model. We will build more tables and relationships into our data model, and commit all of these changes to version control using Alembic.

What we’ve already covered

In our previous blog post:

• We used SQLAlchemy to create the first iteration of a rudimentary data model, with three tables.
• We configured Alembic to connect to a fresh development database and migrated our rudimentary data model.

Please refer to the first post in this series for a detailed explanation of SQLAlchemy and Alembic, and how it is relevant in our workflows here.

What we’ll cover in this post

• Creating a many-to-many association between orders and products.
• Creating a many-to-many association between products and tags.
• Committing all of our changes to version control with Alembic.

Getting started

At this point, we have a rudimentary data model with three tables, and only two relationships.

There’s orders,customers and products, with a one-to-many relationship between customers and orders, and a many-to-one relationship between orders and customers.

The classes for these tables in our ORM were defined as follows:

class Customer(Base):
    __tablename__ = "customers"

    customer_id = Column(
        Integer, Identity(always=True), primary_key=True
    )
    first_name = Column(String, nullable=False)
    last_name = Column(String, nullable=False)
    email_address = Column(String, nullable=False)

    orders = relationship(
        "Order",
        back_populates="customer",
        cascade="all, delete-orphan",
    )

class Order(Base):
    __tablename__ = "orders"

    order_id = Column(
        Integer, Identity(always=True), primary_key=True
    )
    customer_id = Column(
        Integer,
        ForeignKey("customers.customer_id", ondelete="CASCADE"),
        nullable=False
    )
    order_date = Column(
        DateTime(timezone=True),
        server_default=func.now(),
        nullable=False,
    )

    customer = relationship(
        "Customer",
        back_populates="orders"
    )

class Product(Base):
    __tablename__ = "products"

    product_id = Column(
        Integer, Identity(always=True), primary_key=True
    )
    product_name = Column(
        String, nullable=False
    )
    description = Column(Text)
    price = Column(Float, nullable=False)

There are a lot of changes we need to make to make this data model usable. To start with:

• A connection needs to be built between orders and products. A single order can have multiple products, and a single product can be associated with multiple orders.
• Products need to be associated with tags to enable better categorization within the product catalog. Each product can be associated with multiple tags, and a single tag can be associated with multiple products.

Let’s begin implementing this change in scope within the data model. With each step, we will also commit our work to version control using Alembic.

Many-to-many relationship between orders and products

We want to build a many-to-many relationship between orders and products. We also want to expand this relationship to track the count of each product in an order.

To accomplish that, we will create an association table.

An association table maps two (or more) tables together by using their primary keys as foreign keys. Each foreign key forms a many-to-one relationship between the association table and the individual data table. An association table creates a two-way link between the tables that represent each entity.

The best way to demonstrate this is by example! Let’s begin by creating an association table to group together the products associated with each order.

To generate an association table, the following changes need to be made to models.py. Additions have been highlighted.

models.py:

from sqlalchemy import (
    Column,
    Integer,
    String,
    Float,
    ForeignKey,
    Identity,
    Text,
    DateTime,
)
from sqlalchemy.orm import declarative_base, relationship
from sqlalchemy.sql import func

Base = declarative_base()


class Customer(Base):
    __tablename__ = "customers"

    customer_id = Column(Integer, Identity(always=True), primary_key=True)
    first_name = Column(String, nullable=False)
    last_name = Column(String, nullable=False)
    email_address = Column(String, nullable=False)

    orders = relationship(
        "Order",
        back_populates="customer",
        cascade="all, delete-orphan",
    )


class OrderProduct(Base):
    __tablename__ = "order_products"
    order_id = Column(
        Integer,
        ForeignKey("orders.order_id", ondelete="CASCADE"),
        primary_key=True,
    )

    product_id = Column(
        Integer,
        ForeignKey("products.product_id", ondelete="CASCADE"),
        primary_key=True,
    )

    product_count = Column(Integer, server_default="1", nullable=False)
    order = relationship("Order", back_populates="products")
    product = relationship("Product", back_populates="orders")


class Order(Base):
    __tablename__ = "orders"

    order_id = Column(Integer, Identity(always=True), primary_key=True)
    customer_id = Column(
        Integer, ForeignKey("customers.customer_id", ondelete="CASCADE"), nullable=False
    )
    order_date = Column(
        DateTime(timezone=True),
        server_default=func.now(),
        nullable=False,
    )
    customer = relationship(
        "Customer",
        back_populates="orders",
    )
    products = relationship(
        "OrderProduct",
        back_populates="order",
        cascade="all, delete-orphan",
    )


class Product(Base):
    __tablename__ = "products"

    product_id = Column(Integer, Identity(always=True), primary_key=True)
    product_name = Column(String, nullable=False)
    description = Column(Text)
    price = Column(Float, nullable=False)
    orders = relationship(
        "OrderProduct",
        back_populates="product",
        cascade="all, delete-orphan",
    )

Creating the association table

We create a class to define the association table, since the association table contains an additional column beyond just the associations with Order and Product. This table is OrderProduct.

Theorder_id and product_id columns in this table are designated as foreign keys.

We then create associations between OrderProduct and the entity tables using the relationship parameter. OrderProducts has relationships built to both Order and Product.

Explaining the relationship between Order and Product

There is no direct relationship between Order and Product. This relationship is mediated by the OrderProduct class, which is an association table that decomposes the relationship between Order and Product.

A single instance of an Order can belong to multiple rows of OrderProduct.

A single instance of a Product can belong to multiple rows of OrderProduct.

However, every instance of OrderProduct can only have:

• A single Order

• A single Product.

In essence, the many-to-many relationship between Order and Product is abstracted into being indirect via OrderProduct. Each row in OrderProduct represents a single product within a single order, along with additional metadata about that association, such as the product count for that order.

The following code snippets establish this indirect relationship between Order and Product.

OrderProduct:

    order = relationship("Order", back_populates="products")
    product = relationship("Product", back_populates="orders")

Order:

products = relationship(
        "OrderProduct",
        back_populates="order",
        cascade="all, delete-orphan",
    )

Product:

orders = relationship(
        "OrderProduct",
        back_populates="product",
        cascade="all, delete-orphan",
    )

For the OrderProduct Class, this code:

• Creates a relationship called order, that points towards the Order class. This now establishes a many-to-one relationship between OrderProduct and Order. Each instance of OrderProduct references exactly one Order.

• Creates a relationship called product, that points towards the Product class. This now establishes a many-to-one relationship between OrderProduct and Product. Each instance of OrderProduct references exactly one Product.

For the Order Class, this code:

• Creates a relationship called products, that points towards the OrderProduct class. This now establishes a one-to-many relationship between Order and OrderProducts. A single Order can be referenced by multiple instances of OrderProduct.

For the Product Class, this code:

• Creates a relationship called orders, that points towards the OrderProduct class. This now establishes a one-to-many relationship between Product and OrderProducts. A single Product can be referenced by multiple instances of OrderProduct.

• Similar to the products relationship defined in Order, this is a conceptual representation, which is why it is called orders.

By wiring these connections, we abstract away the many-to-many relationship between Orders and Products.

Auto-generating migrations

We now have a end state of our data model defined in the ORM. To generate a migration, we need to allow Alembic to compare the current state of the database against the ORM, and have it automatically generate a migration script that we must review before applying.

We configured and set up Alembic in the previous blog post, so this time, we just need to run the following command:

alembic revision --autogenerate -m "Created OrderProduct as an association table to track Order-Product association."

This should autogenerate a migration script, located at /path_to_your_project/alembic/versions/960969e75871_created_orderproduct_as_an_association_.py. The versions directory will have the following content:

alembic/versions/
└── a0cf7f5d705d_initial_commit_created_customers_orders_.py
└── 960969e75871_created_orderproduct_as_an_association_.py

The versions directory now contains two migration scripts! One is the migration script for our initial commit in the previous blog post, and one is the newly generated script for creating the OrderProducts table.

The newly generated migration should have the following content:

"""Created OrderProduct as an association table to track Order-Product association.

Revision ID: 960969e75871
Revises: a0cf7f5d705d
Create Date: 2026-01-26 17:30:18.314422

"""
from typing import Sequence, Union

from alembic import op
import sqlalchemy as sa


# revision identifiers, used by Alembic.
revision: str = '960969e75871'
down_revision: Union[str, None] = 'a0cf7f5d705d'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None


def upgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    op.create_table('order_products',
    sa.Column('order_id', sa.Integer(), nullable=False),
    sa.Column('product_id', sa.Integer(), nullable=False),
    sa.Column('product_count', sa.Integer(), server_default='1', nullable=False),
    sa.ForeignKeyConstraint(['order_id'], ['orders.order_id'], ondelete='CASCADE'),
    sa.ForeignKeyConstraint(['product_id'], ['products.product_id'], ondelete='CASCADE'),
    sa.PrimaryKeyConstraint('order_id', 'product_id')
    )
    # ### end Alembic commands ###


def downgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    op.drop_table('order_products')
    # ### end Alembic commands ###

Reviewing the auto-generated migration

Remember that Alembic’s documentation states,

It is always necessary to manually review and correct the candidate migrations that autogenerate produces.

Reviewing our migration script, it looks like:

• Alembic has created a table called order_products.
• order_id and product_id are both foreign keys. They are the primary keys from the tables orders and products respectively.
• order_id and product_id are also primary keys for the order_products table.
• order_products has a column called product_count, with a default value of 1.

This migration script looks good! We can go ahead and apply it to the database.

products = relationship(
        "OrderProduct",
        back_populates="order",
        cascade="all, delete-orphan",
    )

Applying the migration

We can then apply the migration script with the command:

alembic upgrade head

This should give us the following output:

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade a0cf7f5d705d -> 960969e75871, Created OrderProduct as an association table to track Order-Product association.

Connecting to the database shows the following ERD:

We’ve successfully created the association table, and we have built an indirect many-to-many association between Orders and Products! When we examine the contents of the alembic_version table, it now contains a single row with the value 960969e75871, which is the revision it has applied. It applied this one specifically because we had alembic upgrade to the head version, which is the latest version.

Committing to GitHub

At this point, we’ve made changes to models.py and we have a new migration script called 960969e75871_created_orderproduct_as_an_association_.py. Both these changes need to be pushed GitHub. I recommend committing and pushing these changes as a single commit with the same commit message that we used for auto-generating a migration script.

Remember to always commit migration scripts together with model changes.

Further iterating

Let’s expand this data model more. A great addition to this data model would be the ability to create and assign tags to products for greater searchability.

To start with, these tags will be really simple. A tag will be a text-based descriptor that can be applied to any product. For example, a laptop as a product might have a tag called ‘computer’ assigned to it.

To reflect this, we will create a table containing tags, and build a many-to-many relationship between products and tags. Each product can have multiple tags, and each tag can be assigned to multiple products. For this, like before, we will build an association table.

We will do the following:

• Create a class for Tags in the ORM
• Create an association table called product_tags to associate products and tags
• Update the relationships for Products to reflect the new relationship to the Tags class.

Creating a Tags class

models.py (Changes to code have been highlighted):

from sqlalchemy import (
    Column,
    Integer,
    String,
    Float,
    ForeignKey,
    Identity,
    Text,
    DateTime,
    Table,
)
from sqlalchemy.orm import declarative_base, relationship
from sqlalchemy.sql import func

Base = declarative_base()


class Customer(Base):
    __tablename__ = "customers"

    customer_id = Column(Integer, Identity(always=True), primary_key=True)
    first_name = Column(String, nullable=False)
    last_name = Column(String, nullable=False)
    email_address = Column(String, nullable=False)

    orders = relationship(
        "Order",
        back_populates="customer",
        cascade="all, delete-orphan",
    )


class OrderProduct(Base):
    __tablename__ = "order_products"
    order_id = Column(
        Integer,
        ForeignKey("orders.order_id", ondelete="CASCADE"),
        primary_key=True,
    )

    product_id = Column(
        Integer,
        ForeignKey("products.product_id", ondelete="CASCADE"),
        primary_key=True,
    )

    product_count = Column(Integer, server_default="1", nullable=False)
    order = relationship("Order", back_populates="products")
    product = relationship("Product", back_populates="orders")


class Order(Base):
    __tablename__ = "orders"

    order_id = Column(Integer, Identity(always=True), primary_key=True)
    customer_id = Column(
        Integer, ForeignKey("customers.customer_id", ondelete="CASCADE"), nullable=False
    )
    order_date = Column(
        DateTime(timezone=True),
        server_default=func.now(),
        nullable=False,
    )
    customer = relationship(
        "Customer",
        back_populates="orders",
    )
    products = relationship(
        "OrderProduct",
        back_populates="order",
        cascade="all, delete-orphan",
    )


product_tags = Table(
    "product_tags",
    Base.metadata,
    Column(
        "product_id",
        ForeignKey("products.product_id", ondelete="CASCADE"),
        primary_key=True,
    ),
    Column("tag_id", ForeignKey("tags.tag_id", ondelete="CASCADE"), primary_key=True),
)


class Product(Base):
    __tablename__ = "products"

    product_id = Column(Integer, Identity(always=True), primary_key=True)
    product_name = Column(String, nullable=False)
    description = Column(Text)
    price = Column(Float, nullable=False)
    orders = relationship(
        "OrderProduct",
        back_populates="product",
        cascade="all, delete-orphan",
    )
    tags = relationship("Tag", secondary=product_tags, back_populates="products")


class Tag(Base):
    __tablename__ = "tags"

    tag_id = Column(Integer, Identity(always=True), primary_key=True)
    name = Column(String, nullable=False, unique=True)
    products = relationship("Product", secondary=product_tags, back_populates="tags")

Generating a migration

We create an auto-generated migration script with the following command:

alembic revision --autogenerate -m "Created a table for tags, and created a table called product_tags as an association table between products and tags."

Reviewing the migration script

The contents of the migration script are:

"""Created a table for tags, and created a table called product_tags as an association table between products and tags.

Revision ID: 6e58a26e5c70
Revises: 960969e75871
Create Date: 2026-01-26 18:32:43.007353

"""
from typing import Sequence, Union

from alembic import op
import sqlalchemy as sa


# revision identifiers, used by Alembic.
revision: str = '6e58a26e5c70'
down_revision: Union[str, None] = '960969e75871'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None


def upgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    op.create_table('tags',
    sa.Column('tag_id', sa.Integer(), sa.Identity(always=True), nullable=False),
    sa.Column('name', sa.String(), nullable=False),
    sa.PrimaryKeyConstraint('tag_id'),
    sa.UniqueConstraint('name')
    )
    op.create_table('product_tags',
    sa.Column('product_id', sa.Integer(), nullable=False),
    sa.Column('tag_id', sa.Integer(), nullable=False),
    sa.ForeignKeyConstraint(['product_id'], ['products.product_id'], ondelete='CASCADE'),
    sa.ForeignKeyConstraint(['tag_id'], ['tags.tag_id'], ondelete='CASCADE'),
    sa.PrimaryKeyConstraint('product_id', 'tag_id')
    )
    # ### end Alembic commands ###


def downgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    op.drop_table('product_tags')
    op.drop_table('tags')
    # ### end Alembic commands ###

This migration script (with the revision ID 6e58a26e5c70) looks good!

You should now have three scripts inside alembic/versions.

alembic/versions/
├── 6e58a26e5c70_created_a_table_for_tags_and_created_a_.py
├── 960969e75871_created_orderproduct_as_an_association_.py
└── a0cf7f5d705d_initial_commit_created_customers_orders_.py

Applying the migration script

Apply the migration script by running:

alembic upgrade head

Connecting to the database shows the following ERD:

We’ve successfully built a connection between Products and Tags via an association table!

Browsing through alembic/versions should give us an idea of the growth of our data model and how it evolves to match the change in scope of our e-commerce workflow. With these three migration scripts, we’ve gone from a rudimentary data model to a more complex one, while being able to track how the data model has evolved from its conception.

Next Steps

In subsequent blog posts, we will cover:

• Using an Async engine for asynchronous database connections
• Using the alembic_utils library to incorporate views, functions and triggers into the version control system.
• Configuring Alembic to connect to a production environment, and seamlessly bring the production environment’s schema up-to-date with our final revision.

References

• SQLAlchemy’s documentation on many-to-many relationships.
• Alembic’s documentation on auto-generating migrations.

Data Modeling

Data modeling is the process of creating a representation of data that defines the way it is structured and used in complex systems. When it comes to relational databases, designing a normalized data model is essential for efficient database operation, and to make your data consistent and clean.

With a fully normalized data model, data in your relational database becomes:

• Understandable

A good data model can decompose the complexity of real-world systems and the data they generate into a relational model that is easy to read and comprehend, and standardizes the relationship of one data domain with every other data domain.

• Scalable

A well defined and normalized data model becomes scalable, and allows for sustainable development of data-driven solutions for users of your database as the scope of your data grows.

• Modular

A database with a well designed and thoroughly documented data model becomes easy to use by downstream teams such as API designers and front-end developers, minimizing the communication needed for these teams to function independently. Good design will naturally flow downstream of a properly normalized database.

Efficient data model design can help your database be a bedrock for your fellow team members to rely on.

SQLAlchemy

SQLAlchemy is a Python SQL toolkit and ORM (Object Relational Mapper) that allows Python developers to work with relational databases using the Python language. SQLAlchemy is indispensable if you’re a Python developer looking to create a relational data model for your database. The two most significant components of SQLAlchemy are it’s Object Relational Mapper (ORM) and SQLAlchemy Core.

• SQLAlchemy Core

SQLALchemy Core is the foundation of SQLAlchemy, and is independent of the ORM. It provides the connectivity to the database, and by allowing for the construction of SQL expressions that can then be executed against a target database, enabling interaction with a database to in the form of queries.

• SQLAlchemy ORM

The SQLAlchemy ORM (Object Relational Mapper) uses Core as a foundation. The ORM will be the bulk of what we discuss in this blog post. The ORM builds on Core to help create user-defined Python classes that can be mapped into database tables. The ORM is what allows SQLAlchemy to define a data model for a relational database.

Alembic

Alembic is a database migration tool written by the author of SQLAlchemy. It helps create a version-controlled system of database migrations that can be used to upgrade a target database to any version of your data model. It uses SQLAlchemy as the underlying engine. By executing migrations in a sequential order, databases can be upgraded and downgraded across versions, even when you are building from scratch.

Alembic is a great way to test your migrations and schema design in a development environment before transferring these migrations to a production environment. An empty database environment can be converted into a database with a production ready schema with just a single command using Alembic.

What we’ll cover

In this blog post, we will cover how you can begin using SQLAlchemy and Alembic to start developing the first iteration of your data model.

We will do the following:

• Create a virtual environment with SQLAlchemy and Alembic installed
• Initialize Alembic
• Design a simple, rudimentary data model with three tables
• Configure Alembic to connect to your database
• Utilize Alembic to generate a migration script
• Migrate your initial data model to the database for the first time

In future posts, we will cover how you can use Alembic to generate database migrations and help grow your database within a version-controlled system, and how this data model can then be plugged into a FastAPI server as a module to help design a modern API service.

Prerequisites

The code in this blog post was written with Python 3.13.1, Alembic 1.14.0 and SQLAlchemy 2.0.36.

You will need the following prerequisites:

• A PostgreSQL server initialized and running, with a database that has been initialized but is otherwise empty. We will refer to this server as development from here.

• An installation of mamba,micromamba or conda as a package manager to create a new environment. I like to use micromamba. You can find instructions on how to install micromamba here.

We will create a new virtual environment using Python 3.13.1 as the pinned version. Create an environment named data_model as follows:

micromamba create -n data_model python=3.13.1 sqlalchemy=2.0.36 alembic=1.14.0 psycopg2=2.9.9 -y

This will create a micromamba environment with the pinned versions of Python and SQLAlchemy that can then be used to follow this tutorial. This environment can be activated with the command micromamba activate data_model.

Lets now get into actually creating some data models!

Getting Started

Setting up the Alembic environment

Before we can start working with our PostgreSQL database, we will first set up the migration environment using Alembic. The reason we set this up first is because we will use Alembic’s config files to connect to our database first before we can begin creating our data model. Alembic uses SQLAlchemy core’s Engine object as a dependency when connecting to a database.

In the root of your project directory, run:

alembic init alembic

This will generate a directory of scripts. The root of your project directory will look like this:

.
├── alembic
│   ├── README
│   ├── env.py
│   ├── script.py.mako
│   └── versions
└── alembic.ini

The directory now includes these files of importance:

alembic.ini

This is Alembic’s main configuration file. When Alembic is run, it looks in the directory for this file. This file is where you will add the SQLAlchemy URL to connect to your database. You can also define multiple environments to run migrations in.

For now we will go with the boilerplate file generated by the alembic init file. Here is what the default file should look like:

# A generic, single database configuration.

[alembic]
# path to migration scripts
# Use forward slashes (/) also on windows to provide an os agnostic path
script_location = alembic

# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
# Uncomment the line below if you want the files to be prepended with date and time
# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
# for all available tokens
# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s

# sys.path path, will be prepended to sys.path if present.
# defaults to the current working directory.
prepend_sys_path = .

# timezone to use when rendering the date within the migration file
# as well as the filename.
# If specified, requires the python>=3.9 or backports.zoneinfo library.
# Any required deps can installed by adding `alembic[tz]` to the pip requirements
# string value is passed to ZoneInfo()
# leave blank for localtime
# timezone =

# max length of characters to apply to the "slug" field
# truncate_slug_length = 40

# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false

# set to 'true' to allow .pyc and .pyo files without
# a source .py file to be detected as revisions in the
# versions/ directory
# sourceless = false

# version location specification; This defaults
# to alembic/versions.  When using multiple version
# directories, initial revisions must be specified with --version-path.
# The path separator used here should be the separator specified by "version_path_separator" below.
# version_locations = %(here)s/bar:%(here)s/bat:alembic/versions

# version path separator; As mentioned above, this is the character used to split
# version_locations. The default within new alembic.ini files is "os", which uses os.pathsep.
# If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas.
# Valid values for version_path_separator are:
#
# version_path_separator = :
# version_path_separator = ;
# version_path_separator = space
# version_path_separator = newline
version_path_separator = os  # Use os.pathsep. Default configuration used for new projects.

# set to 'true' to search source files recursively
# in each "version_locations" directory
# new in Alembic version 1.10
# recursive_version_locations = false

# the output encoding used when revision files
# are written from script.py.mako
# output_encoding = utf-8

sqlalchemy.url = driver://user:pass@localhost/dbname


[post_write_hooks]
# post_write_hooks defines scripts or Python functions that are run
# on newly generated revision scripts.  See the documentation for further
# detail and examples

# format using "black" - use the console_scripts runner, against the "black" entrypoint
# hooks = black
# black.type = console_scripts
# black.entrypoint = black
# black.options = -l 79 REVISION_SCRIPT_FILENAME

# lint with attempts to fix using "ruff" - use the exec runner, execute a binary
# hooks = ruff
# ruff.type = exec
# ruff.executable = %(here)s/.venv/bin/ruff
# ruff.options = --fix REVISION_SCRIPT_FILENAME

# Logging configuration
[loggers]
keys = root,sqlalchemy,alembic

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = WARNING
handlers = console
qualname =

[logger_sqlalchemy]
level = WARNING
handlers =
qualname = sqlalchemy.engine

[logger_alembic]
level = INFO
handlers =
qualname = alembic

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S

To begin with, we will substitute the sqlalchemy.url parameter for the alembic section with the connection string used for our database. As an example, the following is a dummy connection string.

sqlalchemy.url = postgresql://nk_dev:placeholder_password@localhost/development

In the above code snippet:

• The database driver is postgresql
• The database username is nk_dev
• The password of the user is placeholder_password
• The database host is localhost
• The name of the database is development

alembic.ini

A more in-depth explanation of the alembic.ini file can be found in the official documentation here.

The alembic directory

This directory is the home of the migration environment. The relevant contents of this directory include:

• env.py

This is a Python script that can be run when the migration tool is invoked. You can do some cool stuff with this script, like registering entities like views, functions and triggers, and also generating filters to ignore tables that you don’t want to migrate with Alembic.

• versions/

This directory holds the individual version scripts. This directory will have the scripts in it run sequentially when upgrading to or downgrading to a particular version of your database. As you begin generating migrations and applying them, this folder will grow. Each migration will become it’s own script, containing both an upgrade and a downgrade parameter for that particular version.

Now that all of this is finally sorted, we can get to creating basic data models!

Creating a file for our data models

Create a file in the project directory called models.py. This file will be where we store our user-defined Python classes that will then be mapped into database tables in the database using Alembic to run the actual migrations. Let’s start with a simple scenario where we have commerce data for customers, products and orders. Each customer can submit multiple orders, and each order can have several products in it.

As a basic assumption, this initial data model isn’t bad, but there is plenty of room for improvement and refinement in the future. Let’s start with defining these three tables as Python classes first.

We will build a one-to-many relationship between customers and orders, since every customer can have many orders, and each order belongs to just one customer. We will build a table to define our products as well.

This is what your models.py file will look like:

from sqlalchemy import (
    Column,
    Integer,
    String,
    Float,
    ForeignKey,
    Identity,
    Text,
    DateTime,
)
from sqlalchemy.orm import declarative_base, relationship
from sqlalchemy.sql import func

Base = declarative_base()

class Customer(Base):
    __tablename__ = "customers"

    customer_id = Column(
        Integer, Identity(always=True), primary_key=True
    )
    first_name = Column(String, nullable=False)
    last_name = Column(String, nullable=False)
    email_address = Column(String, nullable=False)

    orders = relationship(
        "Order",
        back_populates="customer",
        cascade="all, delete-orphan",
    )

class Order(Base):
    __tablename__ = "orders"

    order_id = Column(
        Integer, Identity(always=True), primary_key=True
    )
    customer_id = Column(
        Integer,
        ForeignKey("customers.customer_id", ondelete="CASCADE"),
        nullable=False
    )
    order_date = Column(
        DateTime(timezone=True),
        server_default=func.now(),
        nullable=False,
    )

    customer = relationship(
        "Customer",
        back_populates="orders"
    )

class Product(Base):
    __tablename__ = "products"

    product_id = Column(
        Integer, Identity(always=True), primary_key=True
    )
    product_name = Column(
        String, nullable=False
    )
    description = Column(Text)
    price = Column(Float, nullable=False)

With these classes in models.py, We have created the following objects: Customer, Order and Product.

Customer:

This object describes the attributes of the customers. The customer_id column is the primary key of the table, and we initialize it as a serialized integer. We have added some additional parameters to the column (which also apply to the primary keys of every other table).

• always=True means the identity is always generated (even if a value is provided)

first_name,last_name and email_address are string columns that cannot be null, and need to be present in the data. This code snippet:

    orders = relationship(
        "Order",
        back_populates="customer",
        cascade="all, delete-orphan",
    )

establishes a one-to-many relationship between customers and orders.

The cascade behavior ("all", "delete-orphan") means that when a instance of customer is deleted, all the associated orders (Orders that used the ID of the deleted customer as a foreign key) will also be deleted, and these changes will cascade downwards to any future child table of the orders table.

Order:

This object describes the attributes of each individual order. order_id is the primary key of this table. We designate the customer_id column as a foreign key from the customers table with this code snippet:

  customer_id = Column(
        Integer,
        ForeignKey("customers.customer_id", ondelete="CASCADE"),
        nullable=False
    )

The foreign key enforces referential integrity, while nullable=False ensures that each order must be associated with a customer. Orders that are associated with non-existent customers, or have no customer at all, will not be possible.

We also establish a many-to-one relationship between orders and customers using the following code snippet:

  customer = relationship(
        "Customer",
        back_populates="orders"
    )

Product:

This object describes the attributes of each individual product. product_id is the primary key of this table, and this table does not have any relationships with any other tables yet. This data model is currently incomplete, but it is a great starting point at which to begin the development and migration of our data model.

With this file complete, our rudimentary data model has been initialized. Without migrating these changes over to the database, however, these will only live as Python classes, and not as database objects.

Doing our first database migration.

The vast majority of database migration workflows with Alembic are done by auto-generating migration scripts. To begin auto-generating migration scripts, we will first need to connect Alembic to the database, and then point alembic towards the target metadata we would like to migrate the data model to.

Alembic can be configured to connect to the database (pointed to be the sqlalchemy.url parameter in the alembic.ini file). For Alembic to compare the state of the database against our target metadata, we will edit alembic/env.py so that Alembic can access the table metadata object that contains the target. Configuring Alembic therefore has two components:

• Connecting Alembic to the database, so that it can compare the state of the database against the target metadata.
• Importing the target metadata, so that Alembic knows what to compare the state of the database against.

Once alembic/env.py is configured, Alembic can then auto generate the “obvious” migrations based on a comparison. In this case, we want it to compare the empty database against our models, and then auto-generate migrations that we can then review and modify before applying.

We have already defined the database connection string in our configuration file. We will begin working with the env.py to access the table metadata object that contains the target.

Editing env.py to import our models

The following is the default content of alembic/env.py. The sections that we will edit have been highlighted.

from logging.config import fileConfig

from sqlalchemy import engine_from_config
from sqlalchemy import pool

from alembic import context

# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config

# Interpret the config file for Python logging.
# This line sets up loggers basically.
if config.config_file_name is not None:
    fileConfig(config.config_file_name)

# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata
target_metadata = None

# other values from the config, defined by the needs of env.py,
# can be acquired:
# my_important_option = config.get_main_option("my_important_option")
# ... etc.


def run_migrations_offline() -> None:
    """Run migrations in 'offline' mode.

    This configures the context with just a URL
    and not an Engine, though an Engine is acceptable
    here as well.  By skipping the Engine creation
    we don't even need a DBAPI to be available.

    Calls to context.execute() here emit the given string to the
    script output.

    """
    url = config.get_main_option("sqlalchemy.url")
    context.configure(
        url=url,
        target_metadata=target_metadata,
        literal_binds=True,
        dialect_opts={"paramstyle": "named"},
    )

    with context.begin_transaction():
        context.run_migrations()


def run_migrations_online() -> None:
    """Run migrations in 'online' mode.

    In this scenario we need to create an Engine
    and associate a connection with the context.

    """
    connectable = engine_from_config(
        config.get_section(config.config_ini_section, {}),
        prefix="sqlalchemy.",
        poolclass=pool.NullPool,
    )

    with connectable.connect() as connection:
        context.configure(
            connection=connection, target_metadata=target_metadata
        )

        with context.begin_transaction():
            context.run_migrations()


if context.is_offline_mode():
    run_migrations_offline()
else:
    run_migrations_online()

First, we will import the models.py script. Edit the highlighted snippet to add an import statement.

from models import Base

Then we will replace the target_metadata value with the metadata of our data model. We will change the second highlighted section to:

target_metadata = Base.metadata

Our env.py file should now look like the following, with our changes being highlighted.

from logging.config import fileConfig

from sqlalchemy import engine_from_config
from sqlalchemy import pool

from alembic import context
from models import Base

# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config

# Interpret the config file for Python logging.
# This line sets up loggers basically.
if config.config_file_name is not None:
    fileConfig(config.config_file_name)

# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata
target_metadata = Base.metadata

# other values from the config, defined by the needs of env.py,
# can be acquired:
# my_important_option = config.get_main_option("my_important_option")
# ... etc.


def run_migrations_offline() -> None:
    """Run migrations in 'offline' mode.

    This configures the context with just a URL
    and not an Engine, though an Engine is acceptable
    here as well.  By skipping the Engine creation
    we don't even need a DBAPI to be available.

    Calls to context.execute() here emit the given string to the
    script output.

    """
    url = config.get_main_option("sqlalchemy.url")
    context.configure(
        url=url,
        target_metadata=target_metadata,
        literal_binds=True,
        dialect_opts={"paramstyle": "named"},
    )

    with context.begin_transaction():
        context.run_migrations()


def run_migrations_online() -> None:
    """Run migrations in 'online' mode.

    In this scenario we need to create an Engine
    and associate a connection with the context.

    """
    connectable = engine_from_config(
        config.get_section(config.config_ini_section, {}),
        prefix="sqlalchemy.",
        poolclass=pool.NullPool,
    )

    with connectable.connect() as connection:
        context.configure(
            connection=connection, target_metadata=target_metadata
        )

        with context.begin_transaction():
            context.run_migrations()


if context.is_offline_mode():
    run_migrations_offline()
else:
    run_migrations_online()

Auto-generating a migration script with Alembic

Now that env.py and alembic.ini have been configured, and we have created our first data model, we can auto-generate our first migration script! We will run the command:

alembic revision --autogenerate -m "Initial commit, created customers, orders and products"

This code does the following:

• Runs Alembic and creates a new revision.
• Tells Alembic to auto-generate the migration script.
• Creates a message string to use with the revision. In this case, this string is “Initial commit, created customers, orders and products”. This is analogous to a commit message when using git for version control.

You should get output like this:

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.autogenerate.compare] Detected added table 'customers'
INFO  [alembic.autogenerate.compare] Detected added table 'products'
INFO  [alembic.autogenerate.compare] Detected added table 'orders'
  Generating /path_to_your_project/alembic/versions/de5f32c11361_initial_commit_created_customers_orders_.py ...  done

This output means that Alembic has successfully generated a migration script, and the script is located at /path_to_your_project/alembic/versions/de5f32c11361_initial_commit_created_customers_orders_.py. The versions directory will have the following content:

alembic/versions/
└── de5f32c11361_initial_commit_created_customers_orders_.py

The contents of this auto-generated script are:

"""Initial commit, created customers, orders and products

Revision ID: de5f32c11361
Revises:
Create Date: 2026-01-15 16:10:01.002285

"""
from typing import Sequence, Union

from alembic import op
import sqlalchemy as sa


# revision identifiers, used by Alembic.
revision: str = 'de5f32c11361'
down_revision: Union[str, None] = None
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None


def upgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    op.create_table('customers',
    sa.Column('customer_id', sa.Integer(), sa.Identity(always=True), nullable=False),
    sa.Column('first_name', sa.String(), nullable=False),
    sa.Column('last_name', sa.String(), nullable=False),
    sa.Column('email_address', sa.String(), nullable=False),
    sa.PrimaryKeyConstraint('customer_id')
    )
    op.create_table('products',
    sa.Column('product_id', sa.Integer(), sa.Identity(always=True), nullable=False),
    sa.Column('product_name', sa.String(), nullable=False),
    sa.Column('description', sa.Text(), nullable=True),
    sa.Column('price', sa.Float(), nullable=False),
    sa.PrimaryKeyConstraint('product_id')
    )
    op.create_table('orders',
    sa.Column('order_id', sa.Integer(), sa.Identity(always=True), nullable=False),
    sa.Column('customer_id', sa.Integer(), nullable=False),
    sa.Column('order_date', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
    sa.ForeignKeyConstraint(['customer_id'], ['customers.customer_id'], ondelete='CASCADE'),
    sa.PrimaryKeyConstraint('order_id')
    )
    # ### end Alembic commands ###


def downgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    op.drop_table('orders')
    op.drop_table('products')
    op.drop_table('customers')
    # ### end Alembic commands ###

We can see that a migration script has been generated, with an upgrade and a downgrade parameter. Alembic has compared our target metadata against the database, and has generated a migration script complete with the operations needed to upgrade our database to the state defined by our ORM.

Examining the auto-generated migration script

Going through the upgrade function generated by Alembic, we can see that the following migrations have been generated:

• The customers, orders and products tables have been created, with their primary keys being customer_id,order_id and product_id respectively.

• The attributes defined by our models.py file have been correctly added to these newly created tables.

• From this code snippet:

  sa.ForeignKeyConstraint(['customer_id'], ['customers.customer_id'], ondelete='CASCADE'), We can see that the customer_id field in the orders table has been made a foreign key, connecting to the customer_id column in the customers table.

The migration script looks good, and does not need further refinement. Let’s now actually apply our first migration!

Applying the Alembic-generated migration script

We will now apply the migration script with the following command:

alembic upgrade head

This command tells Alembic to upgrade the database to the latest revision (the “head”). Depending on the state of the database, it will work through the scripts located in the versions directory to bring the database’s metadata in line with the most recent migration script.

The output of the command is as follows:

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> de5f32c11361, Initial commit, created customers, orders and products

Since our database is empty and there is only a single migration script, only that script has been run. The last line confirms that migration de5f32c11361 has been successfully applied.

Examining the database to verify our migration

Connect to your database with the database management tool of your choice. I use DBeaver.

The Entity Relationship Diagram for the public schema of your database should look like this.

We’ve successfully applied our first database migration! We’ve created the customers, orders and products tables, and have built a foreign key relationship between customers and orders.

The alembic_version table has been newly created by Alembic. Alembic uses this table to identify the path to upgrade the database from its current state to the version requested by the alembic upgrade command. In our case, we requested the version head.

Alembic works through the migration scripts in the versions directory, starting from the version found in the database’s alembic_version, up to the script that defined the version requested by us, invoking the upgrade() method in each file to get to the target revision. You can check this by examining the data in the alembic_version table. It should have a single entry, corresponding to the revision of the script we just applied.

Next Steps

This is a great first step towards building a comprehensive data model, but there is a long way to go before we can consider migrating our data model from a development environment to a production environment! In subsequent blog posts, we will cover the following:

• Updating our data model with more tables and relationships to create a proper e-commerce database schema.
• Using an Async engine for asynchronous database connections
• Using the alembic_utils library to incorporate views, functions and triggers into the version control system.
• Configuring Alembic to connect to a production environment, and seamlessly bring the production environment’s schema up-to-date with our final revision.

References

• SQLAlchemy documentation, most notably the SQLAlchemy Unified Tutorial Section.

• Alembic’s documentation, and importantly, the section covering Autogeneration.

pgBackRest is an open source tool that allows you to perform automated backup and restore operations for a PostgreSQL server. It allows you to take both full and incremental file system backups of your data. With pgBackRest, you can set up multiple repositories for your backups, both local and remote. You can further customize the configuration to take backups at specific times of the day so that regular operations are not affected, and backup retention and rotation can be customized as needed.

Furthermore, it allows mirroring of backups to cloud-compatible object stores, including S3, Azure, and GCS compatible object stores.

Insecurely configured SFTP backups can expose critical database content to unauthorized access, yet many organizations rely on them for off-site storage. This guide addresses the security gap and will explain the process of setting up remote SFTP repositories in a secure manner. We will create a system user on the SFTP repository and limit their permissions, and perform scheduled backups exclusively via the limited user account.

A good database backup strategy uses multiple repositories of different types, alongside both physical and logical backups. Do not limit yourself to merely SFTP repositories. Remember, with database backups, redundancy is good. This tutorial is simply to explain how to securely configure SFTP backups.

Database Server Requirements

• PostgreSQL 12.0 or newer (this tutorial uses PostgreSQL 17.0)

  • PostgreSQL 12.0+ is required because earlier versions lack WAL archiving features that enable point-in-time recovery when using pgBackRest.

• pgBackRest 2.41 or newer (this tutorial uses pgBackRest 2.55.1)

  • Earlier versions may have limitations with certain authentication methods

• OpenSSH 8.0+ client for improved Ed25519 key support

• libssh2 1.9.0+ for modern cryptographic algorithm support

• Sufficient disk space for temporary backup files before transfer

SFTP Host Requirements

• Linux server with SSH daemon supporting SFTP subsystem

  • Recommended: OpenSSH 8.0+ for optimal security features

• Sufficient storage space for your backup retention needs. Plan for minimum 3x your database size (1x for full backup + 1x for transaction logs + 1x overhead and retention)

• Root access for initial configuration

• Ability to create dedicated system users and groups

• Firewall configured to allow SFTP connections (typically port 22) from your database server IP

Network Requirements

• Stable network connection between database and SFTP servers

• Firewall rules allowing SFTP traffic (TCP port 22 by default)

• Recommended: Internal network connectivity (avoid public internet exposure if possible)

You can check your versions with these commands:

postgres --version
pgbackrest version
ssh -V

The Process

For this process, this guide assumes both servers are on a private network (10.0.0.0/24) protected by a corporate firewall, with the following IP addresses:

• The IP of the database server is 10.0.0.1

• The user that runs the code on the database server is postgres. postgres is a system user and cannot log in via SSH.

• The IP of the SFTP repo is 10.0.0.2

On the database host

• Create an SSH key pair in the postgres user’s home directory.

# Adjust this path based on your system's postgres home
export PGHOME=/var/lib/postgresql
ssh-keygen -f $PGHOME/.ssh/id_ed25519_sftp_key -t ed25519 -N ""

Ensure that the .ssh folder is owned by the postgres user. We use Ed25519 keys because they provide stronger security with shorter keys compared to RSA, and are supported by modern OpenSSH implementations.

• Perform a key scan on the SFTP host to gather the public keys. Make sure you verify the legitimacy of the host server before you build your known hosts file.

ssh-keyscan 10.0.0.2 >> $PGHOME/.ssh/known_hosts

You can validate the host keys via fingerprints by running:

ssh-keygen -lf $PGHOME/.ssh/known_hosts

On the SFTP server host

Create a directory that the SFTP user account will be restricted to.

We create a dedicated /data directory separate from standard system paths to isolate backup data and simplify permission management.

# Creating a folder in the root directory
mkdir /data

# Creating the home directory for the SFTP user
mkdir -p /data/sftpuser/upload

## Setting the appropriate permissions for this filesystem
chmod 701 /data

## /data needs to have root:root permissions
chown root:root /data

Create an SFTP group, followed by an SFTP user with minimal permissions and as a system user.

# Creating an SFTP group
groupadd sftpusers
# Creating the SFTP user as a system user
useradd -g sftpusers -d /data/sftpuser/upload/ -s /usr/sbin/nologin -r sftpuser

This process creates a group called sftpusers and creates a system user in that group called sftpuser, with their home directory being /data/sftpuser/upload.

Change the permissions for the SFTP directory.

## Giving ownership to the sftpusers group for the sftpuser folder
chown root:sftpusers /data/sftpuser

## Giving the sftpuser system account access to the home directory we created
chown sftpuser:sftpusers /data/sftpuser/upload

It is very important that /data/sftpuser/upload must not be owned by root. This is a common cause of failure for chroot setups.

Updating the SSH configuration on the SFTP server.

Add the following to the end /etc/ssh/sshd_config:

# For sftp backups
Match Group sftpusers
  ChrootDirectory /data/%u
  ForceCommand internal-sftp
  AllowTcpForwarding no
  X11Forwarding no

Here’s what this does:

• The first line Match Group sftpusers starts a conditional block that only applies to users in the sftpusers group.

• The second line ChrootDirectory /data/%u (jails) the user to the directory /data/username, where %u is replaced with the username. This ensures the user is jailed to their upload directory and cannot access the broader file system.

• The third line ForceCommand internal-sftp forces the SSH session to use the internal SFTP subsystem. The user cannot get a shell session, and only SFTP is allowed.

• The fourth line AllowTcpForwarding no prevents the user using SSH port forwarding to avoid tunneling through SSH.

• The fifth line X11Forwarding no disables X11 forwarding to prevent GUI application tunneling over SSH.

Validate your SSH config and then Restart the SSH daemon

sudo sshd -T && sudo systemctl restart sshd

Create the SSH directory for the sftpuser account with appropriate permissions

# Creating the .ssh directory
mkdir /data/sftpuser/upload/.ssh

## Creating the authorized_keys and known_hosts files
touch /data/sftpuser/upload/.ssh/authorized_keys
touch /data/sftpuser/upload/.ssh/known_hosts

# Changing permissions for the newly created folders and files
chown -R sftpuser:sftpusers /data/sftpuser/upload/.ssh
chmod 700 /data/sftpuser/upload/.ssh
chmod 640 /data/sftpuser/upload/.ssh/authorized_keys
chmod 644 /data/sftpuser/upload/.ssh/known_hosts

Append the contents of the new public SSH key to the authorized_keys file

On the database host, copy the contents of the public key we created ($PGHOME/.ssh/id_ed25519_sftp_key.pub) and append it to the authorized_keys file on the SFTP user’s SSH directory (’/data/sftpuser/upload/.ssh/authorized_keys’).

Testing

From the database host, test if you can login to the SFTP server using the keys without needing a password.

sftp -i $PGHOME/.ssh/id_ed25519_sftp_key sftpuser@10.0.0.2

If your setup was correct, this will let you login without a password.

Deploying

Update your pgbackrest.config file with the following:

# SFTP Backup Repo
repo2-bundle=y
repo2-path=/upload
# Host
repo2-sftp-host=10.0.0.2
repo2-sftp-host-key-hash-type=sha256
# User
repo2-sftp-host-user=sftpuser
# Key Location
repo2-sftp-private-key-file=/var/lib/postgresql/.ssh/id_ed25519_sftp_key
repo2-sftp-public-key-file=/var/lib/postgresql/.ssh/id_ed25519_sftp_key.pub
repo2-type=sftp

The above configuration file sets up the automation process for transferring backups to the SFTP host via the SFTP service, as the user sftpuser, who has restricted permissions.

An example of a full pgbackrest.conf file: (The default location is /etc/pgbackrest/pgbackrest.conf)

[main]
pg1-path=/usr/local/pgsql/data
pg1-port=5432
pg1-user=postgres
pg1-database=postgres

[global]
# Local Backup Repo
repo1-cipher-pass=xxx # Insert with your own
repo1-cipher-type=aes-256-cbc
repo1-path=/var/lib/pgbackrest
repo1-retention-full=2

# SFTP Backup Repo 1
# This section handles the configuration for your SFTP repo
repo2-bundle=y
repo2-path=/upload
repo2-retention-full=2
# Host
repo2-sftp-host=10.0.0.2
repo2-sftp-host-key-hash-type=sha256
# User
repo2-sftp-host-user=sftpuser
# Key Location
repo2-sftp-private-key-file=/var/lib/postgresql/.ssh/id_ed25519_sftp_key
repo2-sftp-public-key-file=/var/lib/postgresql/.ssh/id_ed25519_sftp_key.pub

# Logging Info
log-path=/var/log/pgbackrest
log-level-console=debug
[global:archive-push]
compress-level=3

Then create the stanza by running the following as the postgres user:

pgbackrest --stanza=main --log-level-console=info stanza-create

If your config is right, then this will succeed with the following output:

P00   INFO: stanza-create command end: completed successfully

Addendum

Using secure key algorithms and hash functions

If you are using older versions of cryptography libraries like libssh2 and openssl, your configuration might be restricted to using older, more outdated key algorithms like RSA or DSS, along with older hash functions like SHA1. I recommend using more secure, newer key algorithms like Ed25519, along with more secure hashing functions like SHA256.

Check the key algorithms supported by your installation of libssh2

Run the following code to see what key types are supported by your installation of libssh2:

strings /usr/lib/x86_64-linux-gnu/libssh2.so.1 | grep ssh-

Your output should return more modern key algorithms like ssh-ed25519. If you output only includes algorithms like ssh-rsa or ssh-dss, then you will need to update your installation of libssh2. Some older servers, like ones running outdated operating systems like Ubuntu 20.04, may not allow you to update your libssh2 library using package managers. In that case, you will have to manually build the newer versions from source.

## Installing dependencies
sudo apt update
sudo apt install build-essential cmake zlib1g-dev libssl-dev

# Downloading and building libssh2 from source
wget https://www.libssh2.org/download/libssh2-1.11.0.tar.gz
tar -xzf libssh2-1.11.0.tar.gz
cd libssh2-1.11.0

# Installing libssh2
./configure
make -j$(nproc)
sudo make install
sudo ldconfig

The new libssh2 library will have it’s binaries located at /usr/local/lib/libssh2.so. You can confirm if it was installed by running:

strings /usr/local/lib/libssh2.so | grep ssh-

You should see the newer key algorithms like ecdsa-sha2-nistp256and ssh-ed25519.

If you built/installed pgBackRest before updating your libssh2 library, you will need to rebuild it from source so that it uses the newer library.

To check if pgBackRest is using the newly installed libssh2 as a dependency, run:

ldd $(which pgbackrest) | grep libssh2

Your output should look like this:

libssh2.so.1 => /usr/local/lib/libssh2.so.1 (0x00007f20882cf000)

This confirms that pgBackRest is using the manually installed libssh2 from /usr/local/lib, rather than the system-wide default at /usr/lib/x86_64-linux-gnu.

References

• pgBackRest documentation

• libssh2

• PostgreSQL backup documentation

The HDFS (Hadoop Distributed File System) and PostgreSQL databases are both powerful tools for data storage, queries and analyses. Each have their own unique strengths that make them well suited for specific tasks.

The HDFS, being distributed across several computing nodes, is robust and amenable to storing massive datasets, provided your computing infrastructure has the prerequisite width (the number of nodes in your cluster) and the depth(the available memory on each individual node). The HDFS is optimized for batch processing of massive datasets, making it suitable for big data applications like data warehousing, log processing, and large-scale data analytics. In fact, Spark, the HDFS’ natural companion, has it’s own machine learning library MLlib, making large scale data analytics very much possible.

PostgreSQL databases, on the other hand, are robust RDBMS optimized for ACID (Atomicity, Consistency, Isolation, Durability) compliance. It provides robust data security and access control mechanisms, making it suitable for applications that require strict data protection. PostgreSQL supports SQL, a powerful query language that enables efficient querying and retrieval of data based on specific criteria. PostgreSQL shines at handling structured, transactional workloads that require data integrity, consistency, and complex querying capabilities.

Why connect the two?

The ability to trivialize Data Ingestion.

Spark has a tool called inferSchema. This is a function that allows for the input schema of data to be automatically inferred. It allows for more flexibility and easier data ingestion, especially when working with semi-structured or unstructured data formats. Provided your input data is cleaned up, the schema dictated by inferSchema is very reliable. It makes specifying the schema for a dataset with a large number of columns easy! This eliminates the need for manually defining the schema, which can be a time-consuming and error-prone process, especially when dealing with datasets containing a large number of columns or complex data structures.

Once this data has it’s schema specified, connecting a HDFS to a PostgreSQL database allows for the rapid, trivial loading of data into database tables.

This helps streamline the Data ingestion process as follows:

• Data storage in HDFS: The raw data files (e.g., CSV, JSON) are stored in the HDFS, which provides distributed storage and fault tolerance.

• Schema inference with Spark: Spark reads the data from the HDFS and automatically creates the schema when using inferSchema. This process ensures that the schema accurately reflects the structure of the input data, especially if it has been cleaned up or transformed.

Even if your data isn’t clean to begin with, Spark allows for the data to be cleaned and then written back out into the HDFS, and inferSchema will ensure the schema of the table is accurate when this file is read back in.

• Loading data into a PostgreSQL database: With the schema inferred by Spark, the data can be easily loaded into PostgreSQL database tables.

And just like that, PostgreSQL gains the ability to ingest data from a wide range of semi-structured or unstructured data sources stored in a HDFS cluster quickly and reliably.

Connecting HDFS to a PostgreSQL database

The prerequisites

This tutorial assumes you have the following prerequisites set up:

• Hadoop Distributed File System (HDFS): You have HDFS installed and running on your cluster. HDFS will serve as the distributed storage system for your data files.

• Apache Spark: You have Apache Spark installed and configured to run on your cluster. Spark will act as the data processing engine to read data from HDFS and load it into PostgreSQL.

• PostgreSQL Database: You have a PostgreSQL database instance up and running, either on the same cluster or a separate server. This is the target database where you want to load your data from HDFS.

Downloading the PostgreSQL JDBC driver

Find the PostgreSQL JDBC driver that is appropriate for your Java version and download it. The JDBC driver provides a standard API for Java applications (like Spark) to interact with relational databases. For example, if you’re using PostgreSQL 14.x and Java 8, you might download the postgresql-42.x.x.jar file.

wget https://jdbc.postgresql.org/download/postgresql-xx.x.x.jar

Once you have the jar file downloaded, place it in the jars folder of your Spark installation.

Edit your spark-env.sh file (which can be found in the conf folder) to have the following line:

export SPARK_CLASSPATH=/path/to/spark-x.x.x-bin-hadoop3/jars/postgresql-xx.x.x.jar

Now restart your HDFS cluster, YARN, and MapReduce services to ensure the changes take effect. You can typically do this by running the stop-all.sh and start-all.sh scripts provided with your Hadoop installation.

Connecting PySpark to PostgreSQL

The following code chunk makes the following assumptions. Change them to suit your use case:

The host name of the server that runs the PostgreSQL database is postgres.server.

The port on which PostgreSQL is listening is 5432 (the default port).

The name of the database you want to load the tables into is postgres.

The name of the user who has connection and write privileges on that database is user.

The password for that user is password.

The raw data (that has been cleaned) is named data_file, and is a CSV file. This data type can be anything that’s loaded on the HDFS, just remember to change the type of reader being used.

# Assumptions and configurations
data_file = "/path/to/data_file.csv"  # Replace with the actual path to your data file on HDFS
table_name = "your_table_name"  # Replace with the desired table name in PostgreSQL

# Read the data from HDFS
data = spark.read.csv(data_file,
                      header=True,
                      sep=',',
                      inferSchema=True)  # inferSchema=True will automatically infer the schema

# PostgreSQL database connection details
jdbc_url = "jdbc:postgresql://postgres.server:5432/postgres"
properties = {
    "user": "user",
    "password": "password",
    "driver": "org.postgresql.Driver"
}

# Write the data to the PostgreSQL table
data.write.jdbc(url=jdbc_url, table=table_name, mode="overwrite", properties=properties)

This code chunk will:

1. Read the data in, and automatically create a schema. The spark.read.csv() method is used to read the data from the specified CSV file located on HDFS. It assumes that the file has a header row, and the columns are separated by commas. The inferSchema=True option tells Spark to automatically infer the schema (column names and data types) from the data.

2. Set up a connection to your PostgreSQL database. The jdbc_url variable specifies the JDBC connection string for the PostgreSQL database, including the hostname, port, and database name. The properties dictionary contains the user credentials (username and password) and the PostgreSQL JDBC driver class name.

3. Write the data to the database with the name table_name. The data.write.jdbc() method is used to write the Spark DataFrame (data) to the specified PostgreSQL table. The url parameter specifies the JDBC connection URL, the table parameter specifies the target table name, and the mode="overwrite" option tells Spark to overwrite the table if it already exists. The properties parameter passes the connection properties, including the user credentials and driver class name.

With this integration between HDFS, Apache Spark, and PostgreSQL, you can efficiently ingest and process large volumes of data stored in the HDFS and load it into your PostgreSQL database. Leveraging Spark’s powerful data processing capabilities and its ability to automatically infer schemas streamlines the data ingestion process, making it more efficient and flexible. By combining the strengths of these two data storage systems, you can unlock new possibilities for storing and analyzing large datasets.

Mamba is intended to be a drop-in replacement for /reimplementation of conda (written in C++).

Mamba has been something that I have implemented into all of my pipelines, since it trivializes the package management process. I do almost all of my work within the context of containers/virtual environments, and mamba makes my work life so much easier.

Previously, I used to use Conda as my package manager of choice, relying on it to cut down the amount of time I would need to build an environment using pip as an installer. Once I discovered Mamba, however, I never went back to Conda.

Why use Mamba over Conda?

Simple. Mamba is just faster.

Conda is implemented in Python, while Mamba utilizes C++ code for critical operations (instead of Python), optimizing it’s performance.

At the same time, Mamba utilizes the same command line parser, package installation and deinstallation code and transaction verification routines as Conda to stay as compatible as possible.

This means that Mamba is a drop-in replacement for Conda. If you know how to use Conda, you know how to use Mamba!

Mamba’s features include:

• multi-threaded parallel downloads of repo data and package files.

• libsolv for much faster dependency solving, which is also used in the RPM package manager of Red Hat, Fedora and OpenSUSE.

• core parts of mamba are implemented in C++.

• utilizes the same command line parser, package installation and deinstallation code and transaction verification routines as Conda to stay as compatible as possible.

Conda excels in creating virtual environments that help keep your base environment clutter-free. Initiating a Conda virtual environment helps you keep the dependencies for a particular project limited to that environment. No need to run the risk of nuking your workflow simply because you had to install Python 2.7 as a dependency for an old package that the maintainer never got around to updating.

Implementing Mamba into your workflow

Implementing Mamba into your workflow is pretty straightforward. The potential options are:

• Mambaforge: The mambaforge installer for conda (similar to conda-forge). This allows usage of the conda-forge channel by default.

• Micromamba: micromamba is a tiny executable version of the mamba package manager. It is a statically linked C++ executable with a separate command line interface. It does not need a base environment and does not come with a default version of Python.

• Miniforge: Similar to miniconda. As of August 2023, virtually similar to mambaforge, and is recommended over mambaforge since it now comes with mamba installed. Miniforge used to only ship with conda.

As of July 2023, Miniforge then started to ship with conda-libmamba-solver. Since conda-libmamba-solver depends on libmambapy, the only difference between Miniforge and Mambaforge was the presence of the mamba Python package. As of August 2023, miniforge added mamba as well to minimize surprises.

• Existing conda Install Simply installing mamba into a pre-existing conda environment. This is not recommended by the official documentation.

Non-containerized mamba

If you’re intending on installing mamba, the official documentation says to ensure that mamba and conda are the only packages to be installed in the base environment.

Note that as of August 2023, miniforge and mambaforge are essentially the same thing.

Given that the official documentation encourages the use of Miniforge over Mambaforge, the rest of this blog post will be looking at installing and using Miniforge.

Miniforge

The official miniforge documentation has a list of installers for different operating systems and architectures.

{fig-align=“center”}

The easiest way to do this is via a non-interactive method, using the following commands (for Unix-like platforms such as Mac OS & Linux)

::: callout-important This section assumes you already have conda installed and set up in the base environment. :::

## Download the appropriate installer for your architecture and OS
wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"

## Run the script in non-interactive mode.
## -p is prefix option. A directory will be created at "${HOME}/conda".
bash Miniforge3-$(uname)-$(uname -m).sh -b -p "${HOME}/conda"

##  Create the path to conda
source "${HOME}/conda/etc/profile.d/conda.sh"

## For mamba support also run the following command
source "${HOME}/conda/etc/profile.d/mamba.sh"

## Activate the base environment
conda activate

This above script installs mamba into the base conda environment and then activates the conda environment, enabling the usage of mamba. Now, using mamba is as simple as replacing all instances of conda with mamba!

Micromamba

The documentation for installation of micromamba can be found here.

micromamba is a tiny executable version of the mamba package manager. It is a statically linked C++ executable with a separate command line interface. It comes with an empty base environment, and does not come with a default version of Python or conda.

The following magic URL always returns the latest available version of micromamba, and the bin/micromamba part is automatically extracted using tar.

# Linux Intel (x86_64):
curl -Ls https://micro.mamba.pm/api/micromamba/linux-64/latest | tar -xvj bin/micromamba
# Linux ARM64:
curl -Ls https://micro.mamba.pm/api/micromamba/linux-aarch64/latest | tar -xvj bin/micromamba
# Linux Power:
curl -Ls https://micro.mamba.pm/api/micromamba/linux-ppc64le/latest | tar -xvj bin/micromamba
# macOS Intel (x86_64):
curl -Ls https://micro.mamba.pm/api/micromamba/osx-64/latest | tar -xvj bin/micromamba
# macOS Silicon/M1 (ARM64):
curl -Ls https://micro.mamba.pm/api/micromamba/osx-arm64/latest | tar -xvj bin/micromamba

After extraction is complete, the micromamba binary can be used.

# Linux/bash:
./bin/micromamba shell init -s bash -p ~/micromamba  # this writes to your .bashrc file
# sourcing the bashrc file incorporates the changes into the running session.
# better yet, restart your terminal!
source ~/.bashrc

The above code chunk incorporates the path to the micromamba binary into the restarted terminal.

You can now activate the base environment, and build other environments.

micromamba activate  # this activates the base environment
micromamba install python=3.6 jupyter -c conda-forge
# or
micromamba create -n env_name xtensor -c conda-forge
micromamba activate env_name

Containerized mamba (specifically Docker)

This is my recommended method of using mamba or micromamba. If you’re using mamba, you’re already encouraged to use conda environments. If you’re using a conda environment, you should 100% consider incorporating the usage of containerization into your scientific workflow.

Containers make scientific pipelines easy to reproduce and easy to share, two concepts which form the bedrock of responsible science.

I have a blog post about containers and why they rule. You can find it here.

Docker-based Miniforge

This is very simple. Simply pull from the condaforge/miniforge3:latest tag.

Start your dockerfile with the line:

FROM condaforge/miniforge3:latest

This uses miniforge as a base image, and now you have a base environment with mamba and conda already installed and configured.

Docker-based Micromamba

This is similar to the above section. Start your dockerfile with the line:

FROM mambaorg/micromamba:latest

A real-world example of why I prefer mamba over conda.

Consider the following two dockerfiles:

conda based dockerfile

FROM condaforge/miniconda3:latest
USER root
RUN conda create -n test_env python=3.11 -y
RUN conda install -n test_env \
    'r-tidyverse' \
    'r-sessioninfo' \
    'r-bedr' \
    'r-dbplyr' \
    'r-rmariadb' \
    'r-seqinr' \
    'r-stringr' \
    'r-e1071' \
    'r-matrixeqtl' \
    'ucsc-blat' \
    'r-immunedeconv' \
    'bioconductor-ensembldb' \
    'bioconductor-bsgenome.hsapiens.ncbi.grch38' \
    'bioconductor-illuminahumanv4.db' \
    'plink' \
    'bioconductor-qvalue'\
    'r-mashr' \
    'git'\
    'r-vcfr' \
    'r-ashr' \
    'r-rstatix' \
    'r-ggpubr' \
    'r-glmnet' \
    'r-janitor' \
    'r-reticulate' \
    -c "bioconda/label/cf201901" -c bioconda -c dnachun \
    -c anaconda -c conda-forge -y \
    && \
    conda clean --all -f -y

RUN echo "source activate test_env" > ~/.bashrc
ENV PATH /opt/conda/envs/test_env/bin:$PATH

mamba based dockerfile

FROM condaforge/miniforge3:latest
USER root
RUN mamba create -n test_env python=3.11 -y
RUN mamba install -n test_env \
    'r-tidyverse' \
    'r-sessioninfo' \
    'r-bedr' \
    'r-dbplyr' \
    'r-rmariadb' \
    'r-seqinr' \
    'r-stringr' \
    'r-e1071' \
    'r-matrixeqtl' \
    'ucsc-blat' \
    'r-immunedeconv' \
    'bioconductor-ensembldb' \
    'bioconductor-bsgenome.hsapiens.ncbi.grch38' \
    'bioconductor-illuminahumanv4.db' \
    'plink' \
    'bioconductor-qvalue'\
    'r-mashr' \
    'git'\
    'r-vcfr' \
    'r-ashr' \
    'r-rstatix' \
    'r-ggpubr' \
    'r-glmnet' \
    'r-janitor' \
    'r-reticulate' \
    -c "bioconda/label/cf201901" -c bioconda -c dnachun \
    -c anaconda -c conda-forge -y \
    && \
    mamba clean --all -f -y

RUN echo "source activate test_env" > ~/.bashrc
ENV PATH /opt/conda/envs/test_env/bin:$PATH

Both dockerfiles are intended to accomplish the same goal. Create a conda env called test_env, install a list of packages in test_env, and then activate the test_env. However, while they accomplish the same goal, the way they both go about it is very different. I’ve told you that the mamba based package installation is superior to using conda since it’s faster, but now let’s look at the output of a docker build command for both of these dockerfiles.

mamba based docker build time

[+] Building 1250.0s (8/8) FINISHED
 => [internal] load build definition from dockerfile                                                                                                                                                        0.4s
 => => transferring dockerfile: 897B                                                                                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                                                                           0.3s
 => => transferring context: 2B                                                                                                                                                                             0.0s
 => [internal] load metadata for docker.io/condaforge/miniforge3:latest                                                                                                                                     0.0s
 => CACHED [1/4] FROM docker.io/condaforge/miniforge3:latest                                                                                                                                                0.0s
 => [2/4] RUN mamba create -n test_env python=3.11 -y                                                                                                                                                      71.8s
 => [3/4] RUN mamba install -n test_env     'r-tidyverse'     'r-sessioninfo'     'r-bedr'     'r-dbplyr'     'r-rmariadb'     'r-seqinr'     'r-stringr'     'r-e1071'     'r-matrixeqtl'     'ucsc-bl  1122.7s
 => [4/4] RUN echo "source activate test_env" > ~/.bashrc                                                                                                                                                   1.5s
 => exporting to image                                                                                                                                                                                     53.4s
 => => exporting layers                                                                                                                                                                                    53.3s
 => => writing image sha256:94fbf957162e3735bcba17769deca5a2203f0ec6c7625d8f67254a085ac41531                                                                                                                0.0s
 => => naming to docker.io/library/mamba_test                                                                                                                                                               0.0s

mamba (miniforge3, more specifically) took a total of 1250 seconds to accomplish this. (Approximately 21 minutes.)

conda based build time

[+] Building 4180.1s (9/9) FINISHED
 => [internal] load build definition from dockerfile                                                                      0.4s
 => => transferring dockerfile: 875B                                                                                      0.0s
 => [internal] load .dockerignore                                                                                         0.3s
 => => transferring context: 2B                                                                                           0.0s
 => [internal] load metadata for docker.io/continuumio/miniconda3:latest                                                  0.7s
 => [auth] continuumio/miniconda3:pull token for registry-1.docker.io                                                     0.0s
 => CACHED [1/4] FROM docker.io/continuumio/miniconda3:latest@sha256:5ee0ed861bce8f67acb0a3e8a416891781db54302d93bf20a3c  0.0s
 => [2/4] RUN conda create -n test_env python=3.11 -y                                                                    30.9s
 => [3/4] RUN echo "source activate test_env" > ~/.bashrc                                                                 1.4s
 => [4/4] RUN conda install     'r-tidyverse'     'r-sessioninfo'     'r-bedr'     'r-dbplyr'     'r-rmariadb'     'r-  4073.2s
 => exporting to image                                                                                                   73.4s
 => => exporting layers                                                                                                  73.2s
 => => writing image sha256:30e994e7715109c9860dc45b40e2e57cac194c035a0239ad4b1a8246c59bf8f6                              0.1s
 => => naming to docker.io/library/conda_test                                                                             0.1s

conda (miniconda3, more specifically) took a total of 4180 seconds to accomplish this. (Approximately 70 minutes.)

conda took about 3.3 times the amount of time mamba did to build the same environment!

libsolv makes mamba shine and helps it outperform regular conda for package management.

This is why I prefer using mamba over conda for environment based package management. Especially when I’m adding stuff and rebuilding containers over and over, this extra saved time makes a big difference when it adds up.

Try mamba yourself! Remember, if you can use conda, you already know how to use mamba!

References

• mamba and micromamba installation documentation

• mamba Github repo

• miniforge Github repo

Docker is a container service that we have discussed previously in my blog posts.

The default storage location for Docker is at /var/lib/docker. Speaking from experience, as images and containers are built over a period of time, especially if there are multiple users using the Docker service, the root filesystem can run into issues where the size of the Docker storage directory can cause potential out-of-space crises, and significantly deteriorate overall system performance.

In this blog post, we will look at transferring the Storage directory to another location. (In my case, I transferred it to a physical hard drive with expanded storage capacity). It’s an essential step to ensure that Docker operations continue to run smoothly and efficiently, particularly in production environments.

The process

The storage transfer process will have the following steps:

• Stopping the docker service.

• Transferring the Docker storage files from /var/lib/docker to it’s new location, while preserving the file’s metadata.

• Renaming the old storage so that Docker will not attempt to reuse it.

• Creating a config file daemon.json at /etc/docker in order to point the Docker service to the new storage directory.

• Modifying the systemctl service file in order for the Docker service to correctly identify the Docker daemon.(NOTE: This may or may not be necessary. Explained with greater detail in it’s relevant section.)

• Reloading the daemons and restarting the Docker service.

Note that these commands need to be run with elevated (superuser) privileges.

Stopping the Docker service.

The Docker service can be stopped with the command:

systemctl stop docker

Systemctl is a Linux command used for controlling and managing system services, including starting, stopping, enabling, and disabling them, among other tasks.

Transferring the Docker storage files from /var/lib/docker to it’s new location, while preserving the file’s metadata.

Creating a new directory for the storage

mkdir /path/to/new/storage

Copying the existing storage to the new location

rsync -aqxP /var/lib/docker /path/to/new/storage

The rsync command is used to synchronize and copy files and directories from one location to another. It stands for remote sync, and is a file synchronization tool.

-aqxP:These are the options and flags used with rsync.

• -a: This option stands for “archive” and is used to perform a recursive copy while preserving file permissions, ownership, timestamps, and more.

• -q: The “quiet” option suppresses non-error messages, making the output less verbose.

• -x: This option ensures that rsync does not cross filesystem boundaries. It prevents rsync from copying data to a different filesystem.

• -P: This combines two options: -P is equivalent to -rlptgD and –partial. It enables resumable copying and shows progress during the transfer.

Renaming the old Docker storage.

mv /var/lib/docker /var/lib/docker.old

Creating a daemon.json file at /etc/docker.

This file will help point the Docker service to the new storage location.

echo '{
  "data-root": "/path/to/new/storage"
}' > /etc/docker/daemon.json

Editing the systemctl file to allow Docker to connect to the Docker Daemon

When I attempted to perform the storage transfer on my own, I originally did not have this step included. When restarting the Docker service, I got the following error whenever I attempted any Docker commands.

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

The Docker daemon was very clearly running (by running systemctl status docker), but the docker cli was unable to connect to the socket file.

This was because I had restarted the Docker daemon as a service using the systemctl command.

I found my service file path from the output of systemctl status docker.

The following is the first two lines of the output produced by this command.

● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)

Edit this file at the specified location.

The following was the content in the file on my system. Pay attention to the line that starts with ExecStart=/usr/bin/dockerd.

[Unit]
Description=Docker Application Container Engine
...

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
...

The -H fd:// means the daemon is using a file descriptor managed by systemctl, without pointing towards the socket file at /var/run/docker.sock.

Modify the line to look like this:

ExecStart=/usr/bin/dockerd -H unix:///var/run/docker.sock --containerd=/run/containerd/containerd.sock

This command can be run to do it as part of a script.

sed -i 's/^ExecStart=.*/ExecStart=\/usr\/bin\/dockerd -H unix:\/\/\/var\/run\/docker.sock --containerd=\/run\/containerd\/containerd.sock/' /lib/systemd/system/docker.service

Restarting the Daemons and the Docker service

systemctl daemon-reload
systemctl start docker

Testing!

• Run docker image ls and docker ps -a commands to see if the images and containers have been succesfully transferred, and Docker is able to see them at the new location.

• Run docker info and look for the line that starts with Docker Root Dir:. Check if it’s pointing towards the new storage location.

Deleting the old storage directory

This is a risky step in my opinion. I have the original storage directory preserved at the /var/lib folder, only having renamed it to docker.old. If you are able to run the Docker service with the new storage location without any issues ( try creating new images and containers!) then it may be a good idea to delete it. (Definitely make a backup of it regardless.)

Bonus Automation!

The following is a bash script that should automate this process for you while providing a way to check the time taken for each successful step.

#!/bin/bash
##################################################
# The purpose of this script is to transfer the Docker storage (root) directory from the root directory (/var/lib/docker) to the hard drive (/path/to/new/storage).

# The following are the steps of this process (in order):

# - Stop the docker service

# - Transfer the Docker storage files from /var/lib/docker to /path/to/new/storage.
        # (Note: This transfer will need to preserve the metadata of the original storage files for Docker to identify it.)

# - Rename the old storage directory so that Docker can no longer identify it.

# - Once the storage has been copied, a new file named daemon.json needs to be placed in /etc/docker to point at the new storage location.

# - Once the storage has been successfully transfered to the new location, the systemctl service file path will need to be modified in order to allow
        # Docker to identify the running Daemon.

# - The daemons need to be reloaded.

# - The Docker service needs to be restarted.

# Note: This script needs to be run as a sudo user.

###################################################

## Stopping the docker service

systemctl stop docker

echo "Docker service stopped at $(date)" >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt


## Transferring the Docker storage files from /var/lib/docker to /path/to/new/storage.

## Creating a new file at /path/to/new/storage

mkdir /path/to/new/storage

echo "New storage directory created at /path/to/new/storage . Beginning cache transfer at $(date)." >> docker_cache_transfer_progress.txt

echo "." >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt

## Copying the Docker storage files

rsync -aqxP /var/lib/docker /path/to/new/storage

echo "Docker storage cloned to /path/to/new/storage at $(date)." >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt

## Renaming the old Docker storage

mv /var/lib/docker /var/lib/docker.old

echo "Old Docker storage renamed to /var/lib/docker.old at $(date)." >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt

## Creating a daemon.json file at /etc/docker

echo '{
  "data-root": "/path/to/new/storage"
}' > /etc/docker/daemon.json

echo "daemon.json file created at /etc/docker/daemon.json at $(date) with the following content:" >> docker_cache_transfer_progress.txt

cat /etc/docker/daemon.json >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt

## Editing the systemctl file
# The following command will replace the line beginning with ExecStart with the appropriate line needed to actually have the systemctl comman connect to the Docker daemon.

sed -i 's/^ExecStart=.*/ExecStart=\/usr\/bin\/dockerd -H unix:\/\/\/var\/run\/docker.sock --containerd=\/run\/containerd\/containerd.sock/' /lib/systemd/system/docker.servi$

echo "systemctl config file modified at $(date)." >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt

## Restarting the daemons

systemctl daemon-reload

echo "daemons reloaded at $(date)" >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt
echo "." >> docker_cache_transfer_progress.txt

## Restarting the Docker service

systemctl start docker

echo "Docker service started at $(date)" >> docker_cache_transfer_progress.txt