Selfhosting and your Homeserver - E03 - DYI CD Pipeline

This is the third episode in this mini-series. Before reading this article it makes sense, to have read the first and second episode.

The previous articles gave an in-depth overview of my homeserver configuration and the applications hosted on it. This article aims to concentrate on my approach to establishing a CD-Pipeline for the server.

What is a CD-Pipeline? #

A CD pipeline (continuous delivery) is usually used together with a CI pipeline (continuous integration). The former allows for automated testing of software tests like unit, integration, and end-to-end tests on code changes, PRs, or scheduled events. We will concentrate on the latter today, which is utilized to enable scripted updates to newer versions of software deployments.
For the CD pipeline, the same triggers as for CI may apply, with the addition of manually starting the pipeline.

Requirements and Choices #

Exemplary we will focus on two deployments of mine. The first one being my website and the second one being this blog. The website can be deployed to a static webserver, while the blog needs to be built from the markdown sources to a static page.

I chose to use Docker as my preferred platform due to my focus on containers. Additionally, as I am utilising GitHub as my Version Control System (VCS), I have opted to utilise GitHub Actions for my CI/CD Environment. This may be a suitable choice for your pipeline too, as they offer a satisfactory free option to begin with. Similar options are available with other services, such as GitLab.

To ensure maximum security in the context of the deployment process, I chose to maintain a private server and refrain from connecting it directly to GitHub or exposing any additional ports or services.

For the website, I opted to activate the CD pipeline and update the content based on the releases set in GitHub's web interface. This approach makes sense to me as I do not want every modification in the repository to immediately reflect on the website. It aligns more with the conventional release schedule often employed by agile software projects. As for the blog, I chose the opposite strategy - the repository should be directly synchronized with the live version at all times. As a result, any modifications to the source code, such as a commit, will trigger the CD pipeline.

Solution #

The solution itself is pretty straightforward. First, we will review the solution for my blog, which is the easier option. I have illustrated the process below, let us examine it step by step.

CD Pipeline with Commit Trigger
  1. User publishes Source Code to GitHub
    • Action: User interacts with GitHub to push source code.
  2. Commit triggers the build of the CD Pipeline
    • Action: The GitHub release event triggers a Continuous Deployment (CD) pipeline.
  3. CD Pipeline builds a Docker image and publishes it to the GitHub container registry
    • Actions:
      • Build Docker image.
      • Publish the Docker image to GitHub Container Registry.
  4. Agent on the Server periodically checks for updates to the registry
    • Action: The server periodically polls the GitHub Container Registry for new images.
  5. Agent finds a new image and triggers redeployment
    • Actions:
      • Detect a new Docker image in the registry.
      • Trigger the deployment process.
  6. Docker Daemon stops the running container and replaces it with the new revision
    • Actions:
      • Stop the currently running container.
      • Deploy and start the new Docker image as a replacement.

This is a rather simple way of solving the problem - keeping in mind the KISS-philosophy. The usage of Containers simplifies the process drastically, in combination with free registries broadly available. The alternative of building the source at the host or dealing with binaries or release packages is much more complicated. The same can be said for the easy to use declarative syntax of GitHub Actions Pipelines in contrast to automation systems like Jenkins.

The Server-side update of the Deployment is completely asynchronous and in passiv communication with the pipeline-server / VCS. Alternatives would be to directly replace the deployment via remote management of the server inside the pipeline or utilize some form of Webhook to instruct the server to do so. Instead, the passive variant with Watchtower is utilized as the Update Agent on the Server. This keeps the process simple and uncluttered, whilst being slow in comparison.

Let us first examine the Configuration on the GitHub side and afterward have a look at the server.

GitHub Implementation #

The following File, checked into the Repository of this blog is used to implement the complete GitHub-side of the CD Process.

name: CD

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${ { github.repository }}

on:
  push:
    branches: [ "main" ]

jobs:
  build:
    permissions:
    contents: read
    packages: write
    runs-on: ubuntu-latest
    steps:
      - name: checkout repository
      uses: actions/checkout@v2
      - name: setup node
        uses: actions/setup-node@v3
        with:
          node-version: 18.x
          cache: 'npm'
      - name: install dependencies    
        run: npm ci
      - name: build static site        
        run: npm run build 
      - name: login to github container registry
        uses: docker/login-action@v1
        with:
          registry: ${ { env.REGISTRY }}
          username: ${ { github.actor }}
          password: ${ { secrets.GITHUB_TOKEN }}
      - name: tag image with version release
        id: meta
        uses: docker/metadata-action@v1
        with:
          images: $/$
      - name: Build and push Docker image
        uses: docker/build-push-action@v2
        with:
          context: .
          push: true
          tags: ${ { steps.meta.outputs.tags }}
          labels: ${ { steps.meta.outputs.labels }}

Server Implementation #

Watchtower is used as a watchdog on the server side, checking for updates to containers and restarting deployments with new versions if available, every hour.

$ docker ps | grep 'watchtower\|jeujeus'
e3810977e168   ghcr.io/jeujeus/homepage:latest                        "/docker-entrypoint.…"    26 hours ago    Up 26 hours                     0.0.0.0:8080->80/tcp, :::8080->80/tcp   homepage
018c880a93ef   ghcr.io/jeujeus/blog:main                              "/docker-entrypoint.…"    3 days ago      Up 3 days                       0.0.0.0:4312->80/tcp, :::4312->80/tcp   blog
cda930d84c51   containrrr/watchtower:latest                           "/watchtower"             2 weeks ago     Up 6 days (healthy)                                                     watchtower

Controlled Release Based Trigger #

Adding to the previous illustration of the CD-process, the process for my website adds the necessity for a release to be created on GitHub in order to trigger the CD-pipeline.

CD Pipeline with Release Trigger