docker

Set up a registry mirror

Once the registry is running and either exposed on the local network with HTTP or via the Internet with HTTPS, you'll need to configure Docker and potentially buildx too.

You can see how we do this within a Firecracker VM, to access the registry over the local Ethernet bridge: https://github.com/self-actuated/hub-mirror/blob/master/action.yml

For the Docker daemon, edit /etc/docker/daemon.json.

{
  "insecure-registries" : ["192.168.128.1:5000" ],
  "registry-mirrors": ["http://192.168.128.1:5000"]
}
  • Give each mirror under registry-mirrors and include the URL scheme
  • If you're using HTTP, without TLS, you need to specify insecure-registries

Then make sure you reload Docker:

(
sudo systemctl daemon-reload && \
sudo systemctl restart docker
)

To try it out, run docker run -ti alpine:latest, you should see the images when you run sudo find /var/lib/registry/

Buildx is a little more complicated to configure.

Create a buildkit.toml

[registry."docker.io"]
  mirrors = ["192.168.128.1:5000"]
  http = true
  insecure = true

[registry."192.168.128.1:5000"]
  http = true
  insecure = true

You can omit http and insecure if you're using TLS and HTTPS.

Then, create a new buildx builder and tell Docker to use it:

docker buildx create --config ~/buildkitd.toml --name mirrored
docker buildx use mirrored

Finally, the buildx command will reference buildkit's configuration instead of Docker's and any base images will be pulled through the mirror.

docker buildx build -f Dockerfile .

We have a custom GitHub Action that makes all of the above just one line:

jobs:
    build:
        runs-on: actuated
        steps:

        - uses: self-actuated/hub-mirror@master

        - name: Pull image using cache
            run: |
            docker pull alpine:latest

TLS is better

We used HTTP for the registry as it's accessed over a kind of loopback device between the VM and the server, however I'd recommend always using TLS where you can.

Perhaps you could even setup your registry on the Internet and use free Let's Encrypt certificates. Caddy or Nginx are simple enough to configure for that.

Then, if you're worried about bandwidth charges - Linode, DigitalOcean and Hetzner all have generous amounts included with 5-10 USD / mo VMs.

And you could also set up an IP allow list, so only your servers or build machines can consume your bandwidth allowance.

Setting up multiple mirrors

ou may want multiple mirrors if you pull images from both docker.io and another registry like gcr.io, ecr.io, ghcr.io or quay.io.

The Docker documentation says that dockerd itself can only support a mirror of the Docker Hub itself. And any information that I found about multiple mirrors only applied to Kubernetes or to buildx.

Each registry mirror needs to run on its own HTTP port and if you're using TLS, will require its own distinct TLS certificate.

For instance, here are the things to change for a second registry mirroring ghcr.io:

storage:
  filesystem:
-    rootdirectory: /var/lib/registry
+    rootdirectory: /var/lib/registry-ghcr

proxy:
-  remoteurl: https://registry-1.docker.io
+  remoteurl: https://ghcr.io
-  username: $USERNAME

http:
-  addr: 192.168.128.1:5000
+  addr: 192.168.128.1:5001

So then, buildx or cri (when using Kubernetes) need to be configured to pull from either of these endpoints.

  • 192.168.128.1:5000 mirrors docker.io
  • 192.168.128.1:5001 mirrors ghcr.io

dockerd itself, can have two mirrors defined, but in my experience it was unable to pull from the mirror for ghcr.io.

So let's look at buildx:

[registry."docker.io"]
  mirrors = ["192.168.128.1:5000"]
  http = true
  insecure = true

[registry."192.168.128.1:5000"]
  http = true
  insecure = true

[registry."ghcr.io"]
  mirrors = ["192.168.128.1:5001"]
  http = true
  insecure = true

[registry."192.168.128.1:5001"]
  http = true
  insecure = true

There's two ways to know if the cache is being used:

  • Check the filesystem for the path set under rootdirectory
  • Enable the access logs for the registry itself

To enable access logs change

log:
  accesslog:
-    disabled: true
+    disabled: false   
-  level: warn
+  level: debug
  formatter: text

In my testing, after running buildx create and buildx use, I then needed a Dockerfile that used both the Docker Hub and GHCR:

FROM alpine:3.17 as alpine
FROM ghcr.io/openfaasltd/figlet as figlet

RUN echo -n "Mirror" | figlet

Running the build with docker buildx build -t mirror-test . gave me access logs on both registries and files under the respective /var/lib/ folders.

For Kubernetes configuration, you need to update the CRI plugin in containerd's toml file: Configure Image Registry.

Beware that CRI is an abstraction layer that sits between containerd and the kubelet, configuring this will not affect buildx, containerd or dockerd.

camunda

docker pull camunda/camunda-bpm-platform:latest
docker run -d --name camunda -p 8080:8080 camunda/camunda-bpm-platform:latest
# open browser with url: http://localhost:8080/camunda-welcome/index.html