Artifact caching

Before I get into GitLab functions and components, I wanted to cover how I manage build and deployment artifacts, as it’s a dependency for a lot of what comes later.

I generally block internet access across my homelab. Aside from specific external APIs, everything is either pre-downloaded or accessed through a caching proxy.

I use Nexus for PyPI, apt, go etc. package pull-through caches. For binaries, however, I’ve moved a couple of times from a basic machine (called Vault - unfortunately this 15 year old machine pre-dated me starting to use Hashicorp Vault), through various technologies and now use Nexus.

The idea of this is threefold:

1. Removes hard dependencies on external services and upstream availability.
2. Makes builds more reproducible (not byte-for-byte reproducible, of course).
3. Lets me control exactly what enters the environment.

All of these artifacts are defined up-front (for binary artifacts and docker, anyway - the others are pull-through, so are dynamically downloaded/cached as needed).

For binary artifacts, I use a small python script, which:

• Defines types of artifacts - helpers for Github and other types of common hosting utilities
• Defines the tools, the versions and the architecture

The end result is that the script becomes a declarative list of everything I expect to exist.

An example for the Netbird GeoLite2 archive looks like this:

file_uploader.process_file(File("https://pkgs.netbird.io/geolocation-dbs/GeoLite2-City/download?suffix=tar.gz", "MaxMind/GeoLite2-City.tar.gz"))

Whilst an example for Hashicorp tools, using a small child class, looks like this:

    for os_, arch in [["linux", "amd64"], ["darwin", "amd64"], ["darwin", "arm64"]]:
        HashicorpReplicator("vault", versions=["1.12.2", "1.16.1", "1.16.2", "1.17.3",
                                               "1.17.11", "1.18.4", "1.19.5"], arch=arch, os_=os_)
        for file_ in replicator.get_files():
            file_uploader.process_file(file_)

This is nice because it describes what I’m trying to download fairly cleanly - it’s not best separation of concerns, but it will do.

A simple artifact storage interface means changing repository implementations is mostly plumbing rather than rewriting the replication logic (yes, this has happened several times):

class IArtifactRepository:

    def get_dest_url(self, file: File) -> str:
        ...

    def upload_file(self, file: File, temp_dir: str) -> None:
        ...

    def file_exists(self, file: File) -> bool:
        ...

But, most importantly, perhaps, is the upload logic, which, simplified, looks like:

    def download(self, file: File):
        if not self._artifact_store.file_exists(file):
            self.check_upstream_file_exists(file)
            return
        stream = self.download_file()
        self._artifact_store.upload_file(file, stream)

If the archive already exists, we still check that it exists upstream.

This is because, even if I am pre-downloading to handle upstream failures, if the artifact store disappears, a rebuild would require a re-replication. This way, whilst my builds should continue to function, I can plan to migrate away from these dependencies as needed. So, rather than having a long-term cache of these artifacts and being bitten in a failure situation where I can’t replicate from the upstream source, I should learn pretty quickly about them.