Docker to the rescue - the tale of the broken NuGet

On the morning of Wednesday, January 27, at 12:00 PM UTC, an incident occurred that caused NuGet package restore operations to fail in some Debian Family Linux environments.

These words, right as I was preparing my laptop, running Pop!_OS 20.10, for an on-site deployment of a dotnet core 3.1 application for a client, made my heart sink. Only a few short hours away from the planned deployment, I would be unable to build the application, as I had been working on my desktop computer up to this point, and had only just pulled the project to my laptop for the deployment itself.

I first noticed something was wrong after running dotnet restore, in order to fetch build dependencies, only to be greeted by a list of errors followed up by;

The repository countersignature’s timestamp found a chain building issue: UntrustedRoot: self signed certificate in certificate chain

Clearly, something somewhere had gone out of sync. Seeing as I don’t regularly use my laptop, this wasn’t entirely unusual and I had encountered similar certificate expiry errors before. However, unlike previous cases, a simple sudo apt update && sudo apt upgrade -y didn’t resolve the errors I was seeing. Updating and upgrading packages should resolve certificate errors on Pop!_OS, as Debian’s ca-certificates is just another package in the package manager, containing root certificates of trusted organizations. After a bit of research, I found a NuGet error ticket for the issues I was experiencing, but the news was not good.

It sounds almost impossible that a piece of software as crucial to software development in 2021 as NuGet could stop functioning on mainstream platforms, but that’s exactly what happened last Wednesday. The incident report, posted on NuGet’s Github issue tracker, provides a detailed description of events, but to summarize, a Microsoft-owned signing certificate became distrusted by Debian and Ubuntu maintainers, due to its older, less-secure nature, but an updated version of the ca-certificates package had not been released in time for the NuGet author signing certificate to be updated to use the new certificates before its own expiry. Due to this, NuGet packages were signed with a now invalid certificate, with no way to validate these packages until repository maintainers release an updated list of trusted root certificates. The incident report lays out the following table;

Debian

Debian 11 (“bullseye”): Not Impacted ✔

Debian 10 (“buster”): Impacted/Resolved ⚠️

Debian 9 (“stretch”): Impacted/Ongoing ❌

Debian .NET Docker Images

Debian 10 Images: Impacted/Resolved ⚠️

January 27: Patched images released with a *-ca-patch-buster-slim suffix to tags

January 30: Updated images released for the standard release tags ✔

Ubuntu

Ubuntu 20.10 (Groovy Gorilla): Impacted/Ongoing ❌

Ubuntu 20.04 (Focal Fossa): Not Impacted ✔

Ubuntu 18.04 (Bionic Beaver): Not Impacted ✔

Ubuntu 20.10, the distribution which my version of Pop!_OS is based on, is, as of yet, unfixed. Under time pressure to get my laptop in shape to build my project, I considered installing one of the fixed, or unaffected versions of Linux into a virtual machine, where I would then be able to build without errors. This was the perfect use-case for Docker, a piece of software used to create “containers” for software, running in isolated environments, functioning effectively as light-weight virtual machines sharing resources with the host kernel. Rather than having to install a whole new operating system, Docker could create a low-level environment with the requires configurations to circumvent my host OS’ NuGet woes.

Microsoft host official Docker images for the dotnet sdk, which work as fully self-contained environments for publishing and running dotnet applications. In order to make use of their image, (having already installed Docker) I simply had to write a quick launch script which would create a container using their reference image, mount my project folder inside the virtualized environment, and run, or publish the application, which I had been unable to do in my host OS;

runInDocker.sh

#!/bin/bash

docker run -it \
    -v $(pwd):/app \
    mcr.microsoft.com/dotnet/sdk:3.1 \
    /bin/sh -c 'cd /app && dotnet run'

buildInDocker.sh

#!/bin/bash

docker run -it \
    -v $(pwd):/app \
    mcr.microsoft.com/dotnet/sdk:3.1 \
    /bin/sh -c 'rm -rf /app/Publish && cd /app && dotnet restore && dotnet publish -c Release --framework netcoreapp3.1 -r win-x86 --no-restore /p:PublishTrimmed=true -o /app/Publish'

In fact, using the Visual Studio Code extension, Docker, I could even work directly inside my running dotnet container, running whatever commands I need within Code’s terminal emulator, which is automatically linked into the container’s terminal. This allows debugging, building, and publishing from within the container, as if working in a native dotnet environment!

With these two scripts, as well as the ability to debug inside a functioning Docker dotnet environment if need be, I had regained all the functionality I needed to be able to build, test, and publish my app on-site. Containerization, and Docker in particular, saved the project deployment from delays and allowed me to keep doing my job - writing code, rather than worrying about inconsistencies in my development environments. I reckon I’ll keep using Docker as my primary method of building projects from now on, as the knowledge that no matter where I run a project, it will behave reliably, is a great reassurance.