A best practice in building container images is to add as little dependencies as possible into the image.
Most application don’t need a shell to run, hence dropping the shell from the container image reduces the size
of the image and reduces the attack surface. Popular base images for that use case is either not using a base image at all
and build the image FROM SCRATCH or use distroless. The downside is that we can’t open a shell inside the running container
for debugging.
Kubernetes introduces ephemeral containers
to spawn a debugging container into a running pod.
To debug build issues or examine a container image before bringing it in production we have to have a look at the content
of the filesystem in the image. In a container image with a shell installed we can create a container instance and launch a shell as entrypoint.
This is not possible in images based on SCRATCH or distroless.
The container engine podman allows us to mount a container image into the filesystem of the developer machine.
This allows us to examine the filesystem content of the container image with the tools installed on the developers machine.
Mounting the image
To demonstrate this feature of podman we want to examine the current distroless base image .
Pull that image:
1
2
3
4
5
6
7
8
9
|
christoph@thinkpad-sport % podman image pull gcr.io/distroless/base-debian10
Trying to pull gcr.io/distroless/base-debian10:latest...
Getting image source signatures
Copying blob 1cfcc1bb8434 done
Copying blob d94d38b8f0e6 done
Copying config af31651e48 done
Writing manifest to image destination
Storing signatures
af31651e48fe5f4a0788ad9812ba41315d3f9ef41fd5665313c619fdd20e0f6e
|
The next step is to create a new user namespace. This allows us to freely explore the container image as root user.
1
2
3
4
|
christoph@thinkpad-sport % podman unshare
root@thinkpad-sport # id
uid=0(root) gid=0(root) groups=0(root),65534(nobody)
|
After this we can mount a container image into the user namespace.
1
2
|
root@thinkpad-sport # podman image mount gcr.io/distroless/base-debian10
/home/christoph/.local/share/containers/storage/vfs/dir/021295fa05fd0d2ac6b0eabbbc525bcfd6e2e1959f6ac5da80f97dabf9dc0fd1
|
The command returns the mount point where the image is mounted.
Let examine the glibc version installed in the distroless image:
1
2
3
4
5
6
7
8
9
|
root@thinkpad-sport # cd /home/christoph/.local/share/containers/storage/vfs/dir/021295fa05fd0d2ac6b0eabbbc525bcfd6e2e1959f6ac5da80f97dabf9dc0fd1
root@thinkpad-sport # ls
bin boot dev etc home lib lib64 proc root run sbin sys tmp usr var
root@thinkpad-sport # ls lib/x86_64-linux-gnu/
ld-2.28.so libanl-2.28.so libc.so.6 libmvec.so.1 libnss_compat-2.28.so libnss_files.so.2 libnss_nis-2.28.so libresolv-2.28.so libthread_db.so.1
ld-linux-x86-64.so.2 libanl.so.1 libdl-2.28.so libm-2.28.so libnss_compat.so.2 libnss_hesiod-2.28.so libnss_nis.so.2 libresolv.so.2 libutil-2.28.so
libBrokenLocale-2.28.so libcrypt-2.28.so libdl.so.2 libm.so.6 libnss_dns-2.28.so libnss_hesiod.so.2 libpcprofile.so librt-2.28.so libutil.so.1
libBrokenLocale.so.1 libcrypt.so.1 libmemusage.so libnsl-2.28.so libnss_dns.so.2 libnss_nisplus-2.28.so libpthread-2.28.so librt.so.1
libSegFault.so libc-2.28.so libmvec-2.28.so libnsl.so.1 libnss_files-2.28.so libnss_nisplus.so.2 libpthread.so.0 libthread_db-1.0.so
|
Based on the filenames we assume a glibc 2.28 installed in the image. Since we know that the image is based on debian we can
have a look at the dpkg database.
1
2
3
4
5
6
|
root@thinkpad-sport # cat -p var/lib/dpkg/status.d/libc6 | head -n 5
Package: libc6
Source: glibc
Version: 2.28-10
Architecture: amd64
Maintainer: GNU Libc Maintainers <debian-glibc@lists.debian.org>
|
So we have the concrete package version 2.28-10.
We can do the same to find the installed openssl version:
1
2
3
4
5
6
|
root@thinkpad-sport # cat var/lib/dpkg/status.d/libssl1 | head -n 5
Package: libssl1.1
Source: openssl
Version: 1.1.1d-0+deb10u4
Architecture: amd64
Maintainer: Debian OpenSSL Team <pkg-openssl-devel@lists.alioth.debian.org>
|
Cleanup
To clean up, exit the user namespace:
1
|
root@thinkpad-sport # exit
|
Last but not least delete the image:
1
2
3
|
christoph@thinkpad-sport % podman image rm gcr.io/distroless/base-debian10
Untagged: gcr.io/distroless/base-debian10:latest
Deleted: af31651e48fe5f4a0788ad9812ba41315d3f9ef41fd5665313c619fdd20e0f6e
|
Alternatives
An alternative to podman is dive. This tool let you explore the image layers in a ncurses
UI. It focuses on the filesystem attributes of files and the changes between the image layers. Hence you can see the names and
the attributes of a file but not the content.