Simon Krenger

Back in May 2026, Red Hat released Red Hat Hardened Images, which are distroless images for various languages and runtimes. The repository used to build these images is https://gitlab.com/redhat/hummingbird/containers and I think these are great base images if you are looking for a more minimal or hardened setup.

Given the nature of these distroless images, there is obviously no shell included in the container image, which makes debugging harder. This means when trying to debug such an image using oc rsh or similar, this will fail with an error message:

$ oc create deployment apache --image=registry.access.redhat.com/hi/httpd:latest
$ oc get po
NAME                      READY   STATUS    RESTARTS   AGE
apache-6bc857d844-qmxcr   1/1     Running   0          65s

$ oc rsh apache-6bc857d844-qmxcr 
2026-07-08T19:27:02.257947Z: executable file `/bin/sh` not found: No such file or directory
command terminated with exit code 1

So to actually debug such an image, we need to use kubectl debug to start an ephemeral container which has all the tools but uses the same process namespace:

$ kubectl debug -it pod/apache-6bc857d844-qmxcr --image=registry.fedoraproject.org/fedora-minimal:44 --target=httpd -- /bin/bash
Targeting container "httpd". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
[..]
Defaulting debug container name to debugger-747b5.
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
bash-5.3$ 
bash-5.3$ ls -l /proc
total 0
dr-xr-xr-x.  9 1000760000 root    0 Jul  8 15:49 1
dr-xr-xr-x.  9 1000760000 root    0 Jul  8 15:49 113
dr-xr-xr-x.  9 1000760000 root    0 Jul  8 15:49 114
dr-xr-xr-x.  9 1000760000 root    0 Jul  8 15:49 2
dr-xr-xr-x.  9 1000760000 root    0 Jul  8 15:49 3
dr-xr-xr-x.  9 1000760000 root    0 Jul  8 15:49 33
dr-xr-xr-x.  9 1000760000 root    0 Jul  8 15:49 9
[..]
bash-5.3$ ls -l /proc/1/root/usr/share/httpd/
total 16
drwxr-xr-x. 3 root root 4096 Jun 13 00:53 error
drwxr-xr-x. 3 root root 8192 Jun 13 00:53 icons
drwxr-xr-x. 1 root root   24 Jul  8 06:54 noindex
drwxr-xr-x. 2 root root   67 Jun 13 00:53 server-status

As you can see from the output above, we need to use kubectl debug , specify the debug image to be used (fedora-minimal in my case) and then target the right container. This command will spawn an ephemeral container in the Pod and will run the specified container image in the same process namespace as the targeted container.

In the output above you can see the PIDs of the original container. Now you can use all the tools of your debug image to troubleshoot your application and you can access the original containers root filesystem via the path /proc/1/root/ if that is necessary.

As a side-note, the native oc debug pod/... defaults to spinning up a copy of the Pod with the entrypoint overridden to /bin/sh. For a distroless container image, this approach still fails because there is no shell inside the image to fall back on, which is why we have to use kubectl debug.