Simon Krenger

I recently bought an NVIDIA Jetson Nano Developer Kit to fiddle around with things like MicroShift or TensorFlow. The board is typically used with L4T (Linux for Tegra) based on Ubuntu 18.04. Fedora can also be installed, although not all drivers (for example for the GPU) are available yet. So after properly updating the system with the latest packages, when starting a container using the nvidia runtime, I got the following error:

docker run -it --rm --runtime nvidia --network host nvcr.io/nvidia/l4t-ml:r32.6.1-py3
[..]
docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall clone3: permission denied: unknown.

As you may know, Docker Desktop on macOS runs a Linux VM in the background to run containers on macOS (since containers are a Linux concept). However, that VM is well hidden from view and you typically only interact with it when you start Docker Desktop or when you need to clean up images in the VM itself.

Sometimes you’ll want to have a shell into that VM, but that turns out to be more complicated than I initially expected. There is however an easily accessible debug shell available.

• First, open a terminal and use socat to open the debug shell socket to the VM using the following command:

$ socat -d -d ~/Library/Containers/com.docker.docker/Data/debug-shell.sock pty,rawer

• socat will print the line “PTY is /dev/ttys010“, to which you can then connect to using screen on another terminal window:

$ screen /dev/ttys0xx

So that will look something like this:

$ socat -d -d ~/Library/Containers/com.docker.docker/Data/debug-shell.sock pty,rawer
2021/01/02 21:28:43 socat[23508] N opening connection to LEN=73 AF=1 "/Users/simon/Library/Containers/com.docker.docker/Data/debug-shell.sock"
2021/01/02 21:28:43 socat[23508] N successfully connected from local address LEN=16 AF=1 ""
2021/01/02 21:28:43 socat[23508] N successfully connected via
2021/01/02 21:28:43 socat[23508] N PTY is /dev/ttys010
2021/01/02 21:28:43 socat[23508] N starting data transfer loop with FDs [5,5] and [6,6]

$ screen /dev/ttys010
/ #
/ # uname -a
Linux docker-desktop 4.19.121-linuxkit #1 SMP Tue Dec 1 17:50:32 UTC 2020 x86_64 Linux

The VM is a very stripped down Alpine image with no package manager available, so you’ll have to make do with what is available.

Quit with CTRL-D, which will also close the socat socket. Thanks to Tatsushi for figuring it out in this GitHub Gist.

In the past few months, on all my machines I have replaced Docker with Podman and mostly the transition has been quite smooth. There are still some rough edges here and there, but the overall experience of using Podman has been great!

However, when trying to start a very simple container, one often runs into the following issue:

$ podman run -p80:80 nginx:latest 
Error: error from slirp4netns while setting up port redirection: map[desc:bad request: add_hostfwd: slirp_add_hostfwd failed]

The error message looks very cryptic, but the issue is quite simple: As a regular user, one is typically not allowed to bind ports < 1024. So by trying to bind port 80, you will get the error above.

The fix is trivial, just use a port greater than 1024:

$ podman run -p8080:80 -d nginx:latest 
22d2be2966e9cb77246a8b698f9024de89f4e6d1a0edfe44209bbe4fd27aa8b5
$ curl localhost:8080
[..]
Welcome to nginx!
[..]

If you really need to use a port number lower than 1024, there are multiple ways to configure that:

• Set net.ipv4.ip_unprivileged_port_start=80or similar in your sysctl
• Add the CAP_NET_BIND_SERVICE capability to your process or user

At their core, containers are just Linux processes that are namespaced. This means in practice, many containers still run as processes on the same host machine. While namespacing processes using cgroups creates very good boundaries between processes, the isolation is still not perfect.

So when using NodeSelectors in OpenShift, you’ll also have to set labels on your nodes. You can find more information on labeling nodes in the OpenShift documentation. Here is how you can add or remove a label from a node or pod:

To add a label to a node or pod:

# oc label node node001.krenger.ch mylabel=myvalue
# oc label pod mypod-34-g0f7k mylabel=myvalue

To remove a label (in the example “mylabel”) from a node or pod:

# oc label node node001.krenger.ch mylabel-
# oc label pod mypod-34-g0f7k mylabel-

You can also use oc label -h to see more options for the oc label command.

So in any larger container orchestrator installation, be it Kubernetes or OpenShift, you will encounter pods that crash regularly and enter the “CrashLoopBackOff” status.

$ oc get pod --all-namespaces 
NAMESPACE         NAME                       READY     STATUS             RESTARTS   AGE
[..]
my-project-1      helloworld-11-9w3ud        1/1       Running            0          7h
my-project-2      myapp-simon-43-7macd       0/1       CrashLoopBackOff   3774       9h

Note the container that has status “CrashLoopBackOff” and 3774 restarts.