NVIDIA provides the Linux4Tegra (L4T) distribution as an image for use with the NVIDIA Jetson Nano. However, once you upgrade the whole system, strange problems will pop up, one of which I have described here: NVIDIA Docker “permission denied: unknown.” on Jetson Nano.
When applying a popular solution described here by adding a new repository to your L4T installation, this will result in interesting error messages such as the following when trying to run L4T-ML containers:
docker run --rm --runtime nvidia -it nvcr.io/nvidia/l4t-ml:r32.7.1-py3 python3 -c "import torch" [..] libcurand.so.10: cannot open shared object file: No such file or directory
It turns out that the
nvidia.github.com repository should NOT be used for the Jetson Nano. So installing the package versions from this repository will result in the libraries not being mounted correctly in the container itself.
First, make sure you are actually using the
nvidia docker runtime and that
docker info | grep nvidia shows the runtime as well. L4T with the
nvidia runtime is configured to automatically mount certain folders into the container (this is defined in the CSV files in
/etc/nvidia-container-runtime/host-files-for-container.d/). When using an unsupported version, these files are not mounted correctly into the container, which in turn will result in error messages such as the following when trying to use PyTorch:
/usr/lib/aarch64-linux-gnu/libcudnn.so.8: file too short libcurand.so.10: cannot open shared object file: No such file or directory
To resolve the issue, remove the
nvidia.github.com repositories from your L4T installation and downgrade the
containerd packages as described here.