Simon Krenger

Missing X-Forwarded-For header in Spring Boot application

, by Simon Krenger

So here is another one from the trenches.

More than once one of our OpenShift Container Platform customers approached us and said something along the lines of: “Help, I cannot see the X-Forwarded-For header in my application, our OpenShift Router is probably configured incorrectly!”.

In such cases, it is often a good idea to check what is really being forwarded to the Pods in the cluster. For this, I typically use my simonkrenger/echoenv container to print the headers received by the application. In many cases, it turns out that the application affected is a Spring Boot application and the header is passed correctly to the Pod itself. But the Spring Boot application does not show the header anyway.

We have observed a behaviour of Spring Boot that leads to the X-Forwarded-For header not being passed to the application, as it is consumed by Spring Boot. In the application.properties of a Spring Boot application, the following setting controls this:

server.use-forward-headers: true

This configuration leads to the header being consumed by Spring Boot and the header not being available in the application. See also the relevant sections in Spring documentation. Good to know.

Exploring the OpenShift etcd with etcdctl

, by Simon Krenger

Kubernetes uses etcd as the persistent store for API data. As etcd is a distributed key-value store, we can also use command line tools to query this store. The examples in this post are for OpenShift 3.x.

Apart from just using get, there is also the possibility to perform the following actions on certain keys:

  • put to write to a key – unless you know what you are doing, don’t touch the Kubernetes data in etcd, as this will manifest in very strange Kubernetes behaviour.
  • del to delete a key – also, this may break your Kubernetes cluster by introducing inconsistencies.
  • watch to keep a watch on an object. This is very helpful to track changes on a certain object.

The get action is probably the most helpful functionality for in-depth API debugging directly within etcd.

Read the rest of this entry »

Red Hat Certified Engineer

, by Simon Krenger

Working for Red Hat certainly has its perks. One of them being that I have access to all the content from Red Hat University and I am able to take Red Hat exams for free. With these perks come of course some expectations. Customers expect a Red Hat TAM to be knowledgeable on a wide range of Red Hat products, even if they are not directly related to the function of the TAM.

The most common certifications for System Administrators and also for new TAMs are the Red Hat Certified System Administrator (RHCSA) and the Red Hat Certified Engineer (RHCE). So after passing my RHCSA exam in December 2019, I passed the exam EX294V8 to become a Red Hat Certified Engineer (RHCE) in mid-February. The next step is obviously to become a Red Hat Certified Architect (RHCA), in my case focussed on Cloud technologies such as OpenShift and Containerisation.

To prepare for the RHCE, I used Red Hat University Online courses (RH294) and also prepared using Tomas Nevars Ansible Sample Exam. As others have already noted, the RHCE for RHEL8 is a pure Ansible exam, so knowing your Ansible playbooks in and out will help you with the exam. The above courses and sample exam are great preparations for the exam itself.

vim settings for YAML files

, by Simon Krenger

For editing YAML, be it for OpenShift / Kubernetes or Ansible, having your editor set up right can help to avoid common mistakes. So here is the minimalistic config in my ~/.vimrc to make working with YAML files easier. I am sure there are even more plugins or settings available, but this minimal set of commands works fine for me:

set ts=2
set sts=2
set sw=2
set expandtab

syntax on
filetype indent plugin on

set ruler
Read the rest of this entry »

Investigating slow DNS resolution in container

, by Simon Krenger

Some time ago, I had a curious case of very slow DNS resolution in a container on OpenShift. The symptoms were as follows:

  • In the PHP application in the container, DNS resolution was very slow with a 5 second delay before the lookup was resolved
  • In the container itself, DNS resolution for curl was very slow, with a 5 second timeout before the lookup was resolved
  • However, using dig in the container itself, DNS resolution was instant
  • Also, on the worker node, the DNS resolution was instant (using both dig and curl)

TL;DR: Since glibc 2.10, glibc performs IPv4 and IPv6 lookups in parallel. When IPv6 fails, there is a 5 second timeout in many cases before the lookup is returned. Disable IPv6 DNS lookups by setting “single-request” in “resolv.conf” or disable the IPv6 stack completely.

Read the rest of this entry »

Podman: “desc:bad request: add_hostfwd: slirp_add_hostfwd failed”

, by Simon Krenger

In the past few months, on all my machines I have replaced Docker with Podman and mostly the transition has been quite smooth. There are still some rough edges here and there, but the overall experience of using Podman has been great!

However, when trying to start a very simple container, one often runs into the following issue:

$ podman run -p80:80 nginx:latest 
Error: error from slirp4netns while setting up port redirection: map[desc:bad request: add_hostfwd: slirp_add_hostfwd failed]

The error message looks very cryptic, but the issue is quite simple: As a regular user, one is typically not allowed to bind ports < 1024. So by trying to bind port 80, you will get the error above.

The fix is trivial, just use a port greater than 1024:

$ podman run -p8080:80 -d nginx:latest 
22d2be2966e9cb77246a8b698f9024de89f4e6d1a0edfe44209bbe4fd27aa8b5
$ curl localhost:8080
[..]
Welcome to nginx!
[..]

If you really need to use a port number lower than 1024, there are multiple ways to configure that:

  • Set net.ipv4.ip_unprivileged_port_start=80or similar in your sysctl
  • Add the CAP_NET_BIND_SERVICE capability to your process or user