Removing X-Forward-For-setting in the nginx config leads to an "unhandled low level error"

  • Used Zammad version: 6.4.0 / 1592470089.65376a81.centos7
  • Used Zammad installation source: (source, package, …) CENTOS package
  • Operating system: centos-release-7-8.2003.0.el7.centos.x86_64
  • Browser + version: Firefox, Chrome, Safari … it’s not browser dependent.

I’m running an external proxy / load balancer HAProxy in front of my Zammad installation. This proxy already adds a “X-Forwarded-For” header when if passes data to the backend, which is in this case my Zammad installation.

As the nginx config of Zammad also adds the “X-Forwarded-For”, Zammad always sees the IP address of my HAProxy (which connects to it) for all sessions instead of the real IP address which is added by my HAProxy.

I thought a proper solution is to remove the line which adds the X-Forwarded-For in the nginx of the Zammd VM, so the nginx config with commended X-Forwarded-For lines looks like that:

#
# this is the nginx config for zammad
#

upstream zammad-railsserver {
    server 127.0.0.1:3000;
}

upstream zammad-websocket {
    server 127.0.0.1:6042;
}

server {
    listen 80;

    # replace 'localhost' with your fqdn if you want to use zammad from remote
    server_name helpdesk.mydomain.de;

    # security - prevent information disclosure about server version
    server_tokens off;

    root /opt/zammad/public;

    access_log /var/log/nginx/zammad.access.log;
    error_log  /var/log/nginx/zammad.error.log;

    client_max_body_size 50M;

    location ~ ^/(assets/|robots.txt|humans.txt|favicon.ico) {
        expires max;
    }

    location /ws {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header CLIENT_IP $remote_addr;
#        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_read_timeout 86400;
        proxy_pass http://zammad-websocket;
    }

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header CLIENT_IP $remote_addr;
#        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_read_timeout 300;
        proxy_pass http://zammad-railsserver;

        gzip on;
        gzip_types text/plain text/xml text/css image/svg+xml application/javascript application/x-javascript application/json application/xml;
        gzip_proxied any;
    }
}

From theory nginx should now not change my X-Forwarded-For which is added by the HAProxy and leave the original X-Forwarded-For as it is. Unfortunately as soon as I comment out these two lines, Zammad does not work anymore and I just see a single text error line:

An unhandled lowlevel error occurred. The application logs may have details.

I’m pretty puzzled why that happens. How can I make Zammad not adding it’s own X-Forwarded-For and use the X-Forwarded-For which is added by my HAProxy in front of it to get the real session IP addresses?

Please have a look at my rant post from ages ago:

Your post is missing valuable information.

Sorry, I’ve edited my post and added the required information :wink:

Zammad 6.4…?
We’re talking about a vanilla Zammad installation from zammad.org …?
Sounds rather odd to me to be honest. Anyway.

Instead of not adding the x-forwarded-for header, you should use the header that your ha-proxy does provide.

As far as I’m aware Zammad does not set X-Forwarded-For headers.
I did a quick search on the source code and from what I can tell we act passively in this context.

The error message you receive sounds interesting, I think your logfiles should help for the place this issue comes from.

Start from your first proxy and go down to the second.

Lastly you can check Zamamds production log, but I doubt you’ll find usefull information.