CSRF token verification failed in docker-compose w/ Caddy downstream despite .env set

[Teal]

Infos:

• Used Zammad version: 5.4.1-29
• Used Zammad installation type: docker-compose
• Operating system: Debian 11
• Browser + version: Firefox latest

Expected behavior:

• I can log in

Actual behavior:

• When submitting the login form, a “CSRF token verification failed!” error is shown and login fails

Steps to reproduce the behavior:

• Deploy Zammad with docker-compose according to the documentation
• Configure Caddy in front of Zammad for TLS termination (one-line reverse_proxy call or commandline)
• Access Zammad and run first-run wizard - it should succeed and log you into the admin account
• Try logging in to Zammad in another browser or private window - this should throw the error and login fails

I have tried adding NGINX_SERVER_SCHEME=https to the .env file and rebuilding the compose env, to no avail. zammad-nginx looks to be configured correctly, passing the $scheme it gets from downstream. Caddy’s logs say that it passes X-Forwarded-Proto: https to nginx (excerpt below):

caddy[189]: {... "request":{"remote_ip":"...","remote_port":"...","proto":"HTTP/1.1","method":"GET","host":"...","uri":"/ws","headers":{"Cdn-Loop":["cloudflare"],"Upgrade":["websocket"],"Accept-Encoding":["gzip"],"Cf-Ray":["..."],"Accept-Language":["en-US,en;q=0.7,de;q=0.3"],"Origin":["https://..."],"Dnt":["1"],"Cf-Connecting-Ip":["..."],"X-Forwarded-For":["..."],"X-Forwarded-Proto":["https"],"Sec-Fetch-Mode":["websocket"], ...

I’m at a bit of a loss. I would rather not hardcode the scheme in zammad-nginx if I can help it, but the change in .env doesn’t seem to take.

[Teal]

I should add:

http_type setting is set to https, which I assume was done automatically by the first-run wizard. I believe that is the correct setting? But I am not sure from reading the docs what that setting should be and why.

[spats]

Make sure that you’ve configured “Fully Qualified Domain Name” and “HTTP type” to HTTPS in the Zammad admin panel:

https://zammad.example.com/#settings/system

Also, try adding RAILS_TRUSTED_PROXIES environment variable with the IP of your reverse proxy.

In the example below I have traefik running on the same host, but I assume it works the same with other reverse proxies:

  zammad-nginx:
    command: ["zammad-nginx"]
    expose:
      - "8080"
    depends_on:
      - zammad-railsserver
    image: ${IMAGE_REPO}:${VERSION}
    restart: ${RESTART}
    volumes:
      - zammad-data:/opt/zammad
    # customisations to work with traefik
    environment:
      - RAILS_TRUSTED_PROXIES=['127.0.0.1', '::1', '0.0.0.0', '::0']
      - NGINX_SERVER_SCHEME=https
    labels:
      - traefik.enable=true
      - traefik.http.routers.zammad.rule=Host(`zammad.example.com`)
      #- traefik.http.services.zammad.loadbalancer.server.port=80
      #- traefik.http.services.zammad.loadbalancer.server.scheme=http

[Teal]

Thank you! FQDN was set correctly already. HTTP Type is already set to HTTPS. I tried setting RAILS_TRUSTED_PROXIES in .env, but it doesn’t seem to make a difference either.

.env added:

NGINX_SERVER_SCHEME=https
RAILS_TRUSTED_PROXIES=['127.0.0.1', '::1', '10.0.0.1']

(the gateway and reverse proxy is at 10.0.0.1, the container running docker compose is at 10.0.3.70)

Docker compose logs excerpt:

zammad-docker-compose-zammad-nginx-1          | 10.0.0.1 - - [24/May/2023:18:50:28 +0000] "GET / HTTP/1.1" 200 1569 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0"
zammad-docker-compose-zammad-railsserver-1    | I, [2023-05-24T18:50:43.342016#1-113320]  INFO -- : Started POST "/api/v1/signin" for 10.0.0.1 at 2023-05-24 18:50:43 +0000
...
zammad-docker-compose-zammad-railsserver-1    | I, [2023-05-24T18:50:43.368095#1-113320]  INFO -- : CSRF token verification failed
zammad-docker-compose-zammad-railsserver-1    | I, [2023-05-24T18:50:43.368389#1-113320]  INFO -- : CSRF token verification failed! (Exceptions::NotAuthorized)
zammad-docker-compose-zammad-railsserver-1    | app/controllers/application_controller/prevents_csrf.rb:36:in `verify_csrf_token'
zammad-docker-compose-zammad-railsserver-1    | app/controllers/application_controller/has_download.rb:21:in `block (4 levels) in <module:HasDownload>'
zammad-docker-compose-zammad-railsserver-1    | app/controllers/application_controller/has_download.rb:20:in `block (3 levels) in <module:HasDownload>'
zammad-docker-compose-zammad-railsserver-1    | app/controllers/application_controller/has_download.rb:19:in `block (2 levels) in <module:HasDownload>'
zammad-docker-compose-zammad-railsserver-1    | app/controllers/application_controller/handles_transitions.rb:16:in `handle_transaction'
zammad-docker-compose-zammad-railsserver-1    | I, [2023-05-24T18:50:43.369634#1-113320]  INFO -- : Completed 401 Unauthorized in 25ms (Views: 0.1ms | ActiveRecord: 20.1ms | Allocations: 2577)
zammad-docker-compose-zammad-nginx-1          | 10.0.0.1 - - [24/May/2023:18:50:43 +0000] "POST /api/v1/signin HTTP/1.1" 401 102 "https://fqdn.here/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0"

I even tried recreating the entire compose env (sans volumes) to no avail.

You put your vars in the yml file instead of the .env - but the .env is still supposed to work, right?
I might try putting the variables directly into the yml as well …

[Teal]

I tried setting the vars in the docker-compose.override.yml and re-running compose with both files given:

docker compose -f docker-compose.yml -f docker-compose.override.yml up -d

This correctly recreated all containers that had their envs changed.

override yml:

---
version: '3'

services:

  zammad-init:
    environment:
      - RAILS_TRUSTED_PROXIES=['127.0.0.1', '::1', '10.0.0.1']
    ports:
      - "8080:8080"

I tried with zammad-init, zammad-nginx, and zammad-railsserver, all unsuccessful. I checked with:

$ docker compose exec -it zammad-railsserver cat config/environments/production.rb | grep trusted_proxies
  config.action_dispatch.trusted_proxies = ['127.0.0.1', '::1', '10.0.0.1']

… and only when overriding zammad-init it would at least show up in the rails env config. However the problem persists unchanged.

I also tried flipping the http_type setting, but that made no observable difference either.

I also tried setting NGINX_SERVER_SCHEME=https for both zammad-init and zammad-nginx, but no luck there either.

I feel like I’m going crazy ^^ I might have to dig deeper into how exactly the docker setup is constructed (where I hoped that docker compose would make things easier…) - I must’ve missed something stupidly obvious.

[Teal]

I can make login work by setting http_type to http, but then all generated URLs break.

[Teal]

I just noticed that the generated nginx conf was still using proxy_set_header X-Forwarded-Proto $scheme; so it looks like the config generation in zammad/docker-entrypoint.sh at 6006f7816d8dc42e5f3f0b6c7b0b2bb51240fea1 · zammad/zammad · GitHub was not running correctly?

[Teal]

It works when setting the setting in docker-compose.override.yml:

$ grep SCHEME docker-compose.override.yml 
      - NGINX_SERVER_SCHEME=override
$ docker compose --file docker-compose.yml --file docker-compose.override.yml up -d
$ docker compose exec zammad-nginx grep Proto /etc/nginx/sites-available/default
        proxy_set_header X-Forwarded-Proto override;
        proxy_set_header X-Forwarded-Proto override;

I guess since the default docker-compose.yml file does not have a reference to NGINX_SERVER_SCHEME, the .env value does not override it. Ultimately this feels like it comes down to a documentation issue. I run a Caddy reverse_proxy on another server and the documentation regarding proxying Zammad in Docker did not appear to apply to that situation.

cf. docker-entrypoint for zammad-nginx fails to adapt nginx config properly · Issue #346 · zammad/zammad-docker-compose · GitHub and Using own webserver/reverseproxy · Issue #345 · zammad/zammad-docker-compose · GitHub