CSRF token verification failed in docker-compose w/ Caddy downstream despite .env set

Infos:

  • Used Zammad version: 5.4.1-29
  • Used Zammad installation type: docker-compose
  • Operating system: Debian 11
  • Browser + version: Firefox latest

Expected behavior:

  • I can log in

Actual behavior:

  • When submitting the login form, a “CSRF token verification failed!” error is shown and login fails

Steps to reproduce the behavior:

  • Deploy Zammad with docker-compose according to the documentation
  • Configure Caddy in front of Zammad for TLS termination (one-line reverse_proxy call or commandline)
  • Access Zammad and run first-run wizard - it should succeed and log you into the admin account
  • Try logging in to Zammad in another browser or private window - this should throw the error and login fails

I have tried adding NGINX_SERVER_SCHEME=https to the .env file and rebuilding the compose env, to no avail. zammad-nginx looks to be configured correctly, passing the $scheme it gets from downstream. Caddy’s logs say that it passes X-Forwarded-Proto: https to nginx (excerpt below):

caddy[189]: {... "request":{"remote_ip":"...","remote_port":"...","proto":"HTTP/1.1","method":"GET","host":"...","uri":"/ws","headers":{"Cdn-Loop":["cloudflare"],"Upgrade":["websocket"],"Accept-Encoding":["gzip"],"Cf-Ray":["..."],"Accept-Language":["en-US,en;q=0.7,de;q=0.3"],"Origin":["https://..."],"Dnt":["1"],"Cf-Connecting-Ip":["..."],"X-Forwarded-For":["..."],"X-Forwarded-Proto":["https"],"Sec-Fetch-Mode":["websocket"], ...

I’m at a bit of a loss. I would rather not hardcode the scheme in zammad-nginx if I can help it, but the change in .env doesn’t seem to take.

I should add:

http_type setting is set to https, which I assume was done automatically by the first-run wizard. I believe that is the correct setting? But I am not sure from reading the docs what that setting should be and why.

Make sure that you’ve configured “Fully Qualified Domain Name” and “HTTP type” to HTTPS in the Zammad admin panel:

https://zammad.example.com/#settings/system

Also, try adding RAILS_TRUSTED_PROXIES environment variable with the IP of your reverse proxy.

In the example below I have traefik running on the same host, but I assume it works the same with other reverse proxies:

  zammad-nginx:
    command: ["zammad-nginx"]
    expose:
      - "8080"
    depends_on:
      - zammad-railsserver
    image: ${IMAGE_REPO}:${VERSION}
    restart: ${RESTART}
    volumes:
      - zammad-data:/opt/zammad
    # customisations to work with traefik
    environment:
      - RAILS_TRUSTED_PROXIES=['127.0.0.1', '::1', '0.0.0.0', '::0']
      - NGINX_SERVER_SCHEME=https
    labels:
      - traefik.enable=true
      - traefik.http.routers.zammad.rule=Host(`zammad.example.com`)
      #- traefik.http.services.zammad.loadbalancer.server.port=80
      #- traefik.http.services.zammad.loadbalancer.server.scheme=http

Thank you! FQDN was set correctly already. HTTP Type is already set to HTTPS. I tried setting RAILS_TRUSTED_PROXIES in .env, but it doesn’t seem to make a difference either.

.env added:

NGINX_SERVER_SCHEME=https
RAILS_TRUSTED_PROXIES=['127.0.0.1', '::1', '10.0.0.1']

(the gateway and reverse proxy is at 10.0.0.1, the container running docker compose is at 10.0.3.70)

Docker compose logs excerpt:

zammad-docker-compose-zammad-nginx-1          | 10.0.0.1 - - [24/May/2023:18:50:28 +0000] "GET / HTTP/1.1" 200 1569 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0"
zammad-docker-compose-zammad-railsserver-1    | I, [2023-05-24T18:50:43.342016#1-113320]  INFO -- : Started POST "/api/v1/signin" for 10.0.0.1 at 2023-05-24 18:50:43 +0000
...
zammad-docker-compose-zammad-railsserver-1    | I, [2023-05-24T18:50:43.368095#1-113320]  INFO -- : CSRF token verification failed
zammad-docker-compose-zammad-railsserver-1    | I, [2023-05-24T18:50:43.368389#1-113320]  INFO -- : CSRF token verification failed! (Exceptions::NotAuthorized)
zammad-docker-compose-zammad-railsserver-1    | app/controllers/application_controller/prevents_csrf.rb:36:in `verify_csrf_token'
zammad-docker-compose-zammad-railsserver-1    | app/controllers/application_controller/has_download.rb:21:in `block (4 levels) in <module:HasDownload>'
zammad-docker-compose-zammad-railsserver-1    | app/controllers/application_controller/has_download.rb:20:in `block (3 levels) in <module:HasDownload>'
zammad-docker-compose-zammad-railsserver-1    | app/controllers/application_controller/has_download.rb:19:in `block (2 levels) in <module:HasDownload>'
zammad-docker-compose-zammad-railsserver-1    | app/controllers/application_controller/handles_transitions.rb:16:in `handle_transaction'
zammad-docker-compose-zammad-railsserver-1    | I, [2023-05-24T18:50:43.369634#1-113320]  INFO -- : Completed 401 Unauthorized in 25ms (Views: 0.1ms | ActiveRecord: 20.1ms | Allocations: 2577)
zammad-docker-compose-zammad-nginx-1          | 10.0.0.1 - - [24/May/2023:18:50:43 +0000] "POST /api/v1/signin HTTP/1.1" 401 102 "https://fqdn.here/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0"

I even tried recreating the entire compose env (sans volumes) to no avail.

You put your vars in the yml file instead of the .env - but the .env is still supposed to work, right?
I might try putting the variables directly into the yml as well …

I tried setting the vars in the docker-compose.override.yml and re-running compose with both files given:

docker compose -f docker-compose.yml -f docker-compose.override.yml up -d

This correctly recreated all containers that had their envs changed.

override yml:

---
version: '3'

services:

  zammad-init:
    environment:
      - RAILS_TRUSTED_PROXIES=['127.0.0.1', '::1', '10.0.0.1']
    ports:
      - "8080:8080"

I tried with zammad-init, zammad-nginx, and zammad-railsserver, all unsuccessful. I checked with:

$ docker compose exec -it zammad-railsserver cat config/environments/production.rb | grep trusted_proxies
  config.action_dispatch.trusted_proxies = ['127.0.0.1', '::1', '10.0.0.1']

… and only when overriding zammad-init it would at least show up in the rails env config. However the problem persists unchanged.

I also tried flipping the http_type setting, but that made no observable difference either.

I also tried setting NGINX_SERVER_SCHEME=https for both zammad-init and zammad-nginx, but no luck there either.

I feel like I’m going crazy ^^ I might have to dig deeper into how exactly the docker setup is constructed (where I hoped that docker compose would make things easier…) - I must’ve missed something stupidly obvious.

I can make login work by setting http_type to http, but then all generated URLs break.

I just noticed that the generated nginx conf was still using proxy_set_header X-Forwarded-Proto $scheme; so it looks like the config generation in zammad/docker-entrypoint.sh at 6006f7816d8dc42e5f3f0b6c7b0b2bb51240fea1 · zammad/zammad · GitHub was not running correctly?