All Webhooks stopped working / Let's Encrypt Root CA expiry


  • Used Zammad version: 4.1.0-1632999524.2d4bb375.stretch
  • Used Zammad installation type: package
  • Operating system: Debian stretch
  • Browser + version: irrelevant

Expected behavior:

  • Webhooks work

Actual behavior:

Webhooks stopped working yesterday around 2pm UTC. This does not seem like a coincidence, it happened exactly at the time of the Let’s Encrypt cross-sign CA expiry:

However, the exception logged by Zammad is not really helpful:

E, [2021-10-01T11:51:11.467431 #1663-47369744681020] ERROR -- : Retrying TriggerWebhookJob in #<Proc:0x0000562a40dc88e8@/opt/zammad/app/jobs/trigger_webhook_job.rb:16 (lambda)> seconds, due to a TriggerWebhookJob::RequestError. The original exception was nil.

Also, there’s no porblem accessing our webhook URL from the same server by hand using cURL or even Ruby:

root@host:~# zammad run rails c
irb(main):015:0> require 'open-uri'
=> false
irb(main):016:0> open('https://…/').read

To debug this further – how does Zammad do webhook web requests? What SSL library and trust store is in use?

Zammad uses your systems openssl dependencies and CAs.
This means that restarting the Zammad service might help already if it works in a fresh rails console.