Zammad / nginx fails after server restore on Digitalocean

[mittkkoo]

Infos:

• Used Zammad version: 6.1
• Used Zammad installation type: package
• Operating system: Ubuntu 22.04.2 LTS (Jammy Jellyfish)
• Browser + version: all

Expected behavior:

• I expect Zammad and all other services to be working as before if I restore a server image

Actual behavior:

• I installed Zammad from package following the docs on a Digitalocean droplet. Everything worked well, also the let’s encrypt SSL. Then I wanted to change the hostname from duckdns.org to a new one. Before beginning I created a Snapshot of the droplet through the UI of Digitalocean. I tried to change the hostname, but didn’t manage to configure nginx, certbot and let’s encrypt correctly. Anyway, I decided to revert everything. So I restored the droplet from the Snapshot. But the site is not reachable.

Steps to reproduce the behavior:

• Install Zammad from package following the docs on an ubuntu 22.04 droplet from Digitalocean
• Configure SSL with let’s encrypt on a test domain from duckdns.org following the docs
• Create a Snapshot of the droplet
• Try to change the hostname to a new one (unsuccessfully)
• Revert all changes by restoring the droplet from the Snapshot
• See if Zammad and the other services work as before?

Here some outputs:

root@yyy-s-2vcpu-4gb-fra1-01:~# systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2023-10-30 17:20:07 UTC; 1h 24min ago
       Docs: man:nginx(8)
    Process: 1598 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)
        CPU: 21ms

Oct 30 17:20:07 zammadpackageubuntu-s-2vcpu-4gb-fra1-01 systemd[1]: Starting A high performance web server and a reverse proxy server...
Oct 30 17:20:07 yyy-s-2vcpu-4gb-fra1-01 nginx[1598]: nginx: [emerg] cannot load certificate "/etc/nginx/ssl/xxx.duckdns.org-fullchain.pem">
Oct 30 17:20:07 yyy-s-2vcpu-4gb-fra1-01 nginx[1598]: nginx: configuration file /etc/nginx/nginx.conf test failed
Oct 30 17:20:07 yyy-s-2vcpu-4gb-fra1-01 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Oct 30 17:20:07 yyy-s-2vcpu-4gb-fra1-01 systemd[1]: nginx.service: Failed with result 'exit-code'.
Oct 30 17:20:07 yyy-s-2vcpu-4gb-fra1-01 systemd[1]: Failed to start A high performance web server and a reverse proxy server.

root@yyy-s-2vcpu-4gb-fra1-01:~# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: xxx.duckdns.org
    Serial Number: 396...
    Key Type: RSA
    Domains: xxx.duckdns.org
    Expiry Date: 202q-0q-15 08:q0:0q+00:00 (VALID: 7q days)
    Certificate Path: /etc/letsencrypt/live/xxx.duckdns.org/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/xxx.duckdns.org/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

root@yyy-s-2vcpu-4gb-fra1-01:~# systemctl reload nginx
nginx.service is not active, cannot reload.

root@yyy-s-2vcpu-4gb-fra1-01:~# certbot --nginx -d xxx.duckdns.org
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Error while running nginx -c /etc/nginx/nginx.conf -t.

nginx: [emerg] cannot load certificate "/etc/nginx/ssl/xxx.duckdns.org-fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/etc/nginx/ssl/xxx.duckdns.org-fullchain.pem, r) error:10000080:BIO routines::no such file)
nginx: configuration file /etc/nginx/nginx.conf test failed

The nginx plugin is not working; there may be problems with your existing configuration.
The error was: MisconfigurationError('Error while running nginx -c /etc/nginx/nginx.conf -t.\n\nnginx: [emerg] cannot load certificate "/etc/nginx/ssl/xxx.duckdns.org-fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/etc/nginx/ssl/xxx.duckdns.org-fullchain.pem, r) error:10000080:BIO routines::no such file)\nnginx: configuration file /etc/nginx/nginx.conf test failed\n')

root@yyy-s-2vcpu-4gb-fra1-01:~# systemctl status certbot
○ certbot.service - Certbot
     Loaded: loaded (/lib/systemd/system/certbot.service; static)
     Active: inactive (dead)
TriggeredBy: ● certbot.timer
       Docs: file:///usr/share/doc/python-certbot-doc/html/index.html
             https://certbot.eff.org/docs

I compared with /opt/zammad/contrib/nginx/zammad_ssl.conf, but didn’t find any differencies.

[MrGeneration]

Please by close attention to paths your system complains about:

Oct 30 17:20:07 yyy-s-2vcpu-4gb-fra1-01 nginx[1598]: nginx: [emerg] cannot load certificate “/etc/nginx/ssl/xxx.duckdns.org-fullchain.pem”>

Usually you don’t use the “live” certificates from the letsencrypt directory but install them to a for you fitting location. You may want to double che ck your paths and configuration of nginx.

[mittkkoo]

I think I know what you mean. But since I did it like in the docs, the certificates are installed.

I now deleted all certificates with certbot delete and reverted zammad.conf to the version under /contrib. Also checked the version in the repo and it is the same.

cp /opt/zammad/contrib/nginx/zammad.conf /etc/nginx/sites-available/zammad.conf

Then I started nginx again
systemctl stop nginx
systemctl start nginx

Nginx works:

# systemctl status nginx.service
...
     Active: active (running) since Tue 2023-10-31 09:44:54 UTC; 1s ago
...

There are no certificates:

# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
No certificates found.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Then I named the configuration according to the docs. And certbot received a certificate:

# certbot --nginx -d xxx.yyyy.zzz
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for xxx.yyyy.zzz

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/xxx.yyyy.zzz/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/xxx.yyyy.zzz/privkey.pem
This certificate expires on 2024-01-29.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

Deploying certificate
Successfully deployed certificate for xxx.yyyy.zzz to /etc/nginx/sites-enabled/zammad.conf
Congratulations! You have successfully enabled HTTPS on https://xxx.yyyy.zzz

Finally I copied zammad_ssl.conf and adjusted the hostname according to the docs. I also compared with the repo so that zammad_ssl.conf is the original one.

But afterwards nginx fails again!

# systemctl reload nginx
Job for nginx.service failed.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.

[mittkkoo]

OK, I solved it!
I had to do this:

wget -q https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem -P /etc/nginx/ssl

and this:

openssl dhparam -out /etc/nginx/ssl/dhparam.pem 4096

While the second is mentioned in the docs (although I didn’t have to do it during the initial installation), the first is only in the comments in zammad_ssl.conf. I think this also should be in the docs!

Also in zammad.conf I had to insert the path, which certbot certificates reported to have created the certificate in. I’m not sure if this is right, but it works.

  ssl_certificate /etc/letsencrypt/live/new.domain.com/fullchain.pem;
  ssl_certificate_key  /etc/letsencrypt/live/new.domain.com/privkey.pem;

Maybe it depends on how the auth was configured, but I had to change the FQDM from the rails console, because I wasn’t able to login otherwise:

zammad run rails c
Setting.set('fqdn', 'new.domain.com')

I had Microsoft as third-party auth and local auth disabled. In this case I had to modify the redirect URI in the App in Azure to begin with the new hostname.