Zammad / nginx fails after server restore on Digitalocean

Infos:

  • Used Zammad version: 6.1
  • Used Zammad installation type: package
  • Operating system: Ubuntu 22.04.2 LTS (Jammy Jellyfish)
  • Browser + version: all

Expected behavior:

  • I expect Zammad and all other services to be working as before if I restore a server image

Actual behavior:

  • I installed Zammad from package following the docs on a Digitalocean droplet. Everything worked well, also the let’s encrypt SSL. Then I wanted to change the hostname from duckdns.org to a new one. Before beginning I created a Snapshot of the droplet through the UI of Digitalocean. I tried to change the hostname, but didn’t manage to configure nginx, certbot and let’s encrypt correctly. Anyway, I decided to revert everything. So I restored the droplet from the Snapshot. But the site is not reachable.

Steps to reproduce the behavior:

  • Install Zammad from package following the docs on an ubuntu 22.04 droplet from Digitalocean
  • Configure SSL with let’s encrypt on a test domain from duckdns.org following the docs
  • Create a Snapshot of the droplet
  • Try to change the hostname to a new one (unsuccessfully)
  • Revert all changes by restoring the droplet from the Snapshot
  • See if Zammad and the other services work as before?

Here some outputs:

root@yyy-s-2vcpu-4gb-fra1-01:~# systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2023-10-30 17:20:07 UTC; 1h 24min ago
       Docs: man:nginx(8)
    Process: 1598 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)
        CPU: 21ms

Oct 30 17:20:07 zammadpackageubuntu-s-2vcpu-4gb-fra1-01 systemd[1]: Starting A high performance web server and a reverse proxy server...
Oct 30 17:20:07 yyy-s-2vcpu-4gb-fra1-01 nginx[1598]: nginx: [emerg] cannot load certificate "/etc/nginx/ssl/xxx.duckdns.org-fullchain.pem">
Oct 30 17:20:07 yyy-s-2vcpu-4gb-fra1-01 nginx[1598]: nginx: configuration file /etc/nginx/nginx.conf test failed
Oct 30 17:20:07 yyy-s-2vcpu-4gb-fra1-01 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Oct 30 17:20:07 yyy-s-2vcpu-4gb-fra1-01 systemd[1]: nginx.service: Failed with result 'exit-code'.
Oct 30 17:20:07 yyy-s-2vcpu-4gb-fra1-01 systemd[1]: Failed to start A high performance web server and a reverse proxy server.

root@yyy-s-2vcpu-4gb-fra1-01:~# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: xxx.duckdns.org
    Serial Number: 396...
    Key Type: RSA
    Domains: xxx.duckdns.org
    Expiry Date: 202q-0q-15 08:q0:0q+00:00 (VALID: 7q days)
    Certificate Path: /etc/letsencrypt/live/xxx.duckdns.org/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/xxx.duckdns.org/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

root@yyy-s-2vcpu-4gb-fra1-01:~# systemctl reload nginx
nginx.service is not active, cannot reload.

root@yyy-s-2vcpu-4gb-fra1-01:~# certbot --nginx -d xxx.duckdns.org
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Error while running nginx -c /etc/nginx/nginx.conf -t.

nginx: [emerg] cannot load certificate "/etc/nginx/ssl/xxx.duckdns.org-fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/etc/nginx/ssl/xxx.duckdns.org-fullchain.pem, r) error:10000080:BIO routines::no such file)
nginx: configuration file /etc/nginx/nginx.conf test failed

The nginx plugin is not working; there may be problems with your existing configuration.
The error was: MisconfigurationError('Error while running nginx -c /etc/nginx/nginx.conf -t.\n\nnginx: [emerg] cannot load certificate "/etc/nginx/ssl/xxx.duckdns.org-fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/etc/nginx/ssl/xxx.duckdns.org-fullchain.pem, r) error:10000080:BIO routines::no such file)\nnginx: configuration file /etc/nginx/nginx.conf test failed\n')

root@yyy-s-2vcpu-4gb-fra1-01:~# systemctl status certbot
○ certbot.service - Certbot
     Loaded: loaded (/lib/systemd/system/certbot.service; static)
     Active: inactive (dead)
TriggeredBy: ● certbot.timer
       Docs: file:///usr/share/doc/python-certbot-doc/html/index.html
             https://certbot.eff.org/docs

I compared with /opt/zammad/contrib/nginx/zammad_ssl.conf, but didn’t find any differencies.

Please by close attention to paths your system complains about:

Oct 30 17:20:07 yyy-s-2vcpu-4gb-fra1-01 nginx[1598]: nginx: [emerg] cannot load certificate “/etc/nginx/ssl/xxx.duckdns.org-fullchain.pem”>

Usually you don’t use the “live” certificates from the letsencrypt directory but install them to a for you fitting location. You may want to double che ck your paths and configuration of nginx.

I think I know what you mean. But since I did it like in the docs, the certificates are installed.

I now deleted all certificates with certbot delete and reverted zammad.conf to the version under /contrib. Also checked the version in the repo and it is the same.

cp /opt/zammad/contrib/nginx/zammad.conf /etc/nginx/sites-available/zammad.conf

Then I started nginx again
systemctl stop nginx
systemctl start nginx

Nginx works:

# systemctl status nginx.service
...
     Active: active (running) since Tue 2023-10-31 09:44:54 UTC; 1s ago
...

There are no certificates:

# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
No certificates found.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Then I named the configuration according to the docs. And certbot received a certificate:

# certbot --nginx -d xxx.yyyy.zzz
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for xxx.yyyy.zzz

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/xxx.yyyy.zzz/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/xxx.yyyy.zzz/privkey.pem
This certificate expires on 2024-01-29.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

Deploying certificate
Successfully deployed certificate for xxx.yyyy.zzz to /etc/nginx/sites-enabled/zammad.conf
Congratulations! You have successfully enabled HTTPS on https://xxx.yyyy.zzz

Finally I copied zammad_ssl.conf and adjusted the hostname according to the docs. I also compared with the repo so that zammad_ssl.conf is the original one.

But afterwards nginx fails again!

# systemctl reload nginx
Job for nginx.service failed.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.

OK, I solved it!
I had to do this:

wget -q https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem -P /etc/nginx/ssl

and this:

openssl dhparam -out /etc/nginx/ssl/dhparam.pem 4096

While the second is mentioned in the docs (although I didn’t have to do it during the initial installation), the first is only in the comments in zammad_ssl.conf. I think this also should be in the docs!

Also in zammad.conf I had to insert the path, which certbot certificates reported to have created the certificate in. I’m not sure if this is right, but it works.

  ssl_certificate /etc/letsencrypt/live/new.domain.com/fullchain.pem;
  ssl_certificate_key  /etc/letsencrypt/live/new.domain.com/privkey.pem;

Maybe it depends on how the auth was configured, but I had to change the FQDM from the rails console, because I wasn’t able to login otherwise:

zammad run rails c
Setting.set('fqdn', 'new.domain.com')

I had Microsoft as third-party auth and local auth disabled. In this case I had to modify the redirect URI in the App in Azure to begin with the new hostname.

This topic was automatically closed 360 days after the last reply. New replies are no longer allowed.