502 Bad Gateway after Updae

Infos:

  • Used Zammad version: 6.1.0
  • Used Zammad installation type: Package
  • Operating system: Ubuntu 18.04 LTS
  • Browser + version: FireFox ESR 128.6.0

Expected behavior:

Run ‘do-release-upgrade’ to upgrade operating system

Actual behavior:

After upgrade operating system web interface no longer accessible => 502 Bad Gateway.

Steps to reproduce the behavior:

Process already performed several times with clone of the original system

Hello,

due to CVE-2024-55578, we wanted to update our self-hosted Zammad from 6.1 to 6.4.x.
Since the operating system is also outdated, we wanted to update both.
We are currently using an Ubuntu server 18.04 LTS and want to upgrade it to 24.04 LTS.

However, as soon as I upgrade our Ubuntu server from 18.04 to 20.04, the web interface of the Zammad server is no longer accessible => 502 Bad Gateway.

All services are running Zammad => OK | Elasticsearch => OK | nginx => OK | Functional test nginx -t => OK

I looked for clues and solutions in other posts in the forum.

In a post with a similar problem, the solution was to use the following commands:
zammad run rails r “Setting.set(‘es_url’, ‘http://127.0.0.1:9200’)”
zammad run rake searchindex:rebuild
zammad run rake db:migrate
systemctl restart nginx

I get an error message with the first command:

/opt/zammad/config/initializers/html_email_style.rb:1:in `': uninitialized constant Setting (NameError)

if ActiveRecord::Base.connection.table_exists?( Setting.table_name )

Then I updated Zammad to 6.4 but the error remains.
I can’t find the specific cause of the error

I also checked the file “html_email_style.rb” but I can’t see what’s wrong.
We haven’t changed anything

Do you have modified your zammad? I don’t remember that we ever had a file called /opt/zammad/config/initializers/html_email_style.rb. Whats the content of that? Maybe disable it. In zammad 6.4.x. there is no html_email_style.rb.

mv /opt/zammad/config/initializers/html_email_style.rb /opt/zammad/config/initializers/html_email_style.rb.bak

Hello rolfschmidt,

thanks for the tip.
I renamed the file and the error from my first post is gone.
The system then reported that the PostgreSQL version is too low.
Version 10+ is required - installed and used 9.5.25.

################################################################

Using “pg_lsclusters” I was then able to see that new PostgreSQL versions were installed during the last update process.

9.5 main 5432 online postgres /var/lib/postgresql/9.5/main /var/log/postgresql/postgresql-9.5-main.log
10 main 5433 online postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
12 main 5434 online postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log
14 main 5435 online postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log

################################################################

“pg_upgrade” could not be performed

Performing consistency checks

Checking cluster versions ok

failure
Consult the last few lines of “pg_upgrade_server.log” for
the probable cause of the failure.

connection to server on socket “/var/lib/postgresql/.s.PGSQL.50432” failed: No such file or directory
Is the server running locally and accepting connections on that socket?

could not connect to source postmaster started with the command:
“/usr/lib/postgresql/9.5/bin/pg_ctl” -w -l “pg_upgrade_server.log” -D “/var/lib/postgresql/9.5/main” -o “-p 50432 -b -c listen_addresses=‘’ -c unix_socket_permissions=0700 -c unix_socket_directories=‘/var/lib/postgresql’” start
Failure, exiting

################################################################

After several attempts to specify the database version, I took a more radical approach

  • Backup of the DB created
  • All versions of PostgreSQL removed 9.5 / 10 / 12 / 14
  • PostgreSQL installed version 17
  • Recovery from DB backup

################################################################

Command “zammad run rails r “Setting.set(‘es_url’, ‘http://127.0.0.1:9200’)”” could now be executed

When I ran “zammad run rake searchindex:rebuild” I got the message:

rake aborted!

Don’t know how to build task ‘searchindex:rebuild’

web interface not accessible but the error changed to 500 Internal Server Errors.

All services are running Zammad => OK | Elasticsearch => OK | nginx => OK | Functional test nginx -t => OK

################################################################

I restarted the server – after that error 500 went away but “CSRF token verification failed!” when trying to log in

Changed the following values ​​in zammad.conf

from => proxy_set_header X-Forwarded-Proto $scheme

to => proxy_set_header X-Forwarded-Proto https;

################################################################

After I restarted the nginx service, another error occurred which I unfortunately hadn’t noted down

With “zammad run rake searchindex:rebuild” I got the message => rake aborted! Don’t know how to build task ‘searchindex:rebuild’

“zammad run rake db:migrate” was possible - some data was also migrated

When I tried to log in, the error message that was no longer known was gone, but now you are logged out again immediately after logging in. Error message in the login mask “The session is no longer valid. Please log in again”

################################################################

What would be the next step?

PS: I’m only writing this in detail so that others might find it helpful :slight_smile:

As always, a good eye and one or two restarts help.
I forgot to delete a character in the zammad.conf at “proxy_set_header X-Forwarded-Proto https;”

I only noticed it because after a restart the error “CSRF token verification failed” occurred again.

After I corrected it and restarted the system, I was able to log in successfully.

I will now upgrade the system version and if everything goes well I will close the ticket.

Thanks again Rolfschmidt for the info.

1 Like

I can’t really tell what to do with the postgres update, but you should keep searching for a solution for that. Your version is old.

About elasticsearch it’s the same. Make sure you have a updated version (Software — Zammad System Documentation documentation).

You can try to curl your elasticsearch to see if there are connection issues:

curl http://127.0.0.1:9200

You could also check the logs for errors grep ERROR -A10 log/production.log

In the worst case just reinstall elasticsearch (Set Up Elasticsearch — Zammad System Documentation documentation). Since the data is temporary you can’t really lose something. Reinstall it and then rebuild your searchindex and you will be fine.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.