All logins denied after update 4.1.0 -> 5.0.2

Infos:

  • Used Zammad version: 5.0.2-1637076120.1ba497d6.centos7
  • Used Zammad installation type: yum from repo
  • Operating system: CentOS 7.9
  • Browser + version: Firefox 94.0.1

Problem Description:

We have updated our Zammad 4.1.0-1626855612.90b59692.centos7 to 5.0.2-1637076120.1ba497d6.centos7 with yum update. The setup is using MariaDB, but we have installed PostgreSQL 10 beforehand, to fulfill the zammad package prerequisites.

The update printed out quite some call traces, but still seemed to be successful in the end (full output can be seen here). However now nobody can login any more. Instead we see

image

In /var/log/zammad/production.log we get the following messages at login:

I, [2021-11-17T16:46:09.096556 #4170-173600]  INFO -- : Started POST "/api/v1/signin" for 172.22.167.42 at 2021-11-17 16:46:09 +0100
I, [2021-11-17T16:46:09.106465 #4170-173600]  INFO -- : Processing by SessionsController#create as JSON
I, [2021-11-17T16:46:09.106554 #4170-173600]  INFO -- :   Parameters: {"username"=>"xxxxxxx@xxxx.xxxxx", "password"=>"[FILTERED]", "fingerprint"=>"-245901113"}
E, [2021-11-17T16:46:09.152213 #4170-173600] ERROR -- : Failed to load Auth::Backend from Setting '{"adapter"=>"Auth::Ldap", "host"=>"localhost", "port"=>389, "bind_dn"=>"cn=Manager,dc=example,dc=org", "bind_pw"=>"example", "uid"=>"mail", "base"=>"dc=example,dc=org", "always_filter"=>"", "always_roles"=>["Admin", "Agent"], "always_groups"=>["Users"], "sync_params"=>{"firstname"=>"sn", "lastname"=>"givenName", "email"=>"mail", "login"=>"mail"}}'
E, [2021-11-17T16:46:09.152325 #4170-173600] ERROR -- : uninitialized constant Auth::Ldap (NameError)
lib/auth/backend.rb:26:in `block in instances'
lib/auth/backend.rb:25:in `each'
lib/auth/backend.rb:25:in `filter_map'
lib/auth/backend.rb:25:in `instances'
lib/auth/backend.rb:13:in `valid?'
lib/auth.rb:37:in `valid?'
app/controllers/sessions_controller.rb:219:in `authenticate_with_password'
app/controllers/sessions_controller.rb:10:in `create'
app/controllers/application_controller/has_download.rb:21:in `block (4 levels) in <module:HasDownload>'
app/controllers/application_controller/has_download.rb:20:in `block (3 levels) in <module:HasDownload>'
app/controllers/application_controller/has_download.rb:19:in `block (2 levels) in <module:HasDownload>'
app/controllers/application_controller/handles_transitions.rb:16:in `handle_transaction'
I, [2021-11-17T16:46:10.169668 #4170-173600]  INFO -- : Login failed. Have you double-checked your credentials and completed the email verification step? (Exceptions::NotAuthorized)
app/controllers/application_controller/authenticates.rb:173:in `raise_unified_login_error'
app/controllers/sessions_controller.rb:219:in `authenticate_with_password'
app/controllers/sessions_controller.rb:10:in `create'
app/controllers/application_controller/has_download.rb:21:in `block (4 levels) in <module:HasDownload>'
app/controllers/application_controller/has_download.rb:20:in `block (3 levels) in <module:HasDownload>'
app/controllers/application_controller/has_download.rb:19:in `block (2 levels) in <module:HasDownload>'
app/controllers/application_controller/handles_transitions.rb:16:in `handle_transaction'
I, [2021-11-17T16:46:10.173194 #4170-173600]  INFO -- : Completed 401 Unauthorized in 1065ms (Views: 0.2ms | ActiveRecord: 2.8ms | Allocations: 7917)

Note, that we don’t have LDAP authentication activated (it never worked for us). There are no obvious errors at service start.

Any, even the slightest, hint about how to fix this is highly appreciated, as currently our ticketing system is down and inaccessible.

Cheers

https://docs.zammad.org/en/latest/install/update.html

Your migrations are not complete.

only the recreation of the ES index was missing and recreating it didn’t solve the problem

i had a simliar failure (and we had some problems in choosing “Owner (here, customers was listed to)” and "Ticket Status (here, also wait unitl was open, so we cant use status open and that) at user which were logged in), in my case i had to do an second try in “apt upgrade”:
apt list --upgradable
Listing… Done
zammad/unknown 5.0.2-1637162754.47b72864.buster amd64 [upgradable from: 5.0.2-1637076120.1ba497d6.buster]

Unpacking zammad (5.0.2-1637162754.47b72864.buster) over (5.0.2-1637076120.1ba497d6.buster) …

After that we was able to login again (and change Ticket Infos as usual).

Sorry for bad english,… :confused:

Thanks @lama, but I’m not sure if a yum reinstall wouldn’t make the situation even worse. At yum update zammad is not offered any more, as it is already - and successfully from the perspective of the package system - installed and up-to-date.

lama’s issue was actually solved because the postinstall script that is run upon every installation in the second run was successful. So it’s only a “indirect” fix.

Have I been not clear enough with your migrations are not complete?
Use this: zammad run rake db:migrate ~

Of course you’ll stop Zammad before and ensure the cache is cleared.

No you haven’t, because this step is not mentioned anywhere the page that you referred me to for the missing steps (Updating Zammad — Zammad documentation) :wink: … but it solved the issue. Thanks a lot!

Why isn’t this step performed automatically by the updater during post-install? At least it could be mentioned together with the hints regarding the missing FQDN and the SELinux settings. This would make Zammad updates so much easier for stressed sysadmins :sweat:

1 Like

Uhm… it actually is performed by the updater during post-install:

In your case it just seemed to have failed (which can happen for a ton of reasons).
There probably have been a few error messages thrown while upgrading. Most likely something like “migration failed” or any errors directly after trying to migrate the db (been there, done that).

If you think some kind of hint has to be inside the docs feel free to create a PR. IMHO you’re best suited for this since you know what you were searching for while having that problem.

cheers

1 Like

Yes, that looks like a valid explanation :slight_smile: . Thank you @svnr-dvnkln. I’m not sure it makes sense to adapt the documentation in this case, as consequently there would have to be side-notes for each kind of possible update failure and myself I’m not capable of interpreting the call trace printed out by the updater to find out, what exactly went wrong in our specific case.

Howver if a missed database migration is a “regular” problem happening after an update/upgrade, then the documentation should probably be adapted. In any case I will make this step part of our own internal documentation as something that should be done in case of issues before opening a thread in the Zammad Forum :slight_smile:

We have a community user that went through some effort to have a first try PR.
It still needs QA and is not exactly in the shape I want it to but there has been effort.

PR’s are welcome.

Looks like you maybe had the same problem i had. I’m only running into this when pg is being updated in the same “process” zammad is being updated. Probably some timing issues(?). Since i’m using a script for my update procedure i just told it to update pg prior to zammad (if any pg update is available) in two different steps. Nothing zammad can fix by itself since your package manager is updating pg and not zammad.

There could be a way to avoid that inside the postinstall script but for that someone would need to debug that further. I’m just guessing here since my “workaround” seems to help (at least on my setup).

cheers

I’ll get this procedure a try next time. Thank you for the hint.