All logins denied after update 4.1.0 -> 5.0.2

fthommen · November 17, 2021, 3:55pm

Infos:

Used Zammad version: 5.0.2-1637076120.1ba497d6.centos7
Used Zammad installation type: yum from repo
Operating system: CentOS 7.9
Browser + version: Firefox 94.0.1

Problem Description:

We have updated our Zammad 4.1.0-1626855612.90b59692.centos7 to 5.0.2-1637076120.1ba497d6.centos7 with yum update. The setup is using MariaDB, but we have installed PostgreSQL 10 beforehand, to fulfill the zammad package prerequisites.

The update printed out quite some call traces, but still seemed to be successful in the end (full output can be seen here). However now nobody can login any more. Instead we see

In /var/log/zammad/production.log we get the following messages at login:

I, [2021-11-17T16:46:09.096556 #4170-173600]  INFO -- : Started POST "/api/v1/signin" for 172.22.167.42 at 2021-11-17 16:46:09 +0100
I, [2021-11-17T16:46:09.106465 #4170-173600]  INFO -- : Processing by SessionsController#create as JSON
I, [2021-11-17T16:46:09.106554 #4170-173600]  INFO -- :   Parameters: {"username"=>"xxxxxxx@xxxx.xxxxx", "password"=>"[FILTERED]", "fingerprint"=>"-245901113"}
E, [2021-11-17T16:46:09.152213 #4170-173600] ERROR -- : Failed to load Auth::Backend from Setting '{"adapter"=>"Auth::Ldap", "host"=>"localhost", "port"=>389, "bind_dn"=>"cn=Manager,dc=example,dc=org", "bind_pw"=>"example", "uid"=>"mail", "base"=>"dc=example,dc=org", "always_filter"=>"", "always_roles"=>["Admin", "Agent"], "always_groups"=>["Users"], "sync_params"=>{"firstname"=>"sn", "lastname"=>"givenName", "email"=>"mail", "login"=>"mail"}}'
E, [2021-11-17T16:46:09.152325 #4170-173600] ERROR -- : uninitialized constant Auth::Ldap (NameError)
lib/auth/backend.rb:26:in `block in instances'
lib/auth/backend.rb:25:in `each'
lib/auth/backend.rb:25:in `filter_map'
lib/auth/backend.rb:25:in `instances'
lib/auth/backend.rb:13:in `valid?'
lib/auth.rb:37:in `valid?'
app/controllers/sessions_controller.rb:219:in `authenticate_with_password'
app/controllers/sessions_controller.rb:10:in `create'
app/controllers/application_controller/has_download.rb:21:in `block (4 levels) in <module:HasDownload>'
app/controllers/application_controller/has_download.rb:20:in `block (3 levels) in <module:HasDownload>'
app/controllers/application_controller/has_download.rb:19:in `block (2 levels) in <module:HasDownload>'
app/controllers/application_controller/handles_transitions.rb:16:in `handle_transaction'
I, [2021-11-17T16:46:10.169668 #4170-173600]  INFO -- : Login failed. Have you double-checked your credentials and completed the email verification step? (Exceptions::NotAuthorized)
app/controllers/application_controller/authenticates.rb:173:in `raise_unified_login_error'
app/controllers/sessions_controller.rb:219:in `authenticate_with_password'
app/controllers/sessions_controller.rb:10:in `create'
app/controllers/application_controller/has_download.rb:21:in `block (4 levels) in <module:HasDownload>'
app/controllers/application_controller/has_download.rb:20:in `block (3 levels) in <module:HasDownload>'
app/controllers/application_controller/has_download.rb:19:in `block (2 levels) in <module:HasDownload>'
app/controllers/application_controller/handles_transitions.rb:16:in `handle_transaction'
I, [2021-11-17T16:46:10.173194 #4170-173600]  INFO -- : Completed 401 Unauthorized in 1065ms (Views: 0.2ms | ActiveRecord: 2.8ms | Allocations: 7917)

Note, that we don’t have LDAP authentication activated (it never worked for us). There are no obvious errors at service start.

Any, even the slightest, hint about how to fix this is highly appreciated, as currently our ticketing system is down and inaccessible.

Cheers

MrGeneration · November 17, 2021, 3:58pm

https://docs.zammad.org/en/latest/install/update.html

Your migrations are not complete.

fthommen · November 17, 2021, 4:54pm

only the recreation of the ES index was missing and recreating it didn’t solve the problem

lama · November 17, 2021, 5:01pm

i had a simliar failure (and we had some problems in choosing “Owner (here, customers was listed to)” and "Ticket Status (here, also wait unitl was open, so we cant use status open and that) at user which were logged in), in my case i had to do an second try in “apt upgrade”:
apt list --upgradable
Listing… Done
zammad/unknown 5.0.2-1637162754.47b72864.buster amd64 [upgradable from: 5.0.2-1637076120.1ba497d6.buster]
→
Unpacking zammad (5.0.2-1637162754.47b72864.buster) over (5.0.2-1637076120.1ba497d6.buster) …

After that we was able to login again (and change Ticket Infos as usual).

Sorry for bad english,…

fthommen · November 17, 2021, 5:24pm

Thanks @lama, but I’m not sure if a yum reinstall wouldn’t make the situation even worse. At yum update zammad is not offered any more, as it is already - and successfully from the perspective of the package system - installed and up-to-date.

MrGeneration · November 17, 2021, 5:25pm

lama’s issue was actually solved because the postinstall script that is run upon every installation in the second run was successful. So it’s only a “indirect” fix.

Have I been not clear enough with your migrations are not complete?
Use this: zammad run rake db:migrate ~

Of course you’ll stop Zammad before and ensure the cache is cleared.

fthommen · November 17, 2021, 5:35pm

No you haven’t, because this step is not mentioned anywhere the page that you referred me to for the missing steps (Updating Zammad — Zammad System Documentation documentation) … but it solved the issue. Thanks a lot!

Why isn’t this step performed automatically by the updater during post-install? At least it could be mentioned together with the hints regarding the missing FQDN and the SELinux settings. This would make Zammad updates so much easier for stressed sysadmins

dvnkln · November 18, 2021, 6:34am

Uhm… it actually is performed by the updater during post-install:

github.com

zammad/zammad/blob/stable/contrib/packager.io/functions#L194

    
      
          
          
  echo "# ... Fixing permission database.yml"
            chown zammad:zammad ${ZAMMAD_DIR}/config/database.yml
          }
          
          
function initialise_database () {
            zammad run rake db:migrate
            zammad run rake db:seed
          }
          
          
function update_database () {
            echo "# database.yml found. Updating db..."
            zammad run rake db:migrate
          }
          
          
function update_translations () {
            echo "# Updating translations..."
            zammad run rails r 'Locale.sync'
            zammad run rails r 'Translation.sync'
          }

In your case it just seemed to have failed (which can happen for a ton of reasons).
There probably have been a few error messages thrown while upgrading. Most likely something like “migration failed” or any errors directly after trying to migrate the db (been there, done that).

If you think some kind of hint has to be inside the docs feel free to create a PR. IMHO you’re best suited for this since you know what you were searching for while having that problem.

cheers

fthommen · November 18, 2021, 8:32am

Yes, that looks like a valid explanation . Thank you @dvnkln. I’m not sure it makes sense to adapt the documentation in this case, as consequently there would have to be side-notes for each kind of possible update failure and myself I’m not capable of interpreting the call trace printed out by the updater to find out, what exactly went wrong in our specific case.

Howver if a missed database migration is a “regular” problem happening after an update/upgrade, then the documentation should probably be adapted. In any case I will make this step part of our own internal documentation as something that should be done in case of issues before opening a thread in the Zammad Forum

MrGeneration · November 18, 2021, 8:39am

We have a community user that went through some effort to have a first try PR.
It still needs QA and is not exactly in the shape I want it to but there has been effort.

github.com/zammad/zammad-documentation

created troubleshooting/faq section and added paragraph about elasticsearch issues

zammad:main ← kaeltis:add-troubleshooting

opened 12:02PM - 20 Jul 21 UTC

kaeltis

+127 -0

After having trouble with missing data from the Web-UI and realizing this seems …to be a common occurence I was asked to create an addition for the documentation. I'm hoping this could be extended in the future to add more common issues and their fixes as well as frequently asked questions to reduce opened GitHub issues without creating the same forum threads again and again. First time using Sphinx / ReStructuredText so I hope I didn't do any major mistakes 😄 Original Issue: https://github.com/zammad/zammad/issues/3662

PR’s are welcome.

dvnkln · November 18, 2021, 12:11pm

Looks like you maybe had the same problem i had. I’m only running into this when pg is being updated in the same “process” zammad is being updated. Probably some timing issues(?). Since i’m using a script for my update procedure i just told it to update pg prior to zammad (if any pg update is available) in two different steps. Nothing zammad can fix by itself since your package manager is updating pg and not zammad.

There could be a way to avoid that inside the postinstall script but for that someone would need to debug that further. I’m just guessing here since my “workaround” seems to help (at least on my setup).

cheers

fthommen · November 19, 2021, 9:06am

I’ll get this procedure a try next time. Thank you for the hint.

system · March 19, 2022, 9:06am

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.