Zammad fails to start after upgrade on UCS (docker)

Technical assistance
1

Infos:

  • Used Zammad version: 3.6.0-42
  • Used Zammad installation type: UCS Appcenter Docker Containers
  • Operating system: UCS
  • Browser + version: Firefox

Expected behavior: Upgrade completes and Zammad becomes available.

Actual behavior: After upgrade, proxy fails (due to app not starting properly). Removing and reinstalling does not solve the problem. There are errors in the installation script that reference PGSQL authentication problems:

5796 actions.configure 21-05-06 10:04:07 [ INFO]: … Checking if our PostgreSQL needs some love
5796 actions.configure 21-05-06 10:04:07 [ INFO]: … => We need to adjust settings on PostgreSQL
5796 actions.configure 21-05-06 10:04:07 [ INFO]: … [Adjusting your PostgreSQL container]
5796 actions.configure 21-05-06 10:04:07 [ INFO]: … => Adding Zammad user [PSQL]
5796 actions.configure 21-05-06 10:04:07 [ WARNING]: createuser: could not connect to database template1: FATAL: role “postgres” does not exist
5796 actions.configure 21-05-06 10:04:07 [ INFO]: … => Adding superuser with createdb to Zammad user [PSQL]
5796 actions.configure 21-05-06 10:04:07 [ WARNING]: psql: FATAL: role “postgres” does not exist
5796 actions.configure 21-05-06 10:04:07 [ INFO]: … => Providing password to Zammad user [PSQL]
5796 actions.configure 21-05-06 10:04:07 [ WARNING]: psql: FATAL: role “postgres” does not exist
5796 actions.configure 21-05-06 10:04:08 [ INFO]: … [Okay - enough love for PostgreSQL on this regard]

Steps to reproduce the behavior: Upgrade UCS from last Zammad release to current.

This is similar, if not the same, to Upgrade to zammad 3.4.0-4 in docker based UCS environment

2

The Install script continually repeats:

5796 actions.configure 21-05-06 10:22:47 [ INFO]: # … Checking Zammad-State: 000 #

Perhaps this is because it can’t get access to the existing PGSQL tables saved from the upgrade?

3

And here is the output of “docker logs zammad_zammad-init_1”

/usr/local/bundle/gems/railties-5.2.4.4/lib/rails/engine.rb:619:in block in <class:Engine>' /usr/local/bundle/gems/railties-5.2.4.4/lib/rails/initializable.rb:32:in instance_exec’
/usr/local/bundle/gems/railties-5.2.4.4/lib/rails/initializable.rb:32:in run' /usr/local/bundle/gems/railties-5.2.4.4/lib/rails/initializable.rb:61:in block in run_initializers’
/usr/local/bundle/gems/railties-5.2.4.4/lib/rails/initializable.rb:50:in each' /usr/local/bundle/gems/railties-5.2.4.4/lib/rails/initializable.rb:50:in tsort_each_child’
/usr/local/bundle/gems/railties-5.2.4.4/lib/rails/initializable.rb:60:in run_initializers' /usr/local/bundle/gems/railties-5.2.4.4/lib/rails/application.rb:361:in initialize!’
/opt/zammad/config/environment.rb:5:in <main>' /usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in require’
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in block in require_with_bootsnap_lfi' /usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in register’
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in require_with_bootsnap_lfi' /usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in require’
/usr/local/bundle/gems/activesupport-5.2.4.4/lib/active_support/dependencies.rb:291:in block in require' /usr/local/bundle/gems/activesupport-5.2.4.4/lib/active_support/dependencies.rb:257:in load_dependency’
/usr/local/bundle/gems/activesupport-5.2.4.4/lib/active_support/dependencies.rb:291:in require' /usr/local/bundle/gems/railties-5.2.4.4/lib/rails/application.rb:337:in require_environment!’
/usr/local/bundle/gems/railties-5.2.4.4/lib/rails/application.rb:520:in block in run_tasks_blocks' /usr/local/bin/bundle:23:in load’
/usr/local/bin/bundle:23:in <main>' Tasks: TOP => db:create => db:load_config => environment (See full trace by running task with --trace) initialising / updating database... rake aborted! PG::ConnectionBad: FATAL: password authentication failed for user "zammad" /usr/local/bundle/gems/pg-0.21.0/lib/pg.rb:56:in initialize’
/usr/local/bundle/gems/pg-0.21.0/lib/pg.rb:56:in new' /usr/local/bundle/gems/pg-0.21.0/lib/pg.rb:56:in connect’
/usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql_adapter.rb:692:in connect' /usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql_adapter.rb:223:in initialize’
/usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql_adapter.rb:48:in new' /usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql_adapter.rb:48:in postgresql_connection’
/usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:830:in new_connection' /usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:874:in checkout_new_connection’
/usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:853:in try_to_checkout_new_connection' /usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:814:in acquire_connection’
/usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:538:in checkout' /usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:382:in connection’
/usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:1033:in retrieve_connection' /usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/connection_handling.rb:118:in retrieve_connection’
/usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/connection_handling.rb:90:in connection' /usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/model_schema.rb:466:in load_schema!’
/usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/attributes.rb:234:in load_schema!' /usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/attribute_decorators.rb:51:in load_schema!’
/usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/model_schema.rb:459:in block in load_schema' /usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/model_schema.rb:456:in load_schema’
/usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/model_schema.rb:346:in attribute_types' /usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/model_schema.rb:370:in type_for_attribute’
/usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/table_metadata.rb:33:in type' /usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/relation/predicate_builder.rb:51:in build’
/usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/relation/predicate_builder.rb:112:in block in expand_from_hash' /usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/relation/predicate_builder.rb:71:in each’
/usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/relation/predicate_builder.rb:71:in flat_map' /usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/relation/predicate_builder.rb:71:in expand_from_hash’
/usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/relation/predicate_builder.rb:21:in build_from_hash' /usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/relation/where_clause_factory.rb:19:in build’
/usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/relation/query_methods.rb:588:in where!' /usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/relation/finder_methods.rb:370:in construct_relation_for_exists’
/usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/relation/finder_methods.rb:320:in exists?' /usr/local/bundle/gems/activerecord-5.2.4.4/lib/active_record/querying.rb:5:in exists?’
/opt/zammad/config/initializers/models_searchable.rb:4:in <main>' /usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:50:in load’
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:50:in load' /usr/local/bundle/gems/activesupport-5.2.4.4/lib/active_support/dependencies.rb:285:in block in load’
/usr/local/bundle/gems/activesupport-5.2.4.4/lib/active_support/dependencies.rb:257:in load_dependency' /usr/local/bundle/gems/activesupport-5.2.4.4/lib/active_support/dependencies.rb:285:in load’
/usr/local/bundle/gems/railties-5.2.4.4/lib/rails/engine.rb:663:in block in load_config_initializer' /usr/local/bundle/gems/activesupport-5.2.4.4/lib/active_support/notifications.rb:170:in instrument’
/usr/local/bundle/gems/railties-5.2.4.4/lib/rails/engine.rb:662:in load_config_initializer' /usr/local/bundle/gems/railties-5.2.4.4/lib/rails/engine.rb:620:in block (2 levels) in class:Engine’
/usr/local/bundle/gems/railties-5.2.4.4/lib/rails/engine.rb:619:in each' /usr/local/bundle/gems/railties-5.2.4.4/lib/rails/engine.rb:619:in block in class:Engine’
/usr/local/bundle/gems/railties-5.2.4.4/lib/rails/initializable.rb:32:in instance_exec' /usr/local/bundle/gems/railties-5.2.4.4/lib/rails/initializable.rb:32:in run’
/usr/local/bundle/gems/railties-5.2.4.4/lib/rails/initializable.rb:61:in block in run_initializers' /usr/local/bundle/gems/railties-5.2.4.4/lib/rails/initializable.rb:50:in each’
/usr/local/bundle/gems/railties-5.2.4.4/lib/rails/initializable.rb:50:in tsort_each_child' /usr/local/bundle/gems/railties-5.2.4.4/lib/rails/initializable.rb:60:in run_initializers’
/usr/local/bundle/gems/railties-5.2.4.4/lib/rails/application.rb:361:in initialize!' /opt/zammad/config/environment.rb:5:in ’
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in require' /usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in block in require_with_bootsnap_lfi’
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in register' /usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in require_with_bootsnap_lfi’
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in require' /usr/local/bundle/gems/activesupport-5.2.4.4/lib/active_support/dependencies.rb:291:in block in require’
/usr/local/bundle/gems/activesupport-5.2.4.4/lib/active_support/dependencies.rb:257:in load_dependency' /usr/local/bundle/gems/activesupport-5.2.4.4/lib/active_support/dependencies.rb:291:in require’
/usr/local/bundle/gems/railties-5.2.4.4/lib/rails/application.rb:337:in require_environment!' /usr/local/bundle/gems/railties-5.2.4.4/lib/rails/application.rb:520:in block in run_tasks_blocks’
/usr/local/bin/bundle:23:in load' /usr/local/bin/bundle:23:in ’
Tasks: TOP => db:create => db:load_config => environment
(See full trace by running task with --trace)
initialising / updating database…
rake aborted!
PG::ConnectionBad: FATAL: password authentication failed for user “zammad”

This repeats.

I do have access to backups pre-upgrade, which contain different secrets for the zammad user. I assume this is the problem – how do I update the secret for the runtime app? I don’t know docker well (although I’ve got nearly 30 years in systems).

4

I logged into the railsserver container:

docker exec -it zammad_zammad-railsserver_1 bash

and changed the DB secret with sed:

sed -i "s/NEWSECRET/OLDSECRET/g" /opt/zammad/config/database.yml

This has allowed startup to complete, but the installation hasn’t filled in things like LDAP schema, so users aren’t able to access the interface properly.

Anyone give me a hint where the stored value is so that I can recover the app?

5

I found the old secret in some metadata files (meta.db and container metadata). Changed it with:

sed -i "s/NEWSECRET/OLDSECRET/g" /var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db

and

find /var/lib/docker/containers. -type f -exec sed -i "s/NEWSECRET/OLDSECRET/g" {} +

This does not solve the problem, as the init process is still pulling the incorrect DB secret. Recursive grep on /var and /etc underway.

6

Old secret found in backup files in /lib/univention-appcenter/backups/appcenter-backup-zammad*

Doubt this is a problem, but I removed all the incorrect backups.

7

Updating the DB secret during install allows the zammad-init to finish and it will then complete the install. However, the LDAP binding is still not functional (I assume the binding user and password are incorrect).

Are LDAP credentials held in the DB? I can’t find them anywhere else. Obviously, I don’t have access to the web front-end.

8

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.