StatusCode: 500 and undefined method `lookup'

Technical assistance
1

Infos:

  • Used Zammad version: 3.3.0 (zammad-docker-compose:3.3.0-19 and up)
  • Installation method (source, package, …): helm chart 2.1.1
  • Operating system: GKE
  • Database + version: Postgresql 11.7
  • Elasticsearch version: 7.6.1
  • Browser + version: Chrome 80.0.3987.132 + Firefox 74.0

Expected behavior:

  • After importing the postgresql dump I’m not able to login into the system. When I try to login using a user that worked well in the 3.2 installation I’m somewhat logged in but cant do anything only getting - the below error message and a corresponding 500 server error pop up in the browser windows

Actual behavior:

  • after trying to login, I’m getting a popup, saying " StatusCode: 500 {“error”:“undefined method `lookup’ for #\u003cClass:0x000055f4cfdba2d0\u003e”}" and now services (scheduler) keep on crashing, I’m onlygetting 503 from nginx when I try to login again…

Steps to reproduce the behavior:

Am I missing some important migration step?

which were:

  • dumped postgresql db to a file - also backed up complete content from /opt/zammad directory
    on the railsserver
  • uninstalled helm chart (also removed all pvc’s)
  • installed new helm chart version (worked well and all pods came up without issues)
  • logged into the postgresql pod, and imported my previously generated dump using psql (worked well after dropping / recreating my zammad database)
  • did NOT restore any content to /opt/zammad on the railsserver nor did I run any rail scripts
  • restarted zammad railsserver

in the first instance, the railsserver starts without any issues or error logs.
Once I try to login it get the said “StatusCode: 500” error in the browser and the railsserver logs says:

E, [2020-04-02T16:40:41.288887 #1-47254983389020] ERROR -- : undefined method `lookup' for #<Class:0x000055f4cfdba2d0> (NoMethodError)
/usr/local/bundle/gems/activerecord-5.2.4.2/lib/active_record/dynamic_matchers.rb:22:in `method_missing'
/opt/zammad/app/models/application_model/can_assets.rb:153:in `block in assets_of_object_list'
/opt/zammad/app/models/application_model/can_assets.rb:151:in `each'
/opt/zammad/app/models/application_model/can_assets.rb:151:in `assets_of_object_list'
/opt/zammad/app/models/activity_stream/assets.rb:43:in `assets'
/opt/zammad/app/controllers/activity_stream_controller.rb:24:in `block in show'
/usr/local/bundle/gems/activerecord-5.2.4.2/lib/active_record/relation/delegation.rb:71:in `each'
/usr/local/bundle/gems/activerecord-5.2.4.2/lib/active_record/relation/delegation.rb:71:in `each'
/opt/zammad/app/controllers/activity_stream_controller.rb:22:in `show'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/abstract_controller/base.rb:194:in `process_action'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_controller/metal/rendering.rb:30:in `process_action'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/usr/local/bundle/gems/activesupport-5.2.4.2/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/opt/zammad/app/controllers/application_controller/has_secure_content_security_policy_for_downloads.rb:18:in `block (4 levels) in <module:HasSecureContentSecurityPolicyForDownloads>'
/usr/local/bundle/gems/activesupport-5.2.4.2/lib/active_support/notifications.rb:180:in `subscribed'
/opt/zammad/app/controllers/application_controller/has_secure_content_security_policy_for_downloads.rb:17:in `block (3 levels) in <module:HasSecureContentSecurityPolicyForDownloads>'
/usr/local/bundle/gems/activesupport-5.2.4.2/lib/active_support/notifications.rb:180:in `subscribed'
/opt/zammad/app/controllers/application_controller/has_secure_content_security_policy_for_downloads.rb:16:in `block (2 levels) in <module:HasSecureContentSecurityPolicyForDownloads>'
/usr/local/bundle/gems/activesupport-5.2.4.2/lib/active_support/callbacks.rb:118:in `instance_exec'
/usr/local/bundle/gems/activesupport-5.2.4.2/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/local/bundle/gems/activesupport-5.2.4.2/lib/active_support/callbacks.rb:136:in `run_callbacks'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/abstract_controller/callbacks.rb:41:in `process_action'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_controller/metal/rescue.rb:22:in `process_action'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/usr/local/bundle/gems/activesupport-5.2.4.2/lib/active_support/notifications.rb:168:in `block in instrument'
/usr/local/bundle/gems/activesupport-5.2.4.2/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/usr/local/bundle/gems/activesupport-5.2.4.2/lib/active_support/notifications.rb:168:in `instrument'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/usr/local/bundle/gems/activerecord-5.2.4.2/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/abstract_controller/base.rb:134:in `process'
/usr/local/bundle/gems/actionview-5.2.4.2/lib/action_view/rendering.rb:32:in `process'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_controller/metal.rb:191:in `dispatch'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_controller/metal.rb:252:in `dispatch'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_dispatch/journey/router.rb:35:in `each'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_dispatch/journey/router.rb:35:in `serve'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_dispatch/routing/route_set.rb:840:in `call'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:420:in `call_app!'
/usr/local/bundle/gems/omniauth-saml-1.10.1/lib/omniauth/strategies/saml.rb:89:in `other_phase'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:190:in `call!'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'
/usr/local/bundle/gems/omniauth-1.9.0/lib/omniauth/builder.rb:64:in `call'
/usr/local/bundle/gems/rack-2.2.2/lib/rack/tempfile_reaper.rb:15:in `call'
/usr/local/bundle/gems/rack-2.2.2/lib/rack/etag.rb:27:in `call'
/usr/local/bundle/gems/rack-2.2.2/lib/rack/conditional_get.rb:27:in `call'
/usr/local/bundle/gems/rack-2.2.2/lib/rack/head.rb:12:in `call'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/local/bundle/gems/rack-2.2.2/lib/rack/session/abstract/id.rb:266:in `context'
/usr/local/bundle/gems/rack-2.2.2/lib/rack/session/abstract/id.rb:260:in `call'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/usr/local/bundle/gems/activesupport-5.2.4.2/lib/active_support/callbacks.rb:98:in `run_callbacks'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/usr/local/bundle/gems/railties-5.2.4.2/lib/rails/rack/logger.rb:38:in `call_app'
/usr/local/bundle/gems/railties-5.2.4.2/lib/rails/rack/logger.rb:28:in `call'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/usr/local/bundle/gems/rack-2.2.2/lib/rack/method_override.rb:24:in `call'
/usr/local/bundle/gems/rack-2.2.2/lib/rack/runtime.rb:22:in `call'
/usr/local/bundle/gems/activesupport-5.2.4.2/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/usr/local/bundle/gems/actionpack-5.2.4.2/lib/action_dispatch/middleware/executor.rb:14:in `call'
/usr/local/bundle/gems/rack-2.2.2/lib/rack/sendfile.rb:110:in `call'
/usr/local/bundle/gems/railties-5.2.4.2/lib/rails/engine.rb:524:in `call'
/usr/local/bundle/gems/puma-3.12.4/lib/puma/configuration.rb:227:in `call'
/usr/local/bundle/gems/puma-3.12.4/lib/puma/server.rb:675:in `handle_request'
/usr/local/bundle/gems/puma-3.12.4/lib/puma/server.rb:476:in `process_client'
/usr/local/bundle/gems/puma-3.12.4/lib/puma/server.rb:334:in `block in run'
/usr/local/bundle/gems/puma-3.12.4/lib/puma/thread_pool.rb:135:in `block in spawn_thread'
/usr/local/bundle/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
I, [2020-04-02T16:40:41.292531 #1-47254983389020]  INFO -- : Completed 500 Internal Server Error in 76ms (Views: 0.2ms | ActiveRecord: 31.9ms)

From now on zammad scheduler keeps on crashing:

E, [2020-04-02T17:12:33.658401 #1-47341416898020] ERROR -- : thread_client 47327396198860 exited with error #<NoMethodError: undefined method `lookup' for #<Class:0x0000561d0ed00558>>
E, [2020-04-02T17:12:33.658485 #1-47341416898020] ERROR -- : /usr/local/bundle/gems/activerecord-5.2.4.2/lib/active_record/dynamic_matchers.rb:22:in `method_missing'
  /opt/zammad/app/models/application_model/can_assets.rb:153:in `block in assets_of_object_list'
  /opt/zammad/app/models/application_model/can_assets.rb:151:in `each'
  /opt/zammad/app/models/application_model/can_assets.rb:151:in `assets_of_object_list'
  /opt/zammad/app/models/activity_stream/assets.rb:43:in `assets'
  /opt/zammad/lib/sessions/backend/activity_stream.rb:35:in `block in load'
  /usr/local/bundle/gems/activerecord-5.2.4.2/lib/active_record/relation/delegation.rb:71:in `each'
  /usr/local/bundle/gems/activerecord-5.2.4.2/lib/active_record/relation/delegation.rb:71:in `each'
  /opt/zammad/lib/sessions/backend/activity_stream.rb:33:in `load'
  /opt/zammad/lib/sessions/backend/activity_stream.rb:54:in `push'
  /opt/zammad/lib/sessions/client.rb:67:in `each'
  /opt/zammad/lib/sessions/client.rb:67:in `block in fetch'
  /opt/zammad/lib/sessions/client.rb:23:in `loop'
  /opt/zammad/lib/sessions/client.rb:23:in `fetch'
  /opt/zammad/lib/sessions/client.rb:7:in `initialize'
  /opt/zammad/lib/sessions.rb:721:in `new'
  /opt/zammad/lib/sessions.rb:721:in `thread_client'
  /opt/zammad/lib/sessions.rb:676:in `block (3 levels) in jobs'
  /usr/local/bundle/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
I, [2020-04-02T17:12:43.405527 #1-47341398935020]  INFO -- : execute Channel.stream (try_count 0)...
2

I’m afraid that I can’t reproduce that on a docker-compose 3.3.0-19 tag :confused:
Personally I doubt that this is something because of kubernetes / helm.

Can you constantly reproduce this with an upgraded instance?
Are there possibly some open Delayed-Jobs that did not yet run in that installation?

After a fresh restore without starting the scheduler, a Delayed::Job.count within a rails console and it should return 0 - if not, it’s possibly coming from a background job.

3

I can basically reproduce this behaviour by creating a brand new zammad 3.3.0 environment, export db dump from “old” zammad 3.2 and import the dump into the 3.3 environment.
I’ve done this now several times and I’m ending up with the same error.

Speaking about Zammad in a Kubernetes environment means that the zammad “pod” consists of 7 (docker) containers - 3 init containers (called “zammad-init”, “postgresql-init”, “elasticsearch-init”) that are run before the actual zammad instance (consisting of 4 containers called “zammad-nginx”, “zammad-railsserver”, “zammad-scheduler”, “zammad-websocket”) is started.
As all of the mentioned containers are handled by kubernetes, its not so easy to dont start the scheduler - at least not without messing around with the zammad kubernetes config or doing changes to the helm chart (which I’m tying to prevent…)
(Of course there are also other pods running elasticsearch, memcached and the postgresqldb)

So I tried two things:

  1. (“old” setup, already messed around a bit) got myself a shell inside “zammad-railsserver” and started a rails console and issued the delayed::Job.count command

     zammad@zammad-2nd-0:~$ rails c
 I, [2020-04-08T08:08:00.174888 #39-47192571397540]  INFO -- : Setting.set('models_searchable', ["Chat::Session", "Organization", "Ticket", "KnowledgeBase::Answer::Translation", "User"])
 Loading production environment (Rails 5.2.4.2)
 irb(main):001:0> Delayed::Job.count
 => 0
 irb(main):002:0>

  2. stopped the zammad kubernetes pod, got a postgresql shell, dropped database and recreated empty zammad database, created grants on new db, imported dump from working 3.2 instance and started the zammad pod again.
    After that I’ve got myself a shell inside the “zammad-railsserver” again:

     zammad@zammad-2nd-0:~$ rails c
 I, [2020-04-08T08:59:43.908381 #82-47154677712320]  INFO -- : Setting.set('models_searchable', ["Chat::Session", "Organization", "Ticket", "KnowledgeBase::Answer::Translation", "User"])
 Loading production environment (Rails 5.2.4.2)
 irb(main):001:0> Delayed::Job.count
 => 0
 irb(main):002:0>

As I have a complete separated test environment, I can delete/resetup at any time or run any test you want me too.

4

hmm… I’m slowly getting the feeling that something within the Kubernetes spectrum is so different.
Just to make sure the application itself “is allright” I took a clean install of Zammad 3.2, put some data into it and configured ldap against an Active Directory (2016).

After synchronization had worked I upgraded to Zammad 3.3 without any issues.

But you said you just restored without running any further upgrades, so this seems to be an issue with upgrading. Please run the following within your railsserver container (after upgrading to 3.3):

rake db:migrate # migrate the database to the current version (quite crucial)
rake assets:precompile # make sure to have current assets compiled
rails r 'Cache.clear' # clear the cache for possible left overs
rake searchindex:rebuild # mandatory with upgrading to 3.3 - should be run by the container automatically

Normally the above will automatically be run within the docker-compose context, but right now I don’t trust your setup far enough. ;-(

Please also have a look remarks on the docker-compose repository for upgrading, it might affect you:

Also, because I don’t have enough knowledge about helm and kubernetes…:
Is it really needed to remove your containers and re create them?
I mean, can’t you just “pull” the new container update?

5

I ran “your” commands - here’s the output…I would say, that went fine so far:

zammad@zammad-2nd-0:~$ rake db:migrate
I, [2020-04-09T12:53:05.008391 #50-47030650234300]  INFO -- : Setting.set('models_searchable', ["Chat::Session", "Organization", "Ticket", "KnowledgeBase::Answer::Translation", "User"])
zammad@zammad-2nd-0:~$ rake assets:precompile
I, [2020-04-09T12:53:27.630799 #53-47318522611140]  INFO -- : Setting.set('models_searchable', ["Chat::Session", "Organization", "Ticket", "KnowledgeBase::Answer::Translation", "User"])
Yarn executable was not detected in the system.
Download Yarn at https://yarnpkg.com/en/docs/install
zammad@zammad-2nd-0:~$ rails r 'Cache.clear'
I, [2020-04-09T12:55:30.648262 #63-47082636961220]  INFO -- : Setting.set('models_searchable', ["Chat::Session", "Organization", "Ticket", "KnowledgeBase::Answer::Translation", "User"])
zammad@zammad-2nd-0:~$ rake searchindex:rebuild
I, [2020-04-09T12:55:54.967518 #67-47112593147320]  INFO -- : Setting.set('models_searchable', ["Chat::Session", "Organization", "Ticket", "KnowledgeBase::Answer::Translation", "User"])
I, [2020-04-09T12:55:55.322253 #67-47112593147320]  INFO -- : # curl -X get "http://zammad-master:9200"
I, [2020-04-09T12:55:55.459799 #67-47112593147320]  INFO -- : # 200
drop indexes...I, [2020-04-09T12:55:55.462049 #67-47112593147320]  INFO -- : # curl -X delete "http://zammad-master:9200/zammad_production_chat_session"
I, [2020-04-09T12:55:55.930883 #67-47112593147320]  INFO -- : # 200
I, [2020-04-09T12:55:55.931976 #67-47112593147320]  INFO -- : # curl -X delete "http://zammad-master:9200/zammad_production_organization"
I, [2020-04-09T12:55:56.133026 #67-47112593147320]  INFO -- : # 200
I, [2020-04-09T12:55:56.134863 #67-47112593147320]  INFO -- : # curl -X delete "http://zammad-master:9200/zammad_production_cti_log"
I, [2020-04-09T12:55:56.346085 #67-47112593147320]  INFO -- : # 200
I, [2020-04-09T12:55:56.346934 #67-47112593147320]  INFO -- : # curl -X delete "http://zammad-master:9200/zammad_production_ticket"
I, [2020-04-09T12:55:56.537739 #67-47112593147320]  INFO -- : # 200
I, [2020-04-09T12:55:56.538466 #67-47112593147320]  INFO -- : # curl -X delete "http://zammad-master:9200/zammad_production_knowledge_base_translation"
I, [2020-04-09T12:55:56.726601 #67-47112593147320]  INFO -- : # 200
I, [2020-04-09T12:55:56.727433 #67-47112593147320]  INFO -- : # curl -X delete "http://zammad-master:9200/zammad_production_knowledge_base_category_translation"
I, [2020-04-09T12:55:56.858205 #67-47112593147320]  INFO -- : # 200
I, [2020-04-09T12:55:56.859015 #67-47112593147320]  INFO -- : # curl -X delete "http://zammad-master:9200/zammad_production_knowledge_base_answer_translation"
I, [2020-04-09T12:55:57.043807 #67-47112593147320]  INFO -- : # 200
I, [2020-04-09T12:55:57.044655 #67-47112593147320]  INFO -- : # curl -X delete "http://zammad-master:9200/zammad_production_user"
I, [2020-04-09T12:55:57.185472 #67-47112593147320]  INFO -- : # 200
I, [2020-04-09T12:55:57.186607 #67-47112593147320]  INFO -- : # curl -X delete "http://zammad-master:9200/zammad_production_stats_store"
I, [2020-04-09T12:55:57.346613 #67-47112593147320]  INFO -- : # 200
done
delete pipeline (pipeline)... I, [2020-04-09T12:55:57.348531 #67-47112593147320]  INFO -- : # curl -X delete "http://zammad-master:9200/_ingest/pipeline/zammad483058387599"
I, [2020-04-09T12:55:57.475991 #67-47112593147320]  INFO -- : # 200
done
create indexes...I, [2020-04-09T12:55:57.505156 #67-47112593147320]  INFO -- : Setting.set('es_multi_index', true)
I, [2020-04-09T12:55:57.510938 #67-47112593147320]  INFO -- : # curl -X put "http://zammad-master:9200/zammad_production_chat_session"
I, [2020-04-09T12:55:57.956919 #67-47112593147320]  INFO -- : # 200
I, [2020-04-09T12:55:57.957652 #67-47112593147320]  INFO -- : # curl -X put "http://zammad-master:9200/zammad_production_organization"
I, [2020-04-09T12:55:58.539511 #67-47112593147320]  INFO -- : # 200
I, [2020-04-09T12:55:58.540668 #67-47112593147320]  INFO -- : # curl -X put "http://zammad-master:9200/zammad_production_cti_log"
I, [2020-04-09T12:55:59.029467 #67-47112593147320]  INFO -- : # 200
I, [2020-04-09T12:55:59.031082 #67-47112593147320]  INFO -- : # curl -X put "http://zammad-master:9200/zammad_production_ticket"
I, [2020-04-09T12:55:59.427694 #67-47112593147320]  INFO -- : # 200
I, [2020-04-09T12:55:59.428452 #67-47112593147320]  INFO -- : # curl -X put "http://zammad-master:9200/zammad_production_knowledge_base_translation"
I, [2020-04-09T12:55:59.673204 #67-47112593147320]  INFO -- : # 200
I, [2020-04-09T12:55:59.673919 #67-47112593147320]  INFO -- : # curl -X put "http://zammad-master:9200/zammad_production_knowledge_base_category_translation"
I, [2020-04-09T12:55:59.971763 #67-47112593147320]  INFO -- : # 200
I, [2020-04-09T12:55:59.972666 #67-47112593147320]  INFO -- : # curl -X put "http://zammad-master:9200/zammad_production_knowledge_base_answer_translation"
I, [2020-04-09T12:56:00.526245 #67-47112593147320]  INFO -- : # 200
I, [2020-04-09T12:56:00.527247 #67-47112593147320]  INFO -- : # curl -X put "http://zammad-master:9200/zammad_production_user"
I, [2020-04-09T12:56:01.036349 #67-47112593147320]  INFO -- : # 200
I, [2020-04-09T12:56:01.037642 #67-47112593147320]  INFO -- : # curl -X put "http://zammad-master:9200/zammad_production_stats_store"
I, [2020-04-09T12:56:01.385771 #67-47112593147320]  INFO -- : # 200
done
create pipeline (pipeline)... I, [2020-04-09T12:56:01.397520 #67-47112593147320]  INFO -- : # curl -X delete "http://zammad-master:9200/_ingest/pipeline/zammad483058387599"
I, [2020-04-09T12:56:01.456961 #67-47112593147320]  INFO -- : # 404
I, [2020-04-09T12:56:01.457097 #67-47112593147320]  INFO -- : # curl -X put "http://zammad-master:9200/_ingest/pipeline/zammad483058387599"
I, [2020-04-09T12:56:01.558430 #67-47112593147320]  INFO -- : # 200
done
reload data...
 reload Chat::Session
  - started at 2020-04-09 12:56:01 UTC
  - took 0 seconds
 reload Organization
  - started at 2020-04-09 12:56:01 UTC
I, [2020-04-09T12:56:01.585511 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_organization/_doc/2?pipeline=zammad483058387599"
I, [2020-04-09T12:56:01.998654 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:02.013839 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_organization/_doc/1?pipeline=zammad483058387599"
I, [2020-04-09T12:56:02.187137 #67-47112593147320]  INFO -- : # 201
  - took 0 seconds
 reload Cti::Log
  - started at 2020-04-09 12:56:02 UTC
  - took 0 seconds
 reload Ticket
  - started at 2020-04-09 12:56:02 UTC
I, [2020-04-09T12:56:02.525312 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_ticket/_doc/16?pipeline=zammad483058387599"
I, [2020-04-09T12:56:02.954697 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:03.032807 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_ticket/_doc/15?pipeline=zammad483058387599"
I, [2020-04-09T12:56:03.328547 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:03.406362 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_ticket/_doc/14?pipeline=zammad483058387599"
I, [2020-04-09T12:56:03.527737 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:03.624499 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_ticket/_doc/13?pipeline=zammad483058387599"
I, [2020-04-09T12:56:03.693402 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:03.750417 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_ticket/_doc/12?pipeline=zammad483058387599"
I, [2020-04-09T12:56:03.778594 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:03.838820 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_ticket/_doc/11?pipeline=zammad483058387599"
I, [2020-04-09T12:56:03.861891 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:03.926785 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_ticket/_doc/10?pipeline=zammad483058387599"
I, [2020-04-09T12:56:04.156587 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:04.231271 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_ticket/_doc/9?pipeline=zammad483058387599"
I, [2020-04-09T12:56:04.268131 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:04.326437 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_ticket/_doc/8?pipeline=zammad483058387599"
I, [2020-04-09T12:56:04.363128 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:04.448812 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_ticket/_doc/7?pipeline=zammad483058387599"
I, [2020-04-09T12:56:04.480199 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:04.589319 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_ticket/_doc/6?pipeline=zammad483058387599"
I, [2020-04-09T12:56:04.747287 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:04.813629 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_ticket/_doc/5?pipeline=zammad483058387599"
I, [2020-04-09T12:56:04.843232 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:04.907982 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_ticket/_doc/4?pipeline=zammad483058387599"
I, [2020-04-09T12:56:04.932965 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:04.991941 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_ticket/_doc/3?pipeline=zammad483058387599"
I, [2020-04-09T12:56:05.028246 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:05.119748 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_ticket/_doc/2?pipeline=zammad483058387599"
I, [2020-04-09T12:56:05.142308 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:05.203691 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_ticket/_doc/1?pipeline=zammad483058387599"
I, [2020-04-09T12:56:05.394356 #67-47112593147320]  INFO -- : # 201
  - took 3 seconds
 reload KnowledgeBase::Translation
  - started at 2020-04-09 12:56:05 UTC
  - took 0 seconds
 reload KnowledgeBase::Category::Translation
  - started at 2020-04-09 12:56:05 UTC
  - took 0 seconds
 reload KnowledgeBase::Answer::Translation
  - started at 2020-04-09 12:56:05 UTC
  - took 0 seconds
 reload User
  - started at 2020-04-09 12:56:05 UTC
I, [2020-04-09T12:56:05.504034 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_user/_doc/6?pipeline=zammad483058387599"
I, [2020-04-09T12:56:05.691968 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:05.729491 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_user/_doc/5?pipeline=zammad483058387599"
I, [2020-04-09T12:56:06.053380 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:06.088693 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_user/_doc/4?pipeline=zammad483058387599"
I, [2020-04-09T12:56:06.163647 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:06.199187 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_user/_doc/3?pipeline=zammad483058387599"
I, [2020-04-09T12:56:06.374689 #67-47112593147320]  INFO -- : # 201
I, [2020-04-09T12:56:06.392097 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_user/_doc/2?pipeline=zammad483058387599"
I, [2020-04-09T12:56:06.438394 #67-47112593147320]  INFO -- : # 201
  - took 0 seconds
 reload StatsStore
  - started at 2020-04-09 12:56:06 UTC
I, [2020-04-09T12:56:06.486992 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_stats_store/_doc/2?pipeline=zammad483058387599"
I, [2020-04-09T12:56:06.563192 #67-47112593147320]  INFO -- : # 200
I, [2020-04-09T12:56:06.637401 #67-47112593147320]  INFO -- : # curl -X post "http://zammad-master:9200/zammad_production_stats_store/_doc/1?pipeline=zammad483058387599"
I, [2020-04-09T12:56:06.663280 #67-47112593147320]  INFO -- : # 200
  - took 0 seconds
zammad@zammad-2nd-0:~$

afterwards I opened up the browser, entered my username/password combination:
image

so it seems, that nothing had really changed.

Just to be sure - the testing system I’ve created the database dump on is configured to run under subdomain-1.somedomain.com - the second test system I’m trying the upgrade procedure with (and where the dump was imported again) is available under subdomain-2.somedomain.com … but that should not be causing that issue, right?

I’ve already went through the upgrade procedures in the zammad-docker-compose repo … but as I’m not really doing an inplace upgrade that seems to not really be suitable for my upgrade path?

I’m using helm (some kind of package manager for kubernetes) and the zammad/zammad-helm chart to install zammad on our kubernetes environment. At the very end of the page at https://github.com/zammad/zammad-helm/tree/master/zammad there are also upgrading instructions, which I’ve followed - basically that is to dump/backup your data, delete everything, setup the new version, import the old data.

Referring to your question if its necessary to remove the containers before starting the new version - well, thats what the instructions on the helm chart are saying… :wink:

btw - I saw that you are located in Berlin - I’m from munich…so, I’m really not trying to break the flow here but if its easier to go in german with our little discussion we also could switch languages… :slight_smile:

6

Thanks for running the commands.
That’s indeed very odd.

Also, I’m sorry, but I’m missing the required knowledge of helm and kubernetes which is why I can’t help you further. I’ve tried to help you as far as I could within application context - in your case something seems to be very special (and I can’t figure it out).

Changing the FQDN shouldn’t be an issue at this point. Something’s breaking within your login routine. That shouldn’t happen at all. You could try to create a local account and then login to that. If it happens with your local accounts as well, it seems like something broke during importing.

Again, that’s just a guess. Technically everyone else using helm and similar should have this exact problem then. Earlier this month I’ve been working with docker-compose in combination with upgrading (3.1 to 3.3) with LDAP activated (and living data inside). The only difference to your scenario is that I haven’t removed the data and had a “normal” in place upgrade.

If a in place upgrade is no option within helm, I can’t tell why and how to solve this issue. I’m sorry.

Sorry, but this is a english only board.
Reason for this is that we want to help as many of our users as possible. If someone else has e.g. your problem, he’ll be able to find the solution (if there’s any). Switching to german would help very little people and would technically be similar to fullfilling a service contract.

I hope you can understand this. :slight_smile:

Maybe @thorsteneckel can help you on this if he wants spare time.
Above error messages might sound familiar to him - if we’re lucky I’m even wrong. :slight_smile:

7

Hey @MrGeneration - thanks for the strong investigation so far. I won’t find the time to look into things in the next time. Please involve Mantas where needed.
I had a glimpse look into the error message. I faced the same error a few weeks ago in my development system when working on the login functionality. However, it was more of a side effect of what I was doing and therefore I just reset my system. I’ll will work further on the login functionality over the next time and will let you know if I can reproduce it and if I was able to resolve it.
From what I can tell so far is that Zammad tries to read out the assets of the ActivityStream class and tries to create/find objects for the activity_stream_object_id of one of the records and fails. The corresponding activity_stream_object_id, ObjectLookup mapping would help.
That’s all I can offer for now.

8

ok MrGeneration - thank you very much for your help anyways. I highly appreciate it!
Regarding the usage of helm / docker - I strongly believe that most installations are still done directly into a vm or straight onto some piece of hardware…maybe I can further down the point the error occures.
If so, I will let you guys know

9

Hey thorsteneckel, I would really like to provide you with any information you need. You are talking about activity_stream_object_id, ObjectLookup mapping - I’m sorry but I dont have a clue where to find this piece of information… can you give me some additional instructions?

10

I’d love to have a developer have a look.
We currently suspect a faulty backup.

As shipping your original backup is rather bad, would it be possible to create a playground with ldap authentication enabled. Beside of that without any productive data - you can ship some test tickets of course.

Then backup that instance and try the restoration. If this brings above issue as well, we’d love to have the playground backup to fiddle around.

If possible you can also check to backup your old instance and try restoring again.

In any case for checking your backup, we’ll need the codebase (so content of /opt/zammad/*) and the dump that has been created.

That’s, at this moment, anything I can offer. Sorry.

11

Hi,

like I’ve already mentioned (but maybe it was overseen in all the information masses) I still have a workin g zammad 3.2 instance up and running.
In a parallel setup I’ve tried the migration to 3.3 by installing a fresh zammad 3.3 environment, db dumping on 3.2 and importing to 3.3
So I can do dumps back and forth just as needed. The only thing I dont have atm is a ldap directory but there some few users existing directly in zammad.

As there is nothing important/secret in the affected zammad testing system, I have no problem in sharing that dump with you.

12

Okay, if it’s not too sensitive I’d like to ask for that dump as direct message on here.
If you need, I can also provide a space to upload it to.

13

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.