Issue on rake searchindex:rebuild after 4.0 upgrade

Technical assistance
1

Infos:

  • Used Zammad version: 4.0.0-1618998319
  • Used Zammad installation type: YUM
  • Operating system: Red Hat Enterprise Linux Server 7.9

Expected behavior:

  • After upgrading to 4.0, I encountered this error in the Health Check API
    Failed to run background job #1 ‘SearchIndexJob’ X time(s) with xx attempt(s).

I ran rake searchindex:rebuild and encountered the error below.

Actual behavior:

  • The rake searchindex:rebuild should finish the process and rebuilt the cache

Steps to reproduce the behavior:

  • Update Zammad using the steps in the documentation: Updating Zammad

  • run zammad run rake searchindex:rebuild

** Invoke searchindex:rebuild (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke searchindex:configured (first_time)
** Invoke environment
** Execute searchindex:configured
** Invoke searchindex:version_supported (first_time)
** Invoke environment
** Execute searchindex:version_supported
** Execute searchindex:rebuild
** Execute searchindex:drop
drop indexes…done
** Execute searchindex:drop_pipeline
delete pipeline (pipeline)… done
** Execute searchindex:create
create indexes…done
** Execute searchindex:create_pipeline
create pipeline (pipeline)… done
** Execute searchindex:reload
reload data…
reload Chat::Session

  • started at 2021-04-28 07:22:52 UTC
  • took 0 seconds
    reload Cti::Log
  • started at 2021-04-28 07:22:52 UTC
  • took 0 seconds
    reload KnowledgeBase::Answer::Translation
  • started at 2021-04-28 07:22:52 UTC
  • took 0 seconds
    reload KnowledgeBase::Category::Translation
  • started at 2021-04-28 07:22:53 UTC
  • took 0 seconds
    reload KnowledgeBase::Translation
  • started at 2021-04-28 07:22:53 UTC
  • took 0 seconds
    reload Ticket::Priority
  • started at 2021-04-28 07:22:53 UTC
  • took 0 seconds
    reload Ticket::State
  • started at 2021-04-28 07:22:53 UTC
  • took 0 seconds
    reload Group
  • started at 2021-04-28 07:22:54 UTC
  • took 0 seconds
    reload Organization
  • started at 2021-04-28 07:22:54 UTC
  • took 0 seconds
    reload Ticket
  • started at 2021-04-28 07:22:55 UTC
  • took 645 seconds
    reload StatsStore
  • started at 2021-04-28 07:33:40 UTC

rake aborted!
Unable to send StatsStore.find(23).search_index_update_backend backend: #<NameError: uninitialized constant TicketArticle>
/opt/zammad/app/models/concerns/has_search_index_backend.rb:231:in rescue in block in search_index_reload' /opt/zammad/app/models/concerns/has_search_index_backend.rb:225:in block in search_index_reload’
/opt/zammad/app/models/concerns/has_search_index_backend.rb:220:in each' /opt/zammad/app/models/concerns/has_search_index_backend.rb:220:in search_index_reload’
/opt/zammad/lib/tasks/search_index_es.rake:48:in block (3 levels) in <main>' /opt/zammad/lib/tasks/search_index_es.rake:44:in each’
/opt/zammad/lib/tasks/search_index_es.rake:44:in block (2 levels) in <main>' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in block in execute’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in each' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in execute’
/opt/zammad/lib/tasks/search_index_es.rake:63:in block (2 levels) in <main>' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in block in execute’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in each' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in execute’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:214:in block in invoke_with_call_chain' /opt/zammad/vendor/ruby-2.6.6/lib/ruby/2.6.0/monitor.rb:235:in mon_synchronize’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in invoke_with_call_chain' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:183:in invoke’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:160:in invoke_task' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in block (2 levels) in top_level’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in each' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in block in top_level’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:125:in run_with_threads' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:110:in top_level’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:83:in block in run' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:186:in standard_exception_handling’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:80:in run' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/exe/rake:27:in <top (required)>’
/opt/zammad/vendor/bundle/ruby/2.6.0/bin/rake:23:in load' /opt/zammad/vendor/bundle/ruby/2.6.0/bin/rake:23:in <top (required)>’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in load' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in kernel_load’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in run' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in exec’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in invoke_command’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in dispatch' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in dispatch’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in start' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in start’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in block in <top (required)>' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in with_friendly_errors’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in <top (required)>' /opt/zammad/bin/bundle:3:in load’
/opt/zammad/bin/bundle:3:in `’

Caused by:
NameError: uninitialized constant TicketArticle
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/active_support.rb:74:in block in load_missing_constant' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/active_support.rb:8:in without_bootsnap_cache’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/active_support.rb:74:in rescue in load_missing_constant' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/active_support.rb:56:in load_missing_constant’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.5/lib/active_support/inflector/methods.rb:283:in const_get' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.5/lib/active_support/inflector/methods.rb:283:in block in constantize’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.5/lib/active_support/inflector/methods.rb:281:in each' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.5/lib/active_support/inflector/methods.rb:281:in inject’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.5/lib/active_support/inflector/methods.rb:281:in constantize' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.5/lib/active_support/core_ext/string/inflections.rb:68:in constantize’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.4.5/lib/active_record/associations/belongs_to_polymorphic_association.rb:9:in klass' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.4.5/lib/active_record/associations/belongs_to_association.rb:77:in find_target?’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.4.5/lib/active_record/associations/association.rb:159:in load_target' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.4.5/lib/active_record/associations/association.rb:46:in reload’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.4.5/lib/active_record/associations/singular_association.rb:9:in reader' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.4.5/lib/active_record/associations/builder/association.rb:108:in stats_storable’
/opt/zammad/app/models/application_model/can_lookup_search_index_attributes.rb:102:in search_index_attribute_method' /opt/zammad/app/models/application_model/can_lookup_search_index_attributes.rb:38:in block in search_index_attribute_lookup’
/opt/zammad/app/models/application_model/can_lookup_search_index_attributes.rb:23:in each' /opt/zammad/app/models/application_model/can_lookup_search_index_attributes.rb:23:in search_index_attribute_lookup’
/opt/zammad/app/models/concerns/has_search_index_backend.rb:178:in search_index_update_backend' /opt/zammad/app/models/concerns/has_search_index_backend.rb:226:in block in search_index_reload’
/opt/zammad/app/models/concerns/has_search_index_backend.rb:220:in each' /opt/zammad/app/models/concerns/has_search_index_backend.rb:220:in search_index_reload’
/opt/zammad/lib/tasks/search_index_es.rake:48:in block (3 levels) in <main>' /opt/zammad/lib/tasks/search_index_es.rake:44:in each’
/opt/zammad/lib/tasks/search_index_es.rake:44:in block (2 levels) in <main>' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in block in execute’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in each' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in execute’
/opt/zammad/lib/tasks/search_index_es.rake:63:in block (2 levels) in <main>' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in block in execute’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in each' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in execute’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:214:in block in invoke_with_call_chain' /opt/zammad/vendor/ruby-2.6.6/lib/ruby/2.6.0/monitor.rb:235:in mon_synchronize’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in invoke_with_call_chain' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:183:in invoke’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:160:in invoke_task' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in block (2 levels) in top_level’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in each' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in block in top_level’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:125:in run_with_threads' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:110:in top_level’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:83:in block in run' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:186:in standard_exception_handling’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:80:in run' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/exe/rake:27:in <top (required)>’
/opt/zammad/vendor/bundle/ruby/2.6.0/bin/rake:23:in load' /opt/zammad/vendor/bundle/ruby/2.6.0/bin/rake:23:in <top (required)>’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in load' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in kernel_load’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in run' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in exec’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in invoke_command’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in dispatch' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in dispatch’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in start' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in start’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in block in <top (required)>' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in with_friendly_errors’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in <top (required)>' /opt/zammad/bin/bundle:3:in load’
/opt/zammad/bin/bundle:3:in `’

Caused by:
NameError: uninitialized constant TicketArticle
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/active_support.rb:58:in block in load_missing_constant' /opt/zammad/vendor/bundle/ruby/2.6.0/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/active_support.rb:16:in allow_bootsnap_retry’
/opt/zammad/vendor/bundle/ruby/2.6.0/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/active_support.rb:57:in `load_missing_constant’

2

Just to add more context to my issue:

  • I queried the StatsStore ID and it seems to be coming from the Dashboard:

    StatsStore.find(23)

    => #<StatsStore id: 23, key: "dashboard", data: {"StatsTicketReopen"=>{"used_for_average"=>0, "percent"=>0, "average_per_agent"=>0.0, "state"=>"supergood", "count"=>0, "total"=>0}, "StatsTicketInProcess"=>{"used_for_average"=>0, "average_per_agent"=>0.0, "state"=>"supergood", "in_process"=>0, "percent"=>0, "total"=>0}, "StatsTicketLoadMeasure"=>{"used_for_average"=>0.0, "average_per_agent"=>0.1, "percent"=>0.0, "state"=>"superbad", "own"=>0, "total"=>2, "average_per_agent_in_percent"=>0.0}, "StatsTicketChannelDistribution"=>{"channels"=>{"email"=>{"icon"=>"email", "inbound"=>0, "outbound"=>6, "inbound_in_percent"=>0, "outbound_in_percent"=>100}, "phone"=>{"icon"=>"phone", "inbound"=>0, "outbound"=>0, "inbound_in_percent"=>0, "outbound_in_percent"=>0}, "web"=>{"icon"=>"web", "inbound"=>0, "outbound"=>0, "inbound_in_percent"=>0, "outbound_in_percent"=>0}}}, "StatsTicketEscalation"=>{"used_for_average"=>0, "average_per_agent"=>0.0, "state"=>"supergood", "own"=>0, "total"=>0}, "StatsTicketWaitingTime"=>{"handling_time"=>0, "average_per_agent"=>0, "state"=>"supergood", "percent"=>0.0}}, created_by_id: 1, created_at: "2020-09-11 03:33:20", updated_at: "2020-09-11 05:20:39", stats_storable_type: "TicketArticle", stats_storable_id: 10>

  • Here’s my version of ElasticSearch
    "version" : {
    "number" : "7.12.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "78722783c38caa25a70982b5b042074cde5d3b3a",
    "build_date" : "2021-03-18T06:17:15.410153305Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
    },

  • Ingest-attachment plugin

    Name: ingest-attachment
    Description: Ingest processor that uses Apache Tika to extract contents
    Version: 7.12.0
    Elasticsearch Version: 7.12.0
    Java Version: 1.8

3

I did another upgrade (slightly newer version of 4.x) on a separate server:

Infos:

  • Used Zammad version: 4.0.0-1620154657.a92b45f6.centos7
  • Used Zammad installation type: YUM
  • Operating system: CentOS Linux 7

I am able to index the Users but I got stuck on StatsStore again with a different ID

uat
uat960×1008 41.6 KB

And I am getting this issue on the API Health check:

Stuck data privacy task (ID 4) detected.

1
11580×384 50.6 KB

4

It seems my StatsStore reindexing problem is an isolated issue, after checking the forums, I could not see a similar problem after upgrading to 4.0.

I tried deleting the rouge index but it broke the Zammad indexing all together.

5

Did you create a backup like described in this page Backup and Restore — Zammad documentation before upgrading?

I had a similar problem where the command zammad run rake searchindex:rebuild couldn’t finish. I rolled back, created the backup and then upgraded again. The second time the command ran without errors.

6

Hi,
to solve this issue, use this steps:

  1. Destroy all temporarily generated stats (will be renewed in step 2):

zammad run rails r "StatsStore.delete_all"

  1. Generate new statistics

zammad run rails r "Stats.generate"

  1. Try to rebuild your elastic search index:

zammad run rake searchindex:rebuild

1 Like
7

Hi Rolf,

Thanks for the explanation. These steps solved my indexing issue.

8

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.