Unable to rebuild Elasticsearch index

Technical assistance
1

Infos:

  • Used Zammad version: 6.4.1
  • Used Zammad installation type: package (zammad-6.4.1-1742994139.560f34ed.centos9.src.rpm)
  • Used Elasticsearch version: 7.17.29
  • Operating system: RHEL 9.6

Expected behavior:

Successfully rebuild the index used by Elasticsearch.

Actual behavior:

When rebuilding the index, it errors out with the following:

Creating indexes... rake aborted!
NoMethodError: undefined method `[]' for nil:NilClass (NoMethodError)

    settings = Setting.get('es_model_settings')[model.name] || {}
                                               ^^^^^^^^^^^^
/opt/zammad/lib/search_index_backend.rb:880:in `model_settings'
/opt/zammad/lib/search_index_backend.rb:918:in `block in create_index'

Steps to reproduce the behavior:

  • Rebuild the index by running sudo zammad run rake zammad:searchindex:rebuild --trace.
** Invoke zammad:searchindex:rebuild (first_time)
** Invoke zammad:searchindex:version_supported (first_time)
** Invoke zammad:searchindex:configured (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute zammad:searchindex:configured
** Execute zammad:searchindex:version_supported
** Execute zammad:searchindex:rebuild
** Execute zammad:searchindex:drop
Dropping indexes... done.
** Execute zammad:searchindex:drop_pipeline
Deleting pipeline... done.
** Execute zammad:searchindex:create
Creating indexes... rake aborted!
NoMethodError: undefined method `[]' for nil:NilClass (NoMethodError)

    settings = Setting.get('es_model_settings')[model.name] || {}
                                               ^^^^^^^^^^^^
/opt/zammad/lib/search_index_backend.rb:880:in `model_settings'
/opt/zammad/lib/search_index_backend.rb:918:in `block in create_index'
/opt/zammad/lib/search_index_backend.rb:912:in `each'
/opt/zammad/lib/search_index_backend.rb:912:in `create_index'
/opt/zammad/lib/tasks/zammad/search_index_es.rake:17:in `block (3 levels) in <main>'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `block in execute'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `each'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `execute'
/opt/zammad/lib/tasks/zammad/search_index_es.rake:59:in `block (3 levels) in <main>'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `block in execute'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `each'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `execute'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `synchronize'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:188:in `invoke'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:188:in `invoke_task'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `block (2 levels) in top_level'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `each'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `block in top_level'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:147:in `run_with_threads'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:132:in `top_level'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:83:in `block in run'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:214:in `standard_exception_handling'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:80:in `run'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/opt/zammad/vendor/bundle/ruby/3.2.0/bin/rake:25:in `load'
/opt/zammad/vendor/bundle/ruby/3.2.0/bin/rake:25:in `<top (required)>'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler/cli/exec.rb:58:in `load'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler/cli/exec.rb:58:in `kernel_load'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler/cli/exec.rb:23:in `run'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler/cli.rb:492:in `exec'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler/cli.rb:34:in `dispatch'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler/cli.rb:28:in `start'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/exe/bundle:37:in `block in <top (required)>'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/opt/zammad/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/exe/bundle:29:in `<top (required)>'
/opt/zammad/bin/bundle:121:in `load'
/opt/zammad/bin/bundle:121:in `<main>'
Tasks: TOP => zammad:searchindex:rebuild

Additional information:

  • Running sudo /opt/zammad/contrib/packager.io/postinstall.sh does not change the reproduced behavior.
# Enforcing Redis...
# Creating Redis bootstart
# Starting Redis server
# Redis server is running.
# (Re)creating init scripts
Nothing to do.
Nothing to do.
Nothing to do.
# Enabling Zammad on boot
# Stopping Zammad
# Clear cache...
# database.yml found. Updating db...
# Updating translations...
# No custom packages detected...
# Configuring Elasticsearch...
-> removing [ingest-attachment]...
-> Installing ingest-attachment
-> Downloading ingest-attachment from elastic
[=================================================] 100%  
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessClassInPackage.sun.java2d.cmm.kcms
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.security.SecurityPermission createAccessControlContext
See https://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

-> Installed ingest-attachment
-> Please restart Elasticsearch to activate any plugins installed
# Enforcing 0600 on database.yml ...
# Starting Zammad
# Creating webserver bootstart
# Restarting webserver nginx
####################################################################################

Add your fully qualified domain name or public IP to servername directive of
nginx, if this installation is done on a remote server. You have to change:
/etc/nginx/conf.d/zammad.conf and restart nginx process.
Otherwise just open http://localhost/ in your browser to start using Zammad.

Remember to enable selinux and firewall rules!

Use the following commands:
        setsebool httpd_can_network_connect on -P
        firewall-cmd --zone=public --add-service=http --permanent
        firewall-cmd --zone=public --add-service=https --permanent
        firewall-cmd --reload

####################################################################################
  • Checking for any pending database migrations does not yield any results.
$ sudo zammad run rails console
Loading production environment (Rails 7.1.5.1)
[1] pry(main)> ActiveRecord::Migration.check_all_pending!
=> nil
  • Upgrading Zammad to 6.5 does not change the reproduced behavior. We originally upgraded to 6.5 and encountered this error, but after reverting to a backup, the same error happens under 6.4.1.
  • Uninstalling Elasticsearch 7.17.29 and replacing it with 8.18.4 does not change the reproduced behavior.

Any help would be appreciated. Thanks!

2

Hi.

What is the output of Setting.get('es_model_settings')?

3

I ran the command in the rails console, and its output was nil.

$ sudo zammad run rails c
Loading production environment (Rails 7.1.5.1)
[1] pry(main)> Setting.get('es_model_settings')
=> nil

Running the command on our production Zammad instance has the same results, and that server is able to rebuild its index without any errors.

4

What’s the output of Setting.find_by(name: 'es_model_settings') on both servers?

5

On the working production server:

$ sudo zammad run rails c
Loading production environment (Rails 7.1.5.1)
[1] pry(main)> Setting.find_by(name: 'es_model_settings')
=> #<Setting:0x00007f92ea71c080
 id: 271,
 title: "Elasticsearch Model Configuration",
 name: "es_model_settings",
 area: "SearchIndex::Elasticsearch",
 description: "Define model configuration for Elasticsearch.",
 options: {},
 state_current: {"value"=>{}},
 state_initial: {"value"=>{}},
 frontend: false,
 preferences: {"online_service_disable"=>true},
 created_at: Fri, 24 Nov 2023 12:08:47.575000000 UTC +00:00,
 updated_at: Fri, 24 Nov 2023 12:08:47.575000000 UTC +00:00>

On the problematic dev server:

$ sudo zammad run rails c
Loading production environment (Rails 7.1.5.1)
[1] pry(main)> Setting.find_by(name: 'es_model_settings')
=> nil