[solved] Elastic Search not able to find contacts or link organisations

  • Used Zammad version: 2.9.
  • Used Zammad installation source: package
  • Operating system: Centos 7
  • Browser + version: Chrome + Firefox

Expected behavior:

  • Search and link a customer to an organisation.
  • Search for a new customer

Actual behavior:

  • no results found

Steps to reproduce the behavior:

  • as Above.

    I, [2019-02-26T12:48:56.162046 #6179-47007779625620] INFO – : Enqueued SearchIndexJob (Job ID: a05ccff7-0c7e-4abf-8e53-20d79755715a) to DelayedJob(default) at 2019-02-26 10:48:59 UTC with arguments: “User”, 339
    I, [2019-02-26T12:48:56.163527 #6179-47007779625620] INFO – : 2019-02-26T12:48:56+0200: [Worker(host:web.domain pid:6179)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=381991) (queue=default) COMPLETED after 0.0294
    I, [2019-02-26T12:48:56.166855 #6179-47007779625620] INFO – : 2019-02-26T12:48:56+0200: [Worker(host:web.domain pid:6179)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=381992) (queue=default) RUNNING
    I, [2019-02-26T12:48:56.180013 #6179-47007779625620] INFO – : Performing SearchIndexJob (Job ID: ca5ca18e-8162-48ab-a303-2ce0f907b817) from DelayedJob(default) with arguments: “User”, 256
    I, [2019-02-26T12:48:56.185695 #6179-47007779625620] INFO – : # curl -X POST “http://127.0.0.1:9200/zammad_production/User/256?pipeline=zammad995361596052
    I, [2019-02-26T12:48:56.189658 #6179-47007779625620] INFO – : # 400
    E, [2019-02-26T12:48:56.190055 #6179-47007779625620] ERROR – : Unable to process POST request to elasticsearch URL ‘http://127.0.0.1:9200/zammad_production/User/256?pipeline=zammad995361596052’. Check the response and payload for detailed information:

    Response:
    #<UserAgent::Result:0x00007f240d196b20 @success=false, @body="{“error”:{“root_cause”:[{“type”:“illegal_argument_exception”,“reason”:“pipeline with id [zammad995361596052] does not exist”}],“type”:“illegal_argument_exception”,“reason”:“pipeline with id [zammad995361596052] does not exist”},“status”:400}", @data=nil, @code=“400”, @content_type=nil, @error=“Client Error: #<Net::HTTPBadRequest 400 Bad Request readbody=true>!”>

    Payload:
    {“id”=>256, “organization_id”=>3, “login”=>“support@email”, “firstname”=>“Support”, “lastname”=>"", “email”=>“support@email”, “web”=>"", “phone”=>"", “fax”=>"", “mobile”=>"", “department”=>"", “street”=>"", “zip”=>"", “city”=>"", “country”=>"", “address”=>"", “vip”=>false, “verified”=>false, “active”=>true, “note”=>"", “last_login”=>nil, “out_of_office”=>false, “out_of_office_start_at”=>nil, “out_of_office_end_at”=>nil, “out_of_office_replacement_id”=>nil, “preferences”=>{}, “updated_by_id”=>1, “created_by_id”=>256, “created_at”=>Fri, 01 Feb 2019 06:17:25 UTC +00:00, “updated_at”=>Tue, 26 Feb 2019 10:48:32 UTC +00:00, “organization”=>“Company”, “permissions”=>[“user_preferences.password”, “user_preferences.language”, “user_preferences.linked_accounts”, “user_preferences.avatar”, “ticket.customer”], “role_ids”=>[3]}

    Payload size: 0M
    E, [2019-02-26T12:48:56.190252 #6179-47007779625620] ERROR – : Error performing SearchIndexJob (Job ID: ca5ca18e-8162-48ab-a303-2ce0f907b817) from DelayedJob(default) in 10.12ms: RuntimeError (Unable to process POST request to elasticsearch URL ‘http://127.0.0.1:9200/zammad_production/User/256?pipeline=zammad995361596052’. Check the response and payload for detailed information:

    Response:
    #<UserAgent::Result:0x00007f240d196b20 @success=false, @body="{“error”:{“root_cause”:[{“type”:“illegal_argument_exception”,“reason”:“pipeline with id [zammad995361596052] does not exist”}],“type”:“illegal_argument_exception”,“reason”:“pipeline with id [zammad995361596052] does not exist”},“status”:400}", @data=nil, @code=“400”, @content_type=nil, @error=“Client Error: #<Net::HTTPBadRequest 400 Bad Request readbody=true>!”>

    Payload:
    {“id”=>256, “organization_id”=>3, “login”=>“support@email”, “firstname”=>“Support”, “lastname”=>"", “email”=>“support@email”, “web”=>"", “phone”=>"", “fax”=>"", “mobile”=>"", “department”=>"", “street”=>"", “zip”=>"", “city”=>"", “country”=>"", “address”=>"", “vip”=>false, “verified”=>false, “active”=>true, “note”=>"", “last_login”=>nil, “out_of_office”=>false, “out_of_office_start_at”=>nil, “out_of_office_end_at”=>nil, “out_of_office_replacement_id”=>nil, “preferences”=>{}, “updated_by_id”=>1, “created_by_id”=>256, “created_at”=>Fri, 01 Feb 2019 06:17:25 UTC +00:00, “updated_at”=>Tue, 26 Feb 2019 10:48:32 UTC +00:00, “organization”=>“Company”, “permissions”=>[“user_preferences.password”, “user_preferences.language”, “user_preferences.linked_accounts”, “user_preferences.avatar”, “ticket.customer”], “role_ids”=>[3]}

    Payload size: 0M):
    /opt/zammad/lib/search_index_backend.rb:229:in add' /opt/zammad/app/models/concerns/has_search_index_backend.rb:66:insearch_index_update_backend’
    /opt/zammad/app/jobs/search_index_job.rb:12:in perform' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activejob-5.1.6.1/lib/active_job/execution.rb:37:inblock in perform_now’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:108:in block in run_callbacks' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/i18n-1.1.1/lib/i18n.rb:284:inwith_locale’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activejob-5.1.6.1/lib/active_job/translation.rb:7:in block (2 levels) in <module:Translation>' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:117:ininstance_exec’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:117:in block in run_callbacks' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activejob-5.1.6.1/lib/active_job/logging.rb:24:inblock (4 levels) in module:Logging
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/notifications.rb:166:in block in instrument' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/notifications/instrumenter.rb:21:ininstrument’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/notifications.rb:166:in instrument' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activejob-5.1.6.1/lib/active_job/logging.rb:23:inblock (3 levels) in module:Logging
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activejob-5.1.6.1/lib/active_job/logging.rb:44:in block in tag_logger' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/tagged_logging.rb:69:inblock in tagged’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/tagged_logging.rb:26:in tagged' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/tagged_logging.rb:69:intagged’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activejob-5.1.6.1/lib/active_job/logging.rb:44:in tag_logger' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activejob-5.1.6.1/lib/active_job/logging.rb:20:inblock (2 levels) in module:Logging
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:117:in instance_exec' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:117:inblock in run_callbacks’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:135:in run_callbacks' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activejob-5.1.6.1/lib/active_job/execution.rb:33:inperform_now’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activejob-5.1.6.1/lib/active_job/execution.rb:22:in block in execute' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:108:inblock in run_callbacks’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activejob-5.1.6.1/lib/active_job/railtie.rb:26:in block (4 levels) in <class:Railtie>' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/execution_wrapper.rb:85:inwrap’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/reloader.rb:68:in block in wrap' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/execution_wrapper.rb:85:inwrap’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/reloader.rb:67:in wrap' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activejob-5.1.6.1/lib/active_job/railtie.rb:25:inblock (3 levels) in class:Railtie
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:117:in instance_exec' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:117:inblock in run_callbacks’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:135:in run_callbacks' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activejob-5.1.6.1/lib/active_job/execution.rb:20:inexecute’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activejob-5.1.6.1/lib/active_job/queue_adapters/delayed_job_adapter.rb:36:in perform' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/delayed_job-4.1.3/lib/delayed/backend/base.rb:81:inblock in invoke_job’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:61:in block in initialize' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:66:inexecute’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:40:in run_callbacks' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/delayed_job-4.1.3/lib/delayed/backend/base.rb:78:ininvoke_job’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:230:in block (2 levels) in run' /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/timeout.rb:93:inblock in timeout’
    /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/timeout.rb:103:in timeout' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:230:inblock in run’
    /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/benchmark.rb:308:in realtime' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:229:inrun’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:312:in block in reserve_and_run_one_job' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:61:inblock in initialize’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:66:in execute' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/delayed_job-4.1.3/lib/delayed/lifecycle.rb:40:inrun_callbacks’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:312:in reserve_and_run_one_job' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:213:inblock in work_off’
    /opt/zammad/vendor/bundle/ruby/2.4.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:212:in times' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/delayed_job-4.1.3/lib/delayed/worker.rb:212:inwork_off’
    /opt/zammad/app/models/scheduler.rb:367:in block (3 levels) in worker' /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/benchmark.rb:308:inrealtime’
    /opt/zammad/app/models/scheduler.rb:365:in block (2 levels) in worker' /opt/zammad/app/models/scheduler.rb:361:inloop’
    /opt/zammad/app/models/scheduler.rb:361:in block in worker' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:inblock in create_with_logging_context’
    E, [2019-02-26T12:48:56.190315 #6179-47007779625620] ERROR – : Retrying SearchIndexJob in 3 seconds, due to a StandardError. The original exception was nil.

Seems all searchindexjobs are failing too…

Payload size: 0M
E, [2019-02-26T13:03:22.333377 #14136-47007493228540] ERROR -- : Error performing SearchIndexJob (Job ID: 49934c8a-15f0-443f-8d31-47c0bae94f24) from DelayedJob(default) in 14.99ms: RuntimeError (Unable to process POST request to elasticsearch URL 'http://127.0.0.1:9200/zammad_production/Organization/3?pipeline=zammad995361596052'. Check the response and payload for detailed information: 

Response:
#<UserAgent::Result:0x00007fdb34b3aee0 @success=false, @body="{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [zammad995361596052] does not exist\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [zammad995361596052] does not exist\"},\"status\":400}", @data=nil, @code="400", @content_type=nil, @error="Client Error: #<Net::HTTPBadRequest 400 Bad Request readbody=true>!">
zammad run rake searchindex:rebuild
drop indexes...done
delete pipeline (pipeline)... done
create indexes...done
create pipeline (pipeline)... rake aborted!
Unable to process PUT request to elasticsearch URL 'http://127.0.0.1:9200/_ingest/pipeline/zammad995361596052'. Check the response and payload for detailed information: 

Response:
#<UserAgent::Result:0x00007ff0180078e0 @success=false, @body="{\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"No processor type exists with name [attachment]\",\"header\":{\"processor_type\":\"foreach\"}}],\"type\":\"parse_exception\",\"reason\":\"No processor type exists with name [attachment]\",\"header\":{\"processor_type\":\"foreach\"}},\"status\":400}", @data=nil, @code="400", @content_type=nil, @error="Client Error: #<Net::HTTPBadRequest 400 Bad Request readbody=true>!">

Payload:
{:description=>"Extract zammad-attachment information from arrays", :processors=>[{:foreach=>{:field=>"article", :ignore_failure=>true, :processor=>{:foreach=>{:field=>"_ingest._value.attachment", :ignore_failure=>true, :processor=>{:attachment=>{:target_field=>"_ingest._value", :field=>"_ingest._value._content", :ignore_failure=>true}}}}}}]}

Payload size: 0M
/opt/zammad/lib/search_index_backend.rb:116:in `block (2 levels) in processors'
/opt/zammad/lib/search_index_backend.rb:76:in `each'
/opt/zammad/lib/search_index_backend.rb:76:in `block in processors'
/opt/zammad/lib/search_index_backend.rb:73:in `each'
/opt/zammad/lib/search_index_backend.rb:73:in `processors'
/opt/zammad/lib/tasks/search_index_es.rake:70:in `block (2 levels) in <top (required)>'
/opt/zammad/lib/tasks/search_index_es.rake:50:in `block (2 levels) in <top (required)>'
/opt/zammad/lib/tasks/search_index_es.rake:147:in `block (2 levels) in <top (required)>'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/cli/exec.rb:74:in `load'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/cli/exec.rb:27:in `run'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/cli.rb:360:in `exec'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor.rb:369:in `dispatch'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/cli.rb:20:in `dispatch'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor/base.rb:444:in `start'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/cli.rb:10:in `start'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/exe/bundle:30:in `block in <top (required)>'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/friendly_errors.rb:121:in `with_friendly_errors'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/exe/bundle:22:in `<top (required)>'
/opt/zammad/bin/bundle:3:in `load'
/opt/zammad/bin/bundle:3:in `<main>'
Tasks: TOP => searchindex:rebuild
(See full trace by running task with --trace)
[2019-02-26T13:18:58,778][DEBUG][o.e.a.b.TransportBulkAction] [6lNx6DJ] failed to execute pipeline [zammad995361596052] for document [zammad_production/Organization/3]
java.lang.IllegalArgumentException: pipeline with id [zammad995361596052] does not exist
        at org.elasticsearch.ingest.PipelineExecutionService.getPipeline(PipelineExecutionService.java:194) ~[elasticsearch-5.6.15.jar:5.6.15]
        at org.elasticsearch.ingest.PipelineExecutionService.access$100(PipelineExecutionService.java:41) ~[elasticsearch-5.6.15.jar:5.6.15]
        at org.elasticsearch.ingest.PipelineExecutionService$2.doRun(PipelineExecutionService.java:88) [elasticsearch-5.6.15.jar:5.6.15]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) [elasticsearch-5.6.15.jar:5.6.15]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.15.jar:5.6.15]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

I had to remove the ‘ingest-attachment’ elastic search plugin, and then re-install it… once that was done I rebuilt the index and it seems to have recovered.

I will report if there are any further errors after the rebuild has completed.

1 Like

Thanks for the Feedback and comprehensive writeup

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.