ElasticSearch Problem

Hello I have a problem with ElasticSearch index.
i search here in the forum and try already everything but not successfull so i hope somebody can give me the correct tip.

U have installed Zammad Version 3.1.x on CentOs
everything runs fine exept the elasticsearch.

elasticsearch is installed, including the ingest-attachment plugin

i can get the elasticsearch statusinfo localhost:9200:

  version: "7.3.2"
  build_type: "rpm"
  lucene_version: "8.1.0"

if i try to rebuild the index i get the following error:

E, [2019-09-18T17:05:23.272733 #27890-47027212506520] ERROR -- : Retrying SearchIndexJob in #<Proc:0x0000558abff52220@/opt/zammad/app/jobs/search_index_job.rb:3 (lambda)> seconds, due to a StandardError. The original exception was nil.
I, [2019-09-18T17:05:23.273899 #27886-69833754397460]  INFO -- : # 0
E, [2019-09-18T17:05:23.274368 #27886-69833754397460] ERROR -- : Unable to process GET request to elasticsearch URL 'http://127.0.0.1:9200/zammad_production_user/_doc/_search'. Check the response and payload for detailed information: 

Response:
#<UserAgent::Result:0x00007f06d42cfbf0 @success=false, @body=nil, @data=nil, @code=0, @content_type=nil, @error="#<RuntimeError: Unable to process http call '#<Net::HTTPServiceUnavailable 503 Service Unavailable readbody=true>'>">

Payload:
{:from=>0, :size=>50, :sort=>[{"created_at"=>{:order=>"desc"}}, "_score"], :query=>{:bool=>{:must=>[{:query_string=>{:default_field=>"role_ids", :query=>"\"2\" OR \"3\""}}, {"query_string"=>{"query"=>"**", "default_operator"=>"AND", "analyze_wildcard"=>true}}]}}}

Payload size: 0M
E, [2019-09-18T17:05:23.274424 #27886-69833754397460] ERROR -- : Unable to process GET request to elasticsearch URL 'http://127.0.0.1:9200/zammad_production_user/_doc/_search'. Check the response and payload for detailed information: 

Response:
#<UserAgent::Result:0x00007f06d42cfbf0 @success=false, @body=nil, @data=nil, @code=0, @content_type=nil, @error="#<RuntimeError: Unable to process http call '#<Net::HTTPServiceUnavailable 503 Service Unavailable readbody=true>'>">

Payload:
{:from=>0, :size=>50, :sort=>[{"created_at"=>{:order=>"desc"}}, "_score"], :query=>{:bool=>{:must=>[{:query_string=>{:default_field=>"role_ids", :query=>"\"2\" OR \"3\""}}, {"query_string"=>{"query"=>"**", "default_operator"=>"AND", "analyze_wildcard"=>true}}]}}}

Payload size: 0M
I, [2019-09-18T17:05:23.276486 #27886-69833754397460]  INFO -- : Completed 200 OK in 40ms (Views: 0.3ms | ActiveRecord: 2.8ms)
I, [2019-09-18T17:05:23.286254 #27890-47027212506520]  INFO -- : Enqueued SearchIndexJob (Job ID: 356cbb53-8371-4bfa-ae83-cca355ce4bd9) to DelayedJob(default) at 2019-09-18 15:06:43 UTC with arguments: "StatsStore", 5
I, [2019-09-18T17:05:23.290000 #27890-47027212506520]  INFO -- : 2019-09-18T17:05:23+0200: [Worker(host:zammand.local pid:27890)] Job SearchIndexJob [356cbb53-8371-4bfa-ae83-cca355ce4bd9] from DelayedJob(default) with arguments: ["StatsStore", 5] (id=56212) (queue=default) COMPLETED after 0.0911
I, [2019-09-18T17:05:23.293924 #27890-47027212506520]  INFO -- : 2019-09-18T17:05:23+0200: [Worker(host:zammand.local pid:27890)] Job SearchIndexJob [dad8a5f5-93c2-4eb4-b083-10b5a67069ba] from DelayedJob(default) with arguments: ["StatsStore", 6] (id=56213) (queue=default) RUNNING
I, [2019-09-18T17:05:23.342531 #27890-47027212506520]  INFO -- : Performing SearchIndexJob (Job ID: dad8a5f5-93c2-4eb4-b083-10b5a67069ba) from DelayedJob(default) with arguments: "StatsStore", 6
I, [2019-09-18T17:05:23.367354 #27890-47027212506520]  INFO -- : # curl -X POST "http://127.0.0.1:9200/zammad_production_stats_store/_doc/6?pipeline=zammad537802081995" \
I, [2019-09-18T17:05:23.376841 #27890-47027212506520]  INFO -- : # 400
E, [2019-09-18T17:05:23.379154 #27890-47027212506520] ERROR -- : Unable to process POST request to elasticsearch URL 'http://127.0.0.1:9200/zammad_production_stats_store/_doc/6?pipeline=zammad537802081995'. Check the response and payload for detailed information: 

Response:
#<UserAgent::Result:0x0000558abe668f48 @success=false, @body="{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [zammad537802081995] does not exist\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [zammad537802081995] does not exist\"},\"status\":400}", @data=nil, @code="400", @content_type=nil, @error="Client Error: #<Net::HTTPBadRequest 400 Bad Request readbody=true>!">

Payload:
{"id"=>6, "stats_store_object_id"=>1, "o_id"=>10, "key"=>"dashboard", "related_o_id"=>nil, "related_stats_store_object_id"=>nil, "data"=>{"StatsTicketLoadMeasure"=>{"used_for_average"=>0.0, "average_per_agent"=>0.4, "percent"=>0.0, "state"=>"superbad", "own"=>0, "total"=>3, "average_per_agent_in_percent"=>0.0}, "StatsTicketWaitingTime"=>{"handling_time"=>0, "average_per_agent"=>0, "state"=>"supergood", "percent"=>0.0}, "StatsTicketEscalation"=>{"used_for_average"=>0, "average_per_agent"=>0.0, "state"=>"supergood", "own"=>0, "total"=>0}, "StatsTicketChannelDistribution"=>{"channels"=>{"email"=>{"icon"=>"email", "inbound"=>6, "outbound"=>5, "inbound_in_percent"=>85, "outbound_in_percent"=>100}, "phone"=>{"icon"=>"phone", "inbound"=>1, "outbound"=>0, "inbound_in_percent"=>14, "outbound_in_percent"=>0}, "twitter"=>{"icon"=>"twitter", "inbound"=>0, "outbound"=>0, "inbound_in_percent"=>0, "outbound_in_percent"=>0}, "facebook"=>{"icon"=>"facebook", "inbound"=>0, "outbound"=>0, "inbound_in_percent"=>0, "outbound_in_percent"=>0}}}, "StatsTicketInProcess"=>{"used_for_average"=>100.0, "average_per_agent"=>37.5, "state"=>"supergood", "in_process"=>1, "percent"=>100.0, "total"=>1}, "StatsTicketReopen"=>{"used_for_average"=>100.0, "percent"=>100.0, "average_per_agent"=>18.8, "state"=>"superbad", "count"=>1, "total"=>1}}, "created_by_id"=>1, "created_at"=>Fri, 13 Sep 2019 07:46:43 UTC +00:00, "updated_at"=>Wed, 18 Sep 2019 14:48:54 UTC +00:00, "stats_store_object"=>"User", "stats_store_object_ref"=>{"id"=>10, "organization_id"=>1, "login"=>"nick.hugentobler@osotec.ch", "firstname"=>"Nick", "lastname"=>"Hugentobler", "email"=>"nick.hugentobler@osotec.ch", "web"=>"", "phone"=>"", "fax"=>"", "mobile"=>"", "department"=>"", "street"=>"", "zip"=>"", "city"=>"", "country"=>"", "address"=>"", "vip"=>false, "verified"=>false, "active"=>true, "note"=>"", "last_login"=>Wed, 18 Sep 2019 11:26:11 UTC +00:00, "out_of_office"=>false, "out_of_office_start_at"=>nil, "out_of_office_end_at"=>nil, "out_of_office_replacement_id"=>nil, "preferences"=>{"notification_config"=>{"matrix"=>{"create"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>true, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "update"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>true, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "reminder_reached"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>false, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "escalation"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>false, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}}}, "locale"=>"de-de", "intro"=>true}, "updated_by_id"=>1, "created_by_id"=>6, "created_at"=>Fri, 13 Sep 2019 07:40:24 UTC +00:00, "updated_at"=>Wed, 18 Sep 2019 14:38:40 UTC +00:00, "organization"=>"osoTec GmbH", "permissions"=>["user_preferences", "user_preferences.password", "user_preferences.notifications", "user_preferences.access_token", "user_preferences.language", "user_preferences.linked_accounts", "user_preferences.device", "user_preferences.avatar", "user_preferences.calendar", "user_preferences.out_of_office", "ticket.agent", "chat.agent", "cti.agent", "knowledge_base.reader"], "role_ids"=>[2]}}

Payload size: 0M
E, [2019-09-18T17:05:23.379507 #27890-47027212506520] ERROR -- : Error performing SearchIndexJob (Job ID: dad8a5f5-93c2-4eb4-b083-10b5a67069ba) from DelayedJob(default) in 36.76ms: RuntimeError (Unable to process POST request to elasticsearch URL 'http://127.0.0.1:9200/zammad_production_stats_store/_doc/6?pipeline=zammad537802081995'. Check the response and payload for detailed information: 

Response:
#<UserAgent::Result:0x0000558abe668f48 @success=false, @body="{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [zammad537802081995] does not exist\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [zammad537802081995] does not exist\"},\"status\":400}", @data=nil, @code="400", @content_type=nil, @error="Client Error: #<Net::HTTPBadRequest 400 Bad Request readbody=true>!">

Payload:
{"id"=>6, "stats_store_object_id"=>1, "o_id"=>10, "key"=>"dashboard", "related_o_id"=>nil, "related_stats_store_object_id"=>nil, "data"=>{"StatsTicketLoadMeasure"=>{"used_for_average"=>0.0, "average_per_agent"=>0.4, "percent"=>0.0, "state"=>"superbad", "own"=>0, "total"=>3, "average_per_agent_in_percent"=>0.0}, "StatsTicketWaitingTime"=>{"handling_time"=>0, "average_per_agent"=>0, "state"=>"supergood", "percent"=>0.0}, "StatsTicketEscalation"=>{"used_for_average"=>0, "average_per_agent"=>0.0, "state"=>"supergood", "own"=>0, "total"=>0}, "StatsTicketChannelDistribution"=>{"channels"=>{"email"=>{"icon"=>"email", "inbound"=>6, "outbound"=>5, "inbound_in_percent"=>85, "outbound_in_percent"=>100}, "phone"=>{"icon"=>"phone", "inbound"=>1, "outbound"=>0, "inbound_in_percent"=>14, "outbound_in_percent"=>0}, "twitter"=>{"icon"=>"twitter", "inbound"=>0, "outbound"=>0, "inbound_in_percent"=>0, "outbound_in_percent"=>0}, "facebook"=>{"icon"=>"facebook", "inbound"=>0, "outbound"=>0, "inbound_in_percent"=>0, "outbound_in_percent"=>0}}}, "StatsTicketInProcess"=>{"used_for_average"=>100.0, "average_per_agent"=>37.5, "state"=>"supergood", "in_process"=>1, "percent"=>100.0, "total"=>1}, "StatsTicketReopen"=>{"used_for_average"=>100.0, "percent"=>100.0, "average_per_agent"=>18.8, "state"=>"superbad", "count"=>1, "total"=>1}}, "created_by_id"=>1, "created_at"=>Fri, 13 Sep 2019 07:46:43 UTC +00:00, "updated_at"=>Wed, 18 Sep 2019 14:48:54 UTC +00:00, "stats_store_object"=>"User", "stats_store_object_ref"=>{"id"=>10, "organization_id"=>1, "login"=>"nick.hugentobler@osotec.ch", "firstname"=>"Nick", "lastname"=>"Hugentobler", "email"=>"nick.hugentobler@osotec.ch", "web"=>"", "phone"=>"", "fax"=>"", "mobile"=>"", "department"=>"", "street"=>"", "zip"=>"", "city"=>"", "country"=>"", "address"=>"", "vip"=>false, "verified"=>false, "active"=>true, "note"=>"", "last_login"=>Wed, 18 Sep 2019 11:26:11 UTC +00:00, "out_of_office"=>false, "out_of_office_start_at"=>nil, "out_of_office_end_at"=>nil, "out_of_office_replacement_id"=>nil, "preferences"=>{"notification_config"=>{"matrix"=>{"create"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>true, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "update"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>true, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "reminder_reached"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>false, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "escalation"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>false, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}}}, "locale"=>"de-de", "intro"=>true}, "updated_by_id"=>1, "created_by_id"=>6, "created_at"=>Fri, 13 Sep 2019 07:40:24 UTC +00:00, "updated_at"=>Wed, 18 Sep 2019 14:38:40 UTC +00:00, "organization"=>"osoTec GmbH", "permissions"=>["user_preferences", "user_preferences.password", "user_preferences.notifications", "user_preferences.access_token", "user_preferences.language", "user_preferences.linked_accounts", "user_preferences.device", "user_preferences.avatar", "user_preferences.calendar", "user_preferences.out_of_office", "ticket.agent", "chat.agent", "cti.agent", "knowledge_base.reader"], "role_ids"=>[2]}}

Payload size: 0M):
/opt/zammad/lib/search_index_backend.rb:230:in `add'
/opt/zammad/app/models/concerns/has_search_index_backend.rb:66:in `search_index_update_backend'
/opt/zammad/app/jobs/search_index_job.rb:14:in `perform'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activejob-5.2.3/lib/active_job/execution.rb:39:in `block in perform_now'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/i18n-1.6.0/lib/i18n.rb:297:in `with_locale'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activejob-5.2.3/lib/active_job/translation.rb:9:in `block (2 levels) in <module:Translation>'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:118:in `instance_exec'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activejob-5.2.3/lib/active_job/logging.rb:26:in `block (4 levels) in <module:Logging>'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `block in instrument'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `instrument'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activejob-5.2.3/lib/active_job/logging.rb:25:in `block (3 levels) in <module:Logging>'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activejob-5.2.3/lib/active_job/logging.rb:46:in `block in tag_logger'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `block in tagged'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:28:in `tagged'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `tagged'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activejob-5.2.3/lib/active_job/logging.rb:46:in `tag_logger'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activejob-5.2.3/lib/active_job/logging.rb:22:in `block (2 levels) in <module:Logging>'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:118:in `instance_exec'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:136:in `run_callbacks'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activejob-5.2.3/lib/active_job/execution.rb:38:in `perform_now'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activejob-5.2.3/lib/active_job/execution.rb:24:in `block in execute'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activejob-5.2.3/lib/active_job/railtie.rb:28:in `block (4 levels) in <class:Railtie>'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/execution_wrapper.rb:87:in `wrap'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/reloader.rb:73:in `block in wrap'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/execution_wrapper.rb:87:in `wrap'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/reloader.rb:72:in `wrap'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activejob-5.2.3/lib/active_job/railtie.rb:27:in `block (3 levels) in <class:Railtie>'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:118:in `instance_exec'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:136:in `run_callbacks'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activejob-5.2.3/lib/active_job/execution.rb:22:in `execute'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activejob-5.2.3/lib/active_job/queue_adapters/delayed_job_adapter.rb:42:in `perform'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/delayed_job-4.1.7/lib/delayed/backend/base.rb:81:in `block in invoke_job'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/delayed_job-4.1.7/lib/delayed/lifecycle.rb:61:in `block in initialize'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/delayed_job-4.1.7/lib/delayed/lifecycle.rb:66:in `execute'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/delayed_job-4.1.7/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/delayed_job-4.1.7/lib/delayed/backend/base.rb:78:in `invoke_job'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/delayed_job-4.1.7/lib/delayed/worker.rb:230:in `block (2 levels) in run'
/opt/zammad/vendor/ruby-2.5.5/lib/ruby/2.5.0/timeout.rb:93:in `block in timeout'
/opt/zammad/vendor/ruby-2.5.5/lib/ruby/2.5.0/timeout.rb:103:in `timeout'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/delayed_job-4.1.7/lib/delayed/worker.rb:230:in `block in run'
/opt/zammad/vendor/ruby-2.5.5/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/delayed_job-4.1.7/lib/delayed/worker.rb:229:in `run'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/delayed_job-4.1.7/lib/delayed/worker.rb:312:in `block in reserve_and_run_one_job'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/delayed_job-4.1.7/lib/delayed/lifecycle.rb:61:in `block in initialize'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/delayed_job-4.1.7/lib/delayed/lifecycle.rb:66:in `execute'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/delayed_job-4.1.7/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/delayed_job-4.1.7/lib/delayed/worker.rb:312:in `reserve_and_run_one_job'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/delayed_job-4.1.7/lib/delayed/worker.rb:213:in `block in work_off'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/delayed_job-4.1.7/lib/delayed/worker.rb:212:in `times'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/delayed_job-4.1.7/lib/delayed/worker.rb:212:in `work_off'
/opt/zammad/app/models/scheduler.rb:367:in `block (3 levels) in worker'
/opt/zammad/vendor/ruby-2.5.5/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
/opt/zammad/app/models/scheduler.rb:365:in `block (2 levels) in worker'
/opt/zammad/app/models/scheduler.rb:361:in `loop'
/opt/zammad/app/models/scheduler.rb:361:in `block in worker'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
E, [2019-09-18T17:05:23.379633 #27890-47027212506520] ERROR -- : Retrying SearchIndexJob in #<Proc:0x0000558abff52220@/opt/zammad/app/jobs/search_index_job.rb:3 (lambda)> seconds, due to a StandardError. The original exception was nil.

so the index is not builded and elasticsearch is not working, i not see any users ect.

what can i do to fix this problem?

thanks for any help in advance!

Sorry but I have no clue what you did there…

How did you configure elasticsearch, is it really running on localhost?
Is there a proxy configured on the system…?

Now really, I’ve never seen Error 400 and 503 in that combination.
If you have no proxy configured, please reindex your searchindex, as indices seem to be missing:

zammad run rake searchindex:rebuild

Hello,
sorry i think i was copy the wrong logentrys.

there is no proxy in the system.

I just try it now again:

elasticsearch.yml: (only the active entrys)

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 127.0.0.1
http.port: 9200
gateway.recover_after_nodes: 3
action.destructive_requires_name: true

Console Log:


[root@zammand ~]# zammad run rails r "Setting.set('es_url', 'http://localhost:9200')"
[root@zammand ~]# systemctl restart elasticsearch
[root@zammand ~]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Don 2019-09-19 13:42:34 CEST; 1min 17s ago
     Docs: http://www.elastic.co
 Main PID: 33135 (java)
    Tasks: 59
   CGroup: /system.slice/elasticsearch.service
           ├─33135 /usr/share/elasticsearch/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddres...
           └─33228 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Sep 19 13:42:34 zammand.local systemd[1]: Started Elasticsearch.
Sep 19 13:42:35 zammand.local elasticsearch[33135]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Hint: Some lines were ellipsized, use -l to show in full.
[root@zammand ~]# systemctl status zammad
● zammad.service
   Loaded: loaded (/etc/systemd/system/zammad.service; enabled; vendor preset: disabled)
   Active: active (running) since Mit 2019-09-18 13:41:11 CEST; 24h ago
 Main PID: 27884 (sleep)
    Tasks: 1
   CGroup: /system.slice/zammad.service
           └─27884 /bin/sleep infinity

Sep 18 13:41:11 zammand.local systemd[1]: Started zammad.service.

[root@zammand ~]# zammad run rake searchindex:rebuild
drop indexes...done
delete pipeline (pipeline)... rake aborted!
Unable to process DELETE request to elasticsearch URL 'http://localhost:9200/_ingest/pipeline/zammad537802081995'. Check the response and payload for detailed information:

Response:
#<UserAgent::Result:0x000055c072430368 @success=false, @body=nil, @data=nil, @code=0, @content_type=nil, @error="#<RuntimeError: Unable to process http call '#<Net::HTTPServiceUnavailable 503 Service Unavailable readbody=true>'>">

Payload:
nil

Payload size: 0M
/opt/zammad/lib/search_index_backend.rb:96:in `block (2 levels) in processors'
/opt/zammad/lib/search_index_backend.rb:79:in `each'
/opt/zammad/lib/search_index_backend.rb:79:in `block in processors'
/opt/zammad/lib/search_index_backend.rb:76:in `each'
/opt/zammad/lib/search_index_backend.rb:76:in `processors'
/opt/zammad/lib/tasks/search_index_es.rake:126:in `block (2 levels) in <main>'
/opt/zammad/lib/tasks/search_index_es.rake:23:in `block (2 levels) in <main>'
/opt/zammad/lib/tasks/search_index_es.rake:151:in `block (2 levels) in <main>'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli/exec.rb:74:in `load'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli/exec.rb:27:in `run'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli.rb:360:in `exec'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor.rb:369:in `dispatch'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli.rb:20:in `dispatch'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor/base.rb:444:in `start'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli.rb:10:in `start'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/exe/bundle:30:in `block in <top (required)>'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/friendly_errors.rb:121:in `with_friendly_errors'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/exe/bundle:22:in `<top (required)>'
/opt/zammad/bin/bundle:3:in `load'
/opt/zammad/bin/bundle:3:in `<main>'
Tasks: TOP => searchindex:rebuild
(See full trace by running task with --trace)
[root@zammand ~]#

from Production.log:

Payload size: 0M
I, [2019-09-19T13:55:14.020278 #34075-46976542844400]  INFO -- : # curl -X GET "http://localhost:9200"
I, [2019-09-19T13:55:14.026806 #34075-46976542844400]  INFO -- : # 200
I, [2019-09-19T13:55:14.027098 #34075-46976542844400]  INFO -- : # curl -X DELETE "http://localhost:9200/_ingest/pipeline/zammad537802081995"
I, [2019-09-19T13:55:15.431543 #27890-47027212506520]  INFO -- : 2019-09-19T13:55:15+0200: [Worker(host:zammand.local pid:27890)] Job SearchIndexJob [acf678c6-7c0a-4858-86b6-22a5490af3b8] from DelayedJob(default) with arguments: ["StatsStore", 4] (id=56422) (queue=default) RUNNING
I, [2019-09-19T13:55:15.471148 #27890-47027212506520]  INFO -- : Performing SearchIndexJob (Job ID: acf678c6-7c0a-4858-86b6-22a5490af3b8) from DelayedJob(default) with arguments: "StatsStore", 4
I, [2019-09-19T13:55:15.491560 #27890-47027212506520]  INFO -- : # curl -X POST "http://localhost:9200/zammad_production_stats_store/_doc/4?pipeline=zammad537802081995" \
I, [2019-09-19T13:55:15.501130 #27890-47027212506520]  INFO -- : # 400
E, [2019-09-19T13:55:15.503573 #27890-47027212506520] ERROR -- : Unable to process POST request to elasticsearch URL 'http://localhost:9200/zammad_production_stats_store/_doc/4?pipeline=zammad537802081995'. Check the response and payload for detailed information: 

Response:
#<UserAgent::Result:0x00007fad687ddcd8 @success=false, @body="{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [zammad537802081995] does not exist\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [zammad537802081995] does not exist\"},\"status\":400}", @data=nil, @code="400", @content_type=nil, @error="Client Error: #<Net::HTTPBadRequest 400 Bad Request readbody=true>!">

Payload:
{"id"=>4, "stats_store_object_id"=>1, "o_id"=>2, "key"=>"dashboard", "related_o_id"=>nil, "related_stats_store_object_id"=>nil, "data"=>{"StatsTicketLoadMeasure"=>{"used_for_average"=>33.3, "average_per_agent"=>0.4, "percent"=>33.3, "state"=>"supergood", "own"=>1, "total"=>3, "average_per_agent_in_percent"=>266.4}, "StatsTicketWaitingTime"=>{"handling_time"=>0, "average_per_agent"=>0, "state"=>"supergood", "percent"=>0.0}, "StatsTicketEscalation"=>{"used_for_average"=>0, "average_per_agent"=>0.0, "state"=>"supergood", "own"=>0, "total"=>0}, "StatsTicketChannelDistribution"=>{"channels"=>{"email"=>{"icon"=>"email", "inbound"=>6, "outbound"=>5, "inbound_in_percent"=>85, "outbound_in_percent"=>100}, "phone"=>{"icon"=>"phone", "inbound"=>1, "outbound"=>0, "inbound_in_percent"=>14, "outbound_in_percent"=>0}, "twitter"=>{"icon"=>"twitter", "inbound"=>0, "outbound"=>0, "inbound_in_percent"=>0, "outbound_in_percent"=>0}, "facebook"=>{"icon"=>"facebook", "inbound"=>0, "outbound"=>0, "inbound_in_percent"=>0, "outbound_in_percent"=>0}}}, "StatsTicketInProcess"=>{"used_for_average"=>50.0, "average_per_agent"=>18.8, "state"=>"supergood", "in_process"=>1, "percent"=>50.0, "total"=>2}, "StatsTicketReopen"=>{"used_for_average"=>0.0, "percent"=>0.0, "average_per_agent"=>18.8, "state"=>"supergood", "count"=>0, "total"=>1}}, "created_by_id"=>1, "created_at"=>Fri, 13 Sep 2019 07:11:52 UTC +00:00, "updated_at"=>Thu, 19 Sep 2019 11:47:24 UTC +00:00, "stats_store_object"=>"User", "stats_store_object_ref"=>{"id"=>2, "organization_id"=>1, "login"=>"stefan.moeller@osotec.ch", "firstname"=>"Stefan", "lastname"=>"Möller", "email"=>"stefan.moeller@osotec.ch", "web"=>"", "phone"=>"", "fax"=>"", "mobile"=>"", "department"=>"", "street"=>"", "zip"=>"", "city"=>"", "country"=>"", "address"=>"", "vip"=>false, "verified"=>false, "active"=>true, "note"=>"", "last_login"=>Wed, 18 Sep 2019 14:38:40 UTC +00:00, "out_of_office"=>false, "out_of_office_start_at"=>nil, "out_of_office_end_at"=>nil, "out_of_office_replacement_id"=>nil, "preferences"=>{"tickets_closed"=>0, "tickets_open"=>1, "notification_config"=>{"matrix"=>{"create"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>true, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "update"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>true, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "reminder_reached"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>false, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "escalation"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>false, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}}}, "locale"=>"de-de", "intro"=>true}, "updated_by_id"=>1, "created_by_id"=>1, "created_at"=>Thu, 29 Aug 2019 16:10:39 UTC +00:00, "updated_at"=>Wed, 18 Sep 2019 14:38:40 UTC +00:00, "organization"=>"osoTec GmbH", "permissions"=>["user_preferences", "user_preferences.password", "user_preferences.notifications", "user_preferences.access_token", "user_preferences.language", "user_preferences.linked_accounts", "user_preferences.device", "user_preferences.avatar", "user_preferences.calendar", "user_preferences.out_of_office", "ticket.agent", "chat.agent", "cti.agent", "knowledge_base.reader"], "role_ids"=>[2]}}

Payload size: 0M
E, [2019-09-19T13:55:15.503957 #27890-47027212506520] ERROR -- : Error performing SearchIndexJob (Job ID: acf678c6-7c0a-4858-86b6-22a5490af3b8) from DelayedJob(default) in 32.59ms: RuntimeError (Unable to process POST request to elasticsearch URL 'http://localhost:9200/zammad_production_stats_store/_doc/4?pipeline=zammad537802081995'. Check the response and payload for detailed information: 

Response:
#<UserAgent::Result:0x00007fad687ddcd8 @success=false, @body="{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [zammad537802081995] does not exist\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [zammad537802081995] does not exist\"},\"status\":400}", @data=nil, @code="400", @content_type=nil, @error="Client Error: #<Net::HTTPBadRequest 400 Bad Request readbody=true>!">

Payload:
{"id"=>4, "stats_store_object_id"=>1, "o_id"=>2, "key"=>"dashboard", "related_o_id"=>nil, "related_stats_store_object_id"=>nil, "data"=>{"StatsTicketLoadMeasure"=>{"used_for_average"=>33.3, "average_per_agent"=>0.4, "percent"=>33.3, "state"=>"supergood", "own"=>1, "total"=>3, "average_per_agent_in_percent"=>266.4}, "StatsTicketWaitingTime"=>{"handling_time"=>0, "average_per_agent"=>0, "state"=>"supergood", "percent"=>0.0}, "StatsTicketEscalation"=>{"used_for_average"=>0, "average_per_agent"=>0.0, "state"=>"supergood", "own"=>0, "total"=>0}, "StatsTicketChannelDistribution"=>{"channels"=>{"email"=>{"icon"=>"email", "inbound"=>6, "outbound"=>5, "inbound_in_percent"=>85, "outbound_in_percent"=>100}, "phone"=>{"icon"=>"phone", "inbound"=>1, "outbound"=>0, "inbound_in_percent"=>14, "outbound_in_percent"=>0}, "twitter"=>{"icon"=>"twitter", "inbound"=>0, "outbound"=>0, "inbound_in_percent"=>0, "outbound_in_percent"=>0}, "facebook"=>{"icon"=>"facebook", "inbound"=>0, "outbound"=>0, "inbound_in_percent"=>0, "outbound_in_percent"=>0}}}, "StatsTicketInProcess"=>{"used_for_average"=>50.0, "average_per_agent"=>18.8, "state"=>"supergood", "in_process"=>1, "percent"=>50.0, "total"=>2}, "StatsTicketReopen"=>{"used_for_average"=>0.0, "percent"=>0.0, "average_per_agent"=>18.8, "state"=>"supergood", "count"=>0, "total"=>1}}, "created_by_id"=>1, "created_at"=>Fri, 13 Sep 2019 07:11:52 UTC +00:00, "updated_at"=>Thu, 19 Sep 2019 11:47:24 UTC +00:00, "stats_store_object"=>"User", "stats_store_object_ref"=>{"id"=>2, "organization_id"=>1, "login"=>"stefan.moeller@osotec.ch", "firstname"=>"Stefan", "lastname"=>"Möller", "email"=>"stefan.moeller@osotec.ch", "web"=>"", "phone"=>"", "fax"=>"", "mobile"=>"", "department"=>"", "street"=>"", "zip"=>"", "city"=>"", "country"=>"", "address"=>"", "vip"=>false, "verified"=>false, "active"=>true, "note"=>"", "last_login"=>Wed, 18 Sep 2019 14:38:40 UTC +00:00, "out_of_office"=>false, "out_of_office_start_at"=>nil, "out_of_office_end_at"=>nil, "out_of_office_replacement_id"=>nil, "preferences"=>{"tickets_closed"=>0, "tickets_open"=>1, "notification_config"=>{"matrix"=>{"create"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>true, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "update"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>true, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "reminder_reached"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>false, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "escalation"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>false, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}}}, "locale"=>"de-de", "intro"=>true}, "updated_by_id"=>1, "created_by_id"=>1, "created_at"=>Thu, 29 Aug 2019 16:10:39 UTC +00:00, "updated_at"=>Wed, 18 Sep 2019 14:38:40 UTC +00:00, "organization"=>"osoTec GmbH", "permissions"=>["user_preferences", "user_preferences.password", "user_preferences.notifications", "user_preferences.access_token", "user_preferences.language", "user_preferences.linked_accounts", "user_preferences.device", "user_preferences.avatar", "user_preferences.calendar", "user_preferences.out_of_office", "ticket.agent", "chat.agent", "cti.agent", "knowledge_base.reader"], "role_ids"=>[2]}}

Payload size: 0M):
/opt/zammad/lib/search_index_backend.rb:230:in `add'
/opt/zammad/app/models/concerns/has_search_index_backend.rb:66:in `search_index_update_backend'
/opt/zammad/app/jobs/search_index_job.rb:14:in `perform'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/activejob-5.2.3/lib/active_job/execution.rb:39:in `block in perform_now'

.....
.....

/opt/zammad/app/models/scheduler.rb:367:in `block (3 levels) in worker'
/opt/zammad/vendor/ruby-2.5.5/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
/opt/zammad/app/models/scheduler.rb:365:in `block (2 levels) in worker'
/opt/zammad/app/models/scheduler.rb:361:in `loop'
/opt/zammad/app/models/scheduler.rb:361:in `block in worker'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
E, [2019-09-19T13:55:15.504091 #27890-47027212506520] ERROR -- : Retrying SearchIndexJob in #<Proc:0x0000558abff52220@/opt/zammad/app/jobs/search_index_job.rb:3 (lambda)> seconds, due to a StandardError. The original exception was nil.
I, [2019-09-19T13:55:15.511124 #27890-47027212506520]  INFO -- : Enqueued SearchIndexJob (Job ID: acf678c6-7c0a-4858-86b6-22a5490af3b8) to DelayedJob(default) at 2019-09-19 11:56:55 UTC with arguments: "StatsStore", 4
I, [2019-09-19T13:55:15.513697 #27890-47027212506520]  INFO -- : 2019-09-19T13:55:15+0200: [Worker(host:zammand.local pid:27890)] Job SearchIndexJob [acf678c6-7c0a-4858-86b6-22a5490af3b8] from DelayedJob(default) with arguments: ["StatsStore", 4] (id=56422) (queue=default) COMPLETED after 0.0820
I, [2019-09-19T13:55:15.518058 #27890-47027212506520]  INFO -- : 2019-09-19T13:55:15+0200: [Worker(host:zammand.local pid:27890)] Job SearchIndexJob [58ae140b-91df-44d7-8a85-e594f28a71d8] from DelayedJob(default) with arguments: ["StatsStore", 1] (id=56423) (queue=default) RUNNING
I, [2019-09-19T13:55:15.554932 #27890-47027212506520]  INFO -- : Performing SearchIndexJob (Job ID: 58ae140b-91df-44d7-8a85-e594f28a71d8) from DelayedJob(default) with arguments: "StatsStore", 1


I, [2019-09-19T13:55:15.595732 #27890-47027212506520]  INFO -- : # curl -X POST "http://localhost:9200/zammad_production_stats_store/_doc/1?pipeline=zammad537802081995" \
I, [2019-09-19T13:55:15.606076 #27890-47027212506520]  INFO -- : # 400
E, [2019-09-19T13:55:15.608639 #27890-47027212506520] ERROR -- : Unable to process POST request to elasticsearch URL 'http://localhost:9200/zammad_production_stats_store/_doc/1?pipeline=zammad537802081995'. Check the response and payload for detailed information: 

Response:
#<UserAgent::Result:0x00007fad684a3888 @success=false, @body="{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [zammad537802081995] does not exist\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [zammad537802081995] does not exist\"},\"status\":400}", @data=nil, @code="400", @content_type=nil, @error="Client Error: #<Net::HTTPBadRequest 400 Bad Request readbody=true>!">

Payload:
{"id"=>1, "stats_store_object_id"=>1, "o_id"=>3, "key"=>"dashboard", "related_o_id"=>nil, "related_stats_store_object_id"=>nil, "data"=>{"StatsTicketLoadMeasure"=>{"used_for_average"=>33.3, "average_per_agent"=>0.4, "percent"=>33.3, "state"=>"supergood", "own"=>1, "total"=>3, "average_per_agent_in_percent"=>266.4}, "StatsTicketWaitingTime"=>{"handling_time"=>0, "average_per_agent"=>0, "state"=>"supergood", "percent"=>0.0}, "StatsTicketEscalation"=>{"used_for_average"=>0, "average_per_agent"=>0.0, "state"=>"supergood", "own"=>0, "total"=>0}, "StatsTicketChannelDistribution"=>{"channels"=>{"email"=>{"icon"=>"email", "inbound"=>6, "outbound"=>5, "inbound_in_percent"=>85, "outbound_in_percent"=>100}, "phone"=>{"icon"=>"phone", "inbound"=>1, "outbound"=>0, "inbound_in_percent"=>14, "outbound_in_percent"=>0}, "twitter"=>{"icon"=>"twitter", "inbound"=>0, "outbound"=>0, "inbound_in_percent"=>0, "outbound_in_percent"=>0}, "facebook"=>{"icon"=>"facebook", "inbound"=>0, "outbound"=>0, "inbound_in_percent"=>0, "outbound_in_percent"=>0}}}, "StatsTicketInProcess"=>{"used_for_average"=>100.0, "average_per_agent"=>18.8, "state"=>"supergood", "in_process"=>2, "percent"=>100.0, "total"=>2}, "StatsTicketReopen"=>{"used_for_average"=>50.0, "percent"=>50.0, "average_per_agent"=>18.8, "state"=>"ok", "count"=>2, "total"=>4}}, "created_by_id"=>1, "created_at"=>Thu, 29 Aug 2019 17:13:04 UTC +00:00, "updated_at"=>Thu, 19 Sep 2019 11:47:25 UTC +00:00, "stats_store_object"=>"User", "stats_store_object_ref"=>{"id"=>3, "organization_id"=>1, "login"=>"remco.overman@osotec.ch", "firstname"=>"Remco", "lastname"=>"Overman", "email"=>"remco.overman@osotec.ch", "web"=>"", "phone"=>"", "fax"=>"", "mobile"=>"", "department"=>"", "street"=>"", "zip"=>"", "city"=>"", "country"=>"", "address"=>"", "vip"=>false, "verified"=>false, "active"=>true, "note"=>"", "last_login"=>Wed, 18 Sep 2019 08:39:52 UTC +00:00, "out_of_office"=>false, "out_of_office_start_at"=>nil, "out_of_office_end_at"=>nil, "out_of_office_replacement_id"=>nil, "preferences"=>{"notification_config"=>{"matrix"=>{"create"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>true, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "update"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>true, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "reminder_reached"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>false, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "escalation"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>false, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}}}, "locale"=>"de-de", "intro"=>true}, "updated_by_id"=>1, "created_by_id"=>1, "created_at"=>Thu, 29 Aug 2019 17:11:01 UTC +00:00, "updated_at"=>Wed, 18 Sep 2019 14:38:40 UTC +00:00, "organization"=>"osoTec GmbH", "permissions"=>["admin", "admin.user", "admin.group", "admin.role", "admin.organization", "admin.overview", "admin.text_module", "admin.time_accounting", "admin.macro", "admin.tag", "admin.calendar", "admin.sla", "admin.scheduler", "admin.report_profile", "admin.channel_web", "admin.channel_formular", "admin.channel_email", "admin.channel_twitter", "admin.channel_facebook", "admin.channel_telegram", "admin.channel_sms", "admin.channel_chat", "admin.branding", "admin.setting_system", "admin.security", "admin.ticket", "admin.package", "admin.integration", "admin.api", "admin.object", "admin.translation", "admin.monitoring", "admin.maintenance", "admin.session", "user_preferences", "user_preferences.password", "user_preferences.notifications", "user_preferences.access_token", "user_preferences.language", "user_preferences.linked_accounts", "user_preferences.device", "user_preferences.avatar", "user_preferences.calendar", "user_preferences.out_of_office", "report", "ticket.agent", "chat.agent", "cti.agent", "admin.knowledge_base", "knowledge_base.editor", "knowledge_base.reader"], "role_ids"=>[1, 2]}}

Payload size: 0M
E, [2019-09-19T13:55:15.609037 #27890-47027212506520] ERROR -- : Error performing SearchIndexJob (Job ID: 58ae140b-91df-44d7-8a85-e594f28a71d8) from DelayedJob(default) in 53.9ms: RuntimeError (Unable to process POST request to elasticsearch URL 'http://localhost:9200/zammad_production_stats_store/_doc/1?pipeline=zammad537802081995'. Check the response and payload for detailed information: 

Response:
#<UserAgent::Result:0x00007fad684a3888 @success=false, @body="{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [zammad537802081995] does not exist\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [zammad537802081995] does not exist\"},\"status\":400}", @data=nil, @code="400", @content_type=nil, @error="Client Error: #<Net::HTTPBadRequest 400 Bad Request readbody=true>!">

Payload:
{"id"=>1, "stats_store_object_id"=>1, "o_id"=>3, "key"=>"dashboard", "related_o_id"=>nil, "related_stats_store_object_id"=>nil, "data"=>{"StatsTicketLoadMeasure"=>{"used_for_average"=>33.3, "average_per_agent"=>0.4, "percent"=>33.3, "state"=>"supergood", "own"=>1, "total"=>3, "average_per_agent_in_percent"=>266.4}, "StatsTicketWaitingTime"=>{"handling_time"=>0, "average_per_agent"=>0, "state"=>"supergood", "percent"=>0.0}, "StatsTicketEscalation"=>{"used_for_average"=>0, "average_per_agent"=>0.0, "state"=>"supergood", "own"=>0, "total"=>0}, "StatsTicketChannelDistribution"=>{"channels"=>{"email"=>{"icon"=>"email", "inbound"=>6, "outbound"=>5, "inbound_in_percent"=>85, "outbound_in_percent"=>100}, "phone"=>{"icon"=>"phone", "inbound"=>1, "outbound"=>0, "inbound_in_percent"=>14, "outbound_in_percent"=>0}, "twitter"=>{"icon"=>"twitter", "inbound"=>0, "outbound"=>0, "inbound_in_percent"=>0, "outbound_in_percent"=>0}, "facebook"=>{"icon"=>"facebook", "inbound"=>0, "outbound"=>0, "inbound_in_percent"=>0, "outbound_in_percent"=>0}}}, "StatsTicketInProcess"=>{"used_for_average"=>100.0, "average_per_agent"=>18.8, "state"=>"supergood", "in_process"=>2, "percent"=>100.0, "total"=>2}, "StatsTicketReopen"=>{"used_for_average"=>50.0, "percent"=>50.0, "average_per_agent"=>18.8, "state"=>"ok", "count"=>2, "total"=>4}}, "created_by_id"=>1, "created_at"=>Thu, 29 Aug 2019 17:13:04 UTC +00:00, "updated_at"=>Thu, 19 Sep 2019 11:47:25 UTC +00:00, "stats_store_object"=>"User", "stats_store_object_ref"=>{"id"=>3, "organization_id"=>1, "login"=>"remco.overman@osotec.ch", "firstname"=>"Remco", "lastname"=>"Overman", "email"=>"remco.overman@osotec.ch", "web"=>"", "phone"=>"", "fax"=>"", "mobile"=>"", "department"=>"", "street"=>"", "zip"=>"", "city"=>"", "country"=>"", "address"=>"", "vip"=>false, "verified"=>false, "active"=>true, "note"=>"", "last_login"=>Wed, 18 Sep 2019 08:39:52 UTC +00:00, "out_of_office"=>false, "out_of_office_start_at"=>nil, "out_of_office_end_at"=>nil, "out_of_office_replacement_id"=>nil, "preferences"=>{"notification_config"=>{"matrix"=>{"create"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>true, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "update"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>true, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "reminder_reached"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>false, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}, "escalation"=>{"criteria"=>{"owned_by_me"=>true, "owned_by_nobody"=>false, "no"=>false}, "channel"=>{"email"=>true, "online"=>true}}}}, "locale"=>"de-de", "intro"=>true}, "updated_by_id"=>1, "created_by_id"=>1, "created_at"=>Thu, 29 Aug 2019 17:11:01 UTC +00:00, "updated_at"=>Wed, 18 Sep 2019 14:38:40 UTC +00:00, "organization"=>"osoTec GmbH", "permissions"=>["admin", "admin.user", "admin.group", "admin.role", "admin.organization", "admin.overview", "admin.text_module", "admin.time_accounting", "admin.macro", "admin.tag", "admin.calendar", "admin.sla", "admin.scheduler", "admin.report_profile", "admin.channel_web", "admin.channel_formular", "admin.channel_email", "admin.channel_twitter", "admin.channel_facebook", "admin.channel_telegram", "admin.channel_sms", "admin.channel_chat", "admin.branding", "admin.setting_system", "admin.security", "admin.ticket", "admin.package", "admin.integration", "admin.api", "admin.object", "admin.translation", "admin.monitoring", "admin.maintenance", "admin.session", "user_preferences", "user_preferences.password", "user_preferences.notifications", "user_preferences.access_token", "user_preferences.language", "user_preferences.linked_accounts", "user_preferences.device", "user_preferences.avatar", "user_preferences.calendar", "user_preferences.out_of_office", "report", "ticket.agent", "chat.agent", "cti.agent", "admin.knowledge_base", "knowledge_base.editor", "knowledge_base.reader"], "role_ids"=>[1, 2]}}

Payload size: 0M):

i hope this will help.

Is it possible to complete reinstall the elasticsearch inkl already existign elasticsearch db’s?
maybe that will help.

Please provide your elasticsearch.log, it might hold the reason why this fails.

Also, please enclose your log you post here within code tags
( ```) to enhance the overview for others :slight_smile:

here is the elasticsearch.log ( reduced to 32000 char) i hope it helps:

[2019-09-19T13:42:33,557][INFO ][o.e.n.Node               ] [zammand.local] stopping ...
[2019-09-19T13:42:33,597][INFO ][o.e.x.w.WatcherService   ] [zammand.local] stopping watch service, reason [shutdown initiated]
[2019-09-19T13:42:33,921][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [zammand.local] [controller/36584] [Main.cc@150] Ml controller exiting
[2019-09-19T13:42:33,924][INFO ][o.e.x.m.p.NativeController] [zammand.local] Native controller process has stopped - no new native processes can be started
[2019-09-19T13:42:33,976][INFO ][o.e.n.Node               ] [zammand.local] stopped
[2019-09-19T13:42:33,988][INFO ][o.e.n.Node               ] [zammand.local] closing ...
[2019-09-19T13:42:34,043][INFO ][o.e.n.Node               ] [zammand.local] closed
[2019-09-19T13:42:48,979][WARN ][o.e.b.JNANatives         ] [zammand.local] Unable to lock JVM Memory: error=12, reason=Nicht genügend Hauptspeicher verfügbar
[2019-09-19T13:42:48,985][WARN ][o.e.b.JNANatives         ] [zammand.local] This can result in part of the JVM being swapped out.
[2019-09-19T13:42:48,986][WARN ][o.e.b.JNANatives         ] [zammand.local] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2019-09-19T13:42:48,986][WARN ][o.e.b.JNANatives         ] [zammand.local] These can be adjusted by modifying /etc/security/limits.conf, for example: 
	# allow user 'elasticsearch' mlockall
	elasticsearch soft memlock unlimited
	elasticsearch hard memlock unlimited
[2019-09-19T13:42:48,987][WARN ][o.e.b.JNANatives         ] [zammand.local] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2019-09-19T13:42:49,549][INFO ][o.e.e.NodeEnvironment    ] [zammand.local] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [42.7gb], net total_space [49.9gb], types [rootfs]
[2019-09-19T13:42:49,551][INFO ][o.e.e.NodeEnvironment    ] [zammand.local] heap size [1007.3mb], compressed ordinary object pointers [true]
[2019-09-19T13:42:49,623][INFO ][o.e.n.Node               ] [zammand.local] node name [zammand.local], node ID [rprUBgxlT7CL1qORo2lP-Q], cluster name [elasticsearch]
[2019-09-19T13:42:49,624][INFO ][o.e.n.Node               ] [zammand.local] version[7.3.2], pid[33135], build[default/rpm/1c1faf1/2019-09-06T14:40:30.409026Z], OS[Linux/3.10.0-1062.1.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.2/12.0.2+10]
[2019-09-19T13:42:49,624][INFO ][o.e.n.Node               ] [zammand.local] JVM home [/usr/share/elasticsearch/jdk]
[2019-09-19T13:42:49,631][INFO ][o.e.n.Node               ] [zammand.local] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-14279445182262004138, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2019-09-19T13:42:51,867][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [aggs-matrix-stats]
[2019-09-19T13:42:51,867][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [analysis-common]
[2019-09-19T13:42:51,868][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [data-frame]
[2019-09-19T13:42:51,868][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [flattened]
[2019-09-19T13:42:51,868][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [ingest-common]
[2019-09-19T13:42:51,868][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [ingest-geoip]
[2019-09-19T13:42:51,869][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [ingest-user-agent]
[2019-09-19T13:42:51,869][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [lang-expression]
[2019-09-19T13:42:51,869][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [lang-mustache]
[2019-09-19T13:42:51,869][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [lang-painless]
[2019-09-19T13:42:51,870][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [mapper-extras]
[2019-09-19T13:42:51,870][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [parent-join]
[2019-09-19T13:42:51,870][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [percolator]
[2019-09-19T13:42:51,870][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [rank-eval]
[2019-09-19T13:42:51,870][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [reindex]
[2019-09-19T13:42:51,871][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [repository-url]
[2019-09-19T13:42:51,871][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [transport-netty4]
[2019-09-19T13:42:51,871][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [vectors]
[2019-09-19T13:42:51,871][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [x-pack-ccr]
[2019-09-19T13:42:51,871][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [x-pack-core]
[2019-09-19T13:42:51,872][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [x-pack-deprecation]
[2019-09-19T13:42:51,872][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [x-pack-graph]
[2019-09-19T13:42:51,872][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [x-pack-ilm]
[2019-09-19T13:42:51,876][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [x-pack-logstash]
[2019-09-19T13:42:51,876][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [x-pack-ml]
[2019-09-19T13:42:51,877][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [x-pack-monitoring]
[2019-09-19T13:42:51,877][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [x-pack-rollup]
[2019-09-19T13:42:51,877][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [x-pack-security]
[2019-09-19T13:42:51,877][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [x-pack-sql]
[2019-09-19T13:42:51,877][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [x-pack-voting-only-node]
[2019-09-19T13:42:51,878][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded module [x-pack-watcher]
[2019-09-19T13:42:51,878][INFO ][o.e.p.PluginsService     ] [zammand.local] loaded plugin [ingest-attachment]
[2019-09-19T13:42:56,521][INFO ][o.e.x.s.a.s.FileRolesStore] [zammand.local] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2019-09-19T13:42:57,521][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [zammand.local] [controller/33228] [Main.cc@110] controller (64 bit): Version 7.3.2 (Build 265429af874fbe) Copyright (c) 2019 Elasticsearch BV
[2019-09-19T13:42:58,312][DEBUG][o.e.a.ActionModule       ] [zammand.local] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-09-19T13:42:58,933][INFO ][o.e.d.DiscoveryModule    ] [zammand.local] using discovery type [zen] and seed hosts providers [settings]
[2019-09-19T13:42:59,785][INFO ][o.e.n.Node               ] [zammand.local] initialized
[2019-09-19T13:42:59,786][INFO ][o.e.n.Node               ] [zammand.local] starting ...
[2019-09-19T13:43:15,027][INFO ][o.e.t.TransportService   ] [zammand.local] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2019-09-19T13:43:15,101][WARN ][o.e.b.BootstrapChecks    ] [zammand.local] memory locking requested for elasticsearch process but memory is not locked
[2019-09-19T13:43:15,102][WARN ][o.e.b.BootstrapChecks    ] [zammand.local] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2019-09-19T13:43:15,120][INFO ][o.e.c.c.Coordinator      ] [zammand.local] cluster UUID [xRmBqtb6RwSJrZo-FaeKXA]
[2019-09-19T13:43:15,145][INFO ][o.e.c.c.ClusterBootstrapService] [zammand.local] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2019-09-19T13:43:15,370][INFO ][o.e.c.s.MasterService    ] [zammand.local] elected-as-master ([1] nodes joined)[{zammand.local}{rprUBgxlT7CL1qORo2lP-Q}{jMBwBWIRQoq6xfauBy3-Pw}{127.0.0.1}{127.0.0.1:9300}{dim}{ml.machine_memory=12419833856, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 11, version: 307, reason: master node changed {previous [], current [{zammand.local}{rprUBgxlT7CL1qORo2lP-Q}{jMBwBWIRQoq6xfauBy3-Pw}{127.0.0.1}{127.0.0.1:9300}{dim}{ml.machine_memory=12419833856, xpack.installed=true, ml.max_open_jobs=20}]}
[2019-09-19T13:43:15,642][INFO ][o.e.c.s.ClusterApplierService] [zammand.local] master node changed {previous [], current [{zammand.local}{rprUBgxlT7CL1qORo2lP-Q}{jMBwBWIRQoq6xfauBy3-Pw}{127.0.0.1}{127.0.0.1:9300}{dim}{ml.machine_memory=12419833856, xpack.installed=true, ml.max_open_jobs=20}]}, term: 11, version: 307, reason: Publication{term=11, version=307}
[2019-09-19T13:43:15,754][INFO ][o.e.h.AbstractHttpServerTransport] [zammand.local] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2019-09-19T13:43:15,755][INFO ][o.e.n.Node               ] [zammand.local] started
[2019-09-19T13:45:03,693][DEBUG][o.e.a.i.DeletePipelineTransportAction] [zammand.local] timed out while retrying [cluster:admin/ingest/pipeline/delete] after failure (timeout [30s])
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:189) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.action.ingest.DeletePipelineTransportAction.checkBlock(DeletePipelineTransportAction.java:73) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.action.ingest.DeletePipelineTransportAction.checkBlock(DeletePipelineTransportAction.java:38) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:170) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:161) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:138) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:58) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:145) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:123) ~[?:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:143) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:121) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:392) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:681) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.deletePipeline(AbstractClient.java:1075) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.rest.action.ingest.RestDeletePipelineAction.lambda$prepareRequest$0(RestDeletePipelineAction.java:48) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:113) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:69) ~[?:?]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:344) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:320) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:370) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:299) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:66) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) ~[?:?]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[?:?]
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-19T13:45:03,711][WARN ][r.suppressed             ] [zammand.local] path: /_ingest/pipeline/zammad537802081995, params: {id=zammad537802081995}
org.elasticsearch.discovery.MasterNotDiscoveredException: ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.onTimeout(TransportMasterNodeAction.java:251) [elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:572) [elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) [elasticsearch-7.3.2.jar:7.3.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:189) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.action.ingest.DeletePipelineTransportAction.checkBlock(DeletePipelineTransportAction.java:73) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.action.ingest.DeletePipelineTransportAction.checkBlock(DeletePipelineTransportAction.java:38) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:170) ~[elasticsearch-7.3.2.jar:7.3.2]
....
....
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	... 1 more


[2019-09-19T13:46:41,732][DEBUG][o.e.a.i.DeletePipelineTransportAction] [zammand.local] timed out while retrying [cluster:admin/ingest/pipeline/delete] after failure (timeout [30s])
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:189) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.action.ingest.DeletePipelineTransportAction.checkBlock(DeletePipelineTransportAction.java:73) ~[elasticsearch-7.3.2.jar:7.3.2]
....
....
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-19T13:46:41,754][WARN ][r.suppressed             ] [zammand.local] path: /_ingest/pipeline/zammad537802081995, params: {id=zammad537802081995}
org.elasticsearch.discovery.MasterNotDiscoveredException: ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.onTimeout(TransportMasterNodeAction.java:251) [elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:572) [elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) [elasticsearch-7.3.2.jar:7.3.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:189) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.action.ingest.DeletePipelineTransportAction.checkBlock(DeletePipelineTransportAction.java:73) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.action.ingest.DeletePipelineTransportAction.checkBlock(DeletePipelineTransportAction.java:38) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:170) ~[elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:161) ~[elasticsearch-7.3.2.jar:7.3.2]
....
....
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	... 1 more
[2019-09-19T13:47:25,961][DEBUG][o.e.a.b.TransportBulkAction] [zammand.local] failed to execute pipeline [zammad537802081995] for document [zammad_production_stats_store/_doc/4]
java.lang.IllegalArgumentException: pipeline with id [zammad537802081995] does not exist
	at org.elasticsearch.ingest.IngestService$3.doRun(IngestService.java:352) [elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.2.jar:7.3.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-19T13:47:26,075][DEBUG][o.e.a.b.TransportBulkAction] [zammand.local] failed to execute pipeline [zammad537802081995] for document [zammad_production_stats_store/_doc/1]
java.lang.IllegalArgumentException: pipeline with id [zammad537802081995] does not exist
	at org.elasticsearch.ingest.IngestService$3.doRun(IngestService.java:352) [elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.2.jar:7.3.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-19T13:47:26,198][DEBUG][o.e.a.b.TransportBulkAction] [zammand.local] failed to execute pipeline [zammad537802081995] for document [zammad_production_stats_store/_doc/2]
java.lang.IllegalArgumentException: pipeline with id [zammad537802081995] does not exist
	at org.elasticsearch.ingest.IngestService$3.doRun(IngestService.java:352) [elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.2.jar:7.3.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.2.jar:7.3.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:835) [?:?]

I was do a backup of the system and reinstall it on a new server, now the index is working without Problems.

1 Like

Glad you could solve your issue.
By the way, this is the reason why you had trouble:

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.