Infos:
- Used Zammad version: 3.6.0
- Used Zammad installation type: (source, package, docker-compose, …) source
- Operating system: CentOS 8
- Browser + version: not applicable
- Elasticsearch version: see below
Expected behavior:
- Ticket should be indexed by Elasticsearch
Actual behavior:
- Request to Elasticsearch errors with Net::ReadTimeout
Steps to reproduce the behavior:
- Create a ticket with about 90MB of attachments
- Start indexing the ticket at an remote ES host (e.g. via rails console)
We have a ticket in our Helpdesk system, that has approx. 90 MB of attachments. When Zammad tries to index that ticket in our ES Installation running on a different server, I get a Net:ReadTimeout error and the indexing fails (see error message below). Indexing in general works fine, only this ticket causes problems.
Our ES setup are two nodes running in Docker on a single host. An Apache HTTP Server acts as a Reverse Proxy in front of the ES nodes. The timeouts on the Apache ProxyPass directive are set to 30s for connect and 5min for data.
Can someone help me to fix this timeout issue?
Elasticsearch version
{
"name" : "odfe-node-01",
"cluster_name" : "odfe-cluster",
"cluster_uuid" : "FzR3tyJ8SYya3etIlXXQIw",
"version" : {
"number" : "7.10.2",
"build_flavor" : "oss",
"build_type" : "tar",
"build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
"build_date" : "2021-01-13T00:42:12.435326Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
Error message from production.log (lines are truncated by less, but the important stuff should be visible)
E, [2021-06-24T08:44:07.163204 #2726-83002660] ERROR -- : Unable to process post request to elasticsearch URL 'https://elastic.local.domain:9200/zammad_42__production_ticket/_doc/8048?pipeline=zammad82163369592'. Check the response and payload for detailed informatio>
Response:
#<UserAgent::Result:0x00007f5af04522c8 @success=false, @body=nil, @data=nil, @code=0, @content_type=nil, @error="#<Net::ReadTimeout: Net::ReadTimeout>">
Payload:
{"id":8048,"group_id":2,"priority_id":2,"state_id":2,"organization_id":null,"number":"20210414420017","title":"TICKET TITLE","owner_id":101,"customer_id":75,"note":null,"first_response_at":"2021-04-14T11:39:30.931Z","first_response_escalation_at":null>
E, [2021-06-24T08:44:07.522521 #2726-83002660] ERROR -- : Error performing SearchIndexJob (Job ID: 83aaf8ea-df5b-42ca-81b7-8f5cdacb1c19) from DelayedJob(default) in 195087.12ms: RuntimeError (Unable to process post request to elasticsearch URL 'https://elastic.local.domain>
Response:
#<UserAgent::Result:0x00007f5af04522c8 @success=false, @body=nil, @data=nil, @code=0, @content_type=nil, @error="#<Net::ReadTimeout: Net::ReadTimeout>">
Payload:
{"id":8048,"group_id":2,"priority_id":2,"state_id":2,"organization_id":null,"number":"20210414420017","title":"TICKET TITLE","owner_id":101,"customer_id":75,"note":null,"first_response_at":"2021-04-14T11:39:30.931Z","first_response_escalation_at":null>
Payload size: 92M):
/opt/zammad/lib/search_index_backend.rb:927:in `make_request_and_validate'
/opt/zammad/lib/search_index_backend.rb:145:in `add'
/opt/zammad/app/models/concerns/has_search_index_backend.rb:173:in `search_index_update_backend'
/opt/zammad/app/jobs/search_index_job.rb:26:in `update_search_index'
/opt/zammad/app/jobs/search_index_job.rb:22:in `perform'
/usr/local/rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/execution.rb:39:in `block in perform_now'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/opt/zammad/app/jobs/concerns/has_active_job_lock.rb:12:in `block (2 levels) in <module:HasActiveJobLock>'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/callbacks.rb:118:in `instance_exec'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/local/rvm/gems/ruby-2.6.6/gems/i18n-1.8.5/lib/i18n.rb:313:in `with_locale'
/usr/local/rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/translation.rb:9:in `block (2 levels) in <module:Translation>'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/callbacks.rb:118:in `instance_exec'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/local/rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/logging.rb:26:in `block (4 levels) in <module:Logging>'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/notifications.rb:168:in `block in instrument'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/notifications.rb:168:in `instrument'
/usr/local/rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/logging.rb:25:in `block (3 levels) in <module:Logging>'
/usr/local/rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/logging.rb:46:in `block in tag_logger'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/tagged_logging.rb:71:in `block in tagged'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/tagged_logging.rb:28:in `tagged'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/tagged_logging.rb:71:in `tagged'
/usr/local/rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/logging.rb:46:in `tag_logger'
/usr/local/rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/logging.rb:22:in `block (2 levels) in <module:Logging>'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/callbacks.rb:118:in `instance_exec'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/callbacks.rb:136:in `run_callbacks'
/usr/local/rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/execution.rb:38:in `perform_now'
/usr/local/rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/execution.rb:24:in `block in execute'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/usr/local/rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/railtie.rb:28:in `block (4 levels) in <class:Railtie>'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/execution_wrapper.rb:87:in `wrap'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/reloader.rb:73:in `block in wrap'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/execution_wrapper.rb:87:in `wrap'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/reloader.rb:72:in `wrap'
/usr/local/rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/railtie.rb:27:in `block (3 levels) in <class:Railtie>'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/callbacks.rb:118:in `instance_exec'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/local/rvm/gems/ruby-2.6.6/gems/activesupport-5.2.4.4/lib/active_support/callbacks.rb:136:in `run_callbacks'
/usr/local/rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/execution.rb:22:in `execute'
/usr/local/rvm/gems/ruby-2.6.6/gems/activejob-5.2.4.4/lib/active_job/queue_adapters/delayed_job_adapter.rb:42:in `perform'
/usr/local/rvm/gems/ruby-2.6.6/gems/delayed_job-4.1.7/lib/delayed/backend/base.rb:81:in `block in invoke_job'
/usr/local/rvm/gems/ruby-2.6.6/gems/delayed_job-4.1.7/lib/delayed/lifecycle.rb:61:in `block in initialize'
/usr/local/rvm/gems/ruby-2.6.6/gems/delayed_job-4.1.7/lib/delayed/lifecycle.rb:66:in `execute'
/usr/local/rvm/gems/ruby-2.6.6/gems/delayed_job-4.1.7/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/usr/local/rvm/gems/ruby-2.6.6/gems/delayed_job-4.1.7/lib/delayed/backend/base.rb:78:in `invoke_job'
/usr/local/rvm/gems/ruby-2.6.6/gems/delayed_job-4.1.7/lib/delayed/worker.rb:230:in `block (2 levels) in run'
/usr/local/rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/timeout.rb:93:in `block in timeout'
/usr/local/rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/timeout.rb:103:in `timeout'
/usr/local/rvm/gems/ruby-2.6.6/gems/delayed_job-4.1.7/lib/delayed/worker.rb:230:in `block in run'
/usr/local/rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/benchmark.rb:308:in `realtime'
/usr/local/rvm/gems/ruby-2.6.6/gems/delayed_job-4.1.7/lib/delayed/worker.rb:229:in `run'
/usr/local/rvm/gems/ruby-2.6.6/gems/delayed_job-4.1.7/lib/delayed/worker.rb:312:in `block in reserve_and_run_one_job'
/usr/local/rvm/gems/ruby-2.6.6/gems/delayed_job-4.1.7/lib/delayed/lifecycle.rb:61:in `block in initialize'
/usr/local/rvm/gems/ruby-2.6.6/gems/delayed_job-4.1.7/lib/delayed/lifecycle.rb:66:in `execute'
/usr/local/rvm/gems/ruby-2.6.6/gems/delayed_job-4.1.7/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/usr/local/rvm/gems/ruby-2.6.6/gems/delayed_job-4.1.7/lib/delayed/worker.rb:312:in `reserve_and_run_one_job'
/usr/local/rvm/gems/ruby-2.6.6/gems/delayed_job-4.1.7/lib/delayed/worker.rb:213:in `block in work_off'
/usr/local/rvm/gems/ruby-2.6.6/gems/delayed_job-4.1.7/lib/delayed/worker.rb:212:in `times'
/usr/local/rvm/gems/ruby-2.6.6/gems/delayed_job-4.1.7/lib/delayed/worker.rb:212:in `work_off'
/opt/zammad/app/models/scheduler.rb:372:in `block (3 levels) in worker'
/usr/local/rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/benchmark.rb:308:in `realtime'
/opt/zammad/app/models/scheduler.rb:370:in `block (2 levels) in worker'
/opt/zammad/app/models/scheduler.rb:366:in `loop'
/opt/zammad/app/models/scheduler.rb:366:in `block in worker'
/usr/local/rvm/gems/ruby-2.6.6/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'