Failed to rebuild elastic index

Hello! I need help with elasticsearch! Has anyone encountered such a problem?

  • Used Zammad version: 6.1.0
  • Used Zammad installation type: package
  • Operating system: Ubuntu 22.04
  • Elastic search: 7.17

Expected behavior: Running zammad run rake searchindex:rebuild should rebuild the search index, but during the process an error occurred.

Actual behavior:

AFBLAQIeAxQAAAgIAJxy+1QvRJkEyCkDANzyAwCiABgAAAAAAAAAAACkgUJ4WALRhNCw0LnQu9GLINGE0LjQvdCw0LvRjNC90YvRhSDQstC10YDRgdC40Lkv0JDQutGCX9C/0L9f0L7RgtGH0LXRgtC90YvRhV/QtNC+0LrRg9C80LXQvdGC0L7Qsl/Qk9CaXzk0XzRf0Y3RgtCw0L9fMTU4NTlf0YFf0L/QvtC00L/QuNGB0YzRjl/Qn9C+0LvRg9GH0LDRgtC10LvRjy5wZGZVVAUAA5gf4WJ1eAsAAQTwAwAABPADAABQSwECHgMUAAAICACccvtUVoJTEqfuXwA63WIAmQAYAAAAAAAAAAAApIHmolsC0YTQsNC50LvRiyDRhNC40L3QsNC70YzQvdGL0YUg0LLQtdGA0YHQuNC5L9CQ0LrRgl/Qv9GA0LjRkdC80LAt0L/QtdGA0LXQtNCw0YfQuF/QvtGC0YfRkdGC0L3Ri9GFX9C00L7QutGD0LzQtdC90YLQvtCyX9C/0L5f0JPQml8wOTRfNF/RjdGC0LDQv18tMTU4NTkucGRmVVQFAAOYH+FidXgLAAEE8AMAAATwAwAAUEsFBgAAAAANAA0AhQkAAGCSuwIAAA==“}]},{“id”:22060,“ticket_id”:5259,“type_id”:10,“sender_id”:1,“from”:“John Jones”,“to”:”“,“cc”:”“,“subject”:”“,“reply_to”:null,“message_id”:null,“in_reply_to”:”",“content_type”:“text/html”,“references”:null,“body”:“agreed”,“internal”:true,“preferences”:{},“updated_by_id”:31,“created_by_id”:31,“origin_by_id”:null,“created_at”:“2022-08-01T13:33:00.496Z”,“updated_at”:“2022-08-01T13:33:00.496Z”,“attachment”:}]}

Payload size: 114M
/opt/zammad/lib/search_index_backend.rb:742:in make_request_and_validate' /opt/zammad/lib/search_index_backend.rb:151:in add’
/opt/zammad/app/models/concerns/has_search_index_backend.rb:141:in search_index_update_backend' /opt/zammad/app/models/concerns/has_search_index_backend.rb:189:in block in search_index_reload’
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/activerecord-6.1.7.6/lib/active_record/relation/delegation.rb:88:in each' /opt/zammad/vendor/bundle/ruby/3.1.0/gems/activerecord-6.1.7.6/lib/active_record/relation/delegation.rb:88:in each’
/opt/zammad/config/initializers/active_record_as_batches.rb:27:in as_batches' /opt/zammad/config/initializers/active_record_as_batches.rb:44:in as_batches’
/opt/zammad/app/models/concerns/has_search_index_backend.rb:186:in search_index_reload' /opt/zammad/lib/tasks/zammad/search_index_es.rake:42:in block (5 levels) in ’
/opt/zammad/lib/tasks/zammad/search_index_es.rake:41:in block (4 levels) in <main>' /opt/zammad/lib/tasks/zammad/search_index_es.rake:39:in each’
/opt/zammad/lib/tasks/zammad/search_index_es.rake:39:in block (3 levels) in <main>' /opt/zammad/lib/tasks/zammad/search_index_es.rake:60:in block (3 levels) in ’
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/exe/rake:27:in <top (required)>' /opt/zammad/bin/bundle:121:in load’
/opt/zammad/bin/bundle:121:in `’
Tasks: TOP => zammad:searchindex:rebuild
(See full trace by running task with --trace)

Tell me please what could be the cause of this problem?

Hi @ykz. Could you please post the error message of the production.log?

Hi @fliebe92 ! I have several identical errors, for different tickets.
Here is an example of one of them:
[12:14:56.006764#283501-6300] ERROR – : Unable to send Ticket.find(5259).search_index_update_backend backend: #<RuntimeError: Unable to process post request to elasticsearch URL ‘http://localhost:9200/tickets_zammad_production_ticket/_doc/5259?pipeline=zammadcb54f5b7-3b9d-4339-9760-e429c93fb762

The error message contains a reason portion of HTTP response depending on the issue.

We’ve only seen incomplete error messages so far which makes it impossible to help.
Just to be sure, make sure that the optional suggestions for ES are setup, because 114MB is a farily big payload.

https://docs.zammad.org/en/latest/install/elasticsearch.html#step-2-suggested-configuration

Hi @MrGeneration! My production.log file size is 5 gigabytes, and the ticket errors contain a lot of unclear information, such as this:


one such message takes up 10% of the logfile.
What other information can I provide?

I actually already mentioned what I’m looking for:

Hi @MrGeneration @fliebe92!
Elastic search indicates a problem with one ticket.

I checked this ticket and found that the agent’s last name had been changed from Edison to Jones. The last name change occurred based on updating the agent data via LDAP.
Previously, this user left a comment in the ticket, which is indicated by an error message in the log file.
And now I see that elasticsearch indicates that Jones - id messages (is null).
That is, elasticsearch tries to associate the comment with the ticket, but because the last name has been changed, it cannot do so.

System
updated User Login from ‘edison’ → ‘jones’
updated User Last name from ‘Edison’ → ‘Jones’
updated User Email from ‘edison@mydomain.com’ → ‘jones@mydomain.com’

Hello @MrGeneration @fliebe92 !
As expected, the problem was related to one ticket. After deleting the ticket using the scheduler, indexing was successful.
Please tell me how I can find out what the problem is with this ticket to avoid problems in the future.

You’ve removed all information and did not share what you’ve been asked to share.

At this point nobody will be able to help you further.