After upgrade to 6.5.0: Failed to run background job #1 'SearchIndexAssociationsJob

Infos:

  • Used Zammad version: 6.5.0
  • Used Zammad installation type: docker compose
  • Operating system: Ubuntu server 22.04.3 LTS (Jammy Jellyfish)
  • Browser + version: Chromium Version 134.0.6998.165 (Official Build) snap (64-bit)

Expected behavior:

{"healthy":true}

Actual behavior:

{"healthy":false,"message":"Failed to run background job #1 'SearchIndexAssociationsJob' 2 time(s) with 16 attempt(s).","issues":["Failed to run background job #1 'SearchIndexAssociationsJob' 2 time(s) with 16 attempt(s)."],"actions":[]}

Steps to reproduce the behavior:

  • Upgrade from v6.4.1 to v6.5.0 today
  • Perform a ElasticSearch index rebuild after upgrade as other posts hinted but with no visible improvements, no return error/message in prompt after running this command
$ sudo docker compose run --rm zammad-railsserver rails r rake zammad:searchindex:rebuild
[+] Creating 3/3
 ✔ Container zammad-zammad-redis-1       Recreated                                                                                                                                       0.4s 
 ✔ Container zammad-zammad-memcached-1   Recreated                                                                                                                                       0.7s 
 ✔ Container zammad-zammad-postgresql-1  Recreated                                                                                                                                       0.4s 
[+] Running 3/3
 ✔ Container zammad-zammad-memcached-1   Started                                                                                                                                         0.7s 
 ✔ Container zammad-zammad-redis-1       Started                                                                                                                                         0.6s 
 ✔ Container zammad-zammad-postgresql-1  Started                                                                                                                                         0.7s 
I, [2025-04-07T19:20:52.852850#1-6640]  INFO -- : Using memcached as Rails cache store.
I, [2025-04-07T19:20:52.853557#1-6640]  INFO -- : Using the Redis back end for Zammad's web socket session store.
I, [2025-04-07T19:20:52.934280#1-6640]  INFO -- : ActionCable is using the redis instance at redis://zammad-redis:6379.
I, [2025-04-07T19:20:55.972439#1-6640]  INFO -- : ActionCable is configured to accept requests from (?-mix:https?:\/\/localhost:\d+),


$

I’m sure I’m missing some small parts, so any pinpoint of my blind spot will be much appreciated. :folded_hands:

maybe try this:

docker-compose exec web bundle exec rails r "Delayed::Job.where(failed_at: nil).each(&:invoke_job)"
docker-compose restart zammad-scheduler

yesterday ,i try to migration 6.3.1 to 6.5.0

docs for this : Share: Migration 6.3.1 to 6.5.0

maybe you can see it for reference

First of all many thanks for your help!

As service “web” is not running according to official docker-compose.yml

I ran it against zammad-scheduler (maybe I was wrong)

I got following errors, which at least make me progressing, but any insight to the most effective way to handle it would be much appreciated:

sudo docker compose run --rm zammad-scheduler rails r "Delayed::Job.where(failed_at: nil).each(&:invoke_job)"
WARN[0000] /opt/stacks/zammad/compose.yaml: `version` is obsolete 
[+] Creating 3/3
 ✔ Container zammad-zammad-postgresql-1  Recreated                                                                                                                                                                                                                                                                                               1.5s 
 ✔ Container zammad-zammad-redis-1       Recreated                                                                                                                                                                                                                                                                                               1.1s 
 ✔ Container zammad-zammad-memcached-1   Recreated                                                                                                                                                                                                                                                                                               1.7s 
[+] Running 3/3
 ✔ Container zammad-zammad-postgresql-1  Started                                                                                                                                                                                                                                                                                                 2.0s 
 ✔ Container zammad-zammad-redis-1       Started                                                                                                                                                                                                                                                                                                 2.1s 
 ✔ Container zammad-zammad-memcached-1   Started                                                                                                                                                                                                                                                                                                 1.7s 
I, [2025-04-08T22:30:14.243538#1-6640]  INFO -- : Using memcached as Rails cache store.
I, [2025-04-08T22:30:14.246348#1-6640]  INFO -- : Using the Redis back end for Zammad's web socket session store.
I, [2025-04-08T22:30:14.353040#1-6640]  INFO -- : ActionCable is using the redis instance at redis://zammad-redis:6379.
I, [2025-04-08T22:30:21.901560#1-6640]  INFO -- : ActionCable is configured to accept requests from (?-mix:https?:\/\/localhost:\d+), https://example.com.
E, [2025-04-08T22:30:37.094126#1-6640] ERROR -- : Unable to process GET request to elasticsearch URL 'http://zammad-elasticsearch:9200/zammad_production_overview/_update_by_query?conflicts=proceed&slices=auto&max_docs=1000'. Check the response and payload for detailed information:

Response:
#<UserAgent::Result:0x00007fc2afc4f508 @success=false, @body="{\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [zammad_production_overview]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"zammad_production_overview\",\"index_uuid\":\"_na_\",\"index\":\"zammad_production_overview\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [zammad_production_overview]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"zammad_production_overview\",\"index_uuid\":\"_na_\",\"index\":\"zammad_production_overview\"},\"status\":404}", @data=nil, @code="404", @content_type=nil, @error="No such file http://zammad-elasticsearch:9200/zammad_production_overview/_update_by_query?conflicts=proceed&slices=auto&max_docs=1000, 404!", @header={"x-elastic-product"=>"Elasticsearch", "content-type"=>"application/json", "content-length"=>"485"}>

Payload:
{"script":{"lang":"painless","source":"ctx._source.updated_by=params.updated_by","params":{"updated_by":{"id":8,"organization_id":null,"login":"REDACTED","firstname":"REDACTED","lastname":"REDACTED","email":"REDACTED.REDACTED@example.com","image":null,"image_source":null,"web":"","phone":"","fax":"","mobile":"","department":"","street":"","zip":"","city":"","country":"","address":"","vip":false,"verified":false,"active":true,"note":"","last_login":"2025-04-08T20:27:21.978Z","source":"saml","login_failed":0,"out_of_office":false,"out_of_office_start_at":null,"out_of_office_end_at":null,"out_of_office_replacement_id":null,"preferences":{"locale":"fr-fr","notification_config":{"matrix":{"create":{"criteria":{"owned_by_me":true,"owned_by_nobody":true,"subscribed":true,"no":false},"channel":{"email":true,"online":true}},"update":{"criteria":{"owned_by_me":true,"subscribed":true,"owned_by_nobody":false,"no":false},"channel":{"email":true,"online":true}},"reminder_reached":{"criteria":{"owned_by_me":true,"owned_by_nobody":false,"subscribed":false,"no":false},"channel":{"email":true,"online":true}},"escalation":{"criteria":{"owned_by_me":true,"owned_by_nobody":false,"subscribed":false,"no":false},"channel":{"email":true,"online":true}}}},"intro":true,"tickets_closed":7,"tickets_open":0,"notification_sound":{"file":"Xylo.mp3","enabled":true},"theme":"light","keyboard_shortcuts_clues":true},"updated_by_id":8,"created_by_id":1,"created_at":"2024-04-16T19:44:13.093Z","updated_at":"2025-04-08T20:27:22.136Z","fullname":"REDACTED"}}},"query":{"bool":{"must":[{"term":{"updated_by_id":8}},{"range":{"updated_by.updated_at":{"lt":"2025-04-08T20:27:22.136Z"}}}]}},"sort":{"id":"desc"}}

Payload size: 0M
E, [2025-04-08T22:30:37.130611#1-6640] ERROR -- : Unable to process GET request to elasticsearch URL 'http://zammad-elasticsearch:9200/zammad_production_public_link/_update_by_query?conflicts=proceed&slices=auto&max_docs=1000'. Check the response and payload for detailed information:

Response:
#<UserAgent::Result:0x00007fc2afc4af58 @success=false, @body="{\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [zammad_production_public_link]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"zammad_production_public_link\",\"index_uuid\":\"_na_\",\"index\":\"zammad_production_public_link\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [zammad_production_public_link]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"zammad_production_public_link\",\"index_uuid\":\"_na_\",\"index\":\"zammad_production_public_link\"},\"status\":404}", @data=nil, @code="404", @content_type=nil, @error="No such file http://zammad-elasticsearch:9200/zammad_production_public_link/_update_by_query?conflicts=proceed&slices=auto&max_docs=1000, 404!", @header={"x-elastic-product"=>"Elasticsearch", "content-type"=>"application/json", "content-length"=>"503"}>

Payload:
{"script":{"lang":"painless","source":"ctx._source.updated_by=params.updated_by","params":{"updated_by":{"id":8,"organization_id":null,"login":"REDACTED","firstname":"REDACTED","lastname":"REDACTED","email":"REDACTED.REDACTED@example.com","image":null,"image_source":null,"web":"","phone":"","fax":"","mobile":"","department":"","street":"","zip":"","city":"","country":"","address":"","vip":false,"verified":false,"active":true,"note":"","last_login":"2025-04-08T20:27:21.978Z","source":"saml","login_failed":0,"out_of_office":false,"out_of_office_start_at":null,"out_of_office_end_at":null,"out_of_office_replacement_id":null,"preferences":{"locale":"fr-fr","notification_config":{"matrix":{"create":{"criteria":{"owned_by_me":true,"owned_by_nobody":true,"subscribed":true,"no":false},"channel":{"email":true,"online":true}},"update":{"criteria":{"owned_by_me":true,"subscribed":true,"owned_by_nobody":false,"no":false},"channel":{"email":true,"online":true}},"reminder_reached":{"criteria":{"owned_by_me":true,"owned_by_nobody":false,"subscribed":false,"no":false},"channel":{"email":true,"online":true}},"escalation":{"criteria":{"owned_by_me":true,"owned_by_nobody":false,"subscribed":false,"no":false},"channel":{"email":true,"online":true}}}},"intro":true,"tickets_closed":7,"tickets_open":0,"notification_sound":{"file":"Xylo.mp3","enabled":true},"theme":"light","keyboard_shortcuts_clues":true},"updated_by_id":8,"created_by_id":1,"created_at":"2024-04-16T19:44:13.093Z","updated_at":"2025-04-08T20:27:22.136Z","fullname":"REDACTED"}}},"query":{"bool":{"must":[{"term":{"updated_by_id":8}},{"range":{"updated_by.updated_at":{"lt":"2025-04-08T20:27:22.136Z"}}}]}},"sort":{"id":"desc"}}

Payload size: 0M
E, [2025-04-08T22:30:37.201085#1-6640] ERROR -- : Unable to process GET request to elasticsearch URL 'http://zammad-elasticsearch:9200/zammad_production_role/_update_by_query?conflicts=proceed&slices=auto&max_docs=1000'. Check the response and payload for detailed information:

Response:
#<UserAgent::Result:0x00007fc2afc45738 @success=false, @body="{\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [zammad_production_role]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"zammad_production_role\",\"index_uuid\":\"_na_\",\"index\":\"zammad_production_role\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [zammad_production_role]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"zammad_production_role\",\"index_uuid\":\"_na_\",\"index\":\"zammad_production_role\"},\"status\":404}", @data=nil, @code="404", @content_type=nil, @error="No such file http://zammad-elasticsearch:9200/zammad_production_role/_update_by_query?conflicts=proceed&slices=auto&max_docs=1000, 404!", @header={"x-elastic-product"=>"Elasticsearch", "content-type"=>"application/json", "content-length"=>"461"}>

Payload:
{"script":{"lang":"painless","source":"ctx._source.updated_by=params.updated_by","params":{"updated_by":{"id":8,"organization_id":null,"login":"REDACTED","firstname":"REDACTED","lastname":"REDACTED","email":"REDACTED.REDACTED@example.com","image":null,"image_source":null,"web":"","phone":"","fax":"","mobile":"","department":"","street":"","zip":"","city":"","country":"","address":"","vip":false,"verified":false,"active":true,"note":"","last_login":"2025-04-08T20:27:21.978Z","source":"saml","login_failed":0,"out_of_office":false,"out_of_office_start_at":null,"out_of_office_end_at":null,"out_of_office_replacement_id":null,"preferences":{"locale":"fr-fr","notification_config":{"matrix":{"create":{"criteria":{"owned_by_me":true,"owned_by_nobody":true,"subscribed":true,"no":false},"channel":{"email":true,"online":true}},"update":{"criteria":{"owned_by_me":true,"subscribed":true,"owned_by_nobody":false,"no":false},"channel":{"email":true,"online":true}},"reminder_reached":{"criteria":{"owned_by_me":true,"owned_by_nobody":false,"subscribed":false,"no":false},"channel":{"email":true,"online":true}},"escalation":{"criteria":{"owned_by_me":true,"owned_by_nobody":false,"subscribed":false,"no":false},"channel":{"email":true,"online":true}}}},"intro":true,"tickets_closed":7,"tickets_open":0,"notification_sound":{"file":"Xylo.mp3","enabled":true},"theme":"light","keyboard_shortcuts_clues":true},"updated_by_id":8,"created_by_id":1,"created_at":"2024-04-16T19:44:13.093Z","updated_at":"2025-04-08T20:27:22.136Z","fullname":"REDACTED"}}},"query":{"bool":{"must":[{"term":{"updated_by_id":8}},{"range":{"updated_by.updated_at":{"lt":"2025-04-08T20:27:22.136Z"}}}]}},"sort":{"id":"desc"}}

Payload size: 0M
E, [2025-04-08T22:30:37.275991#1-6640] ERROR -- : Unable to process GET request to elasticsearch URL 'http://zammad-elasticsearch:9200/zammad_production_text_module/_update_by_query?conflicts=proceed&slices=auto&max_docs=1000'. Check the response and payload for detailed information:

Response:
#<UserAgent::Result:0x00007fc2afc5f340 @success=false, @body="{\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [zammad_production_text_module]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"zammad_production_text_module\",\"index_uuid\":\"_na_\",\"index\":\"zammad_production_text_module\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [zammad_production_text_module]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"zammad_production_text_module\",\"index_uuid\":\"_na_\",\"index\":\"zammad_production_text_module\"},\"status\":404}", @data=nil, @code="404", @content_type=nil, @error="No such file http://zammad-elasticsearch:9200/zammad_production_text_module/_update_by_query?conflicts=proceed&slices=auto&max_docs=1000, 404!", @header={"x-elastic-product"=>"Elasticsearch", "content-type"=>"application/json", "content-length"=>"503"}>

Payload:
{"script":{"lang":"painless","source":"ctx._source.updated_by=params.updated_by","params":{"updated_by":{"id":8,"organization_id":null,"login":"REDACTED","firstname":"REDACTED","lastname":"REDACTED","email":"REDACTED.REDACTED@example.com","image":null,"image_source":null,"web":"","phone":"","fax":"","mobile":"","department":"","street":"","zip":"","city":"","country":"","address":"","vip":false,"verified":false,"active":true,"note":"","last_login":"2025-04-08T20:27:21.978Z","source":"saml","login_failed":0,"out_of_office":false,"out_of_office_start_at":null,"out_of_office_end_at":null,"out_of_office_replacement_id":null,"preferences":{"locale":"fr-fr","notification_config":{"matrix":{"create":{"criteria":{"owned_by_me":true,"owned_by_nobody":true,"subscribed":true,"no":false},"channel":{"email":true,"online":true}},"update":{"criteria":{"owned_by_me":true,"subscribed":true,"owned_by_nobody":false,"no":false},"channel":{"email":true,"online":true}},"reminder_reached":{"criteria":{"owned_by_me":true,"owned_by_nobody":false,"subscribed":false,"no":false},"channel":{"email":true,"online":true}},"escalation":{"criteria":{"owned_by_me":true,"owned_by_nobody":false,"subscribed":false,"no":false},"channel":{"email":true,"online":true}}}},"intro":true,"tickets_closed":7,"tickets_open":0,"notification_sound":{"file":"Xylo.mp3","enabled":true},"theme":"light","keyboard_shortcuts_clues":true},"updated_by_id":8,"created_by_id":1,"created_at":"2024-04-16T19:44:13.093Z","updated_at":"2025-04-08T20:27:22.136Z","fullname":"REDACTED"}}},"query":{"bool":{"must":[{"term":{"updated_by_id":8}},{"range":{"updated_by.updated_at":{"lt":"2025-04-08T20:27:22.136Z"}}}]}},"sort":{"id":"desc"}}

Payload size: 0M

Could it be related to my version of Elastisearch? actually running ELASTICSEARCH_VERSION=8.12.0

Your error message indicates that the rebuild wasn‘t done. As you say that you‘ve run it and based on what you‘ve provided, you‘re running it wrong.

You have rails r in front of the rake portion of the command which obviously won‘t work.

Of course!!

Many thanks @MrGeneration :raising_hands:

Here is the correct command for the records:

sudo docker compose run --rm zammad-railsserver bundle exec rake zammad:searchindex:rebuild

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.