Zammad Health Check fails with SearchIndexAssociationsJob error

Infos:

  • Used Zammad version: 5.3.0
  • Used Zammad installation type: docker-compose
  • Operating system: Ubuntu
  • Browser + version: irrelevant

Expected behavior:

  • /#system/monitoring should say
    Current Status: no issues

Actual behavior:

  • /#system/monitoring says intermittently
    Current Status: Failed to run background job #1 'SearchIndexAssociationsJob' 3 time(s) with 48 attempt(s).

Steps to reproduce the behavior:

  • Not reproducible but happens intermittently on both our larger zammad setups

What we did so far

The issues happened last year in december and I currently do not have any logs to give more insight here. As soon as it happens again I will provide the relevant logs of course. I’m just posting in the hopes maybe someone else had the same issue and could provide hints until we can add our logs to this post.

We found this website on our search and also tried to rebuild the ES indices because why not, since if the culprit are b0rked indices due to some past upgrade issues this could solve the issue. But it did not fix the issue.

We just had a similar issue. Not sure how much of this applies to your situation but I’ll share our learnings.

TL;DR

  • Docker host had been running low on space. Elasticsearch seems to stop writing to disk if less than 90% free disk is available. Issue was resolved by freeing up space on the docker host.

Infos:

  • Zammad 5.2.3-4
  • Install type: docker-compose
  • OS Rocky Linux 8.7

Actual behavior:

  • Zammad search returns incomplete results (searching for a specific tag or group doesn’t include tickets that recently were created)
  • Health check reports errors: 41 failing background jobs. Failed to run background job #1 'SearchIndexAssociationsJob' 8 time(s) with 130 attempt(s). Failed to run background job #2 'SearchIndexJob' 2 time(s) with 36 attempt(s)

What we did so far

  • Reviewed container logs. It’s alerting that it has low disk space (less than 90%)
  • 2023-01-13T15:18:01.415903337Z WARN high disk watermark [90%] exceeded on [blablabla][blablabla][/usr/share/elasticsearch/data/nodes/0]
    *E, [2023-01-13T18:16:32.868307 #1-109360] ERROR -- : Unable to process post request to elasticsearch URL 'http://zammad-elasticsearch:9200/zammad_production_stats_store/_doc/126?pipeline=blablabla'. Check the response and payload for detailed information
  • We cleared out some space. Zammad no longer reported bad health check :+1:

Note

  • Having search broken for a while caused weird search results, even after elasticsearch stopped throwing errors
  • This was resolved by rebuilding the search index
  • In Rails container:
zammad@f2f09d2a66c2:~$ rails searchindex:rebuild
I, [2023-01-13T23:08:04.969239 #940-8060]  INFO -- : Using memcached as Rails cache store.
I, [2023-01-13T23:08:04.969357 #940-8060]  INFO -- : Using Redis as web socket session store.
I, [2023-01-13T23:08:05.217487 #940-8060]  INFO -- : Setting.set('product_logo', "logo.png")
I, [2023-01-13T23:08:06.519300 #940-8060]  INFO -- : Setting.set('models_searchable', ["Chat::Session", "KnowledgeBase::Answer::Translation", "Organization", "Ticket", "User"])
Dropping indexes... done.
Deleting pipeline... done.
Creating indexes... done.
Creating pipeline... done.
Reloading data...
 - Chat::Session...
   done in 0 seconds.                  
 - Cti::Log...
   done in 0 seconds.                  
 - Group...
   done in 0 seconds.                  
 - KnowledgeBase::Answer::Translation...
   done in 0 seconds.                  
 - KnowledgeBase::Category::Translation...
   done in 0 seconds.                  
 - KnowledgeBase::Translation...
   done in 0 seconds.                  
 - Organization...
   done in 0 seconds.                  
 - StatsStore...
   done in 1 seconds.                  
 - Ticket::Priority...
   done in 0 seconds.                  
 - Ticket::State...
   done in 0 seconds.                  
 - Ticket...
   done in 479 seconds.                  
 - User...
   done in 2 seconds.                  
DEPRECATION WARNING: The rake task 'searchindex:rebuild' is deprecated. Use 'zammad:searchindex:rebuild' instead.
zammad@f2f89d2a76c2:~$