/#system/monitoring should say Current Status: no issues
Actual behavior:
/#system/monitoring says intermittently Current Status: Failed to run background job #1 'SearchIndexAssociationsJob' 3 time(s) with 48 attempt(s).
Steps to reproduce the behavior:
Not reproducible but happens intermittently on both our larger zammad setups
What we did so far
The issues happened last year in december and I currently do not have any logs to give more insight here. As soon as it happens again I will provide the relevant logs of course. I’m just posting in the hopes maybe someone else had the same issue and could provide hints until we can add our logs to this post.
We found this website on our search and also tried to rebuild the ES indices because why not, since if the culprit are b0rked indices due to some past upgrade issues this could solve the issue. But it did not fix the issue.
We just had a similar issue. Not sure how much of this applies to your situation but I’ll share our learnings.
TL;DR
Docker host had been running low on space. Elasticsearch seems to stop writing to disk if less than 90% free disk is available. Issue was resolved by freeing up space on the docker host.
Infos:
Zammad 5.2.3-4
Install type: docker-compose
OS Rocky Linux 8.7
Actual behavior:
Zammad search returns incomplete results (searching for a specific tag or group doesn’t include tickets that recently were created)
Health check reports errors: 41 failing background jobs. Failed to run background job #1 'SearchIndexAssociationsJob' 8 time(s) with 130 attempt(s). Failed to run background job #2 'SearchIndexJob' 2 time(s) with 36 attempt(s)
What we did so far
Reviewed container logs. It’s alerting that it has low disk space (less than 90%)
2023-01-13T15:18:01.415903337Z WARN high disk watermark [90%] exceeded on [blablabla][blablabla][/usr/share/elasticsearch/data/nodes/0]
*E, [2023-01-13T18:16:32.868307 #1-109360] ERROR -- : Unable to process post request to elasticsearch URL 'http://zammad-elasticsearch:9200/zammad_production_stats_store/_doc/126?pipeline=blablabla'. Check the response and payload for detailed information
We cleared out some space. Zammad no longer reported bad health check
Note
Having search broken for a while caused weird search results, even after elasticsearch stopped throwing errors
This was resolved by rebuilding the search index
In Rails container:
zammad@f2f09d2a66c2:~$ rails searchindex:rebuild
I, [2023-01-13T23:08:04.969239 #940-8060] INFO -- : Using memcached as Rails cache store.
I, [2023-01-13T23:08:04.969357 #940-8060] INFO -- : Using Redis as web socket session store.
I, [2023-01-13T23:08:05.217487 #940-8060] INFO -- : Setting.set('product_logo', "logo.png")
I, [2023-01-13T23:08:06.519300 #940-8060] INFO -- : Setting.set('models_searchable', ["Chat::Session", "KnowledgeBase::Answer::Translation", "Organization", "Ticket", "User"])
Dropping indexes... done.
Deleting pipeline... done.
Creating indexes... done.
Creating pipeline... done.
Reloading data...
- Chat::Session...
done in 0 seconds.
- Cti::Log...
done in 0 seconds.
- Group...
done in 0 seconds.
- KnowledgeBase::Answer::Translation...
done in 0 seconds.
- KnowledgeBase::Category::Translation...
done in 0 seconds.
- KnowledgeBase::Translation...
done in 0 seconds.
- Organization...
done in 0 seconds.
- StatsStore...
done in 1 seconds.
- Ticket::Priority...
done in 0 seconds.
- Ticket::State...
done in 0 seconds.
- Ticket...
done in 479 seconds.
- User...
done in 2 seconds.
DEPRECATION WARNING: The rake task 'searchindex:rebuild' is deprecated. Use 'zammad:searchindex:rebuild' instead.
zammad@f2f89d2a76c2:~$
