Search suddenly does not work; no update or change since it last worked

Infos:

  • Used Zammad version: 6.5.0-1743574490.568ebbed.bookworm
  • Used Zammad installation type: package
  • Operating system: Debian 12
  • Browser + version: Brave Version 1.77.100, Chromium Version 135.0.7049.84 , Firefox 137.0.2 (64-bit)

Expected behavior:

  • I type in any term in the UI search box, and any result appears

Actual behavior:

  • I type in any term, no search results appear.

Steps to reproduce the behavior:

  • zammad run rails r “p User.where(‘firstname ILIKE ?’, ‘%mike%’).pluck(:email)”

  • This successfully returns a email address that has ‘mike’ in it.

  • curl -X GET “http://localhost:9200/zammad_production_user/_search?q=mike&pretty

  • This returns a detailed result for the same user.

  • Typing in ‘mike’ in the search box in the UI returns ‘There is mo match for your search’.

  • I reinstalled Zammad, uninstalled and reinstalled the ES plugin, ran the ‘zammad run rake zammad:searchindex:rebuild’ multiple times in between every other attempt to solve.

  • I can’t make heads or tails in the logs. Everything looks 5x5 in there to me.

  • Nothing in the system changed over the weekend. We were closed, no no one accessed tickets, let alone anything server level.

  • Settings > Monitoring also shows this:
    Current Status
    emails that could not be processed: 31
    18 failing background jobs
    Failed to run background job #1 ‘SearchIndexAssociationsJob’ 5 time(s) with 55 attempt(s).
    Failed to run background job #2 ‘SearchIndexJob’ 5 time(s) with 55 attempt(s).

Health Check as json:
{“healthy”:false,“message”:“emails that could not be processed: 31;18 failing background jobs;Failed to run background job #1 ‘SearchIndexAssociationsJob’ 5 time(s) with 70 attempt(s).;Failed to run background job #2 ‘SearchIndexJob’ 5 time(s) with 70 attempt(s).”,“issues”:[“emails that could not be processed: 31”,“18 failing background jobs”,“Failed to run background job #1 ‘SearchIndexAssociationsJob’ 5 time(s) with 70 attempt(s).”,“Failed to run background job #2 ‘SearchIndexJob’ 5 time(s) with 70 attempt(s).”],“actions”:,“token”:“REMOVED”}

Just an update - I found this forum post that addressed the Health Check ‘failed to run background job’ issue.

Running zammad run rails r "Delayed::Job.where(failed_at: nil).each(&:invoke_job)", then systemctl restart zammad elasticsearch cleared most of these errors.

Emails that could not be processed: 31 < the only error that stayed.

However, after about 15 minutes, they all returned. Different quantities of times and attempts.

Another weird symptom… I am attempting to add an existing tag to a ticket. I type in the tag, the tag search results autopopulate as they normally do, then suddenly the page refreshes and Zammad reloads. Then the partially typed out tag is now in place, and is saved as a new tag. This occurrs every time I attempt to add a tag to any ticket.

It didn’t occur to me, originally, to test tag searching. I only tried searching for tickets and users in the top left search box.