Failing background jobs.;Failed to run background job #1 'BackgroundJobSearchIndex' 10 time(s)


#1

Zammad 2.4 docker-compose upgraded from 2.3

Hi, I’m seeing that message in settings -> System -> Monitoring

16479 failing background jobs.
Failed to run background job #1 'BackgroundJobSearchIndex' 10 time(s) with 130 attempt(s). 

As far as I can tell search is working. I tried manually running reindex

rake searchindex:rebuild

and it completes ok without error, but the warning count grows.

I didn’t do anything special to upgrade the installation, I just pulled the new images and downed the old ones, and upped the new ones.

I also has an LDAP sync that wouldn’t finish, but found the way to mark it finished and restart it and it synced OK. But, and I don’t know if this is related, when I drag a ticket to the bottom to assign it to someone there’s some weirdness and it doesn’t work as expected. Should I open a separate topic for that do you reckon?


Monitoring - Failed to run background job #1 'BackgroundJobSearchIndex'
#2

Hi @chesty

I had something similar today. My problem was the there was a wrong data type stored in user preferences.

Can you execute the following via “zammad run rails c” and can you tell if you still have failing jobs?

items = SearchIndexBackend.search('preferences.notification_sound.enabled:*', 3000, 'User')
items.each {|item|
  next if !item[:id]
  user = User.find_by(id: item[:id])
  next if !user
  next if !user.preferences
  next if !user.preferences[:notification_sound]
  next if !user.preferences[:notification_sound][:enabled]
  if user.preferences[:notification_sound][:enabled] == 'true'
    user.preferences[:notification_sound][:enabled] = true
    user.save!
    next
  end
  next if user.preferences[:notification_sound][:enabled] != 'false'
  user.preferences[:notification_sound][:enabled] = false
  user.save!
  next
}

Delayed::Job.all.each {|job|
  Delayed::Worker.new.run(job)
} 

Feedback is appreciated.

-Martin


#3

cheers martini,

I’m running it, there are over 15,000 failed jobs that I guess are being run
I’ll get back to you, i imagine it will be tomorrow for me, in about 16 hours.


#4

@martini

It appears to be all fixed. Thanks Martin. All old jobs have completed and no new failed jobs in 12 hours.


#5

Wunderful. There is already an issue https://github.com/zammad/zammad/issues/1933 - I’ll create an migration for that to cleanup all instances (in case there is the same issue).


#6

@martini got the same issue, this statement didn’t fix the problem for me.
Currently: Failed to run background job #1 ‘BackgroundJobSearchIndex’ 4 time(s) with 22 attempt(s).


#7

@martini I had the same issue and your fix helped - it is not giving any errors since.

Thanks


#8

Hey,

did not work for me too - I’m getting the following output (not everything as the output hit terminal buffer limits

I uploaded this as output1.txt, as I hit the character limits of this board <.<
https://cloud.smrg.de/index.php/s/o7F9Owm3hISvlID/download

So now my monitoring has the following state:
11 failing background jobs.
Failed to run background job #1 'BackgroundJobSearchIndex' 10 time(s) with 236 attempt(s).

I then tried it with reindexing and configuring elasticsearch, as I hoped that something is just going wrong (after doc: https://docs.zammad.org/en/latest/install-elasticsearch.html)

Output2.txt is the output of zammad run rake searchindex:rebuild - wasn’t too sucessfull.
https://cloud.smrg.de/index.php/s/vR4z48leMxSObr3/download

I am not sure if this is a bug of zammad, a problem with elasticsearch itself or my instance being trash.

I have elasticsearch version 5.6.6 installed


#9

@MrGeneration, I had the Failling backgroub jobs error again. This time the cause was different. I had an article with an xlsx attachment (I think it was 4M) and the POST request to add it to the index was timing out. I think the timeout 12 seconds by default. I tried bumping it up to 60 and it still timed out. So I added xlsx to the ignore list by running in the “rails c”

Setting.set('es_attachment_ignore', [".png", ".jpg", ".jpeg", ".mpeg", ".mpg", ".mov", ".bin", ".exe", ".box", ".mbox", ".xlsx"])

Your error looks similar in that it’s a ticket that’s causing the problem, but it’s not the same problem. I don’t know what it is but it doesn’t like something about ticket 890.


#10

@chesty thanks for your input. I’ve taken a look into Ticket #890 because I was interested, what makes this ticket so special. My problem: I have no idea, this Ticket has no attachment and actually comes from channel “Twitter”

Maybe a feature?


#11

I’m having the same issue after running a large LDAP sync over the weekend that failed.
Failed to run background job #1 ‘BackgroundJobSearchIndex’ 5 time(s) with 5 attempt(s).

Right now it has broken notifications, both email and on the Zammad dashboard.

I’m running what @martini recommended but it’s going to take some time to finish with the size of our environment. What’s interesting is that a few seconds after running his command it pushes notifications through but not permanently as notifications are not working right now.

Any input would be much appreciated!

-Rick

Zammad version 2.4.x


#12

Hi @Mk1DzL,

I thought the notification_sound bug had been fixed, I just looked in github, I think it was in late april for 2.4.1
Maybe you need to update to the latest 2.4?
Also 2.5 was released a few days ago.


#13

Thanks, @chesty. @martini’s command seems to have fixed that error and if all is healthy I’ll upgrade this Thurday’s during our patch window.

It’s still showing scheduler not running in the monitoring section…I’m looking into that now.

Thanks!

-Rick


#14

My problems have been solved with the upgrade to 2.5 regarding this problem :slight_smile:


#15

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.