Monitoring - Failed to run background job #1 'BackgroundJobSearchIndex'

[andreasklaedtke]

Infos:

• Used Zammad version : 2.6.0-1534493073.5a723290.xenial amd64
• Installation method : via DEB-Package
• Operating system : Ubuntu 16.04.5 LTS
• Database + version : Postgresql 9.5+173ubuntu0.2
• Elasticsearch version: 5.6.10
• Browser + version : Firefox Quantum 61.0.2 (64-Bit)

Expected behavior:

• Zammad Monitoring > Current Status
  OK, no problems in Monitoring.

Actual behavior:

• Failed to run background job #1 ‘BackgroundJobSearchIndex’ 5 time(s) with 59 attempt(s).

Steps to reproduce the behavior:

• receive or send messages with Excel files attached (xlsx, xls)

I have already found a thread for the status messages displayed.

Failing background jobs.;Failed to run background job #1 'BackgroundJobSearchIndex' 10 time(s)

Zammad 2.4 docker-compose upgraded from 2.3 Hi, I’m seeing that message in settings -> System -> Monitoring 16479 failing background jobs. Failed to run background job #1 'BackgroundJobSearchIndex' 10 time(s) with 130 attempt(s). As far as I can tell search is working. I tried manually running reindex rake searchindex:rebuild and it completes ok without error, but the warning count grows. I didn’t do anything special to upgrade the installation, I just pulled the new images and downed the…

I have run the code there. The status in the monitoring section was then “OK”.

items = SearchIndexBackend.search('preferences.notification_sound.enabled:*', 3000, 'User')
items.each {|item|
  next if !item[:id]
  user = User.find_by(id: item[:id])
  next if !user
  next if !user.preferences
  next if !user.preferences[:notification_sound]
  next if !user.preferences[:notification_sound][:enabled]
  if user.preferences[:notification_sound][:enabled] == 'true'
    user.preferences[:notification_sound][:enabled] = true
    user.save!
    next
  end
  next if user.preferences[:notification_sound][:enabled] != 'false'
  user.preferences[:notification_sound][:enabled] = false
  user.save!
  next
}

Delayed::Job.all.each {|job|
  Delayed::Worker.new.run(job)
} 

One day later, the previous monitoring status is displayed again.
When I rerun the code, I get the following messages:

Payload size: 0M
2018-08-17T17:15:07+0200: [Worker(host:zammad pid:24612)] Job BackgroundJobSearchIndex (id=223569) FAILED (12 prior attempts) with RuntimeError: Unable to process POST request to elasticsearch URL 'http://127.0.0.1:9200/zammad_zammad_production/Ticket/1452?pipeline=zammad483532993431'. Check the response and payload for detailed information:


Response:
#<UserAgent::Result:0x000055cf3e129400 @success=false, @body="{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Limit of total fields [1000] in index [zammad_zammad_production] has been exceeded\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Limit of total fields [1000] in index [zammad_zammad_production] has been exceeded\"},\"status\":400}", @data=nil, @code="400", @content_type=nil, @error="Client Error: #<Net::HTTPBadRequest 400 Bad Request readbody=true>!">


2018-08-17T17:15:07+0200: [Worker(host:zammad pid:24612)] Job BackgroundJobSearchIndex (id=227549) COMPLETED after 0.0711
=> [#<Delayed::Backend::ActiveRecord::Job id: 225323, priority: 0, attempts: 12, handler: "--- !ruby/object:BackgroundJobSearchIndex\nobject: ...", last_error: "Unable to process POST request to elasticsearch UR...", run_at: "2018-08-17 21:00:45", locked_at: nil, failed_at: nil, locked_by: nil, queue: nil, created_at: "2018-08-17 13:10:26", updated_at: "2018-08-17 15:15:04">, 

Does anyone have an idea how I can find the cause of this behavior and let the BackgroundJobs run successfully again?

I’ve also added the following options in Elasticsearch:

zammad run rails r "Setting.set('es_attachment_ignore', [ '.png', '.jpg', '.jpeg', '.mpeg', '.mpg', '.mov', '.bin', '.exe', '.box', '.mbox', '.xls', '.xlsx', '.xlsm', '.doc', '.docx' ] )"

Thanks for your help and support.

[chesty]

@andreasklaedtke

Limit of total fields [1000] in index [zammad_zammad_production] has been exceeded

That’s an error from elasticsearch, I’m not sure what exactly is going on so you might want to wait for someone else to respond, but here are some urls on how to increase the field limit. No idea if increasing the field limit is a good idea, or if the error indicates some other problem that needs addressing.

Elasticsearch bulk upload error with PHP - Limit of total fields [1000] in index has been exceeded

Total Fields Limit setting

[RAD-X]

Is there any solution? I have similar or even same problem.

OS: Debian 3.16.51-3+deb8u1 (2018-01-08)
Zammad: 2.6.x
ES: “version” : {
“number” : “5.6.11”,
“build_hash” : “bc3eef4”,
“build_date” : “2018-08-16T15:25:17.293Z”,
“build_snapshot” : false,
“lucene_version” : “6.6.1”
},

Using command

zammad run rake searchindex:rebuild

shows

Limit of total fields [1000] in index [zammad_production] has been exceeded

Also looks like that id of problematic ticket is random and even shows in tickets without any attachment.
My Elastic skills equal to zero… but increasing value using

curl -XPUT "http://127.0.0.1:9200/zammad_production/_settings" -d '
{
"index" : {
"mapping.total_fields.limit" : "2000"
 }
}
}'

didn’t works - look like that indexing config is reseted after using a searchindex:rebuild.