Infos:
- Used Zammad version: 3.1.0-17
- Used Zammad installation source: Official docker hub
- Operating system: Ubuntu 18.04 Server LTS
- Browser + version: Chrome 77
Expected behavior:
- Delayed jobs are processed well, queue is not growing
Actual behavior:
- We have growth around +10k of delayed job DAILY!
Steps to reproduce the behavior:
- Virtual server with 8 cores, 40 Gb of RAM
- Around 20 agents
- Around 17k tickets
- Total 15 overviews
Just keep working in system. At the end of day there is 9-10k of jobs in queue.
Also, I can’t run rake in latest docker container, It gives error:
root@dce5bdfbb7eb:/opt/zammad# rake
I, [2019-09-25T18:49:03.064984 #475-47331145835960] INFO -- : Setting.set('models_searchable', ["Organization", "User", "Ticket", "Chat::Session", "KnowledgeBase::Answer::Translation"])
rake aborted!
LoadError: cannot load such file -- selenium-webdriver
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/usr/local/bundle/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `block in require'
/usr/local/bundle/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency'
/usr/local/bundle/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
/opt/zammad/test/browser_test_helper.rb:4:in `<main>'
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
/usr/local/bundle/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `block in require'
/usr/local/bundle/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency'
/usr/local/bundle/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
/opt/zammad/test/browser/aaa_getting_started_test.rb:1:in `<main>'
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
/usr/local/bundle/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
/usr/local/bundle/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `block in require'
/usr/local/bundle/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency'
/usr/local/bundle/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
/usr/local/bundle/gems/railties-5.2.3/lib/rails/test_unit/runner.rb:50:in `block in load_tests'
/usr/local/bundle/gems/railties-5.2.3/lib/rails/test_unit/runner.rb:50:in `each'
/usr/local/bundle/gems/railties-5.2.3/lib/rails/test_unit/runner.rb:50:in `load_tests'
/usr/local/bundle/gems/railties-5.2.3/lib/rails/test_unit/runner.rb:39:in `run'
/usr/local/bundle/gems/railties-5.2.3/lib/rails/test_unit/runner.rb:35:in `rake_run'
/usr/local/bundle/gems/railties-5.2.3/lib/rails/test_unit/testing.rake:16:in `block in <main>'
/usr/local/bundle/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
Tasks: TOP => default => test
(See full trace by running task with --trace)
Scheduler’s log sometimes has errors:
zammad-scheduler_1 | E, [2019-09-25T19:09:26.660725 #1-69906702651020] ERROR -- : /usr/local/bundle/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:201:in `block in wait_poll'
zammad-scheduler_1 | /usr/local/bundle/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:190:in `loop'
zammad-scheduler_1 | /usr/local/bundle/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:190:in `wait_poll'
zammad-scheduler_1 | /usr/local/bundle/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:151:in `internal_poll'
zammad-scheduler_1 | /usr/local/bundle/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:277:in `internal_poll'
zammad-scheduler_1 | /usr/local/bundle/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:145:in `block in poll'
zammad-scheduler_1 | /usr/local/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
zammad-scheduler_1 | /usr/local/bundle/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:155:in `synchronize'
zammad-scheduler_1 | /usr/local/bundle/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:145:in `poll'
zammad-scheduler_1 | /usr/local/bundle/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:799:in `acquire_connection'
zammad-scheduler_1 | /usr/local/bundle/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:523:in `checkout'
zammad-scheduler_1 | /usr/local/bundle/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:382:in `connection'
zammad-scheduler_1 | /usr/local/bundle/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:1014:in `retrieve_connection'
zammad-scheduler_1 | /usr/local/bundle/gems/activerecord-5.2.3/lib/active_record/connection_handling.rb:118:in `retrieve_connection'
zammad-scheduler_1 | /usr/local/bundle/gems/activerecord-5.2.3/lib/active_record/connection_handling.rb:90:in `connection'
zammad-scheduler_1 | /usr/local/bundle/gems/activerecord-5.2.3/lib/active_record/relation/calculations.rb:196:in `block in pluck'
zammad-scheduler_1 | /usr/local/bundle/gems/activerecord-5.2.3/lib/active_record/relation.rb:584:in `skip_query_cache_if_necessary'
zammad-scheduler_1 | /usr/local/bundle/gems/activerecord-5.2.3/lib/active_record/relation/calculations.rb:196:in `pluck'
zammad-scheduler_1 | /opt/zammad/app/models/ticket/overviews.rb:20:in `all'
zammad-scheduler_1 | /opt/zammad/app/models/ticket/overviews.rb:79:in `index'
Postgres shows 125-175 connections.
Yesterday I’ve truncated this table and performed zammad run rake searchindex:rebuild but today we have again 10k jobs.
What could we do this that? 
Our clients are suffering from big notifications delays.
UPD: Currently it hang at least for 8 minutes on this job:
--- !ruby/object:Transaction::BackgroundJob
item:
:object: Ticket
:object_id: 17416
:user_id: 1074
:created_at: !ruby/object:ActiveSupport::TimeWithZone
utc: &1 2019-09-25 10:15:50.517752491 Z
zone: !ruby/object:ActiveSupport::TimeZone
name: Etc/UTC
time: *1
:type: update
:article_id: 185370
:changes:
time_unit:
- !ruby/object:BigDecimal 18:0.5e1
- !ruby/object:BigDecimal 18:0.8e1
params:
:interface_handle: application_server