Scheduler simply not running

hystiff · May 4, 2020, 11:29am

Infos:

Used Zammad version: 3.3.x
Used Zammad installation source: (source, package, …): https://docs.zammad.org/en/latest/install/debian.html
Operating system: Debian GNU/Linux 9
Browser + version: Chrome 81.0.4044.129 (Official)

Expected behavior:

Scheduled tasks running

Actual behavior:

Scheduler doesn’t run, the scheduled tasks are not launched

Steps to reproduce the behavior:

Create a job, wait…

#####################################

Hi Zammad Community,

We use Zammad since 3 months with 10 agents. Till nos we didn’t need to use schedulers but by now it would become really usefull.

When I create (quite simple) tasks, they are not executed.
One of my tesks should run every 10 minutes but nothing happens, here is some info :

root@ip-172-26-15-154:/home/***# zammad run rails r 'p Delayed::Job.count'
0
root@ip-172-26-15-154:/home/***# zammad run rails r 'p Delayed::Job.first'
nil
root@ip-172-26-15-154:/home/***# zammad run rails r 'p Delayed::Job.last'
nil

Have I forgotten something during the setup?
How can I debug (which commands)?

Hope you will help.
Thanks.

Guillaume.

MrGeneration · May 5, 2020, 10:55am

Hey,

your ideas where good, unfortunately you’re looking into the wrong direction.
The scheduler is called within the Job context - you listing all Jobs would require this command:

zammad run rails r "p Job.pluck(:name)"

Let’s suppose your scheduler task is called “test” - the following attributes might be interesting:
(run the following within a rails console (zammad run rails c):

Job.find_by(name:'test').last_run_at
Job.find_by(name:'test').running
Job.find_by(name:'test').processed
Job.find_by(name:'test').matching

Please provide the outputs of above command.
Possibly Zammad just forgot that the Job has run already or you might have performance issues or are missing something that’s not yet clear.

A good idea would also to check for errors in your logfile, especially for your “every 10 minutes” job, this should appear regulary.

grep "ERROR" /var/log/zammad/production.log

hystiff · May 5, 2020, 11:58am

Hello

Here are the results I get.

zammad run rails r "p Job.pluck(:name)"
xyz@xyz:/home/admin# zammad run rails r "p Job.pluck(:name)"
["cleanspam"]

And,

irb(main):005:0> Job.find_by(name:'cleanspam').running
=> false
irb(main):006:0> Job.find_by(name:'cleanspam').processed
=> 0
irb(main):007:0> Job.find_by(name:'cleanspam').matching
=> 1965

About the errors, I don’t see anything relative to the scheduler :

xyz@xyz/home/admin# grep "ERROR" /var/log/zammad/production.log
E, [2020-05-05T06:50:22.430343 #571-46933866408820] ERROR -- : push ticket_create for user 8
E, [2020-05-05T06:50:22.653448 #571-46933866408820] ERROR -- : push ticket_create for user 3
E, [2020-05-05T06:50:22.711496 #571-46933866408820] ERROR -- : push ticket_create for user 4
E, [2020-05-05T07:20:14.807830 #516-47429898277760] ERROR -- : No route matches [GET] /remote/login (A
ctionController::RoutingError)
E, [2020-05-05T09:17:00.629896 #516-70112903139940] ERROR -- : PG::TRDeadlockDetected: ERROR:  deadloc
k detected
E, [2020-05-05T09:39:55.486770 #571-46933866408820] ERROR -- : push ticket_create for user 1475
E, [2020-05-05T09:39:55.565851 #571-46933866408820] ERROR -- : push ticket_create for user 3
E, [2020-05-05T09:39:55.609547 #571-46933866408820] ERROR -- : push ticket_create for user 34
E, [2020-05-05T09:39:55.686376 #571-46933866408820] ERROR -- : push ticket_create for user 4
E, [2020-05-05T09:39:55.736617 #571-46933866408820] ERROR -- : push ticket_create for user 8

Any ideas?
Many thanks.

Guillaume

MrGeneration · May 5, 2020, 12:04pm

Your system doesn’t seem to be able to keep up with the need to write/read in your database with the need Zammad builts.

This can have various reasons. Either your system has a very slow storage, the system ressources may not fit your concurrent agents working in Zammad or a configuration that is very sub optimal for Zammad to run with acceptable performance.
This should also be noticable during working in Zammad, e.g. long opening times or waiting quite some time for updating tickets.

Please provide the output of the following commands from a rails console:

User.count
Ticket.count
Ticket::Article.count
User.joins(roles: :permissions).where(roles: { active: true }, permissions: { name: 'ticket.agent', active: true }).uniq.count
Sessions.list.count
Overview.count
Group.count
Job.find_by(name:'cleanspam')

Especially the command Sessions.list.count makes the most sense in situations where your system faces the most load during the day (so when everyone is working).

hystiff · May 5, 2020, 12:56pm

The results :

irb(main):001:0> User.count
=> 2031
irb(main):002:0> Ticket.count
=> 3320
irb(main):003:0> Ticket::Article.count
=> 9091
irb(main):004:0> User.joins(roles: :permissions).where(roles: { active: true }, permissions: { name: '
ticket.agent', active: true }).uniq.count
=> 15
irb(main):005:0> Sessions.list.count
=> 4
irb(main):006:0> Overview.count
=> 21
irb(main):007:0> Group.count
=> 10
irb(main):008:0> Job.find_by(name:'cleanspam')
=> #<Job id: 9, name: "cleanspam", timeplan: {"days"=>{"Mon"=>true, "Tue"=>true, "Wed"=>true, "Thu"=>t
rue, "Fri"=>true, "Sat"=>true, "Sun"=>true}, "hours"=>{"0"=>true, "1"=>true, "2"=>true, "3"=>true, "4"
=>true, "5"=>true, "6"=>true, "7"=>true, "8"=>true, "9"=>true, "10"=>true, "11"=>true, "12"=>true, "13
"=>true, "14"=>true, "15"=>true, "16"=>true, "17"=>true, "18"=>true, "19"=>true, "20"=>true, "21"=>tru
e, "22"=>true, "23"=>true}, "minutes"=>{"0"=>true, "10"=>true, "20"=>true, "30"=>true, "40"=>true, "50
"=>true}}, condition: {"ticket.tags"=>{"operator"=>"contains all", "value"=>"spam"}}, perform: {"ticke
t.action"=>{"value"=>"delete"}}, disable_notification: true, last_run_at: nil, next_run_at: "2020-05-0
5 11:40:00", running: false, processed: 0, matching: 1965, pid: nil, note: "", active: true, updated_b
y_id: 3, created_by_id: 3, created_at: "2020-04-30 11:52:50", updated_at: "2020-05-05 11:30:49">
irb(main):009:0>

We use this VM on AWS : RAM 16 Go, 4 vCPU, 320 Go SSD

MrGeneration · May 5, 2020, 3:57pm

According to your Job, it never ran up to now. I also guess “next_run” just keeps going up without any real run.

Please run zammad run rails r "p Scheduler.find_by(name: 'Execute jobs')" and provide the output.

hystiff · May 5, 2020, 6:04pm

I get :

   <Scheduler id: 8, name: "Execute jobs", method: "Job.run", period: 300, running: 0, last_run: "2019-1
    1-28 08:52:21", prio: 2, pid: "", note: nil, error_message: "Failed to run Job.run after 10 tries #<Ru
    ntimeErro...", status: "error", active: false, updated_by_id: 3, created_by_id: 1, created_at: "2019-0
    9-12 06:18:04", updated_at: "2019-11-28 08:52:21">

hystiff · May 6, 2020, 1:27pm

Hi,

I read active: false
But I confirm that this job is displayed as active on the interface (not strikeouted).

Guillaume

MrGeneration · May 11, 2020, 7:58pm

Did you disable Scheduler jobs by any chance? By default it should be active:
zammad run rails r "Scheduler.find(8).update(active: true)

Starting from that moment (give Zammad some grace period (20 minutes), your scheduler jobs should run. Heads up on the naming convention here:

Schedulers on the rails console are not schedulers that are mentioned in the UI.
Those are “Job”(s) on the rails console. Hope that helps to clear the confusion.

hystiff · May 12, 2020, 6:17am

Marvellous, it seems it works !

I don’t understand how/when I disable this, but …SOLVED.
Many thanks @MrGeneration

Guillaume

system · September 9, 2020, 6:17am

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.