Scheduler tasks limited to one CPU thread

KTMobility · July 28, 2022, 2:22pm

Hello,

I have a question regarding the performance of the task scheduler.
I couldn’t find any topic about CPU multithreading.

Info:

Used Zammad version: 5.1.1
Used Zammad installation type: package
Operating system: CentOS 7
Browser + version: Firefox v102

Expected behavior:

Have a fast execution of tasks modifying more than 500 tickets in the scheduler (at least, use a larger CPU percentage of the machine)

Actual behavior:

A task that modifies about 1000 tickets (700/1100/2000+ for testing) in the scheduler runs in about 5 minutes, and ticket changes are visible after about 30 minutes.
The “top” utility indicates 100% utilization (single thread) for the zammad user’s ruby command (script?) when the task is being performed (drops to 0% when finished).
Total CPU usage does not exceed 16% (8 Threads).

Steps to reproduce the behavior:

Create a task in the scheduler which aims to modify a “select” field on a large selection of tickets (1000+).
Example: all tickets in the “open” state in group A are moved to group B with the state “new”.
Have a 4+ thread cpu machine and check the performance monitor (does not exceed 1 thread).

To simply ask: Do you know if it is possible to enable multithreading on zammad’s ruby scripts?

I have also observed that tasks modifying more than 2000 tickets show the correct number of modified elements, but only really modify 2000 (tickets are not modified from 2001+).
A ticket on Github is already open, so I won’t talk about it here.

github.com/zammad/zammad

Scheduler 'processed' incorrect value if above 2000 elements

opened 10:35AM - 12 Nov 19 UTC

MrGeneration

bug ticket verified prioritised by payment

### Infos: * Used Zammad version: 3.1 * Installation method (source, package, ..): any * Operating system: any * Database + version: any * Elasticsearch version: any * Browser + version: any * Ticket-ID: 1058202 ### Expected behavior: Zammad shows the number of touched tickets during scheduler job runtime (exact number). ### Actual behavior: When having more than 2.000 tickets (which is our limit per one run) Zammad will act like it did work on all tickets implied, while it doesn't. This is confusing users, as the "processed" counts implies that Zammad worked on tickets it hasn't. In the exact example, this happened with somewhat around 40.000 tickets - Zammad said `processed: 40.000` but in the next step also says `going to process: 38.000` which confuses. Samples: ```ruby #<Job id: 3, name: "magic", timeplan: {"days"=>{"Mon"=>false, "Tue"=>true, "Wed"=>false, "Thu"=>false, "Fri"=>false, "Sat"=>false, "Sun"=>false}, "hours"=>{"0"=>false, "1"=>false, "2"=>false, "3"=>false, "4"=>false, "5"=>false, "6"=>false, "7"=>true, "8"=>true, "9"=>true, "10"=>true, "11"=>true, "12"=>true, "13"=>true, "14"=>true, "15"=>true, "16"=>true, "17"=>true, "18"=>true, "19"=>false, "20"=>false, "21"=>false, "22"=>false, "23"=>false}, "minutes"=>{"0"=>true, "10"=>false, "20"=>false, "30"=>false, "40"=>true, "50"=>false}}, condition: {"ticket.close_at"=>{"operator"=>"before (relative)", "value"=>"3", "range"=>"month"}, "ticket.tags"=>{"operator"=>"contains one not", "value"=>"komplett_löschen_3m"}}, perform: {"ticket.tags"=>{"operator"=>"add", "value"=>"komplett_löschen_3m"}}, disable_notification: true, last_run_at: "2019-11-12 09:44:51", next_run_at: "2019-11-12 10:00:00", running: false, processed: 38344, matching: 36360, pid: nil, note: "magical tagging", active: true, updated_by_id: 1, created_by_id: 18, created_at: "2019-11-12 07:34:45", updated_at: "2019-11-12 09:44:51"> ``` an hour later: ```ruby #<Job id: 3, name: "magic", timeplan: {"days"=>{"Mon"=>false, "Tue"=>true, "Wed"=>false, "Thu"=>false, "Fri"=>false, "Sat"=>false, "Sun"=>false}, "hours"=>{"0"=>false, "1"=>false, "2"=>false, "3"=>false, "4"=>false, "5"=>false, "6"=>false, "7"=>true, "8"=>true, "9"=>true, "10"=>true, "11"=>true, "12"=>true, "13"=>true, "14"=>true, "15"=>true, "16"=>true, "17"=>true, "18"=>true, "19"=>false, "20"=>false, "21"=>false, "22"=>false, "23"=>false}, "minutes"=>{"0"=>true, "10"=>false, "20"=>false, "30"=>false, "40"=>true, "50"=>false}}, condition: {"ticket.close_at"=>{"operator"=>"before (relative)", "value"=>"3", "range"=>"month"}, "ticket.tags"=>{"operator"=>"contains one not", "value"=>"komplett_löschen_3m"}}, perform: {"ticket.tags"=>{"operator"=>"add", "value"=>"komplett_löschen_3m"}}, disable_notification: true, last_run_at: "2019-11-12 10:10:17", next_run_at: "2019-11-12 10:40:00", running: false, processed: 36396, matching: 34408, pid: nil, note: "magical taggingg", active: true, updated_by_id: 1, created_by_id: 18, created_at: "2019-11-12 07:34:45", updated_at: "2019-11-12 10:10:17"> ``` ### Steps to reproduce the behavior: * create more than 2000 tickets * create a scheduler that shall touch all those tickets and do something with them (e.g. add a tag) * after runtime check what Zammad said it processed and then check what it really did process Yes I'm sure this is a bug and no feature request or a general question.

Thanks !
Regards

KTMobility · August 4, 2022, 10:26am

Spotted !

Ok mates, I think I found my happiness.
I saw that the doc was updated :
https://docs.zammad.org/en/latest/appendix/configure-env-vars.html#performance-tuning

There is a new variable : ZAMMAD_PROCESS_DELAYED_JOBS_WORKERS (comes with Zammad 5.2.1)
I updated my zammad installation from 5.1.1 to 5.2.1 and put “zammad config:set ZAMMAD_PROCESS_DELAYED_JOBS_WORKERS=4” then a restart.

Et voilà : now, a big scheduled job with a 1950 ticket modification run on approx. 8 minutes (before, was 30~40 minutes).
On “top”, we can see the traditionnal zammad’s ruby command with ~90% cpu’s thread, and 4 more ruby commands who take ~80~90%.
The total cpu’s use is about 50~60% (I didn’t want to put more, in case)

Note : the ZAMMAD_SESSION_JOBS_CONCURRENT variable was already set to 4 before this change, but didn’t help me.

Hope this can help !

system · December 2, 2022, 10:26am

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.