We were able to narrow the problem down with the help of the fantastic Zammad support!
As described in Issue #2108 the scheduler get’s blocked by recreating the overviews for all agents as soon as they hit a critical limit (about 30 active agents, each having 30 or more overviews).
To circumvent the problem one can disable the generation of overviews in the main scheduler:
systemctl stop zammad-worker
zammad run rails r 'Scheduler.find_by(method: "Sessions.jobs").update(active: false)'
systemctl start zammad-worker
Afterwards it is possible to start a separate Job to recreate the overviews. This way this process doesn’t interfere with the regular Job queue. Best done in screen/tmux session to keep it running.
zammad run rails r 'Sessions.jobs'
With this my delayed jobs count went down to 0 in a few minutes!