Delayed::Job queue continuously increasing, incoming emails delayed ~1 hour, ActiveRecord::ConnectionTimeoutError (Zammad 7.0.1)

Technical assistance
1

Hello everyone,

I’m experiencing a performance issue with Zammad 7.0.1 and I would appreciate any guidance.

Environment

  • Zammad: 7.0.1-1775747706.81b38d6a.bookworm
  • Installation: Official Debian package (not Docker)
  • OS: Debian 12
  • PostgreSQL: External server (PostgreSQL 17.6)
  • Elasticsearch: Local
  • Redis: Local

Server resources:

  • 8 vCPU
  • 16 GB RAM
  • Elasticsearch heap around 4 GB

Problem

Incoming emails are received immediately by the mail server, but they are converted into Zammad tickets with a delay of approximately one hour.

The delay becomes worse as the system receives more emails during the day.

What we found

Delayed jobs continuously increase instead of decreasing.

Examples:

633

812

866

1064

Running:

zammad run rails r “puts Delayed::Job.group(:queue).count”

returns approximately:

{
“default” => 789,
“communication_inbound” => 23
}

The jobs are mostly:

  • SearchIndexJob
  • SearchIndexAssociationsJob
  • TransactionJob
  • TaskbarUpdateRelatedTasksJob
  • TaskbarUpdateTriggerSubscriptionsJob
  • TicketUserTicketCounterJob

There are no failed delayed jobs.

failed_at count = 0

Worker

Only one worker is running:

zammad-worker-1.service

The worker had previously restarted hundreds of times.

The logs contain:

ActiveRecord::ConnectionTimeoutError:
could not obtain a connection from the pool within 5.000 seconds;
all pooled connections were in use

Database

database.yml

pool: 50

Rails confirms:

ActiveRecord::Base.connection_pool.stat

returns

size: 50
connections: 0
busy: 0
dead: 0
idle: 0
waiting: 0

The connection test succeeds:

ActiveRecord::Base.connection.execute(“SELECT version()”)

works correctly.

Also:

10x SELECT pg_sleep(1)

runs successfully.

PostgreSQL server is not saturated.

max_connections = 200

Very few active connections.

Server utilization

CPU usage is low:

~92% idle

Memory is fine:

16 GB RAM
No swap pressure

Redis is healthy.

Elasticsearch is running correctly.

Question

Has anyone seen Delayed::Job continuously growing on Zammad 7.0.1 together with ActiveRecord::ConnectionTimeoutError?

Is this a known issue with Background Services or should we investigate a specific worker configuration?

Any advice would be greatly appreciated.

Thank you.

Additional information:

The PostgreSQL server is hosted on a separate machine.

Increasing the application server RAM from 8 GB to 16 GB did not solve the problem.

The infrastructure is healthy, but Delayed::Job continues to grow over time.

2

Your database server does not meet the requirements.
Please see the documentation for the expected configuration. While this shouldn’t cause such big problems in terms of delay, but it might well contribute to it.

3

I found the actual worker crash.

The worker is not crashing while processing inbound emails.

It crashes while executing the scheduled job:

Generate user-based stats

The stack trace ends with:

Stats::TicketEscalation.generate
Stats.generate

followed by:

ActiveRecord::ConnectionTimeoutError:
could not obtain a connection from the pool within 5 seconds

After that the worker exits with:

Main process exited, status=1/FAILURE

systemd restarts it automatically.

This explains why the Delayed::Job queue grows and later decreases after the worker is restarted.

Could this be a known issue in 7.0.1?

4

Why are you ignoring my initial suggestion to configure your database server up to requirements?

5

Thank you for your feedback.

I realized that during my initial investigation I was checking the wrong PostgreSQL server. I have now verified the actual database server used by Zammad.

The PostgreSQL server is already tuned and is shared with several applications, not only Zammad. Some current settings are:

  • shared_buffers = 3GB
  • effective_cache_size = 6GB
  • maintenance_work_mem = 768MB
  • max_wal_size = 4GB
  • max_connections = 150

During the issue, PostgreSQL is not saturated. There are around 29 Zammad connections, most of them are idle (ClientRead), and I don’t see lock waits or long-running queries.

I understand your recommendation and I will review the remaining PostgreSQL settings against the official Zammad documentation, especially work_mem and checkpoint_timeout.

However, I’m still trying to understand why incoming emails are delayed by about 15 minutes and why the background worker intermittently exits with:

ActiveRecord::ConnectionTimeoutError:could not obtain a connection from the pool within 5 seconds

If you think there is a specific PostgreSQL setting that could explain this behavior, or if there are additional diagnostics you would like me to collect, I’d be happy to provide them.