Zammad 6.3 performance issues

[gestinet]

Infos:

• Zammad Version: 6.3.1
• Installation Type: Package
• Operating System: Ubuntu 22.04.4 LTS
• Browser + Version: Any browser (currently using Chrome, Version 125.0.6422.113)

Expected Behavior:

• Fast ticket display in overviews, quick API responses, efficient database connections, and overall fast performance.

Actual Behavior:

• A high volume of tickets being created and updated simultaneously causes a backlog of jobs. Zammad starts using 100% CPU and displays the message ‘Connection lost’ to users.

Steps to Reproduce the Behavior:

System Configuration:

• 12 CPU cores
• 32 GB of RAM
• 36 to 42 active users working simultaneously
• 36 active overviews, all with fewer than 500 tickets except one with 6000 tickets
• More than 100 tickets created per day
• Elasticsearch configuration file /etc/elasticsearch/jvm.options.d/custom-options.options limits Elasticsearch to -Xms5g
• Zammad environment variables:
  • ZAMMAD_SESSION_JOBS_CONCURRENT set to 16
  • WEB_CONCURRENCY set to 4
  • MAX_THREADS set to 30
  • MIN_THREADS set to 8
• Memcached is installed, but not sure if I have configured it correctly.
• PostgreSQL configuration file /etc/postgresql/14/main:
  • max_connections increased from 100 to 300
  • shared_buffers increased from 128MB to 4096MB

We have extensively searched community posts and implemented various solutions recommended for similar issues. Despite this, we are still experiencing performance problems. We have observed that other zammad users with a similar or higher number of agents and tickets per day, but with less RAM or CPU cores, are operating without issues. This suggests we might be missing a crucial configuration or optimization.

We are seeking help to improve our system’s performance. Any guidance or suggestions would be greatly appreciated.

[MrGeneration]

@gestinet

• 12 CPU cores

based on this…

@gestinet

• ZAMMAD_SESSION_JOBS_CONCURRENT set to 16
• WEB_CONCURRENCY set to 4

this is highly overbooked. In worst case, you have 20 processes that burst to 100% (or rather would like to).

Also…
that web concurrency doesn’t sound like it fits the number of agents, aim for 6 especially if you have issues with the web ui. Also database max_connection of 300 is far below the requirement for your chosen tuning. 50 connections per process by default. Run the math yourself. Our recommendation is 2000. See:
https://docs.zammad.org/en/latest/appendix/configure-database-server.html

@gestinet

We have observed that other zammad users with a similar or higher number of agents and tickets per day, but with less RAM or CPU cores, are operating without issues.

I’d say it’s not a memory issue. I’d guess your top or (even better) htop would show below 16gb of memory use.

@gestinet

36 active overviews, all with fewer than 500 tickets except one with 6000 tickets

And the one exception is an issue. It’s useless to lookup 6000 tickets and then throw away 3900 of them. See: Overviews

@gestinet

Zammad starts using 100% CPU

The interesting question you should lookup is “what process is it?” There is a difference if it’s the pumas or the background worker.

I’d guess the latter with your overviews and thus session jobs concurrent might be an option. See: Configuration via Environment Variables — Zammad System Documentation documentation