High Latency , Timeouts, tickets twice

[pixelflair]

Hey guys,

we started with zammad for our “Sales-Team” …

About ~300mails per day.

The first day everything was fine, but now the webserver has high pings and the
employees can’t work as fast as possible. Sometimes it needs more than 20sec from “click” until the screen is updated…

This is a BIG Problem!

Our nginx config:

upstream zammad-railsserver {
server localhost:3000;
}

upstream zammad-websocket {
server localhost:6042;
}

server {
listen 80;

# replace 'localhost' with your fqdn if you want to use zammad from remote
server_name WMS-CRM.exxellin.local;
server_name 10.181.201.238;

root /opt/zammad/public;

access_log /var/log/nginx/zammad.access.log;
error_log  /var/log/nginx/zammad.error.log;

client_max_body_size 50M;

location ~ ^/(assets/|robots.txt|humans.txt|favicon.ico) {
    expires max;
}

location /ws {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header CLIENT_IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_read_timeout 86400;
   proxy_pass http://zammad-websocket;
}

location / {
    proxy_set_header Host $http_host;
    proxy_set_header CLIENT_IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_read_timeout 300;
    proxy_pass http://zammad-railsserver;

    gzip on;
    gzip_types text/plain text/xml text/css image/svg+xml application/javascript application/x-javascript a$
    gzip_proxied any;
}

}

[MrGeneration]

How’s the memory usuage and CPU usuage on your Server? What Hardware Specifications does the Server have?

Did you take a look directly on your Server and tried to find out if the whole machine is slow or just zammad?

[pixelflair]

@MrGeneration

Did you take a look directly on your Server and tried to find out if the whole machine is slow or just zammad?

Our IT Zammad with about ~30-40Mails per day with the same hardware runs perfectly…

[MrGeneration]

@pixelflair sorry for not coming back to you.

That’s very strange as the configuration, hardware configuration and of course the monitoring stats look perfectly fine (mine don’t look any worse or better). Until now I always blamed the vHost of my Hosting provider whenever a small delay came.

To be honest until now I didn’t really bother, but when you have to work exessive with it, I can understand that this is really troublesome.

What’s the version of elasticsearch and zammad btw?

[pixelflair]

@MrGeneration

To be honest until now I didn’t really bother, but when you have to work exessive with it, I can understand that this is really troublesome.

What’s the version of elasticsearch and zammad btw?

Zammad 2.3
Elasticsearch 5.X

[MrGeneration]

Both machines are running zammad 2.3?
Did you try the new version of zammad? Maybe this addressing some of your problems.

[pixelflair]

Yes both.

i can update it to version 2.4 at the earliest on friday evening

[martini]

Hi @pixelflair

it seems that you are facing with different issues.

1) 100% cpu time
This seems to be the Zammad scheduler -> this has changed in Zammad 2.4 - the scheduler will use much less cpu time

2) Sporadic delays (e. g. 20 sec.)
Per default Zammad starts with one application server process (with max. 30 threats). If you have several concurrent users it may not be enough (http requests will get queued which leads to high latency). Depending on you need you can start more processes (each with max. 30 threads). Just do the following for 4 application server processes:

zammad config:set WEB_CONCURRENCY=4
systemctl restart zammad

Please note, that each application server process needs memory (just have a look on it that the machine is not swapping). Hint: In case you use elasticsearch on same machine, take about 6 GB RAM for elasticsearch it self (it’s not Zammad related).

3) ramdisk

You will get a noticeable improvement if you use an ramdisk for the /opt/zammad/tmp directory.

To test this, just enter the following for an 8 GB ramdisk (temporary ramdisk):

mount -t tmpfs -o size=8g tmpfs /opt/zammad/tmp

To keep it after reboot, enter the following:

echo "tmpfs       /opt/zammad/tmp tmpfs   nodev,nodiratime,size=8g   0 0" >> /etc/fstab

Let me know if this feedback was helpful!

-Martin

[pixelflair]

I made the second point with the config… and now the system runs much better than before…

the other points will be made tommorrow!

Thank you

[monotek]

Imho instead fiddling around with tmpfs we should give memcached a try as described in: https://github.com/zammad/zammad/issues/1601
If its run on the same server as zammad a unix socket could be used so that there is no additional network latency added.

Maybe even the existing POC of rabbitmq could be helpful.

[datenschuft]

Hi @martini

Is there a way to monitor the usage of application server process/threats.
How many processes/threats are used/free.
Sometimes we have sporadic delays, and I don’t know how many of my processes/threats are in use/free.

now we run WEB_CONCURRENCY=2.
Is there a way to Monitor the usage of zammad processes and threats.

I hope that I can figure out the right value for WEB_CONCURRENCY.