Zammad filling up syslog like crazy

Infos:

  • Used Zammad version: 3.4.x
  • Operating system: Ubuntu

Actual behavior:

  • Since updating Zammad a couple of weeks ago via apt, we were running into an issue with the syslog (var/log/syslog) - it is filling up like crazy with Zammad messages, relatively unspecific ones, not actual error messages.

Here is what can be found in the log:

        Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:42:16Z:client(-) Status: websocket clients: 1
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:42:16Z:client(47186522342540) working...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:42:16Z:client(-) Status: ajax clients: 0
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:42:36Z:client(-) Status: websocket clients: 1
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:42:36Z:client(47186522342540) working...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:42:36Z:client(-) Status: ajax clients: 0
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:42:41Z:client(-) check unused idle connections...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:42:56Z:client(-) Status: websocket clients: 1
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:42:56Z:client(47186522342540) working...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:42:56Z:client(-) Status: ajax clients: 0
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:16Z:client(-) Status: websocket clients: 1
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:16Z:client(47186522342540) working...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:16Z:client(-) Status: ajax clients: 0
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:34Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: message repeated 2 times: [ 2020-07-02T14:43:34Z:client(47186522342540) send data to client]
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:35Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:35Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:36Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:36Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:36Z:client(-) Status: websocket clients: 1
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:36Z:client(47186522342540) working...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:36Z:client(-) Status: ajax clients: 0
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:36Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:37Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:37Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:38Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: message repeated 2 times: [ 2020-07-02T14:43:38Z:client(47186522342540) send data to client]
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:39Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:39Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:40Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: message repeated 2 times: [ 2020-07-02T14:43:40Z:client(47186522342540) send data to client]
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:41Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:41Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:42Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: message repeated 2 times: [ 2020-07-02T14:43:42Z:client(47186522342540) send data to client]
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:43Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:43Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:44Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: message repeated 2 times: [ 2020-07-02T14:43:44Z:client(47186522342540) send data to client]
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:45Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:45Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:46Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: message repeated 2 times: [ 2020-07-02T14:43:46Z:client(47186522342540) send data to client]
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:47Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:47Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:48Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: message repeated 2 times: [ 2020-07-02T14:43:48Z:client(47186522342540) send data to client]
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:49Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:49Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:50Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: message repeated 2 times: [ 2020-07-02T14:43:50Z:client(47186522342540) send data to client]
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:51Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:51Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:52Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:52Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:53Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:54Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: message repeated 2 times: [ 2020-07-02T14:43:54Z:client(47186522342540) send data to client]
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:55Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:56Z:client(-) Status: websocket clients: 1
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:56Z:client(47186522342540) working...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:56Z:client(-) Status: ajax clients: 0
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:57Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:57Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:58Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:58Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:59Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:43:59Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:44:00Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:44:00Z:client(47186522342540) send data to client
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:44:16Z:client(-) Status: websocket clients: 1
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:44:16Z:client(47186522342540) working...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:44:16Z:client(-) Status: ajax clients: 0
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:44:36Z:client(-) Status: websocket clients: 1
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:44:36Z:client(47186522342540) working...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:44:36Z:client(-) Status: ajax clients: 0
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:44:41Z:client(-) check unused idle connections...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:44:56Z:client(-) Status: websocket clients: 1
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:44:56Z:client(47186522342540) working...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:44:56Z:client(-) Status: ajax clients: 0
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:45:16Z:client(-) Status: websocket clients: 1
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:45:16Z:client(47186522342540) working...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:45:16Z:client(-) Status: ajax clients: 0
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:45:36Z:client(-) Status: websocket clients: 1
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:45:36Z:client(47186522342540) working...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:45:36Z:client(-) Status: ajax clients: 0
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:45:56Z:client(-) Status: websocket clients: 1
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:45:56Z:client(47186522342540) working...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:45:56Z:client(-) Status: ajax clients: 0
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:46:16Z:client(-) Status: websocket clients: 1
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:46:16Z:client(47186522342540) working...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:46:16Z:client(-) Status: ajax clients: 0
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:46:36Z:client(-) Status: websocket clients: 1
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:46:36Z:client(47186522342540) working...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:46:36Z:client(-) Status: ajax clients: 0
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:46:41Z:client(-) check unused idle connections...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:46:56Z:client(-) Status: websocket clients: 1
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:46:56Z:client(47186522342540) working...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:46:56Z:client(-) Status: ajax clients: 0
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:47:16Z:client(-) Status: websocket clients: 1
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:47:16Z:client(47186522342540) working...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:47:16Z:client(-) Status: ajax clients: 0
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:47:36Z:client(-) Status: websocket clients: 1
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:47:36Z:client(47186522342540) working...
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:47:36Z:client(-) Status: ajax clients: 0
    Jul  2 16:47:56 h2851867 zammad-websocket-1.service[4652]: 2020-07-02T14:47:47Z:client(47186522342540) send data to client

As you can see, there are tons of lines just for one second, so what is happening is that the log fills up so fast that logrotate (in its current setting) can’t handle it, the server runs out of space and then produces an 502 error - up till then, Zammad itself runs ok, though lacks performance.

Steps to reproduce the behavior:

  • I have removed Zammad via apt and re-installed it (generated a new zammad.conf but the database was kept on the server, so no need for a fresh Zammad set-up). That didn’t solve the issue at all, still filling up the syslog.

Any idea what could cause the issue?

Thanks

Alex

I suggest ensuring that websocket connections do work.
The logging you provided shows that Zammad uses it’s long polling (ajax) fallback to get information from the system.

This also puts more load on your host which you usually want not to happen.

If that long polling information already completely fills your filesystem, I must confess that I think that the production.log logging would even faster if all agents work on the system.

Hi!

Thanks for the advice - just as an update, we decided to start from scratch by backing up as much as we can and then re-installing the server as there seem to have been a couple of issues (Rails wasn’t working correctly anymore, ES was also having more and more issues) and it seems to be working fine as of now.

Regards

Alex