500 Internal Server Errors

DCWNZ · October 12, 2020, 2:45am

Infos:

Used Zammad version: 3.5.x
Used Zammad installation source: (source, package, …) APT Package
Operating system: Ubuntu 20.04
Browser + version: Chrome/Edge 85+

Expected behavior:

Loading Tickets/pages in the Web Interface normally should not produce a “500 - Internal Server Error” warning dialog box

Actual behavior:

Whilst browsing tickets, Agents are experiencing the above Error being displayed on screen, Seemingly at random.

Steps to reproduce the behavior:

Occurs at random. Sometimes several times over the course of a minute, other times once every 10 minutes or longer. Unable to point it to a single action/activity. Logs show all API requests can generate the error.

Apache 2.4.x as Reverse Proxy.
Percona XtraDB Clustered Mysql Database.

Apache Config:

<VirtualHost *:443>
    HostnameLookups Off
    UseCanonicalName Off
    ServerSignature Off

    SSLEngine on
    ServerName ictsupport.indevin.com
    DocumentRoot /opt/zammad/public

    SSLCertificateFile /etc/ssl/private/zammadcctv.cer
    SSLCertificateKeyFile /etc/ssl/private/zammadcctv.key

    ErrorLog /var/log/apache2/zammad-error-ssl.log
    CustomLog /var/log/apache2/zammad-access-ssl.log combined

    SSLProxyEngine On
    SSLProxyVerify     None
    SSLProxyCheckPeerCN Off
    SSLProxyCheckPeerName Off
    SSLProxyCheckPeerExpire Off

    <Proxy 127.0.0.1:3000>
        Require local
    </Proxy>

    ProxyPass /assets !
    ProxyPass /favicon.ico !
    ProxyPass /robots.txt !
    ProxyPass /ws ws://127.0.0.1:6042/ keepalive=On retry=1 acquire=3000 timeout=3000
    ProxyPass / http://127.0.0.1:3000/ connectiontimeout=5 timeout=3000
    ProxyPassReverse / http://127.0.0.1:3000/
    ProxyRequests Off
    ProxyPreserveHost On

    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>

    <Directory "/opt/zammad/public">
        Options FollowSymLinks
        Require all granted
    </Directory>

    RequestHeader set X_FORWARDED_PROTO 'https'
    RequestHeader set X-Forwarded-Ssl on
    RequestHeader unset X-Forwarded-User
</VirtualHost>

Example of Apache Error Log:

172.16.15.19 - - [12/Oct/2020:12:51:05 +1300] "GET /api/v1/tickets/1243?all=true&_=1602449446961 HTTP/1.1" 500 925 "https://ictsupport.indevin.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36 Edg/85.0.564.68"
172.16.15.19 - - [12/Oct/2020:12:51:18 +1300] "GET /api/v1/tickets/1323?all=true&_=1602449446983 HTTP/1.1" 500 378 "https://ictsupport.indevin.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36 Edg/85.0.564.68"
172.16.15.19 - - [12/Oct/2020:13:21:09 +1300] "GET /api/v1/tickets/1260?all=true&_=1602449447011 HTTP/1.1" 500 378 "https://ictsupport.indevin.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36 Edg/85.0.564.68"
172.16.15.19 - - [12/Oct/2020:13:21:20 +1300] "GET /api/v1/tickets/1330?all=true&_=1602449447026 HTTP/1.1" 500 378 "https://ictsupport.indevin.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36 Edg/85.0.564.68"
172.16.15.15 - - [12/Oct/2020:13:24:54 +1300] "POST /api/v1/integration/smime HTTP/1.1" 500 378 "https://itsupport.indevin.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"
172.16.15.15 - - [12/Oct/2020:13:24:59 +1300] "GET /api/v1/ticket_customer?customer_id=157&_=1602462286287 HTTP/1.1" 500 378 "https://itsupport.indevin.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"
172.16.15.15 - - [12/Oct/2020:13:25:01 +1300] "GET /api/v1/tickets/1343?all=true&_=1602462286292 HTTP/1.1" 500 378 "https://itsupport.indevin.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"
172.16.15.19 - - [12/Oct/2020:14:21:13 +1300] "GET /api/v1/tickets/1311?all=true&_=1602449447086 HTTP/1.1" 500 378 "https://ictsupport.indevin.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36 Edg/85.0.564.68"
172.16.15.10 - - [12/Oct/2020:14:47:01 +1300] "POST /api/v1/integration/smime HTTP/1.1" 500 378 "https://ictsupport.indevin.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
172.16.15.19 - - [12/Oct/2020:14:48:05 +1300] "GET /api/v1/tickets/1314?all=true&_=1602449447136 HTTP/1.1" 500 378 "https://ictsupport.indevin.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36 Edg/85.0.564.68"
172.16.15.19 - - [12/Oct/2020:14:48:15 +1300] "GET /api/v1/tickets/1334?all=true&_=1602449447151 HTTP/1.1" 500 378 "https://ictsupport.indevin.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36 Edg/85.0.564.68"
172.16.15.19 - - [12/Oct/2020:14:48:22 +1300] "POST /api/v1/integration/smime HTTP/1.1" 500 378 "https://ictsupport.indevin.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36 Edg/85.0.564.68"
172.16.15.19 - - [12/Oct/2020:14:48:33 +1300] "POST /api/v1/integration/smime HTTP/1.1" 500 378 "https://ictsupport.indevin.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36 Edg/85.0.564.68"
172.16.15.24 - - [12/Oct/2020:14:48:52 +1300] "POST /api/v1/integration/smime HTTP/1.1" 500 378 "https://ictsupport.indevin.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
172.16.15.24 - - [12/Oct/2020:14:48:53 +1300] "GET /api/v1/ticket_attachment/1125/2445/3099?view=inline HTTP/1.1" 500 1506 "https://ictsupport.indevin.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
172.16.15.24 - - [12/Oct/2020:15:17:49 +1300] "GET /api/v1/users/image/14b8505342b17d8f5bc577b66485cd9e HTTP/1.1" 500 959 "https://ictsupport.indevin.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"

Zammad Production Log shows no errors.

This appears to be solely related to Apache, but I could be very wrong.

Can anyone see anything glaringly obvious or should I be looking somewhere else?

Let me know if any further information/logging is required.

Cheers!

rsysadmin · October 12, 2020, 6:37am

Hi there!

Errors on the 500-range mean that your server has issues with its configuration.
Please check that /opt/zammad/public has the right permissions - your Apache user must be able to read the files at least…

HTH.

Best regards

DCWNZ · October 13, 2020, 2:18am

Hi,

Thanks for the response,
Current /opt/zammad/public permissions are:

I have tested by adding the Apache User www-data to the ‘zammad’ user group. However, this doesnt appear to have made a significant difference.
One staff member has stated that they seem to get the issue when updating tickets via the sidepanel in the ticket window, however if they reply to a ticket and then close, everything seems to work fine. I havnt seen this behaviour however.

I have just received the error after bulk updating ~30 tickets. Update was successful, ~30 seconds later I get the 500 error on the webpage.

I should also mention that these erros appear to be non-destructive, they can be clicked away and the UI continues on, they are just frustratingly annoying.

Ive got the Production log if that will help narrow it down, unsure if it will help this but I cant find anything else really obvious.

Cheers,

rsysadmin · October 13, 2020, 6:09am

Hi,

You are welcome.

I am not a Zammad team member, just a plain user with a strong UNIX/Linux SysAdmin background
It seems like Zammad itself is working fine but you seem to be bitten by some bug somewhere…

Please send me those logs so I can check. However, I cannot make any promises that I will find the root cause. You may as well see what others reply … or just wait and see if I can find something

Cheers

rsysadmin · October 13, 2020, 6:11am

BTW - can you please copy & paste the output of:

ls -lRt /opt/zammad/public

?

Thanks

DCWNZ · October 13, 2020, 7:26pm

Hi,

Thanks very much. Once I work out how to send you the log file ill get that sent.

Permissions/Folder Structure as requested below:

/opt/zammad/public:
total 84
drwxr-xr-x 14 zammad zammad 20480 Oct 14 08:06 assets
-rw-r–r-- 1 zammad zammad 737 Oct 9 03:15 401.html
-rw-r–r-- 1 zammad zammad 750 Oct 9 03:15 404.html
-rw-r–r-- 1 zammad zammad 727 Oct 9 03:15 422.html
-rw-r–r-- 1 zammad zammad 633 Oct 9 03:15 500.html
-rw-r–r-- 1 zammad zammad 7692 Oct 9 03:15 apple-touch-icon.png
-rw-r–r-- 1 zammad zammad 32988 Oct 9 03:15 favicon.ico
-rw-r–r-- 1 zammad zammad 214 Oct 9 03:15 robots.txt

/opt/zammad/public/assets:
total 13668
drwxr-xr-x 4 zammad zammad 4096 Oct 14 08:06 images
drwxr-xr-x 2 zammad zammad 12288 Oct 14 08:06 fonts
drwxr-xr-x 2 zammad zammad 4096 Oct 14 08:06 form
drwxr-xr-x 2 zammad zammad 4096 Oct 14 08:06 icon-fonts
drwxr-xr-x 2 zammad zammad 4096 Oct 14 08:06 sounds
drwxr-xr-x 2 zammad zammad 4096 Oct 14 08:06 tests
drwxrwxr-x 2 zammad zammad 4096 Oct 14 08:06 addons
drwxr-xr-x 3 zammad zammad 4096 Oct 14 08:06 chat
drwxrwxr-x 2 zammad zammad 4096 Oct 14 08:06 custom
drwxrwxr-x 3 zammad zammad 4096 Oct 14 08:06 doorkeeper
drwxr-xr-x 2 zammad zammad 4096 Oct 14 08:06 error
-rw-rw-r-- 1 zammad zammad 300714 Oct 9 03:15 application-ed28a90e18590f77c1ac1db16ba6c0be5042109a0682533ac2badcce1fc43401.css
-rw-rw-r-- 1 zammad zammad 56055 Oct 9 03:15 application-ed28a90e18590f77c1ac1db16ba6c0be5042109a0682533ac2badcce1fc43401.css.gz
-rw-rw-r-- 1 zammad zammad 2628584 Sep 25 01:12 application-8bf35218647a61ae31d13eef552c8f9a4c81783c883211ca539d8df1e94e2628.js
-rw-rw-r-- 1 zammad zammad 563814 Sep 25 01:12 application-8bf35218647a61ae31d13eef552c8f9a4c81783c883211ca539d8df1e94e2628.js.gz
-rw-rw-r-- 1 zammad zammad 2628608 Sep 22 23:32 application-79140f8ba410a6c5334cb5dd8d066af8dd85a4227221e04f42e1331acb6c581a.js
-rw-rw-r-- 1 zammad zammad 563821 Sep 22 23:32 application-79140f8ba410a6c5334cb5dd8d066af8dd85a4227221e04f42e1331acb6c581a.js.gz
-rw-rw-r-- 1 zammad zammad 300682 Sep 22 23:32 application-1b6c5b58976055bb6834136fa2eb20fff977c50a3211ae443c250ef9f1f7c0a4.css
-rw-rw-r-- 1 zammad zammad 56051 Sep 22 23:32 application-1b6c5b58976055bb6834136fa2eb20fff977c50a3211ae443c250ef9f1f7c0a4.css.gz
-rw-rw-r-- 1 zammad zammad 24406 Sep 22 23:32 knowledge_base-06e35830f5230d92957e79cb37b27d0cb29d3921a01cd1101a0d79d45b0681a1.css
-rw-rw-r-- 1 zammad zammad 5486 Sep 22 23:32 knowledge_base-06e35830f5230d92957e79cb37b27d0cb29d3921a01cd1101a0d79d45b0681a1.css.gz
-rw-r–r-- 1 zammad zammad 2610093 Sep 22 20:31 application-140daeb61299eda003507eccfe600c4728d2899fdee041ed2df4319b9e9ef5d9.js
-rw-r–r-- 1 zammad zammad 560487 Sep 22 20:31 application-140daeb61299eda003507eccfe600c4728d2899fdee041ed2df4319b9e9ef5d9.js.gz
drwxrwxr-x 3 zammad zammad 4096 Sep 10 17:20 app
-rw-rw-r-- 1 zammad zammad 2609831 Jul 16 03:55 application-ceba5c1d631eb05d308098c4191de07231dcd6f244dda8cba986565906110834.js
-rw-rw-r-- 1 zammad zammad 560347 Jul 16 03:55 application-ceba5c1d631eb05d308098c4191de07231dcd6f244dda8cba986565906110834.js.gz
-rw-rw-r-- 1 zammad zammad 299598 Jun 16 02:50 application-e170a71d8482cf6f690173f4d4c518d97b25e84bbc145ae455fde553bfc73aca.css
-rw-rw-r-- 1 zammad zammad 55835 Jun 16 02:50 application-e170a71d8482cf6f690173f4d4c518d97b25e84bbc145ae455fde553bfc73aca.css.gz
-rw-rw-r-- 1 zammad zammad 2731 Jun 16 02:50 application-print-25adf1ad928517e8672a95590d5449b40243c7ae6749c73d532f8fcc068c2612.css
-rw-rw-r-- 1 zammad zammad 1039 Jun 16 02:50 application-print-25adf1ad928517e8672a95590d5449b40243c7ae6749c73d532f8fcc068c2612.css.gz
-rw-rw-r-- 1 zammad zammad 24366 Jun 16 02:50 knowledge_base-6c6e4490fd043596ea82bb22f0e98a516c5da951a3a2dcffe0c92263210cdc96.css
-rw-rw-r-- 1 zammad zammad 5482 Jun 16 02:50 knowledge_base-6c6e4490fd043596ea82bb22f0e98a516c5da951a3a2dcffe0c92263210cdc96.css.gz
-rw-rw-r-- 1 zammad zammad 5255 Jun 16 02:50 knowledge_base_public-8cabc0d60133581b53207c94bd1db9596279589580c0581616aa14849c0803d6.js
-rw-rw-r-- 1 zammad zammad 1795 Jun 16 02:50 knowledge_base_public-8cabc0d60133581b53207c94bd1db9596279589580c0581616aa14849c0803d6.js.gz
-rw-rw-r-- 1 zammad zammad 12294 Jun 16 02:50 knowledge_base_public_polyfills-90b07a6b9678d0b6c05e6d0fa10d500d3fc0368bfb2725770629d5d4b0691c9c.js
-rw-rw-r-- 1 zammad zammad 4217 Jun 16 02:50 knowledge_base_public_polyfills-90b07a6b9678d0b6c05e6d0fa10d500d3fc0368bfb2725770629d5d4b0691c9c.js.gz
-rw-rw-r-- 1 zammad zammad 2731 Jun 16 02:50 print-25adf1ad928517e8672a95590d5449b40243c7ae6749c73d532f8fcc068c2612.css
-rw-rw-r-- 1 zammad zammad 1039 Jun 16 02:50 print-25adf1ad928517e8672a95590d5449b40243c7ae6749c73d532f8fcc068c2612.css.gz

/opt/zammad/public/assets/images:
total 168
-rw-r–r-- 1 zammad zammad 11564 Oct 14 08:08 a8e4a97422558397b671a76e80125723.png
drwxr-xr-x 2 zammad zammad 4096 Oct 14 08:06 graphics
drwxr-xr-x 2 zammad zammad 20480 Oct 14 08:06 icons
-rw-r–r-- 1 zammad zammad 25852 Oct 9 03:15 avatar-bg.png
-rw-r–r-- 1 zammad zammad 3422 Oct 9 03:15 chat-demo-avatar.png
-rw-r–r-- 1 zammad zammad 97 Oct 9 03:15 colorcircle.gif
-rw-r–r-- 1 zammad zammad 90 Oct 9 03:15 eyedropper.gif
-rw-r–r-- 1 zammad zammad 89729 Oct 9 03:15 icons.svg
-rw-r–r-- 1 zammad zammad 3166 Oct 9 03:15 logo.svg

/opt/zammad/public/assets/images/graphics:
total 12
-rw-r–r-- 1 zammad zammad 2101 Oct 9 03:15 knowledge_base_accordion.svg
-rw-r–r-- 1 zammad zammad 2277 Oct 9 03:15 knowledge_base_grid.svg
-rw-r–r-- 1 zammad zammad 1690 Oct 9 03:15 knowledge_base_list.svg

/opt/zammad/public/assets/images/icons:
total 580
-rw-r–r-- 1 zammad zammad 935 Oct 9 03:15 archived-modifier.svg
-rw-r–r-- 1 zammad zammad 677 Oct 9 03:15 arrow-down.svg
-rw-r–r-- 1 zammad zammad 683 Oct 9 03:15 arrow-left.svg
-rw-r–r-- 1 zammad zammad 689 Oct 9 03:15 arrow-right.svg
-rw-r–r-- 1 zammad zammad 669 Oct 9 03:15 arrow-up.svg
-rw-r–r-- 1 zammad zammad 1242 Oct 9 03:15 bold.svg
-rw-r–r-- 1 zammad zammad 2267 Oct 9 03:15 chain.svg
-rw-r–r-- 1 zammad zammad 805 Oct 9 03:15 chat.svg
-rw-r–r-- 1 zammad zammad 1130 Oct 9 03:15 checkbox-checked.svg
-rw-r–r-- 1 zammad zammad 884 Oct 9 03:15 checkbox-indeterminate.svg
-rw-r–r-- 1 zammad zammad 698 Oct 9 03:15 checkbox.svg
-rw-r–r-- 1 zammad zammad 821 Oct 9 03:15 checkmark.svg
-rw-r–r-- 1 zammad zammad 2242 Oct 9 03:15 clipboard.svg
-rw-r–r-- 1 zammad zammad 1312 Oct 9 03:15 clock.svg
-rw-r–r-- 1 zammad zammad 1121 Oct 9 03:15 cloud.svg
-rw-r–r-- 1 zammad zammad 4342 Oct 9 03:15 cog.svg
-rw-r–r-- 1 zammad zammad 1628 Oct 9 03:15 crown.svg
-rw-r–r-- 1 zammad zammad 1397 Oct 9 03:15 danger.svg
-rw-r–r-- 1 zammad zammad 1581 Oct 9 03:15 dashboard.svg
-rw-r–r-- 1 zammad zammad 1327 Oct 9 03:15 diagonal-cross.svg
-rw-r–r-- 1 zammad zammad 1551 Oct 9 03:15 document.svg
-rw-r–r-- 1 zammad zammad 618 Oct 9 03:15 download.svg
-rw-r–r-- 1 zammad zammad 1123 Oct 9 03:15 draft-modifier.svg
-rw-r–r-- 1 zammad zammad 805 Oct 9 03:15 draggable.svg
-rw-r–r-- 1 zammad zammad 676 Oct 9 03:15 dropdown-list.svg
-rw-r–r-- 1 zammad zammad 742 Oct 9 03:15 email-button.svg
-rw-r–r-- 1 zammad zammad 747 Oct 9 03:15 email.svg
-rw-r–r-- 1 zammad zammad 814 Oct 9 03:15 external.svg
-rw-r–r-- 1 zammad zammad 2108 Oct 9 03:15 eyedropper.svg
-rw-r–r-- 1 zammad zammad 1262 Oct 9 03:15 eye.svg
-rw-r–r-- 1 zammad zammad 861 Oct 9 03:15 facebook-button.svg
-rw-r–r-- 1 zammad zammad 1223 Oct 9 03:15 facebook.svg
-rw-r–r-- 1 zammad zammad 2050 Oct 9 03:15 file-archive.svg
-rw-r–r-- 1 zammad zammad 1952 Oct 9 03:15 file-code.svg
-rw-r–r-- 1 zammad zammad 2055 Oct 9 03:15 file-email.svg
-rw-r–r-- 1 zammad zammad 1147 Oct 9 03:15 file-excel.svg
-rw-r–r-- 1 zammad zammad 2791 Oct 9 03:15 file-pdf.svg
-rw-r–r-- 1 zammad zammad 1366 Oct 9 03:15 file-powerpoint.svg
-rw-r–r-- 1 zammad zammad 1028 Oct 9 03:15 file-text.svg
-rw-r–r-- 1 zammad zammad 1791 Oct 9 03:15 file-unknown.svg
-rw-r–r-- 1 zammad zammad 1166 Oct 9 03:15 file-word.svg
-rw-r–r-- 1 zammad zammad 4354 Oct 9 03:15 form.svg
-rw-r–r-- 1 zammad zammad 696 Oct 9 03:15 forward.svg
-rw-r–r-- 1 zammad zammad 8071 Oct 9 03:15 full-logo.svg
-rw-r–r-- 1 zammad zammad 2213 Oct 9 03:15 github-button.svg
-rw-r–r-- 1 zammad zammad 3134 Oct 9 03:15 gitlab-button.svg
-rw-r–r-- 1 zammad zammad 1726 Oct 9 03:15 google-button.svg
-rw-r–r-- 1 zammad zammad 2921 Oct 9 03:15 group.svg
-rw-r–r-- 1 zammad zammad 1479 Oct 9 03:15 help.svg
-rw-r–r-- 1 zammad zammad 775 Oct 9 03:15 horizontal-rule.svg
-rw-r–r-- 1 zammad zammad 1131 Oct 9 03:15 important.svg
-rw-r–r-- 1 zammad zammad 1643 Oct 9 03:15 inactive-organization.svg
-rw-r–r-- 1 zammad zammad 1414 Oct 9 03:15 inactive-user.svg
-rw-r–r-- 1 zammad zammad 2251 Oct 9 03:15 info.svg
-rw-r–r-- 1 zammad zammad 2289 Oct 9 03:15 in-process.svg
-rw-r–r-- 1 zammad zammad 1930 Oct 9 03:15 internal-modifier.svg
-rw-r–r-- 1 zammad zammad 734 Oct 9 03:15 italic.svg
-rw-r–r-- 1 zammad zammad 854 Oct 9 03:15 knowledge-base-answer.svg
-rw-r–r-- 1 zammad zammad 1728 Oct 9 03:15 knowledge-base.svg
-rw-r–r-- 1 zammad zammad 697 Oct 9 03:15 line-left-arrow.svg
-rw-r–r-- 1 zammad zammad 688 Oct 9 03:15 line-right-arrow.svg
-rw-r–r-- 1 zammad zammad 1385 Oct 9 03:15 linkedin-button.svg
-rw-r–r-- 1 zammad zammad 1228 Oct 9 03:15 list.svg
-rw-r–r-- 1 zammad zammad 563 Oct 9 03:15 loading.svg
-rw-r–r-- 1 zammad zammad 1595 Oct 9 03:15 lock-open.svg
-rw-r–r-- 1 zammad zammad 1656 Oct 9 03:15 lock.svg
-rw-r–r-- 1 zammad zammad 2396 Oct 9 03:15 logo.svg
-rw-r–r-- 1 zammad zammad 7147 Oct 9 03:15 logotype.svg
-rw-r–r-- 1 zammad zammad 980 Oct 9 03:15 long-arrow-down.svg
-rw-r–r-- 1 zammad zammad 950 Oct 9 03:15 long-arrow-right.svg
-rw-r–r-- 1 zammad zammad 1307 Oct 9 03:15 low-priority.svg
-rw-r–r-- 1 zammad zammad 1686 Oct 9 03:15 magnifier.svg
-rw-r–r-- 1 zammad zammad 1229 Oct 9 03:15 marker.svg
-rw-r–r-- 1 zammad zammad 2582 Oct 9 03:15 message.svg
-rw-r–r-- 1 zammad zammad 579 Oct 9 03:15 minus-small.svg
-rw-r–r-- 1 zammad zammad 569 Oct 9 03:15 minus.svg
-rw-r–r-- 1 zammad zammad 2197 Oct 9 03:15 mood-bad.svg
-rw-r–r-- 1 zammad zammad 2819 Oct 9 03:15 mood-good.svg
-rw-r–r-- 1 zammad zammad 2588 Oct 9 03:15 mood-ok.svg
-rw-r–r-- 1 zammad zammad 2832 Oct 9 03:15 mood-sad.svg
-rw-r–r-- 1 zammad zammad 2553 Oct 9 03:15 mood-superbad.svg
-rw-r–r-- 1 zammad zammad 2754 Oct 9 03:15 mood-supergood.svg
-rw-r–r-- 1 zammad zammad 1319 Oct 9 03:15 mute.svg
-rw-r–r-- 1 zammad zammad 2022 Oct 9 03:15 note.svg
-rw-r–r-- 1 zammad zammad 3794 Oct 9 03:15 not-signed.svg
-rw-r–r-- 1 zammad zammad 2023 Oct 9 03:15 oauth2-button.svg
-rw-r–r-- 1 zammad zammad 890 Oct 9 03:15 office365-button.svg
-rw-r–r-- 1 zammad zammad 678 Oct 9 03:15 one-ticket.svg
-rw-r–r-- 1 zammad zammad 1723 Oct 9 03:15 organization.svg
-rw-r–r-- 1 zammad zammad 1889 Oct 9 03:15 outbound-calls.svg
-rw-r–r-- 1 zammad zammad 999 Oct 9 03:15 overflow-button.svg
-rw-r–r-- 1 zammad zammad 1451 Oct 9 03:15 overviews.svg
-rw-r–r-- 1 zammad zammad 1216 Oct 9 03:15 package.svg
-rw-r–r-- 1 zammad zammad 2404 Oct 9 03:15 paperclip.svg
-rw-r–r-- 1 zammad zammad 1097 Oct 9 03:15 pen.svg
-rw-r–r-- 1 zammad zammad 1328 Oct 9 03:15 person.svg
-rw-r–r-- 1 zammad zammad 1655 Oct 9 03:15 phone.svg
-rw-r–r-- 1 zammad zammad 611 Oct 9 03:15 plus-small.svg
-rw-r–r-- 1 zammad zammad 689 Oct 9 03:15 plus.svg
-rw-r–r-- 1 zammad zammad 1510 Oct 9 03:15 printer.svg
-rw-r–r-- 1 zammad zammad 854 Oct 9 03:15 radio-checked.svg
-rw-r–r-- 1 zammad zammad 803 Oct 9 03:15 radio.svg
-rw-r–r-- 1 zammad zammad 734 Oct 9 03:15 rearange.svg
-rw-r–r-- 1 zammad zammad 1857 Oct 9 03:15 received-calls.svg
-rw-r–r-- 1 zammad zammad 5153 Oct 9 03:15 reload.svg
-rw-r–r-- 1 zammad zammad 2543 Oct 9 03:15 reopening.svg
-rw-r–r-- 1 zammad zammad 1057 Oct 9 03:15 reply-all.svg
-rw-r–r-- 1 zammad zammad 895 Oct 9 03:15 reply.svg
-rw-r–r-- 1 zammad zammad 626 Oct 9 03:15 report.svg
-rw-r–r-- 1 zammad zammad 2489 Oct 9 03:15 searchdetail.svg
-rw-r–r-- 1 zammad zammad 3309 Oct 9 03:15 signed.svg
-rw-r–r-- 1 zammad zammad 719 Oct 9 03:15 signout.svg
-rw-r–r-- 1 zammad zammad 567 Oct 9 03:15 small-dot.svg
-rw-r–r-- 1 zammad zammad 2915 Oct 9 03:15 sms.svg
-rw-r–r-- 1 zammad zammad 4028 Oct 9 03:15 spinner-small.svg
-rw-r–r-- 1 zammad zammad 1344 Oct 9 03:15 split.svg
-rw-r–r-- 1 zammad zammad 1541 Oct 9 03:15 sso-button.svg
-rw-r–r-- 1 zammad zammad 857 Oct 9 03:15 status-modified-outer-circle.svg
-rw-r–r-- 1 zammad zammad 561 Oct 9 03:15 status.svg
-rw-r–r-- 1 zammad zammad 1569 Oct 9 03:15 stopwatch.svg
-rw-r–r-- 1 zammad zammad 1406 Oct 9 03:15 strikethrough.svg
-rw-r–r-- 1 zammad zammad 2177 Oct 9 03:15 switchView.svg
-rw-r–r-- 1 zammad zammad 903 Oct 9 03:15 task-state.svg
-rw-r–r-- 1 zammad zammad 5872 Oct 9 03:15 team.svg
-rw-r–r-- 1 zammad zammad 1614 Oct 9 03:15 telegram.svg
-rw-r–r-- 1 zammad zammad 1595 Oct 9 03:15 templates.svg
-rw-r–r-- 1 zammad zammad 1886 Oct 9 03:15 tools.svg
-rw-r–r-- 1 zammad zammad 3570 Oct 9 03:15 total-tickets.svg
-rw-r–r-- 1 zammad zammad 865 Oct 9 03:15 trash.svg
-rw-r–r-- 1 zammad zammad 1909 Oct 9 03:15 twitter-button.svg
-rw-r–r-- 1 zammad zammad 1622 Oct 9 03:15 twitter.svg
-rw-r–r-- 1 zammad zammad 820 Oct 9 03:15 underline.svg
-rw-r–r-- 1 zammad zammad 1889 Oct 9 03:15 unmute.svg
-rw-r–r-- 1 zammad zammad 2129 Oct 9 03:15 unordered-list.svg
-rw-r–r-- 1 zammad zammad 1675 Oct 9 03:15 user.svg
-rw-r–r-- 1 zammad zammad 2824 Oct 9 03:15 web.svg
-rw-r–r-- 1 zammad zammad 2892 Oct 9 03:15 weibo-button.svg
-rw-r–r-- 1 zammad zammad 1770 Oct 9 03:15 zoom-in.svg
-rw-r–r-- 1 zammad zammad 1738 Oct 9 03:15 zoom-out.svg

/opt/zammad/public/assets/fonts:
total 4404
-rw-r–r-- 1 zammad zammad 169279 Oct 9 03:15 FiraSans-Bold.eot
-rw-r–r-- 1 zammad zammad 432468 Oct 9 03:15 FiraSans-Bold.ttf
-rwxr-xr-x 1 zammad zammad 101954 Oct 9 03:15 firasans-bold-webfont-0eb0f6eca42625f8184372baf62c255b.svg
-rwxr-xr-x 1 zammad zammad 72008 Oct 9 03:15 firasans-bold-webfont-84efee074704b5fb303303e922a131cd.ttf
-rwxr-xr-x 1 zammad zammad 29060 Oct 9 03:15 firasans-bold-webfont-877d21cf99f300b8680ef77d12a98b57.woff
-rwxr-xr-x 1 zammad zammad 26110 Oct 9 03:15 firasans-bold-webfont-dbb0e92d31531ddaa5e55dedb9761edb.eot
-rw-r–r-- 1 zammad zammad 195936 Oct 9 03:15 FiraSans-Bold.woff
-rw-r–r-- 1 zammad zammad 138632 Oct 9 03:15 FiraSans-Bold.woff2
-rw-r–r-- 1 zammad zammad 156500 Oct 9 03:15 FiraSans-Light.eot
-rw-r–r-- 1 zammad zammad 391456 Oct 9 03:15 FiraSans-Light.ttf
-rwxr-xr-x 1 zammad zammad 101698 Oct 9 03:15 firasans-light-webfont-3c22e83bb9d69299a0119909eb054741.svg
-rwxr-xr-x 1 zammad zammad 25656 Oct 9 03:15 firasans-light-webfont-72b19bb8b5524e54077bc0a37e9332ec.eot
-rwxr-xr-x 1 zammad zammad 28392 Oct 9 03:15 firasans-light-webfont-8f6d166b428b8c4df0fc8ca43c2622cc.woff
-rwxr-xr-x 1 zammad zammad 72468 Oct 9 03:15 firasans-light-webfont-ee192366bb30edc80ed1168eb2ce44ba.ttf
-rw-r–r-- 1 zammad zammad 180856 Oct 9 03:15 FiraSans-Light.woff
-rw-r–r-- 1 zammad zammad 128040 Oct 9 03:15 FiraSans-Light.woff2
-rw-r–r-- 1 zammad zammad 160608 Oct 9 03:15 FiraSans-Medium.eot
-rw-r–r-- 1 zammad zammad 399404 Oct 9 03:15 FiraSans-Medium.ttf
-rw-r–r-- 1 zammad zammad 184800 Oct 9 03:15 FiraSans-Medium.woff
-rw-r–r-- 1 zammad zammad 131156 Oct 9 03:15 FiraSans-Medium.woff2
-rw-r–r-- 1 zammad zammad 156376 Oct 9 03:15 FiraSans-Regular.eot
-rw-r–r-- 1 zammad zammad 398432 Oct 9 03:15 FiraSans-Regular.ttf
-rwxr-xr-x 1 zammad zammad 101675 Oct 9 03:15 firasans-regular-webfont-27e62a02ccd15c3f3ec82a5b07c68810.svg
-rwxr-xr-x 1 zammad zammad 28852 Oct 9 03:15 firasans-regular-webfont-326b4b46e8b5f04059ea4ea96da1de39.woff
-rwxr-xr-x 1 zammad zammad 72228 Oct 9 03:15 firasans-regular-webfont-39d7571f67e40601a3c7bb8146af339f.ttf
-rwxr-xr-x 1 zammad zammad 26030 Oct 9 03:15 firasans-regular-webfont-a1c53cbcd8f6b5dd419e69baff337383.eot
-rw-r–r-- 1 zammad zammad 181308 Oct 9 03:15 FiraSans-Regular.woff
-rw-r–r-- 1 zammad zammad 127596 Oct 9 03:15 FiraSans-Regular.woff2
-rw-r–r-- 1 zammad zammad 20127 Oct 9 03:15 glyphicons-halflings-regular.eot
-rw-r–r-- 1 zammad zammad 108738 Oct 9 03:15 glyphicons-halflings-regular.svg
-rw-r–r-- 1 zammad zammad 45404 Oct 9 03:15 glyphicons-halflings-regular.ttf
-rw-r–r-- 1 zammad zammad 23424 Oct 9 03:15 glyphicons-halflings-regular.woff
-rw-r–r-- 1 zammad zammad 18028 Oct 9 03:15 glyphicons-halflings-regular.woff2

/opt/zammad/public/assets/form:
total 28
-rw-r–r-- 1 zammad zammad 1187 Oct 9 03:15 form.css
-rw-r–r-- 1 zammad zammad 6723 Oct 9 03:15 form.html
-rw-r–r-- 1 zammad zammad 15309 Oct 9 03:15 form.js

/opt/zammad/public/assets/icon-fonts:
total 800
-rwxr-xr-x 1 zammad zammad 29688 Oct 9 03:15 anticon.json
-rw-r–r-- 1 zammad zammad 46488 Oct 9 03:15 anticon.woff
-rwxr-xr-x 1 zammad zammad 153353 Oct 9 03:15 FontAwesome.json
-rw-r–r-- 1 zammad zammad 110368 Oct 9 03:15 FontAwesome.woff
-rwxr-xr-x 1 zammad zammad 91998 Oct 9 03:15 ionicons.json
-rw-r–r-- 1 zammad zammad 108780 Oct 9 03:15 ionicons.woff
-rwxr-xr-x 1 zammad zammad 100722 Oct 9 03:15 material.json
-rw-r–r-- 1 zammad zammad 109948 Oct 9 03:15 material.woff
-rwxr-xr-x 1 zammad zammad 19834 Oct 9 03:15 Simple-Line-Icons.json
-rw-r–r-- 1 zammad zammad 31340 Oct 9 03:15 Simple-Line-Icons.woff

/opt/zammad/public/assets/sounds:
total 216
-rw-r–r-- 1 zammad zammad 16806 Oct 9 03:15 Bell.mp3
-rw-r–r-- 1 zammad zammad 43036 Oct 9 03:15 chat_message.mp3
-rw-r–r-- 1 zammad zammad 44943 Oct 9 03:15 chat_new.mp3
-rw-r–r-- 1 zammad zammad 12554 Oct 9 03:15 Kalimba.mp3
-rw-r–r-- 1 zammad zammad 11642 Oct 9 03:15 Marimba.mp3
-rw-r–r-- 1 zammad zammad 3558 Oct 9 03:15 Peep.mp3
-rw-r–r-- 1 zammad zammad 7104 Oct 9 03:15 Plop.mp3
-rw-r–r-- 1 zammad zammad 20519 Oct 9 03:15 Ring.mp3
-rw-r–r-- 1 zammad zammad 24946 Oct 9 03:15 Space.mp3
-rw-r–r-- 1 zammad zammad 3455 Oct 9 03:15 Wood.mp3
-rw-r–r-- 1 zammad zammad 9661 Oct 9 03:15 Xylo.mp3

/opt/zammad/public/assets/tests:
total 856
-rw-r–r-- 1 zammad zammad 20 Oct 9 03:15 ajax-test.json
-rw-r–r-- 1 zammad zammad 1165 Oct 9 03:15 color_object.js
-rw-r–r-- 1 zammad zammad 16244 Oct 9 03:15 core.js
-rw-r–r-- 1 zammad zammad 1961 Oct 9 03:15 form_autocompletion_ajax.js
-rw-r–r-- 1 zammad zammad 3535 Oct 9 03:15 form_color.js
-rw-r–r-- 1 zammad zammad 2184 Oct 9 03:15 form_column_select.js
-rw-r–r-- 1 zammad zammad 639 Oct 9 03:15 form_datetime.js
-rw-r–r-- 1 zammad zammad 20552 Oct 9 03:15 form_extended.js
-rw-r–r-- 1 zammad zammad 2859 Oct 9 03:15 form_find.js
-rw-r–r-- 1 zammad zammad 62162 Oct 9 03:15 form.js
-rw-r–r-- 1 zammad zammad 6402 Oct 9 03:15 form_searchable_select.js
-rw-r–r-- 1 zammad zammad 602 Oct 9 03:15 form_skip_rendering.js
-rw-r–r-- 1 zammad zammad 3307 Oct 9 03:15 form_sla_times.js
-rw-r–r-- 1 zammad zammad 12572 Oct 9 03:15 form_ticket_perform_action.js
-rw-r–r-- 1 zammad zammad 4661 Oct 9 03:15 form_timer.js
-rw-r–r-- 1 zammad zammad 5484 Oct 9 03:15 form_tree_select.js
-rw-r–r-- 1 zammad zammad 549 Oct 9 03:15 form_trim.js
-rw-r–r-- 1 zammad zammad 15971 Oct 9 03:15 form_validation.js
-rw-r–r-- 1 zammad zammad 168092 Oct 9 03:15 html_utils.js
-rw-r–r-- 1 zammad zammad 9857 Oct 9 03:15 i18n.js
-rw-r–r-- 1 zammad zammad 1806 Oct 9 03:15 image_service.js
-rw-r–r-- 1 zammad zammad 1215 Oct 9 03:15 kb_video_embeding.js
-rw-r–r-- 1 zammad zammad 1563 Oct 9 03:15 local_storage.js
-rw-r–r-- 1 zammad zammad 1462 Oct 9 03:15 model_binding.js
-rw-r–r-- 1 zammad zammad 14433 Oct 9 03:15 model.js
-rw-r–r-- 1 zammad zammad 3280 Oct 9 03:15 model_ticket.js
-rw-r–r-- 1 zammad zammad 2841 Oct 9 03:15 model_ui.js
-rw-r–r-- 1 zammad zammad 5399 Oct 9 03:15 qunit-1.21.0.css
-rw-r–r-- 1 zammad zammad 111796 Oct 9 03:15 qunit-1.21.0.js
-rw-r–r-- 1 zammad zammad 2599 Oct 9 03:15 session.js
-rw-r–r-- 1 zammad zammad 99213 Oct 9 03:15 syn-0.14.1.js
-rw-r–r-- 1 zammad zammad 92217 Oct 9 03:15 table_extended.js
-rw-r–r-- 1 zammad zammad 41466 Oct 9 03:15 table.js
-rw-r–r-- 1 zammad zammad 14102 Oct 9 03:15 taskbar.js
-rw-r–r-- 1 zammad zammad 2777 Oct 9 03:15 text_module.js
-rw-r–r-- 1 zammad zammad 865 Oct 9 03:15 ticket_macro.js
-rw-r–r-- 1 zammad zammad 29280 Oct 9 03:15 ticket_selector.js
-rw-r–r-- 1 zammad zammad 17294 Oct 9 03:15 ui.js

/opt/zammad/public/assets/addons:
total 8
-rw-rw-r-- 1 zammad zammad 46 Jun 16 02:50 README-93260400b4a86248c0586a965245ec8f40a912e17f08b532c51a5fc3273a1b46.txt
-rw-rw-r-- 1 zammad zammad 65 Jun 16 02:50 README-93260400b4a86248c0586a965245ec8f40a912e17f08b532c51a5fc3273a1b46.txt.gz

/opt/zammad/public/assets/chat:
total 1984
drwxr-xr-x 2 zammad zammad 4096 Oct 14 08:06 views
-rw-r–r-- 1 zammad zammad 67756 Oct 9 03:15 chat.coffee
-rw-r–r-- 1 zammad zammad 8454 Oct 9 03:15 chat.css
-rw-r–r-- 1 zammad zammad 103239 Oct 9 03:15 chat.js
-rw-r–r-- 1 zammad zammad 65822 Oct 9 03:15 chat.min.js
-rw-r–r-- 1 zammad zammad 66898 Oct 9 03:15 chat-no-jquery.coffee
-rw-r–r-- 1 zammad zammad 103570 Oct 9 03:15 chat-no-jquery.js
-rw-r–r-- 1 zammad zammad 67050 Oct 9 03:15 chat-no-jquery.min.js
-rw-r–r-- 1 zammad zammad 7952 Oct 9 03:15 chat.scss
-rw-r–r-- 1 zammad zammad 283 Oct 9 03:15 close.svg
-rw-r–r-- 1 zammad zammad 2138 Oct 9 03:15 gulpfile.js
-rw-r–r-- 1 zammad zammad 39259 Oct 9 03:15 index.html
-rw-r–r-- 1 zammad zammad 84345 Oct 9 03:15 jquery-2.1.4.min.js
-rw-r–r-- 1 zammad zammad 474 Oct 9 03:15 package.json
-rw-r–r-- 1 zammad zammad 767720 Oct 9 03:15 website.png
-rw-r–r-- 1 zammad zammad 99110 Oct 9 03:15 yarn.lock
-rw-r–r-- 1 zammad zammad 40 Oct 9 03:15 znuny.css
-rw-r–r-- 1 zammad zammad 5222 Oct 9 03:15 znuny.html
-rw-r–r-- 1 zammad zammad 4599 Oct 9 03:15 znuny-no-jquery.html
-rw-r–r-- 1 zammad zammad 4616 Oct 9 03:15 znuny-no-jquery-open_by_button.html
-rw-r–r-- 1 zammad zammad 5239 Oct 9 03:15 znuny_open_by_button.html
-rw-r–r-- 1 zammad zammad 463707 Oct 9 03:15 znuny.png

/opt/zammad/public/assets/chat/views:
total 40
-rw-r–r-- 1 zammad zammad 214 Oct 9 03:15 agent.eco
-rw-r–r-- 1 zammad zammad 2242 Oct 9 03:15 chat.eco
-rw-r–r-- 1 zammad zammad 484 Oct 9 03:15 customer_timeout.eco
-rw-r–r-- 1 zammad zammad 273 Oct 9 03:15 loader.eco
-rw-r–r-- 1 zammad zammad 242 Oct 9 03:15 message.eco
-rw-r–r-- 1 zammad zammad 108 Oct 9 03:15 status.eco
-rw-r–r-- 1 zammad zammad 84 Oct 9 03:15 timestamp.eco
-rw-r–r-- 1 zammad zammad 373 Oct 9 03:15 typingIndicator.eco
-rw-r–r-- 1 zammad zammad 383 Oct 9 03:15 waiting.eco
-rw-r–r-- 1 zammad zammad 332 Oct 9 03:15 waiting_list_timeout.eco

/opt/zammad/public/assets/custom:
total 8
-rw-rw-r-- 1 zammad zammad 47 Jun 16 02:50 README-bc77bf2f1b2e58cf0ff57d90b554c900bb6063b60693ed49cc27d6db5b255b81.txt
-rw-rw-r-- 1 zammad zammad 66 Jun 16 02:50 README-bc77bf2f1b2e58cf0ff57d90b554c900bb6063b60693ed49cc27d6db5b255b81.txt.gz

/opt/zammad/public/assets/doorkeeper:
total 148
drwxrwxr-x 2 zammad zammad 4096 Oct 14 08:06 admin
-rw-rw-r-- 1 zammad zammad 125213 Jun 16 02:54 application-ccb41f5b8f004022bb9b8b22d7b824c3800915afbb929eb76f2bb56a0646fcc7.css
-rw-rw-r-- 1 zammad zammad 19670 Jun 16 02:54 application-ccb41f5b8f004022bb9b8b22d7b824c3800915afbb929eb76f2bb56a0646fcc7.css.gz

/opt/zammad/public/assets/doorkeeper/admin:
total 144
-rw-rw-r-- 1 zammad zammad 124648 Jun 16 02:54 application-acb09cd0f6b028edaeb4d974047bb20ba934f4000e405e8fa994507daa1dc59c.css
-rw-rw-r-- 1 zammad zammad 19464 Jun 16 02:54 application-acb09cd0f6b028edaeb4d974047bb20ba934f4000e405e8fa994507daa1dc59c.css.gz

/opt/zammad/public/assets/error:
total 172
-rw-r–r-- 1 zammad zammad 2629 Oct 9 03:15 error-1.svg
-rw-r–r-- 1 zammad zammad 3587 Oct 9 03:15 error-2.svg
-rw-r–r-- 1 zammad zammad 6529 Oct 9 03:15 error-3.svg
-rw-r–r-- 1 zammad zammad 19001 Oct 9 03:15 error-4.svg
-rwxr-xr-x 1 zammad zammad 26030 Oct 9 03:15 firasans-regular-webfont.eot
-rwxr-xr-x 1 zammad zammad 72228 Oct 9 03:15 firasans-regular-webfont.ttf
-rwxr-xr-x 1 zammad zammad 28852 Oct 9 03:15 firasans-regular-webfont.woff
-rw-r–r-- 1 zammad zammad 1502 Oct 9 03:15 style.css

/opt/zammad/public/assets/app:
total 4
drwxrwxr-x 2 zammad zammad 4096 Oct 14 08:06 models

/opt/zammad/public/assets/app/models:
total 8
-rw-rw-r-- 1 zammad zammad 912 Jun 16 02:50 application-7670eaba1c43fb379af5a7df8e2261bf55136087f80a1f39030fc0cf5900c8ee.js
-rw-rw-r-- 1 zammad zammad 464 Jun 16 02:50 application-7670eaba1c43fb379af5a7df8e2261bf55136087f80a1f39030fc0cf5900c8ee.js.gz

DCWNZ · October 13, 2020, 7:50pm

Hi, Found a few MySQL connection errors in the latest production log. They seem to appear at random intervals, sometimes every few minutes, others once every 30 minutes.

E, [2020-10-14T03:02:41.801799 #1938835-47453360183720] ERROR -- : Can't reconnect to database #<Mysql2::Error: Can't connect to MySQL server on 'blensqlproxy.indevin3.local' (104)>

execute Sessions.jobs (try_count 0) exited with error #<Mysql2::Error: Lost connection to MySQL server at 'reading initial communication packet', system error: 104> in: 0.003281481 seconds.

That seems like an incredibly low timeframe. Also the connection is never actually lost, Like I said before the WebUI functions normally and the error can be clicked away… Is something killing the connection after too short a time period perhaps?

Database is a Percona MySQL 3-Node Cluster fronted by dedicated ProxySQL host. Runs ~6 Production Databases currently. No others have any issues.

database.yml file looks like this:

production:
  adapter: mysql2
  database: zammad_db
  pool: 50
  timeout: 5000
  encoding: utf8
  username: **redacted**
  password: **redacted**
  host: blensqlproxy.indevin3.local
  port: 6033

Snippet of Log here:

I, [2020-10-14T03:02:26.012153 #1938835-47453432865060]  INFO -- : ended Channel.fetch took: 1.17064112 seconds.
I, [2020-10-14T03:02:41.797823 #1938835-47453360183720]  INFO -- : Scheduler running...
E, [2020-10-14T03:02:41.801799 #1938835-47453360183720] ERROR -- : Can't reconnect to database #<Mysql2::Error: Can't connect to MySQL server on 'blensqlproxy.indevin3.local' (104)>
I, [2020-10-14T03:02:41.806230 #1938835-47453360183720]  INFO -- : script/scheduler.rb stopped.
I, [2020-10-14T03:02:45.647086 #1941979-47062226254240]  INFO -- : Setting.set('product_logo', "a8e4a97422558397b671a76e80125723.png")
I, [2020-10-14T03:02:48.567508 #1941979-47062226254240]  INFO -- : Setting.set('models_searchable', ["User", "Ticket", "Organization", "KnowledgeBase::Answer::Translation", "Chat::Session"])
I, [2020-10-14T03:02:50.611081 #1941979-47062226254240]  INFO -- : Scheduler started.
I, [2020-10-14T03:02:50.630810 #1941979-47062226254240]  INFO -- : Cleanup of left over locked delayed jobs 2020-10-13 14:02:50 UTC started.
I, [2020-10-14T03:02:50.636810 #1941979-47062226254240]  INFO -- : Cleanup of left over locked delayed jobs 2020-10-13 14:02:50 UTC finished.
I, [2020-10-14T03:02:50.636892 #1941979-47062226254240]  INFO -- : Cleanup of left over import jobs 2020-10-13 14:02:50 UTC started.
I, [2020-10-14T03:02:50.640868 #1941979-47062226254240]  INFO -- : Cleanup of left over import jobs 2020-10-13 14:02:50 UTC finished.
I, [2020-10-14T03:02:50.654221 #1941979-47062226254240]  INFO -- : Scheduler running...
I, [2020-10-14T03:02:50.696929 #1941979-47062275304080]  INFO -- : execute Ticket.process_pending (try_count 0)...
I, [2020-10-14T03:02:50.712981 #1941979-47062275304080]  INFO -- : ended Ticket.process_pending took: 0.045567602 seconds.
I, [2020-10-14T03:02:54.654891 #1941979-47062297381280]  INFO -- : Starting worker thread Delayed::Backend::ActiveRecord::Job
I, [2020-10-14T03:03:00.702803 #1941979-47062292670520]  INFO -- : execute Channel.fetch (try_count 0)...
I, [2020-10-14T03:03:00.705888 #1941979-47062292670520]  INFO -- : fetching imap (outlook.office365.com/ictsupport@indevin.com port=993,ssl=true,starttls=false,folder=Inbox,keep_on_server=false,auth_type=LOGIN)
I, [2020-10-14T03:03:01.826999 #1941979-47062292670520]  INFO -- :  - no message
I, [2020-10-14T03:03:01.827154 #1941979-47062292670520]  INFO -- : done
I, [2020-10-14T03:03:01.870526 #1941979-47062292670520]  INFO -- : ended Channel.fetch took: 1.192323207 seconds.
I, [2020-10-14T03:03:10.722937 #1941979-47062292979520]  INFO -- : execute Channel.stream (try_count 0)...
E, [2020-10-14T03:03:20.705302 #1941979-47062294627020] ERROR -- : execute Sessions.jobs (try_count 0) exited with error #<Mysql2::Error: Lost connection to MySQL server at 'reading initial communication packet', system error: 104> in: 0.003281481 seconds.
I, [2020-10-14T03:03:21.744028 #1941979-47062294627020]  INFO -- : execute Sessions.jobs (try_count 1)...
I, [2020-10-14T03:03:30.757785 #1941979-70040687508800]  INFO -- : execute Job.run (try_count

…
TRIMMED
…

 I, [2020-10-14T03:06:39.762596 #1941979-47062292670520]  INFO -- : fetching imap (outlook.office365.com/ictsupport@indevin.com port=993,ssl=true,starttls=false,folder=Inbox,keep_on_server=false,auth_type=LOGIN)
 I, [2020-10-14T03:06:40.894023 #1941979-47062292670520]  INFO -- :  - no message
 I, [2020-10-14T03:06:40.894186 #1941979-47062292670520]  INFO -- : done
 I, [2020-10-14T03:06:40.931445 #1941979-47062292670520]  INFO -- : ended Channel.fetch took: 1.199031657 seconds.
 I, [2020-10-14T03:07:10.964498 #1941979-47062292670520]  INFO -- : execute Channel.fetch (try_count 0)...
 I, [2020-10-14T03:07:10.969518 #1941979-47062292670520]  INFO -- : fetching imap (outlook.office365.com/ictsupport@indevin.com port=993,ssl=true,starttls=false,folder=Inbox,keep_on_server=false,auth_type=LOGIN)
 I, [2020-10-14T03:07:12.107313 #1941979-47062292670520]  INFO -- :  - no message
 I, [2020-10-14T03:07:12.107625 #1941979-47062292670520]  INFO -- : done
 I, [2020-10-14T03:07:12.158766 #1941979-47062292670520]  INFO -- : ended Channel.fetch took: 1.219443774 seconds.
 I, [2020-10-14T03:07:20.970028 #1941979-47062226254240]  INFO -- : Scheduler running...
E, [2020-10-14T03:07:20.972873 #1941979-47062226254240] ERROR -- : Can't reconnect to database #<Mysql2::Error: Lost connection to MySQL server at 'reading initial communication packet', system error: 104>
I, [2020-10-14T03:07:20.975773 #1941979-47062226254240]  INFO -- : script/scheduler.rb stopped.

rsysadmin · October 14, 2020, 7:49am

File ownership look OK to me.
There was an issue, at least until version 3.4.x, which gave Zammad files root.root ownership so they had to be changed to avoid other issues…

rsysadmin · October 14, 2020, 7:52am

can you manually connect to the MySQL server from the Zammad box?

From the Zammad server, please try the following:

ping -c 3 blensqlproxy.indevin3.local <-- this is to see if the DB server can be reached
mysql zammad_db -h blensqlproxy.indevin3.local -P 6033 -u foo -p bar <-- this is to test connectivity to your MySQL. Replace foo and bar with the proper values

Is port 6033 open?

Looking forward to your reply!

Cheers

DCWNZ · October 14, 2020, 7:04pm

Hiya,

Yep sure can connect, as I said before there are several DB’s sitting on the cluster behind that ProxySQL Host. All Applications use that Host to access the Clustered DB’s over port 6033.

ping -c 3 blensqlproxy.indevin3.local
PING blensqlproxy.indevin3.local (172.16.10.24) 56(84) bytes of data.
64 bytes from BLENSQLPROXY.indevin3.local (172.16.10.24): icmp_seq=1 ttl=64 time=0.362               ms
64 bytes from BLENSQLPROXY.indevin3.local (172.16.10.24): icmp_seq=2 ttl=64 time=0.160               ms
64 bytes from BLENSQLPROXY.indevin3.local (172.16.10.24): icmp_seq=2 ttl=64 time=0.177               ms (DUP!)
64 bytes from BLENSQLPROXY.indevin3.local (172.16.10.24): icmp_seq=3 ttl=64 time=0.345               ms

--- blensqlproxy.indevin3.local ping statistics ---
3 packets transmitted, 3 received, +1 duplicates, 0% packet loss, time 2010ms
rtt min/avg/max/mdev = 0.160/0.261/0.362/0.092 ms

and

 mysql zammad_db -h blensqlproxy.indevin3.local -P 6033 -uzammad_user -p
Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 515835
Server version: 5.7.30 (ProxySQL)

Copyright (c) 2009-2020 Percona LLC and/or its affiliates
Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

I have removed all the bonded interfaces from the Cluster machines and the Proxy, to ensure that is not causing the issue. Confirmed issue remains.

Ill do some more troubleshooting with ProxySQL tomorrow to check connection performance and im going to look at enabling caching also, but Zammad appears to be the only application experiencing these disconnections on the DB (im hoping that statement isnt going to come back and bite me on the ass…)

Thanks for your help thus far.

rsysadmin · October 15, 2020, 7:01am

Hello again,

the next step would be to remove Zammad from the proxy and access it directly, just for testing purposes.
Other than that, I can only tell you that our Zammad installation works flawlessly so far…

Please keep me posted. I am curious about what the solution will be.

Cheers

DCWNZ · October 22, 2020, 3:49am

This appears to have solved the issue. Pointing Zammad directly at a single member of the Cluster works, so there must be something it doenst like about our Load Balancing through ProxySQL.

I incresed timeouts and played with connections, nothing appeared to make a difference.

Since Zammad isnt mission critical for now it will stay this way as I dont have time to troubleshoot it any further, not helped byt the fact that all the errors were at Zammad’s End, the Cluster/ProxySQL isnt reporting any dropped or aborted connections.

If we were using Single Writer Mode with ProxySQL this MAY get around the issue, but that isnt what we are looking to do at this stage.
I feel this is related to the Zammad Host and the Ruby on Rails MySQL Driver being utilised, but I cant prove what is causing it, perhaps some form of timeout or it just isnt happy with ProxySQL Multiplexing. Who knows.

For now ill keep monitoring to see if this is the resolution.

rsysadmin · October 22, 2020, 10:54am

Thansk for the update! Good to know how Zammad behaves in different scenarios.
In our case, it is not mission-critical either, so it runs on a single VM which can be recreated in minutes with Terraform and Ansible and Zammad is installed through my script The only thing we take care of is the database, which is backed up daily.

Cheers

system · February 19, 2021, 10:54am

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.