Zammad instance randomly crashes

[derw0lf]

Infos:

• Used Zammad version: Zammad Version 6.4.0 (newest)
• Used Zammad installation type: package
• Operating system: Ubuntu 22.04
• Browser + version: Chrome,Firefox, Edge etc

Expected behavior:

• Zammad should run without any problems and without craching randomly

Actual behavior:

• Our self hosted zammad instance randomly crashes
• When we check production.log or apache2 error.log we cannot see any errors
• Load of server is very low because we only use (15) e-mail channels as ticket input and we only have 25 agents which are working directly in the system

Does anyhove have a hint for me where to search for any errors when the instance crashes?

Thanks and regards

[MrGeneration]

Would you mind to define “crashes”?
You gotta give us at least something what you’re seeing. Even without “error”, you’d notice that something is out of order, no?

[derw0lf]

Yeah, I just thought the same when I clicked on submit.
When the system crashes we just see a white page and the browser is loading without any changes. Sometimes the browser shows a 502 from the webserver and sometimes there is a proxy error message coming up from apache2. A simple restart of the zammad service directly on the server is not possible so that we need to restart the whole server.
Maybe anyone had the same experience when using apache2 reverse proxy with zammad?

[MrGeneration]

So basically this?

Daily timeout (error 504)

Infos: Used Zammad version: 6.4.0-1732538192.197abebd.jammy Used Zammad installation type: Ubuntu package Operating system: Ubuntu 22.04.5 LTS Browser + version: MS Edge 131.0.2903.70 Actual behavior: Once every day we cannot access Zammad but get the error 504 in the browser. Zammad and Nginx services are running. No errors in /var/log/zammad/production.log. Multiple errors in /var/log/nginx/zammad.error.log like: [error] 1068#1068: *16004 upstream timed out (110: Unknown error) while rea…

[derw0lf]

Yeah, seems to be the same problem as described above. I assume this has a high priority for getting a fix?

[MrGeneration]

You can subscribe to the linked issue in the thread. We’re looking for a fix. no ETA.

[martini]

@derw0lf would be grate if you could share more information about your environment. Can you share the output zammad run env, how often the problem appears and any other useful information you have?

Thanks!

[martini]

When the system crashes we just see a white page and the browser is loading without any changes. Sometimes the browser shows a 502 from the webserver and sometimes there is a proxy error message coming up from apache2. A simple restart of the zammad service directly on the server is not possible so that we need to restart the whole server.

JFI: It seems only “zammad-web” (puma) is affected. There is no need to restart the whole server. Only a restart of “zammad-web” should work (maybe you need to kill the process first).

[derw0lf]

@martini i can share any information on this topic that can help you in order to fix it
here is the output from zammad run env:

SHELL=/bin/bash
ORIGINAL_PWD=/root
BUNDLE_WITHOUT=development:test
RACK_ENV=production
APP_USER=zammad
RUBY_GC_OLDMALLOC_LIMIT_MAX=3000100
MALLOC_ARENA_MAX=2
PWD=/opt/zammad
LOGNAME=root
XDG_SESSION_TYPE=tty
PORT=6000
NODE_ENV=production
APP_HOME=/opt/zammad
NODE_HOME=/opt/zammad/.heroku/node
COREPACK_HOME=/opt/zammad/.heroku/corepack
APP_RUNNER_CLI=systemctl
MOTD_SHOWN=pam
HOME=/home/zammad
LANG=C.UTF-8
BUNDLE_DEPLOYMENT=1
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
RUBY_MALLOC_ARENA_MAX=2
RAILS_SERVE_STATIC_FILES=enabled
APP_WIZARDS=
APP_SAFE_NAME=zammad
SSH_CONNECTION=XXX
RUBY_GC_MALLOC_LIMIT_MAX=2177216
RAILS_LOG_TO_STDOUT=
GEM_PATH=/opt/zammad/vendor/bundle/ruby/3.2.0:
APP_GROUP=zammad
LESSCLOSE=/usr/bin/lesspipe %s %s
XDG_SESSION_CLASS=user
TERM=xterm-256color
BUNDLE_BIN=vendor/bundle/bin
LESSOPEN=| /usr/bin/lesspipe %s
USER=root
APP_NAME=zammad
SHLVL=1
DISABLE_DATABASE_ENVIRONMENT_CHECK=0
XDG_SESSION_ID=1765
SECRET_KEY_BASE=XXXX
XDG_RUNTIME_DIR=/run/user/0
RUBY_GC_MALLOC_LIMIT=1077216
SSH_CLIENT=XXXX
BUNDLE_PATH=vendor/bundle
RUBY_GC_OLDMALLOC_LIMIT=2177216
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
PATH=/opt/zammad/.heroku/node/bin:/opt/zammad/bin:/opt/zammad/vendor/bundle/bin:/opt/zammad/vendor/bundle/ruby/3.2.0/bin:/opt/zammad/.heroku/node/bin:/opt/zammad/.heroku/yarn/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin:/opt/zammad/bin:/opt/zammad/node_modules/.bin
ZAMMAD_RAILS_PORT=3000
RAILS_ENV=production
APP_RUNNER_TYPE=systemd
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus
SSH_TTY=/dev/pts/0
OLDPWD=/
DISABLE_SPRING=1

Last week we had multiple crashes within 1 hour and I just updated the current installed zammad version with a newer one from the repository via apt upgrade. Afterwards there was no problem for 2-3 days and then the services crashed again without any reasons we can see from the logs. The production.log shows no error and the apache error.log shows no error. I just increased the log of the mod_proxy module from apache yesterday so that we have more detailed information on the client connections when the next crash happens.

[tlyc]

Same here! Hope someone can fix the issue soon.

[opdevops]

Seeing similar issues with random crashes spaced days apart that just show a white screen with no obvious accompanying error. Zammad 6.4.0 on Ubuntu 22.04. Please let me know if I can provide more details.

APP_RUNNER_CLI=systemctl
LESSOPEN=| /usr/bin/lesspipe %s
RAILS_LOG_TO_STDOUT=
MAIL=/var/mail/zammad
APP_USER=zammad
APP_GROUP=zammad
USER=zammad
APP_SAFE_NAME=zammad
SHLVL=1
DISABLE_SPRING=1
PORT=6000
HOME=/home/zammad
OLDPWD=/home/ubuntu
ORIGINAL_PWD=/home/ubuntu
RUBY_GC_MALLOC_LIMIT_MAX=2177216
APP_NAME=zammad
BUNDLE_WITHOUT=development:test
LC_TERMINAL_VERSION=3.5.10
SUDO_UID=1000
LOGNAME=zammad
BUNDLE_BIN=vendor/bundle/bin
RACK_ENV=production
TERM=xterm-256color
APP_HOME=/opt/zammad
COREPACK_HOME=/opt/zammad/.heroku/corepack
PATH=/opt/zammad/.heroku/node/bin:/opt/zammad/bin:/opt/zammad/vendor/bundle/bin:/opt/zammad/vendor/bundle/ruby/3.2.0/bin:/opt/zammad/.heroku/node/bin:/opt/zammad/.heroku/yarn/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin:/opt/zammad/bin:/opt/zammad/node_modules/.bin
BUNDLE_DEPLOYMENT=1
BUNDLE_PATH=vendor/bundle
RUBY_GC_OLDMALLOC_LIMIT=2177216
NODE_HOME=/opt/zammad/.heroku/node
RUBY_MALLOC_ARENA_MAX=2
LANG=C.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
SUDO_COMMAND=/usr/bin/su zammad
SHELL=/bin/bash
APP_RUNNER_TYPE=systemd
SUDO_USER=ubuntu
LESSCLOSE=/usr/bin/lesspipe %s %s
LC_TERMINAL=iTerm2
RAILS_ENV=production
SECRET_KEY_BASE=a91242d53a5ea85f3311a1a0ce2e9e00d2ce5838ab5514a7082cd0a9ece4b286e79d37fc81afe355a77e6af1018829c539c386d52d653e50a70ddc0696e5ccb8
MALLOC_ARENA_MAX=2
PWD=/opt/zammad
RAILS_SERVE_STATIC_FILES=enabled
GEM_PATH=/opt/zammad/vendor/bundle/ruby/3.2.0:
RUBY_GC_MALLOC_LIMIT=1077216
RUBY_GC_OLDMALLOC_LIMIT_MAX=3000100
REDIS_URL=redis://redis.zammad.db:6379
APP_WIZARDS=
NODE_ENV=production