Zammad backgroundworker/puma using high RAM

Technical assistance
1

Infos:

  • Used Zammad version: 6.5.0-173
  • Used Zammad installation type: Docker compose
  • Operating system: Ubuntu
  • Browser + version:
  • Tried rake rebuild ES
  • Reboot
  • Update to most recent docker compose version
  • Check first delayed job + count (0 and nil)

Expected behavior:

  • I would expect the background worker to use more RAM on specific moments, a couple times a week.

Actual behavior:

Steps to reproduce the behavior:

  • Deploy the docker-compose setup

The machine has 8GB of ram. I saw other posts where elasticsearch was eating all the ram. How can I find out what the background worker is working on that makes it consume so much ram? Do we just need to upgrade or is there something hanging?

2

I can corroborate this report, though we have a central Postgres and ElasticSearch, i.e. the machine with Zammad on it only has Zammad, memcached and redis on it. We are seeing 2,2GB RAM usage for the background worker after less than 48 hours on a low-traffic Zammad (that was 6 mails and probably less that 10 agents in that timeframe).

The response to some other reports were for example that this is inherent to how Ruby and Rails works is not something I’m satisfied with (we host a rails app at my day job and while I wouldn’t say we don’t leak memory we’re talking thousands of jobs before restarting the worker because it hits a ~500MB ceiling…), I’m also cognisant of the fact that this is not easy to trace. It is also a problem that indeed exists across the RoR community (there’s plethora of gems to restart processes after some condition for example memory usage is hit).

Now on to solutions, the above linked gems won’t work as Zammad has hand-rolled background processing. I guess I’ll explore limiting the RAM for the zammad-scheduler container and forcing restarts that way.

EDIT: clarified that the background-worker has a very high memory usage.

1 Like
3

We are seeing very high memory usage for the Zammad background-worker.rb script and puma as well. Any chance of some feedback from Zammad devs on why this is happening, and what can be done about it?
To further specify, we are around 10 users and our standalone Zammad VM has got 16GB memory…

4

A Zammad instance for this many users should run fine with 8gb of memory, unless you don‘t restrict Elasticsearchs memory usage and tune the shit out of Zammad. That‘s all difficult to tell without further information like tuning settings etc.

5

Here’s my full docker compose and .env:

x-shared:
  zammad-service: &zammad-service
    environment: &zammad-environment
      MEMCACHE_SERVERS: ${MEMCACHE_SERVERS:-zammad-memcached:11211}
      POSTGRESQL_DB: ${POSTGRES_DB:-zammad_production}
      POSTGRESQL_HOST: ${POSTGRES_HOST:-zammad-postgresql}
      POSTGRESQL_USER: ${POSTGRES_USER:-zammad}
      POSTGRESQL_PASS: ${POSTGRES_PASS:-zammad}
      POSTGRESQL_PORT: ${POSTGRES_PORT:-5432}
      POSTGRESQL_OPTIONS: ${POSTGRESQL_OPTIONS:-?pool=50}
      POSTGRESQL_DB_CREATE:
      REDIS_URL: ${REDIS_URL:-redis://zammad-redis:6379}
      S3_URL:
      # Backup settings
      BACKUP_DIR: "${BACKUP_DIR:-/var/tmp/zammad}"
      BACKUP_TIME: "${BACKUP_TIME:-03:00}"
      HOLD_DAYS: "${HOLD_DAYS:-10}"
      TZ: "${TZ:-Europe/Berlin}"
      # Allow passing in these variables via .env:
      AUTOWIZARD_JSON:
      AUTOWIZARD_RELATIVE_PATH:
      ELASTICSEARCH_ENABLED:
      ELASTICSEARCH_SCHEMA:
      ELASTICSEARCH_HOST:
      ELASTICSEARCH_PORT:
      ELASTICSEARCH_USER: ${ELASTICSEARCH_USER:-elastic}
      ELASTICSEARCH_PASS: ${ELASTICSEARCH_PASS:-zammad}
      ELASTICSEARCH_NAMESPACE:
      ELASTICSEARCH_REINDEX:
      ELASTICSEARCH_SSL_VERIFY:
      NGINX_PORT:
      NGINX_CLIENT_MAX_BODY_SIZE:
      NGINX_SERVER_NAME:
      NGINX_SERVER_SCHEME:
      RAILS_TRUSTED_PROXIES:
      ZAMMAD_HTTP_TYPE:
      ZAMMAD_FQDN:
      ZAMMAD_WEB_CONCURRENCY:
      ZAMMAD_PROCESS_SESSIONS_JOBS_WORKERS:
      ZAMMAD_PROCESS_SCHEDULED_JOBS_WORKERS:
      ZAMMAD_PROCESS_DELAYED_JOBS_WORKERS:
      # Variables used by ngingx-proxy container for reverse proxy creations
      # for docs refer to https://github.com/nginx-proxy/nginx-proxy
      VIRTUAL_HOST:
      VIRTUAL_PORT:
      # Variables used by acme-companion for retrieval of LetsEncrypt certificate
      # for docs refer to https://github.com/nginx-proxy/acme-companion
      LETSENCRYPT_HOST:
      LETSENCRYPT_EMAIL:

    image: ${IMAGE_REPO:-ghcr.io/zammad/zammad}:${VERSION:-6.5.2-67}
    restart: ${RESTART:-always}
    volumes:
      - /var/opt/zammad/storage:/opt/zammad/storage
    depends_on:
      - zammad-memcached
      - zammad-postgresql
      - zammad-redis

services:
  zammad-backup:
    <<: *zammad-service
    command: ["zammad-backup"]
    volumes:
      - /var/opt/zammad/backup:/var/tmp/zammad
      - /var/opt/zammad/storage:/opt/zammad/storage:ro
    user: 0:0

  zammad-elasticsearch:
    image: elasticsearch:${ELASTICSEARCH_VERSION:-8.19.9}
    restart: ${RESTART:-always}
    volumes:
      - /var/opt/zammad/elasticsearch:/usr/share/elasticsearch/data
    environment:
      #ELASTICSEARCH_ENABLE_SECURITY: 'true'
      ELASTICSEARCH_SKIP_TRANSPORT_TLS: 'true'
      ELASTICSEARCH_ENABLE_REST_TLS: 'false'
      # ELASTICSEARCH_USER is hardcoded to 'elastic' in the container.
      ELASTICSEARCH_PASSWORD: ${ELASTICSEARCH_PASS:-zammad}
      discovery.type: single-node
      xpack.security.enabled: 'false'
      ES_JAVA_OPTS: ${ELASTICSEARCH_JAVA_OPTS:--Xms1g -Xmx1g}

  zammad-init:
    <<: *zammad-service
    command: ["zammad-init"]
    depends_on:
      - zammad-postgresql
    restart: on-failure
    user: 0:0

  zammad-memcached:
    command: memcached -m 256M
    image: memcached:${MEMCACHE_VERSION:-1.6.40-alpine}
    restart: ${RESTART:-always}

  zammad-nginx:
    <<: *zammad-service
    command: ["zammad-nginx"]
    networks:
      - default
      - zammad-nginx-external-network
    ports: []
    depends_on:
      - zammad-railsserver

  zammad-postgresql:
    environment:
      POSTGRES_DB: ${POSTGRES_DB:-zammad_production}
      POSTGRES_USER: ${POSTGRES_USER:-zammad}
      POSTGRES_PASSWORD: ${POSTGRES_PASS:-zammad}
    image: postgres:${POSTGRES_VERSION:-17.7-alpine}
    restart: ${RESTART:-always}
    volumes:
      - /var/opt/zammad/postgresql:/var/lib/postgresql/data

  zammad-railsserver:
    <<: *zammad-service
    command: ["zammad-railsserver"]

  zammad-redis:
    image: redis:${REDIS_VERSION:-7.4.7-alpine}
    restart: ${RESTART:-always}
    volumes:
      - /var/opt/zammad/redis:/data

  zammad-scheduler:
    <<: *zammad-service
    command: ["zammad-scheduler"]

  zammad-websocket:
    <<: *zammad-service
    command: ["zammad-websocket"]
    networks:
      default: {}
      zammad-net:
        aliases:
          - zammad-websocket

networks:
  zammad-net:
    external: true
  zammad-nginx-external-network:
    external: true
    name: ${ZAMMAD_NGINX_EXTERNAL_NETWORK}

# General
TZ=Europe/Copenhagen
ZAMMAD_FQDN=zammad.my.domain
#VERSION=  # explicitly set Zammad version here (overriding the version set in zammad-docker-compose file)
#ZAMMAD_HTTP_TYPE=https  # Use HTTPS if behind a reverse proxy

# Setting secure passwords
POSTGRES_USER=zammad
POSTGRES_PASS=redacted
POSTGRES_DB=zammad-postgresql
ELASTICSEARCH_PASS=redacted

# NGINX
ZAMMAD_NGINX_EXTERNAL_NETWORK=zammad-net
NGINX_EXPOSE_PORT=8080  # Publicly exposed Zammad port
NGINX_CLIENT_MAX_BODY_SIZE=50M  # Allow larger uploads
RAILS_TRUSTED_PROXIES=^172\.(18|20)\. # Make Zammad Rails trust external nginx proxy to get true IP of clients
#NGINX_SERVER_NAME=zammad.my.domain # make nginx serve only this site

#ZAMMAD_WEBSOCKET_PORT=6042
#ZAMMAD_WEBSOCKET_HOST=zammad-websocket