Memory usage keeps increasing and eventually crashes the server

Technical assistance
1

Infos:

  • Used Zammad version: 7.1.3
  • Used Zammad installation type: https://github.com/zammad/zammad-docker-compose and the current version deployed is 2e99d48
  • Operating system: Ubuntu
  • Browser + version: various/not relevant

Expected behavior:

  • RAM consumption should not constantly increase and eventually bring the server down

Actual behavior:

  • zammad-railserver container memory usage keeps increasing constantly and eventually crashes the server

Steps to reproduce the behavior:

  • run the above mentioned stack ?
2

a few more details:

Zammad version

7.1.3-8facd925.docker

Deployment uses the official zammad/zammad-docker-compose repository via Portainer. Docker-compose revision at deployment:

2e99d48

init: true is enabled for the railsserver container.

Problem

The memory usage of zammad-zammad-railsserver-1 continuously increases during normal operation and does not appear to be released. Eventually the server runs out of memory and crashes.

Restarting only the railsserver container releases the memory, after which the same growth starts again.

The memory growth is entirely attributable to the single Puma process, rather than multiple workers or zombie/child processes.

Example after approximately 54 minutes uptime:

PID    PPID   RSS      VSZ      %MEM %CPU  CMD
7      1      2524344  6598552  15.4  9.5  puma 8.0.2 (tcp://0.0.0.0:3000) [zammad]

This corresponds to approximately 2.4 GiB RSS after less than one hour.

There is only one Puma process:

PID 1: /sbin/docker-init
PID 7: puma 8.0.2

No accumulation of child/zombie processes was observed.

Memory analysis

/proc/<puma-pid>/status :

VmSize:  6598552 kB
VmRSS:   2524488 kB
RssAnon: 2487996 kB
RssFile:   36492 kB
RssShmem:      0 kB
VmData:  5184908 kB
VmSwap:        0 kB

smaps_rollup :

Rss:             2524488 kB
Pss:             2507475 kB
Shared_Clean:      27772 kB
Private_Clean:      8720 kB
Private_Dirty:   2487996 kB
Anonymous:       2487996 kB
Swap:                  0 kB

Therefore almost all of the ~2.5 GB RSS is private anonymous memory belonging to the Puma process. It is not filesystem cache or shared/file-backed memory.

Interestingly, the conventional [heap] mapping accounts for only approximately 193 MB:

[heap]
Size:       210528 kB
Rss:        192656 kB
Anonymous:  192656 kB

There are instead many large anonymous mappings. For example:

64936 kB
58760 kB
58736 kB
54004 kB
53812 kB
53712 kB
53672 kB
52736 kB
51636 kB
50716 kB
...

There were 60 anonymous mappings with sizes between 60 and 66 MB.

At the same time Puma had:

Threads: 17

(21 threads were observed during an earlier measurement.)

No allocator configuration is currently present:

env | grep -E 'MALLOC|LD_PRELOAD|JEMALLOC'

returns nothing.

This pattern may indicate substantial glibc malloc arena retention/fragmentation, although I have not established whether this is the actual cause or a consequence of application-level memory retention.

Rails GC note

A separate rails console showed:

heap_allocated_pages: 1088
heap_live_slots:      1024179
old_objects:           489505

However, these statistics are from the separate Rails console Ruby process and therefore do not represent the Puma process. I include this only to clarify that GC statistics from rails console were not used to diagnose Puma memory.

Observed behaviour

Memory starts relatively low after restarting zammad-railsserver and then grows continuously during operation. It does not return to the previous level when activity decreases.

Immediately before one restart:

Threads: 17
VmRSS:   2530956 kB
RssAnon: 2493336 kB

Eventually this growth exhausts available server memory.

Restarting zammad-zammad-railsserver-1 immediately releases the memory without requiring PostgreSQL, Elasticsearch, scheduler, etc. to be restarted.

As a temporary workaround I am monitoring the railsserver container memory and restarting only that container when it exceeds approximately 3.5 GB.

Questions

Is this memory growth known with Zammad 7.1.3 / Puma 8.0.2?

In particular:

  • Is this amount/rate of Puma RSS growth expected?
  • Are there known memory-retention/leak issues in Zammad 7.1.3?
  • Is Zammad expected to use glibc’s default malloc arena configuration?
  • Is MALLOC_ARENA_MAX or jemalloc recommended/supported for Zammad?
  • Are there Puma/Ruby diagnostics that can be enabled to identify which allocations/objects are responsible for the growth?

The important diagnostic detail is that the growth appears isolated to the single Puma process and consists almost entirely of private anonymous memory, rather than Docker cache, multiple Puma workers, or zombie processes.

3

Ruby processes are generally memory hungry. But without knowing how many concurrent users you have and thus how busy puma is, your questions are impossible to answer.

4

currently just 1 or 2 agents. The thing is that it seems to have started after I upgraded from 7.1.1 to 7.1.2 or 7.1.3 probably the last step…

everything I do on the system increases the RAM footprint but RAM never gets freed.

the dip is restart of the container once it reached 3.5 GiB

Screenshot 2026-08-28 at 15.48.48
Screenshot 2026-08-28 at 15.48.482300×1286 484 KB

5

This change from today could maybe add some insights what we suggest for the future: Maintenance: use jemalloc as memory allocator for package installations. · zammad/zammad@791c607 · GitHub

6

So the Zammad team is aware of problems with memory usage?

7

for other running into this issue as an interim fix I created this docker stack

services:
  zammad-watchdog:
    image: docker:cli
    container_name: zammad-watchdog
    restart: unless-stopped

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

    environment:
      CONTAINER: zammad-zammad-railsserver-1
      LIMIT_MB: "3500"
      CHECK_INTERVAL: "300"
      COOLDOWN: "1800"

    command:
      - /bin/sh
      - -c
      - |
        echo "Watching $$CONTAINER; limit=$$LIMIT_MB MB"

        LAST_RESTART=0

        while true; do
          MEM_USAGE=$$(docker stats \
            --no-stream \
            --format '{{.MemUsage}}' \
            "$$CONTAINER" 2>/dev/null | cut -d/ -f1 | xargs)

          NOW=$$(date +%s)

          case "$$MEM_USAGE" in
            *GiB)
              VALUE=$$(echo "$$MEM_USAGE" | sed 's/GiB//')
              MEM_MB=$$(awk "BEGIN {printf \"%.0f\", $$VALUE * 1024}")
              ;;
            *MiB)
              VALUE=$$(echo "$$MEM_USAGE" | sed 's/MiB//')
              MEM_MB=$$(awk "BEGIN {printf \"%.0f\", $$VALUE}")
              ;;
            *KiB)
              VALUE=$$(echo "$$MEM_USAGE" | sed 's/KiB//')
              MEM_MB=$$(awk "BEGIN {printf \"%.0f\", $$VALUE / 1024}")
              ;;
            *)
              echo "$$(date): Could not parse memory value: '$$MEM_USAGE'"
              MEM_MB=0
              ;;
          esac

          echo "$$(date): Railsserver memory=$$MEM_MB MB ($$MEM_USAGE)"

          if [ "$$MEM_MB" -ge "$$LIMIT_MB" ] && \
             [ $$((NOW - LAST_RESTART)) -ge "$$COOLDOWN" ]; then

            echo "$$(date): Memory $$MEM_MB MB >= $$LIMIT_MB MB - restarting $$CONTAINER"

            docker restart "$$CONTAINER"

            LAST_RESTART=$$NOW
          fi

          sleep "$$CHECK_INTERVAL"
        done

this restarts the zammad-railsserver-1 container when the memory usage reaches 3.5GiB.

I don’t consider this elegant but at least it prevents zammad from dragging down the whole server

8

Not really in this direction here, we identified that the general footprint seems to be higher in some situations, but not that it’s growing endless.

9

@dominikklein any idea on when this could be fixed? As I understand it The jemalloc “fix” would need a new container image and also a new docker-compose.yaml - right?

would it be possible to downgrade the installation to 7.1.1 (I think the issue appeared either when I upgraded to 7.1.2 or 7.1.3).

I think there was a DB upgrade done during one of those upgrades so I suppose I cannot just downgrade…

10

7.1.3 is a security release. Before thinking about a downgrade, you might wanna check the CVEs first to see if you’re willing the risk.

11

I did. check the CVE it seems mainly to be issue that could be exploited when logged in.

Right now with our usage pattern I need to kill rail server every 2 hours. This seems to work but I am unsure if this will not affect the agents at some points or even worse corrupt the data.

Not an easy choice… But I’m leaning towards downgrading…

12

Zammad 7.2 images will contain the changes for jemalloc, which improves the memory handling of Zammad a lot. It changes the behaviour to not only keep adding memory to the Rails processes, but actually freeing it when not needed any more. No docker stack configuration changes required.

This release is planned in a few weeks. Until then, you could try to daily restart your railsserver processes outside of business hours to keep the memory in check. Alternatively (if you know what you’re doing), you could try backporting the commit and building a local image based on stable.

13

How many concurrent agents do you have, and what are your configuration settings for puma (WEB_CONCURRENCY, MIN_THREADS and MAX_THREADS)?

14

currently I am restarting zammad automatically when the memory footprint reaches 3.5 GiB. I created a supervisor stack (see Memory usage keeps increasing and eventually crashes the server - #7 by ruettinger) .

This results in restarting every couple of hours depending on usage). We have max 3-4 concurrent agents. So this is really a PITA currently. and it is something that appeared either with 7.1.2 or 7.1.3…

15

I am using the defaults (no env set for those values)

16

Then it is definitely unexpected and unusual behaviour that we haven’t seen so far.