Backups flooding diskspace in docker

Technical assistance
1

Hi everyone,

since a couple of days we’re facing a strange issue with zammad

  • Used Zammad version: zammad=3.4.0-4
  • Used Zammad installation source: univention-appcenter, docker image
  • Operating system: UCS
  • Browser + version:

Actual behavior:

  • every 20 to 60 seconds tar.gz files like 20201025085308_zammad_files.tar.gz flooding the diskspace in /var/lib/univention-appcenter/apps/zammad/data/backup/
  • when the zammad container is stopped, there won’t be any archives added
  • as soon as I start the container again, it is creating several new archive files.

Is there a sheduled (backup)job, that couldn’t finish up?

Cheers
Sebastian

1 Like
2

Sounds like your backup container or even whole compose is restarting over and over again.
This is the root issue you’re experience. So you’ll have to find the reason for that.

Checking the railsserver logs might be a first start.
Also docker ps will tell you if a container keeps on restarting.

The file /var/lib/univention-appcenter/apps/zammad/conf/backup.env should contain this:

# Zammad environment file for backup container
BACKUP_SLEEP=86400
HOLD_DAYS=3

and tells the container to wait 86400 seconds in between each backup run.
If it runs more often, it will be a restarting container.

It’s possible that this process is triggered several times during an installation or upgrade.
This is something we can’t change because of the way on how UCS actually performs those upgrades.

3

Hi @MrGeneration,

indeed, it’s the backup container itself

931401d77034 19323d7d4c56 “/usr/local/bin/back…” 2 months ago Up 31 seconds 5432/tcp zammad_zammad-backup_1

In the logs I see lots of I/O errors as well as file not found:

tar: /opt/zammad/tmp/cache/bootsnap-compile-cache/99/e46b5031e8d54b: No such file or directory
tar: /opt/zammad/tmp/cache/bootsnap-compile-cache/97: No such file or directory
tar: /opt/zammad/tmp/cache/bootsnap-compile-cache/b6: No such file or directory
tar: /opt/zammad/tmp/cache/bootsnap-compile-cache/86/986610b125f801: No such file or directory

tar: /opt/zammad/tmp/cache/bootsnap-load-path-cache: No such file or directory
tar: error exit delayed from previous errors
20200818211100 - backuping zammad...

tar: removing leading '/' from member names
tar: read error: I/O error
20201008223508 - backuping zammad...
tar: removing leading '/' from member names
tar: read error: I/O error
4

Sorry I can’t help on that topic any further.

5

I’d like to reopen that case here :sunglasses:

In the end, it seems that we have a corrupt file in zammads file based storage:

irb(main):013:0> Store::File.move('File', 'DB') 
Traceback (most recent call last):
        6: from (irb):13
        5: from app/models/store/file.rb:123:in `move'
        4: from app/models/store/file.rb:123:in `each'
        3: from app/models/store/file.rb:127:in `block in move'
        2: from app/models/store/file.rb:62:in `content'
        1: from app/models/store/provider/file.rb:52:in `get'
RuntimeError (Corrupt file in fs /opt/zammad/storage/fs/748e/4942/be637/383fe/724374b/a41090f/1f29244b15583e0a66416c0446f2914f, sha should be 748e4942be637383fe724374ba41090f1f29244b15583e0a66416c0446f2914f but is e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855)

I wonder if there is a way to eigther purge the specific article or the ticket itself. How can I find the referenced ticket in the database?

Cheers
Sebastian

6

I’m very sorry, but this is a very serious and dangerous operation.
For the safety of users and anons copy catting blindly, I can’t provide any input on this topic.

7

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.