Update from 5.0 to 5.4 keeps failing

Technical assistance
1

Infos:

  • Used Zammad version: 5.0 → 5.4
  • Used Zammad installation type: package
  • Operating system: Debian 11
  • Browser + version: Latest Chrome

Expected behavior:

  • Running apt Update && apt upgrade should update Zammad to the newer version

Actual behavior:

  • Zammad Update finishes with a lot of errors, resulting in a the whole install being broken and the browser only showing a 502 Error

Steps to reproduce the behavior:

  • Install Zammad pinned to version 5.0 on new host, recover a backup from different host running zammad 5.0, update Zammad on new Host to 5.4.

Hello,

relatively new to Zammad, I’m currently trying to move a production Environment from an old host to a new one.
Since the current Environment is running Zammad 5.0.x according to the Version Info, I Installed that version on the new host to apply the backup I created on the current System.

This works fine with me being able to login on the new system and everything I backed up before is there.

I then put the new system in Maintenance mode and disabled E-Mails to not send any notifications by accident.

After that I try to update the new system to Zammad 5.4 before I want to also upgrade to 6.0. The Update process fails with multiple errors like this:

opt/zammad/vendor/ruby-3.1.3/lib/ruby/3.1.0/fileutils.rb:243:in `mkdir': File exists @ dir_s_mkdir - /opt/zammad/storage/fs (Errno::EEXIST)

Before those errors appear it looks fine to me:

Get:1 https://dl.packager.io/srv/deb/zammad/zammad/stable-5.4/debian 11/main amd64 zammad amd64 5.4.1-1686035985.315231a1.bullseye [143 MB]
Fetched 143 MB in 9s (15.8 MB/s)
Reading changelogs… Done
(Reading database … 55824 files and directories currently installed.)
Preparing to unpack …/zammad_5.4.1-1686035985.315231a1.bullseye_amd64.deb …
Unpacking zammad (5.4.1-1686035985.315231a1.bullseye) over (5.0.3-1675247826.34800255.bullseye)
dpkg: warning: unable to delete old directory ‘/opt/zammad/vendor/bundle/ruby/2.7.0/specifications’: Directory not empty
dpkg: warning: unable to delete old directory ‘/opt/zammad/vendor/bundle/ruby/2.7.0/extensions/x86_64-linux/2.7.0-static’: Directory not empty
dpkg: warning: unable to delete old directory ‘/opt/zammad/vendor/bundle/ruby/2.7.0/extensions/x86_64-linux’: Directory not empty
dpkg: warning: unable to delete old directory ‘/opt/zammad/vendor/bundle/ruby/2.7.0/extensions’: Directory not empty
dpkg: warning: unable to delete old directory ‘/opt/zammad/vendor/bundle/ruby/2.7.0/gems’: Directory not empty
dpkg: warning: unable to delete old directory ‘/opt/zammad/vendor/bundle/ruby/2.7.0’: Directory not empty
Setting up zammad (5.4.1-1686035985.315231a1.bullseye) …
(Re)creating init scripts
Nothing to do.
Nothing to do.
Nothing to do.
Enabling Zammad on boot
Stopping Zammad
Clear cache…

Basically every Task that tries to access/create /opt/zammad/storage/fs fails.

I tried it with FULL_FS_DUMP='no' as well as FULL_FS_DUMP='yes' in the backup config, always changing those at source and target, but the errors keep being the same.

I’m out of Ideas and neither google nor this forum have led me to any solution so far.

Thanks in advance for any pointers!

2

I tried a few more things, changed the order in which I installed the Updates and/or applied the backup, but so far to no avail. I did get a few different errors to show though.

When installing Zammad 5.4 first, then redis plus adding Redis to the config like this:

Creating /etc/zammad/conf.d/redis
Adding REDIS_URL="redis://localhost:6379"

I get this error at the end of importing the backup:

Clearing Cache …
Traceback (most recent call last):
56: from /opt/zammad/bin/rails:9:in <main>' 55: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in require’
54: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in require_with_bootsnap_lfi' 53: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in register’
52: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in block in require_with_bootsnap_lfi' 51: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in require’
50: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/commands.rb:18:in <main>' 49: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/command.rb:46:in invoke’
48: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/command/base.rb:69:in perform' 47: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor.rb:392:in dispatch’
46: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in invoke_command' 45: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor/command.rb:27:in run’
44: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/commands/runner/runner_command.rb:33:in perform' 43: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/command/actions.rb:15:in require_application_and_environment!’
42: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/command/actions.rb:28:in require_environment!' 41: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/application.rb:339:in require_environment!’
40: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/dependencies.rb:324:in require' 39: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/dependencies.rb:291:in load_dependency’
38: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/dependencies.rb:324:in block in require' 37: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.4.2/lib/zeitwerk/kernel.rb:34:in require’
36: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in require' 35: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in require_with_bootsnap_lfi’
34: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in register' 33: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in block in require_with_bootsnap_lfi’
32: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in require' 31: from /opt/zammad/config/environment.rb:7:in ’
30: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/application.rb:363:in initialize!' 29: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/initializable.rb:60:in run_initializers’
28: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:205:in tsort_each' 27: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:226:in tsort_each’
26: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:347:in each_strongly_connected_component' 25: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:347:in call’
24: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:347:in each' 23: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:349:in block in each_strongly_connected_component’
22: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:415:in each_strongly_connected_component_from' 21: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:415:in call’
20: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/initializable.rb:50:in tsort_each_child' 19: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/initializable.rb:50:in each’
18: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:421:in block in each_strongly_connected_component_from' 17: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:431:in each_strongly_connected_component_from’
16: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:422:in block (2 levels) in each_strongly_connected_component_from' 15: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:350:in block (2 levels) in each_strongly_connected_component’
14: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:228:in block in tsort_each' 13: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/initializable.rb:61:in block in run_initializers’
12: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/initializable.rb:32:in run' 11: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/initializable.rb:32:in instance_exec’
10: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/engine.rb:624:in block in <class:Engine>' 9: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/engine.rb:624:in each’
8: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/engine.rb:625:in block (2 levels) in <class:Engine>' 7: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/engine.rb:665:in load_config_initializer’
6: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/notifications.rb:182:in instrument' 5: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/engine.rb:666:in block in load_config_initializer’
4: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/dependencies.rb:318:in load' 3: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/dependencies.rb:291:in load_dependency’
2: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/dependencies.rb:318:in block in load' 1: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:60:in load’
/opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:60:in `load’: /opt/zammad/config/initializers/active_record_as_batches.rb:23: syntax error, unexpected ‘)’, expecting local variable or method (SyntaxError)
def as_batches(query, &)
^
/opt/zammad/config/initializers/active_record_as_batches.rb:27: syntax error, unexpected ‘)’
records.each(&)
^
/opt/zammad/config/initializers/active_record_as_batches.rb:43: syntax error, unexpected ‘)’, expecting local variable or method
… def as_batches(args = {}, &)
… ^
/opt/zammad/config/initializers/active_record_as_batches.rb:44: syntax error, unexpected ‘)’
…arel, args).as_batches(self, &)
… ^
/opt/zammad/config/initializers/active_record_as_batches.rb:48: class definition in method body
class Relation
^~~~~~~~~~~~~~

If I don’t install redis before the Upgrade to Zammad 6 I get this Error. Redis is being installed on the system during the Upgrade. From what I understood from the Documentation I shouldn’t have to do any manual changes prior to the update to Zammad 6.

/opt/zammad/config/initializers/action_cable_preferences.rb:14:in <main>': uninitialized constant Redis (NameError) 52: from /opt/zammad/bin/rails:9:in ’
51: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in require' 50: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in require_with_bootsnap_lfi’
49: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in register' 48: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in block in require_with_bootsnap_lfi’
47: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in require' 46: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/commands.rb:18:in ’
45: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/command.rb:46:in invoke' 44: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/command/base.rb:69:in perform’
43: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor.rb:392:in dispatch' 42: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in invoke_command’
41: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor/command.rb:27:in run' 40: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/commands/runner/runner_command.rb:33:in perform’
39: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/command/actions.rb:15:in require_application_and_environment!' 38: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/command/actions.rb:28:in require_environment!’
37: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/application.rb:339:in require_environment!' 36: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.4.2/lib/zeitwerk/kernel.rb:34:in require’
35: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in require' 34: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in require_with_bootsnap_lfi’
33: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in register' 32: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in block in require_with_bootsnap_lfi’
31: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in require' 30: from /opt/zammad/config/environment.rb:7:in ’
29: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/application.rb:363:in initialize!' 28: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/initializable.rb:60:in run_initializers’
27: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:205:in tsort_each' 26: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:226:in tsort_each’
25: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:347:in each_strongly_connected_component' 24: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:347:in call’
23: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:347:in each' 22: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:349:in block in each_strongly_connected_component’
21: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:415:in each_strongly_connected_component_from' 20: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:415:in call’
19: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/initializable.rb:50:in tsort_each_child' 18: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/initializable.rb:50:in each’
17: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:421:in block in each_strongly_connected_component_from' 16: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:431:in each_strongly_connected_component_from’
15: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:422:in block (2 levels) in each_strongly_connected_component_from' 14: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:350:in block (2 levels) in each_strongly_connected_component’
13: from /opt/zammad/vendor/ruby-2.7.4/lib/ruby/2.7.0/tsort.rb:228:in block in tsort_each' 12: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/initializable.rb:61:in block in run_initializers’
11: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/initializable.rb:32:in run' 10: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/initializable.rb:32:in instance_exec’
9: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/engine.rb:624:in block in <class:Engine>' 8: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/engine.rb:624:in each’
7: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/engine.rb:625:in block (2 levels) in <class:Engine>' 6: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/engine.rb:665:in load_config_initializer’
5: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/notifications.rb:182:in instrument' 4: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/railties-6.0.4.1/lib/rails/engine.rb:666:in block in load_config_initializer’
3: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:60:in load' 2: from /opt/zammad/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:60:in load’
1: from /opt/zammad/config/initializers/action_cable_preferences.rb:13:in <main>' /opt/zammad/config/initializers/action_cable_preferences.rb:16:in rescue in ': uninitialized constant Redis (NameError)

Anyone with any Idea? I’m completely out of Ideas now.

3

Hi @Hannsr. Please have a look here.

4

Hey,

thanks for that pointer. Not sure I got what you’re referring to specifically.
I checked the Ruby version and as far as I can tell it runs on 3.1.3 currently - and since I used the package installation it should automatically pick the right one by itself, right?

There are only 2 Symlinks in /opt/zammad/bin pointing towards ruby 2.7.4:
racc2y -> ../vendor/ruby-2.7.4/bin/racc2y
y2racc -> ../vendor/ruby-2.7.4/bin/y2racc

As y2racc and racc2y don’t exist in /opt/zammad/vendor/ruby-3.1.3/bin/ I think those are deprecated or something?

Also the other dependencies, from what I can tell, are installed.

Redis is basically just the default setup, I’m only testing at this point so I did not set up any authentication yet.

5

Please follow this documentation page expecting a full filesystem dump:
https://docs.zammad.org/en/latest/appendix/backup-and-restore/migrate-hosts.html

FULL_FS_DUMP may not be available in the 5.0 version you have installed yet, it was introduced during it’s lifetime and safely can be expected starting Zammad 5.1.

Alternatively you can copy this repo directory to your 5.0 instance to get the backup features you desire to reduce workload after importing:

5.0 and 5.4 require different ruby versions. This is handled by the packager (and being bundled) on package installations. Usually your package manager will clean up these during an upgrade if they’re no longer required. However, this won’t work if you migrate those kind of directories to a newer version. You may have to cleanup manually.

1 Like
6

Thank you.
I did the first part as per the docs - I did that before as well, but just making sure again. This leaves me with a working Zammad 5.0 installation, containing all the data I backed up, which is good.

So this means usually, if I just run the regular update, it should handle the newer versions and such by itself without me doing anything? Or do I have to do some cleanup because I update from 5.0 to 5.4?

I simply went with this as there is no further steps I could find:
https://docs.zammad.org/en/latest/install/update.html

Which, again, left me with a broken Zammad install and those errors:

Clear cache…
/opt/zammad/vendor/ruby-3.1.3/lib/ruby/3.1.0/fileutils.rb:243:in `mkdir’: File exists @ dir_s_mkdir - /opt/zammad/storage/fs (Errno::EEXIST)

database.yml found. Updating db…
rake aborted!
Errno::EEXIST: File exists @ dir_s_mkdir - /opt/zammad/storage/fs

Updating translations…
/opt/zammad/vendor/ruby-3.1.3/lib/ruby/3.1.0/fileutils.rb:243:in `mkdir’: File exists @ dir_s_mkdir - /opt/zammad/storage/fs (Errno::EEXIST)

And so on.

Are there any steps in between not documented or what do I miss here?

7

Hey,

so after a few days off I gave it another go.
I tried this:

But to no avail. I still end up with a lot of Errors during the Update from 5.0 to 5.4.
From the Old Host:

cat /opt/zammad/contrib/backup/config
#!/usr/bin/env bash
#
# zammad backup script config
#

BACKUP_DIR='/var/tmp/zammad_backup'
HOLD_DAYS='10'
DEBUG='no'
FULL_FS_DUMP='no'

# Zammad backup started - Wed 06 Sep 2023 07:28:43 PM CEST!

creating file backup...
 ... only with productive data (attachments)
creating postgresql backup...
Ensuring dump permissions ...

# Zammad backuped successfully - Wed 06 Sep 2023 07:28:50 PM CEST!

From the new host:

cat config
#!/usr/bin/env bash
#
# zammad backup script config
#
# Learn more about the options below at
# https://docs.zammad.org/en/latest/appendix/backup-and-restore/configuration.html

BACKUP_DIR='/var/tmp/zammad_backup'
HOLD_DAYS='10'
FULL_FS_DUMP='no'
DEBUG='no'

# Zammad restore started - Thu Sep  7 08:04:01 UTC 2023!

The restore will delete your current database! 
Be sure to have a backup available! 

Please ensure to have twice the storage of the uncompressed backup size! 


Note that the restoration USUALLY requires root permissions as services are stopped! 


Enter 'yes' if you want to proceed!
Restore?: yes
Enter file date to restore: 
20230906192843
File date: 20230906192843
Enter db date to restore: 
20230906192843
DB date: 20230906192843
# Stopping Zammad
# Checking requirements
# ... Dropping current database zammad
Dropped database 'zammad'
# ... Creating database zammad for owner zammad
CREATE DATABASE
# Restoring PostgreSQL DB
# Restoring Files
# Ensuring correct file permissions ...
# Clearing Cache ...
# Starting Zammad

# Zammad restored successfully - Thu Sep  7 08:04:18 UTC 2023!

The update still fails with no such file or directory on a bunch of locations:

Unpacking zammad (5.4.1-1686035985.315231a1.bullseye) over (5.0.3-1675247826.34800255.bullseye) ...
Setting up zammad (5.4.1-1686035985.315231a1.bullseye) ...
# (Re)creating init scripts
Nothing to do.
Nothing to do.
Nothing to do.
# Enabling Zammad on boot
# Stopping Zammad
# Clear cache...
/opt/zammad/app/models/store/provider/file.rb:30:in `binread': No such file or directory @ rb_sysopen - /opt/zammad/storage/fs/8e8b/1cca/b9915/82246/b0fb73a/86dfe5d/e0148333e94267781b7c1226f32e887f (Errno::ENOENT)

I also once tried with duping the whole /storage/fs/ content onto the new host prior to restoring the backup, which had the same effect.

I still don’t fully understand the bit about ruby/rails versions, but I don’t copy anything from the old host but the backup files from var/tmp/zammad_backup/, everything else is a fresh install on a fresh debian 11 Host. I Kill/Create the Host using Terraform between each try so I don’t drag along mistakes when retrying.

I really have no clue what else to try.

I thought about Updating the old system to Zammad 6 first to maybe mitigate some issues and start from scratch on the newest version, but tbh, with those current problems I’m not sure I dare updating a live system.

8

That is correct.

Normally this is not required.

The instance is stopped during the backup right?
This message should only appear if you have either a database that’s out of sync to your storage or you have had data loss in the attachment store.

You can verify this on your functioning installation by running Store::File.verify in a rails console. It will return true if you’re good and scream at you (and abort) if a file is missing. That’s hopefully not the case here…

What you have to ensure, no matter if it’s a test system you use before nuking productive, that the data is consistent. That means if you’re using a clone of your production, make sure to stop it before so data indeed is in sync.

Otherwise we/you may be chaising dead rabbits. That would just burn your time without use.

That approach is the correct behavior. From what I can tell you’re acting correctly. The question is why the restore is so terribly broken for you.

9

Yes, I stopped it as per the docs with systemctl stop zammad, then ran the backup script. I did not wait in between those two steps, are there any background tasks that might need to finish first?

I’ll check that in the evening when there’s no one working on tickets.
In case there is some error, any pointers on what would/could be the next steps?

Just to make sure: I used FULL_FS_DUMP='no' while creating the backup, so all I need to transfer is the created backup files, nothing from /opt/zammad/storage/fs/ ? Because as far as I can tell, it’ll recreate the folder structure like on the current host, but without any files - this is correct behavior of Zammad?

Thanks again for your time!

10

Not sure if I get you right.
You want to start Zammad after the backup script finished not in between it’s steps.

When the script tells you “I’m done” then it’s done and does not any secret magic beside that.
Same applies to Zammad. if you stopped it and no processes of Zammad are running any more, it’s in the state you stopped it.

You can do this during normal work hours without problems. It’s a safe operation.

I’m afraid that’s the part where I can’t provide community based support further.
The issue with missing attachments is that it requires manual cleanup in the datastructure.

This is something I wouldn’t like to share in public because copy cats around might accidently break their neck because scope might be missing. Wouldn’t be the first time.

Apart from that it is a very time consuming process. I had to do it on several instances 2-3 years ago and I burned whole working days just fixing these instances.

Correct.
You only copy the two backup files you receive and restore these exact same files and you should be good to go.

You’re installing a new installation of Zammad. Not even the need to use the getting started wizard or to configure anything. You then restore and Zammad overwrites database and attachment files if they’re in the way. Just like that.

Zammad would create the storage/fs/ folder structure if you tell it you want to store to filesystem and there’s no storage folder yet, it will create the folder at the moment it needs it.

Something else coming to mind while typing here.
I currently expect you’re not symlinking the storage folder to your Zammad installation. The backup script didn’t support that up to … I think Zammad 6. (Didn’t check)

You can double check that by running du -h --max-depth=1 /opt/zammad/storage/ on both the source and the destination (after restoring). The directory sizes should be very close to identical. There might be slight differences if you’re using another filesystem type. But normally you’d expect a 1to1 result.

Worth checking as well

11

Guess it was just lost in translation. I meant that I stop Zammad, run the backup script, then start zammad once the backup is finished. Nothing else.

Fair enough, let’s hope it’s not the issue then.

Yeah, I only run the restore script and after that the new instance is working. What I meant is, that during the restore it’ll recreate the structure as far as I can tell. At least on the new host all the folders are back.

I did not set the original system up so I just double checked and it is indeed symlinked.
So I guess I’ll have to move the files back to the main storage so it’s not symlinked anymore? There is enough space on the system, so that’s no problem, anything I have to look out for except setting the correct file permissions once everything is moved?

So my plan would be: I’d just stop zammad, remove the symlink, copy everything over the the main storage, check/fix permissions and start zammad back up.

12

Yes.
Alternatively you update to Zammad 6.0 (if possible) and backup from there. It should support symlinks by then. There are officially supported since 5.4 or so. Your backup and restore script most likely doesn’t follow the symlink and basically thus shoots your back. :-/

Ummm yes correct. Backing up should work fine as well with that approach on your version as well.

13

I’ve had that thought, but tbh, I’m not sure if I wanna update the production system after this (my first) Zammad encounter with updates. Of course there are Backups, but still. It’s most likely my own fault for not checking for symlinks, but yeah ^^
Once the Update worked on the new system I’ll feel more confident to do so in the future :slight_smile:

I know it’s probably more work, but since the new system won’t have to rely on this external storage solution, I’ll have to move those files anyway at some point.

Thanks again, I’ll report back once I’m done (hopefully).

1 Like
14

I just tried this and got this output. I guess it’s not an error in the files, but more about the version I’m running or something like that?

irb(main):001:0> File::Store.verify 
Traceback (most recent call last):
        1: from (irb):1
NameError (uninitialized constant File::Store)
15

Ahhh I am so sorry I flipped the command :see_no_evil:
It’s Store::File.verify my bad.

16

That also explains why there weren’t many results looking for that command. ^^

The bad news though…

irb(main):001:0> Store::File.verify
read from fs /opt/zammad/storage/fs/0697/cb9d/e12a8/2bb46/57575d6/b3a10b7/78012b2e3ac9e5002a65b733c5eef2cf
Traceback (most recent call last):
        6: from (irb):1
        5: from app/models/store/file.rb:81:in `verify'
        4: from app/models/store/file.rb:82:in `block in verify'
        3: from app/models/store/file.rb:62:in `content'
        2: from app/models/store/provider/file.rb:30:in `get'
        1: from app/models/store/provider/file.rb:30:in `binread'
Errno::ENOENT (No such file or directory @ rb_sysopen - /opt/zammad/storage/fs/0697/cb9d/e12a8/2bb46/57575d6/b3a10b7/78012b2e3ac9e5002a65b733c5eef2cf)

I guess we found the issue :see_no_evil:

17

So, by pure coincidence, we found which file (possibly) is giving us those troubles.
One of our Admin-Users has a broken Avatar, which he can’t change as whenever the menu to change the Avatar is opened, Zammad throws an Error 500 for that particular Menu:

Bildschirmfoto 2023-09-25 um 09.46.48
Bildschirmfoto 2023-09-25 um 09.46.481372×264 16.2 KB

Which is exactly the file Store::File.verify is yelling at me about.

Is there any way of manually fixing that missing/broken Avatar without too much trouble?

We’ve had this particular Problem on a different Zammad Install as well - for the same user, weirdly - and resolved it by simply creating a new User and removing the old one. Since there are a few hundred tickets attached to this one now, I’m afraid this won’t be as easy to fix.

Edit: I poked around the console a bit (just looking for now) and found the user properties and if I understand correctly, I can manipulate those using the console as well, right?
So my Idea would be somehow editing this image: "f6eace6be052cf067e8a366285027e1c", image_source: nil, web: "" into image: nil, image_source: nil, web: "" so it falls back to the default avatar? At least the latter is what’s set to a different user that doesn’t have this issue.

Edit2: I tried that on the new system before I try to update anything and it seems to work like that with u.image = '3664fcaa0103d1f2593fce55385e6da2' in rails console to replace the broken link to a working one. The account now works again, Store::File.verify is still unhappy though with the same error.

Not sure if that’s a viable way or if that would break it further, so if you have any insights on this (even if it’s just telling me to absolutely not do this) it’d be much appreciated.

Any thoughts/Ideas about this?

Thanks!

18

Another Update, I think I might have fixed the issue. More or less, will have to test a bit more.

I found an older answer from @MrGeneration on the forum on how to pluck missing files from the index (that’s what I understood), not gonna put it here as it has a clear warning it may break stuff. I tested it in a DEV Environment.

I think a hand full of files got removed during the process. There wasn’t a summary, so can’t tell really.

So after I ran that script Store::File.verify returns => true. I ran a complete Search Index rebuild without errors. Also the Update to first 5.4 and then 6.11 went smooth, so my test-environment now works on the latest version of Zammad.

Except now my Avatar is broken and won’t be shown anymore, guess that got removed during plucking the fs. But I know how to fix that now. :wink:

19

I’m glad you could solve the issue. :slight_smile:

20

Just as a quick note to whomever might stumble across this same issue.

I resolved it and moved the instance to the new host now, every check runs successful and so far no complaints from the support agents.

What I had to do in the end:

Create a backup as per the docs. Set up a new host with the same Zammad version and restore the backup - again, as per the docs.

I then moved all files from the symlink to /opt/zammad/storage/fs (they got backed up properly by the script btw., even following the symlink) and ran the check Store::File.verify which returned a missing file.

So I ran the second script not posted here to pluck those broken/missing files. After that Store::File.verify returned => True and I ran the Updates from 5.0 to 5.4 to latest, set up Redis and Memcached as well as adding the /cable location in nginx.

So thanks again for all your help and time!

As a side-note: Adding how to add redis to the zammad config, including a password-secured redis, to the docs might be helpful. The Syntax how to pass zammad config:set REDIS_URL= to not only add the IP/Port but also a password would be sufficient.

1 Like