Upgrade from 2.6 to 3.4 and host migration

Infos:

  • old system Zammad version: 2.6.x with ubuntu 16.10, Elastic Search 5.6.4
  • new system Zammad version: 3.4.x with ubuntu 18.04 LTS, Elastic Search 7.8.0
  • Used Zammad installation source: package
  • Operating system: Ubuntu
  • Browser Chrome 84.0.4147.105 (Offizieller Build) (64-Bit)

Expected behavior:

  • Update from very old system to newest System with the backup and restore scripts

What I did so far:

  • We use postresql as database thats why i run the update script as root.

  • On the old system i put zammad in Maintainance Mode and then shut it down(systemctl stop zammad).

  • Run the backupscript as root, and put it on a virtual HDD.

  • On the new system elastic search 7.8.0 is running and working.

  • I mount the virtual HDD there and then run the restore script there. The rights for the backup are rw for root.

  • My backup is around 23Gb

  • I have 65Gb free space and on the virtual HDD 74Gb free

cd /opt/zammad/contrib/backup
systemctl stop zammad
./zammad_restore.sh

at the End:

==> I get several errors in the script before and after. I have hundrets of lines of code if you wanna see that.

  • I have a error on the webinterface Bad Gateway 502.

  • Then I do a apt install zammad

  • Searchindex rebuild

  • then I get the login screen back but then im unable to login with my admin user, which comes over a LDAP link.

    indent preformatted text by 4 spaces

What I like to know

What is the best way to update my installation and go to the latest version, as it looks like this is not working that way. I can do that in smaller steps if someone tells me how. Or am I only missing something.

Any help highly appreciated.

New Information.

I can install the backup, but looks like only a part of the Database is installed.
There are no user, only a nicole.braun@zammad.org is in my database. I set a password to her and give her admin rights, then im able to login.
And only a few settings. So I guess there must me a step in the middle which I need to go, so that the database is migrated correctly.

I don’t see any tickets

Doesn’t seem like your database was restored at all.
Please install Zammad first and then restore your backup.

You can follow the guide on our documentation, it should help you through:
https://docs.zammad.org/en/latest/appendix/backup-and-restore.html

If above doesn’t help, please provide the error messages you stumbled upon, because other wise it’s a lot of fishing in the dark.

Hello

I have logged the putty output and put it in a file.

Console Logfile of restore

After that I get a 502 Bad Gateway error, and if I continue do do what is listed in the manual here Backup-and-restore, I end up on a loginscreen where my credentials are not working.

Regards
Steffen

Hello,

this is the reason for your issues:

# Ensuring correct file rights ...
# ... Dropping current database zammad
e[31mbundler: failed to load command: rake (/opt/zammad/vendor/bundle/ruby/2.4.0/bin/rake)e[0m
LoadError: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory - /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/x86_64-linux/digest/sha1.so
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/source/git.rb:4:in `<top (required)>'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/lockfile_parser.rb:122:in `<class:LockfileParser>'
  /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/lockfile_parser.rb:14:in `<module:Bundler>'
  /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/lockfile_parser.rb:13:in `<top (required)>'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/definition.rb:2:in `<top (required)>'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler.rb:129:in `definition'
  /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler.rb:95:in `setup'
  /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/setup.rb:9:in `<top (required)>'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/digest.rb:16:in `const_missing': library not found for class Digest::MD5 -- digest/md5 (LoadError)
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/source/rubygems/remote.rb:26:in `cache_slug'
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/fetcher/compact_index.rb:107:in `cache_path'
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/fetcher/compact_index.rb:83:in `compact_index_client'
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/cli.rb:636:in `warn_on_outdated_bundler'
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/cli.rb:15:in `ensure in start'
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/cli.rb:16:in `start'
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/exe/bundle:30:in `block in <top (required)>'
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/friendly_errors.rb:121:in `with_friendly_errors'
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/exe/bundle:22:in `<top (required)>'
	from /opt/zammad/bin/bundle:3:in `load'
	from /opt/zammad/bin/bundle:3:in `<main>'
# ... Creating database zammad for owner zammad
ERROR:  database "zammad" already exists
# Restoring PostgrSQL DB

As “ruby 2.4” is mentioned in this traceback I guess that you’re not using your fresh installed Zammad package to restore the backup.

  • install zammad as per documentation
  • copy the backup files from your old host
  • restore as per documentation

As I guess that this is the steps you did, seems like the order in the restore script is off.
This may explain why it wants to use the old ruby version that may not be available in old systems.

Who knows why these errors didn’t strike me during tests.
Anyway, you can fast forward these issues by dropping, re creating the zammad database manually.

dropdb zammad should do the trick, if not you may have to do it in the psql console.
The following command will re-create a new zammad database as needed:
su -c "psql -c \"CREATE DATABASE zammad OWNER zammad;\"" postgres

After that you could just rerun the restore script and technically ignore the traceback error.
Loading the dump then should work without issues, as the database is empty at that moment.

Sorry for that ugly workaround.

Hello

well I use the fresh installed zammad. And then try to restore the old one in that. I try that on next monday and let you know

Hey,

I just wanted to point out that the restore order of our script is at fault:

Regards

Hello,

so I tried it and it looks for me like the database is restored but i get other errors now. I have linked my console output. here you can see what I did and what happened.

  • At the end I get the loginscreen and when I try to login with my account i get a Error Message over the loginscreen “Error ID e_SzqKzu: Please contact your administrator”.
  • Then I tried to manuelly set a password via console and get another SQL error
    i loooked up my user ID which is 23 and then did a

User.find_by(login: ‘xyz’).update!(password: ‘test’)

irb(main):003:0> User.find_by(login: 'xyz').update!(password: 'test')
User.find(23) notify UPDATED 2020-08-24 15:32:59 UTC
Traceback (most recent call last):
    8: from (irb):3
    7: from app/models/concerns/has_search_index_backend.rb:27:in `search_index_update'
    6: from app/jobs/concerns/has_active_job_lock.rb:6:in `block (2 levels) in <module:HasActiveJobLock>'
    5: from app/jobs/concerns/has_active_job_lock.rb:57:in `ensure_active_job_lock_for_enqueue!'
    4: from app/jobs/concerns/has_active_job_lock.rb:86:in `in_active_job_lock_transaction'
    3: from app/jobs/concerns/has_active_job_lock.rb:58:in `block in ensure_active_job_lock_for_enqueue!'
    2: from app/jobs/concerns/has_active_job_lock.rb:107:in `active_job_lock_for_enqueue!'
    1: from app/jobs/concerns/has_active_job_lock.rb:117:in `active_job_lock'
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR:  relation "active_job_locks" does not exist)
LINE 8:                WHERE a.attrelid = '"active_job_locks"'::regc...
                                      ^
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                 pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
                 c.collname, col_description(a.attrelid, a.attnum) AS comment
            FROM pg_attribute a
            LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
            LEFT JOIN pg_type t ON a.atttypid = t.oid
            LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation
           WHERE a.attrelid = '"active_job_locks"'::regclass
             AND a.attnum > 0 AND NOT a.attisdropped
           ORDER BY a.attnum
irb(main):004:0>
# ... Dropping current database zammad
e[31mbundler: failed to load command: rake (/opt/zammad/vendor/bundle/ruby/2.4.0/bin/rake)e[0m
LoadError: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory - /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/x86_64-linux/digest/sha1.so
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/source/git.rb:4:in `<top (required)>'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/lockfile_parser.rb:122:in `<class:LockfileParser>'
  /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/lockfile_parser.rb:14:in `<module:Bundler>'
  /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/lockfile_parser.rb:13:in `<top (required)>'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/definition.rb:2:in `<top (required)>'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler.rb:129:in `definition'
  /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler.rb:95:in `setup'
  /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/setup.rb:9:in `<top (required)>'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/zammad/vendor/ruby-2.4.4/lib/ruby/2.4.0/digest.rb:16:in `const_missing': library not found for class Digest::MD5 -- digest/md5 (LoadError)
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/source/rubygems/remote.rb:26:in `cache_slug'
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/fetcher/compact_index.rb:107:in `cache_path'
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/fetcher/compact_index.rb:83:in `compact_index_client'
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/cli.rb:636:in `warn_on_outdated_bundler'
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/cli.rb:15:in `ensure in start'
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/cli.rb:16:in `start'
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/exe/bundle:30:in `block in <top (required)>'
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/lib/bundler/friendly_errors.rb:121:in `with_friendly_errors'
	from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/bundler-1.15.2/exe/bundle:22:in `<top (required)>'
	from /opt/zammad/bin/bundle:3:in `load'
	from /opt/zammad/bin/bundle:3:in `<main>'
# ... Creating database zammad for owner zammad
ERROR:  database "zammad" already exists
# Restoring PostgrSQL DB

This still seems to be your issue. This is also the reason why the restore script fails.
I gave that hint earlier, it’s affected by this issue: https://github.com/zammad/zammad/issues/3160

Technically the issue is that we restore Zammads filesystem before restoring the database. As we’re using Zammad commands for that and your older Zammad version used another Ruby version which isn’t provided by the new package (would cause overhead). This is the reason we can’t run the Zammad commands because of “missing environment” stuff.

To solve your issue, ensure that the file restoration runs after the database restoration.
If you keep this in mind and follow the restoration documentation, you should be fine.

This block is at fault:

Move lines 156 - 159 to below line 187 and try your restoration on a fresh Zammad installation again.

Hello

well I’m a step forward but still got errors here. again in short what I did so far

	everything done as user root
apt-get update
apt-get upgrade
mount /dev/sdb1 /mnt/backup
cd /opt/zammad/contrib/backup
vi functions
zammad_restore.sh 20200806115104
	Restore Successful
	Webpage Shows  "502 Bad gateway"
systemctl restart nginx
apt-get update
apt install zammad
	0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded
apt-get upgrade
	0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded
dpkg -r --force-depends zammad
apt install zammad
	errors invoke command

But now new errors, I can see in the zammad logfile the following:

I, [2020-09-01T09:04:36.158763 #408469-47305883866100]  INFO -- : execute Stats.generate (try_count 7)...
E, [2020-09-01T09:04:36.181665 #408469-47305883866100] ERROR -- : execute Stats.generate (try_count 7) exited with error #<RuntimeError: No settings with area 'Dashboard::Stats' defined> in: 0.028654739 seconds.
I, [2020-09-01T09:04:37.892646 #408469-47305883878720]  INFO -- : execute Channel.fetch (try_count 0)...
I, [2020-09-01T09:04:37.893787 #408469-47305883878720]  INFO -- : ended Channel.fetch took: 0.007911303 seconds.
I, [2020-09-01T09:04:37.904511 #408469-47305875585560]  INFO -- : execute HttpLog.cleanup (try_count 0)...
I, [2020-09-01T09:04:41.409226 #408469-47305875585560]  INFO -- : ended HttpLog.cleanup took: 3.512542692 seconds.
I, [2020-09-01T09:04:43.442213 #408469-47305882452360]  INFO -- : execute Ticket.process_pending (try_count 9)...
E, [2020-09-01T09:04:43.505156 #408469-47305882452360] ERROR -- : execute Ticket.process_pending (try_count 9) exited with error #<ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "active_job_locks" does not exist
LINE 8:                WHERE a.attrelid = '"active_job_locks"'::regc...
                                          ^
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
                     c.collname, col_description(a.attrelid, a.attnum) AS comment
                FROM pg_attribute a
                LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
                LEFT JOIN pg_type t ON a.atttypid = t.oid
                LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation
               WHERE a.attrelid = '"active_job_locks"'::regclass
                 AND a.attnum > 0 AND NOT a.attisdropped
               ORDER BY a.attnum
> in: 0.076769282 seconds.
E, [2020-09-01T09:04:43.513175 #408469-47305882452360] ERROR -- : Failed to run Ticket.process_pending after 10 tries #<ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "active_job_locks" does not exist
LINE 8:                WHERE a.attrelid = '"active_job_locks"'::regc...
                                          ^
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
                     c.collname, col_description(a.attrelid, a.attnum) AS comment
                FROM pg_attribute a
                LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
                LEFT JOIN pg_type t ON a.atttypid = t.oid
                LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation
               WHERE a.attrelid = '"active_job_locks"'::regclass
                 AND a.attnum > 0 AND NOT a.attisdropped
               ORDER BY a.attnum
>
I, [2020-09-01T09:04:44.193975 #408469-47305883866100]  INFO -- : execute Stats.generate (try_count 8)...
E, [2020-09-01T09:04:44.215135 #408469-47305883866100] ERROR -- : execute Stats.generate (try_count 8) exited with error #<RuntimeError: No settings with area 'Dashboard::Stats' defined> in: 0.029594188 seconds.
I, [2020-09-01T09:04:47.921656 #408469-47305884155840]  INFO -- : execute OnlineNotification.cleanup (try_count 0)...
I, [2020-09-01T09:04:53.230514 #408469-47305883866100]  INFO -- : execute Stats.generate (try_count 9)...
E, [2020-09-01T09:04:53.244443 #408469-47305883866100] ERROR -- : execute Stats.generate (try_count 9) exited with error #<RuntimeError: No settings with area 'Dashboard::Stats' defined> in: 0.024412159 seconds.

I have another Log with throwed errors after the recovery

Logfile from Console

That’s actually the part that was helping quite a lot. Thanks for providing this!

It seems like migrations crash on this migration:

However, that doesn’t make any sense at all which makes me believe that this just looks like it’s the reason. The table Zammad is looking for is created by a migration that comes far later:


I thus suggest the following:

  • stop Zammad systemctl stop zammad
  • manually run the migrations `zammad run rake db:migrate
    • fails? okay! try it again
      • migrations can be sketchy but should complete if they can be fullfilled. If it stops on another part, try again until you eventually have success :frowning: If you’re stuck on the same migration see below point
    • still fails? Please provide the complete output on here
  • if your migrations run through without trouble, start Zammad systemctl start zammad
    • monitor your logfile and instance in overall - your mentioned errors should disappear

Hello MrGeneration

I tried that and done the migrate command several times
after the apt install command i get the following output and I put several times the command “zammad run rake db:migrate”
And here is the output:

Console output

Production.log

I hope you can advise

Cheers
Steffen

Sorry for the delay.
Thanks to the help from developers I now at least know the reason for this issue.

The issue appears on too big version jumps due to code changes that are applied before upgrading the database. The issue is that if you then try to migrate the database, it will still fail, because the code expects things that aren’t available at that moment.

So, in this case, the only way is to upgrade in smaller steps.
I fiddled things out for you already:

From Zammad 2.5.1 or 2.6

  • upgrade to Zammad 3.0
  • upgrade to Zammad 3.4 (stable)

If you don’t want to run the upgrade on your old host any longer, you can install a initial older version of Zammad. To do so, you can change the source file /etc/apt/sources.list.d/zammad.conf.

What you want to replace is stable againste.g.stable-3.0. Install from that and before upgrading remove -3.0` to install to latest stable. Being sticky on old versions is not recommended, of course. :slight_smile:

This may not work for all systems, because newer systems were added later on and are version dependent. To see if your system is supported by the version (or at least available) you can check that on the packager page:

Use the dropdown to select a specific version
image


I checked it for Ubuntu 18 already, it’s supported by Zammad 3.0 so you can start from a 3.0 from scratch there. This should solve your issue (hopefully)