UPgrade to 4.0 broke Elasticsearch

Technical assistance
1
  • Used Zammad version: 4.0
  • Used Zammad installation source: (source, package, …) Debian
  • Operating system: Ubuntu server 16.04
  • Browser + version:

Expected behavior:

not able to search
*

Actual behavior:

error when using search
*

normally if i see this error, i reindex “zammad run rail searchindex:rebuild”

but i got an error after upgrading zammad to 4.0 that 5.6 is not supported,

so i upgarded to 7, now it wont start in the elastic logs i see

java.lang.IllegalStateException: The index [supportdesk_zammad_production_user/WAdCKmlfSZmwXsZATDnWRg] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.12.0.
at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkSupportedVersion(IndexMetadataVerifier.java:85) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.verifyIndexMetadata(IndexMetadataVerifier.java:72) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:235) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.node.Node.start(Node.java:835) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:311) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:406) [elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.12.0.jar:7.12.0]
at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.12.0.jar:7.12.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.12.0.jar:7.12.0]
[2021-03-27T06:11:50,865][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [supportdesk] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: The index [supportdesk_zammad_production_user/WAdCKmlfSZmwXsZATDnWRg] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.12.0.
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.12.0.jar:7.12.0]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.12.0.jar:7.12.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.12.0.jar:7.12.0]
Caused by: java.lang.IllegalStateException: The index [supportdesk_zammad_production_user/WAdCKmlfSZmwXsZATDnWRg] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.12.0.
at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkSupportedVersion(IndexMetadataVerifier.java:85) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.verifyIndexMetadata(IndexMetadataVerifier.java:72) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:235) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.node.Node.start(Node.java:835) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:311) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:406) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.12.0.jar:7.12.0]
… 6 more
[2021-03-27T06:11:50,868][INFO ][o.e.n.Node ] [supportdesk] stopping …
[2021-03-27T06:11:50,879][INFO ][o.e.n.Node ] [supportdesk] stopped
[2021-03-27T06:11:50,879][INFO ][o.e.n.Node ] [supportdesk] closing …
[2021-03-27T06:11:50,891][INFO ][o.e.n.Node ] [supportdesk] closed
[2021-03-27T06:11:50,893][INFO ][o.e.x.m.p.NativeController] [supportdesk] Native controller process has stopped - no new native processes can be started

1 Like
2

Sorry oS 18.04 ubuntu server

3

Now reading the logs message i found the system went from 5.6 ES to 7.X to correctly upgrade to ES 7 from 5.6, then first 6.X should have been installed then index rebuild, so have uninstalled elasticsearch 7 reloaded 5.6 and upgrade that to 6.5 and now the index is rebuild nicely,

3 Likes
4

how do you uninstall elasticsearch 7 without removing zammad?

5

Hi Rifter70,

you need not worry, if you have a back up just do apt-get remove elasticsearch and let it do its thing,
this will uninstall zammad,

then change the elasticsearch repo to 6.x (i guess is what your trying to do),

then apt-get udate && apt-get upgrade -y and let it reinstall zammad and elastic search,

once its done zammad will be back to how it was you wont loose anything,
i also increase

xms and xmx to 4 gig in the jvm.options before index rebuild and restart elasticsearch

then do the rebuild of the elasticsearch index

zammad run rake searchindex:rebuild

we have about 350 gig store and it took about 18 hours

and after that Zammad running smooth and normal

hope that helps

Thanks

1 Like
6

Or you just break dependencies during removal.
A hint is living in the restoration part, as upgrading ES is out of our scope.
https://docs.zammad.org/en/latest/appendix/backup-and-restore.html#when-migrated-from-a-self-hosted-zammad-system

1 Like
7

Hey @KINGRULZ
just wondering if you upgraded to ES 7 again? I just moved Zammad installation from one server to another and upgraded to ES7 which works great. Zammads install docu for ES also says v7 should be ok.
Thanks
https://docs.zammad.org/en/latest/install/elasticsearch.html

8

Upgrading Elasticsearch is out of our scope as mentioned on the docs ( Updating Zammad — Zammad System Documentation documentation ). Their documentation will hopefully help you: Upgrade Elasticsearch | Elasticsearch Guide [8.11] | Elastic

Note however that their upgrade path is a pain. You may be faster and more comfortable with removing the indice storage and start over. That is if Zammad is the only application relying on that ES instance.

9

Hey @MrGeneration,

same issue here. Upgraded Zammad to 4.0 and ES to 7…

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: The index [zammad_production_ticket/gNkWkmDcTNqCU9_i2-XMuw] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.12.0.

Any tips how to remove the index and to start over? Sorry, but i´m not so familiar with ES.

Thanks.

10

Resolved the issue by myself. Uninstalled ES 7 and Zammad. Changed the Repo to ES 6 and reinstalled both. Seems to be working.

11

So no upgrade scripts provided by Zammad after a broken depency which seems not mentioned in changelog?

Zammad upgrading through a regular yum system maintenance on centos7 … no warning or anything about compatibility issues.

So, any way to upgrade the search indexes without uninstalling 7, installing 6, upgrading, uninstalling 6 and installing 7 again?

1 Like
12

So the following solves the issue for me:
/usr/share/elasticsearch/bin/elasticsearch-plugin remove ingest-attachment
yum downgrade elasticsearch-6.8.15-1
/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment

13

Sorry I’m really struggling to stay friendly at this point.
Why do you think we are responsible to updating your third party dependencies in the suggested order?

Our documentation ( Updating Zammad — Zammad documentation ) clearly provides the correct documentation link for elasticsearch which states the correct upgrade path:
Upgrade Elasticsearch | Elasticsearch Guide [7.13] | Elastic

If you don’t follow said path and simply upgrade from 5 to 7 you’re lost. Naturally.
You can find the list of supported ES versions here:
https://docs.zammad.org/en/latest/prerequisites/software.html#elasticsearch-optional

Our release notes also hinted to this change and suggested to upgrade.
We can’t and won’t decide for administrators when it’s time to switch to a newer Elasticsearch version (apart from dependencies on our end). Why you may ask? Because you may use the Elasticsearch for other situations too or even use a cluster out side of our scope.

Elasticsearch is a third party dependency just like MariaDB, MySQL or postgreSQL - we can give tips and hints, but we won’t tell you how to do it. I mean, do you expect a “how to install my vps” in our documentation next?

2 Likes
14

dears, I try many times the issue is when I update zammad to the latest, elasticsearch stops working, but when didn’t do the update and do the normal installation steps over the last version of zammad, it will be updated to the latest and elasticsearch will work fine.

15

@Ahmed-Abbas88 please create your own topic to keep overviews of issues clean.
Make sure to provide more information than you did here because other wise it will be impossible to help you.

16

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.