Problem after upgrade Elasticsearch

MartinGruben · January 12, 2022, 8:17am

Infos:

Used Zammad version: 5.x
Used Zammad installation type: (package)
Operating system: Ubuntu 18.04
Browser + version:

Expected behavior:

I get the following error if i try to start Elasticsearch after upgrade.

Elasticsearch log:

[2022-01-12T08:58:02,992][INFO ][o.e.n.Node ] [support] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-01-12T08:58:02,992][INFO ][o.e.n.Node ] [support] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-13626790051793039139, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms7831m, -Xmx7831m, -XX:MaxDirectMemorySize=4106223616, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2022-01-12T08:58:05,777][INFO ][o.e.p.PluginsService ] [support] loaded module [aggs-matrix-stats]
[2022-01-12T08:58:05,778][INFO ][o.e.p.PluginsService ] [support] loaded module [analysis-common]
[2022-01-12T08:58:05,778][INFO ][o.e.p.PluginsService ] [support] loaded module [constant-keyword]
[2022-01-12T08:58:05,778][INFO ][o.e.p.PluginsService ] [support] loaded module [frozen-indices]
[2022-01-12T08:58:05,778][INFO ][o.e.p.PluginsService ] [support] loaded module [ingest-common]
[2022-01-12T08:58:05,779][INFO ][o.e.p.PluginsService ] [support] loaded module [ingest-geoip]
[2022-01-12T08:58:05,779][INFO ][o.e.p.PluginsService ] [support] loaded module [ingest-user-agent]
[2022-01-12T08:58:05,779][INFO ][o.e.p.PluginsService ] [support] loaded module [kibana]
[2022-01-12T08:58:05,779][INFO ][o.e.p.PluginsService ] [support] loaded module [lang-expression]
[2022-01-12T08:58:05,780][INFO ][o.e.p.PluginsService ] [support] loaded module [lang-mustache]
[2022-01-12T08:58:05,780][INFO ][o.e.p.PluginsService ] [support] loaded module [lang-painless]
[2022-01-12T08:58:05,780][INFO ][o.e.p.PluginsService ] [support] loaded module [legacy-geo]
[2022-01-12T08:58:05,781][INFO ][o.e.p.PluginsService ] [support] loaded module [mapper-extras]
[2022-01-12T08:58:05,781][INFO ][o.e.p.PluginsService ] [support] loaded module [mapper-version]
[2022-01-12T08:58:05,781][INFO ][o.e.p.PluginsService ] [support] loaded module [parent-join]
[2022-01-12T08:58:05,781][INFO ][o.e.p.PluginsService ] [support] loaded module [percolator]
[2022-01-12T08:58:05,781][INFO ][o.e.p.PluginsService ] [support] loaded module [rank-eval]
[2022-01-12T08:58:05,782][INFO ][o.e.p.PluginsService ] [support] loaded module [reindex]
[2022-01-12T08:58:05,782][INFO ][o.e.p.PluginsService ] [support] loaded module [repositories-metering-api]
[2022-01-12T08:58:05,782][INFO ][o.e.p.PluginsService ] [support] loaded module [repository-encrypted]
[2022-01-12T08:58:05,782][INFO ][o.e.p.PluginsService ] [support] loaded module [repository-url]
[2022-01-12T08:58:05,782][INFO ][o.e.p.PluginsService ] [support] loaded module [runtime-fields-common]
[2022-01-12T08:58:05,783][INFO ][o.e.p.PluginsService ] [support] loaded module [search-business-rules]
[2022-01-12T08:58:05,783][INFO ][o.e.p.PluginsService ] [support] loaded module [searchable-snapshots]
[2022-01-12T08:58:05,783][INFO ][o.e.p.PluginsService ] [support] loaded module [snapshot-repo-test-kit]
[2022-01-12T08:58:05,783][INFO ][o.e.p.PluginsService ] [support] loaded module [spatial]
[2022-01-12T08:58:05,784][INFO ][o.e.p.PluginsService ] [support] loaded module [systemd]
[2022-01-12T08:58:05,784][INFO ][o.e.p.PluginsService ] [support] loaded module [transform]
[2022-01-12T08:58:05,784][INFO ][o.e.p.PluginsService ] [support] loaded module [transport-netty4]
[2022-01-12T08:58:05,785][INFO ][o.e.p.PluginsService ] [support] loaded module [unsigned-long]
[2022-01-12T08:58:05,785][INFO ][o.e.p.PluginsService ] [support] loaded module [vector-tile]
[2022-01-12T08:58:05,786][INFO ][o.e.p.PluginsService ] [support] loaded module [vectors]
[2022-01-12T08:58:05,786][INFO ][o.e.p.PluginsService ] [support] loaded module [wildcard]
[2022-01-12T08:58:05,786][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-aggregate-metric]
[2022-01-12T08:58:05,787][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-analytics]
[2022-01-12T08:58:05,787][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-async]
[2022-01-12T08:58:05,787][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-async-search]
[2022-01-12T08:58:05,788][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-autoscaling]
[2022-01-12T08:58:05,788][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-ccr]
[2022-01-12T08:58:05,788][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-core]
[2022-01-12T08:58:05,788][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-data-streams]
[2022-01-12T08:58:05,789][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-deprecation]
[2022-01-12T08:58:05,789][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-enrich]
[2022-01-12T08:58:05,789][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-eql]
[2022-01-12T08:58:05,789][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-fleet]
[2022-01-12T08:58:05,789][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-graph]
[2022-01-12T08:58:05,790][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-identity-provider]
[2022-01-12T08:58:05,790][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-ilm]
[2022-01-12T08:58:05,790][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-logstash]
[2022-01-12T08:58:05,790][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-ml]
[2022-01-12T08:58:05,790][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-monitoring]
[2022-01-12T08:58:05,791][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-ql]
[2022-01-12T08:58:05,791][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-rollup]
[2022-01-12T08:58:05,791][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-security]
[2022-01-12T08:58:05,792][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-shutdown]
[2022-01-12T08:58:05,792][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-sql]
[2022-01-12T08:58:05,792][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-stack]
[2022-01-12T08:58:05,792][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-text-structure]
[2022-01-12T08:58:05,793][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-voting-only-node]
[2022-01-12T08:58:05,793][INFO ][o.e.p.PluginsService ] [support] loaded module [x-pack-watcher]
[2022-01-12T08:58:05,793][INFO ][o.e.p.PluginsService ] [support] loaded plugin [ingest-attachment]
[2022-01-12T08:58:05,833][INFO ][o.e.e.NodeEnvironment ] [support] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [118.1gb], net total_space [150.1gb], types [ext4]
[2022-01-12T08:58:05,835][INFO ][o.e.e.NodeEnvironment ] [support] heap size [7.6gb], compressed ordinary object pointers [true]
[2022-01-12T08:58:05,884][INFO ][o.e.n.Node ] [support] node name [support], node ID [URIpYCQERq6d9T7sWphzpg], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2022-01-12T08:58:11,894][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [support] [controller/3268] [Main.cc@122] controller (64 bit): Version 7.16.2 (Build 77e5cf03d1077d) Copyright (c) 2021 Elasticsearch BV
[2022-01-12T08:58:12,463][INFO ][o.e.x.s.a.s.FileRolesStore] [support] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2022-01-12T08:58:13,810][INFO ][o.e.i.g.ConfigDatabases ] [support] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/etc/elasticsearch/ingest-geoip] for changes
[2022-01-12T08:58:13,812][INFO ][o.e.i.g.DatabaseNodeService] [support] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-13626790051793039139/geoip-databases/URIpYCQERq6d9T7sWphzpg]
[2022-01-12T08:58:14,650][INFO ][o.e.t.NettyAllocator ] [support] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2022-01-12T08:58:14,756][INFO ][o.e.d.DiscoveryModule ] [support] using discovery type [zen] and seed hosts providers [settings]
[2022-01-12T08:58:15,412][INFO ][o.e.g.DanglingIndicesState] [support] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-01-12T08:58:16,159][INFO ][o.e.n.Node ] [support] initialized
[2022-01-12T08:58:16,160][INFO ][o.e.n.Node ] [support] starting …
[2022-01-12T08:58:16,176][INFO ][o.e.x.s.c.f.PersistentCache] [support] persistent cache index loaded
[2022-01-12T08:58:16,177][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [support] deprecation component started
[2022-01-12T08:58:16,294][INFO ][o.e.t.TransportService ] [support] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2022-01-12T08:58:16,350][ERROR][o.e.b.Bootstrap ] [support] Exception
java.lang.IllegalStateException: The index [zammad_production_user/XWWc8Fr8T72Dc4rqVm2tnw] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.16.2.
at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkSupportedVersion(IndexMetadataVerifier.java:98) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.verifyIndexMetadata(IndexMetadataVerifier.java:80) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:258) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:244) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:165) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.node.Node.start(Node.java:1185) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:335) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:443) [elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) [elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) [elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) [elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) [elasticsearch-cli-7.16.2.jar:7.16.2]
at org.elasticsearch.cli.Command.main(Command.java:77) [elasticsearch-cli-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) [elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-7.16.2.jar:7.16.2]
[2022-01-12T08:58:16,356][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [support] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: The index [zammad_production_user/XWWc8Fr8T72Dc4rqVm2tnw] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.16.2.
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.16.2.jar:7.16.2]
at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.16.2.jar:7.16.2]
Caused by: java.lang.IllegalStateException: The index [zammad_production_user/XWWc8Fr8T72Dc4rqVm2tnw] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.16.2.
at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkSupportedVersion(IndexMetadataVerifier.java:98) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.verifyIndexMetadata(IndexMetadataVerifier.java:80) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:258) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:244) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:165) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.node.Node.start(Node.java:1185) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:335) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:443) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.2.jar:7.16.2]
… 6 more
[2022-01-12T08:58:16,371][INFO ][o.e.n.Node ] [support] stopping …
[2022-01-12T08:58:16,385][INFO ][o.e.n.Node ] [support] stopped
[2022-01-12T08:58:16,386][INFO ][o.e.n.Node ] [support] closing …
[2022-01-12T08:58:16,410][INFO ][o.e.n.Node ] [support] closed
[2022-01-12T08:58:16,413][INFO ][o.e.x.m.p.NativeController] [support] Native controller process has stopped - no new native processes can be started

Elasticsearch told that i have to reindex, but it don’t start so i can’t reindex.

ingest-attachment removed and installed new.

Hope anyone can help.

Zammad is running but we can not search or change the customer or anything else.

Thanks for help.

Martin

richie_77 · January 12, 2022, 9:24am

I had the same problem. The simplest solution is to switch to the [REDACTED]-user, and then drop all zammad indexes, start the reindex process and then create a new index.

MrGeneration · January 12, 2022, 10:50am

I am sorry @richie_77 I had to edit your post and redact out the user you were referencing.
It’s not the correct user and also puts other users in danger accidentely doing stuff on their database instead of Elasticsearch.

I did this to protect Zammad users. Feel free to edit back in the correct user you were talking about.

In the mean time OP can read and follow the steps for Elasticsearch upgrades:

richie_77 · January 12, 2022, 11:13am

@MrGeneration i’m sorry, that was not my intention.

I thought, i had done this with that user, because it was the one, which had password-less access to the API-Endpoints from Elasticsearch. I was standing at the same point, where @MartinGruben stands and which is a point, where you imho only can import a backup to follow the ES upgrade guide. In german we say “Das Kind ist da bereits in den Brunnen gefallen”.

Is it ok, if i put my “guide” to dropping the indexes and updating ES in a hidden container with an explicit warning? (First i have to check, if i can reproduce my steps…)

MrGeneration · January 12, 2022, 11:15am

Absolutely fine for me within a spoiler combined with disclaimer / warnings so that last copy cat anon get’s the message.

richie_77 · January 12, 2022, 11:39am

Ok, this is my way to drop the indexes, update es and then run the zammad update:

Warning! Using this guide to make changes on your zammad instance with an database user can do changes on your database, which may lead to a malfunctioned system. Use it at your own risk!

Spoiler

First stop your zammad instance using systemctl zammad stop
Not necessary [Next step login to the database user with su postgres from an root account]

list all your indexes with
curl -XGET http://localhost:9200/_aliases?pretty

If you are sure, that there are only zammad indexes, you can drop them with
curl -XDELETE http://localhost:9200/zammad_production*

With the first curl command, you can check, if the indexes where dropped

not necessary [Now log out from your database user user],
download the latest es package (For me it was
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.8.1-amd64.deb), and install it with dpkg -i elasticsearch-7.8.1-amd64.deb.

Download and install the actual ingest-attachment plugin:

/usr/share/elasticsearch/bin/elasticsearch-plugin remove ingest-attachment
/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment

and restart your es instance with systemctl restart elasticsearch.service.

If es has started, you can run zammad run rake searchindex:rebuild to rebuild the indexes with the new es version. After that, you can upgrade your zammad instance to the newest version.

Run a zammad run rake db:migrate for safety, to go clear, that the database is up to date.

I hope, this helps you @MartinGruben. Unfortunatly, you have to do this on a zammad instance below 5.x, so you have to import a backup from your system or revert a vm snapshot to a state, where zammad is below 5.x.

MrGeneration · January 12, 2022, 11:43am

The user switch is not required on default ES installations for what you’re doing there.

richie_77 · January 12, 2022, 11:48am

I have edited my post. After a long way of trying with the es api endpoints, i had done this step. You are right, I tested it in my actual instance - it can be done without switching the user.

system · May 12, 2022, 11:48am

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.