Zammad Search "elastichsearch" status is failed

alexander.bippus · June 28, 2019, 10:10am

Used Zammad version: 2.9.x
Used Zammad installation source: (source, package, …)
Operating system: Ubuntu 18.04
Browser + version: all Browser FireFox / Chrome / IE /

Expected behavior:

Service “elasticsearch” should be running for Zammad Search

Actual behavior:

Service “elasticsearch” status on “failed”

Steps to reproduce the behavior:

Starting Service “elasticsearch”

● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2019-06-26 09:58:12 CEST; 2 days ago
Docs: http://www.elastic.co
Process: 5102 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=e
Process: 5101 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 5102 (code=exited, status=1/FAILURE)

Jun 26 09:58:07 srvzammad01 systemd[1]: Starting Elasticsearch…
Jun 26 09:58:07 srvzammad01 systemd[1]: Started Elasticsearch.
Jun 26 09:58:12 srvzammad01 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jun 26 09:58:12 srvzammad01 systemd[1]: elasticsearch.service: Failed with result ‘exit-code’.
lines 1-12/12 (END)

Hello Zammad Community,

i hope you can help me in this Issue. It is my first Post, so i don´t know exactly how to communicate my issue with the community.

Since several Days, our Zammad Search is not running and doesn´t start anymore.
It is not the Problem with “unsufficent memory”

Could someone help me?

Nice Regards Alex

martin.von.wittich · June 28, 2019, 10:37am

The error message unfortunately doesn’t say why ElasticSearch didn’t start. Check your logs (maybe /var/log/syslog?), there should be a proper error message somewhere.

If I were to hazard a guess, I’d suspect that vm.max_map_count is too low. Does this help?

sysctl -w vm.max_map_count=262144
systemctl start elasticsearch

If so, then make sure to put that setting e.g. into /etc/sysctl.conf so that it’s not getting lost on reboots.

alexander.bippus · June 28, 2019, 10:45am

Hello Martin.von.wittich!

Thank you for your fast response! I tried your syntax, but i didn´t help.

I checked the path
/var/log/syslog but there is no entry.

I found several logs in
/var/log/zammad

-rw-r–r-- 1 zammad zammad 482709087 Jun 28 12:42 production.log
-rw-r–r-- 1 zammad zammad 740291073 Jun 28 00:06 production.log.1
-rw-r–r-- 1 zammad zammad 662210 Jun 19 00:07 production.log.10.gz
-rw-r–r-- 1 zammad zammad 599290 Jun 18 00:06 production.log.11.gz
-rw-r–r-- 1 zammad zammad 368502 Jun 17 00:06 production.log.12.gz
-rw-r–r-- 1 zammad zammad 368152 Jun 16 00:05 production.log.13.gz
-rw-r–r-- 1 zammad zammad 631934 Jun 15 00:06 production.log.14.gz
-rw-r–r-- 1 zammad zammad 88517092 Jun 27 00:10 production.log.2.gz
-rw-r–r-- 1 zammad zammad 1233126 Jun 26 00:10 production.log.3.gz
-rw-r–r-- 1 zammad zammad 970072 Jun 25 00:07 production.log.4.gz
-rw-r–r-- 1 zammad zammad 368199 Jun 24 00:10 production.log.5.gz
-rw-r–r-- 1 zammad zammad 367228 Jun 23 00:06 production.log.6.gz
-rw-r–r-- 1 zammad zammad 632561 Jun 22 00:08 production.log.7.gz
-rw-r–r-- 1 zammad zammad 683049 Jun 21 00:08 production.log.8.gz
-rw-r–r-- 1 zammad zammad 747761 Jun 20 00:07 production.log.9.gz
-rw-r–r-- 1 zammad zammad 0 Jun 26 09:45 scheduler_err.log
-rw-r–r-- 1 zammad zammad 206 Mai 1 02:05 scheduler_err.log.1
-rw-r–r-- 1 zammad zammad 0 Jun 28 00:06 scheduler_out.log
-rw-r–r-- 1 zammad zammad 423 Jun 28 00:06 scheduler_out.log.1
-rw-r–r-- 1 zammad zammad 165 Mai 30 00:09 scheduler_out.log.10.gz
-rw-r–r-- 1 zammad zammad 191 Mai 29 00:08 scheduler_out.log.11.gz
-rw-r–r-- 1 zammad zammad 148 Mai 25 00:08 scheduler_out.log.12.gz
-rw-r–r-- 1 zammad zammad 171 Mai 24 00:08 scheduler_out.log.13.gz
-rw-r–r-- 1 zammad zammad 152 Mai 23 00:05 scheduler_out.log.14.gz
-rw-r–r-- 1 zammad zammad 143 Jun 27 00:10 scheduler_out.log.2.gz
-rw-r–r-- 1 zammad zammad 161 Jun 13 00:05 scheduler_out.log.3.gz
-rw-r–r-- 1 zammad zammad 159 Jun 12 00:08 scheduler_out.log.4.gz
-rw-r–r-- 1 zammad zammad 151 Jun 8 00:05 scheduler_out.log.5.gz
-rw-r–r-- 1 zammad zammad 150 Jun 7 00:07 scheduler_out.log.6.gz
-rw-r–r-- 1 zammad zammad 151 Jun 6 00:05 scheduler_out.log.7.gz
-rw-r–r-- 1 zammad zammad 148 Jun 4 00:05 scheduler_out.log.8.gz
-rw-r–r-- 1 zammad zammad 149 Jun 1 00:06 scheduler_out.log.9.gz
-rw-r–r-- 1 zammad zammad 0 Apr 5 02:05 searchindex-rebuild.log
-rw-r–r-- 1 zammad zammad 2996 Apr 5 02:05 searchindex-rebuild.log.1

and logs in
/var/log/elasticsearch

elasticsearch-2019-06-26.log
#####################################
#####################################
#####################################

[2019-06-26T09:43:24,577][INFO ][o.e.n.Node ] [V4D95vB] stopping …
[2019-06-26T09:43:24,990][INFO ][o.e.n.Node ] [V4D95vB] stopped
[2019-06-26T09:43:24,990][INFO ][o.e.n.Node ] [V4D95vB] closing …
[2019-06-26T09:43:25,017][INFO ][o.e.n.Node ] [V4D95vB] closed
[2019-06-26T09:43:50,897][INFO ][o.e.n.Node ] [] initializing …
[2019-06-26T09:43:51,116][INFO ][o.e.e.NodeEnvironment ] [V4D95vB] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [43.5gb], net total_space [58.8gb], spins? [possibly], types [ext4]
[2019-06-26T09:43:51,116][INFO ][o.e.e.NodeEnvironment ] [V4D95vB] heap size [1.9gb], compressed ordinary object pointers [true]
[2019-06-26T09:43:51,136][INFO ][o.e.n.Node ] node name [V4D95vB] derived from node ID [V4D95vBITbGwkbT3EQXS9A]; set [node.name] to override
[2019-06-26T09:43:51,137][INFO ][o.e.n.Node ] version[5.6.16], pid[13579], build[3a740d1/2019-03-13T15:33:36.565Z], OS[Linux/4.15.0-47-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_212/25.212-b03]
[2019-06-26T09:43:51,137][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2019-06-26T09:43:52,568][INFO ][o.e.p.PluginsService ] [V4D95vB] loaded module [aggs-matrix-stats]
[2019-06-26T09:43:52,568][INFO ][o.e.p.PluginsService ] [V4D95vB] loaded module [ingest-common]
[2019-06-26T09:43:52,568][INFO ][o.e.p.PluginsService ] [V4D95vB] loaded module [lang-expression]
[2019-06-26T09:43:52,568][INFO ][o.e.p.PluginsService ] [V4D95vB] loaded module [lang-groovy]
[2019-06-26T09:43:52,568][INFO ][o.e.p.PluginsService ] [V4D95vB] loaded module [lang-mustache]
[2019-06-26T09:43:52,568][INFO ][o.e.p.PluginsService ] [V4D95vB] loaded module [lang-painless]
[2019-06-26T09:43:52,569][INFO ][o.e.p.PluginsService ] [V4D95vB] loaded module [parent-join]
[2019-06-26T09:43:52,569][INFO ][o.e.p.PluginsService ] [V4D95vB] loaded module [percolator]
[2019-06-26T09:43:52,569][INFO ][o.e.p.PluginsService ] [V4D95vB] loaded module [reindex]
[2019-06-26T09:43:52,569][INFO ][o.e.p.PluginsService ] [V4D95vB] loaded module [transport-netty3]
[2019-06-26T09:43:52,569][INFO ][o.e.p.PluginsService ] [V4D95vB] loaded module [transport-netty4]
[2019-06-26T09:43:52,569][INFO ][o.e.p.PluginsService ] [V4D95vB] loaded plugin [ingest-attachment]
[2019-06-26T09:43:52,569][INFO ][o.e.p.PluginsService ] [V4D95vB] loaded plugin [mapper-attachments]
[2019-06-26T09:43:52,713][WARN ][d.m.attachment ] [mapper-attachments] plugin has been deprecated and will be replaced by [ingest-attachment] plugin.
[2019-06-26T09:43:53,971][INFO ][o.e.d.DiscoveryModule ] [V4D95vB] using discovery type [zen]
[2019-06-26T09:43:54,390][INFO ][o.e.n.Node ] initialized
[2019-06-26T09:43:54,390][INFO ][o.e.n.Node ] [V4D95vB] starting …
[2019-06-26T09:43:54,603][ERROR][o.e.b.Bootstrap ] [V4D95vB] Exception
org.elasticsearch.transport.BindTransportException: Failed to bind to [9300-9400]
at org.elasticsearch.transport.TcpTransport.bindToPort(TcpTransport.java:783) ~[elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:748) ~[elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:175) ~[?:?]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.transport.TransportService.doStart(TransportService.java:219) ~[elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.node.Node.start(Node.java:694) ~[elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:278) ~[elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:351) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.6.16.jar:5.6.16]
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
at sun.nio.ch.Net.bind(Net.java:433) ~[?:?]
at sun.nio.ch.Net.bind(Net.java:425) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[?:?]
at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:128) ~[?:?]

alexander.bippus · June 28, 2019, 11:11am

Ouh man… I found my Problem.
I was wondering why the Bindings are not Working.

I Fact of some IP changes in our Network and IP Changes of our Helpdesk. The IP is written hardcoded in

/etc/elasticsearch/elasticsearch.yml

I Changed on the current IP and startet the service

systemctl start elasticsearch

and everything went fine again!

Thank you @martin.von.wittich

system · October 26, 2019, 11:11am

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.