Elasticsearch often crashes and cannot be easily restarted

[anhofmann]

Infos:

• Used Zammad version: 2.8.0
• Used Zammad installation source: package
• Operating system: 16.04.5
• Browser + version: 60.4.0esr (64-bit)
• Database + version: Postgresql 9.5+173ubuntu0.2
• Elasticsearch version: 5.6.13

Expected behavior:

• Elasticsearch should run all the time or Zammad shouldn’t be able to crash elasticsearch.

Actual behavior:

Elasticsearch sometimes crashes daily, sometimes it runs a few days without problems. If the elasticsearch daemon crashes, it cannot be started, because it crashes again immediately. If I quit Zammad, I can also start elasticsearch.
The only working procedure to get elasticsearch back to work is as follows:

Execute these commands in a shell:

• “systemctl stop zammad”
• “systemctl start elasticsearch”
• “zammad run rails c”
• Then execute the script found in this post: Failing background jobs.;Failed to run background job #1 'BackgroundJobSearchIndex' 10 time(s)
• wait until the messages on the screen stop, then press “CTRL + D”
• “systemctl start zammad”

Steps to reproduce the behavior:

• I have no idea why and how elasticsearch crashes. I fought my way through the log files of Zammad and elasticsearch and found nothing I could do with.

[MrGeneration]

This doesn’t help to help you at any way (no offense).

Check e.g. dmesg or specially the elasticsearch-logfiles, as this will proberbly hold the root of your problem.

What are the system specs?

[anhofmann]

Doh! dmesg says a lot of times: " Out of memory in UB 146199: OOM killed process 27561 (java)" I think this clearly explains what’s wrong with my server. That doesn’t explain why Elasticsearch crashes immediately when starting, but I fix the OOM Killer problem first and then I report again.

The server has 2 cores and 4 GB RAM. Only Zammad and Elasticsearch run on this server. I guess that’s just not enough, can it be?

[MrGeneration]

According to our documentation you can run Zammad on 2 cores with 4GB of RAM. I did that on my early days as well (until I had too many agents haha)

You can limit the memory usuage of elasticsearch, check out the following article:
https://www.elastic.co/guide/en/elasticsearch/reference/master/heap-size.html#heap-size

2G max should be fine.

Also make sure that your postgresql is not eating too much RAM, how much RAM usuage do you have without elasticsearch? I mean it doesn’t help you to limit elasticsearch if something elses is hungry