Elasticsearch StatusCode: 500 on reporting tab

robinedwards · July 5, 2023, 2:25pm

Infos:

Used Zammad version: Zammad version 6.0.0-1687441991.d5fd26cf.jammy
Used Zammad installation type: (source, package, docker-compose, …) Package
Operating system: Ubuntu Jammy 22.04
Browser + version: firefox 114.01 (64 bit)

Expected behavior:

When I click on the ‘reporting’ (bar) icon at the bottom I get this elastic search error:

Unable to process GET request to elasticsearch URL 'http://localhost:9200/zammad_production_ticket/_search'. Check the response and payload for detailed information:

Response:
#<UserAgent::Result:0x00007f625403fc88 @success=false, @body=nil, @data=nil, @code=0, @content_type=nil, @error="#<EOFError: end of file reached>", @header=nil>

Payload:
{"size":0,"query":{"bool":{"must":[{"range":{"created_at":{"from":"2022-12-31T23:00:00Z","to":"2023-12-31T22:59:59Z"}}},{"bool":{"must":[{"bool":{"must_not":[{"term":{"state.name.keyword":"merged"}}]}}]}}]}},"aggs":{"time_buckets":{"date_histogram":{"field":"created_at","calendar_interval":"month","time_zone":"Europe/Brussels"}}},"sort":[{"updated_at":{"order":"desc"}},"_score"]}

Payload size: 0M 

Elastic search version is: Version: 8.8.1, Build: deb/f8edfccba429b6477927a7c1ce1bc6729521305e/2023-06-05T21:32:25.188464208Z, JVM: 20.0.1

fliebe92 · July 6, 2023, 9:52am

Hi @robinedwards, did you check that you have enough disk space for Elasticsearch?

robinedwards · July 7, 2023, 2:02pm

Hi @fliebe92
Thanks for your reply.
I see from df -h that I have 42gb free space.
However when I run the following command to get allocation size in ES, I get an empty reply.

admin@zammad:~$ curl -XGET 'http://localhost:9200/_cat/allocation'
curl: (52) Empty reply from server

I’m not sure if there is some problem with the installation.
Service status below shows that its running:

admin@zammad:~$ sudo service elasticsearch status
● elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor>
     Active: active (running) since Tue 2023-07-04 16:46:40 UTC; 2 days ago
       Docs: https://www.elastic.co
   Main PID: 205 (java)
      Tasks: 74 (limit: 464103)
     Memory: 2.7G
     CGroup: /system.slice/elasticsearch.service
             ├─205 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+Us>
             ├─527 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.ca>
             └─570 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x8>

Jul 04 16:46:09 zammad systemd[1]: Starting Elasticsearch...
Jul 04 16:46:40 zammad systemd[1]: Started Elasticsearch.
lines 1-14/14 (END)

Any ideas what can the issue be?
Thanks

fliebe92 · July 7, 2023, 6:47pm

What happens if you simply try to execute zammad run rake zammad:searchindex:rebuild? Please note that this can use a lot of time + resources.

robinedwards · July 7, 2023, 7:21pm

if I run the rake command to build the index, I basically get the same error as in the stats/reporting page. Here is the output:

admin@zammad:~$ sudo zammad run rake zammad:searchindex:rebuild
rake aborted!
Unable to process GET request to elasticsearch URL 'http://localhost:9200'. Check the response and payload for detailed information:

Response:
#<UserAgent::Result:0x00007f2a1fe1e3f8 @success=false, @body=nil, @data=nil, @code=0, @content_type=nil, @error="#<EOFError: end of file reached>", @header=nil>

Payload:
null

Payload size: 0M
/opt/zammad/lib/search_index_backend.rb:36:in `info'
/opt/zammad/lib/tasks/zammad/search_index_es.rake:64:in `block (3 levels) in <main>'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/zammad/bin/bundle:121:in `load'
/opt/zammad/bin/bundle:121:in `<main>'
Tasks: TOP => zammad:searchindex:rebuild => zammad:searchindex:version_supported
(See full trace by running task with --trace)

and with --trace flag:

admin@zammad:~$ sudo zammad run rake zammad:searchindex:rebuild --trace
** Invoke zammad:searchindex:rebuild (first_time)
** Invoke zammad:searchindex:version_supported (first_time)
** Invoke zammad:searchindex:configured (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute zammad:searchindex:configured
** Execute zammad:searchindex:version_supported
rake aborted!
Unable to process GET request to elasticsearch URL 'http://localhost:9200'. Check the response and payload for detailed information:

Response:
#<UserAgent::Result:0x00007ff52241a5b8 @success=false, @body=nil, @data=nil, @code=0, @content_type=nil, @error="#<EOFError: end of file reached>", @header=nil>

Payload:
null

Payload size: 0M
/opt/zammad/lib/search_index_backend.rb:36:in `info'
/opt/zammad/lib/tasks/zammad/search_index_es.rake:64:in `block (3 levels) in <main>'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:243:in `block in invoke_prerequisites'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `each'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `invoke_prerequisites'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/zammad/vendor/bundle/ruby/3.1.0/bin/rake:25:in `load'
/opt/zammad/vendor/bundle/ruby/3.1.0/bin/rake:25:in `<top (required)>'
/opt/zammad/vendor/ruby-3.1.3/lib/ruby/3.1.0/bundler/cli/exec.rb:58:in `load'
/opt/zammad/vendor/ruby-3.1.3/lib/ruby/3.1.0/bundler/cli/exec.rb:58:in `kernel_load'
/opt/zammad/vendor/ruby-3.1.3/lib/ruby/3.1.0/bundler/cli/exec.rb:23:in `run'
/opt/zammad/vendor/ruby-3.1.3/lib/ruby/3.1.0/bundler/cli.rb:486:in `exec'
/opt/zammad/vendor/ruby-3.1.3/lib/ruby/3.1.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/zammad/vendor/ruby-3.1.3/lib/ruby/3.1.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/zammad/vendor/ruby-3.1.3/lib/ruby/3.1.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/opt/zammad/vendor/ruby-3.1.3/lib/ruby/3.1.0/bundler/cli.rb:31:in `dispatch'
/opt/zammad/vendor/ruby-3.1.3/lib/ruby/3.1.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/opt/zammad/vendor/ruby-3.1.3/lib/ruby/3.1.0/bundler/cli.rb:25:in `start'
/opt/zammad/vendor/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/bundler-2.3.26/libexec/bundle:48:in `block in <top (required)>'
/opt/zammad/vendor/ruby-3.1.3/lib/ruby/3.1.0/bundler/friendly_errors.rb:120:in `with_friendly_errors'
/opt/zammad/vendor/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/bundler-2.3.26/libexec/bundle:36:in `<top (required)>'
/opt/zammad/bin/bundle:121:in `load'
/opt/zammad/bin/bundle:121:in `<main>'
Tasks: TOP => zammad:searchindex:rebuild => zammad:searchindex:version_supported

fliebe92 · July 7, 2023, 7:57pm

Did you try to use a tool like Elasticvue to access your Elasticsearch installation? I guess it’s broken somehow, but I have no idea why.

fliebe92 · July 7, 2023, 7:59pm

… or simply run this: curl -X GET 'localhost:9200/_cluster/health?pretty'

robinedwards · July 8, 2023, 11:15am

Thanks for this.
After doing some more research, I found that xpack was enabled and because of that it did not accept unauthenticated http requests. Once I disabled xpack in the elasticsearch.yml, everything started to work.

admin@zammad:~$ curl -XGET http://127.0.0.1:9200/_cluster/health?pretty
{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Thank you for your help.

system · July 15, 2023, 11:16am

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.