Elasticsearch index file needs to be re-index in old version

Infos:

  • Used Zammad version: 3.1.x
  • Used Zammad installation source: (source, package, …) package
  • Operating system: Debian 9
  • Browser + version: FF 69

Expected behavior:

  • Reports and users showing

Actual behavior:

  • Reporting unavailable and users not being shown

Steps to reproduce the behavior:

zammad run rails r "Setting.set('es_url', '')"

Fixes users being shown, reports disabled without Elasticsearch.



Hi all

I inherited a zammad system and came across a problem with Elasticsearch.
It seems the helpdesk always ran with Elasticsearch disabled, I tried to enable it so we could have faster searches and use the reporting part.

When I enabled elasticsearch the users in zammad disappeared and instead of Elasticsearch needs to be configure i got:

Unable to process GET request to elasticsearch URL ‘http://127.0.0.1:9200/zammad_production/Ticket/_search’

curl http://localhost:9200
Failed connect to localhost:9200; connection refused

Elasticsearch is on the same server, no proxy’s or whatsoever.

After checking the elasticsearch.yml file (al those settings seem fine) I moved on to the logfiles.

These 2 are the culprits but as I don’t have a lot of experience with elasticsearch I’m hoping someone has had a similar error and knows a solution.

  • failed to read /var/lib/elasticsearch/nodes/0/_state/global-20.st
  • index was created with version [5.5.3] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading

So the first error can be solved by renaming global-20.st to global-20.st.old.
But the index needs to be re-indexed in an old Elasticsearch version and lets say:

  • Removing elasticsearch, removed zammad
  • Installing Elastissearch 6 over the current version just borked the whole server that I needed to put a snapshot back.

Thanks for the help guys.

Sorry but this is out of scope.
You upgraded elasticsearch and ignored the upgrade path of it.

Please consult this page to help yourself:
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html

If you can’t do that anymore, just remove the elasticsearch data directories and restart elasticsearch.
Again, this is an elasticsearch based issue and out of our scope.
Sorry.

Yeah thought so, so I tried installing older version of elasticsearch over 7.3.2.
Atm its working but “zammad run rake searchindex:rebuild” gives and error but users are showing and reporting is working!

Rebuild error (fixed) :white_check_mark:

Edit: 24/09/2019
Saw error about “ingest-attachment” and thought I was using ES 5.5 instead of 5.6 and installed mapper-attachment instead. While I should have just removed and installed the ingest-attachment as in MrGeneration’s solution.

root@XXXXXXXX:~# zammad run rake searchindex:rebuild
drop indexes...done
delete pipeline (pipeline)... done
create indexes...done
create pipeline (pipeline)... rake aborted!
Unable to process PUT request to elasticsearch URL 'http://localhost:9200/_ingest/pipeline/zammad172706490976'. Check the response and payload for detailed information:

Response:
#<UserAgent::Result:0x00005592c7788048 @success=false, @body="{\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"No processor type exists with name [attachment]\",\"header\":{\"processor_type\":\"foreach\"}}],\"type\":\"parse_exception\",\"reason\":\"No processor type exists with name [attachment]\",\"header\":{\"processor_type\":\"foreach\"}},\"status\":400}", @data=nil, @code="400", @content_type=nil, @error="Client Error: #<Net::HTTPBadRequest 400 Bad Request readbody=true>!">

Payload:
{:description=>"Extract zammad-attachment information from arrays", :processors=>[{:foreach=>{:field=>"article", :ignore_failure=>true, :processor=>{:foreach=>{:field=>"_ingest._value.attachment", :ignore_failure=>true, :processor=>{:attachment=>{:target_field=>"_ingest._value", :field=>"_ingest._value._content", :ignore_failure=>true}}}}}}]}

Payload size: 0M
/opt/zammad/lib/search_index_backend.rb:119:in `block (2 levels) in processors'
/opt/zammad/lib/search_index_backend.rb:79:in `each'
/opt/zammad/lib/search_index_backend.rb:79:in `block in processors'
/opt/zammad/lib/search_index_backend.rb:76:in `each'
/opt/zammad/lib/search_index_backend.rb:76:in `processors'
/opt/zammad/lib/tasks/search_index_es.rake:83:in `block (2 levels) in <main>'
/opt/zammad/lib/tasks/search_index_es.rake:67:in `block (2 levels) in <main>'
/opt/zammad/lib/tasks/search_index_es.rake:152:in `block (2 levels) in <main>'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli/exec.rb:74:in `load'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli/exec.rb:27:in `run'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli.rb:360:in `exec'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor.rb:369:in `dispatch'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli.rb:20:in `dispatch'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor/base.rb:444:in `start'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli.rb:10:in `start'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/exe/bundle:30:in `block in <top (required)>'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/friendly_errors.rb:121:in `with_friendly_errors'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/exe/bundle:22:in `<top (required)>'
/opt/zammad/bin/bundle:3:in `load'
/opt/zammad/bin/bundle:3:in `<main>'
Tasks: TOP => searchindex:rebuild
(See full trace by running task with --trace)

Guide to downgrade :white_check_mark:

btw for those stumbling on this thread months/years later this is how I installed an older elasticsearch version (5.5.6) over 7.3.2 so I could read the index file again.

#stopping elasticsearch    
sudo -i service elasticsearch stop
#More info here: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/deb.html 
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.16.deb

OUTPUT (I did overwrite my current elasticsearch settings just to be sure)

dpkg: warning: downgrading elasticsearch from 7.3.2 to 5.6.16
(Reading database ... 84846 files and directories currently installed.)
Preparing to unpack elasticsearch-5.6.16.deb ...
Unpacking elasticsearch (5.6.16) over (7.3.2) ...
Setting up elasticsearch (5.6.16) ...

Configuration file '/etc/elasticsearch/elasticsearch.yml'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** elasticsearch.yml (Y/I/N/O/D/Z) [default=N] ? Y
Installing new version of config file /etc/elasticsearch/elasticsearch.yml ...
Installing new version of config file /etc/elasticsearch/jvm.options ...
Installing new version of config file /etc/elasticsearch/log4j2.properties ...

Configuration file '/usr/lib/systemd/system/elasticsearch.service'
 ==> File on system created by you or by a script.
 ==> File also in package provided by package maintainer.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** elasticsearch.service (Y/I/N/O/D/Z) [default=N] ? Y
Installing new version of config file /usr/lib/systemd/system/elasticsearch.service ...
Installing new version of config file /etc/init.d/elasticsearch ...
Installing new version of config file /etc/default/elasticsearch ...
Processing triggers for systemd (232-25+deb9u12) ...

Check the elasticsearch.yml setting file (found in /etc/elasticsearch) and make changes (localhost, nodename, port,…).

Start elasticsearch and check status

sudo -i service elasticsearch start
sudo systemctl status elasticsearch

Still failed so open logs (found in /var/log/elasticsearch)

Still had the “ingest-attachment” plugin installed that 7.3.2 uses.
Remove it with this command:

sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove ingest-attachment

Also had the following error in the logs:

Error injecting constructor, ElasticsearchException[java.io.IOException: failed to read [id:20, legacy:false, file:/var/lib/elasticsearch/nodes/0/_state/global-20.st]];

“fixed” this by renaming the file @ /var/lib/elasticsearch/nodes/0/_state/

mv global-20.st global-20.st.old

Follow zammad instructions

:arrow_right: https://docs.zammad.org/en/latest/install-elasticsearch.html

Install ingest-attachment

sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment
systemctl restart elasticsearch

Configure Zammad to work with Elasticsearch

zammad run rails r "Setting.set('es_url', 'http://localhost:9200')"

After you have configured Zammad for using Elasticsearch, you need to rebuild the index with the following command:

zammad run rake searchindex:rebuild

1 Like

You tried to install mapper-attachments which is a attachment plugin pre elasticsearch 5.6 (According to your console output, you installed 5.6.16). This should have thrown an error - anway.
Do the following to solve this:

sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove ingest-attachment #safety IF any was installed
sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment
systemctl restart elasticsearch
zammad run rake searchindex:rebuild

@MrGeneration

Saw an error about the “ingest-attachment” in the logs and thought I was using ES 5.5 instead of 5.6.
So removed the ingest-attachmenent and installed the mapper-attachment.

While reinstalling the ingest-attachment would have fixed it.

My bad, thanks for the help!

1 Like

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.