Elasticsearch index file needs to be re-index in old version

FunkStar · September 23, 2019, 11:56am

Infos:

Used Zammad version: 3.1.x
Used Zammad installation source: (source, package, …) package
Operating system: Debian 9
Browser + version: FF 69

Expected behavior:

Reports and users showing

Actual behavior:

Reporting unavailable and users not being shown

Steps to reproduce the behavior:

zammad run rails r "Setting.set('es_url', '')"

Fixes users being shown, reports disabled without Elasticsearch.

Hi all

I inherited a zammad system and came across a problem with Elasticsearch.
It seems the helpdesk always ran with Elasticsearch disabled, I tried to enable it so we could have faster searches and use the reporting part.

When I enabled elasticsearch the users in zammad disappeared and instead of Elasticsearch needs to be configure i got:

Unable to process GET request to elasticsearch URL ‘http://127.0.0.1:9200/zammad_production/Ticket/_search’

curl http://localhost:9200
Failed connect to localhost:9200; connection refused

Elasticsearch is on the same server, no proxy’s or whatsoever.

After checking the elasticsearch.yml file (al those settings seem fine) I moved on to the logfiles.

These 2 are the culprits but as I don’t have a lot of experience with elasticsearch I’m hoping someone has had a similar error and knows a solution.

failed to read /var/lib/elasticsearch/nodes/0/_state/global-20.st
index was created with version [5.5.3] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading

So the first error can be solved by renaming global-20.st to global-20.st.old.
But the index needs to be re-indexed in an old Elasticsearch version and lets say:

Removing elasticsearch, removed zammad
Installing Elastissearch 6 over the current version just borked the whole server that I needed to put a snapshot back.

Thanks for the help guys.

MrGeneration · September 23, 2019, 5:48pm

Sorry but this is out of scope.
You upgraded elasticsearch and ignored the upgrade path of it.

Please consult this page to help yourself:
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html

If you can’t do that anymore, just remove the elasticsearch data directories and restart elasticsearch.
Again, this is an elasticsearch based issue and out of our scope.
Sorry.

FunkStar · September 23, 2019, 8:03pm

Yeah thought so, so I tried installing older version of elasticsearch over 7.3.2.
Atm its working but “zammad run rake searchindex:rebuild” gives and error but users are showing and reporting is working!

Rebuild error (fixed)

Edit: 24/09/2019
Saw error about “ingest-attachment” and thought I was using ES 5.5 instead of 5.6 and installed mapper-attachment instead. While I should have just removed and installed the ingest-attachment as in MrGeneration’s solution.

root@XXXXXXXX:~# zammad run rake searchindex:rebuild
drop indexes...done
delete pipeline (pipeline)... done
create indexes...done
create pipeline (pipeline)... rake aborted!
Unable to process PUT request to elasticsearch URL 'http://localhost:9200/_ingest/pipeline/zammad172706490976'. Check the response and payload for detailed information:

Response:
#<UserAgent::Result:0x00005592c7788048 @success=false, @body="{\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"No processor type exists with name [attachment]\",\"header\":{\"processor_type\":\"foreach\"}}],\"type\":\"parse_exception\",\"reason\":\"No processor type exists with name [attachment]\",\"header\":{\"processor_type\":\"foreach\"}},\"status\":400}", @data=nil, @code="400", @content_type=nil, @error="Client Error: #<Net::HTTPBadRequest 400 Bad Request readbody=true>!">

Payload:
{:description=>"Extract zammad-attachment information from arrays", :processors=>[{:foreach=>{:field=>"article", :ignore_failure=>true, :processor=>{:foreach=>{:field=>"_ingest._value.attachment", :ignore_failure=>true, :processor=>{:attachment=>{:target_field=>"_ingest._value", :field=>"_ingest._value._content", :ignore_failure=>true}}}}}}]}

Payload size: 0M
/opt/zammad/lib/search_index_backend.rb:119:in `block (2 levels) in processors'
/opt/zammad/lib/search_index_backend.rb:79:in `each'
/opt/zammad/lib/search_index_backend.rb:79:in `block in processors'
/opt/zammad/lib/search_index_backend.rb:76:in `each'
/opt/zammad/lib/search_index_backend.rb:76:in `processors'
/opt/zammad/lib/tasks/search_index_es.rake:83:in `block (2 levels) in <main>'
/opt/zammad/lib/tasks/search_index_es.rake:67:in `block (2 levels) in <main>'
/opt/zammad/lib/tasks/search_index_es.rake:152:in `block (2 levels) in <main>'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli/exec.rb:74:in `load'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli/exec.rb:27:in `run'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli.rb:360:in `exec'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor.rb:369:in `dispatch'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli.rb:20:in `dispatch'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor/base.rb:444:in `start'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/cli.rb:10:in `start'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/exe/bundle:30:in `block in <top (required)>'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/lib/bundler/friendly_errors.rb:121:in `with_friendly_errors'
/opt/zammad/vendor/bundle/ruby/2.5.0/gems/bundler-1.15.2/exe/bundle:22:in `<top (required)>'
/opt/zammad/bin/bundle:3:in `load'
/opt/zammad/bin/bundle:3:in `<main>'
Tasks: TOP => searchindex:rebuild
(See full trace by running task with --trace)

Guide to downgrade

btw for those stumbling on this thread months/years later this is how I installed an older elasticsearch version (5.5.6) over 7.3.2 so I could read the index file again.

#stopping elasticsearch    
sudo -i service elasticsearch stop
#More info here: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/deb.html 
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.16.deb

OUTPUT (I did overwrite my current elasticsearch settings just to be sure)

dpkg: warning: downgrading elasticsearch from 7.3.2 to 5.6.16
(Reading database ... 84846 files and directories currently installed.)
Preparing to unpack elasticsearch-5.6.16.deb ...
Unpacking elasticsearch (5.6.16) over (7.3.2) ...
Setting up elasticsearch (5.6.16) ...

Configuration file '/etc/elasticsearch/elasticsearch.yml'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** elasticsearch.yml (Y/I/N/O/D/Z) [default=N] ? Y
Installing new version of config file /etc/elasticsearch/elasticsearch.yml ...
Installing new version of config file /etc/elasticsearch/jvm.options ...
Installing new version of config file /etc/elasticsearch/log4j2.properties ...

Configuration file '/usr/lib/systemd/system/elasticsearch.service'
 ==> File on system created by you or by a script.
 ==> File also in package provided by package maintainer.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** elasticsearch.service (Y/I/N/O/D/Z) [default=N] ? Y
Installing new version of config file /usr/lib/systemd/system/elasticsearch.service ...
Installing new version of config file /etc/init.d/elasticsearch ...
Installing new version of config file /etc/default/elasticsearch ...
Processing triggers for systemd (232-25+deb9u12) ...

Check the elasticsearch.yml setting file (found in /etc/elasticsearch) and make changes (localhost, nodename, port,…).

Start elasticsearch and check status

sudo -i service elasticsearch start
sudo systemctl status elasticsearch

Still failed so open logs (found in /var/log/elasticsearch)

Still had the “ingest-attachment” plugin installed that 7.3.2 uses.
Remove it with this command:

sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove ingest-attachment

Also had the following error in the logs:

Error injecting constructor, ElasticsearchException[java.io.IOException: failed to read [id:20, legacy:false, file:/var/lib/elasticsearch/nodes/0/_state/global-20.st]];

“fixed” this by renaming the file @ /var/lib/elasticsearch/nodes/0/_state/

mv global-20.st global-20.st.old

Follow zammad instructions

Set up Elasticsearch — Zammad System Documentation documentation

Install ingest-attachment

sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment
systemctl restart elasticsearch

Configure Zammad to work with Elasticsearch

zammad run rails r "Setting.set('es_url', 'http://localhost:9200')"

After you have configured Zammad for using Elasticsearch, you need to rebuild the index with the following command:

zammad run rake searchindex:rebuild

MrGeneration · September 24, 2019, 9:30am

You tried to install mapper-attachments which is a attachment plugin pre elasticsearch 5.6 (According to your console output, you installed 5.6.16). This should have thrown an error - anway.
Do the following to solve this:

sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove ingest-attachment #safety IF any was installed
sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment
systemctl restart elasticsearch
zammad run rake searchindex:rebuild

FunkStar · September 24, 2019, 10:50am

@MrGeneration

Saw an error about the “ingest-attachment” in the logs and thought I was using ES 5.5 instead of 5.6.
So removed the ingest-attachmenent and installed the mapper-attachment.

While reinstalling the ingest-attachment would have fixed it.

My bad, thanks for the help!

system · January 22, 2020, 10:50am

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.