Elasticsearch error after importing a backup - Zammad 6.0.0

cretection · July 27, 2023, 9:30am

Infos:

Used Zammad version: 6.0.0
Used Zammad installation type: docker-compose
Operating system: Ubuntu 22.04

I had broken my installation while trying to correct a mistake.

I accidentally did a docker compose down -v.

Then I had to realize that I have to rebuild the system - which is of course clear after the command.

So I restored the last backup to have all my tickets back. This has also worked so far. However, it is noticeable that the automatic completion, for example, with a user to which I want to assign a company no longer works.

If I now execute the command rake zammad:searchindex:rebuild I get:

I, [2023-07-27T09:19:50.384216 #50] INFO -- : ActionCable is using the redis instance at redis://zammad-redis:6379.
I, [2023-07-27T09:19:50.398599#50-6160] INFO -- : Using memcached as Rails cache store.
I, [2023-07-27T09:19:50.398744#50-6160] INFO -- : Using the Redis back end for Zammad's web socket session store.
Elasticsearch is not configured.

So I try to reconfigure the Elasticsearch:

rails r "Setting.set('es_url', 'http://zammad-elasticsearch:9200')"

then I get:

I, [2023-07-27T09:23:01.886172 #57]  INFO -- : ActionCable is using the redis instance at redis://zammad-redis:6379.
I, [2023-07-27T09:23:01.897111#57-5380]  INFO -- : Using memcached as Rails cache store.
I, [2023-07-27T09:23:01.897238#57-5380]  INFO -- : Using the Redis back end for Zammad's web socket session store.
/opt/zammad/app/models/setting.rb:38:in `set': Can't find config setting 'es_url' (RuntimeError)
        from /usr/local/bundle/gems/railties-6.1.7.4/lib/rails/commands/runner/runner_command.rb:45:in `<main>'
        from /usr/local/bundle/gems/railties-6.1.7.4/lib/rails/commands/runner/runner_command.rb:45:in `eval'
        from /usr/local/bundle/gems/railties-6.1.7.4/lib/rails/commands/runner/runner_command.rb:45:in `perform'
        from /usr/local/bundle/gems/thor-1.2.2/lib/thor/command.rb:27:in `run'
        from /usr/local/bundle/gems/thor-1.2.2/lib/thor/invocation.rb:127:in `invoke_command'
        from /usr/local/bundle/gems/thor-1.2.2/lib/thor.rb:392:in `dispatch'
        from /usr/local/bundle/gems/railties-6.1.7.4/lib/rails/command/base.rb:69:in `perform'
        from /usr/local/bundle/gems/railties-6.1.7.4/lib/rails/command.rb:48:in `invoke'
        from /usr/local/bundle/gems/railties-6.1.7.4/lib/rails/commands.rb:18:in `<main>'
        from /usr/local/bundle/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
        from /usr/local/bundle/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
        from bin/rails:5:in `<main>'

When I look in the docker logs I get:

networkcretectionhelpdesk-zammad-scheduler-1      | Payload size: 0M
networkcretectionhelpdesk-zammad-scheduler-1      | I, [2023-07-27T09:27:34.365798#1-112140]  INFO -- : 2023-07-27T09:27:34+0000: [Worker(host:b81a720cff35 pid:1)] Job SearchIndexJob [01f35436-57a9-4aee-ba51-811aeed2037c] from DelayedJob(default) with arguments: ["User", 3] (id=2473) (queue=default) COMPLETED after 0.0919
networkcretectionhelpdesk-zammad-scheduler-1      | I, [2023-07-27T09:27:34.368473#1-112140]  INFO -- : 2023-07-27T09:27:34+0000: [Worker(host:b81a720cff35 pid:1)] Job SearchIndexAssociationsJob [c52685d5-5f04-4a2a-a756-dbe0af608b76] from DelayedJob(default) with arguments: ["User", 3] (id=2474) (queue=default) RUNNING
networkcretectionhelpdesk-zammad-scheduler-1      | E, [2023-07-27T09:27:34.435694#1-112140] ERROR -- : Unable to process post request to elasticsearch URL 'http://zammad-elasticsearch:9200/zammad_production_user/_doc/3?pipeline=zammad303115eb-d61f-44f3-a441-00d94c2f1e59'. Check the response and payload for detailed information:
networkcretectionhelpdesk-zammad-scheduler-1      | 
networkcretectionhelpdesk-zammad-scheduler-1      | 
networkcretectionhelpdesk-zammad-scheduler-1      | Response:
networkcretectionhelpdesk-zammad-scheduler-1      | #<UserAgent::Result:0x00007f21fa434270 @success=false, @body="{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [zammad303115eb-d61f-44f3-a441-00d94c2f1e59] does not exist\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [zammad303115eb-d61f-44f3-a441-00d94c2f1e59] does not exist\"},\"status\":400}", @data=nil, @code="400", @content_type=nil, @error="Client Error: #<Net::HTTPBadRequest 400 Bad Request readbody=true>!", @header={"x-elastic-product"=>"Elasticsearch", "content-type"=>"application/json", "content-length"=>"289"}>

Does anyone have a tip on how to get the search completion running again?

fliebe92 · July 27, 2023, 10:12am

Hi @cretection.

You should ask yourself why this fails. Without this setting your search will not work anymore with ES.

cretection · July 27, 2023, 10:16am

That’s what I was wondering - so I wanted to reset the configuration but I get this error.

However, I have not been able to find help anywhere on how to bypass the runtime error.

When I formulate the command with get I get back nil

fliebe92 · July 27, 2023, 10:17am

I guess there is something more broken. What’s the output of rails r 'pp Setting.all.count'?

cretection · July 27, 2023, 10:29am

You seem to be right. I get as output:

zammad@8bb3fb9773c2:~$ rails r 'pp Setting.all.count'
I, [2023-07-27T10:26:02.845843 #34] INFO -- : ActionCable is using the redis instance at redis://zammad-redis:6379.
I, [2023-07-27T10:26:02.857613#34-5380] INFO -- : Using Zammad's file store as Rails cache store.
I, [2023-07-27T10:26:02.857762#34-5380] INFO -- : Using the Redis back end for Zammad's web socket session store.
0

It would be no problem to rebuild the system. I just wonder if this will happen again the next time I am forced to make a backup.

MrGeneration · July 27, 2023, 10:43am

Are you certain that your compose stack is looking into the correct database? You restored into the correct database?

cretection · July 27, 2023, 10:49am

All data is available in the Zammad UI. All users and all tickets are there.
The only thing that does not work is the auto-completion and associated assignment for example with user and company

I have imported the backup like this:

docker compose exec -it zammad-postgresql bash

psql -U zammad -d zammad_production
\q

dropdb zammad_production -U zammad
createdb zammad_production -U zammad
psql zammad_production < /var/lib/postgresql/data/backup.psql -U zammad
exit

Could it have something to do with the fact that in the Docker Compose setup of Zammad 6 only
/opt/zammad/var
is still included in the backup?

fliebe92 · July 27, 2023, 10:52am

If everything is working fine in the GUI, then I guess the rails r commands you’ve submitted are using the nulldb adapter and have no database access at all.

What happens if you submit rake zammad:searchindex:rebuild?

cretection · July 27, 2023, 10:54am

Just to understand and make sure it’s not that - in which container does this command need to be executed?

I mean I have tried all of them and always get the same error but not that in the end that is where the error lies

fliebe92 · July 27, 2023, 10:55am

AFAIR it is zammad-railsserver.

cretection · July 27, 2023, 10:57am

Then I used the right container:

docker compose exec -it zammad-railsserver rake zammad:searchindex:rebuild
I, [2023-07-27T10:55:42.790837 #37]  INFO -- : ActionCable is using the redis instance at redis://zammad-redis:6379.
I, [2023-07-27T10:55:42.805960#37-6160]  INFO -- : Using memcached as Rails cache store.
I, [2023-07-27T10:55:42.806185#37-6160]  INFO -- : Using the Redis back end for Zammad's web socket session store.
Elasticsearch is not configured.

fliebe92 · July 27, 2023, 11:01am

Oh, damn, I overlooked this in your first post. Sorry.

Maybe this helps: V6.0.0 Zammad-rails return empty values - #12 by pabloR_77

cretection · July 27, 2023, 11:19am

That helped me! When I came across this post I could not do anything with it.

I will leave a short documentation here how I could solve the problem completely with backup and then close the ticket - thanks for the great help

cretection · July 27, 2023, 12:41pm

I made a minor adjustment to my setup configuration and mounted the volume for my backups locally, as opposed to the Docker-Compose.yml recommended on Github. This allows me to run the docker compose down -v command worry-free. If you’re following the standard configuration from Github, make sure to secure the latest backup of the database and files from the respective Docker volume before executing this command.

You can usually find your backups at this path:

/var/lib/docker/volumes/zammad_zammad-backup/_data/backup/

After securing your latest backups, you can proceed with the following actions:

First, you shut down all containers and delete all data:

docker compose down -v

Then, you recreate the containers and stop them:

docker compose up -d
docker compose down

Now, extract the database backup:

gzip -dk /path/of/your/backup/xxxxxxxxxx_zammad_db.psql.gz
mv /path/of/your/backup/xxxxxxxxxx_zammad_db.psql /var/lib/docker/volumes/zammad_postgresql-data/_data/backup_db.psql

Start the database next:

docker compose up -d zammad-postgresql
docker compose exec -it zammad-postgresql bash

First, check if you can access the database:

psql -U zammad -d zammad_production

If successful, enter \q to close the connection to the database.

Now, you delete the database and create a new one with the same name using the following command:

dropdb zammad_production -U zammad
createdb zammad_production -U zammad

Now, you can re-import your backup. Since you’ve copied it into the PostgreSQL volumes, you can access it while inside the container:

psql zammad_production < /var/lib/postgresql/data/backup.psql -U zammad

Finally, you can exit the container and shut down all containers.

exit

In the next step of our guide, we will deal with the old files generated by docker compose up. These will be deleted to make room for the files from your backup:

# Delete all old files in the specified directory
rm -rf /var/lib/docker/volumes/zammad_zammad-var/_data/*

Now we extract your files from the backup:

# Navigate to your backup directory
cd /path/of/your/backup/
# Extract the files from the tar.gz backup
tar xzvf xxxxxxxxxx_zammad_files.tar.gz

After successfully executing this command, you need to copy the extracted data into the correct container:

# Moves the extracted files to the right container
mv opt/zammad/var/* /var/lib/docker/volumes/zammad_zammad-var/_data/

Now all containers can be started again:

# Navigate back to your Docker Compose directory
cd /to-your-docker-compose-files/
# Starts all Docker containers
docker compose up -d

Note: If something doesn’t work properly, it’s worth checking the Nginx logs. Depending on how the data was copied, there might be ownership issues. Therefore, in Nginx, check /opt/zammad/var/ and its contents with ls -al to see if it is owned by the user zammad. If it is owned by root, use chown -R zammad:zammad /opt/zammad/var and restart.

The next step is to regenerate the search index:

# Performs the regeneration of the search index
docker compose exec -it zammad-railsserver rake searchindex:rebuild

If the command runs without errors, everything is fine. However, if you receive the following message:

Elasticsearch is not configured.

it is very likely that the autocomplete search in the UI no longer works. The Elasticsearch configuration documentation might not immediately help here, as the commands do not work straightforwardly under Docker Compose. Therefore, the following steps are necessary for it to work in your environment:

First, you connect to the Rails server container:

# Connects you to the Rails server container
docker exec -u 0 -it zammad-railsserver bash

As the container does not contain an editor, you install one. I recommend nano:

# Updates package lists for upgrades and new installations
apt update  
# Installs the text editor nano
apt install nano  
# Opens the database configuration file with nano
nano config/database.yml

The following values should be entered or added under production:

production:  
  adapter: postgresql  
  database: zammad_production  
  pool: 50  
  timeout: 5000  
  encoding: utf8  
  username: zammad  
  password: zammad  
  host: zammad-postgresql  
  port: 5432

Save your changes and disconnect with exit.

Now you can set the correct link for Elasticsearch:

# Sets the Elasticsearch link in the Zammad settings
docker compose exec -it zammad-railsserver rails r "Setting.set('es_url', 'http://zammad-elasticsearch:9200')"

Now you can execute the actual command that is still needed:

# Initiates the process of rebuilding the search index
docker compose exec -it zammad-railsserver rake zammad:searchindex:rebuild

Remember that these commands are executed in a certain context. It might be necessary to adapt them to your specific environment. Always make sure to create backups of your data and keep them safe before making significant changes to your infrastructure.

system · August 3, 2023, 12:41pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.