ES index 'zammad_production_ticket' not created after starting zammad for the very first time

[Access-InTech]

Infos:

• Used Zammad version: 6.1.0-1697809120.a1de05d1.bullseye amd64
• Used Zammad installation type: package
• Operating system: Debian GNU/Linux 11 (bullseye)
• Browser + version: Firefox 119.0 (64-Bit)

Expected behavior:

• After installing Zammad from package, I initialize the system via auto_wizard.json. When I login and click on Reports (this is how I check if ElasticSearch is working properly), I’m expecting to see the initial reports chart.

Actual behavior:

I’m getting a HTTP 500 with the following message:

Unable to process GET request to elasticsearch URL 'http://localhost:9200/zammad_production_ticket/_search'. Check the response and payload for detailed information:

Response:
#<UserAgent::Result:0x00007f439f537098 @success=false, @body=nil, @data=nil, @code="404", @content_type=nil, @error="No such file http://localhost:9200/zammad_production_ticket/_search, 404!", @header={"x-elastic-product"=>"Elasticsearch", "warning"=>"299 Elasticsearch-7.17.14-774e3bfa4d52e2834e4d9d8d669d77e4e5c1017f \"Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.\"", "content-type"=>"application/json; charset=UTF-8", "content-length"=>"473"}>

Payload:
{"size":0,"query":{"bool":{"must":[{"range":{"created_at":{"from":"2022-12-31T23:00:00Z","to":"2023-12-31T22:59:59Z"}}},{"bool":{"must":[{"bool":{"must_not":[{"term":{"state.name.keyword":"merged"}}]}}]}}]}},"aggs":{"time_buckets":{"date_histogram":{"field":"created_at","calendar_interval":"month","time_zone":"Europe/Berlin"}}},"sort":[{"updated_at":{"order":"desc"}},"_score"]}

Payload size: 0M

This seems to be caused by the missing index zammad_production_ticket:

root@zammadtest:~# curl -XGET -s http://localhost:9200/_cat/indices
green  open .geoip_databases               a1bWZZoJRnSSOAHG88zaXA 1 0 40 0  37.9mb  37.9mb
yellow open zammad_production_stats_store  TK9OoaAaScWDHQCg2OMr5w 1 1  1 1  61.7kb  61.7kb
yellow open zammad_production_organization 3aflmL8BQhewlfKeGQcPJg 1 1  1 3 190.9kb 190.9kb
yellow open zammad_production_user         6vVS_mCyQoux-fDkyY1qHw 1 1  2 9   322kb   322kb
yellow open zammad_production_group        r70j7h7GRFmY7qVYgNiIIQ 1 1  1 2  77.6kb  77.6kb

As soos as I change anything in the ticket Welcome to Zammad!, the index is created and everything works as expected.

Is this a bug, or something that may be related to the auto_wizard.json?

Thanks in advance
@Access-InTech

[Access-InTech]

Hello,

any ideas?

Best regards

[fliebe92]

Hi @Access-InTech.

I was not able to reproduce this, what happens if you just execute the command to re-index everything from scratch? See here.

[Access-InTech]

Hi @fliebe92

I’ll check this after work.

[Access-InTech]

Hi @fliebe92

right after installing:

root@zammadtest:~# curl -XGET -s http://localhost:9200/_cat/indices
green  open .geoip_databases               -BYNE-8dRu-AuXIL8BUG0w 1 0 2 0     2mb     2mb
yellow open zammad_production_stats_store  n_TfE4v6Tqae7fpYiuVWfg 1 1 1 1  61.8kb  61.8kb
yellow open zammad_production_organization BMVXy8xYS6mmifEDeYaBHw 1 1 1 3 190.9kb 190.9kb
yellow open zammad_production_group        I6w2IrHORQWy_rla4W0aXA 1 1 1 2  77.6kb  77.6kb
yellow open zammad_production_user         RsP2f-_pQqCBSFdysh7tVA 1 1 2 4 120.9kb 120.9kb
root@zammadtest:~#

root@zammadtest:~# zammad run rake zammad:searchindex:rebuild
Dropping indexes... done.
Deleting pipeline... done.
Creating indexes... done.
Creating pipeline... rake aborted!
Unable to process put request to elasticsearch URL 'http://localhost:9200/_ingest/pipeline/zammad7393a60f-5635-4b96-9155-cc223ceb44be'. Check the response and payload for detailed information:

Response:
#<UserAgent::Result:0x00007fa5c7cc8f48 @success=false, @body="{\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"No processor type exists with name [attachment]\",\"processor_type\":\"foreach\",\"suppressed\":[{\"type\":\"parse_exception\",\"reason\":\"No processor type exists with name [attachment]\",\"processor_type\":\"foreach\"}]}],\"type\":\"parse_exception\",\"reason\":\"No processor type exists with name [attachment]\",\"processor_type\":\"foreach\",\"suppressed\":[{\"type\":\"parse_exception\",\"reason\":\"No processor type exists with name [attachment]\",\"processor_type\":\"foreach\"}]},\"status\":400}", @data=nil, @code="400", @content_type=nil, @error="Client Error: #<Net::HTTPBadRequest 400 Bad Request readbody=true>!", @header={"x-elastic-product"=>"Elasticsearch", "warning"=>"299 Elasticsearch-7.17.14-774e3bfa4d52e2834e4d9d8d669d77e4e5c1017f \"Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.\"", "content-type"=>"application/json; charset=UTF-8", "content-length"=>"519"}>

Payload:
{"description":"Extract zammad-attachment information from arrays","processors":[{"foreach":{"field":"article","processor":{"foreach":{"field":"_ingest._value.attachment","processor":{"attachment":{"target_field":"_ingest._value","field":"_ingest._value._content","ignore_failure":true,"ignore_missing":true}},"ignore_failure":true,"ignore_missing":true}},"ignore_failure":true,"ignore_missing":true}},{"foreach":{"field":"attachment","processor":{"attachment":{"target_field":"_ingest._value","field":"_ingest._value._content","ignore_failure":true,"ignore_missing":true}},"ignore_failure":true,"ignore_missing":true}}]}

Payload size: 0M
/opt/zammad/lib/search_index_backend.rb:743:in `make_request_and_validate'
/opt/zammad/lib/search_index_backend.rb:88:in `block (2 levels) in processors'
/opt/zammad/lib/search_index_backend.rb:72:in `each'
/opt/zammad/lib/search_index_backend.rb:72:in `block in processors'
/opt/zammad/lib/search_index_backend.rb:69:in `each'
/opt/zammad/lib/search_index_backend.rb:69:in `processors'
/opt/zammad/lib/search_index_backend.rb:933:in `create_pipeline'
/opt/zammad/lib/tasks/zammad/search_index_es.rake:25:in `block (3 levels) in <main>'
/opt/zammad/lib/tasks/zammad/search_index_es.rake:19:in `block (3 levels) in <main>'
/opt/zammad/lib/tasks/zammad/search_index_es.rake:59:in `block (3 levels) in <main>'
/opt/zammad/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/zammad/bin/bundle:121:in `load'
/opt/zammad/bin/bundle:121:in `<main>'
Tasks: TOP => zammad:searchindex:rebuild
(See full trace by running task with --trace)

The same error, but after rebuilding the index, I do not get the error anymore when opening the reports:

root@zammadtest:~# curl -XGET -s http://localhost:9200/_cat/indices
yellow open zammad_production_ticket_state                        PqahXP6ITlm5Hnt0ogx-pg 1 1 0 0 227b 227b
yellow open zammad_production_knowledge_base_category_translation wNSrnaIKQOaKTWweCDncYQ 1 1 0 0 227b 227b
green  open .geoip_databases                                      -BYNE-8dRu-AuXIL8BUG0w 1 0 2 0  2mb  2mb
yellow open zammad_production_knowledge_base_translation          qJ98EDRXQLevFI3F8VeR3w 1 1 0 0 227b 227b
yellow open zammad_production_stats_store                         jff3PXsLRdScZV9fp_FUww 1 1 0 0 227b 227b
yellow open zammad_production_organization                        YLsYZQ83TMKbN_3Kn8EH_g 1 1 0 0 227b 227b
yellow open zammad_production_ticket                              Vlg6tEMxRSK9-iYfaPbTiA 1 1 0 0 227b 227b
yellow open zammad_production_ticket_priority                     KkoUXCtdQZqZ9PCoAKETyg 1 1 0 0 227b 227b
yellow open zammad_production_user                                iLa6T6y3QNaez5J5Zl--Qg 1 1 0 0 227b 227b
yellow open zammad_production_group                               GLQlhW6PQVaNH1pD1IfmCA 1 1 0 0 227b 227b
yellow open zammad_production_cti_log                             OySNaq_lQkiP1k_bvOBGoQ 1 1 0 0 227b 227b
yellow open zammad_production_knowledge_base_answer_translation   Ox44uo_tTki7XcGhQF9Feg 1 1 0 0 227b 227b
yellow open zammad_production_chat_session                        nV8Dx84RSVqiiL4dbxHV3Q 1 1 0 0 227b 227b
root@zammadtest:~#

The missing index and some more has been added or created after reindexing.

[MrGeneration]

Please read the documentation. You’re missing dependencies for Elasticsearch 7.
https://docs.zammad.org/en/latest/install/elasticsearch.html

[Access-InTech]

gnupg and apt-transport-https were missing.
I’ve added a task for installing requirements before installing ElasticSearch.

- name: Install ElasticSearch dependencies
  ansible.builtin.apt:
    name: "{{ item }}"
    state: present
    update_cache: true
  loop:
    - sudo
    - apt-transport-https
    - wget
    - curl
    - gnupg

- name: Install ElasticSearch from Repo
  ansible.builtin.apt:
    name: "elasticsearch={{ es_version }}"
    state: present

Well, sometimes it’s too easy to recognize an issue by yourself.

Thank you bot, @fliebe92 and @MrGeneration

[MrGeneration]

The ingest_attachment plugin was also missing. Just for completness.

[Access-InTech]

The plugin should have been installed right after the installation of ElasticSearch.
If I check for the plugin to be installed, it seems to be ok:

root@zammadtest:~# /usr/share/elasticsearch/bin/elasticsearch-plugin list
ingest-attachment

In case the plugin is missing, it gets installed and then ElasticSearch gets restarted.

tasks file for role elasticsearch

---
---
# tasks file for elasticsearch

- name: Install ElasticSearch repository
  tags:
    - zammad-requirements
    - elasticsearch
    - zammad
  block:
    - name: Get signing keys for apt repositories
      ansible.builtin.get_url:
        url: "{{ es_url }}"
        dest: "/etc/apt/trusted.gpg.d/elasticsearch-{{ es_version.split('.')[0] }}.asc"
        mode: "644"
    - name: Add apt repository
      ansible.builtin.apt_repository:
        repo: "{{ es_repo }}"
        filename: "elasticsearch-{{ es_version.split('.')[0] }}"
        state: present

- name: Install ElasticSearch dependencies
  ansible.builtin.apt:
    name: "{{ item }}"
    state: present
    update_cache: true
  loop:
    - sudo
    - apt-transport-https
    - wget
    - curl
    - gnupg
  tags:
    - zammad-requirements
    - elasticsearch
    - zammad

- name: Install ElasticSearch from Repo
  ansible.builtin.apt:
    name: "elasticsearch={{ es_version }}"
    state: present
  notify:
    - Daemon Reload
    - Enable ElasticSearch
    - Start ElasticSearch
  register: elasticsearch_installed
  tags:
    - zammad-requirements
    - elasticsearch
    - zammad

- name: Flush handlers
  ansible.builtin.meta: flush_handlers
  tags:
    - zammad-requirements
    - elasticsearch
    - zammad

- name: Hold ElasticSearch package
  ansible.builtin.dpkg_selections:
    name: elasticsearch
    selection: hold
  changed_when: elasticsearch_installed.changed
  tags:
    - zammad-requirements
    - elasticsearch
    - zammad

- name: Set custom JVM Heap size
  ansible.builtin.template:
    src: custom-heap-size.options.j2
    dest: /etc/elasticsearch/jvm.options.d/custom-heap-size.options
    mode: 0644
    owner: root
    group: elasticsearch
  notify: Restart ElasticSearch
  tags:
    - zammad-requirements
    - elasticsearch
    - zammad

- name: Flush handlers
  ansible.builtin.meta: flush_handlers
  tags:
    - zammad-requirements
    - elasticsearch
    - zammad

- name: Wait for ElasticSearch to be up and running
  ansible.builtin.wait_for:
    host: localhost
    port: 9200
    delay: 5
    connect_timeout: 1
  tags:
    - zammad-requirements
    - elasticsearch
    - zammad

- name: Check if ElasticSearch Plugin ingest-attachment is installed
  tags:
    - zammad-requirements
    - elasticsearch
    - zammad
  block:
    - name: Check if ElasticSearch Plugin ingest-attachment is installed
      ansible.builtin.command:
        cmd: /usr/share/elasticsearch/bin/elasticsearch-plugin list
      register: ingest_attachment_installed
      changed_when: false
    - name: Install ElasticSearch Plugin ingest-attachment
      when: ingest_attachment_installed.stdout != "ingest-attachment"
      ansible.builtin.shell:
        cmd: |
          set -o pipefail
          echo y | /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment 2>&1
        executable: /bin/bash
      register: install_es_plugin
      changed_when: "'Installed ingest-attachment' in install_es_plugin.stdout"
      notify: Restart ElasticSearch
  rescue:
    - name: Remove failed ElasticSearch Plugin ingest-attachment
      ansible.builtin.shell:
        cmd: |
          set -o pipefail
          echo y | /usr/share/elasticsearch/bin/elasticsearch-plugin remove -s ingest-attachment
        executable: /bin/bash
      register: ingest_attachment_removed
      changed_when: "'removing [ingest-attachment]' in ingest_attachment_removed.stdout"
      notify: Restart ElasticSearch

- name: Flush handlers
  ansible.builtin.meta: flush_handlers
  tags:
    - zammad-requirements
    - elasticsearch
    - zammad

[MrGeneration]

Maybe your Elasticsearch was not restarted.
The last error message you provided indicates a missing plugin, be it due to not being installed or an “ignored” restart of ES for whatever reason.

[...] [{\"type\":\"parse_exception\",\"reason\":\"No processor type exists with name [attachment]\",\"processor_type\":\"foreach\"}]}] [...]

[Access-InTech]

I’l investigate that, too.

Thank you