Error when creating elasticsearch index with custom fields

niondir · June 5, 2025, 6:33pm

Infos:

Used Zammad version: 6.5.0
Elasticsearch Version: 8.17.4
Used Zammad installation type: docker-compose
Operating system: Ubuntu
Browser + version: Chrome

Expected behavior:

Search index is created

Actual behavior:

I run the following command:

docker exec zammad6-zammad-railsserver-1 /docker-entrypoint.sh rake zammad:searchindex:rebuild --trace

No errors in elasticsearch log. But the searchindex:rebuild does report the following error multiple times:

Caused by:
Unable to process post request to elasticsearch URL
'http://zammad-elasticsearch:9200/zammad_production_ticket/_doc/13680?pipeline=zammadf94edf71-edcf-4ed5-b759-a47b0d818d81'.
Check the response and payload for detailed information: Response:
#<UserAgent:: Result: 0x00007f4a09e9c7d0 @success=false,
@body="{\"error\":{\"root_cause\":[{\"type\":\"parsing_exception\",
\"reason\":\"Failed to parse object: expecting token of type [START_OBJECT] but found [VALUE_STRING]\",\"line\":1,\"col\":543}],
\"type\":\"document_parsing_exception\",
\"reason\":\"[1:543] failed to parse field [order] of type [flattened] in document with id '13680'. Preview of field's value: ''\",
\"caused_by\":{\"type\":\"parsing_exception\",\"reason\":\"Failed to parse object: expecting token of type [START_OBJECT] but found [VALUE_STRING]\",
\"line\":1,\"col\":543}},\"status\":400}", 
@data=nil, @code="400", @content_type=nil, @error="Client Error: #<Net::HTTPBadRequest 400 Bad Request readbody=true>!", 
@header={"x-elastic-product"=>"Elasticsearch", "content-type"=>"application/json", "content-length"=>"508"
}>

Payload (Stripped irrelevant fields)

   "id": 13680,
  "created_at": "2025-06-05T13:35:39.588Z",
  "updated_at": "2025-06-05T14:25:51.918Z",
  "offer": "",
  "order": "",
  "last_close_at": "2025-06-05T14:25:51.925Z",
  "checklist_id": null,

It seems that elasticsearch is not accepting empty strings in the offer and order field.

Steps to reproduce the behavior:

Create text fields with name “offer” and “order” in Zammad on ticket level
Run rake zammad:searchindex:rebuild

niondir · June 6, 2025, 9:24pm

I run 6.5.0-53 docker version of zammad.

Here is some context to the error. Is it elasticsearch 8 related?

https://stackoverflow.com/questions/71204453/elasticsearch-8-0-0-mapper-parsing-exception-of-a-string-literal-for-field-type

Either zammad should post the custom field as object in the request JSON to es or the index must be created to support the correct string type.

Another issue is, that index creation is working in general, until the create index script returns with an error:

/opt/zammad/lib/search_index_backend.rb:812:in `make_request_and_validate'
/opt/zammad/lib/search_index_backend.rb:151:in `add'
/opt/zammad/app/models/concerns/has_search_index_backend.rb:154:in `search_index_update_backend'
/opt/zammad/app/models/concerns/has_search_index_backend.rb:208:in `block in search_index_reload'
/usr/local/bundle/gems/parallel-1.26.3/lib/parallel.rb:650:in `call_with_index'
/usr/local/bundle/gems/parallel-1.26.3/lib/parallel.rb:417:in `block in work_direct'
/usr/local/bundle/gems/parallel-1.26.3/lib/parallel.rb:660:in `with_instrumentation'
/usr/local/bundle/gems/parallel-1.26.3/lib/parallel.rb:416:in `work_direct'
/usr/local/bundle/gems/parallel-1.26.3/lib/parallel.rb:285:in `map'
/opt/zammad/app/models/concerns/has_search_index_backend.rb:204:in `search_index_reload'
/opt/zammad/lib/tasks/zammad/search_index_es.rake:42:in `block (5 levels) in <main>'
/usr/local/bundle/gems/benchmark-0.4.0/lib/benchmark.rb:323:in `realtime'
/opt/zammad/lib/tasks/zammad/search_index_es.rake:41:in `block (4 levels) in <main>'
/opt/zammad/lib/tasks/zammad/search_index_es.rake:39:in `each'
/opt/zammad/lib/tasks/zammad/search_index_es.rake:39:in `block (3 levels) in <main>'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/task.rb:281:in `block in execute'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/task.rb:281:in `each'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/task.rb:281:in `execute'
/opt/zammad/lib/tasks/zammad/search_index_es.rake:60:in `block (3 levels) in <main>'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/task.rb:281:in `block in execute'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/task.rb:281:in `each'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/task.rb:281:in `execute'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/task.rb:199:in `synchronize'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/task.rb:188:in `invoke'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/application.rb:188:in `invoke_task'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/application.rb:138:in `block (2 levels) in top_level'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/application.rb:138:in `each'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/application.rb:138:in `block in top_level'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/application.rb:147:in `run_with_threads'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/application.rb:132:in `top_level'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/application.rb:83:in `block in run'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/application.rb:214:in `standard_exception_handling'
/usr/local/bundle/gems/rake-13.2.1/lib/rake/application.rb:80:in `run'
/usr/local/bundle/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/usr/local/bundle/bin/rake:25:in `load'
/usr/local/bundle/bin/rake:25:in `<main>'
Tasks: TOP => zammad:searchindex:rebuild

It’s a problem, that e.g. the user index is not created yet and thus also the user search is not working, even if it could. The zammad:searchindex:rebuild just need to continue with the remaining indexes, eve if a single index fails.

It would also help to get a command to rebuild single indexes or to controll which parts to use es for and where to just use the DB (user search via DB would be perfectly fine for me).

niondir · June 6, 2025, 9:42pm

Seems like “order” is a very bad name for a custom field … I just found the following zammad code that takes all columns called “order” and applies the es schema type “flattened” to it.

I will rename the column and check if it fixes my issue…

github.com/zammad/zammad

lib/search_index_backend.rb

bc0cc2127


      
              }
            }
          
          =end
          
            def self.get_mapping_properties_object(object)
              result = {
                properties: {}
              }
          
              store_columns = %w[preferences data condition condition_selected condition_saved perform options view order match timeplan]
          
              # for elasticsearch 6.x and later
              string_type = 'text'
              string_raw  = { type: 'keyword', ignore_above: 5012 }
              boolean_raw = { type: 'boolean' }
          
              object.columns_hash.each do |key, value|
                if store_columns.include?(key)
                  result[:properties][key] = {
                    type: 'flattened',

DD4PK · June 11, 2025, 9:46am

Did the renaming fix it?

rolfschmidt · June 17, 2025, 11:22am

github.com/zammad/zammad

Ticket field named order leads to the issues when indexing in ES

opened 11:20AM - 17 Jun 25 UTC

rolfschmidt

bug verified

### Used Zammad Version 6.4 ### Environment - Installation method: [e.g. sour…ce, package] - Operating system (if you're unsure: `cat /etc/os-release` ): [e.g. debian 10.4, ubuntu 20.04] - Database + version: [e.g. postgresql 9.3] - Elasticsearch version: [e.g. 7.17] - Browser + version: [e.g. chrome 83, safari 14, firefox 105] ### Actual behaviour Error on search index rebuild ### Expected behaviour No errors https://community.zammad.org/t/error-when-creating-elasticsearch-index-with-custom-fields/17901 ### Steps to reproduce the behaviour - Add tree select for tickets called "order" - Try to rebuild your search index ### Support Ticket _No response_ ### I'm sure this is a bug and no feature request or a general question. no