/api/v1/tickets/search?query=state.name:open suddenly stopped working

Infos:

Expected behavior:

  • Get tickets by search endpoint where state.name:open
  • Or get tickets by any filter option related to search endpoint.

Actual behavior:

Steps to reproduce the behavior:

Another considerations

This is happening with all requests I make to the /tickets/search?query= endpoint. All other requests are correct and work fine. I have a copy in a test environment and it works fine in that one, both are a clone of the other, except the test one hasn’t been interacted with in three days.

It’s very strange that this is happening because as I mentioned, the only thing I’ve been doing is working with the tickets in tests and deleting them in the Rails console with: zammad run rails r “Ticket.destroy_all”

Has anyone experienced this and do you have any idea how to solve it please?

Log from production.log

Payload size: 0M
E, [2024-09-26T21:35:32.284814#809-149040] ERROR -- : Unable to process GET request to elasticsearch URL 'http://127.0.0.1:9200/zammad_production_ticket_state/_update_by_query?conflicts=proceed&slices=auto&max_docs=1000'. Check the response and payload for detailed information:

Response:
#<UserAgent::Result:0x00007f1268d9c638 @success=false, @body=nil, @data=nil, @code="404", @content_type=nil, @error="No such file http://127.0.0.1:9200/zammad_production_ticket_state/_update_by_query?conflicts=proceed&slices=auto&max_docs=1000, 404!", @header={"x-elastic-product"=>"Elasticsearch", "warning"=>"299 Elasticsearch-7.17.21-d38e4b028f4a9784bb74de339ac1b877e2dbea6f \"Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.\"", "content-type"=>"application/json; charset=UTF-8", "content-length"=>"509"}>

Payload:
{"script":{"lang":"painless","source":"ctx._source.created_by=params.created_by","params":{"created_by":{"id":51,"organization_id":2,"login":"USER LOGIN STRING","firstname":"FIRST NAME","lastname":"LAST NAME","email":"EMAIL","image":"IMAGE","image_source":null,"web":"","phone":"","fax":"","mobile":"","department":"","street":"","zip":"","city":"","country":"","address":"","vip":false,"verified":false,"active":true,"note":"","last_login":"2024-09-26T17:05:07.360Z","source":"microsoft_office365","login_failed":0,"out_of_office":false,"out_of_office_start_at":null,"out_of_office_end_at":null,"out_of_office_replacement_id":null,"preferences":{"locale":"en-us","notification_config":{"matrix":{"create":{"criteria":{"owned_by_me":true,"owned_by_nobody":true,"subscribed":true,"no":false},"channel":{"email":true,"online":true}},"update":{"criteria":{"owned_by_me":true,"owned_by_nobody":true,"subscribed":true,"no":false},"channel":{"email":true,"online":true}},"reminder_reached":{"criteria":{"owned_by_me":true,"owned_by_nobody":false,"subscribed":false,"no":false},"channel":{"email":true,"online":true}},"escalation":{"criteria":{"owned_by_me":true,"owned_by_nobody":false,"subscribed":false,"no":false},"channel":{"email":true,"online":true}}}},"tickets_closed":0,"tickets_open":1,"intro":true},"updated_by_id":51,"created_by_id":1,"created_at":"2024-05-23T04:40:45.993Z","updated_at":"2024-09-26T21:24:03.345Z","fullname":"NAME"}}},"query":{"bool":{"must":[{"term":{"created_by_id":51}},{"range":{"created_by.updated_at":{"lt":"2024-09-26T21:24:03.345Z"}}}]}},"sort":{"id":"desc"}}

Payload size: 0M
I, [2024-09-26T21:35:32.298987#809-149040]  INFO -- : 2024-09-26T21:35:32+0000: [Worker(host:servicedesk pid:809)] Job SearchIndexAssociationsJob [905dc937-3748-49f1-82c5-951900e6b180] from DelayedJob(default) with arguments: ["User", 51] (id=166387) (queue=default) COMPLETED after 0.1679
I, [2024-09-26T21:35:35.185987#800-148860]  INFO -- : Started GET "/api/v1/tickets/search?query=state.name:open" for 172.28.7.101 at 2024-09-26 21:35:35 +0000
I, [2024-09-26T21:35:35.196125#800-148860]  INFO -- : Processing by TicketsController#search as */*
I, [2024-09-26T21:35:35.196223#800-148860]  INFO -- :   Parameters: {"query"=>"state.name:open"}
I, [2024-09-26T21:35:35.234434#800-148860]  INFO -- : Completed 200 OK in 38ms (Views: 0.3ms | ActiveRecord: 13.8ms | Allocations: 9073)
I, [2024-09-26T21:35:36.308299#809-149040]  INFO -- : 2024-09-26T21:35:36+0000: [Worker(host:servicedesk pid:809)] Job UserDeviceLogJob [b64e4c84-d218-474f-a53f-84279aef43fb] from DelayedJob(default) with arguments: ["PostmanRuntime/7.42.0", "172.28.7.101", 3, nil, "token_auth"] (id=166404) (queue=default) RUNNING
I, [2024-09-26T21:35:36.338795#809-149040]  INFO -- : 2024-09-26T21:35:36+0000: [Worker(host:servicedesk pid:809)] Job UserDeviceLogJob [b64e4c84-d218-474f-a53f-84279aef43fb] from DelayedJob(default) with arguments: ["PostmanRuntime/7.42.0", "172.28.7.101", 3, nil, "token_auth"] (id=166404) (queue=default) COMPLETED after 0.0304
I, [2024-09-26T21:35:40.909216#809-149020]  INFO -- : ProcessScheduledJobs running...
I, [2024-09-26T21:35:40.912492#809-149020]  INFO -- : Running job thread for 'Generate 'Session' data.' (Sessions.jobs) status is: sleep
I, [2024-09-26T21:35:40.912736#809-149020]  INFO -- : Running job thread for 'Check channels.' (Channel.fetch) status is: sleep
I, [2024-09-26T21:35:40.912802#809-149020]  INFO -- : Running job thread for 'Check 'Channel' streams.' (Channel.stream) status is: sleep
I, [2024-09-26T21:35:40.912855#809-149020]  INFO -- : Running job thread for 'Process ticket escalations.' (Ticket.process_escalation) status is: sleep
I, [2024-09-26T21:35:40.914162#809-149020]  INFO -- : Running job thread for 'Execute planned jobs.' (Job.run) status is: sleep
I, [2024-09-26T21:35:48.355165#809-149040]  INFO -- : 2024-09-26T21:35:48+0000: [Worker(host:servicedesk pid:809)] Job SearchIndexAssociationsJob [f8a015e0-b2f3-4674-afea-f7c3aaebec84] from DelayedJob(default) with arguments: ["User", 55] (id=166397) (queue=default) RUNNING
E, [2024-09-26T21:35:48.417639#809-149040] ERROR -- : Unable to process GET request to elasticsearch URL 'http://127.0.0.1:9200/zammad_production_chat_session/_update_by_query?conflicts=proceed&slices=auto&max_docs=1000'. Check the response and payload for detailed information:

Response:
#<UserAgent::Result:0x00007f12695cc088 @success=false, @body=nil, @data=nil, @code="404", @content_type=nil, @error="No such file http://127.0.0.1:9200/zammad_production_chat_session/_update_by_query?conflicts=proceed&slices=auto&max_docs=1000, 404!", @header={"x-elastic-product"=>"Elasticsearch", "warning"=>"299 Elasticsearch-7.17.21-d38e4b028f4a9784bb74de339ac1b877e2dbea6f \"Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.\"", "content-type"=>"application/json; charset=UTF-8", "content-length"=>"509"}>

The error that I can see in the log is related to ElasticSearch, which I think is not related or is it?

Hi @hacknel. Your Elasticsearch index is broken…

To rebuild it you need this command zammad run rake zammad:searchindex:rebuild. You can also run it in parallel if you have e.g. 8 cores zammad run rake zammad:searchindex:rebuild[8]. It can take a long time, depending on your ticket count.

Thank you very much for the recommendation to try to do the reindex.

I have already done the index rebuild but I still have the problem, elastic search does not generate any error now but that endpoint still does not respond at all, I attach the log:

Log from request (production.log)

I, [2024-09-27T15:20:20.642025#45300-151340]  INFO -- : Started GET "/api/v1/tickets/search?query=state.name:open" for 172.28.7.101 at 2024-09-27 15:20:20 +0000
I, [2024-09-27T15:20:20.654321#45300-151340]  INFO -- : Processing by TicketsController#search as */*
I, [2024-09-27T15:20:20.654408#45300-151340]  INFO -- :   Parameters: {"query"=>"state.name:open"}
I, [2024-09-27T15:20:20.691883#45300-151340]  INFO -- : Completed 200 OK in 37ms (Views: 0.3ms | ActiveRecord: 9.1ms | Allocations: 9038)
I, [2024-09-27T15:20:24.176231#45310-149060]  INFO -- : 2024-09-27T15:20:24+0000: [Worker(host:servicedesk pid:45310)] Job UserDeviceLogJob [1eb0c03a-8a2f-43d9-b03b-628773f03e91] from DelayedJob(default) with arguments: ["PostmanRuntime/7.42.0", "172.28.7.101", 3, nil, "token_auth"] (id=166653) (queue=default) RUNNING
I, [2024-09-27T15:20:24.208440#45310-149060]  INFO -- : 2024-09-27T15:20:24+0000: [Worker(host:servicedesk pid:45310)] Job UserDeviceLogJob [1eb0c03a-8a2f-43d9-b03b-628773f03e91] from DelayedJob(default) with arguments: ["PostmanRuntime/7.42.0", "172.28.7.101", 3, nil, "token_auth"] (id=166653) (queue=default) COMPLETED after 0.0321
I, [2024-09-27T15:20:28.404024#45300-153460]  INFO -- : Started GET "/api/v1/tickets/search?query=state.name:open" for 172.28.7.101 at 2024-09-27 15:20:28 +0000
I, [2024-09-27T15:20:28.416608#45300-153460]  INFO -- : Processing by TicketsController#search as */*
I, [2024-09-27T15:20:28.416686#45300-153460]  INFO -- :   Parameters: {"query"=>"state.name:open"}
I, [2024-09-27T15:20:28.450323#45300-153460]  INFO -- : Completed 200 OK in 34ms (Views: 0.2ms | ActiveRecord: 12.8ms | Allocations: 9040)
I, [2024-09-27T15:20:28.822980#45310-149080]  INFO -- : ProcessScheduledJobs running...
I, [2024-09-27T15:20:28.825845#45310-149080]  INFO -- : Running job thread for 'Check 'Channel' streams.' (Channel.stream) status is: sleep
I, [2024-09-27T15:20:28.825965#45310-149080]  INFO -- : Running job thread for 'Generate 'Session' data.' (Sessions.jobs) status is: sleep
I, [2024-09-27T15:20:28.826166#45310-149080]  INFO -- : Running job thread for 'Process ticket escalations.' (Ticket.process_escalation) status is: sleep
I, [2024-09-27T15:20:28.826228#45310-149080]  INFO -- : Running job thread for 'Check channels.' (Channel.fetch) status is: sleep
I, [2024-09-27T15:20:28.827378#45310-149080]  INFO -- : Running job thread for 'Execute planned jobs.' (Job.run) status is: sleep
I, [2024-09-27T15:20:32.220913#45310-149060]  INFO -- : 2024-09-27T15:20:32+0000: [Worker(host:servicedesk pid:45310)] Job UserDeviceLogJob [0591795e-5e95-45e5-8f82-cd27adda8959] from DelayedJob(default) with arguments: ["PostmanRuntime/7.42.0", "172.28.7.101", 3, nil, "token_auth"] (id=166654) (queue=default) RUNNING
I, [2024-09-27T15:20:32.251537#45310-149060]  INFO -- : 2024-09-27T15:20:32+0000: [Worker(host:servicedesk pid:45310)] Job UserDeviceLogJob [0591795e-5e95-45e5-8f82-cd27adda8959] from DelayedJob(default) with arguments: ["PostmanRuntime/7.42.0", "172.28.7.101", 3, nil, "token_auth"] (id=166654) (queue=default) COMPLETED after 0.0305

I’ve been trying a few things:

  • Upgraded my zammad instance and I’m now on version 6.3.1-1727436232.f6b780c1.jammy
  • Migrated the database to Postgresql, the whole process completed successfully.
  • Upgraded my ubuntu 22.04 version and all packages to the latest versions.

I can’t think of anything else to try :frowning:

I finally found the error…

It turns out that yesterday while fixing user permissions I removed the group memberships (client and agent) from the user from which the token was created to interact with the API, leaving him only the administrator role.

Be careful when removing those permissions because if the user from which the token was created does not have the permissions, a silent error will be generated like the one I mentioned just now.

In the documentation, if possible, they should specify that the user must have all the memberships to the groups from which he tries to obtain information, I will try to add it to the documentation myself.

Thank you very much for your attempts to help, I hope you are doing very well.

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.