Search API performance issue with Elastic Search enabled

[cohnen]

Infos:

• Used Zammad version: 2.9.0
• Used Zammad installation source: (source, package, …) package
• Operating system: Ubuntu 18.04.2 LTS
• Browser + version: Chrome Version 74.0.3729.169 (Official Build) (64-bit)

Expected behavior:

Search API - /api/v1/search?query=test+yo&limit=10&_=1559091877904
Should return a response in ms. time using Elastic Search.
In our logs I can check (And even manually with a direct curl query to ES server) that Elastic Search response is pretty fast i.e. Performed SearchIndexJob (Job ID: e0fd01be-feca-4b82-92ee-f6e19c3b8277) from DelayedJob(default) in 11.9ms
Performed SearchIndexJob (Job ID: a65187b6-26c9-4aee-a98b-d72c5f8f811e) from DelayedJob(default) in 12.0ms

But final query result to search (That it looks in code is a combination of 3 ES queries of ~ 10ms. End up on a response time of > 1second. Is that normal?

Our server is a box with 8 cores + 32GB Ram. ES with dedicated 8GB and right now I’m testing on a no users box, just myself and with a ticket collection of around 5000 items. Launched now with a web concurrency of 4, and rest as default by package installation.

Is there something strange here? Or is it normal search API response times over a second?

Thanks and Regards.

Actual behavior:

Steps to reproduce the behavior:

[MrGeneration]

Maybe?
I’ll leave this here: Asking for help on community

If in case you’re still wonder what we need and want: The used commands on your API and on how you authenticate (basic auth or Token).

It otherwise is impossible to help you and determine if those response times are normal.

[cohnen]

Ok and first sorry so much if was not right my support request. I was trying to give proper details.

Before write it here, I was trying to find a normal benchmark for search API and I was not able to find anything. When I mention API, we are not consuming it externally, the “issue” is with Zammad search UI itself, so basic auth.

It’s something noticeable in our Zammad dashboard, search usually takes 1 to 2 seconds and you can see the loading circle during that.
I was just wondering if that was “normal” for most of the installations or if I was suffering and normally takes ms and not more than a second.

Sample of curl from browser request:
curl ‘https://agentsnew.triporate.com/api/v1/search?query=cohnen&limit=10&_=1559292756304’ -H ‘Cookie: HIDDEN_GUESS_NOT_NEEDED’ -H ‘Content-Type: application/json’ -H ‘Accept: application/json, text/javascript, /; q=0.01’ -H ‘Referer: https://agentsnew.triporate.com/’ -H ‘X-Requested-With: XMLHttpRequest’ -H ‘Connection: keep-alive’ --compressed

Response time: 2.43s. 48.2KB Waiting TTFB 2.34s

Thanks and Regards.

[MrGeneration]

First of all, I apologise for my rather harsh reply.
Seems like I didn’t have enough coffee in my veins

Now to your issue:

Thanks for the input!
That really helps further.

First of all:
I’d like to suggest using API-Tokens. This is the fastest and more realiable way!

You’re indirectly striked by this bug: https://github.com/zammad/zammad/issues/2431

In detail (so why this is being troublesome when using basic authentication)

If you’re logging in via API, Zammad will update the last_login field of your API user.
This update will cause Zammad to fire an elasticsearch index update for that user.

So far so good. The more request you fire to the API within a short period of time, the worse this gets.
I’ve seen anything between 1-20 seconds for 5 small requests (up to missing replies by Zammad).

Now to make this even more interesting:
If your API user is member of an organisation with, say, 10 members, it will not just update and reindex your api user, but the organization PLUS the other 9 members.

This can cause such high loads on the system, that the API gets either extremly slow or unresponsive.

super short:

Use API-Tokens
Now honestly, it will speed up amazingly and will also have the benefit you don’t need to change passwords in every script using that account after a password change.