Js console error and non working comments

Infos:

  • Used Zammad version: 3.3.x
  • Used Zammad installation source: package
  • Operating system: Ubuntu 16.04
  • Browser + version: Firefox 77.0.1

Expected behavior:

Add comment as Ticket owner and Agent (not customer) to ticket should work.

Actual behavior:

I get following JS error in the console of multiple browsers. Service restarts and so on doesn’t change anything. Upgrade from 3.1.x to 3.3.x doesn’t help either

uncaught exception: Unknown operator: is,is application-59e16ce374c83cd20e16d0fa77e0b6130d89e904abb254b65b79b9daa0b96a77.js:25:9949

_selectorMatch …/assets/application-59e16ce374c83cd20e16d0fa77e0b6130d89e904abb254b65b79b9daa0b96a77.js:25

selector …/assets/application-59e16ce374c83cd20e16d0fa77e0b6130d89e904abb254b65b79b9daa0b96a77.js:25

submit …/assets/application-59e16ce374c83cd20e16d0fa77e0b6130d89e904abb254b65b79b9daa0b96a77.js:47

l …/assets/application-59e16ce374c83cd20e16d0fa77e0b6130d89e904abb254b65b79b9daa0b96a77.js:47

s https://support.fritz-lange.de/assets/application-59e16ce374c83cd20e16d0fa77e0b6130d89e904abb254b65b79b9daa0b96a77.js:8

dispatch …/assets/application-59e16ce374c83cd20e16d0fa77e0b6130d89e904abb254b65b79b9daa0b96a77.js:2

handle …/assets/application-59e16ce374c83cd20e16d0fa77e0b6130d89e904abb254b65b79b9daa0b96a77.js:2

Steps to reproduce the behavior:

I can’t tell what was happening. From one day to an other we had this problem

I can’t find any issue. In logs everything seems to work well. The PUT Request for the missing Article I could identify. So it seems the client is sending the update to the backend.

I, [2020-06-15T15:59:50.746526 #24751-70028961841480] INFO – : Started PUT “/api/v1/taskbar/30141” for xxx.xxx.xxx.xxx at 2020-06-15 15:59:50 +0200
I, [2020-06-15T15:59:50.753992 #24751-70028961841480] INFO – : Processing by TaskbarController#update as JSON
I, [2020-06-15T15:59:50.754142 #24751-70028961841480] INFO – : Parameters: {“key”=>“Ticket-9137”, “client_id”=>“123”, “callback”=>“TicketZoom”, “state”=>{“ticket”=>{}, “article”=>{“body”=>"[FILTERED]", “from”=>“xxxxxx”, “ticket_id”=>“9137”, “content_type”=>“text/html”, “sender_id”=>“1”, “type_id”=>“10”, “form_id”=>“965776769”}}, “params”=>{“ticket_id”=>“9137”, “init”=>true}, “prio”=>5, “notify”=>false, “active”=>false, “attachments”=>[], “updated_at”=>“2020-06-15T07:24:09.351Z”, “id”=>“30141”}
I, [2020-06-15T15:59:50.765014 #24751-70028961841480] INFO – : Enqueued UserDeviceLogJob (Job ID: e6a7c911-daf3-4f1a-a427-c0c0d1603223) to DelayedJob(default) with arguments: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36”, “xxx.xxx.xxx.xxx”, 7, “-1486906726”, “session”
I, [2020-06-15T15:59:50.800230 #24751-70028961841480] INFO – : Completed 200 OK in 46ms (Views: 0.6ms | ActiveRecord: 8.3ms)

the client / frontend hang’s in update state and no new article is created.

Ticket.find_by(id:‘9137’).article_ids
=> [38011]

rails returns only the original ticket article but not the new one, as I expected.

i can’t find the error - has anyone an idea where to look next?

Personally I’d guess that the issues you mention are two different issues - at least it feels that way.
Regarding your second article:

Above logfile part looks totally fine, ensure that ticket with ID 9137 really doesn’t have articles and, if it doesn’t that your database wasn’t having a bad day. Looks like database was telling Zammad “yep, written!” but didn’t do that. But that’s fishing in the dark as we have no information on your database setup.

This sounds like the application is trying to communicate with your Zammad Server but fails for whatever reason. This can have various reasons:

  • database is overloaded or you’re having deadlocks (deadlocks can be found by searching for that term in production.log)
    • other database errors like max_connections can be found there too. Maybe your database server limits the remaining connections for administrative use
  • ensure your websocket server is up and running
    • If you reload Zammad and /ws upgrade fails, you’re using the ajax fallback which draws much more load on your system which should be avoided
  • Have a taste of my favorite greps:
    • grep "WARN" logfile
    • grep "ERROR" logfile
  • Ensure your host meets the minimum requirenments of Zammad https://docs.zammad.org/en/latest/prerequisites/hardware.html
  • ensure that all migrations have run (zammad run rake db:migrate to doublecheck)
  • Ensure that you have enough Puma-Workers for your agents and if nobody works but puma runs on 100%, it proberbly hangs

That’s it with my fishing in the dark for now

Hello,

First of all thanks a lot @MrGeneration for your reply. I’m sorry for the big information gap I left behind - caused by the JS error I thought the problem is located in the frontend section, so I didn’t post the exact server setup.

Setup

Zammad Server:

  • Single machine installation

  • VM on ESXi

  • Ubuntu 16.04

  • PostgreSQL 10.12

  • Apache 2.4.29 ReverseProxy

  • Zammad 3.3.x

The machine / VM is well dimensioned with:

  • 10 GB RAM

  • 4 vCPU

PostgreSQL:

  • max_connections: 2000 (limit not reached)

  • Size of temporary file: 3032 kB (rewrite from temp DB to master seems to work)

  • Deadlock Counter for last 2 month: 16 (I think that is acceptable)

On the setup nothing had changed for a while and the error started for all users at once.

If I look at the performance values of the machine and especially PostgreSQL, there is no heavy usage of resources.

At the moment I think, caused by your hint with the AJAX fallback, the problem is somewhere around the /ws endpoint or socket server itself. /var/log/zammad/websocket-server_err.log is totally empty. In ‘tail’ I can see backend is telling that there are no AJAX clients and a bunch of websocket using clients. In my opinion that says the socket is working properly.

The production log doesn’t show up any relevant errors. For days I ‘tail’ now it’s content, but nothing significant show up.

The stranged thing is, the closing of tickets and setting of other ticket attributes works well. Only the writing or saving of articles is not working.

Best regards

Honestly, I don’t see why updates on a table should work and inserts don’t.
This doesn’t seemt o be related to Zammad and I can’t help further on that regard.

You could however fire my favorite greps I mentioned above:

Also a grep "Deadlock" logfile might help.

Logfile wants to be your production log in case that wasn’t clear yet. :slight_smile:

Hey there,

I got your point but in the logs there is no error. production.log and websocket_err.log are clean of errors and warnings. The only error popping up is the one in the frontend JS,

The process is atm:

  • Post an article (mail, comment or internal comment doesn’t matter)
  • JS exception is thrown in Ticket._selectorMatch: Unknown operator: is,is
  • Backend recieves PUT request and obviously handle it without error
  • In Frontend nothing happens only ‘submit’ is greyed out (no feedback over established socket? - socket is definitv ‘online’)
    .* May be the socket handling in frontend is aborted at the exception point?
  • The only way to work further is to discard changes by closing ticket tab and answer warning with ‘discard changes’ or push the button ‘discard changes’ on bottum of the ticket view.
  • PUT request is send to backend wich in my opinion is discarding the changes. I don’t know the exact internal process.

Thinking about the process is hardining my opinion that the JS error is the cause of the problem. Somehow it is realted to the socket server. While the debugging I saw my client switching on the socket side to AJAX, although the socket was establshed.

Is this issue constantly happening or only during high load situations?
This might be a performance issue of some sort. I currently suspect you’re using Firefox when having this issue.

You could try:
zammad run rake assets:precompile to compile the javascript entirely from stretch.
If a clean reload of the application keeps providing this issue and it happens no matter of the situation, please provide a list of addons you’re using in that browser.

If possible also try another browser or try to do the same with completely deactivated addons.


If this only happens in high duty situations, please provide the output of the following commands in such a moment:

open a rails console (zammad run rails c)

Sessions.list.count
Overview.count
Ticket.count
Ticket::Article.count
User.count
Group.count
Delayed::Job.count

Also provide the output of zammad config:get WEB_CONCURRENCY.

Thanks

Hey there,

Unfortunately the issue is happening constantly at all users working with the system. In the meantime I had tried multiple Browsers: Chromium 73.0.3683.75, Chrome 83.0.4103.116, Firefox 77.0.1. Also I tried it with Chrome without plugins enabled and the result is the same.

Last week I had tried the following repair strategies for the assets without success:

  1. Asset cleanup
    1.1. zammad run rake assets:clean
    1.2. zammad run rake assets:precompile
  2. zammad run rake tmp:clear + Asset cleanup
  3. Reinstallation of of assets
    3.1. Download of 3.4. source
    3.2. rm -R /opt/zammad/app/assets/*
    3.3. rm -R /opt/zammad/public/assets/*
    3.2. cp -R /tmp/zammad-3.4.0/app/assets/* /opt/zammad/app/assets
    3.3. cp -R /tmp/zammad-3.4.0/public/assets/* /opt/zammad/public/assets
    3.4. chown -R zammad:zammad /opt/zammad
    3.5. zammad run rake assets:precompile
    3.6. service zammad restart

Your requested facts are the following:

Sessions.list.count
=> 12
Overview.count
=> 22
Ticket.count
=> 9113
Ticket::Article.count
=> 38405
User.count
=> 1651
Group.count
=> 16
Delayed::Job.count
=> 0

zammad config:get WEB_CONCURRENCY
=> 3

Best regards

This is an unchanged vanilla installation of Zammad that’s coming from us, Zammad GmbH provided via packager.io …?

Sounds very odd, because the javascript should be working just fine if it’s not changed or manipulated.
Ensure that no antivirus or “security suite” is playing man in the middle and potentially changing javascript files during loading it.

This could explain why parts of Zammad don’t work.
I’m afraid that this is as far as I can help you with, because anything else goes deep down to your client (doesn’t seem to be a server/application issue) which is out of scope for me.

I’m sorry.

Edit:
What you could do, do ensure your host is not at fault and your client isn’t as well - you could spin up a free trial instance on our hosted setup and check if you experience the issue there.