Ticket Overviews not automatically refreshing

A couple months ago I migrated our Zammad deployment from an ancient EKS cluster (k8s version 1.21) to a new EC2 and RDS instance in AWS. We went from version 5.1.x deployed through Helm to 6.2.0-1705322325.8ff65214.jammy package install on Ubuntu 22.04 LTS. It went smoothly and is faster and better than ever, but the CSRs have complained that their tickets view, no longer automatically refreshes. I’ve been trying for the past few days to figure out what the problem is, but have come up empty handed and would appreciate some assistance on getting this resolved.

Things I’ve tried:

  • Checked for disabled schedulers. zammad run rails r "p Scheduler.where(active: false).pluck(:name)"
    Output: []
  • Checked the overview count. zammad run rails r "p Overview.count"
    Output: 41
  • Checked for delayed jobs. zammad run rails r "p Delayed::Job.count"
    Output: 0
  • Checked web concurrency zammad config:get WEB_CONCURRENCY
    Output: 2
  • Checked job concurrency zammad config:get ZAMMAD_SESSION_JOBS_CONCURRENT
    Output: 2
  • Checked scheduled job workers zammad config:get ZAMMAD_PROCESS_SCHEDULED_JOBS_WORKERS
    Output: 2
  • Checked delayed job workers zammad config:get ZAMMAD_PROCESS_DELAYED_JOBS_WORKERS
    Output: 2
  • Watched the network traffic through a browser console for WS errors and did not find any. Just a Status 101 for switching protocols. I saw the same thing on our old deployment so I’m assuming this is fine.
  • I know something is automatically refreshing because notifications on the bird icon show up just fine without refreshing the page as tickets are added and updated.
  • I’ve checked CPU and memory useage on the EC2 instance and at idle, its around 30% at idle and 50-60% during the day when people are using it.
  • I’ve checked to make sure we are not running out of storage and we are good there with plenty to use and >50% free.
  • I’ve rebooted the instance a few times since the deployment and that does not seem to fix it.

System Specs-
Both are running in the same VPC and Region.

EC2:

  • Type: r7a.xlarge
  • CPU: 4
  • Memory: 30G
  • Storage: 120G

RDS:

  • Class: db.m5.large
  • vCPU: 2
  • Memory: 8G
  • Storage: 60G
  • Engine: PostresSQL
  • Engine Version: 15.5

Hopefully that is enough information to start. I’m pretty new to a lot of this, so I apologize for any missed information or troubleshooting steps.

Hi @Luffy

Very SUS… Actually I don’t have a clue. Your pre-research is very exemplary :+1:

I don’t have any knowledge about Kubernetes, but I can try to give you some ideas:

  1. Do you see events flying around when you select overviews? (F12 → Network tab → Websocket connections [reload needed])

  1. Is your scheduler running in general?
  • Are pending tickets flipped to open?
  • Are Admin → Scheduler executed?
  1. What about your logs production.log are there any ERROR lines?

grep ERROR -A10 /opt/zammad/log/production.log

  1. How does your Jobs Scheduler look like:

rails r "pp Scheduler.find_by(method: 'Sessions.jobs')"

  1. Does it have any impact if you unset your ENVs and restart?

ZAMMAD_SESSION_JOBS_CONCURRENT
ZAMMAD_PROCESS_DELAYED_JOBS_WORKERS

Hey @rolfschmidt

Thanks for for taking a look at this. We are actually running this with an Ubuntu package install now because the k8s cluster we have was experiencing too many issues with zammad. Here is what I was able to get for you:

  1. Events when viewing the WS connections.
    • It just repeats this same thing and about every couple minutes gets a close code of 1006 and repeats:

  1. Is the scheduler running in general. Are pending tickets flipped to open. Are Admin > Scheduler executed:

    • I’m not sure, maybe? The monitoring > current status is showing no issues
    • Pending tickets are flipped to open when our ticket master goes through them.
    • There is nothing in the scheduler section of the settings
  2. Error lines in production log.
    There are quite a few, but nothing that I would know how to fix.

	E, [2024-03-21T15:24:07.324194#1146-382526400] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:24:07.590811#1146-382561200] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:24:07.842485#1146-382769260] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:24:08.067293#1146-382674160] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:24:08.123562#1146-382769340] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:24:08.435867#1147-383171660] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:24:08.621554#1146-382612480] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:24:13.289680#1147-383051620] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:24:13.917119#1146-382561200] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:24:18.021129#1146-382674160] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:24:19.224116#1146-382769260] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:24:32.833037#1146-382561320] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:24:34.236927#1146-382854580] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:24:40.114668#1146-382674160] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:24:40.808651#1146-382711700] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:24:41.054942#1147-383174860] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:24:41.390753#1147-383363820] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:24:41.526294#1147-383212760] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:24:52.396158#1146-382769340] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:24:53.001641#1146-382408140] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:24:58.183906#1146-382786780] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:24:58.413464#1146-382711700] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:25:05.305819#1147-383316820] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:25:05.564939#1147-382671280] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:25:11.324505#1146-382674160] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:25:11.479808#1147-383316820] ERROR -- : No route matches [PUT] /testing-put.txt (ActionController::RoutingError)
	E, [2024-03-21T15:25:11.538802#1146-382769260] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:25:11.588048#1146-382674360] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:25:11.775520#1147-383331300] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:25:12.062133#1147-383171660] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:25:12.353649#1147-383172160] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:25:12.832863#1146-382612520] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:25:15.400585#1147-383212760] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:25:15.566816#1146-382487920] ERROR -- : No route matches [GET] /testing-put.txt (ActionController::RoutingError)
	E, [2024-03-21T15:25:16.679536#1146-382561200] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:25:19.996351#1146-382854420] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:25:20.225981#1146-382711700] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:25:21.564989#1146-382699500] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:25:21.682205#1147-382825400] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: )
	E, [2024-03-21T15:25:29.694514#1146-382535580] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:25:47.231271#1147-383172160] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:25:51.193983#1146-382769340] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:25:57.205309#1147-383303200] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:00.781332#1146-382854400] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:04.111123#1146-382408140] ERROR -- : No route matches [GET] /actuator/env (ActionController::RoutingError)
	E, [2024-03-21T15:26:05.597525#1146-382561320] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:09.412230#1147-383303200] ERROR -- : No route matches [GET] /env (ActionController::RoutingError)
	E, [2024-03-21T15:26:09.946890#1146-382854400] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:09.955441#1146-382678040] ERROR -- : No route matches [GET] /mydomain.zip (ActionController::RoutingError)
	E, [2024-03-21T15:26:09.984520#1146-382769560] ERROR -- : No route matches [GET] /zammad.mydomain.com.rar (ActionController::RoutingError)
	E, [2024-03-21T15:26:10.076484#1146-382854380] ERROR -- : No route matches [GET] /mydomain.com.rar (ActionController::RoutingError)
	E, [2024-03-21T15:26:10.095549#1146-382786780] ERROR -- : No route matches [GET] /zammad.mydomain.com.zip (ActionController::RoutingError)
	E, [2024-03-21T15:26:10.122576#1146-382674360] ERROR -- : No route matches [GET] /mydomain.com.tar.gz (ActionController::RoutingError)
	E, [2024-03-21T15:26:10.128098#1146-382880160] ERROR -- : No route matches [GET] /mydomain.tar.gz (ActionController::RoutingError)
	E, [2024-03-21T15:26:10.529838#1146-382854400] ERROR -- : No route matches [GET] /mydomain.com.zip (ActionController::RoutingError)
	E, [2024-03-21T15:26:10.749239#1147-383174860] ERROR -- : No route matches [GET] /mydomain.rar (ActionController::RoutingError)
	E, [2024-03-21T15:26:10.752169#1147-383375920] ERROR -- : No route matches [GET] /zammad.mydomain.com.tar.gz (ActionController::RoutingError)
	E, [2024-03-21T15:26:15.963292#1147-382988420] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:16.499308#1147-383375920] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:16.737665#1147-382824580] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:17.032454#1146-382769160] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:20.032175#1147-382825400] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:22.991618#1147-383331300] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:25.975539#1146-382487920] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:28.653441#1146-382674360] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:28.884327#1146-382786780] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:29.074616#1146-382854580] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:29.277834#1146-382534560] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:29.475676#1146-382526400] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:29.884334#1146-382561320] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:34.258356#1146-382678040] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:43.086201#1147-383173200] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:48.678657#1146-382884040] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:53.801053#1147-383392900] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:26:58.495467#1146-382535580] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:27:01.331620#1147-383329960] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:27:02.976245#1147-383069140] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:27:04.330163#1146-382854380] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:27:05.584985#1146-382674260] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:27:07.050904#1146-382884040] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:27:07.329139#1146-382769160] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:27:08.053352#1146-382534560] ERROR -- : No route matches [GET] /api/v1 (ActionController::RoutingError)
	E, [2024-03-21T15:28:04.191595#1146-382871880] ERROR -- : No route matches [POST] / (ActionController::RoutingError)
	E, [2024-03-21T15:28:28.568687#1147-383069140] ERROR -- : No route matches [GET] /backup.sql (ActionController::RoutingError)
	E, [2024-03-21T15:28:29.177320#1147-383316820] ERROR -- : No route matches [GET] /database.sql (ActionController::RoutingError)
	E, [2024-03-21T15:28:29.767940#1146-382699500] ERROR -- : No route matches [GET] /db.sql (ActionController::RoutingError)
	E, [2024-03-21T15:28:30.357704#1146-382884040] ERROR -- : No route matches [GET] /dump.sql (ActionController::RoutingError)
	E, [2024-03-21T15:28:30.946823#1146-382880160] ERROR -- : No route matches [GET] /zammad.mydomain.com.sql (ActionController::RoutingError)
	E, [2024-03-21T15:28:36.083241#1147-383172160] ERROR -- : No route matches [GET] /wp-content/force-download.php (ActionController::RoutingError)
	E, [2024-03-21T15:29:00.556247#1147-383329960] ERROR -- : No route matches [GET] /%252e%252e%255c%252e%252e%255c%252e%252e%255c%252e%252e%255c%252e%252e%255c%252e%252e%255c%252e%252e%255c%252e%252e%255cwindows%255c%252e%255cwin%252eini (ActionController::RoutingError)
	E, [2024-03-21T15:31:11.965838#1146-382920600] ERROR -- : No route matches [POST] / (ActionController::RoutingError)
  1. Jobs Scheduler.
 #<Scheduler:0x00007f2178111a48
 id: 7,
 name: "Generate 'Session' data.",
 method: "Sessions.jobs",
 period: 60,
 running: 0,
 last_run: Fri, 09 Feb 2024 06:35:54.008000000 UTC +00:00,
 prio: 1,
 pid: "148780",
 note: nil,
 error_message: "",
 status: "ok",
 active: true,
 updated_by_id: 39,
 created_by_id: 1,
 created_at: Thu, 19 Mar 2020 21:02:31.057000000 UTC +00:00,
 updated_at: Fri, 09 Feb 2024 06:35:54.008000000 UTC +00:00,
timeplan: {}>
  1. Does it have any impact if you unset your ENVs and restart?
    • I can test this after hours today when its not in use

    • I actually just deployed a staging env that is an exact match of the prod and got to test this. It seems to have worked! I will test on prod tonight and report back tomorrow on if it solved the issue or not

    • EDIT: It did not work on the prod deployment for some reason

Hopefully this is useful to you

Based on the provided log, IMO, web sockets are not working in general and this is also the reason why the overviews are getting no updates.

@fliebe92 Any ideas on what I can start looking at? The Nginx config, Application Load Balancer, inbound/outbound ports allowed to the EC2, AWS certificate manager, etc…

I’ve been unable to find good resources on how to run this in AWS with a dedicated EC2 and RDS instance, though considering that the other 98% of the application seems to be working, I’m hoping its just a simple mistake on my part.

Nginx config might be interesting + a good starting point.

To me this looks like your package installation was done with a version prior to 6.0 and when it was updated, the needed changes to the nginx-config haven’t been done.

New Major Release: Zammad 6.0!

Hey @YetAnotherGerrit.

It appears to have the appended information that you mentioned.

Here is my Nginx config -

#
# this is the nginx config for zammad
#

upstream zammad-railsserver {
    server 127.0.0.1:3000;
}

upstream zammad-websocket {
    server 127.0.0.1:6042;
}

server {
    listen 80;
    listen [::]:80;

    # replace 'localhost' with your fqdn if you want to use zammad from remote
    server_name zammad.corporatetools.com;

    # security - prevent information disclosure about server version
    server_tokens off;

    root /opt/zammad/public;

    access_log /var/log/nginx/zammad.access.log;
    error_log  /var/log/nginx/zammad.error.log;

    client_max_body_size 50M;

    location ~ ^/(assets/|robots.txt|humans.txt|favicon.ico|apple-touch-icon.png) {
        expires max;
    }

    # legacy web socket server
    location /ws {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header CLIENT_IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_read_timeout 86400;
        proxy_pass http://zammad-websocket;
    }

    # action cable
    location /cable {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header CLIENT_IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 86400;
        proxy_pass http://zammad-railsserver;
    }

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header CLIENT_IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;

        # Change this line in an SSO setup
        proxy_set_header X-Forwarded-User "";

        proxy_read_timeout 300;
        proxy_pass http://zammad-railsserver;

        gzip on;
        gzip_types text/plain text/xml text/css image/svg+xml application/javascript application/x-javascript application/json application/xml;
        gzip_proxied any;
    }
}

Thanks!

Hi @Luffy. Do you have anything in place that blocks web sockets, e.g. a firewall?

Hey @fliebe92 ,

We have security groups with the following ingress and egress rules -

ingress:
allow 0.0.0.0/0 port 22 > port 22
allow 0.0.0.0/0 port 5432 > port 5432
allow 0.0.0.0/0 port 80 > port 80
allow 0.0.0.0/0 port 995 > port 995
allow 0.0.0.0/0 port 465 > port 465
allow 0.0.0.0/0 port 443 > port 443

egress:
allow all

We also have an application load balancer listening to port 443 that applies the certificate and forwards to a target group on port 80 for the nginx config and getting to the zammad.

Another idea: does your FQDN and HTTP type configured in Zammad match the URL you enter in the browser to reach the instance?

Base — Zammad Admin Documentation documentation

And what happens if you open Zammad in the mobile view https://yourhostname/mobile/ ? Do you get any error messages?

@YetAnotherGerrit

We just type in the domain and it automatically upgrades to https if not specified.

[1] pry(main)> Setting.get('fqdn') 
=> "zammad.example.com"
[2] pry(main)> Setting.get('http_type')
=> "https"

Also not seeing any errors in the network console when going to the mobile version. Just the same 101 status for WS switching protocols.

EDIT: Though the problem does not seem to be an issue on the mobile view. It refreshes just fine without doing it manually.

EDIT #2: Now the normal webview is doing the automatic refresh, though it takes about 4-5 minutes after a ticket is created to do so.

So I think this is resolved?