Zammad behind two apache24 reverse proxies

Infos:

  • Used Zammad version: 2.4.0
  • Used Zammad installation source: source
  • Operating system: ubuntu 18.04
  • Browser + version: chrome 66

Goal

I’m trying to run zammad behind two apache24 reverse proxies:

  • The outer reverse proxy terminated the HTTPS connections and forwards all traffic to the inner reverse proxy
  • The inner reverse proxy forewards the traffic to the zammad processes and serves static content
  • For the inner reverse proxy, I basically used contrib/apache2/zammad.conf

Observations

On the surface, everything seems to work OK.

Issues

Opening the browser console shows lots of errors:

application-100164b3…a42086aa426f24.js:3 [Deprecation] Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check https://xhr.spec.whatwg.org/.
application-100164b3…42086aa426f24.js:21 |  _____                                    _
| / _  / __ _ _ __ ___  _ __ ___   __ _  __| |
| \// / / _` | '_ ` _ \| '_ ` _ \ / _` |/ _` |
|  / //\ (_| | | | | | | | | | | | (_| | (_| |
| /____/\__,_|_| |_| |_|_| |_| |_|\__,_|\__,_|
|
| Hi there, nice to meet you!
|
| Visit http://zammad.com/jobs to learn about our current job openings.
|
| Your Zammad Team
|
blob:https://zammad.…a0-19d5b2e99277:164 WebSocket connection to 'wss://zammad.daemons-point.com/ws' failed: Error during WebSocket handshake: Unexpected response code: 503
blob:https://zammad.…a0-19d5b2e99277:164 WebSocket connection to 'wss://zammad.daemons-point.com:6042/' failed: Error during WebSocket handshake: Unexpected response code: 503
/api/v1/message_receive:1 Failed to load resource: the server responded with a status of 503 (Service Unavailable)
/api/v1/text_modules…e&_=1525963505551:1 Failed to load resource: the server responded with a status of 503 (Service Unavailable)
application-100164b3…42086aa426f24.js:21 App.Model(error)               | error Service Unavailable api/v1/text_modules/?full=true
/api/v1/tickets/4?al…e&_=1525963505553:1 Failed to load resource: the server responded with a status of 503 (Service Unavailable)
/api/v1/message_send:1 Failed to load resource: the server responded with a status of 503 (Service Unavailable)
/api/v1/tickets/5?al…e&_=1525963505556:1 Failed to load resource: the server responded with a status of 503 (Service Unavailable)
/api/v1/tickets/7?al…e&_=1525963505558:1 Failed to load resource: the server responded with a status of 503 (Service Unavailable)
/api/v1/ticket_custo…2&_=1525963505562:1 Failed to load resource: the server responded with a status of 503 (Service Unavailable)
/api/v1/ticket_stats…5&_=1525963505564:1 Failed to load resource: the server responded with a status of 503 (Service Unavailable)
/api/v1/ticket_stats…2&_=1525963505565:1 Failed to load resource: the server responded with a status of 503 (Service Unavailable)
/api/v1/users/guess:…e&_=1525963505567:1 Failed to load resource: the server responded with a status of 404 (Not Found)
application-100164b3…42086aa426f24.js:21 App.Model(error)               | error Not Found api/v1/users/guess:Por?full=true
/api/v1/ticket_stats…6&_=1525963505568:1 Failed to load resource: the server responded with a status of 503 (Service Unavailable)
```

Is there a way to prevent these from happening?

Please provide configuration files (IPs or hostnames don’t matter). Also attach the apache error log please, nobody can help you with this little set of information.

Also, this sample file for apache may help you:

We need to find the source of “503” error.

Thanks for looking into the issue.

  • zammad_ssl.conf doesn’t help me, I’ve used contrib/apache2/zammad.conf for the inner reverse proxy.
    The only difference is the line containing the ServerName. I use my server name there
  • I think the issue comes from the outer apache config. I have to add a special line for handling the /ws context:
    • ProxyPass /ws ws://{myhost}:6042/
  • Omitting the line or trying something like `ProxyPass /ws http://{myhost}/ws’ does not work
  • I don’t like the approach, since traffic to /ws bypasses the 2nd apache reverse proxy

Now to the files you requested:

The outer revproxy:

<VirtualHost *:443 *:6042>
	ServerName  xxxx
	ServerAdmin  yyyy
	DocumentRoot /var/www/html

	ErrorLog ${APACHE_LOG_DIR}/zammad-error.log
	CustomLog ${APACHE_LOG_DIR}/zammad-access.log combined

        # ... lots of ssl options

        #RewriteEngine Off
        ProxyRequests Off
        ProxyVia Off
        ProxyPreserveHost On
        AllowEncodedSlashes NoDecode
	<Proxy *>
		Order deny,allow
		Allow from all
	</Proxy>

        #ProxyPass        /ws ws://{myhost}:6042/
        ProxyPass        /   http://{myhost}/
        ProxyPassReverse /   http://{myhost}/

        <Location />
                Order deny,allow
                Allow from a.b.c.d
                Deny from all
                RequestHeader set X-Forwarded-Proto "https"
        </Location>
</VirtualHost>

The inner revproxy - see contrib/apache2/zammad.conf

The outer error.log - nothing

The inner error log:

[Thu May 10 17:37:12.177494 2018] [mpm_event:notice] [pid 7927:tid 140511715433408] AH00489: Apache/2.4.29 (Ubuntu) configured -- resuming normal operations
[Thu May 10 17:37:12.177661 2018] [core:notice] [pid 7927:tid 140511715433408] AH00094: Command line: '/usr/sbin/apache2'
[Thu May 10 17:37:47.962378 2018] [proxy:warn] [pid 7928:tid 140511381804800] [client 10.2.2.1:52346] AH01144: No protocol handler was valid for the URL /ws (scheme 'ws'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.
[Thu May 10 17:45:23.074597 2018] [proxy:warn] [pid 7929:tid 140511499237120] [client 10.2.2.1:52500] AH01144: No protocol handler was valid for the URL /ws (scheme 'ws'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.
[Thu May 10 17:49:03.226918 2018] [proxy:warn] [pid 7929:tid 140510391957248] [client 10.2.2.1:52602] AH01144: No protocol handler was valid for the URL /ws (scheme 'ws'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.
[Thu May 10 17:54:17.587620 2018] [proxy:warn] [pid 7928:tid 140511365019392] [client 10.2.2.1:52734] AH01144: No protocol handler was valid for the URL /ws/ (scheme 'ws'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.
[Thu May 10 17:54:25.004382 2018] [proxy:warn] [pid 7928:tid 140510308062976] [client 10.2.2.1:52742] AH01144: No protocol handler was valid for the URL /ws/ (scheme 'ws'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.
[Thu May 10 18:07:46.843852 2018] [proxy:warn] [pid 7929:tid 140509846693632] [client 10.2.2.1:53342] AH01144: No protocol handler was valid for the URL /ws (scheme 'ws'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.
[Thu May 10 18:07:51.545295 2018] [proxy:warn] [pid 7929:tid 140509855086336] [client 10.2.2.1:53344] AH01144: No protocol handler was valid for the URL /ws (scheme 'ws'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.
[Thu May 10 18:07:54.888356 2018] [proxy:warn] [pid 7929:tid 140511532807936] [client 10.2.2.1:53346] AH01144: No protocol handler was valid for the URL /ws (scheme 'ws'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.
(END)

The modules “proxy”, “proxy_http” and “proxy_wstunnel” are loaded within the inner revproxy.

Sorry for the delay…

Personally, even on the outer proxy I’d remove *:6042 as you don’t need it there.

Other than that, you’re missing one modules:

  • proxy
  • proxy_http
  • proxy_wstunnel
  • proxy_html

that’s atleast what I need on my suse to run the proxy cleanly with apache.
Could you check if that helps?

No problem. Actually, you’re pretty fast…

I removed it, works as before. I added it a couple of days ago since I’ve seen error within the browser console related to failing connections to port 6042.

blob:https://zammad.…a0-19d5b2e99277:164 WebSocket connection to 'wss://zammad.daemons-point.com:6042/' failed: Error during WebSocket handshake: Unexpected response code: 503

No, I don’t think proxy_html is required. It is used to rewrite links within
the response body. I’ll give it a try anyway and report my results.

Yes, I’ try and report. Thanks a lot! Best regards, Uli

Adding proxy_html didn’t change anything.

  • I still need special treatment of /ws on the outer proxy
  • I still do have to bypass the inner proxy for /ws
  • The logs do look absolutely similar

So I removed proxy_html again.

No Problem, was worth a try.

What’s odd is that your request is looking for Port 6042 - but Websockets (ws://) are Port 80 and 443:

Let’s comment that back in, but remove the port :6042

Did you change anything in the zammad-Setup? Like Proxy-Server or something like that? My Websockets are running via 443 :confused:

We have a similar setup where we have Zammad + NGINX running on an app server and a separate reverse proxy running on NGINX which does HTTPS termination on a separate host.

We needed to set the Upgrade header, and the header Connection to Upgrade
Because otherwise the edge reverse proxy will try to open a web socket connection to the other reverse proxy, and you don’t want that. This is a snippet from our NGINX config of the edge reverse proxy.

  location /ws/ {
    proxy_set_header Host   "xxx";
    proxy_set_header X-Original-URI $request_uri;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host $host:443;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_http_version 1.1;
    ....
}

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.