All of a sudden 502 Bad Gateway

Technical assistance
1

Infos:

So we installed zammad a while ago and it always worked like charm until yesterday when we suddenly got a 404 at first and than (without touching anything) a 502. I’ve seen another Post on here which is resolved but any of the steps in that post didn’t help me anything at all.

  • Not sure, how can i find out?
  • Used Zammad installation source: deb
  • Operating system: Ubuntu 18.04
  • Browser + version: Any

Expected behavior:

  • To work again like bevor

Actual behavior:

  • A 502 Bad Gateway

Steps to reproduce the behavior:

  • i don’t know

Error log from /var/log/nginx/zammad.error.log looks like this
2021/03/12 07:41:13 [error] 234787#234787: *17 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support. solid.at, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: " support.solid.at"

Would appreciate your help

2

it looks like your zammad railsserver isn’t running as nginx can’t connect to it. You could try restarting the zammad service, and inspect the production log of zammad for errors.

3

How do i start the railserver?

4

systemctl restart zammad
see: Install on Debian via DEB — Zammad documentation

5

Thanks, i did that and also stopped it and started it, still no luck. Can you tell me where i’ll find that production log?

6

Production log shows this
Response:
#<UserAgent::Result:0x0000564718366e00 @success=false, @body=nil, @data=nil, @code=0, @content_type=nil, @error="#<Errno::ECONNREFUSED: F ailed to open TCP connection to 127.0.0.1:9200 (Connection refused - connect(2) for “127.0.0.1” port 9200)>">

7

looks like your elasticsearch is also not running; systemctl restart elasticsearch
the default log should be here; /var/log/zammad/production.log see; Configuration via Environment Variables — Zammad documentation

8

Ok now that i’ve done that instead of a 502 i get the website is not reachable

9

In the production log: I haven’t found any error in there - looks good in my opinion
For the nginx error log i still get the same data:
2021/03/12 06:50:00 [error] 1553#1553: *6 connect() failed (111: Connection refused) while connecting to upstream, client: 192 .168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid.at”
2021/03/12 07:00:25 [error] 199908#199908: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:00:26 [error] 199908#199908: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:00:26 [error] 199908#199908: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:00:27 [error] 199908#199908: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:00:29 [error] 199908#199908: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:00:32 [error] 199908#199908: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:00:44 [error] 199908#199908: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:29:06 [error] 199908#199908: *10 connect() failed (111: Connection refused) while connecting to upstream, client : 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.soli d.at”
2021/03/12 07:29:06 [error] 199908#199908: *10 connect() failed (111: Connection refused) while connecting to upstream, client : 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.soli d.at”
2021/03/12 07:29:06 [error] 199908#199908: *10 connect() failed (111: Connection refused) while connecting to upstream, client : 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.soli d.at”
2021/03/12 07:31:29 [error] 234787#234787: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:31:30 [error] 234787#234787: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:31:31 [error] 234787#234787: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:31:32 [error] 234787#234787: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:31:49 [error] 234787#234787: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:31:51 [error] 234787#234787: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:31:51 [error] 234787#234787: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:31:51 [error] 234787#234787: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:31:52 [error] 234787#234787: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:31:52 [error] 234787#234787: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:31:52 [error] 234787#234787: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:31:52 [error] 234787#234787: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:31:52 [error] 234787#234787: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:31:53 [error] 234787#234787: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid .at”
2021/03/12 07:41:13 [error] 234787#234787: *17 connect() failed (111: Connection refused) while connecting to upstream, client : 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.soli d.at”
2021/03/12 07:41:13 [error] 234787#234787: *17 connect() failed (111: Connection refused) while connecting to upstream, client : 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.soli d.at”
2021/03/12 08:05:00 [error] 234787#234787: *21 connect() failed (111: Connection refused) while connecting to upstream, client : 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “192.168.3.20 4”
2021/03/12 08:10:02 [error] 29996#29996: *24 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid. at”
2021/03/12 08:31:30 [error] 29996#29996: *27 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.22.152, server: support.solid.at, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:3000/”, host: “support.solid. at”

10

Hi Michael,

have you updated your Zammad server recently? If so, you may have the original config files with some funny extension. On CentOS 8 it usually is “.rpmsave” or “.rpmnew”. If that is the case, double-check the contents of those files, rename them to the original name and restart nginx and Zammad.

I would also recommend checking that all those auxiliary processes are running.

When you type:

systemctl status zammad[tab][tab]

you will see something like this:

zammad.service zammad-web.service zammad-websocket.service zammad-worker.service
zammad-web-1.service zammad-websocket-1.service zammad-worker-1.service

And a check would look something like this:

systemctl status zammad-worker

  • zammad-worker.service
    Loaded: loaded (/etc/systemd/system/zammad-worker.service; enabled; vendor preset: disabled)
    Active: active (running) since Wed 2020-11-25 10:59:48 CET; 3 months 15 days ago
    Main PID: 6600 (sleep)
    Tasks: 1 (limit: 75130)
    Memory: 180.0K
    CGroup: /system.slice/zammad-worker.service
    ±6600 /bin/sleep infinity

HTH,
Martin

11

Hi Martin, thanks for your reply.
So i recently updated only the security packages from ubuntu as far as i know.

But before that it wasnt working either. So i checked all the statuses of the all the services and all are up and running BUT i just discovered when i try to start nginx i get a failed. Can you help me there?

Here’s the error message:
user@zammad:/etc/nginx/sites-available$ systemctl status nginx.service
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2021-03-12 10:51:50 UTC; 8min ago
Docs: man:nginx(8)
Process: 211857 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Main PID: 234784 (code=exited, status=0/SUCCESS)

Mär 12 10:51:50 zammad systemd[1]: Starting A high performance web server and a reverse proxy server…
Mär 12 10:51:50 zammad nginx[211857]: [87B blob data]
Mär 12 10:51:50 zammad nginx[211857]: nginx: configuration file /etc/nginx/nginx.conf test failed
Mär 12 10:51:50 zammad systemd[1]: nginx.service: Control process exited, code=exited status=1
Mär 12 10:51:50 zammad systemd[1]: nginx.service: Failed with result ‘exit-code’.
Mär 12 10:51:50 zammad systemd[1]: Failed to start A high performance web server and a reverse proxy server.

12

This is what the error log give me back

    sudo nano /var/log/nginx/error.log2021/03/12 08:07:30 [notice] 29678#29678: signal process started
2021/03/12 08:07:47 [notice] 29995#29995: signal process started
2021/03/12 09:05:47 [emerg] 93111#93111: unknown directive "^[[0" in /etc/nginx/sites-enabled/earchindex:rebuild:1
2021/03/12 09:20:53 [emerg] 109950#109950: unknown directive "^[[0" in /etc/nginx/sites-enabled/earchindex:rebuild:1
2021/03/12 09:52:50 [emerg] 145720#145720: unknown directive "^[[0" in /etc/nginx/sites-enabled/earchindex:rebuild:1
2021/03/12 10:19:24 [emerg] 175898#175898: unknown directive "^[[0" in /etc/nginx/sites-enabled/earchindex:rebuild:1
2021/03/12 10:19:59 [emerg] 176535#176535: unknown directive "^[[0" in /etc/nginx/sites-enabled/earchindex:rebuild:1
2021/03/12 10:20:32 [emerg] 177147#177147: unknown directive "^[[0" in /etc/nginx/sites-enabled/earchindex:rebuild:1
2021/03/12 10:21:59 [emerg] 178776#178776: unknown directive "^[[0" in /etc/nginx/sites-enabled/earchindex:rebuild:1
2021/03/12 10:23:14 [emerg] 180164#180164: unknown directive "^[[0" in /etc/nginx/sites-enabled/earchindex:rebuild:1
2021/03/12 10:27:12 [emerg] 184598#184598: unknown directive "^[[0" in /etc/nginx/sites-enabled/earchindex:rebuild:1
2021/03/12 10:27:35 [emerg] 185005#185005: unknown directive "^[[0" in /etc/nginx/sites-enabled/earchindex:rebuild:1
2021/03/12 10:28:15 [emerg] 185755#185755: open() "/usr/share/nginx/conf.d" failed (2: No such file or directory)
2021/03/12 10:28:34 [emerg] 186104#186104: unknown directive "^[[0" in /etc/nginx/sites-enabled/earchindex:rebuild:1
2021/03/12 10:28:47 [emerg] 186317#186317: unknown directive "^[[0" in /etc/nginx/sites-enabled/earchindex:rebuild:1
2021/03/12 10:33:43 [emerg] 191837#191837: unknown directive "^[[0" in /etc/nginx/sites-enabled/earchindex:rebuild:1
2021/03/12 10:34:32 [emerg] 192779#192779: unknown directive "^[[0" in /etc/nginx/sites-enabled/earchindex:rebuild:1
2021/03/12 10:51:32 [emerg] 211544#211544: unknown directive "^[[0" in /etc/nginx/sites-enabled/earchindex:rebuild:1
2021/03/12 10:51:50 [emerg] 211857#211857: unknown directive "^[[0" in /etc/nginx/sites-enabled/earchindex:rebuild:1
2021/03/12 11:04:10 [emerg] 225650#225650: unknown directive "^[[0" in /etc/nginx/sites-enabled/earchindex:rebuild:1
13

Hi Michael,

it seems like something is wrong with your nginx configuration file … I know it’s obvious :slight_smile:
Most probably the path to your SSL certs, I guess.

Please run this:

$ nginx -t

it will spill out where the errors are. If you share them here we sure can help you further.

Once they are fixed, run the command again. If you see this:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

…you’re good to go.

If you still get stuck, please post your nginx config file for Zammad.

Best,
Martin

14

it seems like you have a typo somewhere in your nginx config file :slight_smile: – told you :wink:

15

Hi Martin, thanks for the reply

Here’s the output:
root@zammad:/# nginx -t
nginx: [emerg] unknown directive "n /etc/nginx/sites-enabled/earchindex:rebuild:1
nginx: configuration file /etc/nginx/nginx.conf test failed

16

I also have this earchindex in there, could you tell me what that is?

17

it seems like the file was renamed… it should be searchindex instead.
Anyway, that file is not present in my CentOS 8 installation so I cannot tell you exactly what it is.

With that said, it also seems like you have a typo and the "n " character should not be there (see line 1?)

18

I don’t know what you did to your system, but removing above mentioned file shouldn’t be an issue.
Looks like an accident to me.

19

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.