[solved] General update question

martin.von.wittich · March 27, 2019, 7:47pm

“Bad gateway” sounds like NGINX is unable to connect to the backend rails server; it probably crashed for some reason. Can you check the rails server output? Sorry, don’t exactly know how, because I only have experience with zammad-docker-compose
I assume it’s located somewhere in /var/log/zammad…?

ReisKanzlerTao · March 28, 2019, 7:28am

This log is under /var/log/nginx
There are also 3 other logs (access.log,error.log,zammad.log) But they are either empty or useless)

This is the zammad.error.log

2019/03/27 13:13:04 [error] 16579#16579: *20 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:13:10 [error] 16579#16579: *22 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:13:15 [error] 16579#16579: *24 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:13:19 [error] 16579#16579: *26 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:13:24 [error] 16579#16579: *28 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:13:28 [error] 16579#16579: *30 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:13:33 [error] 16579#16579: *32 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:13:39 [error] 16579#16579: *34 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:13:48 [error] 16579#16579: *36 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:14:03 [error] 16579#16579: *38 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:14:24 [error] 16579#16579: *40 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:14:56 [error] 16579#16579: *42 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:15:44 [error] 16579#16579: *44 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:16:44 [error] 16579#16579: *46 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:17:44 [error] 16579#16579: *48 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:18:44 [error] 16579#16579: *50 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:19:44 [error] 16579#16579: *52 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:20:44 [error] 16579#16579: *54 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 13:21:44 [error] 16579#16579: *56 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:6042/ws", host: "192.168.100.195"
2019/03/27 14:55:49 [error] 16579#16579: *58 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:55:53 [error] 16579#16579: *60 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:56:45 [error] 19851#19851: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:56:48 [error] 19851#19851: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:56:49 [error] 19851#19851: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:56:49 [error] 19851#19851: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:56:49 [error] 19851#19851: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:56:49 [error] 19851#19851: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:56:50 [error] 19851#19851: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:56:50 [error] 19851#19851: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:56:50 [error] 19851#19851: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:56:50 [error] 19851#19851: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:56:50 [error] 19851#19851: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:56:50 [error] 19851#19851: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:56:51 [error] 19851#19851: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:57:11 [error] 19851#19851: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:57:12 [error] 19851#19851: *17 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:57:12 [error] 19851#19851: *17 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:57:12 [error] 19851#19851: *17 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:57:13 [error] 19851#19851: *17 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:57:13 [error] 19851#19851: *17 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:57:13 [error] 19851#19851: *17 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:57:13 [error] 19851#19851: *17 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:57:14 [error] 19851#19851: *17 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:57:14 [error] 19851#19851: *17 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:57:18 [error] 19851#19851: *17 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:57:34 [error] 19851#19851: *17 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:59:36 [error] 518#518: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 14:59:45 [error] 518#518: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"
2019/03/27 16:02:41 [error] 518#518: *5 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.92, server: 192.168.100.195, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "192.168.100.195"

MrGeneration · March 28, 2019, 7:36am

Ensure Zammad is running and listening on port 3000.
If it does, you’ll want to ensure that your firewall is not being in your way.

You might want to restart the Zammad service and check the production log if the error persists.
Might be that the Services seems to be running but isn’t, as something is not okay (I guess database related here)

ReisKanzlerTao · March 28, 2019, 8:20am

Where do I check this ?

MrGeneration · March 28, 2019, 8:25am

Really…?

as root:

systemctl status zammad
lsof|grep 3000

result for last command should be similar to this:

ruby      31971                 zammad    9u     IPv4            6159173        0t0        TCP localhost:3000 (LISTEN)
puma\x200 31971  5833           zammad    9u     IPv4            6159173        0t0        TCP localhost:3000 (LISTEN)
puma\x200 31971  6468           zammad    9u     IPv4            6159173        0t0        TCP localhost:3000 (LISTEN)
puma\x200 31971  6686           zammad    9u     IPv4            6159173        0t0        TCP localhost:3000 (LISTEN)
puma\x200 31971  7121           zammad    9u     IPv4            6159173        0t0        TCP localhost:3000 (LISTEN)
puma\x200 31971 28602           zammad    9u     IPv4            6159173        0t0        TCP localhost:3000 (LISTEN)
ruby-time 31971 32056           zammad    9u     IPv4            6159173        0t0        TCP localhost:3000 (LISTEN)
diagnosti 31971 32138           zammad    9u     IPv4            6159173        0t0        TCP localhost:3000 (LISTEN)
diagnosti 31971 32191           zammad    9u     IPv4            6159173        0t0        TCP localhost:3000 (LISTEN)
diagnosti 31971 32193           zammad    9u     IPv4            6159173        0t0        TCP localhost:3000 (LISTEN)
diagnosti 31971 32194           zammad    9u     IPv4            6159173        0t0        TCP localhost:3000 (LISTEN)

ReisKanzlerTao · March 28, 2019, 8:38am

I’m sorry. I don’t know everything…

So this is the output of your 2 commands

ReisKanzlerTao · March 28, 2019, 12:37pm

I did a restart of zammad,nginx.service and postgresql a few minutes ago and checked the production.log
There isnt any new log. The latest log is from yesterday.

ReisKanzlerTao · March 28, 2019, 2:27pm

Hey Martin,

where do I find the database in docker-compose ?

martin.von.wittich · March 28, 2019, 5:12pm

In the container zammaddockercompose_zammad-postgresql_1. You can access it via:

docker exec -it zammaddockercompose_zammad-postgresql_1 psql zammad_production postgres

Or, what I’ve found more comfortable - install the psql client on your host (package postgresql-client in Debian), and set these variables e.g. in your shell startup script:

zammad ~ # tail -n3 /etc/zsh/zshrc
export PGHOST="$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' zammaddockercompose_zammad-postgresql_1)"
export PGDATABASE=zammad_production
export PGUSER=postgres

Then it’s easily accessible via psql (though you might have to open a new shell when you restart all containers, because the container’s IP might change).

ReisKanzlerTao · March 29, 2019, 8:42am

Can you check the rails server output?

Do you know where to find this log ?
or is the production log also the rail log ?

ReisKanzlerTao · March 29, 2019, 9:35am

@MrGeneration @martin.von.wittich
I think this could be interesting. The postgresql isnt running. It is active but not running like the zammad.service
I found this thread 502 Bad Gateway - Debian 9
But why isnt postgresql.service running ?

This is my /opt/zammad/config/database.yml
Should I change the database entry ? Because my database on the new server doesnt match with the entry in the database.yml
grafik

Hannes · March 29, 2019, 9:49am

Maybe it has been stopped due to an update of the service itself. The exit code suggests it has been a “normal stop” no crash or something else.

ReisKanzlerTao · March 29, 2019, 11:25am

Hmm allright.
I just checked the journalctl: And found this

Hannes · March 29, 2019, 11:51am

Which would be normal if the postgresql is not running. The worker and the websocket server rely on the database

martin.von.wittich · March 29, 2019, 12:29pm

Hmm, are you sure it’s not running? You’ve posted output from psql; if postgres weren’t running, you wouldn’t be able to use psql. You’re most likely falling for the way the postgres units are structered in systemd - you want to query postgresql@<major>.<minor>-<cluster>, and not postgresql. E.g. on Debian stretch with PostgreSQL 9.6 with a cluster “main”:

server ~ # systemctl status postgresql                 
● postgresql.service - PostgreSQL RDBMS
   Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
   Active: active (exited) since Sat 2019-03-09 18:21:31 CET; 2 weeks 5 days ago
 Main PID: 686 (code=exited, status=0/SUCCESS)
      CPU: 5ms
   CGroup: /system.slice/postgresql.service

Mär 09 18:21:31 server systemd[1]: Starting PostgreSQL RDBMS...
Mär 09 18:21:31 server systemd[1]: Started PostgreSQL RDBMS.

server ~ # systemctl status postgresql@9.6-main.service | head
● postgresql@9.6-main.service - PostgreSQL Cluster 9.6-main
   Loaded: loaded (/lib/systemd/system/postgresql@.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/postgresql@.service.d
           └─10-server-postgresql_timeout.conf
   Active: active (running) since Sat 2019-03-09 18:21:31 CET; 2 weeks 5 days ago
 Main PID: 578 (postgres)
   CGroup: /system.slice/system-postgresql.slice/postgresql@9.6-main.service
           ├─  578 /usr/lib/postgresql/9.6/bin/postgres -D /var/lib/postgresql/9.6/main -c config_file=/etc/postgresql/9.6/main/postgresql.conf
           ├─  687 postgres: checkpointer process
           ├─  688 postgres: writer process

ReisKanzlerTao · April 1, 2019, 6:55am

Yeah you guys are right. It is running.
But I got a question about docker.
Where do I list all postgresql tables in Docker-Compose ?
And how ?

martin.von.wittich · April 1, 2019, 9:32am

Open a psql shell - either by running docker exec -it zammaddockercompose_zammad-postgresql_1 psql zammad_production postgres or by using psql on the host as I described earlier, and then run the following command:

\dt

ReisKanzlerTao · April 2, 2019, 6:22am

Where do I find the backup.sh under docker.compose ?
It is not under /usr/local/bin/backup.sh and I cant find it with find / backup.sh

#########Update############
It is under /root/zammad-docker-compose/containers/zammad-postgresql/backup.sh

########Update##########
I got an issue after trying to run the following command

zammad-docker-compose# sudo docker exec zammaddockercompose_zammad-backup_1 /root/zammad-docker-compose/containers/zammad-postgresql/backup.sh zammad-backup-once

OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused “exec: "/root/zammad-docker-compose/containers/zammad-postgresql/backup.sh": stat /root/zammad-docker-compose/containers/zammad-postgresql/backup.sh: no such file or directory”: unknown

########Update##########
I’v solved the problem by copy the backup.sh into the /usr/local/bin path.
Afterwards I executed the following command:

sudo docker exec zammaddockercompose_zammad-backup_1 /usr/local/bin/backup.sh zammad-backup-once

martin.von.wittich · April 2, 2019, 4:25pm

Oh no… you’re mixing something up there. backup.sh is located inside the zammaddockercompose_zammad-backup_1 container:

zammad-test.iserv.eu ~ # docker exec -it zammaddockercompose_zammad-backup_1 /bin/bash
bash-4.4# ls -l /usr/local/bin/backup.sh
-rwxrwxr-x    1 root     root          1434 Mar 15 06:37 /usr/local/bin/backup.sh
bash-4.4#

You cannot run it as docker exec zammaddockercompose_zammad-backup_1 /root/zammad-docker-compose/containers/zammad-postgresql/backup.sh because now you’re trying to run a file from the host (/root/zammad-docker-compose/...) inside the container. That’s not how docker works. That’s also why you’re getting an error message - from the viewpoint of the container, there is no folder /root/zammad-docker-compose.

/root/zammad-docker-compose/containers/zammad-postgresql/backup.sh is used only during the construction of the zammad-postgresql docker image, which is in turn used, among other things, as the image for the zammaddockercompose_zammad-backup_1 container.

You copied the script to /usr/local/bin/backup.sh on the host, I assume? There’s no need for that, and the command should work just fine without the copy. You should probably delete /usr/local/bin/backup.sh on the host again if you’ve put it there, because it doesn’t make any sense to have it there

ReisKanzlerTao · April 3, 2019, 6:21am

Oh ok…
But strangely enough, everything worked.
I was able to install the backup on the new server via a detour.