Hi,
right after the weekend, my Zammad-server serves me an error on monday morning.
On the website, I got the error “503 Service Unavailable”. In the apache2 log I see this error:
[Mon Oct 23 09:31:00.409985 2023] [proxy:error] [pid 3892:tid 140117930927872] (111)Connection refused: AH00957: http: attempt to connect to 127.0.0.1:3000 (127.0.0.1:3000) failed
[Mon Oct 23 09:31:00.410019 2023] [proxy_http:error] [pid 3892:tid 140117930927872] [client 10.215.224.52:63624] AH01114: HTTP: failed to make connection to backend: 127.0.0.1
But first my technical situation:
-Zammad Installation from source
-Version? Can’t find a version-number on cli. Can’t access GUI to look for it. Last update was in beginning of 2023.
-Server: Debian GNU/Linux 11 (bullseye)
-Webserver: Apache2
-Database: PostgreSQL
-Browser: MS-Edge Version 118.0.2088.57 and Firefox 118.0.1
-I don’t have a Firewall running on this machine
-Server is on-premise
I don ‘t know what happened on week and, but after it, Zammad doesn’t work. So, I have to troubleshot this situation. I not sure how.
Up to Now I know:
-Apache error.log shows the message above.
-Zammad.conf looks like this:
ServerTokens Prod
<VirtualHost *:80>
ServerName karlx021.intern.ias-gruppe.de
Redirect permanent / https://fqdn-of-server
</VirtualHost>
<VirtualHost *:443>
SSLEngine on
SSLProtocol all -SSLv2 -SSLv3
SSLCipherSuite EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
SSLCertificateFile /etc/certificate-ias/fqdn-of-server.de.crt
SSLCertificateKeyFile /etc/certificate-ias/ fqdn-of-server.de.key.pem
#SSLCertificateFile /etc/self-signed-certivicate/certificate.pem
#SSLCertificateKeyFile /etc/self-signed-certivicate/key.pem
#SSLCertificateChainFile /etc/ssl/lets-encrypt-x3-cross-signed.pem
SSLOpenSSLConfCmd DHParameters /etc/self-signed-certivicate/dhparam.pem
# replace 'localhost' with your fqdn if you want to use zammad from remote
ServerName localhost
## don't loose time with IP address lookups
HostnameLookups Off
## needed for named virtual hosts
UseCanonicalName Off
## configures the footer on server-generated documents
ServerSignature Off
ProxyRequests Off
ProxyPreserveHost On
<Proxy 127.0.0.1:3000>
Require local
</Proxy>
ProxyPass /assets !
ProxyPass /favicon.ico !
ProxyPass /apple-touch-icon.png !
ProxyPass /robots.txt !
ProxyPass /ws ws://127.0.0.1:6042/
ProxyPass / http://127.0.0.1:3000/
# change this line in an SSO setup
RequestHeader unset X-Forwarded-User
RequestHeader set X_FORWARDED_PROTO 'https'
# Use settings below if proxying does not work and you receive HTTP-Errror 404
# if you use the settings below, make sure to comment out the above two options
# This may not apply to all systems, applies to openSuse
#ProxyPass /ws ws://127.0.0.1:6042/ "retry=1 acque=3000 timeout=600 keepalive=On"
#ProxyPass / http://127.0.0.1:3000/ "retry=1 acque=3000 timeout=600 keepalive=On"
DocumentRoot "/opt/zammad/public"
<Directory />
Options FollowSymLinks
AllowOverride None
</Directory>
<Directory "/opt/zammad/public">
Options FollowSymLinks
Require all granted
</Directory>
</VirtualHost>
Netstat will show me:
# netstat -tlpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN 120 1799 639/nrpe
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 121 815 642/redis-server 12
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 743 666/sshd: /usr/sbin
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 0 12073 637/cupsd
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 119 14522 793/postgres
tcp 0 0 127.0.0.1:6042 0.0.0.0:* LISTEN 118 198694 3372/script/websock
tcp6 0 0 :::443 :::* LISTEN 0 199884 3891/apache2
tcp6 0 0 :::5666 :::* LISTEN 120 1800 639/nrpe
tcp6 0 0 ::1:6379 :::* LISTEN 121 816 642/redis-server 12
tcp6 0 0 :::80 :::* LISTEN 0 199880 3891/apache2
tcp6 0 0 127.0.0.1:9200 :::* LISTEN 117 16104 765/java
tcp6 0 0 ::1:9200 :::* LISTEN 117 17548 765/java
tcp6 0 0 127.0.0.1:9300 :::* LISTEN 117 17536 765/java
tcp6 0 0 ::1:9300 :::* LISTEN 117 17529 765/java
tcp6 0 0 :::22 :::* LISTEN 0 754 666/sshd: /usr/sbin
tcp6 0 0 ::1:631 :::* LISTEN 0 12072 637/cupsd
tcp6 0 0 ::1:5432 :::* LISTEN 119 14521 793/postgres
Postgres log shows me:
2023-10-23 09:32:20.692 CEST [4022] zammad@zammad ERROR: could not serialize access due to read/write dependencies among transactions
2023-10-23 09:32:20.692 CEST [4022] zammad@zammad DETAIL: Reason code: Canceled on identification as a pivot, during commit attempt.
2023-10-23 09:32:20.692 CEST [4022] zammad@zammad HINT: The transaction might succeed if retried.
2023-10-23 09:32:20.692 CEST [4022] zammad@zammad STATEMENT: COMMIT
2023-10-23 09:32:20.692 CEST [4022] zammad@zammad WARNING: there is no transaction in progress
2023-10-23 09:32:20.745 CEST [4023] zammad@zammad ERROR: could not serialize access due to read/write dependencies among transactions
2023-10-23 09:32:20.745 CEST [4023] zammad@zammad DETAIL: Reason code: Canceled on identification as a pivot, during write.
2023-10-23 09:32:20.745 CEST [4023] zammad@zammad HINT: The transaction might succeed if retried.
2023-10-23 09:32:20.745 CEST [4023] zammad@zammad STATEMENT: INSERT INTO "active_job_locks" ("lock_key", "active_job_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4) RETURNING "id"
2023-10-23 09:32:20.746 CEST [4023] zammad@zammad ERROR: current transaction is aborted, commands ignored until end of transaction block
2023-10-23 09:32:20.746 CEST [4023] zammad@zammad STATEMENT: DEALLOCATE a1
2023-10-23 09:32:20.747 CEST [4023] zammad@zammad ERROR: current transaction is aborted, commands ignored until end of transaction block
2023-10-23 09:32:20.747 CEST [4023] zammad@zammad STATEMENT: DEALLOCATE a2
2023-10-23 09:32:20.798 CEST [4022] zammad@zammad ERROR: could not serialize access due to read/write dependencies among transactions
2023-10-23 09:32:20.798 CEST [4022] zammad@zammad DETAIL: Reason code: Canceled on identification as a pivot, during write.
2023-10-23 09:32:20.798 CEST [4022] zammad@zammad HINT: The transaction might succeed if retried.
2023-10-23 09:32:20.798 CEST [4022] zammad@zammad STATEMENT: UPDATE "active_job_locks" SET "updated_at" = $1 WHERE "active_job_locks"."id" = $2
2023-10-23 09:32:20.798 CEST [4022] zammad@zammad ERROR: current transaction is aborted, commands ignored until end of transaction block
2023-10-23 09:32:20.798 CEST [4022] zammad@zammad STATEMENT: DEALLOCATE a1
2023-10-23 09:32:20.799 CEST [4022] zammad@zammad ERROR: current transaction is aborted, commands ignored until end of transaction block
2023-10-23 09:32:20.799 CEST [4022] zammad@zammad STATEMENT: DEALLOCATE a2
2023-10-23 09:32:20.799 CEST [4022] zammad@zammad ERROR: current transaction is aborted, commands ignored until end of transaction block
2023-10-23 09:32:20.799 CEST [4022] zammad@zammad STATEMENT: DEALLOCATE a3
2023-10-23 09:32:20.800 CEST [4022] zammad@zammad ERROR: current transaction is aborted, commands ignored until end of transaction block
2023-10-23 09:32:20.800 CEST [4022] zammad@zammad STATEMENT: DEALLOCATE a4
Question:
in Zammad.conf I have this entry:
ProxyPass / http://127.0.0.1:3000/
But in output of “netstat -tlpen” there is no open Port 3000. Which service should run on port 3000?