I believe our websocket server is broken. I’ve received complaints today that Zammad will no longer refresh at all, and there are error messages in the websocket log:
E, [2019-06-26T07:48:42.569125 #1-47327053803360] ERROR -- : #<ActiveRecord::StatementInvalid: PG::UnableToSend: no connection to the server
: SELECT "settings"."updated_at" FROM "settings" ORDER BY "settings"."updated_at" DESC LIMIT $1>
E, [2019-06-26T07:48:42.569239 #1-47327053803360] ERROR -- : ["/usr/local/bundle/gems/activerecord-5.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:635:in `exec_prepared'", "/usr/local/bundle/gems/activerecord-5.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:635:in `block (2 levels) in exec_cache'", "/usr/local/bundle/gems/activesupport-5.1.7/lib/active_support/dependencies/interlock.rb:46:in `block in permit_concurrent_loads'", "/usr/local/bundle/gems/activesupport-5.1.7/lib/active_support/concurrency/share_lock.rb:185:in `yield_shares'", "/usr/local/bundle/gems/activesupport-5.1.7/lib/active_support/dependencies/interlock.rb:45:in `permit_concurrent_loads'", "/usr/local/bundle/gems/activerecord-5.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:634:in `block in exec_cache'", "/usr/local/bundle/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract_adapter.rb:613:in `block (2 levels) in log'", "/usr/local/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'", "/usr/local/bundle/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract_adapter.rb:612:in `block in log'", "/usr/local/bundle/gems/activesupport-5.1.7/lib/active_support/notifications/instrumenter.rb:21:in `instrument'", "/usr/local/bundle/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract_adapter.rb:604:in `log'", "/usr/local/bundle/gems/activerecord-5.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:633:in `exec_cache'", "/usr/local/bundle/gems/activerecord-5.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:613:in `execute_and_clear'", "/usr/local/bundle/gems/activerecord-5.1.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:79:in `exec_query'", "/usr/local/bundle/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:375:in `select_prepared'", "/usr/local/bundle/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:40:in `select_all'", "/usr/local/bundle/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract/query_cache.rb:97:in `select_all'", "/usr/local/bundle/gems/activerecord-5.1.7/lib/active_record/relation/calculations.rb:176:in `pluck'", "/opt/zammad/app/models/setting.rb:104:in `load'", "/opt/zammad/app/models/setting.rb:55:in `get'", "/opt/zammad/lib/app_version.rb:18:in `get'", "/opt/zammad/lib/app_version.rb:76:in `event_data'", "/opt/zammad/lib/sessions/event/login.rb:19:in `run'", "/opt/zammad/lib/sessions/event.rb:15:in `run'", "/opt/zammad/lib/websocket_server.rb:102:in `onmessage'", "/opt/zammad/lib/websocket_server.rb:25:in `block (3 levels) in run'", "/usr/local/bundle/gems/em-websocket-0.5.1/lib/em-websocket/connection.rb:18:in `trigger_on_message'", "/usr/local/bundle/gems/em-websocket-0.5.1/lib/em-websocket/message_processor_06.rb:52:in `message'", "/usr/local/bundle/gems/em-websocket-0.5.1/lib/em-websocket/framing07.rb:118:in `process_data'", "/usr/local/bundle/gems/em-websocket-0.5.1/lib/em-websocket/handler.rb:47:in `receive_data'", "/usr/local/bundle/gems/em-websocket-0.5.1/lib/em-websocket/connection.rb:76:in `receive_data'", "/usr/local/bundle/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in `run_machine'", "/usr/local/bundle/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in `run'", "/opt/zammad/lib/websocket_server.rb:10:in `run'", "script/websocket-server.rb:130:in `<top (required)>'", "/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:74:in `load'", "/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:74:in `kernel_load'", "/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:28:in `run'", "/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/cli.rb:424:in `exec'", "/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'", "/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'", "/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'", "/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/cli.rb:27:in `dispatch'", "/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'", "/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/cli.rb:18:in `start'", "/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/exe/bundle:30:in `block in <top (required)>'", "/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'", "/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/exe/bundle:22:in `<top (required)>'", "/usr/local/bin/bundle:23:in `load'", "/usr/local/bin/bundle:23:in `<main>'"]
zammad ~ # docker logs --tail 1000 zammaddockercompose_zammad-websocket_1 | grep ERROR | wc -l
26
It’s probably a variation of https://github.com/zammad/zammad-docker-compose/issues/96 - apparently postgres has restarted, and now it probably has the original IP of the init container, and the websocket container is still trying to connect to the old IP:
zammad ~ # docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2bf812c7832f zammad/zammad-docker-compose:zammad-postgresql-3.0.0-9 "/usr/local/bin/back…" 5 days ago Up 5 days 5432/tcp zammaddockercompose_zammad-backup_1
aac3849928c3 zammad/zammad-docker-compose:zammad-3.0.0-9 "/docker-entrypoint.…" 5 days ago Up 5 days 0.0.0.0:8080->80/tcp zammaddockercompose_zammad-nginx_1
c1df4ff33168 zammad/zammad-docker-compose:zammad-3.0.0-9 "/docker-entrypoint.…" 5 days ago Up 5 days zammaddockercompose_zammad-websocket_1
5518aa04d9c2 zammad/zammad-docker-compose:zammad-3.0.0-9 "/docker-entrypoint.…" 5 days ago Up 5 hours zammaddockercompose_zammad-scheduler_1
dae88ac66b43 zammad/zammad-docker-compose:zammad-3.0.0-9 "/docker-entrypoint.…" 5 days ago Up 5 days zammaddockercompose_zammad-railsserver_1
a17e86547ce2 zammad/zammad-docker-compose:zammad-postgresql-3.0.0-9 "docker-entrypoint.s…" 5 days ago Up 24 hours 5432/tcp zammaddockercompose_zammad-postgresql_1
19fa30d60c39 zammad/zammad-docker-compose:zammad-elasticsearch-3.0.0-9 "/bin/bash bin/es-do…" 5 days ago Up 5 days 9200/tcp, 9300/tcp zammaddockercompose_zammad-elasticsearch_1
278378b48f2d zammad/zammad-docker-compose:zammad-memcached-3.0.0-9 "/docker-entrypoint.…" 5 days ago Up 5 days 11211/tcp zammaddockercompose_zammad-memcached_1