Background tasks issues

Technical assistance
1
  • Used Zammad version: 6.3.1-1722007696.43d58aa5.bookworm
  • Used Zammad installation type: DEB
  • Operating system: Debian 12
  • Browser + version: various

Expected behavior:

Email should be fectched, and overview refreshed

Actual behavior:

It works, then crash email are not fetched anymore, need to push F5 to refresh overview…
Restart psql + zammad fix it for a time.

Steps to reproduce the behavior:

No idea !

More infos

16GB RAM
4Core
20 users connected

No delayed job count
WEB_CONCURRENCY = 3
ZAMMAD_PROCESS_SCHEDULED_JOBS_WORKERS = 1
ZAMMAD_SESSION_JOBS_CONCURRENT = 2

Kind of error I get on postgresql:

2024-10-10 11:11:53.887 CEST [15218] zammad@zammad DÉTAIL: Reason code: Canceled on identification as a pivot, during write.
2024-10-10 11:11:53.887 CEST [15218] zammad@zammad ASTUCE : La transaction pourrait réussir après une nouvelle tentative.
2024-10-10 11:11:53.887 CEST [15218] zammad@zammad INSTRUCTION : INSERT INTO “active_job_locks” (“lock_key”, “active_job_id”, “created_at”, “updated_at”) >
2024-10-10 11:11:53.903 CEST [15218] zammad@zammad ERREUR: la transaction est annulée, les commandes sont ignorées jusqu’à la fin du bloc
de la transaction

Kind of error I get on production.log:

E, [2024-10-10T11:16:29.416716#15056-276280] ERROR – : thread_client 159820 exited with error #<ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.011 seconds); all pooled connections were in use>
E, [2024-10-10T11:16:29.432035#15056-276280] ERROR – : /opt/zammad/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/connection_pool/queue.rb:126:in block in wait_poll' E, [2024-10-10T11:16:40.428430#15056-276820] ERROR -- : thread_client 160340 exited with error #<ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.709 seconds); all pooled connections were in use> E, [2024-10-10T11:16:40.756711#15056-276820] ERROR -- : /opt/zammad/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/connection_pool/queue.rb:126:in block in wait_poll’
E, [2024-10-10T11:16:46.052683#15056-276280] ERROR – : thread_client 159820 exited with error #<ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.306 seconds); all pooled connections were in use>
E, [2024-10-10T11:16:46.075358#15056-276280] ERROR – : /opt/zammad/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/connection_pool/queue.rb:126:in block in wait_poll' E, [2024-10-10T11:16:56.757050#15056-276820] ERROR -- : thread_client 160340 exited with error #<ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.336 seconds); all pooled connections were in use> E, [2024-10-10T11:16:56.757147#15056-276820] ERROR -- : /opt/zammad/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/connection_pool/queue.rb:126:in block in wait_poll’
E, [2024-10-10T11:17:03.084059#15056-276280] ERROR – : thread_client 159820 exited with error #<ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 6.302 seconds); all pooled connections were in use>
E, [2024-10-10T11:17:03.107326#15056-276280] ERROR – : /opt/zammad/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/connection_pool/queue.rb:126:in block in wait_poll' E, [2024-10-10T11:17:12.231083#15056-276820] ERROR -- : thread_client 160340 exited with error #<ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.278 seconds); all pooled connections were in use> E, [2024-10-10T11:17:12.246635#15056-276820] ERROR -- : /opt/zammad/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/connection_pool/queue.rb:126:in block in wait_poll’
E, [2024-10-10T11:17:19.732028#15056-276280] ERROR – : thread_client 159820 exited with error #<ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.698 seconds); all pooled connections were in use>
E, [2024-10-10T11:17:19.753217#15056-276280] ERROR – : /opt/zammad/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/connection_pool/queue.rb:126:in block in wait_poll' E, [2024-10-10T11:17:28.069654#15056-276820] ERROR -- : thread_client 160340 exited with error #<ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.459 seconds); all pooled connections were in use> E, [2024-10-10T11:17:28.069767#15056-276820] ERROR -- : /opt/zammad/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/connection_pool/queue.rb:126:in block in wait_poll’
E, [2024-10-10T11:17:36.325619#15056-276280] ERROR – : thread_client 159820 exited with error #<ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.577 seconds); all pooled connections were in use>
E, [2024-10-10T11:17:37.344109#15056-276280] ERROR – : /opt/zammad/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/connection_pool/queue.rb:126:in `block in wait_poll’

Let me know if you need more infos…
Many thanks !

2

Sounds like your concurrency is higher than that you configured.

3

Hello, thanks for answering !
You mean WEB_CONCURRENCY ? It like that since years now, and no issues before.
And as I see, the value in database.yaml cannot be changed (it as no effect)…

4

Apparently the errors you get on PostgreSQL side are unrelated to those on the zammad side, since the timestamps do not match and zammad is claiming it cannot obtain a connection, while PostgreSQL is claiming the transaction has to be aborted.

I think that when the problem rises, restarting only zammad should fix the issue, can you verify?

I still believe that the problem is saying that PostgreSQL is busy, but I don’t know exactly how activerecord pooling works.

5

I thought also the same, the error are unrelated.
I’ll try next time it happen to restart only Zammad…

And I also agree that the error suggest that PostgreSQL is busy when I get the error. But no idea why, and also no idea why it cause trouble on the email fetching and overview refresh…

6

Zammad by default (without any tuning) takes up to 200 database connections, while PostgreSQL by default allows 100 only.

This is reflected as a note / hint by our installation documentation, here’s the relevant part that solves your problem:
https://docs.zammad.org/en/latest/appendix/configure-env-vars.html

7

This topic was automatically closed 360 days after the last reply. New replies are no longer allowed.