Unprocessable mails - PG::UniqueViolation: ERROR: duplicate key value violates unique constraint

[nginx]

Hello zammad team!
About my environment:

1. zammad version: 3.3.0
2. installation method: build from source code
3. OS: CentOS Linux release 7.8.2003
4. database: postgresql-11
5. elk: 7.6.2

Number of posgresql DB - 1
Number of ELK clusters - 1
Number of zammad application nodes - 2

Problem description:
after we added second zammad node, sometimes errors began to appear in the logs:

[root@node02:~]# cat /opt/zammad/log/scheduler_out.log-20200518
"ERROR: Can't process email, you will find it for bug reporting under /opt/zammad/tmp/unprocessable_mail/4d9e4a93b68b4438322bab982c69919f.eml, please create an issue at https://github.com/zammad/zammad/issues"
"ERROR: #<ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_users_on_login"\nDETAIL: Key (lower(login::text))=([ajbastos23@gmail.com](mailto:ajbastos23@gmail.com)) already exists.\n: INSERT INTO "users" ("login", "firstname", "lastname", "email", "updated_by_id", "created_by_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING "id">"
"ERROR: Can't process email, you will find it for bug reporting under /opt/zammad/tmp/unprocessable_mail/62f6155ec112f6ca8dd277fa67861ebf.eml, please create an issue at https://github.com/zammad/zammad/issues"
"ERROR: #<ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_users_on_login"\nDETAIL: Key (lower(login::text))=([generosoq@yahoo.com](mailto:generosoq@yahoo.com)) already exists.\n: INSERT INTO "users" ("login", "firstname", "lastname", "email", "updated_by_id", "created_by_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING "id">"
"ERROR: Can't process email, you will find it for bug reporting under /opt/zammad/tmp/unprocessable_mail/b9c7487984b039faae9e12be32a70a06.eml, please create an issue at https://github.com/zammad/zammad/issues"
"ERROR: #<ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_users_on_login"\nDETAIL: Key (lower(login::text))=([dgb.dejuan@gmail.com](mailto:dgb.dejuan@gmail.com)) already exists.\n: INSERT INTO "users" ("login", "firstname", "lastname", "email", "updated_by_id", "created_by_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING "id">"
"ERROR: Can't process email, you will find it for bug reporting under /opt/zammad/tmp/unprocessable_mail/a12fd678f9d913ec127eb0725a594e26.eml, please create an issue at https://github.com/zammad/zammad/issues"

Most emails are processed without such errors
On both zammad nodes errors accumulate about unprocessable mails.
On both nodes zammad health check monitoring indicates a problem with unprocessable mails

how we can solve this problem?

[nginx]

monitoring tabs screenchots

[MrGeneration]

Can you please tell us how you’re running “several” nodes?
As in high avalibility?

Do unprocessible mails only occur on higher load situations?
Right now I’m having issues to understand your use case and setup.

[nginx]

We have haproxy for load balance across two nodes.
both nodes work simultaneously
i found this issue - https://github.com/zammad/zammad/issues/3057
and try to run rails r ‘Channel::EmailParser.process_unprocessable_mails’
on the first node i got such an error:
“ERROR: #<ActiveRecord::Deadlocked: PG::TRDeadlockDetected: ERROR: deadlock detected\nDETAIL: Process 16821 waits for ShareLock on transaction 4385941; blocked by process 17685.\nProcess 17685 waits for ShareLock on transaction 4385942; blocked by process 16821.\nHINT: See server log for query details.\nCONTEXT: while locking tuple (223,15) in relation “tickets”\n: SELECT “tickets”.* FROM “tickets” WHERE “tickets”.“id” = $1 LIMIT $2 FOR UPDATE>”
but all unprocessible mails were resolved
after that, i try to run rails r ‘Channel::EmailParser.process_unprocessable_mails’ on the second node, all unprocessible mails were resolved without any errors

[nginx]

about haproxy:
we balance only 80 and 443 port, on zammad server we have nginx with ssl configuration

frontend http_https
bind *:80
bind *:443 ssl crt /etc/ssl/certs/cert.pem alpn h2,http/1.1
http-request redirect scheme https unless { ssl_fc }
acl zammad hdr(host) site_name
acl has_end_trailing_slash path_end /
acl path_len_great_one path_len gt 1

use_backend  zammad if zammad

backend zammad
balance roundrobin
cookie SERVERID insert indirect nocache
http-request add-header X-CLIENT-IP %[src]
server node01 10.200.42.153:443 ssl check verify none cookie 1
server node02 10.200.42.166:443 ssl check verify none cookie 2

in zammad environment file set WEB_CONCURRENCY=2

[MrGeneration]

So you do have two identical running Zammad instances, did I understand that correctly?
Zammad correctly only partly supports high availability setups.

I’m very sorry, but supporting this via community based support is impossible.
The application is not stateless which will make your life even harder.

Do you experience such high loads that an HA setup is required…?