Slack Integration Bug

I opened this ticket on GitHub as a bug report. Dominik closed it as a technical issue and Marcel, in his infinite tact, thumbs down emoji’d my requests to reopen. I was able to reproduce the issue in a different server with similar setup.

Used Zammad Version

5.4.0-1680261642.c296586b.focal

Environment

  • Installation method: [e.g. source, package] Package
  • Operating system (if you’re unsure: cat /etc/os-release ): [e.g. debian 10.4, ubuntu 20.04] Ubuntu 22.04.2
  • Database + version: [e.g. postgresql 9.3, mysql 5.7, mariadb 10.3] postgres (PostgreSQL) 14.7 (Ubuntu 14.7-0ubuntu0.22.04.1)
  • Elasticsearch version: [e.g. 7.17] 7.17.9
  • Browser + version: [e.g. chrome 83, safari 14, firefox 105] not browser specific

Actual behaviour

I disabled notifications were for one of the groups that had been enabled for years. Didn’t notice the lack of notifications for all groups for a period of hours. Reenabled for the group notifications were removed for. Still no notifications in Slack. Updated from 5.3.1-1676465796.746f9768.focal to 5.4.0-1680261642.c296586b.focal hoping that would fix the issue. This version brought along bugs that prevented Microsoft 365 integration from functioning. Reverted back to snap of 5.3.1. Still no notifications. Screenshot of most recent log posted with personal information removed. I’m not sure if Response “” is normal because I never checked the logs when it was functioning properly to have noticed a difference.
image

Expected behaviour

Slack notifications received.

Steps to reproduce the behaviour

Unselect a group from ‘Only for these groups’ under Slack integration, then lose all slack integration functionality.

I’ve ran through the logs and there’s a myriad of listings for Slack. The error messages are virtually identical to the first block below. The other block at the bottom is what I’m seeing before the error message posts in the log.

E, [2023-03-31T13:20:35.876434 #906-111620] ERROR – : Unable to post webhook: https://hooks.slack.com/services/[REDACTED]: [#<UserAgent::Result:0x000055be6733d198 @success=false, @Body=nil, @DaTa=nil, @code=0, @content_type=nil, @error=“#<Net::ReadTimeout: Net::ReadTimeout>”, @Header=nil>]

I, [2023-03-31T13:41:18.158134 #906-111620] INFO – : 2023-03-31T13:41:18+0000: [Worker(host:helpdesk pid:906)] Job TicketArticleCommunicateEmailJob [REDACTED] from DelayedJob(default) with arguments: [62011] (id=3229311) (queue=default) RUNNING
I, [2023-03-31T13:41:19.329610 #896-28864360] INFO – : Started GET “/api/v1/http_logs/slack_webhook?limit=50&=1680267654594” for 192.168.53.25 at 2023-03-31 13:41:19 +0000
I, [2023-03-31T13:41:19.335589 #896-28864360] INFO – : Processing by HttpLogsController#index as JSON
I, [2023-03-31T13:41:19.335666 #896-28864360] INFO – : Parameters: {“limit”=>“50”, “
”=>“1680267654594”, “facility”=>“slack_webhook”}
I, [2023-03-31T13:41:19.375004 #896-28864360] INFO – : Completed 200 OK in 39ms (Views: 27.0ms | ActiveRecord: 1.6ms | Allocations: 22238)

In spite of the ‘ReadTimeout’ I can curl and successfully post immediately after a fail from Zammad shows up in the logs.

On the Slack side, I have moved the integration amongst various channels, regenerated the key, and deleted and rebuilt the integration.

On the Zammad side, I disabled/enabled the integration, rebooted the server, disabled the integration rebooted the server and then re-enabled the integration.

Bumping. Including logs from our FTDs showing traffic was allowed out at the time stamps that match up with Zammad’s slack logs.

I adjusted the following settings on app\models\transaction\slack.rb as suggested by the OP here: Slack integration - Unable to post webhook (Net::OpenTimeout: execution expired) · Issue #1519 · zammad/zammad · GitHub

open_timeout from 4 to 20
read_timeout from 10 to 20
total_timeout from 20 to 30

Issue still persists.

Just for shits & giggles, I through the server into the DMZ for a moment to test notifications. Still nada.

You’re not using any proxy server on Zammad do you?
Are you certain that no application aware firewall actually timeouts the request of Zammad?

Maybe something that looks for user agents? This would explain why it might work on your console but not via Zammad from the same host. (Both application aware firewalle and proxy server)

You could cross check this via curl by extending your command by -A 'Zammad User Agent'.
Not that you’d need to send the exact same payload like Zammad as other wise you might not be able to reproduce the same issue.

We are not using a proxy server anywhere in our environment.

Absolutely Certain. I moved the server to the DMZ and opened it to any/any for a moment to verify.

Negative.

[quote="MrGeneration] You could cross check this via curl by extending your command by -A ‘Zammad User Agent’`.
Not that you’d need to send the exact same payload like Zammad as other wise you might not be able to reproduce the same issue. [/quote]

This is what I get when I run the code as "
root@helpdesk:/home/zammad#url=“https://hooks.slack…com/services/[REDACTED]”
message=“Hello MrGeneration!”
curl -X POST -H “Content-type: application/json” -H “User-Agent: Zammad User Agent” --data “{"text":"$message"}” $url
image

Bumping this thread up for assistance.

With the myriad of fixes that came along with upgrading our Cisco FTDs to version 7.2.4 from version 7.2.3 (Cisco Secure Firewall Threat Defense Release Notes, Version 7.2 - Open and Resolved Bugs [Cisco Secure Firewall Threat Defense] - Cisco), apparently the slack notifications as well as another issue we were having on another one of our Zammad instances where email was affected but not on any other of our instances.

I think I speak for most of us when I say THANKS, CISCO for breaking something in a way that makes no sense for it to break!

Please close this one out.

2 Likes

Oh wow that though.
Sorry you had to experience that.

Thread will automatically close within 7 days due to the solved state. :slight_smile:

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.