Microsoft Teams Integration Webhook Error

[lmarschall]

So today, I upgraded our Zammad version to 6.0 to integrate the new Microsoft Teams Webhook, which does not work unfortunately
I followed this documentation: Microsoft Teams Notifications — Zammad documentation

• Used Zammad version: 6.0.0-1687169393.30390930.focal
• Used Zammad installation type: package
• Operating system: Ubuntu 20.04
• Browser + version: Google Chrome (newest version)

Expected behavior:

• Webhook works

Actual behavior:

• Webhook throws error

Steps to reproduce the behavior:

• Create “Incoming Webhook” in Microsoft Teams Channel
• Create predefined Microsoft Teams Webhook in Zammad
• Copy the provided Webhook Url into the newly created webhook
• Create Trigger to activate Webhook

The error logs from /var/log/zammad/production.log

E, [2023-06-20T00:34:28.129877#4744-112200] ERROR – : 2023-06-20T00:34:28+0200: [Worker(host:ticketsystem pid:4744)] Job TriggerWebhookJob [3f5a5891-519e-4112-8383-88eb206fd8e5] from DelayedJob(default) with arguments: [{“_aj_globalid”=>“gid://zammad/Trigger/13”}, {“_aj_globalid”=>“gid://zammad/Ticket/3796”}, nil, {“changes”=>{“_aj_symbol_keys”=>[]}, “user_id”=>1831, “execution_type”=>“trigger”, “event_type”=>“update”, “_aj_ruby2_keywords”=>[“changes”, “user_id”, “execution_type”, “event_type”]}] (id=3143052) (queue=default) FAILED (0 prior attempts) with NoMethodError: undefined method `[]’ for nil:NilClass

    message: match[:message].presence || '',
                  ^^^^^^^^^^

E, [2023-06-20T00:34:28.129983#4744-112200] ERROR – : 2023-06-20T00:34:28+0200: [Worker(host:ticketsystem pid:4744)] Job TriggerWebhookJob [3f5a5891-519e-4112-8383-88eb206fd8e5] from DelayedJob(default) with arguments: [{“_aj_globalid”=>“gid://zammad/Trigger/13”}, {“_aj_globalid”=>“gid://zammad/Ticket/3796”}, nil, {“changes”=>{“_aj_symbol_keys”=>[]}, “user_id”=>1831, “execution_type”=>“trigger”, “event_type”=>“update”, “_aj_ruby2_keywords”=>[“changes”, “user_id”, “execution_type”, “event_type”]}] (id=3143052) (queue=default) FAILED permanently because of 1 consecutive failures

[fliebe92]

Hi @lmarschall

Some information is missing in your initial description:

• Can you post your trigger configuration?
• What actions are you doing in the GUI to execute the trigger?

[lmarschall]

I’m sending a mail to a specific email address, so a ticket for the specific user group is created.

[lmarschall]

I was also testing other triggers, like updating the ticket etc., the webhook protocol shows me a created post request with an empty response, but nothing ever shows up in our Microsoft Teams channel

[fliebe92]

I tested this on my machine as well, it works like a charm.

• Did you make some changes to any files in your Zammad instance, especially in the notification mailer templates?
• Is it working if you’re not sending out a mail in the trigger in parallel?

[lmarschall]

Were can I find the notification mailer template? I deleted the mail notification at the trigger and the webhook request is being requested again. This time this error occurs in the logs

E, [2023-06-20T12:12:29.678764#4744-112200] ERROR – : 2023-06-20T12:12:29+0200: [Worker(host:ticketsystem pid:4744)] Job TriggerWebhookJob [937b99aa-9648-4a5e-98c5-5d92f7b25711] from DelayedJob(default) with arguments: [{“_aj_globalid”=>“gid://zammad/Trigger/13”}, {“_aj_globalid”=>“gid://zammad/Ticket/3801”}, {“_aj_globalid”=>“gid://zammad/Ticket::Article/7964”}, {“changes”=>{“_aj_symbol_keys”=>[]}, “user_id”=>2025, “execution_type”=>“trigger”, “event_type”=>“create”, “_aj_ruby2_keywords”=>[“changes”, “user_id”, “execution_type”, “event_type”]}] (id=3143491) (queue=default) FAILED (4 prior attempts) with TriggerWebhookJob::RequestError: TriggerWebhookJob::RequestError
E, [2023-06-20T12:12:29.678900#4744-112200] ERROR – : 2023-06-20T12:12:29+0200: [Worker(host:ticketsystem pid:4744)] Job TriggerWebhookJob [937b99aa-9648-4a5e-98c5-5d92f7b25711] from DelayedJob(default) with arguments: [{“_aj_globalid”=>“gid://zammad/Trigger/13”}, {“_aj_globalid”=>“gid://zammad/Ticket/3801”}, {“_aj_globalid”=>“gid://zammad/Ticket::Article/7964”}, {“changes”=>{“_aj_symbol_keys”=>[]}, “user_id”=>2025, “execution_type”=>“trigger”, “event_type”=>“create”, “_aj_ruby2_keywords”=>[“changes”, “user_id”, “execution_type”, “event_type”]}] (id=3143491) (queue=default) FAILED permanently because of 5 consecutive failures

[MrGeneration]

Is the host of your Zammad instance behind a proxy or does it have specific firewall rules that e.g. forbid it to communicate with Microsoft?

[lmarschall]

Yes we are using Nginx as a Reverse Proxy infront of Zammad, are there any specific ports I would need to open in order to make webhooks funtional?

[MrGeneration]

Sorry that was not the answer I’m looking for.
I didn’t want to know how you access Zammad via Web but if your server Zammad runs on requires proxy servers to communicate with the internet.

[lmarschall]

No, there are no proxy settings in the system → network tab.

[lmarschall]

Are there any specific settings in Microsoft Teams, in order for the POST request endpoint to be available?

[fliebe92]

At least there was no need to adjust any settings for the account/channel I was using for my test.

I just used another account/channel, this is working fine as well.

[lmarschall]

Ok, thank you for the help, I’ll take a look at our Microsoft Teams setup, maybe the problem is there.

[dominikklein]

Normally, you need only to create an incoming webhook connector and use the mention URL in the created pre-defined Webhook for Microsoft Teams.

Do you have any details from the webhook protocoll?

[lmarschall]

So today, I kept on trying to integrate Zammd in our Teams environment and got some weird behavior. I was sending the following request via curl directly to the Incoming Webhook Endpoint, and it worked flawless. In the Webhook Logs the requests appear as well, but all request do have status code 0. I’m wondering if there is any setting I’m missing which blocks external requests from Zammad?

headers

User-Agent
Zammad User Agent
Accept
*/*
Accept-Encoding
gzip
Host
docsinclouds.webhook.office.com
X-Zammad-Trigger
TRIGGERNAME
X-Zammad-Delivery
GUID
Content-Type
application/json

body

{
  "@type": "MessageCard",
  "@context": "http://schema.org/extensions",
  "themeColor": "#{ticket.current_state_color}",
  "title": "#{ticket.title}",
  "text": "#{notification.message}",
  "sections": [
    {
      "text": "#{notification.changes}"
    },
    {
      "text": "#{notification.body}"
    }
  ],
  "potentialAction": [
    {
      "targets": [
        {
          "os": "default",
          "uri": "#{notification.link}"
        }
      ],
      "@type": "OpenUri",
      "name": "Ticket##{ticket.number}"
    }
  ]
}

[lmarschall]

After disabling ipv6 on our vps webhooks now work. I’ll start investigating how we can get things to work with ipv6, maybe there is a firewall config problem.