Zammad 7.0.0: Channel.fetch_async only fetches active Email::Account-Channels partly, manual Channel.find(ID).fetch works

Technical assistance
1

Infos:

  • Used Zammad version: 7.0.0-1773239210.113beb30.jammy
  • Used Zammad installation type: package
  • Operating system: Ubuntu 22.04 LTS
  • Browser + version: independent of browser, issue observed on backend/scheduler level

Additional environment details:

  • PostgreSQL 16 (external DB Server)
  • Redis local
  • Elasticsearch 7.17.29
  • 8 active channels of type Email::Account

Expected behavior:

  • Channel.fetch_async should consistently process all active Email::Account channels.
  • All active inbound mail channels should receive regular automatic fetch jobs.
  • last_fetch should advance for all active channels, not only for a subset.

Actual behavior:

  • Only a subset of active Email::Account channels is automatically processed.
  • Monitoring reports messages such as:
    Channel: Email::Account in is active but not fetched for about X hours
  • Manual fetch works for affected channels via Rails console, for example:
    Channel.find(ID).fetch
  • After manual fetch, last_fetch updates correctly.
  • This indicates that the affected channels and IMAP credentials are functional. But the channels are not consistently included in automatic fetching.

Observed example:

  • Active channels: 8
  • Automatically processed channels in production.log: 7, 8, 9
  • Channel 11 was processed automatically earlier, but later stopped appearing in automatic ChannelFetchJob entries.
  • Active channels 5, 6, 10 and 12 did not appear in automatic ChannelFetchJob entries, although manual fetch succeeded for them.

Additional observations:

  • Channel.fetch_async runs in general.
  • ProcessScheduledJobs also runs.
  • Scheduler.where(status: "error", active: false) returns no affected schedulers.
  • This does not look like a complete scheduler failure, but rather like selective omission of active email channels during automatic fetch scheduling.

Steps to reproduce the behavior:

  • Configure multiple active inbound IMAP channels (Email::Account).
  • Verify that all channels are active.
  • Observe monitoring and last_fetch values over time.
  • Check production.log for ChannelFetchJob entries.
  • It can be observed that only a subset of active channels receives automatic fetch jobs.
  • Run manual fetch for affected channels, e.g.:
    Channel.find(5).fetch
  • Observe that manual fetch succeeds and last_fetch updates correctly for these channels.
  • Compare this with the missing automatic ChannelFetchJob entries for the same channels.

Potentially relevant notes:

  • This behavior started after upgrading to V7.
  • Reboot / Restart does not fix the problem
  • No Indications in Production.og about errors or that the affected channels even get fetched.
2

Please use english for the whole thread next time. :slight_smile:

Please provide your performance tunings - if I’d had to bet, I’d say parts of the background workers died silently.

3

Yes, thanks for correcting the Title! :wink:

There are currently no explicit performance tuning overrides configured.
From the current system report, the relevant environment values are all unset / default:

"Environment": {
  "RAILS_LOG_TO_STDOUT": false,
  "ZAMMAD_SAFE_MODE": false,
  "ZAMMAD_RAILS_PORT": false,
  "ZAMMAD_WEBSOCKET_PORT": false,
  "WEB_CONCURRENCY": null,
  "ZAMMAD_PROCESS_SESSIONS_JOBS_WORKERS": null,
  "ZAMMAD_PROCESS_SCHEDULED_JOBS_WORKERS": null,
  "ZAMMAD_PROCESS_DELAYED_JOBS_WORKERS": null,
  "ZAMMAD_SESSION_JOBS_CONCURRENT": null
}

If by “performance tunings” you mean something else than the configured worker/environment settings from the system report, please let me know what exactly you would like to see.

4

We did not saw any problems in this direction. Are it always the same channels, which have maybe some relations?

5

Yes, it is always the same channels.

So far, no obvious relation between the affected channels has been identified. They are configured in the same way as the working ones, using the same mail host and the same general connection setup.

6

Chiming in to say we seem to be experiencing the exact same thing. One of our two email channels is fetching regularly.

The other is not. Exact same behaviour as described above, but we only have two channels. So I can at least confirm this behaviour is not isolated to MaFi.

Interim solution is a cronjob running the fetch rails command.

7

What happens if you stop the entire stack and start it back up? Are all channels fetched then? Maybe the “it doesn’t fetch partly” comes over time…?

8

systemctl restart zammad services does not seem to import channels (but I didn’t wait that long… will have a look at logs on Monday).

9

Hi @MaFi

  • Active channels 5, 6, 10 and 12 did not appear in automatic ChannelFetchJob entries, although manual fetch succeeded for them.

Do you mean that after running fetch_async there’re no jobs created for those channels?

Could you run Channel.active.fetchable? Does it return all channels or only those 3?

10

@mantas I’ll chip in: 2/2 of my channels are fetchable and active: true.

11

No like entirely stop it (STOP) and then start. Restart sometimes behaves differently, as odd as it sounds.

12

Fetchable as in channel.fetchable? or Channel.fetchable including said channel? I wonder if something could be wrong with the later.

13

That seems to have fixed it for us. Thank you!

Is this difference between Stop / Restart documented somewhere?

We will monitor the situation further and report back if the issue occurs again.

14

Not really and honestly I don’t see why there should be an extra part in the documentation for it.

Technically Restart nor Stop are utilized.

My input came from sole personal experiences in several occasions, not just in terms of Zammad, but also other systemd services over the time.

15

Hello dear community

Unfortunately I was not able to reliably reproduce this issue. But there were some questionable code bits in adjacent areas which may have fixed this as well.

The fix is in develop branch for people living on the bleeding edge. There’s stable-backport-6043-channel-fetching branch with the fix backported to 7.0 on Github for those who use source installation. As well as packages from the usual distros at packager.io.

It’d be great to get some feedback if this fixes the channel fetching issue!

16

Problem seems to be solved since installing the backport ~10 days ago. I presume this is now included in the 7.0.1 release?

Thank you for the excellent support!

17

Hi @sduzevichiow

Unfortunately we’re still testing this fix. Thus it was not included in 7.0.1. Please stick to the backport branch for now. Said branch is rebased on top of 7.0.1 to include the security fixes. The updated packages coming any moment.

18

Hi everyone,

I’m running Zammad via Docker using the official zammad-docker-compose stack.

Is it possible to use stable-backport-6043-channel-fetching with this setup?

Also, just to confirm: would that be Zammad v7.0.0 with the fix applied?

I hope the fix will soon be mature enough for the stable branch. Thanks @mantas !

Best regards!

19

The stable backport is not available for docker nor available in the current stable version of Zammad 7.

20

Is there a reliable way to tell whether an email channel is still working properly or if it has become stuck?

At the moment, our workaround is to monitor the health check with a cron job and trigger a manual fetch if needed. But the health check only kicks in after one hour, correct?

I’m concerned that this means we could be missing emails for quite some time.

Thanks for any guidance.