Data migration from discourse to zammad

Feature requests
1

Hi,

we are a non-profit and used discourse for internal and external communication over the last years. The external communication has its limits, so we’d like to switch to zammad. Thanks for developing this software, it looks great – and features such as “use References: eMail headers” make it very pleasant.

While doing so, data migration is key. Discourse has the concept of “topics” and “posts”, we’d like to move each “topic” into a separate ticket (and posts depending on its nature - private (whisper) or public (mail got sent out to external party) - as note / reply). Preserving the timestamp of the original post creation would be great.

We figured out that discourse has some APIs to retrieve the raw content (see e.g. https://meta.discourse.org/t/topic-and-category-export-import/38930/11 - which does not include external media), but also the https://discourse/posts/$post_id/raw-email.json endpoint which returns the raw email data).

From the Zammad API docs I read there is an API to create a new ticket (i.e. https://docs.zammad.org/en/latest/api/ticket.html#create) - but is it possible to upload raw eMails via the REST API? Or is it more sensible to have zammad retrieve the old discourse-exported mails via POP/IMAP? Would retrieving via POP/IMAP use the Date header from the mail, or the current timestamp?

To migrate the internal notes (+ mails sent from discourse): does the zammad Article create endpoint (https://docs.zammad.org/en/latest/api/ticket-article.html#create) allow to specify the created_at timestamp for the note?

Is there anyone else who has experience with such a migration (discourse -> zammad)?

Thanks for reading, your reply, and developing this amazing piece of software,

Hannes

2

Hi there,

I’m not sure if this is a feature request or a request for technical assistance.

If you are proposing a new data migration tool for Discourse to Zammad, don’t get your hopes up too high. I don’t think it’s common to migrate from a forum software to a helpdesk software, so I have little hope that this feature will be implemented in the future.

If this a support request, I believe it’s relatively difficult to migrate from Discourse to Zammad. Even if you should be able to migrate it, I think that the end result would be rather unsatisfying. You’re trying to migrate from one software type to another. Other people have failed to migrate data within the same software type. Have you thought about just manually migrating the open requests/tickets and then retiring Discourse with an option for agents to still log in and view old topics? Because personally, I don’t think that the benefits would justify the efforts - of course, I don’t know how important old tickets are to you and your team.

3

Thanks for your reply, Ben. I am not sure either whether this belongs to feature requests, tech support, or dev.

I was explaining the overall goal “migrate data from discourse to zammad”.

The concrete questions are:

  • is the timestamp of an email preserved when fetched (via POP/IMAP), or is the current system time used?
  • can I upload a raw email into zammad via the REST API?
  • can I set the created timestamp of tickets and articles via the REST API?

Thanks for reading,

Hannes

4

Replying to myself: I found some answers in Display timestamp of when incoming email was sent, not ticket created which suggests to enable “import_mode” (in Create Tickets and Articles through API with historical time and date) to allow setting of created_at (and other fields) via the REST API.

I still have not found a way to upload raw emails via the REST API - is there any?

I also came across the recently merged PR about “Import archive mailbox” (https://github.com/zammad/zammad/issues/3142) which looks like it could solve many of my issues.

5

Just a small heads up that archive import for “normal” mail accounts is available since Zammad 3.6.

6

Thanks @MrGeneration for your heads up. I indeed did the migration and used the “archive import” when I got aware of that feature in 3.6.0, released mid November 2020.

For what it is worth (if others have the need), the brief outline:

So, for each topic I gathered a list of post_ids. Discourse offers two endpoints of interest:

  • /posts/$id/raw-email.json returning a json dictionary with a raw_email field
  • /posts/$id returning a json dictionary with raw (text) and meta-information (timestamp, who created it, …)

The first endpoint only returns data if it was an incoming mail into discourse. The second endpoint always returns data.

At this point, for each topic I created a mbox file (basically raw mails, separated by newlines and From ... header), to feed into zammad:

  • incoming mails I just output from the raw-email endpoint (and preserved the Message-Id header)
  • outgoing mails I reconstructed from the metadata (notes: from id is a lookup into the table from the category export “users”, Message-Id is of the form topic/$topic-id/$post-d@hostname (useful since other incoming mails may reference this), if post_type is 4, it is a whisper (internal message), References/In-Reply-To I reconstructed using the reply_to_post_number metadata (and a per-topic table of post_number -> Message-id)

Voila, I had ~400 mbox files that I concatenated and fed to zammad :slight_smile:

1 Like
7

This topic was automatically closed after 416 days. New replies are no longer allowed.