Are multiple runs of OTRS migrator possible - maybe with yearly migrate runs?

[xeno]

Infos:

• Used Zammad version: 3.0.0-1560433985.617324d6.bionic
• Used Zammad installation source: ubuntu package
• Operating system: Ubuntu 18.04
• Browser + version: not relevant for this

Expected behavior:

• Migration of OTRS (installed 2005 and upgraded up to 5.0.34-01) to zammad works like a charm

Actual behavior:

• Having multiple runs now to migrate the OTRS sadly the migration fails on multiple different issue. (will post later the issues in different threads)

major question also to fix the documentation

• can I run multiple times the migrator without doing bad duplication or bad data?

I am using

zammad run rails c

and inside the zammad console

Delayed::Worker.max_run_time = 7.days
Setting.set('import_otrs_endpoint', 'https://otrs.example.com/otrs/public.pl?Action=ZammadMigrator')
Setting.set('import_otrs_endpoint_key', 'abcdefghijklmnopqrstuvwxyz')
Setting.set('import_mode', true)
Setting.set('system_init_done', false)
Import::OTRS.start

Sadly neither google, nor this forum search and even not in the documentation is anything mentioned about rerun of the above commands and what they will produce on the data side.
Can someone help me on this questions:

• can I rerun the migrator as often as I want without doing bad to the data? (idempotent reruns?)
• if yes, can I somehow limit the migrator e.g to migrate the years 2005-2007, next run 2008-2010, next run 2010-2013 and so on?

[xeno]

really no one, who had such issues and therefore no answers to this?

[xeno]

retry… No ideas and hopefully solutions around?

[xeno]

really noone who needs to migrate his old OTRS database and needs some measurable progress?
I am getting stuck with the migration and this feature would help me to get the issue tracked down…

[xeno]

I am about to give up again with Zammad, since I cannot again migrate my data

My migration fails non deterministic with timeouts and since I do not have any savepoint in the migration e.g. with date+time (or also ID of tickets would be okay) I cannot get further with the migration.

Having a database with about 10Mio. Tickets. So from the statistical point, a issue may arrise in such huge migration… Sadly the migration fails, and it fails without any visible savepoint to restart the migration.

The only thing which I have now is to fiddle with configs on OTRS side, to pray the next migration will go through properly. Sadly may prayers are not reaching the OTRS or zammad-migrator.

Tried already about 30 migration runs, without success.

In short:

• can I migrate a batch of data (e.g. 500.000 Tickets, the next 500k, the next …)
  alternatively and the better way
• can I migrate by dates (e.g. 1.1.2013 - 30.6.2013, 1.7.2013 - 31.12.2013, the next …)

This would help to track down the issues and it would ensure a consistent migration, even if any of the intermediate migrations fails.

[MrGeneration]

Sorry but we’re quite busy at the moment.
This is a free community which might not always cover all situations, we’re sorry for that.

Technically at the point of somewhat of 10 mio tickets, you might want to consider commercial services.

Anyway:
Please provide the production log that contains the traceback (error).
It’s enough to post the last view lines above the traceback together with the full traceback.

@thorsteneckel can then have a look what your issue might be when he finds some spare time.

To your “serveral run” question:
We can run a diff-sync, but it requires you to have a completed successfull synch before hand:
https://docs.zammad.org/en/latest/migration-otrs.html#importing-a-diff

At the moment it’s also not possible to just import a specific time period of tickets.
Especially with such a big system like yours I can understand that use case, but we currently can’t cover it by default.

[xeno]

This is a free community which might not always cover all situations, we’re sorry for that.

@MrGeneration do you have different source trees for community and for commercial variant?

I will update my other post with more errors, but all are about connection timeouts…

We can run a diff-sync, but it requires you to have a completed successfull synch before hand:

the diff-sync I have seen, so I hoped a initial sync would be possible also with some sort of limiting data on a timeline (or ID)

[MrGeneration]

@MrGeneration do you have different source trees for community and for commercial variant?

There’s no technical differences between running a commercial Zammad and the Open Source version.
What commercial service will provide though, is direct support by the vendor (Zammad GmbH).

This includes priority and direct contact. Especially if this is very important to you, this might help.
Of course we also do provide migration support if needed.

[thorsteneckel]

Hey @xeno - @MrGeneration is right. A diff import is only available after a full import was completed successfully. 10 Mio tickets is quite a sum. Just for comparison: We’re currently supporting a customer with about 70k tickets which took 13 hours on decent hardware. Following this calculation importing all your data will take about 8 days. The probability is pretty high that something fails in the meantime.

It’s necessary to know what errors you’re facing exactly. Otherwise I’m afraid we can’t help.

I like your proposed features and enhancements but our focus is on issues and features from which at least 80% our user base will profit. This use case is pretty specific.

[xeno]

@MrGeneration & @thorsteneckel my biggest issue is that I do not have any correlation possibility.
The logfile of the importer does not have even a timestamp, so I am getting an error and I have no idea where to look in OTRS since not even a timestamp is available.
So this feature

Feature-Request: Timestamp or some other correlationID required

I can’t see any remarks about the correlationID in the logs. Can you tell me where it comes from? on multi-systems debugging via log you need something to correlate the log messages. The simpliest one is a timestamp in the logs. This is missing in the migrator. The better one is to have UUIDs on both sides, so you clearly identify errors independently if you have tons of parallel threads on any system. What I see in the logs is that your OTRS takes to long to answer that HTTP POST request: …

would be very helpful to get responses from admins.
Afaik rails has a default logging mechanism, which can be configured with a: log formatter
Where can I activate it? And why are the timestamps missing?

The other thing is, that the import fails after about 2-4 h not after days.
To make this more transparent, how long my migration may take, this feature is crucial:

Feature-Request: Simple import progress report

A progress update would be great in the OTRS migrator. Afaik a very simple one would do already a great job: fetch total count of articles from OTRS on start update on every batch (otrs.rb variable “limit”) the processed articles in the form x / processed it would make the life much easier for admins to extrapolate how long the migration will take.

And this feature would help to be able to grep/sed easily the logfiles, if the logging would be consistent:

Feature-Request: Improve log output of migrator

The log output of the migrator uses a bunch of quote chars, which corrupt the output on pasting in this forum. Feature-Request to use double or single quotes to eliminate bad formatting here and of course to make the logfile parseable with other tools. E.g. this chars are used and also not consistently: ’ ` " here the original log output (only copy and pasted) thread#1: loading 20 Ticket starting at 56480... thread#1: POST: https://otrs.example.com/otrs/public.pl?Action=ZammadMigrator thre…

Since I have already exceeded massively my planned effort for the migration, I have deferred my migration plans already. I will post my faced issues combined soon here.

best wishes