Sadly neither google, nor this forum search and even not in the documentation is anything mentioned about rerun of the above commands and what they will produce on the data side.
Can someone help me on this questions:
can I rerun the migrator as often as I want without doing bad to the data? (idempotent reruns?)
if yes, can I somehow limit the migrator e.g to migrate the years 2005-2007, next run 2008-2010, next run 2010-2013 and so on?
I am about to give up again with Zammad, since I cannot again migrate my data
My migration fails non deterministic with timeouts and since I do not have any savepoint in the migration e.g. with date+time (or also ID of tickets would be okay) I cannot get further with the migration.
Having a database with about 10Mio. Tickets. So from the statistical point, a issue may arrise in such huge migration… Sadly the migration fails, and it fails without any visible savepoint to restart the migration.
The only thing which I have now is to fiddle with configs on OTRS side, to pray the next migration will go through properly. Sadly may prayers are not reaching the OTRS or zammad-migrator.
Tried already about 30 migration runs, without success.
can I migrate a batch of data (e.g. 500.000 Tickets, the next 500k, the next …)
alternatively and the better way
can I migrate by dates (e.g. 1.1.2013 - 30.6.2013, 1.7.2013 - 31.12.2013, the next …)
This would help to track down the issues and it would ensure a consistent migration, even if any of the intermediate migrations fails.
At the moment it’s also not possible to just import a specific time period of tickets.
Especially with such a big system like yours I can understand that use case, but we currently can’t cover it by default.
Hey @xeno - @MrGeneration is right. A diff import is only available after a full import was completed successfully. 10 Mio tickets is quite a sum. Just for comparison: We’re currently supporting a customer with about 70k tickets which took 13 hours on decent hardware. Following this calculation importing all your data will take about 8 days. The probability is pretty high that something fails in the meantime.
It’s necessary to know what errors you’re facing exactly. Otherwise I’m afraid we can’t help.
I like your proposed features and enhancements but our focus is on issues and features from which at least 80% our user base will profit. This use case is pretty specific.
@MrGeneration & @thorsteneckel my biggest issue is that I do not have any correlation possibility.
The logfile of the importer does not have even a timestamp, so I am getting an error and I have no idea where to look in OTRS since not even a timestamp is available.
So this feature
would be very helpful to get responses from admins.
Afaik rails has a default logging mechanism, which can be configured with a: log formatter
Where can I activate it? And why are the timestamps missing?
The other thing is, that the import fails after about 2-4 h not after days.
To make this more transparent, how long my migration may take, this feature is crucial:
And this feature would help to be able to grep/sed easily the logfiles, if the logging would be consistent:
Since I have already exceeded massively my planned effort for the migration, I have deferred my migration plans already. I will post my faced issues combined soon here.