Zendesk Migration Incomplete

Info:

  • Used Zammad version: 6.3.1
  • Used Zammad installation type: Docker Compose

Expected behavior:

  • The Zendesk to Zammad migration to process all 82k tickets and import them to the new instance.

Actual behavior:

  • The migration tool stops at either 7k or 8k tickets and then marks itself as complete.

Steps to reproduce the behavior:

  • Start a new Zendesk migration and wait for completion; after a while, it should stop at an arbitrary number of tickets.

Additional Info:

Import Job
#<ImportJob:0x00007f9b7c371af8
 id: 1,
 name: "Import::Zendesk",
 dry_run: false,
 payload: {},
 result:
  {"Groups"=>
    {"skipped"=>0,
     "created"=>13,
     "updated"=>0,
     "unchanged"=>0,
     "failed"=>0,
     "deactivated"=>0,
     "sum"=>13,
     "total"=>13},
   "Users"=>
    {"skipped"=>0,
     "created"=>6242,
     "updated"=>8,
     "unchanged"=>0,
     "failed"=>0,
     "deactivated"=>0,
     "sum"=>6250,
     "total"=>6250},
   "Organizations"=>
    {"skipped"=>0,
     "created"=>1,
     "updated"=>0,
     "unchanged"=>1,
     "failed"=>0,
     "deactivated"=>0,
     "sum"=>2,
     "total"=>2},
   "Tickets"=>
    {"skipped"=>0,
     "created"=>5965,
     "updated"=>1035,
     "unchanged"=>0,
     "failed"=>0,
     "deactivated"=>0,
     "sum"=>7000,
     "total"=>7000}},
 started_at: Tue, 09 Jul 2024 23:35:17.161000000 UTC +00:00,
 finished_at: Wed, 10 Jul 2024 02:59:57.207000000 UTC +00:00,
 created_at: Tue, 09 Jul 2024 23:35:11.108000000 UTC +00:00,
 updated_at: Wed, 10 Jul 2024 02:59:57.211000000 UTC +00:00>

Production.log
waiting for init container to finish install or update...
waiting for init container to finish install or update...
starting railsserver... with WEB_CONCURRENCY=0
Puma starting in single mode...
* Puma version: 6.4.2 (ruby 3.2.4-p170) ("The Eagle of Durango")
*  Min threads: 5
*  Max threads: 30
*  Environment: production
*          PID: 1
I, [2024-07-09T17:29:44.475440 #1]  INFO -- : ActionCable is using the redis instance at redis://zammad-redis:6379.
I, [2024-07-09T17:29:44.494380#1-6920]  INFO -- : Using memcached as Rails cache store.
I, [2024-07-09T17:29:44.494601#1-6920]  INFO -- : Using the Redis back end for Zammad's web socket session store.
* Listening on http://[::]:3000
Use Ctrl-C to stop
- Gracefully stopping, waiting for requests to finish
starting railsserver... with WEB_CONCURRENCY=0
Puma starting in single mode...
* Puma version: 6.4.2 (ruby 3.2.4-p170) ("The Eagle of Durango")
*  Min threads: 5
*  Max threads: 30
*  Environment: production
*          PID: 1
I, [2024-07-09T17:33:05.761655 #1]  INFO -- : ActionCable is using the redis instance at redis://zammad-redis:6379.
I, [2024-07-09T17:33:05.792531#1-6920]  INFO -- : Using memcached as Rails cache store.
I, [2024-07-09T17:33:05.792867#1-6920]  INFO -- : Using the Redis back end for Zammad's web socket session store.
- Gracefully stopping, waiting for requests to finish
! Unable to load application: SignalException: SIGTERM
starting railsserver... with WEB_CONCURRENCY=0
Puma starting in single mode...
* Puma version: 6.4.2 (ruby 3.2.4-p170) ("The Eagle of Durango")
*  Min threads: 5
*  Max threads: 30
*  Environment: production
*          PID: 1
I, [2024-07-09T17:33:21.435601 #1]  INFO -- : ActionCable is using the redis instance at redis://zammad-redis:6379.
I, [2024-07-09T17:33:21.454449#1-6920]  INFO -- : Using memcached as Rails cache store.
I, [2024-07-09T17:33:21.454876#1-6920]  INFO -- : Using the Redis back end for Zammad's web socket session store.
* Listening on http://[::]:3000
Use Ctrl-C to stop
I, [2024-07-09T17:36:02.619318#1-149620]  INFO -- : Started GET "/" for 172.22.0.2 at 2024-07-09 17:36:02 -0600
I, [2024-07-09T17:36:02.788936#1-149620]  INFO -- : Processing by InitController#index as HTML
I, [2024-07-09T17:36:02.857283#1-149620]  INFO -- :   Rendered init/index.html.erb within layouts/application (Duration: 2.4ms | Allocations: 451)
I, [2024-07-09T17:36:02.863748#1-149620]  INFO -- :   Rendered layout layouts/application.html.erb (Duration: 9.1ms | Allocations: 1761)
I, [2024-07-09T17:36:02.864850#1-149620]  INFO -- : Completed 200 OK in 76ms (Views: 15.2ms | ActiveRecord: 18.8ms | Allocations: 7955)
I, [2024-07-09T17:36:03.178477#1-150120]  INFO -- : Started POST "/api/v1/signshow" for 172.22.0.2 at 2024-07-09 17:36:03 -0600
I, [2024-07-09T17:36:03.189860#1-150120]  INFO -- : Processing by SessionsController#show as JSON
I, [2024-07-09T17:36:03.190006#1-150120]  INFO -- :   Parameters: {"fingerprint"=>"663193932"}
I, [2024-07-09T17:36:03.568656#1-150120]  INFO -- : Completed 200 OK in 378ms (Views: 23.7ms | ActiveRecord: 30.5ms | Allocations: 38776)
I, [2024-07-09T17:36:03.590890#1-150180]  INFO -- : Started GET "/api/v1/translations/lang/en-us?_=1720568163085" for 172.22.0.2 at 2024-07-09 17:36:03 -0600
I, [2024-07-09T17:36:03.603673#1-150180]  INFO -- : Processing by TranslationsController#lang as JSON
I, [2024-07-09T17:36:03.603816#1-150180]  INFO -- :   Parameters: {"_"=>"1720568163085", "locale"=>"en-us"}
I, [2024-07-09T17:36:03.973899#1-150180]  INFO -- : Completed 200 OK in 370ms (Views: 71.8ms | ActiveRecord: 133.6ms | Allocations: 115810)
I, [2024-07-09T17:36:04.104608#1-150300]  INFO -- : Started GET "/api/v1/getting_started?_=1720568163086" for 172.22.0.2 at 2024-07-09 17:36:04 -0600
I, [2024-07-09T17:36:04.116018#1-150300]  INFO -- : Processing by GettingStartedController#index as JSON
I, [2024-07-09T17:36:04.116171#1-150300]  INFO -- :   Parameters: {"_"=>"1720568163086"}
I, [2024-07-09T17:36:04.138693#1-150300]  INFO -- : Completed 200 OK in 22ms (Views: 0.5ms | ActiveRecord: 6.3ms | Allocations: 2691)
I, [2024-07-09T17:36:04.307820#1-150360]  INFO -- : Started GET "/api/v1/getting_started?_=1720568163087" for 172.22.0.2 at 2024-07-09 17:36:04 -0600
I, [2024-07-09T17:36:04.318596#1-150360]  INFO -- : Processing by GettingStartedController#index as JSON
I, [2024-07-09T17:36:04.318721#1-150360]  INFO -- :   Parameters: {"_"=>"1720568163087"}
I, [2024-07-09T17:36:04.335466#1-150360]  INFO -- : Completed 200 OK in 17ms (Views: 0.3ms | ActiveRecord: 6.4ms | Allocations: 1893)
I, [2024-07-09T17:36:04.353022#1-149620]  INFO -- : Started GET "/api/v1/import/zendesk/import_status?_=1720568163088" for 172.22.0.2 at 2024-07-09 17:36:04 -0600
I, [2024-07-09T17:36:04.361470#1-149620]  INFO -- : Processing by ImportZendeskController#import_status as JSON
I, [2024-07-09T17:36:04.361580#1-149620]  INFO -- :   Parameters: {"_"=>"1720568163088"}
I, [2024-07-09T17:36:04.382070#1-149620]  INFO -- : Completed 200 OK in 20ms (Views: 0.7ms | ActiveRecord: 6.6ms | Allocations: 2334)

Hi!

Did you start the migration from the WebUI?

I had the same issue importing around 56k tickets from OTRS.

My suggestion is

I set up a staging installation first and tried it a few times until everything worked fine.

Have fun!

@skip I have been using the Console method, which is why my production.log is so barren.

What do you mean by “set up the nginx server timeouts accordingly,” and what parameters for the workers and concurrent jobs would you recommend?

For the Database and parameters I suggest to read the docs and search the forum posts to learn more about it. Do not blindly apply numbers.

Straight from the documentation:
https://docs.zammad.org/en/latest/prerequisites/hardware.html#performance-tuning

https://docs.zammad.org/en/latest/appendix/configure-env-vars.html#performance-tuning

Some interesting threads:

Sorry for not giving straight answers, but it highly depends on your setup.
e.g. Elasticsearch will also eat up resources while indexing, etc…