Zendesk Migration Incomplete

[collin.campbell]

Info:

• Used Zammad version: 6.3.1
• Used Zammad installation type: Docker Compose

Expected behavior:

• The Zendesk to Zammad migration to process all 82k tickets and import them to the new instance.

Actual behavior:

• The migration tool stops at either 7k or 8k tickets and then marks itself as complete.

Steps to reproduce the behavior:

• Start a new Zendesk migration and wait for completion; after a while, it should stop at an arbitrary number of tickets.

Additional Info:

Import Job

#<ImportJob:0x00007f9b7c371af8
 id: 1,
 name: "Import::Zendesk",
 dry_run: false,
 payload: {},
 result:
  {"Groups"=>
    {"skipped"=>0,
     "created"=>13,
     "updated"=>0,
     "unchanged"=>0,
     "failed"=>0,
     "deactivated"=>0,
     "sum"=>13,
     "total"=>13},
   "Users"=>
    {"skipped"=>0,
     "created"=>6242,
     "updated"=>8,
     "unchanged"=>0,
     "failed"=>0,
     "deactivated"=>0,
     "sum"=>6250,
     "total"=>6250},
   "Organizations"=>
    {"skipped"=>0,
     "created"=>1,
     "updated"=>0,
     "unchanged"=>1,
     "failed"=>0,
     "deactivated"=>0,
     "sum"=>2,
     "total"=>2},
   "Tickets"=>
    {"skipped"=>0,
     "created"=>5965,
     "updated"=>1035,
     "unchanged"=>0,
     "failed"=>0,
     "deactivated"=>0,
     "sum"=>7000,
     "total"=>7000}},
 started_at: Tue, 09 Jul 2024 23:35:17.161000000 UTC +00:00,
 finished_at: Wed, 10 Jul 2024 02:59:57.207000000 UTC +00:00,
 created_at: Tue, 09 Jul 2024 23:35:11.108000000 UTC +00:00,
 updated_at: Wed, 10 Jul 2024 02:59:57.211000000 UTC +00:00>

Production.log

waiting for init container to finish install or update...
waiting for init container to finish install or update...
starting railsserver... with WEB_CONCURRENCY=0
Puma starting in single mode...
* Puma version: 6.4.2 (ruby 3.2.4-p170) ("The Eagle of Durango")
*  Min threads: 5
*  Max threads: 30
*  Environment: production
*          PID: 1
I, [2024-07-09T17:29:44.475440 #1]  INFO -- : ActionCable is using the redis instance at redis://zammad-redis:6379.
I, [2024-07-09T17:29:44.494380#1-6920]  INFO -- : Using memcached as Rails cache store.
I, [2024-07-09T17:29:44.494601#1-6920]  INFO -- : Using the Redis back end for Zammad's web socket session store.
* Listening on http://[::]:3000
Use Ctrl-C to stop
- Gracefully stopping, waiting for requests to finish
starting railsserver... with WEB_CONCURRENCY=0
Puma starting in single mode...
* Puma version: 6.4.2 (ruby 3.2.4-p170) ("The Eagle of Durango")
*  Min threads: 5
*  Max threads: 30
*  Environment: production
*          PID: 1
I, [2024-07-09T17:33:05.761655 #1]  INFO -- : ActionCable is using the redis instance at redis://zammad-redis:6379.
I, [2024-07-09T17:33:05.792531#1-6920]  INFO -- : Using memcached as Rails cache store.
I, [2024-07-09T17:33:05.792867#1-6920]  INFO -- : Using the Redis back end for Zammad's web socket session store.
- Gracefully stopping, waiting for requests to finish
! Unable to load application: SignalException: SIGTERM
starting railsserver... with WEB_CONCURRENCY=0
Puma starting in single mode...
* Puma version: 6.4.2 (ruby 3.2.4-p170) ("The Eagle of Durango")
*  Min threads: 5
*  Max threads: 30
*  Environment: production
*          PID: 1
I, [2024-07-09T17:33:21.435601 #1]  INFO -- : ActionCable is using the redis instance at redis://zammad-redis:6379.
I, [2024-07-09T17:33:21.454449#1-6920]  INFO -- : Using memcached as Rails cache store.
I, [2024-07-09T17:33:21.454876#1-6920]  INFO -- : Using the Redis back end for Zammad's web socket session store.
* Listening on http://[::]:3000
Use Ctrl-C to stop
I, [2024-07-09T17:36:02.619318#1-149620]  INFO -- : Started GET "/" for 172.22.0.2 at 2024-07-09 17:36:02 -0600
I, [2024-07-09T17:36:02.788936#1-149620]  INFO -- : Processing by InitController#index as HTML
I, [2024-07-09T17:36:02.857283#1-149620]  INFO -- :   Rendered init/index.html.erb within layouts/application (Duration: 2.4ms | Allocations: 451)
I, [2024-07-09T17:36:02.863748#1-149620]  INFO -- :   Rendered layout layouts/application.html.erb (Duration: 9.1ms | Allocations: 1761)
I, [2024-07-09T17:36:02.864850#1-149620]  INFO -- : Completed 200 OK in 76ms (Views: 15.2ms | ActiveRecord: 18.8ms | Allocations: 7955)
I, [2024-07-09T17:36:03.178477#1-150120]  INFO -- : Started POST "/api/v1/signshow" for 172.22.0.2 at 2024-07-09 17:36:03 -0600
I, [2024-07-09T17:36:03.189860#1-150120]  INFO -- : Processing by SessionsController#show as JSON
I, [2024-07-09T17:36:03.190006#1-150120]  INFO -- :   Parameters: {"fingerprint"=>"663193932"}
I, [2024-07-09T17:36:03.568656#1-150120]  INFO -- : Completed 200 OK in 378ms (Views: 23.7ms | ActiveRecord: 30.5ms | Allocations: 38776)
I, [2024-07-09T17:36:03.590890#1-150180]  INFO -- : Started GET "/api/v1/translations/lang/en-us?_=1720568163085" for 172.22.0.2 at 2024-07-09 17:36:03 -0600
I, [2024-07-09T17:36:03.603673#1-150180]  INFO -- : Processing by TranslationsController#lang as JSON
I, [2024-07-09T17:36:03.603816#1-150180]  INFO -- :   Parameters: {"_"=>"1720568163085", "locale"=>"en-us"}
I, [2024-07-09T17:36:03.973899#1-150180]  INFO -- : Completed 200 OK in 370ms (Views: 71.8ms | ActiveRecord: 133.6ms | Allocations: 115810)
I, [2024-07-09T17:36:04.104608#1-150300]  INFO -- : Started GET "/api/v1/getting_started?_=1720568163086" for 172.22.0.2 at 2024-07-09 17:36:04 -0600
I, [2024-07-09T17:36:04.116018#1-150300]  INFO -- : Processing by GettingStartedController#index as JSON
I, [2024-07-09T17:36:04.116171#1-150300]  INFO -- :   Parameters: {"_"=>"1720568163086"}
I, [2024-07-09T17:36:04.138693#1-150300]  INFO -- : Completed 200 OK in 22ms (Views: 0.5ms | ActiveRecord: 6.3ms | Allocations: 2691)
I, [2024-07-09T17:36:04.307820#1-150360]  INFO -- : Started GET "/api/v1/getting_started?_=1720568163087" for 172.22.0.2 at 2024-07-09 17:36:04 -0600
I, [2024-07-09T17:36:04.318596#1-150360]  INFO -- : Processing by GettingStartedController#index as JSON
I, [2024-07-09T17:36:04.318721#1-150360]  INFO -- :   Parameters: {"_"=>"1720568163087"}
I, [2024-07-09T17:36:04.335466#1-150360]  INFO -- : Completed 200 OK in 17ms (Views: 0.3ms | ActiveRecord: 6.4ms | Allocations: 1893)
I, [2024-07-09T17:36:04.353022#1-149620]  INFO -- : Started GET "/api/v1/import/zendesk/import_status?_=1720568163088" for 172.22.0.2 at 2024-07-09 17:36:04 -0600
I, [2024-07-09T17:36:04.361470#1-149620]  INFO -- : Processing by ImportZendeskController#import_status as JSON
I, [2024-07-09T17:36:04.361580#1-149620]  INFO -- :   Parameters: {"_"=>"1720568163088"}
I, [2024-07-09T17:36:04.382070#1-149620]  INFO -- : Completed 200 OK in 20ms (Views: 0.7ms | ActiveRecord: 6.6ms | Allocations: 2334)

[skip]

Hi!

Did you start the migration from the WebUI?

I had the same issue importing around 56k tickets from OTRS.

My suggestion is

• to set up the nginx server timeouts accordingly
• tune zammad workers and concurrent jobs Configuration via Environment Variables — Zammad System Documentation documentation
• use the console as explained here From Zendesk — Zammad System Documentation documentation
  
  

I set up a staging installation first and tried it a few times until everything worked fine.

Have fun!

[collin.campbell]

@skip I have been using the Console method, which is why my production.log is so barren.

What do you mean by “set up the nginx server timeouts accordingly,” and what parameters for the workers and concurrent jobs would you recommend?

[skip]

For the Database and parameters I suggest to read the docs and search the forum posts to learn more about it. Do not blindly apply numbers.

Straight from the documentation:
https://docs.zammad.org/en/latest/prerequisites/hardware.html#performance-tuning

https://docs.zammad.org/en/latest/appendix/configure-env-vars.html#performance-tuning

Some interesting threads:

Scheduler fails with "ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use>"

That fixes it for me, as well.

Zammad web UI slow / detailed documentation missing: How to troubleshoot?

Infos: Used Zammad version: 5.3.0-1671105844.50103cfd.centos7 Used Zammad installation type: RPM from repository Operating system: Linux (CentOS 7) Browser + version: Firefox, Chrome (various versions on various OSes) Problem Description: Since a few Zammad updates, Zammad web UI actions are slow. It can take up to 5 to 15 (or more) seconds for simple ticket updates like owner or state changes or email replies to become active and updated in the browser. It’s not a browser or connectivity i…

Zammad 6.3 performance issues

Infos: Zammad Version: 6.3.1 Installation Type: Package Operating System: Ubuntu 22.04.4 LTS Browser + Version: Any browser (currently using Chrome, Version 125.0.6422.113) Expected Behavior: Fast ticket display in overviews, quick API responses, efficient database connections, and overall fast performance. Actual Behavior: A high volume of tickets being created and updated simultaneously causes a backlog of jobs. Zammad starts using 100% CPU and displays the message ‘Connection lost’ to …

Scheduler fails with "ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use>"

Infos: Used Zammad version: 6.3.0 Used Zammad installation type: docker-compose Operating system: Ubuntu 22.04.4 LTS (jammy) Browser + version: Chrome 124.0.6367.201 Expected behavior: Scheduler able to finish tasks Actual behavior: Some scheduled tasks never finish (mostly Channel.stream, Sessions.jobs, but also others), showing errors in the zammad-scheduler container. Yet some tasks seem to complete just fine. (The LDAP sync and Office 365 integrations do seem to update). Monitor tab sho…

Sorry for not giving straight answers, but it highly depends on your setup.
e.g. Elasticsearch will also eat up resources while indexing, etc…

[collin.campbell]

Sorry for the wait on my response; I have a lot of projects going on.

Even after adjusting and tuning for best performance and database connectivity, it always stops at exactly 8000 tickets processed.