Problems during migration from OTRS 4.0 to Zammad

plaumannd · January 7, 2021, 1:33pm

Dear Zammad-Community,

hope you can help:

Infos:

Zammad Machine

Used Zammad version: Zammad 3.6
Used Zammad installation source: RPM
Operating system: SLES 12.5
Browser + version: Chrome 87.0.4280.88 (64-Bit)
Additional: 4 Cores, 12 GB RAM

OTRS Machine

Used OTRS version: OTRS 4.0
Used OTRS installation source: RPM
Operating system: SLES 11.4
Browser + version: Chrome 87.0.4280.88 (64-Bit)
Migrator Module Version: 4.1.11
Migrator Repo Version: 4.0.25
Additional: Virtual Machine with 16GB RAM, 4 Cores

Expected behavior:

Imports all Tickets (round about 5,5k) from OTRS to Zammad.

Actual behavior:

Tried two hardware configurations for the OTRS virtual machine, each results in different errors.

** 2 Cores, 2 GB RAM, 1 GB Swap **

Round about 2,1k of 5,5k gets imported into Zammad.
Threads dying due to out of memory error until no thread is alive anymore, than the whole process closes himself.

Tried to incease the amount of Memory and Cores of the OTRS Machine to prevent the memory error and a lack of performance.

** 4 Cores, 16GB Ram, 1 GB Swap **

After 600 Tickets, the OTRS Machine is responding very slow (ssh session, enter commands).
The OTRS Migrator throws ReadTimeOut Errors, because the load on the OTRS Machine is very high:

thread#6: ERROR: #<Net::ReadTimeout: Net::ReadTimeout>
#<Thread:0x00005597f4e270c0@/opt/zammad/vendor/bundle/ruby/2.6.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
9: from /opt/zammad/vendor/bundle/ruby/2.6.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context' 8: from /opt/zammad/lib/import/otrs.rb:102:in block (2 levels) in threaded_import’
7: from /opt/zammad/lib/import/otrs.rb:102:in loop' 6: from /opt/zammad/lib/import/otrs.rb:108:in block (3 levels) in threaded_import’
5: from /opt/zammad/lib/import/otrs.rb:144:in imported?' 4: from /opt/zammad/lib/import/otrs.rb:150:in import_action’
3: from /opt/zammad/lib/import/otrs/requester.rb:35:in load' 2: from /opt/zammad/lib/import/otrs/requester.rb:80:in request_result’
1: from /opt/zammad/lib/import/otrs/requester.rb:93:in request_json' /opt/zammad/lib/import/otrs/requester.rb:133:in post’: Zammad Migrator returned an error (RuntimeError)
Traceback (most recent call last):
9: from lib/import/otrs.rb:102:in block (2 levels) in threaded_import' 8: from lib/import/otrs.rb:102:in loop’
7: from lib/import/otrs.rb:108:in block (3 levels) in threaded_import' 6: from lib/import/otrs.rb:144:in imported?’
5: from lib/import/otrs.rb:150:in import_action' 4: from lib/import/otrs/requester.rb:35:in load’
3: from lib/import/otrs/requester.rb:80:in request_result' 2: from lib/import/otrs/requester.rb:93:in request_json’
1: from lib/import/otrs/requester.rb:133:in `post’
RuntimeError (Zammad Migrator returned an error)
irb(main):007:0> thread#1: ERROR: #<Net::ReadTimeout: Net::ReadTimeout>
…
(Content Collapsed)
…

Steps to reproduce the behavior:

Not exactly reproducable. Tried multiple times with same Settings and the appearence of the Errors are each time different.

Now i want to ask, if it is possible to reduce the amount of threads which gets generated, to reduce the parallel workload.

I would be happy about every suggestion you have for me

MrGeneration · January 14, 2021, 1:46pm

Yes it is!
Had to dig a bit into the code but could find the part where it lifes.

This part hints on how to use it:

github.com

zammad/zammad/blob/develop/lib/import/otrs.rb#L38


      
            updateable_objects
          
            customer_user
          
            threaded_import('Ticket', args)
          
            true
          end
          
          def connection_test
            Import::OTRS::Requester.connection_test
          end
          
          private
          
          def checks
            check_import_mode
            check_system_init_done
            connection_test
          end

By default Zammad seems to use 8 threads which my be an issue on some systems:

github.com

zammad/zammad/blob/develop/lib/import/otrs.rb#L83


      
          # is not reflected to threads. So an import error message appears.
          # Reset needed model column information for each thread.
          reset_database_information
          
          Thread.current[:thread_no]  = thread
          Thread.current[:loop_count] = 0
          
          log "Importing #{remote_object} in steps of #{limit}"
          loop do
            # get the offset for the current thread and loop count
            thread_offset_base = (Thread.current[:thread_no] - 1) * limit
            thread_step        = thread_count * limit
            offset             = (Thread.current[:loop_count] * thread_step) + thread_offset_base + start_offset_base
          
            break if !imported?(
              remote_object: remote_object,
              limit:         limit,
              offset:        offset,
              diff:          args[:diff]
            )

So in your case you’ll want to run a console based import (according to: Migration from OTRS — Zammad System Documentation documentation)

Instead of Import::OTRS.start you’ll want to run Import::OTRS.start(threads: 2).
Please keep in mind that this may drastically increase import duration.

Good luck!

plaumannd · January 19, 2021, 6:58am

Hello MrGeneration,

thank you very much, thats exactly what I searched for. It worked very well.

I guess this topic can be closed as solved

system · May 19, 2021, 6:58am

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.