OTRS Migration invalid byte sequence in UTF-8 (ArgumentError)

Technical assistance
1
  • Used Zammad version: 2.8
  • Used Zammad installation source: source
  • Operating system: ubuntu 18.04 LTS
  • Browser + version: Chrome 71

Expected behavior:

  • Migration from OTRS to Zammad

Actual behavior:

thread#1: Ticket 47776 - 7af138b2e027fca887556e717c37f77434421dd3ede93a32e5874eb0ad5f0f97 - invalid byte sequence in UTF-8: {"Disposition"=>"attachment", "FilesizeRaw"=>16745, "ContentID"=>"", "Content"=>"

thread#4: update Ticket::Article.find_by(id: 344972)
/opt/zammad/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6.1/lib/active_support/core_ext/object/blank.rb:120:in `match?': invalid byte sequence in UTF-8 (ArgumentError)
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6.1/lib/active_support/core_ext/object/blank.rb:120:in `blank?'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activemodel-5.1.6.1/lib/active_model/validator.rb:149:in `block in validate'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activemodel-5.1.6.1/lib/active_model/validator.rb:147:in `each'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activemodel-5.1.6.1/lib/active_model/validator.rb:147:in `validate'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:413:in `block in make_lambda'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:197:in `block (2 levels) in halting'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:601:in `block (2 levels) in default_terminator'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:600:in `catch'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:600:in `block in default_terminator'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:198:in `block in halting'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:507:in `block in invoke_before'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:507:in `each'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:507:in `invoke_before'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:130:in `run_callbacks'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:827:in `_run_validate_callbacks'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activemodel-5.1.6.1/lib/active_model/validations.rb:405:in `run_validations!'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activemodel-5.1.6.1/lib/active_model/validations/callbacks.rb:114:in `block in run_validations!'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:97:in `run_callbacks'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:827:in `_run_validation_callbacks'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activemodel-5.1.6.1/lib/active_model/validations/callbacks.rb:114:in `run_validations!'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activemodel-5.1.6.1/lib/active_model/validations.rb:335:in `valid?'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6.1/lib/active_record/validations.rb:65:in `valid?'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6.1/lib/active_record/validations.rb:82:in `perform_validations'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6.1/lib/active_record/validations.rb:50:in `save!'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6.1/lib/active_record/attribute_methods/dirty.rb:43:in `save!'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6.1/lib/active_record/transactions.rb:313:in `block in save!'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6.1/lib/active_record/transactions.rb:384:in `block in with_transaction_returning_status'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:233:in `transaction'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6.1/lib/active_record/transactions.rb:210:in `transaction'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6.1/lib/active_record/transactions.rb:381:in `with_transaction_returning_status'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6.1/lib/active_record/transactions.rb:313:in `save!'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6.1/lib/active_record/suppressor.rb:46:in `save!'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6.1/lib/active_record/persistence.rb:51:in `create!'
        from /opt/zammad/app/models/store.rb:54:in `add'
        from /opt/zammad/lib/import/otrs/article/attachment_factory.rb:39:in `block in import_single'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `block in transaction'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6.1/lib/active_record/connection_adapters/abstract/transaction.rb:194:in `block in within_new_transaction'
        from /opt/zammad/.rvm/rubies/ruby-2.4.4/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6.1/lib/active_record/connection_adapters/abstract/transaction.rb:191:in `within_new_transaction'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `transaction'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6.1/lib/active_record/transactions.rb:210:in `transaction'
        from /opt/zammad/lib/import/otrs/article/attachment_factory.rb:38:in `import_single'
        from /opt/zammad/lib/import/otrs/article/attachment_factory.rb:22:in `block in perform_import'
        from /opt/zammad/lib/import/otrs/article/attachment_factory.rb:22:in `each'
        from /opt/zammad/lib/import/otrs/article/attachment_factory.rb:22:in `perform_import'
        from /opt/zammad/lib/import/otrs/article/attachment_factory.rb:16:in `import'
        from /opt/zammad/lib/import/otrs/article.rb:41:in `import'
        from /opt/zammad/lib/import/otrs/article.rb:31:in `initialize'
        from /opt/zammad/lib/import/base_factory.rb:36:in `new'
        from /opt/zammad/lib/import/base_factory.rb:36:in `create_instance'
        from /opt/zammad/lib/import/base_factory.rb:12:in `block in import_action'
        from /opt/zammad/lib/import/base_factory.rb:40:in `each'
        from /opt/zammad/lib/import/base_factory.rb:40:in `import_loop'
        from /opt/zammad/lib/import/base_factory.rb:9:in `import_action'
        from /opt/zammad/lib/import/otrs/ticket.rb:40:in `import'
        from /opt/zammad/lib/import/otrs/ticket.rb:30:in `initialize'
        from /opt/zammad/lib/import/base_factory.rb:36:in `new'
        from /opt/zammad/lib/import/base_factory.rb:36:in `create_instance'
        from /opt/zammad/lib/import/base_factory.rb:12:in `block in import_action'
        from /opt/zammad/lib/import/base_factory.rb:40:in `each'
        from /opt/zammad/lib/import/base_factory.rb:40:in `import_loop'
        from /opt/zammad/lib/import/base_factory.rb:9:in `import_action'
        from /opt/zammad/lib/import/otrs.rb:148:in `import_action'
        from /opt/zammad/lib/import/otrs.rb:137:in `imported?'
        from /opt/zammad/lib/import/otrs.rb:101:in `block (3 levels) in threaded_import'
        from /opt/zammad/lib/import/otrs.rb:95:in `loop'
        from /opt/zammad/lib/import/otrs.rb:95:in `block (2 levels) in threaded_import'
        from /opt/zammad/.rvm/gems/ruby-2.4.4/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
zammad@zammad:~$ ^C

Steps to reproduce the behavior:

  • Start Migration from OTRS
2

I was able to delete the corresponding attachment from OTRS.
But there are many others… I dont want to start the process over and over again just to delete one file.

Is it possible to continue the migration process and just skip the corrupt attachment?

3

With this script i’ve renamed all attachment filenames so that they are in UTF8:

in attachments folder:

find . -type f -print0 | \
perl -n0e '$new = $_; if($new =~ s/[^[:ascii:]]/_/g) {
  print("Renaming $_ to $new\n"); rename($_, $new);
}'
2 Likes
4

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.