OTRS 6 migration to Zammad (test instance for evaluation)

[skip]

Hello,

I wanted to start a thread to share how I migrated OTRS 6 to Zammad to evaluate the use of Zammad within our organisation and will use this thread to keep track of my findings and experience.

I am thankful for all contributions to this forum and would like to thank every contributor. This would not have been possible without you!

Use case

IT Support for a regional schools’ network
5 Service centers (sites where agents are located)
~150 Organisations
~230 Sites (an organisation can have more than one site)

OTRS 6

• Version
• OS Centos 7
• Apache2 (httpd)
• 1229 users (agents and customers)
• 51668 tickets

Zammad

• Version 6.2.0-1707172848.16867851.bookworm
• OS Debian 12
  • OS and SSH hardening
  • Unattended updates for security updates
• PostgreSQL 15
• nginx
• Distribution package installation (not docker, not source code)

Migration time 6h on a notebook with two VMs on Hyper-V and 16GB RAM in total

Migration setup

OTRS6 Server

• Activate unsigned packages support
  https://support.example.com/otrs/index.pl?Action=AdminSystemConfiguration;Subaction=View;Setting=Package%3A%3AAllowNotVerifiedPackages;
• Install migration plugins following the official guide
  https://docs.zammad.org/en/latest/migration/otrs.html
• Avoid OOM (Out of memory)
  • Create a file with the path /etc/httpd/conf.d/mpm_prefork.conf, and add the following command block to the file:

  <IfModule mpm_prefork_module>  
  StartServers 5  
  MinSpareServers 5  
  MaxSpareServers 10  
  MaxRequestWorkers 30  
  MaxConnectionsPerChild  0  
  </IfModule>
  

• Test configuration
  httpd -t
• Restart httpd
  systemctl restart httpd

Zammad Server Setup

• Set file storage for attachments before import

  • (as root)

  zammad run rails c
  Setting.get('storage_provider')
  Setting.set('storage_provider', 'File')
  Setting.get('storage_provider')
  Store::File.move('DB','File')
  

• Import via CLI

  • Set parameters, enter import mode and start import
    (continue in the rails console)

  Setting.set('import_otrs_endpoint', 'https://support.example.com/otrs/public.pl?Action=ZammadMigrator')
  Setting.set('import_otrs_endpoint_key', '<YOURKEY>')
  Setting.set('import_mode', true)
  Import::OTRS.start
  

  Once done exit import mode

  Setting.set('import_mode', false)
  Setting.set('system_init_done', true)
  Rails.cache.clear
  

Troubleshooting post import

CSRF token verification failed

Since OTRS 6 was running with a SSL certificate, we need to set up a self signed SSL certificate in Zammad to avoid the issue (reminder this is an evalutation installation and not production, therefore we will not register an SSL cert)

• Set up a self signed SSL for nginx

  apt-get install openssl
  openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/nginx.key -out /etc/ssl/certs/nginx.crt
  openssl dhparam -out /etc/nginx/dhparam.pem 4096
  

  Add /etc/nginx/snippets/self-signed.conf

  ssl_certificate /etc/ssl/certs/nginx.crt;
  ssl_certificate_key /etc/ssl/private/nginx.key;
  ssl_protocols TLSv1.2;
  ssl_prefer_server_ciphers on;
  ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
  ssl_session_timeout 10m;
  ssl_session_cache shared:SSL:10m;
  ssl_session_tickets off;
  ssl_stapling on;
  ssl_stapling_verify on;
  resolver 8.8.8.8 8.8.4.4 valid=300s;
  resolver_timeout 5s;
  add_header X-Frame-Options DENY;
  add_header X-Content-Type-Options nosniff;
  add_header X-XSS-Protection "1; mode=block";
  ssl_dhparam /etc/nginx/dhparam.pem;
  ssl_ecdh_curve secp384r1;
  

Open issues

file-2 attachment

Like https://community.zammad.org/t/file-1-file-2-attachment-after-migration-from-otrs/9276 - No solution found yet.
Since these are old tickets to be kept as archive. I do personally not see it as issue

Rethinking the structure

The goal is to have a correct group/organization structure and notifications for the agents
This is the hardest part as I am trying to figure out how that works correctly.

Example:
1 Organization (A)
3 Sites (A1, A2, A3)
3 Customers (CA1, CA2, CA3)
3 Technicians (TA1, TA2, TA3)

New tickets should be assigned to the correct technician (e.g. TA1 for tickets from CA1 for A1) but all other TA (or TB3 or TC4 for other organizations) should not be notified or see the ticket

Set up email boxes (or better MS365 channels)

Same as above, I still need some time to figure out how the things are connected correctly and how to set it up appropriately. I hope I’ll be able to share the setups too

Open points

Try OpenSearch

I know it is not supported, but it’s a good time to start trying it out before putting it in production

Testing with Agents and Customers

Reporting

Excuse the wall of text and have a good day.

[skip]

I delved into the various setups within Zammad, exploring its configurations involving groups, organizations, roles, users, permissions, and notifications.

From what I understood, what was previously termed a “queue” in OTRS now corresponds to a “group” in Zammad. Although direct connections between groups and users or roles aren’t feasible, group permissions can be established at either the role or user level.

The organization setting serves purposes such as user filtering, trigger definition, and overview establishment, yet it does not impact groups, notifications, or permissions significantly, apart from facilitating ticket sharing among organization-affiliated customers.
Given that our users all share the same email domain managed at a regional level, filtering by domain name isn’t viable.

Use Case:

• A school may belong to a group of schools (Sprengel).
• Support requests from a school are submitted via web and email.
• Each school may have one or more teachers requesting support (customers).
• A school typically has one main technician and a backup technician (agents).

How To:

• Establish one organization per school, rather than per school group (Sprengel), with shared organization disabled.
• (Optional) Modify organization object to add field for school group (Sprengel)
• Create one organization per service center (agent site).
• Set up a distinct group for each school.
• Designate a group as “SUPPORT” to ensure all tickets are routed there, simplifying the user experience for teachers (customers).
• Create one role per service center.
• Link customers to their respective schools (main organization).
• Assign technicians (agents) to the service center (main organization) and schools (secondary organizations).
• For each technician (agent, not role), specify the school(s) (group) they primarily serve as well as their backup school(s) (group), granting them FULL Group Permissions to ensure they receive Zammad notifications and emails.
• Implement a trigger for every group to ensure that when a ticket with status new from a specific school (organization) within the “SUPPORT” group is moved to the corresponding school group (group).

This setup ensures:

• Every teacher’s (customer) request is directed to the designated technician (and backup technician) (agent)
• Only specified technicians (agent) receive notifications via Zammad and email.

Did I overlook anything?

Next on the agenda: configuring the MS365 channel, which may present challenges due to the lack of testing capabilities.

[skip]

Technical update

Zammad is being installed on virtual machines with Ubuntu 22.04 using ansible (for repeatability, compliance, etc…). Mainly inspired by https://github.com/hifis-net/ansible-role-zammad (thank you!)

Issue

Zammad, Elasticsearch and PostgreSQL are being installed on three different servers (redis might follow). Since Zammad has hard dependencies on elasticsearch, postgresql and other packages like redis, there is no way to install services on separate servers.

Solution

To solve it I am building dummy packages of the hard dependecies using equivs on the Zammad server. It is not perfect but makes installing Zammad on a cloud like environment possible.

[skip]

Custom CSS

In this post I figured out how to add some custom CSS e.g. to hide the Activity Stream and First Steps in the Dashboard.

Custom CSS foes

Thanks @Sal! I figured it out! First the permissions on the custom.css file were wrong, then I verified also the public/assets directory and found that all *.css files had the wrong permission. This is how I fixed it: systemctl stop zammad vi /opt/zammad/app/assets/stylesheets/custom/custom.css chmod 644 /opt/zammad/app/assets/stylesheets/custom/custom.css #rw-r-r chmod 664 -R /opt/zammad/public/assets/*.css #rw-rw-r zammad run rake assets:precompile systemctl start zammad Now the CSS is co…

[skip]

In any case this is how I set up Zammad, Postgresql and Elasticsearch-oss on three different servers with ansible

ansible-zammad

It is a custom working solution and might not work with every setup, so please if at all, only use it as reference.

[skip]

Here are the refined updates to my strategy:

In order to ensure consistent replication across various systems such as testing, staging, and production using Ansible or similar tools, I’ve made progress in importing roles, organizations, and workflows utilizing the Zammad API with my humble Python skills through zammad.py, as well as employing the official import function or the Rails console.

One notable challenge I encountered was documented in this post: Import Organizations: IDs do not start with ID 2 when imported using CSV . It required ensuring that the organization import process doesn’t encounter any 504 errors.

To overcome this challenge, I opted to divide the import file into four separate files. This approach guarantees consistency in Organization IDs across different systems—be it staging or production—ensuring smooth transitioning for the subsequent step of utilizing static CSV files to import users consistently with identical properties.

Noteworthy adjustments in the strategy include:

• Transitioning from a one-group-per-school group to a one-per-district group.
• Instead of relying on triggers to ensure that tickets are directed to the appropriate group, I’ve implemented a solution using core workflows. This involves having a workflow to conceal all groups initially, followed by individual workflows for each organization to reveal and designate the relevant group. This approach also resolves the occasional scenario where a customer is affiliated with multiple organizations.

Moving forward, the next phase involves updating user roles and organizations once all components are in place.

[skip]

In the last update, I was able to build scripts to import core workflows, triggers, overviews, roles, and users from JSON or CSV files. This enables me to replicate my setup from one system to another seamlessly.

To move Core Workflows from Staging to Production, it’s necessary to maintain the same structure on both systems. I use Python and API calls to delete all existing workflows and import the new ones. Postman is my tool of choice for making API calls; I call the API with GET in staging, copy the JSON body to a file, and then import it using Python. This method ensures I always have a backup of the last import saved in a repository.

Additionally, when migrating from OTRS, there is an issue with email addresses. OTRS allows multiple users to have the same email address, but this poses a problem in Zammad. Users logging in with a duplicate email address will encounter a blank Zammad.

To address this, you need to clean up the users in Zammad. Export the list of users and change the email addresses of duplicate accounts to random ones. Then, depending on how these accounts were used, you’ll need to decide on the best way to handle them.

See you next time!

[skip]

After four months of hard work setting up DevOps processes, importing data from OTRS, getting familiar with the API, and creating custom tools to ensure reproducibility, I can announce that Zammad officially went live as of July 1st!

Challenges Faced

The journey wasn’t without its hurdles. Here are some notable challenges and how they were tackled:

1. OTRS Import Issues:

• The Znuny exporter tools for OTRS worked well.
  However, we encountered problems where user IDs were scrambled after importing around 1200 users. IDs for the first 1200 users stayed consistent, but from 1200 to 1600, they were randomly reassigned.
  The issue with inconsistent IDs is something that somehow is consistent within the project OTRS Import Discussion

2. Permission ID Changes:

• Silent changes in permission IDs between versions 6.2 and 6.3, especially after the addition of WhatsApp Business, required manual adjustments since IDs were not treated as unique values.

3. API Limitations:

• Not all configurations can be altered through the API, necessitating some manual intervention, particularly for settings.
• Reference: API Settings Issue

4. Translation Workarounds:

• To handle translations, the only was is to use an API workaround to seamlessly add the required translations.
• Reference: Translation Issue

5. Non-Customizable Notifications:

• Setting up notifications for approximately 450 vertical sites was challenging due to the lack of customization options. However, through tables, APIs, and scripts it was possible to manage this too.
• Reference: Notification Customization Discussion

Positive Aspects

Many features work out of the box. Core Workflows are instrumental in customizing the experience. Apart from minor adjustments to notification emails and adding custom triggers, the operator-side experience has been smooth.

Looking Ahead

With the summer season being slow, we expect fewer tickets and more time to refine the setup. From autumn onwards, increased real-life use will provide valuable feedback on Zammad’s performance.

Acknowledgements

A huge thanks to @fliebe92, @MrGeneration, and the entire community for their patience and support. This forum has been an invaluable resource, and with the collective knowledge here, setting up Zammad has been a rewarding experience.

See you around,
Skip