Backblaze S3 broken after update to 6.5

[vanOert]

Infos:

• Used Zammad version: 6.5.0-1743671234.61b4a196.jammy
• Used Zammad installation type: package
• Operating system: Ubuntu 22.04 LTS
• Browser + version: Chrome 134.0.6998

Expected behavior:

• Update from 6.4 to 6.5 should leave working S3 configuration (backblaze b2) for attachment storage intact

Actual behavior:

• Old attachments (before update) can no longer be opened, new attachments can not be imported. Needed to revert to file system in order to keep system operational for incoming tickets.
• Opening old attachments hosted on S3 leads to error message: 422: The change you wanted was rejected. Simple Storage Service malfunction. Please contact your Zammad administrator.
• Attempted searchindex:rebuild leads to error message: Aws::S3::Errors::InvalidArgument: Unsupported header ‘x-amz-checksum-mode’ received for this API call. (Aws::S3::Errors::InvalidArgument)

Steps to reproduce the behavior:

• Configure backblaze S3 bucket in storage.yml in zammad version 6.4 or earlier, update to 6.5
• Note: Test recommended in documentation (zammad run rails r ‘Rails.logger = Logger.new(STDOUT); pp Store::Provider::S3.ping?’) will state “true” even with non working config in 6.5

[MrGeneration]

How does your config, without access credentials, look like…?

[vanOert]

# This is the Zammad storage provider config sample. Copy this file to
# config/zammad/storage.yml and adopt your configuration.

# Amazon AWS S3 compatible simple storage service.
#
# For further information please see the official Amazon AWS S3 guide
# https://docs.aws.amazon.com/AmazonS3/latest/userguide
s3:
  access_key_id: 'redacted'
  secret_access_key: 'redacted'
  region: 'eu-central'
  endpoint: 'https://s3.eu-central-003.backblazeb2.com'
  force_path_style: true
  bucket: 'redacted'

# The above configuration may also be provided in a URL-like format in this
# configuration file or as an environment variable named S3_URL.
# If the url is provided in the configuration file the environment variable is
# ignored.
#s3:
#  url: 'https://key:secret@s3.eu-central-1.amazonaws.com/zammad-storage-bucket?region=eu-central-1&force_path_style=true'

[vanOert]

Thanks for the quick reply in any case. If it’s of any use, we’d be happy to supply you with credentials for a b2 S3 bucket in our account for testing purposes.

[MrGeneration]

Okay I might have bad news.
It seems like backblaze doesn’t support these headers (see S3-Compatible API)

Now I’ve checked the upstream gem that’s responsible in Zammad for this. It has received a version bumb from 1.162 to 1.182 (in between 6.4 and 6.5).

I’ve checked the gems release notes and cannot spot any change that’s connected. There is a option in the gem to disable the checksum_mode (which should solve this issue), however, it is not implemented.

E, [2025-04-03T22:48:26.426956#3508-6760] ERROR -- : Store::Provider::S3: invalid configuration option `:checksum_mode'

Only quick option I see for you right now is a downgrade to Zammad 6.4.2 which contains at least the security fixes, but you’ll loose the new features. At least you’d be able to work, I guess.

Temporary workaround (maybe?) would be downgrade, file move to local storage (for now) and then the update back to 6.5 if that’s helping.

[vanOert]

Thanks a lot for your suggestions. Is there any documentation on downgrading the version? We can live without the new features in 6.5 for now. File move to local storage would also be feasible for now but would cause infra issues for us in the short/midterm.

Regarding the checksum_mode disable, do you have an estimate on how much time it might take to implement that or if we could realistically calculate with that happening in the next weeks/months?

[fliebe92]

Hi @vanOert. Could you please update your S3 configuration as shown below?

s3:
  access_key_id: 'redacted'
  secret_access_key: 'redacted'
  region: 'eu-central'
  endpoint: 'https://s3.eu-central-003.backblazeb2.com'
  force_path_style: true
  bucket: 'redacted'
  request_checksum_calculation: when_required
  response_checksum_validation: when_required

Maybe this helps…

[MrGeneration]

I can confirm that

request_checksum_calculation: when_required
response_checksum_validation: when_required

work with my backblaze trial account, so should work for OP.
This isn’t documented anywhere right…?

[vanOert]

@fliebe92 Changed config according to your suggestion. It seems to be working, we can access attachments from b2 S3 again. Searchindex:rebuild will take quite a while to complete but hasn’t run into the error yet. Thanks a lot. I’ll mark this solution as soon as we’re through.

@MrGeneration thanks for testing

[vanOert]

So, 180k tickets and 175GB of attachments reindexed successfully. Thanks again for the very fast solution. Since there may be others using b2 S3, may I suggest adding this info to your docs?

[fliebe92]

Our docs are on GitHub as well, feel free to make a suggestion as issue here: GitHub · Where software is built

Thanks