Backblaze S3 broken after update to 6.5

Technical assistance
1

Infos:

  • Used Zammad version: 6.5.0-1743671234.61b4a196.jammy
  • Used Zammad installation type: package
  • Operating system: Ubuntu 22.04 LTS
  • Browser + version: Chrome 134.0.6998

Expected behavior:

  • Update from 6.4 to 6.5 should leave working S3 configuration (backblaze b2) for attachment storage intact

Actual behavior:

  • Old attachments (before update) can no longer be opened, new attachments can not be imported. Needed to revert to file system in order to keep system operational for incoming tickets.
  • Opening old attachments hosted on S3 leads to error message: 422: The change you wanted was rejected. Simple Storage Service malfunction. Please contact your Zammad administrator.
  • Attempted searchindex:rebuild leads to error message: Aws::S3::Errors::InvalidArgument: Unsupported header ā€˜x-amz-checksum-modeā€™ received for this API call. (Aws::S3::Errors::InvalidArgument)

Steps to reproduce the behavior:

  • Configure backblaze S3 bucket in storage.yml in zammad version 6.4 or earlier, update to 6.5
  • Note: Test recommended in documentation (zammad run rails r ā€˜Rails.logger = Logger.new(STDOUT); pp Store::Provider::S3.ping?ā€™) will state ā€œtrueā€ even with non working config in 6.5
2

How does your config, without access credentials, look likeā€¦?

1 Like
3 
# This is the Zammad storage provider config sample. Copy this file to
# config/zammad/storage.yml and adopt your configuration.

# Amazon AWS S3 compatible simple storage service.
#
# For further information please see the official Amazon AWS S3 guide
# https://docs.aws.amazon.com/AmazonS3/latest/userguide
s3:
  access_key_id: 'redacted'
  secret_access_key: 'redacted'
  region: 'eu-central'
  endpoint: 'https://s3.eu-central-003.backblazeb2.com'
  force_path_style: true
  bucket: 'redacted'

# The above configuration may also be provided in a URL-like format in this
# configuration file or as an environment variable named S3_URL.
# If the url is provided in the configuration file the environment variable is
# ignored.
#s3:
#  url: 'https://key:secret@s3.eu-central-1.amazonaws.com/zammad-storage-bucket?region=eu-central-1&force_path_style=true'
4

Thanks for the quick reply in any case. If itā€™s of any use, weā€™d be happy to supply you with credentials for a b2 S3 bucket in our account for testing purposes.

5

Okay I might have bad news.
It seems like backblaze doesnā€™t support these headers (see S3-Compatible API)

Now Iā€™ve checked the upstream gem thatā€™s responsible in Zammad for this. It has received a version bumb from 1.162 to 1.182 (in between 6.4 and 6.5).

Iā€™ve checked the gems release notes and cannot spot any change thatā€™s connected. There is a option in the gem to disable the checksum_mode (which should solve this issue), however, it is not implemented.

E, [2025-04-03T22:48:26.426956#3508-6760] ERROR -- : Store::Provider::S3: invalid configuration option `:checksum_mode'

Only quick option I see for you right now is a downgrade to Zammad 6.4.2 which contains at least the security fixes, but youā€™ll loose the new features. At least youā€™d be able to work, I guess.

Temporary workaround (maybe?) would be downgrade, file move to local storage (for now) and then the update back to 6.5 if thatā€™s helping.

6

Thanks a lot for your suggestions. Is there any documentation on downgrading the version? We can live without the new features in 6.5 for now. File move to local storage would also be feasible for now but would cause infra issues for us in the short/midterm.

Regarding the checksum_mode disable, do you have an estimate on how much time it might take to implement that or if we could realistically calculate with that happening in the next weeks/months?

7

Hi @vanOert. Could you please update your S3 configuration as shown below?

s3:
  access_key_id: 'redacted'
  secret_access_key: 'redacted'
  region: 'eu-central'
  endpoint: 'https://s3.eu-central-003.backblazeb2.com'
  force_path_style: true
  bucket: 'redacted'
  request_checksum_calculation: when_required
  response_checksum_validation: when_required

Maybe this helpsā€¦

3 Likes
8

I can confirm that

request_checksum_calculation: when_required
response_checksum_validation: when_required

work with my backblaze trial account, so should work for OP.
This isnā€™t documented anywhere rightā€¦?

1 Like
9

@fliebe92 Changed config according to your suggestion. It seems to be working, we can access attachments from b2 S3 again. Searchindex:rebuild will take quite a while to complete but hasnā€™t run into the error yet. Thanks a lot. Iā€™ll mark this solution as soon as weā€™re through.

@MrGeneration thanks for testing

1 Like