Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-27657

Enable S3 Transfer Acceleration

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 10.10
    • Fix Version/s: 9.10-HF41, 10.10-HF23, 11.1, 2021.0
    • Component/s: S3
    • Release Notes Summary:
       S3 Transfer Acceleration is enabled.
    • Impact type:
      Configuration Change
    • Upgrade notes:
      Hide

      The S3 connector now has new nuxeo.conf parameters to configure S3 accelerate mode:

      • nuxeo.s3storage.accelerateMode (default false)
      • nuxeo.s3storage.transient.accelerateMode (default false) (for direct upload)

      For example:

      nuxeo.s3storage.accelerateMode=true
      nuxeo.s3storage.transient.accelerateMode=true
      

      Note that accelerate mode is incompatible with path-style access (NXP-28526), see S3 documentation.

      Show
      The S3 connector now has new nuxeo.conf parameters to configure S3 accelerate mode: nuxeo.s3storage.accelerateMode  (default false ) nuxeo.s3storage.transient.accelerateMode  (default false ) (for direct upload) For example: nuxeo.s3storage.accelerateMode=true nuxeo.s3storage.transient.accelerateMode=true Note that accelerate mode is incompatible with path-style access ( NXP-28526 ), see S3 documentation.
    • Team:
      FG
    • Sprint:
      nxFG 11.1.12

      Description

      We should enable S3 Transfer Acceleration. This is a setting enabled on specific S3 buckets that enables accelerated uploads (and downloads?) with multi-part transfer. Details here: https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html

      I enabled S3 transfer acceleration on my bucket and set my `nuxeo.s3storage.endpoint` but got `exception_message : To enable accelerate mode, please use AmazonS3ClientBuilder.withAccelerateModeEnabled(true)`.

      (Florent Guillaume) "we're not compatible out of the box. That's because of how the AWS SDK is coded. There is a technical reason though: in accelerate mode, the endpoint in request is regionless. We need the client-wide endpoint to fetch the region information and pick the correct signer."

      Among other things, it looks like this contribution needs to add the "accelerateMode" config:
      https://github.com/nuxeo/marketplace-amazon-s3/blob/master/ear/src/main/resources/s3binaries/nxserver/config/s3directupload-config.xml.nxftl#L67

      e.g.:

      <property name="accelerateMode">${nuxeo.s3storage.transient.accelerateMode}</property>
      

      related to:
      https://github.com/nuxeo/nuxeo/blob/10.10/addons/nuxeo-core-binarymanager-cloud/nuxeo-core-binarymanager-s3/src/main/java/org/nuxeo/ecm/core/storage/sql/S3DirectBatchHandler.java#L144

      Even creating an XML override in Studio and setting the accelerateMode config did not enable this to work.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 hours
                  2h