Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-28526

Make S3 Direct Upload compatible with S3-like storage

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 9.10, 10.10
    • Fix Version/s: 9.10-HF41, 10.10-HF23, 11.1, 2021.0
    • Component/s: S3
    • Release Notes Summary:
      S3 Direct Upload is compatible with S3-like storage.
    • Tags:
    • Backlog priority:
      800
    • Impact type:
      Configuration Change
    • Upgrade notes:
      Hide

      S3 direct upload now has new nuxeo.conf parameters to configure a custom S3 endpoint and activate path-style access:

      • nuxeo.s3storage.transient.endpoint (default empty)
      • nuxeo.s3storage.transient.pathstyleaccess (default false)

      For example:

      nuxeo.s3storage.transient.endpoint=https://s3.us-east-1.amazonaws.com
nuxeo.s3storage.transient.pathstyleaccess=true

      Note that path-style access is incompatible with accelerate mode (NXP-27657), see S3 documentation.

      Show
      S3 direct upload now has new nuxeo.conf parameters to configure a custom S3 endpoint and activate path-style access: nuxeo.s3storage.transient.endpoint  (default empty) nuxeo.s3storage.transient.pathstyleaccess  (default false ) For example: nuxeo.s3storage.transient.endpoint=https://s3.us-east-1.amazonaws.com nuxeo.s3storage.transient.pathstyleaccess=true Note that path-style access is incompatible with accelerate mode ( NXP-27657 ), see S3 documentation.
    • Team:
      FG
    • Sprint:
      nxFG 11.1.12

      Description

      Using S3 Direct Upload with an S3-like storage will fail with the following exception

      ERROR [WebEngineExceptionMapper] com.amazonaws.SdkClientException: Unable to execute HTTP request: sts.amazonaws.com
com.amazonaws.SdkClientException: Unable to execute HTTP request: sts.amazonaws.com
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1175) ~[aws-java-sdk-core-1.11.516.jar:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1121) ~[aws-java-sdk-core-1.11.516.jar:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770) ~[aws-java-sdk-core-1.11.516.jar:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744) ~[aws-java-sdk-core-1.11.516.jar:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) ~[aws-java-sdk-core-1.11.516.jar:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) ~[aws-java-sdk-core-1.11.516.jar:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) ~[aws-java-sdk-core-1.11.516.jar:?]
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) ~[aws-java-sdk-core-1.11.516.jar:?]
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) ~[aws-java-sdk-core-1.11.516.jar:?]
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1368) ~[aws-java-sdk-sts-1.11.516.jar:?]
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1335) ~[aws-java-sdk-sts-1.11.516.jar:?]
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1324) ~[aws-java-sdk-sts-1.11.516.jar:?]
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:491) ~[aws-java-sdk-sts-1.11.516.jar:?]
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:464) ~[aws-java-sdk-sts-1.11.516.jar:?]
        at org.nuxeo.ecm.core.storage.sql.S3DirectBatchHandler.assumeRole(S3DirectBatchHandler.java:215) ~[nuxeo-core-binarymanager-s3-10.10-HF15.jar:?]
        at org.nuxeo.ecm.core.storage.sql.S3DirectBatchHandler.getBatch(S3DirectBatchHandler.java:199) ~[nuxeo-core-binarymanager-s3-10.10-HF15.jar:?]
        at org.nuxeo.ecm.automation.server.jaxrs.batch.BatchManagerComponent.lambda$hasBatch$2(BatchManagerComponent.java:209) ~[nuxeo-automation-server-10.10-HF15.jar:?]

      because the parameter nuxeo.s3storage.endpoint is not used to configure the S3 client.

      https://github.com/nuxeo/nuxeo/blob/master/addons/nuxeo-core-binarymanager-cloud/nuxeo-core-binarymanager-s3/src/main/java/org/nuxeo/ecm/core/storage/sql/S3DirectBatchHandler.java#L175

      Using the method withEndpointConfiguration could provide a solution here

                      .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(S3_ENDPOINT_PARAM))

      In addition, the pathStyleAccess must be configurable as well, similar to what is possible for a blob provider (NXP-25525).

        Attachments

          Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. NXP-25525 Expose the "pathStyleAccess" s3 client option within a new configuration key

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. NXPY-158 Allow S3 custom endpoint for direct upload

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. NXP-27657 Enable S3 Transfer Acceleration

          • Major - Major loss of function.
          • Resolved
          Is referenced in

          [Captain Hook] PR for master: #4165 PR for master: #4165

          [Captain Hook] PR for master: #4294 PR for master: #4294

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 3 hours
                  3h