Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-28526

Make S3 Direct Upload compatible with S3-like storage

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 9.10, 10.10
    • Fix Version/s: 9.10-HF41, 10.10-HF23, 11.1
    • Component/s: S3
    • Release Notes Summary:
      S3 Direct Upload is compatible with S3-like storage.
    • Tags:
    • Backlog priority:
      800
    • Impact type:
      Configuration Change
    • Upgrade notes:
      Hide

      S3 direct upload now has new nuxeo.conf parameters to configure a custom S3 endpoint and activate path-style access:

      • nuxeo.s3storage.transient.endpoint (default empty)
      • nuxeo.s3storage.transient.pathstyleaccess (default false)

      For example:

      nuxeo.s3storage.transient.endpoint=https://s3.us-east-1.amazonaws.com
      nuxeo.s3storage.transient.pathstyleaccess=true
      

      Note that path-style access is incompatible with accelerate mode (NXP-27657), see S3 documentation.

      Show
      S3 direct upload now has new nuxeo.conf parameters to configure a custom S3 endpoint and activate path-style access: nuxeo.s3storage.transient.endpoint  (default empty) nuxeo.s3storage.transient.pathstyleaccess  (default false ) For example: nuxeo.s3storage.transient.endpoint=https://s3.us-east-1.amazonaws.com nuxeo.s3storage.transient.pathstyleaccess=true Note that path-style access is incompatible with accelerate mode ( NXP-27657 ), see S3 documentation.
    • Sprint:
      nxFG 11.1.12
    • Team:
      FG

      Description

      Using S3 Direct Upload with an S3-like storage will fail with the following exception

      ERROR [WebEngineExceptionMapper] com.amazonaws.SdkClientException: Unable to execute HTTP request: sts.amazonaws.com
      com.amazonaws.SdkClientException: Unable to execute HTTP request: sts.amazonaws.com
              at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1175) ~[aws-java-sdk-core-1.11.516.jar:?]
              at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1121) ~[aws-java-sdk-core-1.11.516.jar:?]
              at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770) ~[aws-java-sdk-core-1.11.516.jar:?]
              at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744) ~[aws-java-sdk-core-1.11.516.jar:?]
              at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) ~[aws-java-sdk-core-1.11.516.jar:?]
              at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) ~[aws-java-sdk-core-1.11.516.jar:?]
              at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) ~[aws-java-sdk-core-1.11.516.jar:?]
              at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) ~[aws-java-sdk-core-1.11.516.jar:?]
              at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) ~[aws-java-sdk-core-1.11.516.jar:?]
              at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1368) ~[aws-java-sdk-sts-1.11.516.jar:?]
              at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1335) ~[aws-java-sdk-sts-1.11.516.jar:?]
              at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1324) ~[aws-java-sdk-sts-1.11.516.jar:?]
              at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:491) ~[aws-java-sdk-sts-1.11.516.jar:?]
              at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:464) ~[aws-java-sdk-sts-1.11.516.jar:?]
              at org.nuxeo.ecm.core.storage.sql.S3DirectBatchHandler.assumeRole(S3DirectBatchHandler.java:215) ~[nuxeo-core-binarymanager-s3-10.10-HF15.jar:?]
              at org.nuxeo.ecm.core.storage.sql.S3DirectBatchHandler.getBatch(S3DirectBatchHandler.java:199) ~[nuxeo-core-binarymanager-s3-10.10-HF15.jar:?]
              at org.nuxeo.ecm.automation.server.jaxrs.batch.BatchManagerComponent.lambda$hasBatch$2(BatchManagerComponent.java:209) ~[nuxeo-automation-server-10.10-HF15.jar:?] 

      because the parameter nuxeo.s3storage.endpoint is not used to configure the S3 client.

      https://github.com/nuxeo/nuxeo/blob/master/addons/nuxeo-core-binarymanager-cloud/nuxeo-core-binarymanager-s3/src/main/java/org/nuxeo/ecm/core/storage/sql/S3DirectBatchHandler.java#L175

      Using the method withEndpointConfiguration could provide a solution here

                      .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(S3_ENDPOINT_PARAM))
      

      In addition, the pathStyleAccess must be configurable as well, similar to what is possible for a blob provider (NXP-25525).

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 3 hours
                  3h

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.