Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-26899

S3 multipart part size should be configurable

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 10.10
    • Fix Version/s: 10.10-HF02, 11.1, 2021.0
    • Component/s: S3
    • Release Notes Summary:
      S3 multipart part size is configurable.
    • Tags:
    • Impact type:
      Configuration Change
    • Upgrade notes:
      Hide

      There is a new configuration property nuxeo.s3.multipart.copy.part.size to change the S3 multipart copy part size. The default is 5242880 (5MB). It can be changed with:

        <extension target="org.nuxeo.runtime.ConfigurationService" point="configuration">
    <property name="nuxeo.s3.multipart.copy.part.size">5242880</property>
  </extension>

      Note that the maximum allowed by S3 is 5368709120 (5GB).

      Show
      There is a new configuration property nuxeo.s3.multipart.copy.part.size to change the S3 multipart copy part size. The default is 5242880 (5MB). It can be changed with: <extension target= "org.nuxeo.runtime.ConfigurationService" point= "configuration" > <property name= "nuxeo.s3.multipart.copy.part.size" > 5242880 </property> </extension> Note that the maximum allowed by S3 is 5368709120 (5GB).
    • Sprint:
      nxFG 11.1.4

      Description

      S3Utils class defines a part size that is hard coded to 5MB: 

      https://github.com/nuxeo/nuxeo-core-binarymanager-cloud/blob/master/nuxeo-core-binarymanager-s3/src/main/java/org/nuxeo/ecm/core/storage/sql/S3Utils.java#L56

      This value should be configurable via nuxeo.conf to let customers increase the chunk size.

      Need a backport on 10.10 at least (9.10 would be nice too)

       

        Attachments

          Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. NXP-30595 make S3 transfer parameters configurable

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. NXP-28555 S3 Multipart default size for part is too small

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 hour
                  1h