Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-28555

S3 Multipart default size for part is too small

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: 10.10-HF20, 11.1-SNAPSHOT
    • Fix Version/s: None
    • Component/s: S3

      Description

      Since NXP-26899, it is possible to configure the default part size for multipart upload in S3. However, the default is too small and may end up in lots of API requests which kills performance (and may help hitting rate limits too).

      For instance, on an EC2 instance, the throughput to S3 is roughly 30MB/s for a single PUT, which makes 6 chunks and request per second. We trigger multipart after 5GB, so after 5GB, instead of doing one long request, we need 1000 requests of 5MB to S3 which is probably super inefficient.

      The AWS cli, seems to use chunks of 1GB which seems to be more reasonable.

      Instead of toggling multipart at 5GB, a more reasonable default configuration would be to trigger it at 1.5GB and use chunks of 1GB.

      We could even make it more dynamic by allowing to configure chunk size and toggle multipart upload only when sizeOfBlob > max(chunk*1.5, 5G)

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: