Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-29807

S3 Direct Upload using ETag as high-level blob digest

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: File Upload , S3
    • Story Points:
      2

      Description

      Computing the high-level blob digest (Blob.getDigest()) by attempting to do extra S3 copies without multipart in order to retrieve an MD5 is pointless, because:

      1. There are cases when we can't get an MD5 at all (file > 5GB, managed mode) so the rest of the code has to be prepared to see a non-digest used for the high-level digest.
      2. It requires extra S3 copies and therefore has a financial cost.
      3. It is slow (because doing non-multipart copies of files in the GB range is slow, even when done as a direct S3-to-S3 same-bucket operation).

      Therefore we should just use the S3 ETag as high-level blob digest when computing it in the S3 Direct Upload code.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                fguillaume Florent Guillaume
                Reporter:
                fguillaume Florent Guillaume
                Participants:
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: