Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-30366

S3 blob storage: adding depth-based prefixes to object names

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 10.10, 2021.x, 11.x
    • Fix Version/s: 10.10-HF52, 2021.8
    • Component/s: S3
    • Release Notes Summary:
      Blobs can be stored in S3 with depth-based prefixes.
    • Release Notes Description:
      Hide

      It is possible, using the subDirsDepth configuration property for the blob store, to store objects in S3 using nested subdirectories to avoid having all objects in a flat hierarchy. For instance with a depth of 2 the object d41d8cd98f00b204e9800998ecf8427e will be stored as d4/1d/d41d8cd98f00b204e9800998ecf8427e.

      Example XML configuration:

          <blobprovider name="...">
      <class>org.nuxeo.ecm.blob.s3.S3BlobProvider</class>
      ...
      <property name="subDirsDepth">2</property>
    </blobprovider>

      If XML configuration is not used, and if all S3 blob providers can have the same value for this configuration property, a nuxeo.conf property may be used:

      nuxeo.s3storage.subDirsDepth=2

      Using a depth of 0 gives the standard behavior of storing everything flat.

      Show
      It is possible, using the subDirsDepth configuration property for the blob store, to store objects in S3 using nested subdirectories to avoid having all objects in a flat hierarchy. For instance with a depth of 2 the object d41d8cd98f00b204e9800998ecf8427e will be stored as d4/1d/d41d8cd98f00b204e9800998ecf8427e . Example XML configuration: <blobprovider name= "..." > <class> org.nuxeo.ecm.blob.s3.S3BlobProvider </class> ... <property name= "subDirsDepth" > 2 </property> </blobprovider> If XML configuration is not used, and if all S3 blob providers can have the same value for this configuration property, a nuxeo.conf property may be used: nuxeo.s3storage.subDirsDepth=2 Using a depth of 0 gives the standard behavior of storing everything flat.
    • Tags:
    • Backlog priority:
      750

      Description

      There would be some benefit in using a hierarchical namespace for S3, similar to the default binary manager where we can configure a depth to the binary store.

      This is useful for third-party S3-compatible backends like MinIO, but even for S3 itself, which has rate-limiting per prefix (see https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html for more).

        Attachments

          Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. NXP-30632 Fix blobProviderId still present when moving blob from a S3 blob provider to another S3 blob provider

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. NXP-30682 Create a migration procedure to fix S3 blobs having original blob provider prefix in their key

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          Is referenced in

          [Captain Hook] PR for 10.10: #4923 PR for 10.10: #4923

          [Captain Hook] PR for 2021: #227 PR for 2021: #227

          [Captain Hook] PR for master: #4935 PR for master: #4935

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 days
                  2d