Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-30366

S3 blob storage: adding depth-based prefixes to object names

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 10.10, 2021.x, 11.x
    • Fix Version/s: 10.10-HF52, 2021.8
    • Component/s: S3
    • Release Notes Summary:
      Blobs can be stored in S3 with depth-based prefixes.
    • Release Notes Description:
      Hide

      It is possible, using the subDirsDepth configuration property for the blob store, to store objects in S3 using nested subdirectories to avoid having all objects in a flat hierarchy. For instance with a depth of 2 the object d41d8cd98f00b204e9800998ecf8427e will be stored as d4/1d/d41d8cd98f00b204e9800998ecf8427e.

      Example XML configuration:

          <blobprovider name="...">
            <class>org.nuxeo.ecm.blob.s3.S3BlobProvider</class>
            ...
            <property name="subDirsDepth">2</property>
          </blobprovider>
      

      If XML configuration is not used, and if all S3 blob providers can have the same value for this configuration property, a nuxeo.conf property may be used:

      nuxeo.s3storage.subDirsDepth=2
      

      Using a depth of 0 gives the standard behavior of storing everything flat.

      Show
      It is possible, using the subDirsDepth configuration property for the blob store, to store objects in S3 using nested subdirectories to avoid having all objects in a flat hierarchy. For instance with a depth of 2 the object d41d8cd98f00b204e9800998ecf8427e will be stored as d4/1d/d41d8cd98f00b204e9800998ecf8427e . Example XML configuration: <blobprovider name= "..." > <class> org.nuxeo.ecm.blob.s3.S3BlobProvider </class> ... <property name= "subDirsDepth" > 2 </property> </blobprovider> If XML configuration is not used, and if all S3 blob providers can have the same value for this configuration property, a nuxeo.conf property may be used: nuxeo.s3storage.subDirsDepth=2 Using a depth of 0 gives the standard behavior of storing everything flat.
    • Tags:
    • Backlog priority:
      750

      Description

      There would be some benefit in using a hierarchical namespace for S3, similar to the default binary manager where we can configure a depth to the binary store.

      This is useful for third-party S3-compatible backends like MinIO, but even for S3 itself, which has rate-limiting per prefix (see https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html for more).

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 days
                  2d

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.