Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-27567

Fix first-time s3 directdownload of Stale Lazy Stored Rendition

    XMLWordPrintable

    Details

    • Release Notes Summary:
      A lazy stored rendition is correctly downloaded from S3 DirectDownload since the first time.
    • Backlog priority:
      700
    • Sprint:
      nxplatform 11.1.12, nxplatform 11.1.13, nxplatform 11.1.14, nxplatform 11.1.15
    • Story Points:
      8

      Description

      Reproduction steps on Nuxeo 9.10-HF32 w/JSF UI, Amazon S3 Online Storage, NXP-27316 Patch, KVBlobTransientStore w/SQL KV Storage and s3 BlobProviders installed, and nuxeo.conf property nuxeo.s3storage.directdownload=true:

      1. Add test-lazy-rendition-contribution to the "custom" template without rendition definitions "iamlazy", "lazyAutomation" and "lazyDelayedErrorAutomationRendition".

          cd $NUXEO_HOME/templates/custom
          mkdir config
          wget https://raw.githubusercontent.com/nuxeo/nuxeo/9.10/nuxeo-features/nuxeo-platform-rendition/nuxeo-platform-rendition-core/src/test/resources/test-lazy-rendition-contrib.xml
          mv test-lazy-rendition-contrib.xml test-lazy-rendition-config.xml
          # Manually remove rendition definitions "iamlazy", "lazyAutomation" and "lazyDelayedErrorAutomationRendition" from file "test-lazy-rendition-config.xml"
      

      2. Add "custom" to templates in nuxeo.conf

      3. Create workspace1

          curl -F "cmisaction=createFolder" -F "propertyId[0]=cmis:objectTypeId" -F "propertyValue[0]=Workspace" -F "propertyId[1]=cmis:name" -F "propertyValue[1]=workspace1" -u Administrator:Administrator "http://localhost:8080/nuxeo/json/cmis/default/root/default-domain/workspaces"
      

      4. Create folder1 in workspace1

          issued="1340000000000"; curl -F "cmisaction=createFolder" -F "propertyId[0]=cmis:objectTypeId" -F "propertyValue[0]=Folder" -F "propertyId[1]=cmis:name" -F "propertyValue[1]=folder1" -F "propertyId[2]=dc:issued" -F "propertyValue[2]=${issued}" -u Administrator:Administrator "http://localhost:8080/nuxeo/json/cmis/default/root/default-domain/workspaces/workspace1"
      

      5. Upload ten files into folder1, each of which has a Main File size of approximately 50MB. The objective is to make the folder size sufficiently large to cause the zipTreeExportLazily rendition to take several seconds to generate (will vary, of course, based on test server configuration)

          for i in {1..10}; do dd if=/dev/urandom of=file$i count=102400 && curl -F "cmisaction=createDocument" -F "propertyId[0]=cmis:objectTypeId" -F "propertyValue[0]=File" -F "propertyId[1]=cmis:name" -F "propertyValue[1]=file${i}" -F "content=@file${i}" -u Administrator:Administrator "http://localhost:8080/nuxeo/json/cmis/default/root/default-domain/workspaces/workspace1/folder1"; done
      

      6. Generate a zipTreeExportLazily stored rendition by executing the following script

      #!/bin/bash
      rm -f inprogress export.zip
      while : ; do
        result=$(curl -L -s -D- -OJ -u Administrator:Administrator "http://localhost:8080/nuxeo/json/cmis/default/root/default-domain/workspaces/workspace1/folder1?cmisselector=content&streamId=nuxeo:rendition:zipTreeExportLazily")
        if [[ "$result" == *"Content-Type: application/zip"* ]]; then
          break
        fi
      done
      

      7. Verify the downloaded "export.zip" file matches the folder contents

      8. Add a new file to folder1 by incrementing i each time starting with 11

          i=11; dd if=/dev/urandom of=file$i count=102400 && curl -F "cmisaction=createDocument" -F "propertyId[0]=cmis:objectTypeId" -F "propertyValue[0]=File" -F "propertyId[1]=cmis:name" -F "propertyValue[1]=file${i}" -F "content=@file${i}" -u Administrator:Administrator "http://localhost:8080/nuxeo/json/cmis/default/root/default-domain/workspaces/workspace1/folder1"
      

      9. Increment folder1 property dc:issued value (propertyValue[0]=<previous value + 1> below starting with 1340000000001) so its rendition will be re-generated on next request

          issued="1340000000001"; curl -F "cmisaction=update" -F "propertyId[0]=dc:issued" -F "propertyValue[0]=${issued}" -u Administrator:Administrator "http://localhost:8080/nuxeo/json/cmis/default/root/default-domain/workspaces/workspace1/folder1"
      

      10. Initiate generation of the latest available rendition

          curl -L -s -D- -OJ -u Administrator:Administrator "http://localhost:8080/nuxeo/json/cmis/default/root/default-domain/workspaces/workspace1/folder1?cmisselector=content&streamId=nuxeo:rendition:zipTreeExportLazily"
      

      Expected results: an HTTP 302 response followed by an HTTP 200 response w/Content-Disposition: inline; filename=inprogress and Content-Type suffix stale=true

      Actual results: an HTTP 302 response followed by an HTTP 404 response w/Content-Type: application/xml and the following content:

      <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><BucketName>nuxeoBucket</BucketName><RequestId>tx000000000000b4-005cf6dd25-a8cb9-object</RequestId><HostId>a8cb9-object-object</HostId></Error>
      

      This is a blocker to our future OpenShift deployment. This test was executed using the rook-ceph 1.0.1 Object Storage Gateway S3 interface running on OpenShift 3.11.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 4 days
                  4d

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.