Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-30725

Cold Storage - CheckContentAvailability is failing

    XMLWordPrintable

    Details

      Description

      In some cases, the checkColdStorageContentAvailability fails and so the status is never updated for the documents, for the full instance.

      Below the stacktrace:

      2021-11-24T12:15:00,018 ERROR [Nuxeo-Work-default-105] [org.nuxeo.ecm.core.work.WorkManagerImpl] Uncaught error on thread: Nuxeo-Work-default-105, current work might be lost, WorkManager metrics might be corrupted.2021-11-24T12:15:00,018 ERROR [Nuxeo-Work-default-105] [org.nuxeo.ecm.core.work.WorkManagerImpl] Uncaught error on thread: Nuxeo-Work-default-105, current work might be lost, WorkManager metrics might be corrupted.org.nuxeo.ecm.core.api.NuxeoException: Work failed after 2 retries, class=class org.nuxeo.ecm.core.event.impl.AsyncEventExecutor$ListenerWork id=576406837388256.1593781507 category=checkColdStorageContentAvailability title=Listener checkColdStorageContentAvailability [checkColdStorageContentAvailability] at org.nuxeo.ecm.core.work.AbstractWork.workFailed(AbstractWork.java:439) ~[nuxeo-core-event-10.10-HF44.jar:?] at org.nuxeo.ecm.core.work.AbstractWork.run(AbstractWork.java:395) ~[nuxeo-core-event-10.10-HF44.jar:?] at org.nuxeo.ecm.core.work.WorkHolder.run(WorkHolder.java:57) ~[nuxeo-core-event-10.10-HF44.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?]Caused by: org.nuxeo.ecm.core.api.model.PropertyNotFoundException: coldstorage:coldContent at org.nuxeo.ecm.core.api.impl.DocumentModelImpl.getProperty(DocumentModelImpl.java:1265) ~[nuxeo-core-api-10.10-HF54.jar:?] at org.nuxeo.ecm.core.api.impl.DocumentModelImpl.getPropertyValue(DocumentModelImpl.java:1328) ~[nuxeo-core-api-10.10-HF54.jar:?] at org.nuxeo.coldstorage.service.ColdStorageServiceImpl.getBlobStatus(ColdStorageServiceImpl.java:127) ~[nuxeo-coldstorage-10.0.0-rc.33.jar:?] at org.nuxeo.coldstorage.service.ColdStorageServiceImpl.checkColdStorageContentAvailability(ColdStorageServiceImpl.java:407) ~[nuxeo-coldstorage-10.0.0-rc.33.jar:?] at org.nuxeo.coldstorage.events.CheckColdStorageContentAvailabilityListener.handleEvent(CheckColdStorageContentAvailabilityListener.java:51) ~[nuxeo-coldstorage-10.0.0-rc.33.jar:?] at org.nuxeo.ecm.core.event.impl.AsyncEventExecutor$ListenerWork.work(AsyncEventExecutor.java:221) ~[nuxeo-core-event-10.10-HF44.jar:?] at org.nuxeo.ecm.core.work.AbstractWork.runWorkWithTransaction(AbstractWork.java:493) ~[nuxeo-core-event-10.10-HF44.jar:?] at org.nuxeo.ecm.core.work.AbstractWork.run(AbstractWork.java:383) ~[nuxeo-core-event-10.10-HF44.jar:?] ... 4 more Suppressed: org.nuxeo.ecm.core.api.model.PropertyNotFoundException: coldstorage:coldContent at org.nuxeo.ecm.core.api.impl.DocumentModelImpl.getProperty(DocumentModelImpl.java:1265) ~[nuxeo-core-api-10.10-HF54.jar:?] at org.nuxeo.ecm.core.api.impl.DocumentModelImpl.getPropertyValue(DocumentModelImpl.java:1328) ~[nuxeo-core-api-10.10-HF54.jar:?] at org.nuxeo.coldstorage.service.ColdStorageServiceImpl.getBlobStatus(ColdStorageServiceImpl.java:127) ~[nuxeo-coldstorage-10.0.0-rc.33.jar:?] at org.nuxeo.coldstorage.service.ColdStorageServiceImpl.checkColdStorageContentAvailability(ColdStorageServiceImpl.java:407) ~[nuxeo-coldstorage-10.0.0-rc.33.jar:?] at org.nuxeo.coldstorage.events.CheckColdStorageContentAvailabilityListener.handleEvent(CheckColdStorageContentAvailabilityListener.java:51) ~[nuxeo-coldstorage-10.0.0-rc.33.jar:?] at org.nuxeo.ecm.core.event.impl.AsyncEventExecutor$ListenerWork.work(AsyncEventExecutor.java:221) ~[nuxeo-core-event-10.10-HF44.jar:?] at org.nuxeo.ecm.core.work.AbstractWork.runWorkWithTransaction(AbstractWork.java:493) ~[nuxeo-core-event-10.10-HF44.jar:?] at org.nuxeo.ecm.core.work.AbstractWork.run(AbstractWork.java:383) ~[nuxeo-core-event-10.10-HF44.jar:?] at org.nuxeo.ecm.core.work.WorkHolder.run(WorkHolder.java:57) ~[nuxeo-core-event-10.10-HF44.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Suppressed: org.nuxeo.ecm.core.api.model.PropertyNotFoundException: coldstorage:coldContent at org.nuxeo.ecm.core.api.impl.DocumentModelImpl.getProperty(DocumentModelImpl.java:1265) ~[nuxeo-core-api-10.10-HF54.jar:?] at org.nuxeo.ecm.core.api.impl.DocumentModelImpl.getPropertyValue(DocumentModelImpl.java:1328) ~[nuxeo-core-api-10.10-HF54.jar:?] at org.nuxeo.coldstorage.service.ColdStorageServiceImpl.getBlobStatus(ColdStorageServiceImpl.java:127) ~[nuxeo-coldstorage-10.0.0-rc.33.jar:?] at org.nuxeo.coldstorage.service.ColdStorageServiceImpl.checkColdStorageContentAvailability(ColdStorageServiceImpl.java:407) ~[nuxeo-coldstorage-10.0.0-rc.33.jar:?] at org.nuxeo.coldstorage.events.CheckColdStorageContentAvailabilityListener.handleEvent(CheckColdStorageContentAvailabilityListener.java:51) ~[nuxeo-coldstorage-10.0.0-rc.33.jar:?] at org.nuxeo.ecm.core.event.impl.AsyncEventExecutor$ListenerWork.work(AsyncEventExecutor.java:221) ~[nuxeo-core-event-10.10-HF44.jar:?] at org.nuxeo.ecm.core.work.AbstractWork.runWorkWithTransaction(AbstractWork.java:493) ~[nuxeo-core-event-10.10-HF44.jar:?] at org.nuxeo.ecm.core.work.AbstractWork.run(AbstractWork.java:383) ~[nuxeo-core-event-10.10-HF44.jar:?] at org.nuxeo.ecm.core.work.WorkHolder.run(WorkHolder.java:57) ~[nuxeo-core-event-10.10-HF44.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?]

      Steps to reproduce

      1. Create several documents
      2. Move them to cold storage
      3. Request a restore for one of them
      4. After 6 hours, go back to the restored document
      5. Request a retrieve on another document

      Observed results

      4. There is no more header on the document

      4. The restored document does not contain anymore the coldstorage:coldcontent property

      4. The issue occurs on the checkColdStorageContentAvailability everytime it is triggered

      5. the retrieve document is never updated (due to the failure, nothing is updated on the instance)

      Expected behavior

      4. There is no more header on the document

      4. The restored document does not contain anymore the coldstorage:coldcontent property

      4. The issue doesn't occur on the checkColdStorageContentAvailability everytime it is triggered

      5. the retrieve document is updated once it is available

      Note

      The fix should include to handle documents without coldstorage:coldcontent property but also that a failure for 1 document is not aborting the others check and updates (currently nothing is updated on the instance once it is failing for 1 document)

       

        Attachments

          Issue Links

          is related to

          New Feature - A new feature of the product, which has yet to be developed. NXP-28428 Add a scheduled job to check if blobs being retrieved form Glacier are finally available

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          Is referenced in

          [Captain Hook] PR for 10.10: #174 PR for 10.10: #174

          [Captain Hook] PR for lts-2021: #184 PR for lts-2021: #184

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: