Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-29618

Fix handling of object immutability in 'google-storage' addon during mass import

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 10.10
    • Fix Version/s: 10.10-HF33, 11.3, 2021.0
    • Component/s: BlobManager

      Description

      When performing a mass import using the binary manager from the google-storage addon, it can get the following stacktrace:

      2020-08-31T20:45:14,321 ERROR [http-nio-127.0.0.1-8080-exec-91] [org.nuxeo.ecm.webengine.app.WebEngineExceptionMapper] com.google.cloud.storage.StorageException: The rate of change requests to the object gfusw-qecm400-live/transient_BatchManagerCache/bdea87eb0a53c7d416645ae9a54e615b exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
      com.google.cloud.storage.StorageException: The rate of change requests to the object gfusw-qecm400-live/transient_BatchManagerCache/bdea87eb0a53c7d416645ae9a54e615b exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
              at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:227) ~[google-cloud-storage-1.76.0.jar:1.76.0]
              at com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:308) ~[google-cloud-storage-1.76.0.jar:1.76.0]
              at com.google.cloud.storage.StorageImpl.create(StorageImpl.java:178) ~[google-cloud-storage-1.76.0.jar:1.76.0]
              at com.google.cloud.storage.Bucket.create(Bucket.java:975) ~[google-cloud-storage-1.76.0.jar:1.76.0]
              at org.nuxeo.ecm.core.storage.gcp.GoogleStorageBinaryManager$GCPFileStorage.storeFile(GoogleStorageBinaryManager.java:185) ~[nuxeo-core-binarymanager-gcp-10.10-HF27.jar:?]
              at org.nuxeo.ecm.core.blob.binary.CachingBinaryManager.getBinary(CachingBinaryManager.java:155) ~[nuxeo-core-api-10.10-HF29.jar:?]
              at org.nuxeo.ecm.core.blob.binary.AbstractBinaryManager.getBinary(AbstractBinaryManager.java:130) ~[nuxeo-core-api-10.10-HF29.jar:?]
              at org.nuxeo.ecm.core.blob.binary.BinaryBlobProvider.writeBlob(BinaryBlobProvider.java:133) ~[nuxeo-core-api-10.10-HF29.jar:?]
              at org.nuxeo.ecm.blob.AbstractCloudBinaryManager.writeBlob(AbstractCloudBinaryManager.java:147) ~[nuxeo-core-binarymanager-common-10.10-HF21.jar:?]
              at org.nuxeo.ecm.core.transientstore.keyvalueblob.KeyValueBlobTransientStore.putBlobs(KeyValueBlobTransientStore.java:491) ~[nuxeo-core-cache-10.10-HF21.jar:?]
              at org.nuxeo.ecm.automation.server.jaxrs.batch.Batch.addFile(Batch.java:185) ~[nuxeo-automation-server-10.10-HF29.jar:?]
              at org.nuxeo.ecm.restapi.server.jaxrs.BatchUploadObject.addBlob(BatchUploadObject.java:332) ~[nuxeo-rest-api-server-10.10-HF30.jar:?]
              at org.nuxeo.ecm.restapi.server.jaxrs.BatchUploadObject.uploadNoTransaction(BatchUploadObject.java:279) ~[nuxeo-rest-api-server-10.10-HF30.jar:?]
              at org.nuxeo.ecm.restapi.server.jaxrs.BatchUploadObject.upload(BatchUploadObject.java:187) ~[nuxeo-rest-api-server-10.10-HF30.jar:?]
              ...
      Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 429 Too Many Requests
      {
        "code" : 429,
        "errors" : [ {
          "domain" : "usageLimits",
          "message" : "The rate of change requests to the object gfusw-qecm400-live/transient_BatchManagerCache/bdea87eb0a53c7d416645ae9a54e615b exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
          "reason" : "rateLimitExceeded"
        } ],
        "message" : "The rate of change requests to the object gfusw-qecm400-live/transient_BatchManagerCache/bdea87eb0a53c7d416645ae9a54e615b exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
      }
              at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150) ~[google-api-client-1.25.0.jar:1.25.0]
              at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113) ~[google-api-client-1.25.0.jar:1.25.0]
              at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40) ~[google-api-client-1.25.0.jar:1.25.0]
              at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432) ~[google-api-client-1.25.0.jar:1.25.0]
              at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352) ~[google-api-client-1.25.0.jar:1.25.0]
              at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469) ~[google-api-client-1.25.0.jar:1.25.0]
              at com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:305) ~[google-cloud-storage-1.76.0.jar:1.76.0]
              ... 135 more
      

      The code of GoogleStorageBinaryManager.GCPFileStorage.storeFile is naïve in that, contrary to what we do for S3 or Azure, it doesn't check that if the hash is already stored in GCP before trying to upload it.

      If one tries to upload the same file in a short time, it'll cause this issue because https://cloud.google.com/storage/docs/key-terms#immutability clearly says

      However, a single particular object can only be updated or overwritten up to once per second. For example, if you have an object bar in bucket foo, then you should only upload a new copy of foo/bar about once per second. Updating the same object more frequently than once per second may result in 429 Too Many Requests errors.

        Attachments

          Activity

            People

            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 2 days
                2d

                  PagerDuty

                  Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.