Uploaded image for project: 'Nuxeo Python Client'
  1. Nuxeo Python Client
  2. NXPY-180

Allow to upload to S3 when the bucket prefix is empty

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 3.0.1
    • Component/s: Upload
    • Release Notes Summary:
      S3 and Empty Bucker Prefix
    • Release Notes Description:
      Hide

      There was a bug when using the S3 upload provider and when the configured bucket prefix was empty. Such upload would never complete.

      Show
      There was a bug when using the S3 upload provider and when the configured bucket prefix was empty. Such upload would never complete.
    • Sprint:
      nxDrive 11.2.3
    • Story Points:
      1

      Description

      Error

      Trying to upload a file via S3 Direct Upload will fail if the bucket prefix is not set or empty. All chunks will be uploaded successfully. But when completing the upload, an error will raise (logs from Nuxeo Drive):

      2020-09-08 11:40:55 68792 123145564815360 DEBUG    nuxeo.client       Calling POST 'SERVER/api/v1/upload/batchId-8438eb33-8a69-4fc0-a749-92d89ab355ff/0/complete' with headers={...}, params='{"name": "DeepL.dmg", "fileSize": 17665683, "key": "/15f3fe27-fc80-4628-bb6e-a608889407ed", "bucket": "drive-test-s3encryption-2", "etag": "\\"57624f7a3acfb4b8d50a93f0e299e79d\\""}', kwargs={'timeout': 21600, 'verify': True, 'proxies': None} and cookies=<...>
      2020-09-08 11:40:55 68792 123145564815360 DEBUG    nuxeo.client       Response from 'SERVER/api/v1/upload/batchId-8438eb33-8a69-4fc0-a749-92d89ab355ff/0/complete' [400]: '{"entity-type":"exception","status":400,"message":"java.lang.IllegalArgumentException: Invalid key: /15f3fe27-fc80-4628-bb6e-a608889407ed"}' with headers {...} and cookies <...>
      2020-09-08 11:40:55 68792 123145564815360 ERROR    nxdrive.engine.processor Unknown error
      Traceback (most recent call last):
        File "nuxeo/client.py", line 282, in request
          resp.raise_for_status()
        File "requests/models.py", line 941, in raise_for_status
          raise HTTPError(http_error_msg, response=self)
      requests.exceptions.HTTPError: 400 Client Error:  for url: SERVER/api/v1/upload/batchId-8438eb33-8a69-4fc0-a749-92d89ab355ff/0/complete
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "nxdrive/engine/processor.py", line 283, in _execute
          sync_handler(doc_pair)
        File "nxdrive/engine/processor.py", line 511, in _synchronize_direct_transfer
          doc_pair=doc_pair,
        File "nxdrive/client/remote_client.py", line 397, in upload
          return uploader(self).upload(*args, **kwargs)
        File "nxdrive/client/uploader/direct_transfer.py", line 101, in upload
          remote_parent_ref=doc_pair.remote_parent_ref,
        File "nxdrive/client/uploader/__init__.py", line 83, in upload_impl
          file_path, filename=filename, mime_type=mime_type, **kwargs
        File "nxdrive/client/uploader/__init__.py", line 156, in upload_chunks
          return self._complete_upload(batch, blob)
        File "nxdrive/client/uploader/__init__.py", line 362, in _complete_upload
          batch.complete(timeout=TX_TIMEOUT)
        File "nuxeo/models.py", line 231, in complete
          return self.service.complete(self, **kwargs)
        File "nuxeo/uploads.py", line 310, in complete
          return self.client.request("POST", endpoint, data=params, **kwargs)
        File "nuxeo/client.py", line 285, in request
          raise self._handle_error(exc)
      nuxeo.exceptions.HTTPError: HTTPError(400), error: 'java.lang.IllegalArgumentException: Invalid key: /15f3fe27-fc80-4628-bb6e-a608889407ed', server trace: None
      

      The orignal code was not good: it was trying to handle the case when the bucket prefix was not ending with a slash ("/") and thus the problem here. The faultive code:

      key = "{}/{}".format(s3_info["baseKey"].rstrip("/"), batch.key or blob.name)
      

      It is also useless as the check is done server-side:

      if (!isBlank(bucketNamePrefix) && !bucketNamePrefix.endsWith(DELIMITER)) {
          log.debug(String.format("%s %s S3 bucket prefix should end with '/' : added automatically.",
                  BUCKET_PREFIX_PROPERTY, bucketNamePrefix));
          bucketNamePrefix += DELIMITER;
      }
      

      Temporary Fix

      There is a temporary fix on the server to make uploads to work:

      nuxeo.s3storage.bucket_prefix=/ 
      nuxeo.s3storage.transient.bucket_prefix=/
      

      Fix

      Just use a simple concatenation as it is done on the server.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 hours
                  2h

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.