Critical Error
Trying to upload a file via S3 Direct Upload will fail if the bucket prefix is not set or empty. All chunks will be uploaded successfully. But when completing the upload, an error will raise (logs from Nuxeo Drive):
2020-09-08 11:40:55 68792 123145564815360 DEBUG nuxeo.client Calling POST 'SERVER/api/v1/upload/batchId-8438eb33-8a69-4fc0-a749-92d89ab355ff/0/complete' with headers={...}, params='{"name": "DeepL.dmg", "fileSize": 17665683, "key": "/15f3fe27-fc80-4628-bb6e-a608889407ed", "bucket": "drive-test-s3encryption-2", "etag": "\\"57624f7a3acfb4b8d50a93f0e299e79d\\""}', kwargs={'timeout': 21600, 'verify': True, 'proxies': None} and cookies=<...>
2020-09-08 11:40:55 68792 123145564815360 DEBUG nuxeo.client Response from 'SERVER/api/v1/upload/batchId-8438eb33-8a69-4fc0-a749-92d89ab355ff/0/complete' [400]: '{"entity-type":"exception","status":400,"message":"java.lang.IllegalArgumentException: Invalid key: /15f3fe27-fc80-4628-bb6e-a608889407ed"}' with headers {...} and cookies <...>
2020-09-08 11:40:55 68792 123145564815360 ERROR nxdrive.engine.processor Unknown error
Traceback (most recent call last):
File "nuxeo/client.py", line 282, in request
resp.raise_for_status()
File "requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: for url: SERVER/api/v1/upload/batchId-8438eb33-8a69-4fc0-a749-92d89ab355ff/0/complete
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "nxdrive/engine/processor.py", line 283, in _execute
sync_handler(doc_pair)
File "nxdrive/engine/processor.py", line 511, in _synchronize_direct_transfer
doc_pair=doc_pair,
File "nxdrive/client/remote_client.py", line 397, in upload
return uploader(self).upload(*args, **kwargs)
File "nxdrive/client/uploader/direct_transfer.py", line 101, in upload
remote_parent_ref=doc_pair.remote_parent_ref,
File "nxdrive/client/uploader/__init__.py", line 83, in upload_impl
file_path, filename=filename, mime_type=mime_type, **kwargs
File "nxdrive/client/uploader/__init__.py", line 156, in upload_chunks
return self._complete_upload(batch, blob)
File "nxdrive/client/uploader/__init__.py", line 362, in _complete_upload
batch.complete(timeout=TX_TIMEOUT)
File "nuxeo/models.py", line 231, in complete
return self.service.complete(self, **kwargs)
File "nuxeo/uploads.py", line 310, in complete
return self.client.request("POST", endpoint, data=params, **kwargs)
File "nuxeo/client.py", line 285, in request
raise self._handle_error(exc)
nuxeo.exceptions.HTTPError: HTTPError(400), error: 'java.lang.IllegalArgumentException: Invalid key: /15f3fe27-fc80-4628-bb6e-a608889407ed', server trace: None
The orignal code was not good: it was trying to handle the case when the bucket prefix was not ending with a slash ("/") and thus the problem here. The faultive code:
key = "{}/{}".format(s3_info["baseKey"].rstrip("/"), batch.key or blob.name)
It is also useless as the check is done server-side:
if (!isBlank(bucketNamePrefix) && !bucketNamePrefix.endsWith(DELIMITER)) { log.debug(String.format("%s %s S3 bucket prefix should end with '/' : added automatically.", BUCKET_PREFIX_PROPERTY, bucketNamePrefix)); bucketNamePrefix += DELIMITER; }
Temporary Fix
There is a temporary fix on the server to make uploads to work:
nuxeo.s3storage.bucket_prefix=/
nuxeo.s3storage.transient.bucket_prefix=/
Fix
Just use a simple concatenation as it is done on the server.
- Is referenced in
