Affects Version/s: 10.10-HF20, 11.1-SNAPSHOT
Fix Version/s: None
NXP-26899, it is possible to configure the default part size for multipart upload in S3. However, the default is too small and may end up in lots of API requests which kills performance (and may help hitting rate limits too).
For instance, on an EC2 instance, the throughput to S3 is roughly 30MB/s for a single PUT, which makes 6 chunks and request per second. We trigger multipart after 5GB, so after 5GB, instead of doing one long request, we need 1000 requests of 5MB to S3 which is probably super inefficient.
The AWS cli, seems to use chunks of 1GB which seems to be more reasonable.
Instead of toggling multipart at 5GB, a more reasonable default configuration would be to trigger it at 1.5GB and use chunks of 1GB.
We could even make it more dynamic by allowing to configure chunk size and toggle multipart upload only when sizeOfBlob > max(chunk*1.5, 5G)