-
Type: Task
-
Status: Resolved
-
Priority: Minor
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: None
-
Component/s: Performance
-
Epic Link:
This seems to be complex to get sustainable performance when copying small files on the Snowball.
For example, when generating JPG files, the throughput started at 120 d/s for a few 100,000, but the performance progressively dropped and the throughput is now completly erratic between 35 and 50 docs/s.
The limitation is not the CPU:
- without output in memory I have 16,668 docs/s
- with local FS I have more than 120 docs/s
The idea is to leverage the snowball archive as described in this page:
- pack a few hundreds small files in a big archive file
- add a meta-data to tell S3 to uncompress the file at the storage level