Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-28992

Leverage S3 SnowBall Archive to speedup transfer

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Performance

      Description

      This seems to be complex to get sustainable performance when copying small files on the Snowball.
      For example, when generating JPG files, the throughput started at 120 d/s for a few 100,000, but the performance progressively dropped and the throughput is now completly erratic between 35 and 50 docs/s.

      The limitation is not the CPU:

      • without output in memory I have 16,668 docs/s
      • with local FS I have more than 120 docs/s

      The idea is to leverage the snowball archive as described in this page:

      • pack a few hundreds small files in a big archive file
      • add a meta-data to tell S3 to uncompress the file at the storage level

        Attachments

          Activity

            People

            • Assignee:
              tdelprat Thierry Delprat
              Reporter:
              tdelprat Thierry Delprat
              Participants:
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: