Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-25716

Simplify fulltext extraction

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 10.3
    • Component/s: Core, Events / Works, Streams
    • Tags:
    • Impact type:
      Configuration Change
    • Upgrade notes:
      Hide

      Fulltext maximum size is now 128 KB by default. To change this, the repository configuration can be updated to use another fieldSizeLimit value:

      <fulltext ... fieldSizeLimit="1048576">
  ...
</fulltext>

      A value of 0 means no limit.

      See https://doc.nuxeo.com/nxdoc/repository-configuration/#full-text for more

      Show
      Fulltext maximum size is now 128 KB by default. To change this, the repository configuration can be updated to use another fieldSizeLimit value: <fulltext ... fieldSizeLimit= "1048576" > ... </fulltext> A value of 0 means no limit. See https://doc.nuxeo.com/nxdoc/repository-configuration/#full-text for more
    • Sprint:
      nxFG 10.3.5, nxFG 10.3.6
    • Story Points:
      3

      Description

      FulltextUpdaterWork was created by NXP-10864 in order to write newly-extracted fulltext into the database from a single thread, to avoid overloading some databases. This is separate from the FulltextExtractorWork which does the conversion from the binary into text, but currently does not store this text.

      Because the extracted text has to be passed to a second work, the text is stored in a queue for asynchronous processing. However this may be very large, easily several megabytes, which is a problem because queues like Kafka or Chronicle are not designed to work efficiently with such large volumes.

      The original problem leading to this design was a database we rarely use (SQL Server) and in addition we have other means of solving the concurrency problem these days (retry on ConcurrentUpdateException).

      —> merge back FulltextUpdaterWork into FulltextExtractorWork

        Attachments

          Issue Links

          depends on

          Improvement - An improvement or enhancement to an existing feature or task. NXP-5689 VCS: store PostgreSQL fulltext in clear text

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. NXP-10864 Write to fulltext table single-threaded

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. NXP-8797 Make fulltext normalization algorithm configurable

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Task - A task that needs to be done. NXP-14672 Factor common code for VCS and DBS

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. NXP-17862 Improve fulltext extraction

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. NXP-25279 Make the raw binary text available for processing

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. NXP-26618 FulltextUpdaterWork class not found after migration

          • Major - Major loss of function.
          • Resolved
          is required by

          Bug - A problem which impairs or prevents the functions of the product. NXP-25781 Allow per-field fulltext indexing for MongoDB

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. NXP-25359 FulltextExtractorWork should only get called once per blob

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          Is referenced in

          [Captain Hook] PR for 10.10: #4704 PR for 10.10: #4704

          [Captain Hook] PR for 2021: #99 PR for 2021: #99

          [Captain Hook] PR for master: #4702 PR for master: #4702

          [Captain Hook] PR for master: #4752 PR for master: #4752

          [Captain Hook] PR for master: #4769 PR for master: #4769

          [Captain Hook] PR for master: #4924 PR for master: #4924

          [Captain Hook] PR for master: #4935 PR for master: #4935

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 3 days, 2 hours
                  3d 2h