-
Type:
Bug
-
Status: Resolved
-
Priority:
Minor
-
Resolution: Fixed
-
Affects Version/s: 10.10
-
Fix Version/s: 10.10-HF36, 11.2, 2021.0
-
Component/s: Streams
-
Release Notes Summary:The Bulk scroller is executed with a long transaction to avoid timeout.
-
Tags:
-
Backlog priority:100
-
Team:PLATFORM
-
Sprint:nxplatform 11.1.36, nxplatform #21
-
Story Points:3
Today a Bulk Command with an NXQL query that is slow to process either because of the query or the number of results can timeout for different causes: transaction tx, database socket timeout, or even scroll timeout.
In this case, the default policy is to stop on failure, but there will be a retry for each scroller thread (2 per nodes).
The result can be annoying when nuxeo.core.bulk.scroller.produceImmediate is set to true (like in the perf template) because the materialized records are sent downstream during the scroll and for each retry, this result in duplicated and incomplete processing that could lead to a disk full on Kafka in case of huge repository.
It will be better to increase timeouts (or catch the various timeout exceptions and complete the command in failure).
Also, the perf template should be fixed to remove the maxRetry=3 which is a bad idea for the scroller.
--
We should try to use a transaction timeout with a very long value configurable with the config service with a default of 2 days.
