Minor Nuxeo Stream when using Chronicle Queue implementation (default when Kafka is not available) creates multiple cq4 files:
- there is a cq4 file per stream partition
- there is a cq4 file per consumer group to store consumer position
- each cq4 file has a retention so we keep 4 days of data by default (4 cq4 files after 4 days)
A cq4 file is a sparse file, by default the size is 80MB
because it is a sparse file the real used disk size is only the used data size (can be seen using ls -ls or du -sh on the cq4), at least under linux (windows don't handle sparse file).
Today by default only the AuditWriter uses a stream of one partition this creates 2 cq4 up to 8 cq4 files after 4 days.
(viewed as 160MB to 640MB)
Because BAF is going to use stream extensively we will have lots of cq4 by default, so we need to make sure this cq4 are well sized.
The size can be reduced by setting the block size explicitly with net.openhft.chronicle.queue.impl.single.SingleChronicleQueueBuilder#blockSize it can be reduced up to 1MB. We need to find the right default or make it configurable.
Also Nuxeo Stream UT creates lots of cq4 files that might impact the CI slave performance (to be confirmed depending on how overlay2 fs handles sparse file).
