Clean up orphan binaries after document removal, blob property edition and dispatch

Today, when the default blob dispatcher moves a binary from one blob provider to another (following the blob dispatcher rules definition), it performs a copy and does not delete the source binary if it is orphaned i.e. if it is not referenced by a blob property in the backend anymore.

In the case of the retention feature, when a document becomes a record, its main blob is moved to a dedicated blob provider (hence blob store), see the recommended configuration https://doc.nuxeo.com/nxdoc/nuxeo-retention-installation-standard/#configure-via-xml-contribution:

<extension target="org.nuxeo.ecm.core.blob.DocumentBlobManager" point="configuration">
    <blobdispatcher>
      <class>org.nuxeo.ecm.core.blob.DefaultBlobDispatcher</class>
      <property name="records">records</property>
      <property name="default">default</property>
    </blobdispatcher>
</extension>

since the blob dispatcher performs a copy from the default provider to the record one and does not delete the source blob in the default bucket if unreferenced (orphaned), the main blob is duplicated in both associated blob stores resulting in a storage cost rise. Unless a full orphaned binaries GC is performed which is costly.

This is because the default blob store has a default digest key strategy and the stored blobs are potentially referenced by others documents. It will be too heavy to scan the repository to check if it can be deleted synchronously.

To improve the current state, we'd like to be able to add to a stream as records the keys of the blobs that are candidates for deletion.

A computation consuming this stream will be in charge of querying the database to check that the blob key is not referenced by another document's blob field before proceeding to its removal.

The implementation could leverage the domain event feature.

Note that such an improvement could be leveraged when:

• a document is removed: all the blob keys held in the document blob fields could be added to this stream in order to clean up its binaries.
• a blob field value of a document is edited, the old blob key if any could be added to this stream in order to check if the associated blob could be deleted