Ensure file existence when using batch upload in cluster mode and Cloud storage

This is the best scenario I was able to identify:

1. create a batch batch-01 on node1
2. upload fileA via batch-01 to transient store from node2
   => fileA is uploaded to BatchManagerCache store in GCS and stored on node2 in the local nxbin cache
3. use batch-01 to create a File with fileA on node1
   => fileA is read from BatchManagerCache store in GCS, then stored on node1 in the local nxbin cache and in the final storage on GCS
4. create a new batch batch-02 on node2
5. TransientStore GC is run on node2 (this is random : the scheduler triggers an event which is caught by one of the nodes but here we force the GC on node2)
   => fileA is removed from BatchManagerCache store on GCS and in the local nxbin cache on node2
6. upload fileA via batch-02 to transient store from node1
   => fileA is read in local nxbin cache of node2 (but missing in BatchManagerCache store on GCS
7. use batch-02 to create a File with fileA on node2
   => the file is not in the local cache and not found in BatchManagerCache store on GCS (because of the CG)
   => java.lang.NullPointerException: null at org.nuxeo.ecm.core.blob.binary.CachingBinaryManager.getBinary(CachingBinaryManager.java:141) ~[nuxeo-core-api-2021.33.9.jar:?]

Traces:

• node 1

  2023-02-24T19:59:25,899 WARN  [http-nio-0.0.0.0-8080-exec-9] [org.nuxeo.ecm.core.blob.binary.CachingBinaryManager] Getting file bf81f48be077ac51b1e2bc024d9397a2
  2023-02-24T19:59:26,017 WARN  [http-nio-0.0.0.0-8080-exec-9] [org.nuxeo.ecm.core.blob.binary.CachingBinaryManager] => File bf81f48be077ac51b1e2bc024d9397a2 retrieved from fileStorage org.nuxeo.ecm.core.storage.gcp.GoogleStorageBinaryManager$GCPFileStorage : /opt/nuxeo-server-tomcat-2021.1-GCS/tmp/nxbincache.10716387506675255588/bf81f48be077ac51b1e2bc024d9397a2
  2023-02-24T19:59:26,018 DEBUG [http-nio-0.0.0.0-8080-exec-9] [org.nuxeo.ecm.core.storage.gcp.GoogleStorageBinaryManager] Storing blob with digest: bf81f48be077ac51b1e2bc024d9397a2 to GCS
  2023-02-24T19:59:26,241 DEBUG [http-nio-0.0.0.0-8080-exec-9] [org.nuxeo.ecm.core.storage.gcp.GoogleStorageBinaryManager] Stored blob with digest: bf81f48be077ac51b1e2bc024d9397a2 to GCS in 222ms
  2023-02-24T19:59:26,332 WARN  [Nuxeo-Work-updateThumbListener-2:29459507860953.1879359689] [org.nuxeo.ecm.core.blob.binary.CachingBinaryManager] Getting file bf81f48be077ac51b1e2bc024d9397a2
  2023-02-24T19:59:26,332 WARN  [Nuxeo-Work-updateThumbListener-2:29459507860953.1879359689] [org.nuxeo.ecm.core.blob.binary.CachingBinaryManager] => File bf81f48be077ac51b1e2bc024d9397a2 retrieved from cache : /opt/nuxeo-server-tomcat-2021.1-GCS/tmp/nxbincache.8733906533136329591/bf81f48be077ac51b1e2bc024d9397a2
  
  2023-02-24T20:00:08,869 WARN  [http-nio-0.0.0.0-8080-exec-1] [org.nuxeo.ecm.core.blob.binary.CachingBinaryManager] Getting file bf81f48be077ac51b1e2bc024d9397a2
  2023-02-24T20:00:08,869 WARN  [http-nio-0.0.0.0-8080-exec-1] [org.nuxeo.ecm.core.blob.binary.CachingBinaryManager] => File bf81f48be077ac51b1e2bc024d9397a2 retrieved from cache : /opt/nuxeo-server-tomcat-2021.1-GCS/tmp/nxbincache.8733906533136329591/bf81f48be077ac51b1e2bc024d9397a2
  

• node 2

  2023-02-24T19:56:17,290 DEBUG [http-nio-0.0.0.0-9080-exec-1] [org.nuxeo.ecm.core.storage.gcp.GoogleStorageBinaryManager] Storing blob with digest: bf81f48be077ac51b1e2bc024d9397a2 to GCS
  2023-02-24T19:56:17,572 DEBUG [http-nio-0.0.0.0-9080-exec-1] [org.nuxeo.ecm.core.storage.gcp.GoogleStorageBinaryManager] Stored blob with digest: bf81f48be077ac51b1e2bc024d9397a2 to GCS in 281ms
  
  
  2023-02-24T20:00:00,297 DEBUG [Nuxeo-Work-transientStorageGC-1:BatchManagerCache] [org.nuxeo.ecm.core.transientstore.keyvalueblob.KeyValueBlobTransientStore] GC completed for BatchManagerCache: BinaryManagerStatus [gcDuration=88, numBinaries=1, sizeBinaries=171868, numBinariesGC=2, sizeBinariesGC=107063]
  
  2023-02-24T20:03:07,126 WARN  [http-nio-0.0.0.0-9080-exec-3] [org.nuxeo.ecm.core.blob.binary.CachingBinaryManager] Getting file bf81f48be077ac51b1e2bc024d9397a2
  2023-02-24T20:03:07,151 WARN  [http-nio-0.0.0.0-9080-exec-3] [org.nuxeo.ecm.core.blob.binary.CachingBinaryManager] => File bf81f48be077ac51b1e2bc024d9397a2 not in storage
  2023-02-24T20:03:07,152 ERROR [WebEngineExceptionMapper] java.lang.NullPointerException
  java.lang.NullPointerException: null