Replicate BlobStore between 2 Data Centers

For people deploying in multi-datacenters, we may be able to assert that the infrastructure will provide a way to sync blobs.

However, if this is not the case, we can consider several options:

RSync

Principles

We run rsync between 2 nodes on the 2 DC each of then being connected to the underlying filesystem that is used by the BlobManager.

Advantages

Almost nothing to do.

Limitations

Hard to know "where we are in the replication process" and hence hard to know what will be the amount of list data.

Not sure how it will scale with a very large storage with several millions of files.

BlobManager with async remote write

Principles

We write a BlobManager that will also write, asynchronously, the blobs to the DRP location.

This is a "double write" approach.

Advantages

No external process to run.

Limitations

Handling the async write may be complex: it will be easy with a S3 client, but on the other hand, if we use S3 on AWS, we could simply use built-in features or an event triggered Lambda.

BlobManager + nuxeo-stream + copy process

Principles

We can do use the same principle as the one used for ES:

• have a custom Binary Manager that
  • in addition to writing the Blob also add an entry in nuxeo-stream (just the digest of the Blob to be added)
  • expose an http API to retrieve the Blob based on digest
• have a Kafka Connect running on the target DC
• get the digest from Kafka
• fetch the remote Blob and write it

Advantages

• we can track progress
• the process is automatic and reliable

Limitations

More custom code to deploy and maintain.