When using Elasticsearch TransportClient protocol to communicate with the Elasticsearch cluster.
If a command is sent to a node that is not reachable, the TransportClient the node is dropped and there is a retry the operation on another node.
There is also a heartbeat signal every 5 secs to check if nodes are available. Nodes that do not respond are dropped.
To be able to reconnect to a node that was temporary unavailable, the sniffing option client.transport.sniff must be activated. Which is not the default.
To activate the sniffing In the Nuxeo elasticsearch contribution:
Also with the sniff mode activated, new nodes can be added to the Elasticsearch cluster and used without changing the Nuxeo configuration.
Here is an excerpt about the Elasticsearch documentation:
The Transport client comes with a cluster sniffing feature which
allows it to dynamically add new hosts and remove old ones.
When sniffing is enabled, the transport client will connect to the nodes in its
internal node list, which is built via calls to `addTransportAddress`.
After this, the client will call the internal cluster state API on those nodes
to discover available data nodes. The internal node list of the client will
be replaced with those data nodes only. This list is refreshed every five seconds by default.
Note that the IP addresses the sniffer connects to are the ones declared as the 'publish'
address in those node's elasticsearch config.
Keep in mind that the list might possibly not include the original node it connected to
if that node is not a data node. If, for instance, you initially connect to a
master node, after sniffing, no further requests will go to that master node,
but rather to any data nodes instead. The reason the transport client excludes non-data
nodes is to avoid sending search traffic to master only nodes.
We may choose to set this option to true by default and expose an option to disable it from nuxeo.conf.