What do the different cluster states mean?

Created by Steve Place, Modified on Wed, Oct 30 at 1:13 PM by Steve Place

You may see messages like this in your Stardog log:

[node 1 name]: Cluster membership state changed (previous: ACTIVE; new: ACTIVE)
[node 3 name]: Cluster membership state changed (previous: ACTIVE; new: INACTIVE)
[node 2 name]: Cluster membership state changed (previous: ACTIVE; new: SUSPENDED)
[node 2 name]: Cluster membership state changed (previous: SUSPENDED; new: ACTIVE)

There are 3 states your nodes can be in: ACTIVE, INACTIVE, and SUSPENDED.

ACTIVE - The node is part of the cluster.
INACTIVE - The node is not part of the cluster.
- When a node is INACTIVE, it will not process requests, and it will fail the health check. However, because load balancers only query the health check periodically, there may be windows that requests are still routed to an inactive Stardog node; this results in a NotInCluster exception.
- A node can become INACTIVE for multiple reasons, including because the curator is in the suspended state (see next bullet) or because it is expelled by the coordinator.
SUSPENDED - This state comes from Curator (the client Stardog uses to connect to ZooKeeper) and denotes the connection to ZooKeeper has been lost. The node is not part of the cluster when in this state. When the curator client is SUSPENDED, we immediately move the Stardog node to INACTIVE.

A node can move through these different states and, depending on the sequence of events, potentially rejoin the cluster and become ACTIVE.