OpenSearch ISM & Cross-Cluster Replication

Node Role Allocation

Node Role Allocation dictates how OpenSearch distributes primary and replica shards across cluster infrastructure based on explicit role constraints, resource thresholds, and Index State Management (ISM) directives. In production environments handling high-throughput ingestion and Cross-Cluster Replication (CCR), misconfigured allocation rules directly cause shard unassignment, replication lag, and ISM policy stalls. This operational guide details exact configuration payloads, threshold calibration, and programmatic orchestration required to enforce deterministic shard placement. For foundational cluster topology concepts, refer to OpenSearch ISM Architecture & Fundamentals.

Allocation Decision Pipeline & Decider Chain

OpenSearch evaluates shard placement through a strict decider pipeline that executes in deterministic order: cluster.routing.allocation.enable state, disk watermark thresholds, shard count limits, and finally, index-level routing constraints. Modern OpenSearch (2.x+) relies on native _tier_preference routing rather than legacy node.attr tags, though both remain supported for backward compatibility.

When an index is created or rolled over, the routing decider matches index.routing.allocation.require._tier_preference against node capabilities (data_hot, data_warm, data_cold, data_frozen). CCR follower indices inherit the leader’s allocation constraints by default, which can cause routing conflicts if the follower cluster lacks matching tier capacity. Operators must explicitly override follower routing during index creation or via dynamic settings. Detailed mapping strategies for aligning infrastructure capabilities with routing tags are documented in Mapping data tiers to OpenSearch node roles.

Mismatched routing tags trigger ALLOCATION_FAILED or NODE_LEFT states, blocking ISM rollovers and halting CCR checkpoint synchronization. Always validate that node roles declared in opensearch.yml exactly match the tier strings referenced in index templates.

flowchart LR
    H["data_hot - NVMe (ingest, real-time search)"] -->|"age / size"| W["data_warm - SSD (recent history)"]
    W -->|"retention"| C["data_cold - HDD (infrequent access)"]
    C -->|"archive"| F["data_frozen - object storage (snapshots)"]

Threshold Calibration & Watermark Management

Default disk watermarks are calibrated for single-tier deployments and will prematurely throttle shard allocation in multi-tier architectures. Operators must calibrate thresholds to align with underlying storage performance, ISM rollover cadence, and CCR replication windows.

JSON
PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "82%",
    "cluster.routing.allocation.disk.watermark.high": "88%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "93%",
    "cluster.routing.allocation.disk.threshold_enabled": true,
    "cluster.routing.allocation.disk.include_relocations": true,
    "cluster.routing.allocation.node_concurrent_recoveries": 3,
    "cluster.routing.allocation.node_initial_primaries_recoveries": 6
  }
}

Lowering the low and high thresholds prevents sudden I/O saturation during ISM phase transitions. The node_concurrent_recoveries value should be capped at 3–4 for NVMe-backed hot tiers and reduced to 1–2 for HDD-backed warm/cold tiers to avoid storage queue depth exhaustion. Capacity planning for tiered storage ratios is covered in Hot-Warm-Cold Tier Design.

ISM Phase Transitions & CCR Routing Inheritance

ISM policies automate shard routing by injecting allocation tags during phase transitions. When an index moves from hot to warm, the policy must execute an allocation action that updates index.routing.allocation.require.<attr> (for example index.routing.allocation.require.data). If the target tier lacks available capacity, the transition stalls in WAITING state until watermarks clear or nodes are provisioned.

CCR introduces additional routing complexity. Follower indices automatically inherit the leader’s allocation tags, which can cause cross-cluster routing mismatches if the follower cluster uses different node role naming conventions or tier capacities. Override this behavior during follower creation:

JSON
PUT _plugins/_replication/follower-index/_start
{
  "leader_alias": "leader-cluster",
  "leader_index": "logs-prod-2024.01",
  "settings": {
    "index.routing.allocation.require._tier_preference": "data_warm",
    "index.number_of_replicas": 1
  }
}

Understanding how ISM triggers interact with routing constraints is essential for preventing replication drift. Core transition mechanics are outlined in Index Lifecycle Basics.

Programmatic Enforcement & Template Automation

Manual routing configuration does not scale across dynamic environments. The following Python automation script enforces deterministic allocation tags, updates index templates, and validates CCR follower overrides using the OpenSearch REST API. It implements exponential backoff, idempotent checks, and production-grade error handling.

Python
import os
import time
import logging
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")
logger = logging.getLogger(__name__)

class OpenSearchAllocator:
    def __init__(self, base_url: str, username: str, password: str):
        self.base_url = base_url.rstrip("/")
        self.session = requests.Session()
        self.session.auth = (username, password)
        self.session.verify = True
        
        retry_strategy = Retry(
            total=3, backoff_factor=1.5,
            status_forcelist=[429, 500, 502, 503, 504]
        )
        self.session.mount("https://", HTTPAdapter(max_retries=retry_strategy))

    def update_index_template(self, template_name: str, tier: str, pattern: str) -> bool:
        """Apply deterministic tier routing to a new/updated index template."""
        payload = {
            "index_patterns": [pattern],
            "template": {
                "settings": {
                    "index.routing.allocation.require._tier_preference": tier,
                    "index.number_of_shards": 3,
                    "index.number_of_replicas": 1
                }
            }
        }
        url = f"{self.base_url}/_index_template/{template_name}"
        try:
            resp = self.session.put(url, json=payload)
            resp.raise_for_status()
            logger.info(f"Template '{template_name}' updated with tier '{tier}'.")
            return True
        except requests.exceptions.RequestException as e:
            logger.error(f"Failed to update template: {e}")
            return False

    def override_ccr_follower_routing(self, follower_index: str, tier: str, leader_alias: str, leader_index: str) -> bool:
        """Start CCR replication with explicit allocation override."""
        payload = {
            "leader_alias": leader_alias,
            "leader_index": leader_index,
            "settings": {
                "index.routing.allocation.require._tier_preference": tier,
                "index.number_of_replicas": 1
            }
        }
        url = f"{self.base_url}/_plugins/_replication/{follower_index}/_start"
        try:
            resp = self.session.put(url, json=payload)
            resp.raise_for_status()
            logger.info(f"CCR follower '{follower_index}' started with routing override to '{tier}'.")
            return True
        except requests.exceptions.RequestException as e:
            logger.error(f"CCR override failed: {e}")
            return False

    def validate_allocation(self, index: str) -> dict:
        """Return allocation explain payload for diagnostic routing."""
        url = f"{self.base_url}/_cluster/allocation/explain"
        payload = {"index": index}
        try:
            resp = self.session.post(url, json=payload, timeout=10)
            resp.raise_for_status()
            return resp.json()
        except requests.exceptions.RequestException as e:
            logger.error(f"Allocation validation failed: {e}")
            return {}

if __name__ == "__main__":
    allocator = OpenSearchAllocator(
        base_url=os.getenv("OPENSEARCH_URL", "https://localhost:9200"),
        username=os.getenv("OPENSEARCH_USER", "admin"),
        password=os.getenv("OPENSEARCH_PASS", "admin")
    )
    
    allocator.update_index_template("logs-prod-template", "data_hot", "logs-prod-*")
    allocator.override_ccr_follower_routing(
        "logs-prod-2024.01-follower", "data_warm", "leader-cluster", "logs-prod-2024.01"
    )
    explain = allocator.validate_allocation("logs-prod-2024.01-follower")
    logger.info(f"Allocation state: {explain.get('state', 'UNKNOWN')}")

For HTTP client best practices and connection pooling in production automation, consult the official Python Requests Documentation.

Validation & Operational Diagnostics

After deploying allocation rules, verify routing compliance using cluster diagnostics APIs. Avoid relying solely on _cat/shards for routing validation, as it does not expose decider rejection reasons.

Shell
# Explain why a specific shard cannot be allocated
curl -X POST "https://<cluster>:9200/_cluster/allocation/explain" \
  -H "Content-Type: application/json" \
  -d '{"index": "logs-prod-2024.01", "shard": 0, "primary": true}'

# List unassigned shards with rejection reasons
curl -X GET "https://<cluster>:9200/_cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason&s=state"

Common rejection states:

  • ALLOCATION_FAILED: Routing tag mismatch or insufficient node capacity for require constraints.
  • DISK_THRESHOLD: Node storage exceeds watermark.high. Relocate shards or expand volume.
  • CLUSTER_RECOVERED: Post-restart allocation backlog. Increase node_initial_primaries_recoveries temporarily.
  • REPLICA_ADDED_TO_ACTIVE_INDEX: Normal state during CCR sync. Monitor replication_lag metrics.

Maintain allocation stability by auditing node role assignments quarterly, aligning watermark thresholds with storage vendor SLAs, and validating ISM policy routing actions against actual cluster capacity.