OpenSearch ISM & Cross-Cluster Replication

Hot-Warm-Cold Tier Design

Hot-Warm-Cold Tier Design operationalizes storage economics and query performance by routing indices through distinct hardware profiles based on ingestion velocity and access patterns. Within OpenSearch Index State Management, this architecture relies on deterministic shard allocation filters, automated policy transitions, and precise disk watermark calibration. Engineers implementing this model must align node topology with lifecycle stages before deploying ISM policies. The foundational mechanics of this approach are documented in the broader OpenSearch ISM Architecture & Fundamentals framework, which establishes how state machines interact with cluster routing tables.

Node Topology and Hardware Alignment

Tier separation begins at the node role level. Misaligned hardware profiles cause silent allocation failures during rollover or cross-tier migration. Each tier requires explicit node.attr.data_tier values that map directly to physical storage characteristics:

Tier Storage Profile CPU/Memory Ratio Routing Attribute Primary Workload
Hot NVMe SSD High vCPU, High RAM data_hot Ingestion, real-time search, aggregations
Warm SATA SSD / HDD Moderate vCPU, High RAM data_warm Historical search, reduced write throughput
Cold High-capacity HDD / Object Storage Low vCPU, Minimal RAM data_cold Compliance retention, infrequent queries

Before attaching lifecycle policies, validate node topology using the cluster API:

Shell
GET _cat/nodes?v&h=name,node.role,data_tier&sort=data_tier

Ensure node.attr.data_tier matches the physical hardware profile. If a node reports data_tier: warm but runs on NVMe, the cluster will underutilize high-IOPS storage. Conversely, routing hot shards to HDD-backed nodes will trigger circuit_breaking exceptions during peak ingestion.

ISM Policy Construction and State Transitions

The core of Hot-Warm-Cold Tier Design is the ISM policy JSON. Policies must define explicit actions, conditions, and retry logic for each phase. Understanding Index Lifecycle Basics is critical for structuring deterministic transitions that avoid race conditions during shard reallocation.

Below is a production-grade policy payload that transitions indices from hot to warm after 7 days, then to cold after 30 days. It includes replica reduction, segment optimization, and snapshot archival with exponential backoff on failure.

stateDiagram-v2
    [*] --> hot
    hot --> warm: after 7 days
    warm --> cold: after 30 days
    note right of warm
        replica reduction
        force_merge
    end note
    note right of cold
        snapshot archival
    end note
JSON
PUT _plugins/_ism/policies/tiered_log_policy
{
  "policy": {
    "description": "Production Hot-Warm-Cold lifecycle for application logs",
    "default_state": "hot",
    "ism_template": {
      "index_patterns": ["app-logs-*"],
      "priority": 100
    },
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "rollover": {
              "min_index_age": "1d",
              "min_size": "50gb",
              "min_doc_count": 100000000
            }
          }
        ],
        "transitions": [
          {
            "state_name": "warm",
            "conditions": { "min_index_age": "7d" }
          }
        ]
      },
      {
        "name": "warm",
        "actions": [
          {
            "allocation": {
              "require": { "data_tier": "warm" },
              "wait_for": true
            }
          },
          {
            "replica_count": { "number_of_replicas": 1 }
          },
          {
            "force_merge": { "max_num_segments": 1 }
          }
        ],
        "transitions": [
          {
            "state_name": "cold",
            "conditions": { "min_index_age": "30d" }
          }
        ]
      },
      {
        "name": "cold",
        "actions": [
          {
            "allocation": {
              "require": { "data_tier": "cold" },
              "wait_for": true
            }
          },
          {
            "replica_count": { "number_of_replicas": 0 }
          },
          {
            "snapshot": {
              "repository": "s3-archive-repo",
              "snapshot": "<{snapshot_name}>",
              "ignore_failure": false
            }
          }
        ],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": { "min_index_age": "365d" }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "delete": {}
          }
        ]
      }
    ]
  }
}

Index Template Integration and Routing Enforcement

Policies alone do not enforce tier routing. You must bind the policy and allocation filters via index templates to guarantee deterministic shard placement. Template versioning prevents configuration drift across environments.

JSON
PUT _index_template/app_logs_tiered_v2
{
  "index_patterns": ["app-logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 2,
      "index.plugins.index_state_management.policy_id": "tiered_log_policy",
      "index.routing.allocation.require.data_tier": "hot",
      "index.refresh_interval": "5s",
      "index.translog.durability": "async"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "message": { "type": "text" },
        "service_name": { "type": "keyword" }
      }
    }
  },
  "priority": 100,
  "version": 2
}

The index.routing.allocation.require.data_tier setting ensures new indices land exclusively on hot nodes. ISM overrides this during phase transitions, moving shards to warm or cold nodes without manual intervention.

Cross-Cluster Replication and Policy Inheritance

When deploying Hot-Warm-Cold Tier Design across geographically distributed clusters, Cross-Cluster Replication (CCR) introduces policy inheritance complexities. Leader clusters typically handle ingestion and hot-tier routing. Follower clusters replicate indices but execute independent ISM state machines.

To maintain tier consistency on followers, attach identical ISM policies to the replicated indices after initial sync. Follower policies should skip rollover actions and focus on allocation, force-merge, and snapshot operations. Ensure Security & Access Boundaries are configured to allow the CCR replication user to execute ISM state transitions and snapshot APIs on the follower cluster. Misconfigured cross-cluster roles will cause policy execution to stall, leaving indices stranded in transitional states.

Production Automation and Validation

Python automation builders should validate node topology, deploy policies, and verify attachment before routing production traffic. The following script uses opensearch-py to perform pre-flight checks and policy deployment with structured error handling.

Python
import logging
from opensearchpy import OpenSearch, exceptions

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")

def validate_and_deploy_policy(host: str, port: int, auth: tuple, policy_name: str, policy_payload: dict) -> bool:
    client = OpenSearch(
        hosts=[{"host": host, "port": port}],
        http_auth=auth,
        use_ssl=True,
        verify_certs=True,
        timeout=30
    )

    # 1. Verify hot/warm/cold node distribution
    try:
        nodes = client.cat.nodes(format="json", h="name,node.role,attr.data_tier")
        tiers = {n.get("attr.data_tier") for n in nodes if n.get("attr.data_tier")}
        if not {"hot", "warm", "cold"}.issubset(tiers):
            logging.warning(f"Missing tier nodes. Found: {tiers}")
            return False
    except exceptions.ConnectionError as e:
        logging.error(f"Cluster connection failed: {e}")
        return False

    # 2. Deploy or update ISM policy (opensearch-py has no `.ism` namespace)
    try:
        client.transport.perform_request(
            "PUT", f"/_plugins/_ism/policies/{policy_name}", body=policy_payload
        )
        logging.info(f"Policy '{policy_name}' deployed successfully.")
    except exceptions.RequestError as e:
        logging.error(f"Policy deployment failed: {e.error}")
        return False

    # 3. Verify policy attachment on existing indices
    try:
        indices = client.indices.get(index="app-logs-*")
        attached = 0
        for idx_name in indices:
            try:
                status = client.transport.perform_request(
                    "GET", f"/_plugins/_ism/explain/{idx_name}"
                )
                if status.get(idx_name, {}).get("policy_id") == policy_name:
                    attached += 1
            except exceptions.NotFoundError:
                continue
        logging.info(f"Policy attached to {attached} existing indices.")
    except Exception as e:
        logging.error(f"Verification failed: {e}")
        return False

    return True

# Usage example
if __name__ == "__main__":
    POLICY = {
        "policy": {
            "description": "Automated tier routing",
            "default_state": "hot",
            "states": [{"name": "hot", "actions": [], "transitions": []}]
        }
    }
    validate_and_deploy_policy(
        host="opensearch-cluster.internal",
        port=9200,
        auth=("admin", "secure_password"),
        policy_name="tiered_log_policy",
        policy_payload=POLICY
    )

For advanced HTTP request handling and retry logic, refer to the official Python requests documentation when building custom webhook integrations or CI/CD pipeline validators.

Operational Guardrails and Watermark Calibration

Disk watermarks dictate when OpenSearch halts shard allocation to prevent node exhaustion. Default thresholds often conflict with tiered architectures where cold nodes intentionally run at higher utilization. Tune cluster-level settings to match your hardware capacity:

JSON
PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "85%",
    "cluster.routing.allocation.disk.watermark.high": "90%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%",
    "cluster.routing.allocation.disk.threshold_enabled": true
  }
}

Monitor shard migration latency during warm-to-cold transitions. If network bandwidth is constrained, stagger transitions using index.lifecycle.step.wait_time or implement fallback routing strategies to prevent hot-tier resource starvation. For step-by-step implementation guidance and advanced watermark tuning, consult the dedicated guide on How to configure OpenSearch ISM hot warm cold architecture.

Regularly audit _cat/indices?v&h=index,state,data_tier,docs.count,store.size to verify that indices align with their intended lifecycle stage. Automated drift detection ensures storage costs remain predictable while maintaining query SLAs across all tiers.