Handling async ISM policy execution failures

OpenSearch Index State Management (ISM) executes policy phases asynchronously via background job runners. When a phase transition fails, the initial _plugins/_ism/add API returns 200 OK, masking the underlying failure until the job runner evaluates the next execution window. Handling async ISM policy execution failures requires deterministic state inspection, targeted retry orchestration, and automated fallback logic to prevent index lifecycle drift. Because ISM operates on a scheduler tick rather than synchronous API calls, engineers must decouple policy attachment from execution verification and implement continuous state validation pipelines.

Diagnostic Workflow: Extracting Failure Signatures #

ISM job state is exposed through the explain API. Query the exact index to retrieve the current step, error message, and retry configuration:

GET _plugins/_ism/explain/<index_name>

Parse the state and step_info fields. A FAILED state indicates a terminal or exhausted retry condition. Extract the step_name and failed_step to correlate with phase transition logic. For example, a rollover step failure typically returns a structured JSON payload containing failed_step and step_info.cause. Cross-reference step_info.cause against cluster metrics. If cause references cluster_block_exception, verify write availability and disk watermark thresholds. If it references replication_conflict, the index is likely a Cross-Cluster Replication (CCR) follower. Follower indices cannot execute rollover, delete, or snapshot phases natively. ISM on CCR requires leader-side policy enforcement or manual follower index lifecycle management. For deeper context on scheduler behavior and job runner queuing, review Async Execution Patterns to understand how background runners evaluate phase transitions under load.

Retry Orchestration & Threshold Tuning Strategies #

ISM supports automatic retries with configurable backoff. When a step fails, the job runner applies the policy-defined retry block. If retries are exhausted, the policy halts and requires manual intervention. To force an immediate retry without waiting for the next scheduler tick:

POST _plugins/_ism/retry/<index_name>
{
  "state": "hot",
  "action": "rollover"
}

This payload resets the step state and triggers synchronous evaluation. For persistent failures, adjust retry parameters directly in the policy DSL before reapplying:

"actions": {
  "rollover": {
    "retry": {
      "count": 5,
      "backoff": "exponential",
      "delay": "10m"
    }
  }
}

Threshold tuning strategies often resolve transient failures. If rollover triggers on min_index_age but fails due to shard allocation pressure, decouple the condition from min_size and introduce a max_primary_shard_size guard. This prevents the job runner from attempting a rollover on a partially allocated index. Aligning rollover trigger configuration with cluster capacity metrics reduces false-positive failures during peak ingestion windows.

Python Automation for State Polling & Remediation #

Manual API polling does not scale across hundreds of indices. Implement a Python orchestration loop that queries the explain endpoint, parses failure signatures, and executes targeted retries. The following script uses the opensearch-py client to automate state inspection and remediation:

import os
import logging
from opensearchpy import OpenSearch, ConnectionError

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
logger = logging.getLogger(__name__)

def remediate_ism_failures(hosts, index_pattern):
    client = OpenSearch(
        hosts=hosts,
        http_auth=(os.getenv("OPENSEARCH_USER"), os.getenv("OPENSEARCH_PASSWORD")),
        use_ssl=True,
        verify_certs=True,
        timeout=30
    )
    try:
        response = client.indices.get(index=index_pattern)
        indices = list(response.keys())
    except Exception as e:
        logger.error("Failed to resolve index pattern %s: %s", index_pattern, e)
        return

    for idx in indices:
        try:
            explain_resp = client.transport.perform_request(
                "GET", f"/_plugins/_ism/explain/{idx}"
            )
            # Explain returns index names as top-level keys (plus total_managed_indices).
            idx_data = explain_resp.get(idx, {})
            action = idx_data.get("action", {})
            step = idx_data.get("step", {})

            # Failure is signalled by action.failed / step.status, not by a "FAILED" state.
            if action.get("failed") or step.get("status") == "failed":
                state_name = idx_data.get("state", {}).get("name", "hot")
                failed_action = action.get("name", "unknown")
                cause = idx_data.get("info", {}).get("message", "Unknown")
                logger.error("Index %s failed during action '%s' (state '%s'): %s",
                             idx, failed_action, state_name, cause)

                # An empty retry body re-runs the failed action in the current state.
                client.transport.perform_request(
                    "POST", f"/_plugins/_ism/retry/{idx}"
                )
                logger.info("Triggered retry for %s", idx)
        except ConnectionError as e:
            logger.error("Connection failed during explain check for %s: %s", idx, e)
        except Exception as e:
            logger.exception("Unexpected error processing %s", idx)

Integrate this routine into your CI/CD pipelines or cron-based schedulers. For comprehensive guidance on structuring these workflows and managing policy DSL templates programmatically, consult the ISM Policy Implementation & Python Automation pillar documentation. The script safely handles connection drops, enforces TLS verification, and logs structured failure metadata for downstream alerting. Validate endpoint payloads against the official OpenSearch ISM API Reference to ensure compatibility across cluster versions.

Policy Rollback & Fallback Logic #

When automated retries exhaust and threshold tuning fails to resolve the underlying constraint, implement a deterministic rollback strategy. Remove the failing policy to prevent continuous scheduler churn, then attach a simplified fallback policy that enforces only essential retention or snapshot actions. Use the POST _plugins/_ism/remove/<index> API to detach the broken policy, verify index health, and reapply a corrected DSL. Document all terminal failure signatures in a centralized runbook to accelerate incident response. Aligning error handling & retries with explicit policy rollback strategies ensures cluster stability during prolonged ingestion anomalies. Configure automated health checks to trigger policy detachment when the explain response’s info.message matches persistent infrastructure constraints, such as exhausted disk space or network partition events.