Harness engineering: how to build reliable AI agents

Key takeaways

• Start with a task contract that makes the outcome, scope, invariants, budgets, and escalation conditions explicit.
• Design completion gates before execution so the harness can prove success from current system state rather than a model summary.
• Keep authority deterministic: the model may discover a path, but code and policy own permissions, side effects, and finish conditions.
• Anchor checkpoints to the repository or external state that produced them, and reject stale or incompatible recovery state.
• Choose retry and recovery behavior by side-effect class; ambiguous writes and irreversible actions require reconciliation, not blind repetition.

This guide starts where harness anatomy ends. Once an agent runtime can call tools, maintain state, and loop, the reliability problem becomes concrete: what job is allowed, what side effects are permitted, what evidence proves completion, and what should happen after an interrupted or ambiguous action.

Those controls should be enforced by runtime code and policy rather than left to the model’s memory or prose instructions. The sections below focus on execution reliability. For the broader architecture, tracing, evaluation, and improvement loop, use Arize’s agent harness architecture, tracing, and evaluation guide.

What does harness engineering mean in this guide?

Harness engineering is the task-specific work of governing an agent runtime so a production job is bounded, verifiable, resumable, and recoverable without prescribing the path the model must take. The harness supplies the loop and operational primitives; harness engineering defines the contract, authority, evidence, state, and recovery rules for the job.

An agent harness is the runtime; harness engineering is the reliability work that makes that runtime dependable for a specific class of work. The distinction matters because these controls define the execution envelope, not the trajectory through it. The model can still search, choose tools, compact context, delegate work, and adapt its next step to intermediate evidence. Deterministic code and policy retain control over what may execute, what side effects are permitted, what state can be trusted, and what counts as finished.

For the nine-component anatomy and the framework-versus-harness distinction, see What is an agent harness?.

The term also sits inside a broader shift toward agent-driven software engineering. OpenAI uses “harness engineering” for designing environments, specifying intent, and building feedback loops around coding agents. Andrej Karpathy’s autoresearch makes a bounded loop concrete with one agent-editable training file, a fixed five-minute run budget, a single objective metric, and a keep-or-discard cycle. Hermes, Nous Research’s open-source agent harness, shows a complementary runtime pattern with durable sessions, controlled tool exposure, context compression, and long-running execution.

Regardless of terminology, the engineering boundary is the same: prompts can influence the model, but contracts, permission checks, state verification, and completion gates must be enforced outside the model when they need to be guaranteed.

Harness engineering does not replace agent autonomy with a predefined workflow. It specifies the conditions that must remain true while the model determines how to get there. A workflow can prescribe the path; an agent harness can let the model discover the path while retaining deterministic control over authority, state, side effects, and completion.

What makes a harness reliable in production?

For this guide, reliability means six properties.

Try Arize AX

Build better agents with Arize

Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.

Get started View AX docs

Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.

The reliable harness engineering control loop

A reliable harness lets the model choose the next useful action while the runtime owns the contract, authorization, completion evidence, checkpoint state, and recovery decision.

The loop begins with a task contract and ends only when completion gates pass against current state. Between those points, the model can adapt the path. After each material side effect, the runtime records enough verified state to resume or reconcile without turning “try again” into the default recovery strategy.

How do you engineer a reliable harness?

The following sequence works across coding agents, support agents, data-analysis agents, research agents, and internal automation. The examples use a coding agent asked to add per-client API rate limiting because the artifacts and side effects are easy to inspect.

1. Define a task contract before the model starts

Translate the user request into a compact contract that the runtime can inspect. The contract should specify the outcome, allowed scope, invariants, completion evidence, budgets, and escalation conditions. Keep it short enough for a developer to review and structured enough for code to enforce.

The values below are illustrative. In production, budgets should be tuned to the task class and enforced by the runtime rather than treated as prompt instructions.

task:

id: add-api-rate-limiting
outcome:

description: "Return HTTP 429 after a client exceeds the configured limit"
allowed_changes:

- middleware/**

- config/rate_limit.*

- tests/rate_limit/**
forbidden_changes:

- auth/**

- public_api/**

- database_schema/**
invariants:

- existing_authorization_tests_pass

- unrelated_required_tests_pass

- public_api_contract_unchanged
budgets:

wall_clock_minutes: 30

model_calls: 40

tool_calls: 80

repair_attempts_per_gate: 2
completion_gates:

- targeted_tests.exit_code == 0

- required_suite.exit_code == 0

- diff.outside_allowed_scope == false

- response_contract.status == 429

- verification.workspace_fingerprint == delivery.workspace_fingerprint
escalate_when:

- new_dependency_required

- schema_change_required

- access_outside_sandbox_required

- completion_gate_still_fails_after_budget

A task contract is useful only when the runtime can enforce it. allowed_changes should become a real scope validator. Budgets should stop or escalate execution when exhausted. Completion gates should inspect current artifacts and system state. Escalation conditions should transition the run into a defined state rather than relying on the model to remember that it was supposed to ask for help.

For the rate-limiting task, this means the agent can decide which middleware implementation to inspect and how to fix it, but it cannot modify auth/**, silently add a dependency, consume unlimited retries, or finish because it says the tests passed.

Completion principle: a convincing final message is not evidence. The harness should inspect the current test reports, current diff, current workspace version, required approvals, and any relevant external state. If that evidence is missing, stale, or outside the contract, the run is not complete.

2. Design completion gates around the side effect

Do not use “exactly once” as a generic completion guarantee for external writes. Whether a retry is safe depends on the service’s idempotency semantics, the operation identity, and what can be established from current external state. When the first result is ambiguous, reconciliation should happen before another write is attempted.

3. Separate model discovery from deterministic authority

Use a model when the next useful action depends on ambiguous evidence. Use deterministic code or policy when an unacceptable outcome can be stated exactly. This division lets the agent adapt without allowing natural-language instructions to become the authorization system.

This is the same boundary established earlier: the model owns discovery inside the task envelope; deterministic controls own authority and verification.

Engineer guarantees, not compensations for model weakness

The best harness controls remain useful as models improve. Authorization boundaries, completion evidence, idempotency, durable state, and reconciliation protect system invariants regardless of model capability. By contrast, elaborate routing rules or prompt choreography added only to compensate for a model’s current limitations may become unnecessary as models improve.

Treat those two classes differently. Make guarantees stable and explicit. Keep model-specific scaffolding easy to remove, replace, and evaluate.

4. Bound capabilities and permissions

A capability interface does not replace the harness control plane. MCP standardizes how hosts and clients connect to servers that expose tools, resources, and prompts. Skills package reusable task procedures. CLIs and APIs expose execution surfaces. The harness still decides which capability is visible for the current task and whether a proposed invocation is valid and authorized.

MCP includes protocol-level authorization for supported HTTP deployments, but authorization is optional in the protocol and does not encode your organization’s business policy. Tenant boundaries, data classification, action risk, and user-specific authority still belong in the host, server, or a separate policy layer.

What Arize’s 500-run benchmark showed

On the hardest GitHub analysis tasks, a thin MCP surface averaged about 12 tool calls versus 5 for skills, with more than 6x the cost and 5x the latency. MCP tool fidelity was 0.33. The result reversed when the task mapped cleanly to endpoints: creating a branch and pull request averaged 8 calls with MCP versus 22 with a verbose skill. The lesson is to select capabilities by task shape, composition, authorization, and deployment context rather than declaring one interface universally superior.

Source: MCP vs. CLI Skills for agents: what our eval found. The experiment used GitHub tasks and a thin REST-style MCP server, so the measured ratios should not be generalized to every MCP architecture.

Use both interfaces and let the harness choose by task shape. Use the CLI for local workflows, tools with deep training-data coverage, and work that benefits from composition. You should use MCP when the tool is remote or proprietary, when you need OAuth and per-user authorization, when real state spans steps, or when you want an entire agent behind a single tool call. The harness, not the prompt, should decide which surface is visible and authorized for the current stage.

Enforce policy outside the model

A pre-execution policy decision should validate more than a tool name. Check the requesting principal, user or tenant context, capability, resource, arguments, data classification, environment, and current state. A post-execution hook should capture the actual side effect, redact sensitive output, update durable state, and trigger the relevant evaluator or monitor.

External enforcement limits what a prompt injection can cause; it does not eliminate prompt injection. An injected instruction can still persuade the model to request an action that falls inside an overly broad permission rule. Use least privilege, tool-specific parameter validation, sandboxing, output handling, monitoring, and human approval for high-risk or irreversible operations.

5. Make every long-running unit resumable

A long-running task can outlive a model call, worker, browser session, sandbox, or deployment. Divide it into units that end with a durable record of the input version, action, output artifact, verification result, external side effects, and next eligible step. Resume from the last verified unit rather than from the model’s memory of the conversation.

For the rate-limiting change, useful units are: reproduce the current behavior, add a failing test, implement the change, run targeted tests, run the required suite, inspect scope, and prepare the delivery artifact. Each unit should be independently verifiable and safe to skip when its evidence still matches current state.

Checkpoint the state that produced the evidence

A commit ID is not enough to decide whether verified work can be reused. A coding-agent workspace can also contain staged changes, unstaged changes, untracked files, generated artifacts, and execution context that did not exist in the commit.

A useful checkpoint binds a completed unit of work to the state that produced its evidence.

For example, imagine the agent runs the targeted rate-limit tests successfully and records that stage as complete. A developer then changes middleware/rate_limit.py before the worker resumes. The test result may still exist on disk, but it no longer proves anything about the code that will be delivered.

The harness should reject that checkpoint.

There is an important implementation detail here: Git status alone is not a content fingerprint. Two different versions of a modified file can both appear as M middleware/rate_limit.py. An untracked file can change contents while still appearing as ?? file. Recovery state must therefore incorporate the actual changed contents, not only their status labels.

The following Python 3.9+ example fingerprints:

• the current HEAD
• the index tree
• actual unstaged changes to tracked files
• paths, modes, and contents of non-ignored untracked files
• explicitly referenced verification artifacts
• a checkpoint schema version

It also rejects artifacts that resolve outside the repository.

"""Repository-scoped checkpointing for a coding-agent harness. Python 3.9+."""
from __future__ import annotations
import hashlib

import json

import os

import stat

import subprocess

import tempfile

from pathlib import Path

from typing import Dict, List, Optional
SCHEMA_VERSION = 2
AGENT_STATE_DIR = Path(

os.environ.get("AGENT_STATE_DIR", "/tmp/agent-state")

)
def repo_root(start: Optional[Path] = None) -> Path:

start = (start or Path.cwd()).resolve()
out = subprocess.check_output(

[

"git",

"-C",

str(start),

"rev-parse",

"--show-toplevel",

],

text=True,

).strip()
return Path(out).resolve()
def _git_bytes(root: Path, *args: str) -> bytes:

return subprocess.check_output(

["git", "-C", str(root), *args]

)
def _add(h, label: str, data: bytes) -> None:

"""Add a length-delimited field to the fingerprint."""

h.update(label.encode("utf-8"))

h.update(b"")

h.update(len(data).to_bytes(8, "big"))

h.update(data)
def _file_sha256(path: Path) -> bytes:

h = hashlib.sha256()
with path.open("rb") as f:

for chunk in iter(

lambda: f.read(1024 * 1024),

b"",

):

h.update(chunk)
return h.digest()
def workspace_fingerprint(

root: Optional[Path] = None,

) -> str:

"""

Fingerprint committed, staged, tracked-worktree,

and non-ignored untracked state.

"""

root = (root or repo_root()).resolve()

h = hashlib.sha256()
# Committed state.

_add(

h,

"head",

_git_bytes(root, "rev-parse", "HEAD").strip(),

)
# Staged/index state.

_add(

h,

"index",

_git_bytes(root, "write-tree").strip(),

)
# Actual unstaged contents and mode changes

# for tracked files.

worktree_diff = _git_bytes(

root,

"-c",

"diff.external=",

"diff",

"--binary",

"--full-index",

"--no-ext-diff",

"--no-textconv",

"--no-renames",

"--diff-algorithm=myers",

"--no-color",

)
_add(

h,

"tracked_worktree",

worktree_diff,

)
# git diff does not include untracked files.

untracked = _git_bytes(

root,

"ls-files",

"--others",

"--exclude-standard",

"-z",

).split(b"")
for rel_bytes in sorted(

path for path in untracked if path

):

rel = os.fsdecode(rel_bytes)

path = root / rel

mode = path.lstat().st_mode
_add(

h,

"untracked_path",

rel_bytes,

)
_add(

h,

"untracked_mode",

str(stat.S_IMODE(mode)).encode("ascii"),

)
if stat.S_ISLNK(mode):

_add(

h,

"untracked_symlink",

os.fsencode(os.readlink(path)),

)

elif stat.S_ISREG(mode):

_add(

h,

"untracked_file",

_file_sha256(path),

)

else:

raise ValueError(

"unsupported untracked file type: "

f"{rel}"

)
return h.hexdigest()
def _safe_repo_file(

root: Path,

rel: str,

) -> Path:

root = root.resolve()

raw = root / rel
# Keep the example simple: verification artifacts

# must be regular files rather than symlinks.

if raw.is_symlink():

raise ValueError(

f"artifact may not be a symlink: {rel}"

)
path = raw.resolve(strict=True)
if not path.is_relative_to(root):

raise ValueError(

f"artifact escapes repository: {rel}"

)
if not path.is_file():

raise ValueError(

f"artifact is not a regular file: {rel}"

)
return path
def artifact_hashes(

paths: List[str],

root: Optional[Path] = None,

) -> Dict[str, str]:

root = (root or repo_root()).resolve()
return {

rel: _file_sha256(

_safe_repo_file(root, rel)

).hex()

for rel in paths

}
def _checkpoint_path(root: Path) -> Path:

root = root.resolve()
state_dir = (

AGENT_STATE_DIR

.expanduser()

.resolve()

)
# Storing the checkpoint in the worktree would

# make writing the checkpoint alter the state

# that the checkpoint itself fingerprints.

if state_dir.is_relative_to(root):

raise ValueError(

"AGENT_STATE_DIR must not live "

"inside the worktree"

)
return state_dir / "checkpoint.json"
def save_checkpoint(

state: dict,

root: Optional[Path] = None,

) -> Path:

"""

Write the checkpoint atomically within one filesystem.

"""

root = (root or repo_root()).resolve()

checkpoint = _checkpoint_path(root)
checkpoint.parent.mkdir(

parents=True,

exist_ok=True,

)
payload = json.dumps(

state,

indent=2,

sort_keys=True,

).encode("utf-8")
tmp_path = None
try:

with tempfile.NamedTemporaryFile(

mode="wb",

dir=checkpoint.parent,

prefix="checkpoint.",

suffix=".tmp",

delete=False,

) as f:

tmp_path = Path(f.name)

f.write(payload)

f.flush()

os.fsync(f.fileno())
os.replace(

tmp_path,

checkpoint,

)
return checkpoint
finally:

if (

tmp_path is not None

and tmp_path.exists()

):

tmp_path.unlink()
def load_checkpoint(

root: Optional[Path] = None,

) -> Optional[dict]:

"""

Return state only when the schema, workspace,

and referenced artifacts still match.

"""

root = (root or repo_root()).resolve()

checkpoint = _checkpoint_path(root)
if not checkpoint.exists():

return None
try:

state = json.loads(

checkpoint.read_text(

encoding="utf-8"

)

)

except (

OSError,

json.JSONDecodeError,

):

return None
if not isinstance(state, dict):

return None
if (

state.get("schema_version")

!= SCHEMA_VERSION

):

return None
if (

state.get("workspace")

!= workspace_fingerprint(root)

):

return None
artifacts = state.get(

"artifacts",

{},

)
if not isinstance(artifacts, dict):

return None
try:

current_artifacts = artifact_hashes(

list(artifacts),

root,

)

except (

OSError,

ValueError,

):

return None
if current_artifacts != artifacts:

return None
return state
# Example: targeted tests just passed.

root = repo_root()
state = {

"schema_version": SCHEMA_VERSION,

"workspace": workspace_fingerprint(root),

"completed_units": [

"middleware updated",

"targeted tests passed",

],

"next_step": "run integration tests",

"artifacts": artifact_hashes(

["test-results/unit.xml"],

root,

),

}
save_checkpoint(

state,

root,

)
# Later, perhaps in another worker:

state = load_checkpoint(root)
if state is None:

raise SystemExit(

"stale or incompatible checkpoint; "

"enter recovery"

)

The important property is not JSON, SHA-256, or Git specifically. Instead, it’s that the runtime re-establishes the assumptions behind previously verified work before it trusts that work again.

If the workspace changes after targeted tests pass, the test result becomes stale. If the test report itself changes, the artifact hash no longer matches. If the checkpoint schema changes, old state is treated as incompatible instead of being interpreted optimistically.

Define what the fingerprint does not cover

A repository fingerprint is still not a fingerprint of the entire execution environment.

Ignored files are intentionally absent from the example above. So are environment variables, dependency services, container images, feature flags, databases, remote APIs, credentials, and any other external state that can affect whether previous evidence remains valid.

For each resumable stage, ask yourself this: Which inputs could change the meaning of the evidence I am about to reuse?

Bind those inputs to the checkpoint as well.

A useful distinction is:

A checkpoint is reusable only when the inputs that can invalidate its evidence still match or can be reconciled to a known current state.

Durability, concurrency, and scope

The sample is intentionally local and repository-scoped. It assumes one writer owns a worktree while the fingerprint is calculated. If multiple workers can mutate the same workspace concurrently, isolate worktrees or use locking rather than treating a sequence of filesystem reads as an atomic snapshot.

The sample also keeps agent state outside the worktree so writing a checkpoint cannot invalidate its own fingerprint. It rejects unknown checkpoint schemas, corrupt JSON, changed verification artifacts, and artifact paths that resolve outside the repository.

The temporary checkpoint is written in the same directory as the final file, flushed with fsync, and installed with os.replace. Full power-loss durability can require additional filesystem-specific guarantees. Distributed or business-critical runs should usually persist checkpoints and operation state in a transactional database or durable object store rather than a local JSON file.

6. Attach retry and recovery policy to each stage

A generic retry loop treats every failure as if the operation were safe to repeat. Recovery should depend on side effects and on what the runtime can prove about the previous attempt. Every stage needs a timeout, attempt limit, backoff policy, last verified checkpoint, reconciliation action, and escalation condition.

Follow the rate-limiting task through a failure

The rate-limiting example makes the interaction between these controls concrete.

Assume the agent has already added a failing test, implemented the middleware change, and run the targeted test successfully. The harness records a checkpoint tied to the current workspace and test artifact.

Then the worker dies.

On restart:

1. The harness loads the task contract. The agent does not recreate its scope or permissions from conversation history.
2. The runtime recomputes the workspace fingerprint and verification-artifact hashes.
3. If they still match, the targeted-test stage remains valid and the harness continues to the required test suite.
4. If a file changed while the worker was down, the checkpoint is rejected. The harness routes execution back to the first stage whose evidence may now be stale.
5. If the next action requires a forbidden change such as database_schema/**, policy denies it even if the model argues that the change would make the task easier.
6. If a tool call times out after an external write, the runtime does not tell the model simply to “try again.” It reconciles the operation identity and external state before deciding whether another write is safe.
7. The run finishes only after the required tests pass against the current workspace, the diff remains inside the allowed scope, and every completion gate evaluates successfully.

This is the control loop in practice: model judgment determines the next useful action, while deterministic state and policy determine whether that action can execute, whether previous work can be reused, and whether the job is finished.

Polling deserves the same discipline. Prefer an event, webhook, or durable queue. When polling is unavoidable, persist the pending operation, use bounded backoff with jitter, and schedule the next eligible check without keeping the model loop active.

7. Route models and deterministic stages deliberately

The harness should reserve model calls for decisions that benefit from semantic judgment. Retrieval, normalization, exact calculations, schema validation, and invariant checks generally belong in code. Narrow, familiar classifications can use a smaller model; high-impact decisions with conflicting evidence may require a stronger model or human review.

Keep the contract, policy, state model, trace schema, and completion gates stable when the model changes. A model upgrade should be one versioned component of the harness configuration, not a reason to lose comparability across runs.

Treat model changes and harness changes as different experiments

A stronger model can improve planning, tool selection, or recovery reasoning without changing the guarantees of the harness. A harness change can alter permissions, state transitions, checkpoint semantics, retry behavior, or completion logic even when the model remains fixed.

Evaluate those changes separately when possible.

If you change the model, hold the task contract and runtime controls stable so you can measure the effect of model behavior.

If you change the harness, hold the model and representative task set stable so you can measure the effect of the control change.

When both change simultaneously, attribute the result cautiously. A higher task-success rate does not tell you whether the improvement came from better reasoning, broader authority, weaker completion gates, or genuinely better runtime behavior.

When should you use a pipeline, bounded workflow, or agent harness?

Use an agent loop only where the path itself must adapt to evidence at runtime. If the path and branches can be specified reliably in advance, prefer a deterministic pipeline or bounded workflow. Harness engineering begins once the model is allowed to choose among possible next actions and the runtime must govern that autonomy.

The presence of a model does not make a system an agent.

A practical default

Most production systems are hybrids. Keep operations whose path can be stated and tested in deterministic code. Use model judgment where the evidence is ambiguous, and use an agent loop where the next useful action cannot be enumerated reliably in advance. Harness engineering governs that adaptive portion without turning it back into a predefined workflow.

How do you verify a harness change before shipping?

Treat a harness change like a production-system change, not a prompt edit.

The evaluation should answer two different questions:

1. Did the agent become more effective?
2. Did any runtime guarantee become weaker?

A useful test set combines representative tasks, known production failures, and injected infrastructure failures.

For example:

Run the baseline and candidate harness against the same tasks, task contracts, model configuration, environment, and hard gates wherever possible.

For stochastic tasks, use repeated trials rather than interpreting one run as the result.

Measure at least:

Hard gates should be evaluated separately from average task quality. A candidate that completes more tasks by widening permissions or accepting weaker evidence is not necessarily more reliable.

A practical promotion rule is:

Promote only when:

1. no hard scope, authorization, or completion gate regresses;
2. the target reliability metric improves on the representative set;
3. every motivating production failure passes as a regression case;
4. injected interruption and ambiguity cases recover as designed; and
5. latency and cost remain inside accepted operating thresholds.

When a production failure motivates a harness change, save the trace, contract, relevant external state, and expected gate behavior as a permanent regression case. Over time, the regression dataset should become a record of the ways the system has actually failed, not only the ways the team imagined it might fail.

This page stops at that promotion gate. The canonical agent harness architecture, tracing, and evaluation guide covers instrumentation, evaluation levels, metric design, failure analysis, and the full trace-to-improvement loop.

Production harness engineering checklist

The reliability boundary to remember

The purpose of harness engineering is not to remove autonomy from an agent. It is to decide which parts of the system should never depend on autonomy in the first place.

Let the model interpret ambiguous evidence, explore possible solutions, choose among permitted tools, and adapt its plan.

Use deterministic controls to decide:

• what the task allows;
• which actions may execute;
• which state can still be trusted;
• how uncertain side effects are reconciled;
• what evidence proves completion; and
• whether a harness change is safe to promote.

As models improve, the amount of scaffolding needed to help them reason may shrink. Those runtime guarantees remain useful because they protect the system rather than compensate for a particular model’s limitations.

Harness engineering with Arize

Once the runtime owns contracts, gates, checkpoints, and recovery, telemetry provides the evidence needed to prove those controls actually fired. Instrument tool actions, policy decisions, state transitions, retries, approvals, and final artifacts so false completion, duplicate side effects, stale resumes, and policy failures are diagnosable.

Arize Phoenix provides an open-source workflow for tracing, evaluations, datasets, prompt iteration, and experiments. Arize AX adds managed production workflows for observing agents, running online and offline evals, curating datasets, comparing experiments, monitoring quality, and investigating recurring failure modes. Those capabilities let teams test reliability changes against the same traces and regression cases that exposed the failure.

Use this page to design the execution controls; use the agent harness architecture, tracing, and evaluation guide for the end-to-end observability and evaluation workflow. A practical starting point is to send a representative trace, verify one completion or safety gate from observed behavior, and save the first real failure as a regression case.

Frequently asked questions about harness engineering

What is the difference between an agent harness and harness engineering?

An agent harness is the runtime architecture that manages the model loop, context, capabilities, state, permissions, and execution. Harness engineering is the task-specific work of governing that runtime: contracts, policy, completion gates, durable state, recovery, and routing. You can use a packaged harness and still perform substantial harness engineering.

Does harness engineering replace prompt engineering?

No. Prompts remain useful for task instructions, examples, rubrics, tool descriptions, and model judgment. Harness engineering adds controls that a prompt cannot guarantee, including authorization, schema validation, budgets, durable state, completion evidence, and external verification.

Does MCP make agent tool use secure?

MCP standardizes connections and defines security requirements for tools and resources. Supported HTTP deployments can use MCP authorization, but authorization is optional in the protocol and transport-level access is only one layer. The application still needs least privilege, tenant-aware policy, parameter validation, output handling, approval for high-risk actions, monitoring, and audit records.

How do you make a long-running AI agent reliable?

Break the task into resumable units. At each unit, persist the input version, action, verified output, side effects, artifacts, and next eligible step. On restart, compare the checkpoint with current workspace and external state before continuing. Use operation IDs or keys for writes, and reconcile ambiguous outcomes before retrying.

How do you know when an AI agent is finished?

Define completion gates before execution. Gates should inspect current evidence such as test reports, scoped diffs, external records, operation IDs, approvals, and artifact hashes. The harness reaches the finish state only when all required outcome and safety gates pass against the current version of the work.

How do you test a harness reliability change?

Run the baseline and revised harness against the same representative tasks, task contracts, model configuration, and hard gates. Include every known production failure as a regression case, then inject failures that exercise the harness itself: stale checkpoints, worker termination, malformed tool output, forbidden actions, and ambiguous external writes.

Compare task success alongside false-completion rate, hard-gate violations, stale-resume acceptance, recovery success, duplicate side effects, latency, and cost. Promote the change only when the target behavior improves without weakening the guarantees the harness is responsible for enforcing.

For evaluator design and trace-level analysis, use the harness tracing and evaluation guide.

Related Arize resources

• What is an agent harness? The canonical Arize explanation of harness anatomy and the nine-component model.
• Context management in agent harnesses: A focused guide to memory, files, compaction, tool outputs, and subagent context.
• MCP vs. CLI Skills for agents: The full 500-run benchmark behind the capability-selection findings cited above.
• Agent harness architecture, tracing, and evaluation: The canonical resource for generic agent-harness architecture, tracing, evaluation levels, metrics, and the improvement loop.
• How Hermes implements an open source agent harness architecture: A concrete reference for durable sessions, tool exposure, context compression, and long-running execution.
• Agent evaluation metrics: Metrics for task success, quality, safety, cost, latency, and operating views.