The bibliography
behind the harness
Agentic Harness Engineering is the paper. OpenAI and Anthropic wrote the practitioner accounts beside it. This page says what each one argues. The short summary stays on the homepage. How paniolo evolve runs the loop is on evolve research.
Their experiment, not ours
The percentages below are Lin and colleagues' results, on their benchmarks, with their seed harness and their evolution loop. Paniolo has not reproduced them. The OpenAI and Anthropic posts are engineering accounts of how those teams build, not effect sizes. The CLI loop — diagnose, judge, commit or roll back — is on evolve research, including the campaign and the trace debugger this product does not ship.
Hold the model fixed,
edit the harness
Agentic Harness Engineering — Lin et al.
AHE treats the harness as the thing you can edit while the base model stays frozen: system prompt, tool descriptions and implementations, middleware, skills, sub-agent configuration, and long-term memory. Each of those is a file, so a change is a diff you can revert. The loop has three observability pillars. Component observability makes that file set the action space. Experience observability distills raw trajectories into a layered evidence corpus an editor can read, instead of asking it to swallow millions of tokens of logs. Decision observability pairs every edit with a prediction, then checks that prediction against the next round's task outcomes. An edit that fails the check is reverted. The paper's own limit is worth keeping: the loop attributes fixes more reliably than regressions, and stacking edits that each looked good does not add up.
On Terminal-Bench 2, ten iterations raised pass@1 from 69.7% to 77.0%, past the human-designed Codex harness at 71.9% and past the self-evolving baselines they compare (ACE and Training-Free GRPO). The frozen harness, without another evolution pass, transferred to SWE-bench-verified at the paper's highest aggregate success and 12% fewer tokens than the seed. On Terminal-Bench 2 it also gained +5.1 to +10.1 percentage points across three other model families. An ablation put the gain in tools, middleware, and long-term memory. The system prompt alone regressed. The quote on the homepage — tools, middleware, and memory carry the gain — is that ablation, not a Paniolo measurement.
paniolo scan reads the harness as files and scores it. It never writes, which is the diagnostic half of "if the harness moves outcomes, the harness deserves a score." The trace from scan's rules back to other papers is on scan research. The evolution loop that would edit those files — diagnose, a hypothesis, validation, commit or rollback — is paniolo evolve.
The ablation is why the rest of the product is split the way it is. Always-loaded prose stays short, and paniolo scan budgets it, because the paper found the system prompt was the weak component. Skills are a catalog loaded by name. The wiki is the long-term memory: markdown in the repo, linted, and searched with paniolo qmd rather than pasted into the prompt. Those are the layers the ablation said were carrying the result. They are not a claim that a Paniolo harness has been shown to move pass@1.
What the labs wrote
beside the paper
Harness engineering — OpenAI
OpenAI's account of building with Codex is a constraint story, not a benchmark table. The team stopped treating code-writing as the job and started treating the environment as the job: intent specified in the repo, feedback loops the agent can run, and knowledge laid out where an agent can find it. Design docs, execution plans, and a quality grade over domains live in a docs tree. AGENTS.md is a map, not the knowledge base. Custom linters enforce structural rules, and the lint message itself tells the agent how to fix the finding. When human review became the bottleneck, they made the UI, the logs, and the metrics legible to the agent instead of asking people to read more output.
That is the same split Paniolo ships, without their repository. paniolo scan is the linter over the harness, and the finding is the remediation hint. The wiki holds the design notes and decisions so they are not stuffed into AGENTS.md. paniolo stale is the feedback loop on those pages: a code change nominates a claim that may have gone false, and a person or an agent adjudicates it. The post does not measure a scanner. It is why the scanner's job is to make the environment legible, and why a finding that cannot name a fix is an incomplete finding.
Effective harnesses for long-running agents — Anthropic
Anthropic's harness for work that outlasts one context window is two roles and a clean handoff. An initializer session lays down the environment: a startup script, a progress file, and an initial git commit. Later sessions do one increment, then leave the tree in a state the next session can resume — a commit with a real message, and a progress note. The failure they are designing against is an agent that tries to finish the whole task in one window, or that leaves a mess the next window cannot see. Git is the memory that survives compaction.
Paniolo's counterpart is the files between sessions, not a pair of agent prompts. The wiki log and the page history are the progress note. paniolo stale records which claims a commit put in doubt, so the next session inherits an allegation instead of a silent drift. Returning the harness component set to a known snapshot is evolve rollback. We do not ship their initializer agent or their one-feature-per-session coding agent. The lesson we take is narrower: a session that cannot leave a file and a commit has not finished, even if the model thinks it has.
NOOA — NVIDIA
NVIDIA's OO Agents paper collapses an agent into one Python object: methods are the actions, fields are the state, docstrings are the prompts, and type annotations are the contracts. A method whose body is ... is completed at runtime by the agent loop; ordinary bodies stay deterministic. Their claim is that agent behavior should be tested, traced, and refactored like software — and that the harness itself should expose model-callable APIs for context and events.
Paniolo's MCP server is that idea already shipped: qmd, stale, and job submission are harness internals exposed as calls a model can make, not files it must guess at. The piece not taken is the object model — Paniolo's agents stay vendor tooling; what gets the "software, not incantation" treatment here is the harness's own artifacts: skills, rules, and config with a linter over them. Docstrings-as-prompts has a Markdown cousin: a skill's frontmatter is the prompt surface, and the scan gates are its type checker.
WildClawBench
WildClawBench runs agents inside the harnesses people actually deploy — OpenClaw, Claude Code, Codex, Hermes — on 60 human-authored long-horizon tasks with real tools rather than mock APIs. Grading is hybrid: deterministic checks, environment-state auditing of side effects, and an LLM/VLM judge. Across 19 frontier models the best reaches 62.2% — and the number that matters here: switching the harness alone moves a single model by up to 18 points.
Eighteen points is larger than most model-generation deltas on the same suite, which makes it the strongest public measurement that the wrapper is the product, not the model. Paniolo's harness is not one of the four benchmarked, so we claim no score on their suite. The take is the methodology: the honest way to measure a harness is same model, real runtime, real tools — the comparison shape this page already uses.
WildClawBench — arXiv 2026, arxiv.org/abs/2605.10912How the Claude Code team does code review — claude.dev
Claude writes ~90% of Claude Code's code, so the team rebuilt review around the volume: every change runs a ladder of automated layers — verified output before the PR exists, agent review on every PR with findings voted on by a second agent pool, a security pass that blocks, deterministic CI — and humans enter scaled by risk tier of the change, not the author. Two rules carry the trust model: an agent's approval never counts as a person's, and an approval covers exactly one commit. Their eval set is their incident history, re-scored daily; and after one agent↔reviewer pair ran 50+ rounds overnight, they cap review rounds at three and re-review only the new diff.
That is the gate philosophy this page is about, operated by a lab at full scale: deterministic floor first, probabilistic review fenced behind it, human attention rationed by risk — and the honest admission that their approval tiers are still manually tuned (no circuit breaker tightens them when reverts move). Paniolo's check suite and JUDGE gate are the small version of the same architecture; their doubled-merge / −40% revert figures are their telemetry, not something this product measured.
Osmani, Lunt, Wang — claude.dev Engineering, Oct 2026, claude.dev/blog/solving-code-review-at-anthropicHow we made claude.ai 3x faster — claude.dev
A two-week sprint where Claude (in Slack threads, ~150 running at once) proposed, validated, and shipped more than 3,000 performance changes with no customer-facing incident. The governance is the story: every new benchmark had to prove it correlated with wall-clock time before Claude was allowed to hillclimb it; proven benchmarks became CI ratchets that could only tighten; every PR passed automated review plus at least one human approval; anything user-visible rode behind a flag that was classified, ramped, and retired on schedule. Human engineers kept three jobs: ambition ("be braver"), taste (named owners ruled on perceptible changes), and direction (thread scope, sequencing, and when to stop).
That is the gate philosophy applied to throughput rather than review: validate the evaluator before trusting it, turn wins into non-regression contracts, and keep human attention on the three things that do not scale — ambition, taste, and the stop call. Paniolo's evolve gate shares the ratchet instinct: a candidate is a hypothesis until validation says otherwise. Their 3x and zero-incident figures are their telemetry on a domain with unusually clean ground truth (latency), not a rate any harness should promise.
Wang, Attard, G. — claude.dev Engineering, Sep 2026, claude.dev/blog/how-we-made-claude-ai-fasterDynamic concurrency — parallel agents are a cost decision
A field study of 2,124 executions across Codex, Claude Code, and Kimi — over $20K of measured runs — prices when running agents in parallel pays: concurrency helps long-horizon tasks and hurts bounded ones, token spend multiplies by 1.41–3.31×, and failures split into coordination, merge, and task-level classes. SquidAgent derives the decision rule — parallelize only when critical-path savings beat re-exploration plus alignment costs, measured in predicted output tokens because models estimate those more reliably than wall-clock duration.
The take is that "spawn more agents" is a budget decision with a measurable crossover, not a free speedup — and a harness should record the concurrency choice and its cost rather than leave it as a prompt habit. Paniolo does not orchestrate multi-agent runs; the multipliers and the taxonomy are their measured results.
Dynamic concurrency — arXiv 2026, arxiv.org/abs/2610.10263 · SquidAgent — arxiv.org/abs/2610.08647