paniolo scan Overview Docs Research Calibration Compare
Paper → Feature → Stat

The bibliography
behind the scanner

Every headline feature in paniolo scan traces to a published finding. This page is the trace — and, just as deliberately, the limits: where a number is the paper's result and not ours, we say so.

How To Read This Page

Evidence strength, labeled

Not every rule has a peer-reviewed statistic behind it, and pretending otherwise would defeat the point. Each claim below carries one of three labels:

  • Measured — a statistic from the paper's own empirical study. The paper measured the problem; the rule targets it. The paper did not measure our rule's detection rate.
  • Taxonomic — the rule operationalizes a validated category or vocabulary from the paper.
  • Ported / calibrated — engineering practice ported from prior art (AgentLint and peers) or derived from Paniolo's own reference calibration, not from a paper.

One stat we deliberately do not claim: the ~26% task-success improvement reported for failure-targeted feedback (AgentErrorTaxonomy). That result required live trajectory feedback in the paper's environments. An earlier release tagged findings with that vocabulary; the current CLI does not, and it has not reproduced that outcome.

The Papers

Eight studies,
and what they fed

1. Agentic-AI fault taxonomy — arXiv:2603.06847

385 coded faults from 13,602 issues across 40 agentic-AI repos, validated with 145 practitioners (Cronbach's α 0.91). Top root causes: Data Schema Mismatches at the LLM/tool output boundary (28.0%) and Model Interface Instability (11.6%) — together roughly 40% of sampled faults at one seam.

FeedsRulesStrength
LLM-boundary scanning surface llm-output-schema-validated · model-interface-pinned · tool-contract-tests-present Measured problem; heuristic detection (the rules surface likely exposure, they do not find "28% of faults")
28.0% / 11.6% — Agentic-AI fault taxonomy, arXiv 2026, arxiv.org/abs/2603.06847

2. Open Agent Passport — arXiv:2603.20953

A deterministic policy gate held under adversarial pressure: in the paper's CTF, attacker success fell from 74.6% under a permissive policy to 0% under a restrictive one, across 879 attempts — even when the model itself had been socially engineered. Enforcement over prose, demonstrated adversarially.

FeedsRulesStrength
Resource-cap and gate checks; the "deterministic gates beat prose" thesis across the guardrails dimension agent-resource-budget-caps (OAP limits vocabulary) · motivates no-dangerous-auto-approve Measured anchor for the thesis; taxonomic for the rules
74.6% → 0% over 879 attempts — Open Agent Passport CTF, arXiv 2026, arxiv.org/abs/2603.20953

3. HiL-Bench — arXiv:2604.09408

Frontier models drop 53–82 percentage points when task specs go from complete to missing/ambiguous/blocked — and the failure is usually not asking. Epistemic escalation ("I should ask") is a different protocol from authorization escalation ("may I run this"), and most harnesses document neither.

FeedsRulesStrength
The escalation-protocol check no peer scanner has; decision-boundary motivation for vague-rule detection escalation-protocol-discoverability Measured problem; the rule checks the protocol is declared, not that the agent exercises judgment
53–82 pp — HiL-Bench, arXiv 2026, arxiv.org/abs/2604.09408

4. Agentic-AI attack/defense survey — arXiv:2603.11088 (USENIX Security 2026)

A systematization of agent attack vectors (V1–V6) and risks (R1–R7) with defense categories. paniolo-scan's six security-posture rules check that defenses are declared and documented proportionate to the capability the harness grants — tool allowlists (R1/R3), untrusted-input boundaries (R2/R3), memory provenance (R6), high-impact-action confirmation (R6/R7, OWASP LLM06).

FeedsRulesStrength
The security-posture group (guardrails dimension) tool-allowlist-inventory · untrusted-input-action-boundary · memory-write-provenance · high-impact-action-confirmation Taxonomic — conservative posture heuristics, shipped at info/warn pending severity calibration
Risk taxonomy R1–R7 — Attack/defense survey, USENIX Security 2026, arxiv.org/abs/2603.11088

5. AgentErrorTaxonomy / AgentErrorBench — arXiv:2509.25370

A validated taxonomy of runtime agent failures — Memory, Reflection, Planning, Action, System Operation — built from real agent trajectories. An earlier TypeScript release of paniolo-scan tagged each finding with the failure class it was predicted to prevent; the current CLI does not carry those tags, and the taxonomy remains motivation. The paper also reports up to ~26% task-success improvement from failure-targeted feedback in its environments — a result about live trajectory feedback, which we cite as motivation and explicitly do not claim for static scanning.

FeedsSurfaceStrength
Failure-mode tags + exposure report view (earlier release) Not in the current CLI Taxonomic — editorial mapping using the paper's validated vocabulary, reviewable in one source file
Failure-class taxonomy — AgentErrorTaxonomy, arXiv 2025, arxiv.org/abs/2509.25370

6. Reliability science for LLMs — arXiv:2602.16666

Across 14–15 models and 18 months of releases, capability gains produced only modest reliability gains: the same task can succeed or fail run-to-run. Any LLM-judged linter inherits this. An earlier release of paniolo-scan had an opt-in AI review with a K-run majority vote for that reason; the current CLI has no LLM-judged checks, so every finding is deterministic.

FeedsSurfaceStrength
The consistency gate on AI review findings (earlier release) Not in the current CLI Measured motivation; the gate itself is Paniolo engineering
Outcome-consistency findings — Reliability science, arXiv 2026, arxiv.org/abs/2602.16666

7. Codified Context — arXiv:2602.20478

A production study: a ~660-line always-loaded constitution validated over 283 sessions on a 108,256-line codebase, with context infrastructure totaling 24.2% of the repo. These are the anchors behind paniolo-scan's context-budget defaults — and why the default ceiling (250 always-loaded lines) is deliberately tighter for typical, smaller repos.

FeedsRulesStrength
Context budget and always-loaded thresholds adapter-context-budget · always-loaded-budget · sharing analysis framing Measured reference points; calibrated defaults (derivations on the calibration page)
~660 lines / 283 sessions / 24.2% — Codified Context, arXiv 2026, arxiv.org/abs/2602.20478

8. Scanning the Harness — arXiv:2609.07360

The supply-chain study this page's premise predicts: 3,171 public repositories audited for defects in the agent configuration itself — instruction files, skills, commands, hooks, MCP declarations, subagent definitions. 16.0% of setups carried at least one security defect: unpinned MCP servers in 9.8%, broad-looking shell pre-approvals such as Bash(python:*) in 3.1%, and skills that pre-authorize shell use in 3.8%. The defect class is configuration, not model behavior.

FeedsRulesStrength
Existence proof for the scan surface — config files carry an attack surface that ships in repos no-dangerous-auto-approve · tool-allowlist-inventory · hook and MCP posture checks Measured population rate across public repos; not a per-repo verdict
16.0% of 3,171 repos — Scanning the Harness, arXiv 2026, arxiv.org/abs/2609.07360

The vendor's own guidance audit — claude.dev

Anthropic's context-engineering post for the 5-generation models discloses the experiment this page's checks predict: they deleted over 80% of Claude Code's system prompt and measured no loss on their coding evals. The failure mode they describe is a static defect, not a behavior — system prompt, skills, and CLAUDE.md contradicting each other in the same assembled context ("leave documentation as appropriate" vs "DO NOT add comments"), with the model paying resolution cost on every request. Their remediation ships as a command: claude doctor rightsizes skills and instruction files automatically.

Two check classes follow directly: cross-layer contradiction detection (rule files disagreeing with each other is lintable, and the academic work measured the cost — ~35% of jointly-governed trials satisfy colliding rules) and guidance-weight rightsizing (always-loaded text is a hypothesis that must earn its tokens against evals, not a permanent fixture). The 80% figure is one vendor's eval suite on one product — deletion-safety is model-dependent, which is exactly why it's a check rather than a reflex.

Shihipar — claude.dev Engineering, Jul 2026, claude.dev/blog/the-new-rules-of-context-engineering

And the company thesis: AHE — ICLR 2026

Paniolo itself is grounded in Agentic Harness Engineering (Lin et al.): harness design materially shifts task completion with the base model held fixed. Ten iterations lifted pass@1 on Terminal-Bench 2 from 69.7% to 77.0%. The frozen harness transferred to SWE-bench-verified using 12% fewer tokens than the seed. Those are the paper's results. The scanner is that thesis in diagnostic form: if harness quality moves outcomes, harness quality deserves a score. The full trace is on harness research. The short summary stays on the main site.

Credit Where Due

Ported practice, named

Several paniolo-scan capabilities are engineering ports from prior art rather than paper findings, and the right citation is the tool, not a study:

  • Continuous scoring curves and not-run-isn't-zero discipline — scoring-method design ported from AgentLint's scorer (MIT), extended with scoreScope.
  • Security-hygiene checks (auto-approve, hook exfiltration, SHA pinning, pull_request_target, secret scanning) — detection patterns derived from AgentLint's check suite, reimplemented in paniolo-scan's deterministic rule model.

This is also why the comparison page reads the way it does — these tools earned the credit.

See what the research finds in your repo.

npx @paniolo/cli scan — free, diagnostic-only

Quick Start