The bibliography
behind the scanner
Every headline feature in paniolo scan traces to a published finding. This page is the trace — and, just as deliberately, the limits: where a number is the paper's result and not ours, we say so.
Evidence strength, labeled
Not every rule has a peer-reviewed statistic behind it, and pretending otherwise would defeat the point. Each claim below carries one of three labels:
- Measured — a statistic from the paper's own empirical study. The paper measured the problem; the rule targets it. The paper did not measure our rule's detection rate.
- Taxonomic — the rule operationalizes a validated category or vocabulary from the paper.
- Ported / calibrated — engineering practice ported from prior art (AgentLint and peers) or derived from Paniolo's own reference calibration, not from a paper.
One stat we deliberately do not claim: the ~26% task-success improvement reported for failure-targeted feedback (AgentErrorTaxonomy). That result required live trajectory feedback in the paper's environments. An earlier release tagged findings with that vocabulary; the current CLI does not, and it has not reproduced that outcome.
Eight studies,
and what they fed
1. Agentic-AI fault taxonomy — arXiv:2603.06847
385 coded faults from 13,602 issues across 40 agentic-AI repos, validated with 145 practitioners (Cronbach's α 0.91). Top root causes: Data Schema Mismatches at the LLM/tool output boundary (28.0%) and Model Interface Instability (11.6%) — together roughly 40% of sampled faults at one seam.
| Feeds | Rules | Strength |
|---|---|---|
| LLM-boundary scanning surface | llm-output-schema-validated · model-interface-pinned · tool-contract-tests-present |
Measured problem; heuristic detection (the rules surface likely exposure, they do not find "28% of faults") |
2. Open Agent Passport — arXiv:2603.20953
A deterministic policy gate held under adversarial pressure: in the paper's CTF, attacker success fell from 74.6% under a permissive policy to 0% under a restrictive one, across 879 attempts — even when the model itself had been socially engineered. Enforcement over prose, demonstrated adversarially.
| Feeds | Rules | Strength |
|---|---|---|
| Resource-cap and gate checks; the "deterministic gates beat prose" thesis across the guardrails dimension | agent-resource-budget-caps (OAP limits vocabulary) · motivates no-dangerous-auto-approve |
Measured anchor for the thesis; taxonomic for the rules |
3. HiL-Bench — arXiv:2604.09408
Frontier models drop 53–82 percentage points when task specs go from complete to missing/ambiguous/blocked — and the failure is usually not asking. Epistemic escalation ("I should ask") is a different protocol from authorization escalation ("may I run this"), and most harnesses document neither.
| Feeds | Rules | Strength |
|---|---|---|
| The escalation-protocol check no peer scanner has; decision-boundary motivation for vague-rule detection | escalation-protocol-discoverability |
Measured problem; the rule checks the protocol is declared, not that the agent exercises judgment |
4. Agentic-AI attack/defense survey — arXiv:2603.11088 (USENIX Security 2026)
A systematization of agent attack vectors (V1–V6) and risks (R1–R7) with defense categories. paniolo-scan's six security-posture rules check that defenses are declared and documented proportionate to the capability the harness grants — tool allowlists (R1/R3), untrusted-input boundaries (R2/R3), memory provenance (R6), high-impact-action confirmation (R6/R7, OWASP LLM06).
| Feeds | Rules | Strength |
|---|---|---|
| The security-posture group (guardrails dimension) | tool-allowlist-inventory · untrusted-input-action-boundary · memory-write-provenance · high-impact-action-confirmation |
Taxonomic — conservative posture heuristics, shipped at info/warn pending severity calibration |
5. AgentErrorTaxonomy / AgentErrorBench — arXiv:2509.25370
A validated taxonomy of runtime agent failures — Memory, Reflection, Planning, Action, System Operation — built from real agent trajectories. An earlier TypeScript release of paniolo-scan tagged each finding with the failure class it was predicted to prevent; the current CLI does not carry those tags, and the taxonomy remains motivation. The paper also reports up to ~26% task-success improvement from failure-targeted feedback in its environments — a result about live trajectory feedback, which we cite as motivation and explicitly do not claim for static scanning.
| Feeds | Surface | Strength |
|---|---|---|
| Failure-mode tags + exposure report view (earlier release) | Not in the current CLI | Taxonomic — editorial mapping using the paper's validated vocabulary, reviewable in one source file |
6. Reliability science for LLMs — arXiv:2602.16666
Across 14–15 models and 18 months of releases, capability gains produced only modest reliability gains: the same task can succeed or fail run-to-run. Any LLM-judged linter inherits this. An earlier release of paniolo-scan had an opt-in AI review with a K-run majority vote for that reason; the current CLI has no LLM-judged checks, so every finding is deterministic.
| Feeds | Surface | Strength |
|---|---|---|
| The consistency gate on AI review findings (earlier release) | Not in the current CLI | Measured motivation; the gate itself is Paniolo engineering |
7. Codified Context — arXiv:2602.20478
A production study: a ~660-line always-loaded constitution validated over 283 sessions on a 108,256-line codebase, with context infrastructure totaling 24.2% of the repo. These are the anchors behind paniolo-scan's context-budget defaults — and why the default ceiling (250 always-loaded lines) is deliberately tighter for typical, smaller repos.
| Feeds | Rules | Strength |
|---|---|---|
| Context budget and always-loaded thresholds | adapter-context-budget · always-loaded-budget · sharing analysis framing |
Measured reference points; calibrated defaults (derivations on the calibration page) |
8. Scanning the Harness — arXiv:2609.07360
The supply-chain study this page's premise predicts: 3,171 public repositories audited for defects in the agent configuration itself — instruction files, skills, commands, hooks, MCP declarations, subagent definitions. 16.0% of setups carried at least one security defect: unpinned MCP servers in 9.8%, broad-looking shell pre-approvals such as Bash(python:*) in 3.1%, and skills that pre-authorize shell use in 3.8%. The defect class is configuration, not model behavior.
| Feeds | Rules | Strength |
|---|---|---|
| Existence proof for the scan surface — config files carry an attack surface that ships in repos | no-dangerous-auto-approve · tool-allowlist-inventory · hook and MCP posture checks |
Measured population rate across public repos; not a per-repo verdict |
The vendor's own guidance audit — claude.dev
Anthropic's context-engineering post for the 5-generation models discloses the experiment this page's checks predict: they deleted over 80% of Claude Code's system prompt and measured no loss on their coding evals. The failure mode they describe is a static defect, not a behavior — system prompt, skills, and CLAUDE.md contradicting each other in the same assembled context ("leave documentation as appropriate" vs "DO NOT add comments"), with the model paying resolution cost on every request. Their remediation ships as a command: claude doctor rightsizes skills and instruction files automatically.
Two check classes follow directly: cross-layer contradiction detection (rule files disagreeing with each other is lintable, and the academic work measured the cost — ~35% of jointly-governed trials satisfy colliding rules) and guidance-weight rightsizing (always-loaded text is a hypothesis that must earn its tokens against evals, not a permanent fixture). The 80% figure is one vendor's eval suite on one product — deletion-safety is model-dependent, which is exactly why it's a check rather than a reflex.
Shihipar — claude.dev Engineering, Jul 2026, claude.dev/blog/the-new-rules-of-context-engineeringAnd the company thesis: AHE — ICLR 2026
Paniolo itself is grounded in Agentic Harness Engineering (Lin et al.): harness design materially shifts task completion with the base model held fixed. Ten iterations lifted pass@1 on Terminal-Bench 2 from 69.7% to 77.0%. The frozen harness transferred to SWE-bench-verified using 12% fewer tokens than the seed. Those are the paper's results. The scanner is that thesis in diagnostic form: if harness quality moves outcomes, harness quality deserves a score. The full trace is on harness research. The short summary stays on the main site.
Ported practice, named
Several paniolo-scan capabilities are engineering ports from prior art rather than paper findings, and the right citation is the tool, not a study:
- Continuous scoring curves and not-run-isn't-zero discipline — scoring-method design ported from AgentLint's scorer (MIT), extended with
scoreScope. - Security-hygiene checks (auto-approve, hook exfiltration, SHA pinning,
pull_request_target, secret scanning) — detection patterns derived from AgentLint's check suite, reimplemented in paniolo-scan's deterministic rule model.
This is also why the comparison page reads the way it does — these tools earned the credit.
See what the research finds in your repo.
npx @paniolo/cli scan — free, diagnostic-only