A practical study of AI coding agent reliability—from a convincing model report to a result that can actually be accepted into a repository.
Benchmark · methodology · test bank · field research
The completion gap
An agent may confidently report success while changing no files, missing a requirement, or breaking an adjacent scenario.
Task status must therefore follow facts on disk: the diff, tests, scope compliance, and independent review.
Signals from the field
“Don’t prompt Claude. Build a system that prompts itself.” Attention is moving from a single chat to agent systems.
VOLYWe test conditions where specialization and independent verification produce a measurable gain—not whether multi-agent is popular.
Claude Code users discuss usage limits and manual workarounds: multiple accounts, proxy CLIs, and alternative harnesses.
VOLYThis supports the Job: finish a task without quota failure, with transparent fallback and total result cost.
Native Managed Agents already introduce session budgets; basic orchestration is rapidly becoming a platform capability.
VOLYVOLY differentiates on cross-agent task budgets across tools and fallback boundaries—not roles alone.
Large multi-agent systems increase the need for observable environments, bounded roles, and verifiable episodes.
VOLYCollect trustworthy episodes and role metrics first; defer self-play until the evidence is reliable.
Metrics are snapshots of public activity when checked. They provide market context, not VOLY research results.
04 measures →
The core economic unit is neither run cost nor generated code volume. It is the cost of a verified completed task.
accepted tasks / all tasksreported done, but rejectedtotal cost / accepted tasksruns with unrequested changesResearch hypotheses
A multi-agent workflow is not better by definition. It must demonstrate an advantage in quality, result cost, or error recovery.
Reported success is systematically higher than the result verified by tests, diff, and code review.
Architect, developer, and reviewer roles must prove an advantage over one general-purpose agent.
Plan gates, protected paths, and file limits are tested as controls against unrequested changes.
Architecture ≠ outcome
Each architecture receives the same starting commit, task brief, time budget, and pre-frozen acceptance criteria.
30 reproducible tasks
Six dimensions with five tasks each. Every task has a reference commit, an allowed scope, and a machine-checkable acceptance pack.
API, idempotency, concurrency, DB pool
state, accessibility, forms, responsive UI
regression, flaky tests, properties, queues
Docker, healthcheck, CI cache, rollback
traversal, secrets, commands, protected paths
DB → API → UI, flags, timezones, audit
One task up close
We do not ask readers to trust an agent report. Every task follows the same chain of physical evidence.
Make POST /payments idempotent when the same key is submitted twice.
src/payments/** and tests/payments/**. Migrations and configuration are prohibited.
One charge per key; correct concurrent response; existing tests pass.
pytest tests/payments -q · ruff check · git diff --check
Already enforced in VOLY
The study is still in progress, but its core trust boundaries already exist in the product and are covered by tests.
Process completion is not acceptance. A result reaches verified_success only after its deterministic evaluation policy passes.
voly/evaluation · tests/test_evaluation.pyA code-generation role that reports success without reported or detected changes is marked failed.
voly/a2a/hybrid.py · tests/test_hybrid_a2a.pyThe independent judge sees the task, acceptance criteria, and diff, but receives read-only tools with no shell or file writes.
voly/a2a/agentic_judge.py · tests/test_agentic_judge.pyA capability needs six measured outcomes including two held-out outcomes; a routing probe alone is not validation.
voly/capability · tests/test_capability_production_validation.pyReproducible mock benchmark
26.3%fixture savingsA $0.137 baseline versus $0.101 for the fallback chain. Savings occur only on four billing-fallback tasks; happy paths receive no artificial gain. These are deterministic mock-cost fixtures, not production invoices.
python benchmarks/finops-suite/run.py --mode mockFocused evidence, evaluation, judge, FinOps, and production-gate check: 57 tests passed on August 13, 2026. This is not the project’s full test suite.
North star metric
Not the price of an agent response, but the price of a result that passed the checks and can be accepted by the team.
1–2 min survey · 30 min interview
Which AI coding tools does the team use, how often, and for which kinds of tasks?
What counts as done: an agent response, diff, tests, review, merge, or a production metric?
How often does the agent say done, change too much, or create a regression?
Can the team see task cost, and what happens when a quota runs out?
What evidence is required to accept a result without fully repeating the work by hand?
Editorial honesty
In our experiment, N of M tasks passed the acceptance criteria.
On the selected test bank, the role workflow showed…
According to participant self-reports…
Most companies… based on a non-representative sample.
Multi-agent is always better and cheaper.
Present a roadmap or outside figure as a VOLY result.
30-day launch plan
Check implemented / partial / roadmap and available telemetry.
10 tasks × 3 modes; close methodology gaps.
30+ surveys, 5+ interviews, and a wider test bank.
30 tasks; blind review; reproducible analysis.
HTML report, charts, partner wave, and diagnostics.
Open research · 1–2 minutes
We are not asking for a general opinion about AI. Start with the latest task where you spent real time, budget, or team attention.Participants receive the final benchmark first and a short verification-process diagnostic. A public profile is optional.