Evidence before confidence

How we decide whether
a task is truly complete

How VOLY separates an agent’s completion claim from an accepted engineering result.

Take the survey

04 measures →

What counts as real success?

The core economic unit is neither run cost nor generated code volume. It is the cost of a verified completed task.

01

Verified Success Rate

accepted tasks / all tasks
02

False Completion Rate

reported done, but rejected
03

Cost per Verified Task

total cost / accepted tasks
04

Scope Drift

runs with unrequested changes

Research hypotheses

We don’t take architecture at its word

A multi-agent workflow is not better by definition. It must demonstrate an advantage in quality, result cost, or error recovery.

H1

Agent said done ≠ task completed

Reported success is systematically higher than the result verified by tests, diff, and code review.

H2

Roles can reduce false completion

Architect, developer, and reviewer roles must prove an advantage over one general-purpose agent.

H3

Gates can contain scope drift

Plan gates, protected paths, and file limits are tested as controls against unrequested changes.

Architecture ≠ outcome

One task.
Four modes.

Each architecture receives the same starting commit, task brief, time budget, and pre-frozen acceptance criteria.

ModeMechanicWhyStatus
Single agentOne agent does everythingFaster and cheaper for simple tasksBaseline
Supervisor–WorkerAn orchestrator assigns specialistsRole and handoff controlPrimary
Debate / CritiqueAgents challenge the resultComplex decisions and reviewSelective
SwarmMany simple agents in parallelHigh-volume analysisFuture
10 steps

From clean commit
to verified result

  1. RestoreRestore the reference commit and a clean working tree.
  2. FreezeFreeze scope, acceptance criteria, and abort rules.
  3. RunRun each mode with the same time and attempt budget.
  4. CaptureCapture commands, tool calls, tokens, cost, and fallback.
  5. DiffRecord changed files and the diff hash.
  6. TestRun tests, linting, builds, and security checks.
  7. ScopeCheck protected paths and unrequested changes.
  8. ReviewConduct a blind human review of the result.
  9. SeparateStore reported success separately from acceptance.
  10. ResetReset the environment before the next executor.

1–2 min survey · 30 min interview

What happens in real teams?

01Usage

Which AI coding tools does the team use, how often, and for which kinds of tasks?

02Completion

What counts as done: an agent response, diff, tests, review, merge, or a production metric?

03Errors

How often does the agent say done, change too much, or create a regression?

04Economics

Can the team see task cost, and what happens when a quota runs out?

05Trust

What evidence is required to accept a result without fully repeating the work by hand?

Editorial honesty

What we can say.
And what we cannot yet.

We can

In our experiment, N of M tasks passed the acceptance criteria.

On the selected test bank, the role workflow showed…

According to participant self-reports…

Not without data

Most companies… based on a non-representative sample.

Multi-agent is always better and cheaper.

Present a roadmap or outside figure as a VOLY result.