Backend
B01–B05API, idempotency, concurrency, DB pool
30+ reproducible tasks
A test bank,
not a lucky demo
Thirty reproducible engineering tasks with fixed scope, verification commands, and independent acceptance.
Share a real case30 reproducible tasks
A test bank, not a lucky demo
Six dimensions with five tasks each. Every task has a reference commit, an allowed scope, and a machine-checkable acceptance pack.
Frontend
F01–F05state, accessibility, forms, responsive UI
Testing
T01–T05regression, flaky tests, properties, queues
DevOps
D01–D05Docker, healthcheck, CI cache, rollback
Security
S01–S05traversal, secrets, commands, protected paths
Cross-stack
X01–X05DB → API → UI, flags, timezones, audit
One task up close
What a verifiable result looks like
We do not ask readers to trust an agent report. Every task follows the same chain of physical evidence.
01Agent reportCompletion claim
02Git diffActual changes
03TestsMachine checks
04ScopeAllowed files only
05ReviewIndependent assessment
06AcceptedVerified result
View an acceptance pack example +
Make POST /payments idempotent when the same key is submitted twice.
src/payments/** and tests/payments/**. Migrations and configuration are prohibited.
One charge per key; correct concurrent response; existing tests pass.
pytest tests/payments -q · ruff check · git diff --check
Already enforced in VOLY
Mechanisms, not promises
The study is still in progress, but its core trust boundaries already exist in the product and are covered by tests.
Verified success is a distinct state
Process completion is not acceptance. A result reaches verified_success only after its deterministic evaluation policy passes.
voly/evaluation · tests/test_evaluation.pyNo diff, no code success
A code-generation role that reports success without reported or detected changes is marked failed.
voly/a2a/hybrid.py · tests/test_hybrid_a2a.pyThe judge cannot fix its own answer
The independent judge sees the task, acceptance criteria, and diff, but receives read-only tools with no shell or file writes.
voly/a2a/agentic_judge.py · tests/test_agentic_judge.pyEvidence activates capabilities
A capability needs six measured outcomes including two held-out outcomes; a routing probe alone is not validation.
voly/capability · tests/test_capability_production_validation.pyReproducible mock benchmark
26.3%fixture savings8 fixed tasks · 4 executor labels
A $0.137 baseline versus $0.101 for the fallback chain. Savings occur only on four billing-fallback tasks; happy paths receive no artificial gain. These are deterministic mock-cost fixtures, not production invoices.
python benchmarks/finops-suite/run.py --mode mockFocused evidence, evaluation, judge, FinOps, and production-gate check: 57 tests passed on August 13, 2026. This is not the project’s full test suite.
North star metric
Cost per Verified Task=
total cost of all attempts
accepted tasks
accepted tasks
Not the price of an agent response, but the price of a result that passed the checks and can be accepted by the team.