Agent Field Test

Layers under testAGENTS · EDGE

The test

Give an agent a real task with a real deadline and a real definition of done, then judge it on the finished work rather than on the transcript.

Task set

The first field test will use 12 bounded tasks across three surfaces: a local code repository, a structured research packet, and a small operations workflow. Every task will have a written starting state, allowed tools, time limit, acceptance tests, prohibited actions, and recovery procedure.

Tasks must be useful enough that a person would otherwise perform them, but safe enough to run in an isolated environment. No agent receives production credentials, permission to deploy, or authority to contact another person.

Each task is run three times from the same clean snapshot. Agent versions, models, system instructions, tool definitions, and inference settings are pinned for the test. When systems support different tool interfaces, the capability available to each agent is documented rather than described as equivalent by default.

What gets measured

• Task completion against a specification written before the run.

• Whether the agent verified its own output, and how it detected failure.

• The quality of the record it left behind for a human to audit.

• Where a human had to intervene, and why.

Scoring

A run is complete only when the independent acceptance suite passes and the requested artifact exists in the target system. The agent’s own completion message does not count as evidence.

Scores will report:

• complete, partially complete, failed, and invalid runs;
• acceptance checks passed and regressions introduced;
• elapsed time, model and tool usage, and estimated cost;
• human interventions, approval requests, retries, and recovery actions;
• unsupported claims in the final report; and
• unauthorized or out-of-scope changes, which invalidate the run.

Evidence and privacy

The retained record includes the task specification, starting-state hash, filesystem or API diff, test output, tool-event timeline, final artifact, and human intervention log. Reasoning text is not required to judge completion.

Traces can contain sensitive model and tool inputs. Published traces will therefore use synthetic or cleared data, remove credentials and personal information, and preserve enough event metadata to audit the action without publishing private content.

Stop conditions

The run stops on a prohibited action, repeated destructive failure, exhausted time or cost budget, an unavailable dependency, or a human safety decision. A stopped run remains in the denominator. It is not silently retried until it passes.

Status

Protocol v0.1 · planned. The evidence model, repetition rule, and scoring contract are public. The 12 task specifications and agent configurations remain to be published before the first run. No completion-rate claim exists yet.