Runtime evaluations

Runtime evaluations use evidence from workflow runs to assess the quality of work and the cost and time required to produce it. Repeated runs provide evidence for improving workflows and comparing task configurations.

Evidence from execution

Atomic workflows give each task a defined output and acceptance criteria. Evaluating those outputs produces evidence tied to specific work, instructions and configurations.

Dimension	Evidence
Quality	Evaluations, findings and the revisions needed to meet acceptance criteria.
Cost	Execution costs and token usage.
Time	Execution duration and repeated work.

An accepted result alone does not explain how well a workflow performed. The revisions needed to reach that result also matter: two runs can produce acceptable work with different costs and completion times.

Comparable results

Versioned workflows and instructions make it possible to identify what changed between runs. Comparing harness, model and effort configurations against the same workflow version helps separate configuration choices from changes to the workflow itself.

The work being performed still matters. Comparisons need to account for differences in inputs and task difficulty, and use evidence from repeated runs rather than treating one successful result as conclusive.

Evidence for improvement

Runtime evaluations can identify tasks that need frequent revision, expensive configurations or checks that produce unhelpful findings. That evidence informs changes to instructions, context, evaluations and routing.

People establish the priorities and acceptable trade-offs between quality, cost and time. Self-Improvement explains how proposed changes are assessed and authorised; the Workflow Engine provides the definitions, task configurations and runs being evaluated.