# Evaluations

The checked-in evaluations live in
[`evals/`](https://github.com/FrancescoMrn/hiveloom/blob/main/evals/README.md),
each with its harnesses, scoring code, and committed results. They address
task execution and, in 1.3.0, autonomous improvement.
These are historical measurements on specific code revisions, models, and
datasets, not a fresh benchmark of the 1.3.0 checkout. Harness value is
measured per task and model, not assumed — it can be a rescue, a compliance
floor, or nothing at all.

![Task success by model, raw versus the same model inside a hiveloom harness](/assets/01-task-success.png)

## 1. Does a harness rescue a weak model? (article-extractor)

The
[article-extractor benchmark](https://github.com/FrancescoMrn/hiveloom/blob/main/evals/article-extractor/RESULTS.md)
used hiveloom 0.3.1 on 32 live URLs over three epochs (96 runs per arm).
Repeated epochs are not 96 independent samples. Raw and harness arms
use the same prompt and fetch tool; the difference is the harness scaffolding.

| Model / arm | Task success | Hallucination | Cost per success | p50 latency |
|---|---:|---:|---:|---:|
| Claude Haiku 4.5, raw | 3% | 96% | $0.2212 | 5.8s |
| Claude Haiku 4.5 + hiveloom | **65%** | **11%** | **$0.0186** | 10.3s |
| Claude Sonnet 5, raw baseline | 100% | 0% | $0.0073 | 6.4s |
| Qwen 3 4B, raw / harness | 58% / **69%** | 19% / 16% | local | 3.9s / 6.3s |
| Qwen 3.6 35B, raw / harness | 75% / **84%** | 16% / **0%** | local | 13.5s / 16.4s |
| Gemma 4 12B, raw / harness | **92%** / 90% | 1% / **0%** | local | 26.2s / 9.3s |

The Haiku harness gained 61.5 percentage points over raw Haiku and cut cost per
successful result by 12×. Read the rest of the table before generalising from
that: **Haiku's is the only delta that survives a paired test over the 32 URLs**
(p < 0.0001). The local-model deltas are not statistically established over
the 32 URLs;
that does not establish either no effect or harm. In this experiment, raw
Sonnet achieved higher success and lower cost per success than harnessed
Haiku. Measure the tradeoff on your own task and model.

Hallucination is the more robust signal, because the effect sizes are large
relative to the sample. Every output is checked verbatim against a re-fetch of
the live page:

![Hallucination rate, raw versus harnessed](/assets/02-hallucination.png)

![Cost per successful extraction](/assets/03-cost-per-success.png)

## 2. Does it still earn its place on frontier models? (article-digest)

[article-digest](https://github.com/FrancescoMrn/hiveloom/blob/main/evals/article-digest/RESULTS.md)
runs an output-heavy task — a 120–200 word original summary plus five verbatim
quotes and a verbatim outline — on Claude Opus 5 and Sonnet 5. Both arms go
through hiveloom with the same prompt, tool, guardrails, and loop policy; the
raw arms only drop the validators and `loop.require_verification`, so the
measured delta is **validators plus retry-with-feedback**. There were only
five runs per arm: the 100% versus 80% difference is one run per model, not
established general superiority. The prompt requested 120–200 summary words;
the scoring contract accepted 100–260, with explicit quote/outline tolerances
documented in the results.

| Arm | Success | Hallucinated quotes | Cost per success |
|---|---:|---:|---:|
| Opus 5 + hiveloom | **100%** | 0% | $0.0383 |
| Opus 5, raw | 80% | 0% | $0.0320 |
| Sonnet 5 + hiveloom | **100%** | 0% | $0.0142 |
| Sonnet 5, raw | 80% | 0% | $0.0141 |

Neither model fabricated anything. What the raw arms lost was the *contract*:
one run emitted invalid JSON, another a quote outside the required length.
The checks recovered these observed contract failures. This small sample
cannot quantify the reliability gain on future inputs.

## 3. What do frontier models still get wrong? (page-audit)

[page-audit](https://github.com/FrancescoMrn/hiveloom/blob/main/evals/page-audit/RESULTS.md)
targets what remains: exhaustiveness past a truncated tool view, aggregation,
and date arithmetic. The fetch tool clips its digest, and half the pages have
more headings than the digest shows, so no complete answer is reachable from the
tool alone. The metric that matters is not success but whether a wrong answer
arrives **labelled**.

| Arm | Silently wrong | Flagged (`verify_failed`) |
|---|---:|---:|
| Opus 5 + hiveloom | **0/6** | 1/6 |
| Opus 5, raw | 5/6 | 0/6 |
| Sonnet 5 + hiveloom | **0/6** | 1/6 |
| Sonnet 5, raw | 3/6 | 0/6 |

Raw arms confidently returned truncated heading lists and off-by-one day counts.
The harnessed arms either recovered on retry or exited `verify_failed` — no
wrong audit was labelled successful in these six cases per arm. This
illustrates explicit failure reporting; it is not a guarantee for checks or
inputs outside this benchmark.

## Prompt caching compounds it

Prompt caching is on by default for the `claude` provider. In a live measurement
on a 7k-token harness prompt (Haiku 4.5, two-turn run), the first run wrote the
prefix to cache and every later run inside the cache TTL read it back at a tenth
of the input price — **$0.0019 per warm run vs $0.0100 cold, an 81% reduction**.
A harness runs the same prompt shape every time, which is exactly the workload
prompt caching rewards.

![Prompt caching, cold versus warm run cost](/assets/04-prompt-caching.png)

Sample sizes, scoring code, and caveats are in
[`evals/README.md`](https://github.com/FrancescoMrn/hiveloom/blob/main/evals/README.md).


## Autonomous improvement: research-bakeoff

The [research-bakeoff results](https://github.com/FrancescoMrn/hiveloom/blob/main/evals/research-bakeoff/RESULTS.md)
compare research with `evolve --experiment` using the same live director model
and scripted executors. Each run used four planted harnesses, two arms, and
three repeats; fixes were scored on 40 fresh cases unseen by either arm.

| Comparison | Research defects fixed | `evolve --experiment` | Research / evolve spend |
|---|---:|---:|---:|
| First case set | 19/24 | 17/24 | $0.42 / $0.37 |
| Fresh cases after stacking changes | 21/24 | 20/24 | $0.51 / $0.37 |

Both comparisons recorded zero false promotions. **Neither met the
pre-registered 1.5× defect-fix target:** ratios were 1.12× and 1.05×. Research
also took longer. The later product criterion—useful unattended improvement
rather than replacing deliberate evolution—was a disclosed post-hoc change;
it does not erase those results.

Later autonomy runs (eight programs each) tested the engine after additional
fixes:

| Executor | Observed outcome | Limits |
|---|---|---|
| Scripted | 15/16 planted defects fixed; budget, stop, no-drift and no-false-keep checks held in 8/8 programs | Deterministic executor; narrow planted defects and a known unreachable decoy |
| Live Ministral 3B | Fresh-case success rose from 0% to 58% on average; all eight programs stayed within their budgets and stopped themselves | The per-step no-false-keep check held in 6/8; one real false keep motivated the `shifted` fix, validated by replay, and another needed an ablation to distinguish a stepping stone |

These runs used a live `gpt-5-mini` director, fresh case sets, and different
engine revisions as fixes were made. They are evidence about those controlled
tasks, not a clean causal estimate of the current engine's general advantage.
The concepts-mode live check also exposed an invented rate table in a drafted
contract: contract approval is a meaningful part of the workflow.

Read [Research programs](/research) to run your own charter. Choose scoring
that matches real acceptance, keep sealed cases independent, and inspect
unmeasured criteria and confirmation strength before promotion.