Signal-driven evolution

Evolution changes a harness because its runs said something. This document covers how hiveloom finds where the runs point, makes every change a prediction about that place, checks the prediction against the runs that follow, and keeps only the changes that held. It also covers the memory features that let a harness learn from runs without growing a prompt nobody reviews.

runs ──► Hive ──► hiveloom signal ──► evolve (aimed proposal) ──► apply
  ▲                  (free)                 │                        │
  │                                         ▼                        ▼
  └──────────── hiveloom assess ◄── runs of the new version ◄── new version hash
                    (free)          (or: evolve --experiment eval.yaml --yes)

Everything on the left and bottom is counting and costs nothing. The only paid step is the proposal itself, and it is aimed before it is written.

harnesses/signal-lab walks the whole loop offline, with no API key: a located tool error, a reflected lesson, a refuted change reverted and a confirmed one kept.

1. Locate: hiveloom signal #

hiveloom signal ./h            # table view
hiveloom signal ./h --json     # the SignalMap evolution receives

Every ingested run is indexed with features: yes/no facts about what it did or met on the way, never how it ended.

Friction that restates the ending (loop_limit, guardrail_halt, output_validation, verifier_failure) is left out on purpose: as features they would "explain" failure by definition. Those endings stay visible as failure clusters and as status:<status> targets.

For one harness version, the signal map then reports:

• Signals. Each feature's failure rate with and without it, a two-sided Fisher exact p-value, and a Benjamini-Hochberg q-value across all features tested. risk means more common in failures. protective means more common in successes: a tool the good runs call and the bad runs skip is found just as readily as an error. Features that mark exactly the same runs are one signal. The most specific name is shown, and the others are listed as aliases. Each signal names its levers (the spec paths that could change it) and whether evolution may pull them (addressable).
• Failure features. What failed runs share, by prevalence. When there is nothing to contrast (every run failed), prevalence is the only evidence left.
• Mechanisms. Friction counted per (category, component): events, runs, failed runs, recovered events. A change aimed at a mechanism is judged by whether its count falls. Small samples can show that long before they can show a success-rate change.
• Loss classes. Each failed run is charged to one of provider, guardrail, limits, process, tooling, content or other. The first four have a harness-level cause. content means the run finished and was wrong, which no plumbing change reaches. The split bounds what any harness change can buy.
• Quality. The success rate with its 95% Wilson interval, the smallest change this many runs could detect, and the runs per version needed to detect ten points. An underpowered population is reported as underpowered, not mined for patterns.
• Verdict.
  • actionable: a signal survives correction (q ≤ 0.1).
  • suggestive: p ≤ 0.1 but it does not survive correction.
  • diffuse: nothing separates failures, so look at content.
  • underpowered: fewer than two failing or passing runs.
  • no_failures and no_runs: as named.

An external failure label (hiveloom outcome <run> failure) counts as a failure. A run that passed its validators and was wrong anyway is exactly what validators cannot see. Runs ingested before feature indexing are counted as unindexed_runs; re-ingesting their traces includes them.

2. Aim: every proposal names a target #

hiveloom evolve puts the signal map into the proposing prompt as its own section, ahead of the raw report. The proposal must carry a target:

"target": {"signal": "tool_error:http_get", "expect": "decrease", "by": 0.4,
           "rationale": "in 7/9 failures vs 1/11 successes"}

• signal is one of the map's targets (a feature or alias, a mechanism, a status:<status>, success_rate) or metric:<objective> for a configured objective.
• expect is the direction. by is optional: the predicted absolute change, as a fraction of runs.

A missing or unknown target is sent back as repair feedback within the existing three-call budget, like malformed JSON. A proposal that states objective_expectations has already made a checkable prediction, so it needs no separate target. The prediction is stored with the evolution record and is what the assessment checks.

3. Assess: hiveloom assess #

hiveloom assess ./h [--min-runs 5] [--json]

For every applied evolution, the old and new version's runs are compared on the targeted signal, with the success rate as a guard:

• Paired when both versions ran the same eval cases (same eval, case and repetition): McNemar's exact test for rates, the sign test for metrics. Discordant pairs settle questions that two noisy rates cannot.
• Unpaired otherwise: Fisher's exact test for rates, Welch's z for metric means.

Verdicts:

An evolution recorded without a prediction (for example, one applied before 1.2.0) is judged on the success rate alone.

Assessments are also evolution's attempt history. Each applied evolution carries its verdict, or the keep/revert decision an experiment took, so the proposer does not re-propose a refuted change unchanged. Rejected proposals still appear, unmeasured.

4. Experiment: evolve --experiment #

hiveloom evolve ./h --experiment eval.yaml --yes [--rounds 3] \
    [--keep-inconclusive] [--remeasure-baseline]

Each round does the following:

1. Measures the current version on the eval. It does this once, or every round with --remeasure-baseline, so a lucky first draw is not the permanent bar.
2. Locates the signal and asks for one aimed proposal.
3. Applies it. YAML only: code changes are never applied in this mode.
4. Runs the same eval on the new version.
5. Assesses it pair by pair.
6. Keeps a confirmed change and reverts the rest. A revert restores the exact bytes and records the reverse evolution. --keep-inconclusive keeps a change the eval could not decide on.

Decisions are stored on the evolution row, and the next round's attempt history carries them. The eval must run the harness being changed. --yes is required because the loop applies and reverts on its own. Every gate, frozen path, validation and rollback still applies. Each round costs up to three strong-model calls plus one or two eval runs.

Eval size matters. Pairs are what make a small eval decisive: six improved pairs and no worsened ones gives p = 0.03. Six cases that all changed is the smallest sample that can confirm anything at all.

5. Memory that learns and scales #

Relevance-selected memory #

hiveloom set memory.selection relevant --dir ./h
hiveloom set memory.max_selected 8 --dir ./h

With selection: relevant, a run is shown its pinned entries plus the entries that best match its task, up to max_selected. Ranking is tf-idf over title and content, with title matches weighted up, plurals folded, and ties broken by declaration order. It is deterministic: the same task always gets the same section, and the choice is made once per run, so every turn after the first still hits the prompt cache.

The section says how many more lessons are stored. search_memory, registered only in this mode, looks them up read-only. Mark a rule every run needs with pinned: true. The selection is journaled as memory_selected and indexed as memory:<id> features, so the signal map shows a lesson whose presence goes with failure. selection and max_selected are frozen from evolution.

Reflection #

evolution:
  reflect: {enabled: true, on: failure, max_lessons: 1, cooldown_minutes: 30}

After a run (a failed one by default), a strong model reads the run's task, outcome, verifier feedback and friction next to the lessons already held, and drafts at most max_lessons new ones. It is told that no lesson is better than a speculative or one-off one. Drafts are queued with trigger: reflect through the same path as propose_memory:

• redacted;
• deduplicated on content;
• checked against the budgets;
• applied only by proposals apply.

Eval runs never reflect, and the cooldown bounds spend: a reflection that queues nothing still starts it. evolution.reflect is frozen from evolution.

Compaction that keeps what it learned #

Summarize-compaction now carries the previous summary forward. It is lifted out of the transcript into <previous-summary>, with an instruction to keep every identifier, value and ruled-out approach still true. The agent's newest statement is added as a <recent-state> anchor, so the summary does not describe a state the run has already left.

Privacy #

The signal map holds feature names, counts and statistics, never tool inputs, outputs or model text. Successful runs appear in the evolution report with their task and output only under evolution.trace_excerpts.enabled, the same frozen opt-in that governs failure excerpts. Without it, the report says only that passing runs exist and what they cost. Reflection sends one run's capped, redacted task and output to the configured strong model. It is opt-in and frozen.