Mimir — Alpha

The Claude Code comparison identifies full-suite-20260912. The Codex comparison identifies terra-comparison-20260912. Their recorded benchmark commit is 56d9ba28f4c2940d59dcf8b6fa392eeb6acee4e9.

The Mimir source run for the matched-model comparison is mimir-terra-20260912. Both sides name gpt-5.6-terra. The Claude pair instead records claude-sonnet-5 and the native sonnet selection.

The PDFs summarize results, summary files, and manifests. They include task rows and aggregates. This page does not supply all executable inputs or a complete reproduction environment.

To reproduce an experiment, collect its task fixtures, prompts, grading definitions, adapter rules, configuration, versions, and original artifacts. A run identifier helps locate that material but does not replace it.

When a required artifact is unavailable, mark the reproduction incomplete. Do not infer a prompt from a task label. New client releases also require a newly documented experiment.

The Claude comparison shows a mixed timing result. Mimir finished sooner at the median but later at P95. Its recorded elapsed-time standard deviation also increased.

Claude Code comparison
Median: (40.15 - 30.44) / 40.15 × 100 = 24.2% lower
P95: (70.91 - 64.49) / 64.49 × 100 = 10.0% higher
Elapsed standard deviation: 10.33 s versus 17.03 s

Codex comparison
Median: (98.76 - 28.72) / 98.76 × 100 = 70.9% lower
P95: (305.44 - 92.91) / 305.44 × 100 = 69.6% lower

Standard deviation describes variation around the mean. Here, 17.03 seconds is larger than 10.33 seconds. The summary alone does not identify why those attempts varied more.

P95 comes from 18 observations per harness. Such a small sample cannot establish a stable service-level target. The slower observations are a reason to investigate rather than a guarantee about future latency.

For a developer waiting on an answer, a slow interruption may matter more than an improved average. For batch work, failed attempts and recovery time may dominate. Choose measures that fit the actual workflow.

A pilot should therefore retain each attempt, including retries. Report the slower tail with the middle of the distribution. A single percentage cannot describe both ordinary and unusually slow work.

Suppose a team wants to adopt one reviewed verification instruction. That is a different intervention from replacing the runtime with Mimir. Keep the native agent unchanged while testing the instruction.

• Choose task families that represent the intended work.
• Write acceptance checks before collecting results.
• Record model, client version, permissions, and trust.
• Record the existing rules and learning revisions.
• Collect baseline attempts under the agreed method.
• Introduce the one approved revision.
• Collect equivalent attempts using the same checks.
• Retain unsuccessful runs and unavailable measurements.
• Review acceptance results before usage or timing.
• Decide whether the scoped result justifies further adoption.

Include work outside the instruction applicability. That group helps detect unnecessary behavior. Avoid comparing easier changed tasks with harder baseline tasks, and document unavoidable task differences.

Account for setup, review, and recovery separately from execution. Agree when a correctness regression or missing control will stop expansion. Your own criteria should determine the rollout decision.

A comparison can change a runtime, a model, instructions, or several elements together. Its conclusion applies to that combination. Attributing the result to one element requires additional evidence.

This distinction matters for Betterloop evaluations. A change to reviewed team instructions is different from replacing the agent runtime. The September Mimir measurements cannot supply the missing native integration comparison.

Choose the question before choosing a metric. For example, a team may want to know whether a reviewed workflow reduces incomplete submissions. Token usage alone cannot answer that question.

For a native Betterloop pilot, keep the native client in use. Record its version, selected model, trust state, permissions, rules revision, and applicable learning revision.

Use equivalent work with acceptance checks defined in advance. Include tasks where the proposed change should help and tasks where it should have no effect. The latter can reveal unnecessary steps.

Measure administrative work separately. Review time, setup effort, and recovery are part of adoption even when a task finishes sooner. Keep retries and unsuccessful attempts in the record.

A useful decision states which requirement improved, under which conditions. It also states what remains unknown. Broader rollout needs evidence from the work and clients that broader rollout will contain.