Follow the shared learning review path
See where local candidates, assessment, approval, and distribution connect.
Open this resource →Engineering · learning evidence
Test the improvement before asking others to adopt it.
Follow a local candidate through meaningful verification, explicit sharing, independent approval, and scoped outcome evaluation.
A useful local fix deserves a check that can reveal when it is wrong. That check gives its author and reviewer a concrete basis for deciding whether anyone else should use it.
01 / Engineering · learning evidence
Write acceptance while the problem is still clear
An agent can describe work as complete while an important condition remains unmet. Capture that condition before testing the proposed improvement. It should determine whether the result is acceptable.
For a code change, choose behavior that the patch must satisfy. For a document, identify required facts and source checks. For configuration, inspect the effective values that matter to the task.
A test is useful only if it can expose the relevant error. Printing a path or checking that a file exists can pass while behavior remains incorrect. A broad suite may also miss the changed condition.
Define the exclusions as well. A scoped improvement cannot be assessed against every unrelated failure in a repository. Reviewers need to know which work the candidate is intended to help.
02 / Engineering · learning evidence
Four relationships in an outcome record
| Relationship | Question it answers | What it does not establish alone |
|---|---|---|
| Exposure | Was the learning entry supplied during this work? | That this task needed the entry |
| Applicability | Did the task match the stated scope? | That the result met its acceptance conditions |
| Verification | Was the result checked using an explicit method? | That the entry caused the result |
| Attribution | Is the scoped result connected to the entry with a reason? | Scientific causality across changing workloads and environments |
03 / Engineering · learning evidence
Example: catch a recurring configuration omission
In this hypothetical case, a developer repeatedly omits one required configuration check. After substantive work, the developer records a generalized local observation. Repetition or an explicitly verified correction can qualify a draft candidate.
The developer reviews the private candidate and defines where the check should apply. Instructions contain the judgment about applicability. A small script handles the repeatable condition, with a focused test for its behavior.
Next, the developer stages the exact candidate files and selects an explicit verification command. The verifier runs it in the staged directory without a shell. Execution has a two-minute timeout.
File bytes are checked before and after execution. A passing receipt refers to the candidate digest. Editing the candidate changes the verification basis, so an earlier pass cannot verify the edited content.
The developer can retain the candidate privately or dismiss it. Sharing requires inspection of the bounded public submission and explicit confirmation. The local bridge cannot silently publish that private draft.
A script-backed candidate must have passing verification before submission. Its contribution includes a public summary and evidence digests. Private script bytes, paths, commands, and skill instructions are excluded.
betterloop optimize --harness codex --action list
betterloop optimize --harness codex --candidate UUID --action review
betterloop optimize --harness codex --candidate UUID --action stage
betterloop verify --harness codex --candidate UUID -- node tests/check.test.mjsThese commands operate on a local candidate. They do not turn it into a shared executable resource. A receipt records the selected check for identified bytes; reviewers must still assess whether that check is meaningful.
For example, a script that checks only the expected input may miss the omission it is meant to detect. Include an input that should fail and an input that should pass before relying on the result.
04 / Engineering · learning evidence
Associate evidence with the entry it can assess
An exposed entry can be irrelevant to a particular task. An applicable entry can still lack verification. The outcome record needs these distinctions before a non-neutral runtime result is justified.
The contract requires exposure, applicability, and a verification reason. It keeps verified success, attributed failure, and neutral outcomes separate. Generic completion cannot supply the missing evidence for those relationships.
Likewise, a failure in another part of the task does not automatically count against the entry. The recorded reason must connect the relevant failure to its stated applicability. Preserve unrelated failures without inventing that connection.
Even a well-formed result does not establish scientific causality. Model changes, task selection, repository state, and developer choices may affect the outcome. A controlled evaluation needs to address those other variables.
| Observed situation | Appropriate outcome treatment | Reasoning requirement |
|---|---|---|
| Entry supplied, task in scope, explicit check passes | Potential verified success | Connect the passing check to the exposed and applicable entry |
| Failure concerns work outside the entry scope | Neutral for this entry | An unrelated failure does not establish applicability |
| Agent reports completion without a task check | Neutral | The completion statement does not verify acceptance |
| A local candidate test does not pass | Candidate pass absent; no non-neutral fleet outcome | Correct and verify the candidate before using it as an improvement |
| Entry supplied, task in scope, relevant failure identified | Potential attributed failure | Retain the reason linking the scoped failure to the entry |
05 / Engineering · learning evidence
A recommendation does not authorize publication
Automated assessment examines sanitized candidates for reusable value, sufficient evidence, sensitive content, conflicts, and overfitting. It can recommend a memory entry, a skill, or a skill with scripts.
Incomplete evidence or an unavailable assessment service leaves the candidate pending. Time passing does not confer approval. Unsafe, conflicting, or unsupported proposals remain blocked rather than becoming ready by default.
A stored assessment identifies its input and content digests. Changing review text or evidence can require reassessment. A recommendation for one revision must not be treated as approval of later content.
A different human approves the exact eligible memory revision before signing and staged release. Script-backed skills instead use Team Rules and the normal Git review path. That path cannot publish executable fleet packs.
After committing script files, a developer can prepare a reviewable skill reference. It identifies repository, commit, path, digest, runtime, and parameter descriptions. Preparing that reference is still separate from publication.
This gives automated tools a useful supporting role without transferring approval authority. Reviewers can inspect a concrete candidate and its evidence, then decide whether the specified content should be shared.
06 / Engineering · learning evidence
Build a comparison that can reveal a bad tradeoff
Imagine a check that saves time on routine edits but rejects a valid uncommon configuration. A single average might favor the change. The affected case could still make adoption unsuitable.
Select task groups before testing. Include intended matches for the candidate and work outside its applicability. The first group assesses the proposed benefit; the second can reveal unnecessary interference.
Keep retries and unsuccessful attempts. Missing usage measurements should remain unavailable rather than becoming zero. Otherwise the record can make incomplete work appear efficient.
Review correctness before elapsed time, tool calls, or reported token usage. Include slow attempts when measured. Record administrative review and setup separately so their effect remains visible.
For a native Betterloop experiment, retain the chosen native client. Record rules and learning revisions that change. Mimir harness measurements cannot provide a substitute result for that different intervention.
Hypothetical candidate evaluation
Work family: edits that affect required configuration fields
Change: one approved learning revision
Acceptance: targeted behavior check and required task validation
Intended group: changes within the entry applicability
Outside group: edits that should not use the entry
Recorded setup: client, model, rules, learning revision
Observed results: acceptance, retries, timing, tool calls, available usage
Review decision: adopt, amend, or stop under the agreed conditionsState the stop conditions before reviewing aggregate improvements. A correctness regression, unclear scope, or missing required coverage can justify stopping expansion even when another metric improves.
A scoped conclusion is more useful than a broad success claim. It tells the next reviewer which task family was checked and what evidence further adoption still requires.
07 / Engineering · learning evidence
Make weak evidence a concrete next action
When a candidate fails its test, inspect the failure and correct the candidate. If bytes change, review the new identity and repeat verification. A receipt from different content cannot close the issue.
When applicability is uncertain, narrow the stated task family. When a public summary exposes sensitive context, revise it before submission. When assessment is pending, inspect its recorded reason rather than assuming approval.
A conflict with approved learning needs review before release. Publishing contradictory entries would leave an undefined decision to the runtime. Resolve the scope or content difference while the proposal is still inspectable.
If a pilot regresses, retain the failing case and pause wider adoption. Use the documented recovery path to restore the earlier approved process or prepare a reviewed correction. Repeat the relevant acceptance checks afterward.
08 / Engineering · learning evidence
Reviewing the strength of a result
Does a passing candidate command establish complete task correctness?
It records a pass for the selected command and identified candidate bytes. The chosen tests still need to cover the behavior and task conditions that matter.
Can verified success establish that learning caused the benefit?
It provides a scoped outcome record. A causal conclusion needs another evaluation that controls task differences and other variables changing alongside the entry.
What happens to private script material during contribution?
The bounded contribution omits private script bytes, paths, commands, and skill instructions. Sharing executable resources uses the separate repository review workflow.
May an unavailable assessor be treated as an implicit approval?
No. Required assessment remains pending until its evidence and service requirements are satisfied. Availability does not transfer publication authority to the candidate author.
How can a team begin without creating a large program?
Pick one recurring omission and a narrow task family. Write a meaningful acceptance check, review the local candidate, and decide whether its evidence warrants a sharing proposal.