Use diagnostic and recovery instructions
Follow supported setup operations for the selected client.
Open this resource →Workbook · Device evaluation
Explain what this device is ready to do.
Check a developer connection against specific requirements, native activation, signed policy, and observed behavior.
A ready device needs more than installed files. Build a short record that explains which requirements were checked and what the observations establish.
01 / Workbook · Device evaluation
Start with a device that has a real job
A Search developer has completed setup and is ready to change search-api. The team needs reviewed guidance and a restriction on shell executables.
The team also asks whether Codex can enforce its preferred model choice. These requirements need separate answers.
This is a fictional evaluation. It illustrates how to avoid treating one successful installation message as evidence about unrelated controls.
Give each requirement its own row in your working notes. Record the operation involved, the selected agent, and the result you expect.
The resulting record should support a practical decision about this device. It should not imply that every team device has the same state.
Keep the date, native version, setup version, and team release attached to the evaluation. They explain the conditions under which the observations were made.
02 / Workbook · Device evaluation
Describe the requirement at its actual boundary
A statement such as “apply our guidance” needs more detail. Identify the selected resource and a task where its advice should be relevant.
A shell restriction needs an executable and a covered invocation event. A model requirement needs the particular selection path the team intends to restrict.
Support for an explicit model switch does not establish support during startup, resume, fallback, or subagent creation. Those are separate paths.
Use the support contract to distinguish implemented, conditional, advisory, and unavailable behavior. An unknown client version leaves the local support question unresolved.
For instructions, record availability and observed use separately. An agent can retrieve the intended content while still overlooking an important task condition.
A useful evaluation makes these differences visible without turning every observation into a general readiness claim.
03 / Workbook · Device evaluation
Prepare a requirement record
- 01
Name the operation
Write the behavior that matters at the selected boundary.
- 02
Identify the software
Record the native client and installed version.
- 03
Identify the configuration
Record the team release and pinned company revision.
- 04
Classify the requirement
Record its support state from the shared compatibility contract.
- 05
Choose a safe observation
Define an expected success and denial for a supported runtime gate.
- 06
Choose an instruction task
Select observable work that makes the reviewed guidance relevant.
04 / Workbook · Device evaluation
Inspect the example Codex installation
Complete company setup before running the developer evaluation. Confirm that the device is connected to the intended organization and team.
Inspect Codex and Betterloop setup versions. Verify that the chosen versions support the signed resource format required by the configuration.
Review Betterloop hooks through the native /hooks interface. Complete native trust, then start the fresh session required by installation or reload.
npx @betterloop/setup doctor --harness codex --project /path/to/repositoryInspect the complete result rather than a single status line. Authentication, policy, project identity, installed integration, and activation can have different states.
Observed hook invocation establishes activity at that lifecycle boundary. Merely finding the hook script on disk does not establish that it executed.
Resolve any ambiguous project match before using project knowledge. The selected project determines which repository context should become available.
05 / Workbook · Device evaluation
Evaluate instructions with a concrete task
For Search, select a small pagination change. Ask the agent to preserve explicit cursor behavior while correcting the default page-size handling.
Confirm that the intended skill and supporting reference can be retrieved. Compare their exact versions with the reviewed team release.
Then inspect the work itself. Look for acceptance conditions, the relevant caller trace, and evidence from checks that cover the requested change.
A resource can be present while its advice is poorly applied. That result needs task and guidance investigation, not an unsupported claim of enforcement.
Use available output to assess the report. A claimed check without its result cannot establish that the check ran successfully.
Keep observations specific to the task. One good pagination report does not prove that the agent will handle every future backend change correctly.
06 / Workbook · Device evaluation
Make the record easy to review
Codex model restriction remains a gap under the shared contract. A successful shell check cannot stand in for the missing model control.
If the missing requirement is mandatory, resolve that decision before widening the pilot. Keep the useful supported observations without overstating the overall result.
| Team requirement | Evidence for this evaluation | Interpretation |
|---|---|---|
| Selected guidance is available. | Exact identity, version, and retrieval result. | The intended instruction reached the client. |
| Guidance improves the task report. | Task observations beside actual verification output. | The advice was applied in the observed task. |
| Shell restriction operates at the covered event. | Native activation plus harmless allowed and denied cases. | The tested cases demonstrate conditional host coverage. |
| Codex restricts model selection. | The shared client support contract. | The requirement is not yet supported. |
| Project knowledge matches the repository. | Approved identity and explicit project resolution. | Assess this repository context independently. |
| Activity records arrive. | The bounded client reports that were received. | Reports are available; they are not machine attestation. |
07 / Workbook · Device evaluation
Consider local policy and connectivity
Covered hooks use verified local signed policy. The gate does not require a fresh network round trip for every covered operation.
A valid, unexpired local policy can continue through a temporary outage. Background synchronization refreshes state during active use and retries bounded queued events.
Expiry, invalid signatures, and known revocations change the policy evaluation. Supported tool gates deny when their required policy cannot be evaluated.
Native host execution has its own limitation. A disabled hook, timeout, or host hook error can still allow an operation to proceed.
Keep those cases separate in the record. Policy rejection behavior does not establish reliable host execution or an operating-system sandbox.
Event delivery also remains best effort. Do not conclude that an agent was idle simply because no decision record arrived.
08 / Workbook · Device evaluation
Repair one cause at a time
Save the original output before repair. It can distinguish a connection problem from a policy decision or native permission issue.
A repository knowledge mismatch does not usually require disconnecting the entire device. Enrollment and repository caches have different scopes.
| Observation | Repair to investigate | Observation to repeat |
|---|---|---|
| Native authentication is missing. | Complete the supported sign-in for the selected client or integration. | Check access and tool discovery. |
| Scripts exist without activation. | Review trust and the required session restart. | Observe the relevant hook and safe test cases. |
| Signed state is expired. | Restore connectivity and synchronize the connection. | Inspect release identity and affected requirements. |
| Several projects match. | Resolve the intended project explicitly. | Verify the repository knowledge context. |
| A resource cannot be retrieved. | Inspect selected dependencies, release format, and client support. | Retrieve the exact intended version. |
09 / Workbook · Device evaluation
Repeat the evaluation when its inputs change
A connection switch requires a fresh session. Existing background work and MCP subprocesses retain the installation identity captured when they started.
After connectivity recovers, synchronize signed state. Run Doctor for the same repository, then repeat the affected requirement checks.
For migration or removal, follow the supported recovery instructions. Keep native model credentials and unrelated integrations intact.
Claude Code explicit model switches require version 2.1.251 or later. Startup, resume, fallback, and subagents need native managed model settings.
Mimir has its own installation and version requirements. Compare them with the missing requirement before proposing a runtime change.
A useful record can include supported behavior alongside unresolved requirements. Its value comes from the distinction, not from presenting every row as passed.
10 / Workbook · Device evaluation
Questions while recording results
Does the support label describe my current installation?
It describes the implemented capability. Local versions, signed policy, activation, and observed behavior still need examination.
Can decision events replace the safe checks?
No. Events are bounded client reports with best-effort delivery. Inspect the relevant runtime observation as well.
What should an unknown version mean?
Keep support unresolved until the version can be identified. Do not infer local behavior from a product description.
When is the evaluation ready for wider use?
When mandatory requirements have acceptable evidence and the team has explicitly resolved any remaining gaps.
11 / Workbook · Device evaluation
Publish a decision your team can revisit
Summarize the device, repository, release, versions, and actual observations. Name any requirement that remains unsupported or unverified.
Attach the follow-up to the specific missing evidence. The next reviewer should know what to repeat after a repair or release change.