M05-L02 — Test instructions in a fresh session
Evaluate whether repository guidance affects actual work in a fresh session. Start with CP03 or later in a separate exercise folder and use the request below. CP04 later collects the complete skill/request matrix in docs/skills.md. Use a behavior already implemented at that checkpoint. The outcome is a prompt, actual action record, independent review, and verdict.
A continued conversation may carry the command and source facts from earlier messages. Starting a new task makes the repository responsible for supplying those details. A request to summarize active instructions is a useful diagnostic, but a summary alone does not establish that the workflow is followed.
Use this CP03 reporting-documentation case: “Clarify the README's synthetic-input explanation using the project brief. Keep the edit bounded and complete the documented replay check.” Do not paste the exact command into the task. At CP07 or later, adding a test for an implemented behavior is a separate alternative, not a CP03 prerequisite.
Observe context reads, scope, actual command use, and result interpretation. An announced intention to test is not execution. Inspect the changed explanation and check that its claims come from the applicable project brief and preserve synthetic-input limits. Then read the process result and output from the documented wrapper.
Record interface/version, checkpoint, instruction-file identity, prompt, observed actions, changed files, commands/results, and limitations. Independently inspect the diff. A passing test shows relevant guidance used, bounded work, actual available checks, and an accurate completion report. If a compiler is unavailable, the run may be blocked; it is not a passing execution result.
If the run fails, classify the issue before editing instructions. Wrong folder or a hidden extension is discovery. Contradictory active guidance is scope or wording. A nonexistent script is stale documentation. A denied operation is runtime capability or permission. Repair the smallest supported cause and repeat in another fresh session.
Figures SS10-03–05 show fresh task, command use, and repair/rerun. They must come from actual runs. The course's procedure defines success, while your saved evidence determines whether success occurred.
Resources and completion
Use the Sensor Monitor CP03 download and its README.md; project paths in this lesson are relative to that root. Read Chapter 10 for the full lab and explanatory review answers. Figure IDs: SS10-03, SS10-04, SS10-05. Primary references: OpenAI AGENTS.md and OpenAI build skills. Complete the exercise, preserve actual evidence, and use the separate instructor answer key for self-check after attempting the task. Narration scripts are production sources; final transcripts must match the actual narrated edit.
Recording transcript
A clear instruction file is useful, but its wording alone does not prove that the agent used it. In this lesson, we will evaluate a fresh task that relies on repository guidance. We will observe the actual work, review the diff, and record a verdict. Start with checkpoint three or a later preserved exercise copy. Use a behavior and command that exist at that stage.
A continued conversation can hide a discovery problem. You may already have told the agent which document matters and which command to run. If the task succeeds, you cannot tell whether the repository supplied that information or the conversation carried it forward. A fresh session reduces that ambiguity. It is still a practical test, not a guarantee about every future task.
Before starting, read the root instruction file and the request in the written lesson. Confirm the active project root. For this early checkpoint, use a bounded README or report change and the available replay verification. A later checkpoint can use the alternative task of adding a meaningful host assertion test. Do not import later test infrastructure into a stage where it has not been introduced.
The early request asks the agent to clarify the README's explanation of synthetic input using the project brief, keep the edit bounded, and complete the relevant documented replay check. Notice what the request does not contain. We are not pasting the exact command or every instruction into the prompt. The repository should supply the working details through its maintained guidance and references.
You can first ask an inspection-only question about active instruction sources if discovery is uncertain. That is a diagnostic step. Compare the answer with the actual folder and filenames. A correct summary can help locate a problem, but it is not the behavior test. The real question is whether the agent follows the workflow while doing the bounded task.
Start the fresh task and preserve the exact prompt. Watch for relevant context inspection. Does the agent find the project brief and the README? Does it recognize the synthetic-input scope? A final paragraph that sounds sensible is not enough by itself to establish how the task was performed. Keep the available file-read or tool evidence so a reviewer can connect the output to the process.
Next, examine the scope of the edit. The requested outcome is clearer documentation, not a firmware redesign. A bounded diff should change the relevant explanation and preserve unrelated work. If the agent edits policy or target settings without a reason connected to the task, record that as a scope finding. Do not let the attractiveness of the final prose distract you from what actually changed.
Now inspect command use. The task asks for the documented replay verification. An announced intention to run it is a plan. A tool call with an actual process result is execution. Record the command, fixed synthetic output, exit result, and source identity. CP03 does not parse the later fixtures. The dot cmd wrapper is the current Windows entry point, and the source identity should match the exercise you are reviewing.
Interpret the result carefully. A successful synthetic replay supports a bounded software claim for its fixed synthetic input and source. It does not establish physical sensor behavior. The agent's completion report should preserve that distinction. If the command could not run because a required tool was unavailable, the report should say so clearly. A truthful blocked result is useful evidence, but it is not a passing execution check.
Independently inspect the edited README and the output. This is a separate verification step, and your record should identify it as such. If you run a command manually after the agent stopped, that can verify the artifact. It does not retroactively prove that the agent followed the instruction to run it. Keep the agent-behavior verdict and the artifact-verification verdict distinct.
When the run fails, classify the cause before rewriting the instruction file. A wrong folder or hidden filename extension is a discovery problem. A stale command path is a documentation defect. Contradictory active guidance is a scope or wording problem. A denied operation belongs to the runtime environment or permission boundary. Each condition suggests a different repair, so one generic instruction to try harder is unlikely to help.
Make the smallest repair supported by the evidence. Correct a path, remove a demonstrated contradiction, or tighten an overbroad scope sentence. Then repeat the same task in another fresh session. Preserve the initial failed record and the rerun. If several important variables change at once, it becomes difficult to explain why the outcome improved.
Your evidence record needs the checkpoint, interface and version, instruction-file identity, prompt, observed actions, changed-file list, command results, independent review, and verdict. Use concise pointers to longer logs when available. The record should let someone understand both the successful actions and the limitations without trusting a summary that says everything worked.
Pause here to complete the exercise. If your account reaches a limit, or the required tool is unavailable, preserve the prepared task and static review. Mark the runtime gate incomplete and continue with the written material. Do not convert a planned run into a pass merely because its expected behavior is clear. The same honesty applies if the interface does not expose enough evidence to establish a particular action.
When you finish, explain what the test established. A passing run shows that this fresh task used relevant repository context, kept the work bounded, executed the available documented check, and reported the result accurately. The skill is in making that claim precise. You now have a repeatable way to evaluate project instructions after an important wording or tool change, rather than relying on reassurance alone.
A repair should also preserve the original task boundary. If changing guidance requires altering unrelated project policy, pause that expansion and record it separately. The rerun should remain comparable enough to explain what the instruction change accomplished.
Download transcript
