M05-L04 — Test when a skill should activate
Evaluate selection and output quality using separate relevant, unrelated, and incomplete-input cases. Start with CP04, the constraints skill, and docs/skills.md. Your output is an actual behavior matrix, not just a list of test prompts.
First explicitly invoke $datasheet-to-constraints for a relevant extraction request. Use the shipped request to derive Feather product 5477/BME280 interface constraints from the primary links and sections in docs/hardware.md. Inspect which sources are available and actually read. Score each factual row by its source, units, scope, uncertainty, and consequence.
Then start a new task with the same relevant request without naming the skill. This tests implicit selection separately. If the host exposes activation or file-read evidence, preserve it. A table-shaped answer by itself may not establish which procedure was loaded. Record that limit rather than guessing.
For the unrelated case, use the exact request: “Correct spelling in this sentence only: ‘The teminal shows a reading.’” Do not name the skill. The acceptable output is the ordinary spelling correction without a forced constraints workflow. Explicitly invoking the skill would invalidate a negative test of unwanted implicit activation.
For incomplete input, use this refined course scenario, which extends rather than quotes the shipped request:
Map the pins for my ESP32-S3 board. It is different from this repository’s Feather product 5477 example. I have not supplied the exact variant or schematic. Use only this description; do not assume the example board or retrieve other sources. Identify the missing information before giving a pin mapping.
The acceptable behavior identifies the missing exact board reference and does not invent pins. The explicit different-board boundary prevents the surrounding repository from silently supplying the example identity. Preserve this exact source boundary when comparing skill versions. Distinguish missing evidence from low-confidence speculation.
Record prompt, source scope, skill identity, host/version, observed activation, output review, and verdict. A valid file can fail selection. Correct selection can still produce unsupported output. Keep those judgments separate so the repair addresses the actual problem.
If the description is too broad, narrow it and rerun the positive and negative cases. If output invents facts, strengthen the row-by-row evidence check and rerun the incomplete-input case. Change one important variable at a time and preserve failures. Repetition is useful when the prompt or host behavior is uncertain, but a finite test set never guarantees all future behavior.
Resources and completion
Use the Sensor Monitor CP04 download and its README.md; project paths in this lesson are relative to that root. Read Chapter 11 for the full lab and explanatory review answers. Figure IDs: SS11-03, SS11-04, SS11-05. Primary references: OpenAI AGENTS.md and OpenAI build skills. Complete the exercise, preserve actual evidence, and use the separate instructor answer key for self-check after attempting the task. Narration scripts are production sources; final transcripts must match the actual narrated edit.
Recording transcript
A skill needs to work when the task is relevant and stay out of the way when it is not. It also needs to preserve uncertainty when the input is incomplete. In this lesson, we will evaluate those three behaviors separately. Start with checkpoint four, the datasheet-to-constraints skill, and the shipped skills test requests. Your deliverable is an actual behavior record, not merely a list of prompts.
Begin by distinguishing selection from answer quality. The host might select the correct skill and still produce an unsupported table. It might produce a useful answer without enough visible evidence to establish which procedure it loaded. Record these dimensions separately. Otherwise, a polished answer can hide a discovery problem, or a correct activation can receive credit for a technically wrong result.
For the first relevant case, name the skill explicitly. In Codex, use the documented skill selector or dollar-prefixed name available in your interface, as described in the written lesson. Ask it to derive the Feather product five four seven seven and BME280 interface constraints from the primary links and sections in the hardware reference. Preserve the exact prompt and source scope.
Inspect the actual work rather than expecting identical prose across runs. Did it identify the exact board and distinguish the breakout from the chip? Which sources were supplied, and which were actually opened? Does each factual row have a location that supports the claim? Are units and operating conditions preserved? Does the consequence follow from the evidence, or is it an unlabeled leap?
A table can look complete while failing this review. Select one row with a number or pin claim and trace it back to the source. If the cited section does not support it, the row fails. Do not rescue the row because the value sounds familiar. The procedure is supposed to produce traceable constraints, so evidence quality is part of the output contract.
Next, start a fresh task with the relevant request but do not name the skill. This is the implicit-selection case. A description that matches the task should help the host choose the procedure. Preserve any activation or file-read evidence that the interface exposes. If selection is not observable, record that uncertainty and still evaluate the final table on its own merits.
We now need a genuinely unrelated request. In another fresh task, ask only for the spelling correction in this sentence: The teminal shows a reading. Do not mention the skill. The acceptable result corrects teminal to terminal without forcing a hardware-constraints table or an unnecessary source-extraction workflow. Naming the skill here would spoil the negative test because explicit selection changes what the host is being asked to do.
This case checks the description's boundary. If the skill activates for every sentence that mentions a reading, its scope is too broad or another instruction is forcing it. Inspect the actual cause before changing the file. A host behavior, explicit request, or project rule can affect selection. The test record should preserve enough context to explain the result rather than blaming the description automatically.
For incomplete input, use the refined course scenario: the requested ESP32-S3 board differs from the documented product five four seven seven, and no exact variant or schematic is supplied. Use only the given description. Do not assume the repository’s example board or retrieve another source. The acceptable behavior identifies the missing exact board reference and avoids guessing pins. A family name does not establish the board layout. A familiar mapping from another project is not a substitute for the missing source.
You can use an additional fixed-excerpt version when evaluating a different information packet. State that the answer must use only the supplied excerpt. Then check whether missing information remains missing. If the agent quietly retrieves a different board's mapping, it has changed the evidence boundary. A useful next step names the source needed to finish; it does not disguise a guess with a word such as probably.
Record each case in a small matrix. Include the prompt, source boundary, skill identity, interface and version, observable selection, output review, and verdict. Link to the actual response and available tool evidence. A row labeled planned should remain planned until execution occurs. A row labeled blocked should explain the concrete limitation without pretending the procedure passed.
If the positive case fails because the skill is not discovered, inspect the folder, name, and active host state. If the negative case fails because the description is too broad, narrow its task words and boundary. If the incomplete-input case invents facts, strengthen the row-by-row evidence check and add the correct unresolved response. Each repair should address the failure you actually observed.
After changing the description, rerun both the positive and negative cases. A description that never activates can appear to fix unwanted selection while destroying useful selection. After changing uncertainty guidance, rerun the incomplete-input case and inspect the relevant table too. Keep the before-and-after records so the improvement can be assessed without relying on your memory.
Pause here to run the four cases: explicit relevant, implicit relevant, unrelated, and incomplete input. They should use separate fresh tasks. Your prompts and wording may lead to different outputs from the course recording, so grade the actual behavior against the stated criteria. Do not try to force an answer to match a screenshot merely to make the exercise look successful.
When you finish, explain the difference between a valid package, correct selection, and a supported answer. All three matter, and each has different evidence. A finite set of passing cases does not guarantee every future response. It gives you a maintained baseline for this procedure and a practical way to detect regressions after relevant changes. That is a useful engineering role for skill testing.
Keep your source inputs stable when comparing two skill versions. If the excerpt changes at the same time as the procedure, a better answer may reflect better input rather than better instructions. Record either change explicitly.
Download transcript
