WEBVTT

1
00:00:00.000 --> 00:00:04.041
A skill needs to work when the task is
relevant and stay out of the way when

2
00:00:04.075 --> 00:00:08.406
it is not. It also needs to preserve
uncertainty when the input is incomplete.

3
00:00:08.789 --> 00:00:12.005
In this lesson, we will evaluate
those three behaviors separately.

4
00:00:12.608 --> 00:00:15.882
Start with checkpoint four, the
datasheet-to-constraints skill,

5
00:00:15.998 --> 00:00:17.937
and the shipped skills test requests.

6
00:00:18.541 --> 00:00:22.593
Your deliverable is an actual behavior
record, not merely a list of prompts.

7
00:00:22.976 --> 00:00:25.867
Begin by distinguishing
selection from answer quality.

8
00:00:26.192 --> 00:00:30.000
The host might select the correct skill
and still produce an unsupported table.

9
00:00:30.325 --> 00:00:33.936
It might produce a useful answer without
enough visible evidence to establish

10
00:00:34.052 --> 00:00:37.593
which procedure it loaded. Record
these dimensions separately.

11
00:00:37.976 --> 00:00:41.111
Otherwise, a polished answer
can hide a discovery problem,

12
00:00:41.262 --> 00:00:45.035
or a correct activation can receive
credit for a technically wrong result.

13
00:00:45.360 --> 00:00:48.715
For the first relevant case,
name the skill explicitly.

14
00:00:49.040 --> 00:00:53.266
In Codex, use the documented
skill selector or dollar-prefixed

15
00:00:53.348 --> 00:00:56.726
name available in your interface,
as described in the written lesson.

16
00:00:57.051 --> 00:01:02.589
Ask it to derive the Feather product five
four seven seven and BME280 interface

17
00:01:02.636 --> 00:01:06.618
constraints from the primary links and
sections in the hardware reference.

18
00:01:06.897 --> 00:01:09.370
Preserve the exact
prompt and source scope.

19
00:01:09.648 --> 00:01:14.374
Inspect the actual work rather than
expecting identical prose across runs.

20
00:01:14.757 --> 00:01:18.472
Did it identify the exact board and
distinguish the breakout from the chip?

21
00:01:18.855 --> 00:01:22.106
Which sources were supplied,
and which were actually opened?

22
00:01:22.547 --> 00:01:25.902
Does each factual row have a
location that supports the claim?

23
00:01:26.228 --> 00:01:28.584
Are units and operating
conditions preserved?

24
00:01:28.909 --> 00:01:32.950
Does the consequence follow from the
evidence, or is it an unlabeled leap?

25
00:01:33.228 --> 00:01:36.073
A table can look complete
while failing this review.

26
00:01:36.456 --> 00:01:40.658
Select one row with a number or pin
claim and trace it back to the source.

27
00:01:40.983 --> 00:01:44.014
If the cited section does not
support it, the row fails.

28
00:01:44.397 --> 00:01:47.381
Do not rescue the row because
the value sounds familiar.

29
00:01:47.822 --> 00:01:50.666
The procedure is supposed to
produce traceable constraints,

30
00:01:50.782 --> 00:01:53.882
so evidence quality is part
of the output contract.

31
00:01:54.393 --> 00:01:58.665
Next, start a fresh task with the relevant
request but do not name the skill.

32
00:01:59.176 --> 00:02:01.162
This is the implicit-selection case.

33
00:02:01.765 --> 00:02:05.573
A description that matches the task
should help the host choose the procedure.

34
00:02:06.084 --> 00:02:10.438
Preserve any activation or file-read
evidence that the interface exposes.

35
00:02:10.821 --> 00:02:12.389
If selection is not observable,

36
00:02:12.633 --> 00:02:16.731
record that uncertainty and still evaluate
the final table on its own merits.

37
00:02:17.335 --> 00:02:19.842
We now need a genuinely unrelated request.

38
00:02:20.167 --> 00:02:24.161
In another fresh task, ask only for the
spelling correction in this sentence:

39
00:02:24.405 --> 00:02:27.691
The teminal shows a reading.
Do not mention the skill.

40
00:02:27.969 --> 00:02:31.418
The acceptable result corrects
teminal to terminal without forcing

41
00:02:31.487 --> 00:02:35.574
a hardware-constraints table or an
unnecessary source-extraction workflow.

42
00:02:35.957 --> 00:02:39.637
Naming the skill here would spoil
the negative test because explicit

43
00:02:39.707 --> 00:02:42.749
selection changes what the
host is being asked to do.

44
00:02:43.445 --> 00:02:45.709
This case checks the
description's boundary.

45
00:02:45.953 --> 00:02:49.448
If the skill activates for every
sentence that mentions a reading,

46
00:02:49.622 --> 00:02:53.291
its scope is too broad or another
instruction is forcing it.

47
00:02:53.569 --> 00:02:56.751
Inspect the actual cause
before changing the file.

48
00:02:57.076 --> 00:03:01.963
A host behavior, explicit request,
or project rule can affect selection.

49
00:03:02.347 --> 00:03:05.505
The test record should preserve
enough context to explain

50
00:03:05.551 --> 00:03:08.894
the result rather than blaming
the description automatically.

51
00:03:09.219 --> 00:03:12.737
For incomplete input, use
the refined course scenario:

52
00:03:13.016 --> 00:03:17.776
the requested ESP32-S3 board
differs from the documented product

53
00:03:17.892 --> 00:03:22.547
five four seven seven, and no exact
variant or schematic is supplied.

54
00:03:23.151 --> 00:03:24.788
Use only the given description.

55
00:03:25.229 --> 00:03:29.281
Do not assume the repository’s example
board or retrieve another source.

56
00:03:29.490 --> 00:03:33.728
The acceptable behavior identifies the
missing exact board reference and avoids

57
00:03:33.786 --> 00:03:37.872
guessing pins. A family name does
not establish the board layout.

58
00:03:38.151 --> 00:03:42.691
A familiar mapping from another project is
not a substitute for the missing source.

59
00:03:43.016 --> 00:03:45.547
You can use an additional
fixed-excerpt version when

60
00:03:45.582 --> 00:03:47.614
evaluating a different information packet.

61
00:03:48.124 --> 00:03:51.352
State that the answer must
use only the supplied excerpt.

62
00:03:51.793 --> 00:03:54.626
Then check whether missing
information remains missing.

63
00:03:54.951 --> 00:03:57.993
If the agent quietly retrieves
a different board's mapping,

64
00:03:58.202 --> 00:04:00.037
it has changed the evidence boundary.

65
00:04:00.478 --> 00:04:03.264
A useful next step names
the source needed to finish;

66
00:04:03.589 --> 00:04:06.968
it does not disguise a guess
with a word such as probably.

67
00:04:07.409 --> 00:04:09.696
Record each case in a small matrix.

68
00:04:09.975 --> 00:04:13.133
Include the prompt, source
boundary, skill identity,

69
00:04:13.307 --> 00:04:17.568
interface and version, observable
selection, output review, and verdict.

70
00:04:18.009 --> 00:04:20.842
Link to the actual response
and available tool evidence.

71
00:04:21.120 --> 00:04:25.125
A row labeled planned should remain
planned until execution occurs.

72
00:04:25.450 --> 00:04:28.295
A row labeled blocked
should explain the concrete

73
00:04:28.318 --> 00:04:31.058
limitation without pretending
the procedure passed.

74
00:04:31.441 --> 00:04:34.738
If the positive case fails because
the skill is not discovered,

75
00:04:34.947 --> 00:04:38.047
inspect the folder, name,
and active host state.

76
00:04:38.430 --> 00:04:41.855
If the negative case fails because
the description is too broad,

77
00:04:42.099 --> 00:04:44.212
narrow its task words and boundary.

78
00:04:44.723 --> 00:04:47.544
If the incomplete-input
case invents facts,

79
00:04:47.823 --> 00:04:52.455
strengthen the row-by-row evidence check
and add the correct unresolved response.

80
00:04:52.896 --> 00:04:56.367
Each repair should address the
failure you actually observed.

81
00:04:57.064 --> 00:05:01.626
After changing the description, rerun
both the positive and negative cases.

82
00:05:02.068 --> 00:05:05.075
A description that never
activates can appear to fix

83
00:05:05.191 --> 00:05:08.337
unwanted selection while
destroying useful selection.

84
00:05:09.034 --> 00:05:11.112
After changing uncertainty guidance,

85
00:05:11.437 --> 00:05:15.001
rerun the incomplete-input case
and inspect the relevant table

86
00:05:15.047 --> 00:05:18.345
too. Keep the before-and-after
records so the improvement

87
00:05:18.379 --> 00:05:20.759
can be assessed without
relying on your memory.

88
00:05:21.270 --> 00:05:25.090
Pause here to run the four
cases: explicit relevant,

89
00:05:25.217 --> 00:05:28.642
implicit relevant, unrelated,
and incomplete input.

90
00:05:28.967 --> 00:05:32.520
They should use separate fresh
tasks. Your prompts and wording may

91
00:05:32.590 --> 00:05:36.305
lead to different outputs from the
course recording, so grade the actual

92
00:05:36.363 --> 00:05:40.404
behavior against the stated criteria.
Do not try to force an answer

93
00:05:40.473 --> 00:05:44.037
to match a screenshot merely to
make the exercise look successful.

94
00:05:44.421 --> 00:05:47.880
When you finish, explain the
difference between a valid package,

95
00:05:48.124 --> 00:05:50.411
correct selection, and a supported answer.

96
00:05:50.852 --> 00:05:53.720
All three matter, and each
has different evidence.

97
00:05:54.045 --> 00:05:58.677
A finite set of passing cases does
not guarantee every future response.

98
00:05:59.002 --> 00:06:02.752
It gives you a maintained baseline
for this procedure and a practical

99
00:06:02.811 --> 00:06:05.736
way to detect regressions
after relevant changes.

100
00:06:06.061 --> 00:06:08.883
That is a useful engineering
role for skill testing.

101
00:06:09.208 --> 00:06:12.853
Keep your source inputs stable
when comparing two skill versions.

102
00:06:13.294 --> 00:06:16.150
If the excerpt changes at the
same time as the procedure,

103
00:06:16.429 --> 00:06:20.318
a better answer may reflect better
input rather than better instructions.

104
00:06:20.922 --> 00:06:23.453
Record either change explicitly.