WEBVTT

1
00:00:00.000 --> 00:00:01.997
A clear instruction file is useful,

2
00:00:02.148 --> 00:00:05.260
but its wording alone does not
prove that the agent used it.

3
00:00:05.643 --> 00:00:10.612
In this lesson, we will evaluate a fresh
task that relies on repository guidance.

4
00:00:10.937 --> 00:00:15.187
We will observe the actual work,
review the diff, and record a verdict.

5
00:00:15.570 --> 00:00:19.355
Start with checkpoint three or
a later preserved exercise copy.

6
00:00:19.680 --> 00:00:22.582
Use a behavior and command
that exist at that stage.

7
00:00:23.023 --> 00:00:26.100
A continued conversation can
hide a discovery problem.

8
00:00:26.541 --> 00:00:30.639
You may already have told the agent which
document matters and which command to run.

9
00:00:31.081 --> 00:00:34.482
If the task succeeds, you cannot
tell whether the repository

10
00:00:34.564 --> 00:00:38.047
supplied that information or the
conversation carried it forward.

11
00:00:38.650 --> 00:00:42.841
A fresh session reduces that ambiguity.
It is still a practical test,

12
00:00:43.085 --> 00:00:46.696
not a guarantee about every
future task. Before starting,

13
00:00:46.812 --> 00:00:50.260
read the root instruction file and
the request in the written lesson.

14
00:00:50.585 --> 00:00:54.114
Confirm the active project root.
For this early checkpoint,

15
00:00:54.323 --> 00:00:58.572
use a bounded README or report change
and the available replay verification.

16
00:00:59.176 --> 00:01:02.926
A later checkpoint can use the
alternative task of adding a meaningful

17
00:01:03.007 --> 00:01:07.024
host assertion test. Do not import
later test infrastructure into

18
00:01:07.094 --> 00:01:11.227
a stage where it has not been introduced.
The early request asks the agent

19
00:01:11.262 --> 00:01:15.418
to clarify the README's explanation of
synthetic input using the project brief,

20
00:01:15.662 --> 00:01:19.423
keep the edit bounded, and complete
the relevant documented replay check.

21
00:01:19.749 --> 00:01:22.094
Notice what the request does not contain.

22
00:01:22.338 --> 00:01:26.227
We are not pasting the exact command
or every instruction into the prompt.

23
00:01:26.436 --> 00:01:28.874
The repository should
supply the working details

24
00:01:28.909 --> 00:01:31.254
through its maintained
guidance and references.

25
00:01:31.637 --> 00:01:35.457
You can first ask an inspection-only
question about active instruction sources

26
00:01:35.666 --> 00:01:38.986
if discovery is uncertain.
That is a diagnostic step.

27
00:01:39.161 --> 00:01:42.191
Compare the answer with the
actual folder and filenames.

28
00:01:42.574 --> 00:01:46.939
A correct summary can help locate a
problem, but it is not the behavior test.

29
00:01:47.218 --> 00:01:50.875
The real question is whether the
agent follows the workflow while doing

30
00:01:50.921 --> 00:01:55.229
the bounded task. Start the fresh
task and preserve the exact prompt.

31
00:01:55.740 --> 00:01:58.097
Watch for relevant context inspection.

32
00:01:58.538 --> 00:02:01.115
Does the agent find the
project brief and the README?

33
00:02:01.557 --> 00:02:05.829
Does it recognize the synthetic-input
scope? A final paragraph that sounds

34
00:02:05.864 --> 00:02:09.928
sensible is not enough by itself to
establish how the task was performed.

35
00:02:10.438 --> 00:02:14.119
Keep the available file-read or tool
evidence so a reviewer can connect

36
00:02:14.154 --> 00:02:18.159
the output to the process. Next,
examine the scope of the edit.

37
00:02:18.484 --> 00:02:22.989
The requested outcome is clearer
documentation, not a firmware redesign.

38
00:02:23.314 --> 00:02:28.248
A bounded diff should change the relevant
explanation and preserve unrelated work.

39
00:02:28.573 --> 00:02:32.347
If the agent edits policy or target
settings without a reason connected

40
00:02:32.381 --> 00:02:36.166
to the task, record that as
a scope finding. Do not let

41
00:02:36.201 --> 00:02:40.554
the attractiveness of the final prose
distract you from what actually changed.

42
00:02:41.251 --> 00:02:43.062
Now inspect command use.

43
00:02:43.306 --> 00:02:46.754
The task asks for the
documented replay verification.

44
00:02:46.998 --> 00:02:49.494
An announced intention
to run it is a plan.

45
00:02:49.773 --> 00:02:53.372
A tool call with an actual
process result is execution.

46
00:02:53.697 --> 00:02:58.538
Record the command, fixed synthetic
output, exit result, and source identity.

47
00:02:58.863 --> 00:03:01.789
CP03 does not parse the later fixtures.

48
00:03:02.068 --> 00:03:06.305
The dot cmd wrapper is the current Windows
entry point, and the source identity

49
00:03:06.363 --> 00:03:10.427
should match the exercise you are
reviewing. Interpret the result carefully.

50
00:03:10.752 --> 00:03:14.258
A successful synthetic replay
supports a bounded software claim

51
00:03:14.316 --> 00:03:18.205
for its fixed synthetic input and
source. It does not establish physical

52
00:03:18.275 --> 00:03:22.478
sensor behavior. The agent's completion
report should preserve that distinction.

53
00:03:22.803 --> 00:03:26.355
If the command could not run because
a required tool was unavailable,

54
00:03:26.681 --> 00:03:28.457
the report should say so clearly.

55
00:03:29.061 --> 00:03:31.603
A truthful blocked result
is useful evidence,

56
00:03:31.986 --> 00:03:34.413
but it is not a passing execution check.

57
00:03:34.854 --> 00:03:38.082
Independently inspect the
edited README and the output.

58
00:03:38.465 --> 00:03:42.551
This is a separate verification step, and
your record should identify it as such.

59
00:03:43.062 --> 00:03:45.698
If you run a command manually
after the agent stopped,

60
00:03:45.872 --> 00:03:47.393
that can verify the artifact.

61
00:03:47.834 --> 00:03:51.886
It does not retroactively prove that the
agent followed the instruction to run it.

62
00:03:52.164 --> 00:03:56.390
Keep the agent-behavior verdict and the
artifact-verification verdict distinct.

63
00:03:57.087 --> 00:04:01.243
When the run fails, classify the cause
before rewriting the instruction file.

64
00:04:01.522 --> 00:04:05.283
A wrong folder or hidden filename
extension is a discovery problem.

65
00:04:05.492 --> 00:04:08.255
A stale command path is
a documentation defect.

66
00:04:08.580 --> 00:04:12.156
Contradictory active guidance
is a scope or wording problem.

67
00:04:12.400 --> 00:04:16.486
A denied operation belongs to the runtime
environment or permission boundary.

68
00:04:16.870 --> 00:04:19.145
Each condition suggests
a different repair,

69
00:04:19.319 --> 00:04:22.872
so one generic instruction to
try harder is unlikely to help.

70
00:04:23.383 --> 00:04:27.516
Make the smallest repair supported
by the evidence. Correct a path,

71
00:04:27.760 --> 00:04:31.986
remove a demonstrated contradiction,
or tighten an overbroad scope sentence.

72
00:04:32.427 --> 00:04:35.167
Then repeat the same task
in another fresh session.

73
00:04:35.492 --> 00:04:38.093
Preserve the initial failed
record and the rerun.

74
00:04:38.476 --> 00:04:40.856
If several important
variables change at once,

75
00:04:41.100 --> 00:04:44.049
it becomes difficult to explain
why the outcome improved.

76
00:04:44.490 --> 00:04:47.718
Your evidence record needs the
checkpoint, interface and version,

77
00:04:47.927 --> 00:04:51.328
instruction-file identity,
prompt, observed actions,

78
00:04:51.502 --> 00:04:55.543
changed-file list, command results,
independent review, and verdict.

79
00:04:56.146 --> 00:04:59.153
Use concise pointers to
longer logs when available.

80
00:04:59.478 --> 00:05:02.752
The record should let someone
understand both the successful actions

81
00:05:02.810 --> 00:05:06.862
and the limitations without trusting
a summary that says everything worked.

82
00:05:07.245 --> 00:05:10.740
Pause here to complete the exercise.
If your account reaches a limit,

83
00:05:10.798 --> 00:05:12.667
or the required tool is unavailable,

84
00:05:12.911 --> 00:05:15.198
preserve the prepared
task and static review.

85
00:05:15.523 --> 00:05:19.192
Mark the runtime gate incomplete and
continue with the written material.

86
00:05:19.517 --> 00:05:23.140
Do not convert a planned run into
a pass merely because its expected

87
00:05:23.174 --> 00:05:26.866
behavior is clear. The same
honesty applies if the interface

88
00:05:26.901 --> 00:05:30.315
does not expose enough evidence
to establish a particular action.

89
00:05:30.825 --> 00:05:33.786
When you finish, explain
what the test established.

90
00:05:34.111 --> 00:05:38.488
A passing run shows that this fresh
task used relevant repository context,

91
00:05:38.732 --> 00:05:42.180
kept the work bounded, executed
the available documented check,

92
00:05:42.261 --> 00:05:44.084
and reported the result accurately.

93
00:05:44.328 --> 00:05:48.136
The skill is in making that claim
precise. You now have a repeatable way

94
00:05:48.182 --> 00:05:52.025
to evaluate project instructions after
an important wording or tool change,

95
00:05:52.268 --> 00:05:54.382
rather than relying on reassurance alone.

96
00:05:55.194 --> 00:05:58.224
A repair should also preserve
the original task boundary.

97
00:05:58.468 --> 00:06:02.253
If changing guidance requires
altering unrelated project policy,

98
00:06:02.497 --> 00:06:06.073
pause that expansion and record it
separately. The rerun should remain

99
00:06:06.154 --> 00:06:10.078
comparable enough to explain what
the instruction change accomplished.