14 — Debug with Evidence

Debugging starts with a difference between what should happen and what actually happened. A useful assistant helps make that difference smaller and more precise: it reads the relevant source, proposes explanations, and selects a test whose outcome distinguishes them. It is less useful when it responds to an incomplete symptom with a large patch and a confident story.

This chapter uses CP06 and the separate fault checkpoints. You will diagnose a malformed synthetic fixture, then investigate a stale-reading defect. Both can be completed without hardware. The aim is to show the evidence that supports a cause, apply a bounded correction, and demonstrate the original reproducer working afterward. Chapter 15 adds a fuller native test workflow.

Write the symptom before naming the cause

“The sensor is broken” describes neither the observed behavior nor the environment. “The replay exits two at fixture line three when the value is 28C” gives another person something to reproduce. It also avoids implying that a physical sensor was involved when the input came from a file.

Create a short record with five items: starting checkpoint, exact command, expected behavior, actual output/exit, and the smallest input that produces the difference. Add a source hash or commit identity when available. If you changed several files before the failure, include the diff instead of reconstructing the change from memory.

Separate expectation from observation. You might expect a valid numeric sample at timestamp 1000, while the observed adapter error says the supplied token is invalid. Record both. Do not silently edit the fixture and then describe the corrected file as the original reproducer. The history of the experiment matters because it explains what the correction changed.

An assistant can help format this record, but it cannot supply missing output that was never captured. If a terminal session has disappeared, rerun the command in a disposable copy. A recreated run is useful evidence when labeled honestly. It is not the original live chronology.

Locate the failure layer

Before interpreting policy, ask whether the source compiled, the executable started, and the input was accepted. A compiler error means source could not be translated successfully. A linker error means compiled pieces could not be combined as required. A process that cannot start may have a runtime dependency problem. A running process can then reject input or implement the wrong policy.

The layers suggest different next checks. If the compiler reports a missing header, inspect the include path and source layout. If a native executable exits before printing anything, capture its exact exit and inspect the documented runtime DLL path. If JSON output appears and then a line-number error occurs, the process has already started and reached input handling.

A real Windows rehearsal of this project exposed that distinction. The same native test executable initially exited without stdout when its runtime DLL directory was absent from the process PATH. With the compiler distribution's bin directory available to that process, the same executable passed its assertions. The evidence supported an execution-environment cause, not a policy rewrite. The supplied .cmd wrappers account for this process-local dependency.

Do not turn that example into a universal diagnosis for every silent program. It illustrates how to preserve the executable, change one relevant condition, and compare results. A different failure needs its own evidence. Similarly, a remote-control tool failing to start a guest session is not automatically a defect in the program it was supposed to run.

Worked example: a malformed input token

Open CP06 in a new folder and inspect fixtures/invalid.csv. It is explicitly labeled as a malformed synthetic fixture. The first data row supplies a valid number. The second supplies 28C, which does not match the adapter's numeric grammar.

Run the reproducer:

.\scripts\replay.cmd fixtures/invalid.csv
$invalidExit = $LASTEXITCODE

The expected result is process exit two and a diagnostic identifying line three as an invalid sample number. A startup record and the earlier valid sample can already have been printed. Partial output does not mean the rejected token became a valid temperature. Read both stdout and stderr, and keep the line number with the actual file revision.

Now write three candidate explanations before changing anything: the fixture contains an invalid token; the parser rejects an otherwise supported number; or the policy fails after receiving a valid reading. Each explanation points to different evidence. The token in the file, the parser contract, and the point at which the error occurs help distinguish them.

Inspect host/replay.cpp. The adapter accepts a numeric value only when the whole token is consumed and the result is appropriate for an ordinary sample. It also has explicit special tokens for modeled missing and nonfinite input. The suffix C is not a unit annotation supported by this fixture format. Temperature units are part of the documented field meaning, not a suffix to append to each token.

Choose the smallest discriminating test

Make a copy of the fixture named fixtures/lab14-valid.csv. Change only 28C to 28. Run that copied fixture through the same executable path. If the valid token is accepted and the modeled sample reaches policy, the result supports the input-format explanation. You have changed the suspected cause while holding the toolchain and policy constant.

Do not “fix” the example by teaching the parser to ignore arbitrary suffixes. That changes the accepted language and can turn malformed input into a plausible reading. It also prevents the experiment from answering whether the original fixture violated the existing contract. A format change could be a legitimate future task, but it would need its own requirements and tests.

Do not replace the token with zero merely to avoid an error. Zero is a valid numeric measurement with a meaning different from missing or malformed input. Inventing a temperature hides the failure. If the scenario intends missing input, use the documented missing event representation; if it intends a numeric sample, supply that number.

After the copied fixture succeeds, rerun the original invalid.csv and confirm it still rejects the malformed token. That second check establishes that the correction did not weaken the parser. Preserve both command records: accepted corrected input and rejected original invalid input. Together they support a more precise conclusion than one passing run alone.

Use AI to compare explanations

Give the assistant the exact symptom record, the small fixture, relevant parser code, and the format requirement. Ask it to propose a discriminating test before editing source. For example:

The CP06 replay rejects line three of this synthetic fixture with exit two. The row contains 28C. Explain two plausible failure locations using the supplied adapter and policy code, then choose one small test that distinguishes them. Do not change the parser contract or invent a temperature. After the test, state what its result supports and what remains untested.

The assistant's proposal should describe a predicted observation for each explanation. “Try changing the input” is incomplete unless it says which change, which command, and how the result changes the diagnosis. The value of the request is that it makes the reasoning operational and reviewable.

When the result arrives, update the diagnosis. If the corrected token still fails, do not repeat the same edit in a different file. Check whether the command used the copied fixture, whether you are in the intended project folder, and whether the executable was rebuilt from the intended source. Evidence can contradict a plausible first explanation; that is progress when you preserve it.

A second defect: stale data treated as current

Now use the separate FAULT-stale checkpoint in another exercise folder. It is based on the CP07 normal/fault implementation and includes tests that Chapter 15 will explain. For this chapter, use its replay path and source inspection; you do not need to understand the full test harness yet.

The supplied mutation disables the expiry condition in Monitor::poll. A reading that should become stale can remain current. This is a different kind of failure from malformed input: the file is valid, the executable runs, and the output is valid JSON. The meaning of the output is wrong.

Run fixtures/stale-recovery.csv. At timestamp 1000, the last valid sample is current. At 4000 its age is exactly 3000 milliseconds, so it should still be current. At 4001 the age is 3001, and the correct output is FAULT/stale with a null temperature. The defective version fails that expected transition.

Write competing explanations: the timestamps were parsed incorrectly; the expiry condition did not run; or the formatter displayed an old value despite a correct policy state. Inspect the output's timestamp, state, error, and temperature together. A number alone cannot tell you which layer failed.

Trace the stale path without broad changes

Locate the call to monitor.poll(now) in the host adapter, then inspect the policy's poll method and the fail helper. The helper is significant: it sets FAULT, stores the error, marks the value unavailable, and resets the recovery count. A correct expiry path needs the whole transition, not just a different word in the output.

The fault checkpoint inserts a false condition into the expiry guard. In a review, name the precise trigger and consequence: when a current reading becomes older than the freshness limit, the disabled guard prevents fail(Error::Stale) from running. That explanation is supported by the code and the reproducer, rather than by a general suspicion about timing.

A tempting patch changes the printed state to FAULT whenever the timestamp looks old. That can hide the visible symptom while leaving current_ and the recovery counter wrong. The program might continue emitting a numeric stale value or recover incorrectly. Keep state transitions in the policy and representation in the formatter.

Restore the intended expiry guard in the disposable fault copy. Do not change the threshold constant, fixture timestamps, output schema, or recovery requirement. Then run the original stale-recovery sequence again. Check both the failing 4001 event and the later recovery events, because a patch can correct expiry while damaging what happens afterward.

LAB14 — Produce a defensible fault report

Allow about one hour. Submit two short records: the malformed-fixture diagnosis and the stale-policy diagnosis. Each record needs the starting state, original reproducer, competing explanations, chosen test, observed result, bounded correction, regression check, and remaining limits.

For the fixture record, retain invalid.csv unchanged and create the corrected copy. Show that the copied numeric row succeeds while the original malformed row still exits two. Explain why you did not change the parser or substitute a made-up temperature. Name the exact accepted token and its documented unit meaning.

For the stale record, begin from FAULT-stale, not a previously repaired folder. Capture the bad 4001 observation before editing. Correct the disabled expiry guard, preserve the diff, and rerun the same fixture. Expected corrected states are FAULT, NORMAL, NORMAL, FAULT, FAULT, FAULT, FAULT, NORMAL.

The 6000 missing event must interrupt recovery. At 7000, one valid reading is insufficient; only the next valid reading at 8000 allows NORMAL. Include those observations in the regression record. Checking only the first stale transition would leave an important part of the correction unexplored.

Cross-compile the repaired shared source with the unchanged target if the recorded toolchain is available, and save the actual result. A host correction and a firmware compile answer separate questions. If the current build has not been performed, leave that box open and identify the command still required.

Ask an assistant to draft the report from your raw evidence, then review its verbs. Replace “proved the sensor works” with a claim supported by the run, such as “the shared policy expired a synthetic sample at age 3001 milliseconds.” Remove a root-cause statement if the chosen test never actually distinguished the competing explanations.

Worked answer and recovery route

The fixture failure is caused by a token outside the defined input format. Correcting only the token in a copied fixture resolves that scenario while preserving strict rejection of the original. The stale failure is caused by a disabled expiry guard; restoring the policy transition corrects availability and recovery state. These conclusions are narrow enough to reproduce.

A good report includes failure evidence as well as the fixed result. A screenshot of a passing terminal alone cannot establish what was wrong or whether the original case was exercised. Keep the small text log and source diff even when you also capture a full screenshot for explanation.

If you lose track of edits, preserve the current folder and extract the fault checkpoint into a new one. Compare the files and repeat the original command. Do not delete useful work merely to get a clean-looking result. When Git is present, use the restore or revert operation appropriate to an uncommitted or committed change, as practiced in Chapter 7.

Review questions

Why record the process exit separately from the state field? FAULT can be the correct application output from a successful replay. A nonzero process exit describes an execution or input problem in a different channel.

What makes a test discriminating? Different plausible explanations predict different observations under a controlled change. A test that produces the same result under every explanation does little to select among them.

Why can changing only the output label be a bad stale-data fix? It can leave the policy's availability and recovery state incorrect. The visible word improves while the underlying transition remains broken.

When may you call an explanation a root cause? When the relevant evidence supports the mechanism, a bounded correction addresses it, and the original reproducer plus meaningful regression checks behave as required. A plausible narrative alone is insufficient.

Would an oscilloscope help with this fixture typo? It would not test the text parser's accepted language. Instruments are valuable when the uncertainty concerns physical signals or timing; choose evidence appropriate to the question.

Transfer to a real maintenance task

Decide when the investigation is finished before adding more experiments. For these exercises, completion means the original case is explained by a supported mechanism, the smallest correction addresses that mechanism, and the relevant nearby behavior still works. It does not require proving every possible property of the program. An assistant that keeps proposing unrelated improvements after those checks can make the result harder to review.

Write down useful questions that fall outside the bounded fix without silently treating them as new requirements. A physical driver's timeout, for example, may deserve a later task, but it does not explain why a CSV token contains a letter. This separation keeps the report accurate about both what was resolved and what remains. It also lets another engineer resume an unresolved question without repeating a completed diagnosis.

Take a recent failure from your own project and rewrite its description without a proposed cause. Include exact input, output, environment, and expectation. Name two explanations and one observation that would distinguish them. If you cannot reproduce the failure, record what information is missing before changing production code.

Use the assistant to reduce the experiment, not to conceal uncertainty. A small failed test can be more useful than a large speculative patch because it tells you which path deserves attention. The next chapter makes these checks repeatable and demonstrates that a test suite can reject a deliberately wrong change.