15 — Test, Review, and Measure

A passing test is useful when you can explain what would make it fail. This chapter turns the normal/fault behavior from the previous chapters into repeatable checks and then deliberately breaks one comparison. You will see a successful compile followed by meaningful assertion failures, restore the source, and run the same tests again. The exercise shows what the tests establish and where their evidence ends.

Use CP07 in a fresh exercise folder. This checkpoint contains 36 native assertions for the already introduced normal/fault policy, freshness, recovery, scheduling, rollover, and basic output formatting. It does not contain the configurable threshold, hysteresis, or command parser from the Chapter 17 capstone. Do not import those later acceptance tests into an earlier lesson and treat their absence as a defect.

Begin with expected behavior

Before opening the test implementation, read docs/requirements.md. Write three predictions in plain language: a sample exactly 3000 milliseconds old remains current; a sample 3001 milliseconds old becomes stale; and one fresh valid sample after a fault is insufficient for recovery. These statements give the test an independent source of expected behavior.

Now predict an entire short sequence. Start FAULT/missing. Supply valid readings at zero and 1000: the first begins recovery and the second allows NORMAL. Poll at 4000: the last reading is exactly 3000 milliseconds old and remains current. Poll at 4001: it becomes FAULT/stale with no current temperature. The expectation comes from the requirement, not from copying the comparison already in the source.

This independence is easy to lose when asking an assistant for tests. If the prompt only says “test this code,” the answer may encode the current implementation's assumptions. Give the requirement and the code, ask for a case that would distinguish nearby wrong implementations, and review the expected value separately from the mechanics of the test.

An independent expected value need not come from a second person. It can come from a requirement table, an explicit state transition, or a separately reasoned example. The point is to avoid using the same expression to calculate both the implementation's result and the test's expected result.

Execute the actual shared policy

Open scripts/common.cmd. The native build lists the policy and output source files from firmware/sensor-monitor. Open the Arduino sketch directory and confirm those are the same files compiled into the sketch. The host test does not maintain its own alternative version of normal/fault behavior.

Run the test wrapper from PowerShell:

.\scripts\test.cmd
$testExit = $LASTEXITCODE

For the unmodified CP07 checkpoint, the recorded host reference result is PASS 36 assertions; 0 failures; synthetic host tests, with exit zero. Your actual run should be saved with its compiler version and source identity. If a different checkpoint reports a different count, inspect the stage before deciding whether something failed. CP09's full capstone suite has 114 assertions and covers additional behavior.

The wrapper compiles before executing. A compiler failure should stop the process before a test result is claimed. Likewise, an executable that cannot start is not a passing test with missing text. Record the exit and diagnose the documented compiler/runtime location. The .cmd wrappers use a process-local PATH for the native runtime and require no PowerShell execution-policy change.

Read the output, not just the color of a terminal badge. A nonzero exit and a named failing assertion tell you which observation disagreed with the expectation. A compiler error, loader error, failed assertion, and intentionally rejected fixture are distinct events. Keeping their outputs together but clearly labeled makes later diagnosis much easier.

Read a small test as a story

In tests/test_monitor.cpp, find the basic test function. It creates a fresh Monitor, checks its starting state, supplies readings, advances the injected clock, and inspects snapshots. The sequence is short enough to read beside the requirement. No physical sensor or serial connection is needed to produce its inputs.

The CHECK macro increments an assertion count and records a failure when an expression is false. Its line-number report is a pointer into the exact test source used for that run. Line numbers may change after you add a test, which is why the source identity belongs in the evidence record.

Consider these calls from the sequence:

p.sample(1000, 21.0f);
p.poll(4000);
CHECK(p.snapshot(4000).current);
p.poll(4001);
CHECK(p.snapshot(4001).error == Error::Stale);

The numbers are injected timestamps. The test need not wait three real seconds between calls. It asks whether the policy computes the required age and transition for those inputs. It does not measure how promptly an MCU interrupt or USB transaction would occur.

Notice that the test checks both availability and error meaning. A weaker test might check only that a line was printed. That would miss a stale numeric value mislabeled as usable. Choose observations that distinguish the required behavior from a plausible wrong implementation.

Cover failure and recovery together

Invalid-input tests should check what happens after the fault as well as when it begins. CP07 supplies missing input, NaN, and positive infinity. It confirms that invalid data is not current, then exercises valid/invalid recovery sequences. A later valid sample cannot simply erase all preceding state.

Recovery begins with a fresh valid reading but remains in FAULT. Another invalid reading resets the count. Two consecutive fresh valid readings then allow NORMAL. A long stale gap between otherwise valid readings also breaks the sequence. These cases guard against a counter that continues accumulating “good” samples across unrelated failures.

The test should make that distinction visible in its inputs. “Recover after a fault” is too broad if it only sends two valid samples immediately after one missing event. Include the interruption that an incorrect implementation could mishandle. Each additional case earns its place by distinguishing a meaningful error.

Do not turn every possible float value into a separate named test. Select classes of behavior: finite values, missing data, nonfinite results, freshness equality, stale age, and recovery interruption. Within a class, choose examples that exercise an important boundary. This keeps the suite understandable while protecting the decisions the project actually makes.

Understand rollover arithmetic

The clock values are unsigned 32-bit integers. The implementation calculates elapsed time by unsigned subtraction. Under the documented chronological-call assumptions, that handles a single wrap from a large timestamp back to zero. The assumption about call ordering and maximum gap remains part of the contract; arithmetic alone cannot infer an arbitrary number of missed wraps.

CP07 tests a reading at a timestamp one second before wrap and another at zero. It then checks current data at 3000 and stale data at 3001. These cases prove the implemented arithmetic for the injected sequence. They are much more useful than an unsupported comment saying “rollover safe.”

The scheduler has related checks: immediately due at startup, not due too early, due at the interval boundary, and only one acquisition for a delayed turn. A long pause does not cause several identical catch-up acquisitions at one timestamp. That behavior is a design choice which the tests make explicit.

If you transfer this pattern to a different clock type, revisit its width, signedness, ordering assumptions, and expected maximum gap. Do not paste an unsigned-subtraction idiom into code whose inputs have a different meaning. Ask the assistant to explain those assumptions before proposing a portability change.

Prove a boundary test can reject a defect

Make a new disposable CP07 copy for the mutation exercise. In Monitor::poll, the correct guard expires a sample when its age is greater than kStaleAfterMs. Change only that comparison from > to >=. Do not change the requirement or its expected test result to agree with the defect.

This mutation is deliberately small and compiles successfully. It makes the policy expire a reading at exactly 3000 milliseconds, one millisecond earlier than specified. A test that checks only 2999 and 3001 would miss it. The equality case exists to distinguish these nearby implementations.

Run the same test command. In the verified instructor host exercise, the mutation compiled, then the 36-assertion suite exited one with two failures: the ordinary equality check and its rollover counterpart. Restore the original comparison, rebuild, and run again; the same suite returns to zero failures. Those before/after receipts are software evidence from injected inputs, not a physical timing measurement.

The number of failures is secondary to their meaning. Another test layout may report the same defect with a different count. What matters is that the unmodified requirement-based test rejects the wrong behavior, that the failure is not merely a syntax error, and that restoring the source resolves the original case.

Review memory and output boundaries

The policy uses a small fixed state: the current temperature, timestamp, state/error values, recovery count, and availability flag. Inspect how failure changes those values together. A stale transition that changes only the state enum can leave a numeric value marked current, which is why the tests inspect more than one field.

The formatter writes into a bounded buffer and checks whether the complete output fits. CP07 includes a deliberately small buffer case that must report failure. This is a functional check on the formatting API's contract, not a measurement of total firmware memory use. The native executable's address space does not resemble the MCU's complete memory layout.

When reviewing a buffer change, examine both capacity and result handling. A larger buffer can hide a truncation symptom while callers continue ignoring a failed formatting result. A smaller buffer can be correct if the maximum output and failure behavior are understood. The source and tests should explain why the chosen size is sufficient for the current schema.

The capstone later introduces a bounded command buffer. Do not add a parser to CP07 merely to create more memory tests. Test the boundaries of the functionality present at this stage, then extend the suite with the feature that creates a new risk.

Combine tests, replay, and compilation

Native assertions are precise but terse. Replay output provides a readable sequence connecting timestamps, temperatures, states, and errors. Use both: run the basic suite, then run fixtures/stale-recovery.csv and explain the 4000, 4001, 6000, 7000, and 8000 events in ordinary language.

Cross-compilation asks another question: does this source build for the selected firmware target with the recorded dependencies? Save the actual command, full target, exit, and artifact identity. A native pass cannot substitute for that build, and a firmware build cannot substitute for the native behavior checks. The final CP09 Windows fresh-clone build exits zero, with 362,508 program bytes and 54,952 global-variable bytes; those measurements identify that release and do not substitute for this CP07 mutation exercise's build.

Physical timing, sensor accuracy, electrical compatibility, and upload/serial behavior need evidence of their own. List those as unperformed when no bench session occurred. You can complete the core course without hardware, but you cannot relabel a software run as a physical observation to make the report look complete.

An optional automation can run the same commands after a source change. Keep it thin: checkout, identify dependencies, compile, execute, preserve results, and fail on a failed required command. Automation is valuable because it repeats a known check, not because a green badge makes a weak check stronger. Do not add accounts, paid runners, or a new framework merely for this lesson.

LAB15 — Detect and explain a wrong comparison

Allow sixty minutes. Record a clean CP07 test result and the expected equality rule. Create a disposable mutation copy or branch, change only the expiry comparison to >=, and run the unmodified suite. Save the compiler result separately from the failed assertions. Explain which input distinguishes the wrong comparison from the requirement.

Restore the correct source and run the suite again. Inspect the diff to confirm that the mutation is gone. Run the stale-recovery fixture and verify that 4000 remains current while 4001 is stale. Cross-build the restored source when the recorded target toolchain is available and preserve the actual receipt.

Add one independently motivated test of your own. A suitable example is negative infinity: the requirement rejects nonfinite data, while the existing examples already cover NaN and positive infinity. State the expected state, error, and current-data result before writing the assertion. Do not alter production behavior if the new test confirms an already supported case.

Write a verification note with three parts: checks executed and their source identity; observations and their interpretation; remaining physical and environment-specific limits. Include the deliberate mutation failure. A report containing only the final pass omits the evidence that your chosen test could detect the targeted defect.

Worked answer and review questions

The wrong comparison expires a current sample at the inclusive freshness boundary. The test must keep the 3000-millisecond expectation unchanged and reject that implementation. Restoring > returns the suite to the required behavior. The added nonfinite case should enter FAULT/nonfinite and mark temperature unavailable; its assertion supplements the existing class of invalid-input checks.

Why is a compiler error insufficient for the mutation demonstration? It proves the source cannot build, but it does not show that the behavioral test distinguishes a wrong implementation. The seeded comparison must compile and then fail its meaningful assertion.

What is wrong with calculating the expected state using the same production function? The test can reproduce the implementation's error and agree with it. Derive the expectation from the requirement or a separate model of the intended sequence.

Does an injected 3001-millisecond timestamp measure device latency? No. It supplies a value to test arithmetic and policy. Physical response latency needs a defined measurement setup and an observed event.

Is an assertion count a quality score? No. A small suite that detects the relevant fault can be more useful than many assertions checking superficial details. Understand which incorrect behaviors the cases distinguish.

Carry the method forward

When a new test fails, first read its expectation and setup. The implementation may be wrong, but the test may also have supplied an impossible ordering, selected a different checkpoint, or asserted behavior that was never required. Preserve the failure while checking both sides of the contract. The goal is not to defend the test merely because it was written first; the goal is to make the requirement and executable behavior agree for a reason you can explain.

Review test changes with the same care as production changes. Deleting the equality assertion or changing its expected value can make the mutation appear fixed without correcting policy. A passing report after such an edit needs a clear requirement-based justification. When an assistant changes tests and implementation together, inspect the two diffs separately and verify that the original intended behavior has not moved unnoticed.

Choose one boundary in your own project: a buffer length, timer interval, integer range, or recovery count. Write the equality behavior explicitly, then choose values immediately around it. Ask an assistant for a bounded test and review the expected results before accepting the code. Prove the test rejects one plausible wrong implementation in a disposable copy.

Keep the test's purpose readable. A future maintainer should know which requirement would be lost if the case were deleted or weakened. Chapter 16 will package this style of review and evidence into the second skill and a handoff that works without the old chat history.