2. How LLMs Work: The Useful Mental Model

This chapter gives you enough of a language-model mental model to prepare better requests and interpret answers without guessing at hidden reasoning. You will compare answers with and without a short reference passage, then record what the comparison establishes. Begin after Chapter 1 with a browser chat or the supplied original sample responses. No machine-learning mathematics or API account is required.

An assistant can explain a familiar programming idiom beautifully, then invent an option for a library it has never inspected. That is less mysterious when you distinguish generating a likely answer from looking up a verified record. The useful question is not whether the system sounds knowledgeable. It is what information was available, how the answer used it, and which claims survive an independent check.

1. Tokens and next-token generation

A token is a unit of model input or output. It can be a word, part of a word, punctuation, or another encoded unit, depending on the tokenizer and input. Do not equate one token with one English word. Code, identifiers, numbers, and unfamiliar strings can divide differently. Token counts matter because systems have limits on the material they can process and produce in an interaction. OpenAI key concepts

A practical simplified model is that a language model repeatedly predicts a next token using the preceding context, then continues from the growing sequence. Learned patterns can support explanations, transformations, and useful problem solving. The generation process does not automatically consult the current datasheet for a part number or execute a program whose output it describes. Those require supplied references or tools.

Think of a generated command as a proposed piece of text until there is evidence of execution. If an answer shows PASS, that string might be an illustrative expected result. When a terminal actually runs a test, the result belongs to a particular executable, input, and environment. Both kinds of text can look identical on a page. Our evidence labels distinguish them.

This mental model is intentionally limited. Modern systems may use several kinds of models, additional processing, retrieval, and tools. We do not need to infer their private implementation to use them well. We need to recognize that the answer depends on information and constraints supplied to a generation process, and that technical correctness must be checked against the relevant requirement and evidence.

2. Embeddings and attention, intuitively

An embedding is a numerical representation that allows a model or retrieval system to work with relationships among inputs. In ordinary language, related meanings can have useful relationships in that representation. It is not a built-in certificate that two similarly named components are electrically interchangeable. Similar names are precisely where a maker should be careful: a familiar family name can invite the wrong variant assumption.

Attention is a mechanism that lets a model weight relationships among elements of its input representation. The original Transformer paper introduced an architecture centered on attention mechanisms. This supports a conceptual picture of information from several parts of an input contributing to an output; it does not imply that every relevant sentence is always used correctly. Vaswani and colleagues, Attention Is All You Need

Imagine a short project note that states a threshold, unit, and fault behavior. The assistant can connect those details when answering a question about an invalid sample near the boundary. Now imagine burying that note among thousands of lines of unrelated logs and an older contradictory requirement. More material has been supplied, but deciding which statement governs has become harder. Good context is selected and organized, not simply large.

Figure D02 separates learning stages from the current interaction. Figure D03 shows the current request, relevant references, previous messages that are actually retained, and tool results entering a simplified context flow. Neither diagram represents a particular model's internal trace. Their purpose is to show where you can improve the information presented to the system.

D02. Training adjusts model parameters; inference uses the resulting model with current context. Supplying a project note is not the same as retraining.

D03. The current request, selected history, relevant references, and actual tool results enter bounded context. The generated answer still needs verification.

3. Training, post-training, and inference

Training adjusts model parameters using data and an optimization process. Post-training is further work that shapes how a trained model behaves, for example in following instructions. Inference is using the resulting model to process new input and produce output. These terms explain why putting a technical passage into a conversation can affect the next answer without being the same operation as retraining the model.

For the maker, the practical distinction is immediate. If you provide today's build error, that error can inform the current response. You have not thereby updated every future session's knowledge of your project. If you correct a pin mapping in one conversation, a later session may need the corrected project document again. Store durable facts and decisions in the repository, where both people and future tools can inspect them.

Similarly, a model's learned knowledge is not guaranteed to include the latest release of a library or your private circuit modification. Avoid asking it to infer the installed version from a product's general popularity. Supply the exact dependency record and relevant source. The more consequential the version difference, the more important it is to put that version into the task rather than assume the assistant will ask.

This also changes how you interpret a correction. A model can agree with feedback that is wrong. Agreement is not independent confirmation. If you say that a threshold should include equality, the real question remains what the requirement says and whether the implementation satisfies it. Feedback should point to evidence, not merely insist on a preferred answer.

4. Context windows and missing history

The context window is the bounded amount of information available to a model in an interaction. An application may manage long histories by selecting messages or summarizing earlier material. Different products implement this differently. A visible conversation is therefore not a promise that every previous detail remains available in its original form. OpenAI's conversation-state documentation describes context limits and techniques for managing long interactions. Conversation state

Treat critical constraints as project artifacts, not memories you hope the assistant retains. A short task brief can name the requirement document and test command. A handoff can state the current commit, relevant files, open question, and next action. Later chapters build those artifacts. For now, put the essential excerpt directly into the request and ask the assistant to distinguish supplied facts from assumptions.

An effective context packet has enough detail to decide the task and little unrelated material. Include the failing input, expected output, actual output, affected function, versions, and constraints. Omit a full day's log when three lines reproduce the symptom. Keep the full raw evidence locally, but supply the small relevant portion with its origin and any filtering stated. This makes the response easier for you to audit too.

If a conversation drifts, do not argue indefinitely with accumulated assumptions. Save the useful evidence, prepare a concise fresh brief, and start a new interaction. This is a controlled restart, not a trick for improving luck. You should be able to explain what information changed and why the new request is better defined.

5. Variation, hallucination, and confidence

Generated responses may differ across runs. Wording can vary without changing the technical meaning, and a repeated error can remain stable. Do not score quality by exact textual agreement. Compare claims, cited support, and whether the requested artifact satisfies its acceptance criteria. The OpenAI text-generation guide documents generation as model output controlled by input and model settings; it is not a guarantee of reproducible engineering truth. Text generation

We use hallucination here for a generated claim presented as factual when it is unsupported or false, such as a nonexistent API option or invented source section. The word names an output problem; it does not explain every cause. Your recovery is still practical: isolate the claim, inspect the authoritative material, and correct the resulting artifact.

Confidence language is a poor measurement instrument. “Definitely,” a detailed explanation, or an answer repeated by a second assistant can make a claim feel stronger without adding evidence. A second review can be useful because it may identify a missed case or alternative hypothesis. Its value comes from the check it proposes and the evidence it finds, not from counting votes among models.

Ask for a concise rationale, relevant file locations, and verification steps. You do not need a claimed transcript of hidden reasoning. A readable explanation can help you judge a result, but it is itself another output to evaluate. A test with a deliberately wrong comparison will teach you more about a boundary than a page of confident prose defending it.

6. Worked example: give the answer a source

Use this original chapter resource, BenchNote technical note. It is separate from CP00's shorter inclusive-comparison note. It describes a fictional teaching instrument called the BenchNote monitor; it is not a manufacturer specification:

BenchNote accepts a synthetic sample once every 1,000 milliseconds. A sample carries a validity flag and a temperature in degrees Celsius. An invalid sample produces FAULT and its numeric temperature must not select OK or WARN. For this introductory exercise, a valid sample at or above 28.0 degrees produces WARN; a valid sample below 28.0 produces OK. This simplified policy has no hysteresis. It is not the complete capstone requirement.

In a fresh chat, first ask: “How does the BenchNote monitor handle an invalid temperature sample?” Do not supply the note yet. A sound answer may say it lacks the specification. Another may invent a typical behavior. Save the actual answer and label this the without-reference run. Do not lead the assistant by saying that you want a particular fault state.

Then provide the complete original note and ask the same question with this addition: “Use only this note for product-specific claims. Identify what remains unspecified.” Save the answer as the with-reference run. Now the output can be checked against explicit sentences. It should identify FAULT and explain that the invalid number must not select the ordinary states. It should not assert a physical cause, an I2C address, or a recovery delay absent from the note.

These are expected response properties, not a transcript of a recorded chat. If the actual model does something different, preserve it and analyze the difference. A refusal to guess in the first run is a good result. A correct second answer demonstrates that this answer used the supplied material appropriately; it does not establish that every future answer will do so.

Make a table with each product-specific claim as a row. Add columns for answer without reference, answer with reference, supporting sentence, and unresolved detail. The key change should be traceability, not just length. A long answer with an invented restart timer is less useful than two correct sentences that acknowledge the missing timer.

SS02-01 preserves the actual baseline: the new chat conditionally reused earlier code context without a BenchNote rule. SS02-02 shows the supplied-note answer. SS02-03 is the instructor's comparison of that answer and its repeat; both leave recovery unspecified. These observations establish no reliability rate.

7. Lab LAB02: repeat and compare

Run the with-reference request again in a fresh conversation, or compare the two saved illustrative answers in your course resources if access is unavailable. Use exactly the same original note. Record whether each response states the fault behavior, the unit, the valid boundary, and the unspecified recovery behavior accurately. Separate different wording from different claims.

Now ask a new question: “How many consecutive valid samples are required to recover from FAULT?” The correct judgment is that the supplied introductory note does not say. Your deliverable is a comparison worksheet plus a proposed follow-up question for the requirement owner. Do not fill the gap with a common design pattern merely to complete the table.

Expected observations include at least one grounded claim and one explicit unresolved item. If the model invents a recovery count, annotate it as unsupported and point to the absence of such a rule. You have succeeded by detecting that problem. If both answers are accurate, explain why that does not prove universal reliability: you tested a small set of requests against one supplied note.

8. Failure and recovery: the convincing wrong answer

Suppose the assistant says the monitor holds its last state on invalid input, then provides an elegant justification about avoiding alarm chatter. That might be a defensible alternative design, but it contradicts the supplied exercise. Mark the statement against the sentence requiring FAULT. Ask for a corrected answer based on that sentence and for the proposed alternative to be labeled separately.

The recovery is complete when your saved artifact matches the source and preserves any decision still needing an owner. Do not silently alter the source to agree with the answer. When a requirement deserves reconsideration, record a proposed change and its tradeoffs. Engineering judgment includes changing requirements deliberately, but not accidentally through repeated paraphrase.

Completion checklist

• I can explain tokens, context, and inference without claiming access to hidden reasoning.
• I saved or examined answers with and without the original note.
• I compared claims rather than wording and identified missing information.
• I corrected a contradiction by referring to an exact requirement.

Review questions with answers

1. Does a larger context guarantee a better answer? No. Relevant information helps, but extra unrelated or contradictory material can complicate interpretation. Supply a clear, bounded packet and verify its use.

2. Does pasting a datasheet into chat retrain the model? It provides information for the current interaction. That is different from training model parameters. Keep lasting project facts in versioned documents and supply them again when needed.

3. Two answers agree on an undocumented pin. Is it verified? No. Agreement adds no primary reference or measurement. Locate the exact board documentation and inspect the relevant label before relying on the value.

Transfer exercise

Explain to another maker, in fewer than 150 words, why a chat is not a database of verified project facts. Include one example of helpful supplied context and one example of a claim that still needs a test. Your explanation should make the workflow more understandable without dismissing useful assistance.