Evaluation

Series: Building Auditable LLM Systems · Part 2 of 11 An LLM system that learns from historical experience and acts online has two different jobs: produce reusable experience from past evidence; use authorized experience to make a current decision. These jobs may reuse the same engineering layers, but they require different information permissions and prompt responsibilities. The clean boundary is simple: offline processing produces memory; online control consumes it read-only. Why Mixing Offline and Online Stages Is Dangerous When extraction and execution are mixed, experimental boundaries become difficult to defend. ...

Series: Building Auditable LLM Systems · Part 4 of 11 Separate scenario design for an LLM decision system into two layers. The first layer generates worlds that satisfy the physical, operational, and lifecycle constraints of the domain. The second layer organizes those worlds into named task families that define what the model should learn, where it should be tested, and which claims the evaluation can support. These layers answer different questions: ...

Series: Building Auditable LLM Systems · Part 5 of 11 The prompt is the model’s decision interface, not a place to concatenate everything the system knows. A well-designed prompt follows the order of the decision. It explains the task, the action boundary, the current facts, relevant history or external experience, the requested justification, and the output contract. Keep content with different sources and responsibilities in distinct blocks. This structure makes it possible to know what the model actually saw, why it made a judgment, and whether two methods were compared under the same prompt contract. ...