Give Every Decision Point a Shared Information Boundary
Series: Building Auditable LLM Systems · Part 10 of 11 An online evaluation is valid only when every decision point begins from a shared boundary on visible information. An LLM, a deterministic rule, and an online planner may use different data formats and computations. They must still base their decisions on semantically equivalent current state, observed history, and future-information boundaries. A method that sees more information may remain useful as a hindsight upper bound. It is no longer a fair online comparison and cannot by itself prove that the LLM made the wrong decision. ...