acceptodds
Under review as a conference paper at ICLR 2027

CascadeLedger: Counterfactual Auditing of Information and Resource Integrity in LLM Cascades

Abstract

We introduce CascadeLedger, a counterfactual execution-audit framework for budgeted language-model cascades. The protocol enforces call-before-read semantics, binds successful reads to service and cache identities, records availability and component-wise budget transitions, and freezes the event trace before any target or task scorer is joined. The central object is aggregate-equivalent trace validity: two executions can share a final answer, declared resource projection, and service histogram while differing in whether their routing decisions used only legitimately observable information. We complement single-trace invariants with counterfactual prefix non-interference (CPNI), which couples a visible prefix and policy randomness while changing only a declared hidden family. The counterfactual influence frontier (CIF) localizes the earliest violating prefix and a minimal hidden witness; projection-preserving adversarial mutation (PPAM) searches for invalid traces that remain acceptable to a non-counterfactual checker. Across frozen-cache, independently generated, and serving evaluations, the two-layer audit reports a clean false-positive rate of , detects of serialized faults and of unlogged hidden-read faults, and changes of corrupted quality–resource conclusions after deterministic recomputation. The online recorder and offline CPNI audit have separate overhead measurements. Learned routing and semantic verification provide downstream compatibility studies under the same execution contract.

Primary area:
reinforcement learning
Keywords:
Large Language Models, LLM Cascades, Agent Auditing, Counterfactual Analysis, Execution Trace Verification, Information Leakage Detection, Resource Integrity, Reliable AI Systems

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_T5wzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper claims that existing LLM cascade evaluation usually focuses on final answer, call histogram, and cost summary which can be unchanged even when the implementation refers to the target or future output. To mitigate this problem, they propose two layer audit which consists of ledger that test single trace and CPNI which replay cascade after fixing visible state and policy randomness while changing hidden state to verify a existence of hidden read. As a result, CascadeLedger can detect faulty cascade with high recall and a zero false-positive rate.

Critical strengths

None

Critical weaknesses

The existence of the problem is built on limited supporting evidence. Even though this paper is built upon the premise that cache-based routing evaluation can be contaminated by the hidden read, there is little evidence to suggest that this phenomenon actually occurs frequently or that it is difficult to solve without proposed method. Suggested natural fault does not contain the source of the project and no breakdown for the fraction of hidden read. In addition, the authors do not apply this method to existing cascade research or open source project.

Second, overall evaluation is self-fulfiling and the target detection faults are designed by authors. The designed faults itself have a trait that changing hidden value leads to the different action, which CPNI directly tries to test. In addition, I think that just a strict permission that restrict the router’s input as a visible state can solve the most of suggested problem, but the authors do not test this simple baseline. The novelty of overall pipeline is also questionable since each component looks like a direct application of existing testing method, noninterference and metamorphic testing.

Questions for authors

Can you suggest more clear evidence that this problem is frequently occurs and hard to solve without this method?

I think additional clarification for the fairness of each fault design and why simple restriction cannot be applied.

Reviewer_HvdDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CascadeLedger, a framework to address information leakage in LLM cascades. The framework records model calls, observations, and resource consumption to check execution validity. This paper addresses the problem that the router may accidentally see the ground-truth answer at the evaluation stage due to a program bug or other reasons. Its main component, Counterfactual Prefix Non-Interference (CPNI), tests whether changing hidden information, such as ground-truth answers or unrevealed model outputs, changes routing decisions. The authors also introduce methods to localize violations and generate adversarial execution traces. Experiments on synthetic mutations, independent fault generators, and serving workloads report high detection rates.

Critical strengths

The paper provides a method for checking execution traces in LLM cascades to address the possibility that the router in the cascades might have seen the ground-truth answer. The distinction between checking recorded traces and testing hidden information dependencies is useful. The authors also conduct fault-injection experiments and report detection rates, audit overhead, and downstream consequences.

Critical weaknesses
  1. Limited practical relevance. The paper does not convincingly establish that the problem it addresses is common or important in realistic LLM systems. In a standard deployment, the router does not have access to ground-truth answers or outputs from models that have not been called. These information boundaries can also be enforced directly through the system interface. The authors primarily motivate CPNI using hypothetical information leakage and evaluate it through injected faults. Although the paper includes natural implementation faults, it remains unclear how often the specific hidden-information violations targeted by CPNI occur in practice, or why existing access-control mechanisms would be insufficient. This substantially limits the practical significance of the contribution.

  2. Limited methodological novelty. The central idea of CPNI is to change hidden information while holding observable information and randomness fixed, then check whether the action changes. This is closely related to established non-interference testing and information-flow verification. The paper adapts these ideas to LLM cascades, but the underlying methodology appears largely incremental. I do not see a sufficiently strong conceptual or technical advance to justify the novelty.

Questions for authors

Why is counterfactual auditing necessary if the router can be restricted to accessing only legitimately observed information through a rule-based program? What violations can CPNI detect that cannot be prevented by a rule-based program?

Reviewer_6j2mOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work contributes the CascadeLedger framework to detect whether LLM respect the information restriction during LLM cascade routing. Running logs and counter-fact information are utilized to both verify and localize the restriction alignment. They validated this using large and small models from Qwen and llama on GSM8K and ARC-Challenge, and report approximately 99.8% fault detection in main and changed conclusions of a non-trivial number of corrupted comparisons. They also conducted additional evaluation on generated and native implementation faults.

Critical strengths
  1. Both running logs and hidden information are checked, this is quite meaning full for llm that are trained to fully utilize all information reachable.
  2. Good results on the tested setting
Critical weaknesses
  1. The semantic verifier implementation is under explained. It describes human labeling but did not specify the verifier model, prompt, train procedure, etc. This make the results hard to reproduce and the lack of confidence calculation also make it hard to interpret.
  2. The model combination setup is very limited and need justification for whether this framework still work when scaling up. Some example include heterogeneous model, workflow under tool calling, lomg context workflow, etc.
Questions for authors
  1. Can you explain in more detail on how the human labeling affect the semantic verifier?
  2. What if we use models from different families? Does this rely on how strong each model is? When we scale up to more intelligent model, including the router, how to justify that this framework still works? Does this still work with major LLM workflow nowadays under tool calling and long thinking/generation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.