- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper claims that existing LLM cascade evaluation usually focuses on final answer, call histogram, and cost summary which can be unchanged even when the implementation refers to the target or future output. To mitigate this problem, they propose two layer audit which consists of ledger that test single trace and CPNI which replay cascade after fixing visible state and policy randomness while changing hidden state to verify a existence of hidden read. As a result, CascadeLedger can detect faulty cascade with high recall and a zero false-positive rate.
Critical strengths
None
Critical weaknesses
The existence of the problem is built on limited supporting evidence. Even though this paper is built upon the premise that cache-based routing evaluation can be contaminated by the hidden read, there is little evidence to suggest that this phenomenon actually occurs frequently or that it is difficult to solve without proposed method. Suggested natural fault does not contain the source of the project and no breakdown for the fraction of hidden read. In addition, the authors do not apply this method to existing cascade research or open source project.
Second, overall evaluation is self-fulfiling and the target detection faults are designed by authors. The designed faults itself have a trait that changing hidden value leads to the different action, which CPNI directly tries to test. In addition, I think that just a strict permission that restrict the router’s input as a visible state can solve the most of suggested problem, but the authors do not test this simple baseline. The novelty of overall pipeline is also questionable since each component looks like a direct application of existing testing method, noninterference and metamorphic testing.
Questions for authors
Can you suggest more clear evidence that this problem is frequently occurs and hard to solve without this method?
I think additional clarification for the fairness of each fault design and why simple restriction cannot be applied.