- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the inference cost of an LLM-based PR security review system deployed in an enterprise production environment. The system generates candidate findings using multiple LLM detection lenses and deterministic scanners, and a multi-stage adjudication cascade then makes a keep/drop decision for each candidate. Using production telemetry, the authors decompose cost by execution path and find that, on average, the code block given to the judges contains 3.9 times as many changed files as the candidate findings cite. The authors then study two cost interventions: cited-file anchoring (the anchor policy), which restricts the judge's context to the files cited by the candidates, and cost-aware early stopping of the adjudication cascade. They conclude that context pruning, stage stopping, and security correctness are distinct dimensions that require independent empirical validation.
Critical strengths
The research question addressed in this paper has practical significance, and the data are derived from deployments in real-world production environments. Empirical data on the cost structure of multimodel inference are extremely limited in the existing literature. Therefore, the findings of this study and the suggested cost-saving strategies provide valuable insights for the industry.
The authors measured the noise floor by repeating the calls. By asking the same prompt again, they quantified the model’s own fluctuations, allowing them to distinguish the anchor effect from the model’s inherent randomness. The design of this controlled experiment on LLMs is commendable.
Critical weaknesses
The overall savings of 15.37%(15.4%) from early exiting are mentioned repeatedly in the Abstract, Introduction, Figure 4, Section 7, and Conclusions. If I understand correctly, this calculation appears to be based on the following data: the early exiting trials accounted for 59.34% of the baseline adjudication costs, and the reduction in costs within that group was 25.9% (0.5934 × 25.9% ≈ 15.37%). However, other data in the paper point to approximately 5.4%. For example, in Table 5, the baseline cost share for the early exit group is 20.7%, not 59.34%. 20.7% times 25.9% is about 5.4%. In Table 4, the cost index for the cost-aware strategy is 94.6 and the teacher is 100, corresponding to a 5.4% savings. But the “Reduction” column in the same row lists 15.37. Why is there this inconsistency in the data?
The fact that anchor saves tokens is a very intuitive phenomenon, and this can be predicted. The key lies in whether the deleted context contains evidence necessary for the decision. Prior papers have already demonstrated that LLMs are highly sensitive to the content of prompts.
I did not find any measures of correctness in this paper. The evaluation merely checks whether the results match those of the original full-context judge, which makes it impossible to determine whether the changes in judgment caused by the anchor are errors or corrections. Under anchor, what the judge can see is entirely determined by the references in the detection lens, and the content of the PR read by the lens is controlled by the attacker. If a reference is omitted or misled by injected content in the code or comments, all downstream judges will be unable to see critical mitigation code or sinks.
Questions for authors
-
Could the authors please explain the data issue?
-
Could the authors discuss the security risks?
-
The paper uses tokens, characters, and dollars to measure savings in different sections, but these three metrics are not equivalent. Could the authors standardize these metrics rather than using them interchangeably?
Just Some Suggestions:
-
The paper alternates between "stop", "exit", "halt" and "truncate" for the same mechanism, and between "cost cut", "reduction", "saving" and "cost index" for the cost metrics. I suggest defining each term once and using it consistently. In addition, the number of decimal places in the data should be consistent.
-
Several captions run to too many lines and restate results and interpretation already given in the text. Shorter captions that describe only what is shown would improve readability.
-
The content of Appendix E should actually be included in the main text. And there is a formatting issue in Appendix D that may lead readers to mistakenly believe this section is empty.