acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Diff: Cost-Aware Agentic Traversal for Industrial Grade Security Review

Abstract

When state-of-the-art LLMs review code for security vulnerabilities, the true cost bottleneck is not finding bugs: it is adjudicating them to ensure minimal false positives. In production scanning, candidates generated by the scanners - probabilistic as well as deterministic - trigger multi-stage deliberation cascades (cross-vendor debate, rebuttal, and arbitration) where adjudicators read exhaustive repository context to prevent developer fatigue from false alarms. Strikingly, telemetry reveals that judges read 3.9× defects trigger multi-stage deliberation cascades (cross-vendor debate, rebuttal, and arbitration) where adjudicators read exhaustive repository context to prevent developer fatigue from false alarms. Strikingly, telemetry reveals that judges read 3.9× more changed files than candidate findings cite, burning over two-thirds of their token budget on unreferenced code. Grounded in reinforcement learning and sequential decision processes, we formulate cost-aware agentic scanning across two graphs: casting context retrieval as information acquisition over an exploration graph, and cascade termination as an optimal stopping policy over an adjudication graph. Along the first, anchoring judges strictly to candidate-cited evidence slashes judge inference costs by 44% to 58% across repository-disjoint replications. Along the second, sequential cost-aware stopping halts adjudication on a quarter of production runs, cutting their cost by 25.9% (15.4% across all traffic), while preserving 99.86% of final decisions and 99.2% of verified teacher keeps. Crucially, our evaluations reveal that cost optimization is not a free lunch: cited-file anchoring shifts verdicts beyond the judge’s own repeat variability, while agentic navigators fail to outperform character-capped random retrieval. This study delivers the first end-to-end empirical decomposition of cost-fidelity dynamics in deployed agentic review, proving that context pruning, policy-driven stopping, and security correctness cannot be treated as interchangeable optimizations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
llm, codegraph, multi-agent adjudication

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CzfoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the inference cost of an LLM-based PR security review system deployed in an enterprise production environment. The system generates candidate findings using multiple LLM detection lenses and deterministic scanners, and a multi-stage adjudication cascade then makes a keep/drop decision for each candidate. Using production telemetry, the authors decompose cost by execution path and find that, on average, the code block given to the judges contains 3.9 times as many changed files as the candidate findings cite. The authors then study two cost interventions: cited-file anchoring (the anchor policy), which restricts the judge's context to the files cited by the candidates, and cost-aware early stopping of the adjudication cascade. They conclude that context pruning, stage stopping, and security correctness are distinct dimensions that require independent empirical validation.

Critical strengths

The research question addressed in this paper has practical significance, and the data are derived from deployments in real-world production environments. Empirical data on the cost structure of multimodel inference are extremely limited in the existing literature. Therefore, the findings of this study and the suggested cost-saving strategies provide valuable insights for the industry.

The authors measured the noise floor by repeating the calls. By asking the same prompt again, they quantified the model’s own fluctuations, allowing them to distinguish the anchor effect from the model’s inherent randomness. The design of this controlled experiment on LLMs is commendable.

Critical weaknesses

The overall savings of 15.37%(15.4%) from early exiting are mentioned repeatedly in the Abstract, Introduction, Figure 4, Section 7, and Conclusions. If I understand correctly, this calculation appears to be based on the following data: the early exiting trials accounted for 59.34% of the baseline adjudication costs, and the reduction in costs within that group was 25.9% (0.5934 × 25.9% ≈ 15.37%). However, other data in the paper point to approximately 5.4%. For example, in Table 5, the baseline cost share for the early exit group is 20.7%, not 59.34%. 20.7% times 25.9% is about 5.4%. In Table 4, the cost index for the cost-aware strategy is 94.6 and the teacher is 100, corresponding to a 5.4% savings. But the “Reduction” column in the same row lists 15.37. Why is there this inconsistency in the data?

The fact that anchor saves tokens is a very intuitive phenomenon, and this can be predicted. The key lies in whether the deleted context contains evidence necessary for the decision. Prior papers have already demonstrated that LLMs are highly sensitive to the content of prompts.

I did not find any measures of correctness in this paper. The evaluation merely checks whether the results match those of the original full-context judge, which makes it impossible to determine whether the changes in judgment caused by the anchor are errors or corrections. Under anchor, what the judge can see is entirely determined by the references in the detection lens, and the content of the PR read by the lens is controlled by the attacker. If a reference is omitted or misled by injected content in the code or comments, all downstream judges will be unable to see critical mitigation code or sinks.

Questions for authors
  1. Could the authors please explain the data issue?

  2. Could the authors discuss the security risks?

  3. The paper uses tokens, characters, and dollars to measure savings in different sections, but these three metrics are not equivalent. Could the authors standardize these metrics rather than using them interchangeably?

Just Some Suggestions:

  • The paper alternates between "stop", "exit", "halt" and "truncate" for the same mechanism, and between "cost cut", "reduction", "saving" and "cost index" for the cost metrics. I suggest defining each term once and using it consistently. In addition, the number of decimal places in the data should be consistent.

  • Several captions run to too many lines and restate results and interpretation already given in the text. Shorter captions that describe only what is shown would improve readability.

  • The content of Appendix E should actually be included in the main text. And there is a formatting issue in Appendix D that may lead readers to mistakenly believe this section is empty.

Reviewer_gmKaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the cost of the "adjudication" stage (deciding whether a candidate vulnerability is kept or dropped) in a deployed multi-model security code-review system. Using four days of production telemetry from one organization, the authors find that in runs that invoke judges, adjudication accounts for 53.8% of model cost, whereas across all traffic detection accounts for 53.3% and adjudication only 23.9%. Judge B reads on average 3.9 times as many changed files as the candidate findings cite. The authors propose two cost-reduction measures. The first gives the adjudicator only the files cited by the candidates (Anchor). On two samples of 120 PRs each, it cuts cost by 58.5% and 44.0%, but verdicts change relative to full context by 8.5% and 7.9%, which is 3.3 and 3.9 percentage points above the judge's own repeat-call variability. The second uses a logistic-regression model to predict the final decision of the full cascade so that adjudication can stop early. On 210 test runs it reports a 15.37% saving in adjudication cost, 99.86% decision agreement, and one miss among 127 teacher keeps. All "fidelity" measures use the system's own decisions as the reference, not independent vulnerability ground truth.

Critical strengths
  • Real production data and honest reporting. The paper provides a real cost breakdown of a multi-model cascade. It uses repeated calls on the same prompt to estimate the judge's own variability, which separates changes caused by cost reduction from noise. It also runs a repository-disjoint replication under frozen protocols and reports repository-clustered bootstrap intervals. This level of rigor is rare in industry papers and is useful to practitioners.

  • Valuable negative results. The paper shows that an LLM navigator does not outperform random file selection under the same character budget, and that an untrained confidence threshold saves more cost but misses more findings that should be kept. This reminds readers not to look only at cost and overall agreement, but to examine retention of "keep" findings separately.

Critical weaknesses
  • Core numbers are internally inconsistent. The main text states that early-exit runs account for 59.34% of baseline adjudication cost, from which the 15.37% saving is derived. However, Appendix Table 5 gives 20.7% (which corresponds to about 5.4%), and the cost index of 94.6 in Table 4 and the 93.3% call retention in Appendix B also point to a saving of around 5%. Likewise, the 19.26% saving of the confidence threshold is inconsistent with Table 4 (index 90.7, i.e. about 9.3%) and with the per-repository figures in Appendix B (10.6% and 6.9%).

  • No external ground truth, and security quality is not measured. All metrics measure agreement with the system's own decisions. There is no false-positive rate, precision, or recall, even though the paper describes a feedback process with developer replies and expert-labeled findings. No replayed finding had completed human verification. Therefore the claims of "security correctness" and "safely stopping early" in the abstract and conclusion are not validated.

  • Weak statistical evidence and concentrated data. Training uses only one day of data and the development set has only 67 runs. All data come from the same organization within the same four-day window, while the paper itself acknowledges that the reference system keeps drifting.

  • Other issues. Table 2 and Table 3 report different values for the same metric (Sample 1 anchored "changed": 8.7 vs. 7.9). The appendix attributes this to a difference in evaluation convention, but the main tables should be consistent. Data, code, and prompts are not released, so the results cannot be reproduced.

Questions for authors

See the Weaknesses section above.

Reviewer_gJ7fOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on the cost of LLM security review in a production pipeline. The main cost is not finding candidate bugs but adjudicating them, where several LLM judges debate each finding to reduce false positives. This matters because these cascades are expensive, and the judges read much more code than the findings cite. The authors investigate an exploration graph and adjudication graph to reduce such cost. Experimental results on replayed production runs show that cited file anchoring cuts that judge’s cost by 44% to 58%, but changes 3 to 4 points more verdicts than the judge’s own repeat variability. Early stopping cuts adjudication cost by 15.37% while matching 99.86% of the cascade’s decisions and 99.2% of its kept findings.

Critical strengths
  • This paper investigates a security review problem that is often ignored: the cost for LLM reviewers. It studies a deployed system to reduce the cost, especially how much context and how many stages the adjudication needs.
  • Experimental results show that the exploration graph and adjudication graph can reduce the judge and adjudication cost.
Critical weaknesses
  • The methods are simple and straightforward. Anchoring keeps only the cited files. Early stopping is a logistic regression with a threshold. Both are standard ideas from context pruning and learning to defer. The main contribution of this paper is more about a case of an industrial code review system, instead of a generic system or learning problem.
  • There are some overstated claims in the abstract, such as "grounded in reinforcement learning" and "optimal stopping policy”. There is no RL or discussion of the optimality of the stopping policy.
  • In the experiment part, the evaluation metric is not reliable. There is no ground truth or human verification of the decision. It is unclear whether the decision reveals a real vulnerability or just a false positive. Moreover, the reference itself is noisy. Asking the same judge again on the same prompt changes 3.9% to 5.2% of verdicts. It also loses 4.3% to 7.8% of its own kept findings (Table 2). With this much noise, a 99.2% keep retention target is hard to interpret.
  • The evaluation is limited to a 4-day window in 1 organization. Both interventions (Anchoring and early stopping) are tested on a single model configuration.
  • There is no simple baseline or comparison. For example, using a cheaper model as Judge B, reducing context tiers further, or removing a whole stage. Stopping is compared only to untrained gates, not to standard learned deferral or calibrated cascade methods.
  • One repository supplies 65% of the test runs and the only missed keep (§7). Outside this repository, the policy saves only 3.15%, compared with 15.37% overall. This suggests the saving depends heavily on one repository. So the 15.37% figure may not generalize.
Questions for authors

For reproducibility, will you be able to release any kind of things, such as data, code, or prompts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.