acceptodds
Under review as a conference paper at ICLR 2027

Continuous Oversight, Selective Deliberation: Event-Triggered Metareasoning for Judgment Revision

Abstract

Reliable judgment requires ongoing scrutiny while maintaining a responsive decision process: a strong reviewer cannot examine every request, so a system must decide which initial predictions are worth reviewing. Drawing on the metareasoning distinction between monitoring and control, we propose MONITOR, which scores every initial judgment and, under a prescribed review budget, triggers review for the requests with the highest expected correctness gain; the gain is predicted from pre-review information and learned from paired initial and reviewed outcomes. The monitor thus decides whether to investigate further, and the reviewer decides whether to change the outcome. We study two review settings: independent review of the original input, and review conditioned on the initial judgment and its rationale. We evaluate a single shared allocation rule for LLM-to-LLM and statistical-model-to-LLM review, extending the rule beyond LLM pairs, and compare both review settings on three judge benchmarks. Selection improves on random review at matched rates: at 15% review on HaluEval, MONITOR reaches 98.40% accuracy under independent review and 98.80% with the initial judgment, compared with 94.11% and 94.26% for random review. Independent review matches or exceeds full-review accuracy with few calls: 5% review on HaluEval yields 97.80% accuracy versus 97.00% for full review, while 20% on RAGTruth yields 87.50% versus 86.25%. In a single-GPU deployment on 128 HaluEval requests, selective review reaches 92.97% accuracy with 21.09% strong-model calls, compared with 92.19% for direct strong-model use; on 16 timed requests, the nominal 15% policy matches direct strong-model accuracy (87.50%) with mean final-decision times of 3.420 versus 9.175 seconds.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-Judge, Automated Evaluation, Adaptive Reasoning, Decision-Making, Error Correction.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LZCnOpenReview ↗
Rating
2(+1)
Confidence
2(−1)
Informativeness
2(+1)
Presentation
2(+1)
Soundness
1
Read review
Paper summary

The paper proposes MONITOR, a lightweight model that predicts the net correctness gain from reviewing an initial judgment and prioritizes requests under a limited review budget. It learns from paired initial and reviewed outcomes, accounting for both corrections and harmful revisions. Experiments with LLM and statistical-model initial predictors show improvements over random review in the main evaluations. In selected settings, reviewing only 5–20% of requests matches or exceeds full-review accuracy, while a small deployment experiment also demonstrates reduced average latency.

Critical strengths

The paper addresses a practical problem: allocating a limited budget of strong-model reviews to improve the accuracy–cost trade-off.

Critical weaknesses
  • The paper is difficult to read and understand. It exhibits strong signs of AI-generated writing, and the presentation frequently obscures the underlying ideas.
  • The proposed method offers very little insight. The paper does not convincingly justify the introduction of the monitor or demonstrate its practical value.
  • The evaluation is limited in scope: it focuses exclusively on judgment tasks, and the baselines consist only of overly simplistic methods. To convincingly establish the method’s effectiveness, the authors should broaden the evaluation to more diverse tasks and compare against stronger, more relevant baselines.
Questions for authors
  • Could the authors provide a clearer, more concise description of the full pipeline, explicitly separating monitor training, review selection, and judgment revision? The current presentation makes it difficult to identify the essential algorithmic steps.
  • What is the substantive methodological contribution beyond existing learned deferral and model-cascading approaches, particularly Diff-01? Could controlled ablations isolate the contributions of the gain-prediction objective, input features, and regression architecture, and clarify when the monitor provides meaningful practical benefits?
  • How well does the approach generalize beyond the judgment and classification tasks evaluated here? Could the authors broaden the evaluation and include stronger baselines that also observe the initial response, using matched information access and review budgets?
Reviewer_m8uzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a framework for efficient judgment. The proposed method first train a Monitor and then use it to decide whether or not to change the initial judgment by either an LLM judge or a Naive Bayes. If an additional reviewing is triggered, then a strong LLM judge reviews the initial judgment and outputs a final decision. Evaluating on several benchmarks, the proposed method outperforms a random reviewing and a predictive router (Ong et al., 2025), supporting the effectiveness.

Critical strengths
  • This work clearly formulates the problem and mathematically describe those, and the experimental results show superior performance over the baselines authors use.
  • Additional analysis in Section 4 further supports the effectiveness of the proposed method, which makes me leaning towards to accept this work.
Critical weaknesses

However, the following concerns hesitate me to accept this work.

  • Authors only compare the method with Ong et al. (2025), but more relevant works are discussed in Introduction (e.g., Chen et al. (2024b), Jung et al. (2025), Hu et al. (2025)). Is there any particular reason the authors do not compare with these works? If you can compare, then please add those results to prove the effectiveness of the proposed method.
  • I'm a bit worried that large amount of method details are presented in Appendix B.1, which should be in the main paper I think.
Questions for authors
  • Abstract can be written more clearly. Now, sentences are hard to understand before reading Introduction. For instance, there is no explanation of what LLM-to-LLM and statistical-model-to-LLM are. Also, the first few sentences do not smoothly connect relevant works to this work.
  • line 95-97: why do we need independent review of the original input? i guess "review" should target the initial judgment, but authors also add this. Is there any reason? Additional justification of this choice would help readers for better understanding.
Reviewer_gk86OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MONITOR, a selective review framework that determines which initial model judgments should be reviewed by a stronger LLM under a limited review budget. The framework predicts the expected correctness gain of review by considering both beneficial corrections (rescue) and harmful revisions (harm). It supports independent review and review conditioned on the initial judgment and rationale. Experiments across LLM-to-LLM and statistical-model-to-LLM settings demonstrate improvements over random review and predictive routing in several benchmarks. The authors further report that selective review can achieve competitive accuracy with fewer strong-model calls and reduced inference latency.

Critical strengths
  • The paper addresses an important efficiency–accuracy trade-off in LLM-based judgment systems by explicitly accounting for both beneficial and harmful revisions.
  • The proposed framework accommodates different initial predictor families and review settings through a shared expected-gain objective.
  • The paper investigates multiple benchmarks, review budgets, model pairs, and review settings, complemented by additional ablations and deployment-level efficiency measurements.
  • MONITOR demonstrates substantial improvements over random review at matched review rates and, in selected configurations, achieves higher accuracy than reviewing every request.
Critical weaknesses
  • MONITOR's expected-gain formulation closely resembles existing learning-to-defer and quality-aware routing approaches, limiting its methodological novelty. The authors should clarify its contributions beyond prior work.
  • MONITOR uses more information than the predictive-routing baseline(predictive routing estimates model suitability from query features before generation), raising concerns about comparison fairness. More systematic comparisons with information-matched baselines are needed beyond the existing ablations.
  • Performance degradation is observed for certain model pairs and task transfers. Further evaluation of generalizability and robustness across different settings is needed.
  • Training the expected review gain requires ground-truth labels and reviewer outcomes, but the overall efficiency analysis does not sufficiently account for these upfront costs.
Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.