acceptodds
Under review as a conference paper at ICLR 2027

Robust Online Learning Against Biased LLM Judge with Budgeted Calibration

Abstract

Online learning is emerging as a powerful paradigm for adapting LLM systems, often relying on LLM-as-a-judge for feedback. However, such feedback can be systematically biased and may not faithfully reflect true task quality. Clean feedback can correct this mismatch, but is too costly to obtain at every round. This raises a fundamental challenge: how can an online learner leverage abundant but biased feedback with only a limited budget of clean feedback? We formulate this setting as a contextual bandit problem with an always-observed (biased) proxy reward and a limited budget for selectively acquiring clean feedback. We propose Paired-Audit Calibrated Exploration (PACE), which leverages paired proxy-clean observations to estimate true task quality and adaptively allocates the audit budget based on learned proxy-clean disagreement. We evaluate PACE on LLM routing, prompt selection, retrieval configuration selection, and synthetic benchmarks. Across our evaluations, PACE is most effective when proxy-clean disagreement is decision-relevant; on the four primary routing conditions, it reduces mean cumulative clean regret over proxy-only LinUCB by 36.5–58.8%. For theoretical characterization, we study PACE-U, a uniform-auditing variant that isolates paired proxy-clean calibration under linear realizability. PACE-U achieves clean regret, while any learner restricted to at most clean-feedback queries incurs regret on a two-dimensional instance, even with adaptive auditing. Thus, the dependence on the horizon and clean-feedback budget is minimax rate-optimal for fixed dimension.

Primary area:
reinforcement learning
Keywords:
LLM-as-a-Judge, Contextual Bandits, Budgeted Feedback

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZVHCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Mitigating LLM-as-a-judge bias for online learning is important, where acquiring high quality clean feedback (e.g., human feedback) is expensive. In this paper, the authors propose PACE, where the it solves a contextual bandit problem to ask for clean feedback only for contextual regions where the proxy-clean disagreement is high, under a limited auditing budget. PACE maintains three predictors; clean, proxy and discrepancy predictor using the feedback of the bias reward and the clean reward when audited. Based on the three predictors, PACE estimates the expected clean reward of the action in two routes: the direct clean estimation and the proxy minus the discrepancy estimation, then select the minimum estimator for the action selection. The authors propose the PACE-U variant, which guarantees the main regret but does not adaptively allocation the audit budget. The empirical results show that PACE can reduce the mean cumulative clear regret across various setups.

Critical strengths

The problem formulation is well aligned with the motivation.

Addressing the efficiency-bias trade-off in the online learning is an important problem. Formalizing the problem into a contextual bandit problem with two-arms; one cheap and biased, one clean and costs a budget, under a limited auditing budget is well-aligned with the motivation and allows a sound setup for theoretical analysis.

The ablation study follows the expected results

PACE can improve the online learning algorithm especially when the llm-as-a-judge has a severe bias, were as the room of improvement is small when the judge is reliable. The ablation in section 5.3 clearly shows the useful regions of PACE, which is consistent with the design of the methodology.

Critical weaknesses

The empirical results of the main design choices are not isolated

PACE is involved with two major components: (1) the action selection target construction via two optimistic estimates of the clean-reward, and (2) the audit probability calculation via risk predictor. Although their design choices are explained by the authors, a component wise analysis is lacking to empirically support the choice of the methodological design.

  1. The need of two optimistic estimates. The authors explain that the use of both routes will allow complementary effect for an accurate estimation of the clean-reward objective, but the choice of such design is not clearly identified empirically. Comparing PACE with variants that uses each of the route alone can help justify the use of two optimistic routes.
  2. The contribution of adaptive auditing and weighting of PACE. It is unclear which component of the PACE is responsible for the high performance. The authors could test the variant of PACE by removing a single component of the design: (1) PACE without adaptive auditing, and (2) PACE without the observational weighting.
Questions for authors

Please refer to the critical weakness.

5 comments

Thank you for carefully reviewing our work. Here's our response to your questions.

Q1. Contribution of the two optimistic routes

We thank the reviewer for requesting a component-wise analysis. We conducted the ablations using the same question sequences, cached rewards, and 20 matched random seeds, with T=1500T=1500 and a clean-query cap of B=300B=300. Full PACE uses the MLP risk predictor. Each ablation changes only the specified component while retaining the remaining update rules and hyperparameters.

To isolate the contribution of the two action-scoring routes, we compare

St(a)={min⁡Utc(a),Utp→c(a),full PACE,Utc(a),Clean route,Utp→c(a),CAL route.S_t(a)= \begin{cases} \min\\{U_t^c(a),U_t^{p\to c}(a)\\}, & \text{full PACE},\\\\ U_t^c(a), & \text{Clean route},\\\\ U_t^{p\to c}(a), & \text{CAL route}. \end{cases}

Each variant selects an action maximizing its corresponding score. Both single-route variants retain the learned audit-policy rule, importance weighting, and estimator updates. In particular, the Clean-route variant still uses proxy feedback to train the risk predictor; it is not a proxy-free clean-only baseline.

Our motivation for the combined score is to allow both routes to constrain optimistic action selection. For each candidate action, the minimum uses the tighter of the two upper bounds. On the event that both bounds validly cover the clean expected reward, their minimum remains a valid upper bound. Thus, the combined score can exploit whichever route supplies the tighter upper bound.

Table 6 shows that full PACE achieves lower mean cumulative clean regret than the Clean-route variant in 8/12 dataset–judge settings and than the CAL-route variant in 11/12. The combined rule improves on both single-route variants in 8/12 settings. Its advantage over the Clean route is clearest on BBH/GPT-4o and LiveBench: mean regret decreases from 148.60 to 137.60 on BBH/GPT-4o, from 149.56 to 141.81 on LiveBench/Gemma, and from 156.52 to 145.74 on LiveBench/GPT-4o, corresponding to reductions of 7.4%, 5.2%, and 6.9%, respectively.

The reported CAL selection frequency ranges from 2.00% to 20.28% across settings, reaching 15.80% on BBH/Gemma, 20.28% on BBH/GPT-4o, and 13.07% on LiveBench/Gemma. The combined rule is designed to use whichever bound is tighter, so the two routes need not be active equally often. The route ablations directly assess the resulting regret; selection frequency describes how often the CAL bound is used and should not be equated with the fraction of decisions changed or regret saved.

Table 6. Two-route versus single-route action selection. Cumulative clean regret is reported as mean ± sample standard deviation over 20 matched seeds; lower is better. The final column reports PACE's CAL selection frequency. All runs use T=1500T=1500 and an audit cap of B=300B=300.

DatasetJudgeFull PACEClean routeCAL routeCAL selection frequency
BBHGemma148.60 ± 24.60144.60 ± 36.24145.80 ± 18.5815.80%
BBHGPT-4o137.60 ± 14.19148.60 ± 37.87178.60 ± 30.5820.28%
HotpotQAGemma157.62 ± 28.50157.82 ± 28.62196.07 ± 14.582.00%
HotpotQAGPT-4o163.41 ± 50.94163.54 ± 51.78188.62 ± 15.593.80%
IFBenchGemma159.20 ± 51.07158.60 ± 49.93199.00 ± 34.254.23%
IFBenchGPT-4o149.60 ± 57.42143.80 ± 53.18209.20 ± 30.287.97%
LiveBenchGemma141.81 ± 29.16149.56 ± 29.14198.27 ± 53.1613.07%
LiveBenchGPT-4o145.74 ± 17.19156.52 ± 44.51173.32 ± 39.205.20%
Arena-Hard v2Gemma276.15 ± 29.49276.35 ± 29.72306.95 ± 17.362.31%
Arena-Hard v2GPT-4o285.00 ± 32.83284.75 ± 33.80342.70 ± 19.264.81%
MMLU-ProGemma121.80 ± 30.47122.60 ± 29.26144.80 ± 12.872.39%
MMLU-ProGPT-4o111.60 ± 23.48113.60 ± 21.94138.20 ± 8.642.56%

These results demonstrate the empirical value of combining the two routes on several real-data benchmarks, with the strongest gains over the Clean route on BBH/GPT-4o and LiveBench. The relative contribution is task-dependent: for example, both single-route variants have lower mean regret on BBH/Gemma. We will include these ablations to show where combining the two reward estimates improves action selection and how the contribution varies across tasks. Both single-route variants retain risk-based auditing and importance weighting, so these comparisons isolate the action-scoring rule from the continued use of proxy feedback in audit allocation.

Q2. Contribution of adaptive auditing and inverse-propensity weighting

We evaluated audit allocation and observation weighting separately using the same question sequences, cached rewards, and 20 matched random seeds, with T=1500T=1500 and a clean-query cap of B=300B=300. All PACE variants use the MLP risk predictor. Each variant is run independently, so changes to weighting may affect subsequent actions, training samples, and audit decisions.

Adaptive auditing. PACE-uniform replaces risk-based allocation with uniform auditing while retaining the full PACE estimators, residual allowance, and propensity-weighting formula; it is distinct from PACE-U. Full PACE achieves lower mean cumulative clean regret in all 12/12 settings, with reductions of 5.2%–32.8%. This controlled comparison supports the contribution of adaptive auditing relative to uniform allocation.

Separating estimator and risk-predictor weighting. We completed a 2×22\times2 ablation of IPW in the clean-reward, discrepancy, and residual updates ("estimator IPW") and in MLP risk-predictor training:

  • Full PACE: IPW in both groups of updates.
  • No estimator IPW: unit estimator weights, retaining IPW in the risk loss.
  • No risk-predictor IPW: unit risk-loss weights, retaining estimator IPW.
  • No IPW (all updates): unit weights in both groups.

All three weighting ablations retain the risk-based audit-policy rule, both scoring routes, and proxy feedback. Proxy-reward regression always uses unit weights. Removing weighting therefore does not turn PACE into a clean-only baseline.

The updated ablation distinguishes the two roles of weighting. Retaining risk-predictor IPW yields lower mean regret in 10/12 settings when estimator IPW is retained, and in 7/12 when estimator weights are uniform. The latter comparison is between the No estimator IPW and No IPW (all updates) variants. In contrast, retaining estimator IPW yields lower mean regret in 4/12 settings under either risk-loss configuration. Full PACE has lower mean regret than the variant without any IPW in 5/12 settings. These results support a more consistent empirical contribution from risk-predictor weighting, while showing that estimator weighting is not uniformly beneficial.

Table 7. Separate ablations of adaptive auditing and inverse-propensity weighting. All runs use T=1500T=1500 and a clean-query cap of B=300B=300. Regret entries are mean ± sample standard deviation over 20 matched seeds; lower is better. Comparisons in the text refer to sample means.

DatasetJudgeFull PACEUniform auditsNo IPW (all updates)No estimator IPWNo risk-predictor IPW
BBHGemma148.60 ± 24.60156.80 ± 18.35142.10 ± 29.26141.80 ± 31.19130.80 ± 10.73
BBHGPT-4o137.60 ± 14.19156.60 ± 18.18134.90 ± 18.69134.70 ± 16.01136.70 ± 17.45
HotpotQAGemma157.62 ± 28.50198.60 ± 21.77168.54 ± 41.54156.27 ± 35.06167.90 ± 28.64
HotpotQAGPT-4o163.41 ± 50.94198.60 ± 21.77159.39 ± 33.06155.77 ± 37.04165.63 ± 34.60
IFBenchGemma159.20 ± 51.07222.50 ± 40.29145.90 ± 54.27154.20 ± 58.33166.60 ± 48.90
IFBenchGPT-4o149.60 ± 57.42222.75 ± 40.37130.80 ± 36.06135.70 ± 45.03163.00 ± 47.95
LiveBenchGemma141.81 ± 29.16183.12 ± 24.25138.14 ± 22.60147.28 ± 31.27160.55 ± 42.06
LiveBenchGPT-4o145.74 ± 17.19184.10 ± 25.06134.53 ± 24.61127.61 ± 27.67151.84 ± 26.55
Arena-Hard v2Gemma276.15 ± 29.49324.91 ± 12.20281.06 ± 28.27290.26 ± 35.65300.62 ± 36.67
Arena-Hard v2GPT-4o285.00 ± 32.83324.91 ± 12.20291.91 ± 30.99295.51 ± 31.80299.77 ± 34.92
MMLU-ProGemma121.80 ± 30.47145.10 ± 33.64125.35 ± 35.39119.60 ± 32.92123.50 ± 24.65
MMLU-ProGPT-4o111.60 ± 23.48145.10 ± 33.64133.90 ± 37.88125.70 ± 33.84122.25 ± 22.64

Weight-scale and audit-probability controls. To investigate estimator weighting, we added three controls with settings fixed across datasets. The first two retain risk-loss IPW and set estimator weights to either 5=T/B5=T/B or 0.2wt0.2w_t. The latter preserves relative weights and is a fixed scaling, not self-normalized IPW. The third imposes an actual audit-probability floor pt′=max⁡(pt,0.01)p'_t=\max(p_t,0.01) while budget remains, uses 1/pt′1/p'_t in both groups of updates, and respects the same hard budget cap.

Continue on analysis of IPW

Table 8. Summary of the six IPW diagnostics relative to the updated full PACE results. Counts summarize comparisons of mean cumulative clean regret across the 12 dataset–judge settings.

VariantSettings with lower mean regret than full PACE
No estimator IPW8/12
No IPW (all updates)7/12
No risk-predictor IPW2/12
Fixed estimator weights of 50/12
Estimator weights scaled to 0.2wt0.2w_t8/12
Actual audit-probability floor of 0.010/12

Scaling estimator IPW to 0.2wt0.2w_t yields lower mean regret than full PACE in 8/12 settings, whereas fixed estimator weights of 5 yield higher mean regret than both full PACE and unit estimator weights in all 12/12. For example, on IFBench/Gemma, 0.2wt0.2w_t gives mean regret 141.95, versus 159.20 for full PACE. These controls show that estimator-weight scale matters under the fixed regularization and confidence settings. An actual audit-probability floor of 0.01 does not lower mean regret in any of the 12 settings. Neither scale control nor the probability-floor control is uniformly preferable to full PACE.

Mechanistic diagnostics on previously recorded trajectories. We retain the earlier label-composition and fixed-history diagnostics to examine how the implemented weighting operates. The figures in this paragraph and the fixed-history results below describe those diagnostic logs; Tables 7–8 report the updated online regret comparisons. On IFBench/Gemma, the full selected-action stream has a disagreement prevalence of 49.5%, compared with 73.8% in audited samples and 63.6% in the IPW-weighted audited samples. The mean absolute discrepancy from the stream prevalence falls from 24.4 to 14.1 percentage points across seeds. The weighted proportion ∑tMtwtdt/∑tMtwt\sum_t M_tw_td_t/\sum_t M_tw_t is used only for this diagnostic. It shows partial correction of the audited samples' label composition; closer matching of this marginal proportion alone does not establish better reward estimates, action rankings, or regret.

The implementation uses clipped IPW, wt=1/max⁡(pt,0.01)w_t=1/\max(p_t,0.01). With MtM_t denoting the audit indicator and Ht\mathcal H_t the information available before the audit decision,

E[Mtwt∣Ht]=ptmax⁡(pt,0.01).\mathbb E[M_tw_t\mid\mathcal H_t] =\frac{p_t}{\max(p_t,0.01)}.

This factor is below one when 0<pt<0.010 < p_t < 0.01. The clipping controls the largest weight but does not enforce a floor on the actual audit probability; no observations can be recovered after budget exhaustion. The weighting addresses audit selection along the selected-action stream and does not correct action selection itself.

Why estimator-weight scale matters. With identical audit histories, wt≥1w_t\geq1 implies Vw⪰V1V_w\succeq V_1 and hence ∣x∣Vw−1≤∣x∣V1−1\\|x\\|_{V_w^{-1}}\leq\\|x\\|_{V_1^{-1}}. Keeping the ridge and confidence coefficients fixed therefore changes effective regularization, fitted means, and geometric exploration widths. A smaller geometric width does not by itself establish lower estimation noise. In the earlier fixed-history diagnostics, the original IPW produced higher mean clean and calibrated-proxy pairwise-gap prediction errors than unit weights in all 12 settings, together with mean width ratios of 0.939–0.985. These errors are measured against cached reward differences, not known conditional reward means. The diagnostics show effects on fitting and exploration and help interpret the sensitivity observed in the scale controls.

Risk-loss weighting also interacts with regularization: the MLP minimizes a weighted sum of binary cross-entropy plus a fixed L2L_2 penalty. Its weight scale therefore affects effective regularization and finite-step optimization as well as relative sample contributions.

Reviewer_opLKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies online learning with a biased LLM judge and a limited budget of clean feedback. The authors formulate the problem as a contextual bandit setting, where proxy rewards are always available but clean rewards can only be obtained through a limited number of audits.

The paper proposes Paired-Audit Calibrated Exploration (PACE), which uses paired proxy-clean observations to learn the reward discrepancy and correct the proxy reward estimates. PACE also uses a learned disagreement risk model to allocate clean feedback to selected contexts. The authors further introduce PACE-U, a simplified variant with uniform auditing, and provide regret bounds under linear realizability.

Experiments on LLM routing, prompt selection, retrieval configuration selection, and synthetic environments show that PACE can reduce cumulative clean regret compared to proxy-only learning, particularly when proxy-clean disagreement affects action selection.

Critical strengths
  1. Practical and meaningful problem setting

    The paper addresses an important problem in online learning with LLM-based feedback, where cheap proxy rewards may be biased and reliable clean feedback is limited. The use of paired proxy-clean observations to correct the proxy reward is intuitive and well motivated.

  2. Theoretical analysis of budgeted calibration

    The paper provides regret bounds for PACE-U, including a matching lower bound on the dependence on the horizon and clean-feedback budget for fixed dimension. The analysis helps explain the statistical limits of learning with biased proxy rewards and limited clean feedback.

Critical weaknesses
  1. Insufficient baseline comparisons and ablation studies

    The current comparisons do not sufficiently isolate the contributions of paired calibration and adaptive auditing. In particular, additional budget-matched clean-only baselines and ablations that vary only the audit allocation strategy are missing.

  2. Limited justification for disagreement-based auditing

    PACE allocates clean feedback based on the predicted absolute proxy-clean disagreement. However, large disagreement does not necessarily change the optimal action or lead to high clean regret. It is unclear whether this criterion effectively allocates the limited clean-feedback budget to decision-relevant contexts.

Questions for authors
  1. Additional baselines

    Could the authors compare PACE with front-loaded variants of Clean LinUCB and PACE (PACE-F), which use the clean-feedback budget during the first BB rounds? These comparisons should use exactly the same question sequences, cached rewards, audit budgets, and random seed settings as the current experiments. In addition, a comparison with confidence-based auditing methods (e.g., CBM-OFUL from Efroni et al., ICML 2021) would also be useful.

  2. Contribution of adaptive auditing

    PACE and PACE-U differ not only in audit allocation but also in importance weighting and residual correction. Could the authors provide an ablation that changes only the audit allocation strategy (e.g., uniform, front-loaded, and risk-based auditing) while keeping the other components fixed? It would also be helpful to report the cumulative number of audits over time and paired confidence intervals for the regret differences, especially since the improvements over PACE-U are relatively small in some settings.

  3. Disagreement risk and clean regret

    Why does PACE use absolute proxy-clean disagreement rather than a criterion directly related to action ranking or clean regret? For example, large disagreement may preserve the optimal action, while small disagreement may reverse the action ranking. Could the authors analyze whether the predicted disagreement risk is correlated with action-ranking errors or clean regret in the real benchmarks?

2 comments

Thank you for carefully reviewing our work. Here's our response to your questions.

Q1–Q2. Budget-matched baselines and audit allocation

We added front-loaded Clean LinUCB (Clean-front), front-loaded PACE (PACE-F), and CBM-OFUL. Within each dataset–judge setting, all methods use the same question sequences, cached reward matrices, and 20 matched random seeds, with T=1500T=1500 and a clean-feedback cap of B=300B=300. Hyperparameters are fixed across datasets.

To isolate audit allocation, PACE-uniform and PACE-F replace risk-based auditing with uniform auditing and auditing during the first BB rounds, respectively, while retaining the estimator equations, residual-correction rules, and propensity-weighting formula of full PACE. PACE-uniform is distinct from the theoretically analyzed PACE-U. Clean-front likewise queries clean feedback during the first BB rounds.

The results support the benefit of risk-based auditing relative to uniform auditing under this configuration: full PACE achieves lower mean cumulative clean regret than PACE-uniform in all 12/12 dataset–judge settings, with mean regret reductions of 5.2%–32.8%.

The front-loaded baselines provide a more competitive comparison. PACE achieves lower mean cumulative clean regret than PACE-F in 9/12 settings and than Clean-front in 8/12. These results show that front-loading the clean-feedback budget is a strong alternative under this evaluation protocol, with the relative performance of risk-based and front-loaded allocation depending on the dataset and judge.

For the confidence-based baseline, we evaluated the main CBM-OFUL configuration and CBM-OFUL with parameter \texttt{action\\_alpha}=1 using the condition in Algorithm 4 of Efroni et al. (ICML 2021). This configuration consumes all 300 audits in every run. PACE achieves lower mean cumulative clean regret than CBM-OFUL(main) in all settings and with parameter \texttt{action\\_alpha}=1 in 8/12 settings.

Table 4. Comparison of audit-allocation strategies, clean-only baselines, and CBM-OFUL configurations. Entries are cumulative clean regret (mean ± sample standard deviation over 20 matched seeds); lower is better. All runs use T=1500T=1500 and a clean-feedback cap of B=300B=300.

(a) Gemma judge

MethodBBHHotpotQAIFBenchLiveBenchArena-Hard v2MMLU-Pro
PACE148.60 ± 24.60157.62 ± 28.50159.20 ± 51.07141.81 ± 29.16276.15 ± 29.49121.80 ± 30.47
PACE-uniform156.80 ± 18.35198.60 ± 21.77222.50 ± 40.29183.12 ± 24.25324.91 ± 12.20145.10 ± 33.64
PACE-F139.60 ± 24.33170.91 ± 26.29169.60 ± 26.97159.59 ± 33.02296.75 ± 32.51138.75 ± 41.07
Clean-uniform156.80 ± 18.35198.60 ± 21.77222.50 ± 40.29183.19 ± 24.46324.91 ± 12.20145.10 ± 33.64
Clean-front121.40 ± 20.33170.91 ± 26.29126.90 ± 27.39153.80 ± 33.59307.65 ± 32.51138.75 ± 41.07
CBM-OFUL (main)298.05 ± 50.06372.43 ± 33.17423.85 ± 24.60321.55 ± 18.67475.52 ± 22.09178.65 ± 14.93
CBM-OFUL (action_alpha = 1)108.40 ± 13.60170.91 ± 26.29129.40 ± 27.23147.08 ± 27.18296.85 ± 32.79138.75 ± 41.07

(b) GPT-4o judge

MethodBBHHotpotQAIFBenchLiveBenchArena-Hard v2MMLU-Pro
PACE137.60 ± 14.19163.41 ± 50.94149.60 ± 57.42145.74 ± 17.19285.00 ± 32.83111.60 ± 23.48
PACE-uniform156.60 ± 18.18198.60 ± 21.77222.75 ± 40.37184.10 ± 25.06324.91 ± 12.20145.10 ± 33.64
PACE-F119.60 ± 24.33170.91 ± 26.29129.65 ± 26.94169.43 ± 32.64299.75 ± 42.03138.75 ± 41.07
Clean-uniform156.80 ± 18.35198.60 ± 21.77222.50 ± 40.29183.19 ± 24.46324.91 ± 12.20145.10 ± 33.64
Clean-front121.40 ± 20.33170.91 ± 26.29126.90 ± 27.39153.80 ± 33.59307.65 ± 32.51138.75 ± 41.07
CBM-OFUL (main)298.05 ± 50.06372.43 ± 33.17423.85 ± 24.60321.55 ± 18.67475.52 ± 22.09178.65 ± 14.93
CBM-OFUL (action_alpha = 1)108.40 ± 13.60170.91 ± 26.29129.40 ± 27.23147.08 ± 27.18296.85 ± 32.79138.75 ± 41.07
These comparisons isolate the empirical contribution of risk-based auditing within PACE: it improves on uniform allocation while the other algorithmic components are held fixed. The front-loaded results further show that the choice of audit timing matters, and that the advantage over uniform auditing does not translate into uniform superiority over other allocation strategies. We will include these stronger baselines and revise the discussion to reflect both findings.

Q3. Disagreement risk and clean regret

We agree that absolute proxy–clean disagreement is not equivalent to an action-ranking error or clean regret. A large discrepancy shared across actions may preserve the optimal action, while a small discrepancy may reverse the ordering of actions with similar clean rewards.

Our choice of prediction target is motivated by the feedback available under budgeted auditing. A single audit provides a paired observation for the selected action and hence a supervised label for the disagreement event:

d_t=\mathbf{1}\\!\left\\{|F(x_t,a_t)-C(x_t,a_t)|\geq\tau\right\\}

where FF and CC are the cached proxy and clean rewards. Directly labeling ranking errors or clean regret generally requires clean rewards for alternative actions, or additional modeling assumptions. We therefore use the learned score for this disagreement event as a practical audit-allocation heuristic, without treating it as a ranking-error probability or an estimate of the expected regret reduction from an audit.

The results show that predicting disagreement does not consistently translate into strong decision relevance. On IFBench/Gemma, for example, disagreement AUROC is 0.744, but ranking-error AUROC is 0.510.

Table 5. Discrimination of proxy–clean disagreement and proxy-induced ranking errors. Entries are means across 20 seeds, with 95% confidence intervals in brackets. All runs use T=1500T=1500 and B=300B=300. These predictor diagnostics use the previously recorded trajectories; Table 4 reports the updated regret comparisons.

DatasetJudgeDisagreement AUROCRanking-error AUROC
BBHGemma0.607 [0.569, 0.644]0.465 [0.441, 0.488]
BBHGPT-4o0.592 [0.563, 0.622]0.520 [0.498, 0.541]
HotpotQAGemma0.581 [0.572, 0.590]0.533 [0.525, 0.540]
HotpotQAGPT-4o0.576 [0.565, 0.587]0.557 [0.549, 0.564]
IFBenchGemma0.744 [0.732, 0.757]0.510 [0.500, 0.520]
IFBenchGPT-4o0.722 [0.712, 0.732]0.512 [0.501, 0.523]
LiveBenchGemma0.614 [0.597, 0.631]0.544 [0.524, 0.564]
LiveBenchGPT-4o0.619 [0.600, 0.639]0.581 [0.562, 0.599]
Arena-Hard v2Gemma0.580 [0.567, 0.594]0.543 [0.532, 0.553]
Arena-Hard v2GPT-4o0.599 [0.584, 0.614]0.539 [0.530, 0.547]
MMLU-ProGemma0.543 [0.533, 0.552]0.537 [0.519, 0.554]
MMLU-ProGPT-4o0.543 [0.531, 0.555]0.553 [0.536, 0.570]

These diagnostics show that our risk score is not perfectly aligned with action ranking error. Separately, the controlled, budget-matched ablation in our response to Q1–Q2 shows shows that our lower mean cumulative clean regret for risk-based auditing than for uniform auditing in all 12 settings. This supports the audit rule's empirical utility under the evaluated configuration, while leaving the mechanism behind that benefit unresolved. We will clarify this distinction in the paper and present disagreement risk as a heuristic learned from available paired feedback.

Reviewer_J8eYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies online learning with an always-available but potentially biased proxy reward, such as feedback from an LLM judge, while only a limited amount of clean feedback can be queried. The authors propose PACE, a contextual bandit method that uses paired proxy and clean observations to estimate the bias of the proxy and selectively allocates the clean-feedback budget to contexts where disagreement is more likely. The method is evaluated on LLM routing, prompt selection, retrieval configuration selection, and synthetic tasks. The experiments show that PACE can substantially reduce clean regret compared with proxy-only learning when the proxy feedback is systematically misaligned with the true objective. The paper also analyzes a simplified uniform-auditing variant, PACE-U, and provides upper and lower regret bounds under linear realizability assumptions.

Critical strengths

S1: The paper studies a practical problem: how to make online decisions when cheap LLM-based feedback is abundant but potentially biased, while reliable feedback is expensive and limited.

S2: The proposed PACE framework is intuitive and well motivated. In particular, using paired proxy-clean observations to estimate the proxy bias and combining direct clean estimation with a calibrated proxy estimate is a reasonable design.

S3: The controlled experiments are useful for showing when selective auditing helps. The results suggest that the method is especially effective when proxy-clean disagreement is structured and affects the action ranking.

Critical weaknesses

W1: The main experimental proxy bias is largely induced through a fixed style-preference prompt. This makes the experiments controlled and interpretable, but it is unclear whether the same gains would hold for naturally occurring LLM-judge biases or disagreement with human preferences.

W2: The real-data experiments are based on cached replay. Since the online horizon is larger than the number of unique questions in several datasets, the same contexts can appear multiple times. This may make the setting easier than a true online stream with unseen contexts and fresh model generations.

W3: There is a noticeable gap between the practical algorithm and the theoretical analysis. The strongest empirical version, PACE, uses context-dependent auditing and residual correction, while the regret guarantee is only established for PACE-U with uniform auditing and stronger linear realizability assumptions.

W4: The current experiments do not fully isolate the contribution of each component. In particular, it would be useful to separately evaluate the effect of adaptive auditing, residual correction, and inverse-propensity weighting.

Questions for authors
  • Have you evaluated PACE under naturally occurring LLM-judge disagreement, without explicitly inducing a style preference? If so, do you observe similar improvements?

  • Is it possible to provide any theoretical guarantee for the full PACE algorithm with context-dependent auditing, even under additional assumptions?

  • How does PACE perform when each context appears only once? A no-repeat evaluation would help clarify how much the current results benefit from the cached replay setting.

  • Could you provide an ablation that separately removes the adaptive audit policy and the residual correction term?

  • How accurate is the learned disagreement-risk predictor? For example, could you report AUROC, AUPRC, or calibration statistics for predicting proxy-clean disagreement?

3 comments

Thank you for carefully reviewing our work. Here's our response to your questions.

Q1. Naturally occurring LLM-judge disagreement

Have you evaluated PACE under naturally occurring LLM-judge disagreement, without explicitly inducing a style preference? If so, do you observe similar improvements?

To address this concern, we conducted an additional neutral-rubric evaluation on LiveBench, BBH, and HotpotQA by re-scoring the existing model outputs with Gemma. The rubric does not explicitly reward verbosity, confident tone, or formatting. This removes the deliberately induced style preference without assuming that the resulting judge feedback is unbiased.

All comparisons use 20 matched random seeds, with no dataset-specific hyperparameter tuning. As shown in Table 1, PACE achieves lower mean cumulative clean regret than both Proxy LinUCB and uniform Clean LinUCB on all three datasets. Relative to Proxy LinUCB, the reductions are 23.2%, 40.8%, and 24.3% on LiveBench, BBH, and HotpotQA, respectively. The comparison with PACE-U varies across tasks: PACE has lower mean regret on LiveBench and HotpotQA, whereas PACE-U has a slightly lower mean on BBH.

Table 1. Cumulative clean regret under a neutral Gemma judging rubric (lower is better). Values are mean ± standard deviation across 20 matched random seeds. All runs use T=1500T=1500. PACE, PACE-U, and uniform Clean LinUCB use the same audit budget of B=300B=300; Proxy LinUCB uses no clean feedback.

DatasetPACEPACE-UProxy LinUCBClean LinUCB (uniform)
LiveBench182.32 ± 32.52191.03 ± 42.18237.45 ± 49.53196.64 ± 24.43
BBH149.50 ± 41.16147.45 ± 39.87252.60 ± 13.01156.80 ± 18.35
HotpotQA164.03 ± 35.15171.38 ± 32.94216.58 ± 5.84198.60 ± 21.77

These results suggest that PACE's improvements over proxy-only learning extend beyond explicitly induced style preferences on these benchmarks, although their magnitude depends on the task. This neutral-rubric control should not be interpreted as direct validation against human preferences. We will include these results and the corresponding analysis in the revised paper.

Q2. Scope of theoretical guarantees for full PACE

Is it possible to provide any theoretical guarantee for the full PACE algorithm with context-dependent auditing, even under additional assumptions?

Appendix A.6 provides a conditional regret bound for full PACE. Specifically, on the event that the clean-reward, proxy-reward, and discrepancy estimates satisfy the simultaneous error bounds in Eqs. (39)–(41), including valid coverage of residual misspecification, summing the per-round bound in Eq. (44) yields

\mathrm{Reg}_c(T) \le 2\sum_{t=1}^{T} \min\\!\left\\{ \alpha_c u_t^c(A_t),\\, \alpha_p u_t^p(A_t) +\alpha_b u_t^b(A_t) +\epsilon_{J,t}(A_t) \right\\}.

This implication does not require uniform auditing. However, it is conditional on the stated confidence-coverage event and does not by itself establish an explicit regret rate in TT and BB for the learned auditing policy. Obtaining such a rate requires both (i) proving that the simultaneous confidence bounds hold with high probability under adaptive auditing and (ii) controlling the accumulated uncertainty and residual terms through sufficient audit coverage and residual-misspecification control.

The MLP risk predictor in our implementation does not by itself ensure the required audit coverage, and the empirical residual correction does not by itself establish uniformly valid discrepancy bounds. We have not proved these properties for full PACE. Accordingly, our established budget-dependent regret guarantee applies to PACE-U under its stated assumptions, while Appendix A.6 provides a conditional characterization for full PACE. We will make this distinction explicit in the revised paper; an end-to-end analysis of the learned auditing and residual-calibration procedure remains future work.

Q3. No-repeat evaluation

How does PACE perform when each context appears only once? A no-repeat evaluation would help clarify how much the current results benefit from the cached replay setting.

To address the concern about repeated exposure to the same contexts, we conducted a strict no-repeat evaluation. We removed duplicate online context vectors and excluded any online context vector that also appeared in the shared proxy-only warm-up set. Each retained context vector is presented exactly once per run. We set the horizon TT to the size of the retained pool and the clean-query cap to B=⌊0.2T⌋B=\lfloor 0.2T\rfloor.

Table 2 reports results for model routing on IFBench and LiveBench, prompt selection on BBH, and retrieval configuration selection on HotpotQA, using both Gemma and GPT-4o judges. The comparisons include the corresponding proxy-only, budget-matched, and task-specific baselines from the main experiments. All entries report mean cumulative clean regret ± sample standard deviation over 20 matched seeds.

Across the eight dataset–judge combinations, PACE (MLP) achieves lower mean cumulative clean regret than uniform Clean LinUCB in 7/8 settings, than Proxy LinUCB in 7/8, and than PACE-U in 5/8. For the task-specific baselines, PACE (MLP) achieves lower mean regret than each of the two MetaLLM-online configurations in all 4/4 routing settings, than ProTeGi UCB-E in 1/2 BBH settings, and than the MBA-RAG encoder in 2/2 HotpotQA settings. The comparisons are not uniformly favorable: for example, on BBH, uniform Clean LinUCB has lower mean regret under Gemma, while Proxy LinUCB has lower mean regret under GPT-4o.

Table 2. Strict no-repeat evaluation across model routing, prompt selection, and retrieval configuration selection. Entries are cumulative clean regret (mean ± sample standard deviation over 20 matched seeds); lower is better. An audit cap of BB refers to the dataset-specific cap below; methods with a cap of 0 receive no clean feedback. A dash indicates that a task-specific baseline is not applicable.

DatasetTaskHorizon TTClean-query cap BB
IFBenchModel routing27054
LiveBenchModel routing45090
BBHPrompt selection715143
HotpotQARetrieval configuration selection720144

(a) Gemma judge

MethodAudit capIFBenchLiveBenchBBHHotpotQA
Proxy UCB1080.05 ± 3.12150.78 ± 4.81113.80 ± 6.7192.43 ± 5.43
Proxy LinUCB079.25 ± 4.45141.87 ± 6.62115.50 ± 8.2998.94 ± 6.63
MetaLLM-online (α=0\alpha=0)0110.10 ± 8.59135.64 ± 2.41——
MetaLLM-online (α=1\alpha=1)0110.10 ± 8.59135.44 ± 2.09——
ProTeGi UCB-E selector0——109.40 ± 8.56—
MBA-RAG encoder0———90.93 ± 6.38
Proxy LinUCB + RcR_c repl.BB73.70 ± 6.48125.77 ± 6.50100.25 ± 10.58102.64 ± 8.28
Clean LinUCB (uniform)BB63.50 ± 25.9796.49 ± 18.5178.85 ± 12.0396.12 ± 15.79
PACE-UBB52.10 ± 23.9780.30 ± 16.6667.40 ± 8.6986.73 ± 13.13
PACEBB56.70 ± 27.8585.32 ± 32.2183.30 ± 34.8080.95 ± 20.93

(b) GPT-4o judge

MethodAudit capIFBenchLiveBenchBBHHotpotQA
Proxy UCB1075.45 ± 4.57149.80 ± 6.4269.20 ± 4.1083.06 ± 4.15
Proxy LinUCB076.65 ± 5.98124.41 ± 7.8166.45 ± 5.3392.48 ± 6.13
MetaLLM-online (α=0\alpha=0)065.65 ± 47.77161.04 ± 1.07——
MetaLLM-online (α=1\alpha=1)070.30 ± 47.52160.99 ± 1.11——
ProTeGi UCB-E selector0——66.70 ± 6.62—
MBA-RAG encoder0———88.03 ± 8.29
Proxy LinUCB + RcR_c repl.BB70.45 ± 5.62117.90 ± 8.2563.25 ± 4.2997.76 ± 7.20
Clean LinUCB (uniform)BB63.50 ± 25.9796.49 ± 18.5178.85 ± 12.0396.12 ± 15.79
PACE-UBB57.25 ± 24.9695.92 ± 18.8176.90 ± 15.3598.09 ± 21.97
PACEBB55.05 ± 28.4287.47 ± 26.7869.90 ± 11.7481.17 ± 18.18

These results show that improvements over several baselines persist when repeated online context vectors and overlap with the warm-up set are excluded. They support the effectiveness of PACE in this no-repeat setting. We will include the protocol and results in the revised paper.

Q4. Separate ablations of adaptive auditing and the residual allowance

Could you provide an ablation that separately removes the adaptive audit policy and the residual correction term?

We conducted two separate ablations using the same question sequences, cached rewards, and 20 matched random seeds, with T=1500T=1500 and an audit cap of B=300B=300.

First, we replaced risk-based auditing with uniform auditing while retaining the reward estimators, residual allowance, and propensity-weighting formula of full PACE. We call this variant PACE-uniform; it is distinct from the theoretically analyzed PACE-U. Second, we removed the residual allowance by setting ϵJ,t(a)=0\epsilon_{J,t}(a)=0, while retaining the learned audit policy and all other update rules. This second ablation retains the learned proxy–clean bias correction.

As shown in Table 3, full PACE achieves lower mean cumulative clean regret than PACE-uniform in all 12/12 dataset–judge settings, with reductions ranging from 5.2% to 32.8%. This comparison isolates the contribution of risk-based audit allocation while retaining the other algorithmic components.

Retaining the residual allowance yields lower mean cumulative clean regret in 10/12 settings. For example, mean regret decreases from 300.07 to 276.15 on Arena-Hard v2/Gemma and from 158.75 to 141.81 on LiveBench/Gemma. The two exceptions are BBH/Gemma and BBH/GPT-4o, where retaining the allowance increases mean regret by 6.30 and 0.85, respectively. The updated results therefore show a task-dependent contribution from the residual allowance.

Table 3. Separate ablations of adaptive auditing and the residual allowance. Cumulative clean regret is reported as mean ± sample standard deviation over 20 matched seeds; lower is better. All runs use T=1500T=1500 and B=300B=300. Both Δaudit\Delta_{\mathrm{audit}} and Δresidual\Delta_{\mathrm{residual}} are differences between the reported means, defined as full PACE minus the corresponding ablation; negative values favor full PACE. Values are rounded to two decimal places.

DatasetJudgeFull PACEPACE-uniformNo residual allowanceMean difference: Δaudit\Delta_{\mathrm{audit}}Mean difference: Δresidual\Delta_{\mathrm{residual}}
BBHGemma148.60 ± 24.60156.80 ± 18.35142.30 ± 25.28-8.20+6.30
BBHGPT-4o137.60 ± 14.19156.60 ± 18.18136.75 ± 23.83-19.00+0.85
HotpotQAGemma157.62 ± 28.50198.60 ± 21.77168.34 ± 41.72-40.98-10.72
HotpotQAGPT-4o163.41 ± 50.94198.60 ± 21.77165.93 ± 44.55-35.19-2.52
IFBenchGemma159.20 ± 51.07222.50 ± 40.29170.55 ± 53.00-63.30-11.35
IFBenchGPT-4o149.60 ± 57.42222.75 ± 40.37155.10 ± 45.15-73.15-5.50
LiveBenchGemma141.81 ± 29.16183.12 ± 24.25158.75 ± 30.40-41.31-16.94
LiveBenchGPT-4o145.74 ± 17.19184.10 ± 25.06151.28 ± 29.76-38.36-5.54
Arena-Hard v2Gemma276.15 ± 29.49324.91 ± 12.20300.07 ± 30.54-48.76-23.92
Arena-Hard v2GPT-4o285.00 ± 32.83324.91 ± 12.20300.56 ± 37.51-39.91-15.56
MMLU-ProGemma121.80 ± 30.47145.10 ± 33.64127.35 ± 23.43-23.30-5.55
MMLU-ProGPT-4o111.60 ± 23.48145.10 ± 33.64122.90 ± 26.77-33.50-11.30

These results support the contribution of adaptive auditing relative to uniform auditing and show lower mean regret with the residual allowance in most settings. The two ablations separate audit allocation from the allowance used to accommodate possible discrepancy-model misspecification; the latter retains the learned proxy–clean bias correction. We will further revise our work.

Q5. Disagreement-risk predictor

The following diagnostics use the previously recorded PACE trajectories. We evaluated the predictor using pre-update scores. On IFBench with the Gemma judge, it achieves an AUROC of 0.744 and average precision of 0.764, compared with a disagreement prevalence of 0.495. In a post-hoc analysis, 86.7% of samples in the highest-risk 20% exhibit disagreement, versus 49.5% overall. These results show that the scores can concentrate disagreement cases in this setting.

Prediction quality is task-dependent: AUROC is 0.607 on BBH and 0.543 on MMLU-Pro under the same judge. Probability calibration also remains imperfect (IFBench ECE: 0.276), so the scores should not be interpreted as well-calibrated disagreement probabilities. And we will report this limitation and our paper is still under revising.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.