- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Mitigating LLM-as-a-judge bias for online learning is important, where acquiring high quality clean feedback (e.g., human feedback) is expensive. In this paper, the authors propose PACE, where the it solves a contextual bandit problem to ask for clean feedback only for contextual regions where the proxy-clean disagreement is high, under a limited auditing budget. PACE maintains three predictors; clean, proxy and discrepancy predictor using the feedback of the bias reward and the clean reward when audited. Based on the three predictors, PACE estimates the expected clean reward of the action in two routes: the direct clean estimation and the proxy minus the discrepancy estimation, then select the minimum estimator for the action selection. The authors propose the PACE-U variant, which guarantees the main regret but does not adaptively allocation the audit budget. The empirical results show that PACE can reduce the mean cumulative clear regret across various setups.
Critical strengths
The problem formulation is well aligned with the motivation.
Addressing the efficiency-bias trade-off in the online learning is an important problem. Formalizing the problem into a contextual bandit problem with two-arms; one cheap and biased, one clean and costs a budget, under a limited auditing budget is well-aligned with the motivation and allows a sound setup for theoretical analysis.
The ablation study follows the expected results
PACE can improve the online learning algorithm especially when the llm-as-a-judge has a severe bias, were as the room of improvement is small when the judge is reliable. The ablation in section 5.3 clearly shows the useful regions of PACE, which is consistent with the design of the methodology.
Critical weaknesses
The empirical results of the main design choices are not isolated
PACE is involved with two major components: (1) the action selection target construction via two optimistic estimates of the clean-reward, and (2) the audit probability calculation via risk predictor. Although their design choices are explained by the authors, a component wise analysis is lacking to empirically support the choice of the methodological design.
- The need of two optimistic estimates. The authors explain that the use of both routes will allow complementary effect for an accurate estimation of the clean-reward objective, but the choice of such design is not clearly identified empirically. Comparing PACE with variants that uses each of the route alone can help justify the use of two optimistic routes.
- The contribution of adaptive auditing and weighting of PACE. It is unclear which component of the PACE is responsible for the high performance. The authors could test the variant of PACE by removing a single component of the design: (1) PACE without adaptive auditing, and (2) PACE without the observational weighting.
Questions for authors
Please refer to the critical weakness.
5 comments
Q2. Contribution of adaptive auditing and inverse-propensity weighting
We evaluated audit allocation and observation weighting separately using the same question sequences, cached rewards, and 20 matched random seeds, with and a clean-query cap of . All PACE variants use the MLP risk predictor. Each variant is run independently, so changes to weighting may affect subsequent actions, training samples, and audit decisions.
Adaptive auditing. PACE-uniform replaces risk-based allocation with uniform auditing while retaining the full PACE estimators, residual allowance, and propensity-weighting formula; it is distinct from PACE-U. Full PACE achieves lower mean cumulative clean regret in all 12/12 settings, with reductions of 5.2%–32.8%. This controlled comparison supports the contribution of adaptive auditing relative to uniform allocation.
Separating estimator and risk-predictor weighting. We completed a ablation of IPW in the clean-reward, discrepancy, and residual updates ("estimator IPW") and in MLP risk-predictor training:
- Full PACE: IPW in both groups of updates.
- No estimator IPW: unit estimator weights, retaining IPW in the risk loss.
- No risk-predictor IPW: unit risk-loss weights, retaining estimator IPW.
- No IPW (all updates): unit weights in both groups.
All three weighting ablations retain the risk-based audit-policy rule, both scoring routes, and proxy feedback. Proxy-reward regression always uses unit weights. Removing weighting therefore does not turn PACE into a clean-only baseline.
The updated ablation distinguishes the two roles of weighting. Retaining risk-predictor IPW yields lower mean regret in 10/12 settings when estimator IPW is retained, and in 7/12 when estimator weights are uniform. The latter comparison is between the No estimator IPW and No IPW (all updates) variants. In contrast, retaining estimator IPW yields lower mean regret in 4/12 settings under either risk-loss configuration. Full PACE has lower mean regret than the variant without any IPW in 5/12 settings. These results support a more consistent empirical contribution from risk-predictor weighting, while showing that estimator weighting is not uniformly beneficial.
Table 7. Separate ablations of adaptive auditing and inverse-propensity weighting. All runs use and a clean-query cap of . Regret entries are mean ± sample standard deviation over 20 matched seeds; lower is better. Comparisons in the text refer to sample means.
| Dataset | Judge | Full PACE | Uniform audits | No IPW (all updates) | No estimator IPW | No risk-predictor IPW |
|---|---|---|---|---|---|---|
| BBH | Gemma | 148.60 ± 24.60 | 156.80 ± 18.35 | 142.10 ± 29.26 | 141.80 ± 31.19 | 130.80 ± 10.73 |
| BBH | GPT-4o | 137.60 ± 14.19 | 156.60 ± 18.18 | 134.90 ± 18.69 | 134.70 ± 16.01 | 136.70 ± 17.45 |
| HotpotQA | Gemma | 157.62 ± 28.50 | 198.60 ± 21.77 | 168.54 ± 41.54 | 156.27 ± 35.06 | 167.90 ± 28.64 |
| HotpotQA | GPT-4o | 163.41 ± 50.94 | 198.60 ± 21.77 | 159.39 ± 33.06 | 155.77 ± 37.04 | 165.63 ± 34.60 |
| IFBench | Gemma | 159.20 ± 51.07 | 222.50 ± 40.29 | 145.90 ± 54.27 | 154.20 ± 58.33 | 166.60 ± 48.90 |
| IFBench | GPT-4o | 149.60 ± 57.42 | 222.75 ± 40.37 | 130.80 ± 36.06 | 135.70 ± 45.03 | 163.00 ± 47.95 |
| LiveBench | Gemma | 141.81 ± 29.16 | 183.12 ± 24.25 | 138.14 ± 22.60 | 147.28 ± 31.27 | 160.55 ± 42.06 |
| LiveBench | GPT-4o | 145.74 ± 17.19 | 184.10 ± 25.06 | 134.53 ± 24.61 | 127.61 ± 27.67 | 151.84 ± 26.55 |
| Arena-Hard v2 | Gemma | 276.15 ± 29.49 | 324.91 ± 12.20 | 281.06 ± 28.27 | 290.26 ± 35.65 | 300.62 ± 36.67 |
| Arena-Hard v2 | GPT-4o | 285.00 ± 32.83 | 324.91 ± 12.20 | 291.91 ± 30.99 | 295.51 ± 31.80 | 299.77 ± 34.92 |
| MMLU-Pro | Gemma | 121.80 ± 30.47 | 145.10 ± 33.64 | 125.35 ± 35.39 | 119.60 ± 32.92 | 123.50 ± 24.65 |
| MMLU-Pro | GPT-4o | 111.60 ± 23.48 | 145.10 ± 33.64 | 133.90 ± 37.88 | 125.70 ± 33.84 | 122.25 ± 22.64 |
Weight-scale and audit-probability controls. To investigate estimator weighting, we added three controls with settings fixed across datasets. The first two retain risk-loss IPW and set estimator weights to either or . The latter preserves relative weights and is a fixed scaling, not self-normalized IPW. The third imposes an actual audit-probability floor while budget remains, uses in both groups of updates, and respects the same hard budget cap.
Continue on analysis of IPW
Table 8. Summary of the six IPW diagnostics relative to the updated full PACE results. Counts summarize comparisons of mean cumulative clean regret across the 12 dataset–judge settings.
| Variant | Settings with lower mean regret than full PACE |
|---|---|
| No estimator IPW | 8/12 |
| No IPW (all updates) | 7/12 |
| No risk-predictor IPW | 2/12 |
| Fixed estimator weights of 5 | 0/12 |
| Estimator weights scaled to | 8/12 |
| Actual audit-probability floor of 0.01 | 0/12 |
Scaling estimator IPW to yields lower mean regret than full PACE in 8/12 settings, whereas fixed estimator weights of 5 yield higher mean regret than both full PACE and unit estimator weights in all 12/12. For example, on IFBench/Gemma, gives mean regret 141.95, versus 159.20 for full PACE. These controls show that estimator-weight scale matters under the fixed regularization and confidence settings. An actual audit-probability floor of 0.01 does not lower mean regret in any of the 12 settings. Neither scale control nor the probability-floor control is uniformly preferable to full PACE.
Mechanistic diagnostics on previously recorded trajectories. We retain the earlier label-composition and fixed-history diagnostics to examine how the implemented weighting operates. The figures in this paragraph and the fixed-history results below describe those diagnostic logs; Tables 7–8 report the updated online regret comparisons. On IFBench/Gemma, the full selected-action stream has a disagreement prevalence of 49.5%, compared with 73.8% in audited samples and 63.6% in the IPW-weighted audited samples. The mean absolute discrepancy from the stream prevalence falls from 24.4 to 14.1 percentage points across seeds. The weighted proportion is used only for this diagnostic. It shows partial correction of the audited samples' label composition; closer matching of this marginal proportion alone does not establish better reward estimates, action rankings, or regret.
The implementation uses clipped IPW, . With denoting the audit indicator and the information available before the audit decision,
This factor is below one when . The clipping controls the largest weight but does not enforce a floor on the actual audit probability; no observations can be recovered after budget exhaustion. The weighting addresses audit selection along the selected-action stream and does not correct action selection itself.
Why estimator-weight scale matters. With identical audit histories, implies and hence . Keeping the ridge and confidence coefficients fixed therefore changes effective regularization, fitted means, and geometric exploration widths. A smaller geometric width does not by itself establish lower estimation noise. In the earlier fixed-history diagnostics, the original IPW produced higher mean clean and calibrated-proxy pairwise-gap prediction errors than unit weights in all 12 settings, together with mean width ratios of 0.939–0.985. These errors are measured against cached reward differences, not known conditional reward means. The diagnostics show effects on fitting and exploration and help interpret the sensitivity observed in the scale controls.
Risk-loss weighting also interacts with regularization: the MLP minimizes a weighted sum of binary cross-entropy plus a fixed penalty. Its weight scale therefore affects effective regularization and finite-step optimization as well as relative sample contributions.
Thank you for carefully reviewing our work. Here's our response to your questions.
Q1. Contribution of the two optimistic routes
We thank the reviewer for requesting a component-wise analysis. We conducted the ablations using the same question sequences, cached rewards, and 20 matched random seeds, with T=1500 and a clean-query cap of B=300. Full PACE uses the MLP risk predictor. Each ablation changes only the specified component while retaining the remaining update rules and hyperparameters.
To isolate the contribution of the two action-scoring routes, we compare
St(a)=⎩⎨⎧minUtc(a),Utp→c(a),Utc(a),Utp→c(a),full PACE,Clean route,CAL route.
Each variant selects an action maximizing its corresponding score. Both single-route variants retain the learned audit-policy rule, importance weighting, and estimator updates. In particular, the Clean-route variant still uses proxy feedback to train the risk predictor; it is not a proxy-free clean-only baseline.
Our motivation for the combined score is to allow both routes to constrain optimistic action selection. For each candidate action, the minimum uses the tighter of the two upper bounds. On the event that both bounds validly cover the clean expected reward, their minimum remains a valid upper bound. Thus, the combined score can exploit whichever route supplies the tighter upper bound.
Table 6 shows that full PACE achieves lower mean cumulative clean regret than the Clean-route variant in 8/12 dataset–judge settings and than the CAL-route variant in 11/12. The combined rule improves on both single-route variants in 8/12 settings. Its advantage over the Clean route is clearest on BBH/GPT-4o and LiveBench: mean regret decreases from 148.60 to 137.60 on BBH/GPT-4o, from 149.56 to 141.81 on LiveBench/Gemma, and from 156.52 to 145.74 on LiveBench/GPT-4o, corresponding to reductions of 7.4%, 5.2%, and 6.9%, respectively.
The reported CAL selection frequency ranges from 2.00% to 20.28% across settings, reaching 15.80% on BBH/Gemma, 20.28% on BBH/GPT-4o, and 13.07% on LiveBench/Gemma. The combined rule is designed to use whichever bound is tighter, so the two routes need not be active equally often. The route ablations directly assess the resulting regret; selection frequency describes how often the CAL bound is used and should not be equated with the fraction of decisions changed or regret saved.
Table 6. Two-route versus single-route action selection. Cumulative clean regret is reported as mean ± sample standard deviation over 20 matched seeds; lower is better. The final column reports PACE's CAL selection frequency. All runs use T=1500 and an audit cap of B=300.
These results demonstrate the empirical value of combining the two routes on several real-data benchmarks, with the strongest gains over the Clean route on BBH/GPT-4o and LiveBench. The relative contribution is task-dependent: for example, both single-route variants have lower mean regret on BBH/Gemma. We will include these ablations to show where combining the two reward estimates improves action selection and how the contribution varies across tasks. Both single-route variants retain risk-based auditing and importance weighting, so these comparisons isolate the action-scoring rule from the continued use of proxy feedback in audit allocation.