acceptodds
Withdrawn from ICLR 2027

Pay for the Difference: Exact Feedback Reduction for Language Model Comparison

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model iteration requires comparing checkpoints, but not every response score affects the comparisons of interest. Caching avoids rescoring identical responses, but still requires scores whose contributions cancel in the requested comparison. The challenge is to identify necessary scores before acquisition, accounting for the different requirements of means and covariances. Coupled generation increases response sharing while preserving each model’s output distribution; Contrast-Aware Feedback Acquisition (CAFA) then aggregates report coefficients to skip unnecessary scores and exactly recover the fully scored sample statistics. For a fixed sample bank and an otherwise unrestricted scorer, we characterize the necessary and sufficient queries for exact recovery of comparison means, either alone or jointly with ordinary sample covariances. On controlled updates, CAFA eliminates 55.8% and 90.1% of the scoring inputs remaining after caching for Qwen2.5-1.5B-Instruct and SmolLM2-1.7B-Instruct, respectively. Across three online group relative policy optimization (GRPO) trajectories, Independent Star + CAFA reaches an observed root mean square Monte Carlo standard error (RMS MCSE) of at most 0.04 at 46.1–51.3% lower recorded generation-plus-feedback cost than Independent autoregressive (AR) + Cache-only on the evaluated budget grid. All 504 report cells reproduce the fully scored statistics. We provide a modular implementation for existing response banks and external scorers, making exact feedback reduction a reusable step in language-model comparison.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model evaluation, model comparison, feedback acquisition, coupled generation, evaluation efficiency, LLM-as-a-judge, Monte Carlo estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_poHAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks how to compare language-model checkpoints when each responses score is paid for. It proposes contrast-aware feedback acquisition (or "CAFA'), which sums the report coefficients over identical scoring inputs and acquires only the scores whose aggregated coefficients is non-zero, recovering the fully-scored sample mean. Propositions 1 and 2 show that these supports are necessary for an unrestricted scorer. On controlled updates, the authors claim CAFA can remove a majority of cached inputs.

Critical strengths

The observation that the required supports depend on the requested statistic is a good contribution. I think quite naturally, a response can cancel from the mean yet remain necessary for the covariance (Figure 6; L078-079), and an absolute value restores the full support (Corollary 1).

Critical weaknesses

My concerns are attribution of the claimed computational savings, low originality, savings concentrated on ~no-ops, the use of a post-hoc precision threshold, and overall narrow scope.

  1. The abstract (L025) and Figure 1b credit 'Independent Star + CAFA' with 46.1–51.3% lower cost than 'Independent AR + Cache-only', but the arms differ in the samplers as well. The sampler is the classical maximal coupling (L456-457). CAFA alone removes only ~3% of inputs and ~2% of cost for the three-gain report (L425; Table 13). A control for Independent Star + Cache-only is absent.

  2. If I read Proposition 3 correctly, for base-relative zero-sum reports the saving over caching is at most one score per all-copy draw, and within-draw skipping equals batch aggregation in 72 of 72 cells (L1720). It seems to me that the rule amounts to not scoring a response every active policy produced, layered on Wen (2026), whose rule already gets one reward per matched component (Appendix F.5, L2854). That 'all 504 report cells' reproduce full scoring (L027) tests the implementation and says nothing of the method, in my view.

  3. The controlled endpoints are 8 optimiser steps at learning rate 1e-7 (Tables 2 and 4); on SmolLM2 the inputs removed (Table 5) are about 95% of the 2,048 root draws. Removal falls from 83.9% at step 2 (gain -0.002) to 21.6% at step 32 (gain 0.121) in Table 14, and to 2.9-3.8% for the joint report in the GRPO runs (Table 13).

  4. The 0.04 threshold is, in the authors' words, 'a post hoc readout of the completed fixed-budget grid' (Appendix D.2.3). The 46.1-51.3% claim is largely 8 versus 16 draws. MCSE also conditions on 64 fixed prompts, where the step-250 prompt-bootstrap intervals (Table 12) have half-widths of about 0.08-0.10 against an MCSE of 0.02, and coupled MCSE was checked only on synthetic simulations (Appendix F.3).

  5. The evidence in this paper comes from only one recipe (LoRA GRPO), models of 1.5B-1.7B, one 7B trajectory and one fairly cheap judge. Pairwise judges are excluded (L472) and no human raters are used (L061). Because the coupling needs full-sequence probabilities of every checkpoint, it rules out API-only blackbox models. Additionally, any unrecorded overhead equal to the fairly small CNY margin would erase the n=16 advantage (Appendix G.4).

Questions for authors
  1. Please add Independent Star + Cache-only to Figure 1b and Table 1, and TV tree under Wen's own query rule beside TV tree + CAFA.

2.How much of the 46.1-51.3% claim remains attributable to CAFA?

  1. Could cost be reported at several pre-specified thresholds (e.g., 0.02, 0.03, 0.05) from a fitted MCSE–n curve, and relative to the prompt-bootstrap interval width?

  2. Across the 72 GRPO banks, how does realised error against the 64-draw reference compare with the MCSE-based intervals under coupling?

  3. What savings follow where selection is hard (consecutive late checkpoints, larger learning rates, a 7B+ GRPO run)?

Reviewer_pVJAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work targets sample-efficient LLM-judge evaluation when benchmarking fine-tuned checkpoints against a reference policy. The core insight is algebraic: exact-match deduplication alone still queries pairs whose downstream contrastive weights sum to zero. The authors introduce CAFA, which constructs a minimal evaluation support tailored to the user's reporting objective, such as mean comparisons versus second-order covariance estimation. When paired with an Independent Star sampler that enforces maximal coupling between the base and each candidate to preserve marginal invariance while maximizing string collisions, CAFA restricts API queries strictly to non-canceling identities. Key theoretical guarantees include exact statistic recovery under minimal support, support invariance under bijective reparameterization, and support equivalence for token-level Star couplings. On standard benchmarks (Qwen2.5, SmolLM2) and online GRPO checkpoints, CAFA eliminates up to 90.1% of queries beyond naive caching, halving overall feedback expenditure (46.1 to 51.3%) while guaranteeing zero numerical error against exhaustive evaluation. Crucially, the authors delineate operational bounds, showing how the method naturally degrades under high distribution drift or when non-contrastive baseline metrics are reintroduced.

Critical strengths
  1. The problem is practical and clearly framed. Paid judge feedback is a real cost when comparing checkpoints.
  2. The method is sampler-agnostic and easy to combine with other approaches. It works with any joint sampler, including independent sampling. It can be added on top of caching, coupled generation, or budgeted label selection.
  3. The paper releases an installable package with tests, frozen banks and score caches, and scripts that regenerate the reported numbers without GPU or judge calls.
Critical weaknesses
  1. In realistic settings, CAFA provides negligible additive gains beyond coupled sampling. Ablation on fixed Independent Star GRPO banks reveals that CAFA eliminates only 2.9 to 3.8% of queries, accounting for just 1.54 to 2.07% in cost savings for the joint three-gain objective (Table 13). The reported 46.1 to 51.3% savings stem almost entirely from coupling-induced sample reduction (n=8n=8 vs. n=16n=16), a critical distinction that the abstract and Figure 1 fail to disentangle.
  2. The practical scope of this framework is constrained. Independent Star requires white-box token-level densities over a shared vocabulary, immediately ruling out closed APIs, cross-family benchmarks, and pairwise judge protocols. Furthermore, because query sharing hinges entirely on exact string identity, sequence collisions virtually vanish on extended outputs like chain-of-thought traces or agent rollouts.
  3. This paper compares its methods only against caching and other samplers. However, a recent line of work on A/B testing also exploits overlap between similar policies, using importance-weighted estimators whose variance shrinks as the policies get closer [1-2].
  4. The paper does not compare against prediction-powered inference and active inference [4-7], which address the same label-budget problem. The paper should discuss these methods and compare against at least one of them.

[1] Δ-OPEΔ\text{-}{\rm OPE}: Off-Policy Estimation with Pairs of Policies, RecSys 2024. [2] Exploiting Similarities in A/B Testing with Off-Policy Estimation, KDD 2026. [3] Active Statistical Inference, ICML 2024. [4] Stratified Prediction-Powered Inference for Hybrid Language Model Evaluation, NeurIPS 2024.

Questions for authors

Check the weaknesses above.

Reviewer_UoSxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the problem of comparing two language models based on feedback. It introduces Contrast Aware Feedback Acquisition (CAFA) to reduce the amount of feedback needed and thus reduce the cost.

Critical strengths
  • The studied problem of language model comparison is practical.
  • On the tested scenarios, the amount of reduced samples and cost is substantial.
  • The proposed approach theoretically guarantees exact recovery, which makes sure the proposed approach can be safely applied to reduce cost without affecting the comparison.
Critical weaknesses
  • The proposed approach works when there is significant overlap between the two model responses. It is unclear whether the approach works if the models are different. It will be interesting to discuss how similar the models need to be for the proposed approach to work. It will be clearer to state the assumptions for the proposed approach to be able to reduce cost.
  • The proposed approach induces some additional overhead, such as implementing a coupled sampling approach. How does the cost of overhead compare with baselines?
  • The paper evaluates two models (Qwen and SmolLM) and they are both small-scale (1.5B and 1.7B). It will be better to test the approach on larger-scale models, such as 8B or larger. This will help understand whether the proposed approach scale to larger models or not.
Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.