- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a conformal inference framework for LLM-as-a-Judge scoring under covariate shift.
Critical strengths
The core contribution of this work is to replace raw text and embeddings with low-cardinality judgment states Z for shift-aware conformal reweighting.
Critical weaknesses
While the experimental results show improved coverage alongside reduced set size, should the authors report the worst-case subpopulation coverage or group-level coverage? I also wonder how the method can achieve higher coverage and smaller set size simultaneously.
Questions for authors
Beyond the aforementioned weaknesses, descriptions for several symbols in the manuscript are ambiguous.
4 comments
[Additional] Summary of the problem statement, proposed model, and contributions
We have prepared a brief summary for your quick reference.
-
Background
- LaaJ scores are uncertain. We use conformal inference to turn these point scores into intervals with a user-specified coverage guarantee for the corresponding human scores.
- Let denote the input text, such as a prompt–response pair, and its human score. We observe a source calibration set and consider an independent target example: = \left\lbrace (X_{i},Y_{i}) \right\rbrace_{i=1}^{n_{\mathrm{cal}}}, \qquad (X_{i},Y_{i}) \overset{\mathrm{i.i.d.}}{\sim} \mathbf{P}_{0}, \qquad (X,Y) \sim \mathbf{P}_{1}.$$ Our goal is to construct an interval $\widehat{\mathcal{I}}_{\alpha}(X)$ satisfying target-domain marginal coverage: $$\mathbb{P}_{\mathcal{D}_{\mathrm{cal}}\sim\mathbf{P}_{0}^{n_{\mathrm{cal}}},\\,(X,Y)\sim\mathbf{P}_{1}} \left( Y \in \widehat{\mathcal{I}}_{\alpha}(X) \right) \geq 1-\alpha.$$
- How does covariate shift arise? Updating the evaluated model, replacing its backbone, or changing prompts and response policies changes the distribution of evaluated texts. We study covariate shift, where while . Our objective is to reuse source human scores without collecting new target human scores after each such change.
-
The challenge: Estimating calibration weights from high-dimensional text
-
Importance-weighted CI uses the source-to-target density ratio
under source–target overlap. In practice, this ratio is unknown, and we have to estimate it as , typically using density-ratio estimation with text embeddings.
-
Text embeddings can retain formatting, response length, and model-specific phrasing that distinguish domains but carry little information about judgment errors. A density-ratio estimator can rely heavily on these nuisance cues, producing highly variable or poorly estimated weights and unstable calibration.
-
Intuitive example: Suppose target responses use bullet lists more frequently than source responses, while formatting carries little information about judgment errors. A domain classifier may distinguish the domains mainly through formatting and concentrate calibration weight on a few source responses written as bullet lists. The resulting cutoff can then become sensitive to the incidental errors of those few examples.
-
-
The extractor: A representation that retains information needed to transfer calibration errors
-
Input: The text , evaluated under a fixed rubric.
-
Output: A low-dimensional judgment state .
-
From to :
- Decompose the evaluation rubric into concrete questions about the response. (The questions are reused afterward)
- Use an LLM extractor to answer each question with a predefined discrete label.
- Combine these answers: where is the answer to the -th rubric question.
-
Why this representation? Generic embeddings preserve many textual differences. Our rubric-based state restricts the weighting model to evaluation-related attributes, retaining judgment-error information while reducing nuisance cues.
-
How does this address the problem? RoWCI uses for score prediction and calibration weighting. If the conditional distribution of calibration errors given is shared across domains, changes in the overall error distribution can be explained by changes in state frequencies. Under our stated assumptions, reweighting these states therefore transfers source calibration to the target domain without new target human scores.
-
For the complete RoWCI framework, including the extractor and subsequent calibration steps, please refer to Algorithm 1.
-
-
Our contribution
-
Addressing the representation gap: WCI (Tibshirani et al., 2019), doubly robust calibration (Yang et al., 2022), and LR-QR (Joshi et al., 2025) address covariate shift, but applying them to LaaJ requires a representation that supports transferable score calibration. RoWCI constructs this representation using rubric-based judgment states.
-
Providing a reusable representation: Under their respective assumptions, WCI, doubly robust calibration, and LR-QR can all use , separating evaluation-relevant representation design from the calibration algorithm.
-
Connecting theory and evidence: We derive a target-coverage lower bound for imperfect state transfer and empirically show near-nominal coverage, lower variability, and competitive interval sizes across model, policy, and topic shifts.
-
Thank you again for your thoughtful feedback. We have provided additional explanations and ablation results to clarify how RoWCI achieves improved coverage with competitive interval sizes, as well as the precise scope of our theoretical guarantees. We understand that your time is valuable, but we would be very grateful if you could briefly review our responses. If you find that our clarifications adequately address your concerns, we would sincerely appreciate your consideration of higher rating.
Thank you for your response. After reading the other reviewers’ comments, I still have substantial concerns about the current manuscript. This work would likely require major revisions, and I will therefore keep my original score.
[Answer to the mentioned weakness]
Thank you for your valuable time and feedback. We clarify below why coverage and interval size can improve together, how the ablation in Table 3 separates the contributions of scoring and reweighting, and what coverage guarantee RoWCI targets.
1. Why Coverage Can Improve While Interval Size Decreases
Coverage depends on both the interval center and its width. Shrinking an existing interval while keeping its center fixed cannot increase coverage. RoWCI changes both the scorer used to determine the center and the calibration weights used to determine the cutoff. More accurate centers and calibration that better reflects target errors can therefore produce higher coverage with smaller average intervals. The relevant comparison is coverage near the nominal level with competitive interval size; adding reweighting need not reduce interval size.
2. Both factors contribute to the performance improvement
Table 3 in the manuscript separates the contributions of scoring and reweighting using Qwen-3-30B, with HS1 for calibration and HS2 for evaluation. Acc. denotes rounded-score accuracy, and CI denotes unweighted source calibration. Interval entries report coverage / size.
State-based scoring improves point accuracy and produces smaller intervals with source-calibrated CI, but target coverage remains below 90%. Holding this scorer fixed, reweighting recovers nominal coverage with wider intervals. The simultaneous improvement in coverage and size relative to text-based WCI therefore reflects the combined contributions of scoring and reweighting.
3. Scope of the Coverage Guarantee
Our theoretical target is marginal coverage over the target population, with Theorem 3 quantifying degradation from imperfect representation transfer and statistical estimation. The conditional-transfer requirement concerns whether calibration-error distributions agree across domains within a state; it does not imply nominal coverage separately within every state or subgroup. Group-level coverage provides a complementary diagnostic of heterogeneity that marginal coverage does not capture, particularly for predefined subgroups with sufficient samples.