acceptodds
Under review as a conference paper at ICLR 2027

Reliable LLM-as-a-judge Evaluation via Transferable Calibration

Abstract

LLM-as-a-judge (LaaJ) has become a standard tool for scalable evaluation, but its reliability remains a central concern. Conformal inference (CI) can provide user-specified coverage guarantees for LaaJ outputs, but only under exchangeability. Shifts in evaluated models, topics, or response policies break this assumption and invalidate the coverage guarantee. Existing shift-aware CI methods theoretically suggest transferable calibration, but in LaaJ they face a covariate-nuisance bottleneck. We propose Robust Weighted Conformal Inference (RoWCI), which maps each example to a low-cardinality representation \(Z\), capturing coarse, transferable internal evaluation states. Through -state shift correction, RoWCI reuses source calibration data with human scores, without target-domain human scores. We further show that, under covariate shift, RoWCI provides a provable coverage degradation bound in the target domain, even when calibration is transferred without target labels. Empirically, across shifts in evaluated model, response policy, and topic, RoWCI achieves coverage closer to the nominal level with lower variance and competitive interval size than post-hoc LaaJ score calibration baselines. To our knowledge, RoWCI is the first approach to design shift-aware CI for reliable LaaJ evaluation.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLM-as-a-judge, Conformal Inference, Uncertainty, Calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jeLnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a conformal inference framework for LLM-as-a-Judge scoring under covariate shift.

Critical strengths

The core contribution of this work is to replace raw text and embeddings with low-cardinality judgment states Z for shift-aware conformal reweighting.

Critical weaknesses

While the experimental results show improved coverage alongside reduced set size, should the authors report the worst-case subpopulation coverage or group-level coverage? I also wonder how the method can achieve higher coverage and smaller set size simultaneously.

Questions for authors

Beyond the aforementioned weaknesses, descriptions for several symbols in the manuscript are ambiguous.

4 comments
AuthorsOpenReview ↗

[Answer to the mentioned weakness]

Thank you for your valuable time and feedback. We clarify below why coverage and interval size can improve together, how the ablation in Table 3 separates the contributions of scoring and reweighting, and what coverage guarantee RoWCI targets.

1. Why Coverage Can Improve While Interval Size Decreases

Coverage depends on both the interval center and its width. Shrinking an existing interval while keeping its center fixed cannot increase coverage. RoWCI changes both the scorer used to determine the center and the calibration weights used to determine the cutoff. More accurate centers and calibration that better reflects target errors can therefore produce higher coverage with smaller average intervals. The relevant comparison is coverage near the nominal level with competitive interval size; adding reweighting need not reduce interval size.

2. Both factors contribute to the performance improvement

Table 3 in the manuscript separates the contributions of scoring and reweighting using Qwen-3-30B, with HS1 for calibration and HS2 for evaluation. Acc. denotes rounded-score accuracy, and CI denotes unweighted source calibration. Interval entries report coverage / size.

AxissX(X)s_X(X) Acc.CI with sXs_XWCIsZ(Z)s_Z(Z) Acc.CI with sZs_ZRoWCI
Correctness34%84% / 0.4981% / 0.5463%86% / 0.4291% / 0.52
Helpfulness27%81% / 0.4576% / 0.4471%82% / 0.3592% / 0.40

State-based scoring improves point accuracy and produces smaller intervals with source-calibrated CI, but target coverage remains below 90%. Holding this scorer fixed, reweighting recovers nominal coverage with wider intervals. The simultaneous improvement in coverage and size relative to text-based WCI therefore reflects the combined contributions of scoring and reweighting.

3. Scope of the Coverage Guarantee

Our theoretical target is marginal coverage over the target population, with Theorem 3 quantifying degradation from imperfect representation transfer and statistical estimation. The conditional-transfer requirement concerns whether calibration-error distributions agree across domains within a state; it does not imply nominal coverage separately within every state or subgroup. Group-level coverage provides a complementary diagnostic of heterogeneity that marginal coverage does not capture, particularly for predefined subgroups with sufficient samples.

AuthorsOpenReview ↗

[Additional] Summary of the problem statement, proposed model, and contributions

We have prepared a brief summary for your quick reference.

  • Background

    • LaaJ scores are uncertain. We use conformal inference to turn these point scores into intervals with a user-specified coverage guarantee for the corresponding human scores.
    • Let XX denote the input text, such as a prompt–response pair, and YY its human score. We observe a source calibration set and consider an independent target example: = \left\lbrace (X_{i},Y_{i}) \right\rbrace_{i=1}^{n_{\mathrm{cal}}}, \qquad (X_{i},Y_{i}) \overset{\mathrm{i.i.d.}}{\sim} \mathbf{P}_{0}, \qquad (X,Y) \sim \mathbf{P}_{1}.$$ Our goal is to construct an interval $\widehat{\mathcal{I}}_{\alpha}(X)$ satisfying target-domain marginal coverage: $$\mathbb{P}_{\mathcal{D}_{\mathrm{cal}}\sim\mathbf{P}_{0}^{n_{\mathrm{cal}}},\\,(X,Y)\sim\mathbf{P}_{1}} \left( Y \in \widehat{\mathcal{I}}_{\alpha}(X) \right) \geq 1-\alpha.$$
    • How does covariate shift arise? Updating the evaluated model, replacing its backbone, or changing prompts and response policies changes the distribution of evaluated texts. We study covariate shift, where P0X≠P1X\mathbf{P}_0^X\neq\mathbf{P}_1^X while P0(Y∣X)=P1(Y∣X)\mathbf{P}_0(Y\mid X)=\mathbf{P}_1(Y\mid X). Our objective is to reuse source human scores without collecting new target human scores after each such change.
  • The challenge: Estimating calibration weights from high-dimensional text

    • Importance-weighted CI uses the source-to-target density ratio

      w(x)=dP1XdP0X(x),w(x)=\frac{d\mathbf{P}_1^X}{d\mathbf{P}_0^X}(x),

      under source–target overlap. In practice, this ratio is unknown, and we have to estimate it as w^(x)\widehat w(x), typically using density-ratio estimation with text embeddings.

    • Text embeddings can retain formatting, response length, and model-specific phrasing that distinguish domains but carry little information about judgment errors. A density-ratio estimator can rely heavily on these nuisance cues, producing highly variable or poorly estimated weights and unstable calibration.

    • Intuitive example: Suppose target responses use bullet lists more frequently than source responses, while formatting carries little information about judgment errors. A domain classifier may distinguish the domains mainly through formatting and concentrate calibration weight on a few source responses written as bullet lists. The resulting cutoff can then become sensitive to the incidental errors of those few examples.

  • The extractor: A representation that retains information needed to transfer calibration errors

    • Input: The text XX, evaluated under a fixed rubric.

    • Output: A low-dimensional judgment state Z=z(X)Z=z(X).

    • From XX to ZZ:

      • Decompose the evaluation rubric into concrete questions about the response. (The questions are reused afterward)
      • Use an LLM extractor to answer each question with a predefined discrete label.
      • Combine these answers: Z=z(X)=(z1(X),…,zK(X)),Z=z(X)=\bigl(z^1(X),\ldots,z^K(X)\bigr), where zk(X)z^k(X) is the answer to the kk-th rubric question.
    • Why this representation? Generic embeddings preserve many textual differences. Our rubric-based state restricts the weighting model to evaluation-related attributes, retaining judgment-error information while reducing nuisance cues.

    • How does this address the problem? RoWCI uses ZZ for score prediction and calibration weighting. If the conditional distribution of calibration errors given ZZ is shared across domains, changes in the overall error distribution can be explained by changes in state frequencies. Under our stated assumptions, reweighting these states therefore transfers source calibration to the target domain without new target human scores.

    • For the complete RoWCI framework, including the extractor and subsequent calibration steps, please refer to Algorithm 1.

  • Our contribution

    • Addressing the representation gap: WCI (Tibshirani et al., 2019), doubly robust calibration (Yang et al., 2022), and LR-QR (Joshi et al., 2025) address covariate shift, but applying them to LaaJ requires a representation that supports transferable score calibration. RoWCI constructs this representation using rubric-based judgment states.

    • Providing a reusable representation: Under their respective assumptions, WCI, doubly robust calibration, and LR-QR can all use ZZ, separating evaluation-relevant representation design from the calibration algorithm.

    • Connecting theory and evidence: We derive a target-coverage lower bound for imperfect state transfer and empirically show near-nominal coverage, lower variability, and competitive interval sizes across model, policy, and topic shifts.

Thank you again for your thoughtful feedback. We have provided additional explanations and ablation results to clarify how RoWCI achieves improved coverage with competitive interval sizes, as well as the precise scope of our theoretical guarantees. We understand that your time is valuable, but we would be very grateful if you could briefly review our responses. If you find that our clarifications adequately address your concerns, we would sincerely appreciate your consideration of higher rating.

Thank you for your response. After reading the other reviewers’ comments, I still have substantial concerns about the current manuscript. This work would likely require major revisions, and I will therefore keep my original score.

Reviewer_iBWdOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RoWCI, a shift-aware conformal inference framework for LLM-as-a-Judge evaluation under covariate shift. The key idea is to replace simple text-embedding-based density-ratio estimation with reweighting over a low-cardinality judgment-state representation (Z). The authors provide theoretical coverage guarantees under assumptions on the extracted representation and demonstrate improved empirical coverage relative to both post-hoc calibration methods and existing shift-aware conformal inference baselines.

Critical strengths
  • Addresses an important and timely problem: reliable uncertainty quantification for LLM evaluations under shifts.
  • The proposal of constructing a low-dimensional feature representation to improve density-ratio estimation is an interesting direction, which shows a potential to solve the bottlenecks in LLM uncertainty quantification under real-world shifts.
  • Provides an intuitive explanation for why standard embedding-based covariate-shift correction may fail due to nuisance features.
  • Includes theoretical analysis linking representation quality to target coverage degradation.
  • Empirical results show promising improvements over several calibration and conformal baselines across multiple datasets and shift types.
Critical weaknesses
  • The related work appears incomplete. Recent work on conformal prediction for LLMs under distributional shifts should be discussed, and compared as baselines in the empirical experiments. One example is "Lin, Z., Li, Y., Sarna, N., Gao, Y., & von Gablenz, M. (2025). Domain-shift-aware conformal prediction for large language models. arXiv preprint arXiv:2510.05566."
  • The Z-state construction appears largely heuristic, manually, and is not guaranteed to be optimal. The method's success depends on construction of the judgment-state representation (Z), i.e, the extracted state such that it preserves error-relevant information while removing nuisance variation, but how users should construct such states for different tasks and domains is not clear.
  • Sensitive analysis regarding Z-state construction should be performed to understand the effectiveness and limitations of current method.
Questions for authors
  • Have you explored other ways to construct Z, for example, using model internal scores like softmax outputs, or using a small labeled test set to automatically train or finetune Z?
5 comments
AuthorsOpenReview ↗

[W1] Comparison with DS-CP

Thank you for the thoughtful feedback. We additionally compare RoWCI with DS-CP [1], which is closely related to the WCI baseline already evaluated in our paper. Both estimate density ratios from text embeddings and use them to reweight calibration samples. In this additional experiment, we replace the embedding model used in our original WCI experiment with MiniLM, the model adopted by DS-CP. We evaluate the HS1-to-HS2 shift for helpfulness and correctness at α=0.1\alpha=0.1. For RoWCI, we use the updated seven-question rubrics with GPT-5.5 as the state extractor. The table reports coverage (%) and interval size.

MethodHelpfulness coverage (%)SizeCorrectness coverage (%)Size
SCI83.710.335084.470.2870
DS-CP95.240.588993.030.5383
RoWCI90.270.391090.000.3710

Although the target coverage is 90%, DS-CP produces overcoverage on both axes with substantially larger intervals. This pattern is consistent with the sensitivity of text-based density-ratio estimation to nuisance domain cues discussed in Table 2 of the paper. Such cues can help the DRE model distinguish the two domains without identifying the calibration errors that require correction. In comparison, RoWCI achieves coverage close to the desired level with smaller intervals. This comparison supports using a judgment-relevant state for density-ratio estimation.

[1] Lin et al. Domain-Shift-Aware Conformal Prediction for Large Language Models. ICML 2026.

AuthorsOpenReview ↗

[W2, W3] Sensitivity to the subquestion set

We appreciate your suggestion to examine how the choice of rubric questions affects RoWCI. Starting from the HelpSteer annotation guidelines, we repeatedly prompted GPT-6-Luna to generate slightly different sets of seven new questions for each evaluation axis. We evaluate four such sets, Z0Z_0–Z4Z_4. For each set, we use GPT-5.5 to extract the states again and evaluate the resulting intervals under the HS1-to-HS2 shift at α=0.1\alpha=0.1.

MethodHelpfulness coverage (%)SizeCorrectness coverage (%)Size
SCI83.710.33584.470.287
RoWCI (Z0Z_0)90.270.39190.000.371
RoWCI (Z1Z_1)90.110.39789.450.366
RoWCI (Z2Z_2)89.780.39589.590.365
RoWCI (Z3Z_3)90.630.40489.720.360
RoWCI (Z4Z_4)89.840.38789.460.359

Across these question sets, coverage ranges from 89.78% to 90.63% for helpfulness and from 89.45% to 90.00% for correctness, with similar interval sizes. These results indicate that guideline-based question generation and state extraction can be automated with LLMs. Although choosing the questions involves heuristic decisions, the results suggest that coverage is relatively insensitive to small changes in the questions when they jointly cover the central criteria of the evaluation guidelines.

In the following response, we further explain how we arrived at this state design by comparing it with soft states, learned bottleneck representations, and adaptation using labeled target examples.

AuthorsOpenReview ↗

[W2, W3] Alternative approaches to constructing an invariant representation

Our rubric-based ZZ was developed after trying several approaches to learning an invariant representation. As discussed in the first two paragraphs of Section 4.3, the central difficulty is to retain judgment-relevant information while removing nuisance variation without observing target labels. We examine this design problem through the additional methods suggested by the reviewer and a learned bottleneck explored during the development of RoWCI.

We evaluate the HS1-to-HS2 shift for helpfulness and correctness at α=0.1\alpha=0.1. For each alternative representation, we compare SCI (without reweighting) with density-ratio-based reweighting. Each representation has its own fitted scorer, so SCI results can differ across representations. The alternative-representation results are averaged over 20 seeds.

Helpfulness

VariantSCI coverage (%)SCI sizeReweighted coverage (%)Reweighted size
Soft-Z84.210.32083.980.319
Bottleneck84.210.34284.320.343
Adaptation (m=50m=50)85.080.35885.720.361
Adaptation (m=100m=100)85.380.36585.590.364
RoWCI——91.370.510

Correctness

VariantSCI coverage (%)SCI sizeReweighted coverage (%)Reweighted size
Soft-Z83.360.28688.400.349
Bottleneck85.010.28784.140.284
Adaptation (m=50m=50)84.250.30785.900.321
Adaptation (m=100m=100)83.550.30684.850.315
RoWCI——91.100.427

Soft-Z. We replace each hard rubric answer with a probability vector over its possible answers and concatenate the vectors across the questions. In these experiments, the extractor was fixed to Qwen3-30B because internal access to the model was required. Reweighting improves correctness coverage to 88.40%, but helpfulness coverage remains at 83.98%. Soft probabilities retain variation in the extractor's confidence, which can reflect domain-specific wording or style even when the underlying rubric judgment is similar. Such variation can provide additional nuisance cues for DRE. Retaining extraction uncertainty therefore need not produce a more transferable calibration state.

Learned bottleneck. We map MiniLM embeddings to an eight-dimensional representation using source-score supervision, then use this representation for score prediction and density-ratio estimation. Weighted coverage remains at 84.32% for helpfulness and 84.14% for correctness. Compressing an embedding does not ensure that nuisance information is removed. The supervised objective can retain features correlated with source scores even when those relationships change in the target domain. It also does not enforce agreement between the source and target conditional error distributions.

Supervised adaptation. We further fine-tune the bottleneck using 50 or 100 labeled target examples. With 50 labels, weighted coverage improves to 85.72% and 85.90%, but increasing the number to 100 does not yield a consistent further improvement. Target-score supervision provides information for adapting the representation, but minimizing score-prediction error does not directly enforce the conditional stability required for calibration. These results illustrate that optimizing score prediction with additional supervision and preserving a transferable conditional error distribution are distinct objectives.

Among the tested representations, the rubric-based state achieves coverage closest to the nominal 90% level on both axes, without the target labels used by the adaptation variants. The question-set sensitivity experiment further shows that similar coverage can be obtained from different question sets generated from the evaluation guidelines.

Reviewer_6ijxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This is a paper on LLM-as-a-Judge (LaaJ) methods for evaluating settings where there is some sort of context shift between the source and target distributions; here “source” refers to the distribution on which we originally have evaluations, and “target” refers to the distribution we are now evaluating. Let (X,Y) be datapoints, where X is the evaluation text and Y is a human score (assessing e.g., the text’s “usefulness”). Let S = s_X(X) be the judge score and E = |S - Y| be the judge error. The human-scoring distribution remains the same for the source and target evaluation, i.e., P_0(Y | X) = P_1(Y | X). For the source data, we assume we have both X’s and Y’s (i.e., human data is included) but for the target distribution, we only have the target covariates and no human data. The goal is to construct a prediction interval \hat{I}{\alpha}(X) for (X,Y) \sim P_1 such that the probability that Y \in \hat{I}{\alpha}(X) is at least 1 - \alpha. It was hard for me to understand the method proposed, but here’s what I could deduce: the authors find a way to capture some information about the correlation between the errors in P_0 and P_1 — this is what makes the representation Z. If I understand correctly, Z is a codebook-style rubric that “guides” the score generation. Assuming that the rubric stays the same, means that you can transfer knowledge from P_0 to P_1.

Critical strengths

[S1] I actually like the problem that the paper focuses on a lot, and I think it can be quite useful for social science evaluation settings.

Critical weaknesses

[W1] Clarity concerns: In general, I found the introduction of the paper (which included the problem motivation and a high-level overview of the approach) pretty hard to parse. I had fundamental questions about what exactly is the motivation of the problem and why standard methods don’t work which I addressed using Codex myself. Had I not used it, I would not have understood the setup. Some concrete pointers of clarity issues of the intro:

  • I had a hard time understanding why general shift-aware CI methods don’t work in this case. This is paragraph 3 of the intro. There’s some wording issues, but also the high-level argument is unclear. I had to ask an LLM to help me interpret why general methods can’t work.
  • Line 56-57: “turns a raw-text shift into a Z-state frequency shift”: this concept hasn’t been introduced yet. There’s some mention of it in the abstract, but I personally found that it wasn’t a clear exposition of what the concept captures.
  • For Section 3: I would strongly suggest that the authors include the explanation of what exactly Z is in this section because it is unclear otherwise.
  • Section 4: the authors started using the phrase “raw X-space” and “text space”. I could not understand what the difference was, and why we needed the extra jargon.
  • Section 4 in general: the results are either stated in very technical terms with very little high level explanation, or the explanation feels not as clear as it should be.
  • What are some examples of the representation Z mentioned in Section 4?

[W2] It’s unclear to me where exactly the paper’s novelty lies. The paper is connected to works in NLP, Statistics, and the Social Sciences / Political Science. As such, there are several papers (from each of these disciplines) that are potentially related. For example, it seems that a version of the idea of using a low-dimensional representation and estimating the weights of that representation appears in Stojanov et al., 2019. Their work doesn’t construct conformal intervals or scoring rubrics, but the dimension-reduction arguments seem to be similar. The “Z-state-frequency shift” seems to also connect to Bhattacharyya and Barber’s “Group-Weighted Conformal Prediction” although they define “groups” instead of rubric-induced states. I also want to encourage the authors to do a deep literature review for this paper for papers from Political Science methodology.

Additional Comments

  • The AI statement should be more detailed in my view. I can think of many ways to “utilize coding agents” (which is what the paper states) — for example, did the authors use AI to derive any of the results? to brainstorm ideas? to run the experiments?
  • I found that the abstract got technical and jargon-y rather abruptly. I would have preferred a more high-level description of the paper.
Questions for authors

N/A

6 comments
AuthorsOpenReview ↗

[W1] Clarity of presentation

We have revised the Introduction to explain the motivation, the practical difficulty of text-based reweighting, and the definition and role of ZZ more explicitly. The revised text is highlighted in blue in the manuscript. We would be very grateful if you could review the revised manuscript.

The specific changes are as follows:

Introduction: Clarified the distinction between score alignment and coverage guarantees, the need for transferable calibration, and the practical difficulty of estimating importance weights from text.

Section 3: Defined ZZ as a vector of categorical answers to rubric questions and distinguished it from the extracted state Z^\widehat Z.

Section 4 terminology: Replaced “raw XX-space” and “text space” with “text input XX” to remove unnecessary terminology.

Section 4 theory: Revised the explanations of Theorems 1–3 to clarify why state-frequency reweighting suffices under the stated condition, how additional text variation affects exact-weight variance, and how practical approximation errors affect coverage.

Section 4.3 examples: Added concrete rubric questions and answer values to illustrate ZZ, with references to the complete schemas in Appendix A.1 and the worked examples in Table 9.

AuthorsOpenReview ↗

[W2] Comparison with previous studies

Thank you for your valuable time and feedback. While RoWCI may appear similar to the two studies you mentioned, it differs in both the problem it addresses and its core contributions.

Comparison with Stojanov et al. [1].

We share the motivation of performing importance weighting in a low-dimensional, prediction-relevant representation with Stojanov et al. They learn a low-dimensional projection using source labels, aiming to preserve information relevant to predicting the target variable. Conceptually, this introduces a bottleneck that restricts the representation to a smaller space while encouraging it to retain label-relevant information. However, the supervision signal only establishes which features are predictive under the source distribution. In our LaaJ setting, high-dimensional text may contain source-specific patterns, such as characteristic writing styles or formatting, that happen to correlate with human scores. A source-supervised bottleneck can retain these patterns without being penalized, even if their relationship with human scores does not transfer to the target distribution. Since target human scores are unavailable, good source predictive performance alone cannot establish whether the learned representation preserves the conditional distribution needed for target calibration.

We examined a related approach by training a low-dimensional bottleneck on text embeddings using source-score supervision. This alternative did not achieve the nominal target coverage in our experiments. We would appreciate it if the reviewer could refer to our response to Reviewer iBWd, “[W2, W3] Alternative approaches to constructing an invariant representation,” for the setup and results. These findings support our rubric-based construction. An LLM extracts answers to explicit evaluation questions to form the judgment state, without fitting the state extractor to source human scores. Our contribution is therefore not low-dimensional reweighting itself, but a concrete representation design for transferable calibration of LaaJ judgments.

Comparison with Bhattacharyya and Barber [2].

Viewing each judgment state Z=zZ=z as a group and the state-frequency ratio as a group weight reveals a natural connection to group-weighted conformal prediction. However, their work takes the group structure as given and assumes that the joint distribution of the remaining features and response is unchanged within each group. Its contribution is sharper coverage bounds for estimated group weights under this structure. RoWCI addresses a preceding question, which is "how can LaaJ text be transformed into a representation that supports stable reweighting and transferable calibration?" Assigning responses to groups does not by itself ensure that their conditional calibration-error distributions transfer. RoWCI provides a rubric-based procedure for constructing these states and analyzes the coverage loss when the extracted state does not preserve the required conditional distribution. Thus, an ideal transferable state connects RoWCI to group-weighted inference, but the contributions differ. [2] refines inference given a valid group structure, whereas RoWCI constructs and analyzes the state representation needed for calibration under text distribution shift.

[1] Stojanov et al. Low-Dimensional Density Ratio Estimation for Covariate Shift Correction. AISTATS 2019.

[2] Bhattacharyya and Barber. Group-weighted conformal prediction. Electronic Journal of Statistics 2026.

AuthorsOpenReview ↗

[Additional] Summary of the problem statement, proposed model, and contributions

It is unfortunate that you encountered difficulties in reading our paper. We have prepared a brief summary for your quick reference.

  • Background

    • LaaJ scores are uncertain. We use conformal inference to turn these point scores into intervals with a user-specified coverage guarantee for the corresponding human scores.
    • Let XX denote the input text, such as a prompt–response pair, and YY its human score. We observe a source calibration set and consider an independent target example: = \left\lbrace (X_{i},Y_{i}) \right\rbrace_{i=1}^{n_{\mathrm{cal}}}, \qquad (X_{i},Y_{i}) \overset{\mathrm{i.i.d.}}{\sim} \mathbf{P}_{0}, \qquad (X,Y) \sim \mathbf{P}_{1}.$$ Our goal is to construct an interval $\widehat{\mathcal{I}}_{\alpha}(X)$ satisfying target-domain marginal coverage: $$\mathbb{P}_{\mathcal{D}_{\mathrm{cal}}\sim\mathbf{P}_{0}^{n_{\mathrm{cal}}},\\,(X,Y)\sim\mathbf{P}_{1}} \left( Y \in \widehat{\mathcal{I}}_{\alpha}(X) \right) \geq 1-\alpha.$$
    • How does covariate shift arise? Updating the evaluated model, replacing its backbone, or changing prompts and response policies changes the distribution of evaluated texts. We study covariate shift, where P0X≠P1X\mathbf{P}_0^X\neq\mathbf{P}_1^X while P0(Y∣X)=P1(Y∣X)\mathbf{P}_0(Y\mid X)=\mathbf{P}_1(Y\mid X). Our objective is to reuse source human scores without collecting new target human scores after each such change.
  • The challenge: Estimating calibration weights from high-dimensional text

    • Importance-weighted CI uses the source-to-target density ratio

      w(x)=dP1XdP0X(x),w(x)=\frac{d\mathbf{P}_1^X}{d\mathbf{P}_0^X}(x),

      under source–target overlap. In practice, this ratio is unknown, and we have to estimate it as w^(x)\widehat w(x), typically using density-ratio estimation with text embeddings.

    • Text embeddings can retain formatting, response length, and model-specific phrasing that distinguish domains but carry little information about judgment errors. A density-ratio estimator can rely heavily on these nuisance cues, producing highly variable or poorly estimated weights and unstable calibration.

    • Intuitive example: Suppose target responses use bullet lists more frequently than source responses, while formatting carries little information about judgment errors. A domain classifier may distinguish the domains mainly through formatting and concentrate calibration weight on a few source responses written as bullet lists. The resulting cutoff can then become sensitive to the incidental errors of those few examples.

  • The extractor: A representation that retains information needed to transfer calibration errors

    • Input: The text XX, evaluated under a fixed rubric.

    • Output: A low-dimensional judgment state Z=z(X)Z=z(X).

    • From XX to ZZ:

      • Decompose the evaluation rubric into concrete questions about the response. (The questions are reused afterward)
      • Use an LLM extractor to answer each question with a predefined discrete label.
      • Combine these answers: Z=z(X)=(z1(X),…,zK(X)),Z=z(X)=\bigl(z^1(X),\ldots,z^K(X)\bigr), where zk(X)z^k(X) is the answer to the kk-th rubric question.
    • Why this representation? Generic embeddings preserve many textual differences. Our rubric-based state restricts the weighting model to evaluation-related attributes, retaining judgment-error information while reducing nuisance cues.

    • How does this address the problem? RoWCI uses ZZ for score prediction and calibration weighting. If the conditional distribution of calibration errors given ZZ is shared across domains, changes in the overall error distribution can be explained by changes in state frequencies. Under our stated assumptions, reweighting these states therefore transfers source calibration to the target domain without new target human scores.

    • For the complete RoWCI framework, including the extractor and subsequent calibration steps, please refer to Algorithm 1.

  • Our contribution

    • Addressing the representation gap: WCI (Tibshirani et al., 2019), doubly robust calibration (Yang et al., 2022), and LR-QR (Joshi et al., 2025) address covariate shift, but applying them to LaaJ requires a representation that supports transferable score calibration. RoWCI constructs this representation using rubric-based judgment states.

    • Providing a reusable representation: Under their respective assumptions, WCI, doubly robust calibration, and LR-QR can all use ZZ, separating evaluation-relevant representation design from the calibration algorithm.

    • Connecting theory and evidence: We derive a target-coverage lower bound for imperfect state transfer and empirically show near-nominal coverage, lower variability, and competitive interval sizes across model, policy, and topic shifts.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank all three reviewers (6ijx, iBWd, jeLn) for their constructive feedback. Below, we summarize the strengths recognized by the reviewers and our main revisions and additional experiments.

Strengths highlighted by reviewers

  • [6ijx, iBWd] Problem significance: Reliable LLM-as-a-judge evaluation under distribution shift is an important problem. Reviewer 6ijx particularly highlighted its potential usefulness.
  • [iBWd, jeLn] Representation contribution: Using rubric-based judgment states for shift-aware conformal reweighting, with the supporting theory and empirical improvements highlighted by iBWd.
  • [6ijx] Improved exposition: The reviewer described our rebuttal summary as good, the revised Introduction as much clearer, and the added technical explanations as “a great step and in the right direction.”
  • [iBWd] Stronger experimental support: The reviewer stated that the added experiments make the method more sound and better support the contributions, and described the paper as valuable work to publish.

Main concerns and our responses

Clarity of the setup, methodology, and theory ([6ijx, jeLn]). We added Figure 1 to illustrate the complete procedure, from concrete rubric questions and state extraction to reweighting and interval construction. Sections 3–4 now explain what constitutes a judgment state, why state-only scoring is used, and how scoring and calibration weighting contribute to the final interval. A numerical two-state example and expanded explanations accompany the theoretical results.

Positioning and comparison with related work ([6ijx, iBWd]). Section 2 now discusses Stojanov et al., group-weighted conformal prediction, and DS-CP. We clarify RoWCI’s contribution as designing a judgment-relevant representation for transferable LaaJ calibration. In the additional HS1-to-HS2 comparison, RoWCI achieves 90.27% and 90.00% coverage on helpfulness and correctness, respectively, with smaller intervals than DS-CP on both axes.

Construction and robustness of the judgment state ([iBWd]). Appendix C, Table 5 evaluates five question sets generated from the evaluation guidelines. Coverage remains between 89.45% and 90.63% across the two axes, supporting robustness to the tested question variations. Our rebuttal also compares soft states, learned bottlenecks, and adaptation with limited target labels, providing further empirical support for the rubric-based design.

Coverage, interval size, and guarantee scope ([jeLn]). Table 3 separates the contributions of scoring and reweighting: improved interval centers can increase efficiency, while reweighting corrects the calibration cutoff under shift. Their combination explains improved coverage with competitive interval size. We also clarify that our theoretical target is target-population marginal coverage, with Theorem 3 quantifying losses from imperfect representation transfer and statistical estimation; within-state transferability does not itself imply coverage within every subgroup.

These revisions clarify RoWCI’s contribution and strengthen the evidence for transferable calibration through rubric-based judgment states. We sincerely appreciate the reviewers’ constructive suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.