acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Transfer of Representational Signatures for LLM Reliability Detection

Abstract

Large language models (LLMs) are reported to be susceptible to reliability issues including hallucinations and reasoning errors. Existing quality detectors can predict unreliable outputs based on internal representations with decent accuracies. However, training these detectors often require substantial domain-specific labeled data, which limits their applicability to new domains. We propose to address this challenge by transferring representational signatures across related domains. More specifically, we build a statistical framework that first extracts low-dimensional signatures for each domain, then uses cross-fitting to learn composition weights of domain-specific predictors. The weights adapt to each source domain’s predictive relevance to the target domain. Our theoretical analysis in the random matrix regime characterizes how prediction error and weights depend on target sample sizes and the statistical differences across domains. Experiments across multiple hallucination datasets and open-weight LLMs show that, with only 25 labeled questions in the new domain, adaptive transfer improves hallucination-detection AUROC by 7.8%. For reasoning-step prediction, it improves AUROC by 7.7%. The learned representational directions can also be used for activation steering to improve answer correctness with limited degradation in generation quality.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
large language models, hallucination detection, multi-source transfer, mechanistic interpretability, cross-fitting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yLLLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AdaST builds domain-specific reliability detectors from principal components of within-question response contrasts, then combines their scores using question-level cross-fitting and simplex-constrained logistic regression. The paper reports benefits in low-target-data hallucination detection and reasoning-step correctness prediction, analyzes a fixed-subspace least-squares analogue, and explores activation steering.

Critical strengths
  1. AdaST addresses a practical problem with a carefully designed framework. Adapting reliability detectors with limited target labels is valuable for open-weight LLM deployment. Question-level cross-fitting refits both the target representation and detector, addressing the target expert’s in-sample advantage. Source-only layer selection further strengthens the methodological design.
  2. The experiments demonstrate benefits across models, domains, and tasks. At 25 target questions or prefixes, AdaST improves average AUROC over the target-only detector by 7.8 percentage points for hallucination detection and 7.7 percentage points for reasoning-step correctness. Improvements with SinkProbe experts further suggest that the transfer mechanism is useful beyond the default representation.
  3. The analysis provides complementary theoretical and empirical insights. The clearly scoped regression analogue characterizes transfer through domain discrepancies and representation bias, supported by detailed derivations. Learned weights illustrate changing source contributions, while the exploratory steering experiment shows improved semantic accuracy with little change in abstention.
Critical weaknesses
  1. Additional matched-supervision baselines would clarify the contribution of adaptive weighting. Tables 1–4 demonstrate gains over target-only detectors and existing methods. Comparisons with pooled source-and-target training, uniform expert averaging, and conventional stacking without simplex constraints would further establish which benefits come from the particular combination strategy.
  2. The supervision budget and evaluation protocol need clearer reporting. The paper defines the QA budget in terms of original questions, with the 25-question setting corresponding to 375 automatically labeled responses. However, the data used to select kmaxk_{max} and whether any target labels used for this selection count toward the reported budget should be explicitly stated. Reporting per-domain training, validation, and test sizes, the number of repetitions, and the units used to compute standard errors for the main detection experiments would also improve reproducibility. For reasoning-step evaluation, the paper should clarify whether prefixes from the same original problem are kept together across data splits and cross-fitting folds. Reporting retention rates under the PRM thresholds and agreement with independent correctness labels would further help establish the scope of the results.
  3. Automatic labeling and candidate selection limit the scope of the evaluation. Hallucination labels rely on a fixed BLEURT-20 threshold of 0.5, while reasoning examples are selected using extreme scores from Qwen2.5-Math-PRM-7B. The labeling details should be stated more clearly, including the final sample counts and label distributions for QA, as well as per-domain retention rates and final sample counts for reasoning prefixes. In addition, was a sample of these labeled examples manually inspected to assess label quality?
Questions for authors
  1. How much improvement remains over pooled training, uniform averaging, and conventional stacking with exactly the same supervision and tuning budget?
  2. What data are used to select kmaxk_{max}, how are prefixes from the same original problem grouped across splits, and what are the PRM retention rates and levels of agreement with independent correctness labels? Please also identify the units underlying the reported standard errors.
2 comments
AuthorsOpenReview ↗

Weakness 1, 2 (Question1, 2)

We thank the reviewer for the constructive suggestions. We have conducted additional matched-budget comparisons and clarified the supervision, evaluation, and labeling protocols. The main results in the manuscript are based on the full datasets, while the additional experiments reported in this rebuttal use a sampled subset of 1,000 questions per dataset. This accounts for differences in absolute AUROC values between the original results and the additional experiments.

W1. Matched-supervision aggregation baselines

We evaluated pooled source–target training, uniform averaging, similarity-based weighting, and unconstrained stacking using the same target-label budget and evaluation splits within a paired comparison. We consider both a homogeneous QA source pool and a heterogeneous pool that additionally includes seven MATH reasoning domains.

MethodHomogeneousHeterogeneous
AdaST79.8079.72
Pooled training80.2778.03
Similarity-weighted80.1577.30
Uniform averaging79.9768.59
Unconstrained stacking79.1178.50

Pooled training and simple averaging are competitive with AdaST when all sources are related QA domains. However, their performance deteriorates when mismatched MATH sources are introduced, whereas AdaST maintains nearly unchanged performance. These results clarify that the principal advantage of adaptive weighting is its robustness to variations in source relevance.

Revised manuscript: Section 4.2 Table 2, and Appendix B.2 Tables 13–14

W2. Supervision budget and evaluation protocol

Supervision budget and evaluation protocol. We selected kmax⁡k_{\max} through 5-fold cross-validation on source-domain data, considering K∈{8,16,32,64,128}K \in \{8,16,32,64,128\}. The selected kmax⁡=64k_{\max}=64 and model-specific representation layers were fixed across all reported experiments. No additional target labels were used for hyperparameter selection. For QA, we use a 70/30 train–test split at the original-question level, without a separate target validation set. At nT=25n_T=25, we sample 25 training questions, each contributing 15 automatically labeled responses. All responses from the same question remain together during data splitting and three-fold cross-fitting.

Reasoning-step evaluation. For PRM800K, we directly reuse the model-specific representation layers selected for QA, without additional tuning on reasoning data. We group prefixes by their original MATH problem ID (math_id) and use an 80/20 train–test split at the problem level. Cross-fitting also respects these groups, ensuring that prefixes from the same problem never appear in different folds or across training and test sets. Target budgets count retained prefixes sampled from the training groups.

Repetitions and standard errors. The main experiments use three split seeds and three budget-sampling repetitions, yielding nine evaluations per model–target combination. Within-cell standard errors are computed across these nine evaluations. Aggregate standard errors use model–target combinations as the sampling units (30 for QA), while paired comparisons compute differences within each combination before estimating the standard error.

Revised manuscript: Appendix C.1

AuthorsOpenReview ↗

Weakness 3

W3. Automatic labeling and candidate selection

QA sample counts and label distributions. We report the question-level splits and response-label distributions below. Each question contributes 15 generated responses, and the positive rate varies across the five evaluated backbones.

DatasetTrain questionsTest questionsResponses per backboneMean positive rate
NQ2,5271,08354,1508.2%
TriviaQA2,5271,08354,15030.5%
CoQA5,5882,395119,74551.2%
HaluEval-QA7,0003,000150,00034.0%
SimpleQA3,0241,29764,8156.4%
HotpotQA5,1832,222111,07510.7%

PRM retention. For each reasoning prefix, we generate eight candidate next steps and retain the prefix only if at least one candidate receives a PRM score of at least 0.8 and another receives a score of at most 0.2. We select the highest-scoring positive and lowest-scoring negative candidate, producing two labeled steps per retained prefix. The retention statistics, averaged over five backbones, are as follows.

MATH domainOriginal prefixesRetained prefixesRetention rate
Algebra38927871.4%
Counting and Probability28517661.7%
Geometry32519359.4%
Intermediate Algebra84050259.8%
Number Theory31118960.7%
Prealgebra1409970.9%
Precalculus35823465.5%

We assess agreement between PRM predictions and independent human correctness labels.

MATH domainCandidatesRetainedRetention rateAgreementAUROC
Algebra20017085.0%80.6%79.00
Precalculus19715578.7%80.0%76.46
Counting & Probability18212568.7%76.8%72.20
Intermediate Algebra19313871.5%76.1%78.60
Geometry18411260.9%75.9%70.93
Prealgebra18114177.9%74.5%67.63
Number Theory18813471.3%71.6%66.73
Overall1,32597573.6%76.7%73.29

Retention denotes the proportion of individual candidates with PRM scores outside the 0.2–0.8 interval, rather than the proportion of prefixes containing both positive and negative candidates. The retention rates indicate that our filtering procedure preserves a substantial portion of eligible prefixes across all seven domains rather than concentrating on a small subset.

Label quality and sensitivity. We acknowledge that automatic correctness labels are imperfect. For QA, we additionally evaluated token-F1 and Qwen3-14B LLM-judge labels, retraining the detection and transfer pipeline under each criterion. AdaST retains positive transfer gains under both criteria. We also varied the BLEURT threshold from 0.40 to 0.60 and the positive and negative PRM thresholds, observing positive transfer gains across the tested settings.

Revised manuscript: Section 4.1, Appendix B.3 Tables 15–20

Reviewer_csCgOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper focuses on the adaptive transfer of supervised hallucination detection methods between datasets from similar domains. The authors propose a method that first trains a supervised detector using logistic regression on hidden states and then transfers it from the source domain to the target domain through cross-fitting, using only a few instances from the target domain. The authors provide a theoretical analysis showing that the proposed aggregation technique should perform at least as well as target-only detection. The experimental evaluation demonstrates consistent improvements and applicability to other detectors and features.

Critical strengths
  1. The paper considers an important question of the reliable transfer of hallucination detectors between datasets.
  2. The proposed method can be applied to most existing methods and requires only a few instances from the target dataset to achieve good performance.
Critical weaknesses
  1. The design choices of the method are questionable, and some of them seem to be unnecessary. First, the paraphrase-based behavioral subspace component raises several questions: the stability of the paraphrases, how many are needed and why; the necessity of applying PCA, which is not ablated; why PCA is applied only to residuals and not to the hidden states themselves; the use of only the Q±sQ_{\pm}^s subset; the use of only one layer and one token's hidden state, which is also not ablated; and which instances are used for training: all instances or only those from the Q±sQ_{\pm}^s subset. Second, Section 2.1 (bullet a in Figure 1) seems to be completely unnecessary because the proposed target-only detector performs worse than many target-only baselines. The final part of the method, as shown in Section 4.3, works with other methods as well and outperforms the full AdaST method. Thus, I do not see any reason to include this part of the method. Overall, the method should focus more on the ensembling component rather than on the behavioral subspaces, which do not seem to work.
  2. The theoretical analysis covers a least-squares analogue with fixed subspaces, Gaussian design, and fixed-dimension asymptotics, while the method uses logistic regression, data-dependent PCA subspaces, and question-level cross-fitting. The theory therefore does not explain how these assumptions and results connect to the actual method. It also does not provide any intuition for the propositions and definitions.
  3. The paragraph on cross-fitting (lines 197–206) is confusing. What is Std−kStd{-}k? At this stage, all source detectors are trained and fixed. Therefore, is only the target detector trained, while Std−kStd{-}k? is applied to the predictions of the other detectors?
  4. The experimental evaluation is very limited. First, the paper considers rather outdated LLMs, such as OPT, the first version of Gemma, Qwen2, and the first version of Mistral. Therefore, the applicability of the method to current state-of-the-art LLMs is unclear. Second, only QA tasks are considered. It would be very interesting to see how this method performs when transferred to new domains and tasks. Even if it does not work well in these settings, I think it is crucial to investigate this limitation. Third, BLEURT is a rather unreliable choice for evaluating generation correctness. Recent related work suggests using LLM-as-a-judge for more reliable evaluation [1]. Fourth, several supervised baselines are missing: attention-based [2,3,4] and hidden-state-based [5,6]. It also lacks unsupervised baselines [7], which are crucial for positioning a method that requires limited training data. Is the proposed method actually better than unsupervised methods?. Fifth, since cross-fitting seems to be the only component of the method that actually works, it is unclear how much of an improvement it provides. Therefore, it should be evaluated more carefully against appropriate baselines, such as averaging all predictions from the source and target detectors, as well as a weighted average using simple rules (for example, but not limited to, 12N\frac{1}{2N} for each of the NN source detectors and 12\frac{1}{2} for the target detector).
  5. Minor issues:
    1. What does "semantic similarity across domains" mean in the context of line 143?
    2. Why does SinkProbe for OPT differ between Table 3 and Table 2, while it matches for Mistral?

[1] Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results (Santilli et al., ACL 2025)
[2] Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models (Vazhentsev et al., EMNLP 2025)
[3] Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps (Chuang et al., EMNLP 2024)
[4] A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs (Shelmanov et al., EMNLP 2025)
[5] Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models (Vazhentsev et al., NAACL 2025)
[6] Do Androids Know They’re Only Dreaming of Electric Sheep? (CH-Wang et al., Findings 2024)
[7] Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph (Vashurin et al., TACL 2025)

Questions for authors

See weaknesses 1 and 3

5 comments
AuthorsOpenReview ↗

Weakness 1

We thank the reviewer for the detailed feedback. We have conducted additional experiments, revised the theoretical analysis, and clarified several aspects of the implementation. We address the concerns in the order raised. The main results in the manuscript are based on the full datasets, while the additional experiments reported in this rebuttal use a sampled subset of 1,000 questions per dataset. This accounts for differences in absolute AUROC values between the original results and the additional experiments.

W1. Method design and ablations

(a) Paraphrase stability

We conducted additional ablations to examine the effect of paraphrase augmentation and its interaction with cross-fitting.

ParaphrasesWith cross-fittingWithout cross-fitting
P=1P=177.3 (1.6)75.4 (1.6)
P=2P=278.1 (1.4)76.6 (1.5)
P=3P=379.8 (1.1)78.6 (1.1)

AUROC (%). Parentheses report standard errors.

Using multiple paraphrases allows the behavioral subspace to capture correctness-related differences across different formulations of the same question. AUROC increases with the number of paraphrases both with and without cross-fitting, while the standard errors decrease. Cross-fitting provides additional gains across all three settings, indicating that the benefit of paraphrase augmentation does not depend on the aggregation procedure.

The improvement continues through P=3P=3, although the gains come with additional generation costs. With five responses per paraphrase, three views provide 15 responses per question, offering a practical balance between response diversity and computational cost.

(b) Necessity of PCA and the choice of representation

We added ablations removing PCA, applying PCA directly to hidden states, and replacing the subspace detector with a raw hidden-state probe.

Detector constructionTarget-only AUROCWith transfer AUROC
Contrast-based PCA (AdaST)76.52 (1.05)79.80 (1.07)
Raw hidden states76.78 (1.15)79.61 (1.09)
Contrasts without PCA76.35 (0.98)77.87 (0.94)
Hidden-state PCA77.51 (1.01)77.70 (1.02)

We apply PCA to within-question contrasts rather than raw hidden states to reduce question-specific variation and focus on differences between correct and incorrect responses. The ablations show higher mean AUROC than applying PCA directly to hidden states or removing PCA, while raw hidden-state detectors achieve comparable performance after transfer.

(c) Mixed-label subset and training instances

The mixed-label subset contains questions for which both positive and negative responses are observed. This subset is used to construct the contrast vectors, since both classes are needed to compute within-question differences. The logistic detector is fitted on the available labeled training responses, not only those used to construct the contrasts.

(d) Layer and token selection

We select a single layer using source validation and use the final-token representation to keep feature extraction simple, while focusing on behavioral subspace construction and adaptive transfer. To examine the sensitivity to these design choices, we conducted an additional ablation study on the held-out Qwen3-8B-Base model, evaluating five representation layers and two token aggregation methods (final token and mean pooling).

LayerFinal token: Target-onlyFinal token: AdaSTMean pooling: Target-onlyMean pooling: AdaST
25% (L9)80.01 (2.07)87.11 (1.07)85.98 (1.24)90.01 (1.42)
50% (L18)83.90 (1.75)88.17 (1.54)86.61 (1.31)90.79 (1.30)
Selected (L24)84.28 (1.14)88.38 (1.60)86.78 (1.55)90.57 (1.60)
75% (L27)83.46 (1.14)88.02 (1.46)86.30 (1.57)90.17 (1.50)
Last (L36)77.21 (1.69)84.70 (1.62)82.94 (2.14)87.14 (1.98)

These results suggest that the transfer gains are robust to layer and token aggregation choices, although alternative representations can further improve detection performance.

AuthorsOpenReview ↗

Weakness 1, 2, 3

(e) Necessity of the behavioral subspace component

We agree that the original results appeared to suggest that applying AdaST to SinkProbe outperformed our default method. To clarify this comparison, we conducted a matched evaluation across 30 model–target settings using identical experimental protocols. Default AdaST achieves 78.94 AUROC, compared with 77.11 for SinkProbe with transfer. The paired difference is not statistically significant (1.83 ± 1.45). Thus, SinkProbe does not consistently outperform the default method.

ModelDefault target-onlyDefault AdaSTSinkProbe target-onlySinkProbe + transferCombined
OPT-6.7B69.4 (6.4)75.2 (2.4)71.8 (4.6)73.2 (3.9)76.1 (3.0)
Llama-3.1-8B76.9 (3.6)81.6 (3.3)79.4 (3.5)81.6 (3.4)83.5 (3.2)
Gemma-7B79.4 (2.5)81.8 (2.5)70.8 (4.8)73.0 (5.2)82.6 (2.4)
Mistral-7B71.5 (1.6)78.6 (1.5)81.7 (2.1)83.2 (3.1)84.2 (2.2)
DeepSeek-7B74.8 (2.8)77.4 (2.7)72.9 (3.4)74.5 (3.8)76.4 (3.0)
Average74.4078.9475.3177.1180.54

Our main contribution is a flexible adaptive transfer mechanism that is compatible with different base detection algorithms. The behavioral subspace provides one way to construct reliability detectors. AdaST adaptively transfers and aggregates information from source-domain detectors under limited target supervision, making it compatible with different base detection algorithms. We demonstrate this by applying AdaST to SinkProbe in addition to our behavioral-subspace detector. Furthermore, combining the two detector families outperforms both default AdaST and AdaST applied to SinkProbe.

Revised manuscript: Section 2.1 (contrast construction), Section 4.3 Table 3, Appendix B.2 Tables 12–14, Appendix B.3 Table 21, and Appendix C.1.

W2. Theoretical analysis

We agree that the original least-squares analysis did not directly reflect the implemented method. We have replaced it with a logistic-risk analysis of AdaST's simplex-constrained aggregation. The revised Theorem 1 establishes consistency of the learned weights and prediction risk. The analysis no longer requires Gaussian features and accounts for question-level dependence. Lemma 2 also covers estimation errors in score standardization.

As the reviewer suggests placing greater emphasis on aggregation, we treat the PCA subspaces as fixed to focus the theoretical analysis on the transfer mechanism. We have also added intuition for the theoretical results. Theorem 1 shows that the learned weights approach those of the best population combination. Theorem 2 explains when source detectors are useful, if a source provides a direction that reduces logistic risk at the target-only solution, incorporating source information can improve prediction.

Revised manuscript: Section 3 and Appendix D.

W3. Clarification of Cross-Fitting

Thank you for pointing out the ambiguity. The source detectors remain fixed throughout cross-fitting. For each held-out fold kk, only the target subspace and detector are refitted using the other K−1K-1 target folds. Std−k\mathrm{Std}_{-k} denotes score standardization using the mean and standard deviation estimated from these K−1K-1 folds. It is applied to the held-out scores of both source and target detectors. The resulting out-of-fold scores are then used to learn the aggregation weights.

Revised manuscript: Section 2.2, Adaptive Cross-Domain Transfer.

AuthorsOpenReview ↗

Weakness 4

W4. Experimental Evaluation

(a) More recent LLMs

We agree that the original model selection limits the evidence for generalization to newer LLMs. We limit our evaluations to moderate-sized LLMs and evaluated AdaST on Qwen3-8B-Base.

Target datasetTarget-only AUROCAdaST AUROC
NQ82.5(0.6)86.4(0.6)
TriviaQA85.3(0.8)91.0(0.1)
CoQA72.2(1.3)77.1(0.7)
HaluEval-QA84.9(0.7)87.4(0.5)
SimpleQA73.3(1.6)83.2(1.7)
HotpotQA86.5(0.4)88.7(0.3)
Average80.8 (2.6)85.6 (2.0)

The transfer gain is positive on all six targets. While this provides additional evidence on a newer backbone, we acknowledge that broader evaluation across recent model families and scales remains necessary.

(b) Generalization across tasks

We investigate both partially and substantially mismatched source domains. In the partially mismatched setting, we add seven MATH reasoning-step correctness domains to the five QA source domains and evaluate on QA targets. AdaST maintains nearly unchanged AUROC (79.80 to 79.72), assigning only 2.8% of the total aggregation weight to the additional MATH sources. This suggests that AdaST can effectively downweight mismatched sources when relevant sources remain available.

In practical applications, source domains are typically selected based on their relevance to the target task. To examine the limits of adaptive weighting, we additionally consider a deliberately challenging setting in which all six source detectors are trained on QA datasets, while the targets are drawn from counterfactual faithfulness and RAGTruth.

Target task/domainTarget-onlyAdaSTSource weight
Counterfactual / mercury_other86.3184.890.300
Counterfactual / mercury_sc91.5488.600.463
Counterfactual / other_exams86.8381.900.405
RAGTruth / data2txt82.0282.020.000
RAGTruth / qa62.5062.590.071
RAGTruth / summary_cnn_dm57.8456.650.133
RAGTruth / summary_recent_news62.0657.450.272
Average75.5973.440.235

AUROC (%) at nT=25n_T=25. Source weight denotes the total weight assigned to the six QA source detectors.

In this setting, AdaST underperforms target-only by 2.14 ± 0.78 AUROC points on average, with positive transfer on only one of seven targets. These results suggest that AdaST can suppress mismatched sources when relevant sources remain available, but cannot reliably prevent negative transfer when the source pool is substantially mismatched.

(c) Reliability of BLEURT labels

BLEURT is an established learned evaluation metric that has shown strong agreement with human judgments in text generation tasks (Sellam et al., 2020). Nevertheless, we acknowledge that reference-based metrics may not fully capture factual correctness. We therefore evaluate AdaST under two additional labeling criteria: token-F1, a conventional QA metric, and Qwen3-14B as an LLM-as-a-judge.

For each criterion, we relabel the responses and retrain the entire pipeline. AdaST retains positive transfer gains under token-F1 (+5.41 ± 0.63 AUROC points) and LLM-judge labels (+3.98 ± 0.68), although the magnitude of the gains and absolute detection performance vary across criteria. These findings suggest that the transfer improvement is not specific to BLEURT.

Reviewer_hjU5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose a transfer learning method named adaptive signature transfer, which aims to improve LLM reliabiltiy detection tasks when labeled data in tartge doamin are rare. It builds domain-specific detectors by extracting low-dimensional behaviorial subspaces from hidden state contrasts between positive and negative responses to semantically similar paraphrases. Then it maps the detectors into the shared residual space and adpatively combines them using weights learned from cross-fitted target-domain predcitions. The paper evaluates the framework on hallucination detection and reasoning-step correctness, showing consistent improvements over target-only detectors.

Critical strengths
  1. This paper defines transfer learning problem well in this LLM reliability detection task context, and has a clear methodological motivation.
  2. The proposed methodological framework use appropriate transfer learning appraoch to combine representation learning, low-dimensional PCA-based detectors, and cross-fitted weight estimation.
Critical weaknesses

The novelty relative to established multi-source transfer learning methods is marginal. The proposed method combines source-specific and target-specific predictors using weights learned from cross-fitted target predictions. However, its adaptive aggregation resembles established stacking and multi-source transfer methods, leaving its methodological novelty unclear. It is more like adopting the appraoch in a new question setting rather than innovating an interesting new idea.

Questions for authors
  1. How does the propsoed method compare with uniform averaging, target-validation-based stacking, similarity-weighted transfer, and other standard ensemble methods under the same data and computational budgets? Can the authors isolate the contribution of paraphrase-aware subspace construction and cross-fitting through controlled ablations?

  2. What happens when some or all source domains are substantially mismatched with the target? Since the aggregation weights are constrained to be nonnegative and sum to one, can the proposed method reliably suppress unhelpful source detectors, and how often does it outperform the target-only detector on individual target domains rather than only on average?

  3. Can the authors clarify which properties of the implemented logistic-regression-based method are supported by the least-squares analysis, particularly given the data-dependent PCA subspaces and standardized detector scores? Are there conditions under which the theoretical guarantee can be extended to the practical algorithm?

  4. How sensitive are the results to the BLEURT threshold used for hallucination labels and the process reward model thresholds used for reasoning-step correctness? Would the observed transfer gains persist under alternative labeling criteria or a manually validated subset?

3 comments
AuthorsOpenReview ↗

W1 Methodological novelty + Q1 Comparisons and ablations

We thank the reviewer for the constructive feedback. We address each concern below and have incorporated the corresponding changes into the revised manuscript. The main results in the manuscript are based on the full datasets, while the additional experiments reported in this rebuttal use a sampled subset of 1,000 questions per dataset. This accounts for differences in absolute AUROC values between the original results and the additional experiments.

W1. Methodological novelty

We acknowledge that stacking and multi-source transfer is well established for traditional domain adaptation tasks. Our contribution lies in addressing a different problem in LLM reliability detection, where correctness-related signals learned from one domain may not generalize to another. Existing work has examined this cross-domain degradation (Orgad et al., 2025; Alansari et al., 2026) or sought to improve the transferability of individual detectors (Zhang et al., 2025b). SALA (Li et al., 2026) further learns a generalizable hallucination detector from multiple source domains without target supervision. AdaST instead studies adaptive multi-source transfer when a small number of labeled target questions are available.

This setting introduces a difficulty that standard stacking alone does not resolve. The target detector is estimated from very few labeled questions, so learning its weight from in-sample predictions would favor it over source detectors evaluated on unseen target data. We address this by re-estimating the target representation, classifier, and score-standardization statistics within question-level cross-fitting, while keeping the source detectors fixed.

Our revised logistic-risk analysis gives an O(nT−1/2)O(n_T^{-1/2}) convergence rate under the stated conditions and identifies a necessary and sufficient condition for positive transfer. The experiments also show why source selection matters. Adding heterogeneous domains to the QA source pool barely changes AdaST's AUROC (79.80 to 79.72), whereas uniform averaging drops from 79.97 to 68.59. We will make clearer that the contribution is the estimation and adaptive transfer of LLM reliability signals under limited target supervision.

Q1. Comparisons with standard ensembles and component ablations

Aggregation baselines. We added comparisons with pooled source-target training, uniform averaging, similarity-weighted aggregation, conventional stacking without simplex constraints, and the best single source detector under homogeneous and heterogeneous settings. The homogeneous setting uses five remaining QA hallucination-detection domains as source domains. The heterogeneous setting additionally includes seven MATH reasoning-step correctness domains, introducing task-mismatched sources. In both settings, the target is a held-out QA hallucination-detection domain. The table reports AUROC over target-only at nT=25n_T=25, in percentage points.

MethodHomogeneousHeterogeneous
AdaST79.8079.72
Pooled training80.2778.03
Similarity-weighted80.1577.30
Uniform averaging79.9768.59
Conventional stacking79.1178.50
Best single source78.8678.86

Under homogeneous sources, AdaST achieves performance comparable to pooled training, similarity weighting, and uniform averaging. Under heterogeneous sources, AdaST remains robust, while conventional aggregation methods exhibit substantially larger declines. This highlights AdaST's adaptability to varying levels of source-target heterogeneity.

Component ablations. We conducted a factorial ablation to isolate the contributions of paraphrase augmentation and cross-fitting. We evaluate each configuration across 30 model–target settings, formed by six QA datasets and five models. Each dataset serves as the target in turn. The reported standard errors are clustered across the 30 model–target settings.

ParaphrasesWith cross-fittingWithout cross-fitting
P=1 (no paraphrases)77.3 (1.6)75.4 (1.6)
P=278.1 (1.4)76.6 (1.5)
P=379.8 (1.1)78.6 (1.1)

Both paraphrase augmentation and cross-fitting improve detection performance. The benefit of cross-fitting is more pronounced when fewer question versions are available.

Revised manuscript: Section 4.2, Table 2, and Appendix B.2, Tables 13–14.

AuthorsOpenReview ↗

Q2 Robustness to mismatched sources + Q3 Connection between theory and implementation

Q2. Robustness to mismatched source domains and negative transfer

Partially mismatched sources. As shown in Q1, AdaST maintains nearly unchanged AUROC when seven reasoning-step correctness domains are added to the source pool (79.80 to 79.72). To examine how AdaST handles these mismatched sources, we report the learned aggregation weights for each target dataset.

TargetNQTriviaQACoQAHaluEvalSimpleQAHotPotQAMATH
NQ[2]521420210.1
TriviaQA29[3]9177340.8
CoQA56[27]269179.9
HaluEval134123[9]1130.5
SimpleQA191234[8]495.1
HotPotQA134102124[0]0.4

On average, AdaST assigns only 2.8% of the total weight to the seven additional MATH sources, compared with 88.9% for the QA sources and 8.3% for the target detector.

Individual target-domain performance. We further examine whether the gains extend beyond the overall average. The table reports AUROC (%) for each target dataset and the frequency of negative transfer.

TargetTarget-onlyAdaSTNegative Transfer
NQ73.184.31/15
TriviaQA79.583.80/15
CoQA70.673.83/15
HaluEval80.884.63/15
SimpleQA70.080.53/15
HotPotQA71.579.23/15

AdaST improves average AUROC on all six target datasets. In the supplementary paired evaluation, it outperforms target-only in 77 of 90 cases (85.6%), although negative transfer remains possible in individual runs.

Substantially mismatched sources. We further evaluate a more challenging setting in which the entire source pool consists of six QA domains, while the targets are drawn from counterfactual faithfulness and RAGTruth datasets, including counterfactual reasoning, data-to-text generation, and summarization. No source detector is trained on these target domains.

Target domainTarget-onlyAdaSTSource weight
Counterfactual / mercury_other86.3184.890.300
Counterfactual / mercury_sc91.5488.600.463
Counterfactual / other_exams86.8381.900.405
RAGTruth / data2txt82.0282.020.000
RAGTruth / qa62.5062.590.071
RAGTruth / summary_cnn_dm57.8456.650.133
RAGTruth / summary_recent_news62.0657.450.272
Average75.5973.440.235

These results show that AdaST can effectively suppress mismatched sources when relevant sources remain available, but cannot reliably prevent negative transfer when the entire source pool is substantially mismatched.

Revised manuscript: Section 4.2, Table 2, and Appendix B.1, Tables 9–11.

Q3. Connection between theory and implementation

We agree that the original least-squares analysis did not directly provide guarantees for the implemented logistic-regression method. We have therefore replaced it with an analysis of the logistic aggregation used by AdaST. The empirical procedure constructs standardized scores from source and target detectors and learns their aggregation weights using cross-fitted target predictions. Our analysis studies the corresponding population aggregation problem.

Theorem 1 establishes consistency of the learned aggregation weights and the resulting target prediction risk. Its convergence rate accounts for source sample sizes, target sample size, and cross-fitting. When source sample sizes grow proportionally with the target sample size, with a fixed number of sources and folds, the rate is O(nT−1/2)O(n_T^{-1/2}). Theorem 2 establishes that the optimal aggregate strictly improves over the target-only predictor if and only if at least one source has a negative directional derivative of the profiled target logistic risk at the target-only solution.

Score standardization is already covered by the analysis. Lemma 2 bounds the estimation error of standardized detector scores, including errors from the fitted coefficients and target-domain mean and variance estimates. We focus on fixed subspaces because the main theoretical question concerns how detector scores are combined. This also separates the aggregation analysis from the construction of a particular base detector.

Revised manuscript: Section 3, with proofs and assumptions in Appendix D.

AuthorsOpenReview ↗

Q4. Sensitivity to automatic labeling criteria

Q4. Sensitivity to automatic labeling criteria

We conducted additional experiments to assess the sensitivity of AdaST to the labeling criteria used for both tasks.

BLEURT and PRM thresholds. We varied the BLEURT threshold from 0.40 to 0.60. AdaST consistently outperforms target-only across all five thresholds, with AUROC gains ranging from 3.12 to 3.85 points. For reasoning-step correctness, we also tested four PRM threshold pairs, from (0.90, 0.10) to (0.60, 0.40). The transfer gains remain positive across these settings, at approximately 6–9 AUROC points.

Alternative labeling criteria. Beyond varying BLEURT thresholds, we evaluated AdaST using token-level F1 instead of BLEURT to assign correctness labels. Token-F1 provides a deterministic, reference-based criterion that does not depend on a learned evaluation model. We repeated the evaluation across all 30 model–target combinations.

TargetBLEURT gainToken-F1 gain
NQ+11.2+9.6
TriviaQA+4.2+6.1
CoQA+3.3+1.8
HaluEval+3.9+2.6
SimpleQA+10.5+6.5
HotPotQA+7.6+6.1
Overall+6.8+5.4

AUROC (%). Parentheses report SE. Gains are computed from paired model–target comparisons.

We also evaluated an independent Qwen3-14B judge on a stratified subset of 6,000 responses. AdaST retains a positive mean transfer gain of 3.98 AUROC points, although the gain is smaller. These experiments show that the observed transfer gains are robust to the tested labeling thresholds and persist under two alternative automatic labeling criteria.

Revised manuscript: Appendix B.3, Tables 15–20.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.