- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
AdaST builds domain-specific reliability detectors from principal components of within-question response contrasts, then combines their scores using question-level cross-fitting and simplex-constrained logistic regression. The paper reports benefits in low-target-data hallucination detection and reasoning-step correctness prediction, analyzes a fixed-subspace least-squares analogue, and explores activation steering.
Critical strengths
- AdaST addresses a practical problem with a carefully designed framework. Adapting reliability detectors with limited target labels is valuable for open-weight LLM deployment. Question-level cross-fitting refits both the target representation and detector, addressing the target expert’s in-sample advantage. Source-only layer selection further strengthens the methodological design.
- The experiments demonstrate benefits across models, domains, and tasks. At 25 target questions or prefixes, AdaST improves average AUROC over the target-only detector by 7.8 percentage points for hallucination detection and 7.7 percentage points for reasoning-step correctness. Improvements with SinkProbe experts further suggest that the transfer mechanism is useful beyond the default representation.
- The analysis provides complementary theoretical and empirical insights. The clearly scoped regression analogue characterizes transfer through domain discrepancies and representation bias, supported by detailed derivations. Learned weights illustrate changing source contributions, while the exploratory steering experiment shows improved semantic accuracy with little change in abstention.
Critical weaknesses
- Additional matched-supervision baselines would clarify the contribution of adaptive weighting. Tables 1–4 demonstrate gains over target-only detectors and existing methods. Comparisons with pooled source-and-target training, uniform expert averaging, and conventional stacking without simplex constraints would further establish which benefits come from the particular combination strategy.
- The supervision budget and evaluation protocol need clearer reporting. The paper defines the QA budget in terms of original questions, with the 25-question setting corresponding to 375 automatically labeled responses. However, the data used to select and whether any target labels used for this selection count toward the reported budget should be explicitly stated. Reporting per-domain training, validation, and test sizes, the number of repetitions, and the units used to compute standard errors for the main detection experiments would also improve reproducibility. For reasoning-step evaluation, the paper should clarify whether prefixes from the same original problem are kept together across data splits and cross-fitting folds. Reporting retention rates under the PRM thresholds and agreement with independent correctness labels would further help establish the scope of the results.
- Automatic labeling and candidate selection limit the scope of the evaluation. Hallucination labels rely on a fixed BLEURT-20 threshold of 0.5, while reasoning examples are selected using extreme scores from Qwen2.5-Math-PRM-7B. The labeling details should be stated more clearly, including the final sample counts and label distributions for QA, as well as per-domain retention rates and final sample counts for reasoning prefixes. In addition, was a sample of these labeled examples manually inspected to assess label quality?
Questions for authors
- How much improvement remains over pooled training, uniform averaging, and conventional stacking with exactly the same supervision and tuning budget?
- What data are used to select , how are prefixes from the same original problem grouped across splits, and what are the PRM retention rates and levels of agreement with independent correctness labels? Please also identify the units underlying the reported standard errors.
2 comments
Weakness 3
W3. Automatic labeling and candidate selection
QA sample counts and label distributions. We report the question-level splits and response-label distributions below. Each question contributes 15 generated responses, and the positive rate varies across the five evaluated backbones.
| Dataset | Train questions | Test questions | Responses per backbone | Mean positive rate |
|---|---|---|---|---|
| NQ | 2,527 | 1,083 | 54,150 | 8.2% |
| TriviaQA | 2,527 | 1,083 | 54,150 | 30.5% |
| CoQA | 5,588 | 2,395 | 119,745 | 51.2% |
| HaluEval-QA | 7,000 | 3,000 | 150,000 | 34.0% |
| SimpleQA | 3,024 | 1,297 | 64,815 | 6.4% |
| HotpotQA | 5,183 | 2,222 | 111,075 | 10.7% |
PRM retention. For each reasoning prefix, we generate eight candidate next steps and retain the prefix only if at least one candidate receives a PRM score of at least 0.8 and another receives a score of at most 0.2. We select the highest-scoring positive and lowest-scoring negative candidate, producing two labeled steps per retained prefix. The retention statistics, averaged over five backbones, are as follows.
| MATH domain | Original prefixes | Retained prefixes | Retention rate |
|---|---|---|---|
| Algebra | 389 | 278 | 71.4% |
| Counting and Probability | 285 | 176 | 61.7% |
| Geometry | 325 | 193 | 59.4% |
| Intermediate Algebra | 840 | 502 | 59.8% |
| Number Theory | 311 | 189 | 60.7% |
| Prealgebra | 140 | 99 | 70.9% |
| Precalculus | 358 | 234 | 65.5% |
We assess agreement between PRM predictions and independent human correctness labels.
| MATH domain | Candidates | Retained | Retention rate | Agreement | AUROC |
|---|---|---|---|---|---|
| Algebra | 200 | 170 | 85.0% | 80.6% | 79.00 |
| Precalculus | 197 | 155 | 78.7% | 80.0% | 76.46 |
| Counting & Probability | 182 | 125 | 68.7% | 76.8% | 72.20 |
| Intermediate Algebra | 193 | 138 | 71.5% | 76.1% | 78.60 |
| Geometry | 184 | 112 | 60.9% | 75.9% | 70.93 |
| Prealgebra | 181 | 141 | 77.9% | 74.5% | 67.63 |
| Number Theory | 188 | 134 | 71.3% | 71.6% | 66.73 |
| Overall | 1,325 | 975 | 73.6% | 76.7% | 73.29 |
Retention denotes the proportion of individual candidates with PRM scores outside the 0.2–0.8 interval, rather than the proportion of prefixes containing both positive and negative candidates. The retention rates indicate that our filtering procedure preserves a substantial portion of eligible prefixes across all seven domains rather than concentrating on a small subset.
Label quality and sensitivity. We acknowledge that automatic correctness labels are imperfect. For QA, we additionally evaluated token-F1 and Qwen3-14B LLM-judge labels, retraining the detection and transfer pipeline under each criterion. AdaST retains positive transfer gains under both criteria. We also varied the BLEURT threshold from 0.40 to 0.60 and the positive and negative PRM thresholds, observing positive transfer gains across the tested settings.
Revised manuscript: Section 4.1, Appendix B.3 Tables 15–20
Weakness 1, 2 (Question1, 2)
We thank the reviewer for the constructive suggestions. We have conducted additional matched-budget comparisons and clarified the supervision, evaluation, and labeling protocols. The main results in the manuscript are based on the full datasets, while the additional experiments reported in this rebuttal use a sampled subset of 1,000 questions per dataset. This accounts for differences in absolute AUROC values between the original results and the additional experiments.
W1. Matched-supervision aggregation baselines
We evaluated pooled source–target training, uniform averaging, similarity-based weighting, and unconstrained stacking using the same target-label budget and evaluation splits within a paired comparison. We consider both a homogeneous QA source pool and a heterogeneous pool that additionally includes seven MATH reasoning domains.
Pooled training and simple averaging are competitive with AdaST when all sources are related QA domains. However, their performance deteriorates when mismatched MATH sources are introduced, whereas AdaST maintains nearly unchanged performance. These results clarify that the principal advantage of adaptive weighting is its robustness to variations in source relevance.
Revised manuscript: Section 4.2 Table 2, and Appendix B.2 Tables 13–14
W2. Supervision budget and evaluation protocol
Supervision budget and evaluation protocol. We selected kmax through 5-fold cross-validation on source-domain data, considering K∈{8,16,32,64,128}. The selected kmax=64 and model-specific representation layers were fixed across all reported experiments. No additional target labels were used for hyperparameter selection. For QA, we use a 70/30 train–test split at the original-question level, without a separate target validation set. At nT=25, we sample 25 training questions, each contributing 15 automatically labeled responses. All responses from the same question remain together during data splitting and three-fold cross-fitting.
Reasoning-step evaluation. For PRM800K, we directly reuse the model-specific representation layers selected for QA, without additional tuning on reasoning data. We group prefixes by their original MATH problem ID (
math_id) and use an 80/20 train–test split at the problem level. Cross-fitting also respects these groups, ensuring that prefixes from the same problem never appear in different folds or across training and test sets. Target budgets count retained prefixes sampled from the training groups.Repetitions and standard errors. The main experiments use three split seeds and three budget-sampling repetitions, yielding nine evaluations per model–target combination. Within-cell standard errors are computed across these nine evaluations. Aggregate standard errors use model–target combinations as the sampling units (30 for QA), while paired comparisons compute differences within each combination before estimating the standard error.
Revised manuscript: Appendix C.1