- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes conformal maps for anticipating whether a fixed target LLM will succeed or fail on a new instruction, using instruction text and historical instruction–outcome pairs. A pretrained instruction encoder is fine-tuned with a neighborhood-support objective and a distance-preservation regularizer. Class-specific kernel support supplies conformity scores for label-conditional split conformal prediction, with model selection separated from final calibration. Across five QA and reasoning benchmarks and three target models, the paper reports improved singleton rates over the pretrained kernel baseline in 14 of 15 settings. Additional experiments examine supervised alternatives, representation transfer, behavioral organization, retained pretrained geometry, and inspectable local evidence.
Critical strengths
-
The paper provides useful empirical evidence for instruction-level behavioral assessment. The experiments cover five benchmarks and three target models, with improvements over the pretrained kernel baseline in most settings. Comparisons with frozen-embedding probes and end-to-end classifiers, together with the representation–score decomposition and candidate-budget analysis, investigate several plausible explanations for the gains.
-
The statistical protocol and geometric characterization are thoughtfully organized. Training, validation, calibration, and testing are separated, preserving the applicability of label-conditional split conformal coverage under exchangeability. The regularization study characterizes the trade-off between behavioral separation and preservation of pretrained local ordering. Transfer, pooled-training, and stability analyses provide additional information about the learned representations.
Critical weaknesses
-
The methodological advance beyond established assessor models, metric learning, and conformal training needs sharper delineation. Predicting LLM correctness from prompt representations is established in prior assessor research, including PredictaBoard by Pacchiardi et al., 2025. The neighborhood log-probability objective closely follows Equation 6 of NCA by Goldberger et al., 2004, while ConfTr by Stutz et al., 2022 already learns underlying models to improve conformal-set efficiency. The manuscript acknowledges these antecedents and includes useful supervised baselines and a regularization sweep. Nevertheless, its contribution currently appears primarily integrative and empirical, and the comparisons do not fully isolate the benefit of neighborhood training from geometry preservation. A classifier equipped with the same distance-preservation penalty and a matched selection budget would help establish this distinction. The geometry-based success prediction of Kashani et al., 2025 also merits discussion when positioning the contribution. These precedents limit the novelty of individual ingredients without establishing that the complete proposed system is redundant.
-
The connection between the stated method, its theoretical interpretation, and the released implementation remains unresolved. Equation 2 defines , where . However, the released scoring code computes . This introduces a query-dependent scaling that can change class-conditional rankings when support is small, so the logistic-margin identities and Appendix A.7 do not directly describe that implementation. Additionally, the released training loss includes each anchor in its own correct-label support, potentially making the loss small at large bandwidths without useful support from other instructions. The practical effects of these choices have not been quantified, and the authors should confirm whether the released code matches the reported experiments. Independent calibration still supports coverage for a fixed implemented score; the concern is the interpretation and robustness of the reported efficiency gains.
Additional presentation and figure documentation feedback. Figure formatting needs more consistent editorial treatment throughout the manuscript. In Figure 4, the bar-chart labels are substantially smaller than the adjacent tabular text, while Figures 6 and 14 contain dense case descriptions that are difficult to read at normal page scale. Please standardize typography, readable font sizes, color conventions, line and marker styles, legends, panel labels, spacing, and caption formatting across the main paper and appendix. Figure 14 also claims to show the same four anchors as Figure 6, but their R4 examples differ. The examples or captions should be corrected. The AI Use Statement already acknowledges generative AI assistance in creating or modifying figures and tables. Please identify the affected items, the tools used, and whether AI assisted with layout, plotting code, or direct image generation. The authors should also explain how they checked the scientific content and numerical fidelity, and make empirical plots reproducible from the underlying experimental data and plotting scripts.
Questions for authors
-
What methodological contribution should be attributed specifically to the proposed formulation beyond NCA-style representation learning followed by label-conditional calibration? Could a focused comparison give the end-to-end classifier the same geometry-preservation penalty and a matched selection budget, while reporting both singleton rate and retained geometry?
-
Does the released scoring implementation match the reported experiments? Please quantify how often is comparable to or below , and whether numerically stable exact normalization changes score rankings, prediction sets, or the main singleton-rate comparisons.
-
Is including each training anchor in its own support intentional? Please quantify the self-support contribution across sampled bandwidths and provide a diagonal-masked ablation on representative settings under the same evaluation protocol.
-
Appendix B.5 distinguishes the ESS-matched diagnostic bandwidth from the actual prediction bandwidth. Do the evidence regimes and supporting instructions remain similar under the prediction bandwidth, and what fraction of the conformity score is accounted for by the displayed neighbors?
-
Which figures and tables were created or modified with generative AI, and what role did the tools play? Please clarify how numerical plots and illustrative diagrams were produced and verified, including the correspondence between plotted values and experimental outputs.
4 comments
Our response – part 2
W1. & Q1. Novelty and new comparisons
We agree with the reviewer's own conclusion: the precedents "limit the novelty of individual ingredients without establishing that the complete proposed system is redundant". We do not claim novelty for the ingredients (see also our response to W1 of Reviewer 1). The contribution is the formulation and study of a model-specific geometry of instruction space as an evaluation object: historical successes and failures become local structure that simultaneously determines conformal predictions and exposes the evidence supporting them.
- Assessors (Pacchiardi et al., 2025). An assessor outputs a scalar estimate of correctness. Our system outputs a prediction set with finite-sample label-conditional coverage, computed from a geometry that also exposes how much historical evidence surrounds the instruction, which outcome it favors and which evaluated instructions contribute.
- NCA (Goldberger et al., 2004). The training objective is the same, but what is new is its role: the local support score defines the conformity score, and Appendix A.7 shows when improving it reduces ambiguity after the conformal thresholds are recalibrated, which does not follow from better discrimination alone.
- ConfTr (Stutz et al., 2022) trains a classifier through a differentiable calibration step and leaves no geometry to inspect. We now compare with it directly (new Appendix C.16): across the 15 benchmark–model pairs, our map has a higher singleton rate by +0.067 (95% CI [+0.057, +0.077]), and it is significantly higher in 11 of the 15 pairs and lower in 2.
- Kashani et al. (2025). Thank you for the pointer. They keep the pretrained prompt embedding fixed and represent each LLM, in closed form, as one vector; success is predicted by a linear projection onto it, without coverage guarantees. We instead adapt the geometry to one target model, read evidence from neighborhoods and calibrate it into prediction sets. Our closest baseline in form is raw + logistic, a linear readout on the frozen embedding (0.365 against 0.451 for the map, Fig. 4). We will add this discussion to the related work.
Isolating neighborhood training from geometry preservation
We ran the control the reviewer proposes. Our method has three components: the neighborhood loss, the distance-preservation penalty and the kernel conformity score. The control gives the classifier the penalty, and the kernel score, matching the selection budget. These additions move the classifier toward our method one component at a time.
The classifier is trained with cross-entropy plus the same penalty, applied to its embedding, over the same grid and checkpoints as the map, with the same cross-validated selection. For the matched comparison, both encoders are scored with the kernel score and selected over the same candidates, so the two rows differ only in the loss. Cells give the singleton rate at for Qwen3, averaged over five benchmarks.
| no penalty (99 candidates) | with penalty, ( selected, 396 candidates) | |
|---|---|---|
| Classifier encoder (cross-entropy) | 0.383 | 0.437 |
| Conformal map (neighborhood loss, ours) | 0.439 | 0.451 |
| Map minus classifier, singleton rate | +0.056 [+0.041, +0.071] | +0.014 [+0.0002, +0.028] |
With its own probability score, the end-to-end classifier goes from 0.374 to 0.419 with the penalty;the map is +0.032 [+0.017, +0.046] above it.
- Our components transfer to the classifier. The penalty and the kernel score raise the classifier from 0.374 to 0.437 and its retained geometry from 0.13 to 0.43. We agree with the reviewer that a large part of the gap in Fig. 4 is due to these two components. A classifier that adopts them is no longer an independent baseline: it is our pipeline with a different loss.
- The neighborhood loss improves on cross-entropy in both settings, by +0.056 without the penalty and by +0.014 with it. The loss and the penalty act in the same direction, so whichever is added second adds less: this is also why the penalty gives the map a smaller gain than it gives the classifier. With the penalty in place the advantage of the loss is small, and we do not claim otherwise.
- The bandwidth is part of the method, not extra budget. Against the classifier's probability score the map also selects the bandwidth of its kernel score. This is one scalar of the conformity score, selected on validation data without retraining, and a probability score has no counterpart to tune. The matched comparison removes the difference altogether.
What we attribute to the proposed formulation is therefore the combination: a conformity score read from neighbourhoods, a loss that trains the quantity this score uses, and a penalty that keeps the pretrained geometry. No single ingredient accounts for the gain on its own.
Our response – part 3
W2. & Q2. & Q3. Method, theory and released implementation
The released code is the code that produced the reported experiments. We thank the reviewer for the careful reading of the code. Removing the constant from the scoring code leaves the results of the learned map unchanged, and masking the anchor's own term in training does not change them significantly; we quantify both below.
The constant in the scoring code (Q2)
All reported results use , where the constant was meant as a division-by-zero safeguard. Removing the constant does not change the results of the learned map (singleton rate 0.408 with and without it, over the 15 benchmark–model pairs and three seeds of Table 1). We established this by re-running selection, calibration and evaluation with the exact normalization of Equation 2, computed in log space.
How often is comparable to or below ? Never for (smallest observed: 0.22). At the two largest bandwidths of the grid it is more frequent: over all candidate configurations, for 21% of queries at and 31% at ( for 28% and 33%). Among the selected kernel configurations, 39 of 135 use .
Does exact normalization change rankings and prediction sets? For the 96 selected configurations with , nothing changes: scores differ by at most , rankings are identical and no prediction set changes. For the 39 with , prediction sets change for less than 8% of test queries on average. Over all selected configurations, the singleton rate of the learned map is unchanged (Table 1 results stay at 0.408). The map remains the method with the highest singleton rate in 14 of 15 pairs and the overall gain remains significant. Coverage is unchanged (0.90 on average for both classes).
The anchor in its own support (Q3)
The inclusion of the self-term was not a deliberate choice. The loss sums over the whole minibatch without masking , so each anchor adds to the support of its own class. Inclusion of the anchor affects the training loss only. At selection, calibration and inference, the queries are held-out questions and are not in the reference bank, so no self-term enters any reported score.
Self-support contribution across bandwidths. For minibatches of the training size (70) drawn from the MMLU training set (Qwen3), we measured the share of an anchor's own-class support that is the anchor itself, :
| 0.1 | 0.3 | 1 | 3 | 10 | 30 | 100 | sampled | |
|---|---|---|---|---|---|---|---|---|
| Pretrained encoder (initialization) | 0.03 | 0.04 | 0.06 | 0.26 | 0.99 | 1.00 | 1.00 | 0.49 |
| Trained map, | 0.03 | 0.04 | 0.05 | 0.17 | 0.93 | 1.00 | 1.00 | 0.45 |
| Trained map, | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 | 0.04 | 0.06 | 0.03 |
At small bandwidths the anchor is one of about 35 same-outcome items and contributes about 3%. At under the pretrained geometry, the anchor is essentially its whole own-class support, so its loss term is close to zero and those bandwidths contribute little gradient.
Diagonal-masked ablation. We retrained the map with the diagonal masked, changing nothing else, and evaluated it under the same protocol (same splits, 99 candidates, cross-validated selection, calibration and test; MMLU, Qwen3):
| With self-term (paper) | Diagonal masked | Difference (95% paired bootstrap interval) | |
|---|---|---|---|
| 0.456 | 0.488 | +0.032 [−0.003, +0.068] | |
| 0.410 | 0.453 | +0.043 [0.000, +0.085] |
Neither difference is significant, and if anything, it slightly improves our own method. Both variants are well above the pretrained encoder (0.337).
Our response – part 4
W3. and Q5. Presentation and figure documentation
Figure 14. We thank the reviewer for the careful reading. The two figures are rendered separately, and their R4 examples went out of sync while the paper was being revised. We have regenerated Figure 14 so that it shows the same four anchors as Figure 6.
Consistency. All figures are produced by plotting scripts from the stored experimental results, and they share one visual system: a single typeface (DejaVu Sans) in every figure of the main paper and appendix, and one palette (two hues, blue and orange, with neutral grays) used throughout. We agree that the text in some figures is too small at page scale, in particular the bar-chart labels of Figure 4 and the case descriptions of Figures 6 and 14. We will enlarged these to a common minimum size in the camera-ready version providing us with more space for figures.
AI assistance and reproducibility (Q5). AI coding assistance was used for the plotting and table-generation code behind all figures and tables, and in every case its role was limited to writing and editing that code. No figure or table contains generated imagery, and no reported value was produced or edited by an AI tool.
- Numerical plots and tables. Each is produced by a script in the released repository that reads the stored experimental outputs and writes the figure, or the LaTeX table, directly. Plotted and tabulated values are therefore the experimental outputs themselves, with no manual transcription, and the numbers quoted in the text are read from the same files.
- Illustrative diagrams. Figure 1 is a schematic, also drawn by a script; it illustrates the construction and reports no measurements. The example cards of Figures 6 and 14 show actual test instructions and their nearest training instructions, retrieved from the stored embeddings.
- Verification. We reviewed every figure and table against the result files it is generated from, and the evaluation scripts check the data splits of every run against a reference before scoring it.
We have expanded the AI Use Statement accordingly.
Q4. Evidence under the prediction bandwidth
We computed both quantities on all 15 benchmark–model pairs and three seeds (45 cases), comparing the diagnostic bandwidth with the prediction bandwidth selected as in the main results.
Do the supporting instructions remain similar? Yes, they are identical. Neighbors are ordered by distance in the learned map, and this order does not depend on the bandwidth, so the same instructions are displayed under both. At the diagnostic bandwidth, the 3 displayed neighbors account for 19% of the score. The prediction bandwidth, is selected for conformal efficiency and typically aggregates evidence over a much broader portion of the reference bank. At this bandwidth, the three displayed neighbors account for only about 1% of the support. A prediction that pools evidence from many historical instructions cannot be summarized faithfully by only three examples. This broader aggregation also makes the prediction less dependent on any single retrieved example. The displayed neighbors should be interpreted as the nearest concrete exemplars of the learned geometry, rather than as a sparse decomposition of the prediction score. We will adjust our provenance claims accordingly.
Do the evidence regimes remain similar? The regimes are descriptive labels for regions of the evidence plane at the diagnostic bandwidth; they are not used by the prediction, and no reported result depends on them. The quantities they summarize do carry over to the prediction bandwidth: the direction of the evidence (which side of the base rate it favors) agrees between the two bandwidths for 93% of queries, and query rankings by margin and by total support have Spearman correlations of 0.85 and 0.75 (medians over the 45 cases).
Our response – part 1
We appreciate the reviewer’s careful and technically detailed assessment, as well as the positive comments on the empirical evaluation and statistical protocol. The review identified several concrete points that led us to add new controls and clarify both the contribution and the implementation:
In the subsequent parts, we address each point in detail.