- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a post-hoc selective regression method that uses textual context to identify unreliable predictions from a fixed regressor. An LLM takes numerical features, associated text, and the regressor's prediction as inputs and estimates a prediction-error score. A calibrated threshold is then used to accept or reject each prediction. For rejected predictions, a separate prompt extracts supporting textual evidence and generates explanations by underlining relevant text.
Critical strengths
I feel the problem is practically relevant, as assessing the reliability of an existing predictor without modifying it is useful in real-world applications. The main novelty lies in using textual information to estimate prediction errors and provide natural-language explanations for rejected predictions in selective regression.
Critical weaknesses
-
I find the use of textual information for selective regression interesting, but I think the paper needs stronger evidence that it offers clear advantages over existing multimodal loss prediction methods. In particular, the experiments do not directly show whether text can help identify unreliable predictions even when it does not improve outcome prediction.
-
The explanations are generated after scoring, without a clear connection between the selected evidence and the scoring process. Tables 5 and 6 show negative comprehensiveness on several datasets, meaning that removing the selected evidence can actually increase the predicted unreliability score. This raises concerns about whether the explanations truly support the rejection decisions. The qualitative examples are readable and reasonable, but they do not show whether the explanations reflect the model's actual decisions or help human reviewers. I feel the explainability contribution is not sufficiently supported by the current results.
Questions for authors
-
What advantages does the proposed framework offer over a strong multimodal error predictor under the same prediction targets and tuning budgets? Can the authors directly test whether text helps identify unreliable predictions beyond its contribution to outcome prediction?
-
How does the selected evidence compare with random sentence selection or deletion-based attribution methods? Can the authors explain the negative comprehensiveness results and provide stronger evidence that the explanations reflect the rejection scores?
2 comments
Response to Reviewer DM34 (2/2)
W2/Q2-1. Why is extraction separate from scoring?
SFT supervises only the loss estimate, because no supervision is available for explanations. The extracted evidence is therefore a retrospective self-explanation and is not faithful by construction, which is why we evaluate faithfulness empirically.
W2/Q2-2. Do the explanations reflect the rejection scores?
At the sample level, these negative averages are not produced by the majority of rejected samples: on every dataset except Health and Social Good, fewer than half of the rejected samples have negative comprehensiveness (new Tables 15 and 16 in Appendix C.6). Our hypothesis for the negative cases is that removing the extracted sentences also removes context the rejector relies on. In time series, for example, a major event is often described repeatedly across consecutive timestamps. Extracting those sentences then removes a block of context around the event, and the rejector may estimate a larger loss, that is, lower reliability. We state this as a hypothesis, and we leave characterizing the datasets on which faithfulness fails to future work.
To test whether the extracted evidence reflects the sentences the score depends on, we identified, by leave-one-sentence-out occlusion, the sentences whose individual removal lowers the score (score-supporting sentences) and measured how many of them the extracted evidence recovers (capped recall@k), compared with the expected value for k sentences selected uniformly at random (new Table 17 in Appendix C.6).
| Capped recall@k | k = 3, extracted | k = 3, random | k = 7, extracted | k = 7, random | k = 10, extracted | k = 10, random |
|---|---|---|---|---|---|---|
| Airbnb | 0.154 | 0.197 | 0.174 | 0.250 | 0.345 | 0.397 |
| Beer | 0.739 | 0.641 | 0.788 | 0.927 | 0.808 | 0.993 |
| Houses | 0.430 | 0.283 | 0.479 | 0.401 | 0.638 | 0.726 |
| Laptops | 0.052 | 0.141 | 0.064 | 0.208 | 0.069 | 0.378 |
| Mercari | 0.741 | 0.630 | 0.880 | 0.905 | 0.935 | 0.988 |
| Permits | 0.311 | 0.497 | 0.498 | 0.806 | 0.599 | 0.999 |
| Wine | 0.249 | 0.354 | 0.300 | 0.526 | 0.719 | 0.912 |
On Beer, Houses, and Mercari, the extracted evidence recovers score-supporting sentences better than random selection at k = 3 (and at k = 7 on Houses), whereas on the other four datasets it does not. We therefore conclude that the evidence is faithful to the scoring on some datasets and not on others. This is consistent with prior findings that LLM self-explanations can be unfaithful and that faithfulness varies across datasets (Turpin et al., 2023; Madsen et al., 2024).
W2/Q2-3. Do the explanations help human reviewers?
We have not conducted a human study. The case studies show that the evidence and rationales are plausible, not that they help reviewers. Section 5.2.2 states this and lists a human evaluation as future work.
Response to Reviewer DM34 (1/2)
We thank the reviewer for the careful reading and for the concrete experimental requests. We address the questions and concerns below.
W1/Q1-1. Does text help beyond outcome prediction?
Yes, text that does not improve the regressor can therefore still indicate when its prediction is unreliable. For each tabular dataset, we compared the relative MSE reduction of the regressor (TabSTAR with vs. without text) with the relative NAURC reduction of the rejector (Qwen3.5-9B (SFT+GRPO) with vs. without text), both averaged over the four splits. We show the results the following table. On Airbnb, text increases the regressor's MSE by about 5% yet reduces the rejector's NAURC by nearly 40%. On Houses and Wine, text barely changes the regressor's MSE but reduces the rejector's NAURC by about 60%. This supports applying a context-aware rejector post hoc to a multimodal regressor. The time-series datasets are excluded from this analysis because text changes the regressor's MSE by less than 1% on all of them.
W1/Q1-2. Why an LLM rather than a multimodal error predictor?
To our knowledge, no existing selective regression method uses text. SR-TabSTAR, SR-MMTSFlib, and SR-TaTS are not prior methods: we constructed them for our problem setting by replacing the loss predictor of SR-Kernel/SR-kNN with TabSTAR, MMTSFlib, and TaTS. We have revised Section 5.1 to state this explicitly.
The advantage of our LLM-based rejector over a multimodal error predictor is the explanation. In Table 1, the proposed rejector has the best average rank and performs comparably to SR-TabSTAR, the strongest multimodal error predictor we constructed. What SR-TabSTAR and the other baselines cannot do is explain a rejection. None of them produces evidence or a rationale. The problem we pose requires both the rejection and its explanation, and the LLM-based rejector is the component that makes it solvable. Comparable selective regression performance together with natural-language explanations is the advantage we claim.
For a comparison under the same budget, we also added a non-generative rejector built on an LLM of comparable size: a frozen Qwen3-Embedding-8B embedding of the text, concatenated with the numerical input and the prediction, is fed to a GBDT trained on the same target, the realized loss. We show the result in Table 1 in the revised manuscript. The proposed LLM-based rejector outperforms it on 6 of the 7 datasets and on average. The LLM's text representation alone therefore does not account for the performance. Rather, the generative formulation and the fine-tuning contribute. This rejector provides no explanation either.