- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates the use of an LLM to obtain pairwise comparisons given additional evidence and shows that standard mitigation strategies, such as repeated sampling, do not provide further insight into whether the pairwise comparison depends on the LLM's bias (evidence-free preference) or on the supplied evidence. Based on those insights, they propose an outcome-based method, Selective Correction, that does not rely on identifying the decomposition and instead uses a reference selector. In several experiments, they showcase that their method often outperforms the investigated baselines.
Critical strengths
[S1] Theoretical Rigorous: The provided theoretical insights are well explained, and the derivations are easy to follow. The Assumptions are clearly stated, although it is not clear whether they hold in practice, and the outcome is nicely explained. I did not find an error in the proofs whilst checking the Appendix. Overall, I have great trust in the theoretical insights provided.
Critical weaknesses
Summary: As indicated by my confidence, I am not an expert in this field and only have broad knowledge of the investigation conducted. I find the theoretical insights interesting, whilst having questions regarding their practical relevance [Q2]. My main concerns are the experimental evaluations conducted for the method. I would thus encourage the authors to provide further details in this regard.
[W1] Experimental Comparisons: The comparisons between baselines and the Selection Correction do not clearly distinguish which effect was caused by using the LLM judge. Baselines such as Global and Shuffle also use an LLM-based judge, yet it is not clear whether this is DeepSeek or Qwen, as both are used in the Selective Correction. Also, the main experiments do not showcase a pure LLM-based approach (which can only be found in Table 7, yet those settings do not seem to match those in the main experiments. Also, as different reference selectors can be refined and the papers state that they use them, the main Figure 3 does not seem to distinguish between them, or at least this is only done for a few of the provided datasets. I would strongly encourage the authors to provide more details on those points.
[W2] Statistical Significance: Throughout the paper, results of a single run are reported, with only Table 7 in the Appendix showing statistical variance across three seeds only for the LLM-based method. As such, the main experimental results are missing the multiple runs for all the investigated methods, including the proposed method. As such, it is not clear to what extent the results hold across multiple runs. Giving further details here would improve the soundness of the paper regarding the experimental insights.
[W3] Dependence on Reference Selector: The proposed method seems to depend on the chosen reference selector, and it is not entirely clear from the paper which should be chosen in practice. This is further supported by the lack of experimental details in that regard (see [W1]). Also, restricting the LLM to be concise with the reference selector does imply that the LLM-based correction can not strongly change the outcome of the reference selector (based, of course, on weighting in ). I would expect further details on the recommended choices for practical applications of the method.
Questions for authors
[Q1] How is the certainty of the LLM retrieved, and in particular, how is obtained from the LLM? The term „order-symmetrized preference probability“ is not explained much.
[Q2] The theoretical insights of Section 2 are based on the question of the empty configuration not being contained in the affine hull. Practically, I am wondering whether it is a problem that it does not contain the empty configuration, and, as such, how practical and insightful the derived theoretical insights are.
[Q3] Which selector methods are used in Figure 3? Are the shown results consistent over all the dicussed reference selections?
[Q4] To what degree are the empirical insights statistically robust? The main experimental results seem to be missing multiple runs conducted. If you have conducted multiple seeds per dataset, how many were done?
[Q5] Can you give further details on why exactly the threshold has been chosen to be 0.25? I did not fully follow how this number was derived, and I find it rather low. How would the results change with varying these numbers? What happens at the extremes?
7 comments
W2. Statistical significance. Throughout the paper, results of a single run are reported, with only Table 7 in the Appendix showing statistical variance across three seeds only for the LLM-based method. As such, the main experimental results are missing the multiple runs for all the investigated methods, including the proposed method. As such, it is not clear to what extent the results hold across multiple runs. Giving further details here would improve the soundness of the paper regarding the experimental insights.
Q4. Statistical robustness and number of runs. To what degree are the empirical insights statistically robust? The main experimental results seem to be missing multiple runs conducted. If you have conducted multiple seeds per dataset, how many were done?
The revision reports ten repetitions of every corrected method and control. The gains are stable across them (Q4). Selective Correction and its Qwen replication are repeated ten times in every setting of Figure 3, Global Correction ten times wherever it is reported. The shuffled and random controls are repeated ten times and paired with Selective Correction. The Reference is a fixed baseline, which makes every repetition a paired comparison against it.
| Setting | Metric | Ours | Ours − Ref. | Ours − Global | Controls |
|---|---|---|---|---|---|
| Renal TCMR, Top-50 | AUPRC | 0.416 ± 0.022 | +0.032 | +0.041 | |
| GSE272769, EN-50 | AUROC | 0.634 ± 0.010 | +0.006 | +0.014 | |
| Breast pCR, SIS-20 | AUROC | 0.738 ± 0.001 | +0.015 | +0.026 | |
| CREDIT-G, GBM-10 | AUROC | 0.793 ± 0.000 | +0.013 | +0.005 | random |
| Osteoporosis, Top-10 | AUROC | 0.738 ± 0.000 | +0.009 | +0.003 | |
| Darmanis, SIS-10 | AUROC | 0.949 ± 0.001 | +0.035 | +0.028 | |
| JKP153, 2025 held-out | Rank IC | −0.124 ± 0.002 | +0.144 | +0.068 |
Mean ± standard deviation over ten repetitions. Ours − Global uses the mean of the ten repetitions of Global Correction. The last column summarizes the one-sided paired -tests against the shuffled and random controls (Appendix D.1). The CREDIT-G row reports the random control (). Darmanis improves on the Reference by +0.018 to +0.035 across its three settings.
The conclusions hold across runs and across models. Selective Correction improves on the Reference in every reported setting and exceeds Global Correction in mean on all seven datasets. Qwen, with its own judgments, improves on the Reference in most prediction settings and in both JKP153 periods.
What a repetition varies. Each repetition redraws the resamples behind targeting and reselects on validation data. Data partitions, evidence packets, prompts and LLM measurements stay fixed. The spread therefore measures variability from targeting and resampling. Randomness of the LLM is small by construction, since each judgment is read from token log-probabilities. The Qwen replication probes variability across models. The external and temporal test cohorts, Renal TCMR, Breast pCR and JKP153 2025, evaluate the selected sets on data never used for selection.
Revision. Figure 3, Table 1 and Tables 10 and 11 report means and standard deviations over repetitions. Section 4.2 states what each repetition varies. Appendix D.1 gives the paired control tests.
W3. Dependence on the reference selector. The proposed method seems to depend on the chosen reference selector, and it is not entirely clear from the paper which should be chosen in practice. This is further supported by the lack of experimental details in that regard (see [W1]). Also, restricting the LLM to be concise with the reference selector does imply that the LLM-based correction can not strongly change the outcome of the reference selector (based, of course, on weighting in ). I would expect further details on the recommended choices for practical applications of the method.
Choose the selector the analyst would use without an LLM. The method wraps that selector, whose choice and hyperparameters are fixed by data-only validation before any query. With the normal-score anchor and under the conditions of the invariance lemma, the procedure uses only the rankings it produces (Appendix B.3).
Appendix D.5 now gives recommended defaults for a new application. Choose the reference selector and by data-only validation before any query. Compute targeting from at least 200 refits drawn over the scientifically independent sampling unit. Restrict candidate pairs to a window of five ranks around the selection boundary and query a pair when (Q5). Choose on validation data over a grid containing 0. Report the top- certificate with the selected set, together with the local budgets , which bound the change of each score.
Validation can support substantial changes to the selection. The strength sets the limit. The correction interpolates between the two sources,
where the limit holds for coherent soft targets on a connected graph of positive-weight comparisons, is a constant and is the ranking score that the judgments themselves express (Appendix B.6). In between, an isolated hard-target judgment that disagrees with the Reference overturns a margin exactly when its weight, which is proportional to , exceeds that margin. Section 2 shows that the elicitation record cannot say how much the answers deserve, which leaves labeled outcomes to choose . In the planted-truth study of Appendix C.6, perfect reports with every cross-boundary pair queried reduce the utility deficit of the selected set to 8% of that of the Reference at the validation-selected strength. Direct LLM selection in Table 11 is the opposite design. Its mean falls below the Reference in most settings with repeated results, while Selective Correction improves on the Reference in all of them.
Revision. Appendix D.5 adds these defaults and Section 3.4 points to them.
Q1. Extraction of the certainty and of . How is the certainty of the LLM retrieved, and in particular, how is obtained from the LLM? The term "order-symmetrized preference probability" is not explained much.
From token log-probabilities, with both presentation orders. Each query asks for one answer token, A or B. The probability that an item is preferred is read from the renormalized first-token probabilities. Each comparison is queried in both presentation orders, with probabilities and that the same item is preferred. Under the logistic comparison model, averaging the two logits cancels the additive order effect, which is what order-symmetrized means,
Here is the logistic function and the binary entropy. Renal TCMR instead averages the two presentation orders on the probability scale. The certainty equals one for a decisive report and zero for an indifferent one. It measures decisiveness and sets the share of each comparison in the correction budget.
Revision. Appendix E.2, referenced from Section 3.2, gives the formulas and the extraction for each model and dataset.
Thank you for checking the proofs and for the detailed experimental questions, which led to the repeated runs of the revision. We answer Q3 with W1 and Q4 with W2.
Selective Correction, Global Correction and the shuffled control all use DeepSeek V4.1 Flash. The Qwen runs are a separate replication. Section 4.2 now states this. The Reference uses data alone. The shuffled control permutes the DeepSeek judgments across the targeted comparisons and the random control replaces them by random draws. Ours (Qwen) uses Qwen3-32B under the same targeting, evidence and validation, with Qwen2.5-14B-Instruct on JKP153. The effect of the LLM judgments is read from the paired comparison of Selective Correction with the two controls, which keep the queried comparisons and their shares of the weight, reselect λ by the same validation and replace only the judgments. It exceeds both in most comparisons over ten repetitions, covering all seven datasets (Appendix D.1).
The pure LLM approach is evaluated in the main settings. Table 11 (Table 7 of the submission) runs direct LLM-only selection with the same k, candidate universe, data split and downstream predictor as the corresponding main settings. It scores each candidate from the task description and its identity alone, in three independent runs. Its value repeats across rows with the same k because it uses no selector. Selective Correction exceeds both the LLM-only mean and its ensemble on Breast pCR, CREDIT-G, Osteoporosis and the three Darmanis settings with repeated results. The LLM-only mean falls below the Reference in most settings with repeated results, while Selective Correction improves on the Reference in all of them. Against the five LLM-based feature-selection methods of Appendix D.2, Selective Correction attains the highest AUROC in most settings.
Section 4 now names the reference selector of every setting of Figure 3. The gains hold across selectors (Q3). Renal TCMR uses the elastic net with an SIS tie-breaker and k=50, GSE272769 the elastic net with k=50 and Breast pCR sure independence screening with k=20. CREDIT-G uses gradient-boosted permutation ranking and the elastic net with k=10. Osteoporosis uses masked ℓ1-penalized logistic regression with k=10. Darmanis uses SIS-10, SIS-20 and EN-10, with LASSO-10, LASSO-20 and EN-20 added in Tables 10 and 11. JKP153 ranks the 153 factors by estimated alpha over a 120-month window. CREDIT-G and Darmanis report more than one selector. Selective Correction improves on the Reference with both CREDIT-G selectors and with all six Darmanis selectors (Tables 10, 11 and 12).
Revision. Section 4 names the reference selector of every setting. A new Table 16 lists the configuration of each selector. Section 4.2 identifies the methods and controls with their elicitation models and points to Tables 10 and 11.