acceptodds
Under review as a conference paper at ICLR 2027

Identification Limits and a Protected Correction for LLM-Derived Guidance

Abstract

Downstream statistical and machine learning methods increasingly use LLM judgments as external guidance, such as penalty weights, pairwise comparisons or prior scores, often treating the elicited signal as having a stable meaning. Yet when an LLM is shown task-matched evidence, its judgment may combine an evidence-free preference with an evidence-induced shift. We show that repeated querying can estimate this composite judgment arbitrarily precisely without identifying its decomposition. Whenever the queried evidence configurations cannot be combined to reproduce the empty, evidence-free configuration, multiple preference-shift decompositions remain observationally equivalent. This ambiguity persists under adaptive querying and cannot be resolved by increasing the query budget. We therefore develop Selective Correction, whose guarantees depend only on observable quantities. A data-only reference procedure identifies comparisons that could change the decision, an evidence gate restricts queries to those supported by task-matched evidence and the LLM supplies their direction, with its certainty allowed only to attenuate the correction. We prove that any reference ordering whose margin exceeds the sum of its local correction budgets is preserved for every possible set of LLM responses. Across seven datasets, Selective Correction improves over the data-only Reference in every setting while certifying more than 93% of reference orderings in every audited setting. Randomization controls further separate the gains due to LLM content from those due to selective targeting alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-assisted decision making, pairwise judgments, selective querying, robustness, identifiability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1o9UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the use of an LLM to obtain pairwise comparisons given additional evidence and shows that standard mitigation strategies, such as repeated sampling, do not provide further insight into whether the pairwise comparison depends on the LLM's bias (evidence-free preference) or on the supplied evidence. Based on those insights, they propose an outcome-based method, Selective Correction, that does not rely on identifying the decomposition and instead uses a reference selector. In several experiments, they showcase that their method often outperforms the investigated baselines.

Critical strengths

[S1] Theoretical Rigorous: The provided theoretical insights are well explained, and the derivations are easy to follow. The Assumptions are clearly stated, although it is not clear whether they hold in practice, and the outcome is nicely explained. I did not find an error in the proofs whilst checking the Appendix. Overall, I have great trust in the theoretical insights provided.

Critical weaknesses

Summary: As indicated by my confidence, I am not an expert in this field and only have broad knowledge of the investigation conducted. I find the theoretical insights interesting, whilst having questions regarding their practical relevance [Q2]. My main concerns are the experimental evaluations conducted for the method. I would thus encourage the authors to provide further details in this regard.

[W1] Experimental Comparisons: The comparisons between baselines and the Selection Correction do not clearly distinguish which effect was caused by using the LLM judge. Baselines such as Global and Shuffle also use an LLM-based judge, yet it is not clear whether this is DeepSeek or Qwen, as both are used in the Selective Correction. Also, the main experiments do not showcase a pure LLM-based approach (which can only be found in Table 7, yet those settings do not seem to match those in the main experiments. Also, as different reference selectors can be refined and the papers state that they use them, the main Figure 3 does not seem to distinguish between them, or at least this is only done for a few of the provided datasets. I would strongly encourage the authors to provide more details on those points.

[W2] Statistical Significance: Throughout the paper, results of a single run are reported, with only Table 7 in the Appendix showing statistical variance across three seeds only for the LLM-based method. As such, the main experimental results are missing the multiple runs for all the investigated methods, including the proposed method. As such, it is not clear to what extent the results hold across multiple runs. Giving further details here would improve the soundness of the paper regarding the experimental insights.

[W3] Dependence on Reference Selector: The proposed method seems to depend on the chosen reference selector, and it is not entirely clear from the paper which should be chosen in practice. This is further supported by the lack of experimental details in that regard (see [W1]). Also, restricting the LLM to be concise with the reference selector does imply that the LLM-based correction can not strongly change the outcome of the reference selector (based, of course, on weighting in λ\lambda). I would expect further details on the recommended choices for practical applications of the method.

Questions for authors

[Q1] How is the certainty of the LLM retrieved, and in particular, how is pep_e obtained from the LLM? The term „order-symmetrized preference probability“ is not explained much.

[Q2] The theoretical insights of Section 2 are based on the question of the empty configuration not being contained in the affine hull. Practically, I am wondering whether it is a problem that it does not contain the empty configuration, and, as such, how practical and insightful the derived theoretical insights are.

[Q3] Which selector methods are used in Figure 3? Are the shown results consistent over all the dicussed reference selections?

[Q4] To what degree are the empirical insights statistically robust? The main experimental results seem to be missing multiple runs conducted. If you have conducted multiple seeds per dataset, how many were done?

[Q5] Can you give further details on why exactly the threshold has been chosen to be 0.25? I did not fully follow how this number was derived, and I find it rather low. How would the results change with varying these numbers? What happens at the extremes?

7 comments

Thank you for checking the proofs and for the detailed experimental questions, which led to the repeated runs of the revision. We answer Q3 with W1 and Q4 with W2.

W1. Experimental comparisons. The comparisons between baselines and the Selection Correction do not clearly distinguish which effect was caused by using the LLM judge. Baselines such as Global and Shuffle also use an LLM-based judge, yet it is not clear whether this is DeepSeek or Qwen, as both are used in the Selective Correction. Also, the main experiments do not showcase a pure LLM-based approach (which can only be found in Table 7, yet those settings do not seem to match those in the main experiments. Also, as different reference selectors can be refined and the papers state that they use them, the main Figure 3 does not seem to distinguish between them, or at least this is only done for a few of the provided datasets. I would strongly encourage the authors to provide more details on those points.

Q3. Selectors in Figure 3. Which selector methods are used in Figure 3? Are the shown results consistent over all the dicussed reference selections?

Selective Correction, Global Correction and the shuffled control all use DeepSeek V4.1 Flash. The Qwen runs are a separate replication. Section 4.2 now states this. The Reference uses data alone. The shuffled control permutes the DeepSeek judgments across the targeted comparisons and the random control replaces them by random draws. Ours (Qwen) uses Qwen3-32B under the same targeting, evidence and validation, with Qwen2.5-14B-Instruct on JKP153. The effect of the LLM judgments is read from the paired comparison of Selective Correction with the two controls, which keep the queried comparisons and their shares of the weight, reselect λ\lambda by the same validation and replace only the judgments. It exceeds both in most comparisons over ten repetitions, covering all seven datasets (Appendix D.1).

The pure LLM approach is evaluated in the main settings. Table 11 (Table 7 of the submission) runs direct LLM-only selection with the same kk, candidate universe, data split and downstream predictor as the corresponding main settings. It scores each candidate from the task description and its identity alone, in three independent runs. Its value repeats across rows with the same kk because it uses no selector. Selective Correction exceeds both the LLM-only mean and its ensemble on Breast pCR, CREDIT-G, Osteoporosis and the three Darmanis settings with repeated results. The LLM-only mean falls below the Reference in most settings with repeated results, while Selective Correction improves on the Reference in all of them. Against the five LLM-based feature-selection methods of Appendix D.2, Selective Correction attains the highest AUROC in most settings.

Section 4 now names the reference selector of every setting of Figure 3. The gains hold across selectors (Q3). Renal TCMR uses the elastic net with an SIS tie-breaker and k=50k=50, GSE272769 the elastic net with k=50k=50 and Breast pCR sure independence screening with k=20k=20. CREDIT-G uses gradient-boosted permutation ranking and the elastic net with k=10k=10. Osteoporosis uses masked ℓ1\ell_1-penalized logistic regression with k=10k=10. Darmanis uses SIS-10, SIS-20 and EN-10, with LASSO-10, LASSO-20 and EN-20 added in Tables 10 and 11. JKP153 ranks the 153 factors by estimated alpha over a 120-month window. CREDIT-G and Darmanis report more than one selector. Selective Correction improves on the Reference with both CREDIT-G selectors and with all six Darmanis selectors (Tables 10, 11 and 12).

Revision. Section 4 names the reference selector of every setting. A new Table 16 lists the configuration of each selector. Section 4.2 identifies the methods and controls with their elicitation models and points to Tables 10 and 11.

W2. Statistical significance. Throughout the paper, results of a single run are reported, with only Table 7 in the Appendix showing statistical variance across three seeds only for the LLM-based method. As such, the main experimental results are missing the multiple runs for all the investigated methods, including the proposed method. As such, it is not clear to what extent the results hold across multiple runs. Giving further details here would improve the soundness of the paper regarding the experimental insights.

Q4. Statistical robustness and number of runs. To what degree are the empirical insights statistically robust? The main experimental results seem to be missing multiple runs conducted. If you have conducted multiple seeds per dataset, how many were done?

The revision reports ten repetitions of every corrected method and control. The gains are stable across them (Q4). Selective Correction and its Qwen replication are repeated ten times in every setting of Figure 3, Global Correction ten times wherever it is reported. The shuffled and random controls are repeated ten times and paired with Selective Correction. The Reference is a fixed baseline, which makes every repetition a paired comparison against it.

SettingMetricOursOurs − Ref.Ours − GlobalControls
Renal TCMR, Top-50AUPRC0.416 ± 0.022+0.032+0.041p<0.01p < 0.01
GSE272769, EN-50AUROC0.634 ± 0.010+0.006+0.014p<0.02p < 0.02
Breast pCR, SIS-20AUROC0.738 ± 0.001+0.015+0.026p<0.002p < 0.002
CREDIT-G, GBM-10AUROC0.793 ± 0.000+0.013+0.005random p<0.01p < 0.01
Osteoporosis, Top-10AUROC0.738 ± 0.000+0.009+0.003p<0.02p < 0.02
Darmanis, SIS-10AUROC0.949 ± 0.001+0.035+0.028p<0.01p < 0.01
JKP153, 2025 held-outRank IC−0.124 ± 0.002+0.144+0.068p<0.001p < 0.001

Mean ± standard deviation over ten repetitions. Ours − Global uses the mean of the ten repetitions of Global Correction. The last column summarizes the one-sided paired tt-tests against the shuffled and random controls (Appendix D.1). The CREDIT-G row reports the random control (p=0.002p=0.002). Darmanis improves on the Reference by +0.018 to +0.035 across its three settings.

The conclusions hold across runs and across models. Selective Correction improves on the Reference in every reported setting and exceeds Global Correction in mean on all seven datasets. Qwen, with its own judgments, improves on the Reference in most prediction settings and in both JKP153 periods.

What a repetition varies. Each repetition redraws the resamples behind targeting and reselects λ\lambda on validation data. Data partitions, evidence packets, prompts and LLM measurements stay fixed. The spread therefore measures variability from targeting and resampling. Randomness of the LLM is small by construction, since each judgment is read from token log-probabilities. The Qwen replication probes variability across models. The external and temporal test cohorts, Renal TCMR, Breast pCR and JKP153 2025, evaluate the selected sets on data never used for selection.

Revision. Figure 3, Table 1 and Tables 10 and 11 report means and standard deviations over repetitions. Section 4.2 states what each repetition varies. Appendix D.1 gives the paired control tests.

W3. Dependence on the reference selector. The proposed method seems to depend on the chosen reference selector, and it is not entirely clear from the paper which should be chosen in practice. This is further supported by the lack of experimental details in that regard (see [W1]). Also, restricting the LLM to be concise with the reference selector does imply that the LLM-based correction can not strongly change the outcome of the reference selector (based, of course, on weighting in λ\lambda). I would expect further details on the recommended choices for practical applications of the method.

Choose the selector the analyst would use without an LLM. The method wraps that selector, whose choice and hyperparameters are fixed by data-only validation before any query. With the normal-score anchor and under the conditions of the invariance lemma, the procedure uses only the rankings it produces (Appendix B.3).

Appendix D.5 now gives recommended defaults for a new application. Choose the reference selector and kk by data-only validation before any query. Compute targeting from at least 200 refits drawn over the scientifically independent sampling unit. Restrict candidate pairs to a window of five ranks around the selection boundary and query a pair when Uij≥0.25U_{ij}\ge0.25 (Q5). Choose λ\lambda on validation data over a grid containing 0. Report the top-kk certificate with the selected set, together with the local budgets ρj\rho_j, which bound the change of each score.

Validation can support substantial changes to the selection. The strength λ\lambda sets the limit. The correction interpolates between the two sources,

sλ⋆=a  at λ=0,sλ⋆→μ(d)+c1  as λ→∞,s^\star_{\lambda}=a\ \ \text{at }\lambda=0, \qquad s^\star_{\lambda}\to\mu(d)+c \mathbf{1}\ \ \text{as }\lambda\to\infty ,

where the limit holds for coherent soft targets on a connected graph of positive-weight comparisons, cc is a constant and μ(d)\mu(d) is the ranking score that the judgments themselves express (Appendix B.6). In between, an isolated hard-target judgment that disagrees with the Reference overturns a margin ai−aja_i-a_j exactly when its weight, which is proportional to λ\lambda, exceeds that margin. Section 2 shows that the elicitation record cannot say how much the answers deserve, which leaves labeled outcomes to choose λ\lambda. In the planted-truth study of Appendix C.6, perfect reports with every cross-boundary pair queried reduce the utility deficit of the selected set to 8% of that of the Reference at the validation-selected strength. Direct LLM selection in Table 11 is the opposite design. Its mean falls below the Reference in most settings with repeated results, while Selective Correction improves on the Reference in all of them.

Revision. Appendix D.5 adds these defaults and Section 3.4 points to them.

Q1. Extraction of the certainty and of pep_e. How is the certainty of the LLM retrieved, and in particular, how is pep_e obtained from the LLM? The term "order-symmetrized preference probability" is not explained much.

From token log-probabilities, with both presentation orders. Each query asks for one answer token, A or B. The probability that an item is preferred is read from the renormalized first-token probabilities. Each comparison is queried in both presentation orders, with probabilities pe(1)p_e^{(1)} and pe(2)p_e^{(2)} that the same item is preferred. Under the logistic comparison model, averaging the two logits cancels the additive order effect, which is what order-symmetrized means,

pe=σ(12{logit⁡pe(1)+logit⁡pe(2)}),Ce=1−Ent⁡(pe)/log⁡2.p_e=\sigma\Bigl(\tfrac12\bigl\lbrace \operatorname{logit} p_e^{(1)}+\operatorname{logit} p_e^{(2)}\bigr\rbrace \Bigr), \qquad C_e=1-\operatorname{Ent}(p_e)/\log2 .

Here σ\sigma is the logistic function and Ent⁡\operatorname{Ent} the binary entropy. Renal TCMR instead averages the two presentation orders on the probability scale. The certainty equals one for a decisive report and zero for an indifferent one. It measures decisiveness and sets the share of each comparison in the correction budget.

Revision. Appendix E.2, referenced from Section 3.2, gives the formulas and the extraction for each model and dataset.

Reviewer_RALaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the identifiability of LLM-derived guidance and proposes Selective Correction, a framework that selectively incorporates LLM judgments into data-driven rankings. The authors provide theoretical results on identification limits and bounded ranking changes, followed by experiments on seven datasets.

Critical strengths
  • The paper provides a theoretical perspective on the limitations of LLM-derived guidance.

  • Experiments across seven datasets demonstrate improvements over data-only baselines.

Critical weaknesses
  • The identifiability analysis in Proposition 1 and the proposed Selective Correction method are only loosely connected. The method does not directly address the identified ambiguity, and its design appears largely independent of the preceding theoretical results.

  • The theoretical novelty and practical significance of Theorem 1 are limited. Its guarantee essentially follows from bounded score perturbations under quadratic regularization and holds for arbitrary query sets and nonnegative weights. Moreover, preserving reference rankings does not imply correctness, since incorrect but stable rankings are equally protected.

  • Proposition 1 relies on a linear additive decomposition of LLM preferences and evidence-induced shifts, which may not adequately capture complex interactions between evidence and judgments in real LLMs. The non-identifiability result also depends on the available evidence configurations, leaving its practical relevance under alternative elicitation designs unclear.

  • The experiments do not sufficiently establish the necessity of LLM guidance. Randomized controls achieve similar improvements in some settings, suggesting that part of the benefit comes from selective targeting rather than meaningful LLM knowledge. Stronger evidence is needed to isolate the contribution of LLM judgments.

Questions for authors

See weaknesses.

5 comments

Thank you for the four comments. The first two go to the core of the paper and receive the most detailed answers.

W1. Connection between Proposition 1 and Selective Correction. The identifiability analysis in Proposition 1 and the proposed Selective Correction method are only loosely connected. The method does not directly address the identified ambiguity, and its design appears largely independent of the preceding theoretical results.

Section 2 settles what the record can supply. It supplies the judgment precisely and leaves its weight open. Two invariances carry this,

Pq,Δ=Pq+Zγ, Δ−γc0⊤⏟same record, different q (Proposition 1)Ben+(S^)=−Ben−(S^)⏟same record, opposite value (new Corollary 2).\underbrace{\mathbb{P}_{q,\Delta}=\mathbb{P}_{q+Z\gamma,\ \Delta-\gamma c_0^{\top}}}_{\text{same record, different }q\ \text{(Proposition 1)}} \qquad\qquad \underbrace{\mathrm{Ben}_+(\hat S)=- \mathrm{Ben}_-(\hat S)}_{\text{same record, opposite value (new Corollary 2)}} .

Here μ(d)=q+ZΔd\mu(d)=q+Z\Delta d is the judgment under configuration dd, with qq the evidence-free preference and ZΔdZ\Delta d the shift induced by the evidence, γ\gamma is arbitrary and c0⊤d=1c_0^{\top}d=1 for every queried dd. Proposition 1 shows that the designs commonly used to stabilize LLM guidance sharpen the composite judgment without revealing what it means. The new Proposition 2 and its Corollary 2 compare two settings in which every selection rule S^\hat S computed from the record has the same law, while its benefit Ben±(S^)\mathrm{Ben}_\pm(\hat S) over a uniformly random selection has opposite signs (Section 2.3, Appendix A.5). No statistic of the record, not even an exact estimate of qq, can therefore tell how much the judgments should count. This covers the signals commonly used to decide trust, such as self-consistency, verbal confidence, stability under rewording and agreement across models, all computed without evaluation outcomes. Querying buys precision, not trust.

Each design choice of the method follows from this. The corrected scores s⋆s^\star of Equation 5 satisfy

sj⋆−aj=∑e∋j±we{te−σ(sue⋆−sve⋆)}⏟∈[−1,1],ρj:=∑e∋jwe=pλ∑e∋jwˉeCeV,∑jρj=2pλ,s^\star_j-a_j=\sum_{e\ni j}\pm w_e \underbrace{\bigl\lbrace t_e-\sigma(s^\star_{u_e}-s^\star_{v_e})\bigr\rbrace }_{\in[-1,1]}, \qquad \rho_j:=\sum_{e\ni j}w_e=p\lambda\sum_{e\ni j}\frac{\bar w_eC_e}{V}, \qquad \sum_j\rho_j=2p\lambda ,

with aa the reference scores, pp the number of items, σ\sigma the logistic function, tet_e the target that the judgment sets on the comparison e=(ue,ve)e=(u_e,v_e) and wˉeCe/V\bar w_eC_e/V the share of that comparison, its base weight times its certainty, normalized over the queried comparisons. The last two equalities are the new Equation 6 and hold whenever some share is positive. Section 2 fixes three features of these identities. First, no reading of a judgment is certified, which calls for a guarantee that holds for every answer. The bracket is bounded by one whatever the answer, which gives Theorem 1, ∣sj⋆−aj∣≤ρj\lvert s^\star_j-a_j\rvert\le\rho_j. A loss with a bounded derivative is also necessary for such a guarantee (Appendix B.5). Second, more answers and more confident answers add precision and no trust. The total 2pλ2p\lambda is accordingly the same whatever the number of queries and whatever the answers, which only decide where it goes. Third, only labeled outcomes read the value of the guidance. Validation therefore chooses λ\lambda on a grid containing zero, with a held-out shortfall bounded by the validation error (Appendix A.5). The remaining component, where to ask, belongs to the data. Resampling the reference selector locates the comparisons whose order is unstable and matters for the selection.

Each step is verified. In Figure 2B the error against the evidence-free preference is still .654.654 at the largest query budget while the error against an observationally equivalent point falls at the parametric rate, which is precision without meaning. Complementing the evaluation labels turns the mean selected gain of 7.3×10−37.3\times10^{-3} into −7.3×10−3-7.3\times10^{-3} while the record, the correction path and the selected strength stay identical (Appendix C.4), which is one record with opposite values. With the total weight fixed, varying the mean degree of the query graph from 2 to 16 leaves the certified fraction between 0.977 and 0.980 (Appendix C.5). With uninformative reports every fixed positive strength loses on average, while validation keeps the selected strength small and the deficit of the selected set does not increase (Appendices C.3 and C.6). On real data, Selective Correction exceeds Global Correction in mean on all seven datasets with no more queries (Table 1).

Revision. Section 2.3 states the new Proposition 2 and the resulting design constraint. Appendix A.5 states Corollary 2. Section 3 introduces the two budgets. Section 3.2 states the two identities above (Equations 6 and 7). Appendix C adds the verification studies, which the new Section 4.3 and Table 2 summarize in the main text.

W2. Novelty and significance of Theorem 1. The theoretical novelty and practical significance of Theorem 1 are limited. Its guarantee essentially follows from bounded score perturbations under quadratic regularization and holds for arbitrary query sets and nonnegative weights. Moreover, preserving reference rankings does not imply correctness, since incorrect but stable rankings are equally protected.

Protection follows from the same insight. Overriding the Reference pays only where the report is more accurate than the data, which the record cannot reveal. The reviewer is right that a wrong but stable ordering stays protected. On such a pair the method is exactly as good as the Reference. In the calibrated benchmark of Appendix B.6, keeping a reference ordering held with confidence mm errs with probability eD(m)e_{\mathrm D}(m), which decreases in mm, while following a report of accuracy accJ\mathrm{acc}_{\mathrm J} errs with probability 1−accJ1-\mathrm{acc}_{\mathrm J}. Hence

following the report lowers the error  ⟺  eD(m)>1−accJ  ⟺  m<m⋆(accJ),\text{following the report lowers the error} \iff e_{\mathrm D}(m) > 1-\mathrm{acc}_{\mathrm J} \iff m < m^\star(\mathrm{acc}_{\mathrm J}),

with a cutoff m⋆m^\star that grows with the accuracy. The least stable orderings are the first worth opening and the stable ones the last. How far to open depends on accJ\mathrm{acc}_{\mathrm J}, which by Proposition 2 no statistic of the record certifies. The two budgets implement this rule. The query budget admits the pairs that the data order with low confidence, the strength λ\lambda sets the largest reference margin a disagreeing report can overturn and labeled validation chooses λ\lambda in place of the unknown accuracy.

The loss can change only on the pairs the budgets open. There the same budgets certify repairs. Under the task order, with cijc_{ij} the cost of misordering a pair and EijD∈{0,1}E^{\mathrm D}_{ij}\in\lbrace 0,1\rbrace marking a reference error,

L(s⋆)−L(a)=∑(i,j)∈Revcij(1−2EijD),Rev⊆{(i,j):∣ai−aj∣≤ρi+ρj−Wij},L(s^\star)-L(a)=\sum_{(i,j)\in\mathrm{Rev}}c_{ij}\bigl(1-2E^{\mathrm D}_{ij}\bigr), \qquad \mathrm{Rev}\subseteq\bigl\lbrace (i,j):\lvert a_i-a_j\rvert\le\rho_i+\rho_j-W_{ij}\bigr\rbrace ,

where Rev\mathrm{Rev} collects the pairs whose order the correction reverses and WijW_{ij} is the weight on the comparisons joining ii and jj (Appendix B.5). Each reversal either repairs a reference error or introduces one. Every other pair contributes zero, which is the no-harm guarantee. No reversal occurs at λ=0\lambda=0. For fixed reports the thresholds on the right grow linearly with λ\lambda, which makes the strength that validation selects the size of the region open to repair. Reports that point against a reference order with enough weight certify its reversal, just as a large margin certifies its preservation (Appendix B.4). The planted-truth study of Appendix C.6 shows the budgets at work. With uninformative reports, opening every cross-boundary pair at a large fixed strength multiplies the utility deficit of the selected set fivefold, while validation keeps the strength small and the deficit does not grow. With perfect reports and every cross-boundary pair queried, the deficit falls to 8% of that of the Reference at the validation-selected strength. Under the default budget the number of falsely selected items falls at every tested accuracy of the reports. Under the conditions of Appendix B.6, a sampling misordering is more likely to pass the threshold the better separated the pair is in the population. In two sparse-regression designs the default threshold admits over 95% of the harmful pairs in the candidate pools, each joining a falsely selected feature to a wrongly excluded signal (Appendix C.5). Errors that the selector makes in the population become stable as the sample grows and escape direct querying. Theorem 1 holds for any query set and therefore covers query rules that reach them, the next step named in the Discussion.

W2 (continued).

On novelty, the contribution is a guarantee that holds for every judgment, with a characterization of when such a guarantee exists. The mechanism is simple, as the reviewer says. A new proposition (Appendix B.5) shows that it is also necessary. Replace the cross-entropy of Equation 5 by any convex differentiable loss ℓ(ζ;t)\ell(\zeta;t) of the margin ζ\zeta. Then

∣sj⋆−aj∣≤Lρj  for every query set, weighting and target  ⟺  sup⁡ζ,t∣ℓ′(ζ;t)∣≤L.\lvert s^\star_j-a_j\rvert\le L \rho_j \ \text{ for every query set, weighting and target} \iff \sup_{\zeta, t} \lvert\ell'(\zeta;t)\rvert\le L .

The cross-entropy attains L=1L=1. Squared error on the logit scale has an unbounded derivative and admits no such budget. Neither threshold of Theorem 1 can be lowered (Appendix B.4). Its validity for arbitrary query sets makes the certificate practical. Anyone can compute it from the reference scores, the query set and the realized weights. On real data it certifies between 94% and 99% of the reference orderings at the validation-selected strength in the five settings that report it, while the selection improves on the Reference in each (Table 1).

Revision. Appendices B.4 to B.6 add the repair threshold, the signed accounting and the override rule, with proofs. Appendices C.5 and C.6 add the harmful-pair and planted-truth studies. Section 3.3 states why protection is the default and the Discussion the next step.

W3. The additive decomposition and dependence on the design. Proposition 1 relies on a linear additive decomposition of LLM preferences and evidence-induced shifts, which may not adequately capture complex interactions between evidence and judgments in real LLMs. The non-identifiability result also depends on the available evidence configurations, leaving its practical relevance under alternative elicitation designs unclear.

The design rule, query the empty configuration, holds without additivity. The revision separates what each assumption adds (Appendix A.4, with the precise conditions),

SourcesCompositeWhat follows
arbitrary interactionsμ(d)=q+Zg(d)\mu(d)=q+Z g(d)q identified  ⟺  0∈Dq\ \text{identified}\iff 0\in\mathcal{D}
additivityg(d)=Δdg(d)=\Delta dq=∑dαdμ(d), identified  ⟺  0∈aff⁡(D)q=\sum_d\alpha_d \mu(d),\ \text{identified}\iff 0\in\operatorname{aff}(\mathcal{D})
interactions of size at most η\etaμ(d)=q+ZΔd+h(d)\mu(d)=q+Z\Delta d+h(d)∥∑dαdμ(d)−q∥∞≤∥α∥1η\lVert \sum_d\alpha_d \mu(d)-q\rVert _\infty\le\lVert \alpha\rVert _1 \eta

With arbitrary interactions between the sources, qq is identified exactly when the empty configuration is queried. Additivity adds a shortcut, in which a combination of nonempty configurations with coefficients αd\alpha_d stands in for the empty one. Interactions therefore close the shortcut and leave the rule intact.

Approximate additivity gives a sensitivity bound. By the third line, the shortcut estimate of qq still certifies every ordering whose gap exceeds 2∥α∥1η2\lVert \alpha\rVert _1\eta. A crossed design of two sources ss and tt tests a proposed η\eta through the interaction contrast

Jst=μ(es+et)−μ(es)−μ(et)+μ(0),η≥∥Jst∥∞/3,J_{st}=\mu(e_s+e_t)-\mu(e_s)-\mu(e_t)+\mu(0), \qquad \eta\ge\lVert J_{st}\rVert _\infty/3,

with ese_s the configuration that shows source ss alone. The contrast is identified and vanishes under additivity. In the simulation of Appendix C.1 the shortcut estimate converged with the predicted limiting bias, while the design containing the empty configuration kept the n−1/2n^{-1/2} rate.

The result classifies alternative designs before the first query. Figure 2 confirms the criterion on six designs, including the two common ones that show every source at every query or ablate one source at a time. Theorem 1 uses no model of the LLM and holds under any of them.

Revision. Appendix A.4 adds the three-level result, the sensitivity bound and an example without Assumption 1. One sentence of Section 2.3 marks which part of the criterion uses additivity.

Reviewer_FcB9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes pairwise LLM elicitation under varying evidence configurations μ(d)=q+ZΔd\mu(d) = q + Z\Delta d and proves that when the empty configuration lies outside the affine hull of queried configurations (0∉aff(D)0 \notin \text{aff}(\mathcal{D})), the evidence-free preference q=μ(0)q = \mu(0) is observationally equivalent to q+Zγq + Z\gamma under any adaptive protocol (Proposition 1). To safely use LLM pairwise judgments in downstream feature selection and ranking, the authors propose Selective Correction, which queries resampling-unstable boundary pairs (U_{ij} = Q_{ij}B_{ij}) gated by external evidence and minimizes an L2L_2-anchored cross-entropy objective with a deterministic displacement bound ∣sj⋆−aj∣≤ρj|s_j^\star - a_j| \le \rho_j (Theorem 1).

Critical strengths
  1. The identification analysis (Proposition 1, Lemma 1 and Corollary 1) cleanly formalizes why repeated sampling and one-source-at-a-time ablations reduce variance around the composite μ(d)\mu(d) without identifying the decomposition (q,Δ)(q, \Delta).
  2. The protected-region bound ∣sj⋆−aj∣≤ρj=∑e∋jwe|s_j^\star - a_j| \le \rho_j = \sum_{e \ni j} w_e (Theorem 1) follows directly from the 1-Lipschitz stationarity condition of the anchored objective and holds uniformly over all pe∈[0,1]p_e \in [0, 1], requiring no calibration or transitivity assumptions on the LLM.
  3. The empirical evaluation covers 7 datasets across transcriptomics, clinical risk, credit, and finance, incorporating shuffled (pSp_S) and random (pRp_R) controls that separate the effect of boundary targeting from the semantic content of LLM judgments.
Critical weaknesses
  1. Proposition 1 focuses on the non-identifiability of the evidence-free preference q=μ(0)q = \mu(0) when 0∉aff(D)0 \notin \text{aff}(\mathcal{D}). However, if qq were desired, one could directly query the empty prompt d=0d = 0 (as done in Appendix D.5). More importantly, Selective Correction in Section 3 enforces an evidence gate (d≠0d \neq 0) because downstream tasks benefit from the evidence-informed composite μ(d)\mu(d), which is identifiable. The theoretical obstruction in Section 2 therefore does not directly motivate the algorithm in Section 3.
  2. External evidence packets are generated by a "GPT-assisted evidence construction workflow" (Figure 1, Section D.1) that retrieves literature, summarizes effect directions, and applies the evidence gate. The exact prompts, date cutoffs, and filtering rules of this workflow are not specified. For public benchmarks like GSE48581, GSE25065, and JKP153, summarized literature may include studies that analyzed the test cohorts directly.
  3. The objective F(s)F(s) adds 12\frac{1}{2} ∥s−a∥22\|s-a\|^2_2 to logistic cross-entropy terms σ(sue−sve)\sigma(s_{u_e} - s_{v_e}), making both the optimization and the certificate ai−aj>ρi+ρja_i - a_j > \rho_i + \rho_j sensitive to the numerical scale of the reference selector aa (LASSO, SIS, GBM), whose normalization is not specified. Additionally, on CREDIT-G and Darmanis, random and shuffled controls match Selective Correction (Figure 3), indicating that LLM content does not drive the gain on those tasks.
Questions for authors
  1. How is the reference score vector a∈Rpa \in \mathbb{R}^p normalized across different selectors (LASSO, SIS, EN, GBM) before solving Equation 5?
  2. How are the pairwise probability pep_e and certainty CeC_e extracted from DeepSeek V4.1 Flash and Qwen3-32B (token log-probabilities vs. sampled frequencies)?
  3. In Section D.1, how did you verify that the literature retrieved for the transcriptomic and financial evidence packets excluded papers reporting results on the test cohorts (GSE48581, GSE25065, and JKP153 2024/2025)?
3 comments

Thank you for the careful reading and the precise questions. We answer Q3 with W2 and Q1 with W3.

W1. The obstruction of Section 2 and the algorithm of Section 3. Proposition 1 focuses on the non-identifiability of the evidence-free preference q=μ(0)q=\mu(0) when 0∉aff⁡(D)0\notin\operatorname{aff}(\mathcal{D}). However, if qq were desired, one could directly query the empty prompt d=0d=0 (as done in Appendix D.5). More importantly, Selective Correction in Section 3 enforces an evidence gate (d≠0d\neq0) because downstream tasks benefit from the evidence-informed composite μ(d)\mu(d), which is identifiable. The theoretical obstruction in Section 2 therefore does not directly motivate the algorithm in Section 3.

The method works with the composite, as the reviewer says. Section 2 settles how much weight a composite may carry. We agree that the empty configuration identifies qq and that μ(d)\mu(d) is identifiable. Section 2 establishes two invariances:

Pq,Δ=Pq+Zγ, Δ−γc0⊤⏟same record, different q (Proposition 1)Ben+(S^)=−Ben−(S^)⏟same record, opposite value (Corollary 2).\underbrace{\mathbb{P}_{q,\Delta}=\mathbb{P}_{q+Z\gamma,\ \Delta-\gamma c_0^{\top}}}_{\text{same record, different }q\ \text{(Proposition 1)}} \qquad \underbrace{\mathrm{Ben}_+(\hat S)=-\mathrm{Ben}_-(\hat S)}_{\text{same record, opposite value (Corollary 2)}}.

Here μ(d)=q+ZΔd\mu(d)=q+Z\Delta d, γ\gamma is arbitrary, and c0⊤d=1c_0^\top d=1 for every queried dd. The second result is central: under any query design, including the empty prompt, two settings can yield identical elicitation records but opposite benefits for any record-based selector relative to random selection (Section 2.3, Appendix A.5). Thus, even exact identification of qq, self-consistency, verbal confidence, or cross-model agreement cannot establish whether judgments improve the task. The record identifies what the judgments say, not how much they should count.

The method therefore takes each component of the correction from the source that can vouch for it. The corrected scores s⋆s^\star of Equation 5 satisfy

sj⋆−aj=∑e∋j±we{te−σ(sue⋆−sve⋆)}⏟∈[−1,1],∑e∈Ewe≤pλ,s^\star_j-a_j=\sum_{e\ni j}\pm w_e \underbrace{\bigl\lbrace t_e-\sigma(s^\star_{u_e}-s^\star_{v_e})\bigr\rbrace }_{\in[-1,1]}, \qquad \sum_{e\in E}w_e\le p\lambda ,

with aa the reference scores, pp the number of items, σ\sigma the logistic function, tet_e the target that the judgment sets on the comparison e=(ue,ve)e=(u_e,v_e) and a positive sign when j=uej=u_e. Three sources act in this identity. The data and the evidence gate fix the queried comparisons EE before any answer is read. The judgments fix the bracket, a direction of size at most one. Labeled validation outcomes fix λ\lambda and with it the cap pλp\lambda on the total weight. The answers therefore choose directions and shares and cannot raise the cap. Taking absolute values gives Theorem 1, ∣sj⋆−aj∣≤∑e∋jwe\lvert s^\star_j-a_j\rvert\le\sum_{e\ni j}w_e, for every answer the model can return. The evidence gate that the reviewer mentions belongs to the first component. It is a task-specific admissibility rule, separate from identification.

Two experiments illustrate the step. In Appendix C.4, complementing the evaluation labels turns a mean gain of 7.3×10−37.3\times10^{-3} into −7.3×10−3-7.3\times10^{-3}, while the record, the correction path and the selected strength stay identical. This is one record with opposite values. The direct LLM-only selection that the reviewer cites (Appendix D.5 of the submission, now Table 11) reads the model without evidence. Its mean falls below the Reference in most settings with repeated results, while Selective Correction improves on the Reference in all of them. An evidence-free reading is thus available. Its value for the task is settled by labeled outcomes.

Revision. Proposition 2 in Section 2.3 and Corollary 2 in Appendix A.5, with proofs. The end of Section 2.3 and the opening of Section 3 name the two budgets. Section 3.2 states the link between the budgets (Equation 6) and the identity above (Equation 7).

W2. Specification of the evidence workflow and possible test-cohort leakage. External evidence packets are generated by a "GPT-assisted evidence construction workflow" (Figure 1, Section D.1) that retrieves literature, summarizes effect directions, and applies the evidence gate. The exact prompts, date cutoffs, and filtering rules of this workflow are not specified. For public benchmarks like GSE48581, GSE25065, and JKP153, summarized literature may include studies that analyzed the test cohorts directly.

Q3. Exclusion of test-cohort literature. In Section D.1, how did you verify that the literature retrieved for the transcriptomic and financial evidence packets excluded papers reporting results on the test cohorts (GSE48581, GSE25065, and JKP153 2024/2025)?

Appendix E.4 now specifies the workflow in full, with its prompts, date cutoffs and filtering rules. The workflow runs GPT-5.6 Sol. The appendix reproduces the RETRIEVE and ACCEPT prompts of every dataset (Figures 6 to 8). Each item receives two query rounds over at most three authorized indexes, with up to 20 records per index and round and one citation-tracing hop of at most 10 records. Every query, hit and inclusion decision is logged. Supportive, null and discordant findings are all retained. For the six non-JKP datasets the evidence cutoff stated in the prompts is 10 September 2026. JKP153 uses one cutoff per prediction origin, 31 December 2023 for the 2024 validation period and 31 December 2024 for the 2025 held-out period. The RETRIEVE prompt of each dataset excludes the test cohort together with its descendant samples and derivative reports, namely GSE48581 for Renal TCMR, GSE25065 for Breast pCR, GSE272769 for the sepsis task and GSE84465 for Darmanis. CREDIT-G excludes reports of its benchmark outcomes and Osteoporosis the later test batch.

The exclusion is verified by an independent audit of every source (Q3). A separate GPT-5.6 Sol session audits every source against the original record with the ACCEPT prompt, which checks cohort lineage and the exclusion rules. A packet is released only after it passes every audit gate and a manual spot-check. The transcriptomic packets are built from cohorts distinct from the training and evaluation cohorts, namely GSE41998 and GSE32646 for Breast pCR, E-MTAB-4451, E-MTAB-7581 and GSE95233 for GSE272769 and the Ivy Glioblastoma Atlas, GSE116520 and GSE13276 for Darmanis. A new Table 20 reports the scale of every packet. The transcriptomic packets hold between 3,099 and 6,000 gene-by-source records from two or three primary external sources each, the Osteoporosis packets draw on 77 papers and the JKP153 packets hold 214 and 215 approved evidence units from 12 and 13 sources. The judgment model then reads only these frozen packets, under system instructions that forbid tool use, web browsing and external retrieval. We will release every frozen packet with a source-by-source audit against the public test cohorts, their originating studies and the date cutoffs.

Two held-out settings carry structural safeguards. The Osteoporosis cohort is a proprietary hospital dataset that is not publicly available. The JKP153 held-out period, the year 2025, lies after the evidence cutoff of 31 December 2024 that its packets specify, with the 2025 queries locked until the 2024 analysis was frozen. Selective Correction improves on the Reference in both settings. The JKP153 improvement replicates with Qwen2.5-14B-Instruct, a judgment model released before the held-out period began, at a Rank IC of −0.108 against −0.267 for the Reference (Appendix D.3).

Revision. Appendix E.4 reports the workflow with the prompts of every dataset, its authorized sources, retrieval limits, date cutoffs and test-cohort exclusions.

W3. Scale of the reference scores and the CREDIT-G and Darmanis controls. The objective F(s)F(s) adds 12∥s−a∥22\tfrac12\lVert s-a\rVert _2^2 to logistic cross-entropy terms, making both the optimization and the certificate sensitive to the numerical scale of the reference selector (LASSO, SIS, GBM), whose normalization is not specified. Additionally, on CREDIT-G and Darmanis, random and shuffled controls match Selective Correction (Figure 3), indicating that LLM content does not drive the gain on those tasks.

Q1. Normalization of the reference score vector. How is the reference score vector a∈Rpa\in\mathbb{R}^p normalized across different selectors (LASSO, SIS, EN, GBM) before solving Equation 5?

The objective is anchored on a normalized scale. In four datasets it is the normal score of the reference rank, shared across selectors (Q1). For GSE272769, Breast pCR, Darmanis and Osteoporosis,

aj=Φ−1(1−rj−1/2p),a_j=\Phi^{-1} \Bigl(1-\frac{r_j-1/2}{p}\Bigr),

where rjr_j is the reference rank of item jj and Φ\Phi the standard normal distribution function. Renal TCMR divides the raw scores by their maximum. CREDIT-G and JKP153 apply a bounded additive correction on the rank scale in place of Equation 5, which Table 1 now marks. On CREDIT-G, the dataset with the GBM selector, the corrected score is

a~j=p−rjp−1+λhj,∣hj∣≤1,\tilde a_j=\frac{p-r_j}{p-1}+\lambda h_j, \qquad \lvert h_j\rvert\le1,

where hjh_j averages the signed judgments on the comparisons of item jj, weighted by actionability and certainty. Each item therefore moves by at most λ\lambda on that scale for every set of reports (Appendix E.3).

The normal-score anchor settles the scale of the certificate. The same rank-to-score map applies to every selector and its spread sets the unit of the correction budget. A new lemma (Appendix B.3) shows that, under its conditions, a strictly increasing transformation of the fitted scores leaves the query set, the weights, the solution, the certificate and the chosen λ\lambda unchanged.

On both datasets Selective Correction is at least comparable to the controls and higher in most comparisons. Darmanis is a special case. Its data-only Reference is already strong, with an AUROC between 0.91 and 0.98 across the six selectors, which leaves little room above it. Selective Correction still improves on the Reference with all six. Over ten paired repetitions, SIS-10 exceeds the shuffled and random controls (p=0.001p=0.001 and p=0.002p=0.002) and SIS-20 the shuffled control (p=0.002p=0.002). On CREDIT-G, EN-10 exceeds the shuffled control (p=0.003p=0.003) and the random control in every repetition, while GBM-10 exceeds the random control (p=0.002p=0.002). The other comparisons on the two datasets differ by at most 0.005 in AUROC. In the identity of our answer to W1, a control keeps the comparisons and their shares of the weight, reselects λ\lambda by the same validation and replaces only the targets tet_e. The controls can therefore also rise above the Reference, as a planted-truth study confirms (Appendix C.3). The paired difference isolates the content. Across all datasets Selective Correction exceeds both controls in most comparisons (Appendix D.1).

Revision. Sections 3 and 3.2 state the normal-score anchor. Appendix E.3 gives the dataset-specific anchors and Appendix B.3 the invariance lemma. Table 1 marks CREDIT-G and JKP153. Appendix D.1 reports the paired control comparisons.

Q2. Extraction of pep_e and CeC_e. How are the pairwise probability pep_e and certainty CeC_e extracted from DeepSeek V4.1 Flash and Qwen3-32B (token log-probabilities vs. sampled frequencies)?

From token log-probabilities. Each query asks for one answer token, A or B. For DeepSeek V4.1 Flash (API model deepseek-flash) the API returns the log-probabilities of the first output token. We set logprobs=true and top_logprobs=20 with thinking disabled and renormalize over the two answer tokens. For Qwen3-32B, run locally, we read the first-step logits and apply the same normalization. Each comparison is queried in both presentation orders, with probabilities pe(1)p_e^{(1)} and pe(2)p_e^{(2)} that the same item is preferred. Then

pe=σ(12{logit⁡pe(1)+logit⁡pe(2)}),Ce=1−Ent⁡(pe)/log⁡2,p_e=\sigma\Bigl(\tfrac12\bigl\lbrace \operatorname{logit} p_e^{(1)}+\operatorname{logit} p_e^{(2)}\bigr\rbrace \Bigr), \qquad C_e=1-\operatorname{Ent}(p_e)/\log2 ,

with σ\sigma the logistic function and Ent⁡\operatorname{Ent} the binary entropy. Renal TCMR averages the two orders on the probability scale.

Revision. Appendix E.2, referenced from Section 3.2, gives the formulas and the extraction for each model and dataset.

Comments on the paper

1 comment

We thank the three reviewers for their careful reading. The revision is substantial and addresses every comment from the reviewers. Changes are marked in blue in the revised PDF. The algorithm and its implementation are unchanged. The comments concern the theory and the experiments. We summarize our answers here and give the details in each reviewer thread. Point labels combine the reviewer and the comment, for example FcB9-W1.

Theory

  • How Section 2 leads to the method (FcB9-W1, RALa-W1, 1o9U-Q2). Section 2 establishes what can be learned from elicitation: a composite judgment whose precision is limited by the query budget, but whose relevance to the task remains uncertain. More queries can improve precision, but cannot establish whether a judgment should be trusted. Our method addresses this distinction by assigning each part of the correction to an appropriate source of evidence. The data determine which comparisons to query, the elicited judgments indicate the direction of correction, and labeled outcomes determine their influence. We formalize this separation through a new proposition and two identities, and examine its implications in controlled studies.
  • What the guarantee means when the reference is wrong (RALa-W2, 1o9U-W3). Overriding the Reference is beneficial only when a judgment is more accurate than the data, something the elicitation record alone cannot establish. Our method therefore prioritizes protection, guaranteeing no deterioration relative to the Reference on every certified ordering. The two budgets control how much of the Reference can be revised, while new theoretical results characterize when such revisions lead to improvements.
  • How far the assumptions reach (FcB9-W3, RALa-W3). The design rule of Section 2 holds without additivity. The correction is anchored on a normalized, mostly rank-based scale in place of raw selector scores.

Experiments

  • Statistical robustness (1o9U-W2, 1o9U-Q4). The corrected methods and the controls are now repeated ten times in every setting of Figure 3. Selective Correction improves on the Reference in every reported setting and, in mean, on Global Correction on all seven datasets.
  • The contribution of the LLM judgments (FcB9-W3, RALa-W4, 1o9U-W1). Paired controls with shuffled and random judgments separate the content of the judgments from the targeting. Content carries the gain in most comparisons.
  • Protocol and specification (FcB9-W2, FcB9-Q1 to Q3, 1o9U-W1, 1o9U-W3, 1o9U-Q1, 1o9U-Q3, 1o9U-Q5). To ensure reproducibility, the revision specifies the evidence workflow in full, with its prompts, cutoffs and exclusion criteria, together with the score normalization and the extraction of pep_e and CeC_e. We also document the configuration of each dataset and selector, the model behind each comparator, the threshold sensitivity analyses and recommended defaults. Each dataset follows a prespecified evaluation protocol, with all thresholds and tuning grids fixed before held-out evaluation.

Summary of revisions

ConcernRevisionLocation
Link between Section 2 and the methodProposition 2 and Corollary 2, the two budgets and their identitiesSec. 2.3, 3, 3.2, App. A.5
Meaning of Theorem 1Repair threshold, signed loss accounting, override rule, necessity of a bounded loss derivativeSec. 3.3, App. B.4 to B.6
Additivity and interactionsThree-level identification result and sensitivity boundApp. A.4, C.1
MechanismPlanted-truth studies of each componentSec. 4.3, Table 2, App. C
Evidence workflowPrompts, cutoffs, exclusions, audit and evidence scaleApp. E.4, Table 20
Scores and probabilitiesAnchors, invariance lemma, extraction of pep_e and CeC_eApp. B.3, E.2, E.3
Repetitions and controlsTen repetitions, paired controls, LLM-only and cross-model comparisonsSec. 4.2, Table 1, App. D.1 to D.3
Selectors and thresholdSelector of each setting with its configuration, threshold sweeps, defaultsSec. 4, Table 16, App. C.5, D.5

We thank the reviewers again for their time and their constructive comments, which have substantially improved the paper. We would be glad to answer any further questions during the discussion period.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.