acceptodds
Under review as a conference paper at ICLR 2027

CITP: Context-Induced Tilt Posterior

Abstract

Predictive distributions of large language models (LLMs) remain sensitive even to semantically equivalent prompts. However, the variation in predictions across these semantically equivalent prompts can also provide useful information for predictive refinement. We propose **CITP (Context-Induced Tilt Posterior)**, a Bayesian method for refining a frozen LLM's predictive distribution using its logit responses across semantically equivalent prompts. Using a small calibration set of labeled examples from the target task, CITP learns a posterior over how to refine the model's predictions using its responses to semantically equivalent prompts. It then averages predictive distributions over this posterior to account for uncertainty in the learned refinement. Across three model families and nine tasks spanning fixed-label classification and multiple-choice question answering (MCQA), CITP achieves the best macro-averaged normalized Brier score and normalized negative log-likelihood in the main comparison, with relative reductions of 9.5% and 20.2%, respectively, compared with the strongest baseline for each metric. Further analyses show that learning from how predictions change across semantically equivalent prompts can improve predictive performance, and that averaging over the learned posterior provides additional gains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Prompt sensitivity, Calibration, Large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gpzyOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Building on prior findings that prompt-induced variation can improve predictive performance, they propose CITP (Context-Induced Tilt Posterior), a Bayesian framework that refines a frozen LLM’s predictive distribution using its responses to task-equivalent prompts.

CITP estimates a posterior over refinement coefficients using labeled calibration data and determines its final prediction by averaging predictive distributions over a Laplace approximation to this posterior.

They conduct extensive experiments and analyses across multiple model families and benchmarks to demonstrate the contributions of query-matched prompt responses and posterior marginalization.

Critical strengths
  1. The problem statement is clearly defined, and the proposed framework is theoretically well formulated and supported by empirical evidence.
  2. In particular, they provide additional insight into how prompt-induced variation can be used to improve predictive performance.
Critical weaknesses
  1. Further analysis of CITP’s robustness to different prompt constructions and aggregation strategies would help clarify the sources of its performance gains.
  2. They should explicitly clarify the extent to which CITP’s theoretical guarantees apply to its practical implementation.
Questions for authors
  1. How robust are the refinement coefficients to distribution shifts between the calibration and test data?
  2. Additionally, do CITP’s performance gains persist if the prompts or demonstration set change after fitting?
4 comments
AuthorsOpenReview ↗

response to Reviewer gpzy

We thank Reviewer gpzy for the careful review and constructive feedback, and for recognizing the value of using prompt-induced variation for predictive refinement. Changes made in response to your comments are shown in green in the revised manuscript (Section 3.6; Appendix C.8).

W1. Robustness to prompt construction and aggregation strategies.

Tables 2–4 compare aggregation strategies and isolate the effects of query-matched contrasts and posterior averaging.

We also tested 12 predefined eight-view banks, removing one perturbation category and balancing replacement views across the others while keeping the reference and forward-pass budget fixed. CITP was refitted using the same 50 calibration labels for each bank.

T/D/O/F denote template, demonstration-order, option-permutation, and format variants. Each modified-bank row averages three compositions; results are macro-averaged over 27 model–task pairs across ten seeds (mean ± SD).

Method / banknBrier ↓nNLL ↓Accuracy (%) ↑
Scale+Bias0.4679 ± 0.01740.5081 ± 0.026676.34 ± 0.83
CITP, original0.4120 ± 0.00940.4531 ± 0.014179.28 ± 0.48
CITP, without T0.4155 ± 0.00810.4565 ± 0.010879.22 ± 0.40
CITP, without D0.4103 ± 0.00870.4494 ± 0.012279.36 ± 0.41
CITP, without O0.4462 ± 0.01400.4932 ± 0.016277.92 ± 0.70
CITP, without F0.4110 ± 0.00800.4501 ± 0.011979.42 ± 0.53

Across all bank compositions, CITP retained better macro-averaged nBrier, nNLL, and accuracy than Scale+Bias.

W2. Scope of the theoretical guarantees

Thank you for highlighting the need to clarify the scope of our theoretical guarantees.

In response, we clarified in Section 3.6 that Propositions 1–2 establish a residual-alignment criterion and a population-risk improvement bound beyond optimal Scale+Bias, not guarantees for finite-calibration MAP estimates or the Laplace posterior predictive.

Q1. Robustness to distribution shifts between calibration and test data.

Thank you for suggesting this additional analysis.

On five fixed-label tasks, we tested 2×/4× label-prior shifts by reweighting one test class at a time, with coefficients and the prompt bank fixed. This changes empirical class proportions without altering class-conditional distributions.

We average over class shifts using the main aggregation protocol. Entries report mean nBrier/nNLL across class shifts; 1× denotes the original distribution (lower is better).

Weight multiplierScale+BiasWeighted probability ensembleBayesPECITP
1×0.3986 ± 0.0290 / 0.4491 ± 0.04530.3879 ± 0.0140 / 0.5185 ± 0.02540.3893 ± 0.0162 / 0.5017 ± 0.02940.3464 ± 0.0163 / 0.4028 ± 0.0240
2×0.3992 ± 0.0294 / 0.4502 ± 0.04600.3884 ± 0.0140 / 0.5198 ± 0.02580.3898 ± 0.0162 / 0.5029 ± 0.02970.3469 ± 0.0163 / 0.4038 ± 0.0240
4×0.4004 ± 0.0303 / 0.4529 ± 0.04740.3894 ± 0.0141 / 0.5227 ± 0.02680.3907 ± 0.0163 / 0.5055 ± 0.03040.3480 ± 0.0161 / 0.4059 ± 0.0240

CITP retained its advantage in average nBrier and nNLL under both the 2× and 4× shifts.

Additionally, for each method and metric, we averaged the largest loss increase over the 4× class shifts within each model–task–seed cell. CITP showed smaller mean maximum increases in nBrier/nNLL (0.1095/0.1208) than Scale+Bias (0.1238/0.1400) and BayesPE (0.1349/0.1846).

We added these results in Appendix C.8.

Q2. Changes to prompts or demonstrations after fitting.

CITP aims to improve a frozen LLM's predictions by learning how to use responses from a fixed bank of task-equivalent prompts. Its refinement coefficients are fitted to the reference predictions and contrasts of that bank. Reusing the fitted correction under such changes without refitting goes beyond the primary aim of this work.

Nevertheless, we evaluated transfer to nine banks without refitting, varying demonstrations, their order, and class-to-letter mappings while preserving templates, format, and eight-view structure.

Entries are query-weighted scores over 27 model–task pairs, reported as mean ± SD across nine target banks (fixed seed-42 fits; accuracy in %).

MethodnBrier ↓nNLL ↓Accuracy (%) ↑
Temperature scaling0.5551 ± 0.02400.5724 ± 0.021970.08 ± 1.87
Scale+Bias0.5352 ± 0.02280.5986 ± 0.028172.97 ± 1.06
Weighted probability ensemble0.5058 ± 0.02130.6167 ± 0.026674.18 ± 1.04
BayesPE0.5082 ± 0.01920.6035 ± 0.025973.97 ± 1.10
CITP MAP (shared prior)0.5372 ± 0.03150.6366 ± 0.036873.26 ± 1.75
CITP0.5169 ± 0.02960.5587 ± 0.030873.31 ± 1.79

CITP retained better nBrier, nNLL, and accuracy than Scale+Bias under these bank changes.

We thank the reviewer for these valuable suggestions and welcome any further questions or discussion.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer gpzy,

Thank you again for your thoughtful review and constructive feedback.

As the author–reviewer discussion period draws to a close, we would like to kindly follow up on our rebuttal response and revised manuscript, where we provided additional analyses of prompt-bank robustness, distribution shifts, and prompt-bank transfer, along with clarifications of the scope of our theoretical results.

We would sincerely appreciate any further feedback or questions you may have.

Sincerely, The Authors

I appreciate the authors’ thoughtful responses to my review and have carefully read their rebuttal.

The authors have fully addressed all of my concerns.

After reviewing the revised manuscript, I have raised my overall rating from 3 to 4.

Thank you for your follow-up and for reviewing our responses and revised manuscript! We are glad that we could address your concerns.

Reviewer_SAQZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

CITP treats the spread of a frozen LLM's predictions across semantically equivalent prompt variants as a usable correction signal rather than only a failure mode. Per query, centered candidate logits from each view are written as reference-relative contrasts, and the refined logits are affine in a task-shared coefficient vector: a global scale, a class-logit bias in the centered subspace, and signed weights on that query's own contrasts. The contrasts are query-centered so anything shared across queries is absorbed by the bias. A small labeled calibration set fits the coefficients by regularized convex cross-entropy, the posterior is Laplace-approximated at the MAP, and the final prediction averages softmaxes over posterior draws. Section 3.5 shows the family contains temperature scaling, Scale+Bias, simplex and signed affine log-pooling as special cases; Section 3.6 derives a residual-alignment criterion with a lower bound on attainable cross-entropy reduction beyond optimal scale-and-bias. Evaluation spans three model families and nine tasks with predefined candidate sets, 27 model-task pairs, 50 calibration labels per fitted group, 200 test examples, ten outer seeds, and eight forward passes by default. CITP wins macro nBrier by 0.0432 and nNLL by 0.1145 against the best baseline per metric, with paired 95% intervals excluding zero.

Critical strengths
  • The controls are what I would want, and they are what makes the improvement believable rather than a bookkeeping artifact. The shuffled-contrast control (Section 4.2.2 and Table 2) replaces each query's own contrast matrix with a query-independent one while keeping the number of views, the fitting procedure and the budget fixed: nBrier goes from 0.4145 matched to 0.5024 shuffled. That isolates query-specific pairing from "averaging over a perturbation family helps", which is the confound that kills most prompt-ensembling claims. Add ten outer seeds, paired per-cell differences with bootstrap CIs, and a Laplace-versus-NUTS agreement check (mean absolute probability difference 0.0110), and the empirical section is stronger than the field's norm.

  • Section 3.5 converts the baseline comparison into an ablation inside one model class. Setting γ = 0 and b = 0 recovers temperature scaling, u = 0 recovers Scale+Bias, and the raw-contrast reparameterization recovers simplex and signed affine log-pooling. This means the marginal claim is precise and testable rather than "we beat seven prior methods", and Section 3.6 then states a condition under which prompt-contrast directions must help, and the paper measures the predicted relationship: the correlation between the residual-alignment statistic and the observed NLL gain over Scale+Bias.

Critical weaknesses
  • The novelty margin over BayesPE and signed affine log-pooling is thin, and the paper's own decomposition shows it. A signed affine log-pool over the same eight prompt views is essentially CITP's MAP predictor, so the contribution beyond prior work is the affine reparameterization plus Laplace marginalization. Table 1 puts signed affine log-pool at 0.4552 nBrier and CITP MAP at 0.4145, i.e. almost the whole headline gap comes from the scale/bias/centering terms; Table 3 then isolates marginalization at −0.0058 nBrier and −0.0285 nNLL with accuracy unchanged at +0.00. So the "Bayesian" part buys proper-scoring-rule and tail improvements (NLL-CVaR 5.0052→3.7735, wrong-confidence 0.7175→0.6626), not better decisions. Worse for the motivating story, Appendix C.4 shows BayesPE achieves lower nBrier and nNLL at five and ten labels per fitted group, with CITP only ahead at larger budgets — and Section 1 justifies the method precisely by "a small calibration set".

  • The cost of eight forward passes is not paid for in the reported metrics. The strongest nNLL baseline in Table 1 is temperature scaling at 0.5676 — one forward pass, one parameter — and CITP's headline accuracy improvement is only +1.24 percentage points while the gains concentrate in Brier and NLL. The paper notes that prompt variants can be evaluated in parallel, which is true, but there is no wall-clock or throughput number, and no compute-matched alternative such as a single prompt with a stronger calibrator or the same eight views with a plain weighted ensemble tuned on the same 50 labels. A reader who cares about argmax quality rather than calibration cannot currently tell whether CITP is worth 8×.

  • "Task-equivalent" is asserted, never measured. Section 4.1 says the bank is fixed and non-optimized, and Appendix D lists format, demonstration-order and candidate-order perturbations. Candidate-order permutation is known to change the argmax itself — Zheng et al. (2024), cited in Section 2 — so at least some views are not equivalent in the sense the theory assumes. If a subset of views is biased in a consistent direction on the calibration distribution, CITP's fitted u is partly a fit to that bias, and the advantage should shrink when the test set is re-drawn. A drop-one-view ablation, plus reporting the fraction of views whose argmax differs from the reference, would settle this.

  • The calibration arithmetic is tight and unreported. A fitted group has M = 7 contrast weights plus up to K−1 bias dimensions plus γ, sometimes fit on 50 examples, and the reported evaluation units are 200 test examples with three-fold splits where applicable. The convex prior helps, but the paper never discusses effective sample size or how the fitted coefficients behave under a resampled calibration set. For a method whose selling point is Bayesian uncertainty over those coefficients, coefficient-level stability evidence is missing.

Questions for authors
  • Could CITP MAP appear as a row of Table 1, alongside signed affine log-pool, on the same 27-pair macro aggregate? Table 2 reports it on 270 model-task-seed cells, so the two are not directly comparable, and I want to know how much of the headline nBrier gap over signed affine log-pool is the scale term and how much is marginalization. (Clarification; would move my soundness score.)

  • What is the wall-clock and token-throughput overhead of eight views versus one, measured, and how much of the nNLL advantage over temperature scaling remains if the single-prompt budget is matched instead?

  • Please report coefficient stability across the ten outer seeds: the spread of u and of γ within a model-task pair. If the contrast weights swing substantially between seeds, the Laplace posterior width is underestimating what a re-collected calibration set would produce, and the marginalization gains would have a different interpretation.

  • Is there a budget-matched Pareto curve on the calibration axis? Appendix C.4 shows BayesPE winning at 5 and 10 labels, yet the lower block of Table 3 shows your own marginalization effect being largest at exactly that end (Δ_marg of −0.0247 nBrier and −0.1055 nNLL at 5 labels). Those two statements are not obviously consistent, and I would like them reconciled rather than reported separately: a single plot of nBrier/nNLL versus label budget for Scale+Bias, BayesPE, CITP MAP and CITP would make the regime in which CITP is the right choice readable at a glance. Since "small calibration set" is the motivating claim, I think this is answerable from data you already have.

  • Which prompt transformations are genuinely equivalent in your measurement, i.e. how many views flip the reference argmax, and does dropping them change the result? Suggestion for future work if you prefer, but I think a simple count is answerable now.

5 comments
AuthorsOpenReview ↗

response to Reviewer SAQZ

We appreciate Reviewer SAQZ's detailed and constructive review. Changes made in response to your comments are shown in red in the revised manuscript (Sections 3.1, 4.2.1–4.2.2, and 4.2.4; Appendices C.4 and C.9).

We address the related concerns together below.

W1, Q1, Q4: Contribution, posterior marginalization, and calibration budget.

Thank you for emphasizing the need to isolate the contribution of posterior marginalization.

Tables 2–3 are directly comparable: the 270 model–task–seed cells correspond to 27 model–task pairs evaluated over ten seeds, with results averaged within each pair before macro-averaging. We have clarified this aggregation and added CITP MAP to Table 2 (Sections 4.2.1–4.2.2).

Posterior marginalization aims to account for uncertainty in learned refinement coefficients and improve predictive distributions, rather than directly improving argmax accuracy.

Table 4 isolates the effect of marginalization by comparing the MAP predictor and the posterior predictive using the same selected prior precision and MAP fit. As the reviewer notes, accuracy remains nearly unchanged. However, even in the 50-label setting, marginalization yields statistically significant improvements in both nBrier and nNLL. These results suggest that the contribution of posterior marginalization is neither confined to extremely low-label settings nor negligible even in our main 50-label setting.

To further address the reviewer's concern and better understand the effect of posterior marginalization, we have conducted additional analyses and expanded the discussion in Section 4.2.4 and Appendix C.4 of the revised manuscript.

First, we observe that queries with greater predictive disagreement under the coefficient posterior tend to exhibit larger average NLL improvements from marginalization (Figure 5).

In addition, in connection with the coefficient stability concerns raised in W4/Q3, we examined coefficient variability across outer seeds (Figure 6) and conducted a calibration-resampling analysis. The latter shows that model-task pairs with greater variability in fitted predictions tend to have wider conditional posteriors and larger marginalization gains (Figure 7 and Table 11). These findings support the interpretation that the posterior captures uncertainty in the learned refinement and that incorporating this uncertainty through marginalization contributes to improved predictive distributions. Details are provided in our response to W4/Q3.

Regarding calibration-label budget, Figure 8 in the original submission (now Figure 10 in the revised manuscript) already provides the requested comparison of Scale+Bias, BayesPE, CITP MAP, and CITP across different calibration-label budgets.

Method (nNLL ↓)5 labels10 labels20 labels50 labels
BayesPE default0.6014 ± 0.03110.5937 ± 0.03000.5832 ± 0.02810.5707 ± 0.0247
BayesPE tuned0.6063 ± 0.03190.5987 ± 0.03110.5880 ± 0.02860.5717 ± 0.0254
Scale+Bias1.2498 ± 0.29040.9403 ± 0.11090.7834 ± 0.19270.5081 ± 0.0266
CITP MAP (PP prior)0.8036 ± 0.05010.7084 ± 0.03770.6111 ± 0.03280.4816 ± 0.0163
CITP0.6981 ± 0.04360.6180 ± 0.03670.5422 ± 0.02690.4531 ± 0.0141

The same comparison in nBrier shows a consistent budget-dependent pattern.

Method5 labels10 labels20 labels50 labels
BayesPE default0.5114 ± 0.02440.5027 ± 0.02110.4913 ± 0.01820.4794 ± 0.0137
BayesPE tuned0.5174 ± 0.02430.5083 ± 0.02320.4963 ± 0.01870.4804 ± 0.0145
Scale+Bias0.6456 ± 0.07730.5729 ± 0.03410.5137 ± 0.02660.4679 ± 0.0174
CITP MAP (PP prior)0.5977 ± 0.04020.5457 ± 0.03040.4900 ± 0.02220.4178 ± 0.0091
CITP0.5730 ± 0.03540.5263 ± 0.02990.4749 ± 0.02060.4120 ± 0.0094

As the reviewer notes, BayesPE performs better in the extremely low-label regime, whereas CITP achieves better performance with 20 or more calibration labels.

However, BayesPE's superior predictive performance in the extremely low-label regime and the larger marginalization gains observed for CITP in the same regime concern two distinct comparisons. The former compares the final predictive performance of different methods, whereas the latter compares the MAP predictor and the posterior predictive.

Importantly, as discussed above, the benefits of marginalization are not limited to the extremely low-label regime. They persist at both 20 and 50 labels, with non-negligible improvements even in our main 50-label setting (Table 4).

We interpret these findings as additional empirical evidence supporting the role of posterior marginalization in accounting for uncertainty in the learned refinement.

W2, Q2. Computational cost and performance analysis

Thank you for requesting a quantitative evaluation of compute costs. We report measured wall-clock latency and throughput below (Appendix C.9).

ExecutionViewsTime (ms/query) ↓Queries/s ↑Input ktokens/s ↑
Reference, 1 GPU147.79 ± 0.1120.927 ± 0.04711.371 ± 0.025
Sequential, 1 GPU8383.58 ± 0.642.6070 ± 0.004411.352 ± 0.019
Batched, 1 GPU8316.36 ± 0.143.1609 ± 0.001413.7633 ± 0.0061
Parallel 4+4, 2 GPUs8162.96 ± 0.136.1365 ± 0.004826.720 ± 0.021
MethodPostproc. (ms)1 GPU sequential (ms/query) ↓1 GPU batched (ms/query) ↓2 GPU 4+4 (ms/query) ↓2 GPU queries/s ↑2 GPU input ktokens/s ↑
Mean probability ensemble0.20383.77316.56163.166.1326.69
Ensemble + temperature0.30383.88316.66163.266.1326.67
Weighted probability ensemble0.22383.78316.57163.186.1326.68
BayesPE0.40383.98316.76163.366.1226.65
Simplex logarithmic pool0.10383.68316.46163.066.1326.70
Signed affine logarithmic pool0.10383.68316.46163.066.1326.70
CITP MAP0.18383.76316.54163.146.1326.69
CITP MAP (shared prior)0.19383.76316.54163.156.1326.69
CITP2.85386.41319.18165.816.0326.26

Due to hardware constraints, we evaluated one-GPU batching and two-GPU 4+4 parallel execution rather than fully parallel execution of all eight views.

Table 2 already includes a plain weighted eight-view ensemble fitted on the same 50 labels, as well as single-view Scale+Bias, Dirichlet-ODIR, and LinC.

Compared with single-view methods, CITP requires more forward passes but outperforms the evaluated non-CITP eight-view baselines at comparable inference cost. We believe these results provide a concrete basis for assessing whether the predictive gains justify the additional computational cost.

We added this result to Appendix C.9 (Tables 16-17).

W3, Q5. Task equivalence and view ablation

Thank you for asking us to clarify task equivalence.

First, task equivalence in CITP does not require the model's argmax predictions to remain unchanged. Rather, it means that the task-relevant content and candidate semantics are preserved. For option permutations, we map the predictions back to the original candidate order before comparison. Moreover, Propositions 1-2 do not assume argmax invariance.

In fact, CITP is motivated by precisely this prompt sensitivity: a frozen LLM can produce different predictions even for semantically equivalent prompts. An argmax flip is a case where this sensitivity changes the predicted class, rather than a violation of task equivalence. We have further clarified this distinction in Section 3.1 of the revised manuscript.

To address this concern, we selected the view with the highest reference-relative argmax-flip rate on the 50 calibration examples in each fitted group, removed it from calibration and test queries, and refitted CITP using the remaining seven views.

MethodViewsnBrier ↓nNLL ↓Accuracy (%) ↑
Scale+Bias10.468 ± 0.0170.508 ± 0.02776.34 ± 0.83
Original CITP80.4120 ± 0.00940.453 ± 0.01479.28 ± 0.48
CITP: drop highest-flip view70.4223 ± 0.00940.464 ± 0.01378.91 ± 0.49

These results suggest that frequent argmax disagreement does not necessarily make a view uninformative. Rather, even views with frequent argmax flips may contain correction information that CITP can exploit.

For reference, the argmax-flip rates relative to the reference view were 8.34% and 12.68% for template variations, 9.90% and 10.24% for demonstration-order variations, 29.73% and 34.91% for option permutations, and 10.39% for formatting variations.

Our work focuses on exploiting prompt-induced variation using a predefined, non-optimized prompt bank. We agree that combining CITP with more systematic view selection is a valuable direction for future work, and thank the reviewer for this suggestion.

Regarding the concern that CITP may fit calibration-specific biases, we evaluate all methods on held-out test queries using coefficients learned only from calibration data. We also examined synthetic label-prior shifts with the fitted coefficients held fixed, as reported in our response to Reviewer gpzy's Q1. CITP retained its advantage over the evaluated baselines under these shifts.

We also provide additional prompt-bank robustness analyses, including configurations without option-permutation views, in our response to Reviewer gpzy (W1).

W4, Q3. Coefficient stability and posterior uncertainty

We thank the reviewer for raising concerns about coefficient stability and posterior uncertainty when learning from a small calibration set.

Each fitted group has (K+7K+7) coefficients (seven prompt contrasts, (K−1K-1) bias coordinates, and one global scale). In response to the reviewer's request, we added Figure 6 (Appendix C.4), which reports the SDs of (γ\gamma) and each (uju_j) across ten outer seeds for every fitted model–task group.

We also assessed calibration-sampling variability by bootstrapping only the calibration examples, with the prompt bank and test queries fixed.

Spearman correlations were positive for resampling variability vs. posterior width (ρ=0.736)(\rho=0.736), posterior width vs. nNLL marginalization gain (ρ=0.673)(\rho=0.673), and resampling variability vs. marginalization gain (ρ=0.863)(\rho=0.863) (Figure 7; Table 11).

These findings support the interpretation that the posterior meaningfully captures uncertainty in the learned refinement: greater sampling variability is associated with wider posteriors, while marginalizing this uncertainty contributes to improved predictive distributions.

We added these analyses to Section 4.2.4 and Appendix C.4.

W1. Methodological novelty

Lastly, we would like to briefly recap its contributions and clarify their relationship to existing methods.

We added this comparison and discussion to Section 2 (Table 1, marked in blue).

MethodGlobal CalibrationPrompt VariationJoint Component Fit†Bayesian InferenceTheoretical Insight
Temperature scaling○————
Scale+Bias○—△——
mean probability ensemble—○———
Weighted ensemble—○———
BayesPE—○—○Model-error bound from prompt disagreement
Signed affine log pool—○———
CITP○○○○Improvement criterion and risk-reduction bound beyond Scale+Bias

† Joint Component Fit: ○ global calibration + prompt correction; △ multiple global-calibration components; — otherwise.

Based on this comparison, we highlight two complementary aspects of CITP's contribution.

1. A concrete joint Bayesian refinement algorithm with an interpretable decomposition

mean probability ensemble and weighted ensemble aggregate prompt probabilities, BayesPE performs Bayesian prompt aggregation, and signed affine log pooling learns signed prompt weights. However, none of these evaluated formulations jointly infers global calibration and query-specific prompt-contrast corrections within a unified Bayesian refinement model.

In contrast, CITP provides a unified affine formulation that represents global scaling, class bias, and query-specific prompt contrasts as additive correction directions in logit space. In this formulation, the components play distinct roles while remaining affine in a single coefficient vector. This enables global calibration and prompt-contrast coefficients to be jointly learned through a convex MAP objective, with their dependencies captured by a joint posterior. Consequently, uncertainty in both global calibration and query-dependent prompt-contrast corrections can be propagated into the final predictive distribution.

Furthermore, this formulation makes the contributions of global calibration, query-matched prompt contrasts, and posterior marginalization explicit and separately testable.

Moreover, by explicitly separating global calibration from query-dependent prompt-contrast corrections, this formulation provides a natural basis for theoretically analyzing the additional predictive value of prompt sensitivity.

2. A theoretical account of when prompt sensitivity helps

Building on this refinement formulation, we can analyze whether and under what conditions prompt-contrast directions can further reduce population cross-entropy risk beyond optimal Scale+Bias calibration.

Whereas BayesPE derives a general lower bound on expected model error from prompt disagreement, applicable to other semantically equivalent prompt ensembles under an ideal-model invariance assumption, this result does not characterize the predictive improvement attainable from prompt aggregation. In contrast, CITP's Propositions 1–2 require neither an ideal LLM nor such prompt invariance, and directly characterize when prompt-contrast corrections can improve upon optimal Scale+Bias calibration, with a quantitative lower bound on the attainable population cross-entropy risk reduction.

Together, these contributions provide a unified Bayesian refinement framework and a concrete, quantitatively grounded perspective on treating prompt sensitivity not merely as a failure mode to be mitigated, but as a source of learnable correction information.

We thank the reviewer again for the constructive feedback and would welcome any further questions or discussion.

Reviewer_cHBFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents CITP, which uses prediction differences across task-equivalent prompts to correct a frozen LLM’s output. It learns shared correction parameters from a small labeled set and averages predictions over their posterior. The paper gives a theoretical condition for when these corrections help and evaluates the method on three model families and nine tasks. Results show better aggregate probability scores and accuracy than the tested baselines. Ablations suggest that query-specific prompt differences provide useful information, while posterior averaging mainly reduces overconfident errors.

Critical strengths
  1. The central idea is interesting. The same question can get different answers just by changing how it is asked. Rather than treating this instability only as a problem, CITP uses it to improve the model’s answers and confidence estimates. I see value in turning a common failure mode into a learnable correction signal without retraining the LLM.
  2. The controlled comparisons make the results more convincing. The scale-and-bias control, shuffled prompt contrasts, and same-fit comparison with and without posterior averaging help explain where the gains come from. These experiments support the roles of query-matched prompt responses and posterior averaging, giving me more confidence in the design than the headline scores alone would.
Critical weaknesses
  1. The methodological advance may be incremental. CITP’s point predictor can be rewritten as signed logit pooling with scale and bias, and its posterior computation uses a standard Laplace approximation. The overall idea is interesting and the results are encouraging, but I remain unsure how much methodological novelty this combination adds beyond existing pooling and calibration approaches.
  2. The advantage over BayesPE needs a sensitivity check. BayesPE retains its public implementation’s fixed regularization coefficient, while CITP selects its prior strength through cross-validation. Using the default is reasonable, but with so few calibration examples, I would like to know whether tuning this coefficient materially changes the gap. This leaves some uncertainty about robustness, rather than establishing that the comparison is unfair.
Questions for authors

How sensitive is BayesPE to the fixed regularization coefficient of 0.1 under the current prompt bank and calibration budgets? The paper can be improved by checking whether CITP’s advantage remains when this coefficient is selected using the calibration data.

2 comments
AuthorsOpenReview ↗

response to Reviewer cHBF

We thank Reviewer cHBF for the careful review and feedback, and for recognizing the value of our controlled comparisons. Changes made in response to your comments are shown in blue in the revised manuscript (Section 2; Appendix C.7).

W2, Q1. Sensitivity analysis of BayesPE's regularization coefficient

Thank you for suggesting this analysis. We tuned BayesPE via 3-fold CV on the same 50 calibration labels; see our response to Reviewer SAQZ (Q4) for 5–50-label results.

MethodnBrier ↓nNLL ↓Accuracy (%) ↑
BayesPE default0.4794 ± 0.01370.5707 ± 0.024775.39 ± 0.79
BayesPE tuned0.4804 ± 0.01450.5717 ± 0.025475.33 ± 0.89
Scale+Bias0.4679 ± 0.01740.5081 ± 0.026676.34 ± 0.83
CITP MAP (PP prior)0.4178 ± 0.00910.4816 ± 0.016379.28 ± 0.46
CITP0.4120 ± 0.00940.4531 ± 0.014179.28 ± 0.48

BayesPE performed similarly with tuned and default coefficients. Thus CITP's advantage persists even when BayesPE's regularization coefficient is selected using the calibration data. We have added this sensitivity analysis, including detailed experimental settings and results, to Appendix C.7 (Table 13).

W1. About novelty of CITP

We appreciate the reviewer's thoughtful comments on methodological novelty.

A concise version of this comparison and discussion has been added to Section 2 (Table 1) in the revised manuscript.

MethodGlobal CalibrationPrompt VariationJoint Component Fit$^{\dagger}$Bayesian InferenceTheoretical Insight
Temperature scalingO————
Scale+BiasO—△——
mean probability ensemble—O———
Weighted ensemble—O———
BayesPE—O—OModel-error bound from prompt disagreement
Signed affine log pool—O———
CITPOOOOImprovement criterion and risk-reduction bound beyond Scale+Bias

† Joint Component Fit: ○ global calibration + prompt correction; △ multiple global-calibration components; — otherwise.

Based on this comparison, we highlight two complementary aspects of CITP's contribution.

1. A concrete joint Bayesian refinement algorithm with an interpretable decomposition

Mean probability ensemble and weighted ensemble aggregate prompt probabilities, BayesPE performs Bayesian prompt aggregation, and signed affine log pooling learns signed prompt weights. However, none of these evaluated formulations jointly infers global calibration and query-specific prompt-contrast corrections within a unified Bayesian refinement model.

In contrast, CITP provides a unified affine formulation that represents global scaling, class bias, and query-specific prompt contrasts as additive correction directions in logit space. In this formulation, the components play distinct roles while remaining affine in a single coefficient vector. This enables global calibration and prompt-contrast coefficients to be jointly learned through a convex MAP objective, with their dependencies captured by a joint posterior. Consequently, uncertainty in both global calibration and query-dependent prompt-contrast corrections can be propagated into the final predictive distribution.

Furthermore, this formulation makes the contributions of global calibration, query-matched prompt contrasts, and posterior marginalization explicit and separately testable.

Moreover, by explicitly separating global calibration from query-dependent prompt-contrast corrections, this formulation provides a natural basis for theoretically analyzing the additional predictive value of prompt sensitivity.

2. A theoretical account of when prompt sensitivity helps

Building on this refinement formulation, we can analyze whether and under what conditions prompt-contrast directions can further reduce population cross-entropy risk beyond optimal Scale+Bias calibration.

Whereas BayesPE derives a general lower bound on expected model error from prompt disagreement, applicable to other semantically equivalent prompt ensembles under an ideal-model invariance assumption, this result does not characterize the predictive improvement attainable from prompt aggregation. In contrast, CITP's Propositions 1–2 require neither an ideal LLM nor such prompt invariance, and directly characterize when prompt-contrast corrections can improve upon optimal Scale+Bias calibration, with a quantitative lower bound on the attainable population cross-entropy risk reduction.

Together, these contributions provide a unified Bayesian refinement framework and a concrete, quantitatively grounded perspective on treating prompt sensitivity not merely as a failure mode to be mitigated, but as a source of learnable correction information.

We thank the reviewer again for the constructive feedback and would welcome any further questions or discussion.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer cHBF,

Thank you again for your thoughtful review and constructive feedback.

As the author–reviewer discussion period draws to a close, we would like to kindly follow up on our rebuttal response and revised manuscript, where we provided additional results on BayesPE's regularization sensitivity and clarified CITP's methodological and theoretical contributions.

We would sincerely appreciate any further feedback or questions you may have.

Sincerely, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

Dear Area Chair and Reviewers,

Thank you for your feedback and discussion. Reviewer gpzy confirmed that all concerns were addressed and raised the overall rating from 3 to 4. Reviewer SAQZ verified our aggregation clarification and label-budget results and raised Soundness from 2 to 3.

Summary of Responses and Revisions

References below follow the revised manuscript. Results available only in the rebuttal are marked accordingly.

Reviewer / ConcernResponse and key findingsLocation
cHBF W1; SAQZ W1 — NoveltyAdded a comparison with existing methods, clarifying CITP's joint Bayesian estimation and its theory of when prompt responses add value beyond global scale-and-bias calibration.Sec. 2 (Table 1); Secs. 3.3–3.6
cHBF W2/Q1 — BayesPE tuningTuned BayesPE using the same calibration data. Performance remained similar to the default, and CITP retained its advantage in the main 50-label setting.App. C.7 (Table 13)
SAQZ W1/Q1 — Aggregation and posterior averagingClarified that the main tables use the same aggregation protocol and added CITP MAP to Table 2. The existing comparison using the same fit shows significant nBrier/nNLL gains from posterior averaging.Secs. 4.2.1–4.2.2, 4.2.4 (Tables 2–4)
SAQZ W1/Q4 — Calibration budget and posterior averagingClarified two distinct comparisons: BayesPE achieves lower mean nBrier/nNLL at 5/10 labels and CITP at 20/50, while posterior averaging lowers both mean scores relative to the MAP predictor from the same fit at every tested budget.App. C.5 (Fig. 10), Table 4; SAQZ Q4 rebuttal
SAQZ W2/Q2 — Computational costAdded latency and throughput measurements. In the main setting, CITP improves macro-averaged nBrier, nNLL, and accuracy over the non-CITP single- and multi-prompt baselines; the multi-prompt baselines share its forward-pass budget.Tables 2–3; Sec. 4.2.4; App. C.9 (Tables 16–17)
SAQZ W3/Q5 — Task equivalenceClarified semantic equivalence and reported prediction-flip rates. After removing the calibration-selected highest-flip view from calibration and test queries and refitting, CITP retained gains over the global calibration baseline (Scale+Bias).Sec. 3.1; SAQZ W3/Q5 rebuttal (ablation and flip rates)
SAQZ W4/Q3 — StabilityAdded coefficient-wise SDs and calibration-resampling analyses. Greater variability in fitted predictions across resamples is associated with wider posteriors and larger gains from posterior averaging.Sec. 4.2.4; App. C.4 (Figs. 6–7, Table 11)
gpzy W1 — Prompt-bank robustnessTested alternative prompt-bank compositions with refitting. Reported averages remained better than the global calibration baseline in all three metrics.gpzy W1 rebuttal
gpzy W2 — Theoretical scopeClarified the population-level scope of Propositions 1–2 and the empirical evaluation of fitted predictors.Sec. 3.6
gpzy Q1 — Distribution shiftsAdded synthetic label-prior shifts with coefficients fixed. CITP retained lower average nBrier/nNLL than the compared baselines.App. C.8 (Tables 14–15)
gpzy Q2 — Transfer after fittingTested changed prompts and demonstrations without refitting. CITP retained better average scores than the global calibration baseline in all three metrics.gpzy Q2 rebuttal

Summary of CITP

CITP’s affine formulation keeps the roles of global calibration and query-matched prompt corrections explicit while fitting their coefficients jointly through a convex MAP objective and capturing their dependencies in a joint Laplace posterior (Secs. 3.3–3.4). This structure makes their contributions separately testable and lets us derive an improvement criterion and a lower bound on attainable population cross-entropy reduction beyond optimal Scale+Bias (Propositions 1–2). The experiments support this additional value of prompt responses and show further improvements from averaging over the joint posterior (Tables 3–4; Fig. 3). The resulting method improves mean nBrier, nNLL, and accuracy over the evaluated non-CITP baselines in the main setting (Table 2).

Thank you for considering our responses and revised manuscript.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.