- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Building on prior findings that prompt-induced variation can improve predictive performance, they propose CITP (Context-Induced Tilt Posterior), a Bayesian framework that refines a frozen LLM’s predictive distribution using its responses to task-equivalent prompts.
CITP estimates a posterior over refinement coefficients using labeled calibration data and determines its final prediction by averaging predictive distributions over a Laplace approximation to this posterior.
They conduct extensive experiments and analyses across multiple model families and benchmarks to demonstrate the contributions of query-matched prompt responses and posterior marginalization.
Critical strengths
- The problem statement is clearly defined, and the proposed framework is theoretically well formulated and supported by empirical evidence.
- In particular, they provide additional insight into how prompt-induced variation can be used to improve predictive performance.
Critical weaknesses
- Further analysis of CITP’s robustness to different prompt constructions and aggregation strategies would help clarify the sources of its performance gains.
- They should explicitly clarify the extent to which CITP’s theoretical guarantees apply to its practical implementation.
Questions for authors
- How robust are the refinement coefficients to distribution shifts between the calibration and test data?
- Additionally, do CITP’s performance gains persist if the prompts or demonstration set change after fitting?
4 comments
Gentle Reminder
Dear Reviewer gpzy,
Thank you again for your thoughtful review and constructive feedback.
As the author–reviewer discussion period draws to a close, we would like to kindly follow up on our rebuttal response and revised manuscript, where we provided additional analyses of prompt-bank robustness, distribution shifts, and prompt-bank transfer, along with clarifications of the scope of our theoretical results.
We would sincerely appreciate any further feedback or questions you may have.
Sincerely, The Authors
I appreciate the authors’ thoughtful responses to my review and have carefully read their rebuttal.
The authors have fully addressed all of my concerns.
After reviewing the revised manuscript, I have raised my overall rating from 3 to 4.
Thank you for your follow-up and for reviewing our responses and revised manuscript! We are glad that we could address your concerns.
response to Reviewer gpzy
We thank Reviewer gpzy for the careful review and constructive feedback, and for recognizing the value of using prompt-induced variation for predictive refinement. Changes made in response to your comments are shown in green in the revised manuscript (Section 3.6; Appendix C.8).
W1. Robustness to prompt construction and aggregation strategies.
Tables 2–4 compare aggregation strategies and isolate the effects of query-matched contrasts and posterior averaging.
We also tested 12 predefined eight-view banks, removing one perturbation category and balancing replacement views across the others while keeping the reference and forward-pass budget fixed. CITP was refitted using the same 50 calibration labels for each bank.
T/D/O/Fdenote template, demonstration-order, option-permutation, and format variants. Each modified-bank row averages three compositions; results are macro-averaged over 27 model–task pairs across ten seeds (mean ± SD).Across all bank compositions, CITP retained better macro-averaged nBrier, nNLL, and accuracy than Scale+Bias.
W2. Scope of the theoretical guarantees
Thank you for highlighting the need to clarify the scope of our theoretical guarantees.
In response, we clarified in Section 3.6 that Propositions 1–2 establish a residual-alignment criterion and a population-risk improvement bound beyond optimal Scale+Bias, not guarantees for finite-calibration MAP estimates or the Laplace posterior predictive.
Q1. Robustness to distribution shifts between calibration and test data.
Thank you for suggesting this additional analysis.
On five fixed-label tasks, we tested 2×/4× label-prior shifts by reweighting one test class at a time, with coefficients and the prompt bank fixed. This changes empirical class proportions without altering class-conditional distributions.
We average over class shifts using the main aggregation protocol. Entries report mean nBrier/nNLL across class shifts; 1× denotes the original distribution (lower is better).
CITP retained its advantage in average nBrier and nNLL under both the 2× and 4× shifts.
Additionally, for each method and metric, we averaged the largest loss increase over the 4× class shifts within each model–task–seed cell. CITP showed smaller mean maximum increases in nBrier/nNLL (0.1095/0.1208) than Scale+Bias (0.1238/0.1400) and BayesPE (0.1349/0.1846).
We added these results in Appendix C.8.
Q2. Changes to prompts or demonstrations after fitting.
CITP aims to improve a frozen LLM's predictions by learning how to use responses from a fixed bank of task-equivalent prompts. Its refinement coefficients are fitted to the reference predictions and contrasts of that bank. Reusing the fitted correction under such changes without refitting goes beyond the primary aim of this work.
Nevertheless, we evaluated transfer to nine banks without refitting, varying demonstrations, their order, and class-to-letter mappings while preserving templates, format, and eight-view structure.
Entries are query-weighted scores over 27 model–task pairs, reported as mean ± SD across nine target banks (fixed seed-42 fits; accuracy in %).
CITP retained better nBrier, nNLL, and accuracy than Scale+Bias under these bank changes.
We thank the reviewer for these valuable suggestions and welcome any further questions or discussion.