- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper poses a question whether information from critic/verifier can be used to guide answering of a question by a student model. They formalize the problem mathematically and derive a bound based on the ratio between the student's confidence on commitment and the best surviving alternative which specifies when the answer should be switched to the best surviving alternative. The derived approach is evaluated across a variety of policies/budgets/models/datasets and is shown to benefit the correctness of the final answer particularly under conflict aware critic elimination policy with temperature adjusted threshold.
Critical strengths
- The paper looks at the problem both theoretically and empirically. Theoretically it establishes mathematical foundation and proves/derives a number of mathematical properties whose correctness I couldn't verify. Empirically it contains a large suite of experiments, settings, models and datasets.
- Presentation is relatively good adjusted to the density of the material. I especially like Figure 1 visualizing the boundary when the answer should be switched which helps better grasp the mathematical takeaways.
Critical weaknesses
- The density of the content of the paper is at the same time a pro and a con in my eyes. It is extremely dense, with a lot of references on supplementary material which is for the lack of better word quite long. I have a feeling that a lot of work went into this paper and that the authors have to compress it into the format of the conference which resulted in text that tries to present as much as possible at the cost of quality of presentation. This raises a concern of the suitability of the venue considering that the manuscript format is clearly not suitable for the quantity of the work.
- Section 4.1 shows that the method does not really benefit the inference, moreover it arguably harms it on average. While I acknowledge that this is for case without temperature, it just means the solution is highly dependent to a new hyper parameter which I would assume would be highly susceptible to temporal shifts during deployment.
Some issues I came across:
- Line 55 mentions a study that is neither cited nor references
- In Eq (1) it is not defined what is as is supposed to have for domain from my understanding and
- I feel Introduction jumps into the solution too fast and while the answers reveal itself further into the paper I kept wondering as I was reading it: "Where does this rule come from? How are options eliminated? If we eliminate options and renormalize eliminating 98 options shouldn't affect the decision." and so on. Meanwhile I feel it lacks examples of applications of these methods.
Questions for authors
- This method can be considered in a very loose way a form of ensembles. Have you at any point considered relationship with ensembles and incorporating other well-known ensemble methods as baselines to put your solution into perspective?
- Line 310 you mentioned that each cell has 2000 items, however you never specify how 2000 items are sampled? Are they the entire dataset? Are items sampled consistently across pairs/methods/seeds in Table 2? Can you be more precise on this?
- Do you have any insights how having critic and student from the same family affects the results in for example Table 2? It would also be useful to see accuracy of no-critic (just plain student without elimination) in Table 2, so it can be seen whether the proposed solution harms the performance.
- Line 355 states "Temperature fitting and evaluation use separate halves". Does this refer to temperature tuning (as a hyper parameter) and how are halves obtained
- I finished reading the paper and while I saw a number of experiments and text I didn't leave with any conclusive feeling. In my opinion the paper reads more like "this is what we have done" rather than "this is what we have found out". Considering "Conclusion" itself is missing and the paper does not make any conclusive remarks, recommendations or guidelines, could the authors summarize main takeaways/remarks in their response/revision?
2 comments
I thank the authors for their detailed response. While I look at the paper more positively after the changes, from my very limited perspective it does yet warrant being above the acceptance threshold, however it is very close to it.
Response to Reviewer 2srW
Dear Reviewer 2srW,
Many thanks for the careful reading, and for telling us candidly that you finished the paper without a clear sense of its conclusion. That is a fair criticism: we did not communicate the paper's main message well.
For us, the main message is this. A verifier's choice of which answer to reject carries evidence beyond the rejection itself, and that evidence changes the best answer only past a simple threshold. When the verifier follows the rule that spares the model's answer -- as our scripted verifiers do -- using that evidence gains 1.7 to 19.2 points at four options, and when it selects uniformly or eagerly, it gains nothing, exactly as the theory predicts. With the LLM critics we tested, however, it does not reliably help, and we observe that they neither follow the rule they are given nor avoid rejecting the correct answer. And when we enforce the rule in code (the new Section 4.6), the evidence gives a real gain in the pair with the strongest critic, but even there it does not beat simply asking the critic for its answer. The revision now says all of this in a short Conclusion (Section 6), and we have made a light pass through the prose to make it less dense.
Regarding Section 4.1, you are right that it shows no benefit, and we intended that as one of the paper's main findings. A new post-hoc analysis, now in Supplement O, is our attempt to explain why. The temperature correction does not rescue the result: under real four-option critics it changes accuracy by −1.3 to +0.9 points (Section 4.4). On your question about line 355, the temperature is a single number per model and dataset, fitted on a random half of the items (seed 7) and evaluated on the other half, so it is not tuned on the items we report -- although this addresses leakage only, and we did not test whether it transfers across distributions.
Regarding Table 2, the 2,000 items are a fixed, seeded draw: ten-option MMLU-Pro questions from twelve categories chosen with the same seed, shuffled once with seed 7, with the first 2,000 used in every pair, so every method within a pair is scored on the same questions and the same rejections. The revised Table 2 adds the baseline you asked for. Accuracy after eight rounds:
Every method is far above the student alone, mostly because the critics often reject the student's initial wrong answer, and the decoder falls below plain masking in three of the five pairs. The last column is from the corrected re-run (see our general comment).
On same-family pairs, Table 1's same-family pairing (Qwen-14B critic with Qwen-7B student) gives −0.6 [−1.4, +0.2], against −0.4 [−1.0, +0.2] for a cross-family critic with the same student. This neither establishes nor rules out a family effect, since the two critics differ in more than family. Supplement L also contains one exploratory, unregistered same-family pair for Table 2 (Llama-8B student, Llama-8B-Instruct critic), which gives −3.8 against masking, in line with the −8.0 and −5.5 of the two cross-family pairs that use the same critic.
On ensembles, the decoder reweights a single model's distribution and needs no further passes of the student, so it is not itself an ensemble, and we did not run a compute-matched comparison against ensemble methods. The closest experiment is in Supplement M, which builds the distribution from sixteen sampled reasoning chains and compares it with re-prompting over eight fresh chains (+0.6 points, [−1.5, +2.8]). It uses a scripted oracle, missed its parse-validity floor, and is not compute-matched, so we regard it as exploratory only.
Finally, on your specific points. The "confidence study" mentioned early in the Introduction is our own experiment in Section 4.5, and the text now says so. In Equation (1), p(S(r)) is the total probability of the surviving options -- the sum of p(y) over y in S(r) -- and it is now defined; if you meant a different symbol, please let us know. The Introduction now explains how options are eliminated and why renormalizing alone cannot change the answer while the model's answer survives, and gives an example application: a fact-checker with access to records who flags one wrong claim in a draft. And we accept that the paper leans heavily on its supplement; we have used the tenth page for the new experiment and the Conclusion, rather than for expanding what was already there.
Thank you again for the careful reading.
All the best,
the Authors