- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to select reliable structured outputs from multiple candidates generated by a large language model. The method adds a learned quality score to a deterministic constraint penalty, where the quality scorer consists of multiple heterogeneous LoRA members sharing a frozen encoder. The mean energy across members is used for ranking, while the standard deviation is used to estimate uncertainty and further determine whether to append additional generation or reject the answer. Experiments cover five task categories: math, narrative reasoning, travel planning, code generation, and logical reasoning.
Critical strengths
- The method is clear, and it addresses a problem of practical significance. Delegating quality judgments that are hard to verify directly to a learned model, while leaving constraints that can be checked explicitly to a program, is a sound design. A smaller encoder-based scoring model also has the potential to reduce the cost of candidate selection.
- The experiments cover a variety of tasks and are not confined to settings that best suit rule-based checking. On GSM8K and MuSR, the method improves the self-consistency baseline from 96.1% and 64.4% to 97.0% and 67.7%, respectively, showing that the learned scorer can indeed improve candidate selection on some tasks.
- The analysis of the generator-identity shortcut in TACO is quite valuable. Rather than reporting only the relatively high accuracy, the authors further examine the generator selection ratios, the influence of code style, and the changes after retraining the final layer. The paper also reports cases where uncertainty estimation and additional generation fail, and these analyses help delineate the boundaries of the method.
Critical weaknesses
- The proof of Proposition 3.1 does not apply to the mean-energy ranking actually used. Appendix A.1 treats "the majority of members rank correctly" as equivalent to "the mean-energy ranking is correct," but the former depends only on the sign of the energy difference, whereas the latter also depends on its magnitude. For example, if the energy differences of five members between the correct and incorrect candidates are −1, −1, −1, −1, and 10, then four members rank correctly, yet the mean energy still selects the wrong one. The current proof therefore cannot support the paper's claim about the correctness rate of mean-energy ranking.
- The effectiveness of the uncertainty estimation is insufficient to support the paper's central claim. In Table 15, the AUROC of the standard deviation for identifying errors is 0.54, 0.44, and 0.51 on GSM8K, MuSR, and TACO, respectively, which is close to or below chance level. The additional-generation experiments also show no consistent benefit. Moreover, the pairwise ranking loss does not constrain the additive offsets of the individual members' energies, and these offsets affect the standard deviation across members. Although the authors discuss calibration, they still need to explain more fully why this quantity can serve as a reliable uncertainty signal.
- The advantage of the full method over simpler alternatives is not sufficiently clear. On TravelPlanner, the constraint checker alone achieves an average violation score of 0.038, better than the full method's 0.046. In Table 11, single-member and five-member accuracy are also essentially the same on several tasks. This makes it hard for me to judge what necessary benefit energy combination and ensembling each provide. In particular, a direct comparison between homogeneous and heterogeneous LoRA ensembles under the same number of members and the same training budget is missing.
Questions for authors
- Does Proposition 3.1 omit an assumption about the magnitude of the members' energy differences? Please explain the equivalence between mean-energy ranking and majority voting in Appendix A.1. If this step does not hold, which conclusions in the main text would the corrected theory still support?
- Given the same number of members, the same training data, and the same compute budget, does a heterogeneous LoRA ensemble reliably improve error identification or selective prediction over a homogeneous LoRA ensemble? I would also like to see a comparison against a single-scorer confidence baseline, to determine whether the gains indeed come from member disagreement.
- On TravelPlanner, the constraint checker alone outperforms the full method. Does the learned quality term improve aspects not captured by the main metrics, such as the coherence or usefulness of the plans? Could the plans selected by the two methods be directly blind-evaluated, rather than only compared against other models or an oracle?
4 comments
The additional-generation experiments also show no consistent benefit.
We respectfully disagree: regeneration helps where the feedback says what is wrong, which is the case it is designed for (L1561-1564). On TravelPlanner, where the violated constraints become the prompt, the two-pass loop raises pass@1 from 74.0% to 80.0% (Tab. 21; relaxed threshold, L368-369). On TACO, where the failing test is the feedback, it raises pass@1 from 88.6% to 93.2%, at 6 to 67 times lower cost than frontier self-verification (Tab. 22). Where the first pick is already near the oracle, the Distributional EBM keeps it (L1575-1577).
Moreover, the pairwise ranking loss does not constrain the additive offsets of the individual members' energies, and these offsets affect the standard deviation across members.
We agree, and the paper corrects for it (§F.2, L1342-1348). An offset shifts all of a member's energies equally, so it never changes the pick, only . §F.2 therefore centres and unit-scales each member's energies on the validation split before computing . This raises σ-AUROC from 0.54 to 0.58 on GSM8K and from 0.51 to 0.57 on TACO, and pass@1 also rises, by 0.1 and 1.1 pp (Tab. 16).
Although the authors discuss calibration, they still need to explain more fully why this quantity can serve as a reliable uncertainty signal.
We agree that this deserves a fuller explanation; there are three reasons.
First, is uncertainty about quality alone: the constraint term is deterministic and adds no variance (Proposition 3.2). Second, after the offset correction above, it measures on a common scale how much heterogeneous members (Tab. 6) disagree on the selected candidate. Third, it carries information a single scorer does not: on GSM8K it identifies wrong selections better than any member's own confidence (Table R3), and on TravelPlanner and TACO abstaining on the most uncertain selections raises accuracy (Tab. 20).
W3:
- The advantage of the full method over simpler alternatives is not sufficiently clear. On TravelPlanner, the constraint checker alone achieves an average violation score of 0.038, better than the full method's 0.046.
We agree that on TravelPlanner the constraint checker alone is the strongest selector on mean violation: TravelPlanner has an exact checker, which is sufficient for the constraints it checks. The learned scorer is for what such a checker does not measure: on GSM8K, MuSR and TACO it is the whole method and beats every pool heuristic, and on TravelPlanner it selects plans that independent judges prefer (Q3 below).
With no constraint term, the quality scorer beats self-consistency on GSM8K (97.0% against 96.1%), MuSR (67.7% against 64.4%) and TACO (88.6% against 36.4%) (Tab. 1). On TravelPlanner, mean violation counts only the five checked constraints, which the checker optimises by construction; Opus 4.6, judging 120 plans on the original TravelPlanner criteria, rates the Distributional EBM's selections above the oracle's (3.86 against 3.67, L1612-1617), although the oracle plans have the lowest mean violation (0.028). Our answer to Q3 adds the blind comparison of the two methods' plans.
In Table 11, single-member and five-member accuracy are also essentially the same on several tasks. This makes it hard for me to judge what necessary benefit energy combination and ensembling each provide.
The ensemble and the energy combination each have one job, and the paper measures both: the ensemble produces , and the combination adds exact constraints where a task has them.
Ensembling. The members are averaged for ranking, so top-1 accuracy is not what they are for (L1190). With K=1 there is no , and with K=5 supports abstention on TravelPlanner and TACO (Tab. 20). Accuracy holds as members are added (GSM8K 96.9-97.2%, K&K 99.9-100.0%), and on TravelPlanner K≥3 raises zero-violation pass@1 from 24.0% to 28.0% (Tab. 11).
Energy combination. On TravelPlanner, adding the constraint term to the quality scorer lowers mean violation from 0.058 to 0.046 (L371-372), and the combination reaches 99.9% on K&K (Tab. 1). Because the constraint term is deterministic, it does not enter (Proposition 3.2).
W3, Q2:
In particular, a direct comparison between homogeneous and heterogeneous LoRA ensembles under the same number of members and the same training budget is missing. 2. Given the same number of members, the same training data, and the same compute budget, does a heterogeneous LoRA ensemble reliably improve error identification or selective prediction over a homogeneous LoRA ensemble? I would also like to see a comparison against a single-scorer confidence baseline, to determine whether the gains indeed come from member disagreement.
We have added the comparison on MuSR, and the heterogeneous ensemble is the stronger of the two: its members' energy differences are less correlated ( 0.87 against 0.93), which is the quantity the corrected Proposition 3.1 says governs the ensemble, and it selects the correct answer more often (66.9% against 64.9%).
The homogeneous ensemble is also where we started. Our first GSM8K ensembles used five identical adapters (rank 8 on ): every member reached its best validation loss after one epoch and then overfit (validation loss 0.056 rising to 0.211 by epoch 4; the same pattern at a five times lower learning rate), and the members' outputs stayed highly similar (mean pairwise cosine similarity 0.75). We therefore moved to heterogeneous adapters, together with stronger dropout, which is the design in the paper (Tab. 6).
For the matched comparison, the homogeneous ensemble uses five copies of adapter 0 of Tab. 6 (rank 8, alpha 16, ) that differ only in random seed and bagged training split. Everything else matches the heterogeneous ensemble: the same training pairs, validation split, learning rate, batch size, epoch limit and early stopping, and K=5. Both are evaluated on the same 151 MuSR test problems with 32 candidates each.
Same test pool and candidates in both rows (31,730 correct-incorrect pairs). is the -weighted mean correlation of the members' energy differences; adapter-pair correlations range over 0.82-0.91 (heterogeneous) and 0.91-0.96 (homogeneous).
The ordering is the one Proposition 3.1 predicts: lower correlation lets the heterogeneous ensemble rank more pairs correctly than its average member, while the homogeneous ensemble does not improve on its average member. Table R2 compares the two designs on ranking and on the correlation that Proposition 3.1 depends on. For error identification, the comparison we report is our TACO ablation with identical training settings, where the heterogeneous K=5 ensemble reached σ-AUROC 0.75 against 0.36 for a uniform K=3 ensemble. For the single-scorer confidence baseline we use GSM8K, the benchmark with the most problems (1,319). Each method is scored on its own selections: for the ensemble's pick, and each single member's energy margin (best minus second-best candidate) for that member's pick (Table R3). identifies wrong selections better than any of the five single scorers' margins (AUROC 0.54 against 0.30-0.47), and abstaining by raises accuracy at every coverage we test, and at each coverage gives higher accuracy than abstaining by any single scorer's margin (97.4%, 97.3% and 97.2% at 90%, 80% and 60% coverage, against at most 97.0%, 97.0% and 96.7%). The gain in error identification therefore comes from member disagreement, not from a single scorer's confidence.
Each confidence is evaluated on its own method's selections; coverage keeps the most confident fraction of problems. The single members are the five adapters of Tab. 6, each used alone.
Q3: We ran the blind comparison you ask for. On the 40 of 50 TravelPlanner test problems where the constraint checker alone and the Distributional EBM select different plans, Opus 4.6 and Sonnet 4.6 saw both plans in both orders, without method labels, and picked the more coherent and useful plan under the original TravelPlanner criteria. A preference counts only if it holds in both orders; the rest are ties.
Table R4: Blind pairwise comparison on the 40 TravelPlanner test problems where the two methods select different plans (number of problems).
| Distributional EBM's plan | Constraint checker's plan | Tie or order-dependent | |
|---|---|---|---|
| Preferred by Opus 4.6 | 16 | 8 | 16 |
| Preferred by Sonnet 4.6 | 13 | 5 | 22 |
| Lower violation score | 9 | 13 | 18 |
Both judges prefer the Distributional EBM's plan more often, although the checker's plan has the lower violation score more often. The sample is small, so we report this as an indication, not a significant difference. The paper's judge validation (L1612-1617) points the same way: Opus 4.6 rates the Distributional EBM's selections above the oracle's (3.86 against 3.67).
We thank you again. We hope this response addresses your concerns, and that you will consider them in the final assessment. We would be glad to clarify anything further.
Dear Reviewer,
As the discussion period comes to a close soon, we were wondering if you were happy with our detailed responses, corrections and experiments. Kindly let us know if you require any further clarification. We would highly appreciate your feedback.
Many thanks, Authors
Dear Reviewer,
Thank you for your detailed review, for the counterexample to the proof of Proposition 3.1, and for the concrete suggestions on the ensemble and on evaluating TravelPlanner plans. We ran the comparisons you asked for, and answer each point below.
W1, Q1: We agree that the original proof does not apply to mean-energy ranking, and thank you for pointing it out, which is what allowed us to correct it. Please find the corrected Proposition 3.1 above as a global comment, and as a correction in blue in the revised paper (§3.2 and Eq. (2), L193-207; §A.1, L763-799; Tab. 14 and its text, L1267-1289).
Your reading of Appendix A.1 is correct: L771-772 of the submitted version equate a sign count with a mean. The corrected proof starts instead from the exact identity μquality(x,y+)−μquality(x,y−)=dˉ, the average of the adapters' energy differences dk, so the mean ranks a pair correctly if and only if dˉ<0. Magnitude enters through each adapter's mean margin mk and spread sk.
In your example, d=(−1,−1,−1,−1,10), four adapters vote correctly (V=4>K/2=2.5), so the original proof counted the pair as correct; that was the error. The mean gives dˉ=+1.2>0, so the corrected proposition counts the pair as an ensemble error, which is exactly what mean-energy ranking does.
The corrected proposition gives the correctness rate of mean-energy ranking directly: with jointly Gaussian dk, Pensemble=Φ(τmˉ),τ2=Var(dˉ)=K21∑j∑kρjksjsk. Our experiments already rank by the mean energy, so the error is in the proof only and no reported number changes.
Q1, omitted assumption: The corrected Proposition 3.1 needs no assumption on the magnitudes: it models them through each adapter's mean margin mk and spread sk, with the dk jointly Gaussian and a valid correlation matrix (§A.1).
Q1, the equivalence: The corrected §A.1 does not use it. It starts from the exact identity μquality(x,y+)−μquality(x,y−)=dˉ (L769-770), so it describes mean-energy ranking directly.
Q1, what the corrected theory supports: All three conclusions in the main text hold under the corrected proposition. (C1, L204-206) τ increases in every ρjk, so with each adapter's mk and sk held fixed, lower correlation between the adapters' energy differences raises Pensemble; this is the reason for the heterogeneous design. (C2, L206) For adapters of equal strength, ρ=1 gives the single-adapter accuracy Φ(m/s) for every K. (Ceiling, L199-202) For adapters of equal strength and ρ>0, P∞=Φ((m/s)/ρ)<1. The ρ in these statements is now the correlation of the adapters' energy differences, which Tab. 14 reports.
The corrected proposition is also consistent with our experiments. Heterogeneous adapters are less correlated than identical ones (correlation of energy differences 0.87 against 0.93 on MuSR), and, in the direction C1 describes, the heterogeneous ensemble selects the correct answer more often (66.9% against 64.9%) (Table R2). With correlations of 0.845 to 0.941 on the four benchmarks with right-or-wrong labels (Tab. 14), the proposition's pairwise-ranking model implies diminishing returns from adding members, consistent with the limited top-1 changes in Tab. 11.
W2:
We respectfully disagree: the central claim is selection (L71-72), and σquality is claimed as a task-dependent signal (L1308-1310) that raises accuracy where the method acts on it. On TravelPlanner, σ-AUROC for the quality scorer's selection is 0.62, and abstaining on the most uncertain 40% raises zero-violation pass@1 from 28.0% to 36.7%; on TACO, abstaining on 50% raises pass@1 from 88.6% to 90.9% (Tabs. 15 and 20).
On GSM8K the selector makes only about 40 wrong selections in 1,319 problems (L1431-1432), and σquality still finds them better than any single member's confidence in its own selection (Table R3). MuSR is the case the paper reports as a limitation (L435-436).