- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Bootstrap Probabilistic Minimum Bayes Risk (B-PMBR) decoding to reduce the computational cost of MBR decoding. Unlike PMBR, which randomly samples candidate–pseudo-reference pairs, B-PMBR uses model log-probabilities to guide pair selection. It explores three sampling strategies: Top-k, Edge-k, and Gumbel-Top-k. The method also applies z-score normalization before ALS-based matrix completion.
The authors evaluate B-PMBR on WMT22 English–German translation using multiple utility functions, evaluation metrics, and reference sizes. The results show that B-PMBR generally outperforms PMBR, especially with small reference sets, while remaining competitive with full MBR. The paper also reports improved cross-metric evaluation scores in some settings and provides ablation studies on proxy selection and normalization.
Critical strengths
1. Simple and Practical Method
The paper addresses a clear computational bottleneck in MBR decoding. The proposed method is simple and does not require training an additional model. It achieves better generation quality than PMBR in many settings under the same utility evaluation budget, especially with small reference sets. This makes it a practical improvement for efficient MBR decoding.
2. Solid Experimental Validation
The paper evaluates multiple utility functions, sampling strategies, and reference sizes. It also includes runtime comparisons, statistical tests, and ablation studies. These experiments provide useful evidence of the method's effectiveness. The paper is also well organized and easy to follow.
Critical weaknesses
1. Unclear Definition and Justification of Informative Pairs
The paper does not clearly define what makes a pair informative for matrix completion. The proposed proxy simply adds the log-probabilities of two sequences. Why should a high proxy score indicate that a pair is more informative? The paper lacks direct evidence supporting this assumption. This weakens the justification for the main method.
2. Sensitivity to Normalization
Figure 4 shows that B-PMBR, especially Top-k and Edge-k, can have much larger reconstruction errors without z-score normalization than PMBR. Why is the proposed sampling method more sensitive to normalization? Does this suggest that proxy-guided sampling introduces additional instability into ALS? This raises concerns about the robustness of the proposed method.
Questions for authors
1. Informativeness of Proxy Scores
How is the informativeness of a candidate–reference pair defined? Why should the sum of two sequence-level log-probabilities identify informative pairs?
2. Sensitivity to Normalization
Figure 4 shows that B-PMBR can have much larger reconstruction errors without normalization, especially for Top-k and Edge-k. What causes this instability? Does proxy-guided sampling introduce additional instability into ALS?
3. Utility Metric Overfitting
The paper claims that B-PMBR mitigates utility metric overfitting. However, the generalized quality score appears to include the optimized utility metric. Can the authors clarify how this score is computed? Does the improvement remain when the optimized metric is excluded?
4. Consistency of Experimental Results
Some reported generalized quality scores differ across tables. For example, with COMET utility and four references, PMBR obtains -11.82 in Table 1 but -1.68 in Appendix Table 4. What causes this difference? Are different normalization procedures or experimental settings used?
5. Overstated Claims
Some claims appear stronger than the reported results. For example, B-PMBR does not consistently outperform PMBR across all configurations. In Table 2, the log-probability proxy is also not always the best or second-best option.
7 comments
Rebuttal by Authors (Part 2)
Thus, we view the utility metric and the model’s log-probability as two imperfect estimates of translation quality that approach it from different directions. Neither is reliable alone, where utility metrics can overfit to that metric bias, while model probability is known to favor inadequate outputs [1]. Because their failure modes differ, combining them is more robust than either. We examine how the two signals relate by computing Kendall’s between each proxy matrix and the ground-truth utility matrix (WMT-22 En→De, averaged over ) (full result in Appendix I of the revised version):
| Proxy | BLEU | chrF | COMET | BLEURT | BERTScore |
|---|---|---|---|---|---|
| Hypothesis-only | 0.23 | 0.26 | 0.32 | 0.25 | 0.25 |
| Reference-only | 0.22 | 0.24 | 0.15 | 0.19 | 0.24 |
| Joint (ours) | 0.33 | 0.37 | 0.35 | 0.33 | 0.36 |
While all proxies correlate positively with every utility metric, the joint proxy consistently aligns best (0.33–0.37), indicating that both sequences in a pair carry quality information (notably, COMET correlates highest under candidate-only probabilities but lowest under pseudo-reference-only probabilities, followed by BLEURT). Crucially, this moderate correlation is what makes the proxy effective. Table 2 shows that when the proxy is the utility metric itself (BLEU proxy–BLEU utility; chrF proxy–chrF utility), Top- selects the highest-utility pairs directly yet yields lower than the log-probability proxy ( vs. and vs. ). Selecting pairs by the metric alone reinforces its biases, whereas the model-based proxy adds an independent view of quality.
Overall, rather than strictly replicating MBR's final decision, B-PMBR bridges both estimators: pairs are pre-selected by model confidence and explicitly scored by the utility metric, ensuring the selected output is strongly supported by both signals. This explains why diverging from MBR's exact selection reduces metric overfitting in Table 1 and Figure 2—B-PMBR scores slightly lower than MBR on the targeted utility metric, but achieves higher generalized quality (), including on unseen evaluators such as XCOMET and COMETKiwi.
W2, Q2: Sensitivity to normalization
Thank you for your comment. We have added the explanation below to Section 3.3 of our revised manuscript.
Because Top- and Edge- only sample the extreme points, the reconstruction error is larger compared to PMBR without normalization. The cause is not proxy guidance itself but the coverage prioritizing influential pairs, leaving some candidates with only one or two observed entries.
ALS then fits those rows from very few noisy scores, and on raw metric scales this fails for two reasons: A candidate with no observations keeps a near-zero latent factor, so ALS predicts a score of about 0, far from the true values on raw metric scales. Scaling all scores by a constant is equivalent to dividing the ridge penalty by . For raw BLEU scores that span tens of points, provides almost no damping, so a sparsely observed row's predictions fall far outside the metric's range, producing the large, irregular spikes in Figure 4. On raw COMET where the score scales around 0.1, the same is effectively much stronger and shrinks predictions toward 0, producing a persistent offset error without spikes instead. Z-score normalization (§3.3) addresses both: a prediction of 0 corresponds to the mean utility, and operates on a common scale across utility functions.
This sensitivity is not specific to proxy-guided sampling. In PMBR with , each candidate receives on average only observations, i.e., 0.5, 1, 2, and 4 at . At small reference sizes, PMBR therefore faces the same sparse-row regime as proxy-guided sampling. This matches the result in Appendix D, where normalization improves PMBR substantially at and the gain narrows at larger . Sparse rows also affect the decision itself, not only the reconstruction. Using BLEU as a utility function, an inflated row can spuriously win the argmax, whereas on COMET, sparsely observed candidates are shrunk out of contention regardless of quality. Proxy-guided sampling encounters this at every step because it deliberately concentrates the budget, which is why it relies more heavily on normalization.
Therefore, we believe that standardization is an integral component of B-PMBR rather than an optional fix, and that it also benefits PMBR when rows are sparsely observed.
Rebuttal by Authors (Part 3)
Q3: Utility metric overfitting
Thank you for catching this error in our text. We have revised the sentence in Lines 232-233 to clarify that “we combined all evaluation metrics into a unified score, excluding the metric used as the target utility function.” We confirm that our code and all reported results already follow this exact formulation, so no experimental findings are affected.
Q4: Consistency of result and calculation of
Thank you for your comment. The raw evaluation scores and experimental settings for PMBR are identical in both tables. The difference in occurs because the global mean () and standard deviation () used for z-score standardization are computed across the specific pool of baselines and variants compared within each respective table.
Table 1 includes additional strong baselines (SVD-MBR and EnsembleMBR), which shifts the pooled mean upward and changes the group standard deviation compared to the baseline pool in Appendix Table 4. Because all methods within a single table share the exact same and , all within-table comparisons remain strictly apples-to-apples. In the revision, we have clarified as a footnote in Section 4.1 that is standardized within each experiment's comparison group.
Q5: Overstated claims
Thank you for your comment. Regarding the target utility metric (Util. Metric), performance cannot be simplified to a strictly "higher-is-better" or "lower-is-better" dynamic. While a high utility score coupled with a low indicates overfitting to the utility function, a lower utility score does not inherently guarantee superior generalized quality. Furthermore, could the reviewer kindly specify which reported results suggest that B-PMBR fails to consistently outperform PMBR? In Table 2, our claims remain well-supported, as the normalized log-probability proxy (Norm. Log-Prob) consistently achieves the top two performances across all evaluated utility functions.
We want to thank you again for your constructive feedback and questions, which have helped us clarify and strengthen our manuscript. We hope our responses have addressed all of your current concerns. If you have further concerns or questions, we remain happy to discuss them with you.
Best,
References
[1] Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation (Eikema & Aziz, COLING 2020)
Rebuttal by Authors (Part 1)
We would like to thank you for acknowledging our simple and practical approach, extensive experiments, and our easy-to-read paper.
To reduce the length of the response to the comment, some of our revisions are incorporated directly into the paper, and we specify the revision location in the corresponding response. To facilitate tracking in the revised manuscript, all changes addressing your comments have been highlighted in Pink and labeled with the corresponding comment number.
In the comment, we only include parts of the results required to answer your questions and comments. We denote our response below based on the number and category of the reviewer’s response, i.e W1 means Weakness no.1 and Q2 means Question no.2.
Thank you for your comment. We have included our detailed discussion in Appendix I of the revised version.
As noted in Lines 138–145, our proxy scorer estimates the joint probability that a candidate and pseudo-reference pair is high-quality according to the generation model. Our goal is not perfect recovery of the full pairwise matrix because MBR only needs the argmax of u^, so we consider a pair informative if observing it improves the estimate of u^ for candidates likely to compete for the argmax. In ALS, each candidate’s latent factor is fitted to that candidate’s observed entries, and with standardized scores, candidates with few observations are shrunk toward the mean utility. Therefore, the sampler determines which candidates have their expected utility estimated from actual utility scores.
Furthermore, to evaluate how well normalized sequence log-probabilities guide estimation on competitive candidates, we measure
At small reference sizes (M≤64), B-PMBR estimates the expected utility of hMBR far more accurately than uniform sampling. Specifically, Top-k achieves the lowest ∣Δu^(hMBR)∣ across all five utility functions before Gumbel-Top-k and PMBR variants overtake it at larger M, selecting the same candidate as MBR in 29–42% of source sentences as shown in the representative table. This low estimation error confirms that hMBR typically carries high model probability, allowing B-PMBR to estimate its expected utility from explicit metric scores rather than latent factor extrapolation, even though B-PMBR has higher global matrix reconstruction error than PMBR and Normed PMBR. As M grows, uniform sampling observes enough entries per row to lower its estimation error, whereas Top-k and Edge-k plateau because unselected candidates remain unobserved.
continued in the next part