- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the gap between evaluation metrics that respond to task requirements and those that actually verify their satisfaction. The authors propose a two-sided auditing approach to identify such failures and examine their impact on candidate selection and optimization. Using covalent protein–ligand structure prediction as the main case study, they demonstrate how flawed evaluation metrics can lead to incorrect decisions and how targeted corrections can improve results. Additional experiments across code generation, knowledge graphs, scene generation, machine translation, and reward models further explore the broader relevance of the problem.
Critical strengths
-
The paper addresses an important and often overlooked problem: the fact that an evaluation metric can perceive specific task requirements does not mean it can truly verify whether these requirements are satisfied. The authors clearly distinguish this issue into Relation Sensitivity and Relation Completeness, an approach that I find valuable, especially in AI systems that rely on evaluation metrics for model selection and optimization. This problem is practically relevant and may exist widely across many scientific research tasks.
-
A major strength of this paper is its extensive cases across diverse domains, including covalent protein–ligand structure prediction, code generation, knowledge graphs, scene generation, machine translation, and reward models. These cases demonstrate the potential pitfalls of evaluation metrics from multiple perspectives and strengthen the compellingness of the paper’s core argument. Furthermore, beyond analyzing inherent flaws in evaluation metrics, the authors further investigate how these flaws affect candidate selection and generation optimization. Overall, the experiments are extensive and yield many insightful observations.
Critical weaknesses
-
While the paper identifies a meaningful and important problem, I feel that the proposed methodology does not offer sufficient technical novelty. The two-sided auditing approach primarily relies on relatively straightforward interventions and independent constraint checking, which are closely related to existing evaluation and verification techniques. Moreover, the proposed approach mainly helps identify problems in evaluation metrics, while correcting these problems still requires task-specific solutions. Overall, although the paper provides valuable insights, I believe its methodological contribution falls somewhat short of the level of technical innovation expected at a top-tier machine learning conference such as ICLR.
-
Although the paper presents many examples, most only examine specific aspects of the problem rather than the full process: whether evaluation metrics correctly interpret and verify task requirements (Evaluation), whether evaluation failures affect candidate selection (Selection), and whether they further influence generation or optimization (Optimization/Guidance). The covalent protein–ligand case provides the most complete validation using real model predictions, while other domains generally cover only parts of this process or rely on controlled perturbations. Some examples also revisit well-known evaluation issues, such as code similarity not guaranteeing functional correctness. Moreover, even in the main molecular case, the preregistered within-generator experiment and the improvement in overall pose accuracy do not show statistically significant gains. I believe additional comprehensive case studies in other domains, particularly involving realistic AI workflows, would significantly strengthen the paper's contribution and demonstrate the broader practical significance of the identified problem.
Questions for authors
The paper presents many examples across different domains, but the full chain from evaluation failures to their impact on candidate selection and optimization/guidance is mainly demonstrated in the covalent protein–ligand case. Could the authors provide additional case studies in other domains that demonstrate this full chain within realistic AI workflows? I believe such evidence would further strengthen the paper's contributions and better demonstrate the broader practical significance of the identified problem.
4 comments
An inference-time method for diagnosing and repairing missing constraint updates
We propose a method that modifies a pretrained generator's guidance objective or an inverse solver's backward update at inference. The inputs are the existing objective and update rule, the declared constraints and, for inverse problems, the measurement and noise level. The method diagnoses where the requirement loses influence, constructs the corresponding repair, and applies it during inference.
1. Diagnose the missing influence in the actual computation.
Represent a requirement by a readout g_j(x) and valid interval [l_j,u_j]. For each attainable violating side, inspect the relevant interaction list and evaluate the guidance terms on supplied violating states. Measure the change in g_j after each term's descent update. Below l_j, restoration requires an increase in g_j. Above u_j, it requires a decrease. This locates an omitted interaction, a missing penalty side, or a backward path blocked by a clip. The numerical probes diagnose the tested states.
2. Construct the repair selected by that diagnosis.
| What the diagnosis finds | What the method changes |
|---|---|
| A required nonbonded interaction was removed by masking | Restore the eligible interaction set, retaining the declared-bond and local exclusions. |
| An active term restores only one violable side | Add a penalty on the uncovered side, such as max(0,l_j+epsilon-g_j(x)) for a missing lower side. |
| A saturated measurement operation blocks an observation-supported update | Restore the backward derivative only on coordinates flagged by the measurement-and-noise rule below. |
For the supported penalty family, the construction uses the constraint-gradient bounds, opposing update and actual step size to choose sufficient weight and margin conditions. Covered sides receive no additional term. The sufficient conditions apply in the stated region under those bounds. The repaired terms then enter the existing sampler's update.
In Boltz, the chain mask removes relevant nonbonded interactions and the connection term has no lower bound. The method therefore restores the eligible interactions and adds the missing lower-side penalty. Our four-arm experiment isolates the effects of these two interventions.
3. Restore the backward update according to the observation.
For a measurement clip normalized to [-1,1], let z be its pre-clip input, y the observation and sigma the measurement-noise standard deviation. Define
m_i = 1{|z_i|>1 and sign(z_i)y_i < 1-3sigma}.
The forward operation remains q(z)=clip(z). At flagged coordinates the backward derivative is replaced by 1. Other coordinates use the original clip derivative. Thus the observation determines where gradient restoration occurs, without using the unknown clean image.
With z=1.2 and sigma=0.05, y=0.70 gives m=1 and restores the update, while y=0.95 gives m=0 and retains the zero derivative. Both forward values are 1. A solver can therefore react differently to contradictory versus saturation-consistent observations at the same reconstructed value.
The rule changes the inference trajectory through the backward computation. On 42 DAPS-HDR images it improves PSNR by 1.711 dB [0.752,2.729] over the original solver and 2.513 dB [1.580,3.435] over a random mask matched in aggregate mean release rate.
Novelty and the machine-learning contribution.
The structural result explains why pair-exclusion repair can leave invalid global optima. For capped contact scores, under the stated translation assumption, every finite retained pair set admits a bond-violating maximizer. This motivates supplying the omitted requirement to the decision objective.
The constructive contribution maps effective-update diagnostics to the intervention: restore the interaction, supply the missing penalty side, or restore selected backward paths. The imaging algorithm makes the last choice using the actual noisy observation at inference. It provides a concrete modification to how a pretrained model is used, with measured reconstruction gains and controlled ablations.
BPDA supplies backward approximations, Phoompho et al. use a straight-through clip in a diffusion shield, and AdaSTE adapts straight-through estimation during binary-network training. Our inference rule conditions coordinate decisions on the measurement and its noise. Standard penalties and the straight-through primitive are established components. Our claims concern the coverage-based construction and observation-conditioned inference rule. BPDA, Phoompho et al., Appendix B.1, AdaSTE.
Paired repair effects, four-arm ablations, and native-confidence baselines
On the paired Boltz-2.2.1 cohort, joint repair raises bond compliance from 39/63 to 63/63 and reduces severe nonbonded contacts from 10/63 to 0/63. The four-arm comparison shows which intervention changes each endpoint.
The four-arm experiment isolates two different defects in one generator.
All four arms use the same 21 targets and three seeds in Boltz-2.2.1. The diagnosed defects are an omitted interaction set and a missing lower-bound update. We evaluate declared-bond compliance and severe nonbonded cross-chain contacts separately.
| Intervention | Bond compliant | Contact below 2.0 Å |
|---|---|---|
| Stock | 39/63 | 10/63 |
| Interaction-set repair | 41/63 | 0/63 |
| Lower-bound repair | 63/63 | 9/63 |
| Both repairs | 63/63 | 0/63 |
The lower-bound-only ablation fixes all 24 bond failures. It retains the original chain mask, so the omitted nonbonded interactions remain inactive and nine structures still have severe contacts. The interaction-only ablation brings the severe-contact count to zero, while its bond-compliance gain has an interval that crosses zero.
The bond-compliance gain over stock is +38.1 percentage points (95% CI [20.4, 51.7]) with target weighting and +32.9 [12.8, 54.7] with equal weighting of the 13 homology clusters. The cohort was selected using earlier outcomes, so the effect sizes describe those cases. Bond compliance measures geometry targeted by the added penalty, and the contact endpoint tests the separate interaction repair.
An explicit check also adds to native confidence.
On the original frozen six-model panel, ranking by native ligand pLDDT gives 88.49% bond success. Adding the distance check raises it to 95.79%, a paired gain of 7.30 points [2.37, 13.33].
We additionally recovered a separate Boltz-2.2.1 panel of 45 targets with 16 candidates each. A complete local-geometry check, including bond length and adjacent angles, raises success from 77.8% to 91.1% for confidence-head ranking, from 82.2% to 91.1% for ipTM, and from 75.6% to 91.1% for ligand pLDDT. The respective gains are 13.3 [4.4, 24.4], 8.9 [2.2, 17.8], and 15.6 [6.7, 26.7] percentage points.
All three signals therefore miss available candidates meeting the explicit requirement. The check recovers the 41/45 pools containing a passing candidate. It uses predicted coordinates, declared bond atoms and existing pre-2023 reference bands. The calculation requires no target reference pose. These retrospective ranking results measure recovery within the existing candidate pools.
The pose endpoint is evaluated independently of bond compliance.
On the graph-symmetry-corrected 266-target/68-cluster panel, original masked ranking reduces mean ligand RMSD from 13.10 to 11.11 Å, a paired change of −1.99 Å [−3.47, −0.68]. The mean change remains negative when any one cluster is omitted, with point estimates from −2.15 to −1.69 Å. This establishes an average pose-error benefit. The original mask-only RMSD<2 Å success interval still crosses zero.
The additional selector combines native confidence with symmetry-aware agreement between different generators. Repeated seeds of the same generator do not vote for one another. Its decision uses predicted structures and confidence. The gated pipeline raises RMSD<2 Å success from 29.41% to 38.41%, +9.00 points [2.54, 16.25], and gains 3.17 points [0.52, 6.94] over native ligand pLDDT. Every selected pose was independently rescored from its CIF using the corrected graph-mapping evaluator.
Several selector variants were explored on this existing panel, so those additional-selector intervals are nominal and the result is exploratory. The ungated version has similar pose accuracy. We attribute the pose improvement to the complete confidence/agreement selector and evaluate the bond check on its separate relation endpoint.
Repair also improves an image-quality endpoint.
On 42 DAPS-HDR images at 4× gain, the observation-conditioned rule improves PSNR by 1.711 dB [0.752, 2.729] over the original solver. Against random and reversed-criterion masks matched in aggregate mean release rate, the gains are 2.513 [1.580, 3.435] and 5.534 [4.514, 6.520] dB. Per-image and per-step release counts can differ. A separate ten-image comparison gives +6.195 dB [4.785, 7.609] over uniform straight-through, with all ten improving. Those ten images were previously used in other project experiments. These results test task quality as well as the restoration of a backward update in the stated HDR configuration.
The operator-family test uses DAPS with a two-bit quantization operator taken from RED-diff. On 42 images, selected restoration exceeds uniform straight-through by 5.622 dB [5.102, 6.113], with 42/42 improving. This tests another measurement family within the DAPS solver.
Linked evaluation, selection, and optimization evidence beyond molecules
The translation experiment provides an evaluation-to-decoding chain on native model outputs. The InstructScene experiment adds a same-start gradient refinement test on generated indoor object arrangements.
1. Translation uses the same 498 annotated segments and native candidate pools.
Appendix A.14 evaluates Helsinki-NLP/opus-mt-en-fr and facebook/nllb-200-distilled-600M. Each produces 16 samples and one beam output per segment, giving 34 candidates. The declared terminology comes from the source annotations. The compliance check consumes the output and those annotations, without consulting chrF or the reference translation.
| Stage | Evidence linked to the original 498-segment cohort |
|---|---|
| Score evaluation | On the 478 segments with a terminology-compliant human reference, insert that reference as a 35th candidate and rank using the same mean-chrF objective as MBR. A violating model candidate wins over the compliant reference on 9.2% [6.7, 11.9] of segments. |
| Candidate selection | In the original 34-candidate pools, 472/498 contain a fully compliant alternative. Ungated MBR chooses compliant outputs on 436 segments, losing 36 available alternatives. |
| Decode-time optimization | On those same original pools, gating the expected-chrF utility raises terminology compliance from 87.6% to 94.8%, +7.23 points [5.02, 9.64]. |
The reference-insertion arm is a supplementary evaluation using the same source segments and model candidates, with one added human candidate. Its protocol was fixed before computing that endpoint, after the main protocol. Selection and gated decoding use the identical original pools. Of these, 26 have no compliant candidate available, which limits what selection can recover.
The reference comparison counts cases where the winner also violates the declared terminology. This distinguishes the measured failure from ordinary variation among valid translations. Gating recovers available compliant candidates by construction. The measured benefit is declared-term compliance. Human translation quality requires a separate assessment. Table 4 will include this evaluation-to-decoding chain.
2. Scene layouts connect candidate scores, selection losses and refinement on the same rooms.
The prospective cohort contains 32 rooms from 32 buildings excluded from the 64-room development set. Released InstructScene weights generate four native candidates per room. Our external AABB collision score selects one candidate, and every refinement arm starts from that exact layout. Requirements are input spatial relations, evaluated with the author's geometric checker.
A subsequent pool audit finds 26 rooms with a fully compliant candidate. In three rooms the collision score strictly favors a violating layout over the best compliant one. The table includes all three.
| Room | Initial relations met | Selected / best compliant depth, mm | Collision-only refinement | Completed refinement |
|---|---|---|---|---|
| MasterBedroom-86835 | 7/8 | 41.7 / 188.0 | 7/8, 0 mm | 8/8, 0 mm |
| SecondBedroom-8823 | 2/3 | 232.2 / 299.3 | 2/3, 0 mm | 3/3, 0 mm |
| Bedroom-41491 | 5/6 | 3.8 / 385.6 | 5/6, 0 mm | 6/6, 0 mm |
The collision objective selects a violating layout, then reaches zero collision while the relation violation persists. From that same layout, completion recovers all declared relations and also reaches zero collision in all three cases.
Across all 32 rooms, full compliance rises from 24/32 to 31/32. A post-hoc paired analysis gives +21.9 percentage points [9.4, 37.5], with seven rooms improving and none worsening on compliance. Stepwise completion matches uniform completion's compliance and depth in every room with 486 versus 6,400 repair gradients. Total gradients are 6,886 versus 6,400. Subsequent single-thread refinement timing is 6.40 versus 15.24 seconds, including checks.
Across the full cohort, mean depth is 29.0 mm for collision-only refinement and 59.0 mm for completion. A post-hoc control uses completion for recovery and then rejects moves breaking compliance. It also achieves 31/32 compliance and averages 89.0 mm. Stepwise completion reduces depth against it by 30.0 mm [8.7, 56.7], while rejection uses about 11% less CPU time.
What these workflows establish about generality.
Translation tests the principle in discrete decoding. InstructScene tests it with continuous refinement that we attach to native generator outputs. Both use task-specific requirements and checkers. Together they show how restoring an explicit requirement changes the decision across these two workflows.
Technical contributions, statistically significant repair gains, and validation beyond molecules
W1. An impossibility result and constructive repair.
We separately test reading and verifying a requirement. For capped contact scores, under the stated translation assumption, we prove that every finite retained pair set admits a bond-violating maximizer.
Our criterion checks active interactions and restoring updates on attainable violation sides. Completion restores omitted interactions, missing penalty sides, or selected backward updates. For the analyzed family, gradient bounds and step size give sufficient weight and margin conditions. The user declares the relation and range.
A hard gate can exclude invalid candidates. Guidance needs a restoring update on violating sides, where this gate has zero gradient.
Our algorithm uses the measurement and noise to choose which clipped coordinates receive a restored backward update. It builds on the straight-through primitive used by BPDA and Phoompho et al. (2026). On 42 DAPS-HDR images, it gains 1.711 dB [0.752, 2.729] over the original solver and 2.513 dB [1.580, 3.435] over a random mask matched in aggregate mean release rate.
W2. Statistical significance within one generator.
In Boltz-2.2.1, a 2×2 intervention on 21 targets × 3 seeds tests the diagnosed one-sided connection potential and interaction-mask gap. Adding the missing lower bound raises bond compliance 39/63 → 63/63: paired gain 38.1 points [20.4, 51.7] with target weighting and 32.9 [12.8, 54.7] with homology-cluster weighting. Restoring excluded interactions reduces structures with a nonbonded cross-chain contact below 2.0 Å 10/63 → 0/63. Adding the lower bound alone also gives 63/63, repairing all 24 stock failures. This mechanism cohort was selected using earlier results.
On the original 292 targets × 25 candidates, completion replay using archived verifier outputs raises bond success 93.13% → 98.46%, gaining 5.32 points [1.37, 10.49] over masked ranking. This replay recovers available passing candidates. The original mask-only bond-compliance test remains inconclusive.
Pure pose accuracy. On the corrected 266-target/68-cluster panel, masked ranking significantly reduces mean ligand RMSD 13.10 → 11.11 Å, paired −1.99 Å [−3.47, −0.68]. The separate mask-only 2 Å success interval crosses zero. We additionally developed a training-free selector combining native confidence with symmetry-aware cross-generator agreement, excluding repeated-seed self-votes. This gated selector raises RMSD<2 Å success from 29.41% to 38.41%, +9.00 points [2.54, 16.25]; mean RMSD is 10.10 Å. Against native ligand pLDDT, success gains 3.17 points [0.52, 6.94]. The ungated selector has similar pose accuracy. These additional-selector results are exploratory, with nominal intervals after retrospective development on the existing panel.
W2/Q1. A full chain beyond molecules.
On 498 WMT segments, each of two models produces 16 sampled outputs and one beam output.
This connects evaluation, selection, and decode-time optimization. The endpoint is exact term compliance, with inflections, synonyms, and overall translation quality requiring separate assessment.
Prospective scene validation. We froze a predictor on 64 development rooms and tested 32 rooms from 32 different buildings, disjoint from development, using published InstructScene weights, four candidates per room, and external collision-based selection.
Twenty-step runs of each arm predict their relation-satisfaction difference at step 200. Prediction MSE is 0.00306, 80% below zero gain (0.01513) and 75% below development-mean gain (0.01246). Both paired intervals exclude zero. All forecasts choose uniform repair, so room-level selection matches that fixed strategy.
Our stepwise policy checks each unmodified update and enables completion when it violates a declared relation. It raises fully compliant returned rooms 24/32 → 31/32, matches uniform repair's compliance and depth in every room, and uses 92.4% fewer repair gradients (486 vs. 6,400). Repair raises mean depth 29.0 → 59.0 mm. Total gradients are 6,886 versus 6,400. Stepwise-only post-hoc single-thread timing, including checks, is 6.40 versus 15.24 seconds (ratio 0.420 [0.233, 0.639]).
This is postprocessing of public-generator outputs. Table 4 will include the WMT evaluation, selection, and decode-time optimization chain.