acceptodds
Under review as a conference paper at ICLR 2027

Reading Is Not Verifying: Auditing Relation-Conditioned Evaluators for Selection and Guidance

Abstract

Evaluation scores for relation-conditioned generation may respond to a declared relation yet prefer outputs that violate it. We separate these as relation sensitivity and relation completeness and audit each: one intervention changes the declared relation at fixed output, the other breaks the relation at fixed declaration. In covalent protein–ligand structure prediction, a transferred contact score treats the required bond as a clash and accepts only 1.3% of 4,587 deposited complexes. A public pipeline already ships an exclusion for it; a CASP16 entry demotes predictions failing such checks. Excluding graph-local bonded neighbours raises acceptance to 95.8% while keeping sensitivity to distal clashes, yet withholding the false penalty adds no test that the bond exists: for this capped contact-score family, we prove mask-only exclusion relation-incomplete under large ligand translations. The repaired score prefers a broken bond under displacement on 65 of 272 frozen predictions. Bounded rigid-body search finds bond-violating solutions on 93.0% of tasks under the original score and 30.5% under the repair. Despite that gap, the repair improves controlled guidance, preserving 197 of 200 bonds against 0 of 200 unrepaired. Selection depends on the pool: correct-bond selection rises from 36.5% to 73.9% in mixed six-candidate pools, but a preregistered within-generator test does not establish an effect. Five further panels probe the audit's limits. On HumanEval+ a text-similarity score selects a program with explicit failing-test labels on 36.1% of eligible tasks, and an English–Chinese replication gains 18.5 percentage points in terminology compliance under checker-gated selection. Repairing a mismatch can therefore improve decisions without ensuring the requirement is met, so sensitivity and verification need separate tests before a score drives selection or guidance.

Primary area:
datasets and benchmarks
Keywords:
evaluation metrics, relation-conditioned generation, best-of-n selection, inference-time guidance, metric audit, preregistration, molecular structure prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6NVTOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the gap between evaluation metrics that respond to task requirements and those that actually verify their satisfaction. The authors propose a two-sided auditing approach to identify such failures and examine their impact on candidate selection and optimization. Using covalent protein–ligand structure prediction as the main case study, they demonstrate how flawed evaluation metrics can lead to incorrect decisions and how targeted corrections can improve results. Additional experiments across code generation, knowledge graphs, scene generation, machine translation, and reward models further explore the broader relevance of the problem.

Critical strengths
  1. The paper addresses an important and often overlooked problem: the fact that an evaluation metric can perceive specific task requirements does not mean it can truly verify whether these requirements are satisfied. The authors clearly distinguish this issue into Relation Sensitivity and Relation Completeness, an approach that I find valuable, especially in AI systems that rely on evaluation metrics for model selection and optimization. This problem is practically relevant and may exist widely across many scientific research tasks.

  2. A major strength of this paper is its extensive cases across diverse domains, including covalent protein–ligand structure prediction, code generation, knowledge graphs, scene generation, machine translation, and reward models. These cases demonstrate the potential pitfalls of evaluation metrics from multiple perspectives and strengthen the compellingness of the paper’s core argument. Furthermore, beyond analyzing inherent flaws in evaluation metrics, the authors further investigate how these flaws affect candidate selection and generation optimization. Overall, the experiments are extensive and yield many insightful observations.

Critical weaknesses
  1. While the paper identifies a meaningful and important problem, I feel that the proposed methodology does not offer sufficient technical novelty. The two-sided auditing approach primarily relies on relatively straightforward interventions and independent constraint checking, which are closely related to existing evaluation and verification techniques. Moreover, the proposed approach mainly helps identify problems in evaluation metrics, while correcting these problems still requires task-specific solutions. Overall, although the paper provides valuable insights, I believe its methodological contribution falls somewhat short of the level of technical innovation expected at a top-tier machine learning conference such as ICLR.

  2. Although the paper presents many examples, most only examine specific aspects of the problem rather than the full process: whether evaluation metrics correctly interpret and verify task requirements (Evaluation), whether evaluation failures affect candidate selection (Selection), and whether they further influence generation or optimization (Optimization/Guidance). The covalent protein–ligand case provides the most complete validation using real model predictions, while other domains generally cover only parts of this process or rely on controlled perturbations. Some examples also revisit well-known evaluation issues, such as code similarity not guaranteeing functional correctness. Moreover, even in the main molecular case, the preregistered within-generator experiment and the improvement in overall pose accuracy do not show statistically significant gains. I believe additional comprehensive case studies in other domains, particularly involving realistic AI workflows, would significantly strengthen the paper's contribution and demonstrate the broader practical significance of the identified problem.

Questions for authors

The paper presents many examples across different domains, but the full chain from evaluation failures to their impact on candidate selection and optimization/guidance is mainly demonstrated in the covalent protein–ligand case. Could the authors provide additional case studies in other domains that demonstrate this full chain within realistic AI workflows? I believe such evidence would further strengthen the paper's contributions and better demonstrate the broader practical significance of the identified problem.

4 comments
AuthorsOpenReview ↗

Technical contributions, statistically significant repair gains, and validation beyond molecules

W1. An impossibility result and constructive repair.

We separately test reading and verifying a requirement. For capped contact scores, under the stated translation assumption, we prove that every finite retained pair set admits a bond-violating maximizer.

Our criterion checks active interactions and restoring updates on attainable violation sides. Completion restores omitted interactions, missing penalty sides, or selected backward updates. For the analyzed family, gradient bounds and step size give sufficient weight and margin conditions. The user declares the relation and range.

A hard gate can exclude invalid candidates. Guidance needs a restoring update on violating sides, where this gate has zero gradient.

Our algorithm uses the measurement and noise to choose which clipped coordinates receive a restored backward update. It builds on the straight-through primitive used by BPDA and Phoompho et al. (2026). On 42 DAPS-HDR images, it gains 1.711 dB [0.752, 2.729] over the original solver and 2.513 dB [1.580, 3.435] over a random mask matched in aggregate mean release rate.

W2. Statistical significance within one generator.

In Boltz-2.2.1, a 2×2 intervention on 21 targets × 3 seeds tests the diagnosed one-sided connection potential and interaction-mask gap. Adding the missing lower bound raises bond compliance 39/63 → 63/63: paired gain 38.1 points [20.4, 51.7] with target weighting and 32.9 [12.8, 54.7] with homology-cluster weighting. Restoring excluded interactions reduces structures with a nonbonded cross-chain contact below 2.0 Å 10/63 → 0/63. Adding the lower bound alone also gives 63/63, repairing all 24 stock failures. This mechanism cohort was selected using earlier results.

On the original 292 targets × 25 candidates, completion replay using archived verifier outputs raises bond success 93.13% → 98.46%, gaining 5.32 points [1.37, 10.49] over masked ranking. This replay recovers available passing candidates. The original mask-only bond-compliance test remains inconclusive.

Pure pose accuracy. On the corrected 266-target/68-cluster panel, masked ranking significantly reduces mean ligand RMSD 13.10 → 11.11 Å, paired −1.99 Å [−3.47, −0.68]. The separate mask-only 2 Å success interval crosses zero. We additionally developed a training-free selector combining native confidence with symmetry-aware cross-generator agreement, excluding repeated-seed self-votes. This gated selector raises RMSD<2 Å success from 29.41% to 38.41%, +9.00 points [2.54, 16.25]; mean RMSD is 10.10 Å. Against native ligand pLDDT, success gains 3.17 points [0.52, 6.94]. The ungated selector has similar pose accuracy. These additional-selector results are exploratory, with nominal intervals after retrospective development on the existing panel.

W2/Q1. A full chain beyond molecules.

On 498 WMT segments, each of two models produces 16 sampled outputs and one beam output.

  • Evaluation. Replacing a declared term in the reference breaks compliance in 340/380 cases, at a median chrF cost of 0.085.
  • Selection. Of 472 pools containing a fully term-compliant candidate, the original score loses that available option in 36 pools.
  • Optimization. Minimum Bayes risk decoding over sampled candidates achieves 87.6% terminology compliance with chrF utility and 94.8% with requirement-gated utility, a paired +7.23 points [5.02, 9.64] over segments.

This connects evaluation, selection, and decode-time optimization. The endpoint is exact term compliance, with inflections, synonyms, and overall translation quality requiring separate assessment.

Prospective scene validation. We froze a predictor on 64 development rooms and tested 32 rooms from 32 different buildings, disjoint from development, using published InstructScene weights, four candidates per room, and external collision-based selection.

Twenty-step runs of each arm predict their relation-satisfaction difference at step 200. Prediction MSE is 0.00306, 80% below zero gain (0.01513) and 75% below development-mean gain (0.01246). Both paired intervals exclude zero. All forecasts choose uniform repair, so room-level selection matches that fixed strategy.

Our stepwise policy checks each unmodified update and enables completion when it violates a declared relation. It raises fully compliant returned rooms 24/32 → 31/32, matches uniform repair's compliance and depth in every room, and uses 92.4% fewer repair gradients (486 vs. 6,400). Repair raises mean depth 29.0 → 59.0 mm. Total gradients are 6,886 versus 6,400. Stepwise-only post-hoc single-thread timing, including checks, is 6.40 versus 15.24 seconds (ratio 0.420 [0.233, 0.639]).

This is postprocessing of public-generator outputs. Table 4 will include the WMT evaluation, selection, and decode-time optimization chain.

AuthorsOpenReview ↗

An inference-time method for diagnosing and repairing missing constraint updates

We propose a method that modifies a pretrained generator's guidance objective or an inverse solver's backward update at inference. The inputs are the existing objective and update rule, the declared constraints and, for inverse problems, the measurement and noise level. The method diagnoses where the requirement loses influence, constructs the corresponding repair, and applies it during inference.

1. Diagnose the missing influence in the actual computation.

Represent a requirement by a readout g_j(x) and valid interval [l_j,u_j]. For each attainable violating side, inspect the relevant interaction list and evaluate the guidance terms on supplied violating states. Measure the change in g_j after each term's descent update. Below l_j, restoration requires an increase in g_j. Above u_j, it requires a decrease. This locates an omitted interaction, a missing penalty side, or a backward path blocked by a clip. The numerical probes diagnose the tested states.

2. Construct the repair selected by that diagnosis.

What the diagnosis findsWhat the method changes
A required nonbonded interaction was removed by maskingRestore the eligible interaction set, retaining the declared-bond and local exclusions.
An active term restores only one violable sideAdd a penalty on the uncovered side, such as max(0,l_j+epsilon-g_j(x)) for a missing lower side.
A saturated measurement operation blocks an observation-supported updateRestore the backward derivative only on coordinates flagged by the measurement-and-noise rule below.

For the supported penalty family, the construction uses the constraint-gradient bounds, opposing update and actual step size to choose sufficient weight and margin conditions. Covered sides receive no additional term. The sufficient conditions apply in the stated region under those bounds. The repaired terms then enter the existing sampler's update.

In Boltz, the chain mask removes relevant nonbonded interactions and the connection term has no lower bound. The method therefore restores the eligible interactions and adds the missing lower-side penalty. Our four-arm experiment isolates the effects of these two interventions.

3. Restore the backward update according to the observation.

For a measurement clip normalized to [-1,1], let z be its pre-clip input, y the observation and sigma the measurement-noise standard deviation. Define

m_i = 1{|z_i|>1 and sign(z_i)y_i < 1-3sigma}.

The forward operation remains q(z)=clip(z). At flagged coordinates the backward derivative is replaced by 1. Other coordinates use the original clip derivative. Thus the observation determines where gradient restoration occurs, without using the unknown clean image.

With z=1.2 and sigma=0.05, y=0.70 gives m=1 and restores the update, while y=0.95 gives m=0 and retains the zero derivative. Both forward values are 1. A solver can therefore react differently to contradictory versus saturation-consistent observations at the same reconstructed value.

The rule changes the inference trajectory through the backward computation. On 42 DAPS-HDR images it improves PSNR by 1.711 dB [0.752,2.729] over the original solver and 2.513 dB [1.580,3.435] over a random mask matched in aggregate mean release rate.

Novelty and the machine-learning contribution.

The structural result explains why pair-exclusion repair can leave invalid global optima. For capped contact scores, under the stated translation assumption, every finite retained pair set admits a bond-violating maximizer. This motivates supplying the omitted requirement to the decision objective.

The constructive contribution maps effective-update diagnostics to the intervention: restore the interaction, supply the missing penalty side, or restore selected backward paths. The imaging algorithm makes the last choice using the actual noisy observation at inference. It provides a concrete modification to how a pretrained model is used, with measured reconstruction gains and controlled ablations.

BPDA supplies backward approximations, Phoompho et al. use a straight-through clip in a diffusion shield, and AdaSTE adapts straight-through estimation during binary-network training. Our inference rule conditions coordinate decisions on the measurement and its noise. Standard penalties and the straight-through primitive are established components. Our claims concern the coverage-based construction and observation-conditioned inference rule. BPDA, Phoompho et al., Appendix B.1, AdaSTE.

AuthorsOpenReview ↗

Paired repair effects, four-arm ablations, and native-confidence baselines

On the paired Boltz-2.2.1 cohort, joint repair raises bond compliance from 39/63 to 63/63 and reduces severe nonbonded contacts from 10/63 to 0/63. The four-arm comparison shows which intervention changes each endpoint.

The four-arm experiment isolates two different defects in one generator.

All four arms use the same 21 targets and three seeds in Boltz-2.2.1. The diagnosed defects are an omitted interaction set and a missing lower-bound update. We evaluate declared-bond compliance and severe nonbonded cross-chain contacts separately.

InterventionBond compliantContact below 2.0 Å
Stock39/6310/63
Interaction-set repair41/630/63
Lower-bound repair63/639/63
Both repairs63/630/63

The lower-bound-only ablation fixes all 24 bond failures. It retains the original chain mask, so the omitted nonbonded interactions remain inactive and nine structures still have severe contacts. The interaction-only ablation brings the severe-contact count to zero, while its bond-compliance gain has an interval that crosses zero.

The bond-compliance gain over stock is +38.1 percentage points (95% CI [20.4, 51.7]) with target weighting and +32.9 [12.8, 54.7] with equal weighting of the 13 homology clusters. The cohort was selected using earlier outcomes, so the effect sizes describe those cases. Bond compliance measures geometry targeted by the added penalty, and the contact endpoint tests the separate interaction repair.

An explicit check also adds to native confidence.

On the original frozen six-model panel, ranking by native ligand pLDDT gives 88.49% bond success. Adding the distance check raises it to 95.79%, a paired gain of 7.30 points [2.37, 13.33].

We additionally recovered a separate Boltz-2.2.1 panel of 45 targets with 16 candidates each. A complete local-geometry check, including bond length and adjacent angles, raises success from 77.8% to 91.1% for confidence-head ranking, from 82.2% to 91.1% for ipTM, and from 75.6% to 91.1% for ligand pLDDT. The respective gains are 13.3 [4.4, 24.4], 8.9 [2.2, 17.8], and 15.6 [6.7, 26.7] percentage points.

All three signals therefore miss available candidates meeting the explicit requirement. The check recovers the 41/45 pools containing a passing candidate. It uses predicted coordinates, declared bond atoms and existing pre-2023 reference bands. The calculation requires no target reference pose. These retrospective ranking results measure recovery within the existing candidate pools.

The pose endpoint is evaluated independently of bond compliance.

On the graph-symmetry-corrected 266-target/68-cluster panel, original masked ranking reduces mean ligand RMSD from 13.10 to 11.11 Å, a paired change of −1.99 Å [−3.47, −0.68]. The mean change remains negative when any one cluster is omitted, with point estimates from −2.15 to −1.69 Å. This establishes an average pose-error benefit. The original mask-only RMSD<2 Å success interval still crosses zero.

The additional selector combines native confidence with symmetry-aware agreement between different generators. Repeated seeds of the same generator do not vote for one another. Its decision uses predicted structures and confidence. The gated pipeline raises RMSD<2 Å success from 29.41% to 38.41%, +9.00 points [2.54, 16.25], and gains 3.17 points [0.52, 6.94] over native ligand pLDDT. Every selected pose was independently rescored from its CIF using the corrected graph-mapping evaluator.

Several selector variants were explored on this existing panel, so those additional-selector intervals are nominal and the result is exploratory. The ungated version has similar pose accuracy. We attribute the pose improvement to the complete confidence/agreement selector and evaluate the bond check on its separate relation endpoint.

Repair also improves an image-quality endpoint.

On 42 DAPS-HDR images at 4× gain, the observation-conditioned rule improves PSNR by 1.711 dB [0.752, 2.729] over the original solver. Against random and reversed-criterion masks matched in aggregate mean release rate, the gains are 2.513 [1.580, 3.435] and 5.534 [4.514, 6.520] dB. Per-image and per-step release counts can differ. A separate ten-image comparison gives +6.195 dB [4.785, 7.609] over uniform straight-through, with all ten improving. Those ten images were previously used in other project experiments. These results test task quality as well as the restoration of a backward update in the stated HDR configuration.

The operator-family test uses DAPS with a two-bit quantization operator taken from RED-diff. On 42 images, selected restoration exceeds uniform straight-through by 5.622 dB [5.102, 6.113], with 42/42 improving. This tests another measurement family within the DAPS solver.

AuthorsOpenReview ↗

Linked evaluation, selection, and optimization evidence beyond molecules

The translation experiment provides an evaluation-to-decoding chain on native model outputs. The InstructScene experiment adds a same-start gradient refinement test on generated indoor object arrangements.

1. Translation uses the same 498 annotated segments and native candidate pools.

Appendix A.14 evaluates Helsinki-NLP/opus-mt-en-fr and facebook/nllb-200-distilled-600M. Each produces 16 samples and one beam output per segment, giving 34 candidates. The declared terminology comes from the source annotations. The compliance check consumes the output and those annotations, without consulting chrF or the reference translation.

StageEvidence linked to the original 498-segment cohort
Score evaluationOn the 478 segments with a terminology-compliant human reference, insert that reference as a 35th candidate and rank using the same mean-chrF objective as MBR. A violating model candidate wins over the compliant reference on 9.2% [6.7, 11.9] of segments.
Candidate selectionIn the original 34-candidate pools, 472/498 contain a fully compliant alternative. Ungated MBR chooses compliant outputs on 436 segments, losing 36 available alternatives.
Decode-time optimizationOn those same original pools, gating the expected-chrF utility raises terminology compliance from 87.6% to 94.8%, +7.23 points [5.02, 9.64].

The reference-insertion arm is a supplementary evaluation using the same source segments and model candidates, with one added human candidate. Its protocol was fixed before computing that endpoint, after the main protocol. Selection and gated decoding use the identical original pools. Of these, 26 have no compliant candidate available, which limits what selection can recover.

The reference comparison counts cases where the winner also violates the declared terminology. This distinguishes the measured failure from ordinary variation among valid translations. Gating recovers available compliant candidates by construction. The measured benefit is declared-term compliance. Human translation quality requires a separate assessment. Table 4 will include this evaluation-to-decoding chain.

2. Scene layouts connect candidate scores, selection losses and refinement on the same rooms.

The prospective cohort contains 32 rooms from 32 buildings excluded from the 64-room development set. Released InstructScene weights generate four native candidates per room. Our external AABB collision score selects one candidate, and every refinement arm starts from that exact layout. Requirements are input spatial relations, evaluated with the author's geometric checker.

A subsequent pool audit finds 26 rooms with a fully compliant candidate. In three rooms the collision score strictly favors a violating layout over the best compliant one. The table includes all three.

RoomInitial relations metSelected / best compliant depth, mmCollision-only refinementCompleted refinement
MasterBedroom-868357/841.7 / 188.07/8, 0 mm8/8, 0 mm
SecondBedroom-88232/3232.2 / 299.32/3, 0 mm3/3, 0 mm
Bedroom-414915/63.8 / 385.65/6, 0 mm6/6, 0 mm

The collision objective selects a violating layout, then reaches zero collision while the relation violation persists. From that same layout, completion recovers all declared relations and also reaches zero collision in all three cases.

Across all 32 rooms, full compliance rises from 24/32 to 31/32. A post-hoc paired analysis gives +21.9 percentage points [9.4, 37.5], with seven rooms improving and none worsening on compliance. Stepwise completion matches uniform completion's compliance and depth in every room with 486 versus 6,400 repair gradients. Total gradients are 6,886 versus 6,400. Subsequent single-thread refinement timing is 6.40 versus 15.24 seconds, including checks.

Across the full cohort, mean depth is 29.0 mm for collision-only refinement and 59.0 mm for completion. A post-hoc control uses completion for recovery and then rejects moves breaking compliance. It also achieves 31/32 compliance and averages 89.0 mm. Stepwise completion reduces depth against it by 30.0 mm [8.7, 56.7], while rejection uses about 11% less CPU time.

What these workflows establish about generality.

Translation tests the principle in discrete decoding. InstructScene tests it with continuous refinement that we attach to native generator outputs. Both use task-specific requirements and checkers. Together they show how restoring an explicit requirement changes the decision across these two workflows.

Reviewer_uWctOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2(+1)
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies "relation-conditioned evaluators", which are scores s(Z,E) that take an output Z, a requirement E and return a value corresponding to how well Z satisfies E. The authors argue that there are 2 possible failure modes for these evaluators or scores: the score can fail to read the requirement E correctly, or it can fail to verify the output Z correctly under the fixed requirement E. Then, for protein-ligand complexes, the authors propose a repaired score s2s_2 that (I believe) is supposed to address these failure modes.

Critical strengths
  1. The figures (especially Figure 3) are visually quite nice.
Critical weaknesses
  1. The clarity of the paper is very poor, and the paper is hard to follow. The paper is missing a clear statement of the exact problem formulation. What is the context of the existing work? Why is this problem important, and what is novel about the authors' work? These questions should be addressed a lot more clearly in the manuscript. I've read the whole paper carefully and reread some sections multiple times, and unfortunately I still can't answer these questions easily. Here are a few resources the authors might find helpful on how to structure an ML paper well: https://sebastianfarquhar.com/on-research/2024/11/04/how_to_write_ml_papers/, https://jsteinhardt.stat.berkeley.edu/blog/advice-for-authors, https://www.jakobfoerster.com/how-to-ml-paper. Many sections in the paper are missing important citations and context (for example: abstract, line17, "In covalent protein–ligand structure prediction, a transferred contact score treats the required bond as a clash and accepts only 1.3% of 4,587 deposited complexes." Deposited where? What dataset is this from?; Introduction, line 38, "A validity checker in wide use" what validity checker? It should be cited. Section 4.1 line 272: citations for Boltz models, Chai, and Protenix are missing). The mathematical notation is also poorly introduced: line 51, Z is a set of edges; what is E? What are A and B and later C? Are they also sets of edges? They should all be defined.
  2. The paper's contributions are unclear. Currently, the paper is missing a clear explanation of what the novel contributions of this work are and how it builds on top of existing work. Because of this, it's also quite difficult to assess the validity of the experiments shown in the paper.
Questions for authors
  1. In your own words, could you explain the problem setting this paper tackles?
  2. Could you clearly state the novel contributions of this work?
  3. Could you give an example of how you picture this work being used by the community?
8 comments
AuthorsOpenReview ↗

Q1. What problem are we solving?

We study how a machine-learning system uses a score to choose or improve an output subject to an explicit requirement. A program must pass its tests, a translation must use the requested term, and a structure must contain the specified bond. The model proposes outputs. The score ranks them or guides their modification. A task-specific checker tells us which outputs actually satisfy the requirement. If the pool contains a passing answer and the score selects a failing one, the error is in selection from that pool.

How we test the score. Let Z be an output, E the declared requirement, and s(Z,E) the score, with higher values preferred. To test reading, keep Z fixed and change E. Does the score respond to what the task asks for when the output stays the same? To test verification, keep E fixed and compare outputs that pass and fail an independent check. The score needs to prefer the passing output on that comparison. A tie gives the score no preference for compliance. A strict reversal actively favors the violation. A score can respond to E by changing what it measures while leaving the actual fulfillment of E unrewarded.

Structure prediction tests both properties and follows the score into downstream decisions. The task names a ligand atom and a protein atom that must bond. The predictor generates coordinates. The bond checker measures the distance between the named atoms against a 1.2-2.2 angstrom window. A generic contact score can penalize the short distance as a clash. A declaration-aware mask exempts the specified pair and nearby bonded interactions. On the same 4,587 deposited complexes, acceptance rises from 1.3% to 95.8%. AUROC for detecting introduced clashes away from the bond rises from 0.544 to 0.862. For fixed coordinates, changing the declared pair changes which contacts count.

Keep the declaration fixed and move the ligand. Moving it away can improve the remaining contacts while stretching the required bond. On 65/272 targets, a controlled displacement produces a violating position with a repaired score strictly above every compliant position tested. In frozen pools of unmodified predictions, 44/292 targets have a bonded alternative and every top-scoring candidate breaks the bond. Every bonded alternative scored lower. Using the score during generation has a further consequence. On the same 100 targets with two seeds each, the original objective retains 0/200 bonds, while the masked objective retains 197/200. These tests connect what is scored to which output is selected and to where optimization moves it.

Code makes functional correctness directly testable. On HumanEval+, execution supplies the checker and chrF measures similarity to a reference program. Changing the reference can change which program chrF selects. In frozen pools with a passing option, the selected program has explicit test failures on 53/147 eligible tasks. In a separate controlled test, a small edit creates a program that fails execution tests and scores at least as high as every passing completion on 93/105 tasks. The native pools show the selection error, and the edits isolate it in closely matched programs.

Translation shows what changes when we use the checker in the decision. For English-to-French inputs with required target terms, chrF-based minimum Bayes risk decoding ranks translations by agreement with sampled translations. In 36/472 pools containing a fully term-compliant translation, it selects a violation. For example, it picks froid even though a candidate using the required rhume commun is available. Checking terms before ranking raises exact-term compliance from 87.6% to 94.8% on the same 498 pools. An English-to-Chinese panel shows a +18.5 percentage-point gain on segments with a declared term and a compliant reference. This measures compliance with specified terminology.

Scene layout tests optimization in a different system. A declared placement can require one object to rest on another. A collision score can improve as local search moves an object out of that arrangement. With displacements bounded by 0.25 m per axis on 847 initially compliant ProcTHOR scenes, optimization breaks the placement in 91.4% of cases under the original score and 67.4% after pair exclusion. The remaining failures arise even after the score accounts for declared pairs.

Knowledge graphs and reward models probe different parts of the setting. In link prediction, filtering other known-true answers to the same relational query changes which entities count as competitors. This reverses 7/10 pairwise FB15k-237 model orderings, evidence about how the evaluator reads the task. In an exploratory audit of 427 instruction-following pairs, each of six learned reward models sometimes scores a violating response at least as high as its compliant counterpart. Only 3/6 clear the prespecified threshold for responding to declaration changes.

AuthorsOpenReview ↗

Q2. Our method and its novelty

We contribute a framework that diagnoses how an task requirement loses influence on inference and constructs a repair. The submitted work establishes the score audit and structural result. Discussion-period work extends it to objective and backward-update repairs.

How the method works. The inputs are the requirement, its checker or measurement model, the existing score, and the inference update. It specifies which interactions, penalty sides or backward coordinates to modify.

Run two interventions. Hold the output fixed and change the declaration to test reading. Hold the declaration fixed and compare independently checked satisfying and violating outputs to test verification. For selection, use the checker to exclude violations when a passing option exists.

For guidance at state x, express each requirement as g_j(x) in [l_j,u_j]. Inspect which interactions survive the masks. On supplied violating states, measure how each term's update changes g_j. Below l_j, restoration requires an increase. Above u_j, it requires a decrease. This locates missing interactions, uncovered sides and blocked derivatives. Numerical probes diagnose their tested states.

The diagnosis selects the repair. Restore eligible masked interactions while retaining the declared-bond and local exclusions. If the lower side is uncovered, add a term such as max(0,l_j+epsilon-g_j(x)). The upper-side construction reverses the direction. For the supported differentiable penalty family, constraint-gradient bounds, the opposing update and step size give sufficient weight and inward-margin conditions in the stated region. These terms enter the sampler's update.

For a clipped measurement, use the pre-clip value z_i, observation y_i and noise standard deviation sigma. Select |z_i|>1 and sign(z_i)y_i<1-3sigma. Set the backward derivative to 1 at selected coordinates and use the original derivative elsewhere. The forward value is clip(z_i). With z=1.2 and sigma=0.05, y=0.70 triggers restoration, while y=0.95 retains the zero derivative. The observation selects repair locations.

Where the novelty lies relative to existing methods. Metamorphic testing studies responses to input transformations. Vyas et al. evaluate metrics through task invariants encouraged with an augmented loss. Our two interventions distinguish declaration response from preference for fulfillment, then trace their consequences through selection, search and guidance. Segura et al., Vyas et al..

Our proposition shows that, under the stated translation assumption, any finite retained pair set in the analyzed capped contact-score family admits a bond-violating maximizer. Thus pair exclusion still permits invalid optima. Gao et al. study optimizing learned reward proxies. Our audit also covers deterministic scores with omitted relation checks. Gao et al..

The constructive contribution is the mapping from effective-update coverage to repair location and miss

AuthorsOpenReview ↗

How ML practitioners can use the audit and repair method

We envision two immediate uses for ML teams. A team using a pretrained model and a reranker can audit and repair the selection rule from saved candidates. A team that controls a sampler can use the same requirement to diagnose and repair its inference updates. Both start with an explicit requirement, an independent way to check it, and the score or update that affects the output.

Using the audit on a learned reranker. Consider a team deploying best-of-N decoding for an instruction-following language model. Its reward model scores answers against the prompt. A user asks for exactly three bullet points. The team can build a regression test and use a parser to count bullets. With a three-bullet answer held fixed, change only the instruction from three to four bullets and check whether the reward drops. Repeat in reverse with a four-bullet answer. Then hold the three-bullet prompt fixed and compare answers that organize the same facts into three and four bullets. The parser checks the number requested by each prompt, so the same test can cover different requested counts. A tie or a preference for the four-bullet answer flags a verification failure, even when the reward reacts to the requested count.

Next replay the actual best-of-N decision. For each prompt, record whether any candidate passes and whether the reward selects one that passes. If the reward loses a passing answer, evaluate the checker on the candidates and rank the passing subset with the original reward. For pools without a passing answer, test extra sampling or constrained decoding. Report both the availability of passing candidates and the selection success rate among pools with a passing answer, along with an independent answer-quality measure. The saved prompt-output pairs can be rerun when the reward model or sampling policy changes.

The submission provides two pieces of evidence for this workflow. In the exploratory instruction-following audit on 427 matched pairs, each of six reward models sometimes ties or favors a checked violation. In the terminology decoding experiment, the check plugs into inference on the same 498 candidate pools. It recovers the 36 available compliant translations that chrF-MBR had discarded, raising exact-term compliance from 87.6% to 94.8% (+7.23 percentage points [5.02,9.64]). Both policies use the same outputs.

Using the coverage test to modify a sampler. A researcher adding a requirement to a pretrained diffusion model can apply the audit to saved intermediate states. Specify the quantity to control, g(x), and its acceptable interval [l,u]. Take states on the reachable violating sides and

Reviewer_AMX9OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper separates two properties of a relation-conditioned evaluator and shows they are independent. Reading: does changing the declared requirement change the score. Verifying: at fixed requirement, can a violating output tie or outrank a satisfying one. Main case is covalent protein–ligand structure. The contact score takes a minimum over all protein–ligand atom pairs, and the declared bond is by construction the shortest contact — it or its neighbourhood is the closest contact in 98.4% of deposited structures. A typical S–C bond gives 0.52 against a 0.75 cutoff, so the rule rejects 98.7% of 4,587 deposited complexes. Strict crystallographic filters give the same picture (1.1% against 96.3% on 2,591). The repair is the standard connectivity-based exclusion. Acceptance goes to 95.8%, distal-clash AUROC from 0.544 to 0.862, and it passes the reading test with response rate 0.98. The claim is that it still fails verifying: a bounded rigid-body search reaches a bond-violating optimum on 30.5% of 272 predictions within 0.5 Å, and on 44 of 292 unmodified outputs the repaired argmax contains no bonded candidate although one is in the pool.

Critical strengths

S1. The two properties are orthogonal, and the paper shows it instead of asserting it. The repaired score passes reading (response 0.98) and still fails verifying. The reward models decouple the other way: Tulu 3 at 96.6% / 2.4%, Skywork Llama at 68.1% / 25.3%. A.11 cross-tabulates 2,562 cells — 10.3% have the declaration response and still rank the violation at least as high, 9.7% are the converse. This matters because the field assumes a metric that responds to the task specification is thereby checking it.

S2. Guidance is what makes this urgent. Putting a metric inside the sampling loop turns it from a descriptive measure into an optimization target, and the two roles do not have the same error tolerance. Table 3 Arm 1 shows the cost: 0 of 200 generations keep the declared bond, with the trajectory leaving the valid window by the second guidance call. Steering samplers with physical-plausibility terms of this family is becoming common and I have not seen it audited this way.

S3. The diagnosis is structural, not about one package. Table 38 runs three independent steric primitives on the same 247 complexes. Untyped AUROC on an over-long bond is 0.075, 0.517, 0.532; relation-aware it is 0.817, 0.800, 0.746. The 0.075 is below chance, so that rule prefers the broken bond. A.15.1 screens ten published rules and six share the gap, and documents why: the released interface has no representation for an inter-molecular bond, so a user who knows a pair is bonded cannot say so.

S4. The strongest results use unmodified outputs and undirected search. The 44 of 292 involve no intervention at all, and Table 31 shows 26.00 of 26.08 residual points are avoidable selection regret rather than missing candidates. The rigid-body search prescribes no path and its objective carries no bond information, yet 30.5% fail within 0.5 Å, every optimum outscoring its own start, saturating at 34.9% by 1 Å.

Critical weaknesses

W1. The paper raises the problem without solving it, and it is closer to a solution than it seems to realise. I do not hold the absence of a complete fix against the paper — identifying the gap is the contribution, and the positive-test prescription is the right direction. What I would push on is narrower: the data to turn "mask-only exclusion is incomplete" into "here is when it is enough" is already in hand and is not used. Table 31 breaks the 44 failures down by anchor residue (SER 96.7%, CYS 83.3%, LYS 65.1%) and stops. Ligand size, burial, and the distribution of residual near-threshold eligible contacts are the obvious things to test, and E.1 already hints at the mechanism: the failures split both ways, some bonds stretched past 2.2 Å and some compressed below 1.2 Å, whichever raises the smallest eligible ratio. That is what you would expect if the excluded bond is acting as a cost-free degree of freedom the optimizer spends to relieve a tight distal contact. If the 44 selection failures and the ~35% of tasks reachable under rigid-body search turn out to be the same population, that is both a strong internal check and a criterion practitioners could use. If they are not, that is worth knowing too.

Questions for authors
  1. Can you rank the frozen panel by native confidence (ipTM, ligand pLDDT, the Boltz confidence head) and report whether a bond gate adds anything on top? Section 8 says confidence is unaudited. This is the result that would establish decision relevance for the selection half, and I expect it helps your case.
  2. What separates the 44 failing tasks from the rest, beyond anchor residue? And do they overlap with the ~35% reachable under rigid-body search?
1 comment
AuthorsOpenReview ↗

Bond Verification Beyond Native Confidence and the Mechanisms of Selection and Search Failures

Thank you for suggesting how to use the archived structures and search records. We added the following analyses. Unless stated otherwise, intervals are paired 95% bootstrap intervals over the 74 homology clusters.

W1. When is masking enough?

All 44 strict selection failures contained a bonded alternative. They had 2.80 bonded candidates out of six on average, versus 5.02 in the other 248 pools. We measured ligand size, burial, and residual near-threshold contacts on all 292 deposited structures. After accounting for candidate availability, their additional logistic coefficients had intervals containing zero.

This supports a practical rule for the declared bond. If every candidate passes the bond check, masked ranking suffices. In a mixed pool, check the bond before ranking. With no passing candidate, generation or refinement is required.

On the original panel, a fixed 1.2–2.2 Å gate makes every highest-ranked candidate bond-correct in 40/44 strict selection failures. Across all 292 targets, archived bond success rises 73.92% → 95.79%, a gain of 21.87 percentage points [13.48, 31.02]. Joint pose-and-bond success rises 23.36% → 30.24%, gaining 6.88 points [2.45, 12.26]. In pools with 1–3 and 4–5 bonded candidates, bond-success gains are respectively 65.5 [46.1, 83.9] and 20.2 [9.6, 32.4] points.

Checking in descending masked-score order through the first passing score level, including ties, reproduces the full gate's winners on all 292 pools with 372/1,752 checks. One empty pool uses ungated ranking. The distance gate agrees with the archived bond endpoint on 93.7% of candidates.

Q1. Does the check add to native confidence?

Yes for ligand pLDDT on the original frozen six-model panel. We recovered all 1,752 structures and verified their scoring tables against the submission archive. Ranking by mean ligand-heavy-atom pLDDT, the distance gate improves bond success 88.49% → 95.79%, gaining 7.30 points [2.37, 13.33]. Joint success improves 30.43% → 32.18%, gaining 1.75 [0.00, 4.85]. With model-specific confidence percentiles computed outside each evaluated homology cluster, bond success improves 69.39% → 89.31% (+19.92 [11.94, 28.58]), and joint success gains 6.65 [1.93, 12.29] points.

Comparable ipTM and confidence-head values across all six candidates remain missing from the recovered archive. We also evaluate a separate Boltz-2.2.1 cohort of 45 targets × 16 candidates. Its endpoint checks local covalent geometry, including bond length and adjacent angles. Recomputing the complete check from predicted CIFs, declared bond atoms, and existing pre-2023 reference bands reproduces 720/720 labels. This check requires only the prediction and declaration.

Native ranking signalLocal-geometry success, before → after checkGain, points [95% target-bootstrap CI]
Confidence head77.8% → 91.1%+13.3 [4.4, 24.4]
ipTM82.2% → 91.1%+8.9 [2.2, 17.8]
Ligand pLDDT75.6% → 91.1%+15.6 [6.7, 26.7]

This retrospective comparison recovers the 41/45 pools containing a candidate meeting the explicit local-geometry criterion. Joint-success gain intervals touch zero. The broad distance-only gate gives gains of 0, 0, and 2.2 points on this endpoint, showing the importance of checking the complete geometry.

Q2. Overlap and mechanism.

Of the 44 selection failures, 33 belong to the 272-task initially bond-valid search cohort. At 0.5 Å, 16/33 selection-failure tasks and 67/239 other tasks reach a bond-invalid solution. Jaccard overlap is 0.16. At 1 and 2 Å, search failures rise to 93/272 and 95/272, while overlaps remain 16 and 17 (Jaccard approximately 0.15).

All 83 violations at 0.5 Å improve the masked score, with 75 stretched and 8 compressed bonds. Gated search from the same starts keeps the bond valid in 83/83 cases. At the actual predicted starts, the subsequently violating group already has longer bonds (1.834 vs. 1.604 Å, difference +0.231 Å [0.135, 0.324]). Initial contact-score differences include zero. The shallower burial and looser contacts measured on deposited references describe cohort associations.

Exploratory residue strata further locate the failure stage. LYS pools have 0.93 fewer bonded candidates than CYS pools [−1.57, −0.30]. Among selection failures entering search, ungated search breaks valid bonds in 12/15 CYS and 2/11 LYS cases. Same-start gated search preserves all 14. Across the full search cohort, rates are 71/156 CYS versus 5/31 LYS, a cluster-weighted difference of 35.5 points [9.4, 57.1].

We will place these analyses beside Table 31. They connect candidate availability, the selection check, and the same-start search intervention.

Comments on the paper

1 comment

Thank you for the follow-up. We have revised the paper in two ways: we now explain the existing tools before introducing our changes, and we report a matched comparison against stronger reconstruction baselines.

Background and contribution (Sections 2–3). Section 2 now explains hard filtering, geometric validity checks, and metamorphic testing. Section 3.1 then introduces our two tests. Varying the declared requirement at fixed output checks whether the score uses the requirement; varying the output at fixed requirement checks whether the score rewards satisfaction. For image guidance, Section 2 introduces the measurement loss and the standard straight-through derivative, and contrasts it with related approaches. Section 3 then specifies our change to that baseline: we keep the forward measurement and the identity surrogate, and use a local flatness probe and the observed residual to decide where to apply the surrogate.

Final reconstruction quality (Section 4, Table 2 and Figure 2). On 16 FFHQ images with fresh two-bit observations, all methods use the same observation for each image, and the DAPS arms share initialization and sampling noise. Selective STE improves over Uniform STE on all 16 images on each of PSNR, LPIPS, and SSIM, with mean differences of +5.85 dB, −0.466, and +0.354. Against direct bin inversion, the respective mean differences are +4.57 dB, −0.213, and +0.052; all three paired 95% intervals exclude zero. Against projected bin smoothing, the differences are +2.74 dB and −0.190 LPIPS, while the SSIM interval includes zero. Figure 2 shows the middle two examples ranked by PSNR gain over direct inversion.

What the gate ablation shows (Table 12). At a fixed posterior sign rule, adding back the 3σ3\sigma residual cutoff improves PSNR by 3.44 dB (paired 95% CI [2.45, 4.45]). At a fixed 3σ3\sigma cutoff, adding the posterior sign check changes PSNR by −0.12 dB (95% CI [−0.16, −0.08]). This supports retaining the simpler gate in this setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.