acceptodds
Under review as a conference paper at ICLR 2027

Coupled Equivariance Tests Cannot Certify What Action Labels Mean

Abstract

Equivariant outputs guide orientations and actions in downstream programs, but passing a coupled test need not establish that a public label invokes its declared action. Both test branches can execute a 50-degree turn under a label declared to mean 10 degrees. Applying the same group automorphism to the input and expected output transformations reindexes every equivariance residual, preserving the complete residual multiset and reindexing-invariant verdicts. On frozen REKD orientation and MACE force-prediction checkpoints, conventional coupled tests miss all 34 injected shared relabelings but detect all 68 one-sided controls; independent label–state probes detect all 34 shared relabelings. MACE’s mean normalized force equivariance error is 4.85×10⁻¹⁶ even as fixed-label response error increases. In a constructed REKD/HPatches matcher, multiplying rotation labels by five in an optional orientation filter lowers held-out F1@3 px for selecting fixed candidate matches from 0.691 to 0.419 across 24 scenes. A probe-derived boundary correction restores 0.691 without retraining. Coupled consistency alone does not identify the binding needed to use an externally named action.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
equivariance, geometric deep learning, group actions, identifiability, metamorphic testing, image matching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WBrtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2(+1)
Presentation
3(+2)
Soundness
3(+2)
Read review
Paper summary

The paper studies a failure mode where an equivariance test and the tested interface share the same incorrect interpretation of action labels. It shows that such errors can preserve equivariance residuals and proposes independent probes to check the labels against an external convention.

Critical strengths

I appreciate that the authors clearly distinguish injected faults from actual defects in the released systems. The HPatches experiment also gives a concrete example of how an incorrect orientation mapping can affect downstream matching.

Critical weaknesses

I found the paper unnecessarily difficult to follow. It moves between equivariance, action-label semantics, software contracts, and certification without first making the practical task clear. I kept asking what is actually wrong, which component is responsible, and what existing test is supposed to catch it. The many testing tiers and protocol details make these questions harder to follow. I would have understood the motivation much earlier if the paper had started with one complete application example.

Once I understood the setup, I was not convinced that the contribution goes much beyond an interface and oracle problem. The model can remain perfectly equivariant while the interface and its test share the wrong convention. Equivariance is defined relative to the supplied actions; it does not ordinarily promise that those actions match an external declaration. The reindexing argument makes this precise, but I am missing evidence that existing evaluations actually make the stronger inference being challenged. Without that connection, the novelty feels limited.

I also struggled to see what the experiments establish beyond the constructed example. The shared residuals are cached reindexings, so the missed mutations are expected by construction. The HPatches result is more informative, but it shows that deliberately corrupting an orientation decoder hurts a filter and reversing that corruption restores it. The historical e3nn defect is a different failure mode. These results are valid within their stated scope, but they do not yet convince me that the paper has uncovered an important gap in existing practice or offered a substantially better way to address it. The certificate comparison does not resolve this concern either: the simpler baselines achieve the same query counts.

Questions for authors

What is the concrete task here, and what does a practitioner gain from the proposed approach over independently testing the action interface against its specification?

Which existing evaluation practice treats passing equivariance tests as evidence that public action labels have the correct external meaning?

Which experimental result demonstrates something beyond the expected consequences of the injected mapping error? I would particularly like to see what the proposed checks reveal in an existing workflow.

4 comments

Thank you for asking what a direct specification check can tell a practitioner. We now lead with the HPatches filter in §1, trace the decoder and probe in §4.3, and keep the detailed test tiers in Appendix B.1.

Direct specification checks. We added a comparison with a direct known-answer check in §6 and link to it from the witness definition in §2.2. With the same declaration and separating marker positions, either check identifies the binding among all 12 C36 candidates in one query. The independent reference supplies the label meaning that the coupled residual leaves unresolved.

Matching consequences in the original study. The wrong binding changes which candidates survive the filter. The angle adapter can move a 10° deviation from the modal orientation to 50°, across the optional filter’s 30° cutoff. When the marker and angle methods share and correctly implement the binding, the marker-derived inverse restores all 120 original selection masks across 24 held-out scenes. Fixed-candidate F1@3 px returns from 0.419 to 0.691 (original §5 and Table 2, revised §§4.3 and 5).

Which bindings change the filter’s decisions. In a post-review replay on those same pairs, identity and reversal preserve every mask among the 12 invertible C36 bindings. Each of the other ten changes all 120 masks and lowers mean F1 in every scene. Reversal preserves distances from the mode (revised §5 and supplement/postreview/c36/README.md in the ZIP).

What the coupled test sees. In the original checkpoint study, the 34 shared rebindings reorder cached residuals as Proposition 1 predicts. Coupled tests detect all 68 one-sided controls, and independent label–state checks detect all 34 known shared rebindings (revised §5, Table 1). The compiler, generator and greedy baselines each use three queries in total across C36 and O24 (revised Appendix C.1, Table 5).

A released geometric test. In the added AlbumentationsX 2.6.6 replay, all eight oriented-box/corner comparisons passed under the shared flip convention. For the label v, that convention is a horizontal-midline reflection, whereas the public D4 class describes a vertical midline. The released test compares these two computations. Substituting coordinates from the documented geometry exposes exactly the h and v differences (revised §1, Appendix D.1 and public-flip/results.json in the ZIP). Released test · D4 declaration

Thanks for revising the paper and replying carefully. Thanks also for adding the comparison. I will raise my scores for Presentation, Informativeness, and Soundness from 1 to 2. However, I tentatively decide not to change my rating.

I work within “learning on graphs and other geometries & topologies,” which is the primary area the authors selected for this paper. But my main work is on graph learning, and I found the paper’s main example closer to image matching than to graph learning. This is why my confidence score is low. I think the introduction should help a reader from this primary area understand where the problem comes from and why it matters.

I can follow the first example, where a 10° turn becomes a 50° turn. However, the second paragraph then jumps straight to “In our REKD/HPatches matching run.” Neither REKD nor HPatches is introduced or cited there. I looked them up and learned that HPatches (CVPR 2017) is a classic image-matching benchmark and REKD is a method for detecting oriented keypoints. But a few words about what each is would help me follow the example. I had the same difficulty with AlbumentationsX. The paper tells me that two transforms agree, but gives me little context for what people use the library for or why this particular test matters in practice.

To me, papers are supposed to tell a complete story so readers can follow along. Where I'm missing explanations here gets just a few lines. Appendix C.3 dives deep into file paths and SHA-256 hashes. I may regard it as someone wanting to reproduce these experiments needs such detail. As a reader though, I can follow the task now, but I still struggle to see why a direct specification check is not enough here. Sometimes the paper feels like a report written by people who already know the project.

I've struggled to understand the paper's purpose and what it adds. I can follow the observation that an interface and its test can make the same mistake. What I struggle to see is what the proposed method adds in practice when compared to simply checking the interface directly against its specification. In the authors' own comparison, both checks identify the binding in one query. For me, the paper ties together many concepts using case studies. After the revision, I still can't figure out how those cases add up to a full experimental evaluation or what larger contribution they establish.

I want to emphasize that this is my own assessment. If the authors or the AC feel I missed something or judged it incorrectly, I welcome further discussion. In additional, it seems that the authors’ reply came very close to end of the rebuttal period, so I write this response as quickly as I could. If I have overlooked anything or made a serious mistake, please point it out. I will respond as soon as I can.

Thanks for pointing out the missing context. Section 1 now explains what each example uses.

  • REKD finds image keypoints and estimates their orientations.
  • HPatches provides the image pairs for evaluating matching.
  • AlbumentationsX applies geometric transforms to images and their annotations.

This study is about equivariant tests. An equivariant test can pass when both sides use the same wrong angle. With the same reference and observations that distinguish the candidates, a direct specification check and the marker probe each identify the mapping in one query.

Thanks so much for revising the paper again. Now the introduction is clearer and helps me understand this paper better. I’m glad you took my suggestion which further resolves my concern about readability, and I will raise my Presentation score.

But I still remain concerned with how practical the method's contribution is compared against a simple specification check. With ICLR's four-point rating scale, I find it hard to upgrade my overall rating from 2 to 3. For now, I regret that I still intend to keep it unchanged.

I appreciate all of the work you've done to make the experimental comparison more complete and the paper more clear. If after the rebuttal period the AC wants to know more about my assessment, then I'll do my best to explain (as objective as possible) what I support, what I appreciate about your revisions, and where I still have concerns.

Reviewer_t2cZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents a study of the following question: Can equivariance tests be used to establish the public binding of an action label? The key insight is that a shared rebinding of the input transformation and the output oracle can make a system pass the equivariance tests but depart from the public binding of an action label. The paper then proposes the use of a grounded witness, i.e., an independent label-state witness, to check if a public label actually invokes its declared action. The paper also presents an audit of three public systems to support the findings of the study.

Critical strengths
  • The research question is clearly defined, and it seems to be a fundamental question in verifying the semantic correctness of equivariant models.

  • The study is rigorously designed. It includes a formal treatment of the research question and an audit of several systems in different domains

Critical weaknesses
  • The key insight seems to be relatively straightforward, and the findings are largely consistent with expectations. This is probably why the study did not find any actual defect in the three systems they audited.
Questions for authors
  • Would it be possible to find some actual defects in the systems you studied or some other systems?

  • Is there any evidence that developers are not aware of the limitation of equivariance tests and thus rely solely on equivariance tests, i.e., without other kinds of grounded tests?

3 comments

Thank you for asking what released systems and their tests reveal. We added a source replay of a reported robot fault and a check against the documented geometry of a released vision interface (§1, Appendix D.1).

A reported joint-order defect. An Isaac Lab report describes Newton observations and actions mirrored with PhysX joint indices. The reporter observed 0.17–0.42 m simulated discrepancies between mirrored feet. Our post-review replay of a constructed policy and input gives loss 0 with faulty indices versus 1/48 with name-corrected indices. In our replay, both PhysX-reference tests passed on the pre-fix snapshot. The joint-name regression failed on that snapshot and passed on the fixed code. This joint-coordinate fault differs from the theorem’s shared permutation of group labels (revised §1 and Appendix D.1). The ZIP includes the losses in isaaclab/results.json and the regression results in isaaclab/original_regression_results.json. Isaac Lab report

What the released vision test sees. All eight AlbumentationsX 2.6.6 oriented-box/corner comparisons passed our post-review replay. The two computations share a flip convention. For v, both reflect about the horizontal midline. The public D4 class specifies the vertical midline, and lower-level naming uses flip direction. Replacing either path with one based on the documented geometry flagged exactly the h and v comparisons (revised Appendix D.1 and public-flip/results.json in the ZIP). Released test · D4 declaration

The information a coupled check supplies. Shared bijective relabeling permutes complete coupled residuals (original Proposition 1). Distinct signatures from independently specified marker positions identify the binding within the declared family (original Proposition 2, revised §2.2).

Thanks for your response. However, it seems to address only my first question.

Thanks for following up. AlbumentationsX 2.6.6 uses a coupled box-and-corner test and a separate test that compares transformed pixels with fixed expected arrays. The arrays follow the lower-level flip-direction convention. All eight coupled comparisons passed, and checking against the public D4 axis description flagged exactly h and v (Appendix D.1). Fixed-pixel test

Reviewer_LDcxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether coupled equivariant testing can confirm the meaning of action labels in external conventions. The author employs multiple controlled experiments to demonstrate that applying the same set of automorphic re-labeling to both input actions and output references rearranges the test residuals and preserves the group action relationships, thus not altering the determination that is insensitive to re-indexing. For already exposed bindings, within a pre-specified candidate family, independent "label-state" probes can be used to distinguish between different bindings.

Critical strengths
  1. Precisely defined the boundary of a testing capability. Proposition 1 and Corollary 1 explain why even if all coupling residuals are exhausted, it is still impossible to determine a fixed external label binding. The paper distinguishes between the residual reindexing of numerical operators and the preservation of strict group action relationships, and avoids interpreting equivariant consistency as the correctness of physical predictions.
  2. Controlled experiments include meaningful controls and report negative results truthfully. Sharing, inputting one-sided, outputting one-sided, and non-isomorphic interventions help distinguish between expected blind spots and failures of the test without distinction. The HPatches experiment fixes candidate matches, performs pairwise uncertainty analysis by scene, and uses legal renaming and correction to isolate the impact of control boundary conventions. Reporting the applicability failure of Canonical Capsules, the subsequent process of selecting MACE, and the regression of e3nn belong to another category of issues, increasing the transparency of evidence and limiting overinterpretation.
Critical weaknesses
  1. The increment of contributions is limited. The core reindexing equation is directly derived from variable substitution in the residual definition; identifiability and semi-interval conclusions are demonstrated using classical permutation group bases and signature separation. Section 6 and Appendix C.2 have appropriately acknowledged that the certificate algorithm lacks originality, and the precise compiler, like the generator and greedy baseline, also employs three queries. This certificate also relies on complete explicit conventions and specified candidate families.
  2. The actual benefits are primarily validated within the constructed interfaces and fault families. Checkpoint audits inject known mappings, which are then verified by T4 against external conventions; equality sharing of T2 is achieved through caching residual re-indexing. The paper did not identify any naturally occurring cases of this precise fault category; the real-world e3nn flaw clearly falls into another category, and MACE typically directly uses matrices as conventions, without necessarily having the symbol tagging issue introduced in this paper.
Questions for authors
  1. Please explain how the same public binding connects to the two branches of the REKD test, the channel to the angle decoder, and the grid marker probe. During runtime, does the probe observe the actual consumer adapter or directly read the injection mapping? How is candidate family membership established in this deployment? Which behaviors come from upstream interfaces and which are added by the audit framework?
  2. Clarify the existing task results: What does "homography success rate difference is zero" mentioned in Appendix D specifically refer to? How is success defined without re-estimating homography? Please provide existing numerical values and explain what actual information the checking mechanism adds compared to Table 2 without using filtering conditions.
  3. Clarify contribution positioning: Apart from the inherent equality provided by inherited certificate mathematics and cache re-indexing, which theoretical or empirical discovery does the author consider to be a major advancement relative to the referenced independent/identifiable literature? It is best to provide an example to support the claim.
1 comment

Thank you for asking about the probe and downstream geometry. We clarified where the probe observes the binding, what the original geometry measured, and which information the check adds.

How the probe observes the consumer. Revised §4.3 distinguishes the decoder and probe interfaces. Upstream REKD supplies the detector, descriptors, matcher and optional orientation filter. Our added adapter supplies angles and transforms a raster marker. The probe compares its centroid with independent positions for 12 C36 candidates, reading neither the injected mapping nor task images. Controlled injection φ(g)=5g mod 36 establishes family membership. Diagnosis transfers to the decoder when both methods share and correctly implement φ. In the separate checkpoint run, the same mapping by value reindexes the input-transform and output-reference axes of cached residuals as E[φ(g),φ(g)] (§2.1, Eq. (4)). T4 checks the known injection (§2.2 and Appendix B.1).

Original homography estimates and saved replay. Revised §5 reports the original RANSAC counts below and states the 3 px inlier threshold. Success required a valid estimate with mean four-corner error ≤5 px against the supplied homography. Fewer than four matches or an invalid estimate counted as failure. The zero contrast in original Appendix D compares wrong with normal. Six pairs lost success and six gained it across 24 held-out scenes. The offline analysis uses the saved geometry outcomes in supplement/matching/heldout/task_rows.jsonl and the artifact verifier recomputes matching statistics from the masks (revised Appendix D and supplement/scripts/verify_artifact.py in the ZIP).

What correcting the binding recovers. The gain was at match selection. On the same candidates, the probe identifies the wrong binding and restores all 120 intended selection masks without retraining. Against skipping the filter, the corrected binding retains about 33 fewer incorrect candidates per pair. The paired F1 improvement over the wrong binding has 95% scene-stratified interval [0.235, 0.307] (revised §5).

ConditionF1@3 pxCorrect retained matches per pairHomography successes
Probe-corrected / normal0.691148.3392/120
Wrong binding0.41961.8692/120
Unfiltered0.678160.4893/120

For matching, a candidate counts as correct at reprojection error ≤3 px against the supplied homography (revised Table 2 and Appendix B.3).

The fixed-label testing boundary. For the identity model on Z36, relabeling both evaluator actions g→5g preserves zero residuals and group laws, while a caller requesting +1 gets +5. This is the boundary for a fixed declaration. In controlled matching, the analogous change from 10° to 50° crosses the 30° filter window. Moriakov et al. study CycleGAN mapping ambiguity. Here the model stays fixed as both evaluator actions change. Equation (5) extends to approximate models by bounding the gap between fixed-label error and output-action separation on the same prediction by the rebound residual (original §2). In MACE, the rotation matrix can itself specify the action (§4.1).

Classical signature separation identifies a binding when probes give all candidates distinct response sequences (original Proposition 2). In the added comparison, a known-answer check and our probe each identify the C36 binding in one query given the same declaration and separating observations (§6). The compiler, generator and greedy baselines each use three queries in total across C36 and O24 (Appendix C.1, Table 5).

Next test for geometry. Section 7 now proposes holding the geometric estimator fixed while varying view overlap and candidate outlier rates to test when restoring match selection reduces corner error or increases homography success.

Comments on the paper

1 comment

When both branches of an equivariance test use the same one-to-one relabeling, they reorder the full set of discrepancies the test compares, including nonzero ones. For exact actions a group automorphism also preserves the group laws (original Proposition 1 and Corollary 1). In the original cached checkpoint controls, the coupled tests detected the one-sided relabelings and gave the same verdicts under shared ones.

We thank the reviewers for asking what a passing equivariance test establishes about a named action. We revised the abstract and §1 around the HPatches image-matching filter, which keeps matches with consistent orientations. We moved the six test definitions to Appendix B.1. The focus on how these tests can miss label errors motivated the title change from Coupled Equivariance Tests Cannot Certify What Action Labels Mean to Why Coupled Equivariance Tests Can Miss Action-Label Errors.

To address what a practitioner gains over directly checking the interface, we added a comparison using the same declaration and observations that distinguish the candidate label mappings. A known-answer check and the marker probe each identify the binding in one query (§§2.2 and 6).

The original HPatches probe restored all 120 original masks and recovered fixed-candidate F1 from 0.419 to 0.691 (Table 2). The added replay on the same pairs shows that reversal preserves the masks and that the other nonidentity relabelings reduce mean F1 in every scene (§5). Revised §5 reports the original homography success counts and criterion separately from match selection. Section 7 proposes varying view overlap and outlier rates with the estimator held fixed to learn when restored selection improves geometry.

We added two source-level examples (§1, Appendix D.1). Our Isaac Lab replay tests a reported joint-order fault. The released AlbumentationsX box/corner test passes under a shared flip convention, and a comparison with independently specified coordinates reveals the differences from the documented axes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.