- Rating
- 2
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 3(+2)
- Soundness
- 3(+2)
Read review
Paper summary
The paper studies a failure mode where an equivariance test and the tested interface share the same incorrect interpretation of action labels. It shows that such errors can preserve equivariance residuals and proposes independent probes to check the labels against an external convention.
Critical strengths
I appreciate that the authors clearly distinguish injected faults from actual defects in the released systems. The HPatches experiment also gives a concrete example of how an incorrect orientation mapping can affect downstream matching.
Critical weaknesses
I found the paper unnecessarily difficult to follow. It moves between equivariance, action-label semantics, software contracts, and certification without first making the practical task clear. I kept asking what is actually wrong, which component is responsible, and what existing test is supposed to catch it. The many testing tiers and protocol details make these questions harder to follow. I would have understood the motivation much earlier if the paper had started with one complete application example.
Once I understood the setup, I was not convinced that the contribution goes much beyond an interface and oracle problem. The model can remain perfectly equivariant while the interface and its test share the wrong convention. Equivariance is defined relative to the supplied actions; it does not ordinarily promise that those actions match an external declaration. The reindexing argument makes this precise, but I am missing evidence that existing evaluations actually make the stronger inference being challenged. Without that connection, the novelty feels limited.
I also struggled to see what the experiments establish beyond the constructed example. The shared residuals are cached reindexings, so the missed mutations are expected by construction. The HPatches result is more informative, but it shows that deliberately corrupting an orientation decoder hurts a filter and reversing that corruption restores it. The historical e3nn defect is a different failure mode. These results are valid within their stated scope, but they do not yet convince me that the paper has uncovered an important gap in existing practice or offered a substantially better way to address it. The certificate comparison does not resolve this concern either: the simpler baselines achieve the same query counts.
Questions for authors
What is the concrete task here, and what does a practitioner gain from the proposed approach over independently testing the action interface against its specification?
Which existing evaluation practice treats passing equivariance tests as evidence that public action labels have the correct external meaning?
Which experimental result demonstrates something beyond the expected consequences of the injected mapping error? I would particularly like to see what the proposed checks reveal in an existing workflow.
4 comments
Thanks for revising the paper and replying carefully. Thanks also for adding the comparison. I will raise my scores for Presentation, Informativeness, and Soundness from 1 to 2. However, I tentatively decide not to change my rating.
I work within “learning on graphs and other geometries & topologies,” which is the primary area the authors selected for this paper. But my main work is on graph learning, and I found the paper’s main example closer to image matching than to graph learning. This is why my confidence score is low. I think the introduction should help a reader from this primary area understand where the problem comes from and why it matters.
I can follow the first example, where a 10° turn becomes a 50° turn. However, the second paragraph then jumps straight to “In our REKD/HPatches matching run.” Neither REKD nor HPatches is introduced or cited there. I looked them up and learned that HPatches (CVPR 2017) is a classic image-matching benchmark and REKD is a method for detecting oriented keypoints. But a few words about what each is would help me follow the example. I had the same difficulty with AlbumentationsX. The paper tells me that two transforms agree, but gives me little context for what people use the library for or why this particular test matters in practice.
To me, papers are supposed to tell a complete story so readers can follow along. Where I'm missing explanations here gets just a few lines. Appendix C.3 dives deep into file paths and SHA-256 hashes. I may regard it as someone wanting to reproduce these experiments needs such detail. As a reader though, I can follow the task now, but I still struggle to see why a direct specification check is not enough here. Sometimes the paper feels like a report written by people who already know the project.
I've struggled to understand the paper's purpose and what it adds. I can follow the observation that an interface and its test can make the same mistake. What I struggle to see is what the proposed method adds in practice when compared to simply checking the interface directly against its specification. In the authors' own comparison, both checks identify the binding in one query. For me, the paper ties together many concepts using case studies. After the revision, I still can't figure out how those cases add up to a full experimental evaluation or what larger contribution they establish.
I want to emphasize that this is my own assessment. If the authors or the AC feel I missed something or judged it incorrectly, I welcome further discussion. In additional, it seems that the authors’ reply came very close to end of the rebuttal period, so I write this response as quickly as I could. If I have overlooked anything or made a serious mistake, please point it out. I will respond as soon as I can.
Thanks for pointing out the missing context. Section 1 now explains what each example uses.
- REKD finds image keypoints and estimates their orientations.
- HPatches provides the image pairs for evaluating matching.
- AlbumentationsX applies geometric transforms to images and their annotations.
This study is about equivariant tests. An equivariant test can pass when both sides use the same wrong angle. With the same reference and observations that distinguish the candidates, a direct specification check and the marker probe each identify the mapping in one query.
Thanks so much for revising the paper again. Now the introduction is clearer and helps me understand this paper better. I’m glad you took my suggestion which further resolves my concern about readability, and I will raise my Presentation score.
But I still remain concerned with how practical the method's contribution is compared against a simple specification check. With ICLR's four-point rating scale, I find it hard to upgrade my overall rating from 2 to 3. For now, I regret that I still intend to keep it unchanged.
I appreciate all of the work you've done to make the experimental comparison more complete and the paper more clear. If after the rebuttal period the AC wants to know more about my assessment, then I'll do my best to explain (as objective as possible) what I support, what I appreciate about your revisions, and where I still have concerns.
Thank you for asking what a direct specification check can tell a practitioner. We now lead with the HPatches filter in §1, trace the decoder and probe in §4.3, and keep the detailed test tiers in Appendix B.1.
Direct specification checks. We added a comparison with a direct known-answer check in §6 and link to it from the witness definition in §2.2. With the same declaration and separating marker positions, either check identifies the binding among all 12 C36 candidates in one query. The independent reference supplies the label meaning that the coupled residual leaves unresolved.
Matching consequences in the original study. The wrong binding changes which candidates survive the filter. The angle adapter can move a 10° deviation from the modal orientation to 50°, across the optional filter’s 30° cutoff. When the marker and angle methods share and correctly implement the binding, the marker-derived inverse restores all 120 original selection masks across 24 held-out scenes. Fixed-candidate F1@3 px returns from 0.419 to 0.691 (original §5 and Table 2, revised §§4.3 and 5).
Which bindings change the filter’s decisions. In a post-review replay on those same pairs, identity and reversal preserve every mask among the 12 invertible C36 bindings. Each of the other ten changes all 120 masks and lowers mean F1 in every scene. Reversal preserves distances from the mode (revised §5 and
supplement/postreview/c36/README.mdin the ZIP).What the coupled test sees. In the original checkpoint study, the 34 shared rebindings reorder cached residuals as Proposition 1 predicts. Coupled tests detect all 68 one-sided controls, and independent label–state checks detect all 34 known shared rebindings (revised §5, Table 1). The compiler, generator and greedy baselines each use three queries in total across C36 and O24 (revised Appendix C.1, Table 5).
A released geometric test. In the added AlbumentationsX 2.6.6 replay, all eight oriented-box/corner comparisons passed under the shared flip convention. For the label
v, that convention is a horizontal-midline reflection, whereas the public D4 class describes a vertical midline. The released test compares these two computations. Substituting coordinates from the documented geometry exposes exactly thehandvdifferences (revised §1, Appendix D.1 andpublic-flip/results.jsonin the ZIP). Released test · D4 declaration