- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies record misbinding, where valid infromation is incorrectly linked to a subject. It also introduces a controlled benchmark which seperates subject presence, assignment validtiy and task support. A multi stage pipeline combines record level verification with proposed ATRA, context guard and answer checks. Experiments show ta reduction in unauthorized record use as well as transfer to unseen datasets
Critical strengths
RecordAuth-Diag is a diagnostic dataset which isolates misbindings from other failure types. Work identifies a possibly interesting failure mode, and provides a structured method of detecting and handling such issues. Evaluation framework seperates out different states and helps explainability, evaluating where the system failed.
Critical weaknesses
The distinct contribution is insufficiently clear. The paper combines retrieval, similarity-based verification, rejection rules, and answer checking, but does not clearly establish what new method these components introduce. ATRA’s advantage over the matched Flat-MLP is also encoder-dependent, with no clear improvement on SigLIP2. The diagnostic benchmark is useful, but the broader methodological contribution needs stronger justification.
The presentation makes the work difficult to assess. Dense terminology, undefined acronyms, and frequent changes in context make it difficult to follow the work. This affects more than readability: it is difficult to trace how P/E/S determines each decision and which comparisons support each claim. A worked example and a clearer mapping between conditions, safeguards, and outcomes are needed.
Questions for authors
- Contribution clarification: What is the main new insight beyond combining record matching, rejection, and support checks? How do P/E/S and the controlled interventions differ from existing attribution and entity-resolution formulations?
- Method clarification: What benefit does ATRA provide over the matched Flat-MLP, given the lack of a clear advantage with SigLIP2? Which existing results isolate its contribution from the deterministic guard and answer checks?
- Pipeline clarification: Could one concrete example trace a record through P/E/S, selection, readout, and checking, including the conditions that produce NULL? This would help establish exactly what the system verifies and what remains outside its guarantees.
5 comments
Thank you to the authors for the detailed response and the extra experiments. The worked example and Algorithm 1 help a lot. The Bob/Alice walkthrough makes it much easier to follow how P/E/S leads to each decision and when the system returns NULL. The feedback-path intervention also helps, since it measures ATRA before the guard and answer checks. I hope these changes are included in the main manuscript as well for easier readability.
I still have two main concerns.
First, the gain from ATRA over the matched MLP is small. With SigLIP2, the pipeline gain is 2.14 points on DAVIS and 1.94 on VOT. The response says SigLIP2 features may leave fewer ambiguous matches. This is an assumption, and no experiment supports it. The mean-replacement test shows that the trained ATRA relies on its feedback paths. It does not show that these paths beat a model trained without them, which is the MLP comparison. Replacing group feedback also drops coverage from 65.75% to 1.92%. This looks more like the model being pushed outside its training range than proof that group feedback helps.
Second, much of the safety seems to come from the fixed rules. Before the guards, the ATRA selector still exposes ineligible records in 6.53% / 9.73% of cases. The final unsafe rate is about 1%. The algorithm also requires ATRA to agree with Triad and keeps only the records both select. In the walkthrough, the Kyoto record is simply assumed to be excluded. It does not show how ATRA rejects it, and that is the core of the method.
The benchmark is useful, but the method's new contribution is still unclear to me. I will keep my score.
Further response to eDKk
Thank you for taking the time to read our revision and replies, and for continuing the discussion. Our earlier response did not fully answer your concerns. Your feedback has helped us clarify the contribution itself, which we value beyond the score. We appreciate this and welcome further discussion.
Our reading of your main concern is: does ATRA's learned feedback add value over a simpler trained selector, and does that value exist before the fixed safeguards? We previously used a feedback intervention to explain the mechanism. You are right that this does not replace a trained comparison. We now connect that comparison to pre-guard results and actual record decisions.
1. Does training with feedback help?
We trained a new control without group/bank feedback to the card head. It keeps the same 24 relational features and group/NULL branch. A wider local head matches capacity: 55,415 parameters versus ATRA's 55,427. Both start from scratch with the same five identity-disjoint folds, five seeds, losses and optimizer, and their own validation-calibrated thresholds. There is no DAVIS/VOT tuning.
Both use the same active/image-present masks and Field readout, before Triad intersection and candidate checks. Cells show accuracy / ineligible-record exposure / non-NULL coverage (%); accuracy includes correct NULLs.
| SigLIP2 test set | Retrained no-feedback | Retrained ATRA |
|---|---|---|
| DAVIS, n=560 | 79.11 / 10.18 / 67.86 | 83.04 / 9.29 / 70.36 |
| VOT, n=310 | 83.23 / 10.97 / 72.58 | 85.16 / 7.74 / 70.97 |
On DAVIS, accuracy improves by 3.93 points, with correct non-NULL answers increasing from 316 to 338. On VOT, unsafe answers fall from 34 to 24, while correct non-NULL answers rise from 187 to 192. This directly tests learning with feedback, replacing our earlier conjecture about SigLIP2 ambiguity.
2. Where does the improvement enter the pipeline?
We next return to the original locked ATRA and Flat-MLP models and give both the same Field readout, before Triad intersection and candidate checks. This is a new analysis of their saved predictions, separate from the retraining above. Accuracy includes correct NULLs; the other columns are counts.
| Source | Selector | Accuracy | Correct non-NULL | Unsafe | All non-NULL |
|---|---|---|---|---|---|
| DAVIS | Flat-MLP | 77.68% | 317 | 64 | 393 |
| DAVIS | ATRA | 82.86% | 336 | 52 | 390 |
| VOT | Flat-MLP | 80.32% | 182 | 39 | 226 |
| VOT | ATRA | 85.48% | 193 | 23 | 220 |
ATRA is 5.18/5.16 points ahead before safeguards, with 19/11 more correct non-NULL answers and 12/16 fewer unsafe ones. The final-pipeline gaps are 2.14/1.94 points. Separating these stages shows that ATRA improves the records supplied to the common executor, while the fixed checks further screen its answers.
3. Which mistakes does this avoid?
To replace the assumed Kyoto exclusion with actual decisions, here are two logged cases. Scores come from the representative runs selected by the original ensemble rule; both models choose the correct subject group.
- Rejecting a misassigned record (DAVIS, bear::1). January 8's Topaz image belongs to rhino::1; January 7's Indigo remains valid. ATRA scores Topaz 0.6136, below its 0.9081 threshold, and retains Indigo at 0.9987. MLP retains Topaz (0.9157, threshold 0.5000). The same latest-record rule therefore returns Indigo for ATRA and Topaz for MLP. The checks accept ATRA's correct answer but turn MLP's answer into NULL.
- Keeping valid support (VOT, hand). January 5's Blue Station image is copied into another group, but the original record remains valid. At the same 0.5000 threshold, ATRA retains it (0.9474), while MLP drops it (0.4991). The same readout returns Blue Station for ATRA but the older January 4 Hazel Garden for MLP. Both pass the checks: only ATRA answers correctly.
Across all 56 DAVIS / 31 VOT queries, a new clean-to-one-way-misbinding analysis also finds that 13/8 previously correct queries become wrong or abstained with MLP, versus 4/1 with ATRA, using the same final readout and checks.
Together, training with feedback improves selection; that improvement is visible before fixed safeguards; and it helps reject misleading records while keeping valid support that MLP misses. This is the contribution we want to make clearer. Thank you again for helping us explain it better.
Supplementary Response to Reviewer eDKk
Thank you for the time and effort you have devoted to reviewing our paper. Based on our reading of your review, the central concern about our method is what ATRA's learned architecture contributes and how its contribution can be isolated from matching features, deterministic safeguards, and answer checks. Our initial response explained the set-feedback architecture and compared ATRA with MLP under shared downstream processing. To further identify the effective computation, we have added a direct intervention on the learned group- and bank-feedback paths.
A direct intervention on the learned feedback paths.
We evaluate the trained ATRA selector on all 3,690 held-out RecordAuth-Diag cases, from 369 parents and 87 identities, using both encoders and the original five folds and five seeds. All 24 per-card features, learned local card representations, group/NULL scores, weights, standardizers, thresholds, and latest-record readout are held fixed. We measure selector outputs before the deterministic triad guard and candidate-answer checks.
We replace the case-specific learned group context, bank context, or both at the card-decision head with activation means computed from the original seen training conditions in each fold. This directly tests the role of the current record set's learned representations in individual card decisions. The intact implementation exactly reproduces the saved answers, selected groups, and retained sets for both encoders.
Cells show task accuracy / unauthorized-record exposure / coverage (%), each over all 3,690 cases. Accuracy includes correct NULLs; exposure means that a selected set contains an ineligible record, and coverage means a non-NULL selection.
Bank feedback improves the discrimination of ineligible records. With intact bank feedback, exposure is 6.53%/9.73% for SigLIP2/DINOv2, compared with 24.55%/24.66% after mean replacement, while coverage differs by only 1.06/0.65 percentage points. The intact feedback therefore filters substantially more ineligible records at similar coverage. The exposure reductions are 18.02/14.93 points, with identity-clustered 95% intervals of [16.71, 19.30]/[13.28, 16.51]. This supports the role of bank-specific context in distinguishing applicable records from competing evidence.
Group feedback supports useful record retention. Replacing group feedback reduces coverage from 65.75%/60.62% to 1.92%/8.35%, accompanied by a substantial accuracy decrease. The intact group representation thus helps the trained selector admit records that support correct answers. Together, the two interventions connect learned set context to both useful retention and the filtering of ineligible records.
These results identify a concrete contribution of ATRA's hierarchical design: learned representations of the current group and memory bank feed back into record-level authorization, helping retain useful records and reject ineligible ones. The effect is observable with the same relational inputs and group scores, before downstream safeguards, providing direct evidence that learned set feedback contributes to reliable memory use.