- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies whether a model's answers can be explained using the intended objects in a visual task. The model judges equality between the first two of four selected objects, while spatial rules restrict the full selection. A shortcut using an endpoint can pass every allowed change to one object because exposing it requires changing more than one object. The authors complete the allowed comparisons to find errors that local checks leave unresolved, then study internal interventions and repairs on fixed inputs using controlled models and one adapted vision-language model.
Critical strengths
The claims about the tested inputs are precise and sound. The support test asks whether the recorded answers can be explained using the intended objects alone. Passing it need not mean the answers are correct or show how the network computed them. After a local pass, completing the allowed comparisons and including a correct reference answer for each reached pair of task objects makes consistency equivalent to correctness on that completed set. Untested pairs remain outside this claim. The certificates distinguish contradictions, completed checks and unresolved comparisons, with sharp bounds on what missing queries could establish.
The numerical results are supported by reconstruction from saved outputs, including checks of 81,887 generated answers. Small floating-point differences under a newer runtime changed no counts or decisions. The interventions test both control over an answer and whether that control transfers to mistakes the model makes naturally. Repairs keep inputs fixed, include controls that learn repairs directly and receive confirmation on new images. A secondary check of saved records also finds lower support gaps, a measure of inconsistent answers, for every repair method. All methods correct more mistakes than they introduce. The prespecified comparison favors Joint over Semantic on accuracy and completion errors, and the authors clearly report this negative result. Disclosed differences in capacity and construction cost limit the comparison to the tested repair recipes.
Critical weaknesses
The novelty is mainly the combination and careful evaluation of established ideas. Dependency consistency, constrained testing, invariance audits and representation interventions have substantial prior work, which the authors acknowledge. The selected-input certificates and connected experiments are useful, but the theoretical and methodological increment is modest. A brief comparison with the closest work would make the contribution easier to assess.
The evidence for broader relevance is limited. The rules governing visual selections are imposed experimentally, and the vision-language experiments use one adapted base model. The three repair pipelines share that model. Fresh images establish performance on new inputs within this setting; independent model replication remains open. The authors disclose these limits. An application with naturally required constraints, or another capable adapted model, would strengthen the contribution's generality. These are ways to increase its significance, rather than required repairs to the finite claims.
Questions for authors
Which files and scripts from the original model experiments are available to reviewers? The supplied archive rebuilds statistics from saved outputs. Please distinguish those checks from the files needed to repeat training or inference. Could the main text add a short timeline and consistent study names to distinguish discovery, development and final confirmation? Putting counting units beside denominators, distinguishing support gap from completion errors, and labeling logical bounds separately from statistical intervals would also make the results easier to follow.
9 comments
Response to Reviewer Tn76 (2/5)
W2. Independent model-family replication and naturally required constraints.
Thank you for recommending validation across model families and naturally required constraints. The audit finding replicates across independently adapted VLM families and executable-history constraints.
For cross-family replication, we independently adapt InternVL3-8B and InternVL3-38B and compare their audits with adapted Qwen on the reopened 256-image panel. Each audit counts contradictions among that model's own local passes.
| Adapted VLM | D accuracy (%) | Contradictions / local passes (images) |
|---|---|---|
| Qwen3-VL-30B-A3B | 97.040 | 10 / 234 |
| InternVL3-8B | 97.409 | 6 / 234 |
| InternVL3-38B | 98.300 | 7 / 249 |
InternVL3-38B passes the frozen capability gate with 127/128 Four and 124/128 Pair inputs correct. Together, the two adaptations reproduce the audit finding in a second capable model family at two scales. The detailed intervention and repair analyses use task-adapted Qwen.
In AI2-THOR, the actions and observation interval require executable histories, making legality an environmental condition. We verify the frozen native Qwen model's development gate and find three completion contradictions among 42 locally passing question groups in 11 rooms.
To isolate the effect of this condition, we compare executable and relaxed neighborhoods at the same edit distance across all 29 admitted rooms. The model, question, first image, later-image pools, and legal reference remain fixed. For one correct-local-pass example, we obtain 0% executable-local versus 42.86% relaxed-local detection at six queries. A separate executable history supplies the legal witness.
Relaxation gives two additional development-panel detections, each with a legal-reference witness; one has an incorrect root. The aggregate gain has a descriptive room-bootstrap interval that crosses zero. The separate 18-room panel contains 71 always-correct groups, one consistently wrong group, and no context-dependent flips. We retain all 19 geometry exclusions from the original 48 rooms.
Open-output tests complement these model-family and environment evaluations. We use frozen, unadapted Qwen on 32 GQA images and preserve target-region source pixels. We obtain independent blinded review from two human raters and a separate independent blinded follow-up from a third human rater. Together, these tests extend the evidence to another capable model family, real-image descriptions, and executable histories.
Revised manuscript: We add independent model-family audits in Section 3.3 and Table 4, with adaptation details in Appendix B.3. We extend Section 3.4 and Appendix T with real-image descriptions, independent blinded human review, executable-history tests, and the same-edit-distance control. We add the full-domain environment decomposition in Table 50. We distinguish independent model adaptations from correction trajectories and label panel reuse in Figure 2.
Response to Reviewer Tn76 (3/5)
We thank you for the practical guidance on reproducibility and presentation. We make the work easier to inspect and reproduce through executable materials, a study timeline, and explicit counting units and uncertainty types.
Q1. Available files for statistical reconstruction, inference, and training.
We appreciate your request to distinguish statistical reconstruction from full model execution. The anonymous repository supports statistical reconstruction, checkpoint inference, and executable training.
Our anonymous reproducibility repository separates these tasks into executable paths for statistical reconstruction, checkpoint inference, task adaptation, basis construction, and correction training.
| Purpose | Files and inputs |
|---|---|
| Recompute statistics | statistics/Anonymous_Statistics_Revised.zip, containing original/reproduce_all.py, saved responses, and memberships |
| Run trained checkpoints | checkpoint_inference/run_checkpoint_inference.py, final adapter, correctors, 32 images, and requirements.lock |
| Repeat task adaptation | training/original/train_lora_adapted.py and the original ordered 512-image panel |
| Build intervention bases | Original basis_contract.py and basis-construction inputs |
| Train corrections | Original reader, objective, and Joint modules; 965 natural inputs, 1,536 Pixel inputs, and 1,536 training pairs |
| Check provenance | ARTIFACT_INDEX.md, TRAINING.md, source indices, base-model revision, and hashes |
For training, we provide the portable entry point training/portable/train.py with adapt, basis, and repair modes. We check the download and execution paths as follows.
- Anonymous download. We verify login-free access and a complete download. We match all 3,330 files, including 77 weight files, against the release checksums and pass the downloaded integrity checker.
- Checkpoint replay. We install the isolated environment and extract the archive elsewhere, then reproduce 96/96 responses. We match labels, texts, and first tokens, with zero maximum margin drift.
- Training and restart. We run six real-model GPU cases covering basis fitting, Semantic repair, and Joint-Slim repair for both attributes. We compare eight uninterrupted updates against four updates plus separate-process continuation to step eight. We obtain exact matches in parameters, optimizer, RNG, sample order, losses, and call counts.
- Scoring and generation. We verify the four short repair checkpoints on 64 inputs.
- Task adaptation. We verify the adaptation entry point through a two-update LoRA smoke test.
These checks cover both saved-result reconstruction and executable model inference and training. The artifact index distinguishes execution acceptance from full scientific retraining and pins public base-model revisions and hashes.
Revised manuscript: We add the anonymous repository link to the Reproducibility Statement and distinguish saved-output reconstruction from model-execution materials. We add an inventory of inference, adaptation, basis, and repair files in Appendix U and Table 51, and separate execution acceptance checks from full scientific retraining. We provide the execution paths and verification records in the linked repository's ARTIFACT_INDEX.md and VALIDATION.md.
Response to Reviewer Tn76 (1/5)
We thank you for your overall positive evaluation, careful reading, and suggestions on positioning and validation. We respond with a closest-work comparison, model-family and environment evaluations, and executable reproduction materials. We address W1-W2 below and your Questions For Authors in Part 2.
W1. Clarification of novelty and comparison with the closest work.
We thank you for encouraging a more precise comparison with the closest work. Our contribution links legal reachability to checkable visual-support conclusions before and after repair.
The distinction is between dependence complexity and legal reachability. Our controlled visual construction shows that even endpoint-only dependence can escape every legal one-object context edit. The number of edits needed to reveal a failure can therefore overstate dependence complexity. We replicate the audit finding on capable VLMs and apply the same support checks after repair.
This distinction motivates completing the legal comparisons that a local audit leaves unchecked. Building on dependency consistency, we return revealing counterexamples, completed support checks, and explicit unresolved scope. The closest-work comparison below connects each foundation to the specific question we test.
To test the practical value of completion, we compare query orders with identical outputs and the same checker. At budget 16, we obtain 2.157 versus 1.843 completed repeated-support groups and bound widths of 0.0750 versus 0.0945, at equal mean query use. Completion therefore resolves more support checks per query, alongside comparable error discovery.
Completion also gives a precise correctness criterion. A correct anchor and consistent predictions throughout a selected pair's legal completion establish correctness on that completed set. Untested pairs remain a separate category. This criterion specifies how to interpret local passes and re-audit repair.
Revised manuscript: We revise Section 1 to connect the problem, closest approaches, missing comparison, and contribution. We add the equal-query evaluation in Section 2.4 and Table 2, and a closest-work comparison in Appendix P and Table 42. We distinguish consistency, the correct-anchor certificate, and logical bounds in Section 2.3, and separate semantic control, repair, and re-audit in Sections 4.2 and 5.2-5.4.