acceptodds
Under review as a conference paper at ICLR 2027

ANSWERS ARE NOT ROUTES: WHAT LOCAL AUDITS ESTABLISH UNDER VISUAL SUPPORT CONSTRAINTS

Abstract

Vision-language models (VLMs) can answer correctly and pass local tests while still depending on visual context unrelated to the task. When admissible inputs must satisfy constraints, those tests can omit the comparisons that would expose this dependence. We address this problem with a support-auditing procedure built on the Intervenable Causal Route Bottleneck (I-CRB), an interface for controlling object inputs. For each locally tested task pair, the audit completes its legal contexts and checks whether predictions agree when task inputs are unchanged. It returns counterexamples, completed support checks or unresolved comparisons, with finite-domain guarantees and bounds for partial queries. In a constrained four-object task, we show that even a one-object shortcut can evade every legal one-object context change. Evaluating a task-adapted VLM on 256 new images, our audit uncovers previously undetected context dependence in ten of the 234 images that pass all local tests. Donor-free corrections raise image-equal full-domain accuracy from 97.04% to 97.82–98.34%, with net repair in all twelve runs. Re-auditing identifies remaining support violations; matched comparisons do not establish a mechanism-specific repair advantage. Our approach makes hidden context dependence testable and tracks both accuracy and visual support after model updates.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, model auditing, interpretability, counterfactual interventions, visual question answering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Tn76OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies whether a model's answers can be explained using the intended objects in a visual task. The model judges equality between the first two of four selected objects, while spatial rules restrict the full selection. A shortcut using an endpoint can pass every allowed change to one object because exposing it requires changing more than one object. The authors complete the allowed comparisons to find errors that local checks leave unresolved, then study internal interventions and repairs on fixed inputs using controlled models and one adapted vision-language model.

Critical strengths

The claims about the tested inputs are precise and sound. The support test asks whether the recorded answers can be explained using the intended objects alone. Passing it need not mean the answers are correct or show how the network computed them. After a local pass, completing the allowed comparisons and including a correct reference answer for each reached pair of task objects makes consistency equivalent to correctness on that completed set. Untested pairs remain outside this claim. The certificates distinguish contradictions, completed checks and unresolved comparisons, with sharp bounds on what missing queries could establish.

The numerical results are supported by reconstruction from saved outputs, including checks of 81,887 generated answers. Small floating-point differences under a newer runtime changed no counts or decisions. The interventions test both control over an answer and whether that control transfers to mistakes the model makes naturally. Repairs keep inputs fixed, include controls that learn repairs directly and receive confirmation on new images. A secondary check of saved records also finds lower support gaps, a measure of inconsistent answers, for every repair method. All methods correct more mistakes than they introduce. The prespecified comparison favors Joint over Semantic on accuracy and completion errors, and the authors clearly report this negative result. Disclosed differences in capacity and construction cost limit the comparison to the tested repair recipes.

Critical weaknesses

The novelty is mainly the combination and careful evaluation of established ideas. Dependency consistency, constrained testing, invariance audits and representation interventions have substantial prior work, which the authors acknowledge. The selected-input certificates and connected experiments are useful, but the theoretical and methodological increment is modest. A brief comparison with the closest work would make the contribution easier to assess.

The evidence for broader relevance is limited. The rules governing visual selections are imposed experimentally, and the vision-language experiments use one adapted base model. The three repair pipelines share that model. Fresh images establish performance on new inputs within this setting; independent model replication remains open. The authors disclose these limits. An application with naturally required constraints, or another capable adapted model, would strengthen the contribution's generality. These are ways to increase its significance, rather than required repairs to the finite claims.

Questions for authors

Which files and scripts from the original model experiments are available to reviewers? The supplied archive rebuilds statistics from saved outputs. Please distinguish those checks from the files needed to repeat training or inference. Could the main text add a short timeline and consistent study names to distinguish discovery, development and final confirmation? Putting counting units beside denominators, distinguishing support gap from completion errors, and labeling logical bounds separately from statistical intervals would also make the results easier to follow.

9 comments
AuthorsOpenReview ↗

Response to Reviewer Tn76 (1/5)

We thank you for your overall positive evaluation, careful reading, and suggestions on positioning and validation. We respond with a closest-work comparison, model-family and environment evaluations, and executable reproduction materials. We address W1-W2 below and your Questions For Authors in Part 2.


W1. Clarification of novelty and comparison with the closest work.

We thank you for encouraging a more precise comparison with the closest work. Our contribution links legal reachability to checkable visual-support conclusions before and after repair.

The distinction is between dependence complexity and legal reachability. Our controlled visual construction shows that even endpoint-only dependence can escape every legal one-object context edit. The number of edits needed to reveal a failure can therefore overstate dependence complexity. We replicate the audit finding on capable VLMs and apply the same support checks after repair.

This distinction motivates completing the legal comparisons that a local audit leaves unchecked. Building on dependency consistency, we return revealing counterexamples, completed support checks, and explicit unresolved scope. The closest-work comparison below connects each foundation to the specific question we test.

Closest foundationQuestion addressed here
Functional and approximate dependenceWhich visual-support conclusions follow from legal comparisons and correct anchors?
Constrained locating arraysCan simple visual dependence evade local legal comparisons, and which missing comparisons expose it?
Counterfactual VQA and visual-feature supervisionDoes an expanded-domain error contradict task-only support or occur at an untested task pair?
Distributed alignment and representation tuning, including DAS and ReFTHow do semantic answer control, natural-error transfer, and donor-free repair relate on the final intervention spaces?

To test the practical value of completion, we compare query orders with identical outputs and the same checker. At budget 16, we obtain 2.157 versus 1.843 completed repeated-support groups and bound widths of 0.0750 versus 0.0945, at equal mean query use. Completion therefore resolves more support checks per query, alongside comparable error discovery.

Completion also gives a precise correctness criterion. A correct anchor and consistent predictions throughout a selected pair's legal completion establish correctness on that completed set. Untested pairs remain a separate category. This criterion specifies how to interpret local passes and re-audit repair.

Revised manuscript: We revise Section 1 to connect the problem, closest approaches, missing comparison, and contribution. We add the equal-query evaluation in Section 2.4 and Table 2, and a closest-work comparison in Appendix P and Table 42. We distinguish consistency, the correct-anchor certificate, and logical bounds in Section 2.3, and separate semantic control, repair, and re-audit in Sections 4.2 and 5.2-5.4.

AuthorsOpenReview ↗

Response to Reviewer Tn76 (2/5)

W2. Independent model-family replication and naturally required constraints.

Thank you for recommending validation across model families and naturally required constraints. The audit finding replicates across independently adapted VLM families and executable-history constraints.

For cross-family replication, we independently adapt InternVL3-8B and InternVL3-38B and compare their audits with adapted Qwen on the reopened 256-image panel. Each audit counts contradictions among that model's own local passes.

Adapted VLMD accuracy (%)Contradictions / local passes (images)
Qwen3-VL-30B-A3B97.04010 / 234
InternVL3-8B97.4096 / 234
InternVL3-38B98.3007 / 249

InternVL3-38B passes the frozen capability gate with 127/128 Four and 124/128 Pair inputs correct. Together, the two adaptations reproduce the audit finding in a second capable model family at two scales. The detailed intervention and repair analyses use task-adapted Qwen.

In AI2-THOR, the actions and observation interval require executable histories, making legality an environmental condition. We verify the frozen native Qwen model's development gate and find three completion contradictions among 42 locally passing question groups in 11 rooms.

To isolate the effect of this condition, we compare executable and relaxed neighborhoods at the same edit distance across all 29 admitted rooms. The model, question, first image, later-image pools, and legal reference remain fixed. For one correct-local-pass example, we obtain 0% executable-local versus 42.86% relaxed-local detection at six queries. A separate executable history supplies the legal witness.

Relaxation gives two additional development-panel detections, each with a legal-reference witness; one has an incorrect root. The aggregate gain has a descriptive room-bootstrap interval that crosses zero. The separate 18-room panel contains 71 always-correct groups, one consistently wrong group, and no context-dependent flips. We retain all 19 geometry exclusions from the original 48 rooms.

Open-output tests complement these model-family and environment evaluations. We use frozen, unadapted Qwen on 32 GQA images and preserve target-region source pixels. We obtain independent blinded review from two human raters and a separate independent blinded follow-up from a third human rater. Together, these tests extend the evidence to another capable model family, real-image descriptions, and executable histories.

Revised manuscript: We add independent model-family audits in Section 3.3 and Table 4, with adaptation details in Appendix B.3. We extend Section 3.4 and Appendix T with real-image descriptions, independent blinded human review, executable-history tests, and the same-edit-distance control. We add the full-domain environment decomposition in Table 50. We distinguish independent model adaptations from correction trajectories and label panel reuse in Figure 2.

AuthorsOpenReview ↗

Response to Reviewer Tn76 (3/5)

We thank you for the practical guidance on reproducibility and presentation. We make the work easier to inspect and reproduce through executable materials, a study timeline, and explicit counting units and uncertainty types.


Q1. Available files for statistical reconstruction, inference, and training.

We appreciate your request to distinguish statistical reconstruction from full model execution. The anonymous repository supports statistical reconstruction, checkpoint inference, and executable training.

Our anonymous reproducibility repository separates these tasks into executable paths for statistical reconstruction, checkpoint inference, task adaptation, basis construction, and correction training.

PurposeFiles and inputs
Recompute statisticsstatistics/Anonymous_Statistics_Revised.zip, containing original/reproduce_all.py, saved responses, and memberships
Run trained checkpointscheckpoint_inference/run_checkpoint_inference.py, final adapter, correctors, 32 images, and requirements.lock
Repeat task adaptationtraining/original/train_lora_adapted.py and the original ordered 512-image panel
Build intervention basesOriginal basis_contract.py and basis-construction inputs
Train correctionsOriginal reader, objective, and Joint modules; 965 natural inputs, 1,536 Pixel inputs, and 1,536 training pairs
Check provenanceARTIFACT_INDEX.md, TRAINING.md, source indices, base-model revision, and hashes

For training, we provide the portable entry point training/portable/train.py with adapt, basis, and repair modes. We check the download and execution paths as follows.

  • Anonymous download. We verify login-free access and a complete download. We match all 3,330 files, including 77 weight files, against the release checksums and pass the downloaded integrity checker.
  • Checkpoint replay. We install the isolated environment and extract the archive elsewhere, then reproduce 96/96 responses. We match labels, texts, and first tokens, with zero maximum margin drift.
  • Training and restart. We run six real-model GPU cases covering basis fitting, Semantic repair, and Joint-Slim repair for both attributes. We compare eight uninterrupted updates against four updates plus separate-process continuation to step eight. We obtain exact matches in parameters, optimizer, RNG, sample order, losses, and call counts.
  • Scoring and generation. We verify the four short repair checkpoints on 64 inputs.
  • Task adaptation. We verify the adaptation entry point through a two-update LoRA smoke test.

These checks cover both saved-result reconstruction and executable model inference and training. The artifact index distinguishes execution acceptance from full scientific retraining and pins public base-model revisions and hashes.

Revised manuscript: We add the anonymous repository link to the Reproducibility Statement and distinguish saved-output reconstruction from model-execution materials. We add an inventory of inference, adaptation, basis, and repair files in Appendix U and Table 51, and separate execution acceptance checks from full scientific retraining. We provide the execution paths and verification records in the linked repository's ARTIFACT_INDEX.md and VALIDATION.md.

Reviewer_DvFsOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that correct answers and successful local perturbation tests do not establish that a VLM relies solely on task-relevant visual information. To address this, the authors propose the I-CRB audit, which enumerates all legal contexts for the task pairs reached by local tests and checks prediction consistency while keeping those pairs fixed, returning either contradictions, completed checks, or unresolved comparisons. Theoretically, the method provides finite-domain guarantees and bounds for incomplete audits, where "route" denotes an ordered object selection rather than an internal neural pathway. Validation experiments on CLEVR with controlled prediction heads and a task-adapted Qwen3-VL-30B-A3B model show that, on 256 new images, 10 of the 234 images that passed every local test failed the expanded consistency audit; internal replacement of task-object representations repaired 8 of those 10 cases, although this does not identify a unique causal mechanism.

Critical strengths
  1. Clear and coherent presentation.

The paper is well structured, with a logical progression of ideas and a clear presentation of the main arguments. The organization makes the technical discussion relatively easy to follow.

  1. A useful formalization of constrained visual-support auditing.

The finite-domain formulation provides a clear account of what a local audit can and cannot establish. In particular, the distinction between legal reachability and dependence complexity is conceptually useful and helps clarify the scope and limitations of the proposed auditing framework.

Critical weaknesses
  1. The broader importance and generalizability of the research question remain unclear.

    The main results are established on a highly structured, finite-domain CLEVR task with binary answers, known task-relevant object positions, and enumerable legal routes. These assumptions make exact support completion and certification possible, but they may not hold in realistic VLM applications. For example, image captioning has open-ended outputs, no unique task-relevant object subset, and no obvious equivalence relation for determining whether two responses represent the same answer.

  2. The methodological novelty appears limited relative to the complexity of the presentation.

    At a high level, the proposed audit enumerates or queries legal contexts, groups inputs with unchanged task-relevant variables, and checks whether the model output remains consistent. The theoretical framework provides useful terminology, certificates, and bounds, but the underlying procedure is close to exhaustive constrained counterfactual testing. The paper should more clearly distinguish its technical contribution from straightforward enumeration and consistency checking, and demonstrate where its formalism provides practical or computational advantages.

  3. Several important ablations remain incomplete despite the large number of controls.

    The paper compares many training objectives and write spaces, but does not systematically study the sensitivity to the intervention rank, selected layers or streams, number of patched positions, completion coverage, construction-set size, or correction-network capacity. Moreover, Joint is not parameter- or total-compute-matched to the fixed-space methods. These confounds make it difficult to isolate whether performance differences arise from the proposed mechanism, model capacity, or optimization budget. Equal-capacity and equal-compute comparisons would strengthen the mechanism-specific claims.

  4. The audit does not evaluate stochastic generation behavior.

    Each input is evaluated using greedy decoding, so the reported support violations characterize one deterministic decoding policy rather than the model’s full response distribution. A model may appear consistent under greedy decoding while assigning substantial probability to inconsistent answers, or may exhibit different support behavior across sampled generations. The authors should evaluate repeated rollouts under multiple decoding temperatures and random seeds.

Questions for authors

see weakness.

8 comments
AuthorsOpenReview ↗

Response to Reviewer DvFs (1/6)

We thank you for the detailed assessment and concrete experimental recommendations. To address the points you raise, we add cross-family audits, real-image descriptions with human review, executable-history controls, and equal-query comparisons. We address W1-W2 here, W3-W4 in Part 2, and Questions For Authors in Part 3.


W1. Applicability to open outputs, realistic inputs, and environment-required constraints.

We appreciate your emphasis on open outputs, realistic inputs, and environment-required constraints. We identify umbrella/curtain and metal/wood description conflicts despite unchanged target-region source pixels.

The support hypothesis depends on the application. For a marked-region description, the relevant question is whether that region suffices for the requested description. Other applications can specify different candidate supports.

To test this hypothesis with open outputs, we use frozen, unadapted Qwen3-VL-30B-A3B to describe marked regions on 32 GQA images. We compare two context-removal conditions that preserve target-region source pixels. After automatic screening, we obtain independent blinded ratings from two human raters on 40 stratified pairs from 25 images. Their pre-follow-up agreement is 37/40 pairs (92.5%, Cohen's kappa 0.625).

We identify the following agreed conflicts between descriptions of fixed target-region source pixels. We shorten the descriptions here for readability.

Original contextModified context
White, fabric-covered umbrellaYellow fabric curtain
Metal shelving unitBookshelf made of wood

Both raters label zero conflicts on all eight repeat/paraphrase controls. We also obtain a separate independent blinded follow-up from a third human rater. Every original rating and disagreement remains in the record. We keep the standing/nearby-ball pair separate because those descriptions can coexist. These case-level semantic refutations extend the audit to open outputs; the finite binary certificate retains its stated output contract.

For model-family replication, we independently task-adapt InternVL3-38B and verify its frozen gate with 127/128 Four and 124/128 Pair inputs correct. On the reopened 256-image panel, we obtain 98.300% D accuracy and completion contradictions in 7/249 locally passing images. The additional InternVL3-8B evaluation gives 97.409% accuracy and 6/234 such images. Each audit uses that model's own local passes.

These adaptations replicate the audit finding across model families. The detailed intervention and repair studies use task-adapted Qwen.

Environment-required constraints provide a further test. In AI2-THOR, the actions and observation interval require executable histories. After verifying the frozen native Qwen model's development capability gate, we find three completion contradictions among 42 locally passing question groups in 11 development rooms.

To isolate legality from edit distance, we compare executable and relaxed one-edit neighborhoods across all 29 admitted rooms. The model, question, first image, later-image pools, and legal reference remain fixed.

ComparisonExecutable localRelaxed local
Development groups with contradictions, full neighborhoods1 / 443 / 44
Correct-local-pass example, detection at six queries0%42.86%

Both additional detections have separate legal-reference witnesses; one has an incorrect root. The correct-local-pass example isolates a revealing comparison that executable-history constraints exclude at equal query use. We count non-executable recombinations separately from legal counterexamples. The aggregate gain has a descriptive 95% room-bootstrap interval of [-1.43, 6.87] percentage points.

The separate 18-room panel behaves differently, with 71 groups correct throughout the legal domain, one consistently wrong, and zero context-dependent flips. We retain all 19 geometry exclusions from the original 48 rooms. This decomposition distinguishes support failures from genuinely consistent behavior.

Revised manuscript: We add cross-family audits, human-reviewed real-image descriptions, and executable-history experiments in Sections 3.3-3.4 and Table 4. We detail open-output screening and independent blinded human review in Appendix T.1. We extend Appendix T.2-T.3 and Table 50 with the executable-history protocol, exclusions, full-domain decomposition, and same-edit-distance comparison. We clarify the finite certificate's input and output contract in Section 2.3.

AuthorsOpenReview ↗

Response to Reviewer DvFs (2/6)

W2. Technical contribution beyond enumeration and practical information gains.

Thank you for asking us to make the practical value beyond enumeration explicit. Completing support groups resolves more support checks and tightens logical bounds at equal query use.

The purpose of completion is to turn a local pass into a precisely scoped support assessment. I-CRB completes selected same-support groups and returns support counterexamples, completed selected-pair checks, and the remaining unqueried comparisons. With a correct anchor for each selected task pair, consistency throughout its legal completion certifies correctness there. Errors at untested task pairs remain a separate category. This criterion makes local passes inspectable before and after repair, taking legal reachability into account.

To quantify this benefit, we compare query orders using identical saved predictions and the same checker across seven budgets and 32 fixed orders.

At a budget of 16 queriesSupport-group completionLocal-then-uniform
Completed repeated-support groups2.1571.843
Logical-gap bound width0.07500.0945
Actual queries, including root11.08211.082
Images with discovered errors (%)11.47511.536

The means account for smaller domains that exhaust before the query cap. At equal query use, completion gives more completed support checks and tighter logical bounds, alongside similar error discovery.

Stopping-rule replay tests whether the bounds also reduce query use at a declared tolerance. At the 5% gap threshold, we use 13.341 mean queries with two-sided bounds, versus 13.628 with lower-bound-only stopping and 14.090 with enumeration. All three rules use the same grouped order and give zero incorrect replay decisions. At zero tolerance, two-sided and ordinary counterexample stopping coincide.

Revised manuscript: We add the equal-query experiment in Section 2.4 and Table 2, and extend Appendix E.1-E.2 with all seven budgets and stopping-rule comparisons. We reorganize Section 2 around the input contract, completion, and the three evidence outcomes in Table 1. We distinguish the inherited consistency criterion, correct-anchor certificate, and unresolved-scope bounds in Section 2.3, and add the closest-work comparison in Appendix P.

AuthorsOpenReview ↗

Response to Reviewer DvFs (3/6)

We appreciate the specific controls you propose for sensitivity, fair comparison, and stochastic outputs. We address them through additional retraining ablations, matched repair comparisons, and repeated stochastic generation. We present W3-W4 below.


W3. Sensitivity analyses and parameter- and training-cost-matched comparisons.

Thank you for detailing the sensitivity and cost controls that strengthen this comparison. Semantic gives net repair, while time-matched Joint-Slim gives stronger rank-4 repair on the reopened panel.

Following your recommendations, we first evaluate all six requested sensitivity axes.

  • Rank. We separately train Semantic and direct ranks 2/4/8. We evaluate rank-2/rank-8 with three trajectories on the reused 64-image panel, including the earlier seed, and two new trajectories on the reopened 256-image panel.
  • Layers/streams. We retrain main and DeepStack-8/16/24 systems and obtain 98.182/97.904/97.724/97.062% D accuracy on 64 images. We report post-training only/leave-one-out tests separately.
  • Patched positions. We retrain task-18 and task-50 systems, obtaining 98.314/98.077% on 64 images with the same single seed. We compare task-50, suffix-50, and fixed-random non-task-50 across three paired trajectories, obtaining 98.451/97.115/97.185% on 256 images.
  • Completion coverage. We evaluate budgets 1/2/4/8/16/32/64 with 32 fixed orders and the same checker.
  • Constraint strength. We relax 3 to 2 to 1 to 0 relations in a fixed order, obtaining mean one-edit neighborhood sizes of 5.184/8.754/12.328/16.609. We separately evaluate All-distinct and the same-edit-distance environment control, and hold the original legal reference fixed.
  • Correction capacity. We retrain reader widths 8/16/24 and obtain 97.886/98.167/98.179% on 64 images.

The listed accuracies give equal weight to each image. Stream, position-count, and reader-width tests use one seed, while rank and primary location comparisons use the repetitions above. All correction trajectories share one adapted VLM.

To separate capacity from repair performance, we compare Joint-Slim's 135,732 parameters per attribute with the Semantic corrector's 135,760. We account separately for Semantic's 32,768-parameter basis.

The timing control accounts for complete training runs. For each Semantic run, we include 576 basis updates and 512 repair updates, plus extra basis-state capture amortized over three trajectories. All six direct-learning fits match this time target within 0.1%. We stop at 711-757 steps according to elapsed training time, independently of evaluation performance.

Call-count matching provides a separate control, with 2,112 gradient-forward requests in the earlier 704-step comparison. We itemize capture, I/O, forward/backward counts, and deployment in Appendix Q. These controls quantify parameter count, measured training time, and forward/backward call counts, each with its own accounting definition.

With these controls in place, we evaluate three paired trajectories on the reopened 256-image panel at both declared amplitudes.

Method / amplitudeD accuracy (%)Bad prefixes (%)Regression (%)
Semantic / 0.597.8793.5330.272
Time-matched Joint-Slim / 0.597.9993.5110.115
Semantic / 197.9833.3140.645
Time-matched Joint-Slim / 198.4502.4240.284
Reviewer_U9LbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the limitations of local counterfactual auditing for VLMs under constrained input domains. The authors propose a support-auditing procedure that checks prediction consistency across legal contexts sharing the same task-relevant objects. They also investigate activation interventions and donor-free representation corrections on a task-adapted VLM.

Critical strengths
  1. The paper highlights an interesting limitation of local auditing: simple context dependencies may remain undetected when admissibility constraints prevent informative comparisons.

  2. The paper provides a systematic auditing framework that distinguishes prediction correctness from support consistency. Its use of explicit counterexamples, completed checks, and unresolved comparisons makes the conclusions of an audit more precise.

Critical weaknesses

1.The theoretical contribution seems somewhat limited. The finite-domain certification results and support-gap bounds largely follow from classical functional dependency arguments. The constrained visual setting is interesting, but the theoretical novelty could be better justified.

2.The benefit of mechanism-guided repair remains unclear. Semantic shows limited natural-error transfer (Table 22), while Joint achieves better accuracy and bad-prefix rates (Table 6). Despite its larger parameter count, Joint's stronger performance raises questions about the value of intervention-trained directions.

3.The experiments do not sufficiently support the paper's broader contributions. Evaluation is limited to a single adapted VLM and synthetic inputs, while key ablations and simpler baselines are missing.

Questions for authors

Given the better confirmation performance of Joint, could the authors clarify what practical benefit the intervention-trained Semantic directions offer over direct low-rank learning?

Since the relevant object crops are already identified, would training directly on the Pair interface be a useful baseline? This might help clarify when the more complex intervention procedure is needed.

Table 16 shows that replacing non-task positions can also restore errors. Could the authors explain the choice to restrict corrections to task-interior positions?

9 comments
AuthorsOpenReview ↗

Response to Reviewer U9Lb (1/6)

We appreciate the time and care you devote to evaluating our work. We address your recommendations with additional model-family audits, simpler trained baselines, design ablations, and matched repair comparisons. We address W1-W3 below and your Questions For Authors in Part 2.


W1. Clarification of the theoretical contribution and its practical value.

Thank you for highlighting the connection between the theoretical result and its practical value. Simple shortcuts can evade legal local edits; completing support groups resolves more checks per query.

Our theoretical result shows that even an endpoint-only shortcut can pass every legal one-object context change when a revealing comparison requires additional legal edits. The number of edits needed to reveal a failure can therefore overstate the complexity of the underlying dependence.

This distinction motivates completing the legal comparisons for selected task pairs rather than interpreting a local pass alone. Building on functional-dependency consistency, I-CRB checks predictions throughout each selected pair's legal completion. A correct anchor and consistency throughout that completion together certify correctness on the completed set. The audit reports support counterexamples, completed selected-pair checks, and unresolved scope; we track errors at untested task pairs separately.

To measure the practical value of this completion strategy, we compare query orders using identical saved predictions and the same checker. Across seven budgets and 32 fixed orders, the comparison at budget 16 gives:

  • More completed repeated-support groups, 2.157 versus 1.843 for local-then-uniform queries.
  • Narrower logical-bound widths, 0.0750 versus 0.0945, at the same mean query count.
  • Similar error discovery, 11.475% versus 11.536% of images.

These results show that completion resolves more support assessments per query, alongside similar error discovery.

Revised manuscript: We add the equal-query experiment in Section 2.4 and Table 2, with all budgets and decision replay in Appendix E.1-E.2. We separately state the support criterion and correct-anchor certificate in Section 2.3. We revise Section 1 and Appendix P to clarify the legal-reachability contribution and its relationship to these foundations.

AuthorsOpenReview ↗

Response to Reviewer U9Lb (2/6)

W2. Practical benefits of Semantic directions and matched repair comparisons.

We appreciate your recommendation to test the practical benefit of Semantic directions against matched alternatives. Semantic enables targeted attribute control and net repair; matched direct learning gives stronger rank-4 repair.

To test repair under matched conditions, we compare Joint-Slim's 135,732 trainable parameters per attribute with the Semantic corrector's 135,760. We account separately for Semantic's 32,768-parameter basis. The time target includes basis fitting, repair training, and extra basis-state capture amortized over three trajectories. All six attribute-by-trajectory runs match that target within 0.1%.

For this comparison, we evaluate three paired correction trajectories on the reopened 256-image panel, using one shared adapted Qwen VLM. D accuracy and regression give equal weight to each image, and bad-prefix rates first average within each image. Repaired / introduced counts denote mean input counts per trajectory. The uncorrected Four baseline has 97.040% D accuracy and 5.358% bad prefixes.

Amplitude 0.5

MetricSemanticTime-matched Joint-Slim
D accuracy (%)97.87997.999
Bad prefixes (%)3.5333.511
Regression (%)0.2720.115
Repaired / introduced77.33 / 14.6776.67 / 6.67

Amplitude 1

MetricSemanticTime-matched Joint-Slim
D accuracy (%)97.98398.450
Bad prefixes (%)3.3142.424
Regression (%)0.6450.284
Repaired / introduced105.67 / 35.00114.67 / 18.33

The calibrated Semantic update improves accuracy and completion consistency. At amplitude 0.5, it satisfies the development checks and the prespecified 0.5% mean-regression point-estimate criterion on this panel. Regression is 0.272% (95% image-bootstrap interval [0.065%, 0.546%]). At amplitude 1, regression rises to 0.645%, above the threshold. Joint-Slim gives stronger repair by +0.467 accuracy points (descriptive paired-image 95% interval [0.129, 0.840]).

To recheck visual support, we re-audit rank-4 Semantic at amplitude 0.5. Completion finds contradictions in 7/240, 5/237, and 3/243 locally passing images for trajectories 17/29/43, versus 10/234 for Four. Each denominator counts that system's own local passes.

At rank 8 and amplitude 0.5, Semantic improves accuracy by 0.437 points [0.149, 0.761] and gives a bad-prefix difference of -0.856 points [-1.673, -0.107], with 0.086 additional regression points [0.017, 0.171]. We compare paired trajectories 29/43 on the reopened 256 images and report Semantic-minus-direct differences with descriptive 95% paired-image bootstrap intervals.

These comparisons distinguish two useful capabilities. Semantic provides attribute-specific answer control, which we test through exchanges with specified expected answers in Q1. Joint-Slim provides stronger main rank-4 repair. Evaluating semantic control, natural-error transfer, and donor-free correction separately makes each operation's contribution directly testable.

Revised manuscript: We add near-parameter-matched and complete training-time-matched repair comparisons alongside the original Joint results in Sections 5.2-5.3 and main-text Tables 5-6. We expand Appendix Q with training-time, capture, I/O, operation-count, and deployment measurements, plus paired rank repetitions. We summarize the rank-8 accuracy and completion-consistency gains alongside the regression cost in Section 5.3 and report all three paired differences and intervals in Appendix Q.3. We connect Semantic's re-audit to Four's local-pass and completion counts in Section 5.4. We revise Section 4.2 to distinguish semantic control from natural-error transfer.

AuthorsOpenReview ↗

Response to Reviewer U9Lb (3/6)

W3. Broader validation through ablations, simpler baselines, and additional models and inputs.

Thank you for identifying these opportunities to strengthen the experimental evaluation. The audit finding replicates across capable VLM families, and ablations quantify key correction design choices.

Building on your recommendations, we add ablations, trained Pair baselines, cross-family audits, and external-input tests.

FactorWhat we testEvidence
Intervention rankWe train ranks 2/4/8 separatelyRank-2/rank-8 have three trajectories on 64 images and two new trajectories on 256 images
Visual streamWe retrain Main / DeepStack-8/16/24 separately98.182 / 97.904 / 97.724 / 97.062% accuracy; 64 images, one seed
Write-position countWe retrain 18 versus 50 task positions98.314 / 98.077% accuracy; 64 images, same single seed
Write regionWe compare task / suffix / fixed-random non-task, 50 positions each98.451 / 97.115 / 97.185% accuracy; 256 images, three paired trajectories
Correction capacityWe retrain reader widths 8/16/2497.886 / 98.167 / 98.179% accuracy; 64 images, one seed
Completion coverageWe compare query budgets 1/2/4/8/16/32/6432 fixed orders per budget, with the same checker
Constraint strengthWe retain 3/2/1/0 relations and test All-distinctMean one-edit neighborhood sizes 5.184 / 8.754 / 12.328 / 16.609; All-distinct separately

The table reports image-weighted D accuracy in the listed variant order. We use the reused 64-image development panel and reopened 256-image evaluation panel; all correction trajectories share one adapted VLM. Task-interior writing gives the highest accuracy in the write-region comparison, while the other ablations quantify stream and capacity sensitivity.

For the simpler-interface comparison, we train Four, cropped Pair, and fixed-canvas Pair with three matched adaptation seeds. On the reopened 256 images, we obtain 97.071%, 97.431%, and 97.717% accuracy, respectively. Both Pair variants have zero completion misses among local passes in all three runs. Fixed-canvas Pair preserves task pixels, layout, and processor scale while blanking context tiles. We detail these interface and write-region comparisons in Q2-Q3, and capacity and training-time controls in W2.

To test replication across model families, we independently adapt InternVL3-8B and InternVL3-38B. We compare their audits with adapted Qwen3-VL-30B-A3B on the same reopened 256-image panel, counting contradictions among each model's own locally passing images.

Adapted VLMD accuracy (%)Contradictions / local passes (images)
Qwen3-VL-30B-A3B97.04010 / 234
InternVL3-8B97.4096 / 234
InternVL3-38B98.3007 / 249

InternVL3-38B passes the frozen capability gate with 127/128 Four and 124/128 Pair development inputs correct. Its completion contradictions therefore reproduce the audit finding in another capable model family at a comparable scale.

  • Real images and open outputs. We use frozen, unadapted Qwen to describe 32 GQA images while preserving target-region source pixels. We obtain independent blinded ratings from two human raters on 40 stratified pairs from 25 images, with agreement on 37/40 pairs before follow-up review. We obtain a separate independent blinded follow-up from a third human rater. We identify agreed umbrella/curtain and metal/wood conflicts.
  • Environment-defined constraints. We audit executable AI2-THOR histories and find three completion contradictions among 42 locally passing question groups in 11 development rooms. We also evaluate a separate 18-room panel and find 71 groups correct throughout the legal domain and one consistently wrong.

Revised manuscript: We add cross-family audits in Section 3.3 and Table 4, and real-image and executable-history experiments in Section 3.4. We extend Section 5.3 with trained-Pair and matched write-region comparisons. We add design-sensitivity and trained-interface experiments in Appendix R and Tables 46-48, paired rank repetitions in Appendix Q, and external protocols, human review, and environment decomposition in Appendix T.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Response

We sincerely thank all reviewers for their time, effort, and constructive feedback, as well as the area chairs, senior area chairs, and program chairs for their careful consideration throughout the review process. As the review process does not allow further discussion with reviewers, we would like to summarize our rebuttal and clarify how we have addressed each concern.

Our paper studies how to audit a vision-language model when only some visual comparisons are possible. A model can pass every legal local test yet fail when only the context changes. The Intervenable Causal Route Bottleneck (I-CRB) finds these failures and distinguishes checking that the task inputs suffice within a stated scope from testing whether either can be omitted. The legal input structure tells us when even a complete check leaves the second question open and a different comparison is needed. We probe the failures with internal interventions and re-audit the same inputs after repair. We thank the reviewers and Area Chair for their feedback.

Additions During the Rebuttal

Method and comparison design (DvFs, U9Lb, Tn76). Beyond checking agreement on a fixed set, we give a pre-query criterion for whether completing the legal contexts of tested pairs could distinguish an omission. In a new 256-scene CLEVR study, we ablate only the requirement to hold context fixed. In the prespecified subset of 90 local passes where the full target table distinguishes both omissions but the local table does not, matched contrasts rule out both omissions in 89/90 scenes versus 49/90 for the label-aware control, with 257 additional queries per policy (Sections 2.3-2.4; Appendix C.7).

Broader evidence (U9Lb, DvFs, Tn76). We reproduce the finding in separately adapted InternVL3-8B and InternVL3-38B. For DvFs's questions about decoding and open-ended outputs, we test three sampling temperatures and add blinded assessments of real-image descriptions. For Tn76's navigation follow-up, we audit 512 MiniWorld episodes under wall, doorway, and action constraints. In Figure 8, both local answers are correct, yet one of eight executable histories yields the wrong answer for the same task image.

Repair comparisons (U9Lb, DvFs). We train the simpler Pair baseline, compare task and non-task write locations at a fixed number of positions, and add controls for parameter count and measured training time. The main rank-four comparison favors direct learning. At rank eight and amplitude 0.5, Semantic lowers the completion bad-prefix rate by 0.856 percentage points relative to direct learning, with a 0.086-point increase in regression (Section 5.3; Appendices Q-R).

Presentation and reproducibility (U9Lb, Tn76). To address U9Lb's figure comment, we redraw the figures to show what stays fixed and make the interventions easier to compare (Figures 3 and 4). For Tn76, we add a study timeline, clarify counting units and study roles, and separate statistical reconstruction from executable inference and training in the anonymous artifact (Figure 2; Appendix U).

We believe that these clarifications and additional experiments show how we have addressed the concerns from all reviewers and further highlight the value of our work.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.