acceptodds
Under review as a conference paper at ICLR 2027

What Forced-Choice Hides in Vision-Language Model Modality Conflict

Abstract

Vision-language models (VLMs) often favor language-consistent answers over visual evidence under forced-choice image-text conflict. Such evaluation cannot distinguish preference for the language-referenced answer from behavior induced by the absence of an explicit mismatch response. We investigate this distinction using temporal-scale estimation on 500 natural images spanning five ordered temporal bands, comparing a 5-way forced-choice prompt with a 6-way prompt that adds a mismatch option. Five open-weight VLMs select the mismatch option on 56–91% of contradiction items but at most 4% of standard items; without it, they favor language-consistent temporal bands. Paired item-level analysis shows that 50–90% of language-consistent forced-choice responses shift to mismatch even at the smallest contradiction magnitude. Mismatch signaling generally increases with contradiction magnitude, while residual temporal-band responses remain language-aligned. Under the 5-way condition, linear probes recover the vision-consistent band throughout the network, while the model's own unembedding favors the language-consistent answer in later layers. Query-token activation patching strongly shifts outputs toward the vision-consistent answer. This establishes the causal influence of the query representation under the intervention. Removing the image reduces standard-condition accuracy by only 0–18 percentage points. These results show that forced-choice evaluation can obscure the distinction between mismatch signaling and language-aligned temporal selection.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, modality conflict, temporal reasoning, multimodal evaluation, visual grounding, language bias, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cc5aOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how VLMs estimate temporal bands from images and how their answers are influenced by visual evidence or semantic knowledge. The authors devise a behavioral task that asks VLMs about an image depicting an action and examines how the prompt influences their forced-choice estimates of the approximate temporal band. They find that VLMs’ temporal-band estimates follow language-consistent answers.

Critical strengths

None identified.

Critical weaknesses

It is unclear why temporal reasoning, specifically temporal-band estimation, is an interesting question to study. I do not understand which aspects of VLMs this investigation helps us understand. The paper devotes substantial space to reporting paired response transitions between five-way and six-way choices, but the motivation for this comparison and its takeaway remain unclear.

I find the paper difficult to comprehend. The motivation, problem setup, and methods are poorly explained. The paper reports numerous evaluations, but what these evaluations measure and why particular design choices were made are very unclear. The choices of prompt format and the five-way condition appear arbitrary, and the central comparison between five-way and six-way forced-choice prompts is poorly motivated.

Questions for authors

What do the authors mean by “temporal band” and “temporal-scale estimation”? These terms need clear definitions; their current meanings are vague.

Why is temporal-band estimation an interesting question to study, and what does it help us understand about VLMs?

What motivates the prompt design, the five-way condition, and the comparison between five-way and six-way forced-choice prompts?

Why are paired response transitions central to the analysis, and what should readers take away from them?

How is the text-only baseline evaluated on VLMs?

Regarding multimodal interactions, please refer to the visual saliency paper by Tekker et al.: https://arxiv.org/pdf/2610.12650.

1 comment

We thank the reviewer for the careful reading. We provide clarifications and new analyses addressing each concern below.

Scientific question and methodological stakes (W1, Q2). Forced-choice evaluation under image-text conflict is the dominant paradigm for measuring VLM modality preference in the current literature. The standard interpretation of a language-consistent forced-choice answer is that the model disregarded the visual evidence. Our paper tests this interpretation by extending the response space to include an explicit mismatch option. On the same items, most answers classified as language-consistent under forced choice become explicit mismatch reports when this option is available. The paper identifies a methodological problem in the dominant evaluation setup and provides behavioral and mechanistic evidence for reinterpreting its results.

Definitions (Q1). Temporal-band estimation is the task of estimating how long a visible process has been ongoing to reach its current state, with answer choices corresponding to five ordinal duration bands: A (seconds-minutes), B (hours-days), C (weeks-months), D (years-decades), E (decades-millennia). We chose this task because natural processes have characteristic durations. Water boiling is a minutes-scale process, while mountain erosion is a millennia-scale process. When the image depicts one process, and the question specifies a different one, visual evidence and process-description priors support different temporal bands. The ordinal structure also permits measuring conflict magnitude by band-distance.

Prompt design (W2, Q3). The two prompt formats isolate the role of the response space. Both use identical image-question pairs and the same five temporal bands (A–E). The six-way format adds option F, “The described process is not visible in the image.” This is the minimal change needed to test whether a language-consistent response under forced choice reflects modality preference or the absence of a mismatch option. The five-way condition reproduces the forced-choice setup of earlier modality-preference evaluations, while the six-way condition allows the model to report a mismatch explicitly.

Paired response transitions (Q4). Aggregate rates do not identify which individual predictions change between formats. We conducted a new paired item-level transition analysis on the same 500 contradictory examples per model.

ModelL→F (%)V→F (%)L→L (%)
Qwen3-VL90.285.38.3
InternVL3.558.645.341.4
Phi-456.552.941.0
LLaVA-NeXT57.754.227.8
Pixtral55.464.342.9

L/V denote language-/vision-consistent responses; F denotes mismatch. Rates are conditional on the five-way response.

Across models, 55.4 to 90.2 percent of language-consistent five-way responses and 45.3 to 85.3 percent of vision-consistent responses switch to F when the option becomes available. A language-consistent forced-choice answer, interpreted in earlier studies as disregard for visual evidence, usually becomes an explicit mismatch report when the response space permits it. However, 8.3 to 42.9 percent of language-consistent responses persist under the six-way format, indicating that the forced-choice setup obscures mismatch detection without fully accounting for language-consistent responding.

Text-only baseline (Q5). The text-only baseline, specified in Appendix A.2.5 of the paper, uses the standard five-way prompt with the image removed. The process description, answer choices, and output instruction remain unchanged. Removing the image reduces standard-condition accuracy by 0 to 18 percentage points across the five models. Process descriptions alone support many correct temporal-band answers, which bears on how standard-condition accuracy should be interpreted.

Relation to Teker et al. (Q6). Teker et al. (2026) study word-color interference through the Stroop paradigm. They evaluate contrastive and generative VLMs, but their representational-similarity and concept-steering analyses focus on CLIP. Our work differs in four respects. First, we probe and patch hidden states in five generative VLMs from four language-model families. Second, query-token activation patching tests causal influence on answer selection, whereas their interventions modify CLIP image embeddings and measure similarity changes. Third, we analyze ordinal temporal-band distance; their study varies word-color congruence and visual salience. Fourth, we compare matched prompts with and without an explicit mismatch option. Although Teker et al. categorize free-form responses mentioning both cues, they do not manipulate the availability of a dedicated mismatch response. We will cite their study in Related Work.

Writing clarity. We will revise the introduction and methods to state the research question earlier, define temporal bands, and explain the five-way/six-way comparison.

Reviewer_Ysb9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an image-text contradiction task for studying how vision-language models handle conflicts between visual and linguistic evidence, including an ordinal measure of the size of the contradiction. The authors also provide a mechanistic analysis of the forced-choice (5-way) setting of this task. Their main contribution is the task itself, together with the finding that forced-choice evaluation hides the fact that models very often flag the contradiction when given an explicit mismatch option. This is validated across five models, and the mechanistic analysis uses four complementary methods.

Critical strengths
  • The paper introduces a dataset that combines an ordinal measure of contradiction distance with an explicit mismatch response option in a temporal-scale setting. This allows the authors to show interesting results about how conflict severity affects behaviour. It is particularly interesting to see how the answers shift to the mismatch option increasing at larger distances. This dataset also gives the community a reusable testbed for studying conflict severity, which prior modality-preference benchmarks mostly lack.
  • The experiments are comprehensive and cover a wide range of models and mechanistic interpretability methods. Additionally the main finding regarding forced-choices is well supported. Adding an explicit mismatch option shows that forced-choice evaluation pushes models into a modality conflict, and that models identify the contradiction most of the time when allowed to. This is important as it suggests that forced-choice evaluations of modality preference may conflate language reliance with the absence of a mismatch response.
Critical weaknesses
  • The mechanistic analysis is thorough. However, I believe it should have been conducted on the 6-way prompt as well as the 5-way forced-choice prompt. As the authors’ main finding shows, the forced-choice setting obscures the correct behaviour (the mismatch option), and when this option is available it is chosen most of the time. Table 2 showed that for all but two model and band-distance pairs the language consistent responses switched to the mismatch option 50% or more of the time. As such, most of the items analysed mechanistically are ones the model would flag as a mismatch if it could. This analysis could then be describing what the model does when forced to answer a question it recognises as unanswerable, rather than a genuine preference for language over vision. As such, I think the mechanistic analysis, while well executed, is applied to the less informative setting. I would be more interested in understanding how the model navigates this conflict in the setting where it can actually give a correct answer, and why in those cases it still sometimes favours the language option. If the authors provided this analysis with the same setup, I think it would significantly strengthen the paper.
  • I am concerned about the validity of the probe results shown in Figure 2. Here the results show that the visual-band information is recoverable from the activations across the full depth of the model. However, the language-band probe consistently has close to chance accuracy, and the paper does not explain why. This is surprising given that the model regularly produces the language band answer, and the logit lens decodes it at up to ~0.71 in later layers (Figure 2D), meaning the information should be recoverable from the model’s activations. A more detailed explanation of this finding would be necessary to have more confidence in the results.
Questions for authors

Why does the language-band probe perform so much worse than the visual-band probe? (see weaknesses point 2 for more information)

5 comments

We thank the reviewer for the careful engagement and provide additional analyses addressing the concerns.

W1. The five-way and six-way conditions open two complementary windows onto modality conflict. The five-way condition isolates the residual language-aligned pathway that governs responses when mismatch reporting is unavailable. The six-way condition exposes the mismatch-reporting pathway. Our mechanistic protocol targets the first because the paper's central interpretive claim concerns how forced-choice evaluations have been read in the modality-preference literature.

We conducted a new logit-lens analysis extending the mechanistic evidence to the six-way condition. The table below reports final-layer decoding rates for language-consistent answers (L) under both formats and for the mismatch option (F) under six-way:

Model5-way L6-way L6-way F
Qwen3-VL0.510.050.92
InternVL3.50.530.230.58
LLaVA-NeXT0.390.150.56
Phi-40.700.310.57
Pixtral0.490.200.66

Language-consistent decoding decreases across all five models when F becomes available. F dominates six-way final-layer predictions, ranging from 0.56 to 0.92, with Qwen3-VL showing the sharpest shift (0.51 to 0.05 for L alongside 0.92 for F). Under forced choice, the final-layer trajectory ends on a language-consistent band. Under six-way, it ends on F. The final-layer readout tracks the behavioral pattern.

We additionally conducted a paired item-level analysis on the same 500 contradictory examples. Across models, 55.4 to 90.2 percent of language-consistent five-way responses transition to F under six-way. Vision-consistent responses move in the same direction, with 45.3 to 85.3 percent transitioning to F:

ModelL → F (%)V → F (%)
Qwen3-VL90.285.3
InternVL3.558.645.3
Phi-456.552.9
LLaVA-NeXT57.754.2
Pixtral55.464.3

Both modality-consistent response patterns contain a compressed mismatch signal. This supports the paper's central claim. The five-way mechanistic protocol remains the appropriate scope for characterizing the residual language-aligned pathway that persists when mismatch reporting is available. We will integrate the six-way logit-lens analysis and the paired transition analysis into the main text of the camera-ready version.

W2 and Q. Linear probing and the logit lens measure different constructs. A linear probe tests whether a band can be recovered from a hidden representation using a trained linear classifier. The logit lens reports what the model's own output projection favors at a given layer. A band may not be reliably recovered by the trained probe while still receiving a high score under the model's output projection.

The language band is defined by the substituted process in the question. The process can influence the model's answer through its interaction with the query token and the output projection without being encoded as a separable feature that a linear probe can recover across items. Our activation-patching results identify where this interaction occurs. For Qwen3-VL, query-token patching recovers the vision-consistent answer at a rate of approximately 0.99, compared with 0.0055 for image-token patching and 0.10 for process-description patching. These results indicate that the query-token state, rather than the process-text representation, causally shapes the final answer under contradiction.

The near-chance language-band probe accuracy is consistent with this account. The language-aligned output emerges from a late-layer computation routed through the query token, not from a language-band representation reliably recovered by the trained linear probe. Our claim is limited to the following: the vision-consistent band is recoverable throughout the network, while the model's final answer selection under contradiction depends on the query-token computation. We will add this clarification to Section 4.2 in the camera-ready version.

W1

Thank you for running these additional experiments and for the paired transition analysis. However, I am still not fully convinced regarding this point. I do not follow the argument that the five-way protocol is the appropriate setting for “characterizing the residual language-aligned pathway that persists when mismatch reporting is available”, since that can only be observed in the six-way setting, where mismatch reporting is actually available. Additionally, could you clarify whether there is any difference between the logit lens at the final layer and the model’s normal decoding? It is presented in the paper and in this response as part of the mechanistic analysis, but if I understand correctly, it is simply what the model would output as part of its normal decoding. This seems consistent with the six-way values in the response, which closely match the behavioural results in Table 1.

W2 and Q

Thank you for your clarification. However, I am not yet convinced that this explains why the linear probe doesn’t work. The output projection used by the logit lens analysis is also a linear map that is applied to the same hidden representation as the probe. Given that the logit lens recovers this information in later layers, it suggests that at this point it is a linearly decodable feature in the hidden representation. So the fact that the probe cannot recover this information suggests there may have been a problem with how the probe was trained or evaluated. If you are suggesting that the query-token state “causally shapes the final answer under contradiction”, then this also suggests an issue with the probe, since the probe is trained on the query-token state (Section 3). If the language-consistent answer is formed in that state and can be read out linearly by the model’s own output projection, a correctly trained linear probe on that same state should also be able to recover it, at least in the later layers. Further explanation on why this is not the case would be helpful.

We thank the reviewer for the follow-up.

W1. In the five-way condition, L frequently wins over V, allowing us to investigate this outcome through logit-lens trajectories and activation patching. We have also conducted logit-lens analysis under the six-way condition. Across all five models, F eventually overtakes both L and V in decoding rate and remains dominant thereafter, with the onset of sustained dominance ranging from 46.2% to 92.3% of normalized decoder depth (Table R1). The intermediate-layer decoding rates further illustrate this pattern (Table R2). For example, at layer 31 of Qwen3-VL, F decoding reaches 91.0%, whereas L decoding is only 5.0%. This contrast motivates the five-way analysis to examine how L comes to dominate V, while the six-way analysis shows that this pattern is substantially attenuated when F is available.

Table R1: Onset of sustained F dominance

ModelLayer of sustained F dominanceNormalized depth
Qwen3-VL2982.9%
InternVL3.52982.9%
LLaVA-NeXT1754.8%
Phi-43692.3%
Pixtral1846.2%

Table R2: Intermediate-layer F and L decoding rates

ModelLayerF decodingL decoding
Qwen3-VL3191.0%5.0%
InternVL3.53154.2%25.1%
LLaVA-NeXT2847.2%17.1%
Phi-43649.6%35.9%
Pixtral3551.6%25.3%

The item-level transition data connects the two conditions: 55.4 to 90.2 percent of five-way L responses shift to F under six-way. The five-way mechanistic analysis examines how L prevails over V when a mismatch response is unavailable, while the six-way results show that this outcome frequently changes when F is introduced. The five-way analysis is not intended to explain the L responses that persist under the six-way analysis.

On the logit lens, the reviewer is correct that final-layer decoding is equivalent to normal model decoding when restricted to the same answer choices. The reported values match Table 1 because they measure the same quantity at the same point. We should not have presented these values as independent mechanistic evidence. The mechanistic insight comes from the intermediate-layer trajectories. In five-way, the L-consistent prediction strengthens across later decoder layers while the V-consistent prediction weakens, and their crossover tracks the model’s final answer selection. The final-layer endpoint adds no mechanistic information beyond the behavioral result. In the camera-ready version, we will give greater emphasis to the intermediate-layer trajectories and include the six-way results, where F dominates and L remains comparatively weak.

W2 and Q. This requires a methodological clarification that we should have provided earlier. The linear probe and the logit lens operate on different token representations. The probe uses mean-pooled image-token hidden states at each layer, whereas the logit lens applies the output projection to the query-token hidden state. Under causal attention, image tokens precede the question text and cannot attend to the subsequently introduced process description. Consequently, their hidden states cannot incorporate the substituted temporal process that defines the language band. Thus, the near-chance language-band probe accuracy is expected given the information available to these tokens, not a failure of probe training.

The query token appears after the full prompt and can attend to both the image tokens and the process description. Its output projection has access to language-band information, allowing the logit lens to recover language-consistent predictions at later layers. The reviewer is correct that a probe operating on the same representation should, in principle, recover information accessible to the logit lens. However, our probe and logit-lens analyses use different token representations. Section 3 did not make this distinction clear. We will correct it in the camera-ready version.

The findings are consistent with this clarification. The vision-consistent temporal band remains linearly recoverable from image-token representations throughout the network, even under contradiction when the model selects the language-consistent answer. Visual information remains accessible in these representations despite the conflicting final response. The logit lens shows increasingly language-aligned predictions at the query-token position in later layers. For Qwen3-VL, activation patching provides causal evidence for this token's role: query-token patching recovers the vision-consistent answer at approximately 0.99, compared with 0.0055 for image-token patching and 0.10 for process-description patching. These results identify the query-token state as the principal site of causal influence on final answer selection among the tested patching locations. Hence, the probe and logit-lens findings are consistent once their different token representations are taken into account.

Reviewer_v61COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how answer choices affect VLM behavior when an image and its accompanying question refer to different processes. The authors evaluate five models on 500 images using a temporal-scale estimation task. They compare a five-choice prompt with a six-choice prompt that adds “the described process is not visible in the image“. Many language-consistent answers shift to this mismatch option when it becomes available. Additional probing and activation-patching experiments examine the relationship between internal representations and final answers. The main finding is that forced-choice answers alone provide an incomplete picture of model behavior under image-text conflict.

Critical strengths

The comparison between five-choice and six-choice prompts provides a useful evaluation insight. The idea is simple and interesting.

  • The same items are evaluated with and without an explicit mismatch option.
  • Models frequently select the mismatch option when the image and question refer to different processes (56–91%), but rarely select it when they match (at most 4%).
  • For many of the same questions, models switch from answering about the named process to reporting that it is absent from the image when option F is added.
  • These results show why a forced-choice answer should not automatically be interpreted as evidence that the model failed to notice the mismatch.

The behavioral finding is supported across several models and by informative baselines.

  • The evaluation covers five models from four language-model families.
  • The text-only baseline is informative. It shows that models can answer many standard questions using the process description alone. Correct answers therefore do not necessarily indicate that the models used the image.
  • Together, these comparisons make the paper useful for interpreting multimodal evaluation results.
Critical weaknesses

The effect of temporal-band distance is not clearly separated from differences between items.

  • The comparison does not isolate the effect of temporal distance. Each image is paired with only one contradictory process, so different distances contain different images and process pairs. Changes in model behavior could therefore reflect the items being compared rather than distance itself.
  • The distance-4 comparison may be affected by the types of processes included. distance-4 only compares the shortest and longest temporal bands. Models may find these particular processes easier to distinguish, so the results cannot be attributed to temporal distance alone.
  • A larger temporal-band distance does not necessarily mean a stronger image-text mismatch. Two processes can have very different typical durations, but this alone does not tell us how visually or semantically different they are. The paper therefore needs stronger justification for interpreting temporal-band distance as conflict strength.

The activation-patching results do not clearly explain how the model chooses between image and text information.

  • The intervention does not isolate visual information. It replaces the entire query-token state with a state from the standard condition. This state contains information from both the image and the matching question, so the effect cannot be attributed to visual information alone.
  • Successful recovery may reflect transfer of an answer preference. The standard-run state may already favor the vision-consistent answer. Copying this state could transfer that preference without showing how the model resolves the original image-text conflict.
  • Recovery at the final layer provides limited mechanistic evidence. The patched state directly feeds the model’s output layer. Replacing it with a state from a correctly answered standard run makes recovery expected. The earlier-layer effects are more informative, but still require controls to distinguish visual-information transfer from answer transfer.
  • The mechanistic analysis does not directly explain the main behavioral finding. All mechanistic experiments use the five-choice prompt. So they do not establish why models switch to the mismatch response when the sixth option is added.

The findings have limited coverage across model types. The evaluation includes only five open-weight models. Although activation-level analyses may not be possible for closed-weight models, the five-choice versus six-choice behavioral comparison could still be conducted. Such results would help assess whether the main finding extends beyond the tested open-weight models.

Questions for authors
  1. Do the distance-related trends remain after controlling for temporal band and process identity, especially given that distance 4 includes only endpoint-band images?
  2. Does query-token patching restore the use of visual information, or simply transfer the standard run’s answer preference? What evidence distinguishes these explanations?
2 comments

We thank the reviewer for the careful assessment and address each concern with new analyses.

W1/Q1. We agree that band-distance is an ordinal descriptor and that process identity can confound distance effects. We conducted new logistic regressions predicting mismatch selection (F) from distance under three control settings: visual-band adjustment, which compares distance effects within each visual band; joint adjustment for both visual and language bands; and a regression restricted to items with endpoint visual bands (Seconds-Minutes or Decades-Millennia), which tests the reviewer's concern about distance-4.

ModelVisual-band ORBoth-band OREndpoint-only OR
Qwen3-VL1.71 [1.15, 2.55]NC1.80
InternVL3.51.31 [1.07, 1.60]1.33 [0.99, 1.78]1.41
Phi-41.24 [1.02, 1.52]1.15 [0.87, 1.51]1.33
LLaVA-NeXT1.84 [1.48, 2.28]1.52 [1.13, 2.04]1.86
Pixtral1.05 [0.86, 1.29]1.46 [1.01, 2.09]NE

NC = nonconvergent; NE = not estimable within the restricted sample.

Under visual-band adjustment, four of five models show positive associations with 95% CIs excluding 1. The endpoint-only regression reproduces positive associations for the same four models, which shows the distance trend does not arise solely from differences between endpoint and non-endpoint bands. Under joint adjustment for both bands, only LLaVA-NeXT and Pixtral retain intervals excluding 1. The trend is partially supported and depends on band composition. In the camera-ready version, we will describe the distance analysis as a descriptive pattern without an independent causal claim.

The paper's central claim does not depend on the distance analysis. The five-way and six-way evaluations use identical image-question pairs and differ only in whether F is available. The paired comparison supports the main finding about forced-choice compression of mismatch signals independently of the distance stratification.

W2/Q2. The reviewer raises the possibility that query-token patching transfers an answer preference. Our specificity controls distinguish this from a visual-information account. For Qwen3-VL, we patched three positions with activations from the standard run:

Patched positionVision-answer recovery
Query token~0.99
Image tokens0.0055
Process-description tokens0.10

If recovery simply reflected a transfer of answer preference from the standard run, patching image tokens or process-description tokens should produce comparable recovery, since their activations also come from a run that produces the vision-consistent answer. Instead, non-query-token patching yields near-zero recovery. The effect is specific to query-token patching, not a general consequence of inserting standard-run activations.

The layer-wise variation provides additional evidence. Recovery depth varies widely across models. Pixtral and LLaVA-NeXT transition by middle layers (approximately 0.52-0.58 normalized depth), Phi-4 and Qwen3-VL transition at 0.64-0.68, and InternVL3.5 transitions only at 0.86. If the recovery reflected a generic final-layer answer-transfer effect, depth profiles would be uniform across models. The observed variation instead reflects a model-dependent computation routed through the query-token position.

The mechanistic case does not rest on patching alone. Linear probes recover the vision-consistent band from hidden states at 80-88 percent accuracy across the network, while the model's own unembedding favors the language-consistent answer. This probing-versus-readout dissociation holds independently of the patching intervention and shows that vision-consistent information is present in the hidden states even when it does not shape the final answer. We will tighten this distinction in Section 4.2 of the camera-ready version.

W3. The five open-weight VLMs were chosen to span four language-model families (Qwen3, Llama 3, Mistral, Phi), which allowed us to test whether the information-versus-routing pattern replicates across independently trained language backbones. Open-weight access was necessary for the linear probing, logit lens, and activation-patching analyses. The reviewer is right that the behavioral comparison could extend to closed-weight models, and we will note this in the Discussion section. A paired item-level analysis we additionally conducted on the same 500 examples finds that 55.4-90.2 percent of language-consistent five-way responses and 45.3-85.3 percent of vision-consistent responses transition to F under six-way across all five models, which strengthens the behavioral generality within the evaluated open-weight set.

Thanks for the thorough response. The additional analyses provided some clarity, but my concerns about the confounding of temporal-band distance with item/process identity and the isolation of visual information from answer-preference transfer remain partially unresolved. Therefore, I will maintain my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.