- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes an evaluation framework for vision-language models (VLMs) to determine whether their failures on structured causal-reasoning outputs stem from a lack of causal reasoning or from difficulty producing the requested output format. The framework compares structured outputs against free-form text, tests four structural formats, and varies the level of prompt scaffolding. It also applies targeted fine-tuning to examine whether the identified failures reflect data limitations or architectural limits. Six VLMs are evaluated, with outputs scored by two LLM judges and validated against human-annotated response
Critical strengths
-
Clear presentation and well-motivated problem. The paper is well organized and easy to follow. It targets an important and under-examined problem: whether failures in structured causal-reasoning evaluations reflect a lack of reasoning or a failure to produce the requested format. The proposed framework is conceptually simple, yet it covers the main confounding dimensions (output format and prompt scaffolding) systematically.
-
Concrete and well-designed experimental setup. The experiments are specified in detail: four formats, three scaffolding levels, two LLM judges with human validation, and a rule-based failure taxonomy. The choice of models is clearly justified, spanning five training approaches across a 2023–2025 capability range. The selection of LLaVA-CoT and MiniGPT-4 for fine-tuning is likewise explained in terms of architectural contrast on a shared blind spot. Overall, the setup is clean and easy to understand.
Critical weaknesses
-
Validation of the framework. The human study validates the LLM judges, not the framework’s ability to separate format production from causal competence. Existing benchmarks (CELLO, MM-CoT, CausalVLBench) are discussed only qualitatively, with no empirical comparison. The novelty of score-compliance pairing over Long et al. (2025), who already disentangle format compliance from task accuracy, should also be clarified.
-
Limited model range. Only open-source 2023–2025 models are evaluated. Layers 1–3 require only inference, so applying them to recent API-based models (GPT, Gemini, Claude) seems feasible. This would test whether the reported format sensitivity persists in stronger models, which the authors themselves leave open.
Questions for authors
- Have the authors evaluated any of the six models on existing causal-reasoning benchmarks (e.g., CELLO, CausalVLBench)? Do the model rankings from the proposed framework agree with them?
- How does score-compliance pairing differ methodologically from the format-compliance/accuracy disentanglement proposed by Long et al. (2025)?
- The decoding configuration (greedy vs. sampling, temperature, top-p) is not reported for either the evaluated VLMs or the LLM judges. Since format compliance is the central measure, did the authors test whether the results, especially Cross-Format Substitution rates, are sensitive to the decoding strategy?
2 comments
Thank you for the authors' response. The response addressed my concerns and I will maintain my rating.
We thank the reviewer for the careful reading. We address the questions on validation, model coverage, and reproducibility.
W1. We conducted two new analyses to validate the framework's separation of format from content quality, beyond the judge reliability established by the human study (r=0.84 for structure).
First, the failure taxonomy provides diagnostic classification independent of the LLM judges. At C3, Cross-Format Substitution accounts for 95.67% of MiniGPT-4's arrow responses (Table 3). These are not reasoning failures. The model generates valid causal content but in numbered-list notation instead of arrows. A compliance score alone classifies all of them as failures without distinguishing incorrect causal relationships from structures expressed in another format. The taxonomy recovers this distinction.
We also ran a new compliant-only analysis, restricting L3 C3 structural scores to format-compliant responses:
The results show that even among compliant responses, adjacency content quality remains well below linearized content quality for every model. The gap is in the causal content, not in judge agreement or compliance rates.
On CELLO, MM-CoT, and CausalVLBench, we have not run direct comparisons. These benchmarks evaluate causal reasoning using a single structural representation per task. Rankings from such benchmarks may reflect format-following ability alongside causal-content quality. The MiniGPT-4 example above illustrates why. A comparison showing whether models that rank differently on CELLO also show different compliance-content profiles in our framework would be informative. We will discuss this in the camera-ready version.
W2. We agree that API-based models are feasible for Layers 1-3 and would strengthen the generalizability claim. Qwen3-VL provides an initial test: at C3, its compliance reaches 97.18% for arrows, 97.77% for semicolons, and 97.77% for numbered lists, but falls to 44.24% for adjacency lists, with Cross-Format Substitution accounting for 53.02% of adjacency responses (Table 3). Improved linearized adherence does not eliminate difficulty with parent-child representations. We do not extend these rates to untested frontier models. We will note API model evaluation as future work in the camera-ready version.
Q1. Addressed under W1. We have not run the six models on CELLO or CausalVLBench and will discuss the relationship in the camera-ready version.
Q2. Long et al. (2025) study the separation between task accuracy and format compliance for LLM classification tasks (multiple-choice, true/false, extraction) with binary compliance measures. We build on this distinction through experiments on VLM causal-response generation. Our framework adds structure-first generation, continuous 0-10 quality scoring for both structure and text, a five-category failure taxonomy, and fine-tuning experiments. The taxonomy captures phenomena specific to generative output. Cross-Format Substitution has no analog in classification settings where the output is a label. (We note that the first author's family name is Do. We will correct the citation in the camera-ready version.)
Q3. Inference configurations are as follows. LLaVA-RLHF uses temperature=0.2, top-p=0.6. MiniGPT-4 uses temperature=0.1, top-p=0.9. Qwen3-VL uses checkpoint defaults of temperature=0.7, top-p=0.8, top-k=20. LLaVA-NeXT, LLaVA-CoT, and CogVLM2 use generation defaults without explicit sampling overrides. All models use max_new_tokens=512. We did not standardize decoding settings across models, and we have not tested alternative configurations. Cross-Format Substitution rates may depend partly on decoding strategy, and we will note this as a limitation alongside the full configuration table in the camera-ready version.