acceptodds
Under review as a conference paper at ICLR 2027

R-CARE Reliability-Aware Concept Graph Conditioning and Evidence Verification for Visual Figurative Language Reasoning

Abstract

Visual figurative language understanding requires multimodal models to reason beyond the literal content of images and text by capturing implicit relations among visual cues, linguistic concepts, and figurative meanings. However, existing methods remain limited in structuring cross-modal figurative relations and in mitigating noise and semantic bias in automatically generated auxiliary knowledge. Moreover, explanations that are fluent and highly probable under the model may still be inconsistent with the multimodal evidence in the input. To address these challenges, we propose R-CARE (Reliability-aware Concept graph Adaptive Rhetorical Reasoning with Evidence Verification), a framework for visual figurative language reasoning. R-CARE first constructs a structured rhetorical concept graph that explicitly organizes visual entities, textual concepts, and potential figurative meanings. It then performs concept graph conditioned decoding, adaptively integrating graph structure and fine-grained semantic information based on the multimodal context while constraining their influence to mitigate interference from unreliable knowledge. Furthermore, R-CARE decouples label prediction from explanation verification. Given a fixed predicted label, it generates multiple candidate explanations, evaluates their consistency with the multimodal evidence using the model's internal multimodal representations, and selects the most evidence-consistent candidate subject to a generation-likelihood constraint. On the V-FLUTE benchmark, R-CARE achieves F1@0, F1@53, and F1@60 scores of 89.74, 78.14, and 60.85, respectively, surpassing MAPPER by 5.20, 8.45, and 9.13 percentage points. Code is available at: https://github.com/dahfs/R-CARE

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Figurative Language Reasoning, Multimodal Reasoning, Concept Graphs, Reliability-Aware Conditioning, Evidence Verification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jmBAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies visual figurative language reasoning, where a model predicts the relation between an image and a claim and generates an explanation. R-CARE constructs a rhetorical concept graph from an image description and the claim, representing visual entities, textual concepts, figurative anchors, and their relations. Its RACD component conditions a bounded low-rank readout on structural and semantic graph information, while DPEV first selects a relation label and then chooses among explanations generated under that fixed label using an internal evidence score and a generation-likelihood constraint. The paper evaluates the system on V-FLUTE, reporting improvements over the main baselines, component ablations, analyses of residual modulation and candidate selection, three seed configurations, and a human evaluation of final outputs. These results demonstrate gains in the reported task metrics; the proposed interpretation in terms of more reliable knowledge use and better evidence-grounded explanation selection is assessed separately below.

Critical strengths
  1. The reported gains remain substantial against a meaningful description-enhanced internal baseline. Table 2 shows that LLaVA+CD already improves over LLaVA-SFT, yet the full system further improves F1@0, F1@53, and F1@60 by 4.64, 6.78, and 6.80 percentage points. Table 1 also reports gains of 5.20, 8.45, and 9.13 points over MAPPER. Appendix D reports three seed configurations with means of 89.58 ± 0.22, 77.95 ± 0.16, and 59.79 ± 1.20. These results provide a substantive empirical reason to consider the work and make it inappropriate to attribute all gains merely to adding image descriptions. They do not, by themselves, establish the significance of each smaller component gain.

  2. The auxiliary representation is well aligned with figurative reasoning. Section 3.3 and Appendix A explicitly organize rhetorical anchors, conceptual mappings, and visual-evidence links alongside visual entities and claim concepts. This addresses the gap between recognizing literal content and interpreting nonliteral meaning. The construction prompt and worked graph example make the intended knowledge representation inspectable. The value here is the task-specific formulation; the independent benefit of graph structure over the same semantic content still needs clearer isolation.

  3. The bounded readout provides a precise and interpretable control on knowledge injection. Eqs. (8)–(9) constrain the magnitude of the injected residual through norm clipping followed by bounded, nonnegative token gating. Under nonnegative values of the residual-scale coefficient and gate cap, the Euclidean norm of the applied perturbation cannot exceed the norm of the original hidden state multiplied by those two coefficients. Figure 2 examines the residual before and after clipping and gating. This provides a concrete mechanism for limiting representation changes. It does not guarantee factual correctness, evidence grounding, or prediction stability.

  4. DPEV has a coherent conservative selection rule and an explicit quality–cost trade-off. Section 3.5 and Eqs. (10)–(18) distinguish label selection from explanation selection, require evidence-score improvement while constraining generation likelihood, and retain the default explanation when no candidate qualifies. Appendix B reports both performance and time as the candidate count changes. The nonmonotonic F1@60 results are informative about the limits of increasing search. This design is useful for applications where the explanation is to be refined while keeping the explicitly selected decision fixed.

  5. The verifier’s same-label, locally edited negative construction targets a relevant failure mode. Section 3.6, Eq. (20), and the supplied warrant-replacement code construct negatives without relying solely on changing the relation label. Replacing a local explanatory component while retaining the label and surrounding context is a plausible way to teach the verifier to assess input-specific support. This is better aligned with the intended within-label selection task than a supervision signal that could be solved through label disagreement alone. The remaining issue is the quality of the resulting negatives, discussed below.

  6. The human evaluation provides useful evidence about the final system beyond automatic metrics. Appendix E and Table 4 describe 99 stratified examples, two raters, a scoring rubric, agreement statistics, and bootstrap intervals. The overall normalized score of 63.01, with interval [56.19, 69.57], and the variation across groups help expose the system’s limitations. This supports a bounded claim about the quality of final outputs and improves the transparency of the empirical evaluation. It should not be interpreted as an accuracy percentage or as a paired demonstration that DPEV improves grounding.

Critical weaknesses
  1. The stated conditioning equations do not clearly match the final controller construction in the supplied implementation. Section 3.4, Eqs. (5)–(6), uses a three-way softmax over relation, structure, and semantic states and includes the elementwise absolute difference between the structural and semantic states and their elementwise product in the condition. In contrast, the final inference construction inherits ClassificationPreservingController.encode_condition from rgc_semantic_readout_stage_3.py, which uses independent bounded sigmoid gates, combines gated graph and semantic residuals with the relation state through LayerNorm, and constructs the condition using the residuals and interactions between the relation state and the resulting fused state. This is a concrete description–implementation discrepancy, not a demonstrated mathematical impossibility or evidence that the results are false. Nevertheless, it prevents a clear determination of which architecture the main results validate. The model class, configuration, and checkpoint provenance for Tables 1–3 should be identified and reconciled with the equations.

  2. The implemented training procedure contains an additional parameter-level constraint that is not adequately described by the stated objective. The final staged training code includes ReliabilityTrustController.parameter_trust_loss in rgc_semantic_readout_stage_8.py and adds it through the model’s training forward path. It measures semantic-readout parameter drift relative to a reference checkpoint and applies a dual-weighted and quadratic penalty when a trust radius is exceeded. This differs from the activation-level residual clipping in Eq. (8), but its role is not apparent from Section 3.6 and Eq. (19). Without the actual configuration and logs, it is not possible to establish whether the penalty was active or consequential. The concern is that the reliability mechanisms and training budget associated with the reported results are incompletely specified. A stage-by-stage objective and configuration table would make this fixable issue much easier to assess. This concern is related to the architecture correspondence issue above and should not be counted as a wholly separate failure.

  3. The ablations do not yet isolate the contribution of graph structure and reliability control from the contribution of additional semantic knowledge. Table 2 establishes improvements beyond the description-enhanced baseline, which is valuable. However, removing RACD also changes access to and processing of the graph-derived information. A comparison using the same graph content through a simpler representation or conditioning mechanism would more directly test the claimed value of the proposed knowledge-use mechanism. Likewise, Figure 2 establishes bounded modulation but does not show that incorrect graph knowledge is selectively suppressed. This matters because the paper’s contribution is framed around reliable incorporation of structured knowledge, rather than only the usefulness of extra generated semantic content. The configuration matching for the recomputed MAPPER baseline should also be made explicit; the current evidence is insufficient to conclude that the comparison is unfair.

  4. The evidence does not directly establish that DPEV selects more grounded explanations within a candidate set. Figure 3(b) reports a cross-instance association between internal scores and ExplanationScore for default explanations, including a Spearman correlation of 0.214. Such an association can reflect differences in example difficulty and does not establish within-input candidate ranking. ExplanationScore also does not directly measure support from the image and claim. Appendix E evaluates final outputs without a paired default-versus-selected comparison, and its overall rubric combines multiple dimensions. Consequently, the central interpretation that DPEV improves evidence grounding is less directly supported than the aggregate task-metric improvements. A blinded paired assessment using the same fixed labels and candidate pools, particularly among the 204 replacements reported in Figure 3, could substantially address this concern.

  5. The semantic validity of the verifier’s constructed negatives is not sufficiently characterized. The same-label local replacement procedure is sensible, but the supplied materials do not establish how often a replacement truly removes support from the current input, how many usable pairs remain after filtering, or whether textual artifacts provide an easier discrimination cue. These are unresolved possibilities, not confirmed failures of the training data. They matter because the interpretation of the learned score as evidence consistency depends on the supervision. A modest audit of constructed pairs, with counts and representative examples, could strengthen the claim without requiring a new benchmark. This concern is connected to the previous one and should not be double-counted as an independent lack of grounding evidence.

  6. The provided supplementary package leaves important custom dependencies unresolved. The supplied train.py references train_moe_创新点1.py at line 11328 and test_moe_创新点1.py at line 9822; its loaders look for these scripts in the external project directory specified by RIFT_ORIG_ROOT. Neither script is included in the provided archive, so their implementations could not be inspected. The evaluation wrapper also depends on an external compute_metrics_bscore_bleurt.py. The feature and data preparation steps also require resources not fully reconstructible from the supplied examples. The package contains useful readable source, but the available materials do not currently establish a complete path from the benchmark data to the reported scores. This is more consequential than the absence of large model weights alone because it limits verification of the implemented model and evaluation procedure. The concern applies to the provided archive; it is not a claim that every external resource maintained by the authors is unavailable.

Questions for authors
  1. Which exact controller produced the main results? Please map Tables 1–3 to the actual model class, configuration, and checkpoint lineage, and explain the correspondence between Eqs. (5)–(6) and the gated-residual implementation of ClassificationPreservingController.encode_condition. If the implementation is the evaluated method, a corrected equation and an explicit mapping would largely resolve the architecture-identification concern. If a different implementation produced the results, please identify and provide that version.

  2. What objectives and parameter constraints were active in each training stage? Please specify the trainable and frozen modules, learning rates, number of epochs, initialization checkpoints, and auxiliary losses for each stage. In particular, was parameter_trust_loss enabled in the reported runs, and did it become nonzero? A configuration and a compact summary of the relevant logs would clarify whether the paper’s stated objective adequately describes the evaluated system and would change the assessment of the training-specification concern.

  3. How much of RACD’s gain remains when the auxiliary semantic content is held fixed? Is there an existing comparison that supplies the same graph-derived content through text or a simpler conditioning mechanism? Please also clarify the backbone, image-description cache, preprocessing, and training conditions used for the recomputed MAPPER result. A modest matched-content comparison, or directly relevant existing evidence, would help determine whether the proposed structure and reliability mechanism contribute beyond the additional information itself.

  4. Does DPEV improve input support when the label and candidate pool are held fixed? Using the label selected by Eq. (10) and the same three-candidate candidate pool, can you compare the default explanation with the selected explanation in a blinded paired audit? A representative subset of the reported replacements is sufficient if accompanied by the sampling procedure and counts of improved, worse, and tied cases. Evidence that the selections improve input support, rather than only reference similarity, would directly strengthen the central DPEV claim.

  5. How reliable are the constructed same-label negative pairs? Please report the number of eligible pairs after filtering and provide a small audit distinguishing unsupported substitutions, substitutions that remain plausible or supported, and cases with obvious editing artifacts. A low rate of semantic mislabeling and little dependence on artifacts would increase confidence that the verifier learns evidence consistency rather than an easier textual discrimination task.

  6. Can the custom dependencies and data preparation be made unambiguous? Please provide versioned sources or include the project-specific modules required by RIFT_ORIG_ROOT, the evaluation implementation, and the feature-generation procedure. For the generated graphs, either the actual caches or a sufficiently specified generation procedure would help identify the experimental inputs. Resolving these dependencies would materially improve the ability to verify the architecture, metrics, and reported configuration.

  7. How were the candidate count, selection thresholds, and checkpoints chosen? Please identify the data split used to select the candidate count, the evidence-score margin, the generation-likelihood tolerance, and the final checkpoints, and distinguish selection experiments from post hoc test-set analyses. Appendix B’s test-set curve does not by itself establish test-set tuning. A clear validation-based selection account would remove this uncertainty; a different procedure would affect the interpretation of the reported generalization results.

  8. Do training and inference use the same explanation-state region under the actual tokenizer? The training explanation_state_mask uses shifted labels and a fixed label-token cutoff, whereas the inference explanation_token_mask identifies the explanation boundary from decoded output and excludes special tokens. Please show token IDs, EOS handling, and selected hidden-state indices for both relation labels. The differing rules may select equivalent regions for some tokenizations, so this is a clarification request rather than a confirmed off-by-one error. If they select different regions in the reported setup, an assessment of the effect on verifier scores would determine whether the difference is consequential.

3 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

Q1:Which exact controller produced the main results?

Thank you for raising this important question regarding implementation clarity and reproducibility. To clarify, all reported main results are produced by the gated-residual controller, implemented as ReliabilityTrustController, constructed through make_v8_readout_classes with ReliabilityTrustReadoutConfig. It inherits ClassificationPreservingController.encode_condition without overriding its implementation.

For clarity, we will make the correspondence between Eqs. (5)–(6) and the actual implementation more explicit. Specifically, given a=[hr;hg;hs;gstr]a=[h_r;h_g;h_s;g_{\mathrm{str}}], the independent graph and semantic gates are defined as bj=0.03+0.32σ(wj⊤a+cj)b_j=0.03+0.32\sigma(w_j^\top a+c_j), where j∈{g,s}j\in\{g,s\}. The corresponding residuals are Δg=bgϕg(hg)\Delta_g=b_g\phi_g(h_g) and Δs=bsϕs(hs)\Delta_s=b_s\phi_s(h_s), followed by hf=LN(hr+Δg+Δs)h_f=\mathrm{LN}(h_r+\Delta_g+\Delta_s). The final condition representation is obtained as z=fcond([hr;hf;Δg;Δs;∣hr−hf∣;hr⊙hf])z=f_{\mathrm{cond}}([h_r;h_f;\Delta_g;\Delta_s;|h_r-h_f|;h_r\odot h_f]). Here, ϕg\phi_g and ϕs\phi_s are separate Linear–GELU–LayerNorm projections. These operations correspond directly to graph_reliability/semantic_reliability, graph_residual/semantic_residual, fused_norm, and condition_encoder in the implementation. The saved configuration specifies a relation/condition dimension of 256, readout rank of 32, residual norm ratio of 0.06, and readout gate cap of 0.12.

Regarding checkpoint lineage, Table 1 (R-CARE), Table 2 (Full), and Table 3 (Main) use the same checkpoints_llava_final_combined_r4/model.pt checkpoint. The two Table 2 ablations, w/o DPEV and w/o RACD, use the innovation1_r4 and innovation2_r4 checkpoints, respectively. The additional Table 3 runs follow the same controller implementation with seed pairs (123, 123) and (456, 456).

We will further clarify the controller formulation, implementation mapping, and checkpoint lineage in the revised manuscript. To facilitate reproducibility, we also plan to release the corresponding model weights, configuration files, and implementation details upon publication. These clarifications make the evaluated architecture and its experimental lineage more transparent without changing the reported results.

Q2:What objectives and parameter constraints were active in each training stage?

Thank you for requesting this clarification. R-CARE is trained through four sequential stages, with the pretrained LLaVA backbone frozen throughout. Stage 1 (Joint Adaptation): Initialized from LLaVA-v1.6-Vicuna-7B, LoRA and RACD are trained for 3 epochs with learning rates of 2×10−42\times10^{-4} and 1×10−41\times10^{-4}, respectively. Stage 2 (Controller Refinement): Initialized from Stage 1, LoRA is frozen and the controller is trained for 1 epoch at 1×10−51\times10^{-5}, with adaptive soft-negative and tail weighting. Stage 3 (Semantic Readout Refinement): Initialized from Stage 2, only the semantic readout and alignment projections are trained for 1 epoch at 5×10−65\times10^{-6}. Stage 4 (Verifier Training): Initialized from Stage 3, the generator, RACD, and upstream evidence encoders are frozen, and only the verifier is trained for 2 epochs at 2×10−42\times10^{-4}.

Stages 1–2 optimize GroupDRO generation, label prediction (0.4), counterfactual contrast (0.02), and semantic alignment (0.03). Stage 3 uses GroupDRO, label prediction, and decoder alignment (0.015), together with the configured parameter_trust_loss (radius 5×10−45\times10^{-4}). Importantly, although enabled, this constraint remained inactive: its penalty and dual multiplier were zero, and the maximum logged RMS parameter drift was only 5.54×10−55.54\times10^{-5}. Stage 4 optimizes the verifier ranking-and-anchor objective (weight 0.2). Eq. (19) describes Stage 3 rather than the entire training procedure. We will clarify the stage-wise objectives and configurations in the revised manuscript, and release the corresponding configurations, logs, and model weights upon publication.

Q3:How much of RACD’s gain remains when the auxiliary semantic content is held fixed?

Thank you for this valuable suggestion. We conducted matched-content comparisons using both LLaVA and Qwen3-VL-8B-Instruct. On LLaVA, with DPEV disabled, RACD outperforms Graph-as-Text by 1.76, 1.36, and 0.13 points on F1@0, F1@53, and F1@60, respectively. On Qwen3-VL-8B-Instruct, RACD with ordinary decoding improves F1@0 and F1@53 by 1.43 and 0.83 points over Graph-as-Text SFT, while complete R-CARE achieves gains of 2.27, 2.21, and 0.13 points across the three metrics. These results indicate that the improvements go beyond simply providing additional semantic information.

Detailed results are provided in Table 1 of our response to reviewer vtvs and Table 2 of our response to reviewer 3G8A. For MAPPER, we followed its reported architecture and training configuration; further reproduction details are provided in Rebuttal (2/2) to reviewer 3G8A.

AuthorsOpenReview ↗

Rebuttal (2/3)

Q4:Does DPEV improve input support when the label and candidate pool are held fixed?

Thank you for this valuable suggestion. To examine whether DPEV genuinely relies on input visual evidence, we conducted controlled image-swapping experiments while keeping predicted labels and candidate explanations fixed. Replacing only the verifier's image-evidence representation reduced the evidence score, while swapping images throughout the scoring pipeline decreased scores in 95.99% of examples and changed explanation selection in 24.20% of cases. These results demonstrate that DPEV is sensitive to input visual evidence rather than relying solely on textual information. Furthermore, our fixed-label and identical-candidate-pool comparison shows that DPEV improves explanation quality over default selection. The corresponding quantitative results are provided in Table 2 of our response to reviewer vtvs. We acknowledge that a blinded human paired audit would provide complementary evidence of input-support quality and leave this evaluation for future work.

Q5:How reliable are the constructed same-label negative pairs?

Thank you for raising this concern. Of the 4,578 training instances, 2,877 yield eligible explanation spans, and 2,854 unique same-label negative pairs remain after donor matching and the runtime length-ratio filter. Each negative replaces an exact span with one from a different training instance matched by label, partition, and rule family, while preserving the label and surrounding explanation. These constraints establish structural eligibility but do not guarantee that every substitution is semantically unsupported. To assess this distinction, we have prepared a fixed random sample of 100 pairs for independent two-annotator review, recording semantic support and editing artifacts separately because they can overlap. Our preliminary surface-feature diagnostic achieves 73.13% classification accuracy, indicating that superficial cues are learnable in the constructed pairs. This does not establish that the multimodal verifier relies on those cues, but it prevents us from claiming negligible artifact dependence. We therefore characterize these pairs as weakly supervised mismatches rather than guaranteed semantic counterfactuals.

Q6:Can the custom dependencies and data preparation be made unambiguous?

Thank you for highlighting this reproducibility issue. We have added a public, versioned reproducibility supplement that provides the project-specific dependencies and modules, evaluation code and configurations, feature-generation procedures, and actual feature and graph caches with SHA256 checksums. We also document the preprocessing pipeline and metric computation, with the archived per-instance scores reproducing all 101 thresholds for the three main branches.The released feature tensors and graph caches serve as the authoritative inputs for the reported experiments.

Q7:How were the candidate count, selection thresholds, and checkpoints chosen?

Thank you for highlighting this ambiguity. We clarify the configuration and checkpoint selection procedures as follows.

•Candidate count: K=3K=3 was inherited from the existing three-beam generation configuration used in v16, rather than selected through the subsequent Appendix B test-set analysis. No documented validation-based selection was performed.

•Selection thresholds: The evidence margin (0.05) was adopted as a conservative heuristic based on an earlier implementation, while the likelihood tolerance (0.10 nat per explanation token) was specified for the v16 run. Neither was selected through a documented validation search or calibration.

•Checkpoints: All reported R4 checkpoints correspond to the terminal states of fixed training schedules (3/1/1/2 epochs across the four stages), without validation- or test-based epoch selection. The verifier was trained on the full training set, including the 445-example internal development partition.

•Data splits: The official 726-example validation split was not used for selecting these configurations or checkpoints. Appendix B reports subsequent test-set sensitivity analyses, not the original selection procedure.

We acknowledge that test-set metrics were consulted when comparing earlier method iterations. Therefore, although the reported settings and checkpoints were not selected through test-based optimization, we cannot claim that the entire development process was independent of test feedback. We will clarify these distinctions and appropriately qualify our generalization claims.

AuthorsOpenReview ↗

Rebuttal (3/3)

Q8:Do training and inference use the same explanation-state region under the actual tokenizer?

Thank you for this careful clarification request. We audited the actual tokenizer and confirmed that the training and inference masks select slightly different hidden-state regions in the reported implementation.

The historical LLaVA loader uses LlamaTokenizer (use_fast=False, legacy=False, EOS ID = 2). The actual tokenization and selected response-local hidden states are summarized below, where NN denotes the number of non-EOS response tokens.

LabelPrefix Token IDsTraining MaskInference Mask
Entailment[875, 737, 358, 29889]3,…,N−13,\ldots,N-14,…,N−14,\ldots,N-1
Contradiction[23949, 29889]3,…,N−13,\ldots,N-12,…,N−12,\ldots,N-1

Thus, the training mask includes the label-ending period state for entailment but excludes the first explanation-token state for contradiction. The terminal boundary is consistent: training supervises EOS, retaining the final explanation state that predicts EOS, while excluding the EOS-token state itself.

We further audited 4,578 positive training sequences, 2,854 eligible negative sequences, and 2,164 test candidates. To quantify the impact, we fixed the checkpoint, candidate hidden states, evidence representations, labels, likelihoods, and selection thresholds, changing only the scoring mask. Across the test candidates, the mean, median, and maximum absolute evidence-score changes were 0.009563, 0.008218, and 0.062120, respectively. The final explanation selection changed in only 1 of 723 examples (0.138%), with all predicted labels preserved. In this case, the candidate's evidence gain decreased from 0.051499 to 0.049682, crossing the fixed 0.05 replacement threshold.

These results confirm a genuine but limited training–inference mask discrepancy, with measurable effects on evidence scores but minimal impact on final explanation selection in the evaluated test set. We will clarify the tokenization and masking procedures in the revised manuscript. We note that this fixed-checkpoint intervention does not establish the effect of retraining with a unified mask or guarantee unchanged explanation quality.

Reviewer_3G8AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes R-CARE for visual figurative language reasoning. The method constructs rhetorical concept graphs to capture relationships between images and text, introduces reliability-aware graph-conditioned decoding (RACD), and uses evidence verification (DPEV) to select more reliable explanations. Experiments on V-FLUTE show improvements over existing methods in both prediction accuracy and explanation quality.

Critical strengths
  1. Improving figurative language understanding and explanation reliability is a meaningful challenge in multimodal reasoning.

  2. The method achieves clear improvements over MAPPER on V-FLUTE. The ablation studies also demonstrate the contributions of the proposed components.

Critical weaknesses
  1. Insufficient comparisons with recent models. Most baseline models are relatively outdated, such as LLaVA-1.6, GPT-4o, and Gemini-1.5-Pro. Comparisons with stronger recent multimodal models and simpler knowledge-augmented baselines are missing. This makes it difficult to assess the practical advantage and necessity of the proposed method.

  2. High complexity with insufficient justification. The framework introduces many components, mathematical formulations, and hyperparameters, but provides limited sensitivity analysis. It is unclear whether such a complex design is necessary. Moreover, the method presentation is somewhat difficult to follow, with excessive mathematical details and insufficient correspondence between the framework figure and the text. The evidence for improved explanation reliability is also limited.

Questions for authors
  1. Can the authors compare R-CARE with stronger recent multimodal models and simpler baselines using the same external knowledge? This would help clarify whether the improvements come from the proposed architecture or the additional knowledge.

  2. How sensitive is the method to key hyperparameters, particularly those controlling knowledge injection and evidence verification? Are the improvements stable across different settings?

  3. DPEV is designed to preserve the predicted label during explanation verification. However, removing DPEV changes F1@0 from 89.74 to 88.58 in Table 2. Could the authors clarify whether the ablation also changes the label prediction procedure or training configuration? If so, how can the contribution of explanation verification be isolated?

4 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Q1: Can the authors compare R-CARE with stronger recent multimodal models ... whether the improvements come from the proposed architecture or the additional knowledge?

Thank you for this valuable suggestion. We extended our evaluation in two directions: comparison with a recent task-specific method, RIFT [1], and controlled comparisons using the stronger multimodal backbone Qwen3-VL-8B-Instruct [2] with identical external concept graphs.

First, compared with the published V-FLUTE results of RIFT, R-CARE achieves improvements of 1.22, 2.07, and 1.38 percentage points on F1@0, F1@53, and F1@60, respectively, using the same LLaVA-v1.6-Vicuna-7B backbone.

Second, we conducted controlled comparisons among ordinary SFT, Graph-as-Text SFT, and RACD-based variants using Qwen3-VL-8B-Instruct. Notably, simply appending concept graphs does not improve performance over ordinary SFT. In contrast, compared with Graph-as-Text SFT, the complete R-CARE framework improves F1@0, F1@53, and F1@60 by 2.27, 2.21, and 0.13 percentage points, respectively. These results suggest that the improvements arise not merely from additional external knowledge, but also from its structured integration and evidence-aware verification. We note that F1@60 is slightly lower than that of the label-routing variant, indicating that DPEV does not uniformly improve all explanation thresholds.

MethodF1@0F1@53F1@60
RIFT [1]88.5276.0759.47
R-CARE (Ours)89.7478.1460.85
MethodGraphF1@0F1@53F1@60
Ordinary SFTNo90.0879.3966.80
Graph-as-Text SFTYes89.7878.8466.80
RACD (Ordinary Decoding)Yes91.2179.6766.65
RACD + DPEV Label RoutingYes92.0580.5067.34
RACD + Complete DPEVYes92.0581.0566.93

[1] Xiaojian Lou, Qimeng Yang, Hui Song, Yunjing Jiang, Jinmiao Song, and Xinjun Pei. 2026. RIFT: Relation Modeling and Transfer for Visual Figurative Entailment. Findings of the Association for Computational Linguistics: EMNLP 2026, pp. 13711–13725.

[2] Shuai Bai et al. 2025. Qwen3-VL Technical Report. arXiv preprint arXiv:2511.21631.

Q2: How sensitive is the method to key hyperparameters ...? Are the improvements stable across different settings?

Thank you for raising this important question. We conducted a controlled sensitivity analysis using the same frozen R-CARE checkpoint, independently varying the key hyperparameters of knowledge injection and evidence verification.

For knowledge injection, we perturbed the token-gating cap, residual norm bound, and channel-modulation amplitude by ±50%. Across all seven configurations, the predicted labels remained identical on all 723 test examples, with F1@0 consistently at 89.74. F1@53 ranged from 77.45 to 78.27, while F1@60 ranged from 59.75 to 60.85, indicating relatively stable performance despite moderate variations in explanation quality.

For evidence verification, we evaluated all 25 combinations of the evidence margin (τe\tau_e) and likelihood slack (τℓ\tau_\ell), keeping predicted labels, candidate pools, and candidate scores fixed. All 20 configurations with nonzero likelihood slack improved both explanation-aware metrics over the same-pool default explanation, with gains of 0.14–0.55 points on F1@53 and 0.13–0.55 points on F1@60. In contrast, the five zero-slack configurations matched the default baseline. Notably, at the default evidence margin (τe=0.05\tau_e = 0.05), all four positive-slack settings selected exactly the same explanations.

Overall, these results demonstrate that R-CARE maintains stable performance across a broad range of tested hyperparameter settings, while also revealing moderate sensitivity in explanation quality. This analysis characterizes inference-time robustness under fixed trained weights.

SettingF1@0F1@53F1@60
Default89.7478.1460.85
Gating cap (-50%)89.7477.7259.75
Gating cap (+50%)89.7477.4560.02
Norm bound (-50%)89.7477.7259.75
Norm bound (+50%)89.7477.5959.89
Channel modulation (-50%)89.7478.2760.58
Channel modulation (+50%)89.7478.0060.71
AuthorsOpenReview ↗

Rebuttal (2/2)

Q3: DPEV is designed to preserve the predicted label during explanation verification ... how can the contribution of explanation verification be isolated?

Thank you for pointing out this ambiguity. The original ``w/o DPEV'' ablation removes both explicit label selection and explanation verification, reverting to conventional joint label-and-explanation generation. Therefore, the F1@0 decrease from 89.74 to 88.58 is caused by the change in label prediction procedure, rather than explanation verification itself. Both variants follow the same generator and RACD training protocol, while the full model additionally trains the verifier with the generator and RACD frozen.

To isolate the contribution of explanation verification, we conducted a controlled comparison using the same trained model, fixed predicted labels, and identical candidate explanation pools, varying only the selection strategy. All 723 predicted labels remain unchanged, yielding identical F1@0 (89.74). Compared with default explanation selection (89.74/77.72/60.44), DPEV achieves 89.74/78.14/60.85 on F1@0/F1@53/F1@60, improving the latter two metrics by approximately 0.41 percentage points each. This controlled comparison confirms that DPEV improves explanation quality independently of label prediction. We will clarify the scope of the original ablation and include this additional comparison to better distinguish the contributions of label selection and explanation verification.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns. The new results provide useful clarifications and partially strengthen the empirical evidence, but my main reservations remain: the framework is relatively complex, while the incremental benefits, particularly for explanation reliability, are limited.

Thank you for your thoughtful feedback and for taking the time to review our rebuttal. We sincerely appreciate your consideration of our additional experiments and clarifications.

Reviewer_vtvsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes R-CARE, a framework for visual figurative language understanding. The method first constructs a rhetorical concept graph using a large language model, then integrates the structural and semantic information from the graph into LLaVA via Reliability-Aware Concept Graph Conditioned Decoding (RACD), and finally scores and selects among multiple candidate explanations through Decision-Preserving Endogenous Multimodal Evidence Verification (DPEV). Experiments on the V-FLUTE benchmark achieve F1@0, F1@53, and F1@60 scores of 89.74, 78.14, and 60.85, respectively, outperforming the MAPPER baseline reported in the paper.

Critical strengths
  1. The paper addresses important challenges in visual figurative language understanding—specifically, the potential unreliability of automatically generated external knowledge and the lack of evidence grounding in generated explanations. The research motivation is sound, and the overall framework is well-structured.

  2. The empirical results are strong, surpassing compared baselines across all three primary metrics. The authors also provide module ablations, multi-seed evaluations, inference cost analyses, and human evaluations, offering relatively broad experimental coverage.

Critical weaknesses

W1. Limited methodological novelty. Concept graph augmentation, dynamic gating, low-rank adaptation, and candidate explanation reranking have all been explored in prior literature. This work primarily combines these existing techniques into a pipeline, but lacks comparisons with simpler alternatives, making it difficult to judge whether such a complex architectural design is truly necessary.

W2. Insufficient validation of RACD's contribution. The paper relies on ChatGPT to generate rhetorical concept graphs, which already contain explicit interpretations of the figurative meanings. However, the experiments do not compare against a baseline that directly feeds these graphs as plain text into LLaVA, nor do they evaluate performance under random or corrupted graphs. Consequently, it is unclear whether the performance gains stem from the external knowledge provided by the powerful LLM or from the proposed RACD mechanism itself. Furthermore, fine-grained ablations isolating dynamic routing, norm clipping, and token-level gating are missing.

Questions for authors
  1. What is the performance if the ChatGPT-generated concept graphs are serialized and fed directly into LLaVA as plain text? Could the authors also provide ablations with random/corrupted graphs, as well as variants without token gating or norm clipping, to verify the necessity of RACD's specific design?

  2. Can the authors decouple label prediction from explanation reranking when evaluating DPEV? Specifically, under a fixed predicted label and an identical pool of candidate explanations, how does DPEV compare against selecting the default explanation or the highest-likelihood explanation?

  3. Could the authors provide more direct evidence that DPEV genuinely leverages visual information—for example, by swapping input images or perturbing visual facts in candidate explanations to observe whether the evidence score responds appropriately? A pairwise human evaluation comparing explanations before and after DPEV replacement would also be highly informative.

  4. Please clarify the exact reproduction setup for MAPPER, as well as the source and construction procedure of negative samples for training the DPEV verifier. These details are essential for assessing experimental fairness and reproducibility.

4 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We sincerely thank you for the detailed and constructive comments, which have helped us improve our work.

Q1: What is the performance if the ChatGPT-generated concept graphs are serialized and fed directly into LLaVA as plain text? ... Could the authors also provide ablations to verify the necessity of RACD's specific design?

Thank you for the valuable suggestion. We conducted additional ablation experiments to investigate the effects of graph serialization, graph reliability, token gating, and norm clipping, with DPEV disabled across all settings for a fair comparison. Compared with direct graph serialization, RACD achieves improvements of 1.76, 1.36, and 0.13 points on F1@0, F1@53, and F1@60, respectively. Replacing the original graphs with random graphs consistently degrades performance across all metrics, highlighting the importance of meaningful graph semantics. Furthermore, replacing adaptive token gating with a fixed coefficient reduces F1@53 and F1@60 by 0.31 and 1.16 points, respectively, despite a slight improvement in F1@0. Removing norm clipping leads to decreases of 0.72, 0.32, and 2.23 points across the three metrics. These results demonstrate the complementary roles of structured graph conditioning, adaptive token gating, and norm clipping in improving reasoning and explanation quality. The detailed results are presented below.

SettingF1@0F1@53F1@60
RACD (Original)88.5876.4659.74
Graph-as-Text86.8275.1059.61
Random Concept Graph87.5875.4658.60
Fixed Token Gate (0.04)88.7176.1558.58
w/o Norm Clipping87.8676.1457.51

Q2: Can the authors decouple label prediction from explanation reranking ... how does DPEV compare against selecting the default explanation or the highest-likelihood explanation?

Thank you for this valuable suggestion. To isolate the contribution of DPEV, we conducted a controlled comparison using the same frozen checkpoint, fixed predicted labels, and identical candidate explanation pools, varying only the explanation-selection strategy. All three strategies achieve identical F1@0 (89.74), confirming that label prediction remains unchanged. DPEV achieves the best F1@53 and F1@60 scores (78.14 and 60.85), outperforming both default beam-0 selection (77.72 and 60.44) and highest-likelihood selection (77.44 and 59.89). Specifically, DPEV yields improvements of 0.42/0.41 and 0.70/0.96 percentage points over these two baselines, respectively. These results demonstrate that DPEV improves explanation quality through evidence-aware reranking, independently of label prediction and candidate generation.

Selection StrategyF1@0F1@53F1@60
Default Beam-089.7477.7260.44
Highest Likelihood89.7477.4459.89
DPEV (Ours)89.7478.1460.85

Q3: Could the authors provide more direct evidence that DPEV genuinely leverages visual information ...? A pairwise human evaluation comparing explanations before and after DPEV replacement would also be highly informative.

Thank you for this valuable suggestion. We conducted controlled image-swapping experiments on all 723 test examples while keeping predicted labels, candidate explanations, and non-visual inputs fixed. Replacing only the verifier's image-evidence representation reduced the evidence score by an average of 0.01235 (95% CI: [0.00793, 0.01692]), demonstrating sensitivity to visual evidence. Swapping the image throughout the scoring pipeline resulted in a larger average decrease of 0.47770 (95% CI: [0.44408, 0.51173]), with scores decreasing in 95.99% of examples and explanation selection changing in 24.20% of cases. While the latter intervention also affects image-conditioned representations beyond the verifier, these results provide direct evidence that DPEV incorporates visual information rather than relying solely on textual cues. We also acknowledge the value of pairwise human evaluation and plan to investigate it in future work.

AuthorsOpenReview ↗

Rebuttal (2/2)

Q4: Please clarify the exact reproduction setup for MAPPER ... These details are essential for assessing experimental fairness and reproducibility.

Thank you for pointing out the need for further clarification. For MAPPER, we followed the architecture, prompts, and training settings reported in the original paper, using a frozen LLaVA-v1.5-7B describer and a LoRA-finetuned LLaVA-v1.5-7B thinker. The thinker was trained for three epochs with batch size 4, learning rate 2.5×10−52.5\times10^{-5}, LoRA rank 128, cosine scheduling, warmup ratio 0.03, and maximum sequence length 2048. Since the original repository does not provide the required evaluation script, we evaluated MAPPER and all other compared methods on the same 723 test instances using a unified implementation of F1@0/53/60, with explanation quality measured by the average of BERTScore and BLEURT. Thus, our reported MAPPER results are reproduced under a common evaluation protocol rather than copied from the original paper.

For DPEV, positive samples are the original training explanations, while negative samples are constructed exclusively from training data through same-label rationale-span replacement. Specifically, we extract reasoning spans using deterministic rules and replace them with spans retrieved from other training instances sharing the same relation label and reasoning-rule family. Retrieval is based on image–claim similarity, s=0.55cos⁡(EI,EI′)+0.45cos⁡(Ec,Ec′)s = 0.55\cos(E_I, E'_I) + 0.45\cos(E_c, E'_c), with a minimum similarity of 0.05, minimum span-length ratio of 0.5, and maximum lexical Jaccard similarity of 0.25. Only the selected span is replaced, preserving the original input, label, and remaining explanation. These automatically constructed silver negatives encourage evidence-sensitive discrimination while reducing label-related shortcuts, although they are not assumed to be human-verified errors. The generator remains frozen during verifier training, and label-prefix tokens are excluded from verifier scoring. Importantly, no test labels or reference explanations are used in negative construction.

Reviewer_vtvsOpenReview ↗

Final Assessment After Rebuttal

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns.

The new Graph-as-Text baseline and component ablations provide useful evidence for RACD, and the fixed-label, identical-candidate-pool comparison successfully separates DPEV's reranking contribution from label prediction. The clarification of MAPPER's reproduction setup and the negative sample construction is also helpful. I consider these concerns largely addressed.

However, I still have some reservations:

  1. The independent gains of the proposed mechanisms remain relatively limited. Compared with Graph-as-Text, RACD improves F1@0 and F1@53 by 1.76 and 1.36 points, respectively, but improves F1@60 by only 0.13 points. Similarly, DPEV improves F1@53 and F1@60 by only 0.42 and 0.41 points over default beam selection. While these are positive results, their magnitude raises questions about whether the additional architectural complexity is sufficiently justified.

  2. The evidence for improved explanation reliability remains insufficient. The image-swapping experiments demonstrate that DPEV is sensitive to visual information, but this does not necessarily establish that it can identify explanations containing incorrect or unsupported visual claims. A paired human evaluation comparing explanations before and after DPEV selection would provide more direct evidence. Moreover, the additional Qwen3-VL experiments show that DPEV does not consistently improve F1@60 across backbones.

  3. Some concerns about implementation consistency and experimental methodology remain. The clarification regarding the difference between the reported controller equations and the actual implementation is important, but the paper should accurately describe the architecture used in the experiments. I am also concerned by the acknowledgment that test-set results were consulted during earlier method development, while the official validation set was not used for configuration selection. Although this does not invalidate the reported results, it makes the generalization of small performance improvements more difficult to assess.

Overall, the rebuttal has meaningfully strengthened the experimental evidence and resolved several of my original concerns. Nevertheless, I remain unconvinced that the current results sufficiently demonstrate the advantages of the proposed complex framework, particularly regarding explanation reliability.

I therefore maintain my original rating of Weak Reject.

Thank you for your thoughtful feedback and for taking the time to review our rebuttal. We sincerely appreciate your consideration of our additional experiments and clarifications.

Comments on the paper

1 comment

Dear Reviewers,

The author-reviewer discussion period is approaching its end. Please take a moment to review the authors' rebuttal and confirm whether your concerns have been addressed.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.