- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies visual figurative language reasoning, where a model predicts the relation between an image and a claim and generates an explanation. R-CARE constructs a rhetorical concept graph from an image description and the claim, representing visual entities, textual concepts, figurative anchors, and their relations. Its RACD component conditions a bounded low-rank readout on structural and semantic graph information, while DPEV first selects a relation label and then chooses among explanations generated under that fixed label using an internal evidence score and a generation-likelihood constraint. The paper evaluates the system on V-FLUTE, reporting improvements over the main baselines, component ablations, analyses of residual modulation and candidate selection, three seed configurations, and a human evaluation of final outputs. These results demonstrate gains in the reported task metrics; the proposed interpretation in terms of more reliable knowledge use and better evidence-grounded explanation selection is assessed separately below.
Critical strengths
-
The reported gains remain substantial against a meaningful description-enhanced internal baseline. Table 2 shows that LLaVA+CD already improves over LLaVA-SFT, yet the full system further improves F1@0, F1@53, and F1@60 by 4.64, 6.78, and 6.80 percentage points. Table 1 also reports gains of 5.20, 8.45, and 9.13 points over MAPPER. Appendix D reports three seed configurations with means of 89.58 ± 0.22, 77.95 ± 0.16, and 59.79 ± 1.20. These results provide a substantive empirical reason to consider the work and make it inappropriate to attribute all gains merely to adding image descriptions. They do not, by themselves, establish the significance of each smaller component gain.
-
The auxiliary representation is well aligned with figurative reasoning. Section 3.3 and Appendix A explicitly organize rhetorical anchors, conceptual mappings, and visual-evidence links alongside visual entities and claim concepts. This addresses the gap between recognizing literal content and interpreting nonliteral meaning. The construction prompt and worked graph example make the intended knowledge representation inspectable. The value here is the task-specific formulation; the independent benefit of graph structure over the same semantic content still needs clearer isolation.
-
The bounded readout provides a precise and interpretable control on knowledge injection. Eqs. (8)–(9) constrain the magnitude of the injected residual through norm clipping followed by bounded, nonnegative token gating. Under nonnegative values of the residual-scale coefficient and gate cap, the Euclidean norm of the applied perturbation cannot exceed the norm of the original hidden state multiplied by those two coefficients. Figure 2 examines the residual before and after clipping and gating. This provides a concrete mechanism for limiting representation changes. It does not guarantee factual correctness, evidence grounding, or prediction stability.
-
DPEV has a coherent conservative selection rule and an explicit quality–cost trade-off. Section 3.5 and Eqs. (10)–(18) distinguish label selection from explanation selection, require evidence-score improvement while constraining generation likelihood, and retain the default explanation when no candidate qualifies. Appendix B reports both performance and time as the candidate count changes. The nonmonotonic F1@60 results are informative about the limits of increasing search. This design is useful for applications where the explanation is to be refined while keeping the explicitly selected decision fixed.
-
The verifier’s same-label, locally edited negative construction targets a relevant failure mode. Section 3.6, Eq. (20), and the supplied warrant-replacement code construct negatives without relying solely on changing the relation label. Replacing a local explanatory component while retaining the label and surrounding context is a plausible way to teach the verifier to assess input-specific support. This is better aligned with the intended within-label selection task than a supervision signal that could be solved through label disagreement alone. The remaining issue is the quality of the resulting negatives, discussed below.
-
The human evaluation provides useful evidence about the final system beyond automatic metrics. Appendix E and Table 4 describe 99 stratified examples, two raters, a scoring rubric, agreement statistics, and bootstrap intervals. The overall normalized score of 63.01, with interval [56.19, 69.57], and the variation across groups help expose the system’s limitations. This supports a bounded claim about the quality of final outputs and improves the transparency of the empirical evaluation. It should not be interpreted as an accuracy percentage or as a paired demonstration that DPEV improves grounding.
Critical weaknesses
-
The stated conditioning equations do not clearly match the final controller construction in the supplied implementation. Section 3.4, Eqs. (5)–(6), uses a three-way softmax over relation, structure, and semantic states and includes the elementwise absolute difference between the structural and semantic states and their elementwise product in the condition. In contrast, the final inference construction inherits
ClassificationPreservingController.encode_conditionfromrgc_semantic_readout_stage_3.py, which uses independent bounded sigmoid gates, combines gated graph and semantic residuals with the relation state through LayerNorm, and constructs the condition using the residuals and interactions between the relation state and the resulting fused state. This is a concrete description–implementation discrepancy, not a demonstrated mathematical impossibility or evidence that the results are false. Nevertheless, it prevents a clear determination of which architecture the main results validate. The model class, configuration, and checkpoint provenance for Tables 1–3 should be identified and reconciled with the equations. -
The implemented training procedure contains an additional parameter-level constraint that is not adequately described by the stated objective. The final staged training code includes
ReliabilityTrustController.parameter_trust_lossinrgc_semantic_readout_stage_8.pyand adds it through the model’s training forward path. It measures semantic-readout parameter drift relative to a reference checkpoint and applies a dual-weighted and quadratic penalty when a trust radius is exceeded. This differs from the activation-level residual clipping in Eq. (8), but its role is not apparent from Section 3.6 and Eq. (19). Without the actual configuration and logs, it is not possible to establish whether the penalty was active or consequential. The concern is that the reliability mechanisms and training budget associated with the reported results are incompletely specified. A stage-by-stage objective and configuration table would make this fixable issue much easier to assess. This concern is related to the architecture correspondence issue above and should not be counted as a wholly separate failure. -
The ablations do not yet isolate the contribution of graph structure and reliability control from the contribution of additional semantic knowledge. Table 2 establishes improvements beyond the description-enhanced baseline, which is valuable. However, removing RACD also changes access to and processing of the graph-derived information. A comparison using the same graph content through a simpler representation or conditioning mechanism would more directly test the claimed value of the proposed knowledge-use mechanism. Likewise, Figure 2 establishes bounded modulation but does not show that incorrect graph knowledge is selectively suppressed. This matters because the paper’s contribution is framed around reliable incorporation of structured knowledge, rather than only the usefulness of extra generated semantic content. The configuration matching for the recomputed MAPPER baseline should also be made explicit; the current evidence is insufficient to conclude that the comparison is unfair.
-
The evidence does not directly establish that DPEV selects more grounded explanations within a candidate set. Figure 3(b) reports a cross-instance association between internal scores and ExplanationScore for default explanations, including a Spearman correlation of 0.214. Such an association can reflect differences in example difficulty and does not establish within-input candidate ranking. ExplanationScore also does not directly measure support from the image and claim. Appendix E evaluates final outputs without a paired default-versus-selected comparison, and its overall rubric combines multiple dimensions. Consequently, the central interpretation that DPEV improves evidence grounding is less directly supported than the aggregate task-metric improvements. A blinded paired assessment using the same fixed labels and candidate pools, particularly among the 204 replacements reported in Figure 3, could substantially address this concern.
-
The semantic validity of the verifier’s constructed negatives is not sufficiently characterized. The same-label local replacement procedure is sensible, but the supplied materials do not establish how often a replacement truly removes support from the current input, how many usable pairs remain after filtering, or whether textual artifacts provide an easier discrimination cue. These are unresolved possibilities, not confirmed failures of the training data. They matter because the interpretation of the learned score as evidence consistency depends on the supervision. A modest audit of constructed pairs, with counts and representative examples, could strengthen the claim without requiring a new benchmark. This concern is connected to the previous one and should not be double-counted as an independent lack of grounding evidence.
-
The provided supplementary package leaves important custom dependencies unresolved. The supplied
train.pyreferencestrain_moe_创新点1.pyat line 11328 andtest_moe_创新点1.pyat line 9822; its loaders look for these scripts in the external project directory specified byRIFT_ORIG_ROOT. Neither script is included in the provided archive, so their implementations could not be inspected. The evaluation wrapper also depends on an externalcompute_metrics_bscore_bleurt.py. The feature and data preparation steps also require resources not fully reconstructible from the supplied examples. The package contains useful readable source, but the available materials do not currently establish a complete path from the benchmark data to the reported scores. This is more consequential than the absence of large model weights alone because it limits verification of the implemented model and evaluation procedure. The concern applies to the provided archive; it is not a claim that every external resource maintained by the authors is unavailable.
Questions for authors
-
Which exact controller produced the main results? Please map Tables 1–3 to the actual model class, configuration, and checkpoint lineage, and explain the correspondence between Eqs. (5)–(6) and the gated-residual implementation of
ClassificationPreservingController.encode_condition. If the implementation is the evaluated method, a corrected equation and an explicit mapping would largely resolve the architecture-identification concern. If a different implementation produced the results, please identify and provide that version. -
What objectives and parameter constraints were active in each training stage? Please specify the trainable and frozen modules, learning rates, number of epochs, initialization checkpoints, and auxiliary losses for each stage. In particular, was
parameter_trust_lossenabled in the reported runs, and did it become nonzero? A configuration and a compact summary of the relevant logs would clarify whether the paper’s stated objective adequately describes the evaluated system and would change the assessment of the training-specification concern. -
How much of RACD’s gain remains when the auxiliary semantic content is held fixed? Is there an existing comparison that supplies the same graph-derived content through text or a simpler conditioning mechanism? Please also clarify the backbone, image-description cache, preprocessing, and training conditions used for the recomputed MAPPER result. A modest matched-content comparison, or directly relevant existing evidence, would help determine whether the proposed structure and reliability mechanism contribute beyond the additional information itself.
-
Does DPEV improve input support when the label and candidate pool are held fixed? Using the label selected by Eq. (10) and the same three-candidate candidate pool, can you compare the default explanation with the selected explanation in a blinded paired audit? A representative subset of the reported replacements is sufficient if accompanied by the sampling procedure and counts of improved, worse, and tied cases. Evidence that the selections improve input support, rather than only reference similarity, would directly strengthen the central DPEV claim.
-
How reliable are the constructed same-label negative pairs? Please report the number of eligible pairs after filtering and provide a small audit distinguishing unsupported substitutions, substitutions that remain plausible or supported, and cases with obvious editing artifacts. A low rate of semantic mislabeling and little dependence on artifacts would increase confidence that the verifier learns evidence consistency rather than an easier textual discrimination task.
-
Can the custom dependencies and data preparation be made unambiguous? Please provide versioned sources or include the project-specific modules required by
RIFT_ORIG_ROOT, the evaluation implementation, and the feature-generation procedure. For the generated graphs, either the actual caches or a sufficiently specified generation procedure would help identify the experimental inputs. Resolving these dependencies would materially improve the ability to verify the architecture, metrics, and reported configuration. -
How were the candidate count, selection thresholds, and checkpoints chosen? Please identify the data split used to select the candidate count, the evidence-score margin, the generation-likelihood tolerance, and the final checkpoints, and distinguish selection experiments from post hoc test-set analyses. Appendix B’s test-set curve does not by itself establish test-set tuning. A clear validation-based selection account would remove this uncertainty; a different procedure would affect the interpretation of the reported generalization results.
-
Do training and inference use the same explanation-state region under the actual tokenizer? The training
explanation_state_maskuses shifted labels and a fixed label-token cutoff, whereas the inferenceexplanation_token_maskidentifies the explanation boundary from decoded output and excludes special tokens. Please show token IDs, EOS handling, and selected hidden-state indices for both relation labels. The differing rules may select equivalent regions for some tokenizations, so this is a clarification request rather than a confirmed off-by-one error. If they select different regions in the reported setup, an assessment of the effect on verifier scores would determine whether the difference is consequential.
3 comments
Rebuttal (2/3)
Q4:Does DPEV improve input support when the label and candidate pool are held fixed?
Thank you for this valuable suggestion. To examine whether DPEV genuinely relies on input visual evidence, we conducted controlled image-swapping experiments while keeping predicted labels and candidate explanations fixed. Replacing only the verifier's image-evidence representation reduced the evidence score, while swapping images throughout the scoring pipeline decreased scores in 95.99% of examples and changed explanation selection in 24.20% of cases. These results demonstrate that DPEV is sensitive to input visual evidence rather than relying solely on textual information. Furthermore, our fixed-label and identical-candidate-pool comparison shows that DPEV improves explanation quality over default selection. The corresponding quantitative results are provided in Table 2 of our response to reviewer vtvs. We acknowledge that a blinded human paired audit would provide complementary evidence of input-support quality and leave this evaluation for future work.
Q5:How reliable are the constructed same-label negative pairs?
Thank you for raising this concern. Of the 4,578 training instances, 2,877 yield eligible explanation spans, and 2,854 unique same-label negative pairs remain after donor matching and the runtime length-ratio filter. Each negative replaces an exact span with one from a different training instance matched by label, partition, and rule family, while preserving the label and surrounding explanation. These constraints establish structural eligibility but do not guarantee that every substitution is semantically unsupported. To assess this distinction, we have prepared a fixed random sample of 100 pairs for independent two-annotator review, recording semantic support and editing artifacts separately because they can overlap. Our preliminary surface-feature diagnostic achieves 73.13% classification accuracy, indicating that superficial cues are learnable in the constructed pairs. This does not establish that the multimodal verifier relies on those cues, but it prevents us from claiming negligible artifact dependence. We therefore characterize these pairs as weakly supervised mismatches rather than guaranteed semantic counterfactuals.
Q6:Can the custom dependencies and data preparation be made unambiguous?
Thank you for highlighting this reproducibility issue. We have added a public, versioned reproducibility supplement that provides the project-specific dependencies and modules, evaluation code and configurations, feature-generation procedures, and actual feature and graph caches with SHA256 checksums. We also document the preprocessing pipeline and metric computation, with the archived per-instance scores reproducing all 101 thresholds for the three main branches.The released feature tensors and graph caches serve as the authoritative inputs for the reported experiments.
Q7:How were the candidate count, selection thresholds, and checkpoints chosen?
Thank you for highlighting this ambiguity. We clarify the configuration and checkpoint selection procedures as follows.
•Candidate count: was inherited from the existing three-beam generation configuration used in v16, rather than selected through the subsequent Appendix B test-set analysis. No documented validation-based selection was performed.
•Selection thresholds: The evidence margin (0.05) was adopted as a conservative heuristic based on an earlier implementation, while the likelihood tolerance (0.10 nat per explanation token) was specified for the v16 run. Neither was selected through a documented validation search or calibration.
•Checkpoints: All reported R4 checkpoints correspond to the terminal states of fixed training schedules (3/1/1/2 epochs across the four stages), without validation- or test-based epoch selection. The verifier was trained on the full training set, including the 445-example internal development partition.
•Data splits: The official 726-example validation split was not used for selecting these configurations or checkpoints. Appendix B reports subsequent test-set sensitivity analyses, not the original selection procedure.
We acknowledge that test-set metrics were consulted when comparing earlier method iterations. Therefore, although the reported settings and checkpoints were not selected through test-based optimization, we cannot claim that the entire development process was independent of test feedback. We will clarify these distinctions and appropriately qualify our generalization claims.
Rebuttal (3/3)
Q8:Do training and inference use the same explanation-state region under the actual tokenizer?
Thank you for this careful clarification request. We audited the actual tokenizer and confirmed that the training and inference masks select slightly different hidden-state regions in the reported implementation.
The historical LLaVA loader uses LlamaTokenizer (use_fast=False, legacy=False, EOS ID = 2). The actual tokenization and selected response-local hidden states are summarized below, where denotes the number of non-EOS response tokens.
| Label | Prefix Token IDs | Training Mask | Inference Mask |
|---|---|---|---|
| Entailment | [875, 737, 358, 29889] | ||
| Contradiction | [23949, 29889] |
Thus, the training mask includes the label-ending period state for entailment but excludes the first explanation-token state for contradiction. The terminal boundary is consistent: training supervises EOS, retaining the final explanation state that predicts EOS, while excluding the EOS-token state itself.
We further audited 4,578 positive training sequences, 2,854 eligible negative sequences, and 2,164 test candidates. To quantify the impact, we fixed the checkpoint, candidate hidden states, evidence representations, labels, likelihoods, and selection thresholds, changing only the scoring mask. Across the test candidates, the mean, median, and maximum absolute evidence-score changes were 0.009563, 0.008218, and 0.062120, respectively. The final explanation selection changed in only 1 of 723 examples (0.138%), with all predicted labels preserved. In this case, the candidate's evidence gain decreased from 0.051499 to 0.049682, crossing the fixed 0.05 replacement threshold.
These results confirm a genuine but limited training–inference mask discrepancy, with measurable effects on evidence scores but minimal impact on final explanation selection in the evaluated test set. We will clarify the tokenization and masking procedures in the revised manuscript. We note that this fixed-checkpoint intervention does not establish the effect of retraining with a unified mask or guarantee unchanged explanation quality.
Rebuttal (1/3)
Q1:Which exact controller produced the main results?
Thank you for raising this important question regarding implementation clarity and reproducibility. To clarify, all reported main results are produced by the gated-residual controller, implemented as
ReliabilityTrustController, constructed throughmake_v8_readout_classeswithReliabilityTrustReadoutConfig. It inheritsClassificationPreservingController.encode_conditionwithout overriding its implementation.For clarity, we will make the correspondence between Eqs. (5)–(6) and the actual implementation more explicit. Specifically, given a=[hr;hg;hs;gstr], the independent graph and semantic gates are defined as bj=0.03+0.32σ(wj⊤a+cj), where j∈{g,s}. The corresponding residuals are Δg=bgϕg(hg) and Δs=bsϕs(hs), followed by hf=LN(hr+Δg+Δs). The final condition representation is obtained as z=fcond([hr;hf;Δg;Δs;∣hr−hf∣;hr⊙hf]). Here, ϕg and ϕs are separate Linear–GELU–LayerNorm projections. These operations correspond directly to
graph_reliability/semantic_reliability,graph_residual/semantic_residual,fused_norm, andcondition_encoderin the implementation. The saved configuration specifies a relation/condition dimension of 256, readout rank of 32, residual norm ratio of 0.06, and readout gate cap of 0.12.Regarding checkpoint lineage, Table 1 (R-CARE), Table 2 (Full), and Table 3 (Main) use the same
checkpoints_llava_final_combined_r4/model.ptcheckpoint. The two Table 2 ablations, w/o DPEV and w/o RACD, use theinnovation1_r4andinnovation2_r4checkpoints, respectively. The additional Table 3 runs follow the same controller implementation with seed pairs (123, 123) and (456, 456).We will further clarify the controller formulation, implementation mapping, and checkpoint lineage in the revised manuscript. To facilitate reproducibility, we also plan to release the corresponding model weights, configuration files, and implementation details upon publication. These clarifications make the evaluated architecture and its experimental lineage more transparent without changing the reported results.
Q2:What objectives and parameter constraints were active in each training stage?
Thank you for requesting this clarification. R-CARE is trained through four sequential stages, with the pretrained LLaVA backbone frozen throughout. Stage 1 (Joint Adaptation): Initialized from LLaVA-v1.6-Vicuna-7B, LoRA and RACD are trained for 3 epochs with learning rates of 2×10−4 and 1×10−4, respectively. Stage 2 (Controller Refinement): Initialized from Stage 1, LoRA is frozen and the controller is trained for 1 epoch at 1×10−5, with adaptive soft-negative and tail weighting. Stage 3 (Semantic Readout Refinement): Initialized from Stage 2, only the semantic readout and alignment projections are trained for 1 epoch at 5×10−6. Stage 4 (Verifier Training): Initialized from Stage 3, the generator, RACD, and upstream evidence encoders are frozen, and only the verifier is trained for 2 epochs at 2×10−4.
Stages 1–2 optimize GroupDRO generation, label prediction (0.4), counterfactual contrast (0.02), and semantic alignment (0.03). Stage 3 uses GroupDRO, label prediction, and decoder alignment (0.015), together with the configured parameter_trust_loss (radius 5×10−4). Importantly, although enabled, this constraint remained inactive: its penalty and dual multiplier were zero, and the maximum logged RMS parameter drift was only 5.54×10−5. Stage 4 optimizes the verifier ranking-and-anchor objective (weight 0.2). Eq. (19) describes Stage 3 rather than the entire training procedure. We will clarify the stage-wise objectives and configurations in the revised manuscript, and release the corresponding configurations, logs, and model weights upon publication.
Q3:How much of RACD’s gain remains when the auxiliary semantic content is held fixed?
Thank you for this valuable suggestion. We conducted matched-content comparisons using both LLaVA and Qwen3-VL-8B-Instruct. On LLaVA, with DPEV disabled, RACD outperforms Graph-as-Text by 1.76, 1.36, and 0.13 points on F1@0, F1@53, and F1@60, respectively. On Qwen3-VL-8B-Instruct, RACD with ordinary decoding improves F1@0 and F1@53 by 1.43 and 0.83 points over Graph-as-Text SFT, while complete R-CARE achieves gains of 2.27, 2.21, and 0.13 points across the three metrics. These results indicate that the improvements go beyond simply providing additional semantic information.
Detailed results are provided in Table 1 of our response to reviewer vtvs and Table 2 of our response to reviewer 3G8A. For MAPPER, we followed its reported architecture and training configuration; further reproduction details are provided in Rebuttal (2/2) to reviewer 3G8A.