- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes DERIVE, a framework for improving evidence-based reasoning in multimodal long-document question answering. Its motivation is that locating the correct evidence does not ensure that a model reads, combines, and uses that evidence correctly. DERIVE represents reasoning as Verifiable Execution chains, which connect document-grounded values through executable operations. It also introduces Evidence Intervention, which modifies answer-determining document content and recomputes the corresponding reasoning targets. The model is trained through supervised fine-tuning followed by reinforcement learning with rewards for answer correctness, evidence grounding, execution consistency, and intervention responsiveness. Experiments use Qwen2.5-VL-3B and 7B across six benchmarks, with the largest reported improvements on arithmetic-intensive document questions.
Critical strengths
-
The paper addresses an important limitation beyond evidence retrieval.
The observation that models still fail when relevant evidence is provided motivates studying evidence interpretation and use. This is a useful direction for document question answering, where retrieving the right page is only one step toward producing a reliable answer.
-
The proposed representation makes parts of the reasoning process directly checkable.
Explicit READ operations, variable references, and executable computations allow the system to inspect evidence provenance and intermediate calculations. Combining this representation with controlled document edits provides a concrete way to evaluate whether generated answers respond appropriately to changes in relevant evidence.
-
The evaluation covers multiple tasks and includes useful component analyses.
The experimental design includes six benchmarks, two model sizes, and two datasets excluded from training. The paper also examines paired evidence edits, different answer types, training stages, reward-signal availability, and computational costs. These analyses provide several perspectives on the proposed approach.
Critical weaknesses
-
Important related methods and executable-reasoning baselines are missing.
[DocSeeker][docseeker] and [LAT][lat] are particularly relevant because they already combine structured reasoning, evidence supervision, and reinforcement learning, yet neither is included in the experimental comparisons. [PoT][pot]/[PAL][pal]-style baselines are also important for determining how much of the arithmetic improvement comes from representing computations as executable programs. Additional relevant document systems include [HiEvi-RAG][hievi], [Doc-V*][docv], and [DocLens][doclens], although comparisons with these systems should account for differences in backbone and inference budget. Without these comparisons, the contribution beyond existing evidence-grounded and program-based reasoning methods remains insufficiently established.
-
The evidence does not fully support the claim that the model's reasoning dependencies have been verified.
Successful execution of a generated chain demonstrates a property of the output program, while correct answers on paired document edits provide behavioral evidence of responsiveness. Neither establishes that the model actually relied on the displayed chain when generating its answer. Furthermore, a program can produce correct outputs for an original value and one replacement while implementing an incorrect general rule. More varied interventions, irrelevant-evidence controls, and independent checks of chain-answer consistency are needed to support the stronger dependency-verification claims.
-
The verifier checks only part of what reliable evidence-grounded reasoning requires.
The grounding reward checks whether a value occurs on the cited page, rather than explicitly verifying the predicted bounding box or its semantic role. Repeated numbers can therefore receive grounding credit despite being associated with the wrong row, year, or entity. Similarly, executable consistency establishes that an operation was performed correctly, but not that the operation was appropriate or that all necessary evidence was included. These limitations weaken the interpretation of the rewards as comprehensive process verification.
-
The ablations do not fully isolate the sources of improvement.
Freezing the retrieval head does not guarantee identical retrieved pages because the upstream language-model layers are updated through LoRA. Some improvements could therefore involve changes in evidence selection. Figure 4 also compares SFT against GRPO with intervention data, changing both the training procedure and the data simultaneously. Consequently, the corresponding gain cannot be attributed solely to additional intervention data. Stronger controls should hold retrieved pages, training examples, update counts, and rollout budgets constant.
-
The intervention evaluation and statistical reporting are insufficiently documented.
The paper does not adequately specify the source split, selection criteria, filtering rate, or document-level separation of the 505 intervention pairs. This matters because the base model's original-state accuracy on this set is much lower than its main TAT-DQA result. The intervention labels are also generated by re-executing the original chain, which can preserve semantic mistakes in that chain. Finally, the main comparisons lack reported uncertainty or repeated-run results, making it difficult to assess the reliability of relatively small improvements.
-
Numerical inconsistencies and incomplete methodological reporting undermine confidence in the conclusions.
The manuscript mixes evaluation metrics in its baseline comparisons, reports overall scores that do not match the corresponding subgroup results, and contains inconsistencies in conditional accuracy and parameter percentages. Its stated GRPO objective also omits important elements of the standard formulation without explaining the actual implementation. These issues require correction and access to the underlying evaluation details before the reported comparisons and component-level conclusions can be assessed confidently.
Questions for authors
The following list distinguishes directly checkable errors from issues that could potentially be explained by an undocumented evaluation or implementation choice.
-
Table 1 mixes EM and ANLS in the same benchmark columns.
Section 4.1 states that SlideVQA and TAT-DQA are evaluated using Exact Match. However, the borrowed DocR1 results—71.9 for SlideVQA and 64.8 for TAT-DQA—correspond to results reported as ANLS in the original paper. The published [DocR1][docr1] table explicitly uses ANLS and reports 71.96 and 64.80. The borrowed mPLUG-DocOwl2 results also match values in that ANLS table. These scores cannot be treated as directly comparable EM results without reevaluation.
-
Table 2's overall scores do not match its subgroup scores.
The manuscript states that it evaluates the complete 1,091-question MMLongBench-Doc benchmark. The corresponding [official dataset description][mmlong] contains 485 single-page, 360 multi-page, and 246 unanswerable questions.
Using these counts to aggregate the reported subgroup scores gives:
Model Single-page Multi-page Unanswerable Recomputed overall Reported overall DERIVE-3B 38.2 16.7 39.5 31.40 33.5 DERIVE-7B 39.5 18.4 66.0 38.51 35.6 The calculation is:
= \frac{485\,\mathrm{SIN}+360\,\mathrm{MUL}+246\,\mathrm{UNA}}{1091}.$$ These differences are too large to result from rounding. If different subsets, metrics, or dataset versions were used, they must be identified explicitly. -
Table 3 contains an inconsistent conditional accuracy.
For free-CoT+GRPO, original accuracy is 51.2% and joint accuracy is 41.8%. Under the table's pooled interpretation, the conditional intervention accuracy should be:
rather than the reported 79.6%. If the columns instead contain separately averaged statistics across runs, the aggregation procedure must be specified.
-
Two Table 3 percentages are incompatible with a single binary evaluation over 505 pairs.
With 505 examples, 258 and 259 correct predictions produce 51.1% and 51.3% after rounding; neither produces the reported 51.2%. Likewise, 230 and 231 correct predictions produce 45.5% and 45.7%, rather than the reported 45.6%. These values could arise from averaging multiple runs, but no such protocol is reported.
-
Table 5's answerability and document-length breakdowns cannot both describe the same evaluation under a common averaging scheme.
For the baseline, the answerable and unanswerable scores are 44.7 and 0.4. Since every document-length group scores at most 39.7, the unanswerable fraction must be at least approximately 11.3%.
For VE-chain SFT, the answerable and unanswerable scores are 42.4 and 85.1. Since every document-length group scores at most 44.3, the unanswerable fraction must be at most approximately 4.45%.
The same evaluation set cannot satisfy both constraints. Different subsets or aggregation rules would need to be disclosed.
-
The improvement stated in Section 4.2 is arithmetically incorrect.
The paper states that DERIVE-7B improves over URaG-7B by 2.1 points on MMLongBench-Doc. Table 2 instead gives:
-
Figure 2 contains an incorrect intermediate computation after intervention.
The edited chain changes the evidence value from 172.1 to 172.2 while retaining the other value at 304.7. The subtraction should therefore produce 132.5, but the intermediate result remains 132.6, while the final answer is shown as 132.5. The example is internally inconsistent.
-
Table 6 uses a denominator inconsistent with its caption.
The caption states that parameter percentages exclude output embeddings. However, the reported percentages correspond approximately to the total model size. For example:
rather than the reported 1.593%. Similarly:
rather than 0.973%. The main text's 1.57% figure for the 3B model also differs from the table's approximately 1.59%.
-
Equation 9 does not specify the standard GRPO objective.
The stated loss is an advantage-weighted log-likelihood objective. It omits the old-policy probability ratio, clipping, and the KL term present in the [original GRPO formulation][grpo]. This may be an intentional simplification or variant, but the actual objective, update schedule, and relevant hyperparameters must be provided. The omission alone does not establish that the implementation is incorrect.
-
The statement about avoiding negative advantages for semantically correct answers is mathematically inaccurate.
Recognizing alternative valid answers prevents false incorrectness labels. It does not prevent negative group-relative advantages:
A correct response with a lower process reward than other correct responses can still receive a negative advantage.