- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies evidence selection and answer-input compression for long-document visual question answering. It introduces ELDER, a dual-role framework built on a shared vision-language model. An Evidence Extractor selects complete, pre-parsed document elements, and an Evidence-only Generator answers using only their original image crops. Teacher-based support checks and a reader that does not receive reference answers construct families of acceptable evidence sets for supervised initialization. The proposed MRE-GRPO uses nested sampling to evaluate evidence sets through mean answer scores and visual-token costs, while separately optimizing answers conditioned on fixed evidence. Across five benchmarks, the paper reports an average score of 67.6. Relative to dual-role RL without the visual-token cost penalty, ELDER reduces mean Generator input visual tokens by 43.2%, with a 0.3-point decrease in average score. Role-loss ablations, full-document training controls, and frozen-checkpoint combinations further examine the contributions of selection and answering.
Critical strengths
S1. The method addresses a practically relevant quality–input-cost tradeoff with a coherent training design. Selecting complete document elements, preserving their internal layouts, and restricting answering to the selected crops is a reasonable approach to reducing irrelevant visual context in long documents. MRE-GRPO distinguishes quality–cost comparisons across evidence sets from answer optimization under fixed evidence, and explicitly handles invalid selections, empty sets, and all-zero feedback. This provides a clear, operational approach to jointly learning evidence selection and evidence use. [Sections 3.3–3.5 and Appendix A.3] S2. The internal controls provide meaningful evidence for the overall framework’s empirical value. Beyond the five-benchmark results and role-loss ablations, Table 3 includes full-document controls matched by data progress or training GPU-hours, as well as combinations of frozen Extractor and Generator checkpoints. Full ELDER exceeds the strongest full-document control in Table 3 by 4.0 points on average. With either role fixed, replacing the other with its RL checkpoint still improves scores. These experiments support contributions from both evidence selection and evidence-conditioned answering, and strengthen the case beyond simply applying additional training. [Tables 1–3 and Section 4.4]
Critical weaknesses
W1. Important experimental details are missing, limiting the verification and reproducibility of the central quality–cost results. The paper defines the visual budget , controller step size , calibrated target , and fixed image preprocessing, but does not report the concrete values or settings needed to reproduce the experiments. Training sampling and evaluation decoding configurations are also insufficiently specified. These choices directly determine the executable selection space, visual-token counts, and the scale of the cost penalty. Table 4 reports question–evidence-set pairs without the actual numbers of unique questions, role-specific training records, or active RL questions, while the compute-matched controls in Table 3 lack numerical GPU-hour accounting. External baseline results are also not individually mapped to imported or reproduced sources and evaluation settings. These omissions make the central quality–cost comparisons and their fairness difficult to verify fully. [Section 4.1 and Appendices A.1–A.3, C–D] W2. The distinctive design choices and the support quality of the selected evidence are not sufficiently validated. Tables 2–3 establish benefits from the overall framework, role losses, and cost term, but do not directly isolate the gains from multiple-set supervision or nested set evaluation over simpler alternatives. Evaluation also relies primarily on answer scores and visual-token counts, without an independent assessment of whether the selected regions support the answers. Figure 6 illustrates the reference answer and evidence structure rather than reporting model predictions and selection traces. Although the authors explicitly acknowledge that answer correctness does not establish evidence sufficiency, this empirical gap still limits our understanding of what the model learns about evidence. Moreover, adaptive control improves the average score over fixed cost by approximately 0.3 points, without uncertainty estimates across repeated runs, leaving the necessity of the additional mechanism unclear. [Tables 2–3 and Appendices A.1, A.3, F–G]
Questions for authors
1.Q1. Experimental protocol and reproducibility Could you provide the complete configuration and data statistics, including , , the calibrated , page and crop preprocessing, training sampling and evaluation decoding settings, the actual numbers of unique SFT/RL questions and documents, and the evaluation split and sample count for each benchmark? What are the numerical training GPU-hours for the methods in Table 3? Which results in Table 1 are imported versus reproduced, and are their evaluation settings comparable? This information would directly affect my confidence in the main comparisons. 2.Q2. Independent contribution of multiple-set supervision When source questions and training exposure are kept comparable, how does using only one verified evidence set per question compare with the current multiple-set supervision? A controlled comparison on a representative benchmark would help determine whether multiple-set supervision provides value beyond additional training exposure. 3.Q3. Stability of the adaptive controller’s gains Is the small improvement over fixed consistent across repeated runs? Could you provide repeated-run results for a representative setting, together with paired per-example uncertainty analysis and the cost-coefficient trajectory, to clarify the controller’s actual effect? This would affect my assessment of whether the additional control mechanism is necessary.