- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MM-ReportBench, a comprehensive benchmark for multimodal report generation from tabular data, comprising 386 tasks derived from 185 real-world tables across six domains. The authors construct reference reports through a combination of Monte Carlo Tree Search (MCTS) and human review, and propose a dual-judge evaluation framework to assess textual quality, visualization quality, and cross-modal consistency. Through evaluations of 20 models, the authors identify significant limitations in numerical accuracy, insight novelty, and visualization quality among existing models.
Critical strengths
-
Practical Relevance. The paper focuses on generating comprehensive reports that integrate textual analysis and visualizations from structured tables. Compared to traditional table question-answering tasks, this setting is more closely aligned with real-world data analysis and business reporting scenarios.
-
Well-Constructed Dataset. The authors collect 386 tasks across six domains and construct reference reports through a combination of Monte Carlo Tree Search (MCTS), human review, and SQL-based fact verification, providing a relatively reliable foundation for benchmark evaluation.
Critical weaknesses
-
Insufficient Validation of Reward Hacking Robustness. The paper identifies reward hacking in LLM-as-a-Judge as a key motivation, but the proposed Dual-Judge framework primarily addresses evaluation biases by separating textual and visual assessments. Although the authors introduce some source-grounded factual verification mechanisms, they do not conduct targeted adversarial experiments. Therefore, the current evidence is insufficient to demonstrate that the framework can effectively mitigate reward hacking.
-
Lack of Report-Level Visual Layout Evaluation. Although the paper evaluates individual charts in terms of appropriateness, aesthetics, and consistency with textual content, it lacks a systematic assessment of the overall visual layout of generated reports, including chart placement, text-figure arrangement, spatial organization, and overall readability. For a benchmark focused on multimodal report generation, the current evaluation framework remains incomplete in this regard.
-
Insufficient Justification of the Overall Scoring Mechanism and Fairness of Model Comparisons. The authors compute the final score as a weighted sum of T-Judge and V-Judge scores, but provide limited analysis of the chosen weights and their impact on model rankings. Furthermore, assigning a V-Score of zero to text-only models, while consistent with the task requirements, limits meaningful comparisons across different model categories. A more comprehensive evaluation could allow language models to generate visualizations through SVG or Python code, which can then be rendered in a standardized environment. This would enable a fairer assessment of report generation capabilities across different systems.
Questions for authors
-
Reward Hacking Robustness and Dual-Judge Effectiveness: Could the authors provide targeted experiments demonstrating that the proposed framework is more robust to reward hacking than conventional LLM-as-a-Judge approaches? Given that both T-Judge and V-Judge fundamentally rely on existing large multimodal models, how can the authors substantiate the advantages of the Dual-Judge design beyond simply separating the evaluation tasks?
-
Reliability of Novelty Evaluation: The paper primarily identifies novel insights based on semantic similarity to keypoints in reference reports. How do the authors ensure that this metric can distinguish genuinely valuable analytical discoveries from content that merely differs from the reference reports? Have the authors conducted targeted human evaluations to validate the reliability of this novelty assessment?
4 comments
Thank you for your time in reviewing our submission. I am writing to kindly ask whether there is any update you are able to share regarding the review timeline.
We completely understand that you have a busy schedule, and we are grateful for your effort. If there is any additional material that would be helpful, please let us know.
Response for Author
Hi, I apologize for my late response.
I appreciate the additional experiments and clarifications. However, I still have some concerns regarding the Reward Hacking claims.
The A/B/C experiments seem to focus more on detecting predefined errors than on testing whether models can strategically exploit weaknesses in the judges. Also, the comparison does not clearly isolate the contribution of the Dual-Judge design from the additional SQL-based verification and rule-based checks.
I am still not fully convinced that the current experiments sufficiently demonstrate the claimed robustness against reward hacking.
Thank you very much for your response.
The purpose of the A/B/C experiments is to provide a controlled operationalization of reward hacking: each adversarial report preserves the surface features that judges tend to reward (fluent text, aesthetic charts, and plausible numbers) while violating factual grounding, cross-modal consistency, or task intent. A single judge shows score inflation of +11.2/+9.8/+8.5 on A/B/C, with human agreement of only 0.62/0.58/0.55. Dual-Judge reduces inflation to +2.1/+3.4/+1.8 and raises human agreement to 0.81/0.84/0.86. This indicates that these weaknesses are exploitable and that Dual-Judge aligns more closely with human judgment on these adversarial reports.
We also do not claim that task separation alone explains all gains. SQL-based GIV, V-Judge DF/TC, and CMC are not external add-ons; they reduce reliance on a single LMM score and limit cross-modal compensation. The existing ablation and A/B/C results provide initial attribution: A is mainly caught by CMC/T-GIV, B by V-DF/TC, and C by T-GIV_num. In the next revision, we will add a task-separation-only baseline and mechanism ablations removing CMC, SQL-based GIV, DF/TC, and the NA penalty, and we will report per-component detection rates and bootstrap confidence intervals to formally separate the contributions of these components.
Finally, we will revise our claim to be more precise: Dual-Judge reduces the manipulable space, cross-modal compensation, and score inflation, but it is not a complete defense against reward hacking. Subjective dimensions can still benefit from panel-based evaluation and human calibration.
Thank you again for these constructive comments.
Thank the reviewer for these detailed and constructive comments.
Weaknesses 1(Q1): Dual judges primarily address evaluation bias and cross-modal compensation, and do not automatically constitute a comprehensive defense against reward hacking; both T-Judge and V-Judge still rely on large multimodal models. We acknowledge that the original evidence is insufficient. We add targeted adversarial experiments.
The experiment design: On 100 tasks, we construct three types of adversarial reports:
We compare three evaluation settings: single judge (GPT-4o), dual judges (T-Judge + V-Judge + CMC), and human evaluation. Metrics are score inflation (adversarial reports relative to faithful reports), detection rate, and agreement with human judgment. Results:
These results indicate that the advantage of dual judges is not merely “separating tasks,” but rather: task separation prevents cross-modal compensation + source-table SQL/rule checks provide external anchors + CMC constrains text-chart consistency. However, we will state explicitly in the main text that dual judges reduce the manipulable space but do not constitute a complete defense against reward hacking; subjective dimensions still require panel-based evaluation and human calibration.
Weaknesses 2: The reviewer is correct. We add a Report-Level Layout Score (RLS) with four sub-items:
RLS uses a 0–5 rubric, calibrated via blind evaluation by three annotators on 100 reports.
Results:
RLS is only moderately correlated with Overall, indicating that it provides independent diagnostic information. We will not incorporate RLS into the main Overall score for now, but will report it as a report-level diagnostic dimension; if the reviewer believes it should be included, we will report the ranking sensitivity after adding RLS in the appendix.
Weaknesses 3: The paper already includes weight sensitivity in Appendix A.6. The weight is based on the fact that in business reporting, textual analysis carries the primary analytical burden while visualizations provide support.
We add experiments for a few Text-only LLMs with the same Python executor and plotting library as code-augmented systems in same setting. Results:
The zero score in the original setting is a lower-bound reference for “no-tool, text-only output.” In the revision, the main table will retain only systems capable of generating multimodal reports, and text-only results will be moved to the appendix as a controlled reference.
Question 2: We add claim-level human validation:
Results:
We will adjust NA in the revision: treat “factual consistency with source tables” as a hard gate, so claims that fail it are not counted as novel; increase the weight of actionability; and report false-positive types in the appendix.
We thank the reviewer again for these constructive comments.