acceptodds
Under review as a conference paper at ICLR 2027

MM-ReportBench: A Comprehensive Benchmark for Multimodal Report Generation from Tables

Abstract

The ability to generate multimodal reports integrating fluent narratives with faithfulvisualizations from tabular data, has become a cornerstone capability for mod-ern vision-language models in practical domains such as finance, healthcare, andpolicy making. However, existing benchmarks are limited by single-modalityfocus, simplistic metrics, novelty evaluation, and visualization quality. We ad-dress these limitations with MM-ReportBench, a multimodal report evaluationbenchmark covering 386 tasks across 185 real-world tables. Each task is pairedwith an expert-curated reference report featuring standardized chapter structuresand verifiable key insight points, constructed through a hybrid pipeline of multi-modal reports generation based on Monte Carlo Tree Search and rigorous humanquality control. For robust evaluation, we design a dual-judge evaluation frame-work: Text Judge Model assesses textual quality along structural completeness,grounded insight verification with numerical fact-checking against source tables,and novelty analysis that distinguishes genuine analytical discovery from triv-ial paraphrasing; Visualization Judge Model evaluates generated visualizationsacross data fidelity, expressiveness, aesthetics, and technical correctness. They alsoevaluate cross-modal consistency. Extensive experiments on 20 state-of-the-artmodels reveal critical weaknesses: even the strongest model achieves only 68.9overall score, with numerical hallucination affecting 38.5% of reports and gen-uinely novel insights remaining rare. Our ablation studies further demonstratethat single-judge evaluation consistently inflates scores by 8–12 points comparedto dual-judge assessment, confirming that rigorous and reliable benchmarkingdemands multi-faceted, complementary evaluation probes. The source data and code is available at https://anonymous.4open.science/r/MReportBench.

Primary area:
datasets and benchmarks
Keywords:
Multimodal report generation, table-to-report, dual-judge evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_q5s3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MM-ReportBench, a comprehensive benchmark for multimodal report generation from tabular data, comprising 386 tasks derived from 185 real-world tables across six domains. The authors construct reference reports through a combination of Monte Carlo Tree Search (MCTS) and human review, and propose a dual-judge evaluation framework to assess textual quality, visualization quality, and cross-modal consistency. Through evaluations of 20 models, the authors identify significant limitations in numerical accuracy, insight novelty, and visualization quality among existing models.

Critical strengths
  1. Practical Relevance. The paper focuses on generating comprehensive reports that integrate textual analysis and visualizations from structured tables. Compared to traditional table question-answering tasks, this setting is more closely aligned with real-world data analysis and business reporting scenarios.

  2. Well-Constructed Dataset. The authors collect 386 tasks across six domains and construct reference reports through a combination of Monte Carlo Tree Search (MCTS), human review, and SQL-based fact verification, providing a relatively reliable foundation for benchmark evaluation.

Critical weaknesses
  1. Insufficient Validation of Reward Hacking Robustness. The paper identifies reward hacking in LLM-as-a-Judge as a key motivation, but the proposed Dual-Judge framework primarily addresses evaluation biases by separating textual and visual assessments. Although the authors introduce some source-grounded factual verification mechanisms, they do not conduct targeted adversarial experiments. Therefore, the current evidence is insufficient to demonstrate that the framework can effectively mitigate reward hacking.

  2. Lack of Report-Level Visual Layout Evaluation. Although the paper evaluates individual charts in terms of appropriateness, aesthetics, and consistency with textual content, it lacks a systematic assessment of the overall visual layout of generated reports, including chart placement, text-figure arrangement, spatial organization, and overall readability. For a benchmark focused on multimodal report generation, the current evaluation framework remains incomplete in this regard.

  3. Insufficient Justification of the Overall Scoring Mechanism and Fairness of Model Comparisons. The authors compute the final score as a weighted sum of T-Judge and V-Judge scores, but provide limited analysis of the chosen weights and their impact on model rankings. Furthermore, assigning a V-Score of zero to text-only models, while consistent with the task requirements, limits meaningful comparisons across different model categories. A more comprehensive evaluation could allow language models to generate visualizations through SVG or Python code, which can then be rendered in a standardized environment. This would enable a fairer assessment of report generation capabilities across different systems.

Questions for authors
  1. Reward Hacking Robustness and Dual-Judge Effectiveness: Could the authors provide targeted experiments demonstrating that the proposed framework is more robust to reward hacking than conventional LLM-as-a-Judge approaches? Given that both T-Judge and V-Judge fundamentally rely on existing large multimodal models, how can the authors substantiate the advantages of the Dual-Judge design beyond simply separating the evaluation tasks?

  2. Reliability of Novelty Evaluation: The paper primarily identifies novel insights based on semantic similarity to keypoints in reference reports. How do the authors ensure that this metric can distinguish genuinely valuable analytical discoveries from content that merely differs from the reference reports? Have the authors conducted targeted human evaluations to validate the reliability of this novelty assessment?

4 comments

Thank the reviewer for these detailed and constructive comments.

Weaknesses 1(Q1): Dual judges primarily address evaluation bias and cross-modal compensation, and do not automatically constitute a comprehensive defense against reward hacking; both T-Judge and V-Judge still rely on large multimodal models. We acknowledge that the original evidence is insufficient. We add targeted adversarial experiments.

The experiment design: On 100 tasks, we construct three types of adversarial reports:

  • A: charts are correct, but the text contains exaggerated conclusions unsupported by the source tables;
  • B: text is correct, but charts are aesthetically pleasing while containing incorrect data;
  • C: numbers appear plausible but cannot be verified by SQL against the source tables.

We compare three evaluation settings: single judge (GPT-4o), dual judges (T-Judge + V-Judge + CMC), and human evaluation. Metrics are score inflation (adversarial reports relative to faithful reports), detection rate, and agreement with human judgment. Results:

Adversarial typeSingle-judge score inflationDual-judge score inflationSingle-judge agreement with humansDual-judge agreement with humans
A Text exaggeration, correct charts+11.2+2.10.620.81
B Beautiful charts but incorrect data+9.8+3.40.580.84
C Numbers not verifiable against source tables+8.5+1.80.550.86

These results indicate that the advantage of dual judges is not merely “separating tasks,” but rather: task separation prevents cross-modal compensation + source-table SQL/rule checks provide external anchors + CMC constrains text-chart consistency. However, we will state explicitly in the main text that dual judges reduce the manipulable space but do not constitute a complete defense against reward hacking; subjective dimensions still require panel-based evaluation and human calibration.

Weaknesses 2: The reviewer is correct. We add a Report-Level Layout Score (RLS) with four sub-items:

  • whether charts are placed close to the relevant discussion;
  • whether text-figure arrangement is clear and unobstructed;
  • whether spatial organization is balanced and layout is consistent across sections;
  • overall readability, including font size, legends, whitespace, and multi-chart layout.

RLS uses a 0–5 rubric, calibrated via blind evaluation by three annotators on 100 reports.

Results:

MetricResult
Spearman between RLS and human layout ratings0.76
Human ICC(2,k)0.82
Correlation between RLS and Overall0.41
DeepSeek-R1 RLS74.8
Gemini-2.5-Pro RLS72.1

RLS is only moderately correlated with Overall, indicating that it provides independent diagnostic information. We will not incorporate RLS into the main Overall score for now, but will report it as a report-level diagnostic dimension; if the reviewer believes it should be included, we will report the ranking sensitivity after adding RLS in the appendix.

Weaknesses 3: The paper already includes weight sensitivity in Appendix A.6. The weight is based on the fact that in business reporting, textual analysis carries the primary analytical burden while visualizations provide support.

We add experiments for a few Text-only LLMs with the same Python executor and plotting library as code-augmented systems in same setting. Results:

ModelOriginal Text-only Overall+ Code Tool Overall
GPT-4o-text22.357.4
DeepSeek-V321.356.1
Qwen3-32B20.354.8

The zero score in the original setting is a lower-bound reference for “no-tool, text-only output.” In the revision, the main table will retain only systems capable of generating multimodal reports, and text-only results will be moved to the appendix as a controlled reference.

Question 2: We add claim-level human validation:

  • Extract 200 claims judged as novel by NA from 100 reports;
  • Three annotators perform blind evaluation; a “genuine discovery” must simultaneously satisfy: non-trivial, actionable, factually consistent with source tables, and not a paraphrase of a reference keypoint;
  • Compute NA precision/recall/F1, Spearman correlation with human novelty ratings, and error types.

Results:

MetricResult
NA claim-level Precision0.76
NA claim-level Recall0.71
NA claim-level F10.73
Spearman between NA and human novelty ratings0.71
Fleiss’ κ\kappa among human annotators0.80
“Different but trivial” false-positive rate18%
“Correct but not actionable” false-positive rate12%

We will adjust NA in the revision: treat “factual consistency with source tables” as a hard gate, so claims that fail it are not counted as novel; increase the weight of actionability; and report false-positive types in the appendix.

We thank the reviewer again for these constructive comments.

Thank you for your time in reviewing our submission. I am writing to kindly ask whether there is any update you are able to share regarding the review timeline.

We completely understand that you have a busy schedule, and we are grateful for your effort. If there is any additional material that would be helpful, please let us know.

Reviewer_q5s3OpenReview ↗

Response for Author

Hi, I apologize for my late response.

I appreciate the additional experiments and clarifications. However, I still have some concerns regarding the Reward Hacking claims.

The A/B/C experiments seem to focus more on detecting predefined errors than on testing whether models can strategically exploit weaknesses in the judges. Also, the comparison does not clearly isolate the contribution of the Dual-Judge design from the additional SQL-based verification and rule-based checks.

I am still not fully convinced that the current experiments sufficiently demonstrate the claimed robustness against reward hacking.

Thank you very much for your response.

The purpose of the A/B/C experiments is to provide a controlled operationalization of reward hacking: each adversarial report preserves the surface features that judges tend to reward (fluent text, aesthetic charts, and plausible numbers) while violating factual grounding, cross-modal consistency, or task intent. A single judge shows score inflation of +11.2/+9.8/+8.5 on A/B/C, with human agreement of only 0.62/0.58/0.55. Dual-Judge reduces inflation to +2.1/+3.4/+1.8 and raises human agreement to 0.81/0.84/0.86. This indicates that these weaknesses are exploitable and that Dual-Judge aligns more closely with human judgment on these adversarial reports.

We also do not claim that task separation alone explains all gains. SQL-based GIV, V-Judge DF/TC, and CMC are not external add-ons; they reduce reliance on a single LMM score and limit cross-modal compensation. The existing ablation and A/B/C results provide initial attribution: A is mainly caught by CMC/T-GIV, B by V-DF/TC, and C by T-GIV_num. In the next revision, we will add a task-separation-only baseline and mechanism ablations removing CMC, SQL-based GIV, DF/TC, and the NA penalty, and we will report per-component detection rates and bootstrap confidence intervals to formally separate the contributions of these components.

Finally, we will revise our claim to be more precise: Dual-Judge reduces the manipulable space, cross-modal compensation, and score inflation, but it is not a complete defense against reward hacking. Subjective dimensions can still benefit from panel-based evaluation and human calibration.

Thank you again for these constructive comments.

Reviewer_3XhfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed MM-ReportBench to evaluate the multimodal report generation ability of LLMs with 386 tasks and 185 tables. It uses reference reports and keypoints to evaluate the generated reports based on text, insight and cross-modal consistency. Comprehensive experiments and analysis are provided.

Critical strengths

The task is meaningful, and the evaluation contains meaningful metrics. Evaluation is thorough with 20 models / systems.

Critical weaknesses
  1. The core novelty of this benchmark is the evaluation of multimodal report, but its distinction against prior works such as InsightBench[1], DataNarrative [2], Multimodal DeepResearcher [3], FinSight [4] needs discussion.
  2. For text-only models, the visual score is set to zero. This could be unfair as text-only models can also generate code.
  3. The judge only uses GPT-4o, which is outdated and not as capable as the models evaluated. This affects the effectiveness of judge and makes it suspicious to overfitting the taste of the single model.
  4. Some scores in Table 2 are confusing. V-Score is defined as the average of DF, EX, AE, TC, but in the DeepSeek-R1 line, the V-Score is 80.2, while the average of the four metrics is 81.025.

References [1] Sahu, Gaurav, et al. "Insightbench: Evaluating business analytics agents through multi-step insight generation." International Conference on Learning Representations. Vol. 2025. 2025. [2] Islam, Mohammed Saidul, et al. "DataNarrative: Automated data-driven storytelling with visualizations and texts." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. [3] Yang, Zhaorui, et al. "Multimodal deepresearcher: Generating text-chart interleaved reports from scratch with agentic framework." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 40. 2026. [4] Jin, Jiajie, et al. "Finsight: Towards real-world financial deep research." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Questions for authors
  1. Some scores in Table 2 are confusing. V-Score is defined as the average of DF, EX, AE, TC, but in the DeepSeek-R1 line, the V-Score is 80.2, while the average of the four metrics is 81.025. Why?
  2. Why do you use GPT-4o (with vision) as the primary judge model?
  3. Is the vision capability of VLMs evaluated used during the generation of reports?s
3 comments

We thank the reviewer for these constructive comments. We respond point by point:

Weaknesses 1: Thank you for point this out. We will add this comparison table in Related Work. Cells are kept short, using ✓/△/✗ to denote “explicitly supported / partially or not central / not supported.”

WorkStructured data InputMultimodal ReportTable-Grounded Fact VerificationChart Quality EvaluationCross-Modal ConsistencyVerifiable References / KeypointsbilingualEvaluation Method / Ground Truth
InsightBench✓✗△✗✗△✗LLaMA-3 bidirectional evaluation
DataNarrative✓✓✗△△✗✗Model-based evaluation + human evaluation
Multimodal DeepResearcher / MultimodalReportBench✗✓✗✓△✗✗Dedicated metrics + win-rate comparisons
FinSight✓✓△△△✗✗Factual accuracy, analytical depth, presentation quality
MM-ReportBench (ours)✓✓✓✓✓✓✓Dual-judge T-Judge + V-Judge

The core innovation of MM-ReportBench lies in constructing, for real-world business tables, the only benchmark that simultaneously covers bilingual multimodal report generation, source-table fact verification, chart quality and cross-modal consistency, verifiable keypoints, and dual-judge evaluation—thereby providing an auditable, diagnostic, and cross-modal-compensation-resistant evaluation standard for enterprise AI data analysis and report generation.

Weaknesses 2: Thank you for point this out. The zero visual score corresponds to a no-tool, text-only setting: these models are asked to produce only the textual report and are not given a code executor. This setting isolates native multimodal generation capability. In our opinion, It would be better to remove "Text-only LLMs" from the main table and the main ranking and downgrade it to a controlled comparison setting in the appendix. This setting deviates from the core task of the benchmark: generating multimodal report.

Weaknesses 3: We agree that GPT-4o is not the strongest available judge. GPT-4o was chosen because it was stable, cost-effective, vision-capable, and reproducible at the time of experiments. We use a judge panel rather than a single judge. We will report scores from GPT-4o, Qwen3.5-9B, and Claude-3.5-Sonnet, and use their average or consensus for the main results.

CategoryModelGPT-4o OverallQwen3.5-9B OverallClaude-3.5-Sonnet OverallThree-Judge Average
VLMGPT-4o64.363.163.763.70
VLMGPT-4.166.165.365.665.67
VLMGemini-2.5-Pro66.863.066.365.37
VLMGemini-3-Flash63.860.462.362.17
VLMGemini-3.5-Flash65.063.765.264.63
VLMClaude-4.5-Sonnet64.864.466.665.27
VLMClaude Opus 4.763.661.064.863.13
VLMQwen3-VL-235B61.465.162.963.13
Code-augmentedDeepSeek-R168.971.867.269.30
Code-augmentedQwen3-Coder-32B63.065.762.463.70
Code-augmentedTableGPT253.456.152.854.10
Deep ResearchGemini Deep Research67.966.571.368.57
Deep ResearchChatGPT Deep Research65.762.367.165.03
Deep ResearchPerplexity Sonar DR63.962.562.062.80

Conclusion: The three-judge results show that the main findings are largely robust, DeepSeek-R1 remains the top model (average 69.3), and code-augmented systems still lead, but individual scores shift substantially across judges (e.g., Qwen3-VL: 61.4 vs. 65.1; Gemini Deep Research: 67.9 vs. 71.3), and several adjacent rank swaps occur. This confirms that GPT-4o-only judging is insufficient and that a multi-judge panel is necessary. The accurate claim is therefore that overall rankings are largely stable across judges.

Question 1 (Weaknesses 4):Thank you for point this out. DF,EX,AE and TC are on 0-5 scale at first; The V-Score is the average of DF, EX, AE and TC. Then the four values and V-Score are normalized to [0,100] respectively. Because the displayed dimension scores are individually normalized and rounded, their simple average does not exactly reproduce the displayed V-Score. We will calculate the V-Score after normalization of the four values and report 81.0 for DeepSeek-R1 line.

Question 2:GPT-4o with vision was chosen for stability, cost, reproducibility, and vision support at the time of experiments, and it was calibrated against human ratings.

Question 3:The task input is structured tables plus a text instruction. During generation, a VLM’s vision capability is used when the model produces charts through native multimodal chart generation/rendering (Vision-Language Models); if charts are generated via code execution, the model’s vision capability is not used in the generation stage (Code-augmented Models).

Thank you again for these constructive comments.

Thank you for the comparison with prior benchmarks. These address my main concerns, so I have raised my rating.

Thank you very much for your positive response, which has helped us sharpen the contribution and presentation of the benchmark.

Reviewer_DV5WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MM-ReportBench, a benchmark for generating multimodal reports from tabular data. It includes 386 tasks based on 185 tables across six business domains, along with curated reference reports and a dual-judge evaluation framework for assessing text, visualizations, and cross-modal consistency. The authors evaluate 20 models and report weaknesses in numerical accuracy, insight novelty, and visualization quality.

Critical strengths
  1. The task is relevant and more demanding than table question answering or text-only table-to-report generation: it evaluates narrative, charts, and their consistency together.
  2. The benchmark covers multiple domains and includes large and complex tables, which could support useful analysis of current models’ limitations.
  3. The paper attempts to evaluate more than surface-level fluency, including numerical grounding, chart fidelity, and insight quality.
Critical weaknesses
  1. The justification for some evaluation metrics is insufficient. For example, when calculating GIVcov_{cov}, the paper uses probabilities rather than embedding-based similarity, but does not explain why this choice is appropriate or how it affects the metric’s validity. Moreover, many math terms are used without explanation, making it hard to understand.

  2. The text and visualization dimensions are largely evaluated separately, leaving the benchmark’s central contribution unclear. Existing work already evaluates text generation and visualization quality individually. Although MM-ReportBench also considers cross-modal consistency, the paper does not clearly explain how this joint evaluation goes beyond those prior approaches or how the text and visualization scores are combined to assess the quality of a multimodal report as a whole.

  3. The evidence that the judge scores are reliable is insufficiently documented. GPT-4o is used for several subjective dimensions, including structural completeness, novelty, expressiveness, and aesthetics. The paper reports correlations with human ratings on 100 reports, but does not provide enough detail about how reports were assigned to raters, how the rubric was calibrated, or how the correlations and inter-rater agreement were computed.

Questions for authors

See the weaknesses above

5 comments

Thank you for these detailed and constructive comments.

Response 1. We agree with the reviewer. Placing the full derivation in Appendix A.3 is not sufficient: the main-text equation for GIV_cov currently uses several mathematical terms without defining them, which harms readability and makes the metric look under-specified. We will revise the main text. The goal of GIV_cov is "coverage": whether each reference keypoint is covered by at least one generated sentence, while penalizing coverage obtained through many repetitive or generic sentences . We therefore use normalized mutual information (NMI) rather than directly averaging cosine similarities. NMI rewards both coverage and unique matching, and penalizes omissions and repetition. Average similarity is affected by sentence length, generic sentences, and the number of keypoints, making it difficult to interpret as a “coverage percentage”; Also similarity can be inflated by repeated sentences and insensitive to omissions. We compare GIV_cov with “average maximum BERTScore matching” on 100 stratified sampled reports:

Coverage metricSpearman ρ\rho with human coverage ratingCorrelation with report length (lower is better)Score inflation from repeated sentences
Average maximum BERTScore matching0.660.41+12.3
NMI (our GIV_cov)0.790.12+3.1

These results show that NMI is more consistent with human coverage judgments and is less affected by length and repetition artifacts.

Response 2: Our core contribution is: under the same task, same source tables, and same reference reports, models are required to generate text and embedded charts simultaneously, and a dual-judge plus cross-modal consistency design explicitly constrains the two modalities from compensating for each other in evaluation. Unlike prior work that evaluates text generation or chart quality in isolation, MM-ReportBench evaluates the complete multimodal report.

We add a newly added joint-evaluation ablation experiment:

SettingOverallT-JudgeV-JudgeDescription
Gemini-2.5-Pro full multimodal66.859.078.4Table 2
Remove visualizations, text only38.2—0Figure 3
Visual only, no text46.5——Figure 3
DeepSeek-R1 full68.961.3580.2Table 2
DeepSeek-R1 w/o CMC66.458.980.2New: Overall −2.5-2.5

Removing visualizations not only sets the V score to zero, but also lowers T-Judge GIV from 57.3 to 49.1 and NA from 35.5 to 29.3. This suggests that jointly generating charts in turn pushes the model to verify data more carefully; textual quality is not independent of visuals. Visual-only without text scores only 46.5, indicating that charts alone are insufficient for a high-quality report without narrative. These results support the necessity of “joint evaluation,” rather than a simple sum of two independent evaluations.

Response 3:

  • We stratify 100 generated reports from the 386 tasks by six domains, four model categories, and Overall score quintiles, ensuring coverage of strong/medium/weak models.
  • Each report is independently scored by 3 annotators: 2 domain experts and 1 senior expert. Annotators are blind to model identity, judge scores, and other annotators’ scores.
  • The rating dimensions are consistent with the judges: SC, GIV, NA, EX, AE, and Overall.

Rubric calibration: Two rounds of calibration are conducted before formal annotation:

  • Using 20 calibration reports, we provide positive and negative examples for each score anchor (e.g., anchors for SC at 1/3/5, and poor/medium/excellent chart examples for AE);
  • After the 3 annotators independently score, disagreements are discussed and rubric wording is revised; the calibration set is excluded from the formal 100 reports.

We add a set of reliability verification experiments for judges:

DimensionJudge–human Spearman ρ\rho95% CIHuman ICC(2,k)Proportion within 1 point of human mean
SC0.78[0.69, 0.85]0.8486%
GIV0.83[0.75, 0.89]0.8789%
NA0.71[0.60, 0.80]0.7982%
EX0.74[0.64, 0.82]0.8184%
AE0.69[0.58, 0.78]0.7783%
Overall0.82[0.74, 0.88]0.8687%

Human Fleiss’ κ\kappa (ordinal bins) is 0.76–0.84.

As an additional robustness check on the judge model, we rerun the 100 reports with Qwen3.5-9B and Claude-3.5-Sonnet: their Overall Spearman correlations with the GPT-4o judge are 0.88 and 0.85, respectively, and model-ranking Kendall τ\tau is 0.91 and 0.87, respectively. This suggests that the overall scores and model rankings are largely robust to the choice of judge model, although subjective dimensions may still exhibit model-specific variation and warrant further validation.

We thank the reviewer again for these constructive comments.

Thank you for your response.

Q1. I appreciate the additional ablation study and the formula provided in A3. I suggest briefly introducing the math terms in the main text so that readers can see how the scores are computed.

Q2. The authors state, “This differs from prior work that separately evaluates text generation or chart quality.” However, most of the proposed metrics still assess text and visualizations separately.

Thank you very much for your response.

Q1. Thank you for your suggestion. We will add a concise explanation of the GIV_cov terms directly in the main text (Section 3.3), rather than leaving them only in Appendix A.3. Specifically, after Equation (3), we will add a short paragraph defining:

  • kik_i: the ii-th reference keypoint;
  • sjs_j: the jj-th generated report sentence;
  • P(ki)P(k_i): the prior probability of keypoint kik_i;
  • P(sj∣ki)P(s_j \mid k_i): a temperature-scaled softmax over BERTScore similarities between kik_i and all generated sentences;
  • P(ki,sj)P(k_i, s_j) and P(sj)P(s_j): the joint and marginal probabilities derived from the above

Q2. The statement is easy to misread, and we replace it with a more precise statement: “Unlike prior work that evaluates text generation or chart quality in isolation, MM-ReportBench evaluates the complete multimodal report.” T-Judge and V-Judge are separated for diagnostic purposes and to prevent cross-modal compensation, while CMC and source-table-grounded fact verification explicitly couple the two modalities within the same task.

Thank you very much again for the valuable suggestions. We hope the revised version will merit your positive consideration.

Comments on the paper

1 comment

We sincerely thank the reviewers for their constructive and helpful feedback. We have carefully revised the manuscript as follows:

Main text

(1) In Section 2 (Related Work), we have added a discussion of prior work, including InsightBench, DataNarrative, Multimodal DeepResearcher, and FinSight, together with a comparison table to clarify the distinctions between these works and MM-ReportBench.

(2) In Section 3.3, we have added an introduction to the mathematical terms used in the computation of GIV_cov, so that readers can more easily follow how the score is calculated.

Appendix

A.4 adds a controlled comparison between GIV_cov and average maximum BERTScore matching.

A.7 adds Judge Panel Experiment Results.

A.9 adds Human Evaluation and Consistency.

We are grateful for the reviewers’ careful reading and thoughtful suggestions, which have helped us improve the clarity and rigor of the manuscript. We believe these revisions address the concerns raised and make the paper more transparent and rigorous.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.