- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the issue of content faithfulness when automatically generating presentation videos from research papers. The authors note that while existing systems can produce fluent and complete videos, they may present claims unsupported by the source paper or misstate numerical values and comparative relationships. To address this, the paper introduces the PRESENTFAITH benchmark, which extracts atomic claims from video narrations and slides, then verifies them against evidence indexed within the paper—down to the level of individual table cells. Furthermore, the authors propose the EVI-IR generation framework, which tracks the evidence source for every generated claim and identifies and corrects inconsistencies prior to video rendering. Experiments conducted on 40 research papers demonstrate that, compared to the strongest baseline, EVI-IR reduces the proportion of unfaithful claims from 17.6% to 12.5% and numerical discrepancies from 9.7% to 2.6%. Simultaneously, it increases the number of paper-supported claims per minute while maintaining the overall quality of the presentation.
Critical strengths
- Build fine-grained fidelity reviews for final videos. It checks the assertions in the narration and slides one by one, and locates the numbers to specific table cells in the paper, thereby discovering the problem of "the numbers exist, but the attribution or comparison relationship is wrong."
- Move evidence verification earlier into the video generation process. Each assertion is associated with the original text evidence when it is generated, and is rendered after being checked and partially corrected. Experiments show that this reduces claims of disloyalty while maintaining video presentation quality.
Critical weaknesses
- The "verifiable" scope of the method is limited. EVI-IR’s numerical checks primarily address relationships between, at most, two evidence units; broader conclusions drawn across multiple experimental setups, information embedded within figures, and descriptive assertions still largely rely on the judgment of language models. The paper reports that 12.5% of the assertions regarding the videos remain unfaithful, yet it does not quantify the specific proportions of the various remaining types of errors.
- Experiments are not yet sufficient to support broad applicability. The 40 test papers were screened and all were numerically dense English papers; the main experiments all used LaTeX source files. The paper acknowledges the degradation in quality of evidence indexing resulting from PDF parsing, but does not test this input in the main experiment. Therefore, there is currently no evidence whether papers with fewer tables, relying on complex graphics to express results, or only PDFs will achieve similar benefits.
Questions for authors
- On the same batch of manually annotated video judgments, compared with existing visual language model evaluation methods, how many more errors of "numbers correspond to wrong objects" or "comparison directions indicate the opposite" were identified by PRESENTFAITH, and how many more false positives were added?
- How will EVI-IR's proportion of unfaithful assertions, evidence pointer accuracy, and abstention rate change if the input is only PDF, or if the paper's conclusions are mainly supported by complex graphs and multiple experimental results?
2 comments
Q1 and Q2
>>> Q1: Detailed of the PresentFaith Results
>>> A3:
Thanks for your questions and it is valuable.
Supplementary experiment
Our verifier already agrees with the human majority on 93.4% of claims. However, as requested, we conducted additional experiments to compare its results with those of visual language model: We replaced the step in PRESENTFAITH where it determines whether a paper supports a claim in a video with a visual language model judge (Gemini-2.5-Flash). Then, on the same manually annotated sentences, we compared the two and the results are as follows:
| Human-labeled category | claims | VLM judge detects | PRESENTFAITH detects |
|---|---|---|---|
| Number bound to the wrong object (row/method/setting) | 71 | 27 (38.0%) | 63 (88.7%) |
| Reversed comparison direction | 46 | 19 (41.3%) | 41 (89.1%) |
| Other unfaithful claims | 135 | 104 (77.0%) | 113 (83.7%) |
| False positives on clean claims (n = 988) | — | 13 (1.3%) | 16 (1.6%) |
| Agreement with human majority (all 1240) | — | 85.1% | 93.4% |
Summary
On the same claims, PRESENTFAITH finds 58 more wrong-object and reversed-comparison errors (+36 and +22) than the VLM judge, at the cost of 3 more false positives (1.3% → 1.6% FPR). The VLM judge misses most of these relational errors because every value in them does appear in the paper, so a judge that checks for the value's presence accepts the claim. This matches the injected-error study in Appendix B.3 (38% / 31% detection). PRESENTFAITH instead looks up the cells the claim names and recomputes the relation. We have added this comparison to Table 14 (Appendix C.11).
>>> Q2: Evaluation of EVI-IR for different inputs
>>> A4:
(a) PDF input
As we mentioned in A2-(b), we have reported the evaluation results of pdf as the input.
(b) Complex papers
Experimental Setup
Thank you very much for your points out the shortcomings. Full annotation is costly, so we selected 3 additional papers outside PRESENTFAITH. In each, more than half of the key claims in the abstract and conclusion are supported only by figures or by three or more experimental results. We ran EVI-IR and PaperTalker on them. In addition, we also ran PaperTalker on these papers, in order to facilitate the comparison of the variation range. We have added the details in Table 15 (Appendix C.12).
| Setting | System | UCR↓ | NCR↓ | Ptr. acc.↑ | Abstain (%) |
|---|---|---|---|---|---|
| PRESENTFAITH (40 papers) | PaperTalker | 17.6 | 9.7 | — | — |
| EVI-IR | 12.5 | 2.6 | 84.6 | 12.0 | |
| Complex (3 papers) | PaperTalker | 26.9 | 11.8 | — | — |
| EVI-IR | 19.8 | 3.4 | 81.7 | 21.5 |
Summary
- Both systems degrade on these papers, and EVI-IR still keeps a clear margin. UCR rises 12.5 → 19.8 (per-paper range 17.2–22.6), and the gap to PaperTalker is 7.1 points. NCR barely changes, because table-backed numeric claims are still checked symbolically.
- Most of the increase comes from where the symbolic layer does not reach. Of the 7.3-point UCR increase, figure-only information accounts for 4.1 points and relations over three or more units for 2.2. These are the same two categories identified in Appendix C.8.
- The pointers that are emitted stay accurate, while abstention nearly doubles. Claims about figure internals have no index unit to point to, so the system abstains instead of emitting a wrong pointer. Abstained atoms are still rendered without a binding check, however, and they are the main source of the higher UCR.
These results point to two extensions in the future: evidence units for figure internals and n-ary relation checks. We have added both to the Limitations section as future work.
Thanks Reviewer Z8kJ for the helpful suggestions
We greatly appreciate Reviewer Z8kJ's helpful comments, which has helped us to further refine this work.
>>> W1: Scope of verification and breakdown of error types>>> A1:(a) Scope of numerical verification.
Currently, our numerical checks do involve at most two evidence units, and some broader conclusions still rely on the judgment of language models. We deliberately keep the checks simple because this is the first work to verify paper-to-video generation content at the claim level. We chose a reliable and simple form - pairwise comparison. It has already proven to be superior to previous work, so we did not delve further into this aspect in this paper.
However, we fully agree with your suggestion. As stated in the limitation section, we will support more numerical types of checks in subsequent work and improve the numerical verification mechanism, , which is very meaningful.
(b) Breakdown of error types.
We thank the reviewer’s finding .As requested, we have reported the specific proportions of EVI-IR's unfaithful claims rate (UCR 12.5%) as
Table 11 (Appendix C.8)and summarize it here:Setup
We collected all 1,442 unfaithful claims from EVI-IR's delivered videos. Each claim was traced back to its IR atom (pointer, type, V1–V4 outcome, repair history). Two non-author annotators then assigned it a single error type from a taxonomy fixed before labeling; a third resolved disagreements (κ = 0.74).
Results
It can be seen from the table that the residual is dominated by the scope of verification (71%, 8.9 pts). And errors inside the symbolic layer total only 0.9 pts. In addition, about 20% consists of verifier false positives, perception errors, and rendering alteration.
>>> W2 Broad applicability>>> A2:Thanks for your suggestions, it’s very important for us. We will make the following explanations:
(a) For the dataset diversity.
We selected 40 papers to avoid including unrepresentative ones and ensure that the dataset is not overly influenced by certain special articles in the evaluation results. Although the screening process was conducted, the complexity and focus of these 40 papers vary, and there are also some papers with fewer tables and relying on complex graphical representations to express their results. Table 1 reports these differences, which to some extent reflects the robustness of EVI-IR. However, we fully agree with your suggestion to explore more complex situations, as it is meaningful. In the next study, we will collect more articles in the dataset that rely on complex charts as conclusions. Additionally, we also plan to expand the data scale and domain diversity in the same way.
(b) For PDF input.
Our system supports PDF input. Our focus is on the link between the paper itself and the generated claims, while PDF introduces an extra OCR step and thus extra loss, so the main experiments only reported the better LaTeX-based results. As requested, we report the evaluation results with PDF as input below (for clarity, we also copy the LaTeX-based results of EVI-IR):
The results show that the evaluation result of PDF as the input has a slight decrease compared to LaTeX, but the overall performance is still good. So the generation quality of EVI-IR is not significantly affected by the input type.