acceptodds
Under review as a conference paper at ICLR 2027

FigAct: Turning Scientific Figures into Active Canvases for Explanation

25 views

Abstract

Scientific figures are designed to communicate information visually, yet MLLMs typically explain them by translating their visual content back into text. This requires readers to manually map the resulting explanations back to the figure. Inspired by how people present visual information, we introduce FigAct, a framework that transforms static scientific figures into question-conditioned visual presentations by acting directly on their existing graphical elements. Like a human presenter, FigAct generates a sequence of short narrations, grounds each narration in the corresponding visual evidence, and applies visual actions to guide the viewer's attention. We develop a hierarchical search strategy for efficient element localization, reducing token usage by approximately 40x. We further train FigAct-8B using three task-specific rewards for grounding accuracy, search efficiency, and rendering quality. We further build a human-verified benchmark from figures in real-world scientific papers to evaluate the ability of MLLMs to generate grounded visual explanations. Our results demonstrate the effectiveness of FigAct and show that treating scientific figures as presentation canvases makes explanations clearer and easier to follow.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Understanding, Scientific Figure, Multimodal Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_usv5OpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

FigAct turns questions about scientific figures into visual presentations through a Plan–Ground–Act pipeline. It combines hierarchical SVG search with a deterministic renderer and trains FigAct-8B through SFT and RL. The paper evaluates the framework on 400 real-world question–figure pairs and in a user study.

Critical strengths

Linking narration directly to elements in the original figure can reduce the effort readers spend mapping text back to visual evidence. The framework improves grounding and action execution across several backbones. The appendix provides implementation details, ablations, and failure cases, giving the work practical and research value.

Critical weaknesses
  1. The evaluation does not sufficiently distinguish successful presentation generation from improved understanding.

Comp. and Exec. approach saturation after adding the framework, but mainly measure output completeness and executability. Video–narration similarity and model-based presentation scores also provide limited evidence that added annotations or inferred relationships are scientifically correct. The user study offers encouraging evidence, but it is unclear whether conditions use identical narration and comparable viewing time. Uncertainty accounting for participant and question differences is also missing. It is therefore difficult to attribute the improvement in comprehension specifically to visual actions.

  1. The independent benefits of hierarchical search and training remain unclear, and some reported results are inconsistent.

The framework changes SVG grouping, visual inputs, search, and action execution together. The dual-view ablation is useful, but a comparison between flat selection and hierarchical search under the same grouped representation, views, and budget is still needed. The reported 40–50× reduction concerns SVG tokens; the total cost of visual inputs, model calls, and rendering remains unclear. FigAct-8B improves substantially over its own baseline, but its grounding F1 and Pres-Q remain below Claude Fable 5.1 and GPT-6 Astra when both use FigAct. The paper should establish the smaller model’s quality–cost trade-off more clearly. In particular, RL raises grounding F1 from 52.67 to 62.27 while reducing Pres-Q from 82.47 to 82.16, leaving its benefit to final presentation quality uncertain. The appendix explains that SFT search trajectories are reconstructed from known final targets, but the supplementary package does not provide inspectable training trajectories or run records. More evidence is needed on how this ideal-path supervision transfers to actual search and whether RL adds value beyond an additional SFT budget. The numerical inconsistencies also matter: SFT grounding F1 is 52.67 in the main table but 60.67 in Table 8, while SFT Pres-Q is 82.47 in the main table but 80.56 in Table 6.

Questions for authors
  1. Does the user study control narration content and viewing time? How large is the benefit of visual actions after accounting for participant and question differences?
  2. What does “reference propagation” mean? Do downstream stages receive predictions or reference annotations, and which results evaluate the complete predicted pipeline?
  3. Under matched representations and budgets, what are the separate gains from hierarchical search and RL? What is FigAct-8B’s quality–cost trade-off relative to the stronger models?
  4. Can you provide training trajectories and run records and reconcile the inconsistent SFT results across tables?

I see value in the task and practical promise in the framework, but the evidence for comprehension gains, component contributions, and training benefits is not yet sufficient. I would consider raising my score if the authors clarify the evaluation protocol, reconcile the reported results, and provide targeted evidence for the independent benefits of visual actions and training.

Reviewer_A8KhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

FigAct is a framework that converts figures in scientific papers into visual presentations that explain the content step by step in response to user questions. This method consists of three stages: Plan, which breaks down the explanation into multiple short steps; Ground, which identifies elements in the figure corresponding to each step; and Act, which selects visual operations such as zooming, highlighting, arrows, and annotations. In the Ground stage, the SVG is reorganized into a geometric hierarchical structure, and relevant elements are efficiently identified through a coarse-to-fine search, significantly reducing token usage compared to directly handling the entire SVG. Furthermore, FigAct-8B is trained using supervised and reinforcement learning based on Qwen3-VL-8B. Evaluation using real-world scientific figures demonstrates improvements in grounding, visual action generation, and overall presentation quality.

Critical strengths
  1. This paper addresses the problem of generating visual presentations based on scientific figures, rather than simply generating captions. This problem setting is highly novel and well-suited to real-world scientific communication. In particular, the concept of presenting information in stages while linking explanatory text to visual evidence within the figure aims to reduce the cognitive load on readers, making it a highly practical problem setting.

  2. FigAct breaks down the explanatory content into three stages; Plan, which structures the explanation; Ground, which identifies corresponding elements within the figure; and Act, which determines the visual presentation method, resulting in a clear overall design. In particular, the Ground stage reorganizes complex SVG into a geometric hierarchical structure and performs coarse-to-fine search, thereby efficiently identifying relevant elements while significantly reducing token usage. Furthermore, since this framework can be applied to multiple proprietary and open-source MLLMs, it holds high value as a model-agnostic method.

Critical weaknesses
  1. Table 1 demonstrates the effectiveness of FigAct primarily using automated evaluation metrics; however, the extent to which these metrics correspond to human understanding or perceived comprehensibility has not been established. Although the 400 real-world benchmark examples are constructed using model-assisted annotation followed by human review and refinement, the reliability of the resulting annotations is not quantitatively assessed, for example through inter-annotator agreement or independent re-annotation. Furthermore, Pres-Q relies on an MLLM judge, yet its agreement or correlation with human evaluations is not reported; given that these metrics form the basis of the paper’s main quantitative conclusions, their validity should be examined more carefully.

  2. The human evaluation in Table 3 does not include a comparison between using FigAct and not using FigAct with the same backbone. Consequently, it is impossible to directly determine whether the improvements in comprehension and visual clarity observed in the human evaluation are due to FigAct or to the capabilities of the underlying MLLM itself. Given that one of the central claims of this paper is that FigAct makes scientific figures easier for humans to understand, this lack of a matched comparison constitutes a significant evaluative weakness.

Questions for authors

Please see the weaknesses.

Reviewer_bjifOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes Visual Question Presentation (VQP), which produces a temporally structured presentation that acts directly on the figure's existing vector elements instead of answering a question about a scientific figure with text alone. Each presentation "beat" pairs a short narration with a set of grounded SVG elements and high-level visual actions, executed by a deterministic renderer. The proposed FigAct framework decomposes generation into three stages: Plan, Ground, and Act. Experiments reported results show large grounding gains for all backbones with roughly 40–50x lower SVG token usage, further gains from SFT+RL.

Critical strengths

1. A genuinely new and cleanly formalized task. Relative to the cited figure/chart-QA lines (SPIQA, ChartQAPro, MathVista) and paper-to-slides/video systems that treat figures as static assets, answering by acting on the original figure's existing elements through temporally ordered beats is a distinct problem.

2. Well-designed supervision and training detail. Grounding trajectories are reconstructed deterministically from final target, matching the inference-time protocol and avoiding noisy per-step annotation.

3. Broad ablation coverage. Individual RL rewards (Table 2), context vs. isolated views with a stricter [email protected] metric (Table 5), standard vs. stage-aware GRPO under identical initialization (Table 6), rollout group size (Table 7), synthetic-to-real transfer (Table 8), complexity scaling (Fig. 5), and action distributions (Table 9). The RL reward encoder (PE-Core) is deliberately separated from the NVA evaluation encoder (Qwen3-VL-Embedding). They gives a complete ablation variations for understanding the usage of components.

Critical weaknesses

1. No external or intermediate baselines; gains partly measure input infeasibility. The only non-FigAct baseline feeds the raw ~1M-token SVG to the backbone (Sec. 7.1, p. 7), a setting dominated by context overflow (25.5% fit; InternVL3.5-8B: 0.00 completion; Table 1, p. 8), cited narrated-explanation systems (TheoremExplainAgent, Paper2Video, PresentAgent) are never compared, and no intermediate baseline (single-round Set-of-Mark over the same grouped forest, or the simplified grouped SVG fed directly) decomposes gains among preprocessing, grouping, and multi-round search. The paper also never states how over-context SVGs were handled for baselines or how raw element predictions are canonicalized against forest-derived references, and all automatic metrics are author-defined on the authors' own benchmark.

2. Token-efficiency headline omits the image tokens FigAct introduces. The 40–50x reduction counts only SVG text tokens, ignoring the Set-of-Mark context view plus isolated candidate views rendered every search round (Sec. 4.2, p. 4, itself notes image-token pressure); the abstract drops the "SVG" qualifier entirely, and no total-token or latency accounting appears anywhere.

Questions for authors

The authors are expected to stress the questions raised in the weakness section.

8 comments
AuthorsOpenReview ↗

Response for Reviewer bjif (Part 1/5)

We thank the reviewer for the thoughtful review and for recognizing the task formulation, supervision design, and ablation coverage. Below, we provide additional baseline comparisons and efficiency reporting.

Q1: Baseline fairness and Missing intermediate baselines

Your Concern: Gains over non-FigAct baselines may partly reflect context overflow. Intermediate baselines are needed to distinguish the contributions of grouping and hierarchical search.

Our Response: We extend the view ablation in Appendix E.1 (Table 5) with additional intermediate baselines to disentangle the contributions of grouping, hierarchical search, and candidate visualization.

  • Existing ablation: Table 5 compares context-only (Set-of-Mark), isolated-only, and dual-view inputs over the same grouped forest with hierarchical search.
  • New Evaluation. We add two single-round flat-selection baselines: A uses ungrouped element candidates, while B uses grouped candidates. C–E retain the original settings and evaluate the effect of the context/isolated view.
    • Contribution of grouping (A vs. B). With dual-view inputs and no hierarchical search, grouping improves Elem. F1 by 2.08 and 5.28 points for Gemini 3.5 Flash and Qwen3-VL-8B, respectively.
    • Contribution of hierarchical search (B vs. E). Hierarchical search further improves Elem. F1 by 7.01 and 6.82 points for Gemini 3.5 Flash and Qwen3-VL-8B, respectively.
    • Contribution of dual view (C–E). Compared with context-only Set-of-Mark input, the isolated view provides substantially improvement. Dual view achieves the best Elem. F1 and GS scores for both models, while isolated-only yields slightly higher Search-E for Qwen3-VL-8B.
ModelIndexGroupingContext view (SoM)Isolated viewHierarchical searchElem. F1 ↑[email protected] ↑[email protected] ↑Search-E ↑
Gemini 3.5 FlashA—✓✓—59.7660.9137.48—
B✓✓✓—61.8463.5840.97—
C✓✓—✓60.9964.4740.5389.50
D✓—✓✓67.0986.8758.6395.18
E✓✓✓✓68.8589.6359.8295.35
Qwen3-VL-8BA—✓✓—16.2520.473.43—
B✓✓✓—21.5323.9413.77—
C✓✓—✓17.7521.873.9063.75
D✓—✓✓26.4225.1016.1769.73
E✓✓✓✓28.3527.8316.9268.26
  • Why not compare with a grouped-SVG baseline? Grouping only records geometric relationships among the original SVG elements in a separate JSON file; it does not simplify or shorten the SVG itself. We revised Sec. 4.2 to make this distinction explicit.
AuthorsOpenReview ↗

Response for Reviewer bjif (Part 2/5)

Q2. Why not compare with related visual explanation systems?

Your Concern: Missing comparisons with external baselines like the cited work TheoremExplainAgent, Paper2Video, and PresentAgent.

Our Response: These systems are relevant, but they do not produce outputs aligned with the VQP task. TheoremExplainAgent generates animations from textual/theorem input, while Paper2Video and PresentAgent generate document-level videos or slides. In contrast, VQP requires a question-conditioned sequence of grounded actions over the original SVG elements. As a result, these systems do not output the element-level grounding, beat structure, or action predictions required by our benchmark, so metrics such as Elem. F1, Search-E, and Exec. cannot be meaningfully computed for them.

As a complementary comparison, our human study uses baseline conditions that capture capabilities shared across several lines of prior work. Text condition represents standard VQA-style figure understanding. Highlight condition captures visual grounding and attention guidance, as in evidence-localization tasks and systems such as Paper2Video that use cursor movement to direct attention. Full FigAct further adds dynamic presentation actions over the original figure.

AuthorsOpenReview ↗

Response for Reviewer bjif (Part 3/5)

Q3. How are over-context SVGs handled for the baselines?

Our Response: If a baseline cannot complete a sample because the SVG exceeds its context limit, this is reflected in its completion rate. Other metrics, like grounding F1, are computed only over successfully completed samples. Therefore, over-context failures are not additionally counted as failures in these conditional metrics. We added a clarification to the Completion metric description in Appendix D.

Q4. How are raw element predictions canonicalized against forest-derived references?

Our Response: All grounding outputs are canonicalized to the same set of terminal SVG element IDs before evaluation. Forest-level group predictions are recursively expanded to their underlying terminal elements, while raw element predictions remain unchanged. Elem. F1 is then computed against the reference terminal-element set in this shared ID space. We added this implementation detail to Appendix D.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response for Reviewer A8Kh (Part 2/3)

Q2. Human Validation of Pres-Q

Your Concern: Pres-Q uses an MLLM judge, but its agreement with human judgments is unknown.

Our Response: We clarify that Pres-Q is not intended as a proxy for human comprehension. It is a scalable, rubric-based automatic metric for evaluating presentation quality, following recent multimodal presentation evaluation practices (e.g., PresentAgent and Paper2Video). Human comprehension is evaluated separately through our controlled human study.

AuthorsOpenReview ↗

Response for Reviewer A8Kh (Part 3/3)

Q3. Missing Matched Human Baseline

Your Concern: Table 3 lacks a same-backbone comparison with and without FigAct.

Our Response: We agree that a same-backbone comparison provides an important complementary end-to-end control.

  • Original design. Table 3 was designed to factorize the human-facing contributions of FigAct: Text → Highlight measures the benefit of grounding explanations in visual evidence, while Highlight → Full Presentation measures the additional benefit of dynamic presentation actions.

  • New matched baseline. We additionally evaluate Claude Fable 5.1 with vs. without FigAct under the same human-evaluation protocol. Samples exceeding the FigAct-off model's context window are excluded for a fair comparison. We also added semi-structured interviews and full implementation details to the appendix D2.

  • Results. FigAct improves all four measures: comprehension accuracy increases from 81.5% to 86.7% (+5.2 pp), while comprehension ease, visual clarity, and visual appeal improve by +0.32, +0.68, and +0.47, respectively. In addition, 7/18 participants noted that FigAct-off presentations relied more heavily on repetitive local highlighting and showed less diversity in visual actions.

MethodFigActComp. Acc. ↑Comp. Ease ↑Visual Clarity ↑Visual Appeal ↑
Claude Fable 5.1✗81.5%5.064.925.17
Claude Fable 5.1✓86.7%5.385.605.64
AuthorsOpenReview ↗

Response for Reviewer usv5

Q1. Robustness of comprehension gains from visual actions to participant and question differences

We conducted mixed-effects analyses with crossed random intercepts for participants and question–figure pairs (Appendix D.3, Table 5). Specifically, we fitted a binomial mixed-effects model for comprehension accuracy and linear mixed-effects models for the three Likert-scale measures: Outcome ~ Condition + (1|Participant) + (1|Question-Figure Pair).

Both FigAct-8B (p = 0.036) and Claude + FigAct (p = 0.006) show statistically significant improvements in comprehension accuracy over Highlight, with significant positive effects also observed in comprehension ease, visual clarity, and visual appeal. Although participant-level variance is consistently larger than question-level variance, these effects remain statistically significant after accounting for both sources of variability, supporting the robustness of benefits of visual actions.

Outcome / EffectCoef.SEp
Comprehension Accuracy
Intercept (Highlight)1.1250.284–
Claude Fable 5.1 + FigAct0.9290.3350.006**
FigAct-8B0.6150.2940.036*
Question–Figure Pair Var0.2880.101–
Participant Var0.4630.142–
Comprehension Ease
Intercept (Highlight)4.6600.148–
Claude Fable 5.1 + FigAct0.6920.2300.003**
FigAct-8B0.4590.1940.018*
Question–Figure Pair Var0.1760.074–
Participant Var0.3810.108–
Visual Clarity
Intercept (Highlight)4.8700.165–
Claude Fable 5.1 + FigAct0.7340.180< 0.001***
FigAct-8B0.6820.196< 0.001***
Question–Figure Pair Var0.1380.086–
Participant Var0.3020.091–
Visual Appeal
Intercept (Highlight)4.7800.161–
Claude Fable 5.1 + FigAct0.8260.183< 0.001***
FigAct-8B0.6630.2080.0014**
Question–Figure Pair Var0.2140.073–
Participant Var0.4260.116–

Significance: * p < 0.05; ** p < 0.01; *** p < 0.001.


Q2. Matched-Compute SFT vs. RL

We report a matched-compute comparison between continued SFT and RL, both starting from the same SFT checkpoint (Appendix E.1, Table 11).

  • Limited gains from continued SFT. Additional SFT improves Elem. F1 by only +1.18 and NVA by +0.07, while Pres-Q decreases from 82.47 to 81.93, indicating limited benefits from further supervised training at this stage.

  • RL delivers substantially greater gains while better preserving presentation quality. Under matched compute, RL outperforms continued SFT by +8.42 Elem. F1 and +2.50 NVA, with a higher Pres-Q (82.16 vs. 81.93). Notably, although further training slightly reduces Pres-Q relative to the original SFT checkpoint in both settings, the decrease is smaller with RL (−0.31 vs. −0.54), despite its substantially larger grounding and alignment gains.

SettingElem. F1 ↑Search-E ↑Exec. ↑NVA ↑Pres-Q ↑
SFT checkpoint52.6783.0495.6669.6882.47
+ Continued SFT53.8582.9196.1169.7581.93
+ RL62.2784.6897.1072.2582.16
AuthorsOpenReview ↗

Summary of Revisions

Dear Reviewers and ACs,

We introduce Visual Question Presentation, a new task that transforms a static scientific figure into a question-conditioned, dynamic visual presentation. We develop a model-agnostic framework that consistently improves different MLLMs while reducing token usage by 40–50×. We also contribute a high-quality real-world benchmark with human-verifiable evaluation and a scalable supervision strategy for training an open 8B model.

We appreciate the reviewers' recognition of (1) the novelty and practical value of Visual Question Presentation (Reviewers bjif, A8Kh, and usv5), (2) the clear, model-agnostic Plan–Ground–Act framework (Reviewer A8Kh), and (3) the supervision design and broad ablation coverage (Reviewer bjif), and (4) the framework’s broad applicability and consistent gains across MLLMs (Reviewers A8Kh and usv5).

We summarize the reviewer feedback and key updates below. We have uploaded a revised PDF in which every change is shown in blue.

Additional Experiments and Analysis

  • Intermediate Grounding Baselines: We added flat-selection baselines to distinguish the contributions of grouping and hierarchical search. (Reviewer bjif Q1; Reviewer usv5 Q5; Appendix E1)
  • Efficiency Measurements: We added reporting of image tokens, model calls, and latency. (Reviewer bjif Q6; Reviewer usv5 Q4, Appendix E2, Table 13)
  • Matched-Compute Continued SFT vs. RL: We compared continued SFT with RL under the same additional training budget. (Reviewer usv5, Appendix E.1, Table 11)
  • Robustness Analysis to Participant and Question Differences: We added mixed-effects analyses with crossed random intercepts for participants and question–figure pairs. (Reviewer usv5, follow-up Q1; Appendix D3, Table 5)
  • Matched Same-Backbone Human Evaluation: We added a new end-to-end baseline and provided semi-structured interviews results. (Reviewer A8Kh Q3; Appendix D3, Table 6) -Independent Annotation Reliability Audit: We added inter-annotator agreement analysis on the benchmark. (Reviewer A8Kh Q1; Appendix B)

Clarifications

  • SVG Grouping: We clarified that grouping is a method to record geometric relationships among SVG elements, it is not a new representation that simplify or shorten the original SVG source. (Reviewer bjif Q1)
  • Task Scope and Related Systems: We clarified how VQP differs from related document-to-presentation and theorem-to-animation systems in its inputs, outputs, and actions on existing figure elements. (Reviewer bjif Q2)
  • Grounding Input-Token Accounting: We clarified that the reported token usage already includes all inputs across all grounding calls and rounds: SVG source and text prompts for the baseline, and text prompts and images for FigAct. (Reviewer bjif Q6)
  • Benchmark and Metric Design: We clarified the contribution of the human-verified VQP benchmark and how our evaluation combines prior work with task-specific metrics to assess feasibility, stage-wise performance, and end-to-end presentation quality. (Reviewer bjif Q5)
  • Human-Study Protocol: We provided detail protoco including shared narration, viewing window, question-type distributions. (Reviewer usv5 Q1)
  • FigAct-8B’s Quality-Cost Trade-off: We clarified the scalability of synthetic supervision without proprietary-model distillation, together with the gains training on an open-weight 8B model. (Reviewer usv5 Q6)

Thank you once again to all the reviewers for your constructive feedback, which has helped us enhance our work to make it more solid and comprehensive. We believe these extensive experiments and revisions have strengthened our paper and addressed the concerns raised.

Best regards,

All Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.