- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This Paper formulate the Slides2MindMap task, proposes S2M-Bench, a benchmark for evaluating automatic mind-map generation from lecture slides and AutoMindMap, an agentic framework for constructing mind-map from slides inspired by the Structure Building Framework. The authors evaluate the generated mind-map using ground-truth based metrics, structure conformity and VLM-Judge.
Critical strengths
- The proposed benchmark is large and multi-dimensional, which could serve to evaluate automatic methods in this subdomain.
- The pipeline of AutoMindMap is also well designed, with multiple stages for skeleton construction, iterative knowledge, and refinement.
Critical weaknesses
-
The evaluation mainly focuses on the generated map itself or its similarity to the human reference but does not directly evaluate whether the generated map is faithfully grounded in the original slides. There might therefore be a hallucination problem. Based on the prompt of VLM-Judge, only the generated map, excluding the original slides, is given to the VLM for evaluation. Therefore, the Content Accuracy score seems to mainly evaluate map itself, rather than the faithfulness between the generated map and the original slides.
-
This task is subjective, and many possible taxonomies may exist. As the authors show in Table 18, the EC-F1 of human re-annotation is only 54.73. In addition, only limited details about the ground-truth annotation process are provided in the paper. Therefore, I am skeptical about the reliability of the GT-based metrics for this open-ended task.
Questions for authors
-
More details about the annotation process need to be clarified. For example, how many annotators were involved for each course? Were the annotations constructed independently? How were the annotation instructions and rubrics provided, and how were disagreements resolved?
-
Is there any evaluation that directly measures whether the generated concepts and relations are supported by the original slides?
I thank the authors for their replies, which address many of my concerns. Specifically, I appreciate the more detailed discussion of related work, and the responses highlight the unique advantages of the proposed AutoMindMap. In addition, I appreciate the comparison with the latest model; the limitations of SOTA models on this benchmark add value to this work. I remain somewhat reserved about the novelty of the proposed modules. Although the authors highlight five modules in the rebuttal, some appear incremental to me. For example, step-wise association and repair involves similarity-based retrieval and prompting an LLM to check structural validity. However, the contributions in scale and evaluation are clear to me. I will increase my rating.