- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents MangaBench, a benchmark for evaluating manga understanding of LLMs/agents. It consists of 954 manually authored questions across eight series, covering both English and Japanese. It is categorized into single-hop, cross-scene aggregation, and multi-hop, and the frontier agents score far below the human baseline in all categories.
Critical strengths
- A first benchmark proposal that considers a multi-volume setting that requires substantially larger context than similar benchmarks.
- Large-scale, manually crafted question-answer pairs with up-to-date evaluation, including Opus 5.5 and GPT-6 Astra, still show below-human performance, making the benchmark practically relevant.
Critical weaknesses
- Closed-book ablation (Table 4) shows that there is only a 4% gain from actually providing manga as input. This either indicates (i) the score comes from memorization/training data contamination, or (ii) many questions are so trivial that they can be solved by common knowledge. Either way, it means the score does not necessarily reflect manga processing ability.
Questions for authors
N/A
3 comments
Thank you for the rebuttal. I appreciate the additional breakdown tables to understand the scores better. However, I don't think this addresses the weakness itself. It is unclear whether the benchmark can truly evaluate the models' manga understanding ability if half of the problems (of the whole benchmark) can be solved without manga. Overall, I still lean towards weak rejection.
Thank you for the follow-up. The series used in MangaBench are widely discussed online, so major plot points and the overarching thread of the story are likely encoded in model weights. This is why closed-book accuracy is the highest on questions addressing facts on the story level. On finer-grained tasks, however, page access proves necessary: models perform significantly better on cross-scene linking (+29.2), scene-level presence and absence (+20.6), and state tracking (+16.9) with open book. We kept the knowledge-answerable questions on purpose: dropping questions purely because models perform well on them would bias the benchmark and provide an incomplete representation of current capabilities.
The closed-book results also demonstrate that reading the volumes is necessary and still difficult; within the 449 questions GPT-6-Astra fails without the manga, only 32.7% are answered correctly even with open-book access. This shows the benchmark’s ability to measure model reading capabilities and leaves substantial headroom.
GPT-6-Astra: Closed-book versus open-book results across four series
Thank you for highlighting this concern. We agree that some questions can be answered without accessing the manga. However, the 4% gain does not sufficiently show whether the closed-book or open-book condition solve the same questions correctly. We have extended the closed-book evaluation to four series using GPT-6-Astra and compared outcomes on the same 897 questions across 4 series.
Here, we show that 147 questions are answered correctly only with manga access, while only 31 are answered correctly only with closed-book. Among the 449 questions that closed-book answered incorrectly, open-book evaluation answered 32.7% of these correctly. The per-series results show improvements across all four evaluated series as well. While it is true that a good portion of the benchmark can be answered without manga access, the questions missed in closed-book evaluation let us (and future researchers) test whether reading systems can find answers that the model fails to provide without the pages. We also provide a breakdown by task type using GPT-6-Astra.