acceptodds
Under review as a conference paper at ICLR 2027

MangaBench: Can Multimodal Models Reason Across Thousands of Manga Pages?

Abstract

As Multimodal Large Language Models (MLLMs) scale to context windows of up to one million tokens, many long-context benchmarks increasingly test whether models can effectively use their context. However, evaluating model understanding across long, connected visual narratives remains an open challenge, and recent benchmarks fail to evaluate model reasoning over content beyond the context window limit. To address this gap, we introduce MangaBench, a benchmark for evaluating reasoning and understanding over long visual stories. Its tasks require models to aggregate evidence and to track entities, relationships, and states across volumes. Manga is a particularly challenging medium because its information-dense panels depict selected narrative moments, and later scenes may depend on previous visual details. MangaBench pairs manually constructed questions with an evaluation of multimodal models using native agents and an RLM-inspired reader. We compare task performance across model–harness configurations. Our evaluation shows that even frontier models using these harnesses struggle with counting and listing, and with state tracking across volumes. These results highlight the need for models and agent harnesses that can more effectively gather visual evidence to reason over extended narratives.

Primary area:
datasets and benchmarks
Keywords:
benchmarks, multimodal, long-context, MLLMs, visual reasoning, narrative understanding, agents, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oMVrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents MangaBench, a benchmark for evaluating manga understanding of LLMs/agents. It consists of 954 manually authored questions across eight series, covering both English and Japanese. It is categorized into single-hop, cross-scene aggregation, and multi-hop, and the frontier agents score far below the human baseline in all categories.

Critical strengths
  • A first benchmark proposal that considers a multi-volume setting that requires substantially larger context than similar benchmarks.
  • Large-scale, manually crafted question-answer pairs with up-to-date evaluation, including Opus 5.5 and GPT-6 Astra, still show below-human performance, making the benchmark practically relevant.
Critical weaknesses
  • Closed-book ablation (Table 4) shows that there is only a 4% gain from actually providing manga as input. This either indicates (i) the score comes from memorization/training data contamination, or (ii) many questions are so trivial that they can be solved by common knowledge. Either way, it means the score does not necessarily reflect manga processing ability.
Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

GPT-6-Astra: Closed-book versus open-book results across four series

Thank you for highlighting this concern. We agree that some questions can be answered without accessing the manga. However, the 4% gain does not sufficiently show whether the closed-book or open-book condition solve the same questions correctly. We have extended the closed-book evaluation to four series using GPT-6-Astra and compared outcomes on the same 897 questions across 4 series.

+----------------+-----------+-------------+------------+------------+
| Series         | Questions | Closed-book | With manga | Difference |
+----------------+-----------+-------------+------------+------------+
| Blue Period    |       102 |       58.8% |      66.7% |       +7.8 |
| Chainsaw Man   |        89 |       51.7% |      68.5% |      +16.9 |
| Golden Kamuy   |       513 |       47.2% |      60.8% |      +13.6 |
| Jujutsu Kaisen |       193 |       51.8% |      63.7% |      +11.9 |
+----------------+-----------+-------------+------------+------------+
| Overall        |       897 |       49.9% |      62.9% |      +12.9 |
+----------------+-----------+-------------+------------+------------+

Here, we show that 147 questions are answered correctly only with manga access, while only 31 are answered correctly only with closed-book. Among the 449 questions that closed-book answered incorrectly, open-book evaluation answered 32.7% of these correctly. The per-series results show improvements across all four evaluated series as well. While it is true that a good portion of the benchmark can be answered without manga access, the questions missed in closed-book evaluation let us (and future researchers) test whether reading systems can find answers that the model fails to provide without the pages. We also provide a breakdown by task type using GPT-6-Astra.

+-----------------------------------+-----------+-------------+------------+------------+
| Task type                         | Questions | Closed-book | With manga | Difference |
+-----------------------------------+-----------+-------------+------------+------------+
| Needle retrieval                  |       110 |       77.3% |      90.0% |      +12.7 |
| Scene-level presence and absence  |       170 |       55.3% |      75.9% |      +20.6 |
| Counting and listing              |       127 |       26.8% |      33.1% |       +6.3 |
| Selection and grouping            |        83 |       38.6% |      45.8% |       +7.2 |
| Comparison and ranking            |        38 |       60.5% |      57.9% |       -2.6 |
| Corpus-level absence verification |        32 |       62.5% |      78.1% |      +15.6 |
| State tracking                    |        89 |       39.3% |      56.2% |      +16.9 |
| Ordering and chronology           |       200 |       53.0% |      63.0% |      +10.0 |
| Cross-scene linking               |        48 |       39.6% |      68.8% |      +29.2 |
+-----------------------------------+-----------+-------------+------------+------------+
| Overall                           |       897 |       49.9% |      62.9% |      +12.9 |
+-----------------------------------+-----------+-------------+------------+------------+

Thank you for the rebuttal. I appreciate the additional breakdown tables to understand the scores better. However, I don't think this addresses the weakness itself. It is unclear whether the benchmark can truly evaluate the models' manga understanding ability if half of the problems (of the whole benchmark) can be solved without manga. Overall, I still lean towards weak rejection.

Thank you for the follow-up. The series used in MangaBench are widely discussed online, so major plot points and the overarching thread of the story are likely encoded in model weights. This is why closed-book accuracy is the highest on questions addressing facts on the story level. On finer-grained tasks, however, page access proves necessary: models perform significantly better on cross-scene linking (+29.2), scene-level presence and absence (+20.6), and state tracking (+16.9) with open book. We kept the knowledge-answerable questions on purpose: dropping questions purely because models perform well on them would bias the benchmark and provide an incomplete representation of current capabilities.

The closed-book results also demonstrate that reading the volumes is necessary and still difficult; within the 449 questions GPT-6-Astra fails without the manga, only 32.7% are answered correctly even with open-book access. This shows the benchmark’s ability to measure model reading capabilities and leaves substantial headroom.

Reviewer_X7EiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MangaBench, a benchmark of 954 manually authored questions over eight manga series, designed to evaluate multimodal models’ ability to retrieve and combine evidence across long visual narratives. The authors evaluate several frontier models with native agent harnesses and an RLM-inspired reader, finding persistent difficulties with counting, listing, and state tracking.

Critical strengths
  1. The benchmark targets a meaningful gap: reasoning over visual narratives that span multiple volumes, rather than isolated panels or short excerpts.
  2. Questions include supporting page references and evidence explanations, with independent review and additional quality control.
  3. The evaluation includes useful task breakdowns, input ablations, and a failure analysis that distinguishes evidence-gathering errors from answer-construction errors.
Critical weaknesses
  1. The benchmark does not yet establish that correct answers depend on visual evidence. On the Jujutsu Kaisen subset, the closed-book condition reaches 54.4% accuracy, close to the 58.5% achieved with full input. This raises a substantial concern that prior knowledge or question cues can solve many items. The paper acknowledges this limitation but does not report item-level shortcut analysis or show how much performance remains on questions that are demonstrably unsolvable without the pages. As a result, the benchmark’s central claim about evaluating visually grounded long-range reasoning is not fully supported.

  2. The question distribution is highly imbalanced across series, while results are mostly aggregated. Golden Kamuy contributes 513 of the 954 questions, whereas two series contribute only 5 and 15. This makes the overall scores heavily dependent on one title and leaves performance on several titles too sparsely measured to support broad conclusions about manga understanding. Per-series results and a clearer sampling rationale are needed to establish that the findings generalize across stories.

  3. The harness comparisons are not controlled enough to support conclusions about reading architectures. Native agents receive all questions for a series in one shared session, while RLM questions are batched in groups of up to 20; the harnesses also differ in prompts, tools, and available context. Neither has a total token or cost budget. The paper appropriately notes that observed differences cannot be attributed to architecture alone, but these protocol differences still make the headline model–harness comparisons difficult to interpret as evidence for the relative effectiveness of the evaluated approaches.

Questions for authors

See the questions above

3 comments
AuthorsOpenReview ↗

Closed-book performance analysis

  1. The benchmark does not yet establish that correct answers depend on visual evidence. On the Jujutsu Kaisen subset, the closed-book condition reaches 54.4% accuracy, close to the 58.5% achieved with full input. This raises a substantial concern that prior knowledge or question cues can solve many items. The paper acknowledges this limitation but does not report item-level shortcut analysis or show how much performance remains on questions that are demonstrably unsolvable without the pages. As a result, the benchmark’s central claim about evaluating visually grounded long-range reasoning is not fully supported.

Thank you for identifying this limitation. We fully agree that a correct answer does not mean that the model used the manga pages. To understand this, we have extended closed-book evaluation to four series, using GPT-6-Astra and compared individual outcomes on the same 897 questions.

By series:

+----------------+-----------+-------------+------------+------------+
| Series         | Questions | Closed-book | With manga | Difference |
+----------------+-----------+-------------+------------+------------+
| Blue Period    |       102 |       58.8% |      66.7% |       +7.8 |
| Chainsaw Man   |        89 |       51.7% |      68.5% |      +16.9 |
| Golden Kamuy   |       513 |       47.2% |      60.8% |      +13.6 |
| Jujutsu Kaisen |       193 |       51.8% |      63.7% |      +11.9 |
+----------------+-----------+-------------+------------+------------+
| Overall        |       897 |       49.9% |      62.9% |      +12.9 |
+----------------+-----------+-------------+------------+------------+

By task type:

+-----------------------------------+-----------+-------------+------------+------------+
| Task type                         | Questions | Closed-book | With manga | Difference |
+-----------------------------------+-----------+-------------+------------+------------+
| Needle retrieval                  |       110 |       77.3% |      90.0% |      +12.7 |
| Scene-level presence and absence  |       170 |       55.3% |      75.9% |      +20.6 |
| Counting and listing              |       127 |       26.8% |      33.1% |       +6.3 |
| Selection and grouping            |        83 |       38.6% |      45.8% |       +7.2 |
| Comparison and ranking            |        38 |       60.5% |      57.9% |       -2.6 |
| Corpus-level absence verification |        32 |       62.5% |      78.1% |      +15.6 |
| State tracking                    |        89 |       39.3% |      56.2% |      +16.9 |
| Ordering and chronology           |       200 |       53.0% |      63.0% |      +10.0 |
| Cross-scene linking               |        48 |       39.6% |      68.8% |      +29.2 |
+-----------------------------------+-----------+-------------+------------+------------+
| Overall                           |       897 |       49.9% |      62.9% |      +12.9 |
+-----------------------------------+-----------+-------------+------------+------------+

We show that closed-book performance is at 49.9% overall, compared to 62.9% for Codex with full corpus access. There are 147 questions answered correctly only with manga access, while 31 answered correctly only in the closed-book scenario. Among the 449 questions that closed-book answered incorrectly, 32.7% were answered correctly by corpus access.

This provides the requested item-level comparison and measure performance on closed-book evaluation. However, we cannot assert that this subset (i.e., questions failed by closed-book) is demonstratably unsolvable. Our comparison here also does has yet to distinguish reliance on dialogue vs. visuals. We will report these distinctions explicitly and narrow overall claims regarding evidence attributed to visual evidence.

AuthorsOpenReview ↗

Series-imbalance and harness comparison

  1. The question distribution is highly imbalanced across series, while results are mostly aggregated. Golden Kamuy contributes 513 of the 954 questions, whereas two series contribute only 5 and 15. This makes the overall scores heavily dependent on one title and leaves performance on several titles too sparsely measured to support broad conclusions about manga understanding. Per-series results and a clearer sampling rationale are needed to establish that the findings generalize across stories.

We agree with this point here and that the aggregate score is strongly influenced by Golden Kamuy, and that the current distribution is not a balanced sample across all 8 series. We also agree that the two series pointed out are not enough to support any reliable conclusion regarding those series. We provide the following additional matched evaluation, which gives per-series results:


+----------------+-----------+-------------+------------+
| Series         | Questions | Closed-book | With manga |
+----------------+-----------+-------------+------------+
| Blue Period    |       102 |       58.8% |      66.7% |
| Chainsaw Man   |        89 |       51.7% |      68.5% |
| Golden Kamuy   |       513 |       47.2% |      60.8% |
| Jujutsu Kaisen |       193 |       51.8% |      63.7% |
+----------------+-----------+-------------+------------+

Equally weighting the series gives 52.4% closed-book accuracy and 64.9% with manga access. Excluding Golden Kamuy entirely gives 53.6% and 65.6%, respectively. Therefore the improvement in this comparison is not driven solely by Golden Kamuy.

On the point of distribution imbalance: the unequal question counts reflect differences in annotator expertise across series. All our annotators knew Golden Kamuy well, while expertise in other titles varied. We will make this sampling rationale explicit and report per-series results and equally weighted series averages alongside question-weighted accuracy. We will also limit conclusions about series with very few questions.

  1. The harness comparisons are not controlled enough to support conclusions about reading architectures. Native agents receive all questions for a series in one shared session, while RLM questions are batched in groups of up to 20; the harnesses also differ in prompts, tools, and available context. Neither has a total token or cost budget. The paper appropriately notes that observed differences cannot be attributed to architecture alone, but these protocol differences still make the headline model–harness comparisons difficult to interpret as evidence for the relative effectiveness of the evaluated approaches.

We agree that the reported comparisons cannot isolate effects of reading architecture. Our main contribution is the benchmark and its evaluation of complete reading systems. The model–harness results show performance under the documented configurations; they do not establish whether one architecture is inherently better than another. We will make this interpretation consistent throughout the paper and remove any wording that attributes differences specifically to architecture.

Our limited evaluation budget prevented additional runs under matched token or cost budgets. We will acknowledge this limitation and therefore report the performance of complete systems under their documented configurations, without claiming that one harness is more efficient under a common budget. We will make this limitation explicit and report resource usage where available.

Thanks for your response. I have read through your additional results. After consideration, I decided to maintain my score.

Reviewer_AaZ5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MangaBench, a benchmark for evaluating multimodal models and agents over long manga narratives spanning thousands of pages. It contains 954 manually authored questions across eight manga series and evaluates single-hop retrieval, cross-scene aggregation, and multi-hop reasoning using frontier MLLMs with several agent harnesses. The benchmark is ambitious in scale however, I have significant concerns about novelty positioning, dataset balance, shortcut solvability, statistical support, and the reproducibility of the evaluated systems.

Critical strengths

The benchmark addresses a meaningful and difficult setting. The long-context aspect is genuine rather than artificial: cross-scene aggregation has a median evidence gap of 944 pages and multi-hop reasoning 542 pages; 82.5% and 74.1% of those questions respectively span multiple volumes. This is a clear step beyond most existing manga benchmarks that operate on panels, spreads, or short sequences.

The annotation pipeline is also a strong point. Questions are manually authored, a second annotator independently answers each question before seeing the reference, exhaustive-list and absence questions receive additional checking, unresolved cases can be reviewed by a third annotator, and ambiguous or duplicate questions are removed. The final set contains 954 of 1,026 candidate questions, with roughly 15% revised prior to acceptance. The authors additionally report substantial manual validation effort.

The paper is unusually transparent about several confounds. It explicitly states that agent-harness comparisons differ in system prompts, tools, batching, and context, and therefore cannot isolate the effect of the reading architecture alone. It also acknowledges that high closed-book performance means correct answers do not necessarily imply visual grounding. This transparency is valuable.

The task taxonomy is practically useful. Distinguishing needle retrieval, counting/listing, state tracking, chronology, cross-scene linking, and related capabilities gives more diagnostic information than reporting only overall accuracy.

Critical weaknesses
  1. The novelty should be positioned more narrowly in terms of novelty. Re:Verse already introduced long-form manga narrative understanding. While MangaBench is not redundant the novelty is primarily scaling to full-page, multi-volume, beyond-context agentic reasoning. Claims suggesting that long-range manga understanding or temporal reasoning are themselves entirely unexplored are overstated. Additionally, the recently published RE:COGNIZE (this does not factor into my evaluation given its proximity to the deadline) formalizes sequential manga reading for character identity maintenance, including open-set identity emergence and re-identification.

  2. The dataset is extremely imbalanced across series. Golden Kamuy contributes 513 of 954 questions, whereas Land of the Lustrous contributes only 5; several other titles contribute fewer than 20. Consequently, overall accuracy is dominated by one series and should not be interpreted as balanced performance across eight narratives. The paper does not provide a convincing justification for this distribution or a macro-averaged per-series score.

  3. Shortcut solvability is a serious concern. On Jujutsu Kaisen, GPT-5.6-Sol reaches 54.4% closed-book accuracy compared with 58.5% using full input; single-hop accuracy is 64.0% closed-book versus 65.1% with manga access. Since these are popular published works, parametric knowledge is an obvious confound. For a benchmark intended to measure visual grounding and long-context retrieval, closed-book evaluation should be much broader across titles and models.

  4. The model–harness comparisons are not controlled. Native agents receive all selected questions for a series within one shared persistent session, whereas RLM configurations use batches of up to 20 questions and do not retain findings between sessions. The resulting comparisons conflate model capability, memory persistence, batching, available context, tool interfaces, and orchestration strategy.

  5. There is also no common resource budget. The paper explicitly states that the harnesses have no total token or cost limit. On Blue Period, for example, the RLM setup uses roughly 8.38M tokens, versus approximately 182K–310K for compressed full-context alternatives. Accuracy without cost- or token-normalized reporting is insufficient for comparing agentic reading systems. The statistical support is weak. The paper reports single accuracy values without repeated-run variance, bootstrap confidence intervals, or significance testing for the main rankings. Given stochastic agent behavior and strong clustering of questions within only eight series, the paper should quantify both run-level and dataset-level uncertainty.

  6. Task difficulty is also confounded with answer format. List questions receive zero credit if even one item is missing or one extra item is present, with no partial credit. Counting/listing and selection/grouping tasks are therefore evaluated under inherently harsher output criteria than many short-answer retrieval questions.

  7. The human baseline is under-characterized. Humans achieve only 84.0% overall, despite full corpus access and no time limit. The paper does not adequately analyze the reasoning behind the human failure. Maybe due to ambiguity, search burden, annotation errors, or strict all-or-nothing scoring.

  8. The LLM-judge validation description is also unclear. The paper states that the authors manually checked “every model answer” and agreed with the judge on 935/954 decisions, even though multiple model–harness configurations are evaluated over the same 954 questions. It is therefore unclear which 954 outputs were reviewed and how representative this validation is.

  9. A fatal practical flaw in this benchmark is its reproducibility and copyright friction. The authors explicitly state they 'do not distribute the copyrighted pages' and rely on researchers to independently purchase, scan, and align 30,147 pages across 162 volumes just to run the benchmark. Proposing a dataset that cannot be legally distributed or easily reproduced by peers is an astronomical barrier to entry and unacceptable for an ICLR benchmark track submission. A promised future 'hosted evaluation' leaderboard does not substitute for a usable, open artifact at the time of review.

  10. Furthermore, the failure analysis (Section 5.5) is non-systematic. Manually auditing only 139 hand-picked failures specifically from the Codex configuration (and 23 from RLM) fails to represent the broader spectrum of models, tools, and harnesses tested. Drawing generalized conclusions about model failures from a narrow, unrepresentative sampling is methodologically unsound.

Questions for authors

Please refer to weaknesses in the previous section.

4 comments
AuthorsOpenReview ↗

Novelty, dataset imbalance

Thank you for your review. We will address each weakness one by one.

  1. The novelty should be positioned more narrowly in terms of novelty. Re:Verse already introduced long-form manga narrative understanding. While MangaBench is not redundant the novelty is primarily scaling to full-page, multi-volume, beyond-context agentic reasoning. Claims suggesting that long-range manga understanding or temporal reasoning are themselves entirely unexplored are overstated. Additionally, the recently published RE:COGNIZE (this does not factor into my evaluation given its proximity to the deadline) formalizes sequential manga reading for character identity maintenance, including open-set identity emergence and re-identification.

We agree that the contribution should be positioned specifically around full-page, multi-volume evaluation. Our introduction already distinguishes MangaBench from Re:Verse, and we will make this distinction consistent throughout the paper. We do not intend to claim that manga narrative understanding or temporal reasoning was previously unexplored. Our contribution is evaluating questions over extended narratives, with evidence and dependencies distributed across multiple volumes. We also appreciate the reference to Re:Cognize and will discuss its sequential character-identity evaluation alongside our broader question-answering setting.

  1. The dataset is extremely imbalanced across series. Golden Kamuy contributes 513 of 954 questions, whereas Land of the Lustrous contributes only 5; several other titles contribute fewer than 20. Consequently, overall accuracy is dominated by one series and should not be interpreted as balanced performance across eight narratives. The paper does not provide a convincing justification for this distribution or a macro-averaged per-series score.

We agree with this point here and that the aggregate score is strongly influenced by Golden Kamuy, and that the current distribution is not a balanced sample across all 8 series. We also agree that the two series pointed out are not enough to support any reliable conclusion regarding those series. We provide the following additional matched evaluation, which gives per-series results:


+----------------+-----------+-------------+------------+
| Series         | Questions | Closed-book | With manga |
+----------------+-----------+-------------+------------+
| Blue Period    |       102 |       58.8% |      66.7% |
| Chainsaw Man   |        89 |       51.7% |      68.5% |
| Golden Kamuy   |       513 |       47.2% |      60.8% |
| Jujutsu Kaisen |       193 |       51.8% |      63.7% |
+----------------+-----------+-------------+------------+

Equally weighting the series gives 52.4% closed-book accuracy and 64.9% with manga access. Excluding Golden Kamuy entirely gives 53.6% and 65.6%, respectively. Therefore the improvement in this comparison is not driven solely by Golden Kamuy.

On the point of distribution imbalance: the unequal question counts reflect differences in annotator expertise across series. All our annotators knew Golden Kamuy well, while expertise in other titles varied. We will make this sampling rationale explicit and report per-series results and equally weighted series averages alongside question-weighted accuracy. We will also limit conclusions about series with very few questions.

AuthorsOpenReview ↗

Shortcut solvability, harness comparison

  1. Shortcut solvability is a serious concern. On Jujutsu Kaisen, GPT-5.6-Sol reaches 54.4% closed-book accuracy compared with 58.5% using full input; single-hop accuracy is 64.0% closed-book versus 65.1% with manga access. Since these are popular published works, parametric knowledge is an obvious confound. For a benchmark intended to measure visual grounding and long-context retrieval, closed-book evaluation should be much broader across titles and models.

We fully agree with this point here regarding shortcut solvability, and that a correct answer does not mean that the model used the manga pages. To understand this, we have extended closed-book evaluation to four series, using GPT-6-Astra and compared individual outcomes on the same 897 questions.

By series:

+----------------+-----------+-------------+------------+------------+
| Series         | Questions | Closed-book | With manga | Difference |
+----------------+-----------+-------------+------------+------------+
| Blue Period    |       102 |       58.8% |      66.7% |       +7.8 |
| Chainsaw Man   |        89 |       51.7% |      68.5% |      +16.9 |
| Golden Kamuy   |       513 |       47.2% |      60.8% |      +13.6 |
| Jujutsu Kaisen |       193 |       51.8% |      63.7% |      +11.9 |
+----------------+-----------+-------------+------------+------------+
| Overall        |       897 |       49.9% |      62.9% |      +12.9 |
+----------------+-----------+-------------+------------+------------+

By task type:

+-----------------------------------+-----------+-------------+------------+------------+
| Task type                         | Questions | Closed-book | With manga | Difference |
+-----------------------------------+-----------+-------------+------------+------------+
| Needle retrieval                  |       110 |       77.3% |      90.0% |      +12.7 |
| Scene-level presence and absence  |       170 |       55.3% |      75.9% |      +20.6 |
| Counting and listing              |       127 |       26.8% |      33.1% |       +6.3 |
| Selection and grouping            |        83 |       38.6% |      45.8% |       +7.2 |
| Comparison and ranking            |        38 |       60.5% |      57.9% |       -2.6 |
| Corpus-level absence verification |        32 |       62.5% |      78.1% |      +15.6 |
| State tracking                    |        89 |       39.3% |      56.2% |      +16.9 |
| Ordering and chronology           |       200 |       53.0% |      63.0% |      +10.0 |
| Cross-scene linking               |        48 |       39.6% |      68.8% |      +29.2 |
+-----------------------------------+-----------+-------------+------------+------------+
| Overall                           |       897 |       49.9% |      62.9% |      +12.9 |
+-----------------------------------+-----------+-------------+------------+------------+

We show that closed-book performance is at 49.9% overall, compared to 62.9% for Codex with full corpus access. There are 147 questions answered correctly only with manga access, while 31 answered correctly only in the closed-book scenario. Among the 449 questions that closed-book answered incorrectly, 32.7% were answered correctly by corpus access.

This provides the requested item-level comparison and measure performance on closed-book evaluation. However, we cannot assert that this subset (i.e., questions failed by closed-book) is demonstratably unsolvable. Our comparison here also does has yet to distinguish reliance on dialogue vs. visuals. We will report these distinctions explicitly and narrow overall claims regarding evidence attributed to visual evidence.

  1. The model–harness comparisons are not controlled. Native agents receive all selected questions for a series within one shared persistent session, whereas RLM configurations use batches of up to 20 questions and do not retain findings between sessions. The resulting comparisons conflate model capability, memory persistence, batching, available context, tool interfaces, and orchestration strategy.

We agree that the reported comparisons cannot isolate effects of reading architecture. Our main contribution is the benchmark and its evaluation of complete reading systems. The model–harness results show performance under the documented configurations; they do not establish whether one architecture is inherently better than another. We will make this interpretation consistent throughout the paper and remove any wording that attributes differences specifically to architecture.

Our limited evaluation budget prevented additional runs under matched token or cost budgets. We will acknowledge this limitation and therefore report the performance of complete systems under their documented configurations, without claiming that one harness is more efficient under a common budget. We will make this limitation explicit and report resource usage where available.

AuthorsOpenReview ↗

Resource budget, task difficulty, human baseline, LLM-judge

  1. There is also no common resource budget. The paper explicitly states that the harnesses have no total token or cost limit. On Blue Period, for example, the RLM setup uses roughly 8.38M tokens, versus approximately 182K–310K for compressed full-context alternatives. Accuracy without cost- or token-normalized reporting is insufficient for comparing agentic reading systems. The statistical support is weak. The paper reports single accuracy values without repeated-run variance, bootstrap confidence intervals, or significance testing for the main rankings. Given stochastic agent behavior and strong clustering of questions within only eight series, the paper should quantify both run-level and dataset-level uncertainty.

We agree that single-run scores do not quantify stochastic variation. Our limited evaluation budget prevented repeated evaluations across all configurations, and we will make clear that the results do not establish statistically reliable rankings between closely scoring systems.

Dataset-level uncertainty and run-level variability are separate issues. We will report uncertainty from the available scored outcomes with the sampling assumptions stated explicitly, including the limitations imposed by clustering within a small number of series. Such analyses cannot replace repeated runs or establish generalization to unseen narratives.

The Blue Period full-context experiment examines what happens when the entire manga is compressed sufficiently to fit into one model context. It compares this setting with systems that access pages through reading tools. The difference in token usage follows from these different input representations and execution procedures.

The purpose is to examine the performance of compressed full-context input, rather than establish an efficiency advantage under equal resource budgets. A matched-budget comparison would address a separate question. We will clarify this purpose and report accuracy alongside the available resource measurements.

  1. Task difficulty is also confounded with answer format. List questions receive zero credit if even one item is missing or one extra item is present, with no partial credit. Counting/listing and selection/grouping tasks are therefore evaluated under inherently harsher output criteria than many short-answer retrieval questions.

Exact-set scoring reflects the requirement of exhaustive-list questions: omitting a requested item or adding an unsupported item means the requested list is incomplete or incorrect. We therefore retain binary accuracy as the measure of complete task success.

We agree, however, that longer answers create more opportunities for error. Differences between task categories cannot be attributed entirely to reasoning difficulty. We will make this limitation explicit and report performance by answer format alongside the task breakdowns.

  1. The human baseline is under-characterized. Humans achieve only 84.0% overall, despite full corpus access and no time limit. The paper does not adequately analyze the reasoning behind the human failure. Maybe due to ambiguity, search burden, annotation errors, or strict all-or-nothing scoring.

We agree that the human baseline requires more characterization. The reported 84.0% is the observed performance of participating annotators under the stated protocol, rather than a universal human ceiling. Annotators answered questions authored by others, with corpus access and no time limit. We will clarify participant allocation and the scoring procedure. Based on our analysis, most of the errors come from aggregation across long contexts, hence the 84.0% human performance.

  1. The LLM-judge validation description is also unclear. The paper states that the authors manually checked “every model answer” and agreed with the judge on 935/954 decisions, even though multiple model–harness configurations are evaluated over the same 954 questions. It is therefore unclear which 954 outputs were reviewed and how representative this validation is.

We agree that “every model answer” is insufficiently precise when multiple model–harness configurations are evaluated. We will identify the exact outputs included in the 954-answer audit, explain their selection, and restrict the reported 935/954 agreement to that reviewed collection.

AuthorsOpenReview ↗

Reproducibility, failure analysis

  1. A fatal practical flaw in this benchmark is its reproducibility and copyright friction. The authors explicitly state they 'do not distribute the copyrighted pages' and rely on researchers to independently purchase, scan, and align 30,147 pages across 162 volumes just to run the benchmark. Proposing a dataset that cannot be legally distributed or easily reproduced by peers is an astronomical barrier to entry and unacceptable for an ICLR benchmark track submission. A promised future 'hosted evaluation' leaderboard does not substitute for a usable, open artifact at the time of review.

MangaBench contributes manually authored questions, reference answers, supporting evidence annotations, and an evaluation protocol. These annotations require substantial expert work over extended narratives and remain useful research contributions when the underlying commercial works cannot be redistributed with them.

We nevertheless agree that acquiring and aligning the source material creates a substantial reproduction burden. Individual series can be evaluated separately, so initial use does not require preparing all 162 volumes; reproducing the complete aggregate result still requires the full evaluated corpus. We will distinguish these requirements clearly.

Research resources can also provide access under restrictions rather than unrestricted redistribution. Manga109, for example, provides author-authorized research access while prohibiting recipients from redistributing the dataset. However, its established acquisition route is an advantage that our release does not currently match. Manga109 terms.

We will describe precisely which artifacts are available and what remains necessary to reproduce the main results. The synthetic example demonstrates the workflow, while hosted evaluation remains a planned accessibility measure. Neither will be presented as a substitute for an already available reproduction of the full benchmark.

  1. Furthermore, the failure analysis (Section 5.5) is non-systematic. Manually auditing only 139 hand-picked failures specifically from the Codex configuration (and 23 from RLM) fails to represent the broader spectrum of models, tools, and harnesses tested. Drawing generalized conclusions about model failures from a narrow, unrepresentative sampling is methodologically unsound.

We agree that this analysis cannot estimate failure prevalence across all evaluated systems. The paired analysis covers 139 questions selected from Codex failures and their corresponding RLM responses. The plotted distributions contain 52 resolved Codex failures and 23 resolved RLM failures; 23 is the number of resolved RLM failures, not the total number of RLM responses inspected.

We will make the selection procedure and denominators explicit. The examples demonstrate particular failure mechanisms in the inspected records. Their frequencies are conditional on this selected, resolvable subset and should not be generalized to the complete benchmark or used to rank harnesses.

Comments on the paper

1 comment

Dear Reviewers,

The authors have submitted their rebuttal and tried to address the concerns raised in your reviews. If you haven't already, we would appreciate it if you could take the time to read their responses and engage in the discussion, particularly to assess whether your concerns have been adequately addressed.

Thank you for your time and valuable contributions to the review process!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.