- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ITJoint, a benchmark for query-driven image-text joint extraction from long documents. The task requires extracting relevant text with all corresponding image regions. The benchmark contains 316 queries, 910 answer instances and 2,455 document pages across 41 documents. The documents mainly covers medicine and archaeology, together with biology, art, and agriculture. The authors evaluate four MLLMs and propose an agentic baseline.
Critical strengths
Query-driven extraction is a practical task, and the design of task taxonomy captures meaningful challenges beyond conventional textual QA benchmarks. The authors conduct substantial baseline analysis, providing insightful evidence about the fine-grained localization bottlenecks and errors.
Critical weaknesses
- The main Joint metric combines semantic association, image-set completeness and bounding-box accuracy, and Soft Joint also retains the same IoU evalutaions. Although the IoU analysis partially address this issue, they do not independently measure whether models identify the correct image-text correspondence. Consequently, low scores and model differences remain difficult to attribute to association failures versus coordinate-generation errors.
- The benchmarks contains only 41 documents, with medicine and archaeology accounting for over 70%, providing limited support for broader generalization.
- The annotation workflow is carefully designed, additional clarification would be helpful: did the two independent checkers separately record corrections and missing instances before their findings were consolidated? Reporting the overlap and differences in their findings, particularly for set completeness and text-only instances, would provide additional insight into the verification process.
Questions for authors
Please see Critical Weaknesses above.
6 comments
Official Comment by Authors – Part 2
Continued from W1.
Case-by-case inspection statistics.
We further inspected these 108 failures individually. In 23 cases, the predicted box covers at least 90% of the GT, with an average coverage of 98.62%, but the median predicted-box area is 2.08 times the GT area. Another 51 predicted boxes partially overlap with the GT. Among the 41 cases where other candidate images are present and the distance and overlap criteria agree, 37 (90.2%) have prediction boxes closer to the GT and with greater overlap with it. Taken together, among GPT's 108 localization failures, 60 cases (55.6%) have prediction boxes that clearly cover or are closer to the correct image. Their error patterns are more consistent with inaccurate bounding-box coordinates than with selecting a different, incorrect image (paper lines 1389–1402).
Official Comment by Authors – Part 3
W2: The dataset has limited document-domain coverage.
Response:
We sincerely thank the reviewer for carefully considering the dataset's domain coverage and its implications for interpreting the results. This gives us an opportunity to provide additional evaluation statistics across different domains.
Regarding the relatively limited dataset size and domain coverage.
We devoted 3.5 months to manually annotating ITJoint, with independent checking and final adjudication procedures and rigorous annotation guidelines to ensure its quality. At its current scale, ITJoint is sufficient as a benchmark to support our analysis of different image–text relationships and help identify the main problems faced by models. Several other manually annotated benchmarks have datasets of comparable sizes. For example, ImplicitMemBench [1] (ACL 2026, Best Resource Paper) contains 300 samples, S2I-Bench [2] (CVPR 2026) contains 400 questions, and Video-MMMU [3] (ACL 2026) contains 900 manually annotated questions.
Although its domain coverage is relatively limited, our paper primarily analyzes model performance under different image–text relationships. In this respect, ITJoint provides sufficient coverage of image–text relationships from both query-level and instance-level perspectives.
Reporting results by domain, with confidence intervals using documents as the sampling unit.
Although our analysis of model capabilities primarily focuses on different image–text relationships, reporting evaluation results across domains can also help us gain a more comprehensive understanding of the image–text joint extraction task. We therefore further computed Strict Joint F1 for four models under both standalone inference and Q2IT settings, stratified by domain, and calculated 95% confidence intervals using 20,000 bootstrap resamples with documents as the sampling unit. The results are shown below (paper lines 1076–1093):
| Domain (No. of documents) | Setting | Gemini | GPT | Qwen | GLM |
|---|---|---|---|---|---|
| Overall (41) | Standalone | 50.72 [43.39, 58.29] | 18.49 [13.12, 24.74] | 26.30 [19.16, 33.79] | 12.79 [8.46, 17.58] |
| Q2IT | 72.21 [66.72, 77.66] | 65.28 [58.94, 71.59] | 49.23 [42.83, 55.93] | 53.68 [46.93, 60.77] | |
| Medicine (16) | Standalone | 62.54 [53.42, 73.10] | 20.90 [12.94, 30.51] | 36.46 [22.66, 51.57] | 17.60 [9.91, 26.90] |
| Q2IT | 71.97 [63.98, 80.72] | 67.13 [57.30, 77.39] | 51.53 [40.29, 63.92] | 58.92 [46.92, 70.98] | |
| Archaeology (14) | Standalone | 35.96 [26.63, 46.81] | 12.76 [5.12, 22.52] | 13.54 [7.35, 19.04] | 7.54 [2.68, 13.49] |
| Q2IT | 75.55 [66.70, 84.13] | 63.06 [50.98, 75.45] | 44.15 [37.40, 51.45] | 46.65 [38.08, 56.50] | |
| Biology (7) | Standalone | 42.74 [27.55, 57.27] | 15.38 [6.98, 25.71] | 26.99 [13.37, 41.12] | 12.09 [3.77, 22.46] |
| Q2IT | 61.20 [44.16, 76.89] | 55.08 [46.40, 63.37] | 45.39 [33.33, 56.19] | 44.37 [35.70, 52.57] | |
| Art (3) | Standalone | 58.78 [25.71, 92.86] | 40.05 [11.36, 73.21] | 20.62 [1.14, 34.77] | 14.09 [0.00, 25.00] |
| Q2IT | 84.04 [67.05, 93.30] | 83.58 [62.88, 94.05] | 67.52 [18.33, 96.43] | 74.66 [43.56, 94.64] | |
| Agriculture (1) | Standalone | 73.47 [NA] | 19.48 [NA] | 25.00 [NA] | 0.00 [NA] |
| Q2IT | 73.96 [NA] | 82.68 [NA] | 54.14 [NA] | 63.45 [NA] |
The results show that, although the four standalone models perform differently across domains, none consistently achieves high scores in every domain. This indicates that the challenges of image–text joint extraction are not confined to medical and archaeological documents; they can also be observed in biology, art, and agriculture.
[1] Qin C, Feng X, Ma W, et al. ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models. ACL, 2026.
[2] Xie Q, Yuan K, Qu Y, et al. Score2Instruct: Scaling Up Video Quality-Centric Instructions via Automated Dimension Scoring. CVPR, 2026.
[3] Hu K, Wu P, Pu F, et al. Video-MMMU: Evaluating Knowledge Acquisition from Multidisciplinary Professional Videos. ACL, 2026.
Official Comment by Authors – Part 1
We sincerely thank the reviewer for the thoughtful evaluation and constructive suggestions regarding error attribution, domain coverage, and annotation checking. We respond to each point below.
Response:
We sincerely thank the reviewer for pointing out that IoU alone cannot distinguish incorrect image selection from coordinate deviations. This suggestion prompted us to conduct a more detailed diagnosis of localization errors.
Under ITJoint's strict evaluation, the IoU between a predicted box and the GT box must reach 0.75 for the image localization evaluation to be passed. However, failure to meet this threshold may indicate either that a model selected the wrong image or that it found the correct image but failed to predict its boundaries accurately. To investigate this issue further, we first selected GPT and Gemini, which show a substantial performance gap, for analysis. We then focused on GPT, which has the lower Joint score, and supplemented our analysis with quantitative IoP and IoG results and a case-by-case examination of the causes of its lost points.
Additional evidence already included in the paper.
The localization-method ablation for GPT under the Q2IT framework (paper lines 1347–1371) shows that, while keeping the Evidence Agent and Page Agent unchanged, simply switching GPT from direct coordinate generation to candidate bounding-box selection increases its 1-to-1 score from 0.170 to 0.534 and its 1-to-N score from 0.078 to 0.452. This indicates that GPT is capable of identifying and selecting the correct images, but directly generating accurate image coordinates remains a clear bottleneck.
Quantitative analysis of IoP and IoG.
To investigate whether IoU localization failures result from missing the correct target or from inaccurate bounding-box boundaries, we compared IoU, IoG, and IoP on 122 Single-Positive queries containing exactly one GT image. In our task, IoG reflects how completely the predicted box covers the target, IoP reflects how effectively it excludes irrelevant regions, and IoU jointly assesses both aspects.
We measured the pass rates of all three metrics at the same threshold of 0.75, keeping the original predictions and all other evaluation requirements unchanged. Since IoU ≥ 0.75 (the IoU threshold required for successful image matching in our task) necessarily implies both IoG ≥ 0.75 and IoP ≥ 0.75, the difference between the IoG and IoU pass rates at the same threshold directly measures the proportion of queries whose predicted boxes cover at least 75% of the GT region but fail the strict IoU criterion. For these cases, examining whether IoP falls below 0.75 helps identify predicted boxes that include excessive irrelevant regions, suggesting inaccurate boundaries rather than a complete failure to cover the correct target. The results are shown below (paper lines 1375–1386):
The results indicate that:
Some strict localization failures are more consistent with inaccurate bounding-box boundaries than with a complete failure to cover the correct target. Both GPT and Gemini exhibit substantially higher IoG pass rates than IoU pass rates, with a difference of 40.16 percentage points for GPT. This indicates that many of GPT's predicted boxes already cover most of the target region but still fail the strict localization evaluation. Furthermore, among GPT's 108 IoU localization failures, 42 cases satisfy IoG ≥ 0.75 but IoP < 0.75. These cases have a mean IoG of 93.81% and a mean IoP of 52.38%, indicating that the predicted boxes cover nearly the entire target while including substantial irrelevant regions. This provides further geometric evidence that some localization failures arise from inaccurate bounding-box boundaries.
Differences in bounding-box boundary accuracy may be an important factor contributing to the performance gap between GPT and Gemini. Gemini outperforms GPT on IoU, IoP, and IoG, but the gap between the two models narrows from 35.24 percentage points under IoU to 26.23 percentage points under IoG. Since IoG does not penalize predicted boxes for including regions outside the GT, this narrower gap suggests that, in addition to differences in target coverage, GPT's weaker control over bounding-box boundaries may further widen the performance gap under strict IoU evaluation.