acceptodds
Under review as a conference paper at ICLR 2027

ITJoint: Benchmarking and Diagnosing MLLMs on Query-Driven Image-Text Joint Extraction from Long Documents

Abstract

In domain-specific multimodal long documents, images and text jointly convey complex knowledge that cannot be fully captured by plain text alone. Structuring such information into usable multimodal data is therefore a practical demand. Motivated by this demand, we systematically formulate a new research problem, query-driven image-text joint extraction from multimodal long documents, and develop a two-level taxonomy covering challenges arising from user intent and document content. Further, we construct ITJoint, a high-quality, manually annotated benchmark for systematically evaluating and diagnosing MLLM capabilities on this problem, comprising 2,455 pages of domain-specific documents with numerous non-decorative images and 910 answer instances. Our evaluation reveals three key limitations of current MLLMs: they tend to over-rely on visual similarity for image-text association, struggle to exhaustively recover all associated images in one-to-many correspondences, and exhibit a clear gap between page localization and fine-grained image localization. To examine whether these bottlenecks can be mitigated at the system level, we further introduce Q2IT as a strong multi-agent baseline. Our analyses show that multi-agent approaches can compensate for MLLM weaknesses and better leverage their strengths through specialized external tools and structured strategy design, while remaining vulnerable to error propagation across stages. The complete dataset, prompts, and code are publicly available at https://anonymous.4open.science/r/FullResources-72C7/.

Primary area:
datasets and benchmarks
Keywords:
benchmark, diagnostic evaluation, document understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RASbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ITJoint, a benchmark for query-driven image-text joint extraction from long documents. The task requires extracting relevant text with all corresponding image regions. The benchmark contains 316 queries, 910 answer instances and 2,455 document pages across 41 documents. The documents mainly covers medicine and archaeology, together with biology, art, and agriculture. The authors evaluate four MLLMs and propose an agentic baseline.

Critical strengths

Query-driven extraction is a practical task, and the design of task taxonomy captures meaningful challenges beyond conventional textual QA benchmarks. The authors conduct substantial baseline analysis, providing insightful evidence about the fine-grained localization bottlenecks and errors.

Critical weaknesses
  1. The main Joint metric combines semantic association, image-set completeness and bounding-box accuracy, and Soft Joint also retains the same IoU evalutaions. Although the IoU analysis partially address this issue, they do not independently measure whether models identify the correct image-text correspondence. Consequently, low scores and model differences remain difficult to attribute to association failures versus coordinate-generation errors.
  2. The benchmarks contains only 41 documents, with medicine and archaeology accounting for over 70%, providing limited support for broader generalization.
  3. The annotation workflow is carefully designed, additional clarification would be helpful: did the two independent checkers separately record corrections and missing instances before their findings were consolidated? Reporting the overlap and differences in their findings, particularly for set completeness and text-only instances, would provide additional insight into the verification process.
Questions for authors

Please see Critical Weaknesses above.

6 comments
AuthorsOpenReview ↗

Official Comment by Authors – Part 1

We sincerely thank the reviewer for the thoughtful evaluation and constructive suggestions regarding error attribution, domain coverage, and annotation checking. We respond to each point below.


W1: The IoU analysis does not sufficiently establish whether low scores and model performance differences are attributable to association failures or coordinate generation errors.

Response:

We sincerely thank the reviewer for pointing out that IoU alone cannot distinguish incorrect image selection from coordinate deviations. This suggestion prompted us to conduct a more detailed diagnosis of localization errors.

Under ITJoint's strict evaluation, the IoU between a predicted box and the GT box must reach 0.75 for the image localization evaluation to be passed. However, failure to meet this threshold may indicate either that a model selected the wrong image or that it found the correct image but failed to predict its boundaries accurately. To investigate this issue further, we first selected GPT and Gemini, which show a substantial performance gap, for analysis. We then focused on GPT, which has the lower Joint score, and supplemented our analysis with quantitative IoP and IoG results and a case-by-case examination of the causes of its lost points.

Additional evidence already included in the paper.

The localization-method ablation for GPT under the Q2IT framework (paper lines 1347–1371) shows that, while keeping the Evidence Agent and Page Agent unchanged, simply switching GPT from direct coordinate generation to candidate bounding-box selection increases its 1-to-1 score from 0.170 to 0.534 and its 1-to-N score from 0.078 to 0.452. This indicates that GPT is capable of identifying and selecting the correct images, but directly generating accurate image coordinates remains a clear bottleneck.

Quantitative analysis of IoP and IoG.

To investigate whether IoU localization failures result from missing the correct target or from inaccurate bounding-box boundaries, we compared IoU, IoG, and IoP on 122 Single-Positive queries containing exactly one GT image. In our task, IoG reflects how completely the predicted box covers the target, IoP reflects how effectively it excludes irrelevant regions, and IoU jointly assesses both aspects.

We measured the pass rates of all three metrics at the same threshold of 0.75, keeping the original predictions and all other evaluation requirements unchanged. Since IoU ≥ 0.75 (the IoU threshold required for successful image matching in our task) necessarily implies both IoG ≥ 0.75 and IoP ≥ 0.75, the difference between the IoG and IoU pass rates at the same threshold directly measures the proportion of queries whose predicted boxes cover at least 75% of the GT region but fail the strict IoU criterion. For these cases, examining whether IoP falls below 0.75 helps identify predicted boxes that include excessive irrelevant regions, suggesting inaccurate boundaries rather than a complete failure to cover the correct target. The results are shown below (paper lines 1375–1386):

ModelIoU ≥ 0.75IoP ≥ 0.75IoG ≥ 0.75Both IoP and IoG ≥ 0.75
GPT11.48%18.85%51.64%17.21%
Gemini46.72%49.18%77.87%48.36%

The results indicate that:

  • Some strict localization failures are more consistent with inaccurate bounding-box boundaries than with a complete failure to cover the correct target. Both GPT and Gemini exhibit substantially higher IoG pass rates than IoU pass rates, with a difference of 40.16 percentage points for GPT. This indicates that many of GPT's predicted boxes already cover most of the target region but still fail the strict localization evaluation. Furthermore, among GPT's 108 IoU localization failures, 42 cases satisfy IoG ≥ 0.75 but IoP < 0.75. These cases have a mean IoG of 93.81% and a mean IoP of 52.38%, indicating that the predicted boxes cover nearly the entire target while including substantial irrelevant regions. This provides further geometric evidence that some localization failures arise from inaccurate bounding-box boundaries.

  • Differences in bounding-box boundary accuracy may be an important factor contributing to the performance gap between GPT and Gemini. Gemini outperforms GPT on IoU, IoP, and IoG, but the gap between the two models narrows from 35.24 percentage points under IoU to 26.23 percentage points under IoG. Since IoG does not penalize predicted boxes for including regions outside the GT, this narrower gap suggests that, in addition to differences in target coverage, GPT's weaker control over bounding-box boundaries may further widen the performance gap under strict IoU evaluation.

AuthorsOpenReview ↗

Official Comment by Authors – Part 2

Continued from W1.

Case-by-case inspection statistics.

We further inspected these 108 failures individually. In 23 cases, the predicted box covers at least 90% of the GT, with an average coverage of 98.62%, but the median predicted-box area is 2.08 times the GT area. Another 51 predicted boxes partially overlap with the GT. Among the 41 cases where other candidate images are present and the distance and overlap criteria agree, 37 (90.2%) have prediction boxes closer to the GT and with greater overlap with it. Taken together, among GPT's 108 localization failures, 60 cases (55.6%) have prediction boxes that clearly cover or are closer to the correct image. Their error patterns are more consistent with inaccurate bounding-box coordinates than with selecting a different, incorrect image (paper lines 1389–1402).

AuthorsOpenReview ↗

Official Comment by Authors – Part 3

W2: The dataset has limited document-domain coverage.

Response:

We sincerely thank the reviewer for carefully considering the dataset's domain coverage and its implications for interpreting the results. This gives us an opportunity to provide additional evaluation statistics across different domains.

Regarding the relatively limited dataset size and domain coverage.

We devoted 3.5 months to manually annotating ITJoint, with independent checking and final adjudication procedures and rigorous annotation guidelines to ensure its quality. At its current scale, ITJoint is sufficient as a benchmark to support our analysis of different image–text relationships and help identify the main problems faced by models. Several other manually annotated benchmarks have datasets of comparable sizes. For example, ImplicitMemBench [1] (ACL 2026, Best Resource Paper) contains 300 samples, S2I-Bench [2] (CVPR 2026) contains 400 questions, and Video-MMMU [3] (ACL 2026) contains 900 manually annotated questions.

Although its domain coverage is relatively limited, our paper primarily analyzes model performance under different image–text relationships. In this respect, ITJoint provides sufficient coverage of image–text relationships from both query-level and instance-level perspectives.

Reporting results by domain, with confidence intervals using documents as the sampling unit.

Although our analysis of model capabilities primarily focuses on different image–text relationships, reporting evaluation results across domains can also help us gain a more comprehensive understanding of the image–text joint extraction task. We therefore further computed Strict Joint F1 for four models under both standalone inference and Q2IT settings, stratified by domain, and calculated 95% confidence intervals using 20,000 bootstrap resamples with documents as the sampling unit. The results are shown below (paper lines 1076–1093):

Domain (No. of documents)SettingGeminiGPTQwenGLM
Overall (41)Standalone50.72 [43.39, 58.29]18.49 [13.12, 24.74]26.30 [19.16, 33.79]12.79 [8.46, 17.58]
Q2IT72.21 [66.72, 77.66]65.28 [58.94, 71.59]49.23 [42.83, 55.93]53.68 [46.93, 60.77]
Medicine (16)Standalone62.54 [53.42, 73.10]20.90 [12.94, 30.51]36.46 [22.66, 51.57]17.60 [9.91, 26.90]
Q2IT71.97 [63.98, 80.72]67.13 [57.30, 77.39]51.53 [40.29, 63.92]58.92 [46.92, 70.98]
Archaeology (14)Standalone35.96 [26.63, 46.81]12.76 [5.12, 22.52]13.54 [7.35, 19.04]7.54 [2.68, 13.49]
Q2IT75.55 [66.70, 84.13]63.06 [50.98, 75.45]44.15 [37.40, 51.45]46.65 [38.08, 56.50]
Biology (7)Standalone42.74 [27.55, 57.27]15.38 [6.98, 25.71]26.99 [13.37, 41.12]12.09 [3.77, 22.46]
Q2IT61.20 [44.16, 76.89]55.08 [46.40, 63.37]45.39 [33.33, 56.19]44.37 [35.70, 52.57]
Art (3)Standalone58.78 [25.71, 92.86]40.05 [11.36, 73.21]20.62 [1.14, 34.77]14.09 [0.00, 25.00]
Q2IT84.04 [67.05, 93.30]83.58 [62.88, 94.05]67.52 [18.33, 96.43]74.66 [43.56, 94.64]
Agriculture (1)Standalone73.47 [NA]19.48 [NA]25.00 [NA]0.00 [NA]
Q2IT73.96 [NA]82.68 [NA]54.14 [NA]63.45 [NA]

The results show that, although the four standalone models perform differently across domains, none consistently achieves high scores in every domain. This indicates that the challenges of image–text joint extraction are not confined to medical and archaeological documents; they can also be observed in biology, art, and agriculture.

[1] Qin C, Feng X, Ma W, et al. ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models. ACL, 2026.

[2] Xie Q, Yuan K, Qu Y, et al. Score2Instruct: Scaling Up Video Quality-Centric Instructions via Automated Dimension Scoring. CVPR, 2026.

[3] Hu K, Wu P, Pu F, et al. Video-MMMU: Evaluating Knowledge Acquisition from Multidisciplinary Professional Videos. ACL, 2026.

Reviewer_gYtZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a query-driven image-text joint extraction task for long multimodal documents. Its core contributions include developing a two-level taxonomy covering user query types and image-text relationships within documents, and constructing the manually annotated ITJoint benchmark based on this taxonomy. The benchmark systematically evaluates the capabilities of existing multimodal large language models in image-text association, cross-page retrieval, exhaustive image extraction, and fine-grained localization. In addition, the paper introduces Q2IT, a multi-agent baseline, to investigate the effects of external tools and staged collaboration on task performance. Overall, the paper's main value lies in clarifying the gap between existing document understanding tasks and the requirements of joint image-text extraction, while providing targeted evaluation resources and error analyses.

Critical strengths
  1. The research problem has practical value. Existing document question-answering tasks typically focus on textual answers and the localization of supporting evidence, whereas this paper requires extracting query-relevant textual attributes together with their corresponding images. This task addresses practical needs such as document knowledge organization and multimodal dataset construction, giving it a degree of independence and application value.

  2. The annotation process demonstrates a degree of rigor. The paper adopts a procedure involving multiple rounds of self-checking, verification by two independent reviewers, and final adjudication by experts. It also explicitly requires set queries to cover all qualifying instances as comprehensively as possible. These measures help improve annotation reliability.

  3. The experimental analysis is relatively thorough. The paper not only compares four multimodal large language models but also analyzes missed images in multi-image mappings, the performance gap between page-level and image-level localization, and incorrect image-text associations. The comparison between Q2IT and a simplified pipeline using the same tools also provides preliminary evidence for examining the role of structured multi-agent strategies.

Critical weaknesses
  1. The definition of long documents and the associated dataset statistics are insufficiently specified. The paper only states that each document should contain a continuous page range spanning tens to over one hundred pages. However, it does not provide an operational definition of a long document or specify the minimum page count required to qualify as one. Why would an 18-page document not qualify? The shortest, average, and longest documents in the actual dataset contain 22, 59.88, and 178 pages, respectively. This is a considerable range. The rationale for selecting documents ranging from tens to over one hundred consecutive pages therefore requires further clarification.

  2. Concerns about dataset quality. Although ITJoint contains 2,455 document pages and 910 answer instances, the dataset consists of only 41 source documents, which is relatively small. I understand that collecting long documents is challenging and that experimental time may be limited, but the dataset should be expanded further. The domain distribution of ITJoint is 39.0% medicine, 34.2% archaeology, 17.1% biology, 7.3% art, and 2.4% agriculture. Across the 41 source documents, these percentages correspond to exactly 16 medical documents, 14 archaeological documents, 7 biological documents, 3 art documents, and 1 agricultural document. These figures further highlight the limitations of the dataset's scale. Moreover, although the paper describes the benchmark as bilingual and reports the domain distributions of the Chinese and English subsets, the underlying distribution across languages can be inferred to some extent from the limited number of source documents. For example, the agriculture domain is represented by only one document.

  3. Copyright concerns regarding the data. The paper states that the documents were obtained from public sources and recommendations by students or researchers with relevant domain expertise. The documents were then screened in two rounds based on domain relevance, diversity of image-text relationships, and the number of non-decorative images. However, domain-specific atlases and medical books are often subject to copyright restrictions, and the paper does not adequately clarify the copyright status of these materials or whether their redistribution is permitted.

  4. Inconsistent or difficult-to-reconcile statistical definitions. The main text reports 166 instances for the 1-to-N category. If Figure 2 reports 177 instances for 1-to-0 and 161 negative single-instance queries, only approximately 16 empty-image instances remain among set queries. Table 1, however, reports 148 positive instances and 107 negative instances for negative set queries. These numbers do not align with the definition of negative instances as instances with no corresponding images. This discrepancy undermines confidence in the reported category distributions and conclusions concerning negative examples.

  5. Insufficient mechanistic evidence for the claim of "over-reliance on visual similarity." Table 2 only establishes that the model may return an image even when its textual answer is correct. This behavior could result from visual similarity, but it could also reflect an assumption that an image must always be provided or a tendency to select a nearby image. The paper relies primarily on a qualitative example in Figure 10, without statistically examining whether the incorrectly selected images are the most visually similar regions on the page or whether stronger layout cues were overlooked.

  6. Ambiguities in the evaluation protocol. Instance matching does not use the entity identifiers eje_j in the corresponding formula. Instead, it performs greedy matching based on bounding boxes, followed by text matching for 1-to-0 instances. Algorithm 1 invokes an LLM judge for instances with images, whereas 1-to-0 instances are marked as text-correct directly after successful text matching. Thus, two different standards for textual correctness appear to coexist. Furthermore, the argument that greedy matching "does not affect the results" relies on the assumption that a single predicted box cannot legally match multiple ground-truth instances at an IoU threshold of 0.75. This assumption requires empirical verification, particularly for composite figures and images of adjacent artifacts.

Questions for authors
  1. Please report the exact number of source documents in the abstract or introduction.

  2. For the substantial performance gap between GPT and Gemini on positive single-instance queries (0.099 versus 0.447), how much of the difference is attributable to coordinate formatting, refusal behavior, page-number mapping, and parsing failures?

  3. What are the sources, licenses, and redistribution permissions of the 41 source documents?

  4. Are the page images processed by the independent models constructed by stitching three columns together? Please report page-level, image-level, and joint scores separately for documents processed without stitching and long documents processed with stitching. In addition, please provide a single-model pipeline without stitching, such as page-by-page processing, retrieval followed by inspection of only a small number of original pages, or native PDF input. If the performance gap disappears under such a setup, which of the three identified bottlenecks remain valid?

  5. The gap between textual correctness and joint correctness for 1-to-0 cases is interpreted as evidence of over-reliance on visual similarity (with gaps of 0.105 for Gemini, 0.226 for GPT, 0.327 for Qwen, and 0.324 for GLM). However, the gap itself only demonstrates that providing images changes the outcome. Figure 10 presents a single example. Meanwhile, the relationship between negative-instance counts in Table 1 and Figure 2 remains unclear, and the double-review process covers only three queries. Could missing distant figures cause correct predictions of empty image sets to be marked incorrect, or cause genuine images found by the model to be classified as hallucinations? Please clarify this issue further.

  6. Has greedy matching ever encountered a composite figure in which one predicted box has an IoU of at least 0.75 with two ground-truth boxes? Why does Algorithm 1 not invoke TextJudge for 1-to-0 instances, and is the textual evaluation standard consistent with that used for instances containing images? How is partial correctness handled for answers containing multiple attributes, such as the attribution and length in Figure 8?

13 comments
AuthorsOpenReview ↗

Official Comment by Authors – Part 1

We sincerely thank the reviewer for the thorough assessment and constructive suggestions regarding dataset construction, evaluation, and experimental analysis. We address each point below.


W1: The definition of long documents and the related dataset statistics are insufficiently clear.

Response:

We apologize that the original manuscript did not sufficiently explain the rationale for selecting document lengths or clearly present the page-count distribution, making it more difficult for the reviewer to understand the dataset.

ITJoint selects consecutive pages from source documents to preserve semantic completeness and the necessary image–text context. As a result, document lengths vary with the actual content, and the shortest document currently contains 22 pages. Previous studies have likewise not established a uniform minimum page count for long documents. For example, MMLongBench-Doc [1] uses 15 pages, whereas LongDocURL [2] uses 50 pages.

Because we hope this task can eventually be used in real-world knowledge organization or dataset construction, we selected documents ranging from several dozen to more than one hundred pages to reflect, as closely as possible, variations in document length in practical applications.

To present ITJoint's document-length composition more clearly, we further calculated the complete page-count distribution and added it to paper lines 990–1002:

Document Length (Pages)Number of DocumentsPercentage
22–401229.3%
41–601024.4%
61–801331.7%
81–10037.3%
101–17837.3%
Total41100.0%

In addition, Appendix F.2 of our paper already shows that all four models exhibit declining performance as document length increases.

[1] Ma Y, Zang Y, Chen L, et al. MMLongBench-Doc: Benchmarking Long-Context Document Understanding with Visualizations. NeurIPS, 2024.

[2] Deng C, Yuan J, Bu P, et al. LongDocURL: A Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating. ACL, 2025.

W2: The dataset is relatively small and has limited domain coverage.

Response:

We sincerely thank the reviewer for carefully considering the dataset's size, quality, and domain coverage. These specific questions have helped us explain the benchmark's design objectives more clearly.

We devoted 3.5 months to manually annotating ITJoint, with independent checking and final adjudication procedures and rigorous annotation guidelines to ensure its quality. At its current scale, ITJoint can serve as a benchmark supporting our analysis of different image–text relationships and helping us identify the main problems faced by models. Several other manually annotated benchmarks have datasets of comparable sizes. For example, ImplicitMemBench [1] (ACL 2026, Best Resource Paper) contains 300 samples, S2I-Bench [2] (CVPR 2026) contains 400 questions, and Video-MMMU [3] (ACL 2026) contains 900 manually annotated questions.

Although its domain coverage is relatively limited, our paper primarily analyzes model performance under different image–text relationships. In this respect, ITJoint provides detailed coverage of different image–text relationships at both the query and instance levels.

[1] Qin C, Feng X, Ma W, et al. ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models. ACL, 2026.

[2] Xie Q, Yuan K, Qu Y, et al. Score2Instruct: Scaling Up Video Quality-Centric Instructions via Automated Dimension Scoring. CVPR, 2026.

[3] Hu K, Wu P, Pu F, et al. Video-MMMU: Evaluating Knowledge Acquisition from Multidisciplinary Professional Videos. ACL, 2026.

AuthorsOpenReview ↗

Official Comment by Authors – Part 2

W3/Q3: The paper does not explain the copyright status of the dataset or whether future researchers can use it.

Response:

We apologize that the original manuscript did not sufficiently explain the copyright and licensing conditions of the data sources, and we thank the reviewer for reminding us to provide this important information. We have checked the copyright statements and official terms of use of the source books individually, compiled a list of bibliographic details, copyright information, and relevant links, and uploaded it to the anonymous resource link in the paper. Our review of these statements and terms indicates that the relevant materials may be used for noncommercial research or educational purposes under specified conditions. For example, the Metropolitan Museum of Art permits qualifying noncommercial educational use, while Springer Nature, Wiley, Cambridge, and Taylor & Francis also provide pathways for research use subject to conditions. Our annotation JSON has been made available at the anonymous link, and researchers may use ITJoint in compliance with the applicable copyright and licensing conditions (paper lines 709–716).

W4: The negative-example statistics in Figure 2 and Table 1 appear inconsistent.

Response:

We apologize that the original manuscript did not clearly explain the meanings of Pos. and Neg. in Figure 2, causing difficulty when the reviewer checked the statistics. In fact, the numbers 161 and 70 in Figure 2 represent the counts of positive and negative single-instance queries, respectively. The 70 negative single-instance queries each correspond to one 1-to-0 instance. Adding the 107 negative instances contained in the negative Set Queries in Table 1 gives exactly the 177 1-to-0 instances shown in Figure 2: 70 + 107 = 177. Therefore, our statistics are consistent. We have clarified in paper lines 178–181 that Pos. and Neg. stand for Positive and Negative, respectively.

W5/Q5: There is insufficient mechanistic evidence for the claim of “over-reliance on visual similarity”; annotation review appears to involve only 3 queries, raising concerns about missed images on distant pages.

Response:

We sincerely thank the reviewer for the detailed and specific comments on the evidence supporting visual-similarity attribution, negative-example analysis, and annotation completeness. These suggestions helped us conduct further targeted checks and experiments.

Regarding the impact of visual similarity on incorrect image–text associations.

Our analysis of the impact of visual similarity in the paper was primarily based on case observations and lacked sufficient quantitative evidence. We therefore conducted the following additional experiments (paper lines 1110–1121):

(1) We first examined whether these incorrect associations contradicted explicit evidence in the documents.

We first inspected 1-to-0 cases in which the textual answer was correct but an image was incorrectly returned. We found that 52.9% of Gemini's cases and 62.5% of Qwen's cases contained explicit evidence that ruled out the returned image.

This shows that, in a substantial proportion of erroneous cases, the document already provides evidence that rules out the incorrect image, yet the model still establishes an incorrect association.

(2) Next, we examined whether these incorrectly selected images had high image–text similarity.

For the cases above with explicit exclusion evidence, we used SigLIP2 [1] to compute image–text similarity between the incorrectly selected images and the target descriptions, using random regions on the same page and other real images as controls. Compared with random regions on the same page, the incorrectly selected images had similarity scores above the control mean in 100% of Gemini's cases and 95.0% of Qwen's cases. The proportions in which they ranked first in similarity were 66.7% and 80.0%, respectively. Even when other real images were used as controls, the incorrectly selected images still scored above the mean similarity of the other images in 80.0% of Gemini's cases and 75.0% of Qwen's cases.

These results show that the images incorrectly selected by the models generally have high image–text similarity to the target descriptions.

AuthorsOpenReview ↗

Official Comment by Authors – Part 3

Continued from W5/Q5.

(3) However, the models might simply favor images located near the textual evidence. We therefore further examined the page locations of the incorrectly selected images.

Among erroneous cases where the page containing the textual evidence also had illustrations, Gemini selected an incorrect image from another page rather than an image on the evidence page in 50.0% of cases, while Qwen did so in 38.9% of cases. For these cross-page selection errors, we further compared the image–text similarity of the incorrectly selected images and the illustrations on the evidence page to the same target description. The results showed that the incorrectly selected images had higher mean similarity in 100% of Gemini's cases and 71.4% of Qwen's cases.

This indicates that models do not always select the image closest to the textual evidence. In some cases, even when an incorrect image is on another page, the model still selects the image that is more similar to the target description. This provides further evidence that visual similarity may affect image–text associations.

We sincerely hope that these supplementary analyses provide stronger empirical support for the relationship between visual similarity and incorrect image–text associations and help alleviate your concerns about the sufficiency of the relevant evidence, at least to some extent.

Regarding whether models tend to associate an image with every textual answer.

In our original experimental prompts, we explicitly instructed the models to return an empty image set if no corresponding image exists. Previous research has also shown that these model families have relatively good multimodal instruction-following capabilities [2].

Regarding the scope of human review of dataset annotations.

The wording “three queries raised disagreements” in the original paper was insufficiently clear and led to questions about the scope of annotation review. In fact, all 316 queries were independently checked by two reviewers. Only the disagreements involving 3 queries remained unresolved after discussion and required final adjudication. We have clarified the distinction between independent checking and final adjudication in paper lines 254–265 and 685–706.

Regarding the risk of missing images on other pages.

We devoted 3.5 months to manually annotating ITJoint, with independent checking and final adjudication procedures and rigorous annotation guidelines to ensure that the dataset does not contain omissions or similar problems. Specifically, annotators and reviewers must check the entire document page by page and confirm all corresponding images for each instance, including images located on pages different from the textual evidence.

[1] Tschannen M, Gritsenko A, Wang X, et al. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. arXiv:2502.14786, 2025.

[2] Qian Y, Ye H, Fauconnier J P, et al. MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs. ICLR, 2025.

Reviewer_UfxAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ITJoint, a benchmark for extracting text attributes and their corresponding images from long documents based on user queries. It contains 41 documents, 316 queries, and 910 answer instances. The authors evaluate four multimodal models and propose Q2IT, a three-stage system for text retrieval, page localization, and image selection.

Critical strengths
  1. The benchmark includes cases with no matching image, several matching images, and text and images on different pages. It checks whether models find all the images for an item and return no image when none is available. These errors would be missed by checking the text answer alone.
  2. According to Table 14, replacing region selection with direct box prediction lowers GPT's score from 0.534 to 0.170 on one-to-one cases, with the earlier stages unchanged. This supports the finding that models struggle to predict image boxes even after the search has been narrowed to a few pages.
Critical weaknesses
  1. The dataset contains only 41 documents and 316 queries. Medicine and archaeology account for 73.2% of the documents. Results by domain and language are not reported, and there are no confidence intervals. With this size and distribution, it is hard to tell whether the findings hold for other kinds of documents.
  2. Table 10 reports 97.7% and 97.5% agreement with human judgments for direct inference and Q2IT, respectively. However, these checks cover only the text scores for Gemini outputs. The paper does not explain how the two reviewers' judgments were used to compute these rates or report their agreement with each other. Section 4.2 describes several rounds of annotation checks, but agreement on the full answer sets and text-image matches is not reported.
Questions for authors
  1. Could the authors report results by domain and language, along with confidence intervals using documents as the sampling unit?
  2. How were the human judgments used to calculate the rates in Table 10, and how often did the two reviewers agree? Are there agreement results for the annotated answer sets and text-image matches?
5 comments
AuthorsOpenReview ↗

Official Comment by Authors – Part 1

We sincerely thank the reviewer for the thoughtful assessment and for recognizing ITJoint's coverage and localization analysis. We address your suggestions on stratified evaluation and agreement statistics below.


W1/Q1: Given the relatively small dataset size and limited domain coverage, it is recommended to report results by domain and language and provide confidence intervals using documents as the sampling unit.

Response:

We sincerely thank the reviewer for carefully highlighting the importance of domain- and language-stratified results and document-level confidence intervals. This suggestion has helped us present the evaluation results across different settings more comprehensively.

Regarding the relatively limited dataset size and domain coverage.

We devoted 3.5 months to manually annotating ITJoint, with independent checking and final adjudication procedures and rigorous annotation guidelines to ensure its quality. At its current scale, ITJoint is sufficient as a benchmark to support our analysis of different image–text relationships and help identify the main problems faced by models. Several other manually annotated benchmarks have datasets of comparable sizes. For example, ImplicitMemBench [1] (ACL 2026, Best Resource Paper) contains 300 samples, S2I-Bench [2] (CVPR 2026) contains 400 questions, and Video-MMMU [3] (ACL 2026) contains 900 manually annotated questions.

Although its domain coverage is relatively limited, our paper primarily analyzes model performance under different image–text relationships. In this respect, ITJoint provides detailed coverage of different image–text relationships at both the query and instance levels.

Reporting results by domain and language, with confidence intervals using documents as the sampling unit.

Although our analysis of model capabilities primarily focuses on different image–text relationships, reporting evaluation results across domains can also help us gain a more comprehensive understanding of the image–text joint extraction task. We therefore further computed Strict Joint F1 for four models under both standalone inference and Q2IT settings, stratified by domain, and calculated 95% confidence intervals using 20,000 bootstrap resamples with documents as the sampling unit. The results are shown below (paper lines 1076–1093):

Domain (No. of documents)SettingGeminiGPTQwenGLM
Overall (41)Standalone50.72 [43.39, 58.29]18.49 [13.12, 24.74]26.30 [19.16, 33.79]12.79 [8.46, 17.58]
Q2IT72.21 [66.72, 77.66]65.28 [58.94, 71.59]49.23 [42.83, 55.93]53.68 [46.93, 60.77]
Medicine (16)Standalone62.54 [53.42, 73.10]20.90 [12.94, 30.51]36.46 [22.66, 51.57]17.60 [9.91, 26.90]
Q2IT71.97 [63.98, 80.72]67.13 [57.30, 77.39]51.53 [40.29, 63.92]58.92 [46.92, 70.98]
Archaeology (14)Standalone35.96 [26.63, 46.81]12.76 [5.12, 22.52]13.54 [7.35, 19.04]7.54 [2.68, 13.49]
Q2IT75.55 [66.70, 84.13]63.06 [50.98, 75.45]44.15 [37.40, 51.45]46.65 [38.08, 56.50]
Biology (7)Standalone42.74 [27.55, 57.27]15.38 [6.98, 25.71]26.99 [13.37, 41.12]12.09 [3.77, 22.46]
Q2IT61.20 [44.16, 76.89]55.08 [46.40, 63.37]45.39 [33.33, 56.19]44.37 [35.70, 52.57]
Art (3)Standalone58.78 [25.71, 92.86]40.05 [11.36, 73.21]20.62 [1.14, 34.77]14.09 [0.00, 25.00]
Q2IT84.04 [67.05, 93.30]83.58 [62.88, 94.05]67.52 [18.33, 96.43]74.66 [43.56, 94.64]
Agriculture (1)Standalone73.47 [NA]19.48 [NA]25.00 [NA]0.00 [NA]
Q2IT73.96 [NA]82.68 [NA]54.14 [NA]63.45 [NA]

The results show that, although the four standalone models perform differently across domains, none consistently achieves high scores in every domain. This indicates that the challenges of image–text joint extraction are not confined to medical and archaeological documents; they can also be observed in biology, art, and agriculture.

[1] Qin C, Feng X, Ma W, et al. ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models. ACL, 2026.

[2] Xie Q, Yuan K, Qu Y, et al. Score2Instruct: Scaling Up Video Quality-Centric Instructions via Automated Dimension Scoring. CVPR, 2026.

[3] Hu K, Wu P, Pu F, et al. Video-MMMU: Evaluating Knowledge Acquisition from Multidisciplinary Professional Videos. ACL, 2026.

AuthorsOpenReview ↗

Official Comment by Authors – Part 2

Continued from W1/Q1.

Additionally, our cross-domain case study (Appendix F.4, Table 13) provides further evidence that challenges in image-text association, one-to-many image recovery, and fine-grained image localization can also arise in an additional domain (geology).

ITJoint includes both Chinese and English queries. To better understand model performance across different language settings, we also calculated Strict Joint F1 separately by query language, using the same document-level bootstrap procedure to obtain confidence intervals (paper lines 1096–1108).

Query languageSettingGeminiGPTQwenGLM
ChineseStandalone45.74 [33.84, 58.76]14.96 [7.27, 24.77]30.27 [19.56, 41.98]16.23 [8.00, 25.70]
Q2IT75.62 [67.39, 83.76]67.00 [57.56, 76.99]55.04 [47.17, 63.97]58.09 [48.09, 68.71]
EnglishStandalone54.41 [45.93, 63.38]21.09 [13.97, 29.08]23.17 [14.54, 33.24]10.24 [5.93, 14.97]
Q2IT69.70 [62.59, 76.99]64.01 [55.43, 72.39]44.89 [36.29, 54.29]50.43 [41.61, 59.53]

The results show that, although the four standalone models perform differently on Chinese and English queries, their overall scores remain low. This suggests that the challenges of image–text joint extraction are not limited to a particular language. It also shows that the model capability limitations revealed by ITJoint can be observed in both language settings.

W2/Q2: The computation of human–model agreement lacks details, and inter-reviewer agreement is not reported. Agreement between annotators on complete answer sets and text–image matching is also not reported.

Response:

We sincerely thank the reviewer for the detailed suggestions regarding the human evaluation procedure and annotation agreement. These suggestions prompted us to provide further details on the independent checking process and the corresponding agreement statistics.

The computation of human–model agreement lacks details, and inter-reviewer agreement is not reported.

Although Appendix F.5 of our paper reports agreement between the LLM Judge and human judgments, it does not describe the two reviewers' judgment process or the calculation of the agreement rates in detail. We provide these details here and report the agreement rate between the two reviewers.

For the human judgment procedure, we asked two reviewers to independently examine all text Judge results for Gemini in the Standalone (728 records) and Q2IT (825 records) settings. Each reviewer independently determined whether the Judge was correct and recorded cases where they disagreed with its judgment. To facilitate item-by-item inspection, we organized each record in the following format:

Question: Original question
GT: Reference answer
Prediction: Predicted answer
Judge result: True / False

The 97.7% and 97.5% reported in Table 14 of the paper are the arithmetic means of the two reviewers' respective agreement rates with the Judge. We further computed agreement between the two reviewers and human–Judge agreement after combining the cases flagged by either reviewer. The results are shown below (paper lines 1050–1059):

SettingA vs. JudgeB vs. JudgeAverage human–Judge agreementInter-reviewer agreementHuman–Judge agreement after merging disagreements
Standalone97.25%98.08%97.7%96.15%95.74%
Q2IT97.82%97.21%97.5%95.52%95.27%

The corresponding human–Judge review records are available for verification through the anonymous resource link in Footnote 1 (paper line 30).

AuthorsOpenReview ↗

Official Comment by Authors – Part 3

Continued from W2/Q2.

The textual answers in ITJoint are primarily short attribute values with clearly defined reference answers. For such answers, the LLM Judge mainly determines whether a model prediction is semantically equivalent to the GT, rather than assigning subjective scores to complex, open-ended responses. We also measured the lengths of predicted answers from four models under both settings, as shown below (paper lines 1061–1073):

SettingModelMean length (tokens)Median length (tokens)≤64 tokens
StandaloneGemini19.591596.82%
StandaloneGPT-5.425.301995.42%
StandaloneQwen35.782790.26%
StandaloneGLM-4.5V29.062392.99%
Q2ITGemini10.97697.94%
Q2ITGPT-5.414.251097.90%
Q2ITQwen23.681391.64%
Q2ITGLM-4.5V13.52796.71%

These results show that predicted answers from all four models in both settings are predominantly short texts, indicating a degree of similarity in their textual outputs.

In addition, LLM-as-a-Judge is being increasingly adopted in research and is gradually gaining acceptance within the research community. For example, recent studies reported human–model agreement rates of 97.0% [1], 95.7% [2], and 92.0% / 93.2% [3], illustrating the practical use of this evaluation approach in different tasks.

Agreement between annotators on complete answer sets and text–image matching was not reported.

All 316 queries in ITJoint underwent independent checking by two reviewers. We further computed their agreement rates on complete answer entity sets and text–image correspondences before joint discussion and final adjudication. The results are shown below (paper lines 685–696):

Checked itemAgreement rate
Complete answer entity set (all queries)96.20%
Complete answer entity set (set queries)85.88%
Text–image correspondence (complete image sets)96.15%

These results show high agreement between the two reviewers on the complete answer sets and image–text correspondences across all queries. In comparison, the agreement rate for set queries is 85.88%, reflecting the greater difficulty of identifying every qualifying instance. When disagreements arose during checking, the two reviewers first discussed them against the original documents; unresolved issues were then submitted to the final adjudicator. We have provided the corresponding human review records for verification through the anonymous resource link in Footnote 1 on page 1 (paper line 30).

[1] Steyvers M, Tejeda H, Kumar A, et al. What Large Language Models Know and What People Think They Know. Nature Machine Intelligence, 2025.

[2] Kaduri O, Bagon S, Dekel T. What's in the Image? A Deep-Dive into the Vision of Vision Language Models. CVPR, 2025.

[3] Azadani M N, Wang Y, Zhu Y, et al. VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence. arXiv:2605.20676, 2026.


Thank you again for your valuable feedback. We welcome further questions during the discussion and will do our best to provide clarifications and additional analyses.

Comments on the paper

1 comment
Area_Chair_ReJYOpenReview ↗

Post-Rebuttal Discussion

The authors have provided detailed responses and additional analyses. Please review the rebuttal and discuss whether your main concerns have been adequately addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.