- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper descrives an approach for learning visual document embeddings that incorporates both full-page similarity and regional evidence similarity during training. To identify evidence regions, the method measures the similarity between query tokens and OCR bounding boxes on the document page. A novel loss function is introduced to enforce that query token embeddings share higher similarity with visual patches inside the evidence region than with patches outside it. Region-level evidence is utilized only during model training; at inference, retrieval is performed at the page level. Experimental evaluations on visual retrieval datasets show performance improvement over the page-level only baseline.
Critical strengths
- Grounding visual page embeddings with region-based OCR evidence provides a novel way to learn fine-grained visual-semantic representations without introducing extra latency at inference time.
- The method demonstrates performance improvement over the baseline trained using only full-page visual similarity across most of the selected benchmark datasets.
Critical weaknesses
- The writing in the Introduction and Related Work / State of the Art sections is difficult to follow and understand without reading the rest of the paper.
- Experimental setting and results: 1) The evaluation relies exclusively on NDCG@k. Other standard retrieval metrics, such as MRR, Precision@K, and Recall@K are missing and would be helpful to aspects of retrieval behavior. 2) valuation is restricted to four subsets from the ViDoRe benchmark, rather than utilizing the complete benchmark or testing on additional visual document retrieval benchmarks. 3) Although overall performance averaged across the four evaluated ViDoRe subsets improves, the proposed method fails to outperform the baseline in some of the individual subsets. Some discussion explaining these performance difference across subsets is necessary along with evaluation on other datasets to test the generalization of the method.
Questions for authors
- The paper does not specify the granularity of the OCR bounding boxes used to detect region evidence (words, lines, paragraphs, cells in a table, ...). Box granularity can impact visual overall retrieval performance and should be discussed and evaluated.
- Some presentation isssues: 1) at the end of Section 3.2, settings A3 and A4 are mentioned prior to being formally defined.2) Table 1 displays the proposed method's results in bold even in cases where competing models achieve higher scores. Common practice uses bold formatting to denote the best score across all compared methods.