- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
MosaicDrive combines question-conditioned camera/tool selection, a source-aware evidence store, and an Arbiter that checks conflicting observations and requests additional evidence. The paper introduces MOSICQA and evaluates acquisition policies, answer-model training, and conflict handling using automatic answer grading, an annotation-based acquisition benchmark, and synthetic and real disagreement studies.
Critical strengths
-
The evaluation includes useful diagnostic controls: matched-budget random selection, image removal/replacement, and scene-level confidence intervals. The transparent reporting of negative results helps distinguish evidence acquisition from actual evidence use.
-
The separate annotation-based acquisition benchmark provides a more credible assessment of camera selection than agreement with generated answers, using disjoint scenes and explicit visibility controls.
Critical weaknesses
-
Limited methodological novelty. Question-conditioned visual selection, progressive tool-use training, and conflict-aware evidence coordination are already explored in Frame-Voyager, DriveAgent-R1, and InfoCoordiBridge, respectively. MosaicDrive primarily combines fixed acquisition mappings with threshold checks and re-query heuristics (section 3.2 and 3.3). The substantive methodological advance beyond this integration remains unclear.
-
The introduction claims that unseen regions produce no uncertainty and that adjusting retrieval thresholds cannot help. However, an agent can recognize missing evidence from the question and its observation history. The paper neither defines the relevant confidence signal precisely nor evaluates a question- and coverage-aware uncertainty baseline.
-
The fixed acquisition prior lacks convincing effectiveness. On the annotation-based benchmark, category-mapped selection achieves 59.5% accuracy, essentially matching random selection at 59.4% with the same camera budget (Table 3). Thus, the high agreement with legacy camera annotations in Table 4 does not establish that the prior selects more useful evidence.
-
Successful offline acquisition uses one training-derived camera-subset label per named region (section 4.3; Appendix F). A direct lookup of those same labels is not evaluated. Consequently, the reported gains do not establish what policy learning contributes beyond reproducing a region-to-camera rule.
-
MOSICQA uses generated, unreviewed references (section 4.1). More concerningly, removing all images improves the SFT model’s grade by 2.4 points, while substituting another scene’s images has no significant effect (section 4.6). The full system also scores below front-only input, 24.8 versus 28.0 (Table 2).
Questions for authors
-
How does the learned acquisition policy compare with directly executing its training-derived region-to-camera lookup table across the reported acquisition costs?
-
Which evaluated setting demonstrates improved factual answer correctness from executable conflict-triggered re-query, rather than simulated returns or abstention? Can this benefit be established when either source may be erroneous, with answer coverage and query budget controlled
-
Tier 2 lacks corresponding validation evidence. Table 1 marks semantic verification as evaluated on conflict subsets, but Appendix D states that these studies contain only structured distance claims and exercise Tier 1 and re-query. The semantic tier’s evaluation status needs clarification.
-
The “matched-budget” baseline does not match executed calls. Tables 2 and 13 report 8.4 calls for random selection versus 7.7 for the proposed mapping. Please clarify whether matching concerns tool types, requests, or actual executed acquisition cost.