- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper compares joint pretraining, supervised fusion of independently pretrained encoders, and LLM-based inference-time composition across 16 datasets and multiple label budgets. It examines how the preferred approach changes with label availability and introduces a diagnostic-based router to predict which approach will work best on a new task. The evaluation uses shared training and selection protocols, component controls, and grouped validation. The results show budget-dependent changes in the preferred system and suggest that task and representation diagnostics can improve selection within the evaluated configurations.
Critical strengths
- The paper addresses two practically useful questions: how the preferred approach depends on the amount of supervision, and whether that preference can be predicted before fitting all candidate systems. I find this combination particularly interesting because it connects empirical comparisons to decisions practitioners actually need to make.
- The experimental design is careful and systematic. Shared evaluation protocols and component controls help distinguish overall system performance from the benefit of combining modalities, while grouped validation provides meaningful evidence of selection performance on unseen datasets. These choices strengthen the credibility of the empirical findings.
Critical weaknesses
- It is unclear how much the observed preferences reflect alignment paradigms rather than the particular checkpoints and their pretraining histories. The paper does not sufficiently establish that its main findings persist across competitive implementations of each approach. The absence of multimodal LLMs that directly process images also leaves an important practical alternative untested on vision–language tasks. Together, these limitations make broader recommendations about which paradigm to choose less convincing.
- The evidence for the router’s generalizability remains limited. Although the benchmark covers several domains, the routing evaluation contains only 15 dataset groups and largely fixed candidate configurations. Held-out-dataset performance is useful evidence, but it does not establish that the selection rules transfer to new domains or different candidate models. Since providing guidance for new tasks is a central contribution, this uncertainty materially affects my assessment.
Questions for authors
Clarification requests:
- What considerations drove the choice of default checkpoints, and how consistently do the existing encoder and controller ablations preserve the main budget-dependent preferences? A clear account of which conclusions remain stable across these changes would help assess their generality.
- What is the rationale for excluding multimodal LLMs that directly receive images from the vision–language comparisons? How should this exclusion limit the practical interpretation of the Agent results?
- What specific predictive capability or scientific insight does the router add beyond prior diagnostic-based model-selection work? Clarifying this distinction would help assess the contribution beyond the benchmark itself.
Suggestions for additional validation: Evaluation on held-out domains and alternative candidate models would strengthen the generalization evidence. Testing under a fixed total annotation budget, including validation labels, would also clarify the practical meaning of the budget-dependent findings. These are requests for additional evidence, rather than ambiguities in the current protocol that can be resolved through explanation alone.