- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper tackles multi-view selection for LLM tasks. It proposes Adaptive View Harness (AVH) which leverages historical multi-view behavior to propose adaptive views for new queries without additional target LLM calls using predictive relative utility. AVH is evaluated across eight model--benchmark settings yielding gains of up to +3.42 pp compared to using the original input.
Critical strengths
- The paper is clear and easy to follow.
- Additional analysis separates improvement from global dataset-level view preference and instance-level preference, as well as giving the maximum possible improvement with an oracle, which helps put the performance improvement in perspective.
Critical weaknesses
- The improvement in performance seems marginal compared to just using the original prompt and yet it requires access to historical multi-view interactions, which may limit its use in most practical settings. It is not not compared to a baseline that uses any other type of view selection or prompt transformation. Moreover, the method doesn't show any improvement on three of the four models on the MMLU-Pro dataset while the fallback avoids negative performance.
- The three views proposed by the authors (OV, LV, and OLV) seem very simple, and it's unclear why they would enhance performance in a predictable and consistent way that can be leveraged by a trained predictor.
Questions for authors
- Why were these three view construction methods in particular chosen? Have you experimented with more sophisticated ways to construct prompts, and do you believe this could help the method achieve stronger results
- Have you tried assessing the number of training samples needed to obtain reliable predictors for view selection?
- Have you tried comparing AVH to a prompt optimization baseline that has access to the same historical data?
- The GLM model seems to benefit the most from AVH despite not having more Oracle headroom than the other models, which would suggest that its sensitivity to view choice is not larger. Have you investigated why?
2 comments
Thank you for the response and the additional experiments.
My main concerns about the source of the reported gains remain. The added comparisons suggest that simpler task-level selection or routing achieves similar results, so the benefit of the proposed instance-level selection is still unclear. The parseability analysis also indicates that some of the largest improvements may come from changes in output formatting rather than better answers.
These points leave my concerns about the soundness of the method and its evaluation unresolved. I maintain my score.
Thank you for reading our paper and for your positive comments on the presentation and Oracle analysis. We appreciate your suggestions and have revised the experimental comparisons and discussion of the results.
We agree that collecting multi-view responses requires an initial investment. For each target model, we use 5,260 historical responses on BBH and 12,000 on MMLU-Pro during development. Once the selector is trained, each new question requires one target-model call. AVH achieves positive gains in five of eight settings, while CA retains Identity in the remaining three. We see this approach as more suitable for fixed models serving many questions from a stable task distribution. We've clarified the data requirements and deployment procedure in Section 4.1 and Appendix B.
We added a fixed-view adaptation of Prompt-OIRL using the same four candidate views and historical responses. We trained a separate query–view reward model and included the results in Section 4.3, with its configuration described in Appendix D. We also added task-level and correctness-based routing comparisons in Appendix D. These experiments help us examine AVH alongside other selection strategies under matched target-model responses.
We chose option ordering and layout as two presentation factors. The four combinations give us ID, OV, LV, and OLV, while keeping the question, options, answer labels, and instruction unchanged. Even these simple changes produce different outcomes. For example, on Hunyuan/BBH, LV improves accuracy from 90.03% to 91.93%, while OV reduces it to 86.30%. We use these variations to study whether historical responses can help select a suitable presentation for a new question. We've expanded the explanation in Section 4.1 and Appendix A.
This is an interesting observation. Oracle headroom measures the potential improvement available across views, while AVH must predict which view to use before seeing the model's response. For GLM, output parseability appears to play an important role. On MMLU-Pro, the parse success rate increases from 66.4% with Identity to 80.8% with AVH, alongside a 3.20 pp accuracy gain. On the all-four-parseable subset, the accuracy difference is −2.04 pp. We've added these results and their interpretation in Section 4.4 and Appendix F.