- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Traditional representations are typically defined by a model’s hidden states, making them difficult to directly transfer across different models and requiring retraining when the underlying model is replaced. This paper explores an alternative approach that defines representations using a fixed set of natural-language questions, providing a model-independent semantic coordinate system. The resulting question-based representations can be transferred across different models. The authors validate this idea through extensive experiments. First, they show that a fixed question bank indeed provides cross-model coordinate compatibility, while aggregating a large number of questions can mitigate individual measurement noise across models. The resulting representation demonstrates transferability across model families, generations, and prompting languages, and can even extend to closed APIs and human annotators.
Critical strengths
- The representation portability studied in this paper is innovative and meaningful. The authors validate the transferability of representations across models through strategies such as label-free standardization.
- The experimental evaluation is relatively comprehensive. The authors investigate transferability across different model families, model scales, datasets, and languages. In addition, the paper provides a relatively comprehensive measurement theory to support the proposed representation framework.
Critical weaknesses
Although the proposed method demonstrates cross-model transferability, its practical efficiency remains a concern, and the paper lacks a systematic analysis of the associated computational cost. The experiments suggest that the question-based representations constructed by smaller models are not necessarily accurate. Therefore, constructing the 768-dimensional representation typically requires multiple inference runs with relatively large models, which may introduce substantial computational overhead in practical deployment. Although the number of questions can be reduced to some extent, the overall computational cost is still an order of magnitude higher than that of traditional hidden-state-based representations.
Questions for authors
- What are the potential approaches for improving computational efficiency? At present, this appears to be the most critical bottleneck of the proposed approach.
- Why do different models sometimes produce inconsistent or even opposite responses on individual question dimensions? Is this primarily because the question content is not sufficiently related to the video input, or is it due to noise in the model responses?
- Since the number of questions is closely tied to inference efficiency, are there effective strategies for reducing the number of questions while preserving representation quality and transferability?
- What characteristics make a question a good and compact semantic coordinate?
Please review the authors’ responses and join the discussion
Dear Reviewers,
The authors have responded to all reviews. Please read their responses and the other reviewers’ comments, then post a brief follow-up indicating which concerns, if any, remain and whether the responses have changed your assessment. If your concerns have been resolved, a short confirmation would also be helpful.
Thank you for your time and for contributing to the discussion.
Best regards,
Your AC