acceptodds
Under review as a conference paper at ICLR 2027

Models Are Mortal, Measurements Are Not: A Portable, Model-Agnostic Representation for Video Misinformation

Abstract

A representation belongs to the model that produced it: replace the model and everything fitted on it must be rebuilt, and a closed model exposes no representation at all. We ask whether the coordinates of a representation can instead be defined in language, as a fixed bank of yes/no questions that any model can answer, and test this on short-video misinformation across eleven frozen vision-language models, two prompting languages, a closed API and human annotators. A linear readout fitted once on any one open model deploys unchanged on the other ten, losing only .040 AUC on average, and, with same-event corroboration, comes within a point of a fine-tuned detector's accuracy. A matched control separates two properties, compatibility and validity. A bank of questions unrelated to the task also transfers across models, yet loses much of its discrimination across datasets, so fixing the coordinates buys compatibility while their content buys validity. A measurement view splits each reading into what the question measures and how reliably a model reads it; it accounts for why aggregates transfer when single questions disagree, and locates where reading thins, above all in the smallest models and in collapsed answer channels. In simpler terms, the questions hold the representation, and a model is only one way of reading it.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, vision-language models, model-free representation, cross-model transfer, measurement invariance, natural-language question bank, short-video misinformation detection, multimodal learning, psychometrics, frozen foundation models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_sAjFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Traditional representations are typically defined by a model’s hidden states, making them difficult to directly transfer across different models and requiring retraining when the underlying model is replaced. This paper explores an alternative approach that defines representations using a fixed set of natural-language questions, providing a model-independent semantic coordinate system. The resulting question-based representations can be transferred across different models. The authors validate this idea through extensive experiments. First, they show that a fixed question bank indeed provides cross-model coordinate compatibility, while aggregating a large number of questions can mitigate individual measurement noise across models. The resulting representation demonstrates transferability across model families, generations, and prompting languages, and can even extend to closed APIs and human annotators.

Critical strengths
  1. The representation portability studied in this paper is innovative and meaningful. The authors validate the transferability of representations across models through strategies such as label-free standardization.
  2. The experimental evaluation is relatively comprehensive. The authors investigate transferability across different model families, model scales, datasets, and languages. In addition, the paper provides a relatively comprehensive measurement theory to support the proposed representation framework.
Critical weaknesses

Although the proposed method demonstrates cross-model transferability, its practical efficiency remains a concern, and the paper lacks a systematic analysis of the associated computational cost. The experiments suggest that the question-based representations constructed by smaller models are not necessarily accurate. Therefore, constructing the 768-dimensional representation typically requires multiple inference runs with relatively large models, which may introduce substantial computational overhead in practical deployment. Although the number of questions can be reduced to some extent, the overall computational cost is still an order of magnitude higher than that of traditional hidden-state-based representations.

Questions for authors
  1. What are the potential approaches for improving computational efficiency? At present, this appears to be the most critical bottleneck of the proposed approach.
  2. Why do different models sometimes produce inconsistent or even opposite responses on individual question dimensions? Is this primarily because the question content is not sufficiently related to the video input, or is it due to noise in the model responses?
  3. Since the number of questions is closely tied to inference efficiency, are there effective strategies for reducing the number of questions while preserving representation quality and transferability?
  4. What characteristics make a question a good and compact semantic coordinate?
Reviewer_K9kwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Conventional representations depend on the model that produces them, making a trained predictor difficult to reuse when the underlying model changes. This paper proposes a portable representation defined by a fixed bank of natural-language questions. Different models answer the same questions, and their standardized responses are passed to a shared linear classifier. Experiments on video misinformation show that the classifier transfers across frozen vision-language models with a small average loss in performance, with additional evaluations using a closed model and human annotators.

Critical strengths

1 A portable representation. This paper defines feature coordinates through shared questions, allowing the same classifier to be reused across models without aligning their hidden representations. 2 Interpretable features. Each coordinate corresponds to a specific question about the input, making the representation and its contribution to a prediction easier to inspect. 3 Extensive transfer analysis. Experiments cover multiple models, languages, and datasets. The unrelated-question control is particularly useful for distinguishing whether features merely transfer across models or also capture information relevant to the task.

Critical weaknesses

1 The questions may be capturing topic information rather than truthfulness. The authors' own negative-control experiment shows that a bank of truth-irrelevant questions still achieves ~0.80 AUC in-domain, close to the real bank's 0.876. This suggests the method may largely rely on topic cues rather than genuine misinformation signals, meaning its validity is only established in cross-dataset settings.

2 Deployment still requires per-target recalibration. Although the classifier weights are fixed, each new target model or domain requires re-estimating per-question means and standard deviations from an unlabeled pool. If the topic distribution or misinformation prevalence shifts, these calibration statistics will shift too, so the decision rule is not truly stable across deployments.

3 The claimed universality across models is overstated. The reported "3B-parameter threshold" masks a continuous degradation in transfer performance and conflates model size with model family. Notably, small models remain accurate in self-fit settings (~0.887 AUC at 1B).

4 The strongest results benefit from unequal supervision and computation. The question subsets used for closed-model and human-annotator transfers were selected using labeled development data, blurring the line between a portable representation and an adapted one. Furthermore, the best-performing variant uses same-event corroboration and requires answering hundreds of questions per video, making the comparison to single-pass detectors unfair in terms of both evidence and computational cost.

5 The paper would benefit from further polishing to improve clarity and readability.

Questions for authors

1 How much of the in-domain discriminative power survives after controlling for video topic, given that a truth-irrelevant question bank still achieves ~0.80 AUC?

2 How sensitive is the real/fake decision threshold to changes in the size, topic mix, or fake prevalence of the unlabeled calibration pool?

3 Can the authors disentangle the reported "3B threshold" into a continuous transfer-loss curve and separate model size from base-LM family, given that 1B models remain accurate in self-fit?

4 Can the authors report closed-model and human-annotator transfer with a fixed, label-free question set, and compare accuracy under matched inference cost and single-video evidence?

Reviewer_Lx3nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Instead of using hidden-layer features, this paper uses answers to a fixed set of yes/no questions as features. A classifier trained on one model can be directly used with another model without retraining. Experiments on short-video misinformation detection show an average AUC drop of only 0.04 across 11 open-source models, with accuracy close to fine-tuned models.

Critical strengths
  1. The idea is simple and clear, and the experiments on transfer across open-source models are comprehensive.

  2. The control experiment using task-irrelevant questions is well designed, and the authors honestly report the unsuccessful results.

Critical weaknesses
  1. The main performance results are not very reliable. Task-irrelevant questions can achieve 0.80 AUC even in-domain, suggesting that the model mainly learns topic information rather than real vs. fake cues. Also "same-event corroboration" uses the dataset's event annotations, and on FakeTT, it even uses videos from later dates, raising concerns about information leakage. Removing this part leads to a significant performance drop.

  2. The advantages are not obvious. For transfer across open-source models, existing methods perform almost the same as the proposed method (0.869 vs. 0.870). The main unique advantage is that it can work with closed-source models and humans, but these results are the weakest. Moreover, some test labels were used during question selection. The paper also lacks the most direct comparison: retraining the classifier on a new model using a small number of labeled samples.

Questions for authors
  1. If the classifier is retrained on a new model using about 100 labeled samples, how does its performance compare to direct transfer?

  2. What are the results without same-event corroboration? Can these results be reported as the main results?

  3. Is there a way to predict whether the method will work on a new model without using any labeled samples?

Reviewer_eerVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a representation whose coordinates are defined by a fixed bank of 768 binary natural-language questions rather than by any model's hidden states. Each VLM answers the same frozen questions about a video's declared inputs; the resulting answer sheet is standardized per question from unlabelled data, and a single L2-regularized logistic regression fitted on one source model is reused verbatim on others. Across 11 frozen open-source VLMs, a closed API (GPT-5.5), and two human annotators, cross-model transfer on FakeSV/FakeTT loses .040 AUC on average among the open models. The theoretical backbone is a congeneric measurement model from psychometrics that decomposes each reading into a shared content tendency and a reader-specific reliability, explaining why item-level disagreement washes out under linear aggregation. A second construct (harmful meme detection) replicates the within-dataset transfer properties but fails across datasets.

Critical strengths

S1. A clean and practically valuable formulation

Decoupling the coordinate system from model internals and anchoring it to natural language is a simple idea with clear engineering payoff: replacing a model requires only ~100 unlabelled videos to re-estimate per-question statistics, while the bank, the readout coefficients, and the decision logic stay fixed. The entire portable instrument consists of three text files.

S2. The measurement-theory framing goes beyond post-hoc narrative

The congeneric model (Eq. 2–3) is not just an interpretive gloss. It predicts why single-item readings disagree while aggregates transfer (Eq. 4), it yields a saturation law with no free parameter that tracks the empirical curve closely (Eq. 5–6, Fig. 3 right), and the fusion predictions in App. B were registered before observation and came close to the observed values. This is the part of the paper with the longest shelf life.

S3. Well-designed controls

The negative-control bank — questions unrelated to veracity, drafted by the same generators and frozen before scoring — cleanly separates compatibility (fixed coordinates transfer across models) from validity (task-relevant content transfers across datasets). The pre-registration of criteria for the GPT-5.5, annotator, and negative-control arms, together with the candid disclosure of post-test-observation decisions in App. J, sets a high methodological standard.

S4. Thorough ablation and selection-bias analysis

The design ablations (Tabs. A.2–A.3) and the four selection-bias experiments (Tab. A.5) make a convincing case that using all questions with regularization is more robust than cherry-picking subsets. The greedy-selection result (cross-dataset AUC drops by .033) and the demonstration that selection touching the evaluation pool inflates comparisons from +.022 to +.061 are particularly informative.

Critical weaknesses

W1. Narrow task scope relative to the generality of the claims (Major)

The full experimental pipeline — cross-model transfer, cross-dataset transfer, negative control, saturation law — is developed on a single construct: short-video misinformation detection across two datasets (FakeSV and FakeTT) from overlapping research groups. The second construct (harmful memes, App. I) occupies 1.5 pages and its cross-dataset transfer fails (.62/.66, matching the shuffled baseline). The authors attribute this to cultural dependence, which is plausible, but the net result is that the framework has been validated end-to-end on one task.

The title and abstract present the work as a general representational paradigm. The evidence supports a qualified claim: a portable representation for video misinformation, with promising but incomplete generalization beyond it. At least one additional construct where cross-model and cross-dataset transfer both hold would substantially strengthen the generality argument. Tasks with less explicit cue structure (e.g., visual grounding, sentiment analysis) would be particularly informative, since video misinformation is cue-rich by nature.

W2. Model-family homogeneity inflates transfer numbers (Major)

Of the 11 open-source readers, 5 are Qwen VLMs and 5 are InternVL VLMs. The paper notes (Sec. 3) that InternVL2.5-1B, InternVL2.5-4B, and InternVL3-8B all build on Qwen2.5 language models, and Tab. C.1 confirms their specific backbones (Qwen2.5-0.5B, Qwen2.5-3B, and Qwen2.5-7B respectively). Only InternVL2.5-2B and InternVL2.5-8B use a non-Qwen backbone (InternLM2.5). This means 8 of the 11 readers (73%) share a Qwen language backbone; only InternVL2.5-2B, InternVL2.5-8B, and Pixtral are truly non-Qwen. Pixtral is furthermore the sole reader from an entirely independent laboratory (different language model, English questions only), and it has one of the highest transfer losses (.056).

The reported mean loss of .040 is therefore heavily weighted toward shared-backbone pairs. This does not invalidate the results, but it means the headline number overstates the degree of model-agnosticism. Including 1–2 readers from fully independent families (e.g., LLaVA-OneVision, Phi-3.5-Vision, CogVLM2) and reporting the loss separately for Qwen-backbone vs. non-Qwen-backbone pairs would clarify how much of the transfer is due to backbone similarity versus the question-based coordinate system itself.

Questions for authors

Q1. What is the transfer loss for a reader from a fully independent model family (no shared Qwen backbone)? Even one additional data point (e.g., LLaVA-OneVision or Phi-3.5-Vision) would clarify whether .040 is a genuine cross-architecture number or partly a family effect.

Q2. Can the content-swap ablation for corroboration be conducted? Replacing same-event excerpts with equal-count excerpts from unrelated events would directly test whether the channel measures corroborative content or merely material availability.

Q3. For the meme cross-dataset failure: is this a question-design issue (solvable with a better bank) or an inherent validity gap between the two benchmarks? If the former, what design principles would guide a more transferable bank?

Comments on the paper

1 comment
Area_Chair_xXq7OpenReview ↗

Please review the authors’ responses and join the discussion

Dear Reviewers,

The authors have responded to all reviews. Please read their responses and the other reviewers’ comments, then post a brief follow-up indicating which concerns, if any, remain and whether the responses have changed your assessment. If your concerns have been resolved, a short confirmation would also be helpful.

Thank you for your time and for contributing to the discussion.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.