acceptodds
Withdrawn from ICLR 2027

Beyond a Single Surrogate: Consistency-Weighted Alignment for Instruction-Free Multimodal Extension

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Extending a frozen LLM to a new modality can be reduced to training a lightweight projector, provided the alignment target is the LLM's own response distribution rather than external annotation; recent instruction-free alignment-only methods realize this with a caption as a semantic surrogate, whose supervision captures semantics well but — as prior work itself acknowledges — may under-represent super-semantic attributes. We take that acknowledged limitation as our starting point: surrogate fidelity is currently neither measured per sample nor acted upon, so when a single caption fails to stand in for the signal, the projector is silently taught to reproduce content the encoder never carried. **CoSA** makes surrogate fidelity an explicit, per-sample quantity: it constructs multiple surrogates per sample at differing granularity and viewpoint — deliberately spanning the super-semantic attributes a single caption tends to omit — elicits a response from the same frozen LLM for each, and uses the disagreement among these responses as a fidelity estimate, weighting rather than filtering samples so low-fidelity examples contribute proportionally instead of injecting unbacked supervision. It converts the previously implicit anti-shortcut argument into an **explicit constraint plus a measurable diagnostic** — a lightweight reverse probe reads surrogate-relevant attributes back out of the projected representation, while a modality-ablated control set quantifies how much accuracy survives when the signal is removed — and exploits the fidelity estimate for surrogate-aware active sampling, directing the generation budget where surrogate coverage is weakest. On MMAU-Pro and four audio-language benchmarks, CoSA improves over single-surrogate instruction-free alignment by 2.6 points while keeping the encoder and LLM fully frozen and using 42% of the alignment data; we find the previously reported saturation is not purely an encoder-capacity ceiling — over a third of the gap on closed-set tasks is recovered by reweighting alone — and the reverse probe reduces language-prior shortcut behavior, measured as a 4.8-point accuracy drop on audio-absent control inputs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal alignment, instruction-free extension, audio-language models, surrogate fidelity, data curation, frozen LLM

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 5 reviews, above 0% of the others.
Reviewer_cV5TOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes CoSA to align audio with a frozen LLM using multiple textual descriptions. It trains a projector using responses generated from these descriptions, weights samples by response consistency, and adds an auxiliary probe to preserve audio attributes.

Critical strengths
  1. Motivation: Building LLMs that understand audio is an important research topic (as covered in Audio LLM, speech-aware LLM literature). Many methods use captions to align audio with pre-trained LLMs, but a single caption may miss important details. Like the authors try to motivate in the introduction, a caption describing a door closing may not mention footsteps before it or someone speaking at the same time. I think using multiple descriptions covering different "views" is a reasonable direction.
Critical weaknesses
  1. Writing. I found the paper difficult to follow. For example, the introduction should explain more clearly what a "surrogate" is and how it is used in training. A clearer overview of the training process and the role of each component before introducing the equations would help readers understand the method.

  2. Missing details and experiments. The paper does not clearly explain how the different views are generated. What's the extra cost of CoSA compared to other standard methods? The paper also needs to cover more backbones and more comprehensive experiments including quantitative studies.

Questions for authors

None

Reviewer_89YhOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CoSA for instruction-free audio-language alignment through multi-view semantic surrogates and consistency-weighted projector training. It additionally introduces a training-only reverse probe to constrain the recoverability of acoustic attributes and an active acquisition strategy to supplement surrogate coverage. With the audio encoder and language model frozen, the paper reports an average improvement of 2.6 points across four audio-understanding benchmarks and two encoder settings, while using 42% of the alignment samples used by the single-surrogate baseline.

Critical strengths

1. The paper addresses a meaningful supervision problem in audio-language alignment. A single caption can omit important acoustic information, including temporal order, source relationships, and delivery characteristics, thereby limiting what the model can learn. Expanding coverage through multiple descriptive views and adjusting supervision according to reliability are worthwhile directions. The frozen-backbone setting with a lightweight trainable interface also has potential practical value.

2. Several aspects of the formulation and its conceptual boundaries are carefully stated. The authors compare predictive distributions on shared response prefixes and explicitly distinguish a mixture of sequence distributions from a simple average of token-level conditionals. They also acknowledge that consistency does not establish acoustic truth and that recovering an attribute with a probe does not imply that the language model uses it. These distinctions help clarify the method’s assumptions and limitations.

Critical weaknesses

1. The audio-ablation experiment does not establish that the reverse probe increases the model’s use of audio evidence. In Table 2, adding the probe leaves audio-conditioned accuracy unchanged at 57.0, while audio-absent accuracy increases from 47.3 to 52.2. The gap therefore shrinks from 9.7 to 4.8 because performance without audio improves, not because performance with audio improves. This does not substantiate the claim in Section 5.3 that the probe reduces language-prior shortcutting.

The audio-absent results also require clarification. Under the protocol in Section 4.5, zero prefixes bypass the projector, the probe is not used at inference, and the language model remains frozen. If both variants use identical inputs, decoding procedures, and scoring rules, their expected audio-absent accuracy should not systematically differ as a result of probe training. The authors should clarify whether the difference between 47.3 and 52.2 reflects sampling variability or unreported configuration differences. Moreover, zero prefixes may introduce out-of-distribution effects, so the ablation gap alone cannot establish more faithful use of audio.

2. The implementation and ablation of consistency weighting are insufficiently specified to establish its independent contribution. Table 3 treats consistency reweighting on a single caption as a separate setting, but does not distinguish the number of views used to compute the weights from those used to construct the training targets. If each sample truly has only one view, the stated fallback assigns the same fixed weight to every sample, which cancels after normalization and produces no reweighting effect. If multiple views are used to compute weights while only one caption supplies the training target, this should be stated explicitly. The weighting expressions in Equation (5) and the experimental section also need to be reconciled.

Necessary details concerning the training corpus, selection of the 42% subset, probe attribute labels, and constraint implementation remain unclear. The paper also lacks a clearly reported comparison between uniform and consistency-based weighting under fixed multi-view supervision. Consequently, it is difficult to determine whether the gains arise from richer supervision, weighting, or other training choices. Direct validation that the consistency score reflects the acoustic reliability of the supervision is also missing.

Questions for authors

1. Why do the two CoSA variants obtain different audio-absent scores in Table 2, and how do these results support the anti-shortcut claim? Please clarify whether both variants use the same frozen language model, zero-prefix construction, prompt template, decoding procedure, and scoring rules, and provide repeated evaluations or uncertainty estimates. After establishing the source of the difference, please explain what additional evidence supports increased use of audio, such as matched shuffled-audio controls.

2. What exactly changes in the “consistency reweighting only” setting in Table 3? Please separately specify the number of views used to compute weights and construct training targets, together with the actual weighting formula. Please also provide the necessary details of the training data and subset selection, probe supervision, and constraint implementation. Comparing uniform and consistency-based weighting with the multi-view supervision and training budget held fixed would more directly establish whether the weighting mechanism itself is effective.

Reviewer_y7BHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CoSA, an extension of instruction-free alignment-only framework that keeps the audio encoder and LLM frozen. It constructs multiple textual surrogates per audio sample, measures disagreement between their LLM response distributions, and uses this score to weight alignment supervision. It also introduces a training-only attribute probe and an active surrogate acquisition policy. The paper reports an average improvement of 2.6 points across four benchmarks using 42% of the original audio samples.

Critical strengths
  1. The paper addresses a meaningful limitation of caption-based alignment. A single caption can omit temporal, paralinguistic, or relational information. Exploring richer supervision while retaining a frozen backbone is a reasonable research direction.

  2. It compares response distributions on shared prefixes, thus avoiding confounding different generation histories.

  3. The method also explicitly recognizes that agreement does not guarantee acoustic correctness and that probe accuracy does not prove the LLM uses the recovered information.

Critical weaknesses
  1. Section 4.5 replaces projected audio vectors with zeros, bypassing the projector. With the same frozen LLM and matched inputs, different projector checkpoints should produce the same null-input distribution. However, Table 2 reports accuracies of 42.5%, 47.3% and 52.2%. Furthermore, the smaller ablation drop does not establish reduced shortcutting: adding the probe leaves intact accuracy unchanged and only increases null accuracy.

  2. The paper does not identify the actual acoustic source used to construct richer surrogates or specify the probe's attribute labels and their extraction.

  3. Using fewer unique audio samples does not establish lower supervision or compute cost when each sample requires multiple surrogate generations and distribution comparisons.

  4. The paper explicitly builds on [1] and uses its published results as baselines, but the corresponding bibliographic reference is missing.

[1] Zhou, X., Shao, Y., Li, J., & Yu, D. (2026). "Alignment is all you need: Instruction-free training for general audio-language models". arXiv. https://arxiv.org/abs/2608.18132

Questions for authors
  1. What differs between the three zero-prefix evaluations in Table 2? Please explain how projector training changes predictions after its output has been replaced entirely by zeros.

  2. Which models or annotations supply the acoustic evidence for surrogate construction and probe labels?

Reviewer_Nu5COpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CoSA, a method for audio-language alignment that does not require instruction fine-tuning. Unlike traditional approaches that rely on a single audio description, CoSA utilizes multiple descriptions from diverse perspectives and adjusts training weights based on their consistency. Furthermore, the method incorporates an reverse probe and an active sampling strategy to enhance alignment performance. Experiments conducted across four audio benchmarks demonstrate that CoSA achieves an average improvement of 2.6 percentage points over the single-surrogate baseline, while utilizing only 42% of the alignment data.

Critical strengths
  1. The paper addresses practical challenges in instruction-free audio-language alignment. Since a single audio description may omit important information, leveraging semantic surrogates from multiple perspectives to provide richer supervision is a reasonable approach.
  2. The proposed method achieves consistent improvements across two different audio encoders.
Critical weaknesses
  1. The interpretation of some experimental results is not sufficiently supported. In Table 2, the gap between the intact and audio-absent conditions decreases from 9.7 to 4.8 points after introducing the reverse-probe constraint. However, this is primarily due to an increase in audio-absent accuracy from 47.3 to 52.2, while intact accuracy remains unchanged at 57.0. It is unclear how this result supports the claim that the reverse-probe constraint reduces reliance on language-prior shortcuts.
  2. The experimental evaluation is insufficient. Although the paper presents some ablation studies, it does not adequately verify the independent contributions of the reverse-probe constraint and fidelity-aware active sampling across the evaluated benchmarks. Furthermore, since CoSA requires generating multiple semantic surrogates for each sample, the additional surrogate-generation cost should also be considered when discussing data efficiency.
Questions for authors
  1. In Table 2, why does adding the reverse probe increase audio-absent accuracy while leaving intact accuracy unchanged? How does this result demonstrate reduced reliance on language-prior shortcuts?
  2. In Table 3, how are the consistency weights computed in the single-caption setting?
  3. How are the surrogate descriptions from different viewpoints generated and verified? It would be helpful if the authors could provide the exact prompts and some actual generated examples to show whether these descriptions introduce new acoustic information.
  4. Additional ablation studies would help strengthen the paper's analysis.
  5. The paper appears to be missing the AI Use Statement section.
Reviewer_6jyQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CoSA for instruction-free audio-language alignment with a frozen audio encoder and frozen LLM. It constructs multiple descriptive surrogates per audio sample, measures disagreement between their induced response distributions on shared prefixes, and uses this proxy to weight response supervision. A training-only reverse probe encourages recovery of supported attributes from the projected representation, while an acquisition policy allocates additional surrogate-generation effort. Table 1 reports approximately 2.6 percentage points of average improvement across MMAU, MMAR, MMSU, and MMAU-Pro using 42% of the baseline alignment data, for two encoders. However, the interpretation of the audio-ablated results and inconsistencies between the formal method and experimental setup prevent these results from establishing the main grounding and efficiency claims.

Critical strengths
  1. The paper addresses a consequential supervision problem: a fluent textual surrogate may omit acoustic information or provide claims unsupported by the signal. Sections 4.1–4.2 explicitly distinguish consistency from correctness and acknowledge both correlated errors and disagreement between complementary valid descriptions. This is a useful framing for evaluating instruction-free alignment.

  2. The formal treatment usefully distinguishes sequence mixtures from averages of token conditionals, compares teachers on shared response prefixes, and specifies global weight normalization. Together with the proposed probe and paired audio-ablation protocol, these provide a reasonably concrete framework for testing the hypothesis, although the reported experiments do not yet validate its central claims.

Critical weaknesses
  1. The anti-shortcut conclusion is not supported by Table 2. Adding the reverse probe leaves intact accuracy at 57.0 but increases audio-absent accuracy from 47.3 to 52.2; the intact-minus-null gap therefore decreases from 9.7 to 4.8. Section 5.3 interprets the smaller gap as evidence for the reverse-probe mechanism, but greater retention of performance without audio does not demonstrate reduced reliance on language priors. The 4.8 figure is an intact-minus-null difference, not a decrease in null-input accuracy caused by the probe. Section 4.5 itself cautions that such gaps do not establish grounding. This undermines a central claimed contribution and requires corrected interpretation and additional controlled evidence, rather than only a wording change.

  2. The experiments do not unambiguously instantiate the specified method or isolate its benefits. Equation (5) uses a positive floor and exponent, whereas Section 5.1 and Appendix A use F_i = 1 - D_i and call it strictly positive, although the stated range permits zero. Section 4.6 retains the original sample collection and acquires extra views, whereas Appendix A trains for two epochs on a 42% subset whose selection is not explained. The experimental surrogate-generation description also does not identify the audio-grounded source supporting fine acoustic and speaker/style attributes. The reported tables do not provide the matched-subset, uniform multi-view, and matched-generation-budget acquisition comparisons prescribed in Section 4.7, nor uncertainty across seeds. Consequently, neither the contribution of the individual mechanisms nor the overall cost advantage is established.

Questions for authors
  1. Clarification and correction: How should Table 2 be interpreted given that adding the reverse probe raises audio-absent accuracy from 47.3 to 52.2 at unchanged intact accuracy? Please clarify what the 4.8-point claim refers to and provide evidence that distinguishes improved acoustic grounding from null-prefix behavior, such as paired shuffled-audio controls with uncertainty estimates.

  2. Implementation clarification: Which weighting rule was actually used, and what were the values of epsilon and gamma? How was the 42% subset selected, and how does subset training relate to Section 4.6’s full-collection objective? What audio-conditioned source or annotations support the five experimental surrogate views and the reverse-probe attribute labels?

  3. Evidence needed to assess attribution and efficiency: Can the authors provide comparisons on the same audio subset between single-surrogate supervision, uniform multi-surrogate supervision, and consistency weighting, as well as uniform versus active acquisition at matched generation budgets? Please include total generation/teacher-computation costs and variation across independently trained seeds.

  4. Result provenance clarification: Lines 111–113 state that the experimental sections reproduce the original paper’s setup and results, whereas Table 1 identifies new CoSA rows. Please distinguish cited baseline results from newly executed experiments and provide the corresponding configurations and reproducibility details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.