acceptodds
Withdrawn from ICLR 2027

Follow Me If You Can:Auditing Understanding-to-Generation Transfer

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding-only training can change image generation through shared parameters, but the resulting scores support different claims about transfer. We develop an audit framework separating source validity, target influence, and attribute correspondence. Fixed-checkpoint image and instruction controls test the source gain; paired trained and frozen states measure generation influence; conditioning swaps probe the requested value expressed by the update. We characterize a blind spot of norm-matched swaps: changes along a shared direction yield identical interventions even when their amplitudes depend on the prompt. Across six unified architectures, counting and spatial experiments reveal different answer–generation response patterns. In a focused counting audit, a percentage-point answer gain becomes under digit requests at fixed weights. A standard low-rank adaptation (LoRA) reference also improves answers without a consistent generation-score gain. Joint readouts distinguish recipient improvement from donor-value following: a percentage-point recipient gain accompanies a donor change. Mixed same-budget prompt-contrast calibration bounds inference from non-following. The audit ties each interpretation of transfer to the comparison that supports it.

Primary area:
interpretability and explainable AI
Keywords:
unified multimodal models, cross-task transfer, capability evaluation, controlled interventions, image generation, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Fb98OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how to assess understanding-to-generation transfer in unified multimodal models. Rather than treating changes in understanding and generation scores as direct evidence of capability transfer, the authors decompose the claim into three components: source validity, i.e., whether the observed understanding gain reflects the intended visual attribute; target influence, i.e., whether the understanding-trained update causally changes generation; and attribute correspondence, i.e., whether the generation change follows the specific attribute value associated with the update.
To audit these components, the paper introduces removable square-core updates trained only on understanding tasks, fixed-checkpoint image and instruction controls, paired full-versus-frozen generation comparisons with matched randomness, and donor-replacement interventions that swap norm-matched conditioning changes between prompts. The study evaluates counting and spatial-relation tasks across six unified multimodal architectures, with an additional standard LoRA reference. The experiments show that source-side answer gains can be highly instruction-dependent, understanding and generation changes exhibit heterogeneous and sometimes opposite-sign relationships across models, and donor replacement does not consistently induce donor-value following. The paper also analyzes limitations of norm-matched swaps and uses prompt-contrast calibration to qualify the interpretation of negative intervention results.

Critical strengths
  1. The paper addresses an important and timely question for unified multimodal models. A central motivation for unified architectures is that understanding and generation may share representations or capabilities, but it is nontrivial to determine whether improvements on one side genuinely transfer to the other. The paper usefully separates architectural sharing from stronger claims about capability transfer, and I found this framing conceptually valuable.
  2. The audit framework is carefully designed and empirically broad. The decomposition into source validity, target influence, and attribute correspondence provides a clear progression of increasingly stronger claims, and the corresponding controls—fixed-checkpoint instruction/image interventions, paired full-vs.-frozen generation, and donor replacement with dual readouts—are thoughtful attempts to rule out alternative explanations. The evaluation also covers six unified architectures, two attribute families (counting and spatial relations), and an additional LoRA reference, which makes the empirical study substantially more informative than a single-model case study.
Critical weaknesses
  1. The audit framework is stronger at ruling out overly strong transfer claims than at positively establishing transfer. Many results remain “unresolved” because of prompt sensitivity, metric ambiguity, norm-matching blind spots, or carrier limitations. The paper would be substantially stronger with a convincing positive-control setting showing that the full framework can successfully detect known attribute-level transfer.
  2. The scope of the evidence is somewhat narrow relative to the paper’s broad framing. The instruction-control experiment establishes that the measured source gain is prompt-dependent, but not necessarily that the original gain is non-visual, and the paper does not show how unusual this sensitivity is compared with standard VQA/adaptation settings. More broadly, the substantive analysis is limited to counting and spatial relations with only a few independent training runs, which weakens broad generalization to understanding-to-generation transfer as a whole.
Questions for authors
  1. Broader task coverage. The current empirical study mainly focuses on counting and spatial relations. Could the authors provide evidence on at least one qualitatively different type of capability (e.g., semantic recognition, factual/visual knowledge, or reasoning-related attributes) to support that the proposed audit framework captures a broader phenomenon rather than failure modes specific to these two relatively low-level compositional tasks? This would materially affect how broadly I interpret the paper’s conclusions.
  2. What can the framework establish positively? Could the authors clarify what empirical outcome would constitute a positive conclusion of successful understanding-to-generation transfer under their framework? At present, many analyses are effective at showing that a stronger interpretation is not justified, but it is less clear what the framework can positively establish beyond target influence. A concrete decision criterion or worked positive example would help clarify the scientific claim of the framework.
  3. Positive control. Could the authors provide a convincing positive-control setting in which attribute-level transfer is known to exist by construction, and then demonstrate that the proposed source-validity / target-influence / attribute-correspondence pipeline successfully recovers it? Given the acknowledged blind spots of norm matching and carrier choice, such a control would help distinguish genuine absence of evidence from limited sensitivity of the audit.

If the authors can satisfactorily address these concerns in the rebuttal, I would consider increasing my rating.

Reviewer_gV5AOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors provide a new view about how to interpret understanding-to-generation transfer in UMMs. Specifically, the paper proposes an auditing framework that separates source validity, target influence, and attribute correspondence. Experiments across 6 UMMs on counting and spatial-relation tasks demonstrates that there may exists the situation that understanding gains can depend strongly on answer instructions, even understanding and generation scores do not essentially improve together.

Critical strengths
  • This paper proposes a useful framework for interpreting understanding-to-generation transfer. The auditing framework aims to separates source validity, target influence, and attribute correspondence.
  • The paper also conducts extensive experimental controls and results. The authors evaluate 6 UMMs on counting and spatial-relation tasks.
Critical weaknesses
  • The construct validity of donor replacement in the third experiment (Attribute Correspondence) is insufficiently established. The training-induced representation differences do not necessarily encode the intended attributes. It is therefore unclear whether donor replacement can reliably measure attribute correspondence.

  • The experimental evaluation is quite limited in task diversity. Although the authors evaluate six UMMs, the experiments only cover two tasks (counting and spatial-relation). It remains unclear whether the proposed auditing framework generalizes to other visual capabilities.

Questions for authors
  1. How do the authors validate/proof that donor replacement reliably measures attribute correspondence?

  2. Can the authors provide evidence that the framework generalizes beyond counting and spatial-relation tasks?

Reviewer_152JOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors study whether VQA fine-tuning on unified multimodal models actually improves image generation. They test six models on counting and spatial tasks using a removable square-core adapter (plus a LoRA baseline) and a three-step audit: VQA controls, adapter removal at generation, and cross-prompt conditioning swaps.

Critical strengths
  • The prompt-control test on BLIP3-o (Tab. 1, Fig. 3) is insightful because changing only the answer prompt makes the counting gain drop to near zero.
  • The authors report negative results honestly, fixing shown examples beforehand and keeping failed control runs, which makes their results credible.
Critical weaknesses
  • Poor writing and paper organization. The paper is very hard to parse, and I honestly needed help from AI just to understand what it is doing. Even after reading the abstract and introduction, I could not tell what the task is because the authors use custom terms before defining them (carrier, cohort) or never define them, as if readers already know everything. I also do not see how Eq. 1 is obtained: they state it is removable and bounded, yet never justify the tanh form, the constants, or how "high-energy" rows and columns are chosen (and omit its scaling factor in Fig. 2a). Every table cited in Section 6 is also put in the appendix.
  • No positive control for the audit. The square-core adapter fails to improve counting on four of six models (Tab. 2), and the LoRA baseline has more gains on blank gray and permuted images than on real images (Tab. 6), showing strong answer-prior bias [2] that leaves visual counting unresolved (as the authors themselves conclude in Section 6). As they also note, claiming failed transfer requires a model that actually learned the source skill; without a positive control such as standard LoRA on DiMOO [1], we cannot tell if this audit works or just rejects everything.
  • Missing error bars and unclear run counts. They mention saving bootstrap intervals in Appendix A.21, yet report none in any table. Run counts for DiMOO, Janus-Pro, and Show-o2 in Tab. 3 are also unclear: one appendix line suggests four runs, whereas every score changes in steps of one prompt out of 128, suggesting a single run.

[1]. Transferability Between Understanding and Generation in Unified Multimodal Models. Jiwon Kang, et. al. ECCV 2026.

[2]. Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering. Aishwarya Agrawal, et. al. CVPR 2018.

Questions for authors
  1. Can you clarify Section 1 and Section 3.1, define custom terms at first use, and justify Eq. 1 and its constants?
  2. How does the audit behave on a known positive-transfer case such as standard LoRA on DiMOO [1]?
  3. How many runs were used for DiMOO, Janus-Pro, and Show-o2 in Tab. 3, and can you show the bootstrap intervals and per-run numbers cited in Section 6?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.