acceptodds
Under review as a conference paper at ICLR 2027

Tracing Prompt-Conditioned Control from Allocation to Realized Actuation in Frozen VLMs

Abstract

We analyze inference-time control authority through the measurable sequence (). In one paired checkpoint, Dynamic-to-Uniform composition yields equal measured accuracies. Across five training seeds, the replacement changes the Prompt-derived state while retaining a zero-median allocation response and unchanged dominant allocation indices. Direct allocation and Prompt-input replacements produce downstream continuous responses at different controller coordinates, revealing a recurrent dissociation between conditioning variation, continuous allocation, realized actuation, and decision-boundary crossing.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Inference-time control, Vision-Language Models (VLMs), control authority, representation learning, stage-wise evaluation, prompt conditioning, model interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6MFSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper asks whether a learned prompt-composition signal in a representation-level adapter actually influences a frozen VLM's output at inference time. The audited adapter is right before the LM head. The paper breaks the path from prompt to prediction into stages and measures the change at each stage under several interventions. In experiments, Dynamic-to-Uniform changes the conditioning state, but the router weights and accuracy barely change.

Critical strengths

None.

Critical weaknesses
  • The main result is an almost null effect presented as a finding, and the experiments are too limited to support it. The experiments are small and narrow (just 900 training and 100 evaluation examples), Table 1 is a single seed, and there's one small VLM. Also, the evaluated model predicts option A for every example (Sec 5.6), so the "no prediction change" results are not meaningful. "Norm-matched random directions" are mentioned as a control in Section 6, but no results are reported.
  • The writing and presentation of this paper are really hard to follow, so I could not verify the claims. The paper contradicts itself about prompt tuning (Sec 5.1 vs Appendix A.1.1), notation is used before it's defined (see questions), components and configurations are never explained (VEFT, KSL, KLAFT, PAAF, etc.), and Figure 1 does not match the experiments.
Questions for authors
  • Did the authors check Figure 1 before submitting it? Is this really the final version you want to present?
  • What is M,dm,K,dh,V,Y,Am,AwM, d_m, K, d_h, V, \mathcal{Y}, \mathcal{A}_m, \mathcal{A}_w in Section 3? Did the authors ever define the variables before using them?
  • This paper cites two anonymous papers under review. Can you provide more self-contained details of the two papers since they are not available to reviewers?
  • What exactly is "Dynamic-to-Uniform composition"? I guess it is defined in Section 4.1 but why the authors use it without explaining it in the abstract?
  • What is adyn(h)a_dyn(h) in line 148? Is it the a(h)a(h) defined in Eq.2?
  • In Section 5.6, all predctions select option A. Are the same accuracies in Table 1, just because the model always answers A? Does the result still hold with a model that does not always predict the same option?
  • Section 6 (line 477) says "norm-matched random directions provide an additional specificity control." but I don't think there are results for this... Where are the results?
  • Section 5.1 says input-levle prompt tuning is omitted but Apendix A.a.a says role-aware prompts are prepended to the input and trained. What is true?
Reviewer_XcrPOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how prompt-conditioned control signals propagate through a frozen VLM, from prompt composition and routing to intermediate representations and final predictions. Through interventions such as Dynamic-to-Uniform prompt replacement, direct allocation replacement, and prompt-input zeroing, the authors find that changes in prompt conditioning affect intermediate representations but have limited influence on downstream routing and final predictions.

Critical strengths
  1. Meaningful research question. The paper attempts to distinguish changes in intermediate representations from their actual influence on model behavior, which is relevant to understanding prompt-based adaptation.

  2. Multiple intervention analyses. The authors investigate different stages of the controller and validate some observations across multiple training seeds, providing useful diagnostic evidence.

Critical weaknesses
  1. Limited mechanistic insight. The paper mainly measures differences between intermediate variables before and after interventions, without clearly identifying the underlying mechanisms responsible for weak downstream responses. The analysis therefore remains largely descriptive rather than providing a deeper mechanistic explanation.

  2. The analysis builds on existing prompt-routing and activation-intervention techniques. The findings are largely restricted to a specific VLM and controller configuration, with insufficient evidence of novel insights or broader applicability beyond the evaluated setting.

  3. Presentation issues. Figure 1 contains noticeable alignment and layout problems, making the main architecture difficult to follow.

Questions for authors
  1. The paper observes that changes in prompt conditioning have limited effects on downstream representations and predictions. What specific mechanisms cause this attenuation? Could the authors conduct targeted interventions to identify the primary causes and verify whether addressing them restores meaningful prompt control over model outputs?

  2. The current analysis focuses on a specific SmolVLM and controller configuration. Does the observed discrepancy between intermediate representation changes and final prediction changes also occur in other prompt-based adaptation methods or VLMs of different scales? Which findings do the authors consider generalizable beyond the evaluated architecture?

Reviewer_bGjLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how prompt-conditioned control signals propagate through different stages of a frozen Vision-Language Model (VLM). The authors focus on whether changes in learned prompt composition lead to measurable differences in downstream allocation, representation updates, and final predictions.

To study this problem, the authors analyze a VEFT/KSL-based controller and introduce a stage-wise intervention framework that separates prompt composition, conditioning states, continuous allocation, realized actuation, and decision-level responses. The analysis includes Dynamic-to-Uniform prompt replacement, direct allocation replacement, fixed-allocation replay, and prompt-input zeroing.

Experiments on SmolVLM-Instruct show that changing prompt composition modifies the conditioning states but produces limited changes in downstream allocation. Across five training seeds, the allocation response has a zero median, with no changes in dominant routing indices. Direct allocation replacement produces larger downstream responses, while prompt-input zeroing changes continuous representations and output probabilities without changing the evaluated top-1 decisions. The authors argue that conditioning variation, allocation, realized actuation, and decision-boundary crossing should be evaluated separately.

Critical strengths

S1: Interesting Research Question and Stage-Wise Analysis.

The paper studies an interesting question about whether learned prompt conditioning actually influences downstream model behavior. The distinction between changes in conditioning states, continuous allocation, realized actuation, and final decisions is meaningful. The stage-wise analysis provides a useful perspective for understanding the internal behavior of prompt-conditioned controllers.

S2: Controlled Interventions and Experimental Analysis.

The authors design several interventions at different stages of the controller, allowing them to examine downstream responses under different conditions. The five-seed analysis provides evidence that the weak allocation response under Dynamic-to-Uniform replacement is recurrent within the evaluated configuration. The direct allocation replacement and fixed-allocation replay experiments also provide useful complementary evidence about the controller's downstream behavior.

Critical weaknesses

W1: Problem Formulation and Presentation Clarity.

The paper does not clearly explain the core problem and the motivation behind the proposed analysis, making its main contribution difficult to identify. The presentation also lacks a clear and intuitive illustration of the overall framework. In particular, Figure 1 contains overlapping elements, and the dense notation makes the control flow difficult to follow. A clearer problem statement, a better-organized figure, and simpler explanations would improve readability.

W2: Limited Experimental Evaluation.

The experiments are mainly conducted on a single compact VLM with a relatively small evaluation set. Although five training seeds are considered, the external performance comparison is primarily reported for one checkpoint. Moreover, Section 5.6 shows that all evaluated examples predict option A, which raises concerns about whether the absence of decision changes reflects the proposed mechanism or prediction bias. More diverse evaluation settings would strengthen the conclusions.

W3: Limited Explanation of the Observed Control Behavior.

The paper aims to understand whether learned prompt conditioning influences downstream control, but the current interventions do not fully explain why the observed responses are weak. In particular, Dynamic-to-Uniform replacement examines only one intervention direction, while direct allocation replacement operates at a different perturbation scale. Additional controlled comparisons would help distinguish weak prompt influence from limited sensitivity to the chosen intervention.

Questions for authors

Q1: Problem Formulation and Presentation.

Could the authors clarify the main research problem and how the proposed analysis addresses it? It would also be helpful to improve Figure 1 by removing overlapping elements, reducing notation density, and making the relationships between control stages and interventions easier to follow.

Q2: Experimental Generalizability.

Could the authors clarify whether the findings generalize to other models or evaluation settings? In particular, does the absence of decision changes still hold when the model produces more diverse predictions rather than consistently selecting option A?

Q3: Intervention Design.

Could the authors explain why Dynamic-to-Uniform replacement is sufficient to investigate prompt-conditioned control? Would magnitude-matched or alternative prompt-state interventions help clarify the source of the weak downstream responses?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.