acceptodds
Under review as a conference paper at ICLR 2027

Auditing Optimization and Probabilistic Reliability in Frozen Vision-Language Representation Intervention

Abstract

We study whether optimization fit, probabilistic prediction quality, and discrete behavior move together under scheduled representation intervention in a frozen vision-language model. PAAF organizes the analysis into a deterministic Formation Schedule, normalized token-wise Allocation, and the Action that applies the resulting vector field. We compare the held-out training objective with Accuracy, ECE, Brier score, and NLL. In paired-coordinate KnG evaluations ( design), gain scheduling lowers the held-out training objective by while increasing NLL by and Brier score by . A controlled Allocation perturbation changes internal readouts but changes only 3 of 500 decisions. These results show that optimization fit, probabilistic prediction quality, and decision sensitivity are distinct properties under the evaluated intervention configuration. The practical implication is that representation-intervention schedules should be selected using held-out proper scores and decision-level checks, not the training objective alone.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Representation intervention, frozen vision-language models, probabilistic reliability, calibration, optimization schedules, token-wise allocation, proper scoring rules

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZAfqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

With a framework called PAAF, the paper tests whether lower training loss also brings better probabilities and answers under scheduled intervention in a frozen SmolVLM. Gain scheduling lowers held-out loss but worsens Q-Bench2 Brier score and NLL at similar accuracy, and an allocation shuffle changes 3 of 500 answers.

Critical strengths
  1. The main comparison is carefully controlled, so I believe the narrow result. The 2 × 2 design varies the gain and temperature schedules separately, with matched seeds and time averages checked in logs.

  2. The paper is careful with its claims. It reports Brier and NLL next to accuracy, bases its main statistics on seeds rather than questions, and states its limits openly.

Critical weaknesses
  1. The main lesson seems to be known already, and this is the main reason for my rating. Guo et al. [1], cited by the paper, showed that loss and calibration can move apart, as the paper's own LoReFT sweep also shows (Table 12). PAAF itself mostly puts existing parts (VEFT [2], KnG [3], KSL [4]) into a new frame, so the framework adds little on its own.

  2. The evidence seems too narrow for the paper's general advice. The main reliability results come from one backbone, one layer, one schedule shape, one benchmark family (Q-Bench), and three seeds. Brier and NLL are also shown only for Paired-coordinate KnG, which the paper says reacts differently from the other two Actions (Section 4.5).

  3. A bigger gain at test time may explain the main result, which would weaken the lesson that matched averages are not enough. The gain ramps up during training, so the Gain-only model may be tested with a bigger gain than the Fixed model. The "fixed terminal" variant in Table 1 would check this, but its results are not shown.

  4. The loss and the proper scores are measured on different data, and the training data is not named. The held-out objective is token cross-entropy plus allocation regularizers on held-out training data, and the cross-entropy part is itself a form of NLL. So the result may show a gap between in-domain fit and transfer to Q-Bench2, not a split between fit and probability quality.

  5. The allocation test seems too weak to show that decision sensitivity is a separate property. The token shuffle barely moves the internal readouts (linear CKA 0.99996, routing JSD 0.00212), so few answer changes would be expected. The paper itself notes the test does not separate weak coupling from a perturbation that is too small.

  6. The audited setups bring little benefit over the frozen model, which limits the practical value of the audit. On Q-Bench (Q1), every reported VEFT-family setup is less accurate than zero-shot (Tables 4 and 14). On Q-Bench2, they beat zero-shot accuracy by under two points but have worse ECE.


References

[1] Chuan Guo et al. On calibration of modern neural networks. ICML, 2017.

[2] Noriaki Kawamae. One-shot multi-view visual conversational recommendation. KDD, 2025.

[3] Noriaki Kawamae. Multi level vision language adapter. ICDM, 2025.

[4] Noriaki Kawamae. Knowledge-aligned domain shift tuning for efficient adaptation in large language models. KDD, 2025.

Questions for authors

Questions:

  1. Which gain and temperature values do you use at test time? If the proper-score gap remains against the fixed-terminal variant, the schedule claim would be more convincing and my score could rise.
  2. What is the training data? If the gap remains when loss and proper scores use the same data, the central claim would be much stronger.
  3. Does post-hoc temperature scaling of the four answer logits, as in Guo et al. [1], remove the Brier and NLL gap? If not, the reliability cost would matter more in practice.

Suggestions:

  1. Try stronger allocation changes, such as uniform weights or removing vector fields, to see when answers start to move.
  2. Report Brier, NLL, and accuracy for all three Actions, and repeat the 2 × 2 test on a second backbone.
  3. Use fewer new terms, and fix the table numbers in Appendix B.3, which point to Tables 1 to 3 instead of 13 to 15.
Reviewer_5qkXOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether optimization fit, probabilistic prediction quality and discrete decisions move together when a frozen VLM (SmolVLM-Instruct) is adapted by scheduled representation intervention. It introduces PAAF (Policy–Allocation–Action framework): a deterministic Formation Schedule for the intervention gain αs, residual gain βs and allocation temperature τs; a token-wise softmax/sparsemax Allocation over K=5 learned vector fields; and an Action (VEFT, output-residual KnG or paired-coordinate KnG) that applies the field just before the frozen LM head. Validation: a 3-seed Action×Schedule factorial and a 2×2 gain-by-temperature factorization on Q-Bench/Q-Bench2 and VisIT-Bench. Main finding: gain scheduling lowers the held-out training objective by 0.0999 but worsens Q-Bench2 ECE (+0.062), Brier (+0.040) and NLL (+0.073) with unchanged accuracy (−0.0013); a cyclic Allocation shuffle changes internal routing but only 3/500 decisions. The authors conclude that intervention schedules should be selected with held-out proper scores and decision-level checks, not the training objective alone.

Critical strengths
  1. Careful matched design – fixed time-average controls (verified against per-step logs) and a 2×2 gain×temperature factorization with seed-blocked linear models isolate gain scheduling as the driver of both the objective drop and the proper-score degradation — consequential because the dissociation claim is attributed to one component rather than to the joint schedule.

  2. Proper scores (ECE, Brier, NLL) and correctness transitions are reported beside accuracy, with carefully scoped limitations — consequential because it shows that unchanged accuracy can hide a sizeable shift in confidence.

Critical weaknesses
  1. Limited novelty and an expected finding. The PAAF components (VEFT, KnG, KSL) come from prior work, and the paper explicitly does not claim novelty for the gated multi-branch design or the softmax/sparsemax comparison. The remaining message – a lower training loss does not imply better calibration – is well established. Moreover, the "held-out training objective" is token-level CE on answer+rationale data plus allocation regularizers, whereas the proper scores are computed on four-option-renormalized Q-Bench2 MCQ probabilities, so a divergence between two different tasks is not surprising.

  2. Narrow and weak empirical basis. All confirmatory evidence comes from one small frozen VLM, one intervention site (final text block 23) and three seeds (12–18 run-level observations), with many metrics tested and no multiplicity correction; the only Action-dependent reliability effect (Q1 ECE interaction, 95% CI [−0.181, −0.002]) barely excludes zero and is not replicated on Q2. LoReFT/PEFT baselines use different quantization and are only contextual. In Table 14 the VEFT-family variants are worse than zero-shot on Q1 accuracy (48.2–52.6 vs 56.8) and ECE (0.146–0.299 vs 0.052) and worse than LoReFT on Q2 — consequential because the studied intervention family does not yet give a useful operating point, which limits the practical value of the schedule-selection recommendation.

Questions for authors

Clarification 1. Does the ECE/Brier/NLL degradation from gain scheduling persist after post-hoc temperature scaling fitted on a held-out split? If one temperature removes it, the dissociation is a logit-scale effect of αs rather than a deeper reliability problem.

Clarification 2. The time-averages are matched, but can the effect of trajectory shape be separated from general path-dependence, e.g., with the fixed-initial, fixed-terminal and random variants listed in Table 1 but not reported?

Reviewer_YxNSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the relationship between optimization performance, probabilistic reliability, and decision behavior in frozen vision-language models. The authors introduce the Policy–Allocation–Action Framework (PAAF) to analyze representation interventions under different scheduling and allocation strategies. Experiments reveal that improved optimization objectives do not necessarily translate into better calibration or decision quality. The study further examines how allocation perturbations affect internal representations and downstream predictions.

Critical strengths

1.The paper addresses an important issue by examining whether optimization improvements correspond to better calibration and decision behavior. This perspective complements conventional adaptation studies that primarily emphasize task accuracy.

2.The authors employ matched scheduling comparisons, factorial experiments, and multiple random seeds. The evaluation considers accuracy, ECE, Brier score, NLL, and internal allocation responses, providing a reasonably systematic analysis of intervention behavior.

Critical weaknesses

1.The held-out training objective includes token-level cross-entropy and allocation regularization, whereas probabilistic reliability is evaluated using downstream multiple-choice predictions. These objectives are not directly equivalent, making their divergence less surprising. Additional controlled comparisons using aligned training and evaluation objectives are needed to establish the underlying mechanism.

2.The main experiments rely on a single frozen SmolVLM backbone, with interventions primarily applied near the final language-model head. Moreover, the confirmatory analyses use only three training seeds. Evaluations across additional VLM architectures, model scales, and intervention locations would strengthen the generality of the conclusions.

3.Although gain scheduling is associated with increased confidence and worse proper scores, the paper does not clearly identify whether these effects arise from logit scaling, representation changes, allocation dynamics, or optimization regularization. Comparisons with simple post-hoc temperature scaling and additional logit-level analyses would help establish whether the observed degradation reflects a substantive representation-level problem.

Questions for authors

1.What are the key methodological contributions of PAAF beyond reorganizing existing representation intervention and scheduling mechanisms?

2.Could the authors evaluate whether the optimization–reliability discrepancy persists when training and evaluation objectives are more closely aligned?

3.Can simple post-hoc calibration methods, such as temperature scaling, mitigate the observed reliability degradation without changing prediction accuracy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.