acceptodds
Under review as a conference paper at ICLR 2027

Candidate-Conditioned Alignment for Stable Test-Time Adaptation in Cross-Domain VQA

Abstract

Vision-language models (VLMs) have achieved remarkable progress in visual question answering (VQA), yet their performance often degrades under domain shifts. Test-time domain adaptation (TTDA) offers an attractive solution by adapting models directly on unlabeled target data during inference. Existing TTDA methods, however, typically optimize prediction confidence over a shared answer space, an assumption that is inconsistent with the question-dependent prediction structure of VQA. As a result, optimization may progressively become biased toward dominant answers and produce question-insensitive predictions under domain shift. To address this issue, we propose a Candidate-Conditioned Alignment framework for VQA test-time domain adaptation. Instead of adapting over the global answer space, our method performs adaptation within instance-specific candidate sets through a local energy-based image-text alignment objective, enabling stable optimization while preserving question-conditioned reasoning. A lightweight text adapter further supports efficient adaptation without updating the pretrained backbone. Extensive experiments on six vision-language models and six cross-domain VQA benchmarks demonstrate consistent improvements across diverse domains, achieving an average accuracy gain of 18.8% over direct candidate-based inference under the same candidate setting.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Visual Question Answering, Test-Time Adaptation, Vision-Language Models, Conditional Reasoning, Domain Shift

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fGd1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets test-time domain adaptation (TTDA) for cross-domain visual question answering (VQA). Existing entropy-minimization based TTDA approaches suffer from structural collapse on VQA tasks: models converge toward dominant frequent answers and lose question-conditioned reasoning capability. The authors trace this failure to optimization over a globally shared answer vocabulary, while VQA naturally features instance-specific candidate answer sets per input sample. They propose a candidate-conditioned alignment framework, which performs local energy-based image-text alignment loss inside each per-sample candidate set. Only a lightweight text adapter is updated, keeping the pre-trained backbone frozen. Extensive experiments are conducted across six vision-language models and six cross-domain VQA benchmarks. Comparisons against general TTDA baselines and specialized VQA-TTDA methods, together with ablation and diagnostic studies, are provided to validate the proposal.

Critical strengths

Strength 1: This paper systematically dissects the internal mechanism of structural collapse when entropy‑minimization based TTA is applied to VQA. Rather than only empirically reproducing this failure case, the authors provide formal propositions to explain why constant predictors become stable minimizers under shared‑parameter entropy optimization.

Strength 2: The evaluations include comparisons against both general TTA baselines and existing VQA‑specific TTA methods. Results are reported for the offline batch setting as well as the more difficult online single‑step scenario. Ablation studies are also carried out to verify loss components, adapter design options, and hyper‑parameter sensitivity.

Critical weaknesses

Weakness 1: One central prerequisite for this pipeline is access to instance‑level candidate sets during inference. All experiments in the paper rely on oracle candidate sets provided by benchmarks. This strong assumption, however, is not sufficiently emphasized within the main text. In real‑world open‑world VQA, pre‑built candidate sets are not readily available. External retrieval or generative components would be required instead, inevitably introducing additional approximation error. This represents a notable practical deployment constraint and considerably limits the approach’s applicable scope.

Weakness 2: CLIP exhibits an unusually large accuracy jump from 0.0588% up to 91.1% on EarthVQA. While diagnostic plots in both the main paper and appendix attribute this sharp gain to structural mismatch between pre‑trained similarity geometry and the dataset’s answer space, controlled cross‑dataset experiments are still absent. We cannot cleanly disentangle how much improvement stems from the alignment objective itself versus fixing severe dataset‑specific pre‑training bias unique to EarthVQA. This outlying result raises legitimate concerns about the generalizability of the reported performance gains.

Questions for authors

Q1 (Sec 3.3, Appendix B): You only update the gated adapter placed on the text branch. Drawing on the theory laid out in Appendix B, could you explain why image‑branch updates yield such limited gains? Is the instance‑wise gradient‑decoupling property simply invalid for the image encoder, or are there alternative feature‑level explanations?

Q2 (Sec 4.7, Table 7, Appendix C.5): Under online single‑step adaptation, your approach suffers noticeable performance drops and its advantages shrink compared to the offline setting. The appendix documents this behaviour but offers little root‑cause analysis. Is this performance degradation mainly caused by noisy pseudo‑positive labels, insufficient update steps per sample, or limited optimization room within each candidate set?

Q3 (Sec 3.2, Appendix D.2): All of your experiments rely on oracle candidate sets which guarantee that ground‑truth answers are included. How would your framework behave if the ground‑truth answer is not retrieved during candidate‑set construction at test time?

Q4 (Sec 3.2, Appendix C.4 Table 12): Your main experiments fix K=4. Larger candidate‑set K hurts prediction accuracy while also increasing computational cost. Could you provide empirical or theoretical guidance on picking an appropriate K to balance accuracy and compute cost for real‑world deployment?

Q5 (Sec 4.8, Table 8, Appendix D.5): Your gated residual adapter outperforms LoRA. While the appendix describes its architecture, comparisons on gradient update dynamics are missing. Could you clarify the key differences in gradient update behaviour between your gated residual adapter and LoRA under test‑time adaptation, to help explain the observed performance gap?

Q6 (Conclusion, Appendix D.6): Appendix D.6 mentions extending towards fully open‑ended generative VQA as future work. Could you briefly outline what you see as the most central technical obstacles when porting the core candidate‑conditioned alignment idea over to generative VQA systems such as LLaVA‑style models?

1 comment
AuthorsOpenReview ↗

Response to Reviewer fGd1: Candidate Sets and Adapter Design

(1) Text versus image adaptation. Table 8 reports CLIP image-adapter accuracy 44.8%/42.7%/53.6%, versus text-adapter 53.7%/57.4%/59.6% on VQA-RAD/SLAKE/OVQA. A text residual changes candidate representations while retaining visual embeddings; an image adapter changes the visual representation used in the comparison. The accuracy preference concerns these evaluated configurations. Text Adapter/LoRA drift is not used as evidence for an image/text gradient mechanism.

(2) Single-pass degradation. Table 7 reports CLIP EarthVQA 91.1% offline and 10.6% online, the latter below the 25% uniform four-choice expectation. C.1 describes offline batch 4 with repeated exposure and online batch 1 without replay. Table 26 separates fixed budgets: CLIP reaches 32.0%/65.0%/88.5%/91.1% at epochs 1/2/5/10. More offline exposure accompanies larger gains, but update count is not isolated from batch, order, selection and prediction timing. The executed original checkpoint rule and online update/predict order still require documentation.

(3) Missing reference. Section 3.2 makes oracle inclusion explicit. Without it, a selector cannot output an absent answer. In Appendix E.7/Table 30, question retrieval at K=4/8/16 has recall 42.0%/56.0%/68.0% and conditional ranking accuracy 61.0%/56.0%/45.0%, giving end-to-end 25.6%/31.4%/30.6%. For percentage inputs, end-to-end accuracy equals recall times conditional accuracy divided by 100 and cannot exceed recall. Question-type filtering is reported separately; the exact retriever, answer pool and index are not specified sufficiently for reproduction.

(4) Choosing K. Table 12 evaluates K=4/8/16 for Open questions while Closed candidates remain binary. CLIP SLAKE accuracy decreases 57.4%/47.8%/40.2%; OVQA decreases 59.6%/53.1%/51.8% as distractors increase. Table 14 gives CLIP Ours latency 0.116 to 0.174 s/sample and DFN-5B 0.258 to 0.354 under that timing protocol. With non-oracle retrieval, coverage improves with K, but Table 30's best listed question-retrieval end-to-end score is at K=8. Ranking difficulty, coverage and cost therefore jointly determine K; a larger set is not uniformly better.

(5) Gated Adapter versus text LoRA. This comparison is separate from branch selection. Table 8 gives CLIP text-LoRA 44.4%/52.0%/55.1%. The gated adapter modifies frozen encoder-output features; LoRA updates selected internal linear maps through a low-rank parameterization. E.9/Table 33 reports SLAKE accuracy 52.0% versus 57.4%, feature drift 0.051 versus 0.025, and final gate 0.55. This is a structural and empirical comparison, not proof that smaller drift causes better optimization. Executed LoRA rank, targets and scaling still need to be specified.

(6) Generative extension. Appendix D.6 identifies candidate-generation coverage, answer equivalence, sequence/length calibration and propagation of generator errors as the main obstacles. Retrieval-then-ranking is a distinct evaluation from oracle ranking, and neither establishes unrestricted generative VQA performance.

Reviewer_V3rQOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a candidate-conditioned test-time adaptation framework for cross-domain visual question answering (VQA). Instead of minimizing prediction entropy over a shared answer space, the method performs image-text alignment within instance-specific candidate sets. It combines positive alignment, a hard-negative margin loss, and a feature-space consistency regularizer, while updating a lightweight text adapter and keeping the pretrained encoders frozen. Experiments on six vision-language models and six VQA benchmarks show improvements over direct candidate-based inference and several existing test-time adaptation methods.

Critical strengths
  • The paper identifies a meaningful limitation of conventional entropy-based adaptation for question-dependent VQA predictions.

  • The evaluation covers six VLMs and six cross-domain VQA benchmarks.

Critical weaknesses
  • The highest-scoring candidate is treated as a pseudo-positive and further reinforced by the alignment and margin losses. Incorrect initial predictions may therefore be amplified.

  • The reported CLIP accuracy on EarthVQA increases from 0.0588% to 91.1% under offline adaptation, but drops to 10.6% in the online single-step setting. Given the stated four-candidate evaluation protocol, where each candidate set contains one reference answer and three randomly sampled negatives, these results require clarification.

  • Open-answer tasks are converted into candidate-ranking problems, and all evaluated models predict by ranking candidate answers using image-text similarity. Consequently, the experiments do not establish whether the method generalizes to open-ended answer generation or modern generative VQA models.

  • The paper does not sufficiently quantify the adaptation time, inference latency, or memory consumption, making it difficult to assess its practical efficiency relative to existing TTA methods.

Questions for authors
  • How does the proposed objective prevent incorrect pseudo-labels from being progressively reinforced?

  • Can the authors evaluate the method on generative VQA models or standard open-ended VQA settings to establish whether the approach generalizes beyond candidate ranking?

  • What are the end-to-end adaptation time, per-sample latency, and peak memory consumption compared with existing TTA methods?

1 comment
AuthorsOpenReview ↗

Response to Reviewer V3rQ: Pseudo-Labels and Online Adaptation

(1) Incorrect pseudo-positives. Sections 3 and Appendix B state that positive alignment and the hard-negative hinge can strengthen an incorrect winner. Shared adapter parameters still couple samples, and consistency constrains sensitivity rather than correctness. Appendix E.5 separates endpoint accuracy from confidence. Table 27 reports CLIP EarthVQA accuracy 91.1% at epoch 10 and 89.8% at epoch 50, although the margin increases 0.139 to 0.146 and entropy decreases 0.53 to 0.46 nats. Increasing confidence can therefore accompany declining accuracy. The endpoint-compatible transition table has been removed. Initial and final accuracy alone do not identify how many errors were corrected or introduced.

(2) Offline versus online. Table 7 reports CLIP EarthVQA 91.1% offline versus 10.6% online. The online value is below the 25% uniform four-choice expectation; we do not describe it as strong streaming performance. C.1 distinguishes repeated offline exposure, batch 4, from one pass with batch 1 and one update per question. Table 26 separately reports CLIP 32.0%, 65.0%, 88.5% and 91.1% at fixed epochs 1, 2, 5 and 10, followed by 89.8% at 50. MetaCLIP2 changes from 14.2% at epoch 1 to 51.6% at 10; DFN-5B changes from 10.1% to 36.5%. This trajectory is distinct from the original checkpoint column. Update count, batch, order, checkpoint choice and prediction timing must be separated before assigning one cause to the offline/online gap. The executed original selection rule and before/after-update convention remain unspecified.

(3) Efficiency. Appendix C.6/Table 14 gives CLIP Base/Ours 0.113/0.116 s/sample under its update-stage timing boundary. E.8/Table 32 separately reports full-path Base/Ours 0.151/0.181 s/sample for CLIP and 0.293/0.351 for DFN-5B; Ours peak memory is 1890/5600 MiB. The adaptation full path includes loading, image/candidate encoding or cache access, scoring, backward/update, rescoring and device synchronization; Base omits the optimizer update. Its boundary differs from the update-stage figures. Table 32's 192.0/372.4 s are latency-times-question-count equivalents, not independently measured whole-dataset wall time. The precise cache and instrumentation configuration is needed to reproduce these timings.

(4) Generative scope. Section 3.2 and Appendix D.6 restrict the empirical claim to reference-containing candidate ranking. Table 30 evaluates retrieval followed by ranking separately; a missing reference imposes a recall ceiling. Extending to generative VQA requires candidate coverage, semantic-equivalence handling, sequence/length calibration and control of generator errors. The current ranking results do not validate unrestricted generation.

Reviewer_Sfv2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies prediction collapse when entropy-based test-time adaptation is applied to VQA over a shared answer vocabulary. It proposes adapting a lightweight text adapter within question-specific candidate sets, using pseudo-label alignment, a margin loss, and a consistency loss. Experiments cover six vision-language models and six VQA datasets.

Critical strengths
  • The comparison with entropy minimization is useful. Table 5 compares the two objectives under the same candidate-answer setting. The proposed objective performs better in all nine model dataset combinations, providing evidence for the value of the alignment loss.

  • The experiments cover a broad range of models and domains. The method updates only a small text adapter, making it a potentially practical approach. The comparisons of adaptation strategies and loss components also help explain its behavior.

Critical weaknesses
  • The unusually low baseline accuracies need a clearer explanation. Table 2 reports near-zero full-vocabulary accuracy but does not include results before adaptation. Without that comparison, it is difficult to determine whether entropy minimization causes the failure or whether the model already fails in this setting.

  • The EarthVQA results raise a related concern. All six models achieve only about 0.04–0.12% baseline accuracy on a four-choice task, compared with 25% for random guessing. CLIP then improves from 0.0588% to 91.1%. The paper acknowledges this anomaly, but the analysis does not establish whether the improvement requires visual understanding or mainly reflects patterns in the candidate answers, such as length and frequency.

  • Please report the full-vocabulary baseline before adaptation and add controls using shuffled images, the shortest candidate, and candidate-frequency heuristics. These would help determine what drives the gains.

  • The model-selection procedure is unclear. The main paper does not explain how hyperparameters and the reported checkpoint are selected without test labels. This matters because Figure 5 shows accuracy changing over training. Please state the selection rule, whether test accuracy influenced any choices, and whether all methods use comparable adaptation budgets. Reporting results at a fixed, predefined stopping point would make the comparison easier to assess.

  • Some overall accuracies appear inconsistent with the Open and Closed results. In Table 3, DFN-5B reports 69.3% Open, 55.5% Closed, and 61.9% Overall on SLAKE. Using the approximate Open/Closed proportions implied by the other rows gives an overall accuracy of about 63.9%. Similarly, the OVQA values imply about 68.6%, rather than the reported 66.5%. Please provide the subset sizes and reconcile these numbers.

  • The candidate construction limits the scope of the conclusions. Each open-answer candidate set contains the ground-truth answer and three sampled distractors. This is a valid controlled multiple-choice setting, but it assumes the correct answer is already available among the candidates. The paper should distinguish improvements within this setting from improvements in open-ended VQA and explain how candidates would be obtained at deployment. The description of the protocol as “standard” also needs a supporting reference.

  • The objective does not clearly prevent incorrect predictions from becoming more confident. Both the alignment and margin losses reinforce the model’s current highest-scoring candidate, even when that candidate is wrong. Candidate conditioning alone therefore does not establish that collapse or confirmation bias is avoided. The paper should explain this limitation and provide evidence that adaptation preserves meaningful differences between candidate representations.

  • The baseline and uncertainty reporting could be stronger. Since preventing collapse is the central motivation, a comparison with entropy minimization plus a prediction-diversity regularizer would be particularly informative. Baseline adaptation settings should also be reported clearly. Table 4 provides results across seeds for selected experiments, but the smaller gains would be easier to interpret with paired uncertainty estimates and an explanation of whether candidate sampling changes across runs.

Questions for authors
  • What is the no-adaptation full-vocabulary accuracy per Table 2 cell (and in what units), which answer does the collapsed predictor produce, why is Fig. 2(b) at 0.00 already at the first logged epoch if that answer is "yes" or "no" , and what are the per-answer prediction counts before adaptation, at epoch 1 and at the final epoch?
  • How can all six backbones score 0.04-0.12% on 4-way EarthVQA, and what is the ground-truth position distribution within C_i?
  • Were hyperparameters or the reported epoch chosen on test accuracy, and if not, by what label-free criterion?
  • Please reconcile DFN-5B SLAKE 61.9 (implied 63.6-64.1), OVQA 66.5 (implied 68.4-68.8), and Table 19 vs. Table 3 OVQA Closed (were closed sets changed per App. D.3 l. 1433-1434, or kept at {yes, no}?).
  • Given k-hat_i = c*, grad L_neg raises s_{i,c*}; can any non-trivial statement distinguish the objective from entropy minimization under shared parameters, including on closed questions?
  • Please state H, the L_cons temperature, whether alpha is trainable and why Table 18's alpha = 0 nearly equals no adaptation for CLIP/MetaCLIP2 but not for DFN-5B, the (q, c) template/truncation, how type-consistent negatives are built for yes/no EarthVQA questions and why Table 17 has one Base, and which TPT implementation (Shu et al. NeurIPS 2022 vs. arXiv "Efficient TPT") with what augmentations/steps.
1 comment
AuthorsOpenReview ↗

Response to Reviewer Sfv2: Evaluation and Reproducibility

(1) Initialization and collapse. Tables 2 and 21-22 separate pretrained full-vocabulary scoring from subsequent adaptation. For CLIP on VQA-RAD/SLAKE/OVQA, Table 21 reports initial accuracy 12.9%/10.4%/8.6%, epoch-1 0.0%/0.2%/0.4%, and final 0.0%/0.0%/0.05%. Final dominant-answer frequencies are 99.8%/99.5%/99.1%, with 2/3/4 unique predictions. Table 22 separately compares full fine-tuning, LoRA and text adapters across three models. A nonzero adapter residual is distinct from the pretrained reference. Saved traces contain index frequencies but no vocabulary-to-text map. Dominant-answer text requires that mapping; a first logged epoch is not automatically an epoch-zero observation.

(2) EarthVQA controls. Table 3 retains CLIP Base/Ours 0.0588%/91.1%. Table 23 reports 91.1% after candidate permutation with reference-index remapping, 32.4% with shuffled images, 38.2% with shuffled questions and 28.6% with mismatched images during adaptation/evaluation. Table 24 gives shortest/longest/frequency/question-only accuracy 30.6%/12.3%/33.8%/41.5%. Figure 6 supplies distributional diagnostics. Image-shuffle accuracy need not equal uniform guessing because question/candidate information remains. These controls support paired-visual dependence alongside textual priors, not exclusion of every mapping error. The extremely low Base still needs its per-run denominator or aggregation reconciled.

(3) Selection and budget. Appendix C.1 describes AdamW (learning rate/weight decay 1e-4), offline batch 4 with up to 50 passes, and online batch 1 with one update per sample. Tables 25-26 evaluate fixed epochs separately from the original Offline column. CLIP EarthVQA is 32.0%/65.0%/88.5%/91.1% at epochs 1/2/5/10. The executed original checkpoint rule, test-accuracy involvement and online before/after-update order are not documented; we cannot claim label-free selection for Table 3.

(4) Overall and hard candidates. DFN-5B SLAKE remains 69.3/55.5/61.9; OVQA remains 57.6/75.8/66.5. Appendix E.1 defines count-based accuracy, while Table 20 describes benchmark statistics rather than validated per-run subset counts. Effective subset/correct counts have not been recovered, so no empirical correction to Table 3 is claimed. Table 19 now compares only Open-answer hard candidates: CLIP Base is 20.6%/30.5%/26.1%, versus 46.3%/46.8%/49.5% for alignment on VQA-RAD/SLAKE/OVQA. Closed and Overall columns are excluded because the perturbation targets Open distractors. This narrows the comparison without claiming to explain the historical frozen-Closed discrepancy.

(5) Oracle candidates. Section 3.2 includes the reference during benchmark candidate construction; the loss does not receive its identity. The evaluated task is candidate ranking, not unrestricted answer generation.

(6) Incorrect pseudo-positives. Alignment and the hinge can reinforce an incorrect winner; consistency limits local sensitivity, not correctness. The endpoint-compatible decomposition has been removed because it cannot substitute for paired sample predictions. Table 27 reports margin 0.139 to 0.146 from epochs 10 to 50 while accuracy falls 91.1% to 89.8%; greater confidence need not mean greater accuracy.

(7) Stronger entropy. Table 29 reports CLIP candidate entropy 42.8%/33.9%/48.2%, entropy plus diversity 46.3%/43.8%/52.5%, versus alignment 53.7%/57.4%/59.6%. Table 28 separately treats full-vocabulary diversity. These are configuration comparisons; loss-only attribution requires matched candidates, adapter, optimizer, seeds, updates and selection rules. Diversity uses answer identities, not option positions.

(8) Reproducibility and uncertainty. Sections 3.3/C describe a two-layer ReLU adapter and trainable gate initially 0.5. Numerical H, temperature, prompt/truncation, seeds, executed LoRA rank/targets/scaling and TPT version are not documented sufficiently to supply exact settings. Table 31 reports paired 95% gain intervals [8.1, 15.1]/[12.7, 19.6]/[6.8, 11.9] pp under image-cluster resampling; its seed/iterations and paired outputs also need documentation. These settings are necessary to reproduce the comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.