- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper targets test-time domain adaptation (TTDA) for cross-domain visual question answering (VQA). Existing entropy-minimization based TTDA approaches suffer from structural collapse on VQA tasks: models converge toward dominant frequent answers and lose question-conditioned reasoning capability. The authors trace this failure to optimization over a globally shared answer vocabulary, while VQA naturally features instance-specific candidate answer sets per input sample. They propose a candidate-conditioned alignment framework, which performs local energy-based image-text alignment loss inside each per-sample candidate set. Only a lightweight text adapter is updated, keeping the pre-trained backbone frozen. Extensive experiments are conducted across six vision-language models and six cross-domain VQA benchmarks. Comparisons against general TTDA baselines and specialized VQA-TTDA methods, together with ablation and diagnostic studies, are provided to validate the proposal.
Critical strengths
Strength 1: This paper systematically dissects the internal mechanism of structural collapse when entropy‑minimization based TTA is applied to VQA. Rather than only empirically reproducing this failure case, the authors provide formal propositions to explain why constant predictors become stable minimizers under shared‑parameter entropy optimization.
Strength 2: The evaluations include comparisons against both general TTA baselines and existing VQA‑specific TTA methods. Results are reported for the offline batch setting as well as the more difficult online single‑step scenario. Ablation studies are also carried out to verify loss components, adapter design options, and hyper‑parameter sensitivity.
Critical weaknesses
Weakness 1: One central prerequisite for this pipeline is access to instance‑level candidate sets during inference. All experiments in the paper rely on oracle candidate sets provided by benchmarks. This strong assumption, however, is not sufficiently emphasized within the main text. In real‑world open‑world VQA, pre‑built candidate sets are not readily available. External retrieval or generative components would be required instead, inevitably introducing additional approximation error. This represents a notable practical deployment constraint and considerably limits the approach’s applicable scope.
Weakness 2: CLIP exhibits an unusually large accuracy jump from 0.0588% up to 91.1% on EarthVQA. While diagnostic plots in both the main paper and appendix attribute this sharp gain to structural mismatch between pre‑trained similarity geometry and the dataset’s answer space, controlled cross‑dataset experiments are still absent. We cannot cleanly disentangle how much improvement stems from the alignment objective itself versus fixing severe dataset‑specific pre‑training bias unique to EarthVQA. This outlying result raises legitimate concerns about the generalizability of the reported performance gains.
Questions for authors
Q1 (Sec 3.3, Appendix B): You only update the gated adapter placed on the text branch. Drawing on the theory laid out in Appendix B, could you explain why image‑branch updates yield such limited gains? Is the instance‑wise gradient‑decoupling property simply invalid for the image encoder, or are there alternative feature‑level explanations?
Q2 (Sec 4.7, Table 7, Appendix C.5): Under online single‑step adaptation, your approach suffers noticeable performance drops and its advantages shrink compared to the offline setting. The appendix documents this behaviour but offers little root‑cause analysis. Is this performance degradation mainly caused by noisy pseudo‑positive labels, insufficient update steps per sample, or limited optimization room within each candidate set?
Q3 (Sec 3.2, Appendix D.2): All of your experiments rely on oracle candidate sets which guarantee that ground‑truth answers are included. How would your framework behave if the ground‑truth answer is not retrieved during candidate‑set construction at test time?
Q4 (Sec 3.2, Appendix C.4 Table 12): Your main experiments fix K=4. Larger candidate‑set K hurts prediction accuracy while also increasing computational cost. Could you provide empirical or theoretical guidance on picking an appropriate K to balance accuracy and compute cost for real‑world deployment?
Q5 (Sec 4.8, Table 8, Appendix D.5): Your gated residual adapter outperforms LoRA. While the appendix describes its architecture, comparisons on gradient update dynamics are missing. Could you clarify the key differences in gradient update behaviour between your gated residual adapter and LoRA under test‑time adaptation, to help explain the observed performance gap?
Q6 (Conclusion, Appendix D.6): Appendix D.6 mentions extending towards fully open‑ended generative VQA as future work. Could you briefly outline what you see as the most central technical obstacles when porting the core candidate‑conditioned alignment idea over to generative VQA systems such as LLaVA‑style models?
Response to Reviewer fGd1: Candidate Sets and Adapter Design
(1) Text versus image adaptation. Table 8 reports CLIP image-adapter accuracy 44.8%/42.7%/53.6%, versus text-adapter 53.7%/57.4%/59.6% on VQA-RAD/SLAKE/OVQA. A text residual changes candidate representations while retaining visual embeddings; an image adapter changes the visual representation used in the comparison. The accuracy preference concerns these evaluated configurations. Text Adapter/LoRA drift is not used as evidence for an image/text gradient mechanism.
(2) Single-pass degradation. Table 7 reports CLIP EarthVQA 91.1% offline and 10.6% online, the latter below the 25% uniform four-choice expectation. C.1 describes offline batch 4 with repeated exposure and online batch 1 without replay. Table 26 separates fixed budgets: CLIP reaches 32.0%/65.0%/88.5%/91.1% at epochs 1/2/5/10. More offline exposure accompanies larger gains, but update count is not isolated from batch, order, selection and prediction timing. The executed original checkpoint rule and online update/predict order still require documentation.
(3) Missing reference. Section 3.2 makes oracle inclusion explicit. Without it, a selector cannot output an absent answer. In Appendix E.7/Table 30, question retrieval at K=4/8/16 has recall 42.0%/56.0%/68.0% and conditional ranking accuracy 61.0%/56.0%/45.0%, giving end-to-end 25.6%/31.4%/30.6%. For percentage inputs, end-to-end accuracy equals recall times conditional accuracy divided by 100 and cannot exceed recall. Question-type filtering is reported separately; the exact retriever, answer pool and index are not specified sufficiently for reproduction.
(4) Choosing K. Table 12 evaluates K=4/8/16 for Open questions while Closed candidates remain binary. CLIP SLAKE accuracy decreases 57.4%/47.8%/40.2%; OVQA decreases 59.6%/53.1%/51.8% as distractors increase. Table 14 gives CLIP Ours latency 0.116 to 0.174 s/sample and DFN-5B 0.258 to 0.354 under that timing protocol. With non-oracle retrieval, coverage improves with K, but Table 30's best listed question-retrieval end-to-end score is at K=8. Ranking difficulty, coverage and cost therefore jointly determine K; a larger set is not uniformly better.
(5) Gated Adapter versus text LoRA. This comparison is separate from branch selection. Table 8 gives CLIP text-LoRA 44.4%/52.0%/55.1%. The gated adapter modifies frozen encoder-output features; LoRA updates selected internal linear maps through a low-rank parameterization. E.9/Table 33 reports SLAKE accuracy 52.0% versus 57.4%, feature drift 0.051 versus 0.025, and final gate 0.55. This is a structural and empirical comparison, not proof that smaller drift causes better optimization. Executed LoRA rank, targets and scaling still need to be specified.
(6) Generative extension. Appendix D.6 identifies candidate-generation coverage, answer equivalence, sequence/length calibration and propagation of generator errors as the main obstacles. Retrieval-then-ranking is a distinct evaluation from oracle ranking, and neither establishes unrestricted generative VQA performance.