acceptodds
Under review as a conference paper at ICLR 2027

Generating to Generalize: Socratic Questioning for Compositional Continual VQA

Abstract

Continual visual question answering (VQA) requires models to retain previously learned knowledge while reasoning compositionally across different tasks, yet existing generative continual learning methods primarily emphasize knowledge retention and overlook cross-task composition. We address this limitation with a Mixture-of-Experts framework comprising dual-purpose experts that learn visual question answering (VQA) and visual question generation (VQG) within a shared multimodal large language model backbone. Each expert is implemented via low-rank adaptation to preserve task-specific knowledge. In contrast, knowledge distilled from prior experts provides complementary views to new experts, enabling compositional generalization without storing past raw experiences. A task-agnostic router selects experts at inference time. To more rigorously evaluate compositional inference in continual VQA, we introduce a human-annotated benchmark comprising real-world compositional questions. Extensive experiments on both standard and our compositional VQA benchmarks demonstrate that our approach retains knowledge and achieves stronger compositional generalization than prior generative continual learning methods.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual VQA, VQA, MoEs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_e9HUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies compositional continual visual question answering. It keeps a frozen, task-specific pair of low-rank VQA and visual-question-generation experts for each sequential task. When learning a new task, earlier experts generate filtered question–answer turns on the current task's images, and the new expert learns from these additional views. At inference, a language-only router selects one expert. The paper also introduces a 500-example human-annotated compositional test set and a 41,981-question COCO-GQA split. On LLaVA-7B, final average performance on VQACL is 61.20%, versus 54.50% for the strongest listed continual baseline; accuracy on the human compositional set is 51.40%, versus 43.20% for GaB.

Critical strengths

S1. The paper identifies a worthwhile gap between retaining individual VQA skills and answering questions that require several learned skills. The additional human-annotated test set records intended task combinations, which gives the continual-learning setting a more direct compositional evaluation than per-task accuracy alone (Section 5). S2. The reported gains are substantial across the main evaluations. The LLaVA-7B model reaches 61.20% final average performance on VQACL, 64.40% on COCO-GQA, and 51.40% on the human compositional set (Table 1 and Figure 3); the BLIP-2 appendix also reports the highest average performance among its listed baselines (Table 8). The Socratic ablation improves COCO-GQA by 9.87 points and the human set by 6.00 points (Table 5 and Appendix Table 11), making the proposed supervision worth further study. S3. The method gives a concrete way to reuse prior experts without retaining raw training images. Freezing earlier adapters protects their task-specific parameters, while generated turns provide training-time cross-task views. The paper also tests a final-expert-only variant and a direct two-adapter merge (Appendix Tables 9–10), which helps clarify why it uses a routed expert pool at inference.

Critical weaknesses

W1. The task-agnostic routing evidence does not yet cover the paper's central compositional setting. The router is trained to assign one task label to each question, and Figure 3(c) evaluates that prediction on single-task VQACL questions. A compositional question has two or more intended task labels, but inference still selects only one expert. Although Appendix Table 10 shows that the full pool beats the final expert overall, it does not show which expert receives each composition type or whether that expert was trained after all required skills became available. Errors from routing and errors from the selected expert are therefore difficult to separate. Please analyze routing on multi-skill questions and performance by selected expert and task pair. W2. The ablations do not fully isolate Socratic questioning from more training. Table 5 removes generated turns entirely. Table 4 matches the number of question–answer pairs, but changes current versus past images and studies only the last task with a single expert. Its gains are modest (51.45 versus 49.61 on VQACL; 61.76 versus 60.80 on COCO-GQA), and the human set is not evaluated. A matched current-image control with equal turns and tokens but generic, shuffled, or single-expert questions would test whether complementary prior-task views cause the compositional gain. Earlier VQA experts' pseudo-answer accuracy also needs measurement. W3. The new benchmark is promising, but its validity for skill composition needs stronger evidence. It has 500 questions on 418 images across 100 composition types (Section 5), with uneven type frequencies in Figure 2. Annotators saw existing VQACL question–answer triples; low-confidence items were filtered using self-ratings and peer review of a random subset. The paper does not report independent verification that each accepted question truly requires every assigned skill, agreement on task labels, or answer accuracy after annotation. COCO-GQA adds scale, but its scene-graph questions are not labeled as compositions of VQACL task types (Appendix F–G). Per-type results, an independent audit, and uncertainty intervals would make the compositional conclusions more secure. W4. The storage comparison in Table 7 is not made at the same point in the task sequence. It contrasts a 770 MB replay buffer with about 233 MB of expert parameters per task; at ten tasks, the stated expert cost alone is about 2.33 GB, before the router and shared backbone. Thus, the table does not show a lower total storage footprint for the reported ten-task experiment. Moreover, each new task queries all earlier VQG and VQA experts on current images, while the paper gives no end-to-end training time, inference time, or peak-memory comparison. These totals are needed to assess the claimed practical overhead of the expandable design.

Questions for authors

Q1. For W1, how are multi-skill questions routed among single-label experts? Please report expert-selection distributions and accuracy by composition type and learning step, including cases where the selected expert was trained before one required skill was introduced. Q2. For W2, can the authors add equal-turn and equal-token current-image controls using generic or shuffled previous-task questions, and measure the factual correctness of generated pseudo-answers? Reporting these on both compositional benchmarks would identify what Socratic supervision contributes beyond extra training data. Q3. For W3–W4, can the authors independently audit a stratified sample for answer correctness and genuine multi-skill necessity, give per-type or paired-bootstrap uncertainty, and compare total adapter storage and end-to-end training/inference cost at ten tasks with the replay baselines?

1 comment

We thank the reviewer for taking the time to review our paper.


Q1. Routing of multi-skill questions.

We thank the reviewer for this suggestion. Our framework enables compositional reasoning through Socratic knowledge transfer during training, rather than combining multiple experts at inference. Each expert retains its primary/dominant task specialization while acquiring knowledge from previous tasks. Therefore, our router selects an expert based on the dominant component of a compositional question.

We analyze whether the router selects an expert corresponding to at least one required component task across learning steps.

Step2345678910Avg
Hit rate (%)100.088.8982.7884.5281.4082.0881.1177.1576.6083.84

The router achieves 83.84% average component-task alignment across learning steps. We further compare router-based selection against always selecting the latest required-task expert on the compositional questions.

Selection policyAccuracy
Latest required expert47.8%
Router-selected expert51.4%

The router improves compositional accuracy by 3.6 percentage points. This suggests that expert specialization remains relevant despite cross-task knowledge transfer, and selecting dominant / most important skilled expert improves performance. Socratic supervision enables knowledge sharing across experts without requiring identical compositional performance from every expert. Additionally, Tab. 11 provides evidence for Socratic supervision with improved performance on all three benchmarks (average improvement of +6.18 pp).


Q2. Controls beyond extra training data.

Our Replay vs. Socratic questioning (Tab. 4) uses additional replay images for the Replay method, while our method reuses the current task image (i.e matched QAs but additional images for the replay method). We thank the reviewer for the suggested comparison with matched training samples. Following the suggestion, we ran a matched current-image control on the last task. We kept all original last training data and added shuffled questions. So both models see identical questions, answers, image counts, and training settings. Socratic questioning achieved best performance on all ten tasks and both compositional benchmarks. We note that the suggested shuffling pairs QAs with the wrong images, so part of the drop may reflect label noise.

MethodVQACLCOCO-GQAComp.
Matched34.8042.7137.2
Ours51.4561.7642.4

Pseudo-answer correctness.

We checked 200 randomly sampled generated Q&A pairs. An author judged 81.5% of pseudo-answers correct, 15% wrong, and 3.5% ambiguous(unable to answer the question for the corresponding image). We additionally judged the same with pairs using GPT-5.5 as a judge. GPT-5.5 judged 85.5% correct. Pseudo-labels are therefore mostly correct, with little generation noise.


Q3. Audit, uncertainty, and cost.

Audit. An annotator with an independent reviewer audited a stratified sample of 100 questions; reference-answer agreement was 86.0%, and task-label (at least two) agreement was 77.0%.

Uncertainty. A paired bootstrap over questions (10,000 resamples) gives our gain over GaB on COCO-GQA as +5.42 pp (64.40 vs. 58.98; 95% CI [4.97, 5.86]; positive in every resample).

Storage. Our expert pool grows linearly at about 233 MB per task, or 2.33 GB at ten tasks, which exceeds a 770 MB replay buffer. The advantage of our design is that we store no raw past-task images.

Inference cost.

On 100 test questions (10 per task, ~2.5 answer tokens on average), end-to-end inference latency matches a single expert:

MethodMean latencyMedianRelative
Ours244.7 ms225.5 ms1.00×
Single expert240.5 ms220.2 ms0.98×
CL-MoE756.1 ms678.5 ms3.09×

Training cost.

End-to-end training time over all ten tasks on three A100 40GB GPUs, including Socratic generation by past experts for our method:

MethodTraining timeRelative
Replay~18 h0.51×
Ours~35 h1.00×
CL-MoE~42 h1.20×

Socratic generation makes training about 1.9× slower than replay, though still faster than CL-MoE. The overhead stays manageable because experts are independent and share the backbone and current-task images. This is a trade-off: the extra cost is paid once during training, while inference—incurred on every query—matches single-model latency and is about 3× faster than CL-MoE. Additionally, our method improved performance (an average improvement of +6.77 pp compared to the second-best baseline) on all benchmarks (VQACL, COCO-GQA, Compositional VQA).


We thank the reviewer for the review, which has strengthened our paper. We will incorporate these analyses and address any remaining minor comments in the revised version.

Reviewer_6SQNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper tackles the problem of continual learning for VLMs in the context of Visual Question Answering (VQA) by adding a generative (VQG) module to re-use former concepts. At each new task, a new expert comprised of a VQA and a VQG module is added to the adapter of the VLM (Q-Former in the case of BLIP). The new VQG is first trained to generate plausible questions on an image given the answers of questions for this image. Then, the VQG module of each former expert is used to generate questions on the images of the current task, and its VQA answers it. These new question-answer pairs are filtered, and used to augment the training data of the current task. Using this so-called Socratic questioning procedure, the authors claim that the model can deal with composed concepts (e.g. going from "How many cars are there?" to "How many blue car are there?") better than other approaches. It also has a router, which predicts the nature of the task in the question and passes it to the appropriate expert. The authors claim that this exemplar-free approach can substantially improve results, faring 7 points above the second-best approach on VQACL. It is done at the expense of extra parameters, since the model adds an expert at each task, but the authors stress that it is reasonably small for their sequence of ten tasks. They also propose a new compositional VQA dataset (called Compositional VQA) with 500 images paired with several question-answer pairs, which combine two or more tasks (e.g. location and recognition).

Critical strengths
  • Study of a meaningful problem: MLLMs have trouble combining concepts they learn continually
  • Clearly written and structured
  • Assessment of the generated questions: good point and sound procedure
  • Promising results
Critical weaknesses
  • Disconnect between claims and methodology: The paper heavily emphasizes composition, yet lacks a specific mechanism to enforce or measure it. As acknowledged in lines 77-78, the method functions essentially as a standard Continual Learning (CL) approach that achieves better results via specialized experts; its compositional improvements appear incidental rather than by design.
  • Clarity on expert architecture: The paper would greatly benefit from an early, explicit description of the experts (similar to the information currently relegated to Figure 5). It is not immediately clear what these experts are, how they are constructed, or that exactly one expert is assigned per task.
  • Dataset scale: The proposed compositional evaluation set is severely limited, containing only 500 images (a subset from VQACL). This scale is insufficient for robust evaluation, especially when divided across 10 distinct tasks.
  • Scalability and CL validity: Categorizing this as a standard CL method is highly debatable, as the parameter budget increases linearly with the number of tasks at a non-negligible cost (~233 MB per task). Compared to a standard 770 MB replay buffer, the proposed method only offers a memory advantage for three tasks or fewer. For 10 tasks, the footprint expands to approximately 2.3 GB. Furthermore, training cost analysis is entirely missing, which is critical given that every earlier expert must generate questions for each new task.
  • Questionable contribution of the MoE architecture: The experimental results cast doubt on whether the Mixture of Experts (MoE) routing is the true source of the performance gains. The final expert alone scores 56.79, outperforming all baselines (GaB 54.50, QUAD 53.40, CL-MoE 51.34) without needing the router. This single expert also surpasses baselines on both compositional benchmarks (62.54 vs. 58.98 for GaB on COCO-GQA, and 47.60 vs. 43.20 on compositional VQA). This indicates that the vast majority of the improvement actually stems from question–answer pairs generated by earlier experts (effectively acting as generative replay). Because the router and frozen pool yield only minor marginal gains, the core argument supporting the MoE design is undermined.
  • Benchmark difficulty and trivial routing: The router achieving near 100% accuracy raises concerns about the actual difficulty of the benchmark. Assuming a near-perfect router that cleanly delegates to task-specific models, the CL problem effectively collapses; the challenge is reduced to simply fine-tuning on the current task without having to actively mitigate catastrophic forgetting.
  • Limited novelty: Generating questions from past tasks is an established technique already utilized in GaB (using the same network) and QUAD (using standard replay). The primary distinction in this work is the reliance on a separate LLM for generation, which limits the overall methodological novelty.
Questions for authors

Questions:

  • The loss LVQAL_{VQA} depends on ϕVQG\phi_{VQG} via sampling, but the gradient is most likely stopped. Therefore, the formulation of Equation (4) is confusing. Please clarify this point.
  • Why not include experiments on NExT-QA, as done in concurrent works?
  • Did the authors try applying the Socratic questioning approach to a setting with a single VQA expert? I understand that multiple VQA experts are used to sequentially produce answers and that modifying this would break the proposed approach; however, it would still be helpful to understand if the VQA experts provide better answers on new, unseen question-image pairs compared to a single expert trained on many questions.
  • Conversely, did you try removing the Socratic questioning (i.e., no questioning) and evaluating the resulting experts?
  • As far as I understand, the VQG generates likely questions for a given expert and image. These likely have a high chance of being single-task questions, since each expert was trained on a single task. Therefore, it is unclear how this process leads to better compositional results. Did you measure the share of questions generated by former experts' VQGs that were actually compositional? If this share is low, how do you explain the improved compositional capabilities?
  • To the best of my understanding, the router feeds the question to a single expert at test time. Is this not somewhat contrary to the spirit of composition that the authors claim to foster? Why not use a dynamic router that can call a combination of experts and aggregate their results?
  • Benchmarking: Are the reported baseline results taken directly from their respective papers, or are they from your own re-implementations? For example, the QUAD numbers do not match QUAD's own LLaVA-7B results (53.40 reported here vs. 44.16 in their paper).
  • Equation (4): Why formulate this as a single loss given that the training procedure is alternating, and the VQA and VQG modules do not share parameters (meaning they can be trained separately)? What is the purpose of the balancing coefficients?
  • How did the authors measure the performance of the baseline approaches? Did you re-implement them, reuse the authors' code, or use their reported results directly?
  • Did the authors measure zero-shot results prior to any training?

Suggestions:

  • The idea of specialized experts and completely isolating parameters was already proposed in Expert Gate (Aljundi et al., CVPR 2017); perhaps it would be relevant to cite it.
  • Text following Equation (3), line 248: The definition of Δθ(ΦVQGt)\Delta \theta(\Phi_{VQG}^t) is only provided in the appendix, which caused confusion during the initial read.
  • MoE architectures typically operate at the block level, meaning the system's output is a mixture of multiple experts. In the proposed method, it seems only a single task expert is selected and used, though this is not entirely clear. Please clarify this point.
  • Clarity: An early, explicit description of the experts (essentially the information in Figure 5) would be very helpful. It is not immediately clear early in the text what these experts are, how they are constructed, and that there is exactly one expert per task.
3 comments

We thank the reviewer. Our contribution is Socratic cross-task supervision, in which frozen past experts provide complementary views on current-task images, along with a compositional benchmark across multiple linguistic tasks.


Q1/Q8 (Eq. 4).

Past experts generate QA pairs offline, with no gradient through decoding. Current VQG and VQA use disjoint LoRA parameters: Eq. (4) is separable, optimized alternately with λVQG=λVQA=1\lambda_{VQG}=\lambda_{VQA}=1. We will clarify that the weights do not balance shared gradients. Eq. (4) was intended to state this unit's per-task objective compactly.


Q2 (NExT-QA).

To our knowledge, the continual NExT-QA benchmark has been evaluated only with VL-T5-based architectures using pre-extracted video features. Its protocol is not directly compatible with an image-pretrained MLLM (i.e., LLaVA), since it would require a new video-feature adapter. We ported our approach to VL-T5 under QUAD's pre-extracted-video-feature protocol. Using QUAD's reported baselines:

MethodAP ↑Forget ↓
VQACL30.864.12
QUAD31.702.91
Ours (VL-T5)32.390.11

Q3 (single expert / MoE contribution).

Appendix Table 10 evaluates the final Socratic-trained expert without routing at inference: 56.79 AP/62.54 COCO-GQA/47.60 compositional VQA, versus 61.20/64.40/51.40 with the full pool. Thus, the pool adds +4.41 AP beyond a strong final expert. That final expert still required past experts for training. A single VQA expert depends on its own generations; both generation noise and forgetting compound over the task sequence. A single VQA expert with Socratic VQG experts performs 44.60 v.s 51.40 for the full pull on compositional VQA.


Q4/Q5 (does Socratic questioning cause composition?).

Generated questions are single-task by design. Our proposed mechanism is cross-task grounding: prior skills (e.g., color) are applied to current-task images (e.g., counting), creating complementary supervision in the same visual context. Images queried by ≥2 earlier tasks rise from 4.90% to 36.83%; this is multi-task coverage, not a compositional-question rate. Two controlled comparisons support this: (i) with the same expert pool and router, removing Socratic QA reduces VQACL AP 61.20→58.54, COCO-GQA 64.40→54.53, and human compositional VQA 51.40→45.40 (Table 11); (ii) with equal QA counts, Socratic supervision on current images exceeds replay on past images: VQACL 51.45 vs. 49.61, COCO-GQA 61.76 vs. 60.80 (Table 4). We will explicitly distinguish compositional generalization from enforced composition.


Q6 (single-expert routing).

Our framework places compositional reasoning at training time rather than at inference. Through Socratic training, each expert specializes in its own task (dominant task) while also learning earlier tasks' skills on its images, so a single selected expert already carries cross-task knowledge. Combining the top two experts at inference does not help:

MethodRec.Loc.Jud.Com.Cou.Act.Col.Typ.Sub.Cau.AP
Top-2 merge52.1138.4677.8871.0747.0672.5871.4857.9754.3623.9656.69
Ours56.6644.0281.3774.4953.1575.5574.7961.8559.6630.4161.20

Q7/Q9 (baselines).

All baselines except GaB and QUAD use CL-MoE's LLaVA-7B results; we ported GaB from the released code.

QUAD's submitted 53.40 was an approximate reimplementation because QUAD's cross-attention distillation lacks an exact decoder-only LLaVA analog, and the QUAD paper does not provide LLaVA implementation details (also released code based on VL-T5). QUAD results remain lower than our reported results. GaB remains second at 54.50, versus ours 61.20. We will clarify result provenance.


Q10 (zero-shot).

Yes, we measure zero-shot results. Pretrained LLaVA-7B struggles in VQACL metrics as it produces long sentences. Using the contains metric, it scores 37.10% (10-task mean) and 36.00% (compositional).


Due to space limitations, we continue our rebuttal in the comment below.

Other concerns.


(i) Scale/cost:

Storage. Our expert pool grows linearly at about 233 MB per task, or 2.33 GB at ten tasks, which exceeds a 770 MB replay buffer. The advantage of our design is that we store no raw past-task images.

Inference cost.

On 100 test questions (10 per task, ~2.5 answer tokens on average), end-to-end inference latency matches a single expert:

MethodMean latencyMedianRelative
Ours244.7 ms225.5 ms1.00×
Single expert240.5 ms220.2 ms0.98×
CL-MoE756.1 ms678.5 ms3.09×

Training cost.

End-to-end training time over all ten tasks on three A100 40GB GPUs, including Socratic generation by past experts for our method:

MethodTraining timeRelative
Replay~18 h0.51×
Ours~35 h1.00×
CL-MoE~42 h1.20×

Socratic generation makes training about 1.9× slower than replay, though still faster than CL-MoE. The overhead stays manageable because experts are independent and share the backbone and current-task images. This is a trade-off: the extra cost is paid once during training, while inference—incurred on every query—matches single-model latency and is about 3× faster than CL-MoE. Additionally, our method improved performance (an average improvement of +6.77 pp compared to the second-best baseline) on all benchmarks (VQACL, COCO-GQA, Compositional VQA).


(ii) Benchmark:

The human set has 500 questions on 418 images across 100 composition types; we also evaluate on 41,981 COCO-GQA questions, with consistent gains, and we will acknowledge the small-set limitation.


(iii) Routing difficulty:

Near-perfect task-ID prediction reflects VQACL's linguistically separable tasks. The no-Socratic ablation retains the same router yet loses 9.87 points on COCO-GQA.


We will address the other minor issues in the revised version of our manuscript.

Reviewer_6SQNOpenReview ↗

post rebuttal

I want to thank the authors for their very detailed response and the additional experiments. The answers address several of my questions. However, my two main concerns remain:

  1. Source of the gains on VQACL. Table 11 shows that the experts and the router and no Socratic questioning already reach 58.54 AP. That is above every baseline (eg GaB 54.50), and Socratic questioning adds very little. So most of the gain over the SOTA comes from parameter isolation combined with near-perfect routing. In that regime, the problem is close to training one model per task, not continual learning. The rebuttal answers with results on COCO-GQA but does not address the VQACL side. The headline "+7 points" is not really fair and should acknowledge what comes from the experts and the router.

  2. Composition is claimed but not measured. In the rebuttal, the authors confirm that the generated questions are single-task. They report multi-task coverage (36.83%) instead of the share of compositional questions. So the evidence for composition remains indirect.

I may have missed it but it seems that no revised PDF has been uploaded. So I cannot check the promised clarifications. At this stage, I keep my score.

Reviewer_HHoAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a dual-purpose Mixture-of-Experts (MoE) framework for continual visual question answering that jointly addresses catastrophic forgetting and compositional reasoning: it isolates task-specific knowledge via low-rank adapter experts (each with paired VQA and visual question generation branches), transfers and fuses knowledge across tasks through a "Socratic" inter-expert questioning mechanism that generates multi-view supervision on current-task images without storing raw exemplars, and enables task-agnostic inference with a lightweight language-based router. The authors also introduce the first human-annotated benchmark for task compositionality in continual VQA, and show on VQACL, COCO-GQA, and their new benchmark that the method outperforms prior continual VQA approaches in both knowledge retention and compositional generalization.

Critical strengths
  1. Meaningful research problem: The paper addresses an important and practically relevant challenge in continual VQA.
  2. Coherent methodological design: The proposed components are logically motivated and work together within a consistent framework.
  3. Practically valuable benchmark: The construction of a benchmark for evaluating cross-task knowledge composition provides a useful resource for future research.
  4. Relatively comprehensive experiments: The experimental evaluation covers multiple aspects of the proposed method.
Critical weaknesses
  1. Insufficient comparison with MoE baselines: the comparison with existing MoE methods is relatively limited.
  2. Lack of direct evidence for the source of the gains from Socratic supervision.
  3. Tension between the linear growth of the expert pool and the "scalability" claim.
  4. Routing has not yet been validated in compositional scenarios.
Questions for authors
  1. MoE baseline comparisons: The comparison with existing MoE methods is relatively limited. Could you include more representative MoE methods as baselines to more comprehensively validate the effectiveness of your proposed method and its relative advantages?
  2. Source of gains from Socratic supervision: Tab. 5 only shows that Socratic supervision is effective overall, but it is not sufficient to establish that the gains stem from the "composition" mechanism rather than from stronger regularization or additional supervision signals. Could you add a controlled experiment using an equal amount of non-expert-generated supervision as a comparison, to isolate and identify the specific source of the gains?
  3. Linear growth of the expert pool vs. the "scalability" claim: Assigning one expert per task causes storage overhead to grow linearly with the number of tasks. Could you add an analysis of how routing accuracy, storage footprint, and inference latency change as the number of tasks increases to 20/50, in order to support the scalability claim?
  4. Routing in compositional scenarios: The near-perfect routing accuracy in Fig. 3(c) is obtained on the in-distribution VQACL test set, which may not represent routing quality on compositional questions. Could you report routing accuracy and the expert-selection distribution on the compositional benchmarks, to verify that task-agnostic inference is indeed effective in the compositional setting that is the central focus of this paper?
3 comments

We thank the reviewer for recognizing the importance of the problem, the coherence of our design, the value of our compositional benchmark, and the breadth of our experiments. We address each question below.


Q1. MoE baseline comparisons

We thank the reviewer for suggesting additional MoE baselines. We address the concern by adding SMoLoRA (ICCV 2025) to our comparison. We use the official SMoLoRA released code and use the same settings as in our paper. All baselines use the same LLaVA backbone. Our Socratic-style complementary views and expandable design ensure gains in both unique-task and compositional performance compared with the MoE baselines.

MethodVQACLCOCO-GQACompositional VQA
SMoLoRA57.6361.7944.40
CL-MoE51.3453.9239.60
Ours61.2064.4051.40

Q2. Source of gains from Socratic supervision

We agree that Tab. 5 only shows that Socratic supervision is effective overall. We design a controlled experiment in our "Replay vs. Socratic questioning" (see Tab. 4). Here, the Replay baseline uses additional replay images, while our method reuses the current task image (i.e matched QAs but additional images for the replay method). This suggests that the Socratic supervision improves overall performance (49.61 vs 51.45), not stronger regularization or additional supervision signals. To address the reviewer's concern, we use a matched setting where both baselines see identical questions, answers, image counts, and training settings. Socratic questioning achieved best performance on all ten tasks and both compositional benchmarks.

MethodVQACLCOCO-GQAComp.
Matched34.8042.7137.2
Ours51.4561.7642.4

Q3. Linear growth of the expert pool vs. the "scalability" claim

We agree that the expert pool grows linearly; however, the storage footprint is manageable at ∼233 MB per task, or ∼2.33 GB after ten tasks. The advantage of our design is that we do not store any raw past-task images. Additionally, our design enables compositional generalization, which improves performance on unseen task compositions. So compositional generalization itself restricts overall linear task growth requirements through generalization.

Figure 3C shows routing accuracy across learning steps. To test with increased task size, we increase the last layer of the router; also, since VQACL defines ten tasks, we cannot measure accuracy, so we only show the latency results up to 100 tasks:

  • Router: 0.56 / 0.52 / 0.52 / 0.53 / 0.53 ms with 1 / 5 / 10 / 50 / 100 tasks, so flat.

We also calculate overall inference latency. All baselines answered the same 100 test questions (10 per task), and average answer lengths were the same (about 2.5 tokens). The table below shows the overall latency comparison:

MethodMean latency ↓Median ↓vs Ours
Ours244.7 ms225.5 ms1.0×
CL-MoE756.1 ms678.5 ms3.1× slower
SMoLoRA571.2 ms516 ms2.3× slower

Q4. Routing in compositional scenarios

To address the reviewer's concern, we analyze whether the router selects an expert corresponding to at least one required compositional task across learning steps.

Step2345678910Avg
Hit rate (%)100.088.8982.7884.5281.4082.0881.1177.1576.6083.84

The results suggest that task-agnostic inference remains efficient in the compositional setting, with compositional performance of 51.40 vs. the second-best baseline, GaB, at 43.20.


We have addressed each of the reviewer's concerns above and will revise the paper accordingly. We thank the reviewer again for the constructive comments, which have helped improve our work.

Thank you for adding SMoLoRA and the routing analysis. My main concern remains that shuffling image–QA pairs introduces label noise, making it difficult to isolate the benefit of Socratic supervision. Re-evaluating Continual VQA (CVPR 2026) also shows that shared answer vocabularies can inflate apparent retention under the standard VQACL protocol. The additional compositional results are useful, and robustness under a debiased continual VQA protocol such as UCo-VQA remains open. I am keeping my rating.

We thank the reviewer for replying and taking the time to read our rebuttal.

R1. Label noise in the shuffle control.

We agree that shuffling confounds grounding with label noise; we noted this in our response to (this) reviewer Q2. To address the concern, we add a control in which the same LLaVA pretrained backbone used throughout our experiments, without any past experts, generates the same number of Q&A pairs on the same current-task images. All pairs are correctly grounded, so the only difference from our method is whether the questions and answers carry past experts' task knowledge. We use the same pretrained backbone, rather than instruction-tuned LLaVA or a stronger external model, because those were trained on VQA v2 or would inject outside knowledge, making the comparison unfair.

The pretrained backbone produces long, sentence-style answers, and the expert trained on them inherits this format, so exact-match VQA accuracy collapses for reasons unrelated to content. We therefore also report a relaxed metric, contains, that counts an answer as correct if it contains the ground truth. This metric favors long answers, so it is conservative toward our method.

Supervision (exact / contains)VQACL.COCO-GQAComp.
Zero-shot pretrained-backbone Q&As3.89 / 40.080.78 / 49.941.20 / 37.40
Socratic expert Q&As (ours)51.45 / 52.3161.76 / 61.8942.40 / 43.00

Even under the relaxed metric, Socratic supervision outperforms correctly grounded zero-shot supervision by 12.2 points on the 10-task average, 12.0 on COCO-GQA, and 5.6 on the compositional set. Together with Table 4 in the paper (equal Q&As, current vs. past images) and the shuffle control (Rebuttal Q2), this indicates that the gain comes from past experts' task knowledge rather than from additional supervision.


R2. Shared answer vocabularies.

We thank the reviewer for pointing to this concurrent work, Re-evaluating Continual VQA (CVPR 2026). Given the time constraints of the rebuttal period, we will discuss this protocol in the revision and leave evaluation under its disjoint-answer splits to future work.


We thank the reviewer again for their constructive feedback.

Comments on the paper

1 comment
Area_Chair_frEgOpenReview ↗

Urgent Reminder: Reviewer Discussion Deadline

Dear Reviewers,

This is a friendly but urgent reminder that the discussion period will end on November 18, 2026.

Please carefully review the authors' rebuttals and actively participate in the ongoing discussions. If you have not yet responded to the authors or initiated a discussion, please do so as soon as possible.

In particular, reviewers are expected to address the authors' clarifications and responses, provide constructive follow-up feedback, and update their assessments and scores when appropriate.

Please ensure that your final evaluations reflect your independent judgment and take into account the information provided during the rebuttal and discussion process.

Active and timely participation is essential to ensuring a fair, thorough, and constructive review process.

Thank you for your time, effort, and cooperation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.